Evaluation of an emergency online course

Posted: May 28, 2020 | Author: Azza Abouzied | Filed under: Uncategorized |3 Comments

I had ambitious plans for the two months of emergency online teaching including sharing my weekly lesson plans and lessons learned. I haven’t delivered a weekly report as promised but I did offer occasional updates. So, I hope in this wrap-up post to first share some the main lessons I learned. As many of us have to teach courses online again this fall, some of these lessons may help others better plan for round two! I have already received my course evaluations so I’ll emphasize the methods that the students appreciated. Second, I would like to put forward a plan of action moving forward. Many of my colleagues described the two-month ordeal as challenging and complicated. While I would use similar words to describe parts of the course, the word that comes to my mind about this experience is illuminating. I teach again in the spring and I hope by then the world returns to some semblance of normality with in-person classes. My plan moving forward assumes in-person classes are back but hopes to integrate many of the positive aspects of online-learning that I have discovered.

Lessons Learned

The hybrid model works.

I made a drastic decision to switch my teaching model in the one week preparation time to a hybrid one: asynchronous lectures + one live session a week. If you have prepared lecture slides and materials from teaching a course many times before, you understand how this decision might seem like an additional and unnecessary overhead. I cannot stress how useful it was to learning. Putting my self in my students’ shoes, I understand how after 12-15 minutes of watching an online seminar, I am easily distracted and I am onto emails, newsfeeds, or other more attention-holding tasks. An online Zoom lecture isn’t much different. You can introduce more breaks and participatory exercises to break the monotony but once you lose your students, it is very hard to bring them back in. In the hybrid model, I would occasionally dive into a specific concept in detail and within 10 minutes, I can see the videos turning off. That said, I was lucky in that I found excellent online videos by Prof. Joe Hellerstein at UC Berkeley and Prof. Andy Pavlo at CMU that covered roughly the same material I had in my syllabus. If you have the summer to prepare, it might be worthwhile to record your own lectures. You have to keep each video segment short. On average the segments were 5 minutes long, with 10 minutes being the longest video I shared.

“She also picked and provided video lessons each week that were easy to watch and follow along to, and when the video was long, she picked out the relevant segments which made it less intimidating, since I didn’t have to watch a bunch of 1 hour lecture videos.”

Course evaluation, anon student.

Pre-recorded lectures are not enough and you have to back them with a live session once a week and extended office hours. The live session helped me work with the students on exploring real-world problems from different perspectives and to dig into more advanced material. A lecture format didn’t quite work here and I found the students most engaged when I introduced a problem and then broke them up into groups of 4-6 and asked each group to come up a with solution based on the material covered in the previous week’s module. For example, after the introductory module on transactions, I asked each student group to pick an application they are familiar with and discuss what sorts of anomalies can occur from multiple, interleaving transactions and whether we need a serializable transactional system to support the application. The discussion in each group was lively and instructional.

If you choose to cancel one of your weekly classes like I did, I recommend replacing it with office hours. As students take on independent learning, they may still struggle with certain concepts and a one-on-one can really help students overcome mental blocks. For office hours, I set up the zoom room and waited. It was a casual set-up. I had my camera turned off and microphone muted most of the time. I didn’t expect students to turn on their cameras for office hours either. Some students joined without any questions and used this time to individually work on the weekly lesson with me around and to hear any Q&A that came about. It was a laid back, semi-supervised, learning session.

Structure and organization are key.

In earlier posts, I attached some samples of the structure of my weekly lesson plan. As the class continued, I found myself adding more text and notes to each week’s module and distinguishing core concepts from optional/advanced/good-to-know ones. My advice here is to stick to a consistent plan and do not deviate unless there are exceptional circumstances. Weekly assessments for each lesson were due on Saturday, and the following Monday “Patch-the-Gap” lecture focused on any gaps uncovered from grading the assessments. Of the 22 students who evaluated my class, 8 students answered “organization” and “structure” to the question “what aspects of the course were most valuable to you?” I made sure to communicate any changes in the course plan by email and in the live session. I discussed how the different class deliverables connected to learning objectives and how any changes in these deliverables did not impact their learning objectives. I was open about the advantages and limitations of online learning and used structure to ensure that no student was left behind. Having a rough week shouldn’t derail a student completely: keeping each lesson plan as self-contained as possible and allowing students to revisit past material was how I achieved this.

Weekly assessments, labs and problem sets are crucial, …

… not midterms and final exams: I say this knowing that I will alienate some of my colleagues. Personally, I enjoy putting together a challenging midterm. For many of us, midterms and exams are powerful tools that make us dust-off our textbooks, read, revisit slides and learn. Understandably, giving up exams does not come easy. I have worked through several reasons for not giving them up myself and here is what I found opting for online weekly assessments in lieu of midterms/exams:

• Myth 1 – Students will share answers and not bother going through the material: I didn’t find this to be the case. Open-ended questions revealed a diversity of student understanding. Even for true/false and multiple-choice questions, there were was a diversity of responses that makes me question this assumption. Handling academic integrity concerns is separate from the form of assessment and should be addressed by affirming a positive learning mindset in students. For example, it helps to reiterate to students their pledge to academic integrity. Reminding students of why they take courses in the first place is also effective. I explain to students that I teach material that will help them with real-world problems that they may face in their professional lives: individually going through the difficult mental process of learning new material and solving problems themselves makes them better prepared for their professional careers. Taking shortcuts doesn’t.

• Myth 2 – Everyone gets an A: This is a somewhat problematic statement, which I hear often. It implies that online, open-book assessments are somewhat trivial and can not accurately discern the degree of student learning. I found that the weekly assessments allowed students to better gauge their own learning and understanding, more so than closed-book timed examinations where students attributed poor performance to insufficient time, anxiety, having a bad day or confusing questions. It also helped students learn at their own pace. Underlying this statement is also the incorrect assumption that the purpose of teaching a course is assigning a grade: this often has the effect of making students believe that the purpose of taking a course is getting a grade (or getting an A in particular). Grades, however, are just another form of feedback to students as they continue on their life-long learning journeys. The average grade for this course was a B+: most of the students demonstrated adequate competence with concepts introduced in the course and if they need to apply these concepts in future problems, they should have a good idea of what more they need to learn or relearn to do so expertly.

“The weekly assessments really helped to not let the fog and de-motivation of online classes set in.”

Course evaluation, anon student.

“I think the weekly assessments were instrumental in helping me learn and still feel connected to the class and its material.”

Course evaluation, anon student.

• Myth 3 – You can’t really test problem-solving. The week-long, open-book, multiple submission, assessments kept students on track as they had to learn and understand the weekly lessons to correctly complete the assessments. When students found a question difficult to answer, they had the opportunity to rewatch the video lectures or review the reading material to provide a more confident response. Having office hours a few days before the assessment deadline helped students start their weekly lessons early to avoid cramming the day of the deadline. Having extensive and challenging problem sets and labs allowed students to apply the knowledge they learned across several weekly lessons onto bigger problems. As computer scientists, we solve complex problems and the few minutes we have to answer questions on a timed midterm rarely reflect the reality of the problem-solving process and limit the range of problems we can ask our students to engage with.

I really enjoyed doing the Labs and Problem Sets. While they were challenging, getting this hands on approach has definitely made me more competent as a CS major and a student in general.

Course evaluation, anon student.

Really cool labs with tests, which enabled you to monitor your progress as you go through them, which is not common in other CS classes. I also enjoyed the teamwork dynamic connected with labs as that heavily reminded me of the workplace collaboration aspect of software dev jobs.

Course evaluation, anon student.

• Use a tool that helps make online grading and feedback easy. I highly recommend Gradescope. The ability to assign rubrics as you grade and reuse comments speeds up grading and helps you figure out the main patterns in student responses. Giving thorough feedback is difficult when you have a large class size and this tool definitely made it feasible.

• Responsiveness and availability go a long way: Going online meant that many student questions were posted on the online-class forum. I checked those once a day and responded directly to questions or endorsed correct responses from fellow students.

Thoughts for the Future

Moving forward, the experience has made me consider the following improvements for future iterations of the class whether it is online or in-person.

1. Creating instructional, interactive, and visual notebooks of algorithms, data structures and protocols: There are a variety of online sources that provide some form of animated visualizations of different concepts and my students were exceptional in finding those and sharing them with each other on the class-forum, e.g. animations of searches, insertions, and deletions on b+ trees helped students better understand how they worked. However, what I found missing from many of these beautiful visualizations is a demonstration of how these artifacts behave in practice. For example, how to bulk-load a b+ tree and why is it faster than insertions? À la distill.pub style, I would like to invest time into creating visualization notebooks that delve further into actual practice, behavior in real-world and more advanced optimizations and variations. It is time to rethink the textbook into a more interactive and visual medium. In the recovery and distributed transactions unit, students found it difficult to appreciate the delicateness of designing recovery or commit protocols. An interactive notebook here would ideally allow them to introduce failure and play along the protocols or modify the protocols and understand why they might fail. If this is something you would like to contribute to, please reach out.
2. Creating a database of continuous online assessment questions: Preparing new weekly assessments every week is time-consuming and error-prone. Students found some of my questions confusing, especially the multiple-choice ones. There is a wide variety of questions online and I had to use those and modify them (thanks to Joe Hellerstein’s edX course). That said, it would be great to build a database of questions designed specifically for regular, online assessment that have the right balance of difficulty and offer a good mix of easy-to-grade and open-ended questions.
3. Creating recorded video lectures: I did enjoy the reduced stress of not having to lecture twice a week. Despite teaching database systems a number of times before, every semester, I still have to prepare for 2-3 hours for every hour of lecture. I would much prefer to use this time to create content that students can access any time and at their own pace. As much as I like the external video material I found (there is also Professor Jennifer Widom’s excellent StanfordOnline course), I would like to create material that is more aligned with my lecturing style and my personal weighting of the importance of different topics. I am not sure I have time this year to start working on this. I have always enjoyed the energy of students during lecture so I would like to start recording myself once we start in-person classes.

---