Fighting back the tide of AI Learning Shortcuts, with AI

Posted: April 15, 2024 | Author: | Filed under: Teaching | Leave a comment

Any conversation on teaching with AI needs to start from “how do we learn?”. Here I find Barabara Oakley’s many books on learning, Mindshift, A Mind for Numbers, Learning How to Learn, etc to provide an honest answer: we learn “with effort” and lots of it! Low-effort tasks like listening to lectures, reviewing notes, rereading notes are not going to give students the learning gains they need, at least not beyond passing an imminent exam that tests our capacity for word-association. Actively recalling a lecture and readings through recreating notes, solving problems, asking questions, getting help when stuck, and repeated rehearsals over periods of time, would however. These are all effortful. Most of Oakley’s learning tips are not about eliminating such efforts. Rather, they are about setting up the right environment (place and time) and, more importantly the right mindset (motivation, and warming-up with pomodoro techniques) conducive for learning that lasts. 

Now that we understand the reality of how effortful learning is, we can talk about the impact of AI on education. Marc Watkins’s post on the plethora of Ed-Tech AI tools promising to remove the effort from learning should ring alarm bells in the heads of many educators. There are two cases to explore here: educators are doing the right thing, and educators are not doing the right thing.

The second case is widely discussed and is definitely the stance many struggling students take and for good reason. Remember the days when you would hear teaching descriptions along these lines: “Its like throwing pies at a wall and hoping something sticks”. Now I hear: “Its like aiming a fire hose at students and hoping they drink!” I can’t shake the image of battered and drenched students from my head. As a computer scientist, I can relate. The research field is constantly evolving, with an ever-growing complexity (or incomprehensibility) that makes one envy Lacan’s students. AI tools that summarize papers, hour-long lectures and talks, and answer questions can help me get a handle on the hose and push it away, but I am completely aware of how deficient my learning is when doing so. Our fields have grown, our primary sources have multiplied, and we are trying to bring our undergrads up to the cutting edge — this ain’t working.  Let’s simplisticly (and incorrectly for the sake of illustration) assume that our brains are a somewhat fixed volume — a cuboid — with its width representing different disciplines, its length representing the different subfileds within a discipline and its depth representing concepts and techniques within each subfield. If we keep increasing the width and length, the depth’s got to give. And even if we promote learning in ways that share concepts across fields and disciplines and allow our brain to succinctly codify knowledge, student motivation and time cannot keep up with what we are expecting them to learn. If we don’t distill our fields into meaningful trickles that students can build on as they see fit, then they will turn to these AI tools to push away the fire hose. If I assign five primary source readings a week, I should not expect any deep grasp of these readings and if students can get the cliff notes of these readings from an AI tool, then that is fair game. With that much assigned, a shallow, superficial understanding is all that one can hope for after all. Which poses the question: why do so in the first place? If we aren’t distilling our fields then we are leaving it up to our students to do so in ways that boost the sales of these AI tools and we can’t fault them for not learning as we intended. The same argument extends to our writing assignments, labs, problem sets, etc. For our students, using AI “shortcuts” is not about reducing effort but about surviving the term. Perhaps our great challenge as educators is not about curtailing AI or other shortcuts, but about taking the responsibility of curation and pruning to ensure depth that matters.

This brings us to the first case, where we are doing the right things and assigning the right kind of learning work. In this world, I assign one primary source reading and engage my students critically in it over a reasonable period of time, and I also sufficiently motivate the students, so they believe in the value of learning the material. The same AI tools that helped with the second case are detrimental to learning. A tool that does “the effort of learning” for students strips them of learning. We need to have conversations with students about how they should learn to preempt these behaviors. These conversations must stress how to use AI tools appropriately, i.e. to set up the right environment and mindset to get the effort rolling. This can be difficult when there is an abundance of tools that aim to eliminate effort and an inaccessibility to tools that actually promote learning. 

In joint work with NYUAD alum, Liam Richards, and Prof. Nancy Gleason, we built one such AI tool, ReaderQuizzer. To help students with active-reading, ReaderQuizzer generates questions at one of two reading levels, comprehension and analysis, to help students stay on track and self-assess their reading at every page. This tool is in contrast with other AI tools that provide answers to questions asked of a reading or that summarize it. Our tool puts the effort on the student but helps guide and maintain it, akin to the 20-minute pomodoro timer that gets student started on any learning exercise. ReaderQuizzer for now is locked in as a research prototype, inaccessible to the many students who could benefit from it. The same is true for many possible AI education tools designed by educators who understand how learning works. Another tool my lab is building is a syllabus reading tool that transforms syllabi into actionable learning tasks directly in your calendar. Why are they locked? First, institutions unlike start-ups are held more accountable from a legal perspective. Does a student uploading a reading onto a tool like ReaderQuizzer constitute a copyright violation? Until a legal team settles this, higher education institutes may not wish to risk legal battles with publishers. Second, a resistance and a fear of change, breaching the subject of open access to syllabi can lead to hours of heated faculty debate. Third, institutions don’t have the IT/tech development resources to launch such tools. Where to host the servers, who manages the subscriptions, what are the right subscriptions to purchase and so on. 

Outside academia, the incentives are such that tools that offer certain features, especially those that promise “shortcuts” will inherently attract more students. If there was a magic pill that keeps one healthy and fit, would we take the pill or sweat for a tedious hour everyday? When the side-effects aren’t obvious — you aren’t truly learning — but there are immediate gains — you complete the assignment or pass the test — its hard to make the right choices. Naturally, bad tools will flourish. 

So what can be done? We can create in-house AI Ed-Tech sand boxes where both legal and development teams support on a tool by tool basis rather than aim to resolve grander issues for all. Uploading certain readings can violate copyright but not all do. A single subscription or LLM model may not work for all possible tools but since money doesn’t seem to be the primary resource constraint for now, let 100 tools flourish each with their own subscription or LLM models. What we shouldn’t do is wait until the market is flooded by bad options that we can’t pull our students back from.

Week 1 of emergency-online-teaching DB systems

Posted: March 23, 2020 | Author: | Filed under: Teaching | 1 Comment

Week 1 Lessons covered: Hashing (Cuckoo, Extendible, Linear) and Buffer Pool Management.

Lessons learned from Assessments

The weekly assessments are a mix of auto-graded (e.g. what value causes an infinite loop in a given cuckoo hashing structure) and open-ended questions (e.g. given the pin-counter, can you do away with the dirty bit?). I used Gradescope’s online assignment feature and it worked well – I highly recommend it. With 25 students, I could still read through and grade open-ended questions — it did take most of my Sunday morning!

I found that students came to Zoom’s office hours to validate their understanding of how certain algorithms worked. So the auto-graded aspects did ensure that students worked through these algorithms individually.

The open-ended questions showed me gaps that need to be addressed. Usually, these questions explored different design tradeoffs (e.g. students often drew an analogy between a B+ tree’s fill factor and a hashing table’s utilization factor and split on a rather low utilization threshold of 2/3.) It is spring break this week but next week my in-class Zoom session will target these specific areas such as why would a DBMS rely on an OS file cache and what are some side-effects/problems that could occur?

Re: Academic Integrity: We discussed online academic integrity at length during our Zoom class and I had my students digitally pledge to maintain (using a piazza poll) online academic integrity. It is definitely worth while to do so. It may seem obvious but there are things that aren’t. For example, asking students not to record zoom sessions without explicit permission and consent from others especially if you have students that would not participate if they know they are being recorded, etc.

Lessons learned from office hours

I guess what would have helped most during office hours was to have back-up worked examples, especially if your weekly unit covers specific data-structures/algorithms, etc. I had a student ask about why do you need local-depth in extendible hashing. As I tried to explain that you need to know at local overflows if you need to double the slot directory or not and if local-depth < global-depth, you don’t need to, it became clear that it would have been much easier (less confusing) if I worked through some ready-made examples on the screen.

Lessons learned from the forum

Students do like to help each other. My students shared links to external materials that helped them understand something better than the videos, assigned reading materials or textbook. They also answered each other’s questions much faster (maybe even better) than I would have.

If you don’t have a class forum, start one!

General student feedback

Some students returned to their home countries to stay with family. Others got caught in between the global travel bans and are stuck (neither at home, nor in the UAE!) I received some extension requests but even those students worked through most of the material and online assessments. I think the students do appreciate/prefer the new format. Students sent “thank you” notes. This made me feel much better about the asynchronous approach at least for this week.

More weekly updates to come.