The rapid emergence of sophisticated AI agents has forced educators to revisit long‑standing assumptions about how knowledge is assessed and retained. In a senior‑level machine‑learning‑in‑production course, the instructor observed that even before the public debut of ChatGPT, early language models could generate plausible answers to reading quizzes without ever seeing the source material. This early warning sign highlighted a growing mismatch between traditional take‑home assignments and the capabilities of generative tools. Rather than reacting with punitive measures, the educator chose to redesign the assessment landscape while preserving the core learning objectives. The experience underscores a broader trend: as AI becomes capable of producing coherent text and code, the value of measuring rote reproduction diminishes, pushing educators toward evaluating higher‑order skills such as judgment, communication, and collaborative problem‑solving. Institutions that recognize this shift early can avoid a scramble for stop‑gap solutions and instead build curricula that leverage AI as a partner in learning rather than viewing it solely as a threat to academic integrity.
Despite the upheaval in assessment methods, the fundamental goals of the course have remained remarkably stable. The instructor emphasizes that the course is not about memorizing syntax or mastering a particular framework; instead, it targets engineering tradeoffs, risk anticipation, mitigation strategies, and effective teamwork. These competencies are inherently human‑centric and resistant to full automation because they require situational judgment, ethical reasoning, and the ability to negotiate conflicting priorities among stakeholders. By keeping these objectives unchanged, the educator ensures that students still graduate with the ability to weigh the pros and cons of deploying a model in a noisy production environment, to anticipate failure modes before they manifest, and to collaborate across disciplinary boundaries. This approach also provides a clear north star when deciding which classroom activities to retain, modify, or discard in the face of ever‑more capable AI agents.
One of the most consequential policy decisions was to grant students unrestricted permission to use AI tools in virtually every course activity, with the sole exception of formal written and oral examinations. The rationale behind this permissive stance is twofold. First, attempting to police AI usage is practically infeasible; detection tools lag behind model capabilities, and enforcing attribution would create an adversarial atmosphere that undermines trust. Second, the instructor believes that learning to wield AI responsibly is an essential graduate‑level competency, akin to mastering a library or a version‑control system. By normalizing AI use and encouraging experimentation, students can develop intuition about when a model’s output is reliable and when it warrants skepticism. This policy also aligns with industry expectations, where engineers are routinely expected to augment their workflow with AI‑powered code assistants, data‑labeling tools, and automated testing frameworks.
The rise of AI has unintentionally eroded several evidence‑based teaching practices that decades of educational research have shown to promote deep learning. For instance, the literature consistently favors frequent low‑stakes assessments—such as weekly homework, short quizzes, and iterative drafts—over a reliance on a few high‑stakes exams because spaced retrieval and timely feedback strengthen memory consolidation. Yet when students can offload these low‑stakes tasks to an AI agent, the intended benefits disappear; the work no longer reflects personal effort, and feedback loops are broken. Consequently, the instructor has been compelled to shift weight toward more supervised, in‑person evaluations like exams and live TA interactions, even though this move conflicts with the ideal of distributing assessment load evenly throughout the term. The tension illustrates a broader dilemma: preserving pedagogical integrity while adapting to a technological environment that can simulate student work with unsettling fidelity.
In response to the erosion of traditional take‑home assignments, the instructor eliminated all components that required original written text generated outside of class. While solution reports that merely describe a system and point to relevant code snippets are still collected, they are treated as navigational aids rather than measures of comprehension, and AI‑generated drafts are accepted without penalty. By contrast, reflective prompts that once asked students to articulate challenges, propose teamwork improvements, or identify data‑quality risks have been deemed obsolete because a language model can produce plausible answers with minimal prompting. To preserve the reflective dimension, the course now mandates a brief, structured conversation with a teaching assistant after each assignment. During this 15‑minute check‑in, students discuss the same prompts verbally, allowing the instructor to probe understanding, request clarification, and assess whether the learner can defend their reasoning in real time.
The oral check‑ins are scored on a pass/fail basis and contribute 20 % of the points for each assignment, with the option to retry without penalty until a satisfactory demonstration is achieved. This design mitigates fairness concerns that could arise from subjective TA grading, as the primary consequence of a weak performance is additional time spent preparing rather than a loss of credit. The instructor encourages TAs to set a high bar for the initial attempt, knowing that many students will need a second try to articulate their thoughts clearly. Importantly, the format accommodates learners with anxiety or disabilities through established accommodation pathways, paralleling the support offered for written exams. By shifting the assessment of reflective thinking to a spoken dialogue, the course also nurtures a professional competency—clear, concise technical communication—that written reflections often fail to develop as effectively.
Reading quizzes, once a staple for ensuring engagement with assigned papers, have been largely abandoned in their original format. The instructor concluded that conducting in‑person, pen‑and‑paper quizzes would impose unnecessary stress and memorization burdens that detract from the low‑stakes, exploratory spirit intended for such activities. Online quizzes were retained for a period merely as a signal that students should glance at the papers, even though it was widely understood that most would consult an AI model for answers. Currently, the reading list has been halved and stripped of any point value; instead, key concepts from the papers are woven into live lectures and class discussions. While many learners still rely on LLMs to surface talking points during those discussions, the students who do read the primary sources tend to gain richer insights, illustrating that the incentive structure can still motivate genuine engagement when the assessment pressure is removed.
Weekly laboratory sessions, designed to introduce tools such as Kafka, Grafana, Docker, and Weights & Biases, have undergone a parallel transformation. These bite‑sized tasks are deliberately scoped to give novices a foothold, which also makes them highly susceptible to automation by coding agents. To preserve the educational value, the course now requires students to demonstrate completion to a TA in person and answer a few probing questions about what they built and why certain choices were made. The pass/fail grading, coupled with unlimited attempts until the lab period ends, encourages iteration without penalizing early missteps. By leveraging the TA interaction as a gatekeeper, the instructor ensures that learners cannot simply copy a bot‑generated script and claim mastery; instead, they must be able to articulate the underlying principles, troubleshoot issues, and connect the tool to the broader MLOps workflow.
To keep assignments challenging in an era where AI can generate functional code with minimal prompting, the instructor has deliberately increased the scale and complexity of the projects. The initial screening task once involved extending a modest Instagram‑clone (approximately twelve thousand lines of code) with two AI‑powered features—a challenge that originally hinged on navigating dependencies, understanding a compact codebase, and integrating unfamiliar libraries. By late 2025, advanced agents such as Claude Code could solve the entire assignment end‑to‑end, producing both code and write‑up without human intervention. In response, the course migrated to a substantially larger codebase: the open‑source team‑chat application Zulip, which exceeds half a million lines of code. Here, the requested features demand several hundred lines of modifications across front‑end and back‑end layers, necessitating genuine exploration of the architecture, debugging, and integration work that current agents cannot complete autonomously.
Grading bottlenecks have long plagued courses that rely on extensive written deliverables, and the advent of AI‑generated content has exacerbated the imbalance between rapid production and slower human review. To address this, the instructor built an automated assessment pipeline that leverages institutionally approved language models as judges. The system parses student submissions divided into multiple markdown files, runs each file through a rubric‑driven prompt, and returns a binary pass/needs‑review verdict accompanied by explanatory comments. While the occasional pass verdict is spot‑checked for quality, the majority of the TA’s effort is redirected to the needs‑review subset, where human judgment adds nuance that the model may miss. Consequently, teaching assistants report spending 50 % to 80 % less time on routine grading, freeing them to invest more energy in direct student interaction, feedback sessions, and mentorship—activities that both staff and learners find more rewarding.
The same LLM‑as‑a‑judge concept has been extended to provide instantaneous feedback on in‑class collaborative exercises. During each lecture, students participate in think‑pair‑share breakout sessions, post their responses to a shared Slack channel, and receive automated evaluations from a custom bot. The bot checks each answer against a set of criteria derived from that day’s lecture material—criteria that are not disclosed to students to prevent gaming the system. A second prompt then translates the raw evaluation into constructive, actionable suggestions, which are posted as threaded replies. Learners can iterate on their answers, resubmit, and receive further feedback, creating a rapid‑cycle improvement loop that mimics the iterative nature of professional engineering work. The instructor notes that calibrating the prompts—tuning sensitivity and specificity—often benefits from using an AI agent itself to draft the initial rubric language based on lecture slides and sample solutions.
Looking ahead, the educator offers several actionable recommendations for peers navigating the AI‑transformed classroom. First, articulate learning objectives that emphasize judgment, communication, and teamwork rather than rote replication; these are the competencies that AI cannot fully replace. Second, embrace transparent policies that permit AI use while designing assessments that require live explanation, demonstration, or iteration—such as TA check‑ins, video demos, or in‑person labs. Third, invest in lightweight automation pipelines (LLM‑based rubric checks, automated feedback bots) to reclaim instructor time for high‑touch mentoring. Fourth, continuously reassess the difficulty of assignments by benchmarking against the latest model releases, scaling up scope or introducing intentional failure cases to surface overreliance on automation. Finally, advocate for institutional support—such as subsidized AI licenses or scholarships—to ensure equitable access, recognizing that the cost of AI subscriptions is modest compared to traditional textbook expenses. By following these steps, educators can turn the disruption of generative AI into an opportunity to cultivate deeper, more resilient engineering talent.