Duolingo's Angel Lee on AI Discernment vs. Approval

Artificial intelligence is rapidly integrating into every facet of our lives, promising efficiency, speed, and unprecedented capabilities. Yet, as AI systems become more sophisticated, a critical question emerges: are we designing them to truly augment human intelligence, or merely to solicit our approval? Angel Ortmann Lee, a software engineer at Duolingo, tackled this profound challenge in her compelling presentation, "Build AI Systems for Discernment, Not Approval." Her insights offer a crucial roadmap for developers aiming to create AI that empowers critical thinking and accountability, rather than fostering blind trust.

The Core Philosophy: Discernment, Not Approval

Lee's central thesis challenges the conventional approach to AI implementation. Often, AI is designed to present a solution or a decision, with the expectation that human users will simply approve it. This creates a passive interaction dynamic, where the human acts as a rubber stamp. Instead, Lee advocates for building systems that encourage discernment – the ability to judge well, to perceive and understand differences, and to make informed decisions based on critical analysis.

This shift is not merely semantic; it represents a fundamental change in how we conceive of human-AI collaboration. It moves beyond AI as an oracle and positions it as a sophisticated tool that provides insights, raises flags, and presents data, requiring human expertise to interpret, validate, and ultimately take responsibility for the final outcome. This is especially vital in applications where the stakes are high, and errors can have significant consequences.

Understanding Human-in-the-Loop AI (and its Pitfalls)

At the heart of building discerning AI systems lies the concept of Human-in-the-Loop (HITL) AI. Lee defined HITL AI as a system where a human actively participates in the operation, supervision, or decision-making process of an automated system. The intention behind HITL is noble: to ensure accuracy, safety, accountability, and ethical decision-making in complex or sensitive scenarios.

The typical flow is often simplified as: Model -> Human -> Decision. However, Lee emphasized that this process is rarely linear. True human-AI collaboration involves cyclical feedback loops, where human input not only leads to a decision but also informs and improves the underlying AI model. This continuous interaction is essential for refining AI performance and adapting to nuances that automated systems might miss.

While the theory of HITL AI is sound, its practical implementation often falls prey to human psychological biases, particularly as our familiarity and trust in technology grow. This increasing reliance, coupled with decreasing caution, sets the stage for a significant challenge: automation bias.

The Silent Threat: Automation Bias

Automation bias is a cognitive shortcut where humans tend to favor and over-rely on automated systems, often without sufficient critical evaluation. It's the phenomenon of blindly trusting AI decisions, even when they might be incorrect or incomplete. Lee illustrated this with everyday examples:

• GPS Navigation: Following directions without questioning whether they are the most efficient or accurate route, even if common sense suggests otherwise.
• Search Engine Results: Accepting the top search results as definitive truths without cross-referencing or critical thought.

These seemingly innocuous examples take on a more sinister character in high-stakes environments. Lee cited a compelling study where participants, when presented with AI-generated answers on an exam, accepted those answers 80% of the time, even when the AI's suggestions were wrong. This alarming statistic underscores a significant reliance on AI output, often at the expense of independent human judgment.

The research further revealed the detrimental impact of this bias: when the AI was correct, human performance improved by a notable 25%. However, when the AI was incorrect, human performance decreased by 15%. This suggests that humans aren't just supplementing their thinking with AI; they are, in some cases, supplanting it entirely. The implications for critical fields like college admissions, visa applications, or medical diagnoses are profound, where errors influenced by automation bias can have life-altering consequences.

Case Study: The Duolingo English Test and AI Integrity

To concretely demonstrate the challenges and solutions related to automation bias, Lee delved into a detailed case study: the Duolingo English Test (DET). The DET is a high-stakes online English proficiency exam, recognized by over 6,000 institutions globally. Ensuring the legitimacy and integrity of test scores is paramount, as these scores directly impact academic and professional opportunities for test-takers.

The DET employs a multi-layered security approach, combining advanced AI with human oversight. This includes:

• Identity verification.
• A locked-down testing environment to prevent external interference.
• AI-assisted monitoring for suspicious behavior.
• Human proctor review of flagged sessions.

One specific area of focus for AI-assisted monitoring is detecting "copy-typing," where a test-taker reproduces text from another source rather than composing it independently. Duolingo utilizes a sophisticated CNN-Transformer model that analyzes keystroke patterns to differentiate between genuine composition and transcription. The model is designed with a conservative threshold, aiming for a very low false positive rate to minimize incorrect flags.

However, even with a robust model, the risk of automation bias in human proctors remained. Duolingo conducted an experiment where they introduced fake AI signals for copy-typing during human review. The results were telling: human reviewers endorsed these false alerts at nearly coin-flip rates. This provided concrete evidence that even experienced proctors were susceptible to automation bias, demonstrating a tendency to trust the AI's signal even when it was fabricated.

Engineering for Discernment: Duolingo's Approach

Faced with clear evidence of automation bias, Duolingo recognized the need for a strategic intervention. The solution wasn't to remove the human or the AI, but to fundamentally redesign the human-AI interaction loop. New proctoring guidelines were implemented, emphasizing a crucial shift in perspective:

An AI signal should be treated as a preliminary alert, not a definitive judgment. Human reviewers were explicitly instructed that they must find independent, verifiable evidence to uphold a flag. This meant moving away from a system where AI simply presented a verdict, and humans confirmed it, to one where AI provided a strong hint, and humans acted as investigators to confirm or deny.

This principle is what Lee termed "engineering the reasoning." It's about designing the AI system and its interface in a way that actively elicits critical thinking and independent judgment from the human user. It forces humans to engage with the problem, analyze the evidence, and take ownership of the decision, rather than passively accepting the AI's output.

Key Principles for Designing Human-AI Interaction

To effectively "engineer the reasoning" and foster discernment, Lee outlined several actionable principles for AI developers and product designers:

1. Structure Inputs and Outputs to be Specific, Not Just Walls of Text:
   AI outputs should be clear, concise, and highlight the most relevant information. Instead of presenting a large, undifferentiated block of text, the interface should guide the human's attention to specific data points, anomalies, or areas of concern. This helps focus human review and makes the task of discernment more manageable and less prone to oversight.

2. Highlight Assumptions Made by the Model and Ask for Sign-Off:
   Transparency is key. AI models operate based on certain assumptions and parameters. Making these explicit to the human user empowers them to evaluate the context and potential limitations of the AI's suggestion. Asking for explicit sign-off on these assumptions ensures that the human is actively engaging with the AI's reasoning, not just its conclusion.

3. Build in Friction and Review Gates to Slow Down Deliberate Thought Where Needed:
   Counter-intuitively, sometimes the best design choice is to add friction. In high-stakes scenarios, rapid decision-making can lead to errors. Implementing deliberate review gates, mandatory pauses, or requiring multiple confirmations can force humans to slow down, critically evaluate the information, and prevent knee-jerk approvals. This ensures that crucial decisions are made with the necessary deliberation.

4. Collect Explicit Feedback to Create a Taxonomy of Signals for Model Evaluation and Improvement:
   Every human interaction with an AI system is an opportunity for learning. Instead of just a binary "approve/reject," systems should be designed to collect rich, explicit feedback. This feedback can create a detailed taxonomy of why a human disagreed with the AI, what additional evidence they considered, or what nuances were missed. This data is invaluable for analytics, fine-tuning the AI model, and fostering continuous improvement.

Cultivating a Virtuous AI Cycle

Lee emphasized that interaction design fundamentally determines whether an AI system operates in a "vicious cycle" or a "virtuous cycle."

• Vicious Cycle: The AI makes confident calls, humans uncritically rubber-stamp them, and the AI, receiving only approvals, becomes even more confident, perpetuating the cycle of potential bias and unverified decisions.

• Virtuous Cycle: The interface actively forces independent human judgment. Disagreements between human and AI are meticulously logged and analyzed. This feedback is then used to identify where the model was wrong, leading to targeted improvements in the AI. This creates a positive feedback loop where the AI gets smarter, and human discernment becomes more refined.

Crucially, every single interaction within a human-AI loop generates a label. This label, whether it's an approval, a rejection, or a detailed piece of feedback, is data. By consciously designing for a virtuous cycle, we transform these interactions into a rich dataset that drives continuous innovation and enhances the reliability and trustworthiness of AI systems.

Conclusion: Beyond Automation, Towards Augmented Intelligence

Angel Ortmann Lee's insights from Duolingo provide a powerful reminder that the future of AI isn't about replacing human intelligence, but about augmenting it responsibly. By building AI systems for discernment, not approval, we move towards a paradigm where technology serves as a powerful assistant that elevates human capabilities, rather than diminishing our critical faculties.

For developers and product managers, this means a deliberate focus on the human-AI interface. It requires asking not just "Can the AI do this?" but "How can the AI help humans do this better and more thoughtfully?" By embracing principles of transparency, friction, and explicit feedback, we can design AI systems that transform humans into skilled investigators, ensuring that the promise of artificial intelligence is realized with integrity, accountability, and genuine human insight.