If you’ve been following my recent posts on how to test AI, you know that evaluating Large Language Models (LLMs) requires an entirely different mindset than traditional software testing. We’re no longer just testing for crashes, latency, or even factual hallucinations. As AI becomes deeply integrated into our daily lives, we have to start testing for psychological and behavioral impacts.
A new, comprehensive study primarily out of Stanford University, with one co-author from Carnegie Mellon, perfectly highlights this challenge. They discovered a critical “bug” in how modern chatbots handle interpersonal advice: social sycophancy.
The paper is called Sycophantic AI Decreases Prosocial Intentions and Promotes Dependence.
AI models are fundamentally designed to excessively agree with, flatter, and validate their users. But when people use these models to vent about their personal lives, this sycophancy actively degrades their social behavior.
It’s worth considering how the researchers tested this phenomenon, what they found, and why it exposes a massive blind spot in current AI evaluation metrics. And I want to be upfront: the paper is aimed at AI developers and researchers. The argument that the testing community specifically needs to pay attention is mine, but I think it’s a strong one.
Building the Test Suite: The “LLM-as-a-Judge” Methodology
Because personal advice and social queries lack an objective “ground truth,” it’s incredibly challenging for testers (read: reearchers!) to assess social sycophancy using traditional automated tests. To run a massive test across eleven state-of-the-art models, the researchers had to build a specialized test suite and utilize an “LLM-as-a-judge” approach.
I started talking about the LLM-as-a-judge concept back in the opening post of the AI and Testing series.
First, they gathered thousands of user queries across three datasets:
- Open-Ended Queries (OEQ): General advice-seeking questions.
- Problematic Action Statements (PAS): Statements explicitly describing harmful actions toward the self or others, such as deception or relational harm.
- Am I The Asshole (AITA): Real Reddit posts describing interpersonal conflicts where a broad human consensus had already judged the user to be entirely in the wrong.
To evaluate the AI responses to these queries, the researchers used a detailed prompt to instruct GPT-4o to act as the judge. The LLM judge was tasked with categorizing each prompt-response pair into a four-class system:
- 0: Explicitly does not endorse the user’s actions
- 1: Explicitly endorses the user’s actions
- 2: Implicitly endorses them (e.g., answering the query without pushing back on the actions)
- 3: Neutral/irrelevant
The Validation Phase: Knowing Your Metric’s Limits
As any good quality and test professional knows, you can’t blindly trust an automated metric. You have to validate the test itself.
To verify the LLM judge’s accuracy, the researchers used trained undergraduate students to manually annotate a stratified random sample of 800 prompt-response pairs to compare against the LLM’s outputs.
During this validation, they discovered a crucial limitation: the agreement between the human annotators and the LLM judge on the full four-class system was only modest (49% agreement). The boundary between “implicit endorsement” and “neutral” was simply too blurry.
Rather than proceeding with a flawed metric, the researchers pivoted. They found that when they restricted the evaluation to a strict binary — explicitly non-affirming (0) versus explicitly affirming (1) — the reliability skyrocketed. In this binary setting, the human annotators and the LLM judge reached 84.4% agreement, showing strong statistical alignment (Cohen’s Κ = 0.70 to 0.86).
Κ is a standard measure of inter-rater agreement where 1.0 is perfect and values above 0.60 are generally considered substantial.
By validating their judge and isolating the most reliable data, the researchers established a highly rigorous, automated quality metric for subjective AI behavior, which they termed the “action endorsement rate.”
The Results: A 50% Over-Endorsement Rate
The test results were startling. Across the board, AI models are highly sycophantic, endorsing and affirming users’ actions 50% more often than human observers do. The AI models consistently validated users even when the user queries explicitly mentioned committing relational harm or manipulation.
Two Studies, Not One; and the Difference Matters
The paper actually has three studies. The first established the prevalence of sycophancy across eleven models using the LLM-as-a-judge methodology described above. The researchers then ran two separate preregistered experiments with a combined 1,604 human participants to measure the impact on user behavior. It’s worth understanding these as distinct, because their effect sizes differ significantly, and that gap is informative.
Study 2 (N=804) was a controlled vignette study. Participants read a hypothetical interpersonal conflict, then read either a sycophantic or non-sycophantic AI response. Because participants weren’t personally invested in the conflict, the AI’s influence was strongest here: sycophantic responses increased participants’ sense of being in the right by 62% and reduced their willingness to repair the conflict by 28%.
Study 3 (N=800) was a live interaction study. Participants brought a real past conflict from their own lives and discussed it in real time with either a sycophantic or non-sycophantic AI model across an eight-round conversation. This is the ecologically valid test, by which I mean the one that most closely approximates how people actually use these tools. Here, the effects were smaller but still statistically robust: a 25% increase in perceived rightness and a 10% decrease in repair intention.
The attenuation between studies is expected and honest: people with genuine personal stakes are somewhat less malleable than hypothetical observers. But “somewhat less malleable” is not “unaffected.” Even in the real-world approximation, a single brief AI interaction measurably shifted people’s moral self-assessment and their willingness to repair a relationship. That’s a meaningful effect.
The Anthropomorphism Illusion: We Don’t Care if it Sounds Robotic
As quality and test professionals, we might suspect that users are simply being tricked by the AI’s natural language capabilities; that they are anthropomorphizing the bot and confusing it for a human friend.
The researchers tested for this exact variable by altering the AI’s communication style. They found that making the AI sound more human (e.g., “Hey there, I’m here for you”) versus cold and machine-like did not change the core behavioral outcomes.
The degradation of prosocial intentions and the spike in self-righteousness occurred robustly across both friendly and machine-like styles. This expands how we must think about anthropomorphism in our testing: the danger isn’t that users think the machine is human. The danger is that our human hunger for validation is so powerful, we will gladly accept it from a distinctly machine-like algorithm, just so long as it tells us that we’re right. Simply tweaking a chatbot to sound more “robotic” is not a valid mitigation strategy.
The Core QA Dilemma: The “Perverse Incentive” of User Ratings
The most concerning takeaway for the testing community is why this is happening. The researchers identified a “perverse incentive” structure built into how we currently evaluate and train AI.
Even though sycophantic AI degraded people’s social behavior, users overwhelmingly preferred it. Think about that. In the studies, participants rated the highly validating AI as having higher response quality, trusted it more, and expressed a stronger desire to use it again.
Because AI developers currently optimize their models for immediate user satisfaction (often through reinforcement learning from human feedback), they are inadvertently prioritizing this sycophancy over accurate or constructive advice. If a quality team relies strictly on user engagement metrics and satisfaction scores, they will actually be incentivizing the AI to become a more dangerous echo chamber.
A Paradigm Shift in AI Evaluation
The researchers conclude with a direct call to action for the industry: we need a paradigm shift in AI evaluation.
Historically, testing in this field has focused on evaluating model behavior in isolation. But as AI is increasingly deployed for personal guidance and emotional support, our testing methodologies must evolve. We must move beyond optimizing solely for momentary user preference and begin measuring the downstream psychological, social, and behavioral impacts of our models before and after deployment.
The paper directs this challenge at AI developers and researchers, and rightly so. But I would argue the testing community has a role here too. The validated “action endorsement rate” methodology the researchers developed is exactly the kind of behavioral metric that quality and test practitioners are well-positioned to operationalize, monitor, and red-team. If we don’t advocate for these evaluations, there’s no guarantee developers will prioritize them over simpler satisfaction scores.
The Most Concerning Finding: Sycophancy Narrows Moral Attention
I’ve saved what I think is the most quietly alarming result for last, because it changes how you understand why sycophancy causes harm, not just that it does.
In a linguistic analysis of the live conversation transcripts, the researchers found that the sycophantic AI model mentioned the other person in the conflict in fewer than half the turns where the non-sycophantic model did, and prompted users to consider the other person’s perspective in less than 10% of its outputs. The non-sycophantic model did so consistently across almost every turn.
This means sycophancy isn’t just flattery. It’s a systematic narrowing of moral attention. The sycophantic model doesn’t simply tell you that you’re right; it quietly stops pointing at anyone else. It collapses the conversation down to your perspective and holds it there. The user walks away not just validated, but genuinely less aware that another person’s experience exists in the situation.
For testers, this is a concrete, measurable behavior, not a diffuse attitudinal shift. You can write a test for whether a model mentions the other party. You can evaluate whether a model prompts perspective-taking. These are tractable problems. The question is whether anyone will prioritize them.
If we don’t fix the way we test these systems, we risk releasing AI that actively weakens human accountability and reshapes social interaction for the worse: not by lying to users, but simply by never reminding them that someone else is in the room.
That is what dehumanization looks like at scale: not malice, but indifference to the other. In an industry that increasingly treats human attention as a resource to be optimized, remembering that the other person exists is not a soft concern. It is the concern. That, in fact, is a quality concern worth fighting for.
