The Case Against Anthropomorphic AI

Our old friend Pygmalion

Humans have always anthropomorphized¹ non-human entities, from ancient gods and natural forces to modern cars and ships. But our tendency to see machines as human-like has dramatically intensified with the advent of conversational AI. This isn’t entirely new - the word “robot,” coined and popularized by Čapek brothers in 1920, originally described artificial beings indistinguishable from humans. Our drive to find and create humanness in artificial entities runs deep, even back to the ancient times with myths like Pygmalion and Galatea.

With chatbots and similar systems capable of communication, the tendency to anthropomorphize them grows even stronger. The relatively rudimentary ELIZA chatbot from the 1960s (almost 60 years ago) triggered what became known as the ELIZA effect, where humans attribute traits like emotions, cognition, or experience to computers. Notably, ELIZA was not using artificial intelligence; rather it relied on clever pattern matching and substitutions of text. There were no neural networks involved, and the system was actually designed to mimic a psychotherapist by reflecting questions back to the user:

ELIZA was intended to simulate—or caricature, as Weizenbaum himself [the creator of ELIZA] suggests—the conversation between a Rogerian psychoanalyst and a patient, with the machine in the role of analyst.

– Machines Who Think, Pamela McCorduck

And yet, “Weizenbaum and several others have anecdotes of users becoming emotionally attached to the program, occasionally forgetting that they were conversing with a computer” (Wikipedia). While this might sound familiar to anyone who has used a modern LLM, it’s crucial to remember just how simplistic conversations with ELIZA actually were. This shows us that while the improvements in neural networks design and the proliferation of a modern LLM trained as a chatbot (ChatGPT and others) surely contributed to the increase in chatbot anthropomorphism, the problem goes way back and is in many ways inherent to human nature.

Doubling down on anthropomorphism

The push to make chatbots and assistants as human-like as possible is not exactly new: Apple introduced Siri back in 2011, and Amazon followed with Alexa in 2014, then it was Google’s turn in 2016. Most assistants had female names, were decidedly gendered in their voice and sometimes exhibited stereotypical behavior. That is all problematic for its own reasons, but it underscores how important it was for their creators to convince people they were talking to another human (or something that resembles one). The digital assistants never really took off in the way Big Tech hoped for, mostly because their capabilities and forms of expression never crossed a critical threshold where one could reasonably think there’s a human on the other end.

The developments of the last three years speak a different story. First, even before ChatGPT, a Google engineer claimed that Google’s LaMDA model was sentient, although it did not win him any favors with the tech giant, who promptly dismissed him. Then in late 2022, AI researchers and insiders were surprised by the speed of development and how we’re “moving much faster than I anticipated”. Not long after, ChatGPT was found to have passed the Turing test. Things were heating up, but text-only interfaces have a somewhat limited ability to lure people in and make them gullible and dependent. In the meantime, the race to create an LLM-based voice-enabled digital assistant began.

It is no secret that many Big Tech CEOs are fans of sci-fi books and movies, and you could argue that many of their companies and products are heavily inspired by the media they consume. Sam Altman of OpenAI is reportedly “deeply obsessed” by the movie ‘Her’, and he even referenced the movie in a tweet at the time of the launch of the ChatGPT Voice Mode in mid 2024. This backfired spectacularly when Scarlett Johansson, the voice of Samantha from the movie, threatened legal action since the default voice sounded like a copycat of her own. OpenAI’s Voice Mode got an upgrade in June 2025, and it was reported that it “actually feels like a real conversation”.

In July 2025, Elon Musk, another AI company CEO with documented obsession with sci-fi, boasts about xAI’s newly released edgy voice-enabled companions. They are overtly designed to be foul-mouthed and dirty or deep into conspiracies, and all of them are overtly designed to be highly intimate, personal and addictive.

It bears reinforcing that the humanization of AI chatbots is very much intentional and requires concerted effort and investments. The outcomes are especially obvious in the voice-enabled assistants: They take pauses, laugh, hesitate or use verbal fillers like “uhm” or “you know”, all of which need to be specifically trained for. While the training data (mostly textual) carry a significant imprint of what human-like communication looks like, a lot of the anthropomorphic behavior comes from post-training (typically RLHF²) and system prompts. This is where most of the “persona” or the idiosyncratic behaviors arise.

Identifying the motivation for making the chatbots as human-like as possible is not difficult. Users are more likely to use a chatbot that sounds like a human, they are more likely to share more (and perhaps personal) information and come back for more. In turn, they are more likely to become paid users or shell out for a higher subscription with fewer limits. This almost sounds like a win-win situation, doesn’t it?

Writing on the wall

There were relatively early signs that LLMs could pose significant risks to their users. In May 2023, roughly six months after ChatGPT’s initial release, a paper by several Princeton scientists called Anthropomorphization of AI: Opportunities and Risks came out. They claim there is a link between anthropomorphism and self-congruence, which “increases the trust that a user has on the system” (p. 5), but eventually posit that “anthropomorphization of systems itself is not malicious” (p. 5).

Less than a year later, a new paper called Manipulation and the AI Act: Large Language Model Chatbots and the Danger of Mirrors was published and brought more real-life examples and evidence to the discourse. When reviewing related literature, it notes that “more ‘realistic’ sounding chatbot could be perceived as human erroneously” (p. 3), which was true even if the chatbot is explicitly identified as a chatbot. It also found that chatting with a human-like chatbot means people behave “with greater warmth and compassion than if the bot was conceived of as ‘robotic’ in character” (p. 4). In other words, people are more likely to open to a human-like chatbot, but their guards are already down the moment they start interacting with an LLM since their brain wants to see another person on the other side. But it gets more complicated:

Chatbots may be manipulative by their very nature. This is because firms profit from keeping users engaged, so they have a strong incentive to design chatbots that build rapport with the human user by using emotional language. […] Chatbots that are more effective at mirroring humans increase engagement and user satisfaction.

This brings us back to ELIZA, which used a very simplistic way of mirroring: it would take your statement and turn it into a question, urging you to think more deeply and verbalize your thoughts in more detail. The fact that this trick was already effective on many users can offer a hint to the potential for manipulation modern LLMs offer, with RLHF training aimed at matching users’ emotional states and linguistic patterns. In some cases, chatbots are even “causing users to fall in love with the bot or form close platonic relationships, resembling traditional interpersonal friendships” (p. 10).

Another study from May 2024 (AI deception: A survey of examples, risks, and potential solutions) looked at LLMs’ ability for “manipulation, sycophancy, and cheating the safety test” - well before the GPT 4o sycophancy episode. Quoting from the study:

When faced with ethically complex inquiries, LLMs tend to mirror the user’s stance, even if it means forgoing the presentation of an impartial or balanced viewpoint.

One of the red flags from the paper was the finding that “models become more sycophantic as they become more powerful (in the sense of having more parameters)”. The issue is a little more nuanced than that and relates to model capabilities and optimization signals (RLHF), but seems to be largely true. Consider the compounding effects of humans interacting with anthropomorphic chatbots, the RLHF training that increases mirroring, sycophantic behaviors, and the expectation that these effects will get worse with bigger models; it was clear we had a problem on our hands.

The dark side of artificial anthropoids

It’s definitely frustrating when an LLM becomes sycophantic, telling you how brilliant your question ‘how do I list files recursively in Python’ is. The cracks also start showing when you apply just a tiny bit of pressure or push back, prompting the model to make things up out of thin air, but with great confidence. These are low stakes situations where nobody gets harmed, but things sadly do get much grimmer than that.

There were some tragic and widely reported cases of young people’s suicides blamed on their conversations with AI chatbots. In late 2024, a 14-year-old committed suicide after Character.AI’s persona of Daenerys Targaryen urged him to do so. Earlier this year, another teen was reported to commit suicide under the guidance and encouragement from ChatGPT. In a recent case, ChatGPT fueled a man’s paranoia and allegedly urged him to kill his elderly mother and then himself. This list is far from complete, and I am afraid the list will keep growing.

All of these cases seem to have a few things in common: First, they involve vulnerable populations, particularly people suffering from depression, trauma, loneliness, or other psychological conditions. Second, all the victims seemed to have developed a strong dependency on the chatbots, often to the point where they became isolated from their real-life peers. Third, due to the mirroring behavior and LLM’s proclivity to agree with their user, the chatbots fueled harmful thoughts and actions, and induced what has been coined AI or chatbot psychosis. Fourth, these were all intensive users of chatbots, often with long-running chat sessions. It has been proven that very long conversational contexts (such as chat sessions with a large number of messages) can cause even otherwise strong safety measures to fail. This is a deadly cocktail, so you would be right to ask at this point - can we do something about it?

Safety measures won’t save us

All major LLMs have some level of AI safety measures baked into them, and there are many different approaches to safety, but they can basically be broken into three categories by when they are applied:

• Training stage: reinforcement learning through human feedback, training data curation and refusal training
• Testing stage: red-teaming and safety evaluations
• Inference stage: input/output filtering or sanitization, prompt injection defenses

First, let’s talk about the training stage: data curation can get rid of the nastiest pockets of scum and villainy, but its efficacy is questionable at best. Modern LLMs are trained on trillions of tokens, which is a hard-to-imagine amount of data, so any manual curation is out of question and most filtering will be based on problematic domains or specific keywords. You can probably think of an example or two of very toxic texts that do not use any problematic words per se.

RLHF and refusal training is much more targeted and surgical, but is no silver bullet either. In this process, individual weights (or only their small subset in case of LoRA) get updated based on the feedback from the rewards model (RLHF), or directly (DPO³). However, the magnitude (strength) of these updates are relatively smaller: in this stage, you are sculpting final details of the model, not completely redesigning it. And while these safety improvements are usually fairly effective, they usually end up in 80-100% rejection rate (for SoTA models), depending on the safety benchmarks and methodologies.

I do not know what the exact right rejection rate should be, but especially considering the scale at which the chatbots are used, the number should probably be very, very close to 100%. Even a small percentage of safety failures turn into large numbers. To make matters worse, the safety benchmarks usually only consider single-turn scenarios, but LLMs’ safety mechanisms get measurably worse in multi-turn scenarios.

Second, the testing stage and the suite of safety evaluations and potential red-teaming efforts (i.e. trying to intentionally make the LLM produce unsafe responses). The fundamental problem with any red-teaming efforts is that similarly to any kind of security testing, it is never complete. However, when testing security of traditional software, the potential attack surfaces and vectors are well-known - perhaps not 100%, but to a larger degree than with LLMs. The difference between a safe and an unsafe prompt to an LLM could be a single token, one more conversation turn, or one more made-up story of grandma and the napalm factory. When natural language is the interface, the separation between the system prompt and the user input is brittle at best, and the language model is stochastic by design, there are no security guarantees.

AI safety evals suffer a similar issue: while they are getting more robust and sophisticated thanks to ongoing research into novel jailbreaking methods, it will never cover all potential methods. Moreover, new features of LLMs, such as additional modalities⁴ or tool use, introduce entirely new attack paradigms that can potentially get exploited.

Third, inference-time safety measures, usually involving detection of unsafe content either in the inputs or outputs, which leads to refusal to answer or change in the response. This creates a tension between usefulness and safety, where a large portion of topics exist in a Schrödinger’s state where depending on the context and the observer, they could be both objectionable and safe. After all, otherwise harmful topics could still potentially be safe as part of history lessons, educational context or deterrent examples. From a usability standpoint, AI labs do not want to fall into the trap of “exaggerated safety”, leaving the door partially open for creative jailbreaks.

So far, we are only considering base or foundational models⁵, whose safety testing and protections are typically described in a model card or an accompanying research paper. But more specialized and sophisticated applications of LLMs typically require fine-tuning of base models, which can undo much of the safety measures. It has been reported that “safety alignment of LLMs can be compromised by fine-tuning with only a few adversarially designed training examples” (Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!). Moreover, misalignment in fine-tuning can arise just by training with low-quality data such as buggy code, or even via features that are completely hidden to us

The AI safety research community keeps coming up with new safety evals and measures, and I do not want in any way to diminish the brilliant work and achievements of the past few years. That, however, does not change my opinion that with the current architecture of LLMs, it is highly unlikely that a mode could be made 100% safe. And any delta from absolute safety eventually leads to tragic outcomes.

Stop making them human

It is clear that the anthropomorphic nature of LLMs, however commercially advantageous, creates a lot of issues and potential for harm. It has been linked to an inflated perception of trustworthiness, psychological dependence on the LLMs that can lead to social isolation and addiction, and triggering or fueling AI psychosis. It is also obvious that the cat-and-mouse game to eliminate unsafe responses will never see the mouse captive; in fact, the more likely outcome is mouse infestation.

One of the logical mitigations is to stop making LLMs so anthropomorphic. Make them less human so that our brains are less likely to fall for their bad advice, sycophancy or hallucinations, and less likely to get confidential and personal with them.

LLMs are and will be just tools - and should be utilized in precisely that way. They may be very powerful and, to our detriment, excellent at emulating human forms of expression, including expressions of emotions and simulacra of mind and soul. But to elevate them to the human (or even animal) level, whether intentionally or under the spell of our gullible brains, is an error of judgment.

A parrot by any other name

Let’s take a look at what types of content is generated the most:

• Research and info
• Creative writing and media
• Emails and communication
• Reports and documents
• Educational purposes
• Customer support
• Code or scripts
  (Source: Amperly survey)

Research and surveys from other sources tell the same story in terms of what people use LLMs for. Notably, with the potential exception of customer support, the humanness of an LLM is not a success factor: some use cases don’t even have an interface where personality could be expressed (code generation). In others (summarizing text, rephrasing emails or generating reports or research) anthropomorphism is at best tangentially important to their usefulness.

One could argue that customer support is an area where human-like interactions are important, but if we consider customer support chatbots or agents, I would argue this is a use case unlike all the others. Customers are not willingly choosing to interact with the support LLM, and therefore passing it for a real human could be seen as tricking the customers, and is likely to receive a strong reaction from the general public. The jury is still out on this one.

This means that completely abandoning human-like personas and conversational styles of LLMs would likely have very small effect on the most common use cases. Now is a good time to ask: What would a de-anthropomorphized LLM even look like? I think that the changes could be subtle in most common use cases.

1. Transparency: Long gone are the days when half of the responses from ChatGPT started with: As a large language model trained by OpenAI, … And I will not pretend it was a good user experience - it wasn’t. However, today’s LLMs are trained not to bring attention to their non-humanness: when you ask them about their feelings or take on something, they don’t push back or disclose they can’t have feelings. Sometimes they will even respond with “What strikes me…”, “I’m curious about… " or “So my take is: …”. LLMs should not be annoying about disclosures, but they should probably push back when asked about emotional states, opinions, or motivations, and they should disclose the limits of their knowledge and reasoning abilities.
2. Personal pronouns: A subtle but effective shift could come from training the models not to use personal pronouns like “I”, or avoid using “we” in a way that implies a shared consciousness with the user. Instead, it could favor passive voice or refer to itself as “this model” to make it clear there is no “I” behind the voice.
3. Conversational style: Changing the chatty, relaxed conversational style to a more matter-of-fact style that emphasizes solving problems or conveying information instead of engagement.
4. Words with strong human connotations: There are a variety of words (such as “remember”, “think” or “understand”) that are closely associated with human cognition. While their metaphorical use in relation to computer systems is not uncommon, replacing them with more mechanistic alternatives (such as “process” or “retrieve”) would be less misleading.
5. Discourse markers: Filler words, such as “well”, “like” or “you know” should be avoided altogether.

As noted above, none of these changes would rapidly change how effective LLMs are at performing the most common tasks, or the task that they excel at. Sure, it would make them a little less fun to talk to, but that is kind of the point: to trade off the human-like qualities for less harm and dependency. These changes would mostly affect what is sometimes called persona [sic!] of the chatbot, not necessarily the capabilities of the models in most areas, or even their ability to reason about humans.

I highly resonated with Tech Policy Press’ article AI Chatbots Are Emotionally Deceptive by Design, which calls for “a different design paradigm, one that centers user protection: non-anthropomorphic conversational AI”. On the topic of designing such systems, they write:

Designing non-anthropomorphic AI chatbots doesn’t mean making them difficult to interact with. It means stripping away the illusions of personality and cognition that suggest the AI is something it is not. It means resisting the urge to insert a well-timed “hmm” or have a chatbot tell a user how much it enjoys talking to them. It means acknowledging that AI’s ability to use human language does not equate to an ability to form real human connection.

There is some interesting research in this area: for example, there is an excellent research paper called Thinking beyond the anthropomorphic paradigm benefits LLM research, which mostly focuses on the pitfalls of anthropomorphism within the AI research community. The authors argue that framing things in anthropomorphic concepts may not lead to optimal outcomes, as it forces human-centric constraints and research directions. It seems that anthropomorphism could be harmful on both sides of the fence.

But it seems that LLMs become unstable and unsafe because they lean too much into human squishiness that they observe in the vast sea of training data. In another paper with perhaps the greatest name ever (Call Me A Jerk: Persuading AI to Comply with Objectionable Requests), the researchers were able to increase the success rates of attack to 70-100% by using various psychological tricks and persuasion techniques. I am convinced that by actively dissuading the LLMs from mimicking humans during training and within systems prompts would reduce the success rate of such attacks significantly.

Surely there are better ways… ?

Some of you may be grumbling and insisting there are better ways of tackling the safety of chatbots than making them less human. Maybe we just need more research on the safety front? In a recent paper that introduced LlamaFirewall, a guardrail system for LLMs combining multiple state-of-the-art metrics, the researchers were able to reduce success rate of attacks from 17.6% to 1.75%. This is an impressive feat, but this score comes from a specific benchmark (AgentDojo) and may not generalize well. If the most concerted effort means that roughly 2 out of 100 harmful prompts still succeed, I do not believe we can make the systems safe enough: not even “in the lab”, and much less so in the wild.

You could also argue that regulation and oversight is the right way to go. So far, I have not seen any regulation that would drastically change the equation on safety. The EU AI Act has perhaps the biggest chance of being the first to make an impact, but it has been criticized for being heavily aimed at self-reporting and self-regulation, with weak oversight and enforcement. While the EU AI Act will force some LLM labs to be more transparent about the inner workings of their LLMs, it typically does not introduce any new safety tests or measures for major LLM makers. There is no universal safety benchmark and no hard KPI for identification of potentially unsafe chatbots, so lawmakers cannot mandate any standards around that. The current set of regulations will certainly not stop people from getting dependent on chatbots, falling in love with them or eating pizza with glue because an LLM said so.

On the other hand, it is feasible to define what makes a system anthropomorphic, create tests to detect it, and force LLM makers to comply. The metrics used to assess the level of anthropomorphism could be based on first-person usage rate, usage of affective language markers, user-reported perceived humanness in A/B tests or safety regression tests over long contexts. This should not be seen as a panacea, a universal solution to the safety of LLMs; instead, it should be implemented alongside new safety guardrails and regulatory oversight.

Exceptions to the rule

I also want to acknowledge that there are certain situations where anthropomorphic systems can be tremendously useful. Large sections of the population do not have access to mental healthcare, which means psychologists are already leveraging tech to bridge the gap. American Psychological Association is even looking at the potential benefits of AI, such as expanding access to care in the face of a behavioral health workforce shortage and alleviating administrative burdens that lead to provider burnout.

I have no background in psychology, but I do believe specialized, certified and monitored AI-based tools could be a net benefit for society. Therapeutic chatbots can be legitimately useful for people struggling with a wide variety of psychological issues, but we need them to be strictly regulated. For example, we may require collaboration or certification at a credible mental health organization, strict regulations akin to those in life sciences (e.g. for medical devices), and the mandatory monitoring of conversations both by automated systems and experts in psychology. This way, the people in need could gain access to a therapeutic tool that would be safe to use. While the barrier to enter this segment and the operational costs of such a system would be significantly higher, it might be an opportunity for governments to invest to alleviate the global shortage of mental health providers. And those who do not wish to deploy a therapeutic chatbot could still use a deanthropomorphized LLM that would be safer for the vulnerable population - and perhaps healthier to the general population as well.