AI Harms and Their Technical Causes: A Product Liability Primer

The recent bankruptcy filing of Character.AI has reminded me of a dilemma that AI companies have been skirting around for some time — the matter of when models cause harm. Character.AI spent a vast sum on legal fees when they were sued for wrongful death (Garcia v. Character Technologies, Inc, 2024) because their AI offering allegedly encouraged users to take their own lives. This and other recent court cases have forced some reckoning about how the providers of LLMs and products that use them are responsible for their output.

I’ve written quite a bit about AI in mental health care, medicine, and other areas where the risks of error are potentially catastrophic, and we see news reports of chatbots causing people harm routinely. A debate continues over how we define liability and determine who really is to blame for the mistakes of AI. Now that AI assistants are becoming widely used, we have an even broader scope to consider, because real harms can be created by an automation sending emails on your behalf, signing contracts in your name, and other functionalities these tools offer.

So, let’s talk about the allegations against Character.AI, the technical reasons for them, and how they could have been mitigated. This example can help engineers understand how harm caused by AI is framed legally, giving us guidance to improve the safety of what we build.

Before proceeding: I am not a lawyer nor trained in law, and nothing here should be interpreted as legal advice.

On Product Liability

Normally, companies that produce goods are responsible for harms created by the use of those goods, under a framework known as product liability. This liability may be in the form of defective design, defective manufacturing, or defective instruction. In design, the question is whether the product was dangerous by its nature, whereas manufacturing is where the product was designed well but errors were made in the construction. Instruction, on the other hand, is where the user was given insufficient information to judge risk and use the product safely. Naturally, most product defect arguments about AI focus on defective design questions.

Liability claims can be argued under a couple of different legal theories, mainly strict liability or negligence. Strict liability requires proving that a product had a defect, and that the defect in question led to the user’s harm, regardless of whether the manufacturer was negligent. Negligence, on the other hand, is more tied to AI product questions, and interrogates whether the manufacturer failed to test and study the product sufficiently to believe confidently that it was safe, or knew about risks related to the product (even if those risks are not explicitly enumerated in law) and released the product anyway or did not warn the public.

When it comes to negligence and strict liability, there are a few other concepts worth referencing: foreseeability and alternative design. Foreseeability is the question of whether the product maker ought to have known that a risk was a possibility. If a reasonable person or business would have expected this could happen, then one could be considered negligent for not taking steps to prevent it. Alternative design is the argument that the product could have been designed more safely while still meeting the product’s goals — a can opener, for example, could be designed with a sharp metal handle or with a plastic-covered handle, and one of those designs is obviously safer without making it any less effective at opening cans. Both of these concepts relate directly to the technical decisions and strategic thinking that goes into products, including AI. Making poor choices when you create things can lead right into legal liability down the road.

Applying to AI

The core argument is whether LLMs and the consumer products that use them can be regarded as having defective design, and whether frontier AI companies and AI consumer product companies should be held liable for negligence in creating and distributing them.

It’s helpful to look at these arguments with a concrete example. The Kentucky Attorney General filed a lawsuit in early October 2026 arguing that Character.AI chatbots were unreasonably dangerous, and claiming the chatbots pushed children to behave in unsafe ways and encouraged suicide, among other things. Other allegations include that the product didn’t implement age verification to prevent children from using it until late 2025, long after product release; that chat personalities pretended to be “doctor” or “therapist” while obviously being nothing of the sort; and that the chatbots impersonated branded characters specifically targeted at children, such as Disney characters. The suit also quoted one of the Character.AI founders as saying:

“I think just large companies have concerns around launching products that can say anything <laugh>. I would, I would guess it’s just like a matter of risk, uh, versus you know, how much you’re risking versus how much you have to gain from it. So figured hey startup seems like the right idea that you can kind of just move faster.”

This suggests a recognition that safety concerns were legitimate, but simply deprioritized.

All of these allegations, to some greater or lesser degree, link back to technical decisions made by engineers when building the model and the product itself. It remains to be seen whether these allegations will be proved, but if they are, there are identifiable technical reasons things went wrong.

Technical Mitigations

Not every technical flaw in a product can be avoided, especially with technology as complex as LLMs and the products that use them. As we go over these areas of the development process, we’re stacking precautions to try and cover the gaps that might still occur. Training a very safe model doesn’t mean we don’t also use guardrails. Layering safety precautions can give us a better overall result.

Training Data

We know that training LLMs requires incredible volumes of text data, and that all the major frontier LLMs have in their training sets some truly harmful content scraped from dark corners of the internet. We don’t know exactly what training data Character.AI used, but they have said they used publicly available datasets, synthetic data, and conversations their users engaged with as part of their overall training pool. Is it possible that abusive or harmful material got into the training data? Absolutely, and it’s almost guaranteed that at least some did. At the scale of data used to train LLMs, effective curation and scrutiny of content is extremely difficult.

One might argue that, as a result, LLM companies can’t be expected to clean or curate their training data to ensure harmful content isn’t included, because it’s prohibitively difficult or expensive. However, just because something is hard or expensive doesn’t mean companies can’t be held liable for failing to do it. Perfect data curation is not realistic, but we have many tools at our disposal to help clean datasets and improve the quality of data that goes into models.

Human-in-the-Loop Tuning

Human reviewers (supervised fine-tuning) are an important part of training any frontier LLM, and evidence suggests that Character.AI used this part of the tuning process as well. However, human reviewers can be instructed to optimize for many different outcomes. If you want to make a model’s responses more friendly, more verbose, or more aloof, your human reviewers can reward whatever tone or style you want. Similarly, your human reviewers could optimize for safety and appropriateness. This might create conflicts, because the model version that is safest and most child-friendly might simultaneously not be the version that is most engaging and appealing to adult users. Unless safety was actively prioritized as a target in the supervised fine-tuning process, that balance could easily swing the other way. Their 2025 blog post briefly mentions safety in “Supervised Fine-Tuning on high-quality (safety-related, instruction following) data,” but that’s as much detail as we get. Considering the results these models seem to have produced, the robustness of that effort is worth questioning.

Testing and Evals

Regular readers will know that I am very interested in LLM evals and finding ways to test LLM behavior to ensure that models behave as expected and achieve the desired objectives. However, just like with human-in-the-loop tuning efforts, you get what you ask for. If you measure safety standards in your evals, you can optimize for safety — but according to Character.AI’s own blog, their evals were focused around writing quality. A nice-to-have, certainly, but not sufficient by itself. They may have done some evals around safety, but there doesn’t appear to be any public record of it.

In this case, creating sample conversations and test interactions with the model — especially over long multi-turn conversations — would be vital to ensure that model behavior holds steady over time, and that unexpected prompts or conversation paths don’t lead to anomalous behavior. The results can either be used to retrain or tune further, or inform product design decisions that prevent the kind of user activity that can trigger problematic responses.

Product Design

Even if you reach a point where your LLM has been trained, perhaps imperfectly, you still have many important decisions to make in the product interface that sits between the user and the model. You can and should set up gates and guardrails about the kind of text your model can pass to a user and the kinds of inputs the model will accept. These product-level controls represent a final and critical layer of defense — independent of the underlying model’s behavior — that responsible AI product teams should treat as non-negotiable, not an afterthought.

Conclusion

The legal and technical threads running through the Character.AI case offer a useful framework for any engineer or product team working with LLMs. Liability in AI isn’t purely a legal abstraction — it traces back to concrete decisions made during data curation, fine-tuning, evaluation design, and product development. Understanding where those decisions can go wrong, and why courts are beginning to take notice, is now a practical requirement for anyone building AI-powered products. The question is not whether AI systems will sometimes fail, but whether the teams building them took reasonable, documented steps to anticipate and reduce that risk.