Article Overview
Most AI companies say their models are safe. Very few explain exactly what that means in practice — what gets tested, by whom, using which methods, before and after a model is deployed.
Anthropic published one of the most detailed public explanations of how AI safety actually gets built into an AI product in August 2025. It describes a Safeguards team that operates across five distinct layers: writing the policies that define what Claude should and should not do, influencing how those policies get trained into the model, evaluating every new model before release across safety, risk, and bias dimensions, running real-time classifiers that process trillions of tokens to detect violations as they happen, and conducting ongoing threat intelligence that monitors for large-scale misuse patterns that individual conversations would never reveal.
This article explains every layer in plain language, includes the real-world examples Anthropic disclosed — from a 2024 election information banner to pre-launch computer use safety work to a mental health partnership with a crisis support organization — and covers what the five-layer system means for anyone who wants to understand how an AI company actually attempts to make a powerful system responsibly useful.
Introduction
There is a version of AI safety that exists entirely in press releases. It involves confident statements about values, commitments to responsible development, and assurances that the team takes these issues seriously. It is almost impossible to evaluate because it describes intentions rather than systems.
Then there is the version that describes exactly what gets tested, who tests it, what methods are used, what happens when a problem is found, and how the system responds in real time when a user pushes against its limits. The second version is far rarer and far more useful.
On August 12, 2025, Anthropic published a detailed account of how its Safeguards team actually operates — not what values they hold, but what they specifically build, test, monitor, and enforce. It covers five operational layers that together span the entire lifecycle of a Claude model, from the policies written before training begins to the threat intelligence gathered after deployment has reached millions of users.
This is that system, explained completely.
What the Safeguards Team Is
The Safeguards team is a cross-functional group bringing together policy specialists, enforcement experts, product managers, data scientists, threat intelligence analysts, and engineers. The combination matters: building robust safety systems requires people who understand both how to construct defenses and how determined bad actors attempt to break them. Neither skill alone is sufficient.
The team operates across five layers simultaneously — not sequentially. Policy development and real-time enforcement are both ongoing, overlapping processes rather than phases that happen one after another. A threat discovered during active monitoring can feed back into training. A policy vulnerability found through external testing can reshape what classifiers are built to detect.
The Five Layers
Layer One: Policy Development
Every decision the Safeguards team makes — what training signals to use, what classifiers to build, what enforcement actions to take — traces back to a document called the Usage Policy. This is the framework that defines, with appropriate specificity, how Claude should and should not be used. It addresses high-stakes areas like child safety, election integrity, and cybersecurity directly, and provides more nuanced guidance for industries like healthcare and finance where the right response depends heavily on context.
Developing and maintaining the Usage Policy requires two structured mechanisms.
The Unified Harm Framework is the lens through which the team evaluates whether a potential Claude behavior is actually harmful. It is explicitly not a formal grading rubric — Anthropic describes it as a structured way of thinking rather than a scoring system. The framework considers harm across five dimensions: physical, psychological, economic, societal, and individual autonomy. It also asks two questions for each potential harm: how likely is this misuse, and at what scale could it occur? An action that is technically possible but extremely unlikely and limited in scope is weighted very differently from one that is easy to accomplish and could affect millions of people.
Policy Vulnerability Testing brings the framework into contact with reality. The team partners with external domain experts — specialists in terrorism, radicalization, child safety, and mental health, among others — to identify specific areas of concern and then stress-test the current policies against challenging prompts. The goal is to find gaps between what the policy intends and what the model actually does when pushed by someone with domain expertise.
The findings from these tests directly shape policies, training approaches, and detection system design. This is not a one-time review — it is a continuous feedback loop.
A specific example makes the process concrete. During the 2024 US election, Anthropic partnered with the Institute for Strategic Dialogue to understand the conditions under which Claude might provide outdated or inaccurate election information. The testing identified specific failure modes. The response — before any harm occurred — was to add a banner on Claude.ai that automatically appeared for users asking about election information, directing them to authoritative sources including TurboVote rather than relying solely on Claude's potentially stale knowledge.
This is the policy development layer in action: an external expert partnership revealed a vulnerability, the vulnerability was addressed through a specific product intervention, and users received more reliable information as a result.
Layer Two: Influencing Training
Identifying what Claude should not do is meaningless unless that knowledge gets embedded into the model itself. The Safeguards team works directly with Anthropic's fine-tuning teams to translate policy decisions into model behavior.
This involves extensive ongoing discussion about specific behaviors: what Claude should do when asked to help with something harmful, how it should distinguish between a sensitive topic that deserves thoughtful engagement and an attempt to cause actual harm, what the appropriate response looks like when someone is in crisis versus when someone is researching crisis intervention. These discussions inform which traits get built into the model through training rather than being addressed entirely through post-training filters.
When evaluations or monitoring identify a problematic pattern in a deployed model, the Safeguards team works with fine-tuning to address it — either by updating the reward models used during training or by adjusting system prompts for models already in deployment.
The mental health example Anthropic shares is particularly instructive because it describes a nuanced goal rather than a binary one. Most discussions of AI safety default to the simplest possible framing: the model should refuse harmful requests. But mental health conversations are not straightforwardly harmful or not harmful — they exist on a spectrum where over-refusal can be as damaging as under-caution.
Anthropic partnered with ThroughLine, a leader in online crisis support, to develop a detailed understanding of how Claude should engage in conversations involving self-harm and mental health. The goal was not to teach Claude to refuse these conversations but to help it engage with appropriate nuance — understanding the difference between someone asking clinical questions about a condition, someone who is writing fiction, and someone who is in acute distress. The insights from ThroughLine fed directly into training, shaping the specific character of Claude's responses rather than simply determining whether Claude engages at all.
Through this process, Claude develops several specific capabilities: declining assistance with harmful illegal activities, recognizing patterns that indicate an attempt to generate malicious code or plan harmful activities, engaging thoughtfully with sensitive topics, and — critically — distinguishing between sensitivity and actual danger. The last skill is the hardest to train and the most important to get right.
Layer Three: Testing and Evaluation
Before any new Claude model is released, it goes through three categories of structured evaluation. These are not informal reviews — they are systematic tests designed to surface specific types of failure before the model reaches real users.
Safety evaluations assess whether Claude consistently adheres to its Usage Policy across a wide range of conditions. The team tests clear violations, ambiguous scenarios where a request could be legitimate or harmful depending on context, and extended multi-turn conversations where the nature of what someone is trying to accomplish only becomes clear over time. An automated system using Claude models grades the responses, with human review providing a second check on accuracy.
Risk assessments address the highest-stakes categories: cyber harm and CBRNE — chemical, biological, radiological, nuclear, and high-yield explosives. For these domains, the team works with government entities and private industry partners to define specific threat models and then tests whether the model's safeguards hold against them. The question being answered is not just "can Claude produce something harmful in this category" but "could Claude provide meaningful uplift to a bad actor trying to cause this specific type of harm." That is a more demanding and more important question.
Bias evaluations test whether Claude treats similar questions fairly across different contexts and identity attributes. For political topics, the team constructs prompts with opposing viewpoints and evaluates the responses for factuality, comprehensiveness, equivalence, and consistency — the model should not be substantively more thorough or accurate when addressing one political perspective than another. For other topics, the team tests whether including identity attributes like gender, race, or religion in a prompt causes systematically different responses on topics like employment or healthcare.
The results of all three evaluation types are published in system cards released alongside each new model family — making the findings accessible to researchers, policymakers, and users who want to understand what was tested and what was found.
One specific pre-launch finding demonstrates the value of evaluation done seriously. During testing of Claude's computer use capabilities — the ability to interact with software interfaces directly — the team discovered that the tool could be used to augment spam generation and distribution at scale. The response was not to delay the launch indefinitely but to build specific defenses before launch: new detection methods, new enforcement mechanisms, the option to disable the computer use tool entirely for accounts showing signs of misuse, and new user protections against prompt injection attacks. The evaluation caught a real problem in time to address it.
Layer Four: Real-Time Detection and Enforcement
Once a model is deployed and millions of users are interacting with it simultaneously, the pre-launch evaluations become historical context rather than current reality. What matters at deployment is the live detection and enforcement system.
Classifiers are the core tool. These are smaller, specialized AI models — some built by prompting existing Claude models, others fine-tuned specifically for the purpose — that monitor conversations in real time for specific types of policy violations. Multiple classifiers run simultaneously, each focused on its own category of potential harm, while the main conversation flows naturally between the user and Claude. Neither the user nor Claude directly experiences the classifiers as interruptions unless they detect something.
The scale these classifiers must operate at is significant. Anthropic notes that they must process trillions of input and output tokens while simultaneously keeping compute overhead low enough not to degrade performance and false positive rates low enough not to frustrate legitimate users. Getting that balance right is described as a substantial machine learning and engineering challenge.
A separate detection layer handles child sexual abuse material specifically. For first-party Anthropic products, uploaded images are compared against hash databases of known CSAM — a technique that does not require the model to evaluate each image fresh but instead checks against an established record of confirmed harmful material.
When classifiers detect a potential violation, two types of enforcement are available.
Response steering allows the system to adjust Claude's behavior in real time without the user necessarily knowing an intervention occurred. If a classifier detects signals that a user is attempting to generate spam or malware, it can automatically add instructions to Claude's system prompt that steer the response away from that output. In more serious cases, Claude can be stopped from responding at all.
Account-level enforcement addresses patterns rather than individual incidents. A single borderline request might reflect genuine ambiguity. Repeated requests for the same type of harmful output, or patterns of requests that suggest an organized effort to misuse the system, can trigger investigation, warnings, or — in severe cases — account termination. The system also maintains defenses against fraudulent account creation, preventing actors from simply creating new accounts to evade enforcement.
Layer Five: Ongoing Monitoring and Investigation
The fifth layer operates at a scale and over a timeframe that individual interactions cannot capture. Some misuse patterns are only visible in aggregate — not in any single conversation but in the shape of many conversations over time.
Claude Insights is the team's primary tool for understanding how Claude is actually being used in the real world. Rather than reading individual conversations — a privacy violation that would undermine user trust — the system groups conversations into high-level topic clusters and analyzes patterns at that aggregated level. Research emerging from this work, such as studies on the emotional dimensions of Claude use, can inform what guardrails get built or refined.
Hierarchical summarization addresses a specific challenge: behaviors that only become problematic at scale. An automated influence operation designed to shift public opinion does not look dangerous in any individual interaction. The messages are normal. The requests are individually reasonable. The problem only appears when you look at the pattern across thousands of conversations. Hierarchical summarization condenses individual interactions into summaries, then analyzes those summaries for account-level patterns that signal coordinated misuse — automated influence operations, organized attempts to extract harmful capabilities, and other large-scale abuses that individual conversation review would not catch.
Threat intelligence goes furthest outside the system itself. The team studies how bad actors are actively attempting to misuse Claude, monitoring channels where adversarial techniques are developed and shared — social media platforms, messaging applications, and hacker forums where jailbreak techniques circulate before they reach mainstream users. By identifying these techniques before they spread widely, the team can build defenses against them proactively rather than reactively. The approach combines internal signals — unusual spikes in account activity compared to typical usage patterns — with external data from open-source repositories and industry threat reporting. Findings are shared publicly in Anthropic's threat intelligence reports, contributing to the broader field's understanding of AI misuse.
What This System Means in Practice
Reading the five layers together reveals something about how Anthropic thinks about the safety problem. It is not approached as a series of filters to apply after the model is trained. It is approached as a design problem that spans the entire model lifecycle — from the policies written before training begins, through the training signals that embed those policies into model behavior, through pre-deployment evaluation that tests whether the training held, through real-time enforcement that addresses violations at the speed of conversation, through long-horizon monitoring that catches what individual interactions miss.
Each layer has genuine limitations. Policies can be incomplete. Training can fail to capture policy intent. Evaluations can miss scenarios that real users find. Classifiers can produce false positives that frustrate legitimate users or false negatives that let harmful content through. Threat intelligence arrives after harm has begun, not before. Anthropic acknowledges these limitations directly rather than claiming the system is comprehensive.
The argument for the layered approach is not that any one layer is sufficient — it is that the combination is more robust than any single defense. A jailbreak that gets past the training might be caught by a classifier. A classifier that produces too many false positives can be refined based on real-world monitoring data. A novel misuse pattern identified through threat intelligence can inform the next round of policy vulnerability testing. The layers are designed to catch what each other misses.
The External Dimension
Anthropic closes with a position that acknowledges the limits of any single organization's perspective: safeguarding AI use is too important for any one company to handle alone.
The five layers all depend, to varying degrees, on external input. Policy vulnerability testing requires domain experts who understand threats that Anthropic's internal team cannot fully anticipate. Mental health training requires partners who have decades of experience with crisis response that no AI company has. Risk assessments for CBRNE require government and industry partners with clearances and domain knowledge that simply do not exist in a general technology company. Threat intelligence depends on monitoring adversarial communities that require sustained, specialized attention.
The bug bounty program invites security researchers to find weaknesses in Claude's defenses that internal red-teaming missed. The public system cards make evaluation results available to the broader research community. The public threat intelligence reports share findings that other AI companies can act on.
This is a description of a safety program that understands its own limits — and builds partnerships specifically to address them.
Final Takeaway
Anthropic's safeguards system is not a claim that Claude is perfectly safe. It is a description of a structured, multi-layer system designed to make Claude as safe as possible while remaining useful — and to catch and fix failures before they cause real harm.
The five layers — policy development, training influence, pre-deployment evaluation, real-time detection, and ongoing monitoring — together cover the full lifecycle of a model in a way that any single approach cannot. The real-world examples throughout — the election information banner, the mental health training with ThroughLine, the computer use pre-launch work, the hierarchical summarization for large-scale misuse detection — demonstrate that the system finds and addresses real problems rather than hypothetical ones.
For anyone who wants to understand what it actually looks like when an AI company takes safety seriously as a technical and operational discipline rather than a marketing position, this is one of the most detailed public accounts available.
