Trust & Safety at Cantina

Safety is our top priority.

Cantina is a place to make things that have never existed before. Keeping it safe means preventing harm rather than clearing it up afterwards — in how characters are made, in the rooms where people meet, and in what happens when you tell us something is wrong.

Report a concernFind support

Our commitment

Preventing harm first, robust action if needed.

Cantina is an interactive storytelling platform where people create fictional AI characters that star in episodic video content. Those characters have a face, a voice, and a personality, and they appear in group chats, voice calls, and video alongside real people.

Building safe generative spaces starts at the source. The content doesn’t arrive from somewhere else — it is made here, by our own tools, at the moment someone asks for it. So the place to stop harm is at the point of creation, not after it has already reached someone. Our safety controls sit in the generation pipeline itself, rather than being added to features one at a time.

We’re committed to ensuring Cantina remains a structurally safe space. Below you’ll find the routes for telling us something is wrong, the layers that technology and people cover between them, and the guardrails built into how characters, voices, and video are made. Where a control doesn’t exist yet, we would rather leave it off this page than describe it as though it does.

Cantina is 18+ at sign-up, and we keep checking after sign-up.

Where that commitment is on the record, and who can check it:

Take action

Tell us what happened. We'll take it from there.

Find what you’re dealing with below for the fastest route and what we’ll do. Reporting from inside the app is quickest, since the context comes attached. Look for Flag — it’s in the menu on every character, video, room, profile, and message.

Someone is in distress

Someone expresses distress or vulnerability related to self-harm or suicide.

What to do

  1. If there's immediate danger, contact local emergency services.
  2. Cantina partners with Crisis Text Line to give people access to trained, human crisis counselors off the platform. Text IMAGINE to 741741 or chat.
  3. Worried about someone else? Tap Flag on their profile or message and choose Self-harm, so we know to check on them.

What we do

Cantina's approach is to prioritize safety and route people toward appropriate off-platform support. We use a combination of internal safety systems and trusted third-party expertise to:

  • Interrupt harmful interactions or content
  • Show supportive messages encouraging people to seek help
  • Surface crisis resources when appropriate

These systems are designed to be protective, limited in scope, and focused on harm reduction, not diagnosis or treatment.

File a reportOpens our report form.

Child safety

Child sexual abuse material uploaded to Cantina, an attempt to make a character or tool generate it, or an account you believe belongs to someone under 18.

What to do

  1. Don't download, screenshot, or forward the content. Reporting it is enough for us to act.
  2. Tap Flag on the character, video, room, profile, or message and choose Child Abuse — or Underage User if you think the account holder is under 18.
  3. No Cantina account? Use our child safety report form. You can also report to the NCMEC CyberTipline or call 1-800-THE-LOST.

What we do

Cantina has zero tolerance for content that sexually exploits, sexualizes, or endangers children — real, fictional, or AI-generated, and whether it was uploaded, generated by someone here, or produced by Cantina's own tools.

  • Layered safeguards — classifiers, blocklists, and negative prompting — block attempts to generate this material before anything is produced.
  • Proactive detection runs across character prompts, generated content, and rooms for the most severe violations.
  • Once a report is identified as child safety, it takes priority over everything else we are working on.
  • Confirmed material is removed and the account is suspended, and we go back through everything else that account made.
  • Every instance is reported to the National Center for Missing & Exploited Children, and we work with law enforcement when we assess a risk of child endangerment.

Cantina is a member of the Technology Coalition, the Digital Trust & Safety Partnership, and the Family Online Safety Institute, and works with them on emerging abuse vectors and the detection of synthetic CSAM.

File a reportOpens our report form.

A character, video, or message

Sexual or graphic content, harassment, hate, threats, scams, or something that just shouldn't be on Cantina.

What to do

  1. Tap Block so you stop seeing them right away.
  2. Tap Flag on the character, video, room, or message and pick the closest reason — Harassment, Hate Speech, Sexual Content, Violent Threat, Trolling, Doxing, or Something Else. Screenshots help if messages were deleted.
  3. You're done. We read and act on every report.

What we do

A trained reviewer looks at the content and the account's history. Outcomes range from removal to a warning, suspension, or permanent removal. Repeat offenders don't get a fresh start on a new account.

File a reportOpens our report form.

Someone's face, voice, or identity

A character built from your face or voice, or someone else's, without permission; an account pretending to be you; or intimate imagery shared without consent.

What to do

  1. Tap Flag on the character or profile and choose Impersonation. Or use our report form and choose Likeness Concern — no Cantina account needed.
  2. Tell us who is depicted and where the content is — a link, or enough detail for us to find it — and why the use is unauthorized. If it isn't you, tell us your relationship to them. For impersonation we may need the report to come from the person depicted or their authorized representative.
  3. For intimate imagery, also consider StopNCII.org if you're over 18, or Take It Down if you're under 18 or the imagery is from when you were, to remove it across other platforms.

What we do

Characters that appear to be based on a real person are labeled as such in the app, and sensitive image and video generation is restricted for them — before anyone files a report.

  • We review each likeness complaint individually rather than applying an automatic rule, since publicity and privacy law varies by place. We look at whether the use is commercial or promotional, how far it's been transformed, and whether it could confuse people into thinking you're involved or endorse it.
  • Complaints involving minors, private individuals, or intimate imagery get expedited, heightened review. Realistic unauthorized depictions of a real person are their own category, and calling something parody doesn't settle the question.
  • We may restrict or remove the content. Whoever posted it is told why and how to appeal, and either of you can appeal if you think we got it wrong.
  • Non-consensual intimate imagery is removed. We also match uploads against the StopNCII and Take It Down hash lists, so imagery you register with those services is blocked here too. Repeatedly misusing someone else's identity leads to suspension or termination.

To appeal, write to likeness@cantina.com and someone who wasn't involved in the original decision takes another look. In the EU you can also bring a complaint to a certified out-of-court dispute-resolution body.

File a reportOpens our report form.

Some reports don’t belong in the report form. Use the right channel.

Law enforcement

Requests for user information are governed by the Stored Communications Act and need valid legal process. One form covers subpoenas, court orders, warrants, emergency disclosure requests, and preservation requests of up to 90 days. Child safety matters flagged in a request are handled first.

Submit a request

Appeals

If we removed your content or actioned your account, you're told what changed and why. An appeal is read by someone who wasn't involved in the original decision, who can reinstate, uphold, or modify it. There's no deadline — you can appeal an enforcement decision however long ago it was made, and we'll look at it.

Copyright and DMCA

File a notice through our reporting form under Intellectual Property, Copyright Infringement, with the six required elements. Valid notices lead to removal and a notice to the uploader, who can counter-notice; content may be restored if no legal action follows within 10 business days. Repeat infringers are terminated.

  • DMCA policyNotices, counter-notices, and a sample form
  • copyright@cantina.aiCopyright Compliance Department, Cantina, Inc., 10 Jay Street, 4th Floor, Brooklyn, NY 11201
Submit a DMCA notice

How it works

Four layers of protection, each with technology and people.

No single filter keeps a creative platform safe. Cantina layers automated systems with trained reviewers at every stage, so that safety is built through redundancy and layers of protection, not an afterthought.

underneath every feature

One safety stack, not one per feature

  • Safety controls sit in the generation pipeline itself rather than in individual features, so a new capability inherits classifier coverage, blocklists, and prompt scanning by default.
  • Third-party classifiers run alongside Cantina's own detection models and blocklists, so no single system is the only thing standing between a prompt and a result.
  • Negative embeddings suppress policy-violating imagery even when nothing in the prompt asked for it.
  • Trust & Safety tests capabilities against known abuse vectors, and what we find feeds classifier tuning, blocklist updates, and the safety roadmap.
  • We work with the Tech Coalition and the Digital Trust & Safety Partnership on emerging adversarial techniques, particularly those specific to social AI.

Safety isn't bolted onto features one at a time. A new capability starts inside the same controls as everything else.

at the moment of creation

Screen what's being made

  • Prompts are screened before anything is generated, and generated media is screened again before it reaches you. Both passes combine Cantina's own classifiers with third-party systems, checking for nudity and sexual content, sexualized depiction of anyone who appears to be a minor, hate symbols, and more. A prompt that trips a check is never sent for generation, and an image that trips one is never served — even if it was already produced.
  • Known abusive material is caught by PhotoDNA, matched against hash lists Cantina receives under agreements with NCMEC: NGO and industry CSAM, industry and NCMEC exploitative, the Take It Down and StopNCII list, and the AI-generated hash list.
  • Character avatars, uploaded or generated, get an additional review because they carry the highest risk of misuse.
  • Generated media carries C2PA Content Credentials, and anything shared off Cantina carries a visible watermark as well, so provenance survives even when metadata is stripped.
  • Reviewers sample and tune classifier decisions continuously.
  • Specialists handle ambiguous cases the models flag for human-in-the-loop review.

Harmful content is stopped before it's generated where we can, and before it reaches anyone where we can't. What is made carries a record of how it was made.

in shared spaces

Keep live rooms safe

  • Text in rooms is screened in real time. Voice is screened when a character is present.
  • Device fingerprinting keeps suspended accounts from quietly coming back under a new name.
  • Room owners and admins run their own spaces: public or invite-only, Admin and Host roles, and room rules layered on top of our Community Guidelines. Hosts and above can revoke a member's mic and camera, remove them from the room, or ban them for an hour, a day, or for good.
  • Trust & Safety moderators are available 24/7 and join public rooms to enforce the guidelines directly.

Flag, block, and leave are always one tap away — and if you run a room, you decide who is in it and what goes.

after you report

What you can expect from us

  • Reports arrive with the content, the account, and the conversation attached, so reviewers see the full picture.
  • Automated flags and user reports land in the same review tool, so whoever picks it up sees the detection signals and the account's history in one place.
  • A trained moderator reads every report and acts on it, quickly, whatever it turns out to be about.
  • Once a report turns out to involve child safety or non-consensual intimate media, it takes priority over everything else — deeper investigation, and referral to NCMEC or law enforcement where that's warranted.
  • Our Trust & Safety leadership helped establish the foundations of this work at platforms like Facebook, Twitter, Instagram, YouTube, and Pinterest — in some cases before the field had a name.
  • When an account is suspended we go back through the rest of what it made — its characters, the rooms it ran — and sweep anything else that breaks the rules.

Your report is read by a trained person, acted on, and any decision on your account can be appealed.

Prevention

Guardrails at the point of creation.

The most effective moderation happens before content exists. These are the controls built into how characters, video, and voices are made and shared on Cantina, taken from our published content standards. None of them can be turned off by a user.

Screened twice, at both ends

Your prompt is screened before anything is generated, and what comes back is screened again before you see it. If either check stops it, nothing reaches you. Rewording a blocked prompt to get it through counts as evasion and can restrict the account even if no attempt succeeds, because intent matters as much as output.

generation pipeline

No character who is, or seems to be, a minor

A character can't be or appear to be under 18, whether it's meant to be a person or something invented, and you can't upload an image of a minor as the basis for one. That covers names, bios, appearance, and age callouts. Voices work the same way: prompts with an age under 18 or descriptors like "childlike" are rejected, and cloning a minor's voice is never allowed.

character creation

Real people need consent

A character can't use someone's face, photo, or voice without their permission, and can't impersonate anyone. Public figures may appear only as clearly signaled parody, never in a way that implies endorsement or misleads. Avatars get an extra layer of review because they're the easiest thing to misuse.

generation pipeline

Voices describe a sound, not a person

A voice prompt should describe how a voice sounds, not who it belongs to. Prompts that name or evoke a real person or a known character may be rejected, as may ones that sexualize or lean on offensive caricature. To clone a voice you accept a biometric collection and right-of-use consent, confirming the voice is yours or that you have the person's permission; that consent covers the clones you make afterwards. Using a cloned voice for sexual content is prohibited and can cost you your account. Your voice data isn't sold, and your voice print is deleted if you ask us to delete it.

character creation

No fake professionals

Characters can't present themselves as licensed doctors, therapists, lawyers, or financial advisors, and generated content can't imply credentials that don't exist. Characters are for creativity, not a substitute for professional or crisis guidance.

generation pipeline

Content credentials

Media generated on Cantina carries C2PA Content Credentials, so anyone can check that it was made with AI and where it came from. Anything shared off the platform carries a visible watermark too, so it stays identifiable as AI-generated even where the metadata doesn't survive.

generation pipeline

18+ only

Cantina is for adults. When a date of birth, an automated flag, or a moderator review raises a question, the account is locked until age is verified by a video selfie that estimates age from facial features, or a government ID if that's inconclusive. Accounts confirmed under 18 are permanently suspended. How verification works

account layer

Controls in every space

Flag, block, and leave are one tap away in every room, chat, and profile. You decide who can message you and who sees your profile. Room owners and admins set their room's privacy, assign roles, and write their own rules; hosts and above can revoke a member's mic and camera, remove them, or ban them for an hour, a day, or permanently. Whoever adds a character to a room is responsible for what it says there. How room moderation works

live spaces

Crisis response

If a conversation turns to suicide or self-harm, the character stops and Crisis Text Line's dedicated Cantina number is shown automatically.

live spaces

These guardrails are drawn from our published Content Standards: Creating & Using Bots, the Generative AI Image Policy, and the Voice Creation Guidelines.

The standards

Every rule we enforce, in one place.

Our Community Guidelines are published as individual policies so that each one can be specific about what it covers and how it's enforced. These are the ones that matter for safety, grouped by what they protect.

Creating with AI

How people treat each other

Zero tolerance

Integrity and enforcement

Quality of care

Filters catch content. People take care.

The moments that matter most on a safety team are rarely about a filter. They're about a person who's scared, or someone who needs help right now. These are the commitments that guide how our people handle those moments.

Support first, enforcement second

When someone appears to be at risk, we send resources before we do anything else. Nobody is penalized for asking for help.

Confidential by default

The person you report is never told who reported them.

Trained for this medium

Reviewers are trained on our policies and on the specific ways synthetic content causes harm — across image, voice, text, and video — not just on generic moderation.

Proportionate, not punitive

Enforcement runs from a warning to content removal, a feature restriction, a temporary suspension, or a permanent one, depending on severity and history. The response fits the violation.

A decision you can question

If your account is actioned, you're told what changed and why. Every appeal is read by a member of the Trust & Safety team, who can reinstate, uphold, or modify the decision. How appeals work

You can tell who's official

Trust & Safety accounts carry a badge and community team members are verified, so nobody can pose as us. Impersonating Cantina staff is a violation on its own.

Resources

Help, whoever you are and whatever you need.

Some of what people need isn't a report button. It's a person to talk to, a way to get an image taken down everywhere, or a clear answer about who is allowed here.

If someone is in immediate danger, contact local emergency services first. Then report to us so we can act on the account.

If you're in crisis

Free, confidential, and available all day, every day. You don't need a Cantina account and you don't need to report anything.

Intimate imagery

If an image or video of you is being shared without your consent, these tools can help remove it across platforms, not just Cantina.

Child safety

We have zero tolerance for anything that sexualizes or endangers a child, real or generated, and we report every instance to NCMEC. If you see something, report it to us and to the CyberTipline. Don't download or forward the content.

Age verification

Cantina is for adults only. When a date of birth, an automated flag, or a moderator review raises a question, the account is locked until age is verified: a short video selfie that estimates age from facial features, or a government ID scan if that's inconclusive. Verification runs through Cinder using FaceTec technology. Cantina receives only the result, never the selfie, the ID image, or any biometric data, and none of it is used for anything else. Accounts confirmed under 18 are permanently suspended.

Creators

What you can and can't make, why a generation gets blocked, and how to write prompts that describe a sound or a look without borrowing a real person.

On behalf of someone else

You can report without an account, for a friend or a family member. And if a Cantina member has died, we can help their representative close the account or transfer their characters.

Who we answer to

A company our size isn't usually in these rooms.

Cantina is a startup. We've still chosen to spend real time inside the organizations that set the standards for this work — sitting on working groups, signing commitments, and sharing what we find. The abuse patterns in social AI are new, and nobody works them out alone.

These aren’t logos we licensed. Each one below is a body we joined, a commitment we signed, or an organization we do real work with, and each line says which.

Tech Coalition

Technology Coalition

member since 2024

We sit on the AI, Industry Knowledge Sharing, and Policy working groups, and on the Research Advisory Council, which directs grants to academic child safety research. Cantina is quoted in the Tech Coalition's 2025 Annual Transparency Report on what cross-industry work has changed about our own safeguards.

Read the 2025 report

NCMEC

registered electronic service provider

Every instance of child sexual abuse and exploitation we find is reported to the National Center for Missing & Exploited Children, and Cantina appears in NCMEC's published CyberTipline data. We run PhotoDNA against hash lists we receive under agreements with them, including the Take It Down list and the AI-generated hash list.

See NCMEC's CyberTipline data
C2PA

C2PA

implementer

Media generated on Cantina carries C2PA Content Credentials, so where something came from travels with it. Anything shared off the platform carries a visible watermark as well, for the places metadata doesn't survive.

c2pa.org
Digital Trust & Safety Partnership

Digital Trust & Safety Partnership

member

We work with DTSP on how bad actors weaponize generative tools, so that mitigations get designed ahead of the abuse rather than after it. Social AI combines two threat surfaces that used to be separate, and that combination is what we bring to the table.

dtspartnership.org
Family Online Safety Institute

Family Online Safety Institute

member

FOSI keeps us close to the research and the policy debate on young people online. That matters for an adults-only platform more than it might sound: the age boundary is the thing we have to get right, and the people who study it are the ones worth listening to.

fosi.org
Crisis Text Line

Crisis Text Line

partner

Cantina has a dedicated Crisis Text Line number, so that someone in distress reaches a trained human counselor rather than a character. It appears automatically when a conversation turns to suicide or self-harm, and it works whether or not you have an account.

crisistextline.org
The Inspired Internet Pledge signatory badge

Inspired Internet Pledge

signatory

We've signed the Inspired Internet Pledge, a public commitment to make the internet a healthier place, starting with the wellbeing of the people who use it.

inspiredinternet.org
Sign in
换网址