What should be established before buying a generator?

Where the provider's boundary sits, whether anything is adjustable, and what is recorded when something is refused or allowed. The first is discovered by trying rather than by reading, the second is usually less than expected, and the third is what somebody will ask for later when a specific output has to be accounted for.

Why do two tools answer the same request differently?

Because the restricted categories are set by each provider and are not the same. That is a product decision rather than a technical limit, which is why a request declined by one is fulfilled by another, and why comparing on capability alone gives a misleading picture of what a tool will actually do for you.

Does training data matter to this?

It has become part of the same question. What a model was trained on affects what a buyer is exposed to when output ships, which is why licensed sourcing and indemnification are now sold together on some products. It is moderation reasoning applied upstream rather than to any individual piece of output.

Glossary

Content moderation

Content moderation is checking material against rules about what is permitted, and with generative tools it runs in two directions at once: what users submit, and what the product itself generates.

In plain terms

Deciding what is allowed through. The familiar version checks what people upload or write. The version that arrives with a generator also checks what the product itself makes, which is a different problem: the material did not exist until your system created it, and your organisation is the one that made it.

Why it matters

Because buying a generator means buying a publishing capability, and the checks that come with it are now part of the purchase rather than a detail. What a tool refuses to make, what it will make that you would rather it did not, and who is answerable for the output are procurement questions. They tend to be discovered after deployment, usually by someone outside the team that chose the tool.

How it works

Checks sit on the way in and on the way out, and generative products made the outgoing half matter for organisations that never needed it. Incoming moderation is the older discipline and is well understood. Outgoing moderation asks whether the thing just produced should be allowed to leave, which is a question a company that has never published user content suddenly has to answer about itself.

Providers moderate at their own boundary, and their rules are theirs rather than yours. Every large generator restricts categories of output regardless of what a customer wants, which is why two products given the same request return different results. That boundary is not configurable in the way buyers often expect, and discovering where it sits is a practical exercise rather than a documentation exercise.

It has become a governed feature rather than a background service, and enterprise products say so. An avatar video platform in this guide lists moderation and consent controls alongside security as the things enterprise buyers require, which places moderation in the purchasing conversation rather than in the terms of service. That is a shift in where the topic is decided.

The two failure directions are permanent and the tuning between them is the whole job. Strict settings block legitimate work, which generates complaints immediately and visibly. Loose settings let things through, which generates nothing until it generates a great deal. Only one of those failures arrives as feedback, so the pressure on any moderation setting is consistently toward loosening it.

Where the material came from is now part of the same conversation, which is new. A generator trained on licensed and public-domain content rather than scraped data changes what a buyer is exposed to when the output ships, and one such tool in this guide pairs that sourcing with indemnification. That is moderation reasoning applied to the training set rather than to any individual output.

The old problem and the new one

The two problems share a name and differ in the one respect that matters, which is authorship. Moderating submissions is a question about somebody else's material and about what you are willing to carry. Moderating output is a question about your own material, produced by a system your organisation chose, configured and pointed at a task, and there is no third party to attribute it to. That difference changes who is answerable and it changes what a mistake means. It also explains why the outgoing check is so often missing: teams adopting a generator are thinking about capability and cost, both of which are visible in a trial, while the moderation question only becomes concrete once real output is going somewhere real. The useful move is to treat the outgoing check as part of the purchase rather than as something to add once the volume justifies it, because by then the volume is the problem.

Seen in the wild

An enterprise video platform listing moderation and consent controls among the things enterprise buyers require, alongside security.
Synthesia
A generator trained on licensed and public-domain content with enterprise indemnification available, which optimises for safe and usable over spectacular.
Adobe Firefly
A support agent that answers from your help content and hands off to a person when it reaches its limit, which is the boundary drawn at the outgoing edge.
Tidio Lyro

Common misconceptions

People assume

This is a social platform problem.

In fact

It was, until products started producing the content. Any organisation generating images, video, audio or customer-facing text has acquired an outgoing moderation question it did not previously have, and it is a harder one than the incoming version because the organisation itself is the author.

People assume

The provider's moderation covers us.

In fact

It enforces their rules at their boundary and is not adjustable to yours. Material well within what they permit can still be entirely wrong for your audience, your sector or your brand, and nothing in their settings knows that. What leaves your organisation is moderated by you or by nobody.

Telling them apart

Content moderation vs Guardrails

Content moderation

Judging material against rules about what is permitted.

Guardrails

Any check wrapped around a model, including limits on cost, scope and actions.

Moderation is one kind of guardrail, and the one about content. The most valuable guardrails are often not about content at all.

Questions

What should be established before buying a generator?: Where the provider's boundary sits, whether anything is adjustable, and what is recorded when something is refused or allowed. The first is discovered by trying rather than by reading, the second is usually less than expected, and the third is what somebody will ask for later when a specific output has to be accounted for.
Why do two tools answer the same request differently?: Because the restricted categories are set by each provider and are not the same. That is a product decision rather than a technical limit, which is why a request declined by one is fulfilled by another, and why comparing on capability alone gives a misleading picture of what a tool will actually do for you.
Does training data matter to this?: It has become part of the same question. What a model was trained on affects what a buyer is exposed to when output ships, which is why licensed sourcing and indemnification are now sold together on some products. It is moderation reasoning applied upstream rather than to any individual piece of output.

Key takeaways

Two directions now: what people submit, and what the product itself produces.
The outgoing half is new for most organisations, and harder, because you are the author.
Provider rules apply at their boundary, are not yours, and are largely not adjustable.
Only one of the two failure directions complains, so the pressure is always toward loosening.
Training-data sourcing has joined the conversation, sold alongside indemnification.