If your application accepts any form of user input and generates AI responses based on that input, you absolutely need content moderation in place. OpenAI’s moderation endpoint is completely free to use and helps you systematically filter harmful content before it causes serious problems for your users or your platform’s reputation.
Table of contents
Open Table of contents
Why Moderation Matters
Without proper moderation systems in place, your application becomes a vector for hate speech, violent content, harassment, and other deeply harmful material that can seriously damage users and destroy your platform’s reputation, and you’re legally and ethically responsible for all of it. Content moderation isn’t an optional nice-to-have feature for production applications, it’s an absolute requirement for any platform that accepts user-generated content or allows users to interact with AI systems.
What the Endpoint Does
The moderation endpoint analyzes text content and classifies it across several specific categories of potentially harmful content:
hateandhate/threateningfor content that expresses or incites hatredharassmentandharassment/threateningfor content targeting individualsself-harm,self-harm/intent,self-harm/instructionsfor content promoting self-injurysexualandsexual/minorsfor adult content and child safety violationsviolenceandviolence/graphicfor violent or disturbing content
For each category that it evaluates, the endpoint returns both a simple binary flag indicating whether the content violated that category and a confidence score showing how certain the model is about that classification.
Why you should use this endpoint: The service is completely free with generous rate limits that work for most applications, it responds incredibly fast with typical latency of only 100-200ms which won’t slow down your user experience, the underlying models have high accuracy and are regularly updated to catch new harmful patterns, and using it helps you meet regulatory requirements for content safety in various jurisdictions.
Basic Implementation
The actual implementation is remarkably straightforward and requires just a few lines of code:
const moderation = await openai.moderations.create({ input: text });
const result = moderation.results[0];
if (result.flagged) {
// Handle flagged content
} else {
// Process normally
}That’s genuinely all you need to get started with basic content moderation in your application.
When to Moderate
User input moderation: Always check user-submitted content before you process it or pass it to your AI model, and if the moderation endpoint flags the content, immediately return an error message to the user without processing their request.
AI output moderation: Check the AI-generated response after your model produces it but before you display it to users, and if the generated content gets flagged, return a safe fallback message instead of the potentially harmful AI output.
Best practice for production systems: Moderate both user input and AI output to create defense in depth, because user input could try to trick your AI into generating harmful content, and AI models can occasionally produce inappropriate output even from innocent prompts.
Handling Flagged Content
Block completely without explanation: This approach provides the strongest safety guarantees by preventing any harmful content from reaching users, but it offers no educational feedback to help users understand what they did wrong or how to correct their behavior.
Explain which categories were violated: This approach is more educational and helps users understand your platform’s boundaries, but it reveals your detection methods which sophisticated bad actors could potentially use to circumvent your moderation systems.
Apply category-specific responses based on severity: Handle critical violations like sexual content involving minors with immediate account bans, respond to moderate violations like general harassment with warnings and temporary restrictions, and deal with mild violations like borderline language by silently filtering the content without notifying the user.
Best Practices
Fail closed for maximum safety: If the moderation endpoint is unavailable or returns an error for any reason, block the content by default rather than allowing potentially harmful material through your system, because a temporarily degraded user experience is vastly preferable to exposing users to dangerous content.
Log all flagged content for review and analysis: Maintain detailed records of every piece of flagged content including the userId for accountability, the actual content that was flagged, which specific categories triggered the flags, the confidence scores for each category, and a timestamp for when the violation occurred.
Implement a progressive strike system: Apply consequences that escalate with repeated violations, where critical violations like child safety issues result in immediate permanent bans, moderate violations accumulate as strikes where three strikes equals a permanent ban, and mild violations result in temporary restrictions or warnings.
Combine automated blocking with human review: Automatically block content with high confidence scores above 0.9 where the system is very certain about the violation, but queue content with low confidence scores for human review to catch edge cases and reduce false positives that could frustrate legitimate users.
Provide a fair appeal process: Allow users to contest moderation decisions they believe are false positives within a reasonable deadline like 30 days, because even the best automated systems make mistakes and users deserve a path to remedy unjust restrictions on their accounts.
Advanced: Context-Aware Moderation
Different contexts within your application require different moderation thresholds to balance safety with user experience: public chat channels need strict thresholds because harmful content has wide blast radius and affects many users, private direct messages can use moderate thresholds since the impact is limited to consenting adults, and creative writing tools need lenient thresholds because authors often write about difficult topics that aren’t actually harmful in a storytelling context.
Compliance Considerations
GDPR compliance for European users: Pseudonymize the userId in your logs to protect user privacy, hash the actual content rather than storing it in plain text, and set automatic retention expiry to delete moderation logs after 90 days unless you have a legitimate ongoing need for them.
Age-based restrictions for minor protection: Implement significantly stricter moderation thresholds for users under 18 years old, such as blocking any sexual content with a confidence score above 0.1 for minors while using the standard threshold of 0.5 for adults, because children need additional protection from inappropriate material.
Conclusion
Moderate all user input before processing it through your systems, seriously consider moderating AI output as well to catch model misbehavior, fail closed when errors occur to prioritize safety, log all flagged content for review and pattern analysis, implement graduated responses that match the severity of violations, and provide a fair appeal process for users who believe they were wrongly flagged.
Content moderation protects both your users and your platform from genuine harm and creates a safer environment for everyone, and implementing it responsibly is about safety and community standards, not censorship or controlling speech.