Content moderation

Filter abusive language in comments, chat, usernames and reviews before it reaches other users.

The problem#

Open text fields invite spam and abuse. Manual moderation does not scale, and keyword lists miss misspellings and context. You want a fast first pass that handles the clear cases and queues the uncertain ones for people.

Signals to combine#

• Profanity detection: a riskScore and an isSafe verdict for a piece of text, with the option to list the matched words.
• Email scoring and IP reputation for repeat offenders who register throwaway accounts.
• Blacklists for users or networks you have already banned.

Integration flow#

1. When a user submits text, call profanity detection from your server. Pass userID so abusive authors are traceable in events.
2. If isSafe is true, publish immediately.
3. If it is unsafe with a high riskScore, reject and explain which rule was broken.
4. For borderline results, publish as pending and queue for a moderator.
5. Count repeated violations per user and escalate to a temporary mute.

<?php
$text = $_POST['comment'];

$url = 'https://gurdx.cretip.com/api/scoring/profanity?' . http_build_query([
    'text' => $text,
    'userID' => $user->id,
]);
$res = json_decode(file_get_contents($url, false, stream_context_create(['http' => [
    'header' => 'Authorization: Bearer ' . getenv('GURDX_KEY'),
    'timeout' => 4,
]])), true);

if (($res['status'] ?? '') !== 'success') {
    $state = 'pending';           // fail safe: queue for a human
} elseif ($res['data']['isSafe']) {
    $state = 'published';
} elseif ($res['data']['riskScore'] >= 0.8) {
    $state = 'rejected';
} else {
    $state = 'pending';
}

Suggested thresholds#

The score for this endpoint is on a 0 to 1 scale, and Gurdx raises an event from 0.5 upwards.

Use stricter limits for children's audiences and looser ones for adult forums. Send long texts in chunks if you hit request limits.

Warning: Classifiers make mistakes, particularly with quotes, reclaimed words and mixed languages. Let users appeal and review a sample of auto-rejected content regularly.

Send a chat alert for profanity events to your moderators' channel. Related: Fake account prevention.