Anonymous chat attracts two kinds of people: someone bored on a Tuesday night who wants to say hi, and someone who already knows what they want to do in a room full of strangers. Bored People Chat has to protect the first group from the second, without making honest regulars feel like they're talking through a prison intercom.
This post is a tour of how moderation actually works under the hood: what runs instantly when you hit send, what runs in the background, how bans escalate, and why a brand-new nickname gets treated differently from someone who's been around for a week.
Safety is the product
Most of the engineering principles in BPC come back to one idea: the room has to feel safe before it can feel alive. People open up in chat when they trust that creeps, slurs, and drive-by harassment won't land unchecked. That means moderation isn't a feature we bolted on later. It's the foundation.
The system is built in layers so sends stay fast. Heavy checks don't block the WebSocket. Borderline harm can be caught retroactively. The worst stuff (CSAM, grooming, minors) gets zero tolerance, no second chances.
Three layers, one pipeline
flowchart TB
Client["You hit send"]
Filters["Wordlists and pattern filters"]
Score["Toxic score"]
Enforce["Threshold checks"]
Queue["Moderation queue"]
Worker["Background worker"]
OpenAI["OpenAI Moderation API"]
Redact["Profanity redacted"]
Hide["Message hidden"]
Soft["Soft block"]
Hard["Hard block (nick and IP)"]
Client --> Filters
Filters --> Redact
Filters --> Hide
Filters --> Score
Score --> Enforce
Enforce --> Soft
Enforce --> Hard
Client -->|"new visitors: msgs 1-5, then every 10th"| Queue
Queue --> Worker
Worker --> OpenAI
OpenAI --> Score
OpenAI --> Enforce
Layer 1 is synchronous rule-based filtering on every message: slur lists, profanity in English and Spanish, solicitation patterns, homoglyph and leet-speak bypass detection. Profanity gets redacted. High-risk patterns (self-harm prompts, off-platform DM extraction, CSAM bypass phrases, genocide advocacy) get the message hidden. Some patterns trigger an immediate soft block.
Layer 2 is async. For users who aren't veterans, we enqueue a background job on the first five messages in a session, then every tenth message after that. A worker fetches the last ten messages and sends them to the OpenAI Moderation API (free at our scale). Results add to a persistent toxic score and can trigger bans even after a message already went out.
Layer 3 is human: trusted moderators, admin tools, user reports, and Discord alerts when auto-ban fires.
New visitors vs regulars
Not everyone in the room gets the same scrutiny. That's intentional.
New visitors (anyone in their first 20 chat messages on a nickname) face extra friction and stricter automated checks:
| Rule | New visitor | Regular / veteran |
|---|---|---|
| Wait before first message | 30 seconds after joining | None |
| Message length | 100 characters max | 2,000 characters |
| Send rate | Once every 10 seconds | Normal room limits |
| Room reactions | Not yet | Allowed |
| Local word filters | Strict: ambiguous terms, spam patterns | Lenient for earned veterans |
| Background ML checks | Yes, first 5 msgs, then every 10 | Skipped for veterans |
| Spam auto-blocks | Full enforcement | Relaxed for veterans |
The goal isn't punishment. It's risk management. A nickname that appeared thirty seconds ago is the highest-risk moment in anonymous chat: trolls, spammers, and creeps don't need a week of history to ruin a room. Slowing new accounts down gives filters and background checks time to work, and gives the room a beat to notice someone acting badly.
Veterans earn lighter treatment. Once a nickname has been seen on 3+ separate days and sent 30+ messages, they qualify for veteran moderation status:
- Lenient local filters: ambiguous terms and some spam heuristics that trip false positives on normal conversation are relaxed. Slurs, hidden-message patterns, and Tier-2 safety rules still apply. Trust is earned; zero-tolerance rules are not.
- No background ML queue: we stop enqueueing OpenAI checks for that nickname. Veterans have a track record; we'd rather not waste API calls (and false positives) on people who've been harmless for days.
- Relaxed spam rate limits: same idea. Repeat offenders still get caught; one-off mistakes don't auto-block as aggressively.
Veteran status follows the nickname, not the browser tab. Reconnect with a recovery code and your history comes with you. Spin up a fresh random name and you're a new visitor again, by design.
Trusted moderators are a small allowlisted set of volunteers. They skip automated filters entirely (so they can discuss moderation openly), can hard-block abusive nicks and IPs from chat, and help when automation isn't enough.
flowchart LR
New["New visitor\nfirst 20 messages"]
Vet["Veteran\n3+ days, 30+ msgs"]
Trusted["Trusted mod\nallowlisted"]
New -->|"earn trust"| Vet
Vet -->|"volunteer role"| Trusted
New --> N1["Strict filters\nML queue\nsend limits"]
Vet --> V1["Lenient filters\nno ML queue"]
Trusted --> T1["Skip filters\nmanual block"]
The toxic score
Every filtered, hidden, or flagged message adds points to a score stored on your session and nickname profile. Examples:
| What happened | Points |
|---|---|
| Profanity redacted | +1 |
| Slur detected | +2 extra |
| Message hidden (kys, DM prompt, etc.) | +5 |
| Hate / genocide advocacy | +10 |
| CSAM / minor content | +50 |
Scores decay over time: −5 every 7 days without a new hit. That helps one-off mistakes fade. It does not automatically unban someone already blocked.
When the score crosses thresholds:
- ≥ 50 → soft block (messages stored but not broadcast)
- ≥ 80 → hard block on the nickname
- CSAM / sexual-minors (any source) → immediate hard ban on nickname and IP, message delete, disconnect. No score accumulation required. One strike.
Soft block vs hard block
| Soft block | Hard block | |
|---|---|---|
| Can you reconnect? | Often yes, with a new session or nick | Nick blocked; IP rejected at connect |
| Do others see your messages? | No, stored as softBlocked |
Messages soft-deleted |
| Typical trigger | 3 filtered messages in one minute, or toxic score | Tier-2 content, repeat abuse, trusted mod action |
We prefer soft ban first for harassment-tier behavior. Shared IPs (school, café, mobile carrier) make blind IP bans risky for lower tiers. Tier 2 is different: IP block is required.
What we deliberately don't do
- No public toxic score UI. The number exists for enforcement, not shame.
- No DMs or whispers. Private channels are predator-friendly. Conversation stays in the shared room where others can witness and moderators can act.
- No local LLM on Lambda. Too slow, too expensive, wrong tool for our traffic. OpenAI's moderation endpoint is free and runs async.
- No blocking every borderline message synchronously. Sends stay snappy; retroactive delete + ban handles what filters miss on the first pass.
Reports, admins, and alerts
Users can report a nickname from chat. Trusted mods can /block abusive accounts (nick + all known IPs). Admins have HTTP tools for nick and IP blocks. When auto-ban fires, a Discord webhook pings the team with the display name, score, tags, and reason, enough context to review without reading the whole room log.
The honest tradeoff
Stricter rules for new visitors means a regular who makes a new alt for fun gets treated like a stranger again. Lenient rules for veterans means we accept a small amount of false-negative risk on people with a clean track record. Tier-2 rules mean we accept shared-IP collateral when the alternative is leaving CSAM content in the room.
Anonymous chat will never be perfectly fair and perfectly safe at the same time. We're optimizing for safe enough that vulnerable people stay, and light enough that regulars don't leave.
If you're new: the first few messages are supposed to feel a little constrained. That's the room checking you out, same as you'd eye a stranger who just walked in.
If you've been around: we see the days and the message count. We trust you more, not with a free pass, but with fewer false alarms.
That's the moderation system. Layers, scores, async checks, and different rules for different trust levels, all in service of one campfire that doesn't burn the people sitting closest to it.
Comments
Loading comments…