Content moderation settings
Set up content moderation for an app in the console: maintain word lists, connect a third-party moderation service, set up automatic penalties, and handle moderation records and user reports. For what content is checked, when, and how it affects APIs and clients, see Content moderation.
Content moderation lives in two places:
- The "Content moderation" tab of app details, for per-app settings and handling, with sub-tabs "Settings", "Word lists", "Try it", "Moderation records", "Platform actions", "User violations" and "Stats";
- "Moderation accounts" in the left menu, which manages the tenant's third-party moderation accounts, shared by all apps under the tenant.
Who can use it
Moderation records, reports and violations involve users' chat content, so they're only open to owner and admin:
| Action | owner | admin | developer |
|---|---|---|---|
| View settings, word lists and stats; use "Try it" | ✓ | ✓ | ✓ |
| Change settings; create, edit and delete word lists; add and delete words | ✓ | ✓ | |
| View, add, edit, disable, delete and verify moderation accounts | ✓ | ✓ | |
| View moderation records and reports, view content, make decisions, escalate to the platform, view platform actions | ✓ | ✓ | |
| View and clear users' violation records | ✓ | ✓ |
developer only sees "Settings", "Word lists", "Try it" and "Stats" under "Content moderation", read-only, and has no "Moderation accounts" in the left menu.
Adding, editing (including disabling and enabling) and deleting moderation accounts require identity verification; verifying an account doesn't. Viewing a moderation record's content needs no identity verification, but every view is written to the operation log.
Getting started
A new app starts like this: platform rules apply; your rules are on for all scenarios but there are no word lists yet, so nothing is blocked; nothing is sent to third-party moderation; no automatic penalties; client reports are allowed. A typical start:
- Create a word list under "Word lists" and add words; see Word lists;
- Enter a few pieces of text in "Try it" and check the matches and replacements are what you expect;
- To check images, voice and video, or to check text more accurately, add an account under "Moderation accounts", then choose it for the app under "Settings" and turn on third-party moderation for the scenarios you need; see Third-party moderation;
- Add automatic penalty rules under "Settings" if needed;
- Day to day, handle content sent for review and user reports under "Moderation records".
Settings
"Settings" shows the app's current moderation settings, which owner and admin can change. Click "Save" after changing them; servers usually apply them within 1 second, 30 seconds at most. Click "Discard changes" to drop unsaved changes. If saving says the data was changed by someone else, someone changed the settings after you opened the page; the page reloads the latest settings, so make your changes again.
Platform settings
"Platform settings (read-only)" shows what the platform set for this app, which you can't change:
| Item | Description |
|---|---|
| Platform rules | The platform word lists and the categories the platform enforces apply to all content in the app; matching content is reviewed and handled by the platform |
| Platform account's daily checks | The most third-party checks the platform does for this app per day, 0 for unlimited. Beyond that, only word lists are checked for the rest of the day |
| Categories the platform enforces | Categories in third-party results handled by platform rules, by default politics, violence and terrorism, and pornography |
| Third-party moderation the platform does for this app | The scenarios and modes the platform checks for this app with its own account. "None" means only the platform word lists are used |
Checks with the platform's account count toward the app's usage and are billed by plan; see Metered items. Checks with your own moderation account don't count; you settle those with the provider.
Checks by scenario
One row per scenario; for what each scenario means, see What is checked and when:
| Column | Description |
|---|---|
| Tenant rules | When off, this scenario is only checked against platform rules; your word lists and third-party moderation are no longer used for it |
| Send text to third-party moderation | "Don't send" or "After": checked after sending or saving, and violations are recalled or cleared. Uploaded files and chat room attributes are only checked against word lists and show "Not supported" |
| Send media to third-party moderation | How images, voice and video are checked: "After" is used for messages, chat room messages, uploaded files and avatars in user and group profiles |
| Also check server-submitted content | By default, content your server (OpenAPI) and the console submit is only checked against platform rules; when on, it's also checked against your rules; see Server-submitted content. Not supported for chat room attributes |
| Platform third-party moderation | The third-party moderation the platform does for this scenario, read-only |
Third-party moderation
| Item | Description |
|---|---|
| Moderation account | The tenant moderation account this app uses. "Don't use a tenant account" uses only the third-party moderation the platform turned on for this app. If any scenario has third-party moderation on, an account must be chosen, or saving says "Third-party moderation requires a moderation account of this tenant". Disabled accounts can't be chosen |
Third-party checks only run after content is sent or saved, so sending and saving never wait for the provider. When the platform account's per-second quota runs out, check tasks queue up; once the daily checks run out, content is only checked against word lists for the rest of the day.
The "Handling third-party results (by category)" table sets how third-party results are handled per category:
| Column | Options |
|---|---|
| When it suggests blocking | "Reject" (default): sync checks refuse to write it, and after-the-fact checks automatically recall or block it and count a violation for the author; "Review": written as usual and added to moderation records for manual review |
| When it suggests manual review | "Review" (default); "Ignore": written as usual with no moderation record |
Categories marked "Platform enforced" are handled by platform rules, and these settings don't apply to them.
Custom messages
The body of custom messages (custom) is defined by you. With "Check all strings in body" (default), every string value in it is checked; with "Don't check", custom messages get no text check. ext is never checked.
Automatic penalties
Click "Add rule" to add an automatic penalty rule, up to 5; with no rules there are no automatic penalties. Each rule has:
| Item | Description |
|---|---|
| Time window | "Within N hours", 1 to 720 hours |
| Violations | "N violations", 1 to 1000 |
| Categories | Check "All categories" to count all violations; otherwise only the chosen categories count |
| Penalty | "Mute": choose one or more of one-to-one chats, groups and chat rooms; "Ban" |
| Duration | A duration or "Permanent", up to 10 years |
After each violation is recorded, each rule counts the violations in its time window, and applies when the count is reached and the rule hasn't already penalized this user in this window. Penalties are applied as your app and can be lifted under "IM users"; see Users, groups, chat rooms and messages. For what counts as a violation, see Violation records and automatic penalties.
Word lists
"Word lists" lists all the app's word lists. All members can view lists and words; owner and admin can click "New word list", and enable, disable, edit and delete lists in the list. Each app can have up to 20 word lists.
When creating a list, fill in:
| Field | Description |
|---|---|
| Name | 1 to 64 characters, such as "Ad spam" |
| Kind | "Block list" or "Allow list", can't be changed after creation. Allow words clear false positives: a blocked word that falls entirely inside a longer allowed word doesn't count |
| On match | How a block list handles matches: "Reject", "Replace with *" or "Review" |
| Match mode | "Contains": matches if the word appears in the text; "Exact": matches only if the whole text, after normalization, equals the word, for reserved nicknames (such as "Official support"). Allow lists can only use "Contains" |
| Category | The category recorded in moderation records and violations on a match |
| Enabled | Disabled lists aren't matched; their words are kept |
| Scenarios | Check "All scenarios", or choose one or more scenarios |
Click a list's name or "Words" to open its words:
- Word is the normalized form; Original is the text as submitted. You can search by normalized prefix;
- Bulk add: one word per line, up to 1000 at a time. Afterwards it shows the counts and words that were added, duplicated (equal to an existing one after normalization) and invalid (too short, too long, or containing control characters). Block words are 2 to 32 characters after normalization, allow words 2 to 64;
- Bulk delete: one word per line, either original or normalized; or check words in the list and click "Delete selected".
A list can hold up to 10,000 words, and an app up to 50,000 in total. For normalization rules and how allow words work, see Word lists. Deleting a list deletes its words too, and they can't be recovered.
Try it
"Try it" checks a piece of text against all of the app's current rules, so you can verify right after editing word lists. It creates no moderation records or violations and doesn't count in stats. All members can use it, up to 60 times a minute per console account.
| Item | Description |
|---|---|
| Scenario | Checks with the word lists and settings that apply to this scenario |
| Text, or several fields | Checks one piece of text, or up to 10 fields at once (such as a group profile's name, description and announcement) |
| Also call third-party moderation | Calls third-party moderation per this scenario's settings; scenarios without it aren't called. Calls count toward the provider's usage and fees |
The result shows pass, reject or replace (with the replaced text), the third-party suggestion and category, and the matched word lists, categories, handling, matched words and positions. Matches from platform word lists only show "Platform rules" and the category, not the matched words.
Moderation accounts
"Moderation accounts" in the left menu manages the tenant's own third-party moderation accounts; only owner and admin can view and manage them. Accounts are shared by all apps under the tenant and are only used once chosen in an app's "Content moderation → Settings". You pay the provider directly. Each tenant can have up to 5 accounts.
Click "Add account" and fill in:
| Field | Description |
|---|---|
| Name | 1 to 64 characters, such as "Aliyun-prod" |
| Provider | One of the providers in the list, can't be changed after creation |
| Region | Determines the address used to call the provider; options in the table below. The service address is set by provider and region and can't be entered yourself |
| Max calls per second | 1 to 1000, 20 by default; fill in the quota the provider gave you |
| AccessKey ID | The provider's access key ID, up to 128 characters |
| AccessKey Secret | The provider's access key |
| Other provider settings | Provider-specific non-sensitive settings as keys and values; available keys in the table below |
| Provider | Region | Access key | Other settings |
|---|---|---|---|
| Alibaba Cloud Content Moderation | cn-shanghai, cn-beijing, cn-hangzhou, cn-shenzhen, cn-chengdu, ap-southeast-1 | AccessKey ID and AccessKey Secret | Optional text_service, image_service, audio_service, video_service: replace the default services, such as a service you copied in the Alibaba Cloud console. Defaults: text chat_detection (messages and chat room messages) and comment_detection (other scenarios), image baselineCheck, voice audio_media_detection, video videoDetection |
| Tencent Cloud Tianyu | ap-guangzhou, ap-shanghai, ap-beijing, ap-singapore | SecretId and SecretKey | Optional biz_type (a policy shared by all check types), and text_biz_type, image_biz_type, audio_biz_type, video_biz_type (replace biz_type per type); with none, Tencent Cloud's default policy is used |
| NetEase Yidun | Empty or cn-hangzhou | SecretId and SecretKey | The BusinessId of each check type in the Yidun console: text_business_id and image_business_id are required; audio_business_id and video_business_id too if you use voice and video |
Other settings' values are 1 to 64 letters, digits or underscores (Yidun BusinessIds also allow hyphens).
Before saving, the provider is called once with a fixed harmless text; if that fails it isn't saved, and the provider's error code is shown. Access keys are stored encrypted and never shown again on any page or API; the list shows only the first and last 4 characters of the AccessKey ID.
In the account list you can:
- Edit: change the name, region, calls per second and other settings. Leaving the AccessKey ID and secret empty means unchanged; filling in either re-verifies with the provider;
- Verify: call the provider once with a fixed harmless text; the result shows in the "Last verified" column. Problems hit while calling the provider, such as a wrong key or insufficient balance, also show in this column;
- Disable, enable: once disabled, apps using it treat third-party moderation as an unavailable account: no checks are made;
- Delete: the key is deleted too and can't be recovered. Accounts still used by apps under "Used by" can't be deleted; switch those apps to another account or "Don't use a tenant account" first.
Moderation records
"Moderation records" lists the app's content that needs manual handling: matches sent for review, content automatically handled by after-the-fact checks, and user reports. By default only open records (pending review, auto-handled) are listed, by priority, then newest first.
Filter by status, scenario, target, category, source (word list, third-party moderation, report), author's username and creation time; check "Oldest first" to sort oldest to newest when clearing a backlog; check "Escalated to platform" to list records this app escalated to the platform, whose status you can only view.
| Column | Description |
|---|---|
| Priority | "Highest": includes a category the platform enforces, or a report for fraud, politics, violence and terrorism, or minors; "High": auto-handled, or reported by 3 or more people; the rest are "Normal" |
| Status | "Pending review", "Auto-handled" (awaiting confirmation), "Violation", "No violation", "Closed". For closed records you can see why: not handled for over 30 days, merged into a platform record, or target deleted |
| Scenario, target | The content checked, such as a message, user profile, group nickname or file; report targets also include users, groups and chat rooms |
| Author | The user who posted the content. Content your server submitted and reported groups and chat rooms have no author |
| Category, source, reports | The matched category, the source (word list, third-party moderation, report, URL can't be checked, escalated to platform), and the number of reporters |
Each target has at most one open record at a time; later matches and reports all go to that record.
View content
Lists and records don't include users' content. Click a row to open the record, then click "View content" in "Content and reports" to load the content and reports:
- The content is a snapshot from when it was first found. If the message was later recalled or the profile changed, you still see the original; if the message was edited or the profile changed later, those later versions are listed separately;
- Reports list the reporter, reason, description, evidence messages and how it was reported (from the client, or by your server on the user's behalf);
- Images, voice and video get view URLs valid for 10 minutes. External URLs submitted by users aren't loaded automatically, only opened after you confirm, and the other server then sees your IP address. Files blocked by platform rules can't be viewed;
- Every view is written to the operation log (
moderation.content_view), up to 120 views per account per minute; - Content is cleared 30 days after the record closes; only the record and actions can be viewed then.
Make a decision
Under "Make a decision" in the record, choose:
- Confirm violation: you can choose actions at the same time, and whether to "Count as the author's violation" (yes by default; you can skip it if the author's account was stolen, for example);
- No violation: closes the record.
Available actions:
| Target | Actions |
|---|---|
| Messages, chat room messages | Recall the message, recall the chat room message; block files (by default all of this app's files in the message, or enter file IDs) |
| Files | Block the file |
| User profiles, group nicknames, reported users | Clear profile: choose the fields to clear; with none chosen, all checked fields are cleared |
| Group profiles, reported groups | Clear profile, ban the group |
| Chat room profiles, reported chat rooms | Clear profile, ban the chat room |
| Chat room attributes | Delete the chat room attribute |
| Content with an author, reported users | Mute the author (choose scope and duration), ban the author (choose duration) |
Notes and reasons on group join requests and friend requests can't be cleared; you can only mute or ban the author. Clearing a profile only clears fields still holding the checked value; fields the user changed since aren't affected. Actions run after you submit; refresh shortly afterwards to see results under "Actions from this record": pending, done, skipped (such as when the target was deleted or the message already recalled) or failed.
Confirming a violation on an auto-handled record doesn't repeat the actions or count the violation again; it only runs the extra actions you chose this time.
Changing a decision: within 30 days of confirming a violation, you can change it to no violation. This deletes the violations recorded under your rules from this record and unblocks the files it blocked. Recalled messages can't be restored, and mutes and bans already applied aren't lifted automatically; lift them by hand under "IM users". Records decided as no violation can't be changed to violation; wait for a new match or report if there's a new violation.
Bulk decisions: check records on the current page that have no decision yet and click "Bulk decide", up to 100 at a time, for handling a wave of spam ads. Bulk decisions can only choose content actions (recall, block files, clear profiles, delete attributes), not mutes or bans; each record is handled separately, and records someone else already decided fail.
When two people handle a record at the same time, the later submission says the record was changed and shows its latest status.
Escalate to the platform
When content may be illegal and needs the platform to handle it, click "Escalate to platform" in the record, optionally with a note. The platform's reviewers then handle the record; you can see its status under "Escalated to platform", but can no longer decide it. Platform reviewers may also handle this app's moderation records directly, and only the platform can change decisions it made.
Platform actions
"Platform actions" lists actions taken in this app as the platform after platform rules matched: recalled messages, blocked files, bans and so on, showing the target, category, action, status and time, without content. The platform keeps evidence of messages it recalled, and you can't lift files the platform blocked or its bans.
User violations
Enter a username under "User violations" to look up:
- Violation count: the user's current number of violations, used by automatic penalty rules;
- Recent penalties: mutes and bans from automatic penalties and manual decisions, with scope, duration, reason and status;
- Violation records: each violation's time, scenario, category, source (rejected by sync check, auto-handled after the fact, manually confirmed), rule (platform or tenant) and matching moderation record.
Click "Clear tenant-rule violations" to delete all of this user's violations recorded under your rules, lowering the count accordingly, for example after confirming their account was stolen. Platform-rule violations are kept, and mutes and bans already applied aren't lifted. Violation records are kept for 180 days.
Stats
"Stats" shows content moderation figures by day and scenario, up to 90 days at a time, filterable by scenario. Dates follow Beijing time, and figures lag by at most 5 minutes.
| Metric | Description |
|---|---|
| Checks | Number of checks |
| Rejected, replaced, sent for review | Number of checks resulting in reject, replace or review |
| Auto-handled | Number of automatic recalls or blocks by after-the-fact checks |
| Third-party checks | Number of third-party calls, with those using the platform account in parentheses |
| Cache hits | Images, voice and video matching an existing result without calling the provider |
| Check failures | Number of failed third-party calls |
| Sync timeouts | Deprecated, always 0: third-party checks are no longer synchronous |
| Over quota, not checked | Checks that only used word lists because the per-minute after-the-fact rate or the platform account's daily checks were exceeded |
| Reports | Number of reports |
The top of the page also shows the number of records pending review and how long the oldest has waited (excluding records escalated to the platform).
Operation log
Content moderation actions are written to the operation log:
| Action type | Description |
|---|---|
moderation.config_update | Change settings |
moderation.word_list_create, moderation.word_list_update, moderation.word_list_delete | Create, edit, delete word lists |
moderation.words_add, moderation.words_remove | Add, delete words; the change lists the words |
moderation.provider_account_create, moderation.provider_account_update, moderation.provider_account_delete, moderation.provider_account_verify | Add, edit, delete, verify moderation accounts, without keys |
moderation.content_view | View a moderation record's content; only which record is recorded |
moderation.item_decide, moderation.item_escalate | Make a decision, escalate to the platform |
moderation.violations_clear | Clear a user's violations |
Automatic actions and automatic penalties are logged as the system by the features that carry them out, such as recalling messages or banning users; for actions from manual decisions, the operator is the person who made the decision.