feat(ai-aws-content-moderation): add request_check_roles and request_check_mode - #13773
feat(ai-aws-content-moderation): add request_check_roles and request_check_mode#13773shreemaan-abhishek wants to merge 4 commits into
Conversation
…check_mode Bring the request-side selectivity of ai-aliyun-content-moderation to the AWS plugin: `request_check_roles` picks which message roles are moderated (user/tool/system) and `request_check_mode` limits user/tool moderation to the newest turn instead of re-scoring the whole conversation on every request. system is always moderated when enabled, since it can be poisoned by malicious ToolCall arguments. Defaults keep today's coverage: all roles, every message. A protocol that cannot extract a configured role now goes through fail_mode instead of silently passing the content unmoderated.
…-check-roles # Conflicts: # apisix/plugins/ai-aws-content-moderation.lua # docs/en/latest/plugins/ai-aws-content-moderation.md
membphis
left a comment
There was a problem hiding this comment.
P1 — must be fixed before merge
The previous default path used proto.extract_request_content(), which moderated text from every message role. This change replaces it with request_check_roles = ["user", "tool", "system"]; the schema rejects assistant, so an unchanged configuration silently stops moderating assistant-role content and cannot restore the old coverage.
Client-supplied assistant history is still part of the LLM context, so harmful content placed only in an assistant message can bypass request-side moderation. This also conflicts with the stated goal that the defaults preserve existing coverage.
Please preserve assistant-role moderation by default—either make assistant configurable and enabled by default, or retain the legacy all-role extraction path for the default configuration—and add a regression test proving that toxic assistant-only content is still sent to Comprehend and rejected.
…fault Client-supplied assistant history is part of the LLM context, so harmful content placed only in an assistant message must not bypass request-side moderation. Adds assistant to request_check_roles and to its default, which restores the coverage the previous extract_request_content path had.
bf240a3
|
@membphis you're right, and thanks for catching it — an unchanged config silently losing assistant-role coverage was not the intent, and the enum made it unrecoverable. Fixed in bf240a3 by putting
Test coverage, in
One related gap this surfaced, which I've filed rather than folded in here: Local run: all 66 tests in |
membphis
left a comment
There was a problem hiding this comment.
[P1] Include the developer role in the default moderation set
The fixed default and schema enum contain user, tool, system, and assistant, but omit the OpenAI developer role. After upgrading, developer messages can therefore bypass request moderation silently, and users cannot add that role explicitly.
Please treat developer like system by default, or preserve the previous role-agnostic catch-all; allow it in the schema; and add regressions for both Chat Completions and Responses inputs.
Description
ai-aws-content-moderationmoderates the request side by callingextract_request_content, which takes every message of every role and scores the whole conversation on every turn.ai-aliyun-content-moderationhas had role and turn selectivity since #13646; this brings the same options to the AWS plugin.request_check_roles(array, default["user","tool","system"]) selects which message roles are moderated.user/toolfollowrequest_check_mode;systemis moderated on every request because it can be poisoned by malicious ToolCall arguments overwriting the system prompt.request_check_mode(last/all, defaultall) limitsuser/toolmoderation to the latest consecutive block of selected-role messages, so history is not re-scored (and re-billed to Comprehend) each turn.All selected roles' text is collected and scored in a single Comprehend call:
DetectToxicContenttakes a flat list of text segments with no role field, so per-role calls would only cost extra requests.No new extraction code is needed —
extract_turn_content(body, mode, roles)andextract_system_content(body)already exist on every protocol adapter from #13646.Defaults preserve current coverage rather than mirroring the aliyun plugin's
["user"]+last: this plugin moderates everything today, and a narrower default would silently moderate less after an upgrade. Selectivity is opt-in. Assistant content is the one thing that drops out, since no extractor collects it (it is the LLM's own prior output, echoed back by the client).One behavior change beyond the new options: when the active protocol cannot extract a configured role (
passthroughimplements neither extractor), the request goes throughbinding.on_unsupportedsofail_modedecides, instead of silently passing the content unmoderated. With the defaultfail_mode: skipnothing changes.Checklist
t/plugin/ai-aws-content-moderation.tTEST 36-49: default role/mode coverage,lastvsall, role selection, schema validation, and the unsupported-protocolfail_modepath)docs/en/latest/plugins/ai-aws-content-moderation.mdand the zh translation)prove -I. t/plugin/ai-aws-content-moderation.t— all 49 tests pass;make lintclean