Skip to content

feat(ai-aws-content-moderation): add request_check_roles and request_check_mode - #13773

Open
shreemaan-abhishek wants to merge 4 commits into
apache:masterfrom
shreemaan-abhishek:feat/aws-cm-request-check-roles
Open

feat(ai-aws-content-moderation): add request_check_roles and request_check_mode#13773
shreemaan-abhishek wants to merge 4 commits into
apache:masterfrom
shreemaan-abhishek:feat/aws-cm-request-check-roles

Conversation

@shreemaan-abhishek

Copy link
Copy Markdown
Contributor

Description

ai-aws-content-moderation moderates the request side by calling extract_request_content, which takes every message of every role and scores the whole conversation on every turn. ai-aliyun-content-moderation has had role and turn selectivity since #13646; this brings the same options to the AWS plugin.

  • request_check_roles (array, default ["user","tool","system"]) selects which message roles are moderated. user/tool follow request_check_mode; system is moderated on every request because it can be poisoned by malicious ToolCall arguments overwriting the system prompt.
  • request_check_mode (last / all, default all) limits user/tool moderation to the latest consecutive block of selected-role messages, so history is not re-scored (and re-billed to Comprehend) each turn.

All selected roles' text is collected and scored in a single Comprehend call: DetectToxicContent takes a flat list of text segments with no role field, so per-role calls would only cost extra requests.

No new extraction code is needed — extract_turn_content(body, mode, roles) and extract_system_content(body) already exist on every protocol adapter from #13646.

Defaults preserve current coverage rather than mirroring the aliyun plugin's ["user"] + last: this plugin moderates everything today, and a narrower default would silently moderate less after an upgrade. Selectivity is opt-in. Assistant content is the one thing that drops out, since no extractor collects it (it is the LLM's own prior output, echoed back by the client).

One behavior change beyond the new options: when the active protocol cannot extract a configured role (passthrough implements neither extractor), the request goes through binding.on_unsupported so fail_mode decides, instead of silently passing the content unmoderated. With the default fail_mode: skip nothing changes.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change (t/plugin/ai-aws-content-moderation.t TEST 36-49: default role/mode coverage, last vs all, role selection, schema validation, and the unsupported-protocol fail_mode path)
  • I have updated the documentation (docs/en/latest/plugins/ai-aws-content-moderation.md and the zh translation)
  • I have verified locally: prove -I. t/plugin/ai-aws-content-moderation.t — all 49 tests pass; make lint clean

…check_mode

Bring the request-side selectivity of ai-aliyun-content-moderation to the
AWS plugin: `request_check_roles` picks which message roles are moderated
(user/tool/system) and `request_check_mode` limits user/tool moderation to
the newest turn instead of re-scoring the whole conversation on every
request. system is always moderated when enabled, since it can be poisoned
by malicious ToolCall arguments.

Defaults keep today's coverage: all roles, every message. A protocol that
cannot extract a configured role now goes through fail_mode instead of
silently passing the content unmoderated.
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request plugin labels Aug 3, 2026
…-check-roles

# Conflicts:
#	apisix/plugins/ai-aws-content-moderation.lua
#	docs/en/latest/plugins/ai-aws-content-moderation.md
nic-6443
nic-6443 previously approved these changes Aug 4, 2026
AlinsRan
AlinsRan previously approved these changes Aug 4, 2026

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 — must be fixed before merge

The previous default path used proto.extract_request_content(), which moderated text from every message role. This change replaces it with request_check_roles = ["user", "tool", "system"]; the schema rejects assistant, so an unchanged configuration silently stops moderating assistant-role content and cannot restore the old coverage.

Client-supplied assistant history is still part of the LLM context, so harmful content placed only in an assistant message can bypass request-side moderation. This also conflicts with the stated goal that the defaults preserve existing coverage.

Please preserve assistant-role moderation by default—either make assistant configurable and enabled by default, or retain the legacy all-role extraction path for the default configuration—and add a regression test proving that toxic assistant-only content is still sent to Comprehend and rejected.

…fault

Client-supplied assistant history is part of the LLM context, so harmful
content placed only in an assistant message must not bypass request-side
moderation. Adds assistant to request_check_roles and to its default, which
restores the coverage the previous extract_request_content path had.
@shreemaan-abhishek
shreemaan-abhishek dismissed stale reviews from AlinsRan and nic-6443 via bf240a3 August 4, 2026 09:57
@shreemaan-abhishek

Copy link
Copy Markdown
Contributor Author

@membphis you're right, and thanks for catching it — an unchanged config silently losing assistant-role coverage was not the intent, and the enum made it unrecoverable.

Fixed in bf240a3 by putting assistant back in scope by default rather than only making it configurable:

  • request_check_roles enum is now user / tool / system / assistant, and the default is ["user","tool","system","assistant"], so a config that sets nothing moderates the same roles the old extract_request_content path did.
  • The role set is no longer hardcoded to user/tool: everything except system is now routed through extract_turn_content, which already keys on roles[message.role] in all four protocol adapters, so no change was needed under ai-protocols/.

Test coverage, in t/plugin/ai-aws-content-moderation.t:

  • TEST 55 is now the regression test you asked for: a request whose only toxic text sits in an assistant message is sent to Comprehend (asserted via grep_error_log on the moderated text) and denied with 400.
  • TEST 62 pins the schema: ["assistant"] accepted, unknown roles rejected.
  • TEST 65/66 pin one consequence worth knowing: with request_check_mode: last, selecting assistant widens the moderated block instead of narrowing it, because assistant turns no longer terminate the trailing block. Documented in the attribute table; drop assistant from the roles to moderate only the newest turn.

One related gap this surfaced, which I've filed rather than folded in here: extract_system_content matches role == "system" exactly, so OpenAI's newer developer role is moderated by neither this plugin nor ai-aliyun-content-moderation. It needs a change in the protocol adapters and affects both plugins, so it deserves its own PR.

Local run: all 66 tests in t/plugin/ai-aws-content-moderation.t pass, make lint clean.

@membphis membphis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Include the developer role in the default moderation set

The fixed default and schema enum contain user, tool, system, and assistant, but omit the OpenAI developer role. After upgrading, developer messages can therefore bypass request moderation silently, and users cannot add that role explicitly.

Please treat developer like system by default, or preserve the previous role-agnostic catch-all; allow it in the schema; and add regressions for both Chat Completions and Responses inputs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request plugin size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants