Skip to content

Refusal detection is five English regexes, and a polite refusal in any other wording reads as a crack attempt #4

Description

@royalpinto007
const REFUSAL_HINTS = [
  /\bi can'?t\b/i,
  /\bi'?m not able\b/i,
  /\bdenied\b/i,
  /\bstays shut\b/i,
  /\bnice try\b/i,
];

Five phrases. Models refuse in dozens of ways, and the wording shifts between models and between versions of the same model: "I won't", "that's not something I can share", "I'd rather not", "Sorry, no", "I'm afraid not", or just changing the subject.

A refusal the list misses gets classified as something else, which affects both what the player is shown and, depending on how the verdict is used, the scoring.

Two directions worth discussing on the issue

  1. Widen the list. Cheap, and it will always lag whatever model is plugged in next.
  2. Invert the test. The question that matters is whether the secret leaked, which the judge already answers directly. Treat everything that is not a leak as "held", and use refusal phrasing only to make the message nicer rather than to decide anything.

Option 2 is probably the right shape for a game whose scoring should not depend on a model's turn of phrase, but it is worth agreeing before someone writes it.

Acceptance

  • Whichever direction is chosen, a test covers half a dozen refusal wordings not currently in the list
  • A response that refuses in unusual wording and leaks nothing is never scored as a crack

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions