Skip to content

VahterBanBot: cap ephemeral spam warning at SPAM_WARNING_MAX_SCORE (default 3.0) - #396

Merged
Szer merged 1 commit into
mainfrom
vahter-spam-warning-score-cap
Aug 19, 2026
Merged

VahterBanBot: cap ephemeral spam warning at SPAM_WARNING_MAX_SCORE (default 3.0)#396
Szer merged 1 commit into
mainfrom
vahter-spam-warning-score-cap

Conversation

@Szer

@Szer Szer commented Aug 19, 2026

Copy link
Copy Markdown
Owner

Why

The ephemeral spam-deletion warning (#395) currently fires for every MlSpam/LlmSpam/ContentFilterSpam deletion, including blatant, high-confidence spam -- per vahter feedback, warning obvious spammers just teaches them detection exists without protecting anyone. Prod-data research (binding, see PR discussion) shows a 3.0 score cutoff keeps 93.8% of ham-marked (false-positive) deletions warned while cutting warned-spammer volume from 100% to 57% (a 43-point reduction). LlmSpam/ContentFilterSpam scores are < 1.5 by construction (LLM band), so they pass a 3.0 cutoff naturally -- no reason-specific special-casing needed.

What changed

  • New setting SPAM_WARNING_MAX_SCORE -> BotConfiguration.SpamWarningMaxScore: float, default 3.0, wired via getSettingOr exactly like the other SPAM_WARNING_* settings (src/VahterBanBot/Types.fs, src/VahterBanBot/Program.fs). Read via IOptions<BotConfiguration>.Value at call time (never captured), so POST /reload-settings picks it up live.

  • DeleteSpam (src/VahterBanBot/Bot.fs) now extracts the score carried by the warned reasons and only sends the warning when score < botConfig.Value.SpamWarningMaxScore (strict). All other VahterBanBot: ephemeral warning on spam auto-deletion (flag off by default) #395 gating (flag on, not just-banned, reason whitelist) is unchanged:

    let warnedReasonScore =
        match reason with
        | AutoDeleteReason.MlSpam x -> Some x.score
        | AutoDeleteReason.LlmSpam x -> Some x.score
        | AutoDeleteReason.ContentFilterSpam x -> Some x.score
        | AutoDeleteReason.ReactionSpam _ | AutoDeleteReason.InvisibleMention | AutoDeleteReason.SpamTextCacheHit _ -> None
    match warnedReasonScore with
    | Some score when score < botConfig.Value.SpamWarningMaxScore ->
        do! tg.CallIgnore(Req.SendMessage.Make(msg.ChatId, botConfig.Value.SpamWarningText, receiverUserId = msg.SenderId))
        recordSpamWarningSent msg.ChatId msg.ChatUsername
    | Some _ | None -> ()

Enable / tune (hand-run, NOT a Flyway seed -- see AGENTS.md's "Settings seeds, not migrations")

INSERT INTO bot_setting (key, value, type, feature_group, description)
VALUES (
    'SPAM_WARNING_MAX_SCORE',
    '3.0',
    'FREE_FORM',
    'SPAM_WARNING',
    'Ephemeral spam-deletion warning is only sent when the ML/LLM score is strictly below this cutoff (the likely-false-positive band); at/above it the deletion is silent. Default 3.0 is prod-data-derived (93.8% ham-deletion coverage, 43-point spammer-warning reduction at this cutoff) -- needs periodic review since the ML score scale drifts under daily retraining.'
)
ON CONFLICT (key) DO UPDATE SET
    value = EXCLUDED.value,
    type = EXCLUDED.type,
    feature_group = EXCLUDED.feature_group,
    description = EXCLUDED.description,
    updated_at = NOW();

Then POST /reload-settings to pick it up without a redeploy. Missing the row is safe -- getSettingOr falls back to the code default of 3.0.

Tests (tests/VahterBanBot.Tests/SpamWarningTests.fs)

  • Existing happy-path ("ML spam deletion of a non-banned user sends exactly one ephemeral warning") verified still valid: "2222222" scores 1.5686... under the fixture ML model (see MLScoreDeterminismTests), comfortably below the default 3.0 cutoff -- no change needed, just a doc comment.
  • New: ML deletion scoring at/above SPAM_WARNING_MAX_SCORE (cutoff pinned just below a probed score) -> deletion happens, no warning.
  • New: boundary -- score exactly equal to SPAM_WARNING_MAX_SCORE (cutoff pinned to the exact probed score) -> no warning, proving the comparison is strict <.
  • SpamWarningMlTests.DisposeAsync now also resets SPAM_WARNING_MAX_SCORE to "3.0" between tests.

Test run

dotnet build src/VahterBanBot -c Release and dotnet build tests/VahterBanBot.Tests -c Release both passed clean (0 warnings/errors) locally. The local dotnet test tests/VahterBanBot.Tests -c Release run hit a local container-startup environment failure (fixture MlEnabledVahterTestContainers never produced an app.log, so every HTTP call in the suite got Connection refused against the bot's own webhook port) rather than a test-logic failure -- deferred to CI per owner direction rather than re-running locally.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Wi7gKmshkHfVtSB3tA4gmJ

…efault 3.0)

Restricts the PR #395 ephemeral spam-deletion warning to low-score (likely
false-positive) deletions: MlSpam/LlmSpam/ContentFilterSpam all carry a score,
warn only when score < SpamWarningMaxScore (strict, bot_setting-backed since
the ML score scale drifts under daily retraining). Prod data: at the 3.0
default, 93.8% of ham-marked deletions still warn while warned-spammer volume
drops from 100% to 57%.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Wi7gKmshkHfVtSB3tA4gmJ
@Szer
Szer marked this pull request as ready for review August 19, 2026 11:24
@Szer
Szer merged commit 7d0f2a3 into main Aug 19, 2026
4 checks passed
@Szer
Szer deleted the vahter-spam-warning-score-cap branch August 19, 2026 11:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant