Enhance content moderation and admin panel features - #109
Open
joan-code6 wants to merge 17 commits into
Open
Conversation
Author
|
btw the contact mahad on slack thingy redierects to slack.hackclub.com which allows user which dont have slack to still reach you / join slack and get a tutorial |
Author
Add content_violations table with type, category, content, and status fields. Add user_flag_settings table for opt-in forced review. Add reviewStatus, violation counts, and lastViolationAt columns to users table. Add content_hash, dismissed, and violation_event_id columns to content_violations. Remove obsolete pending_charges table.
Make moderation API keys optional. Add review sampling rate, violation thresholds (weekly/monthly/strict), internal API key, and admin credentials with defaults.
Add moderation module for classifying input/output content via OpenAI moderation API, with category mapping and hash-based caching.
Implement tiered review system: normal users sampled at configured rate, flagged users always reviewed, and strict mode blocks until review passes. Track weekly/monthly violation counts and automatically escalate status. Support forced review opt-in for apps exposed to public users.
Remove pending charges reservation system. Query request logs directly for daily spending calculation. Default limit raised to $8. Add /admin route registration. Redirect banned users to /banned page. Improve auth middleware null safety.
Add input/output moderation to general chat, moderation, and OCR proxies. Add sample-based review triggering and strict-mode content checking. Add violations endpoint with manual/unban/override support on /internal. Add content review endpoint for moderation events on /api.
Add moderation page with violation history, review mode toggle, and stats. Update header navigation, dashboard cards, and activity logs with moderation status indicators. Simplify auth login redirect logic. Add forced review opt-in endpoint on auth routes.
Add basic auth-protected admin panel with dashboard overview of violation stats. Add user search, detail view with violation history, ban/unban controls, and status management. Add violation review interface for admins to dismiss or confirm flagged content.
Add dedicated /banned page showing violation history, appeal information, and next steps for banned users instead of a plain 403 error.
Export Violation type. Remove pending charges references from AppVariables. Remove unused blocked prompts config. Simplify DashboardRequestLog type and models list. Remove unused stats tracking.
skyfallwastaken
requested changes
Jun 30, 2026
Comment on lines
+34
to
+36
| // Admin panel credentials (defaults) | ||
| ADMIN_USERNAME: "string = 'admin'", | ||
| ADMIN_PASSWORD: "string = 'test'", |
Member
There was a problem hiding this comment.
Can you refactor this to make it be an is_admin field on the users?
Comment on lines
+131
to
+132
| <path d="M5.042 15.165a2.528 2.528 0 0 1-2.52 2.523A2.528 2.528 0 0 1 0 15.165a2.527 2.527 0 0 1 2.522-2.52h2.52v2.52zM6.313 15.165a2.527 2.527 0 0 1 2.521-2.52 2.527 2.527 0 0 1 2.521 2.52v6.313A2.528 2.528 0 0 1 8.834 24a2.528 2.528 0 0 1-2.521-2.522v-6.313zM8.834 5.042a2.528 2.528 0 0 1-2.521-2.52A2.528 2.528 0 0 1 8.834 0a2.528 2.528 0 0 1 2.521 2.522v2.52H8.834zM8.834 6.313a2.528 2.528 0 0 1 2.521 2.521 2.528 2.528 0 0 1-2.521 2.521H2.522A2.528 2.528 0 0 1 0 8.834a2.528 2.528 0 0 1 2.522-2.521h6.312zM18.956 8.834a2.528 2.528 0 0 1 2.522-2.521A2.528 2.528 0 0 1 24 8.834a2.528 2.528 0 0 1-2.522 2.521h-2.522V8.834zM17.688 8.834a2.528 2.528 0 0 1-2.523 2.521 2.527 2.527 0 0 1-2.52-2.521V2.522A2.527 2.527 0 0 1 15.165 0a2.528 2.528 0 0 1 2.523 2.522v6.312zM15.165 18.956a2.528 2.528 0 0 1 2.523 2.522A2.528 2.528 0 0 1 15.165 24a2.527 2.527 0 0 1-2.52-2.522v-2.522h2.52zM15.165 17.688a2.527 2.527 0 0 1-2.52-2.523 2.526 2.526 0 0 1 2.52-2.52h6.313A2.527 2.527 0 0 1 24 15.165a2.528 2.528 0 0 1-2.522 2.523h-6.313z" /> | ||
| </svg> |
Member
There was a problem hiding this comment.
This needs to be in the icons.tsx please!
| <title>Slack</title> | ||
| <path d="M5.042 15.165a2.528 2.528 0 0 1-2.52 2.523A2.528 2.528 0 0 1 0 15.165a2.527 2.527 0 0 1 2.522-2.52h2.52v2.52zM6.313 15.165a2.527 2.527 0 0 1 2.521-2.52 2.527 2.527 0 0 1 2.521 2.52v6.313A2.528 2.528 0 0 1 8.834 24a2.528 2.528 0 0 1-2.521-2.522v-6.313zM8.834 5.042a2.528 2.528 0 0 1-2.521-2.52A2.528 2.528 0 0 1 8.834 0a2.528 2.528 0 0 1 2.521 2.522v2.52H8.834zM8.834 6.313a2.528 2.528 0 0 1 2.521 2.521 2.528 2.528 0 0 1-2.521 2.521H2.522A2.528 2.528 0 0 1 0 8.834a2.528 2.528 0 0 1 2.522-2.521h6.312zM18.956 8.834a2.528 2.528 0 0 1 2.522-2.521A2.528 2.528 0 0 1 24 8.834a2.528 2.528 0 0 1-2.522 2.521h-2.522V8.834zM17.688 8.834a2.528 2.528 0 0 1-2.523 2.521 2.527 2.527 0 0 1-2.52-2.521V2.522A2.527 2.527 0 0 1 15.165 0a2.528 2.528 0 0 1 2.523 2.522v6.312zM15.165 18.956a2.528 2.528 0 0 1 2.523 2.522A2.528 2.528 0 0 1 15.165 24a2.527 2.527 0 0 1-2.52-2.522v-2.522h2.52zM15.165 17.688a2.527 2.527 0 0 1-2.52-2.523 2.526 2.526 0 0 1 2.52-2.52h6.313A2.527 2.527 0 0 1 24 15.165a2.528 2.528 0 0 1-2.522 2.523h-6.313z" /> | ||
| </svg> | ||
| Write @mahad on Slack |
Comment on lines
+76
to
+109
| {violations.map((violation) => ( | ||
| <div | ||
| key={violation.id} | ||
| class="bg-brand-bg p-4 rounded-xl flex items-center justify-between" | ||
| > | ||
| <div class="flex items-center gap-4"> | ||
| <div | ||
| class={`w-2 h-2 rounded-full ${violation.category === "sexual" || violation.category === "sexual/minors" ? "bg-pink-500" : violation.category === "violence" ? "bg-red-500" : "bg-orange-500"}`} | ||
| /> | ||
| <div> | ||
| <div class="flex items-center gap-2"> | ||
| <span | ||
| class={`text-xs px-2 py-0.5 rounded-full capitalize ${getCategoryBadgeColor(violation.category)}`} | ||
| > | ||
| {violation.category} | ||
| </span> | ||
| <span class="text-xs text-brand-text capitalize"> | ||
| {violation.type} | ||
| </span> | ||
| </div> | ||
| <div class="text-xs text-brand-text mt-1"> | ||
| {violation.createdAt | ||
| ? new Date(violation.createdAt).toLocaleString() | ||
| : "Unknown date"} | ||
| </div> | ||
| {violation.content && ( | ||
| <div class="mt-2 text-xs text-brand-text/70 bg-brand-surface/50 rounded p-2 max-h-32 overflow-y-auto break-words font-mono"> | ||
| {violation.content} | ||
| </div> | ||
| )} | ||
| </div> | ||
| </div> | ||
| </div> | ||
| ))} |
Member
There was a problem hiding this comment.
This should prolly be a component
Comment on lines
+17
to
+25
| const modKey = env.OPENAI_MODERATION_API_KEY || env.OPENAI_API_KEY; | ||
| const modUrl = | ||
| env.OPENAI_MODERATION_API_URL || "https://api.openai.com/v1/moderations"; | ||
|
|
||
| if (!modKey) { | ||
| throw new HTTPException(503, { | ||
| message: "Moderation API key is not configured", | ||
| }); | ||
| } |
Member
There was a problem hiding this comment.
This is not necessary - ArkType/env.ts should be enforcing this
| const spent = parseFloat(usage?.totalCost || "0"); | ||
|
|
||
| const spent = await computeSpent(user.id); | ||
| if (spent >= limit) { |
Member
There was a problem hiding this comment.
I'm confused, why does this remove overLimitMessage?
Comment on lines
+7
to
+16
| const token = getCookie(c, "admin_session"); | ||
|
|
||
| if (!token) { | ||
| return c.redirect("/admin/login"); | ||
| } | ||
|
|
||
| // simple session check: cookie must equal '1' (set on successful login) | ||
| if (token !== "1") { | ||
| throw new HTTPException(401, { message: "Admin authentication required" }); | ||
| } |
Member
There was a problem hiding this comment.
What the hell is this 😭😭😭😭😭
(This is a vuln)
Comment on lines
+30
to
+33
| type PgErrorLike = { | ||
| message?: string; | ||
| code?: string; | ||
| }; |
Member
There was a problem hiding this comment.
this does not seem necessary
Comment on lines
+91
to
+108
| const pgError = ( | ||
| typeof e === "object" && e !== null ? e : {} | ||
| ) as PgErrorLike; | ||
| const msg = String(pgError.message || e); | ||
| // If DB doesn't have the new column yet, retry without it for backwards compatibility | ||
| if (msg.includes("violation_event_id") || pgError.code === "42703") { | ||
| const legacyValues = values.map((v) => { | ||
| const { violationEventId: _violationEventId, ...rest } = v; | ||
| return rest; | ||
| }); | ||
| try { | ||
| await db.insert(contentViolations).values(legacyValues); | ||
| } catch (e2) { | ||
| console.error("Failed to record violation (legacy insert):", e2); | ||
| } | ||
| } else { | ||
| console.error("Failed to record violation:", e); | ||
| } |
Member
There was a problem hiding this comment.
This is not necessary because the migrations get run on startup
- Replace env-based admin credentials with user.isAdmin - Secure admin/internal auth via session_token - Move banned-page icons to Icons.tsx and extract ViolationCard - Link ban appeal to Mahad's Slack profile - Restore overLimitMessage helper - Remove manual moderation key check; rely on env validation - Remove legacy violation-insert fallback - Fix missing drizzle journal entries for migrations 0017/0018
skyfallwastaken
requested changes
Jul 15, 2026
|
|
||
| services: | ||
| postgres: | ||
| image: postgres:16-alpine |
Member
There was a problem hiding this comment.
We use Postgres 18 in production
Comment on lines
+8
to
+10
| POSTGRES_USER: postgres | ||
| POSTGRES_PASSWORD: postgres | ||
| POSTGRES_DB: hc-ai |
Member
There was a problem hiding this comment.
You would need to add this to the example .env
| export type User = typeof users.$inferSelect; | ||
| type ApiKey = typeof apiKeys.$inferSelect; | ||
| type RequestLog = typeof requestLogs.$inferSelect; | ||
| export type Violation = typeof contentViolations.$inferSelect; |
Member
There was a problem hiding this comment.
This belongs in schema.ts
| env.ALLOWED_EMBEDDING_MODELS, | ||
| ); | ||
|
|
||
| export const reviewConfig = { |
Member
There was a problem hiding this comment.
Defaults belong in the ArkType type (also a morph to break it down would be better here)
Comment on lines
-6
to
-14
| // JSON-encode a value for safe inlining inside a <script> tag. Escapes `<`, | ||
| // `>`, and U+2028/U+2029 to prevent breaking out of the script context. | ||
| const jsonForScript = (value: unknown): string => | ||
| JSON.stringify(value) | ||
| .replace(/</g, "\\u003c") | ||
| .replace(/>/g, "\\u003e") | ||
| .replace(/\u2028/g, "\\u2028") | ||
| .replace(/\u2029/g, "\\u2029"); | ||
|
|
Member
There was a problem hiding this comment.
Why are you removing this
Comment on lines
-69
to
-76
| "You are Codex, a coding agent based on GPT-5", | ||
| "You and the user share one workspace, and your job is to collaborate with them until their goal is genuinely handled.", | ||
| "Runs a command in a PTY, returning output or a session ID for ongoing interaction", | ||
|
|
||
| // Trae | ||
| "This tool is Trae's context engine. It:\n1. Takes in a natural language description of the code you are looking for;\n2. Uses a proprietary retrieval/embedding model suite that produces the highest-quality recall of relevant code snippets from across the codebase;\n3. Maintains a real-time index of the codebase, so the results are always up-to-date and reflects", | ||
| '- Fast file pattern matching tool that works with any codebase size\n- Supports glob patterns like "/*.js" or "src//*.ts"\n- Returns matching file paths sorted by modification time', | ||
| "Use this tool when you need to ask the user questions during execution. This allows you to:\n1. Gather user preferences or requirements\n2. Clarify ambiguous instructions\n3. Get decisions on implementation choices as you work", |
Member
There was a problem hiding this comment.
Why are we removing this?
Comment on lines
+117
to
+142
| { | ||
| "idx": 16, | ||
| "version": "7", | ||
| "when": 1779302500000, | ||
| "tag": "0016_add_content_column", | ||
| "breakpoints": true | ||
| }, | ||
| { | ||
| "idx": 17, | ||
| "version": "7", | ||
| "when": 1779302600000, | ||
| "tag": "0017_add_violation_event_id", | ||
| "breakpoints": true | ||
| }, | ||
| { | ||
| "idx": 18, | ||
| "version": "7", | ||
| "when": 1779302700000, | ||
| "tag": "0018_add_violation_dismissed", | ||
| "breakpoints": true | ||
| }, | ||
| { | ||
| "idx": 19, | ||
| "version": "7", | ||
| "when": 1779400000000, | ||
| "tag": "0019_add_is_admin_to_users", |
Member
There was a problem hiding this comment.
Have you... actually run these migrations 😭
(the when ending in 00000 for multiple migrations is a bit suspect
Member
There was a problem hiding this comment.
Why are you deleting this
| throw new HTTPException(403, { | ||
| message: "You are banned from using this service.", | ||
| }); | ||
| if (result.user.isBanned || result.user.reviewStatus === "banned") { |
Member
There was a problem hiding this comment.
Why are there two banned fields? This should really be one
| c.set("user", apiKey.user); | ||
|
|
||
| if (apiKey.user.isBanned) { | ||
| if (apiKey.user.isBanned || apiKey.user.reviewStatus === "banned") { |
Member
There was a problem hiding this comment.
Why are there two banned fields? This should really be one
skyfallwastaken
requested changes
Jul 15, 2026
skyfallwastaken
left a comment
skyfallwastaken
left a comment
Member
There was a problem hiding this comment.
Also this should really use a local classifier imho, for latency. + the mod code should ideally be in a background job of some sort
- Move inline dashboard icon to Icons.tsx (AlertTriangle) - Fix banned.tsx: slack.hackclub.com link + remove slackId fallbacks - Remove global stats from admin dashboard, fix name fallbacks, componentize admin views - Restore blocked-prompts.ts entries (Codex CLI, Trae) - Restore price.ts + add cost column back to activity - Fix migration journal timestamps (realistic ms values) - Consolidate banned fields: remove isBanned, use reviewStatus only - Add local moderation classifier (regex classifier + async background moderation for all output - Make moderation API keys optional, add guard to moderations proxy
skyfallwastaken
requested changes
Jul 15, 2026
skyfallwastaken
left a comment
skyfallwastaken
left a comment
Member
There was a problem hiding this comment.
- Consolidate banned fields: remove isBanned, use reviewStatus only
Please don't do this 😭 It removes pre-existing bans
skyfallwastaken
requested changes
Jul 15, 2026
Member
There was a problem hiding this comment.
This is not what I would call accurate, we should be using a proper classifier here
| </p> | ||
| <a | ||
| href="https://hackclub.slack.com/team/@mahad" | ||
| href="https://slack.hackclub.com/" |
Member
There was a problem hiding this comment.
??? why are we linking to the slack signup page
Comment on lines
+12
to
+13
| OPENAI_MODERATION_API_KEY: "string", | ||
| OPENAI_MODERATION_API_URL: "string", | ||
| "OPENAI_MODERATION_API_KEY?": "string", | ||
| "OPENAI_MODERATION_API_URL?": "string", |
Member
There was a problem hiding this comment.
These are already required for the moderation API support (that was there before this PR) so this is not necessary
skyfallwastaken
requested changes
Jul 17, 2026
Author
There was a problem hiding this comment.
for horizon they wanted a deployed version i tried to do it quickly but reverted it (it went into the pr by mistake)
| POSTGRES_PASSWORD: postgres | ||
| POSTGRES_DB: hc-ai | ||
| ports: | ||
| - "5432:5432" |
Member
There was a problem hiding this comment.
Make this not run on 5432 since lots of people already have other Postgres DBs running on their device!
| *.log | ||
| scratch | ||
| package-lock.json | ||
| AGENTS.md No newline at end of file |
|
|
||
| // Cline | ||
| "You are Cline, a software engineering AI. Your mission is to execute precisely what is requested - implement exactly what was asked for, with the simplest solution that fulfills all requirements.", | ||
| "You are Cline, a software engineering AI. Your mission is to execute precisely what was requested - implement exactly what was asked for, with the simplest solution that fulfills all requirements.", |
Member
There was a problem hiding this comment.
Please keep BOTH the old and the new versions! So we can catch old Cline versions
| '- mode: (required) The slug of the mode to start the new task in (e.g., "code", "debug", "architect").', | ||
| "By specifying line ranges, you can efficiently read specific portions of large files without loading the entire file into memory.", | ||
| 'The tool outputs line-numbered content (e.g. "1 | const x = 1") for easy reference when creating diffs or discussing code', | ||
| 'The tool outputs line-numbered content (e.g., "1 | const x = 1") for easy reference when creating diffs or discussing code', |
Member
There was a problem hiding this comment.
Please keep BOTH the old and the new versions! So we can catch old KC versions
| @@ -0,0 +1 @@ | |||
| ALTER TABLE "users" DROP COLUMN IF EXISTS "is_banned"; | |||
Member
There was a problem hiding this comment.
We should not be dropping our existing bans for no reason
| -- Ensure pgcrypto extension for gen_random_uuid() | ||
| CREATE EXTENSION IF NOT EXISTS pgcrypto; | ||
|
|
||
| -- Add the violation_event_id column if it doesn't exist yet |
Member
There was a problem hiding this comment.
This comment is not necessary
Member
There was a problem hiding this comment.
(the first one is fine tho)
| isIdvVerified: boolean("is_idv_verified").notNull().default(false), | ||
| skipIdv: boolean("skip_idv").notNull().default(false), | ||
| isBanned: boolean("is_banned").notNull().default(false), | ||
| isAdmin: boolean("is_admin").notNull().default(false), |
Member
There was a problem hiding this comment.
(See previous downvote comment)
| isBanned: boolean("is_banned").notNull().default(false), | ||
| isAdmin: boolean("is_admin").notNull().default(false), | ||
| agentBannerDismissedAt: timestamp("agent_banner_dismissed_at"), | ||
| reviewStatus: text("review_status").notNull().default("normal"), // "normal" | "flagged" | "strict" | "banned" |
Member
There was a problem hiding this comment.
Drizzle has enum support - use it!
| ], | ||
| ); | ||
|
|
||
| export const pendingCharges = pgTable( |
Member
There was a problem hiding this comment.
Why are we removing pendingCharges?
… feedback - Restore pending_charges billing (reserve/release, $4 default limit) that the PR accidentally reverted - Restore activity pagination/model names, OCR base64 document validation, and PostHog script fixes from upstream - Regenerate moderation migrations with drizzle-kit on top of upstream's 0015_regular_winter_soldier, with real snapshots and timestamps; carry over existing is_banned users to review_status - Fix ArkType env defaults syntax that crashed the app on boot; moderation API config is now enforced at startup - Validate admin user status filter against the review_status enum - DM link now points at hackclub.slack.com, compose port reverted to 5432, and .env.example documents DATABASE_URL - Remove duplicate output moderation calls and unused exports (knip/typecheck/lint all green)
There was a problem hiding this comment.
Pull request overview
This PR introduces a first-pass content violation system (OpenAI moderation-backed), adds user-facing violations visibility + “forced review” opt-in, and ships an admin panel to browse/dismiss violations and ban/unban users.
Changes:
- Add moderation/violation tracking (DB schema + review logic) and integrate it into proxy request handling.
- Add user violations UI (dashboard link, violations page, banned page) including forced-review toggle.
- Add admin panel (dashboard/users/violations) with ban/unban and dismissal workflows, plus internal/admin auth plumbing.
Reviewed changes
Copilot reviewed 36 out of 37 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| src/views/moderation.tsx | New user-facing violations/review-status view with forced-review toggle. |
| src/views/dashboard.tsx | Adds a quick link to the violations page. |
| src/views/components/StatCard.tsx | Makes stat values nullable/optional and renders “-” when absent. |
| src/views/components/Icons.tsx | Adds an AlertTriangle icon used by the dashboard quick link. |
| src/views/components/Header.tsx | Adds admin-mode navigation + makes user optional/null-safe. |
| src/views/banned.tsx | New banned landing page showing recent violations and appeal link. |
| src/views/admin/violations.tsx | New admin violations list with filters and dismissal action. |
| src/views/admin/users.tsx | New admin user search/list with status filters. |
| src/views/admin/user-detail.tsx | New admin user detail page with ban/unban and violation history. |
| src/views/admin/dashboard.tsx | New admin dashboard with moderation stats and recently banned users. |
| src/views/admin/components.tsx | Shared admin UI components/types (tabs, status badges, DTOs). |
| src/views/admin.tsx | Re-exports admin views for route usage. |
| src/routes/proxy/v1/moderations.ts | Refactor to explicit async handler (no behavior change intended). |
| src/routes/proxy/v1/general.ts | Integrates review/forced-review + moderation gating + pending-charge release on upstream errors. |
| src/routes/proxy/shared.ts | Ensures pending charge is released after logging requests. |
| src/routes/internal.ts | Adds internal/admin auth middleware and user violation/flag-settings endpoints. |
| src/routes/dashboard.tsx | Adds /banned and /dashboard/violations routes; redirects banned users appropriately. |
| src/routes/auth.ts | Adds a development-only “dev user” login shortcut; minor catch cleanup. |
| src/routes/api.tsx | Adds authenticated JSON endpoints for violations and forced-review settings. |
| src/routes/admin.tsx | Adds admin routes for dashboards/users/violations + moderation actions. |
| src/middleware/auth.ts | Switches banned logic to reviewStatus === "banned" and redirects web users to /banned. |
| src/middleware/admin.ts | Adds requireAdmin middleware enforcing user.isAdmin. |
| src/lib/stats.ts | Adjusts model stats query (removes HAVING filter). |
| src/lib/review.ts | Implements review decisioning, violation recording/counting, stats queries, and flag settings. |
| src/lib/moderation.ts | Adds OpenAI moderation API wrapper + flagged-category extraction. |
| src/lib/models.ts | Minor unused param/comment cleanup. |
| src/index.ts | Mounts the new /admin router. |
| src/env.ts | Adds review/threshold env vars + exports reviewConfig and optional INTERNAL_API_KEY. |
| src/db/schema.ts | Adds review status enum, violations tables, flag settings, admin flag; migrates away from is_banned. |
| drizzle/meta/0017_snapshot.json | Drizzle snapshot for new schema state. |
| drizzle/meta/0016_snapshot.json | Drizzle snapshot for prior step in migration chain. |
| drizzle/meta/_journal.json | Records the two new migrations. |
| drizzle/0017_nice_shard.sql | Migrates existing bans into review_status then drops is_banned. |
| drizzle/0016_even_fallen_one.sql | Creates new moderation tables/enum and extends users schema. |
| docker-compose.yml | Adds local Postgres service for development. |
| .gitignore | Ignores package-lock.json and AGENTS.md. |
| .env.example | Updates DATABASE_URL example to a local Postgres URL. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+239
to
+245
| return c.json( | ||
| { | ||
| error: "Content prohibited by moderation", | ||
| categories, | ||
| }, | ||
| 400, | ||
| ); |
| }[]; | ||
| } | ||
|
|
||
| async function trigger_review(content: string[]): Promise<ModerationResult> { |
| * Classify content using the OpenAI moderation API. | ||
| */ | ||
| export async function moderate(content: string[]): Promise<ModerationResult> { | ||
| return trigger_review(content); |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…_review to camelCase - Release pending spending reservation when blocking output moderation flags content (previously leaked until TTL cleanup) - Rename trigger_review to triggerReview for codebase consistency
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey mahad,
i have created a violation system that will help you dealing with nsfw and co.
The OpenAI api is free (but needs a active billing acc).
I also added a small Admin panel were you can manually ban unban and review violations.
For users with apps were everyone can use the AI they can activate the "strict" mode in the violations page.
This means every review will be reviewed before being send to the AI. If in that case a violation should get detected it wont count towards the generall violation limit and just return with a clear error (403 forbidden i think but dont take my word).
It follows this concept i made in slack:
Here are some screenshots from my local dev env