Skip to content

Fix 13 audit findings (startup blocker, credential leaks, retrieval correctness) + CI gates - #1

Merged
ShreeBohara merged 2 commits into
mainfrom
fix/audit-batch-1
Aug 9, 2026
Merged

Fix 13 audit findings (startup blocker, credential leaks, retrieval correctness) + CI gates#1
ShreeBohara merged 2 commits into
mainfrom
fix/audit-batch-1

Conversation

@ShreeBohara

Copy link
Copy Markdown
Owner

Findings from a full-repository audit, ordered by cost today over risk of changing it. Every item was reproduced before the fix and re-verified after.

Why this is worth reviewing first

Two of these are not latent:

  • docker compose up could never start the API. pydantic-settings JSON-decodes List[str] inside EnvSettingsSource before field validators run, so the comma-separated CORS_ORIGINS that docker-compose.yml ships as its own default raised SettingsError at import of src.config — before uvicorn bound a port. config.py now reads it as a str and splits in a property.
  • The GitHub token could be exfiltrated by an unauthenticated request. The token was injected whenever "github.com" in url, which also matches github.com.attacker.tld. POST /api/repos/ needs no auth, so an attacker-chosen host received the token. Host is now compared exactly against the parsed hostname, git stderr is redacted before it is raised, and .dockerignore stops COPY apps/api . baking a live .env into the image.

What changed

Startup / deploymentCORS_ORIGINS parsing; blank env assignments fall back to declared defaults rather than becoming ""; compose forwards LLM_PROVIDER, EMBEDDING_PROVIDER, GITHUB_TOKEN, ANTHROPIC_*, OLLAMA_* and the embedding rate-limit knobs, which its hardcoded allowlist had been dropping (Anthropic and Ollama were unreachable under the documented Docker path).

Credentials — exact-host token injection; token redacted from indexing_error, which the public /api/repos/{id}/progress SSE endpoint streams; .env and .venv excluded from the Docker build context.

Retrieval correctness

  • Answer cache key now includes a conversation-history digest. The prompt contained history but the key did not, so a fresh session was served another session's history-conditioned answer for the full 30-minute TTL.
  • A stream failure with nothing delivered now raises instead of yielding [Error: ...] as a token. That text was being cached and persisted as the assistant's answer, so one transient upstream error poisoned a question for every later asker.
  • tree_sitter_parser uses node.text instead of slicing the decoded str with start_byte/end_byte. Those are byte offsets, so a single multi-byte character mis-aligned every chunk after it in the file.
  • .tsx is parsed with language_tsx(). The plain TypeScript grammar cannot parse JSX, and has_error was never checked — tree-sitter returns an ERROR tree rather than raising, so the existing except clause never fired. Chunks are now checked and fall back to raw indexing.
  • Ollama 404s raise instead of being retried and failed-open, and fail-open gained a failure-ratio ceiling. The default model name was a HuggingFace id Ollama cannot resolve, so every chunk exhausted a 10-attempt backoff and landed as a zero vector — which scores every chunk identically and makes retrieval arbitrary at full confidence.

API surface — quiz input bounded and rate-limited (it was the only LLM route with neither, and the quiz bucket it needed did not exist, so the limit call would have been a silent no-op); /health uses .value (a str-Enum f-strings to IndexingStatus.COMPLETED, so demo mode reported degraded permanently) and an unreachable LLM now actually fails the check; api-client.ts reads the error body once and handles FastAPI's array detail, so 422s and non-JSON bodies stop collapsing to a generic message.

Dependenciestree-sitter>=0.22 (Parser(language) is 0.22+; the old floor permitted a version where importing the parser raises TypeError and the API cannot boot), chromadb>=1.0, plus the undeclared pyyaml and pytest-cov.

Second commit: CI gates that passed for the wrong reasons

  • CI ran pytest tests/unit tests/integration, silently skipping tests/test_parser.py at the tests/ root — so CI never constructed a tree-sitter parser, and local runs covered more than CI on the same commit. Now pytest tests.
  • pnpm install --frozen-lockfile, matching vercel.json. CI was repairing lockfile drift that the Vercel build then rejected, so a PR could go green while the deploy failed on the same commit.
  • test is vitest run (bare vitest only exits when it detects CI, so pnpm test hung locally); type-check runs next typegen first, since Next 16 generates route types only during dev/build; turbo clean now resolves, so root pnpm clean reaches its rm -rf node_modules instead of short-circuiting.
  • turbo.json: test depends on ^build not build, dropping a full next build --webpack before the first test; globalDependencies points at apps/web/.env.local, the file that actually exists.
  • .env.example / docker/README.md contradicted each other on where .env goes and neither was right for both paths — local dev reads apps/api/.env, Compose expands ${VAR} from docker/.env. Also commented out the OPENAI_API_KEY=sk placeholder, which was a truthy value that left the app believing it was configured.

Verification

Gate Result
pytest tests 89 passed, 0 failed
ruff check src tests clean
vitest run (CI unset) 12 passed, exits
next typegen types generated
tsc --noEmit 0 errors in project source

pytest also could not run at all on a clean install before this: pyproject's addopts requires pytest-cov, which requirements.txt never declared.

Not included

  • .tsx and byte-offset fixes are correct in code but existing indexes need a re-index to benefit — 28 .tsx files in the pinned demo repo are still mis-chunked.
  • Chroma's implicit l2 space (embeddings are normalized for cosine) is deliberately left for that same re-index, since the space is immutable after collection creation.
  • parse_github_url still accepts non-GitHub hosts, so git clone can reach internal addresses. The credential exposure is closed; restricting the host set is a functional decision.

🤖 Generated with Claude Code

ShreeBohara and others added 2 commits August 9, 2026 08:26
…ctness

Ranked batch from the repository audit. Every item was reproduced before the fix
and re-verified after; ruff, pytest (89 passed incl. tests/test_parser.py),
vitest (12 passed) and tsc are green.

Startup / deployment
- config.py: CORS_ORIGINS is read as a raw str and parsed in a property.
  pydantic-settings JSON-decodes List[str] inside EnvSettingsSource *before*
  field validators run, so docker-compose's comma-separated default raised
  SettingsError at import and `docker compose up` could never start the API.
- config.py: a blank env assignment (FOO=) now falls back to the declared
  default for the 12 settings where blank is always a mistake. A blank
  LOCAL_EMBEDDING_MODEL was otherwise a request for the model named "".
- docker-compose.yml: forward LLM_PROVIDER, EMBEDDING_PROVIDER, GITHUB_TOKEN,
  ANTHROPIC_*, OLLAMA_*, DEBUG and the embedding rate-limit knobs. The api
  service had a hardcoded allowlist and no env_file, so Anthropic, Ollama and
  private-repo cloning were unreachable under the documented Docker path.

Credentials
- repo_manager.py: inject the GitHub token only when the parsed hostname is
  exactly github.com/www.github.com. The previous `"github.com" in url` test
  also matched github.com.attacker.tld, sending the token to it via an
  unauthenticated POST /api/repos/.
- repo_manager.py: redact the token from git stderr before it is raised. That
  message is persisted to Repository.indexing_error and streamed by the public
  /api/repos/{id}/progress endpoint.
- .dockerignore: exclude .env (any depth) and .venv. `COPY apps/api .` was
  baking the developer's live OPENAI_API_KEY into the image, plus a ~470MB
  Darwin-built venv.

Retrieval correctness
- chat_cache.py/pipeline.py: include a conversation-history digest in the
  answer cache key. The prompt contains history but the key did not, so a
  fresh session was served another session's history-conditioned answer for
  the 30-minute TTL.
- llm/*: on a stream failure with nothing yet delivered, raise instead of
  yielding "[Error: ...]" as a token. That text was being cached and persisted
  as the assistant's answer, so one transient upstream error poisoned that
  question for every later asker. Partial streams still get an inline marker,
  and pipeline.py refuses to cache a response containing it.
- tree_sitter_parser.py: use node.text instead of slicing the decoded str with
  start_byte/end_byte. Those are byte offsets, so one multi-byte character
  mis-aligned every chunk after it in the file.
- tree_sitter_parser.py: parse .tsx with language_tsx(). The plain TypeScript
  grammar cannot parse JSX, so every .tsx file produced an ERROR tree.
- tree_sitter_parser.py/indexing_service.py: surface has_errors and fall back to
  raw indexing. tree-sitter returns an ERROR tree rather than raising, so the
  existing except-clause never fired for a grammar mismatch.
- ollama_embeddings.py: a 404 now raises OllamaModelNotFound (not retried, not
  failed-open) and fail-open gained a failure-ratio ceiling. The default model
  was a HuggingFace id Ollama cannot resolve, so every chunk exhausted a
  10-attempt backoff and landed as a zero vector - which scores every chunk
  identically and makes retrieval arbitrary at full confidence.
- config.py: default local_embedding_model to the Ollama tag nomic-embed-text
  (the class default was already correct; only config overrode it).

API surface
- learning.py: bound GenerateQuizRequest.context_content and apply the demo
  soft limit. It was the only LLM route with neither, so an unauthenticated
  caller could send an unbounded body straight to the model; added the missing
  "quiz" bucket, without which the limit call would have been a silent no-op.
- main.py: /health uses status.value (a str-Enum f-strings to
  "IndexingStatus.COMPLETED", so demo mode reported "degraded" permanently) and
  the LLM check now reports "error: ..." so it actually fails critical_ok
  instead of passing while the provider is unreachable.
- api-client.ts: read the error body once. res.json() consumes the stream even
  on a parse failure, so the res.text() fallback always threw and every
  non-JSON error body became a generic message. Also handle FastAPI's array
  `detail`, since typeof [] === 'object' turned every 422 into that fallback.

Dependencies
- requirements.txt: tree-sitter>=0.22 (Parser(language) and single-arg
  Language() are 0.22+; the 0.21 floor permitted a version where importing the
  parser raises TypeError and the API cannot boot), chromadb>=1.0
  (chromadb.errors.NotFoundError), plus the undeclared pyyaml (imported by the
  /openapi.yaml route) and pytest-cov (required by pyproject's addopts, so
  pytest could not start on a clean install).
- conftest.py: AsyncClient(transport=ASGITransport(...)); the app= shortcut was
  removed in httpx 0.28.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- ci.yml: run `pytest tests`, not `tests/unit tests/integration`.
  tests/test_parser.py sits at the tests/ root, so CI silently skipped ~20 cases
  and never constructed a tree-sitter parser -- while local runs (pyproject
  testpaths = ["tests"]) covered more than CI did on the same commit.
- ci.yml: install only ruff ad hoc, pinned to 0.16.2. pytest/pytest-asyncio/
  pytest-cov/httpx are now declared in requirements.txt, so a fresh clone can run
  the suite; pinning stops an unrelated PR going red when ruff adds a rule.
  Dropped pytest-mock, which nothing in the repo uses.
- ci.yml: `pnpm install --frozen-lockfile`, matching apps/web/vercel.json. CI was
  repairing lockfile drift that the Vercel build then rejected, so a PR could go
  green while the production deploy failed on the same commit.
- apps/web/package.json: `test` is `vitest run`; `test:watch` keeps the watcher.
  Bare `vitest` only switches to run mode when it detects CI, so the documented
  `pnpm test` hung forever locally inside a turbo task that expects an exit.
- apps/web/package.json: `type-check` runs `next typegen && tsc --noEmit`. Next 16
  generates route types during dev/build only, so a bare tsc validated no route
  types -- and tsconfig includes .next/types/**, absent on a clean checkout.
- apps/web/package.json + turbo.json: add the `clean` task that root
  `pnpm clean` already invoked. `turbo clean` failed on an undefined task, so the
  `&&` short-circuited and node_modules was never removed.
- turbo.json: `test` dependsOn `^build` instead of `build`. It was running a full
  `next build --webpack` before a single vitest test, which is neither needed nor
  what CI does.
- turbo.json: globalDependencies points at apps/web/.env.local, the file that
  actually feeds the build. The root .env.local it named does not exist, so a
  cached web build could ship a stale inlined NEXT_PUBLIC_* value.
- .env.example: comment out the OPENAI_API_KEY placeholder. `sk` is a non-empty
  value, so a copied file left the app believing it was configured.
- .env.example + docker/README.md: state where .env goes, which differs by path.
  Local dev reads apps/api/.env (config.py loads ".env" relative to the cwd, and
  the README starts uvicorn from apps/api); Compose expands ${VAR} from
  docker/.env and ignores a repo-root .env. The two docs previously contradicted
  each other and neither was right for both cases.

Verified: pytest 89 passed, vitest 12 passed and now exits with CI unset,
next typegen succeeds, turbo clean resolves, ruff clean, and tsc reports zero
errors in project source.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
codebaseqa-web Ready Ready Preview Aug 9, 2026 3:28pm

@ShreeBohara ShreeBohara changed the title Fix 15 audit findings: startup blocker, credential leaks, retrieval correctness, CI gates Fix 13 audit findings (startup blocker, credential leaks, retrieval correctness) + CI gates Aug 9, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bed0605207

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

context_content: str
# Bounded like ChatMessageCreate.content (schemas.py): this string goes straight
# into an LLM prompt, so an unbounded body is an unmetered spend vector.
context_content: str = Field(..., max_length=20000)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid rejecting long lessons before truncation

When a generated lesson's content_markdown exceeds 20 KB, the frontend still sends the full lesson body to this endpoint (api.generateQuiz(..., content.content_markdown)), but LearningService.generate_quiz() only uses context_content[:2000] in the actual prompt. This new validation rejects those long lessons with a 422 before the server can use the safe 2 KB slice, so users can no longer generate quizzes for longer lessons even though no extra LLM spend would occur. Consider truncating before validation or having the client send the already-bounded excerpt.

Useful? React with 👍 / 👎.

@ShreeBohara
ShreeBohara merged commit 320fc56 into main Aug 9, 2026
4 checks passed
@ShreeBohara
ShreeBohara deleted the fix/audit-batch-1 branch August 9, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant