Skip to content

test: add metering E2E integration tests + Ollama CI tier#1208

Closed
pyramation wants to merge 3 commits into
mainfrom
feat/metering-e2e-tests
Closed

test: add metering E2E integration tests + Ollama CI tier#1208
pyramation wants to merge 3 commits into
mainfrom
feat/metering-e2e-tests

Conversation

@pyramation
Copy link
Copy Markdown
Contributor

@pyramation pyramation commented May 21, 2026

Summary

Test #3 from constructive-planning#911: full metering pipeline E2E tests + new Ollama CI tier.

Metering E2E tests (metering-e2e.test.ts, 15 tests)

Config resolution (3 tests):

  • getLlmBillingConfig resolves billing + inference log config from metaschema tables
  • Returns null for unknown database_id

Billing functions (4 tests):

  • check_billing_quota returns true when unlimited (no credits row)
  • Returns true when under limit, false when over limit
  • record_usage writes to balance + ledger with metadata

Inference log (1 test):

  • logInferenceUsage inserts row with all fields (service, operation, cache tokens, etc.)

meteredEmbed with mock embedder (4 tests):

  • Calls embedder + records usage when quota allows
  • Returns quotaExceeded when limit reached
  • Passes through unmetered when no meter slug configured
  • request_id propagates to billing ledger metadata

meteredEmbed with real Ollama (3 tests, skipped when unavailable):

  • Full pipeline: Ollama embed → billing check → record_usage → inference_log + ledger
  • Quota enforcement blocks real Ollama calls
  • Multiple calls accumulate in balance and ledger

CI Tier 4 — Ollama tests (run-tests.yaml)

New CI job with PostgreSQL + Ollama service containers:

  • Uses bash /dev/tcp health check (Ollama image doesn't have curl)
  • Pulls nomic-embed-text model before test run
  • Runs only metering-e2e tests (existing graphile-llm.test.ts tests require schema setup from pg-tests tier)
  • Tests gracefully skip Ollama-specific cases when Ollama is unavailable (local dev)

Test infrastructure (metering-setup.sql)

Minimal stub schemas matching what config-cache.ts queries:

  • metaschema_modules_public.billing_module + inference_log_module
  • Simplified but functional check_billing_quota + record_usage functions
  • usage_public.usage_log_inferences table
  • Seeded meter rows for embedding + inference

Review & Testing Checklist for Human

  • Verify the Ollama CI tier doesn't slow down the critical path (it runs in parallel with other tiers)
  • Check that metering-setup.sql stub functions match the real billing function signatures
  • Confirm nomic-embed-text is a reasonable model choice for CI (small, fast)

Notes

  • Companion PR for Test readme #2 (inference rollup tests): constructive-db#1292 (merged)
  • The 3 Ollama tests are describe.skip'd when SKIP_OLLAMA_TESTS=1 or Ollama is unreachable — safe for local dev
  • The setup SQL uses CREATE FUNCTION (not CREATE OR REPLACE) — these are test-only stubs in ephemeral databases

Link to Devin session: https://app.devin.ai/sessions/2b5a29d83d3f478e8d3d972653b4879c
Requested by: @pyramation

Test #3 from constructive-planning#911: full metering pipeline tests.

Metering E2E tests (15 tests):
- Config resolution: getLlmBillingConfig reads metaschema tables
- Billing functions: check_billing_quota + record_usage SQL stubs
- Inference log: logInferenceUsage writes correct fields
- meteredEmbed with mock embedder: quota check, record_usage, quota exceeded, request_id propagation
- meteredEmbed with real Ollama: live inference + billing pipeline (skipped when Ollama unavailable)

CI Tier 4 (ollama-tests):
- PostgreSQL + Ollama service containers
- Pulls nomic-embed-text model before test run
- Runs graphile-llm full test suite including Ollama E2E
@devin-ai-integration
Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@blacksmith-sh

This comment has been minimized.

@devin-ai-integration
Copy link
Copy Markdown
Contributor

Closing — metering E2E tests should live in constructive-db where the real billing infrastructure exists, not stub SQL in the open source repo.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant