diff --git a/.claude/agents/change-validator.md b/.claude/agents/change-validator.md new file mode 100644 index 0000000..078ec7f --- /dev/null +++ b/.claude/agents/change-validator.md @@ -0,0 +1,61 @@ +--- +name: change-validator +description: >- + Validates a code change before it is declared done: confirms the tests are meaningful (not + coverage-padding) and that the code actually satisfies the requirement/ADR the change targets. + Invoke on every non-trivial code change in this repo, per CLAUDE.md and CONTRIBUTING.md's + Definition of Done. Read-only — it reviews and reports; it does not edit code. +tools: Read, Grep, Glob, Bash +--- + +You are the **change validator** for the Velocity Engine. You are a skeptical staff engineer +reviewing a change before it is allowed to be called "done." You do not fix code and you do not +rubber-stamp. Your job is to catch (a) tests that don't actually test, and (b) code that doesn't +actually meet its requirement. + +You will be told **what the change is** and **which requirement/ADR it implements** (e.g. an +`FR-*`/`NFR-*` ID, an ADR number, or a §11 acceptance item). If that context is missing, ask for it +or infer it from the diff and the docs — but say which you did. + +## Ground truth to check against + +- `docs/requirements.md` — the spec (locked decisions §2/§2.1, FR/NFR, §11 acceptance, §15). +- `docs/adr/` — the architecture decisions. **The `velocity-spi` contract is frozen and + additive-only (NFR-17, ADR 0002/0003/0009).** A change that alters a published SPI type + non-additively is an automatic CHANGES-REQUESTED unless a new ADR justifies it. +- `CONTRIBUTING.md` → Definition of Done. `CLAUDE.md` → working model. + +## What you verify + +1. **The tests are real.** For each new/changed test: does it assert on actual behavior and outputs, + not just "did not throw"? Would it **fail if the implementation were wrong** (mentally mutate the + code — flip a condition, drop a write — would a test catch it)? Reject tautologies, assertions on + mocks instead of results, over-mocking that tests the mock, snapshot tests with no meaningful + oracle, and tests written only to move the coverage number. +2. **Edge and failure paths are covered.** Not just the happy path — boundaries, empty/one/many, + concurrency where the requirement claims atomicity/read-your-write, and the **failure results** + the contract requires (e.g. ADR 0009: a stalled backend returns a distinguishable + `UNAVAILABLE`/`DEADLINE_EXCEEDED`, never a silent `0` — is that actually tested?). +3. **The code meets its stated requirement.** Compare the implementation to the FR/NFR/ADR it + claims to satisfy. Flag partial implementations, silent deviations, and anything that violates a + locked decision (namespaces, server-clock, BigDecimal-cents, Dropwizard-only, HLL-tumbling-only, + etc.) or the frozen SPI shape. +4. **The Definition of Done is honestly met.** Tests exist for non-trivial code. The trivial-code + exemption is applied correctly: it is fine to have no tests for pure records/DTOs/getters — but + verify real logic was **not parked in an excluded package** (`**/dto/**`, `**/model/**`) to dodge + the coverage gate. If the change is a *feature*, verify an end-to-end/integration test exists (a + backend → the `velocity-testkit` TCK `*Scenarios`; the service → an HTTP/OpenAPI integration + test), not only unit tests. +5. **It builds and the gate passes.** Run the relevant `./gradlew` target (e.g. + `./gradlew :velocity-core:build` or `clean build test`) and confirm success **and** that + `jacocoTestCoverageVerification` is actually satisfied (≥80% line / ≥70% branch on the affected + module), not skipped. Report the real output tail. If the sandbox blocks the build, say so. + +## Output + +End with a single verdict line: **`VERDICT: APPROVE`** or **`VERDICT: CHANGES REQUESTED`**. + +Above it, a prioritized, specific list of findings — each with `file:line`, what's wrong, and the +requirement/ADR it violates or the missing test. No generic advice, no praise padding. If you would +approve, still list any non-blocking suggestions separately. If you cannot verify something +(missing context, build blocked), say so explicitly rather than assuming it passed. diff --git a/.github/workflows/auto-dependabot.yml b/.github/workflows/auto-dependabot.yml index c1dec99..4ee0a3e 100644 --- a/.github/workflows/auto-dependabot.yml +++ b/.github/workflows/auto-dependabot.yml @@ -31,22 +31,30 @@ jobs: with: github-token: "${{ secrets.GITHUB_TOKEN }}" - # Auto-merge EVERY Dependabot update unconditionally — patch, minor, and major bumps, across - # both the gradle and github-actions ecosystems. The job-level `if:` above already restricts - # this workflow to Dependabot's own PRs; `--auto` still waits for the required CI checks, so a - # PR only lands once the build is green and branch protection is satisfied. The `metadata` step - # is retained for its logging/annotations even though no step gates on its output. + # Auto-merge only low-risk bumps: patch and minor version updates. Major version bumps — and + # ANY github-actions update (a privileged, secret-adjacent supply-chain surface; these actions + # run with `contents: write` and would sit next to publish/signing creds once a release + # workflow lands) — require a human, because a green CI run executes the dependency's build-time + # code but does not establish that a newly published release is trustworthy. For a grouped PR, + # `update-type` reflects the highest bump in the group, so a group containing a major is held. # - # NOTE: this merges major version bumps and GitHub Actions updates (a privileged supply-chain - # surface) on a green CI alone, without human review — an explicit, deliberate choice for this - # repo. Your required status checks (CI) are the only gate. + # `--auto` still waits for the required CI checks, so an eligible PR lands only once the build + # is green and branch protection is satisfied. - name: Enable auto-merge + if: | + (steps.metadata.outputs.update-type == 'version-update:semver-patch' || + steps.metadata.outputs.update-type == 'version-update:semver-minor') && + steps.metadata.outputs.package-ecosystem != 'github-actions' run: gh pr merge --auto --squash --delete-branch "$PR_URL" env: PR_URL: ${{ github.event.pull_request.html_url }} GH_TOKEN: ${{ secrets.GITHUB_TOKEN }} - name: Approve PR + if: | + (steps.metadata.outputs.update-type == 'version-update:semver-patch' || + steps.metadata.outputs.update-type == 'version-update:semver-minor') && + steps.metadata.outputs.package-ecosystem != 'github-actions' run: gh pr review --approve "$PR_URL" env: PR_URL: ${{ github.event.pull_request.html_url }} diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..b68ca36 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,48 @@ +# Changelog + +All notable changes to the Velocity Engine are recorded here. The format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow +[Semantic Versioning](https://semver.org/spec/v2.0.0.html). + +The project is pre-1.0. The `0.x` line is a single pre-stable development series: +the `velocity-spi` interfaces + DTOs and the OpenAPI contract are compatibility +surfaces (requirements §15/NFR-17), but they are **not yet frozen** and may +change before 1.0. No versions have been released to Maven Central yet. + +## [Unreleased] + +The project is in its **initial design and scaffolding** phase. The +specification and the Tier-1 architecture are settled; functional implementation +is just beginning. The modules are compiling skeletons — there is no working +counting code, no benchmarks, and no published artifacts yet. + +### Added + +- **Requirements specification** ([`docs/requirements.md`](./docs/requirements.md), + Draft v0.2) — functional and non-functional requirements, the locked scope + decisions (D1–D8) and resolved parameters (P1–P18), the architecture and SPI + design (§6), and the review-driven revisions that shape the SPI (§15). +- **Tier-1 Architecture Decision Records** ([`docs/adr/0001`–`0008`](./docs/adr/)) — + the decisions that gate freezing the published `velocity-spi` contract: record + ADRs, `velocity-spi` as a standalone contract module, SPI capability mix-ins, + the mandatory conformance TCK, HLL restricted to tumbling windows, the + distinct exact→HLL threshold and merge rule, read-your-write as a declared + capability, and the seed/backfill contract. +- **Gradle multi-module skeleton** — Phase 1 modules `velocity-spi`, + `velocity-core`, `velocity-api`, and `velocity-testkit` as compiling stubs, + with `build-logic` convention plugins, the `gradle/libs.versions.toml` version + catalog, and nmcp → Sonatype Central Portal publishing wiring, mirroring the + sibling `../pk-auth` project. +- **Project governance documentation** — `README.md`, `GOVERNANCE.md`, + `SECURITY.md`, `CONTRIBUTING.md`, and this changelog (requirements GR-2/GR-3). + +### Notes + +- Nothing is published to Maven Central yet; the current version is + `0.1.0-SNAPSHOT`. +- v1 will ship two production backends — Postgres (JDBI, tumbling reference) and + Redis (sliding, hot-path reference); see the roadmap in + [`GOVERNANCE.md`](./GOVERNANCE.md) and the v1 acceptance criteria in + requirements §11. + +[Unreleased]: https://github.com/codeheadsystems/velocity-engine/commits/main diff --git a/CLAUDE.md b/CLAUDE.md index 2869a9c..1b2d1bc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,8 +81,9 @@ Dependency versions are centralized in `gradle/libs.versions.toml` (add new libs the header `// SPDX-License-Identifier: BSD-3-Clause` (spotless adds it via `spotlessApply`). Library modules compile with `-Xlint:all -Werror` + Error Prone, so warnings fail the build. - **Coverage gates** (`jacocoTestCoverageVerification`, part of `check`): library modules floor at - LINE ≥70% / BRANCH ≥55%; `velocity-core` raises LINE to ≥80%. Raise floors as coverage improves; a - module only restates limits it *raises*. + **LINE ≥80% / BRANCH ≥70%**; a module may restate a *higher* limit but never lower. Trivial/generated + code (records/DTOs, `**/dto/**`, `**/model/**`, generated DI) is excluded from the denominator, so + the floor applies to logic. See the working model below and `CONTRIBUTING.md` → Definition of Done. - **Spotless must run fresh** — it is marked incompatible with the configuration cache (a google-java-format/Guava class-loading bug). Seeing "Configuration cache entry discarded … spotless…" is expected, not an error. The build cache is also disabled globally in `gradle.properties` for the @@ -94,12 +95,39 @@ Dependency versions are centralized in `gradle/libs.versions.toml` (add new libs for stub modules; `isFailOnError = false` in `velocity.java-conventions` keeps the build green. This self-resolves once a module gains public types — do not "fix" it by weakening anything else. +## Working model — how to write code here (required) + +`CONTRIBUTING.md` → **Definition of Done** is binding for every code change. In short: tests ship +with the code; **≥80% line / ≥70% branch** on the affected module (a hard build gate); **no tests for +trivial code** (records/DTOs/getters and generated code are excluded from coverage — do not pad, and +do not hide real logic in an excluded package); and **a feature is not done until it has an +end-to-end/integration test** (a backend → the `velocity-testkit` conformance TCK; the service → an +HTTP/OpenAPI integration test). + +For any **non-trivial** change, follow this loop before reporting the work complete: + +1. Write the implementation **and** its tests together (unit tests for logic; the e2e/TCK test for a + feature). +2. Run `./gradlew clean build test` (or the affected module's `build`) and confirm it is green, + including `jacocoTestCoverageVerification`. **Never report a change done on a red or skipped gate.** +3. **Invoke the `change-validator` sub-agent** (`.claude/agents/change-validator.md`) via the Agent + tool, telling it what the change is and which requirement/ADR (`FR-*`/`NFR-*`/ADR-N/§11 item) it + implements. It independently checks that the tests are *meaningful* (would fail if the code were + broken) and that the code actually satisfies that requirement and the frozen `velocity-spi` + contract (NFR-17, additive-only). +4. If it returns `CHANGES REQUESTED`, address the findings and re-validate. Only call the change done + after it returns `APPROVE` (or you have explicitly resolved each finding with the user). + +Trivial, no-runtime-surface changes (docs, comments, a pure DTO/record addition, build-comment tweaks) +do not require the sub-agent — but a DTO addition still goes through the build. When in doubt, validate. + ## CI / automation (`.github/`) `ci.yml`'s **`build` job** (`./gradlew clean build test` on JDK 21) is the required status check. Dependabot (`dependabot.yml`) runs **weekly on Tuesday** for the `gradle` and `github-actions` -ecosystems, grouped into one PR each. `auto-dependabot.yml` **auto-approves and auto-merges every -Dependabot PR** (all update types, both ecosystems) once CI is green — so CI is the *only* gate -protecting `main`; keep it meaningful. Auto-merge also needs three repo settings enabled (allow -auto-merge, allow Actions to approve PRs, and branch protection requiring the `build` check) — see the -header comment in `auto-dependabot.yml`. +ecosystems, grouped into one PR each. `auto-dependabot.yml` auto-approves and auto-merges only +**patch/minor** Dependabot PRs once CI is green; **major bumps and all github-actions updates are +held for human review** (a green build runs a dependency's build-time code but doesn't prove a new +release is trustworthy). Auto-merge also needs three repo settings enabled (allow auto-merge, allow +Actions to approve PRs, and branch protection requiring the `build` check) — see the header comment +in `auto-dependabot.yml`. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..0930244 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,123 @@ +# Contributing + +Contributions are welcome. This project is early — the design is settled and +implementation is starting — so the most valuable contributions right now are +faithful to the specification and the frozen contract. Please read this before +opening a pull request. + +## Working agreements + +1. **Spec-first.** [`docs/requirements.md`](./docs/requirements.md) is the + source of truth. Its §2 locked decisions (D1–D8) and §2.1 resolved parameters + (P1–P18) constrain scope; honor them, and cite requirement IDs (`FR-*`, + `NFR-*`, `P*`, `D*`, `§15 R*`) in code comments, commits, and ADRs where they + apply. +2. **ADR-first for design changes.** Non-trivial, cross-module decisions get an + ADR under [`docs/adr/`](./docs/adr/) (Nygard format, numbered sequentially). + This is **mandatory** for anything touching the **frozen `velocity-spi` + contract** — the SPI interfaces + DTOs and the OpenAPI document are + compatibility surfaces (requirements §15/NFR-17), and SPI evolution is + **additive-only** with new capability fields defaulting to "unsupported." + Read ADRs `0003` (capability mix-ins), `0005`/`0006` (HLL/distinct), `0007` + (read-your-write), and `0008` (seed) before reshaping the SPI. +3. **Dependencies go through the version catalog.** New libraries are added in + [`gradle/libs.versions.toml`](./gradle/libs.versions.toml) — never inline a + version in a module's `build.gradle.kts` — and are justified in the commit + message or an ADR. +4. **No `TODO` in `main`** unless it is paired with a tracking issue link. +5. **Correct → tested → fast.** Don't optimize prematurely. + +## Enforced conventions + +These are gated by the build; `./gradlew clean build test` fails if any is +violated, so run it before you push. + +- **Formatting.** `google-java-format`, applied by Spotless. Run + `./gradlew spotlessApply` to format. +- **SPDX license header.** Every `.java` file must start with + `// SPDX-License-Identifier: BSD-3-Clause`. Spotless adds it via + `spotlessApply` and `spotlessCheck` (part of `check`) enforces it. +- **Warnings are errors.** Library modules compile with `-Xlint:all -Werror` + plus Error Prone; a warning fails the build. +- **Coverage floors** (`jacocoTestCoverageVerification`, part of `check`): + library modules floor at **LINE ≥ 80% / BRANCH ≥ 70%**; a module may restate a + *higher* limit but never a lower one. Trivial/generated code is excluded from + the denominator (see the Definition of Done below), so the floor applies to + logic, not boilerplate. +- **Nullability.** jspecify annotations; make nullness explicit at public + surfaces. + +Two build quirks are expected, not errors: + +- Spotless is marked incompatible with the configuration cache (a + google-java-format/Guava class-loading bug), so + `Configuration cache entry discarded … spotless…` is normal. The build cache + is disabled globally in `gradle.properties` for the same reason. +- `javadoc` prints "No public or protected classes found to document" for stub + modules; `isFailOnError = false` keeps the build green. This self-resolves once + a module gains public types — do not "fix" it by weakening anything else. + +## Definition of Done + +A change is not complete until all of the following hold. This applies to every +contributor, human or AI-assisted. + +1. **Tests exist for the code the change adds or alters.** New or changed + behavior ships with unit tests in the same PR. Code without tests is not done. +2. **≥ 80% line and ≥ 70% branch coverage** on the affected library module, + enforced by `jacocoTestCoverageVerification`. This is a hard gate, not a + target. +3. **Trivial code is exempt — and only trivial code.** Do **not** write tests for + getters/setters, plain records/DTOs, or generated code, and do not pad + coverage with tests that assert nothing. Such code is excluded from the + coverage denominator: JaCoCo already filters record-generated members, and the + build additionally excludes `**/dto/**`, `**/model/**`, and generated DI + classes (`velocity.test-conventions`). A PR that adds *only* DTOs/records may + therefore legitimately add no tests and still pass. **Never park real logic in + an excluded package to dodge the gate** — that is the one thing this carve-out + must not become. +4. **A feature is not done until it has an end-to-end / integration test.** Unit + tests prove units; a feature must also be exercised end to end before it + counts as complete. For a backend, that is the `velocity-testkit` conformance + TCK (`*Scenarios`, ADR `0004`) run against the real engine (Testcontainers / + local emulator). For the service tier, it is an integration test through the + HTTP/OpenAPI surface. These map to the §11 acceptance criteria. +5. **`./gradlew clean build test` passes locally** before the PR is opened. + +The build enforces (2) and (5); (1), (3), and (4) are review responsibilities. +An AI-assisted change must additionally pass the sub-agent validation described +in [`CLAUDE.md`](./CLAUDE.md) (a reviewer agent checks that the tests are +*meaningful* and that the code actually satisfies the requirement/ADR it targets) +before it is considered done. + +## Build & test + +```sh +./gradlew clean build test # full build + all module tests (the CI gate) +./gradlew :velocity-core:test # one module's tests +./gradlew :velocity-core:test --tests 'com.codeheadsystems.velocity.core.FooTest' # one class +./gradlew :velocity-core:test --tests 'com.codeheadsystems.velocity.core.FooTest.myCase' # one method +./gradlew spotlessApply # auto-format + apply the SPDX header +./gradlew jacocoTestReport # coverage HTML at /build/reports/jacoco/ +``` + +**JDK 21** is required; Gradle's toolchain will fetch one if it is not present. + +When you add or change a backend, remember every `velocity-backend-*` must pass +the shared conformance TCK (`*Scenarios`) in `velocity-testkit` for its +**declared** capabilities, including the negative tests (HLL-on-sliding +rejected; read-your-write flagged, not silently wrong, on a `besteffort` +backend). See ADR `0004`. + +## Pull request flow + +1. For anything beyond a small fix, open an issue first so the approach can be + agreed — especially for changes touching the SPI contract. +2. Branch from `main`, keep commits small and atomic, and write clear commit + messages that cite the relevant requirement or ADR. +3. Ensure `./gradlew clean build test` passes locally. +4. Open the pull request against `main`. CI runs the same command on JDK 21 and + is the required status check. + +See [`GOVERNANCE.md`](./GOVERNANCE.md) for the decision model and how one becomes +a maintainer. diff --git a/GOVERNANCE.md b/GOVERNANCE.md new file mode 100644 index 0000000..d6bd553 --- /dev/null +++ b/GOVERNANCE.md @@ -0,0 +1,97 @@ +# Governance + +This document describes how the Velocity Engine is governed: who makes +decisions, how those decisions are recorded, how the roadmap is sequenced, and +how you can become a contributor or maintainer. It is deliberately honest about +the project's youth — this is requirement **GR-2** (governance & support model). + +## Maintainers + +| Maintainer | Role | Contact | +|------------|------|---------| +| Ned Wolpert | Lead maintainer, architect | ned.wolpert@gmail.com | + +**Bus factor: one.** Today this is a single-maintainer project. There is no +foundation, steering committee, or company behind it, and adopters should size +their self-support risk accordingly: a permissive BSD-3-Clause license grants +you broad rights, but a license is not a maintenance guarantee. If you are +considering the engine on a fraud/abuse-critical path, plan to be able to +self-support the code you depend on. + +## Decision model + +While the project has a single maintainer, decisions are made by the maintainer, +informed by issues, discussion, and the constraints already captured in the +specification. As the project attracts contributors, the intent is to move +toward lazy consensus — a proposal that draws no sustained objection within a +reasonable window is accepted — with the lead maintainer as the tiebreaker. + +Two things constrain every decision and are not casually revisited: + +- **The specification.** [`docs/requirements.md`](./docs/requirements.md) is the + source of truth for scope and intent. Its §2 *locked* scope decisions (D1–D8) + and §2.1 *resolved* parameters (P1–P18) were decided at kickoff and constrain + everything below them; changing one is a deliberate act, recorded in an ADR. +- **The compatibility surfaces.** The `velocity-spi` interfaces + DTOs and the + OpenAPI contract are compatibility surfaces (NFR-17). SPI evolution is + **additive-only**, with new capability fields defaulting to "unsupported." + +## How decisions are recorded + +Non-trivial, cross-module decisions are captured as **Architecture Decision +Records** under [`docs/adr/`](./docs/adr/), in +[Nygard format](https://cognitect.com/blog/2011/11/15/documenting-architecture-decisions), +numbered sequentially. ADRs are append-only: when a decision is reversed, a new +ADR supersedes the old one rather than editing history. + +The initial batch (`0001`–`0008`) records the **Tier-1** decisions that gate +freezing the published `velocity-spi` contract — the decisions that are +expensive to get wrong because they would force a re-cut of a published SPI. +Anything that touches the frozen SPI contract is **ADR-first**: write the ADR +before the code. + +## Roadmap & phasing + +The roadmap is phased by module build order, described in requirements +[§6.2](./docs/requirements.md#62-proposed-module-layout-mirrors-pk-auth) and +[§11](./docs/requirements.md#11-acceptance-criteria-v1-done). The phases are not +speculative wishlist items — each gates the next: + +- **Phase 1 (in progress).** The contract and engine skeleton: `velocity-spi`, + `velocity-core`, `velocity-api`, `velocity-testkit`. The Tier-1 ADRs must be + settled here because the SPI is frozen at the end of this work. +- **Phase 2 (v1).** The two committed production backends — + `velocity-backend-jdbi` (Postgres, tumbling reference) and + `velocity-backend-redis` (sliding, hot-path reference) — plus the + `velocity-dropwizard` service and the generated `velocity-client-java`. Both + window shapes are exercised through the SPI's sliding/tumbling capability split + before the contract is considered stable. v1 "done" is defined by the + acceptance criteria in §11. +- **Phase 3.** `velocity-backend-dynamodb` (volume). +- **Phase 4.** `velocity-backend-s3` (cost, approximate). + +Enterprise-readiness commitments (governance, security disclosure, supply-chain +provenance, admin audit log, data export/migration, cost model, load-tested +production-readiness proof) are tracked as the **GR-** requirements in +[§16](./docs/requirements.md#16-governance--enterprise-readiness). This document +and [`SECURITY.md`](./SECURITY.md) are themselves GR-2 and GR-3. + +## Becoming a contributor + +Contributions are welcome. The path is ordinary open-source practice: + +1. Read [`CONTRIBUTING.md`](./CONTRIBUTING.md) for the build, the enforced + conventions, and the ADR-first expectation for design changes. +2. For anything beyond a small fix, open an issue first so the approach can be + agreed before you invest in a pull request — especially for anything touching + the `velocity-spi` contract, which is a frozen compatibility surface. +3. Submit a pull request against `main`. CI (`./gradlew clean build test` on + JDK 21) is the required status check. + +## Becoming a maintainer + +There is no committee to petition. Maintainership is earned through a sustained +track record of high-quality contributions and reviews, and is extended by the +lead maintainer. Growing the maintainer set — and thereby the bus factor — is an +explicit goal for the project, not an afterthought. If you are contributing +regularly and want to take on more responsibility, say so. diff --git a/README.md b/README.md new file mode 100644 index 0000000..2c65262 --- /dev/null +++ b/README.md @@ -0,0 +1,134 @@ +# velocity-engine + +> **Status: pre-release (0.1.0-SNAPSHOT).** The design is complete and the +> Tier-1 architecture is decided; **implementation is just starting.** The +> modules today are compiling skeletons (`package-info.java` and stubs) — there +> is **no functional counting code, no benchmarks, and nothing published to +> Maven Central yet.** This README describes what the engine *is designed to be* +> and points at the authoritative design; do not read it as a description of +> shipped, working features. + +A backend-agnostic **counting substrate** for the JVM: it records events and +answers *"how many / how much / how many distinct X has happened for this +subject in the last N units of time?"* — the primitive behind rate limiting, +fraud/abuse detection, quota enforcement, and anomaly monitoring. + +## What it is — and what it is not + +The Velocity Engine is a **counting substrate, not a decisioning platform.** It +records events and returns aggregated velocities (counts, sums, distinct +cardinalities) over time windows. **The caller owns every threshold, rule, and +ALLOW/REVIEW/DENY decision.** A rules/decisioning layer can be built *on top of* +the engine, but that is explicitly out of scope for v1. If you are evaluating +this for a fraud or abuse path, read that sentence again: the counting is ours, +the decision is yours. (This is requirement **GR-1** — positioning honesty.) + +The core abstraction is the **feature**: a named velocity counter — +`(subject-type, aggregation, window[, dimension])` given a stable name such as +`card.count.1h` or `card.distinct.ip.1d`. One `record()` call fans a single +event out to every configured feature it matches, updating them together and +returning the post-update velocities. + +## Design tenets + +- **Pluggable backends.** The storage backend is an SPI in its own module + (`velocity-spi`); nothing in the engine hard-depends on any one datastore. +- **Capability-driven.** Each backend *declares* what it supports — which + windows, which aggregations, exactness, maximum retention, read-your-write + strength, idempotency. The engine validates every feature definition against + the backend it targets and **fast-rejects** what a backend cannot do rather + than silently degrading. It never pretends a backend can do something it can't. +- **Dropwizard is *the* framework.** The only supported standalone runtime is + Dropwizard. Unlike the sibling `pk-auth` project, the engine deliberately does + **not** ship Spring, Micronaut, or any other framework adapter. The counting + logic is a plain library, so it *can* be embedded in an existing process, but + the only shipped, supported service is the Dropwizard one. +- **Portable config & data.** Feature definitions import/export as **YAML**; + event records import as **YAML or JSON**. Configuration and onboarding data are + versionable, reviewable, and portable between environments. + +## Architecture at a glance + +``` + HTTP/JSON clients ─▶ velocity-dropwizard (stateless delivery tier) + │ calls + embedded library ─▶ velocity-core (record()/query(), fan-out, YAML I/O) + │ depends on + velocity-spi (the contract: capability mix-ins + DTOs) + │ implemented by + ┌───────────────┬─────┴──────┬────────────────┬───────────────┐ + backend-jdbi backend-redis backend-dynamodb backend-s3 testkit + (Postgres, (sliding, (volume) (cost, (in-memory + tumbling, v1) hot-path, v1) approx) + TCK) +``` + +The central design seam is **where windowing lives**: `velocity-core` resolves +fan-out to backend-neutral *intents* `(feature, subject, aggregation, +member|value)` and does **not** compute buckets or storage keys. The **backend** +owns everything time-shaped — bucket keying, sliding-vs-tumbling semantics, +eviction/TTL, and its own key schema. There is intentionally **no single +storage-key schema** (Redis ZSET keys ≠ DynamoDB items ≠ S3 paths). The SPI is +therefore not one fat interface but **capability mix-ins**: a backend implements +only what it declares. + +## Modules + +Phase 1 modules exist as skeletons today; later phases are planned. + +| Module | Purpose | Phase | +|--------|---------|-------| +| `velocity-spi` | The contract: capability mix-ins (`CountStore`, `SumStore`, `DistinctStore`, `SlidingSupport`, `TumblingSupport`, `SeedSupport`), `BackendCapabilities`, shared DTOs | 1 | +| `velocity-core` | Engine: `record()`/`query()`, feature/fan-out resolver, window/aggregation model, YAML I/O | 1 | +| `velocity-api` | OpenAPI 3.1 document + shared API DTOs (source of truth for the client) | 1 | +| `velocity-testkit` | In-memory SPI implementation, fixtures, and the **conformance TCK** every backend must pass | 1 | +| `velocity-backend-jdbi` | PostgreSQL/JDBI backend — **v1 tumbling reference** | 2 | +| `velocity-backend-redis` | Redis backend (sliding, exact, low-latency) — **v1 sliding/hot-path reference** | 2 | +| `velocity-dropwizard` | Dropwizard bundle, Jersey resources, Dagger wiring, API-key auth | 2 | +| `velocity-client-java` | Java client generated from the OpenAPI document | 2 | +| `velocity-backend-dynamodb` | DynamoDB backend (volume; single-table, atomic ADD) | 3 | +| `velocity-backend-s3` | S3 backend (cost; tumbling, approximate) | 4 | + +**Both Postgres and Redis are committed v1 production backends** — Postgres +proves exact record-and-return against infrastructure most adopters already run; +Redis proves the true-sliding, low-latency hot path the fraud/gateway use cases +need. DynamoDB (volume) and S3 (cost) follow. + +## Stack + +- **Java 21** (language + toolchain), Gradle multi-module with `build-logic` + convention plugins. +- **Dagger 2** for compile-time dependency injection — no Spring, no runtime + reflective IoC container. +- **Dropwizard** for the service tier (the only standalone runtime). +- **`BigDecimal` denominated in cents** for money — never binary floating point. +- **jspecify** nullability annotations; **Jackson 3** for JSON. +- **BSD-3-Clause** across all modules, SPDX headers on every source file. + +## Build + +```sh +./gradlew clean build test # full build + all module tests (the CI gate) +``` + +Requires **JDK 21** (Gradle's toolchain will fetch one if it is not present). + +## Where the design lives + +This project is design-first: the specification and the decision records are the +source of truth, ahead of the code. + +- **[`docs/requirements.md`](./docs/requirements.md)** — the authoritative spec. + Start with §1 (what it is), §2.1 (locked parameters P1–P18), §6 (architecture + and the SPI), and §15 (the review-driven revisions that shape the SPI). +- **[`docs/adr/`](./docs/adr/)** — the Architecture Decision Records. The Tier-1 + batch (`0001`–`0008`) records the decisions that **gate freezing the published + `velocity-spi` contract**. +- **[`CONTRIBUTING.md`](./CONTRIBUTING.md)** · **[`GOVERNANCE.md`](./GOVERNANCE.md)** + · **[`SECURITY.md`](./SECURITY.md)** · **[`CHANGELOG.md`](./CHANGELOG.md)**. + +The project deliberately mirrors the conventions of the sibling project at +`../pk-auth` (build-logic, version catalog, nmcp publishing, ADR style). + +## License + +BSD-3-Clause — see [`LICENSE`](./LICENSE). Copyright © 2026 Ned Wolpert. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..cb1c69a --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,92 @@ +# Security Policy + +The Velocity Engine is designed to sit on abuse- and fraud-critical paths, so +security reports are taken seriously and handled with priority. Thank you for +helping keep it and its users safe. This is requirement **GR-3** (coordinated +security disclosure). + +## Supported versions + +The project is **pre-1.0 and pre-release**: the only supported line is the +development trunk on `main`. There are no published releases yet, so there is no +back-port window — fixes land on `main`. + +| Version | Supported | +| ------- | --------- | +| `main` (pre-release, `0.x-SNAPSHOT`) | :white_check_mark: | +| Anything else | :x: (nothing else exists yet) | + +Once the project cuts releases, this table will move to "fixes land on the +latest published line; Maven Central releases are immutable, so a fix ships as a +new release rather than a re-publish — upgrade to pick it up." + +## Reporting a vulnerability + +**Please do not open a public issue, pull request, or discussion for a suspected +vulnerability.** Public disclosure before a fix is available puts downstream +users at risk. + +Use one of these private channels instead: + +1. **GitHub private vulnerability reporting (preferred).** On the repository, go + to the **Security** tab → **Report a vulnerability**. This creates a private + advisory thread visible only to you and the maintainer. +2. **Email.** Send details to **ned.wolpert@gmail.com** with + `velocity-engine security` in the subject line. + +Please include, as far as you can: + +- The affected module(s) and commit/version (e.g. `velocity-core` at commit + `abc1234`). +- A description of the issue and its impact — what an attacker can do. +- Steps to reproduce — a minimal proof of concept, failing test, or request + sequence is ideal. +- Any relevant configuration (backend, feature definitions, auth setup). + +## What to expect + +- **Acknowledgement** within **3 business days**. +- An initial assessment (severity, affected surface, whether it reproduces) + within **10 business days**. +- A fix targeted within **90 days** of triage; complex issues may take longer, + and we will say so. +- **Coordinated disclosure.** We ask that you keep the report private until a fix + is available and a GitHub Security Advisory is published. We are happy to + credit you unless you prefer to remain anonymous. + +Because this is a single-maintainer project (see [`GOVERNANCE.md`](./GOVERNANCE.md)), +these are good-faith targets, not a staffed SLA. Response may be slower during +travel or other absences; we will communicate if a timeline slips. + +## Scope + +This policy covers the code in this repository: the `velocity-*` library +modules, the Dropwizard service, the wire contract (OpenAPI), and the generated +client, as those land. + +Security is **capability-driven** in this engine, and several security-relevant +behaviors are explicit requirements rather than incidental — reports against +them are in scope and welcome: + +- **Namespace-scoped authorization** (NFR-21): each API key is bound to an + allowed-namespace set; a request whose path namespace is outside that scope is + denied by default. Cross-tenant read/poisoning is a security bug. +- **PII at rest** (FR-38): exact DISTINCT dimension values (IPs, tokens, device + IDs) are keyed-hashed at rest with a per-namespace salt; raw PII must not + persist. +- **Hot-path failure contract** (NFR-19): `record()`/`query()` must fail fast + within the caller deadline and return a distinguishable `UNAVAILABLE` / + `DEADLINE_EXCEEDED` — never a silent `0` that a caller could misread as "no + velocity." +- **Self-overload protection** (NFR-22): bounded concurrency / load-shedding and + per-namespace fairness, so one tenant's flood cannot degrade another's latency. + +This policy does **not** cover the responsibilities a deploying operator +retains — TLS termination, network ingress, secrets management, backend +datastore hardening, and the decisioning/authorization logic the caller builds +*on top of* the engine's counts (the engine is a counting substrate, not a +decisioning platform — see [`README.md`](./README.md) and requirement GR-1). + +> **Maturity note.** This project is pre-release, largely AI-authored, and has +> not undergone a formal third-party security review unless a note in the +> repository says otherwise. Evaluate it accordingly before production use. diff --git a/build-logic/src/main/kotlin/velocity.test-conventions.gradle.kts b/build-logic/src/main/kotlin/velocity.test-conventions.gradle.kts index 0a30dc8..1f5fc99 100644 --- a/build-logic/src/main/kotlin/velocity.test-conventions.gradle.kts +++ b/build-logic/src/main/kotlin/velocity.test-conventions.gradle.kts @@ -23,38 +23,61 @@ jacoco { toolVersion = "0.8.12" } +// Coverage excludes trivial/generated code so the 80% line / 70% branch gate (below) applies to +// LOGIC, not boilerplate. JaCoCo already filters record-generated accessors/equals/hashCode/toString, +// so a pure record/DTO contributes ~nothing to the denominator; these patterns additionally drop +// DTO/model packages and generated DI code from BOTH the report and the gate. A change that adds only +// DTOs/records therefore passes without artificial tests — the "no tests for trivial code" carve-out +// (see CONTRIBUTING.md → Definition of Done). Non-trivial logic MUST NOT be parked in an excluded +// package to dodge the gate. +val coverageExcludes = listOf( + "**/dto/**", + "**/model/**", + "**/dagger/**", + "**/Dagger*", + "**/*_Factory.class", + "**/*_MembersInjector.class", + "**/*_Provide*Factory.class", + "**/package-info.class", + "**/module-info.class", +) + tasks.named("jacocoTestReport") { dependsOn(tasks.named("test")) reports { xml.required.set(true) html.required.set(true) } + classDirectories.setFrom( + files(classDirectories.files.map { fileTree(it) { exclude(coverageExcludes) } }), + ) } tasks.named("check") { dependsOn(tasks.named("jacocoTestReport")) } -// Baseline coverage gate for published library modules: LINE ≥70%, BRANCH ≥55%. Keyed off -// `java-library` (applied by velocity.library-conventions) so it covers every library module but -// NOT the example apps, which apply test-conventions only for the JaCoCo report. A module may -// layer on a STRICTER rule in its own build.gradle.kts — Gradle enforces all rules, so a module -// override need only state the limits it raises, never restate this floor. Floors are static: -// bump them as coverage improves (jacocoTestReport's HTML shows current numbers). -// -// Dagger-generated code (Dagger*, *_Factory, *_MembersInjector) is excluded per-module in the -// modules that use Dagger (the Dropwizard tier); see NFR-14. +// Coverage gate for published library modules: LINE ≥80%, BRANCH ≥70%. Keyed off `java-library` +// (applied by velocity.library-conventions) so it covers every library module but NOT the example +// apps, which apply test-conventions only for the report. A module may layer on a STRICTER rule in +// its own build.gradle.kts — Gradle enforces all rules, so an override need only state the limits it +// raises, never restate this floor. This gate is what makes "tests required, ≥80% coverage" a +// REQUIREMENT and not a suggestion; the exclusions above keep it honest about trivial code. See +// CONTRIBUTING.md → Definition of Done and requirements NFR-14. plugins.withId("java-library") { tasks.named("jacocoTestCoverageVerification") { + classDirectories.setFrom( + files(classDirectories.files.map { fileTree(it) { exclude(coverageExcludes) } }), + ) violationRules { rule { limit { counter = "LINE" - minimum = "0.70".toBigDecimal() + minimum = "0.80".toBigDecimal() } limit { counter = "BRANCH" - minimum = "0.55".toBigDecimal() + minimum = "0.70".toBigDecimal() } } } diff --git a/docs/adr/0002-velocity-spi-standalone-module.md b/docs/adr/0002-velocity-spi-standalone-module.md index 57ed278..2cba733 100644 --- a/docs/adr/0002-velocity-spi-standalone-module.md +++ b/docs/adr/0002-velocity-spi-standalone-module.md @@ -59,9 +59,15 @@ velocity-core ──depends-on──► velocity-spi ◄──depends-on─ `velocity-core` depends on `velocity-spi`. Every `velocity-backend-*` depends on `velocity-spi`. Backends **do not** depend on `velocity-core` — a backend author needs the contract, never the engine's internals. `velocity-spi` itself depends on as little as possible (jspecify annotations, -the money type; no Dropwizard, no Dagger, no Jackson-bound types beyond what the DTOs strictly need) -so that implementing a backend does not drag the engine's framework choices onto the backend author's -classpath. +the money type; no Dropwizard, no Dagger, **no Jackson**) so that implementing a backend does not drag +the engine's framework choices onto the backend author's classpath. + +**Resolved (v0.2 re-review):** the SPI is **serialization-neutral**. Its DTOs (intents, +`FeatureResult`/`FeatureValue`, `BackendCapabilities`; see [ADR 0009](0009-hot-path-result-dto.md)) +are plain records with no Jackson binding; Jackson 3 lives in `velocity-core` / the wire layer, which +owns JSON (NFR-4). The skeleton originally put `jackson` on `velocity-spi`'s `api` surface — that +would have frozen Jackson into the compatibility surface (NFR-17); it was removed so the code matches +this ADR. `velocity-spi` is the versioned compatibility surface named in NFR-17: it evolves **additive-only**, with new capability fields defaulting to "unsupported." diff --git a/docs/adr/0006-distinct-exact-hll-threshold.md b/docs/adr/0006-distinct-exact-hll-threshold.md index d795514..5da1ef3 100644 --- a/docs/adr/0006-distinct-exact-hll-threshold.md +++ b/docs/adr/0006-distinct-exact-hll-threshold.md @@ -58,6 +58,17 @@ Rationale: native HLL implementation use its stock semantics without re-tuning. The 12 KiB sketch also sits far under DynamoDB's 400 KB item ceiling, so an HLL bucket always fits where an exact set might not. + **Interop caveat (v0.2 re-review):** equal precision `p` is *necessary but not sufficient* for two + HLL implementations to interoperate — Redis's native HLL and any JVM HLL library use different + register encodings and hash functions, so their serialized sketches neither merge nor deserialize + across implementations. This is harmless while a sketch never leaves the backend that produced it. + We therefore scope an HLL sketch as **opaque, same-implementation-only bytes**: a backend may use + its native/library HLL internally, but a serialized `hllSketch` (e.g. in a seed/export, ADR 0008) + is only meaningful to the same implementation that wrote it. **Cross-backend migration of HLL + sketches is explicitly not supported** in v1 (only exact members / counts migrate across backends); + a canonical wire HLL (pinned algorithm + hash + layout) can be added additively later if that need + arises. + Both numbers are **defaults on `BackendCapabilities`**, not hardcoded in core (P18): a backend documents its own values at dev time (§8 note, OQ-A) and may raise/lower them, but the TCK (ADR 0004) verifies whatever it declares. diff --git a/docs/adr/0007-read-your-write-declared-capability.md b/docs/adr/0007-read-your-write-declared-capability.md index dadfa70..721874a 100644 --- a/docs/adr/0007-read-your-write-declared-capability.md +++ b/docs/adr/0007-read-your-write-declared-capability.md @@ -4,7 +4,9 @@ Date: 2026-07-18 ## Status -Accepted. +Accepted. **Completed by [ADR 0009](0009-hot-path-result-dto.md)**, which freezes the rest of the +hot-path result shape (the failure variant, per-feature apply status, and definition-version stamp) +that this ADR deferred. ## Context diff --git a/docs/adr/0008-seed-backfill-contract.md b/docs/adr/0008-seed-backfill-contract.md index a0be9a4..874afed 100644 --- a/docs/adr/0008-seed-backfill-contract.md +++ b/docs/adr/0008-seed-backfill-contract.md @@ -47,6 +47,10 @@ BucketValue = (bucketStart, bucketEnd, aggregate) ADR 0006 — members are keyed-hashed at rest, FR-38) or with a pre-computed HLL sketch of the agreed precision `p` (ADR 0006, tumbling only per ADR 0005). A backend rejects a seed whose distinct representation it cannot store (e.g. an HLL sketch on a sliding feature — ADR 0005). + An `hllSketch` is **opaque, same-implementation-only bytes** (ADR 0006 interop caveat): a backend + accepts an `hllSketch` seed only from its own implementation and MUST reject a foreign one. **Exact + members migrate across backends; HLL sketches do not** — cross-backend HLL migration is out of scope + for v1. - Seeded state is **flagged onboarding-seeded**, distinct from organically recorded state (FR-32), so it is auditable (§16 GR-5 admin audit log) and distinguishable in exports (GR-6). @@ -83,9 +87,11 @@ that an unrepresentable seed (e.g. sliding HLL, or a seed on a backend that does identically (FR-14) — instead of the wrong-apportionment failure a single-total seed would cause. - **Optionality is honest.** A backend that genuinely cannot represent seeded buckets declares so and rejects seeds cleanly, rather than the engine pretending seeding works everywhere (§1). -- **Migration story has a foundation.** Per-bucket seed is the ingest side of the export/migration - path business adopters asked for (GR-6): export counter data as bucket values, seed it into another - backend. +- **Migration story has a foundation (for exact state).** Per-bucket seed is the ingest side of the + export/migration path business adopters asked for (GR-6): export counter data as bucket values, seed + it into another backend. This holds for COUNT, SUM, and **exact** DISTINCT; **HLL-sketch distinct + does not migrate across backends** (opaque, same-implementation-only — ADR 0006), so GR-6's + cross-backend migration is scoped to exact aggregates in v1. ### Negative diff --git a/docs/adr/0009-hot-path-result-dto.md b/docs/adr/0009-hot-path-result-dto.md new file mode 100644 index 0000000..03f0a49 --- /dev/null +++ b/docs/adr/0009-hot-path-result-dto.md @@ -0,0 +1,131 @@ +# 9. The hot-path result is a frozen value-or-failure sum type + +Date: 2026-07-18 + +## Status + +Accepted. Completes [ADR 0007](0007-read-your-write-declared-capability.md), which froze only the +read-your-write level onto the result and explicitly deferred the rest of the result's shape. + +## Context + +Every `apply()` (record) and `query()` call on the `velocity-spi` data-plane mix-ins +([ADR 0003](0003-spi-capability-mixins.md)) returns a **feature-value result**. Because +`velocity-spi` is a *published* compatibility surface with additive-only evolution (NFR-17), the +*shape* of that result — the fields it can carry — is frozen the moment the SPI ships. Behavior can +be filled in later; the envelope cannot grow a new field without re-cutting the published DTO. + +The v0.2 review (§15) put four decision-bearing pieces of information onto this one result, but +tiered them across phases: + +- **read-your-write level** (§15 R2 / ADR 0007) — Tier-1, *already frozen* onto the result. +- **per-feature apply status** `applied|failed|skipped` (§15 R3 / FR-34) — was Tier-2. +- **a distinguishable failure** `UNAVAILABLE|DEADLINE_EXCEEDED|…`, never a silent `0` + (§15 R2 second half / NFR-19) — was Tier-2, and ADR 0007 explicitly deferred it "to its own + Tier-2 ADR." +- **the feature-definition version/hash** the value was computed under (§15 R12 / FR-40) — was Tier-3. + +The architect and both adopter reviews (fraud/rules and gateway) independently reached the same +conclusion on re-review: these are not *behavior*, they are the *shape of the result*. A result that +can only ever *be* a value cannot later *become* a distinguishable `DEADLINE_EXCEEDED`; adding that +variant in phase 2 re-cuts the published DTO — the exact phase-3 re-cut the §15 tiering exists to +prevent. For a synchronous authorization caller, "did this count fail, or is it truly zero?" is *the* +correctness question, and the sliding cardinality-cap outcome ([ADR 0005](0005-hll-tumbling-only.md)) +needs a defined result too. This result rides on every call, so it is the single highest-value thing +to freeze correctly. + +The *enforcement* of these fields — deadline plumbing, load-shedding, hot-reload skew handling — stays +Tier-2/Tier-3. Only the **envelope** is pulled into the Tier-1 freeze here. + +## Decision + +Freeze the data-plane result as a **value-or-failure sum type**, carrying all decision-bearing fields, +before the SPI ships. Fields whose behavior lands later are present in the frozen shape from day one +and populated when the behavior arrives. + +``` +FeatureResult = Success { FeatureValue } + | Failure { FailureCode code, String detail } + +FeatureValue = { FeatureRef feature, + Number value, // BigDecimal cents for SUM (P3); long for COUNT/DISTINCT + Exactness exactness, // EXACT | APPROXIMATE (FR-7) + ReadYourWriteLevel readYourWriteLevel, // ATOMIC | SNAPSHOT | BESTEFFORT (ADR 0007) + String definitionVersionHash, // FR-40; nullable until R12 lands + WindowBounds windowBounds, // FR-7 + Instant asOf } // FR-7 + +FailureCode = UNAVAILABLE // backend down/partitioned (NFR-19) + | DEADLINE_EXCEEDED // caller deadline hit before a result (NFR-19) + | CARDINALITY_CAP_EXCEEDED // sliding exact-distinct over its cap (ADR 0005) + | UNSUPPORTED_WINDOW // FR-13 + | VALIDATION // malformed request + | ... // extensible; new codes are an ADDITIVE change + +ApplyResult = { List perFeature, ... } +PerFeature = { FeatureRef feature, + ApplyStatus status, // APPLIED | FAILED | SKIPPED (FR-34) + FeatureResult result } +``` + +Rules that make the freeze hold: + +1. **A result is a value OR a distinguishable failure — never an ambiguous `0`.** A backend that is + down/slow returns `Failure{UNAVAILABLE}` / `Failure{DEADLINE_EXCEEDED}`, so the caller can choose + fail-open vs fail-closed deterministically. Returning `Success{value: 0}` for "I don't know" is + forbidden and is a `velocity-testkit` TCK negative test ([ADR 0004](0004-mandatory-conformance-tck.md)). +2. **`FailureCode` is an extensible enum; adding a code is additive** (consumers must tolerate unknown + codes — treat as a generic failure). This is the one place growth is expected. +3. **`definitionVersionHash` is part of the frozen shape now, nullable until FR-40's behavior lands.** + Same for populating apply-status semantics — the field exists at freeze; its value is filled in + when the feature (hot-reload versioning, load-shedding) ships. +4. **The result carries the read-your-write level *per feature value*, not per call** — a single + `record()` fans out across backends (ADR 0007), so each returned value states its own level. +5. **These DTOs are serialization-neutral** (see [ADR 0002](0002-velocity-spi-standalone-module.md), + amended): plain `velocity-spi` records with no Jackson binding; `velocity-core` and the wire layer + own JSON. The AR-5 HTTP problem model gains matching `DEADLINE_EXCEEDED` and `OVERLOADED` types. + +## Consequences + +### Positive + +- The SPI can be frozen and implemented behind without a foreseeable phase-2/3 re-cut of the + universal result DTO — the concrete goal of §15's tiering. +- The gateway operator's milestone-1 test and the fraud adopter's "all five fields in one response" + requirement are satisfiable against the *frozen* contract, not a future one. +- `CARDINALITY_CAP_EXCEEDED` gives ADR 0005's sliding exact-distinct cap a defined, typed outcome + instead of an undefined reject/truncate. + +### Negative + +- More surface is frozen up front, including fields (`definitionVersionHash`, some `FailureCode`s) + whose *behavior* does not exist in phase 1 — a small "declared but not yet populated" gap that must + be documented so implementers don't assume a populated value. +- Callers must handle the `Failure` branch from day one, even before load-shedding/deadline + enforcement is wired — slightly more caller code for behavior that is initially rare. + +### Neutral + +- The *enforcement* ADRs remain to be written (Tier-2: the deadline/bounded-failure behavior of + NFR-19; load-shed/fairness NFR-22). This ADR freezes only the shape they will populate. + +## Alternatives considered + +- **Freeze only the value; add the failure variant additively later (ADR 0007's deferral).** Rejected: + a sum type cannot gain a `Failure` arm additively without changing the published type; every + consumer's exhaustiveness/deserialization breaks. This is the re-cut we are preventing. +- **Model failure out-of-band (exceptions instead of a result variant).** Rejected for the *per-feature* + case: one fan-out `record()` can partially fail, so failure must be expressible per returned feature + alongside successes in the same `ApplyResult`, which an all-or-nothing thrown exception cannot do. + +## References + +- Requirements §15 R2/R3/R12, FR-2, FR-7, FR-34, FR-40, NFR-19, NFR-17, AR-5. +- [ADR 0003](0003-spi-capability-mixins.md), [ADR 0004](0004-mandatory-conformance-tck.md), + [ADR 0005](0005-hll-tumbling-only.md), [ADR 0007](0007-read-your-write-declared-capability.md). + +## Open follow-ups + +- Tier-2 ADR: hot-path deadline & bounded-failure *behavior* (BR-6 item (o)) — how the deadline is + plumbed and enforced, and how `UNAVAILABLE`/`DEADLINE_EXCEEDED` are produced. +- Tier-2 ADR: load-shedding & per-namespace fairness (NFR-22), including the `OVERLOADED` result path. diff --git a/docs/adr/README.md b/docs/adr/README.md index 076524f..f6bebe6 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -6,13 +6,15 @@ numbered sequentially. ADRs are append-only: when a decision is reversed, a new ADR is added and the prior one is marked `Superseded by NNNN`. See [ADR 0001](./0001-record-architecture-decisions.md) for the format itself. -The initial batch (`0001`–`0008`) records the **Tier-1** decisions from requirements +The initial batch (`0001`–`0009`) records the **Tier-1** decisions from requirements [§15](../requirements.md#15-review-driven-revisions-v02) — the decisions that **gate freezing the -published `velocity-spi` contract** and so must be recorded before code. The remaining BR-6 items -(Dagger-for-Dropwizard, track-latest Dropwizard, OpenAPI-first client generation, stateless tier, -server-clock event time, feature-as-core-abstraction, `BigDecimal`-cents money, API-key auth, JDBI -reference backend) and the Tier-2/Tier-3 decisions (hot-path deadline & bounded failure, -namespace-scoped authz, keyed-hash-at-rest) will be added as those phases land. +published `velocity-spi` contract** and so must be recorded before code. `0009` was added after the +v0.2 re-review: it completes `0007` by freezing the *full* hot-path result shape (value-or-failure) +before the SPI ships. The remaining BR-6 items (Dagger-for-Dropwizard, track-latest Dropwizard, +OpenAPI-first client generation, stateless tier, server-clock event time, feature-as-core-abstraction, +`BigDecimal`-cents money, API-key auth, JDBI reference backend) and the Tier-2/Tier-3 *behavior* +decisions (hot-path deadline & bounded-failure enforcement, load-shedding & fairness, namespace-scoped +authz, keyed-hash-at-rest) will be added as those phases land. | # | Status | Date | Title | |---|---|---|---| @@ -24,3 +26,4 @@ namespace-scoped authz, keyed-hash-at-rest) will be added as those phases land. | [0006](./0006-distinct-exact-hll-threshold.md) | Accepted | 2026-07-18 | DISTINCT exact→HLL threshold is a backend-clampable capability with a per-bucket merge rule | | [0007](./0007-read-your-write-declared-capability.md) | Accepted | 2026-07-18 | Read-your-write is a declared capability, not a universal guarantee | | [0008](./0008-seed-backfill-contract.md) | Accepted | 2026-07-18 | Seed/backfill contract is resolved before SPI freeze | +| [0009](./0009-hot-path-result-dto.md) | Accepted | 2026-07-18 | Hot-path result is a frozen value-or-failure sum type (completes 0007) | diff --git a/docs/requirements.md b/docs/requirements.md index 6462d1c..6fa813f 100644 --- a/docs/requirements.md +++ b/docs/requirements.md @@ -1,6 +1,6 @@ # Velocity Engine — Requirements -> Status: **Draft v0.2** · Date: 2026-07-18 · Owner: Ned Wolpert +> Status: **Draft v0.3** · Date: 2026-07-18 · Owner: Ned Wolpert > License: **BSD-3-Clause** · Group: `com.codeheadsystems` · Root package: `com.codeheadsystems.velocity` This document captures the functional and non-functional requirements for the Velocity Engine, @@ -18,6 +18,15 @@ Anything not yet decided lives in [§13 Open Questions](#13-open-questions). > tumbling-only v1) are also corrected inline below. Where §15 amends an earlier clause, the earlier > clause carries a "→ see §15" pointer. +> **v0.3 changelog.** After a second pass by the same four reviewers (evaluating readiness to start +> building), the following were folded in: the hot-path result DTO is **frozen complete in phase 1** +> as a value-or-failure sum type (**[ADR 0009](adr/0009-hot-path-result-dto.md)**, completing ADR 0007); +> `velocity-spi` is **serialization-neutral** (Jackson moves to `velocity-core`); HLL sketches are +> **same-implementation-only** (cross-backend HLL migration dropped from v1); Postgres **and** Redis +> are both committed **production-grade** v1 backends; and new acceptance gates were added +> (idempotency-exactness #15, seed-implemented #16) along with KMS key-custody for R11 and +> `deadline-exceeded`/`overloaded` problem types in AR-5. + --- ## 1. Purpose & Vision @@ -151,9 +160,12 @@ record( key**; cross-key/cross-backend fan-out atomicity is a declared capability, and `record()` reports per-feature apply status. → see [§15 R3](#15-review-driven-revisions-v02). - **FR-2** `record()` SHALL, in the same call, return the post-increment velocities for the affected - `(subject, aggregation, window)` tuples. A caller MAY request a subset of windows/aggregations to - return to bound payload size. The *strength* of "post-increment" (read-your-write) is per D4's - declared capability, not universal. → see [§15 R2](#15-review-driven-revisions-v02). + `(subject, aggregation, window)` tuples, as the frozen **`FeatureResult` value-or-failure** type + ([ADR 0009](adr/0009-hot-path-result-dto.md)): per feature, a value (with read-your-write level, + `exact|approximate`, definition-version hash) or a distinguishable failure, plus an apply status. + A caller MAY request a subset of windows/aggregations to bound payload size. The *strength* of + "post-increment" (read-your-write) is per D4's declared capability, not universal. + → see [§15 R2](#15-review-driven-revisions-v02). - **FR-3** Event time SHALL be the server's ingest clock (D7); for sliding windows the **backend clock is authoritative** (not the pod clock). Any client-supplied timestamp is ignored in v1 (MAY be captured for audit but MUST NOT affect bucketing). → see [§15 R9](#15-review-driven-revisions-v02). @@ -343,8 +355,8 @@ existing event data *in*. This is a first-class concern, distinct from the hot ` │ depends on ┌───────────────────▼─────────────────────────┐ │ velocity-spi (the contract module) │ - │ • VelocityStore • BackendCapabilities │ - │ • shared SPI DTOs (updates, feature value)│ + │ • capability mix-ins • BackendCapabilities│ + │ • intents • FeatureResult (value|failure) │ └───────────────────┬─────────────────────────┘ │ implemented by ┌──────────────┬─────────────┼───────────────┬──────────────┐ @@ -369,9 +381,15 @@ owns bucket keying, sliding/tumbling semantics, eviction, and its own key schema 1. **Data-plane mix-ins** — a backend implements only the ones it declares: `CountStore`, `SumStore`, `DistinctStore`, `SlidingSupport`, `TumblingSupport`, `SeedSupport` (onboarding `seed(...)`, FR-32 — its schema resolved before freeze per - [§15 R3b](#15-review-driven-revisions-v02)). Common ops: `apply(intents) -> featureValues`, - `query(tuples) -> featureValues`, `purge(...)`. `apply`/`query` honor a **caller deadline** and - return a **read-your-write level** per result ([§15 R2](#15-review-driven-revisions-v02)). + [§15 R3b](#15-review-driven-revisions-v02)). Common ops: `apply(intents) -> ApplyResult`, + `query(tuples) -> FeatureResult[]`, `purge(...)`. `apply`/`query` honor a **caller deadline**, and + the result is a **frozen value-or-failure sum type** — `FeatureResult = Success{FeatureValue} | + Failure{FailureCode}` — carrying, per feature, the read-your-write level, `exact|approximate`, + the definition-version hash, and (for `apply`) the per-feature apply status. This full shape is + frozen in phase 1 (**[ADR 0009](adr/0009-hot-path-result-dto.md)**), so a stalled backend returns a + distinguishable `UNAVAILABLE`/`DEADLINE_EXCEEDED`, never a silent `0`. The SPI DTOs are + **serialization-neutral** (plain records, no Jackson; JSON lives in `velocity-core`/the wire layer — + [ADR 0002](adr/0002-velocity-spi-standalone-module.md), [§15 R2b](#15-review-driven-revisions-v02)). 2. **`BackendCapabilities`** — the descriptor: supported windows (duration/type/exactness), supported aggregations, `distinctHllSliding`, distinct exact-cardinality clamp & default threshold, **`maxRetention`** (FR-22a), read-your-write level (`atomic|snapshot|besteffort`), idempotency @@ -385,7 +403,7 @@ including negative tests ([§15 R7](#15-review-driven-revisions-v02)). | Module | Purpose | Build phase | Publishes | |--------|---------|-------------|-----------| -| `velocity-spi` | **The contract module**: `VelocityStore`, `BackendCapabilities`, shared SPI DTOs | 1 | ✅ | +| `velocity-spi` | **The contract module**: capability mix-ins, `BackendCapabilities`, serialization-neutral SPI DTOs (`FeatureResult`) | 1 | ✅ | | `velocity-core` | Engine, feature/fan-out resolver, window/aggregation model, YAML I/O (depends on `velocity-spi`) | 1 | ✅ | | `velocity-api` | OpenAPI 3.1 document + shared API DTOs (source of truth) | 1 | ✅ (spec artifact) | | `velocity-testkit` | In-memory `velocity-spi` impl, fixtures, Testcontainers helpers | 1 | ✅ | @@ -422,7 +440,10 @@ including negative tests ([§15 R7](#15-review-driven-revisions-v02)). on the shared API DTOs + a standard HTTP client. - **AR-5** Errors SHALL use a consistent problem model (RFC 9457 `application/problem+json` recommended) distinguishing: unknown namespace, unsupported window, validation error, backend - unavailable, rate-limited. + **unavailable**, **`deadline-exceeded`** (distinct from unavailable — a caller's fail-open/closed + policy may differ; NFR-19), **`overloaded`** (load-shed, NFR-22), and rate-limited. These mirror the + SPI `FailureCode`s ([ADR 0009](adr/0009-hot-path-result-dto.md)) so a wire caller can branch the same + way an embedded caller does. --- @@ -494,8 +515,10 @@ FR-12, FR-22a) rather than papering over differences. (proven by Testcontainers-backed test). 2. **`velocity-backend-redis` (sliding, hot-path) is in v1** and implements true-sliding COUNT/SUM and exact sliding DISTINCT (ZSET-bounded), proven exact + read-your-write (`atomic`) under - concurrent load. *(Two backends in v1 → the SPI's sliding/tumbling capability split is exercised - before freeze.)* + concurrent load. *(Both Postgres and Redis are committed v1 **production-grade** backends — both + are load-tested with published SLOs and DR behavior per [GR-8](#16-governance--enterprise-readiness), + not just TCK-passing — and the two window shapes exercise the SPI's sliding/tumbling split before + freeze.)* 3. **SPI conformance TCK** ([§15 R7](#15-review-driven-revisions-v02)): both v1 backends pass the shared `*Scenarios` suite in `velocity-testkit`, including negative tests (HLL-on-sliding rejected; read-your-write on a `besteffort` backend flagged, not silently wrong). @@ -522,6 +545,13 @@ FR-12, FR-22a) rather than papering over differences. 13. All modules build on Java 21 with Gradle, pass coverage gates, and produce Central-publishable signed artifacts (dry-run). 14. Admin **audit log** records purge/seed/config-change actions ([§16](#16-governance--enterprise-readiness)). +15. **Idempotency-exactness** ([§15 R15](#15-review-driven-revisions-v02)): JDBI and Redis declare + idempotency support, and a **retry-storm test** proves COUNT stays exact under duplicate + idempotency-key `record()`s (no double-count → no false fraud hit / false rate-limit trip). +16. **Seed actually implemented** ([§15 R3b](#15-review-driven-revisions-v02), ADR 0008): ≥1 v1 backend + implements `SeedSupport`, and a `SeedSupportScenarios` test proves a **seeded** bucket and a + **recorded** bucket merge identically through the same windowed read (so the seed contract is real + in v1, not shelfware). --- @@ -624,13 +654,24 @@ re-cut of a published SPI at phase 3. block unbounded. New NFR-19. Also: **read-your-write is a declared capability** (`atomic|snapshot|besteffort`), SHALL for exact backends, best-effort + `approximate`-flagged otherwise; reconciles D4, FR-2, NFR-7 (which previously overclaimed a universal guarantee). + **v0.2 re-review update:** the *shape* of the result (the `Failure{UNAVAILABLE|DEADLINE_EXCEEDED|…}` + variant) is **frozen in phase 1** via **[ADR 0009](adr/0009-hot-path-result-dto.md)** — only the + deadline/bounded-failure *enforcement* stays Tier-2. Freezing the shape late would re-cut a + published DTO. +- **R2b — `velocity-spi` is serialization-neutral.** *(Architect re-review, user decision.)* SPI DTOs + are plain records with **no Jackson binding**; Jackson 3 is required in `velocity-core`/the wire + layer, which owns JSON. Keeps the engine's serialization stack off a third-party backend author's + classpath and out of the frozen surface (NFR-17). Amends ADR 0002; the skeleton's `jackson` on the + `velocity-spi` `api` surface was removed. - **R3 — Per-feature apply status + single-backend decision domain.** *(Rules A, Architect B6.)* Since one `record()` can fan out across features on *different* backends (FR-16 binds a feature to one backend), `record()` MUST return a **per-feature apply status** (`applied|failed|skipped`) and the engine MUST document whether fan-out is all-or-nothing or partial (and identify every feature that did not apply). A caller MUST be able to **constrain all features a single decision reads to one backend**, so they share one consistency + latency domain. New FR-34, FR-37, NFR-20. Note DynamoDB - `TransactWriteItems` caps atomic fan-out at 100 items — declare `maxAtomicFanOut`. + `TransactWriteItems` caps atomic fan-out at 100 items — declare `maxAtomicFanOut`. **v0.2 re-review + update:** the per-feature apply-status *field* is **frozen in phase 1** on the result DTO + ([ADR 0009](adr/0009-hot-path-result-dto.md)); gated in acceptance #4. - **R4 — Namespace-scoped authorization (not just authentication).** *(Security d.)* Each API key MUST be **bound to an explicit allowed-namespace set**; a request whose path namespace is outside the key's scope MUST be **denied by default**. Authz is in scope for v1 (today any caller can pass any @@ -639,7 +680,10 @@ re-cut of a published SPI at phase 3. 4, Rules, Architect B4.)* Exact distinct persists raw dimension values (IPs, tokens, device IDs). For DISTINCT dimensions the engine MUST support **keyed hashing/tokenization at rest** (per-namespace salt), so exact-distinct sets never store raw PII; HLL of hashed values is unaffected. New FR-38; - amends NFR-12 (which only covered logging). + amends NFR-12 (which only covered logging). **v0.2 re-review update (key custody):** the salt/key + MUST live in a **separate trust domain (KMS/secret store), not co-resident** with the distinct sets — + otherwise a low-entropy value like an IPv4 (2³² space) is brute-forceable offline after a single DB + dump, defeating the "no raw PII" guarantee. High-entropy dimensions (tokens) are fine either way. - **R15 — Idempotency as the inline default.** *(Security, Rules, Architect B6.)* At-least-once (NFR-8) makes COUNT provably inexact under retries. v1 reference backends (JDBI, Redis) MUST declare idempotency support; an idempotency key is the recommended inline posture; §8 "Exact" means "exact @@ -670,6 +714,9 @@ re-cut of a published SPI at phase 3. - **R12 — Feature values carry their definition version/hash.** *(Rules D.)* Hot-reload (P6) can change a feature's window/threshold under a running rule; every feature value MUST carry the **definition-version/hash** it was computed under, so a caller can detect drift. New FR-40. + **v0.2 re-review update:** the `definitionVersionHash` *field* is **frozen in phase 1** on the result + DTO ([ADR 0009](adr/0009-hot-path-result-dto.md)), nullable until the hot-reload versioning behavior + lands; gated in acceptance #9. - **R13 — Feature-discovery read API.** *(Rules D.)* Add `GET /v1/{namespace}/features` (list + by version) so a rules layer can discover, pin, and diff feature definitions (today only `/capabilities` exists). New FR-41; amends AR-2. @@ -693,6 +740,13 @@ re-cut of a published SPI at phase 3. failure · `NFR-20` single-backend decision domain · `NFR-21` namespace-scoped authz · `NFR-22` self-protection & per-namespace fairness. +**Frozen in phase 1 by [ADR 0009](adr/0009-hot-path-result-dto.md)** (v0.2 re-review): the hot-path +result is a value-or-failure sum type carrying — per feature — the read-your-write level, `exact| +approximate`, the `definitionVersionHash` (FR-40), an apply status (FR-34), and a distinguishable +`FailureCode` (NFR-19). The *shape* is frozen now (only the enforcement behavior of R2/R10/R18 stays +Tier-2/3), because this DTO rides on every `apply()`/`query()` call and could not gain those fields +additively without re-cutting a published contract. + --- ## 16. Governance & Enterprise Readiness @@ -718,9 +772,13 @@ are project/operational commitments to document. - **GR-6 Data export / migration & exit story.** Feature definitions are portable YAML (low config lock-in, good), but counter **data** lock-in lives at the storage layer. Provide a documented path to **export counter data and migrate between backends**, and treat each backend's key schema as a - versioned surface (NFR-17). + versioned surface (NFR-17). Migration covers COUNT, SUM, and **exact** DISTINCT; **HLL-sketch + distinct does not migrate across backends** (opaque, same-implementation-only — ADR 0006/0008), + a v1 scope limit. - **GR-7 Capacity & cost model.** Publish **cost-per-million-events** guidance per backend so TCO is - quantifiable; recommend adopters commit to **one backend** for a pilot rather than running all four. + quantifiable. The project itself commits to **two** production-grade backends in v1 (Postgres + + Redis, per §11 #2 / GR-8), but still recommends an **adopter** pilot on **one** backend rather than + running all of them. - **GR-8 Production-readiness proof.** Beyond functional acceptance (§11), publish **load-tested throughput, tail-latency, and DR/failure behavior** for at least the v1 reference backends (Postgres, Redis), tied to the committed SLOs (OQ-A). diff --git a/velocity-core/build.gradle.kts b/velocity-core/build.gradle.kts index 6462131..4576434 100644 --- a/velocity-core/build.gradle.kts +++ b/velocity-core/build.gradle.kts @@ -27,9 +27,10 @@ dependencies { testRuntimeOnly(libs.logback.classic) } -// velocity.test-conventions sets the baseline gate (LINE ≥0.70, BRANCH ≥0.55) for every library -// module. velocity-core is the engine and is held to a stricter ≥80% line bar (NFR-14); the -// baseline branch floor applies via the convention. +// velocity.test-conventions sets the baseline gate for every library module (now LINE ≥0.80, +// BRANCH ≥0.70). velocity-core is the engine and pins its LINE floor at ≥0.80 explicitly so that, +// even if the shared baseline is ever relaxed, the core stays at the higher bar (NFR-14). Gradle +// enforces all rules, so this restates only the limit it guarantees. tasks.named("jacocoTestCoverageVerification") { violationRules { rule { diff --git a/velocity-spi/build.gradle.kts b/velocity-spi/build.gradle.kts index a91f824..9d23544 100644 --- a/velocity-spi/build.gradle.kts +++ b/velocity-spi/build.gradle.kts @@ -12,8 +12,12 @@ dependencies { // jspecify nullability annotations are part of the published compatibility surface (NFR-17), // so they are `api`, not `compileOnly` as in the baseline java-conventions. api(libs.jspecify) - // SPI DTOs (intents, feature values) serialize with Jackson 3 on the wire (NFR-4). - api(libs.bundles.jackson) + + // NOTE: velocity-spi is deliberately SERIALIZATION-NEUTRAL (ADR 0002, ADR 0009). The SPI DTOs + // (intents, FeatureResult/FeatureValue, BackendCapabilities) are plain records with NO Jackson + // binding, so a third-party backend author does not inherit the engine's serialization stack on + // their classpath as a frozen part of the contract. Jackson 3 lives in velocity-core / the wire + // layer, which owns JSON (NFR-4). Do not add Jackson to this module's `api` surface. testRuntimeOnly(libs.logback.classic) }