diff --git a/docs/DEVELOPMENT_NOTES.md b/docs/DEVELOPMENT_NOTES.md new file mode 100644 index 0000000..8ffd743 --- /dev/null +++ b/docs/DEVELOPMENT_NOTES.md @@ -0,0 +1,167 @@ +# Development notes + +Dated notes on work in progress. This file records what is being changed and why, including +findings that are unflattering to the current implementation. Nothing here describes a released +capability unless it says so. + +--- + +## 2026-08: replacing fixed constants with data-derived parameters + +**What this work is.** We are removing magic numbers from the pipeline and replacing them with +parameters the data determines for itself. A threshold should be a stated percentile of a +distribution the corpus actually produces, not a number someone picked because it sounded strict. + +**Why it became necessary.** Fixed constants survive contact with small, clean test corpora. They +fail quietly on large ones. This work was forced by running the pipeline on very large, dense +personal text, and by beginning to explore organizational corpora, where the same constants that +looked reasonable at small scale turned out to sit in the wrong part of the distribution entirely. +Nothing errored. The numbers simply stopped meaning what their names implied. + +**How we are testing the claim.** By measuring components that had never been measured. Most of what +follows is therefore a defect report on our own code. It is published because a project that reports +only what worked is not one whose measurements anybody should trust, and because each defect below +is an instance of the same underlying pattern: a constant that was never checked against the +distribution it operates in. + +**Status: this work sits on an unmerged branch. Nothing described here is in a release.** Installing +from `main` today gets the previously released behavior. + +### The distance conversion was wrong, and the documentation taught it + +ChromaDB's distance metric is per collection. Our fact collection is created with cosine distance, +where the conversion to similarity is `1 - d`. Collections created without an explicit space use +Chroma's `l2`, which returns a **squared** euclidean distance, so the conversion there is +`1 - d / 2`. + +The code used `1 - d^2 / 2` everywhere. That is wrong for both spaces. A nominal 0.85 similarity +gate was therefore firing at a true cosine of roughly 0.45. + +Two things about this are worth more than the bug itself: + +1. The same wrong conversion existed at three call sites and survived its own fix three times, + because each fix corrected the site that had been noticed and not the others. +2. `docs/core/ARCHITECTURE.md` stated the wrong formula as fact. A document that states a formula is + where the next contributor learns to reimplement it. The formula now lives in exactly one + function, and the architecture doc describes rather than specifies it. + +The lesson we took: enumerate every call site before calling a fix done, and do not let a formula +exist in more than one place. + +### A guard that could never fire + +A faithfulness check compared an authored summary against its supporting facts and flagged the +summary when the best match fell below 0.35. + +Measured against the encoder now in use, randomly paired, unrelated facts from the same corpus have +a median cosine similarity around 0.62, and the 0.1st percentile sits near 0.38. The guard also +takes the maximum over a summary's supporting facts, so every one of them would have to fall below +0.35 simultaneously. In the operating range the threshold provides no discrimination: an unrelated +pair of facts clears it almost as reliably as a related one. + +An earlier draft of this note said random pairs have a minimum similarity of 0.358 and that the +guard could therefore never fire. That was a sample minimum over 600 pairs. Measured across roughly +eight million within-corpus pairs the true minimum is 0.276, and about 0.015% of pairs fall below +0.35. The guard is effectively unreachable for any on-topic English summary, which is the claim the +evidence supports; "never" is a stronger claim than the data licenses. + +Its own source comment already said the value was inferred rather than calibrated. The comment was +correct and nobody acted on it. + +The general form of this problem: a threshold expressed as an absolute number is meaningless without +the distribution it sits in. We now check candidate thresholds against a random-pair null before +trusting them, and we have started expressing thresholds as percentiles of a corpus's own +distribution rather than as fixed constants. + +### The deduplication step merges facts that are not duplicates + +Our extraction pipeline compares each new fact against similar stored facts and asks a model whether +the new one is a duplicate, a refinement, or genuinely new. + +We sampled 100 of those decisions, labelled each pair independently with a stronger model in two +passes (the second with the pair order reversed, so that a changed answer indicates position bias +rather than noise), and compared the labels to what the pipeline decided. + +The result, stated at the strength the sample supports: + +Across the full population of decisions in that band, the pipeline chose to keep a fact as new in +**4.1%** of the cases it examined (610 of 14,771), while the labelled sample puts roughly two thirds +of that band in the keep-as-new category. The gap is large and one-directional. + +The per-cell rates from the sample point the same way, but their confidence intervals are wide +enough that the individual percentages should not be quoted as findings. A Fisher exact test on the +split gives p = 0.001, so the direction is solid even though the magnitudes are not pinned down. + +The bias has a plausible cause. The prompt asks "Is this a duplicate?", which invites a yes, and the +measured base rate of true duplicates in that band is about one third. + +Two limitations of our own method, since they cut against the conclusion. The labelling rubric +instructed the labeller to be strict about calling pairs distinct, which biases the ground truth +toward finding the pipeline too merge-happy. And the measurement harness supplied the adjudicator +with one candidate neighbour where production supplies three, so it was not scored on exactly the +input it sees. Both need fixing before these numbers are used for anything beyond direction. + +We also found that the similarity gate itself is doing real work and should be kept: above it, +roughly a third of pairs are genuine restatements; below it, in the sample, none were. + +### What the merge behaviour revealed, which mattered more than the bug + +The refinement action did not merge text. It wrote the new fact and marked the older one superseded, +which removed it from every downstream query. That is a soft delete, so the rows survive and the +decisions remain auditable, which is the only reason this analysis was possible at all. + +Examining those pairs changed our reading of the problem. Roughly 46% were genuine refinements where +the newer fact said everything the older one said and more. Around 18% changed the predicate +entirely, which is not a refinement but a **second interpretation** of the same material. + +And a meaningful share were pairs that appear contradictory but are better understood as two true +statements about the same underlying dimension, resolving differently in different contexts. A +person can be inconsistent about the same thing. When our pipeline saw such a pair, it deleted one +half and kept the other, which destroys the finding: the pair is what identifies the dimension. + +The conclusion we drew is that the action was detecting something real, two facts bearing on the +same construct, and had only destructive ways to record it. So we are not removing it. We are +renaming it to describe what it detects, recording a typed relation between the two facts, and +keeping both facts active. Disagreement between facts becomes an edge in the data rather than an +argument to be settled at write time. + +This also gave us an answer to a design question that had been open for a while: how to preserve +disagreement without introducing a second component whose job is to argue. A typed edge between two +facts needs no such component. + +### The test suite was green throughout + +769 tests, all passing, for the entire period in which the above was true. (It was 756 before this +block of work added tests; the count is not the point.) + +The suite verifies that data moves in the shapes the test author expected under inputs the test +author invented. No test loads a real embedding model or a real vector store. The test for the +faithfulness guard uses mocked three-dimensional vectors at cosine ~1.0 and ~0.0, so any threshold +between roughly 0.05 and 0.9 passes it identically. The test proves that a comparison operator +works. It cannot see that the threshold sits below the noise floor. + +A mechanical census found that 29 of 107 module-level constants in the configuration are never +referenced anywhere in the source. One of them is referenced only by a test that asserts it equals +its own literal value, which is how a constant can be dead and appear live to coverage tooling at +the same time. + +We are restructuring the harness into three tiers, split by determinism rather than by cost: + +1. Fast hermetic unit tests, which is what exists today, keeping its current job. +2. Tests that require a real pinned encoder and a real ephemeral vector store. These cost no API + money and take a minute or two. They are required whenever a change touches a threshold, the + extraction path, the verifier, or the encoder pin. They break when the encoder changes, which is + the point: an encoder change invalidates every threshold calibrated against the old one. +3. Gold-labelled measurements that cost real money and are not deterministic. These stay out of the + test suite, because a test that silently skips without an API key reads as coverage. They are + scripts with committed result artifacts, plus a cheap test in tier one that fails when the + measured component changes and the measurement has not been re-run. + +### What is not fixed + +The renaming and typed relations are designed and not implemented. The faithfulness guard is +diagnosed and not recalibrated. The verifier still computes a contradiction score and does not use +it, so a claim its own evidence contradicts and a claim with no evidence at all currently return the +same verdict. Facts that were superseded by the merge behaviour have been triaged but not restored. + +We will update this note as those land.