Skip to content

Fix run duplication problem#18

Closed
AStaroverov wants to merge 3 commits into
mainfrom
feat/content-hash-namespacing
Closed

Fix run duplication problem#18
AStaroverov wants to merge 3 commits into
mainfrom
feat/content-hash-namespacing

Conversation

@AStaroverov

@AStaroverov AStaroverov commented Jun 12, 2026

Copy link
Copy Markdown

Namespace exported property columns by a content hash of the source table (pl7.app/contentHash, derived from the backend CanonicalID) instead of the per-block blockId, and drop the blockId-keyed trace step id. Identical results across blocks/projects now produce content-identical columns that dedupe downstream instead of being made unique per block; different results stay distinct. The heavy Python step and Parquet import already deduped; this extends dedup to the exported pframe identity.

Greptile Summary

This PR replaces per-block blockId namespacing on exported PFrame column domains with a content-derived hash (pl7.app/contentHash) pulled from info().CanonicalID of the Python output tables. The pFrame-building logic is split into a new process.tpl.tengo render template so that the Python outputs arrive as resolved render inputs (making CanonicalID available), enabling identical results across different blocks or projects to share the same column identity and deduplicate downstream.

  • workflow/src/process.tpl.tengo — new render template that imports the TSV outputs into Parquet-backed PColumns, derives a content hash from each source table, and stamps columns with pl7.app/contentHash in their domain instead of pl7.app/blockId; the trace step no longer carries an id field so it is content-identical across runs.
  • workflow/src/main.tpl.tengo — removes the inline pFrame-building (and xsv/pSpec imports), imports the new processTpl, and defers the import+stamp work via render.create; the params JSON resource is built with canonical.encode so its CID is deterministic across runs.
  • workflow/src/columns.lib.tengo and test/src/wf.test.ts — comment and TODO-description updates to reflect the terminology change from blockId to contentHash.

Confidence Score: 5/5

Safe to merge — the split into a separate render template is architecturally clean and the content-hash derivation is correct for resolved render inputs.

The core change (moving pFrame building into process.tpl.tengo so Python outputs arrive as resolved resources) achieves its goal cleanly. The canonical JSON params resource, sorted chain key lists, and removal of the blockId trace id all cooperate to make the render CID deterministic. The aaFractionTsv is always produced by the exec (empty file in non-peptide mode) so passing it unconditionally as a render input is safe. No new logic defects were found beyond what was already raised in prior review threads.

workflow/src/process.tpl.tengo — the contentHash helper has an unguarded fallback branch (covered by a prior thread); no other files require special attention.

Important Files Changed

Filename Overview
workflow/src/process.tpl.tengo New render template: imports TSV outputs, derives content hash via info().CanonicalID, and stamps columns with pl7.app/contentHash. Logic is correct for the resolved-input context; the contentHash fallback branch has an existing unguarded access (flagged in prior thread).
workflow/src/main.tpl.tengo Removes inline pFrame building; defers to processTpl via render.create with canonical params. aaFractionTsv is always produced by the exec (empty body in non-peptide mode) so passing it unconditionally as a render input is safe.
workflow/src/columns.lib.tengo Comment-only updates replacing blockId terminology with contentHash; no logic changes.
test/src/wf.test.ts TODO test description updates only; no executable test logic changed.
.changeset/content-hash-namespacing.md New changeset file documenting the patch-level version bump and the content-hash namespacing change.

Sequence Diagram

sequenceDiagram
    participant MB as main.tpl.tengo
    participant PY as Python exec
    participant RC as render.create
    participant PT as process.tpl.tengo
    participant PF as pFrameBuilder

    MB->>PY: exec input.tsv + plan.json
    PY-->>MB: properties.tsv, aa_fraction.tsv

    MB->>RC: render.create processTpl with properties, aaFraction, params
    Note over RC: dedups on content-addressed CID

    RC->>PT: body with resolved inputs
    PT->>PT: contentHash via info().CanonicalID
    PT->>PT: xsv.importFile properties to scalarOut
    PT->>PT: xsv.importFile aaFraction to aaOut (peptide only)
    PT->>PF: pf.add with stamp(spec, contentHash)
    Note over PT,PF: injects pl7.app/contentHash into domain
    PF-->>PT: built pFrame
    PT-->>RC: propertiesPf and exportPframe

    RC-->>MB: processResult
    MB-->>MB: outputs.propertiesPf and exports.properties
Loading

Reviews (2): Last reviewed commit: "Fix contentHash: read CanonicalID in a r..." | Re-trigger Greptile

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request replaces the per-block blockId with a content hash (pl7.app/contentHash) derived from the source table's CanonicalID to namespace exported property columns, enabling downstream deduplication of identical results. This logic is moved from main.tpl.tengo to a new post-processing template process.tpl.tengo. Feedback on the changes points out a potential runtime panic in process.tpl.tengo when accessing i.Value without checking if the key exists in the map.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread workflow/src/process.tpl.tengo Outdated
AStaroverov and others added 2 commits June 12, 2026 13:57
Stamp pl7.app/contentHash (base32 of the source table's CanonicalID) on the
exported property columns instead of the per-block blockId, and drop the
blockId-keyed trace step id. Identical results across blocks/projects now
produce content-identical columns that dedupe downstream, while different
results stay distinct. Mirrors the humanization-score approach. The Python
step and Parquet import already deduped; this extends it to the export.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
info().CanonicalID is only readable once the table resource is resolved, which
it is not in main's body (the same body that creates the Python exec — the
saveFile output field has Value==0, so smart.resource(i.Value) errors with
"null resource id"). Move the TSV import + content-tagging + trace stamping
into a new process.tpl.tengo that receives the Python output tables as resolved
render inputs. Also probe info() with maps.containsKey (it is a strict map;
accessing a missing key throws).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AStaroverov AStaroverov force-pushed the feat/content-hash-namespacing branch from 7900e67 to ccadf23 Compare June 12, 2026 11:57
Comment thread workflow/src/process.tpl.tengo Outdated
Comment on lines +27 to +34
contentHash := func(table) {
i := table.info()
if maps.containsKey(i, "CanonicalID") {
return ll.base32Encode(i.CanonicalID)
}
ri := smart.resource(i.Value).info()
return ll.base32Encode(ri.CanonicalID)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unguarded accesses in the fallback branch of contentHash

The first branch correctly uses maps.containsKey(i, "CanonicalID") before accessing i.CanonicalID, but the fallback branch accesses i.Value and then ri.CanonicalID without the same protection. Since info() is documented as a strict map that throws on missing keys, if a resource arrives with neither CanonicalID nor Value (e.g. an error-state or future resource shape), or if the dereferenced resource lacks CanonicalID, the function throws a cryptic "key not found" rather than surfacing a useful error. Consider guarding with maps.containsKey(i, "Value") before the dereference, and similarly for ri.CanonicalID.

Prompt To Fix With AI
This is a comment left during a code review.
Path: workflow/src/process.tpl.tengo
Line: 27-34

Comment:
**Unguarded accesses in the fallback branch of `contentHash`**

The first branch correctly uses `maps.containsKey(i, "CanonicalID")` before accessing `i.CanonicalID`, but the fallback branch accesses `i.Value` and then `ri.CanonicalID` without the same protection. Since `info()` is documented as a strict map that throws on missing keys, if a resource arrives with neither `CanonicalID` nor `Value` (e.g. an error-state or future resource shape), or if the dereferenced resource lacks `CanonicalID`, the function throws a cryptic "key not found" rather than surfacing a useful error. Consider guarding with `maps.containsKey(i, "Value")` before the dereference, and similarly for `ri.CanonicalID`.

How can I resolve this? If you propose a fix, please make it concise.

Comment thread workflow/src/process.tpl.tengo Outdated
@AStaroverov AStaroverov requested a review from PaulNewling June 12, 2026 12:10
@AStaroverov

Copy link
Copy Markdown
Author

@greptileai

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant