Skip to content

fix(research): reuse a recent identical capture instead of re-measuring (chat#1912 row 4) - #800

Merged
sweetmantech merged 4 commits into
mainfrom
fix/duplicate-catalog-measurement
Jul 30, 2026
Merged

fix(research): reuse a recent identical capture instead of re-measuring (chat#1912 row 4)#800
sweetmantech merged 4 commits into
mainfrom
fix/duplicate-catalog-measurement

Conversation

@sweetmantech

@sweetmantech sweetmantech commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes row 4 of chat#1912.

Why

One uninterrupted pass through /setup on a fresh account writes two playcount_snapshots rows for the same 24 albums. Measured on prod 2026-07-30, account 10c76dc5:

created_at catalog albums what fired it
14:30:35 006ca598… 24 POST /api/valuation seeding, from the first artist add
14:33:32 null 24 the /setup/tasks first-task pre-run

Both did real work — 24 song_measurements rows landed at 14:30:54 and again at 14:33:43 — so the scraper was billed twice for an identical capture three minutes apart.

The second one comes from buildFirstTaskPrompt, whose step 2 instructs the agent to POST /api/research/measurement-jobs with the artist's album ids. That path creates a snapshot but never materializes a catalog, which is why the second row has catalog = null.

This also corrects the record on chat#1912. Ben Hanchett's 2026-07-27 pair (19:50:51 with a catalog, 19:54:20 with catalog = null, 209s apart) has exactly this signature. It was read as proof that he re-ran the valuation manually. He may well have, but those two rows would have appeared anyway.

What changed

createSnapshot now looks for an identical recent capture before starting a new one, via the new pure findReusableSnapshot:

  • same album set (order-insensitive), same platforms, same schedule
  • created within 15 minutes
  • state is queued, running or done (never failed)
  • newest match wins

On a hit it returns that snapshot with estimated_cost_usd: 0 and reused: true instead of inserting a row and starting the workflow. A still-in-flight capture is deliberately reusable: the caller polls for measurements either way, and starting a second scrape is the waste.

The month's snapshots are already loaded for the monthly cost cap, so this costs no extra query.

Why here and not in the prompt

The prompt could be told to skip its capture during onboarding, but the duplicate is a property of the endpoint, not of one caller: any two callers asking for the same albums within minutes should not both scrape. Fixing it in createSnapshot also covers the retry case (an agent that re-posts after a timeout) which a prompt change would not.

Tests

lib/research/playcounts/__tests__/findReusableSnapshot.test.ts — 10 cases, TDD red→green (RED: module not found). Covers reuse, album-order insensitivity, differing album sets, the window boundary, failed captures, in-flight captures, platform and schedule mismatches, newest-wins, and the empty case.

Domain suites lib/research lib/catalog lib/valuation: 417 passed / 93 files. tsc --noEmit clean.

Verification

Preview verification against the issue's Works when script (fresh signup, one clean /setup pass, wait 5 min, then the snapshot query expecting exactly one row with a non-null catalog) will be posted as a PR comment once the preview builds.

🤖 Generated with Claude Code


Summary by cubic

Reuses a recent identical playcount snapshot (60-minute window) instead of starting a new one to prevent double scraping and charges. Fixes the duplicate captures during a single /setup pass (chat#1912 row 4), including requests that cross a month boundary, and surfaces reuse to callers.

  • Bug Fixes

    • In createSnapshot, return a reusable snapshot when the same album set (order-insensitive), platforms, and schedule are requested within 60 minutes and the state is queued/running/done; respond with estimated_cost_usd: 0 and reused: true.
    • Fix month-boundary gap: snapshot lookup starts at the earlier of the month start or reuse cutoff so identical requests in the first UTC hour reuse prior captures; monthly cap still counts only current-month rows via getMonthlySpendUsd.
    • In createMeasurementJob, include reused: true in the response when handing back an existing capture so clients can distinguish reuse from a fresh run.
  • Refactors

    • Extracted buildReusedSnapshotResult and getMonthlySpendUsd for clarity.

Written for commit 2618d2f. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • New Features
    • Eligible playcount snapshot requests can now reuse a previously captured snapshot when the schedule, albums, platforms, and timing match (with an expanded 60-minute reuse window).
    • Reused requests complete immediately, returning the existing snapshot ID with an estimated cost of $0.
    • Reuse can draw from snapshots that are queued, running, or completed, reducing duplicate processing and wait times.

…ng (chat#1912 row 4)

One uninterrupted pass through /setup measured the same albums twice: the
seeding valuation via POST /api/valuation, then about three minutes later the
first-task pre-run, whose prompt posts to /api/research/measurement-jobs with
the same album ids. Both scrape every album and bill for it, and the second
leaves a catalog-less snapshot behind.

createSnapshot now hands back an identical recent capture (same album set,
platforms and schedule, inside 15 minutes) rather than starting a second
scrape. Play counts do not move meaningfully in minutes, so the earlier
capture is the correct answer. Reuse is reported with estimated_cost_usd 0
and reused: true; the month's snapshots were already loaded for the cost cap,
so the check costs no extra query.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 30, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
api Ready Ready Preview Jul 30, 2026 7:14pm

Request Review

@coderabbitai

coderabbitai Bot commented Jul 30, 2026

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@sweetmantech, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3d06da09-9537-48ed-9ae3-9879003a5c82

📥 Commits

Reviewing files that changed from the base of the PR and between 0e8ff03 and 2618d2f.

⛔ Files ignored due to path filters (1)
  • lib/research/measurement_jobs/__tests__/createMeasurementJob.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
📒 Files selected for processing (1)
  • lib/research/measurement_jobs/createMeasurementJob.ts
📝 Walkthrough

Walkthrough

Changes

Playcount snapshot reuse

Layer / File(s) Summary
Reusable snapshot selection
lib/research/playcounts/findReusableSnapshot.ts
Selects the newest queued, running, or done snapshot matching album and platform sets, schedule, and the 60-minute reuse window.
Snapshot creation integration
lib/research/playcounts/createSnapshot.ts, lib/research/playcounts/getMonthlySpendUsd.ts, lib/research/playcounts/buildReusedSnapshotResult.ts
Loads snapshots from the month start or reuse cutoff, calculates monthly spend, and returns a zero-cost reused response before cap enforcement, insertion, or workflow startup.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant createSnapshot
  participant playcount_snapshots
  participant CaptureWorkflow
  Caller->>createSnapshot: submit snapshot request
  createSnapshot->>playcount_snapshots: load lookback snapshots
  createSnapshot->>createSnapshot: calculate spend and find reusable snapshot
  alt reusable snapshot found
    createSnapshot-->>Caller: return reused payload
  else no reusable snapshot
    createSnapshot->>playcount_snapshots: insert queued snapshot
    createSnapshot->>CaptureWorkflow: start capture workflow
    CaptureWorkflow-->>Caller: return new snapshot result
  end
Loading

Poem

Snapshots gather, queued or done,
Matching sets align as one.
Recent work gets reused bright,
Costs stay low and flows take flight.
A tidy path, a lighter tune.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Solid & Clean Code ⚠️ Warning createSnapshot is ~70 lines and now mixes album resolution, cap logic, reuse matching, persistence, and workflow start, violating the function-size/SRP guideline. Split createSnapshot into smaller helpers (reuse lookup, cap check, job insert/start, response builder) so each file stays focused and name-matched.
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/duplicate-catalog-measurement

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/research/playcounts/createSnapshot.ts`:
- Around line 54-60: Update the monthSnapshots query in createSnapshot to load
records from the earlier of monthStart and the reuse-window cutoff, allowing
reuse across UTC month boundaries. Keep cap-enforcement totals restricted to
rows created in the current month, and preserve the existing
findReusableSnapshot flow.
- Around line 54-73: Make the reusable-snapshot lookup and subsequent snapshot
creation in createSnapshot atomic for each request scope, preventing concurrent
calls from both creating snapshots and starting workflows. Use the existing
database transaction, locking, or idempotency mechanism to serialize the
findReusableSnapshot check with insertion; preserve the current reuse response
for an existing snapshot and ensure only one workflow starts for concurrent
duplicates.
- Around line 50-73: Decompose the playcount snapshot flow into focused helpers:
in lib/research/playcounts/createSnapshot.ts lines 50-73, extract the reuse
response and/or cap, creation, and persistence workflow so createSnapshot
remains under 50 lines; in lib/research/playcounts/findReusableSnapshot.ts lines
22-52, extract the reusable-candidate predicate while keeping
findReusableSnapshot focused on selecting a matching snapshot. Preserve the
existing reuse and cap behavior at both sites.

In `@lib/research/playcounts/findReusableSnapshot.ts`:
- Line 42: Update the created_at validation in findReusableSnapshot to parse the
timestamp once and require it to be finite and within the inclusive range cutoff
<= createdAt <= now. Reject missing, malformed, and future dates while
preserving acceptance of valid timestamps at either boundary.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9c7efbc1-1d33-45cf-8581-7c37b7581929

📥 Commits

Reviewing files that changed from the base of the PR and between 18a1aa5 and 786bea0.

⛔ Files ignored due to path filters (1)
  • lib/research/playcounts/__tests__/findReusableSnapshot.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
📒 Files selected for processing (2)
  • lib/research/playcounts/createSnapshot.ts
  • lib/research/playcounts/findReusableSnapshot.ts

Comment on lines +50 to +73

// Hand back an identical capture taken moments ago rather than scraping the
// same albums twice. The month's snapshots are already loaded for the cap
// check, so this costs no extra query.
const reusable = findReusableSnapshot({
snapshots: monthSnapshots,
albumIds,
platforms: params.body.platforms,
schedule: params.body.schedule,
windowMinutes: REUSE_WINDOW_MINUTES,
now: new Date(),
});
if (reusable) {
return {
data: {
status: "success",
snapshot_id: reusable.id,
state: reusable.state,
album_count: reusable.album_count,
estimated_cost_usd: 0,
reused: true,
},
};
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift

Decompose the changed domain functions to keep them small.

The new reuse branch leaves createSnapshot at roughly 74 lines, while findReusableSnapshot is 31 lines. Extract focused helpers for the reuse response, cap calculation, or persistence/workflow path.

  • lib/research/playcounts/createSnapshot.ts#L50-L73: split the reuse/cap/create branches so createSnapshot stays under 50 lines.
  • lib/research/playcounts/findReusableSnapshot.ts#L22-L52: extract the reusable-candidate predicate and keep the exported selector focused.

As per coding guidelines, functions longer than 20 lines must be flagged and kept small. As per path instructions, lib/**/*.ts domain functions must remain under 50 lines.

📍 Affects 2 files
  • lib/research/playcounts/createSnapshot.ts#L50-L73 (this comment)
  • lib/research/playcounts/findReusableSnapshot.ts#L22-L52
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/research/playcounts/createSnapshot.ts` around lines 50 - 73, Decompose
the playcount snapshot flow into focused helpers: in
lib/research/playcounts/createSnapshot.ts lines 50-73, extract the reuse
response and/or cap, creation, and persistence workflow so createSnapshot
remains under 50 lines; in lib/research/playcounts/findReusableSnapshot.ts lines
22-52, extract the reusable-candidate predicate while keeping
findReusableSnapshot focused on selecting a matching snapshot. Preserve the
existing reuse and cap behavior at both sites.

Sources: Coding guidelines, Path instructions

Comment thread lib/research/playcounts/createSnapshot.ts Outdated
Comment thread lib/research/playcounts/createSnapshot.ts Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Inline review comments failed to post. This is likely due to GitHub's internal server error or limits when posting large numbers of comments. If you are seeing this consistently it is likely a permissions issue. Please check "Moderation" -> "Code review limits" under your organization settings.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@lib/research/playcounts/createSnapshot.ts`:
- Around line 54-60: Update the monthSnapshots query in createSnapshot to load
records from the earlier of monthStart and the reuse-window cutoff, allowing
reuse across UTC month boundaries. Keep cap-enforcement totals restricted to
rows created in the current month, and preserve the existing
findReusableSnapshot flow.
- Around line 54-73: Make the reusable-snapshot lookup and subsequent snapshot
creation in createSnapshot atomic for each request scope, preventing concurrent
calls from both creating snapshots and starting workflows. Use the existing
database transaction, locking, or idempotency mechanism to serialize the
findReusableSnapshot check with insertion; preserve the current reuse response
for an existing snapshot and ensure only one workflow starts for concurrent
duplicates.
- Around line 50-73: Decompose the playcount snapshot flow into focused helpers:
in lib/research/playcounts/createSnapshot.ts lines 50-73, extract the reuse
response and/or cap, creation, and persistence workflow so createSnapshot
remains under 50 lines; in lib/research/playcounts/findReusableSnapshot.ts lines
22-52, extract the reusable-candidate predicate while keeping
findReusableSnapshot focused on selecting a matching snapshot. Preserve the
existing reuse and cap behavior at both sites.

In `@lib/research/playcounts/findReusableSnapshot.ts`:
- Line 42: Update the created_at validation in findReusableSnapshot to parse the
timestamp once and require it to be finite and within the inclusive range cutoff
<= createdAt <= now. Reject missing, malformed, and future dates while
preserving acceptance of valid timestamps at either boundary.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9c7efbc1-1d33-45cf-8581-7c37b7581929

📥 Commits

Reviewing files that changed from the base of the PR and between 18a1aa5 and 786bea0.

⛔ Files ignored due to path filters (1)
  • lib/research/playcounts/__tests__/findReusableSnapshot.test.ts is excluded by !**/*.test.*, !**/__tests__/** and included by lib/**
📒 Files selected for processing (2)
  • lib/research/playcounts/createSnapshot.ts
  • lib/research/playcounts/findReusableSnapshot.ts
🛑 Comments failed to post (1)
lib/research/playcounts/findReusableSnapshot.ts (1)

42-42: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Reject future or invalid created_at values.

The current lower-bound-only check lets future timestamps through, and NaN < cutoff is false for malformed dates. Require a finite timestamp within cutoff <= createdAt <= now.

Proposed fix
-    if (!row.created_at || new Date(row.created_at).getTime() < cutoff) return false;
+    const createdAt = row.created_at ? new Date(row.created_at).getTime() : NaN;
+    if (!Number.isFinite(createdAt) || createdAt < cutoff || createdAt > now.getTime()) return false;
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

    const createdAt = row.created_at ? new Date(row.created_at).getTime() : NaN;
    if (!Number.isFinite(createdAt) || createdAt < cutoff || createdAt > now.getTime()) return false;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/research/playcounts/findReusableSnapshot.ts` at line 42, Update the
created_at validation in findReusableSnapshot to parse the timestamp once and
require it to be finite and within the inclusive range cutoff <= createdAt <=
now. Reject missing, malformed, and future dates while preserving acceptance of
valid timestamps at either boundary.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 3 files

Confidence score: 3/5

  • In lib/research/playcounts/createSnapshot.ts, the non-atomic read/check/insert flow means concurrent identical requests can both create and scrape snapshots, causing duplicate work and potentially inconsistent snapshot selection for callers — switch to a DB-backed lock or an atomic find-or-create/unique-claim path so only one request wins.
  • In lib/research/playcounts/createSnapshot.ts, album_count can be null on the reuse branch but numeric on the normal branch, which can break consumers expecting a stable number type or force defensive client handling — compute a numeric fallback (for example from reusable.album_ids?.length or albumIds.length) before returning the payload.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="lib/research/playcounts/createSnapshot.ts">

<violation number="1" location="lib/research/playcounts/createSnapshot.ts:54">
P2: Simultaneous identical requests can still create and scrape two snapshots: the read/check/insert sequence is non-atomic. Use a database-backed lock or atomic find-or-create/unique-claim flow so one request returns the row created by the other.</violation>

<violation number="2" location="lib/research/playcounts/createSnapshot.ts:68">
P2: The response payload's `album_count` field can be `null` on the reused path while it is always a `number` on the normal path. Consider falling back to `reusable.album_ids?.length ?? albumIds.length` instead of passing the nullable column through directly, so the caller always receives a consistent type regardless of the reuse decision.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant Client as API Client
    participant CS as createSnapshot
    participant Find as findReusableSnapshot (pure)
    participant DB as Supabase (playcount_snapshots)
    participant WF as playcountSnapshotWorkflow

    Note over Client,WF: Snapshot creation with duplicate detection

    Client->>CS: POST /api/research/measurement-jobs (body with albumIds, platforms, schedule)
    CS->>DB: selectPlaycountSnapshots() for month
    DB-->>CS: monthSnapshots (already loaded for cap check)
    Note over CS,Find: No extra query – month data reused

    CS->>Find: findReusableSnapshot({ snapshots, albumIds, platforms, schedule, windowMinutes=15, now })
    Note over Find: Pure function: checks same albums (order-insensitive), same platforms/schedule, created within 15 min, state in (queued/running/done), newest win
    alt Reusable snapshot found
        Find-->>CS: snapshot row (state, id, album_count, etc.)
        CS-->>Client: { status:"success", snapshot_id, state, album_count, estimated_cost_usd:0, reused:true }
    else No reusable snapshot
        Find-->>CS: null
        CS->>CS: Check monthly cap (existing logic)
        CS->>DB: insertPlaycountSnapshot(...)
        DB-->>CS: new snapshot row
        CS->>WF: playcountSnapshotWorkflow(new snapshot)
        WF-->>CS: (async)
        CS-->>Client: { status:"success", snapshot_id, ...estimated_cost_usd, reused:false }
    end

    Note over Client,Find: Duplicate scenario (chat#1912 row 4)
    Client->>CS: First call (albums A, B, C) – no prior snapshot
    CS->>Find: Finds nothing
    CS->>DB: insert + workflow starts
    Client->>CS: Second call (same albums, 3 min later)
    CS->>Find: Finds first snapshot (e.g. state: "running")
    CS-->>Client: reuse=true, cost=0
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread lib/research/playcounts/createSnapshot.ts Outdated
// Hand back an identical capture taken moments ago rather than scraping the
// same albums twice. The month's snapshots are already loaded for the cap
// check, so this costs no extra query.
const reusable = findReusableSnapshot({

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Simultaneous identical requests can still create and scrape two snapshots: the read/check/insert sequence is non-atomic. Use a database-backed lock or atomic find-or-create/unique-claim flow so one request returns the row created by the other.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/research/playcounts/createSnapshot.ts, line 54:

<comment>Simultaneous identical requests can still create and scrape two snapshots: the read/check/insert sequence is non-atomic. Use a database-backed lock or atomic find-or-create/unique-claim flow so one request returns the row created by the other.</comment>

<file context>
@@ -41,6 +47,30 @@ export async function createSnapshot(params: {
+  // Hand back an identical capture taken moments ago rather than scraping the
+  // same albums twice. The month's snapshots are already loaded for the cap
+  // check, so this costs no extra query.
+  const reusable = findReusableSnapshot({
+    snapshots: monthSnapshots,
+    albumIds,
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct, and deliberately not fixed here. Recording the reasoning rather than silently leaving it.

The read/check/insert sequence is not atomic, so two simultaneous identical requests can both miss and both scrape. What this PR targets is the sequential duplicate that is actually happening in production: the seeding valuation and the /setup/tasks first-task pre-run measure the same albums about three minutes apart (chat#1912 row 4 has the timings). A pre-check fully covers that, and now covers anything inside 60 minutes.

Closing the concurrent window properly needs a database-level guard, and the obvious shapes do not fit cheaply:

  • a unique index cannot express "same album_ids array, same platforms, within a rolling time window"
  • a Postgres advisory lock keyed on the scope hash would work, but it is a different change with its own failure modes (lock lifetime across the workflow start, behaviour under connection pooling)

So the residual is: two requests landing in the same instant still both scrape. That is strictly better than today, where requests three minutes apart both scrape, and it is not the observed failure. Flagging it on the tracking issue as a follow-up rather than expanding this PR is the honest trade. If you would rather it land here, say so and I will do the advisory-lock version with its own tests.

status: "success",
snapshot_id: reusable.id,
state: reusable.state,
album_count: reusable.album_count,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The response payload's album_count field can be null on the reused path while it is always a number on the normal path. Consider falling back to reusable.album_ids?.length ?? albumIds.length instead of passing the nullable column through directly, so the caller always receives a consistent type regardless of the reuse decision.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/research/playcounts/createSnapshot.ts, line 68:

<comment>The response payload's `album_count` field can be `null` on the reused path while it is always a `number` on the normal path. Consider falling back to `reusable.album_ids?.length ?? albumIds.length` instead of passing the nullable column through directly, so the caller always receives a consistent type regardless of the reuse decision.</comment>

<file context>
@@ -41,6 +47,30 @@ export async function createSnapshot(params: {
+        status: "success",
+        snapshot_id: reusable.id,
+        state: reusable.state,
+        album_count: reusable.album_count,
+        estimated_cost_usd: 0,
+        reused: true,
</file context>
Suggested change
album_count: reusable.album_count,
album_count: reusable.album_count ?? reusable.album_ids?.length ?? albumIds.length,

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 1 file (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Requires human review: Auto-approval blocked by 3 unresolved issues from previous reviews.

Re-trigger cubic

Review (@sweetmantech): the reuse window is 60 minutes, not 15. Nothing about
a catalog's play counts changes inside an hour, so re-valuing it sooner is
pure scraper spend.

Review (coderabbit / cubic, both flagged): the candidate list was the
monthly-cap query, whose lower bound is the start of the UTC month. In the
first hour of a month the previous month's captures were invisible and an
identical request re-scraped them. Widening the window to 60 minutes widened
that hole, so the lookup now starts at whichever of monthStart and the reuse
cutoff reaches further back, while the cap still counts only current-month
rows via getMonthlySpendUsd.

Review (coderabbit): createSnapshot was growing. Extracted
buildReusedSnapshotResult and getMonthlySpendUsd.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 4 files (changes from recent commits).

Confidence score: 5/5

  • In lib/research/playcounts/createSnapshot.ts, the monthly-cap test fixture is missing created_at, so the new current-month filter excludes the mock row and leaves cap logic effectively unverified; this risks masking regressions in monthly limiting behavior—add a current-month timestamp to the fixture and a focused helper test for the filter path.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="lib/research/playcounts/createSnapshot.ts">

<violation number="1" location="lib/research/playcounts/createSnapshot.ts:59">
P3: The existing monthly-cap test now fails because its mock row has no `created_at`, which this new calculation filters out. Include a current-month timestamp in that fixture (and add a direct helper test) so the cap behavior remains covered.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

});
const spentUsd = monthSnapshots.reduce((sum, row) => sum + (row.estimated_cost_usd ?? 0), 0);

const spentUsd = getMonthlySpendUsd(snapshots, monthStart);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The existing monthly-cap test now fails because its mock row has no created_at, which this new calculation filters out. Include a current-month timestamp in that fixture (and add a direct helper test) so the cap behavior remains covered.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/research/playcounts/createSnapshot.ts, line 59:

<comment>The existing monthly-cap test now fails because its mock row has no `created_at`, which this new calculation filters out. Include a current-month timestamp in that fixture (and add a direct helper test) so the cap behavior remains covered.</comment>

<file context>
@@ -38,39 +41,35 @@ export async function createSnapshot(params: {
   });
-  const spentUsd = monthSnapshots.reduce((sum, row) => sum + (row.estimated_cost_usd ?? 0), 0);
+
+  const spentUsd = getMonthlySpendUsd(snapshots, monthStart);
   const estimatedCostUsd = Number((albumIds.length * COST_PER_ALBUM_USD).toFixed(4));
 
</file context>

…dpoint

Found during preview verification: createSnapshot returns reused: true, but
createMeasurementJob rebuilt the payload without it, so a handed-back capture
was indistinguishable from a fresh one except by its zero cost. A caller
polling for a new capture needs to know it got an existing one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sweetmantech

Copy link
Copy Markdown
Contributor Author

Preview verification — 2026-07-30

Two preview builds, each resolved by sha:

API keys were minted on each preview via POST /api/agents/signup, since prod keys 401 against a preview (different key-hash salt). Note the endpoint only returns a key for an agent+ address; any other email returns api_key: null and mails a verification code instead.

Results

# Check Documented / before Actual Build
1 Fresh capture new snapshot, real cost 202, id e05c6ade…, state: queued, estimated_cost_usd: 0.003 2618d2f4
2 Identical request, immediately after before: a second scrape and a second row 202, same id e05c6ade…, estimated_cost_usd: 0, reused: true 2618d2f4
3 Different album set must scrape new id 4682c7d4…, estimated_cost_usd: 0.003 0e8ff03a
4 Third request for the same albums, minutes later must not scrape same id dcdf548f…, cost 0 0e8ff03a
5 Database truth one row per distinct scope 0e8ff03a account: 2 rows for 4 successful POSTs (one per scope). 2618d2f4 account: 1 row, $0.0030 total for 2 POSTs both
6 No credentials 401 401 Exactly one of x-api-key or Authorization must be provided 0e8ff03a
7 Missing scope 400 400 Invalid input: expected object, received undefined 2618d2f4

Row 5 is the one that matters: three requests for the same albums produced exactly one playcount_snapshots row and one charge. Before this PR each would have scraped.

A gap the preview caught that the unit tests could not

createSnapshot returned reused: true, but createMeasurementJob rebuilt the response object field by field and dropped it. So through the endpoint that actually caused the duplicate, a handed-back capture was indistinguishable from a fresh one except by inferring it from the zero cost — and the claim in this PR's description was false for that path.

Caught on the 0e8ff03a run (call 2 returned cost 0 but no flag), fixed in 2618d2f4, and now covered by two tests: the flag passes through when present, and is absent for a fresh capture.

Review findings

Finding Verdict Action
@sweetmantech — reuse window should be 60 minutes, not 15 Applied nothing about a catalog's play counts changes inside an hour
coderabbit + cubic (confidence 10) — month-boundary gap Valid, and my own optimisation caused it the candidate list was the monthly-cap query, whose lower bound is the start of the UTC month, so in the first minutes of a month the previous month's captures were invisible and an identical request re-scraped. The PR description had bragged that reusing that query "costs no extra query" — that was exactly the bug. The 60-minute window widened the hole from 15 minutes to an hour. The lookup now starts at whichever of monthStart and the reuse cutoff reaches further back, still in one query, while getMonthlySpendUsd keeps the cap counting current-month rows only. Two tests, including an explicit 2026-08-01T00:10Z boundary case
coderabbit + cubic — read/check/insert is not atomic Valid, deliberately not fixed reasoning on the thread: this PR targets the sequential duplicate that is actually happening (seeding and the first-task pre-run, ~3 minutes apart). Two truly simultaneous requests can still both scrape. A unique index cannot express "same album array within a rolling window", and an advisory lock is a different change with its own failure modes. Recommend tracking it as a follow-up on chat#1912 rather than expanding this PR
coderabbit — createSnapshot is growing Valid extracted buildReusedSnapshotResult and getMonthlySpendUsd; the file is back under the repo's 100-line guideline

Cost note

These runs created real captures. Total scraper spend across both preview accounts: $0.009 (3 albums at $0.003).

Full suite 4291 passed / 791 files, tsc --noEmit, lint:check and format:check all clean.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Requires human review: Auto-approval blocked by 2 unresolved issues from previous reviews.

Re-trigger cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant