Skip to content

fix(heartbeat): honor Ally concurrency 15 - #936

Closed
allyblockcast[bot] wants to merge 1 commit into
blo-20396-dispatch-serializationfrom
staff/blo-20742-ally-concurrency
Closed

fix(heartbeat): honor Ally concurrency 15#936
allyblockcast[bot] wants to merge 1 commit into
blo-20396-dispatch-serializationfrom
staff/blo-20742-ally-concurrency

Conversation

@allyblockcast

@allyblockcast allyblockcast Bot commented Aug 2, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip is the open source app people use to manage AI agents for work.
  • Heartbeat dispatch converts queued work into bounded concurrent agent runs.
  • Ally persists heartbeat.maxConcurrentRuns: 15, but the external-lifecycle operational ceiling is 8, so the runtime cannot honor the stored configuration.
  • Raising admission before fixing queue-selection serialization could amplify overlapping scans, so this change is stacked on the strict dispatcher fix in fix(heartbeat): serialize queued-run dispatch instead of bypassing the lock (BLO-20396) #912.
  • This pull request raises only the external operational ceiling to 16 and adds boundary, validation, and concurrent-claim regressions.
  • The benefit is that Ally can use its configured 15 slots without letting concurrent starter calls claim a 16th or allowing unsupported API values to persist.

Linked Issues or Issue Description

Paperclip issue: https://paperclip.blockcast.net/BLO/issues/BLO-20742

Related prerequisite: #912. Prior eight-slot ceiling change: #885.

Bug report fields:

  • What happened: Ally stores maxConcurrentRuns: 15, while external-lifecycle admission clamps it to 8.
  • Expected behavior: a valid persisted value of 15 produces 15 available slots, with values above the supported validator maximum rejected before persistence and external admission bounded by a hard operational ceiling.
  • Steps to reproduce: configure an opencode_k8s agent with concurrencyEnabled: true and maxConcurrentRuns: 15; under backlog, observe no more than 8 active runs before this change.
  • Deployment mode: authenticated Paperclip Kubernetes deployment.

What Changed

  • Raise EXTERNAL_LIFECYCLE_SLOT_CAPACITY from 8 to 16 so Ally's configured 15 is effective.
  • Prove resolveExternalLifecycleConcurrency returns 15 for the configured value and clamps 16/50 to 16.
  • Prove create/update agent validation accepts 15 and rejects HEARTBEAT_POLICY_MAX_CONCURRENT_MAX + 1 before persistence.
  • Add an embedded-Postgres regression that invokes 20 concurrent dispatch starters and asserts exactly 15 runs are claimed while five remain queued.

Verification

  • pnpm exec vitest run packages/shared/src/adapter-types.test.ts server/src/__tests__/heartbeat-external-lifecycle-concurrency-flag.test.ts server/src/__tests__/heartbeat-start-lock.test.ts --reporter=dot (32/32 passed)
  • pnpm exec vitest run server/src/__tests__/heartbeat-dispatch-priority-sort.test.ts -t 'claims at most 15 available slots across concurrent dispatch attempts' --reporter=dot (1/1 passed)
  • pnpm --filter @paperclipai/shared typecheck (passed)
  • pnpm --filter @paperclipai/server typecheck (passed)
  • Pre-rollout capacity at 2026-08-02 02:43 UTC: nine Paperclip nodes expose 368 CPU and ~661 GiB allocatable, with ~221 CPU and ~361 GiB requested. Six observed Ally pods used 7.376 CPU / 3.161 GiB; each requests 1.1 CPU / 2.25 GiB, so scaling from six to 15 adds 9.9 CPU / 20.25 GiB of requests. No Ally FailedScheduling event was observed.

Risks

  • This raises the maximum blast radius of a misbehaving external-lifecycle agent. The per-agent config remains a separate gate, so other agents do not change unless configured above 8.
  • Do not merge or deploy ahead of fix(heartbeat): serialize queued-run dispatch instead of bypassing the lock (BLO-20396) #912. This PR is intentionally stacked on that branch so strict queue-selection serialization lands first.
  • A recent Ally pod had a transient prompt-secret FailedMount; it was not a resource-capacity failure, but post-rollout verification must still watch FailedScheduling, OOMKilled, eviction, start-lock errors, queue age, and review failure rate.
  • Rollback is the one-line operational ceiling revert; setting concurrencyEnabled: false immediately restores one-run containment.

For core feature work, check ROADMAP.md first and discuss it in #dev before opening the PR. Feature PRs that overlap with planned core work may need to be redirected. See CONTRIBUTING.md.

Model Used

  • OpenAI GPT-5.6-sol (openai/gpt-5.6-sol) via OpenCode, with reasoning, repository editing, test execution, GitHub, Paperclip, Prometheus, and Kubernetes read-only tools. Context window size was not exposed by the runtime.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have searched GitHub for duplicate or related PRs and linked them above
  • I described the issue in-PR following the bug template
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • This change has no UI impact
  • No documentation change is required beyond the PR rollout contract
  • I have considered and documented risks above
  • All Paperclip CI gates are green
  • Greptile is 5/5 with no open P2s, recommendations, or follow-ups
  • I will address all Greptile and reviewer comments before requesting merge

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@allyblockcast

allyblockcast Bot commented Aug 2, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-20742

1 similar comment
@allyblockcast

allyblockcast Bot commented Aug 2, 2026

Copy link
Copy Markdown
Author

🔗 Paperclip issue: BLO-20742

@allyblockcast

allyblockcast Bot commented Aug 2, 2026

Copy link
Copy Markdown
Author

@ally please review head 39ed31f with focus on the 15-slot concurrent admission invariant, validator/runtime consistency, and compatibility with stacked prerequisite #912.

@allyblockcast

allyblockcast Bot commented Aug 2, 2026

Copy link
Copy Markdown
Author

Superseded by an exact-head restack on current #912; this branch used a stale managed-local pointer and GitHub correctly reported it conflicted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant