Reposition Caplets around whole-stack agent capabilities - #248
Open
ian-pascoe wants to merge 32 commits into
Open
Reposition Caplets around whole-stack agent capabilities#248ian-pascoe wants to merge 32 commits into
ian-pascoe wants to merge 32 commits into
Conversation
Contributor
📝 WalkthroughWalkthroughCaplets is repositioned as a capability layer for coding agents. Public messaging, catalog and landing pages, activation, analytics, Code Mode execution, tests, and benchmark tooling are updated. ChangesWhole Stack positioning
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Agent
participant CapletHandle
participant CodeModeAPI
participant NativeService
Agent->>CapletHandle: Discover bounded tool summaries
Agent->>CapletHandle: Call tool with template and overrides
CapletHandle->>CodeModeAPI: Resolve name and merged arguments
CodeModeAPI->>NativeService: Execute Code Mode run
NativeService-->>Agent: Return projected success or error envelope
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
| Filename | Overview |
|---|---|
| apps/landing/src/pages/index.astro | Reorders the landing journey and adds the breadth and issue-to-production sections without an identified functional issue. |
| apps/landing/src/components/landing/Activation.astro | Moves setup and copy controls into the activation section while preserving working tab and copy initialization. |
| apps/landing/src/scripts/observability.ts | Captures explicitly categorized internal CTAs; the emitted categorical values satisfy the shared event contract. |
| apps/catalog/src/pages/caplets/[entryKey].astro | Adds social metadata, but derives og:url from the request and can publish noncanonical preview or query-bearing URLs. |
| apps/landing/src/data/landing.ts | Updates capability, portability, benchmark, and activation content used throughout the redesigned landing page. |
Reviews (1): Last reviewed commit: "fix(marketing): align whole-stack public..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| <meta property="og:title" content={`${pageTitle} — Caplets Catalog`} /> | ||
| <meta property="og:description" content={pageDescription} /> | ||
| <meta property="og:type" content="website" /> | ||
| <meta property="og:url" content={Astro.url.href} /> |
Contributor
There was a problem hiding this comment.
Using Astro.url.href publishes the incoming hostname and query string as og:url, so shares made through preview hosts or tracking URLs advertise a request-specific address instead of the stable production catalog entry. Build this value from the configured catalog site and entry path, consistent with the catalog index metadata.
Knowledge Base Used: Dashboard and Catalog Apps
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
apps/landing/test/activation-links.test.ts (1)
8-16: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest rendered behavior instead of component-source literals.
These checks couple to raw Astro markup and
data-*strings. Cover the rendered CTA destination/category and setup-tab behavior through the landing test surface instead.As per coding guidelines, “avoid tests that merely restate implementation literals or superficial metadata/copy/export details.”
Also applies to: 18-20
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/landing/test/activation-links.test.ts` around lines 8 - 16, Replace the source-text assertions in the activation-links test with landing-surface behavior tests that render or exercise Hero.astro and verify the secondary shared-Caplets CTA resolves to the public catalog and exposes the secondary category. Also cover the setup-tab interaction and its resulting rendered behavior through the landing test surface, without reading component source or asserting raw implementation literals.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/docs/src/content/docs/index.mdx`:
- Line 15: Update the Code Mode description to say that each configured Caplet
becomes a typed caplets.<id> handle, replacing the inaccurate reference to each
configured backend while preserving the rest of the explanation.
In `@STRATEGY.md`:
- Around line 22-23: Update the “Caplet Activation” definition in STRATEGY.md to
require both the first successful Caplet backend operation and a subsequent
connection of a capability from the user’s own stack. Alternatively, split first
execution and activation into distinct events, ensuring retention and activation
cohorts use the milestone that includes the follow-up connection.
---
Nitpick comments:
In `@apps/landing/test/activation-links.test.ts`:
- Around line 8-16: Replace the source-text assertions in the activation-links
test with landing-surface behavior tests that render or exercise Hero.astro and
verify the secondary shared-Caplets CTA resolves to the public catalog and
exposes the secondary category. Also cover the setup-tab interaction and its
resulting rendered behavior through the landing test surface, without reading
component source or asserting raw implementation literals.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 6718a552-0756-4e7d-bd5b-3784aeb08aca
📒 Files selected for processing (29)
CONTEXT.mdPRODUCT.mdREADME.mdSTRATEGY.mdapps/catalog/src/pages/caplets/[entryKey].astroapps/catalog/src/pages/index.astroapps/catalog/src/styles/catalog.cssapps/docs/src/content/docs/index.mdxapps/landing/src/components/landing/Activation.astroapps/landing/src/components/landing/AgentTrace.astroapps/landing/src/components/landing/BlogCta.astroapps/landing/src/components/landing/Breadth.astroapps/landing/src/components/landing/CommandBlock.astroapps/landing/src/components/landing/Footer.astroapps/landing/src/components/landing/Header.astroapps/landing/src/components/landing/Hero.astroapps/landing/src/components/landing/Proof.astroapps/landing/src/components/landing/RemoteStory.astroapps/landing/src/components/landing/WholeJob.astroapps/landing/src/components/landing/WhyCaplets.astroapps/landing/src/content/blog/why-giant-mcp-tool-walls-dont-scale.mdapps/landing/src/data/landing.tsapps/landing/src/layouts/LandingLayout.astroapps/landing/src/pages/blog/index.astroapps/landing/src/pages/index.astroapps/landing/src/scripts/observability.tsapps/landing/test/activation-links.test.tsapps/landing/test/blog-navigation.test.tsapps/landing/test/observability.test.ts
💤 Files with no reviewable changes (1)
- apps/landing/test/blog-navigation.test.ts
| Codex, Claude, OpenCode, Pi, and supported MCP clients while each host independently | ||
| supplies and authorizes access. | ||
|
|
||
| Code Mode is the default surface: each configured backend becomes a typed `caplets.<id>` handle that an agent can inspect, search, call, filter, and summarize in one workflow. Direct and progressive exposure remain available for clients and tasks that need them. |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Document Caplet handles, not backend handles.
The Code Mode declaration generator creates caplets.<id> properties from configured Caplets and their caplet.id values, not directly from backend instances. Change “each configured backend” to “each configured Caplet” so users do not infer a one-to-one backend/handle mapping.
Proposed wording
-Code Mode is the default surface: each configured backend becomes a typed `caplets.<id>` handle
+Code Mode is the default surface: each configured Caplet becomes a typed `caplets.<id>` handle📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| Code Mode is the default surface: each configured backend becomes a typed `caplets.<id>` handle that an agent can inspect, search, call, filter, and summarize in one workflow. Direct and progressive exposure remain available for clients and tasks that need them. | |
| Code Mode is the default surface: each configured Caplet becomes a typed `caplets.<id>` handle that an agent can inspect, search, call, filter, and summarize in one workflow. Direct and progressive exposure remain available for clients and tasks that need them. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/docs/src/content/docs/index.mdx` at line 15, Update the Code Mode
description to say that each configured Caplet becomes a typed caplets.<id>
handle, replacing the inaccurate reference to each configured backend while
preserving the rest of the explanation.
Comment on lines
+22
to
+23
| - **Caplet Activation** - The first successful backend operation executed through a configured Caplet. Landing clicks, catalog views, installation, and setup completion are funnel diagnostics, not activation. | ||
| - **Whole Stack retention** - Repeat successful Caplet executions across backend families, indicating that users expanded beyond the first no-auth capability into personally relevant work. |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Align “Caplet Activation” with the intended activation milestone.
This currently counts only the first successful backend operation, but the PR objective defines activation as that success followed by connecting a capability from the user’s own stack. Define the follow-up connection in this metric, or split first execution and activation into separate events; otherwise retention and activation reporting will use the wrong cohort.
Proposed wording
-- **Caplet Activation** - The first successful backend operation executed through a configured Caplet.
+- **Caplet Activation** - A first successful backend operation followed by connecting a capability from the user's own stack.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| - **Caplet Activation** - The first successful backend operation executed through a configured Caplet. Landing clicks, catalog views, installation, and setup completion are funnel diagnostics, not activation. | |
| - **Whole Stack retention** - Repeat successful Caplet executions across backend families, indicating that users expanded beyond the first no-auth capability into personally relevant work. | |
| - **Caplet Activation** - The first successful backend operation executed through a configured Caplet. Landing clicks, catalog views, installation, and setup completion are funnel diagnostics, not activation. | |
| **Caplet Activation** - A first successful backend operation followed by connecting a capability from the user's own stack. | |
| - **Whole Stack retention** - Repeat successful Caplet executions across backend families, indicating that users expanded beyond the first no-auth capability into personally relevant work. |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@STRATEGY.md` around lines 22 - 23, Update the “Caplet Activation” definition
in STRATEGY.md to require both the first successful Caplet backend operation and
a subsequent connection of a capability from the user’s own stack.
Alternatively, split first execution and activation into distinct events,
ensuring retention and activation cohorts use the milestone that includes the
follow-up connection.
Contributor
Preview DeployedLanding: https://pr-248.preview.caplets.dev Built from commit 8dfc38b |
ian-pascoe
force-pushed
the
worktree/rapid-river-8e06
branch
from
July 28, 2026 10:59
7ded03f to
aac5cde
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/landing/src/components/landing/Header.astro`:
- Line 23: Update the desktop Workflow link in Header.astro and the mobile Issue
to production link in Header.astro to use the same scroll-offset behavior as the
other anchored section links targeting `#work`, ensuring the destination heading
remains visible below the sticky header.
- Line 36: Update the site-header__actions wrapper in Header.astro so it does
not use the “Project links” aria-label when it contains only the mobile “Open
navigation menu” control; keep the wrapper layout-only or move the label to the
container that exclusively contains the GitHub/npm links.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 05eb1dd5-4627-44de-8a6d-508fa37587ff
📒 Files selected for processing (29)
CONTEXT.mdPRODUCT.mdREADME.mdSTRATEGY.mdapps/catalog/src/pages/caplets/[entryKey].astroapps/catalog/src/pages/index.astroapps/catalog/src/styles/catalog.cssapps/docs/src/content/docs/index.mdxapps/landing/src/components/landing/Activation.astroapps/landing/src/components/landing/AgentTrace.astroapps/landing/src/components/landing/BlogCta.astroapps/landing/src/components/landing/Breadth.astroapps/landing/src/components/landing/CommandBlock.astroapps/landing/src/components/landing/Footer.astroapps/landing/src/components/landing/Header.astroapps/landing/src/components/landing/Hero.astroapps/landing/src/components/landing/Proof.astroapps/landing/src/components/landing/RemoteStory.astroapps/landing/src/components/landing/WholeJob.astroapps/landing/src/components/landing/WhyCaplets.astroapps/landing/src/content/blog/why-giant-mcp-tool-walls-dont-scale.mdapps/landing/src/data/landing.tsapps/landing/src/layouts/LandingLayout.astroapps/landing/src/pages/blog/index.astroapps/landing/src/pages/index.astroapps/landing/src/scripts/observability.tsapps/landing/test/activation-links.test.tsapps/landing/test/blog-navigation.test.tsapps/landing/test/observability.test.ts
💤 Files with no reviewable changes (1)
- apps/landing/test/blog-navigation.test.ts
🚧 Files skipped from review as they are similar to previous changes (25)
- apps/landing/src/pages/blog/index.astro
- apps/landing/src/components/landing/Footer.astro
- apps/landing/src/pages/index.astro
- CONTEXT.md
- apps/landing/src/components/landing/CommandBlock.astro
- apps/landing/src/layouts/LandingLayout.astro
- apps/landing/src/components/landing/Breadth.astro
- apps/landing/src/components/landing/AgentTrace.astro
- apps/landing/src/components/landing/Proof.astro
- apps/catalog/src/pages/index.astro
- apps/landing/src/components/landing/WholeJob.astro
- apps/landing/src/components/landing/Activation.astro
- apps/landing/test/activation-links.test.ts
- README.md
- apps/catalog/src/styles/catalog.css
- apps/landing/src/components/landing/RemoteStory.astro
- apps/landing/src/data/landing.ts
- apps/catalog/src/pages/caplets/[entryKey].astro
- apps/landing/src/components/landing/WhyCaplets.astro
- apps/landing/src/components/landing/BlogCta.astro
- PRODUCT.md
- apps/docs/src/content/docs/index.mdx
- apps/landing/src/scripts/observability.ts
- STRATEGY.md
- apps/landing/src/components/landing/Hero.astro
| <a class="site-header__section-link inline-flex min-h-11 items-center rounded-md px-3 py-2 transition hover:bg-muted hover:text-foreground focus-visible:ring-3 focus-visible:ring-outline/50 focus-visible:outline-none" href="#remote">Remote</a> | ||
| <a class="site-header__section-link inline-flex min-h-11 items-center rounded-md px-3 py-2 transition hover:bg-muted hover:text-foreground focus-visible:ring-3 focus-visible:ring-outline/50 focus-visible:outline-none" href="#breadth">Backends</a> | ||
| <a class="site-header__section-link inline-flex min-h-11 items-center rounded-md px-3 py-2 transition hover:bg-muted hover:text-foreground focus-visible:ring-3 focus-visible:ring-outline/50 focus-visible:outline-none" href="#portable">Agents</a> | ||
| <a class="site-header__section-link inline-flex min-h-11 items-center rounded-md px-3 py-2 transition hover:bg-muted hover:text-foreground focus-visible:ring-3 focus-visible:ring-outline/50 focus-visible:outline-none" href="#work">Workflow</a> |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Add a scroll offset to the shared #work target.
The target in apps/landing/src/components/landing/WholeJob.astro lacks the offset used by the other anchored sections, so both links can hide the destination heading beneath the sticky header.
apps/landing/src/components/landing/Header.astro#L23-L23: update the desktop Workflow destination behavior.apps/landing/src/components/landing/Header.astro#L64-L64: update the mobile Issue to production destination behavior.
📍 Affects 1 file
apps/landing/src/components/landing/Header.astro#L23-L23(this comment)apps/landing/src/components/landing/Header.astro#L64-L64
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/landing/src/components/landing/Header.astro` at line 23, Update the
desktop Workflow link in Header.astro and the mobile Issue to production link in
Header.astro to use the same scroll-offset behavior as the other anchored
section links targeting `#work`, ensuring the destination heading remains visible
below the sticky header.
| </nav> | ||
|
|
||
| <div class="site-header__actions flex items-center gap-2" aria-label="Project links"> | ||
| <div class="site-header__actions flex items-center gap-2" role="group" aria-label="Project links"> |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Do not label the menu trigger as “Project links”.
On mobile, the GitHub/npm links are hidden and this group contains only the “Open navigation menu” control. Keep the wrapper layout-only, or apply the group label only to a container containing the project links.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/landing/src/components/landing/Header.astro` at line 36, Update the
site-header__actions wrapper in Header.astro so it does not use the “Project
links” aria-label when it contains only the mobile “Open navigation menu”
control; keep the wrapper layout-only or move the label to the container that
exclusively contains the GitHub/npm links.
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/landing/src/components/landing/RemoteStory.astro`:
- Line 15: The RemoteStory “Create and share a Caplet” CTA is categorized as
“docs” instead of the intended secondary CTA. Add an explicit
data-cta-category="secondary" attribute to that link in
apps/landing/src/components/landing/RemoteStory.astro; the related CTA in
apps/landing/src/components/landing/Proof.astro requires no direct change.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 2b62414b-4f74-4c18-aff6-fd90ee9c9fee
📒 Files selected for processing (12)
apps/landing/src/components/landing/Activation.astroapps/landing/src/components/landing/AgentTrace.astroapps/landing/src/components/landing/CommandBlock.astroapps/landing/src/components/landing/Proof.astroapps/landing/src/components/landing/RemoteStory.astroapps/landing/src/components/landing/WhyCaplets.astroapps/landing/src/data/landing.tsapps/landing/src/pages/index.astroapps/landing/src/scripts/agent-trace.tsapps/landing/src/styles/global.csspackages/benchmarks/lib/pi-eval/semantic-judge.tspackages/benchmarks/test/benchmark.test.ts
💤 Files with no reviewable changes (4)
- apps/landing/src/scripts/agent-trace.ts
- apps/landing/src/components/landing/AgentTrace.astro
- apps/landing/src/components/landing/CommandBlock.astro
- apps/landing/src/pages/index.astro
| </div> | ||
| </CardContent> | ||
| </Card> | ||
| <a class="mt-6 inline-flex min-h-11 items-center font-medium text-primary-accent hover:underline focus-visible:ring-3 focus-visible:ring-outline/50 focus-visible:outline-none" href="https://docs.caplets.dev/capabilities/">Create and share a Caplet</a> |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Inspect how ctaCategory() infers a category when data-cta-category is absent
rg -n -A10 "function ctaCategory|parseExplicitCtaCategory" apps/landing/src/scripts/observability.tsRepository: spiritledsoftware/caplets
Length of output: 1476
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Candidate files =="
fd -a 'RemoteStory\.astro|Proof\.astro|Activation\.astro|observability\.ts' apps/landing/src | sed 's#^\./##'
echo
echo "== Observability full relevant section =="
sed -n '1,230p' apps/landing/src/scripts/observability.ts | cat -n
echo
echo "== Landing components snippets =="
for f in apps/landing/src/components/landing/RemoteStory.astro apps/landing/src/components/landing/Proof.astro apps/landing/src/components/landing/Activation.astro; do
if [ -f "$f" ]; then
echo "--- $f ---"
cat -n "$f" | sed -n '1,120p'
fi
done
echo
echo "== All data-cta-category occurrences in landing components/scripts =="
rg -n --hidden --glob '!*node_modules*' --glob '*.{astro,ts,tsx,js,jsx}' 'data-cta-category|ctaCategory\(' apps/landing/src || trueRepository: spiritledsoftware/caplets
Length of output: 19484
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Button implementation =="
fd -a 'button.tsx|button.jsx|button.ts' apps/landing/src/components/starwind | sed 's#^\./##'
for f in $(fd -a 'button.tsx|button.jsx|button.ts' apps/landing/src/components/starwind); do
echo "--- $f ---"
cat -n "$f" | sed -n '1,220p'
done
echo
echo "== Astro render output for Button with dataset attribute =="
python3 - <<'PY'
import re
from pathlib import Path
p = Path("apps/landing/src/components/starwind/button.tsx")
src = p.read_text() if p.exists() else ""
m = re.search(r"data-cta-category|data-cta-category", src)
print("data-cta-category present in button.tsx:", bool(m))
for file in ["apps/landing/src/components/landing/Button.astro","apps/landing/src/components/starwind/button.tsx"]:
p = Path(file)
print(f"exists {file}:", p.exists())
PYRepository: spiritledsoftware/caplets
Length of output: 424
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Button component filenames =="
fd -i -a '^button' apps/landing/src/components/starwind apps/landing/src/components/landing 2>/dev/null | sed 's#^\./##' | head -20
echo
echo "== Search Button definition and dataset spread usage =="
rg -n -C6 'function Button|const Button|export \* from .*button|data-\{|data-cta-category|<button' apps/landing/src/components/starwind apps/landing/src/components/landing 2>/dev/null || trueRepository: spiritledsoftware/caplets
Length of output: 13201
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Button.astro =="
cat -n apps/landing/src/components/starwind/button/Button.astro | sed -n '1,220p'
echo
echo "== Behavioral probe: ctaCategory inference for current untagged CTAs =="
python3 - <<'PY'
from pathlib import Path
src = Path("apps/landing/src/scripts/observability.ts").read_text()
start = src.find("function ctaCategory")
end = src.find("\n}", start)
print(src[start:end+2])
cases = [
("Create and share a Caplet", "inside main", "<a href>"),
("Review the benchmark", "inside main", "<Button>"),
("Connect your own backend", "inside main", "<Button data-cta-category=\"secondary\">"),
]
for text, location, markup in cases:
text_lower = text.lower()
explicit = None
inferred = None
if text_lower.strip() == "connect your own backend":
explicit = "secondary"
elif text_lower.strip() == "create and share a caplet":
inferred = "docs"
elif text_lower.strip() == "review the benchmark":
inferred = "secondary"
print({"text": text, "markup": markup, "explicit": explicit, "inferred": inferred})
PYRepository: spiritledsoftware/caplets
Length of output: 1902
Tag the docs CTA so analytics report it consistently. ctaCategory() already falls back to "secondary" for "Review the benchmark" because it is in <main>, but "Create and share a Caplet" includes "docs" and gets reported as "docs" unless explicitly categorized. Add data-cta-category="secondary" (or whichever explicit category is desired) to the RemoteStory link.
📍 Affects 2 files
apps/landing/src/components/landing/RemoteStory.astro#L15-L15(this comment)apps/landing/src/components/landing/Proof.astro#L20-L23
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/landing/src/components/landing/RemoteStory.astro` at line 15, The
RemoteStory “Create and share a Caplet” CTA is categorized as “docs” instead of
the intended secondary CTA. Add an explicit data-cta-category="secondary"
attribute to that link in apps/landing/src/components/landing/RemoteStory.astro;
the related CTA in apps/landing/src/components/landing/Proof.astro requires no
direct change.
Benchmark entrypoint: bash autoresearch.sh Goal: without overfitting gpt-5.6 or the pi harness/integration, improve the token efficiency while maintaining the success rate
…guidance while preserving discovery, schema, fallback, synthesis, session, recovery, and compact-output semantics.
Result: {"status":"keep","token_burden":10849,"success_rate":1,"surface_tokens":2342,"workflow_tokens":1481,"workload_calls":4}
…guidance without removing tested discovery, schema, triage, output-shaping, or session semantics.
Result: {"status":"keep","token_burden":10633,"success_rate":1,"surface_tokens":2288,"workflow_tokens":1481,"workload_calls":4}
…ng tool description; retained generated Caplet IDs/descriptions plus every handle operation, result envelope, pagination, and debug reference.
Result: {"status":"keep","token_burden":5305,"success_rate":1,"surface_tokens":956,"workflow_tokens":1481,"workload_calls":4}
…e system prompt with one compact purpose/session/recovery line; dedicated tool guidance remains complete.
Result: {"status":"keep","token_burden":5017,"success_rate":1,"surface_tokens":884,"workflow_tokens":1481,"workload_calls":4}
…sion/recovery invariants while preserving all tested session and recovery phrases.
Result: {"status":"keep","token_burden":4909,"success_rate":1,"surface_tokens":857,"workflow_tokens":1481,"workload_calls":4}
…ile retaining all operations, discovery/schema rules, fallback/error handling, result envelopes, evidence synthesis, and debug access.
Result: {"status":"keep","token_burden":4733,"success_rate":1,"surface_tokens":813,"workflow_tokens":1481,"workload_calls":4}
…on-failure wording after focused verification exposed the contract; all workload calls still pass.
Result: {"status":"keep","token_burden":4749,"success_rate":1,"surface_tokens":817,"workflow_tokens":1481,"workload_calls":4}
…ols guidance marker, removing generic discovery instructions already present at the Code Mode level while retaining each Caplet name and domain description.
Result: {"status":"keep","token_burden":4449,"success_rate":1,"surface_tokens":742,"workflow_tokens":1481,"workload_calls":4}
… no longer receives irrelevant snake_case direct-tool instructions, while direct-tool exposures retain their full discovery/schema/triage guidance.
Result: {"status":"keep","token_burden":4049,"success_rate":1,"surface_tokens":642,"workflow_tokens":1481,"workload_calls":4}
Benchmark entrypoint: bash autoresearch.sh Goal: Without overfitting the executor model or the Pi harness/integration, reduce total Code Mode input plus output token burden while maintaining 100% success on deterministic workload and the mcp-real-world-large Pi eval suite.
… signatures, removed duplicated REPL guidance, and shortened the session schema description; deterministic success remains 100%.
Result: {"status":"keep","token_burden":3077,"success_rate":1,"surface_tokens":500,"workflow_tokens":1077,"workload_calls":4}
… while preserving metadata on failures and readiness errors; deterministic workload and focused contracts pass.
Result: {"status":"keep","token_burden":2925,"success_rate":1,"surface_tokens":500,"workflow_tokens":925,"workload_calls":4}
…pt guidance while shortening duplicate system-level session text; all focused native and Code Mode contracts pass.
Result: {"status":"keep","token_burden":2913,"success_rate":1,"surface_tokens":497,"workflow_tokens":925,"workload_calls":4}
… fence scaffolding while retaining discovery, exact signatures, error handling, persistence, and recovery semantics.
Result: {"status":"keep","token_burden":2737,"success_rate":1,"surface_tokens":453,"workflow_tokens":925,"workload_calls":4}
…mpressed duplicated prompt guidance while retaining session creation, reuse, and recovery behavior.
Result: {"status":"keep","token_burden":2661,"success_rate":1,"surface_tokens":434,"workflow_tokens":925,"workload_calls":4}
…t, and omitted the redundant sole-tool heading while retaining exact behavior and mixed-tool guidance.
Result: {"status":"keep","token_burden":2589,"success_rate":1,"surface_tokens":416,"workflow_tokens":925,"workload_calls":4}
…protocol and tightened workflow prose without removing discovery, schema, fallback, or evidence guidance.
Result: {"status":"keep","token_burden":2529,"success_rate":1,"surface_tokens":401,"workflow_tokens":925,"workload_calls":4}
…tails remain available. Deterministic success stayed 100%, and the architecture live replay passed in 25 rounds at 632,480 total tokens.
Result: {"status":"keep","token_burden":2529,"success_rate":1,"surface_tokens":401,"workflow_tokens":925,"workload_calls":4}
…, session reuse, and recovery safety semantics. Deterministic burden fell 32 tokens to 2,497; architecture replay passed at 467,242 total tokens across 22 calls, the best observed replay.
Result: {"status":"keep","token_burden":2497,"success_rate":1,"surface_tokens":393,"workflow_tokens":925,"workload_calls":4}
Benchmark entrypoint: bash autoresearch.sh Goal: Reduce caplets-code-mode total input plus output tokens on the full mcp-real-world-large suite while preserving 100% semantic success and remaining strictly more efficient than executor-mcp and vanilla-mcp, without model-, harness-, or Pi-integration overfitting.
Benchmark entrypoint: bash autoresearch.sh Goal: Reduce caplets-code-mode total input plus output tokens on the full mcp-real-world-large suite while preserving 100% semantic success and remaining strictly more efficient than executor-mcp and vanilla-mcp, without model-, harness-, or Pi-integration overfitting.
Benchmark entrypoint: bash autoresearch.sh Goal: Reduce caplets-code-mode total input plus output tokens on the full mcp-real-world-large suite while preserving 100% semantic success and remaining strictly more efficient than executor-mcp and vanilla-mcp, without model-, harness-, or Pi-integration overfitting.
…mode pass: all modes completed 5/5, Caplets used 696,705 average tokens versus Executor 1,131,991 and Vanilla 883,006. Caplets still averaged 25 provider requests, so the template hints preserved correctness but did not yet solve turn serialization.
Result: {"status":"keep","total_tokens":696705,"success_rate":1,"executor_total_tokens":1131991,"vanilla_total_tokens":883006,"executor_ratio":0.6154686742209081,"vanilla_ratio":0.7890150236804733,"average_calls":24}
… all three modes passed 5/5; Caplets used 555,132 average tokens versus Vanilla 569,598 and Executor 1,229,127. Caplets improved 20.3% from the 696,705 baseline and reduced average requests from 25 to 21.
Result: {"status":"keep","total_tokens":555132,"success_rate":1,"executor_total_tokens":1229127,"vanilla_total_tokens":569598,"executor_ratio":0.45164738875641003,"vanilla_ratio":0.9746031411627147,"average_calls":20}
…diagnostics/logs, and session/recovery metadata produced a new best: all modes passed 5/5; Caplets used 466,726 tokens versus Vanilla 900,810 and Executor 979,600. Caplets improved 15.9% over the prior best and 33.0% over baseline, with requests down to 17.
Result: {"status":"keep","total_tokens":466726,"success_rate":1,"executor_total_tokens":979600,"vanilla_total_tokens":900810,"executor_ratio":0.47644548795426706,"vanilla_ratio":0.5181181381201363,"average_calls":16}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@autoresearch.sh`:
- Around line 4-7: Use a fresh run-scoped directory for output_dir instead of
the shared fixed path, creating it with mktemp -d beneath the existing base
location and including it in cleanup alongside wrapper_dir. Ensure the report
collection and selection logic at the reports assignment and latest-report
selection only sees files generated by the current run.
In `@packages/core/src/code-mode/api.ts`:
- Around line 377-392: Update compactDefaultSearchPage so matching items
identified by summaryMatchesQuery(item, query) are placed before non-matching
items before applying the limit and slice. Preserve the existing limit
selection, input-limit bypass, truncation flag, and page behavior while ensuring
partial-match results retain matching items within the truncated output.
In `@packages/core/src/code-mode/runner.ts`:
- Around line 367-385: Add focused local and remote tests for
projectCodeModeToolEnvelope covering omission of empty diagnostics and logs,
retention of visible logs, preservation of non-empty diagnostics/logs, and
unchanged failure envelopes including error and session metadata. Keep existing
runCodeMode coverage separate and assert the projected response shape directly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9cda6d65-d2b9-448c-9238-f236e76ad8c8
⛔ Files ignored due to path filters (1)
packages/core/src/code-mode/runtime-api.generated.tsis excluded by!**/*.generated.*
📒 Files selected for processing (17)
.changeset/tidy-code-mode-token-burden.mdautoresearch.shpackages/benchmarks/run-token-efficiency.tspackages/core/src/code-mode/api.tspackages/core/src/code-mode/declarations.tspackages/core/src/code-mode/runner.tspackages/core/src/code-mode/runtime-api.d.tspackages/core/src/code-mode/tool.tspackages/core/src/native/remote.tspackages/core/src/native/service.tspackages/core/src/native/tools.tspackages/core/test/code-mode-api.test.tspackages/core/test/code-mode-mcp.test.tspackages/core/test/code-mode-runner.test.tspackages/core/test/native-remote.test.tspackages/core/test/native.test.tspackages/core/test/serve-session.test.ts
Comment on lines
+4
to
+7
| output_dir="${HOME}/.omp/caplets-autoresearch-live" | ||
| wrapper_dir="$(mktemp -d)" | ||
| trap 'rm -rf "$wrapper_dir"' EXIT | ||
| mkdir -p "$output_dir" |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Stale reports can be selected as "latest" from a shared, uncleaned output directory.
output_dir is a fixed path under ${HOME} and is never cleared between runs. At Line 146, reports=("$output_dir"/*.json) collects every JSON report ever written there, and Line 147 picks the lexicographically last entry as "the" report. If filenames from run-pi-eval.ts are not strictly time-sortable, or if an old report happens to sort after the new one, the script silently grades a stale run instead of the current one. Unlike wrapper_dir (created with mktemp -d and cleaned via trap), output_dir has no equivalent isolation.
Use a fresh, run-scoped output directory (for example mktemp -d under the base path) so only the current run's report can be picked up, or select the report by verified modification time after the run started.
🛠️ Proposed fix using a per-run output directory
-output_dir="${HOME}/.omp/caplets-autoresearch-live"
+output_dir="$(mktemp -d "${HOME}/.omp/caplets-autoresearch-live-XXXXXX")"Also applies to: 146-147
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@autoresearch.sh` around lines 4 - 7, Use a fresh run-scoped directory for
output_dir instead of the shared fixed path, creating it with mktemp -d beneath
the existing base location and including it in cleanup alongside wrapper_dir.
Ensure the report collection and selection logic at the reports assignment and
latest-report selection only sees files generated by the current run.
Comment on lines
+377
to
+392
| function compactDefaultSearchPage( | ||
| page: Page<unknown>, | ||
| query: string, | ||
| input: PageInput | undefined, | ||
| ): Page<unknown> { | ||
| if (input?.limit !== undefined) return page; | ||
| const limit = page.items.some((item) => summaryMatchesQuery(item, query)) | ||
| ? DEFAULT_CODE_MODE_LIST_LIMIT | ||
| : DEFAULT_CODE_MODE_FALLBACK_SEARCH_LIMIT; | ||
| if (page.items.length <= limit) return page; | ||
| return { | ||
| ...page, | ||
| items: page.items.slice(0, limit), | ||
| truncated: true, | ||
| }; | ||
| } |
Contributor
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Slicing does not prioritize matching items.
compactDefaultSearchPage picks the truncation size based on whether any item matches the query, using page.items.some((item) => summaryMatchesQuery(item, query)). It then applies page.items.slice(0, limit) on the original, unfiltered order. If only some items in the page match the query and a matching item sits beyond position limit, the current logic keeps the larger limit (10) but still cuts by position, so the actual matching item can end up truncated away while non-matching items ahead of it are kept.
This affects searchTools, searchResources, and searchPrompts, since they all use this helper. The tests at Lines 381-388 only cover the case where every item matches or none match, so this partial-match scenario is untested.
Filter or reorder items so matches are placed first before slicing.
🐛 Proposed fix to prioritize matches before truncation
function compactDefaultSearchPage(
page: Page<unknown>,
query: string,
input: PageInput | undefined,
): Page<unknown> {
if (input?.limit !== undefined) return page;
- const limit = page.items.some((item) => summaryMatchesQuery(item, query))
- ? DEFAULT_CODE_MODE_LIST_LIMIT
- : DEFAULT_CODE_MODE_FALLBACK_SEARCH_LIMIT;
- if (page.items.length <= limit) return page;
- return {
- ...page,
- items: page.items.slice(0, limit),
- truncated: true,
- };
+ const matching = page.items.filter((item) => summaryMatchesQuery(item, query));
+ const items = matching.length > 0 ? matching : page.items;
+ const limit =
+ matching.length > 0 ? DEFAULT_CODE_MODE_LIST_LIMIT : DEFAULT_CODE_MODE_FALLBACK_SEARCH_LIMIT;
+ if (items.length <= limit) return { ...page, items };
+ return {
+ ...page,
+ items: items.slice(0, limit),
+ truncated: true,
+ };
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| function compactDefaultSearchPage( | |
| page: Page<unknown>, | |
| query: string, | |
| input: PageInput | undefined, | |
| ): Page<unknown> { | |
| if (input?.limit !== undefined) return page; | |
| const limit = page.items.some((item) => summaryMatchesQuery(item, query)) | |
| ? DEFAULT_CODE_MODE_LIST_LIMIT | |
| : DEFAULT_CODE_MODE_FALLBACK_SEARCH_LIMIT; | |
| if (page.items.length <= limit) return page; | |
| return { | |
| ...page, | |
| items: page.items.slice(0, limit), | |
| truncated: true, | |
| }; | |
| } | |
| function compactDefaultSearchPage( | |
| page: Page<unknown>, | |
| query: string, | |
| input: PageInput | undefined, | |
| ): Page<unknown> { | |
| if (input?.limit !== undefined) return page; | |
| const matching = page.items.filter((item) => summaryMatchesQuery(item, query)); | |
| const items = matching.length > 0 ? matching : page.items; | |
| const limit = | |
| matching.length > 0 ? DEFAULT_CODE_MODE_LIST_LIMIT : DEFAULT_CODE_MODE_FALLBACK_SEARCH_LIMIT; | |
| if (items.length <= limit) return { ...page, items }; | |
| return { | |
| ...page, | |
| items: items.slice(0, limit), | |
| truncated: true, | |
| }; | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/core/src/code-mode/api.ts` around lines 377 - 392, Update
compactDefaultSearchPage so matching items identified by
summaryMatchesQuery(item, query) are placed before non-matching items before
applying the limit and slice. Preserve the existing limit selection, input-limit
bypass, truncation flag, and page behavior while ensuring partial-match results
retain matching items within the truncated output.
Comment on lines
+367
to
+385
| export function projectCodeModeToolEnvelope(envelope: CodeModeRunEnvelope): unknown { | ||
| if (!envelope.ok) return envelope; | ||
| const logsVisible = | ||
| envelope.logs.entries.length > 0 || | ||
| envelope.logs.truncated || | ||
| envelope.logs.stored || | ||
| envelope.logs.logRef !== undefined; | ||
| return { | ||
| ok: true, | ||
| value: envelope.value, | ||
| ...(envelope.diagnostics.length > 0 ? { diagnostics: envelope.diagnostics } : {}), | ||
| ...(logsVisible ? { logs: envelope.logs } : {}), | ||
| meta: { | ||
| sessionId: envelope.meta.sessionId ?? null, | ||
| sessionStatus: envelope.meta.sessionStatus ?? null, | ||
| recoveryRef: envelope.meta.recoveryRef ?? null, | ||
| }, | ||
| }; | ||
| } |
Contributor
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 \
--glob '*.{test,spec}.{ts,tsx}' \
'projectCodeModeToolEnvelope|executeCodeModeRunNative|executeCodeModeRunRemote|sessionStatus|recoveryRef|diagnostics|logs' \
packages/coreRepository: spiritledsoftware/caplets
Length of output: 50382
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Files matching runner and tests:\n'
git ls-files 'packages/core/**/runner.ts' 'packages/core/**/*runner*.test.ts' 'packages/core/**/*code-mode*.test.ts'
printf '\nrunner outline:\n'
ast-grep outline packages/core/src/code-mode/runner.ts || true
printf '\nrunner relevant section:\n'
sed -n '1,360p' packages/core/src/code-mode/runner.ts | cat -n
printf '\nfunction implementation:\n'
sed -n '360,400p' packages/core/src/code-mode/runner.ts | cat -n
printf '\ntest references in code-mode files only (narrowed):\n'
rg -n -C 4 \
'projectCodeModeToolEnvelope|executeCodeModeRunNative|executeCodeModeRunRemote|sessionStatus|recoveryRef|diagnostics:|logs:' \
packages/core/src packages/core/test \
--glob '*.test.ts' --glob '*.spec.ts' --glob '*.test.tsx' --glob '*.spec.tsx' \
| grep -E 'packages/core/(src|test)/.*\.(test|spec)\.t[sx]$' | head -200Repository: spiritledsoftware/caplets
Length of output: 19042
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Runner test file size:\n'
wc -l packages/core/test/code-mode-runner.test.ts
printf '\nRunner test references to project/run/envelope/remote/local/native/errors:\n'
rg -n -C 3 'projectCodeModeToolEnvelope|executeCodeModeRunNative|executeCodeModeRunRemote|runCodeMode|CodeModeRunEnvelope|diagnostic|error|SESSION|diagnostics|logs|metadata|sessionStatus|recoveryRef' packages/core/test/code-mode-runner.test.ts packages/core/test/code-mode-public-api.test.ts packages/core/test/code-mode-api.test.ts
printf '\nProgrammatic check for direct references:\n'
python3 - <<'PY'
from pathlib import Path
target = Path("packages/core/test/code-mode-runner.test.ts")
src = Path("packages/core/src/code-mode/runner.ts")
for fp in target, Path("packages/core/test/code-mode-public-api.test.ts"), Path("packages/core/test/code-mode-api.test.ts"):
print(f"{'exists' if fp.exists() else 'missing'}: {fp}")
if fp.exists():
txt = fp.read_text()
print(f" projectCodeModeToolEnvelope={txt.find('projectCodeModeToolEnvelope')}")
PYRepository: spiritledsoftware/caplets
Length of output: 32196
Add coverage for projectCodeModeToolEnvelope.
Coverage for the raw runCodeMode envelope does not cover the projected response shape. Add local and remote tests for projectCodeModeToolEnvelope that assert omitted empty diagnostics/logs, retained visible logs, preserved non-empty diagnostics/logs, and unchanged failure envelopes with error/session metadata.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/core/src/code-mode/runner.ts` around lines 367 - 385, Add focused
local and remote tests for projectCodeModeToolEnvelope covering omission of
empty diagnostics and logs, retention of visible logs, preservation of non-empty
diagnostics/logs, and unchanged failure envelopes including error and session
metadata. Keep existing runCodeMode coverage separate and assert the projected
response shape directly.
Source: Coding guidelines
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
pnpm verifyCloses #244
Summary by CodeRabbit