chore(toolkit-docs): restore lost summaries for 6 toolkits#928
Merged
chore(toolkit-docs): restore lost summaries for 6 toolkits#928
Conversation
The summary field for github, googledocs, jira, salesforce, daytona, and linear was silently wiped to null across recent [AUTO] docs update merges (see #926 for the root cause — regeneration fallback paths did not preserve the previous summary). Each restored summary was rewritten from the last known-good version in git history and then updated to reflect the toolkit's CURRENT tools. Notable capability drift accounted for: - github: drop notifications (no notification tools currently); drop CLASSIC_PERSONAL_ACCESS_TOKEN (only GITHUB_SERVER_URL remains); add explicit Projects V2 fields/items and user-centric views (review workload, open items, recent activity). - googledocs: clarify SearchAndRetrieveDocuments returns body content while SearchDocuments is metadata-only; keep File Picker recovery. - jira: no capability drift; tighten wording around single-call batching and name/key/email reference resolution. - salesforce: no capability drift; add explicit mention of related- object enrichment (contact roles, line items) visible in current descriptions. - daytona: drop abstract "metadata for coordination"; add ListRegions, signed vs. standard port preview URLs, snapshots as a separate lifecycle. - linear: drop the comment-anchoring caveat (no longer present in tool descriptions); add pagination for project/initiative descriptions and ManageIssueSubscription. This restoration is intentionally separate from #927 (the code fix) so content review can happen independently of the regression fix. Refs #926 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Apr 17, 2026
…to Arcade auth provider docs Per follow-up feedback, summaries should not repeat OAuth scopes — they belong in the per-toolkit auth provider documentation, not the overview prose. Each of the six restored summaries now: - States whether OAuth is required and names the provider. - Links to the matching Arcade auth provider page under /en/references/auth-providers/<provider> instead of listing scopes. - Where applicable, adds explicit explanations for each secret plus a link to the Arcade Dashboard secret setup flow. The deterministic secret-coverage check in #932 is not changed here — that PR already accepts either the Arcade secret docs link or the Dashboard URL, both of which remain present in these summaries. Refs #926 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jottakka
added a commit
that referenced
this pull request
Apr 18, 2026
…cade auth provider docs Same follow-up as #928 applied to the five toolkits with brand-new summaries. Each OAuth section now states the provider and links to /en/references/auth-providers/<provider> instead of enumerating scopes. Secrets sections were also tightened with per-secret provenance and a link to the Arcade secret setup guide. Refs #926 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jottakka
added a commit
that referenced
this pull request
Apr 18, 2026
…scopes Per follow-up on PRs #928 and #929, the OAuth section of each summary should name the provider and link to the Arcade per-provider auth docs rather than enumerate scopes. Scopes already live on the provider reference page and repeating them in toolkit summaries creates drift every time a provider page updates. Changes: - Add ARCADE_AUTH_PROVIDERS_BASE_URL constant alongside the existing Arcade secret URLs in secret-coherence.ts. - Rewrite the OAuth bullet in toolkit-summary-generator.ts's prompt to require a link to {base}/<providerId> and explicitly forbid listing scopes. - Drop scopes from formatAuth's prompt payload so the model has no stray scope list to fall back on. - README: note the no-scopes-in-summary rule and point to the provider reference pages as the source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
torresmateo
reviewed
Apr 20, 2026
| "subPages": [], | ||
| "generatedAt": "2026-04-02T11:25:02.814Z" | ||
| } No newline at end of file | ||
| "generatedAt": "2026-04-02T11:25:02.814Z", |
Collaborator
There was a problem hiding this comment.
This seems like the generation date is not taking the actual generation date, is this intentional?
torresmateo
approved these changes
Apr 20, 2026
Resolve linear.json: keep restored summary from PR with main's generatedAt. Made-with: Cursor
jottakka
added a commit
that referenced
this pull request
Apr 20, 2026
* chore(toolkit-docs): add missing summaries for 5 toolkits posthog, granola, microsoftoutlookcalendar, microsoftoutlookmail, and zohocreatorapi never received a summary — LLM generation was either skipped or failed on the first auto-update run and the null state has persisted ever since (see #926 for the underlying regression). Each summary is written from the actual tool list in the toolkit JSON, matching the structure the LLM prompt targets (60–140 words, Capabilities bullets, OAuth / Secrets sections) so the rendered overview looks consistent with LLM-generated peers. Refs #926 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(toolkit-docs): drop OAuth scopes from new summaries, link to Arcade auth provider docs Same follow-up as #928 applied to the five toolkits with brand-new summaries. Each OAuth section now states the provider and links to /en/references/auth-providers/<provider> instead of enumerating scopes. Secrets sections were also tightened with per-secret provenance and a link to the Arcade secret setup guide. Refs #926 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
torresmateo noted daytona.json carried an older generatedAt than the actual edit. Refresh timestamps for toolkit JSON touched in this PR where dates were still from prior generator runs (Apr 2 / Apr 10). Made-with: Cursor
jottakka
added a commit
that referenced
this pull request
Apr 28, 2026
…ts (#932) * feat(toolkit-docs-generator): secret-coherence scan + minimal LLM edits When a toolkit loses a secret upstream (typically because the tool that required it was dropped), the rendered docs can continue to mention that secret in the summary and in hand-authored documentation chunks. One concrete example on main: github.json still references GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN after the notification tools were removed in PR #922. Symmetrically, toolkits can end up with current secrets that the summary never mentions, or mention secrets without any link to the Arcade config docs — leaving readers without the information needed to actually configure them. This adds a two-stage pipeline that runs after summary generation: 1. Deterministic scanners (src/merger/secret-coherence.ts) - detectStaleSecretReferences: diffs current vs previous toolkit secret sets and scans summary, toolkit chunks, and per-tool chunks by exact substring for each removed secret. - detectSecretCoverageGaps: flags current secrets missing from the summary and a missing link to the Arcade secret config docs. - groupStaleRefsByTarget: collapses multiple removed-secret hits in the same artifact into a single edit target so the LLM is called at most once per (summary | chunk). 2. Targeted LLM editor (src/llm/secret-edit-generator.ts) - Unlike toolkit-summary-generator (which rewrites from scratch and tends to oversimplify), this editor is prompted to make the smallest possible change: delete sentences/rows that are only about the removed secret, minimally rewrite any sentence that mentions the removed secret alongside other content, and never re-summarize or reorder sections. - A separate fillCoverageGaps method adds missing secret mentions and, when required, the Arcade config docs link — also without rewriting unrelated text. Both steps are wired into DataMerger.enforceSecretCoherence, called after maybeGenerateSummary. The editor is optional: if unconfigured, the scanners still run and emit warnings, but no content is rewritten. Failures in the editor are caught and surfaced as warnings so a single LLM error does not break the run. Wiring changes: - DataMergerConfig gains an optional secretEditGenerator. - CLI gains --llm-editor-provider / --llm-editor-model / --llm-editor-api-key / --llm-editor-base-url / etc., mirrored by LLM_EDITOR_* env vars, with --skip-secret-coherence for the scan-and-edit step. Resolver fails open: a missing API key degrades to scanner-only warnings instead of crashing the run. - Workflow generate-toolkit-docs.yml now passes editor flags pointing at Anthropic + claude-sonnet-4-6 (overridable via secrets) so the editor stays on a stronger model than the gpt-4o-mini used for bulk summary and example generation. Summary prompt updates (src/llm/toolkit-summary-generator.ts): - Drop the hard 60–140 word cap; ask for "compact but complete". - Require each current secret be named in backticks with a one-line factual description of how to obtain it from the provider. - Require the Arcade secret config docs link at the end of the **Secrets** section. Tests: - tests/merger/secret-coherence.test.ts (13 tests) covers scanner behavior across summary, toolkit chunks, tool chunks, coverage gaps, and target grouping. - tests/llm/secret-edit-generator.test.ts (6 tests) exercises the cleanup/coverage flows and the fence-stripping / empty-response guards with a mocked LLM client. - Two new DataMerger integration tests verify that a removed secret surfacing in a doc chunk drives exactly one cleanup call and that the editor-disabled path still emits the warning. - tests/workflows/generate-toolkit-docs.test.ts asserts the new editor flags are present in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toolkit-docs-generator): address ACR findings on secret coherence Two issues surfaced by `/acr-run`: 1. FENCE_PATTERN (secret-edit-generator.ts) was non-greedy and unanchored, so stripOptionalFence stopped at the FIRST inner ``` when the LLM wrapped its edit in a markdown fence and the edit itself contained a fenced code block. Result: the rest of the edit was silently dropped with no error — corrupted doc chunks written to disk. Fix: anchor the pattern to ^…$ and use a greedy capture so the match extends to the outer closing fence. 2. enforceSecretCoherence (data-merger.ts) computed coverage gaps once, before stale cleanup ran. If cleanup modifies the summary and incidentally drops a passage that mentioned a current secret, the pre-cleanup gap snapshot would miss it. Fix: re-run detectSecretCoherenceIssues after applyStaleRefCleanup so the coverage fill sees post-cleanup state. Tests: - Two new fence tests cover (a) preserving inner code blocks when unwrapping the outer fence, and (b) leaving unwrapped responses with inner blocks untouched. - One new DataMerger test proves the coverage editor receives post- cleanup summary content (not a stale snapshot). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(toolkit-docs-generator): raise editor max-tokens default to 8192 4096 was tight. Largest single artifact in current data is a ~6K-char doc chunk (googlenews) ≈ 1.5K output tokens for a minimal-edit rewrite; a summary with no word cap for a 40+ tool toolkit with several secrets can land in the 2–3K output-token range. 8K gives comfortable margin without meaningful cost or latency impact on Sonnet 4.6. Help text updated to match. Callers can still override via --llm-editor-max-tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(toolkit-docs-generator): document secret coherence + loosen per-secret prose cap Two changes: 1. README: new Secret coherence section covering the scan/edit pipeline, the editor CLI flags, the claude-sonnet-4-6 default, fail-open behavior when no API key is set, and a local invocation example. Required/optional CI secrets updated with ANTHROPIC_API_KEY and ANTHROPIC_EDITOR_MODEL. Key CLI options list updated with the new flags. 2. Prompts (summary generator + coverage-fill editor) no longer cap each secret at one line. Instead they ask for as much detail as the secret actually needs — a short URL override may be one line; a scoped API key typically needs several sentences naming the provider dashboard page, required scopes or permissions, and any account tier. Both prompts also request an inline markdown link to the provider's own docs page for how to create/retrieve the secret when the model knows it, and explicitly forbid inventing URLs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(toolkit-docs-generator): summary prompt no longer repeats OAuth scopes Per follow-up on PRs #928 and #929, the OAuth section of each summary should name the provider and link to the Arcade per-provider auth docs rather than enumerate scopes. Scopes already live on the provider reference page and repeating them in toolkit summaries creates drift every time a provider page updates. Changes: - Add ARCADE_AUTH_PROVIDERS_BASE_URL constant alongside the existing Arcade secret URLs in secret-coherence.ts. - Rewrite the OAuth bullet in toolkit-summary-generator.ts's prompt to require a link to {base}/<providerId> and explicitly forbid listing scopes. - Drop scopes from formatAuth's prompt payload so the model has no stray scope list to fall back on. - README: note the no-scopes-in-summary rule and point to the provider reference pages as the source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toolkit-docs-generator): address ACR findings on secret coherence (round 2) Four findings from /acr-run: 1. HIGH (5/5) — ANTHROPIC_EDITOR_MODEL was documented as a fallback env var in the README but never read by resolveSecretEditGenerator. A local dev setting only ANTHROPIC_EDITOR_MODEL would get `model = undefined`, the (provider && model) guard would fire, and the editor would silently stay inactive. Extract resolveEditorModel helper that walks `--llm-editor-model` → LLM_EDITOR_MODEL → ANTHROPIC_EDITOR_MODEL in documented order, and use it from both the resolver and the verbose-log blocks. 2. MEDIUM — --skip-secret-coherence was documented to "disable both the scan and the edit step entirely" but DataMerger never received the flag; enforceSecretCoherence always ran, so coherence warnings still appeared when the user explicitly opted out. Add `skipSecretCoherence` to DataMergerConfig, gate enforcement on it, and pass it through from all three merger construction sites in the CLI. 3. MEDIUM — FENCE_PATTERN matched non-markdown language fences (```python, ```bash, ```json). A documentation chunk whose content was a code block would have its fences stripped, corrupting the edited output. Tightened the pattern to require either an empty, markdown, md, or text tag followed by a newline between the opening fence and the captured content, so language-tagged code blocks fall through stripOptionalFence unchanged. 4. LOW — verbose log showed "model: undefined" when only ANTHROPIC_EDITOR_MODEL was set. Fixed by #1. Tests added: - fence strip preserves `\`\`\`python` and `\`\`\`bash` code blocks verbatim - skipSecretCoherence suppresses both edits and warnings 549 tests pass, type-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(workflow): Node 24 opt-in + focused workflow_dispatch for manual runs Two workflow additions driven by PR #936 feedback: 1. Job-level `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"` opts all JavaScript actions into Node 24 ahead of the 2026-06-02 deprecation. actions/checkout@v4, actions/setup-node@v4, peter-evans/create-pull-request@v7, and pnpm/action-setup@v4 all trigger the "Node.js 20 actions are deprecated" annotation today; the opt-in silences it and matches the runtime we'll be forced onto anyway. 2. New `workflow_dispatch` input `providers`. When set to a comma-separated provider list (e.g. "Github"), the run uses `--providers "$providers"` AND drops `--skip-unchanged` so the secret-coherence scan actually re-evaluates those toolkits — even when the Engine API reports no version change. Scheduled and porter_deploy_succeeded runs keep the previous `--all --skip-unchanged` behavior. This is what lets the #935 demo PR actually exercise the pipeline end-to-end: trigger the workflow with `providers=Github` and the phantom secret gets surfaced + cleaned. Tests added: workflow assertions for the new env var and the providers input fallback structure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): surface per-toolkit merge warnings to stdout in --all runs The stale-secret scanner, coverage-gap detector, and summary-generation failures all push warnings onto `result.warnings`. Per-provider mode already echoes those to stdout (line 848 of cli/index.ts). The --all and regenerate-all paths did not — they only appended to the run log file on disk, which GitHub Actions runs don't expose. Result: on the #935 demo, the workflow ran, the phantom secret was removed from the tool's .secrets array, but no cleanup was applied to the stale doc chunk that still referenced it AND there was no signal in the CI log explaining why. The warnings that would have explained "stale secret detected but edit failed" or "stale secret detected but no editor configured" were present in memory but discarded. This commit prints every non-empty `mergeResult.warnings` to stdout right after `mergeAllToolkits()` returns, in both the `generate --all` and `regenerate --all` paths. Format matches existing spinner output: ⚠ Github: 2 warning(s) - Stale secret reference in toolkit_chunk #4: GITHUB_CLASSIC_... - Secret cleanup edit failed for Github (documentation_chunk): ... 551 tests pass, type-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(workflow): remove focused providers dispatch input Simplify manual toolkit docs runs by removing the workflow_dispatch providers override and restoring the default full run path with --all --skip-unchanged. Made-with: Cursor * fix(toolkit-docs-generator): address cursor review on secret coherence Reuse shared secret collection logic across merger modules and restore stale-summary coverage in data-merger tests. Made-with: Cursor * fix(cli): use editor flag in provider validation message Pass the editor-specific option name to provider validation so invalid --llm-editor-provider values return actionable guidance. Made-with: Cursor --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
summaryfield for github, googledocs, jira, salesforce, daytona, linear — all six had rich, hand-refined overviews that were silently wiped tonullacross recent[AUTO] Adding MCP Servers docs updatemerges.LlmToolkitSummaryGeneratorproduces (60–140 words, Capabilities bullets, OAuth / Secrets sections) so future LLM regeneration stays on-style.Capability drift accounted for
githubCLASSIC_PERSONAL_ACCESS_TOKEN(onlyGITHUB_SERVER_URLremains). Added Projects V2 fields/items and user-centric views (review workload, open items, recent activity).googledocsSearchAndRetrieveDocumentsreturns body content vs.SearchDocumentsmetadata-only. Kept File Picker recovery pattern.jirasalesforcedaytonaListRegions, signed vs. standard port preview URLs, explicit snapshot lifecycle.linearManageIssueSubscription.Why separate from the code fix
The regression fix (#927) lands code changes to
data-merger.ts. This PR is content-only so reviewers can evaluate wording and truthfulness independently — and so either can merge without blocking the other. Without the code fix, the next auto-update run that changes a toolkit's tool signature will wipe these summaries again (signature mismatch → regen; if LLM unavailable/fails → wipe). Recommend merging #927 first.Test plan
LlmToolkitSummaryGenerator)..auth.allScopes./en/mcp-servers/development/github).Refs #926
🤖 Generated with Claude Code
Note
Low Risk
Content-only changes that add back human-readable toolkit
summarytext; no runtime code paths or APIs are modified, so risk is limited to documentation accuracy.Overview
Restores and adds rich
summaryblocks to six generated toolkit JSON descriptors (daytona,github,googledocs,jira,linear,salesforce), covering capabilities plus OAuth/secrets guidance.Also updates the affected files’
generatedAttimestamps and fixes the JSON endings (adds trailing commas/newlines) so the summaries are persisted in the generated artifacts.Reviewed by Cursor Bugbot for commit 3b7a17f. Bugbot is set up for automated code reviews on this repo. Configure here.