chore(toolkit-docs): add missing summaries for 5 toolkits#929
Merged
Conversation
posthog, granola, microsoftoutlookcalendar, microsoftoutlookmail, and zohocreatorapi never received a summary — LLM generation was either skipped or failed on the first auto-update run and the null state has persisted ever since (see #926 for the underlying regression). Each summary is written from the actual tool list in the toolkit JSON, matching the structure the LLM prompt targets (60–140 words, Capabilities bullets, OAuth / Secrets sections) so the rendered overview looks consistent with LLM-generated peers. Refs #926 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…cade auth provider docs Same follow-up as #928 applied to the five toolkits with brand-new summaries. Each OAuth section now states the provider and links to /en/references/auth-providers/<provider> instead of enumerating scopes. Secrets sections were also tightened with per-secret provenance and a link to the Arcade secret setup guide. Refs #926 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
jottakka
added a commit
that referenced
this pull request
Apr 18, 2026
…scopes Per follow-up on PRs #928 and #929, the OAuth section of each summary should name the provider and link to the Arcade per-provider auth docs rather than enumerate scopes. Scopes already live on the provider reference page and repeating them in toolkit summaries creates drift every time a provider page updates. Changes: - Add ARCADE_AUTH_PROVIDERS_BASE_URL constant alongside the existing Arcade secret URLs in secret-coherence.ts. - Rewrite the OAuth bullet in toolkit-summary-generator.ts's prompt to require a link to {base}/<providerId> and explicitly forbid listing scopes. - Drop scopes from formatAuth's prompt payload so the model has no stray scope list to fall back on. - README: note the no-scopes-in-summary rule and point to the provider reference pages as the source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
torresmateo
approved these changes
Apr 20, 2026
byrro
approved these changes
Apr 20, 2026
jottakka
added a commit
that referenced
this pull request
Apr 28, 2026
…ts (#932) * feat(toolkit-docs-generator): secret-coherence scan + minimal LLM edits When a toolkit loses a secret upstream (typically because the tool that required it was dropped), the rendered docs can continue to mention that secret in the summary and in hand-authored documentation chunks. One concrete example on main: github.json still references GITHUB_CLASSIC_PERSONAL_ACCESS_TOKEN after the notification tools were removed in PR #922. Symmetrically, toolkits can end up with current secrets that the summary never mentions, or mention secrets without any link to the Arcade config docs — leaving readers without the information needed to actually configure them. This adds a two-stage pipeline that runs after summary generation: 1. Deterministic scanners (src/merger/secret-coherence.ts) - detectStaleSecretReferences: diffs current vs previous toolkit secret sets and scans summary, toolkit chunks, and per-tool chunks by exact substring for each removed secret. - detectSecretCoverageGaps: flags current secrets missing from the summary and a missing link to the Arcade secret config docs. - groupStaleRefsByTarget: collapses multiple removed-secret hits in the same artifact into a single edit target so the LLM is called at most once per (summary | chunk). 2. Targeted LLM editor (src/llm/secret-edit-generator.ts) - Unlike toolkit-summary-generator (which rewrites from scratch and tends to oversimplify), this editor is prompted to make the smallest possible change: delete sentences/rows that are only about the removed secret, minimally rewrite any sentence that mentions the removed secret alongside other content, and never re-summarize or reorder sections. - A separate fillCoverageGaps method adds missing secret mentions and, when required, the Arcade config docs link — also without rewriting unrelated text. Both steps are wired into DataMerger.enforceSecretCoherence, called after maybeGenerateSummary. The editor is optional: if unconfigured, the scanners still run and emit warnings, but no content is rewritten. Failures in the editor are caught and surfaced as warnings so a single LLM error does not break the run. Wiring changes: - DataMergerConfig gains an optional secretEditGenerator. - CLI gains --llm-editor-provider / --llm-editor-model / --llm-editor-api-key / --llm-editor-base-url / etc., mirrored by LLM_EDITOR_* env vars, with --skip-secret-coherence for the scan-and-edit step. Resolver fails open: a missing API key degrades to scanner-only warnings instead of crashing the run. - Workflow generate-toolkit-docs.yml now passes editor flags pointing at Anthropic + claude-sonnet-4-6 (overridable via secrets) so the editor stays on a stronger model than the gpt-4o-mini used for bulk summary and example generation. Summary prompt updates (src/llm/toolkit-summary-generator.ts): - Drop the hard 60–140 word cap; ask for "compact but complete". - Require each current secret be named in backticks with a one-line factual description of how to obtain it from the provider. - Require the Arcade secret config docs link at the end of the **Secrets** section. Tests: - tests/merger/secret-coherence.test.ts (13 tests) covers scanner behavior across summary, toolkit chunks, tool chunks, coverage gaps, and target grouping. - tests/llm/secret-edit-generator.test.ts (6 tests) exercises the cleanup/coverage flows and the fence-stripping / empty-response guards with a mocked LLM client. - Two new DataMerger integration tests verify that a removed secret surfacing in a doc chunk drives exactly one cleanup call and that the editor-disabled path still emits the warning. - tests/workflows/generate-toolkit-docs.test.ts asserts the new editor flags are present in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toolkit-docs-generator): address ACR findings on secret coherence Two issues surfaced by `/acr-run`: 1. FENCE_PATTERN (secret-edit-generator.ts) was non-greedy and unanchored, so stripOptionalFence stopped at the FIRST inner ``` when the LLM wrapped its edit in a markdown fence and the edit itself contained a fenced code block. Result: the rest of the edit was silently dropped with no error — corrupted doc chunks written to disk. Fix: anchor the pattern to ^…$ and use a greedy capture so the match extends to the outer closing fence. 2. enforceSecretCoherence (data-merger.ts) computed coverage gaps once, before stale cleanup ran. If cleanup modifies the summary and incidentally drops a passage that mentioned a current secret, the pre-cleanup gap snapshot would miss it. Fix: re-run detectSecretCoherenceIssues after applyStaleRefCleanup so the coverage fill sees post-cleanup state. Tests: - Two new fence tests cover (a) preserving inner code blocks when unwrapping the outer fence, and (b) leaving unwrapped responses with inner blocks untouched. - One new DataMerger test proves the coverage editor receives post- cleanup summary content (not a stale snapshot). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(toolkit-docs-generator): raise editor max-tokens default to 8192 4096 was tight. Largest single artifact in current data is a ~6K-char doc chunk (googlenews) ≈ 1.5K output tokens for a minimal-edit rewrite; a summary with no word cap for a 40+ tool toolkit with several secrets can land in the 2–3K output-token range. 8K gives comfortable margin without meaningful cost or latency impact on Sonnet 4.6. Help text updated to match. Callers can still override via --llm-editor-max-tokens. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs(toolkit-docs-generator): document secret coherence + loosen per-secret prose cap Two changes: 1. README: new Secret coherence section covering the scan/edit pipeline, the editor CLI flags, the claude-sonnet-4-6 default, fail-open behavior when no API key is set, and a local invocation example. Required/optional CI secrets updated with ANTHROPIC_API_KEY and ANTHROPIC_EDITOR_MODEL. Key CLI options list updated with the new flags. 2. Prompts (summary generator + coverage-fill editor) no longer cap each secret at one line. Instead they ask for as much detail as the secret actually needs — a short URL override may be one line; a scoped API key typically needs several sentences naming the provider dashboard page, required scopes or permissions, and any account tier. Both prompts also request an inline markdown link to the provider's own docs page for how to create/retrieve the secret when the model knows it, and explicitly forbid inventing URLs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(toolkit-docs-generator): summary prompt no longer repeats OAuth scopes Per follow-up on PRs #928 and #929, the OAuth section of each summary should name the provider and link to the Arcade per-provider auth docs rather than enumerate scopes. Scopes already live on the provider reference page and repeating them in toolkit summaries creates drift every time a provider page updates. Changes: - Add ARCADE_AUTH_PROVIDERS_BASE_URL constant alongside the existing Arcade secret URLs in secret-coherence.ts. - Rewrite the OAuth bullet in toolkit-summary-generator.ts's prompt to require a link to {base}/<providerId> and explicitly forbid listing scopes. - Drop scopes from formatAuth's prompt payload so the model has no stray scope list to fall back on. - README: note the no-scopes-in-summary rule and point to the provider reference pages as the source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(toolkit-docs-generator): address ACR findings on secret coherence (round 2) Four findings from /acr-run: 1. HIGH (5/5) — ANTHROPIC_EDITOR_MODEL was documented as a fallback env var in the README but never read by resolveSecretEditGenerator. A local dev setting only ANTHROPIC_EDITOR_MODEL would get `model = undefined`, the (provider && model) guard would fire, and the editor would silently stay inactive. Extract resolveEditorModel helper that walks `--llm-editor-model` → LLM_EDITOR_MODEL → ANTHROPIC_EDITOR_MODEL in documented order, and use it from both the resolver and the verbose-log blocks. 2. MEDIUM — --skip-secret-coherence was documented to "disable both the scan and the edit step entirely" but DataMerger never received the flag; enforceSecretCoherence always ran, so coherence warnings still appeared when the user explicitly opted out. Add `skipSecretCoherence` to DataMergerConfig, gate enforcement on it, and pass it through from all three merger construction sites in the CLI. 3. MEDIUM — FENCE_PATTERN matched non-markdown language fences (```python, ```bash, ```json). A documentation chunk whose content was a code block would have its fences stripped, corrupting the edited output. Tightened the pattern to require either an empty, markdown, md, or text tag followed by a newline between the opening fence and the captured content, so language-tagged code blocks fall through stripOptionalFence unchanged. 4. LOW — verbose log showed "model: undefined" when only ANTHROPIC_EDITOR_MODEL was set. Fixed by #1. Tests added: - fence strip preserves `\`\`\`python` and `\`\`\`bash` code blocks verbatim - skipSecretCoherence suppresses both edits and warnings 549 tests pass, type-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(workflow): Node 24 opt-in + focused workflow_dispatch for manual runs Two workflow additions driven by PR #936 feedback: 1. Job-level `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"` opts all JavaScript actions into Node 24 ahead of the 2026-06-02 deprecation. actions/checkout@v4, actions/setup-node@v4, peter-evans/create-pull-request@v7, and pnpm/action-setup@v4 all trigger the "Node.js 20 actions are deprecated" annotation today; the opt-in silences it and matches the runtime we'll be forced onto anyway. 2. New `workflow_dispatch` input `providers`. When set to a comma-separated provider list (e.g. "Github"), the run uses `--providers "$providers"` AND drops `--skip-unchanged` so the secret-coherence scan actually re-evaluates those toolkits — even when the Engine API reports no version change. Scheduled and porter_deploy_succeeded runs keep the previous `--all --skip-unchanged` behavior. This is what lets the #935 demo PR actually exercise the pipeline end-to-end: trigger the workflow with `providers=Github` and the phantom secret gets surfaced + cleaned. Tests added: workflow assertions for the new env var and the providers input fallback structure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(cli): surface per-toolkit merge warnings to stdout in --all runs The stale-secret scanner, coverage-gap detector, and summary-generation failures all push warnings onto `result.warnings`. Per-provider mode already echoes those to stdout (line 848 of cli/index.ts). The --all and regenerate-all paths did not — they only appended to the run log file on disk, which GitHub Actions runs don't expose. Result: on the #935 demo, the workflow ran, the phantom secret was removed from the tool's .secrets array, but no cleanup was applied to the stale doc chunk that still referenced it AND there was no signal in the CI log explaining why. The warnings that would have explained "stale secret detected but edit failed" or "stale secret detected but no editor configured" were present in memory but discarded. This commit prints every non-empty `mergeResult.warnings` to stdout right after `mergeAllToolkits()` returns, in both the `generate --all` and `regenerate --all` paths. Format matches existing spinner output: ⚠ Github: 2 warning(s) - Stale secret reference in toolkit_chunk #4: GITHUB_CLASSIC_... - Secret cleanup edit failed for Github (documentation_chunk): ... 551 tests pass, type-check clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(workflow): remove focused providers dispatch input Simplify manual toolkit docs runs by removing the workflow_dispatch providers override and restoring the default full run path with --all --skip-unchanged. Made-with: Cursor * fix(toolkit-docs-generator): address cursor review on secret coherence Reuse shared secret collection logic across merger modules and restore stale-summary coverage in data-merger tests. Made-with: Cursor * fix(cli): use editor flag in provider validation message Pass the editor-specific option name to provider validation so invalid --llm-editor-provider values return actionable guidance. Made-with: Cursor --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
summaryfor 5 toolkits that have always shipped withsummary: nullbecause LLM generation was skipped or failed on their first auto-update run: posthog, granola, microsoftoutlookcalendar, microsoftoutlookmail, zohocreatorapi.toolkit-docs-generator/data/toolkits/<id>.json. Format matches whatLlmToolkitSummaryGeneratorproduces (60–140 words, Capabilities bullets, OAuth / Secrets sections).Why
Without a summary the rendered toolkit page (app/_components/toolkit-docs/components/toolkit-page.tsx:647) omits the prose overview block. Readers land on a page with just the header + tool list and no narrative orientation.
Per-toolkit notes
posthoggranolamicrosoftoutlookcalendaroutlookcalendar(see #874).microsoftoutlookmailoutlookmail(see #874).zohocreatorapiZOHO_SERVER_URLsecret for regional domain selection.Why separate from #927 / #928
Test plan
.auth.allScopesin the JSON.excluded-toolkits.txtorignored-toolkits.txt./en/mcp-servers/productivity/posthog).Refs #926
🤖 Generated with Claude Code
Note
Low Risk
Low risk documentation-only change: adds
summarystrings to existing toolkit JSON metadata without altering tool definitions, auth, or runtime code paths.Overview
Adds
summaryfields to five toolkit definition JSONs (granola,microsoftoutlookcalendar,microsoftoutlookmail,posthog,zohocreatorapi), providing a prose overview plus Capabilities, and where applicable OAuth/Secrets sections.Also normalizes the JSON file endings by keeping
generatedAtand appending the newsummaryfield with proper trailing comma/newline so the metadata is valid and complete.Reviewed by Cursor Bugbot for commit ac753d2. Bugbot is set up for automated code reviews on this repo. Configure here.