From 8007d423fba82a1991446b41008dfa380a68eec3 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 12:24:19 -0400 Subject: [PATCH 01/31] feat(vue-ai-apps): add skill for building AI/agent apps in Vue/Nuxt New skill covering the Vercel AI SDK v5 integration layer for Vue: - streaming-chat-ui: useChat (@ai-sdk/vue), UIMessage parts, manual input ref, status-driven UI, server route with streamText - tool-calling: server tool() + inputSchema + stepCountIs, client part.state machine for tool parts - structured-output: streamObject + useObject for typed partial streaming - error-handling-and-abort: stop(), error ref, onError, retry via regenerate Centers on the Vue-specific surface; treats the volatile AI SDK core as shape with 'verify against installed version' pointers. Keys stay server-side as the security boundary. --- skills/vue-ai-apps/SKILL.md | 51 +++++++++ .../references/error-handling-and-abort.md | 87 +++++++++++++++ .../references/streaming-chat-ui.md | 101 ++++++++++++++++++ .../references/structured-output.md | 90 ++++++++++++++++ skills/vue-ai-apps/references/tool-calling.md | 88 +++++++++++++++ 5 files changed, 417 insertions(+) create mode 100644 skills/vue-ai-apps/SKILL.md create mode 100644 skills/vue-ai-apps/references/error-handling-and-abort.md create mode 100644 skills/vue-ai-apps/references/streaming-chat-ui.md create mode 100644 skills/vue-ai-apps/references/structured-output.md create mode 100644 skills/vue-ai-apps/references/tool-calling.md diff --git a/skills/vue-ai-apps/SKILL.md b/skills/vue-ai-apps/SKILL.md new file mode 100644 index 0000000..e55e035 --- /dev/null +++ b/skills/vue-ai-apps/SKILL.md @@ -0,0 +1,51 @@ +--- +name: vue-ai-apps +description: "Building AI/LLM and agent apps with Vue 3 and Nuxt: streaming chat UIs, the Vercel AI SDK (`ai` + `@ai-sdk/vue`), `useChat`, tool calling, structured output, and abort/error handling. Load for AI chatbots, assistant UIs, LLM streaming, or agent frontends in Vue or Nuxt." +version: "1.0.0" +license: MIT +author: github.com/Pythoughts-labs +--- + +# Vue AI Apps Workflow + +Building the **Vue/Nuxt front end** for LLM and agent features with the Vercel AI SDK. This skill covers the Vue-specific integration; the model layer (`streamText`, `tool`, providers) is shown as shape only — follow the AI SDK docs for the current core API, which moves between minor versions. + +Assumes the foundations in `vue-best-practices` (Composition API, ` + +``` + +**Correct:** +```vue + + + +``` + +## Notes + +- `stop()` aborts the in-flight request; the partial assistant message stays in `messages`. +- `regenerate()` (v5; replaces v4 `reload`) re-runs the last user turn — pair with `clearError()` after a failure. +- `useObject` does not expose `status`; it uses `isLoading` + `error` + `stop()` instead. +- Surface a friendly message to users; keep raw errors, request bodies, and keys out of client logs. + +## Reference +- [AI SDK — useChat reference](https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-chat) +- [AI SDK — Chatbot: status & stop](https://ai-sdk.dev/docs/ai-sdk-ui/chatbot) diff --git a/skills/vue-ai-apps/references/streaming-chat-ui.md b/skills/vue-ai-apps/references/streaming-chat-ui.md new file mode 100644 index 0000000..82361d2 --- /dev/null +++ b/skills/vue-ai-apps/references/streaming-chat-ui.md @@ -0,0 +1,101 @@ +--- +title: Streaming Chat UI with useChat (AI SDK v5) +impact: HIGH +impactDescription: Rendering message.content or managing input inside the hook is the v4 API and silently breaks in v5 +type: capability +tags: [vue3, nuxt, ai-sdk, useChat, streaming, llm, chat] +--- + +# Streaming Chat UI with useChat (AI SDK v5) + +**Impact: HIGH** - In AI SDK v5 the Vue `useChat` composable no longer manages the input field, exposes `status` instead of `isLoading`, and returns messages as `UIMessage[]` built from typed `parts`. Code written for v4 (`input`, `handleSubmit`, `message.content`, `isLoading`) compiles but renders nothing useful. + +The provider API key must stay on the server. The browser calls your own route; your route calls the model. + +## Task Checklist + +- [ ] Keep the provider key and `streamText` call in a server route, never in the component +- [ ] Import `useChat` from `@ai-sdk/vue` (not `@ai-sdk/react`) +- [ ] Own a local `input` ref; send with `sendMessage({ text })` +- [ ] Render `message.parts`, branching on `part.type` +- [ ] Disable the input while `status` is `submitted` or `streaming` + +**Incorrect - v4-style usage, broken in v5:** +```vue + + + +``` + +**Correct - server route (Nuxt/Nitro), `server/api/chat.ts`:** +```ts +import { streamText, convertToModelMessages, type UIMessage } from 'ai' +import { openai } from '@ai-sdk/openai' // key read from env on the server + +export default defineEventHandler(async (event) => { + const { messages } = await readBody<{ messages: UIMessage[] }>(event) + + const result = streamText({ + model: openai('gpt-4o'), + messages: convertToModelMessages(messages), + }) + + // Standard v5 streaming response. (The Nuxt template also shows a + // gateway-wrapped form with toUIMessageStream — both are valid.) + return result.toUIMessageStreamResponse() +}) +``` + +**Correct - component, `pages/index.vue`:** +```vue + + + +``` + +## Notes + +- `useChat` returns Vue refs (`messages.value`, `status.value`); templates unwrap them automatically. +- `status` values: `'submitted'` (sent, awaiting first token) → `'streaming'` (receiving) → `'ready'` (done) → `'error'`. Treat `submitted` + `streaming` as busy. +- `sendMessage` also accepts files/attachments and per-call options; `text` is the common case. +- The core `streamText` signature can change between AI SDK minors — verify against the installed version rather than copying blindly. + +## Reference +- [AI SDK — Nuxt quickstart](https://ai-sdk.dev/docs/getting-started/nuxt) +- [AI SDK — useChat reference](https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-chat) diff --git a/skills/vue-ai-apps/references/structured-output.md b/skills/vue-ai-apps/references/structured-output.md new file mode 100644 index 0000000..6be386b --- /dev/null +++ b/skills/vue-ai-apps/references/structured-output.md @@ -0,0 +1,90 @@ +--- +title: Streaming Structured Output in Vue (AI SDK v5) +impact: MEDIUM +impactDescription: Use streamObject + useObject for typed data; the object streams in partial, so the UI must tolerate undefined fields +type: capability +tags: [vue3, nuxt, ai-sdk, streamObject, useObject, structured-output, zod] +--- + +# Streaming Structured Output in Vue (AI SDK v5) + +**Impact: MEDIUM** - When the model should return a typed object (a recipe, a form, an extraction) rather than chat, use `streamObject` on the server and the `useObject` composable on the client. The object arrives **incrementally as a deep-partial**, so every field can be `undefined` mid-stream. Templates must guard with optional chaining and `v-if`, or they throw while streaming. + +`useObject` uses its own surface (`object`, `submit`, `isLoading`) — it does **not** share `useChat`'s `status`/`sendMessage` API. + +## Task Checklist + +- [ ] Use `streamObject` server-side with a `zod` `schema` +- [ ] Return `result.toTextStreamResponse()` +- [ ] Drive the UI from `useObject`'s `object`, `submit`, `isLoading` +- [ ] Treat every field of `object` as possibly `undefined` while streaming +- [ ] Verify the exact `useObject` export name in your installed `@ai-sdk/vue` + +**Incorrect - assuming the object is complete:** +```vue + +
  • {{ step }}
    • +``` + +**Correct - server route, `server/api/recipe.ts`:** +```ts +import { streamObject } from 'ai' +import { openai } from '@ai-sdk/openai' +import { z } from 'zod' + +export const recipeSchema = z.object({ + recipe: z.object({ + name: z.string(), + steps: z.array(z.string()), + }), +}) + +export default defineEventHandler(async (event) => { + const { dish } = await readBody<{ dish: string }>(event) + + const result = streamObject({ + model: openai('gpt-4o'), + schema: recipeSchema, + prompt: `Generate a recipe for ${dish}`, + }) + + return result.toTextStreamResponse() +}) +``` + +**Correct - component:** +```vue + + + +``` + +## Notes + +- `useObject` is experimental and available for React, Svelte, and **Vue**. The reference docs page only prints the React import, so confirm the Vue export name (`useObject` vs `experimental_useObject`) in `node_modules/@ai-sdk/vue` for your version. +- `object` is typed as `DeepPartial` — TypeScript already forces the optional-chaining discipline above. +- For free-form text streaming use `useChat` (or `useCompletion`); reach for `useObject` only when you need a validated shape. + +## Reference +- [AI SDK — useObject reference](https://ai-sdk.dev/docs/reference/ai-sdk-ui/use-object) +- [AI SDK — streamObject](https://ai-sdk.dev/docs/reference/ai-sdk-core/stream-object) diff --git a/skills/vue-ai-apps/references/tool-calling.md b/skills/vue-ai-apps/references/tool-calling.md new file mode 100644 index 0000000..96b786e --- /dev/null +++ b/skills/vue-ai-apps/references/tool-calling.md @@ -0,0 +1,88 @@ +--- +title: Tool Calling in Vue Chat UIs (AI SDK v5) +impact: HIGH +impactDescription: Tool calls arrive as typed message parts with a state machine; ignoring the state renders blank or stale UI +type: capability +tags: [vue3, nuxt, ai-sdk, tools, function-calling, agents, useChat] +--- + +# Tool Calling in Vue Chat UIs (AI SDK v5) + +**Impact: HIGH** - When the model calls a tool, the result surfaces in the chat as a `parts` entry of type `tool-` with a `state` field. The Vue UI must branch on `part.state` (`input-streaming` → `input-available` → `output-available` / `output-error`) to show progress and results. Rendering the part without checking `state` shows nothing while the tool runs, then throws when accessing `part.output` too early. + +Multi-step agent loops (call tool → feed result back → answer) require `stopWhen` on the server; without it the model stops after the first tool call. + +## Task Checklist + +- [ ] Define tools server-side with `tool({ description, inputSchema: z.object(...), execute })` +- [ ] Use `inputSchema` (v5), not `parameters` (v4) +- [ ] Allow multiple steps with `stopWhen: stepCountIs(n)` for agent behavior +- [ ] In the template, branch tool parts on `part.state` +- [ ] Access `part.input` / `part.output` only in the matching state + +**Incorrect - v4 keys + no state handling:** +```ts +// ❌ `parameters` and `maxSteps` are v4; renamed in v5 +tool({ parameters: z.object({ city: z.string() }), execute }) +streamText({ /* ... */ tools, maxSteps: 5 }) +``` +```vue + +
    {{ part.output.tempC }}
    +``` + +**Correct - server route, `server/api/chat.ts`:** +```ts +import { streamText, tool, convertToModelMessages, stepCountIs, type UIMessage } from 'ai' +import { openai } from '@ai-sdk/openai' +import { z } from 'zod' + +export default defineEventHandler(async (event) => { + const { messages } = await readBody<{ messages: UIMessage[] }>(event) + + const result = streamText({ + model: openai('gpt-4o'), + messages: convertToModelMessages(messages), + stopWhen: stepCountIs(5), // let the model use a tool then answer + tools: { + getWeather: tool({ + description: 'Get the current weather for a city', + inputSchema: z.object({ city: z.string() }), + execute: async ({ city }) => ({ city, tempC: 21 }), + }), + }, + }) + + return result.toUIMessageStreamResponse() +}) +``` + +**Correct - rendering the tool part:** +```vue + +``` + +## Notes + +- Part type follows the pattern `tool-${toolName}`; dynamically registered tools use type `dynamic-tool` with `part.toolName`. +- Useful accessors: `part.input`, `part.output`, `part.toolCallId`, `part.errorText`. +- **Client-side tools** (a `tool()` with no `execute`) are fulfilled from the UI by calling `addToolResult({ tool, toolCallId, output })` returned by `useChat`. +- `stepCountIs` / `stopWhen` and the `tool()` signature are core AI SDK API — confirm against the installed version. + +## Reference +- [AI SDK — Chatbot tool usage](https://ai-sdk.dev/docs/ai-sdk-ui/chatbot-tool-usage) +- [AI SDK — tool() / stopWhen](https://ai-sdk.dev/docs/reference/ai-sdk-core/step-count-is) From c3466d3795854ba88d3326a5196007bd6b3ce731 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 12:24:20 -0400 Subject: [PATCH 02/31] docs(registry): register vue-ai-apps skill Add vue-ai-apps to the marketplace manifest and README skill table, and note it in the bundle description. The MCP server discovers skills from the filesystem, so no server change is required. --- .claude-plugin/marketplace.json | 7 ++++++- README.md | 1 + 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 13b711d..5a362a6 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -9,7 +9,7 @@ { "name": "vue-skills-bundle", "source": "./", - "description": "Install all Vue.js skills at once, including best practices, testing, router, Pinia, JSX, Options API, and debug guides." + "description": "Install all Vue.js skills at once, including best practices, testing, router, Pinia, JSX, Options API, debug guides, and AI apps." }, { "name": "vue-best-practices", @@ -45,6 +45,11 @@ "name": "vue-testing-best-practices", "source": "./skills/vue-testing-best-practices", "description": "Use for Vue.js testing. Covers Vitest, Vue Test Utils, component testing, mocking, testing patterns, and Playwright for E2E testing." + }, + { + "name": "vue-ai-apps", + "source": "./skills/vue-ai-apps", + "description": "Building AI/LLM and agent apps with Vue 3 and Nuxt: streaming chat UIs, the Vercel AI SDK, useChat, tool calling, structured output, and abort/error handling." } ] } diff --git a/README.md b/README.md index 1eb772c..82848d5 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,7 @@ claude mcp add vue-skills -- node /mcp/index.mjs | **vue-testing-best-practices** | Component or E2E tests | Vitest, Vue Test Utils, Playwright | | **vue-jsx-best-practices** | JSX in Vue | Syntax differences from React JSX, plugin config | | **vue-debug-guides** | Debugging Vue 3 | Runtime errors, warnings, async failures, hydration issues | +| **vue-ai-apps** | AI/LLM & agent apps in Vue/Nuxt | Streaming chat UIs, Vercel AI SDK, `useChat`, tool calling, structured output | ## How it works From c053169ffe4b39c078f7b8be1a69afc573e1d44d Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 12:24:20 -0400 Subject: [PATCH 03/31] feat(vue-best-practices): cover Vue 3.5 APIs and 3.6 Vapor mode - reactive-props-destructure: 3.5 stable destructure defaults (replaces withDefaults) and the getter-boundary reactivity gotcha - vue-3-5-helpers: useId, onWatcherCleanup, , lazy hydration - vapor-mode: 3.6 beta, flagged experimental and opt-in only - SKILL.md: link new references under foundations (3.5) and performance (Vapor); bump version 18.1.0 -> 18.2.0 useTemplateRef and watch async cleanup are already covered in vue-debug-guides and intentionally not duplicated. --- skills/vue-best-practices/SKILL.md | 9 +- .../references/reactive-props-destructure.md | 80 ++++++++++++++++ .../references/vapor-mode.md | 65 +++++++++++++ .../references/vue-3-5-helpers.md | 91 +++++++++++++++++++ 4 files changed, 244 insertions(+), 1 deletion(-) create mode 100644 skills/vue-best-practices/references/reactive-props-destructure.md create mode 100644 skills/vue-best-practices/references/vapor-mode.md create mode 100644 skills/vue-best-practices/references/vue-3-5-helpers.md diff --git a/skills/vue-best-practices/SKILL.md b/skills/vue-best-practices/SKILL.md index cfcb8e5..13a2aa7 100644 --- a/skills/vue-best-practices/SKILL.md +++ b/skills/vue-best-practices/SKILL.md @@ -4,7 +4,7 @@ description: MUST be used for Vue.js tasks. Strongly recommends Composition API license: MIT metadata: author: github.com/Pythoughts-labs - version: "18.1.0" + version: "18.2.0" --- # Vue Best Practices Workflow @@ -102,6 +102,11 @@ Entry/root and route view rule: - Keep composable APIs small, typed, and predictable. - Separate feature logic from presentational components. +### Vue 3.5+ APIs (apply on Vue 3.5 or newer) + +- Prefer destructure defaults over `withDefaults()`; mind the getter-boundary rule -> [reactive-props-destructure](references/reactive-props-destructure.md) +- Use 3.5 built-ins before hand-rolling: `useId`, `onWatcherCleanup`, ``, lazy hydration -> [vue-3-5-helpers](references/vue-3-5-helpers.md) + ## 3) Consider optional features only when requirements call for them ### 3.1 Standard optional features @@ -138,6 +143,8 @@ Performance work is a post-functionality pass. Do not optimize before core behav - Over-abstraction in hot list paths -> [perf-avoid-component-abstraction-in-lists](references/perf-avoid-component-abstraction-in-lists.md) - Expensive updates triggered too often -> [updated-hook-performance](references/updated-hook-performance.md) +Experimental: a measured rendering hotspot may justify Vapor Mode (Vue 3.6, opt-in, unstable) -> [vapor-mode](references/vapor-mode.md). Do not enable by default. + ## 5) Final self-check before finishing - Core behavior works and matches requirements. diff --git a/skills/vue-best-practices/references/reactive-props-destructure.md b/skills/vue-best-practices/references/reactive-props-destructure.md new file mode 100644 index 0000000..ba03624 --- /dev/null +++ b/skills/vue-best-practices/references/reactive-props-destructure.md @@ -0,0 +1,80 @@ +--- +title: Reactive Props Destructure (Vue 3.5+) +impact: HIGH +impactDescription: Destructured props lose reactivity when passed across a function boundary, silently breaking watchers and composables +type: best-practice +tags: [vue3, vue35, props, defineProps, reactivity, composition-api, withDefaults] +--- + +# Reactive Props Destructure (Vue 3.5+) + +**Impact: HIGH** - Since Vue 3.5 (stable, on by default), destructuring `defineProps()` keeps reactivity and lets you declare defaults with plain JS syntax — replacing `withDefaults()`. The compiler rewrites each destructured access back to `props.x`. The catch: a destructured prop is only reactive when accessed *inside* a reactive scope (template, `computed`, `watchEffect`). The moment you pass the bare variable **into a function** — `watch(count, …)`, a composable, `toRef(count)` — you pass a snapshot value, and reactivity is lost. + +Vue's compiler warns when it detects a destructured prop passed directly into a function call, but the fix must be applied for the code to work. + +## Task Checklist + +- [ ] Use destructure defaults instead of `withDefaults()` in Vue 3.5+ +- [ ] Access destructured props directly in templates, `computed`, and `watchEffect` +- [ ] When passing a prop across a function boundary, wrap it in a getter `() => prop` +- [ ] Watch a destructured prop with `watch(() => prop, …)`, never `watch(prop, …)` +- [ ] In composables, accept `() => T` and read it with `toValue()` + +**Incorrect - reactivity lost at the function boundary:** +```vue + +``` + +**Correct - wrap in a getter when crossing a function boundary:** +```vue + + + +``` + +**Correct - composable consuming a getter:** +```ts +import { toValue, watchEffect, type MaybeRefOrGetter } from 'vue' + +export function useFetch(id: MaybeRefOrGetter) { + watchEffect(() => { + const current = toValue(id) // normalizes getter | ref | plain value + // fetch with `current`… + }) +} +``` + +## Notes + +- This is the recommended way to set prop defaults in 3.5+. If a component still uses `withDefaults()`, its caveats (e.g. mutable factory defaults) continue to apply — prefer migrating to destructure defaults for new code. +- The getter rule applies **only** across function boundaries. Bare `count` in a template or `computed` body is fully reactive; do not over-wrap. + +## Reference +- [Vue.js — Reactive Props Destructure](https://vuejs.org/guide/components/props.html#reactive-props-destructure) +- [Vue 3.5 release notes](https://blog.vuejs.org/posts/vue-3-5) diff --git a/skills/vue-best-practices/references/vapor-mode.md b/skills/vue-best-practices/references/vapor-mode.md new file mode 100644 index 0000000..ddaf674 --- /dev/null +++ b/skills/vue-best-practices/references/vapor-mode.md @@ -0,0 +1,65 @@ +--- +title: Vapor Mode (Vue 3.6, Experimental) +impact: LOW +impactDescription: Vapor is opt-in and unstable; adopting it app-wide or expecting full ecosystem support is premature +type: capability +tags: [vue3, vue36, vapor, performance, experimental, compiler] +--- + +# Vapor Mode (Vue 3.6, Experimental) + +**Impact: LOW (forward-looking)** - Vapor Mode is an alternative compilation strategy in Vue 3.6 that drops the Virtual DOM and compiles components to direct, fine-grained DOM updates (Solid-like), for smaller bundles and less memory. As of the Vue 3.6 beta it is **feature-complete but unstable** — opt in per component, do not bet a production app on it, and do not present it as the default. + +This is an experimental, version-gated feature. Treat everything below as subject to change. + +## Task Checklist + +- [ ] Do **not** enable Vapor by default; use the standard VDOM build unless a measured hotspot justifies it +- [ ] Opt in **per component** with ` + + +``` + +**Mixing with an existing VDOM app — interop plugin required:** +```ts +import { createApp, vaporInteropPlugin } from 'vue' +import App from './App.vue' + +createApp(App).use(vaporInteropPlugin).mount('#app') +// Vapor and VDOM components can then nest in each other. +``` + +**Fully-Vapor app (smallest baseline, no VDOM runtime):** +```ts +import { createVaporApp } from 'vue' +import App from './App.vue' + +createVaporApp(App).mount('#app') +``` + +## Constraints & caveats (3.6 beta) + +- ` + + +``` + +Do not call `useId()` inside a `computed()`; declare it at setup top level. + +### `onWatcherCleanup()` — cancel in-flight work + +Registers cleanup that runs before the watcher re-fires or on unmount. Must be called **synchronously** within the effect (before any `await`). + +```ts +import { watch, onWatcherCleanup } from 'vue' + +watch(id, (newId) => { + const controller = new AbortController() + fetch(`/api/items/${newId}`, { signal: controller.signal }) + onWatcherCleanup(() => controller.abort()) // abort the stale request +}) +``` + +### `` — target rendered later + +Without `defer`, `` needs its target to already exist. `defer` delays mounting until after the current render tick, so the target can appear later in the same template. + +```vue + +``` + +### Lazy hydration for async components (SSR) + +Defer hydrating heavy, below-the-fold components until they are needed, cutting time-to-interactive. Strategies are imported from `vue`. + +```ts +import { + defineAsyncComponent, + hydrateOnVisible, + hydrateOnIdle, + hydrateOnInteraction, + hydrateOnMediaQuery, +} from 'vue' + +const HeavyChart = defineAsyncComponent({ + loader: () => import('./HeavyChart.vue'), + hydrate: hydrateOnVisible({ rootMargin: '100px' }), // IntersectionObserver + // hydrateOnIdle(timeout?) → requestIdleCallback + // hydrateOnInteraction('click') → hydrates on first interaction + // hydrateOnMediaQuery('(min-width: 768px)')→ hydrates when the query matches +}) +``` + +## Reference +- [Vue.js — Composition API helpers (useId, onWatcherCleanup)](https://vuejs.org/api/composition-api-helpers.html) +- [Vue.js — Teleport (defer)](https://vuejs.org/guide/built-ins/teleport.html) +- [Vue.js — Lazy hydration](https://vuejs.org/guide/components/async.html#lazy-hydration) From e2403091365f05173de048d5d7ea2b926a8e2b13 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 12:24:20 -0400 Subject: [PATCH 04/31] chore(tasks): track eval follow-up for new references Record the deferred eval-driven validation (3 evals x 4 tiers x 3 models per new reference) and the AI SDK items to confirm against an installed version. --- tasks/todo.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 tasks/todo.md diff --git a/tasks/todo.md b/tasks/todo.md new file mode 100644 index 0000000..634d2a8 --- /dev/null +++ b/tasks/todo.md @@ -0,0 +1,28 @@ +# Task: Vue 3.5/3.6 refresh + new vue-ai-apps skill + +## Done + +- New skill `vue-ai-apps` (Vercel AI SDK v5 in Vue/Nuxt): + - `SKILL.md` + references: `streaming-chat-ui`, `tool-calling`, `structured-output`, `error-handling-and-abort` + - Registered in `.claude-plugin/marketplace.json` and `README.md`; MCP server auto-discovers it (no code change) +- `vue-best-practices` refresh (version 18.1.0 → 18.2.0): + - `references/reactive-props-destructure.md` (3.5 stable; replaces `withDefaults`, getter-boundary gotcha) + - `references/vue-3-5-helpers.md` (3.5 stable; `useId`, `onWatcherCleanup`, ``, lazy hydration) + - `references/vapor-mode.md` (3.6 beta, flagged experimental, opt-in only) + - SKILL.md pointers added under section 2 (3.5 APIs) and section 4 (Vapor under perf) + +## Out of scope — follow-up (eval-driven validation per AGENTS.md) + +Each new reference file needs 3 evals × 4 tiers × 3 models. Not done this pass (user chose +"land docs now, track evals later"). Files needing eval suites: + +- vue-ai-apps: streaming-chat-ui, tool-calling, structured-output, error-handling-and-abort +- vue-best-practices: reactive-props-destructure, vue-3-5-helpers +- vapor-mode: experimental — eval only once 3.6 is stable + +## Verify-before-trust notes (flagged in the content, confirm against installed SDK) + +- `@ai-sdk/vue` `useObject` export name (`useObject` vs `experimental_useObject`) — page only showed React import. +- Server return: used `result.toUIMessageStreamResponse()` (standard); Nuxt template also shows a + `createUIMessageStreamResponse`/`toUIMessageStream` gateway-wrapped form. Both valid v5. +- AI SDK core (`streamText`, `tool`, `stepCountIs`) is version-volatile — references say "verify against installed version". From b040869739c123111049d75c1aea689daa6e63c0 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 12:38:44 -0400 Subject: [PATCH 05/31] test(evals): scaffold eval stubs for new references Add eval specs under evals/suites/skills/ for the 6 stable references (vue-ai-apps x4, vue-best-practices x2), 3 scenarios each: - eval.json: query + expected_behavior - eval.ts: starter content-pattern assertions - src stub: clean empty component (no hints, per AGENTS.md) Not yet runnable: the pnpm eval runner and per-scenario build files are not in this repo state (see evals/README.md). vapor-mode is skipped until Vue 3.6 is stable. --- evals/README.md | 33 +++++++++++++++++++ .../scenario-1/eval.json | 11 +++++++ .../scenario-1/eval.ts | 15 +++++++++ .../scenario-1/src/components/ChatBox.vue | 3 ++ .../scenario-2/eval.json | 11 +++++++ .../scenario-2/eval.ts | 15 +++++++++ .../scenario-2/src/components/ChatBox.vue | 3 ++ .../scenario-3/eval.json | 11 +++++++ .../scenario-3/eval.ts | 15 +++++++++ .../scenario-3/src/components/ChatBox.vue | 3 ++ .../streaming-chat-ui/scenario-1/eval.json | 11 +++++++ .../streaming-chat-ui/scenario-1/eval.ts | 22 +++++++++++++ .../scenario-1/src/components/ChatBox.vue | 3 ++ .../streaming-chat-ui/scenario-2/eval.json | 11 +++++++ .../streaming-chat-ui/scenario-2/eval.ts | 22 +++++++++++++ .../scenario-2/src/components/ChatBox.vue | 3 ++ .../streaming-chat-ui/scenario-3/eval.json | 11 +++++++ .../streaming-chat-ui/scenario-3/eval.ts | 22 +++++++++++++ .../scenario-3/src/components/ChatBox.vue | 3 ++ .../structured-output/scenario-1/eval.json | 10 ++++++ .../structured-output/scenario-1/eval.ts | 14 ++++++++ .../src/components/RecipeGenerator.vue | 3 ++ .../structured-output/scenario-2/eval.json | 10 ++++++ .../structured-output/scenario-2/eval.ts | 14 ++++++++ .../src/components/RecipeGenerator.vue | 3 ++ .../structured-output/scenario-3/eval.json | 10 ++++++ .../structured-output/scenario-3/eval.ts | 14 ++++++++ .../src/components/RecipeGenerator.vue | 3 ++ .../tool-calling/scenario-1/eval.json | 11 +++++++ .../tool-calling/scenario-1/eval.ts | 14 ++++++++ .../scenario-1/src/components/ChatBox.vue | 3 ++ .../tool-calling/scenario-2/eval.json | 11 +++++++ .../tool-calling/scenario-2/eval.ts | 14 ++++++++ .../scenario-2/src/components/ChatBox.vue | 3 ++ .../tool-calling/scenario-3/eval.json | 11 +++++++ .../tool-calling/scenario-3/eval.ts | 14 ++++++++ .../scenario-3/src/components/ChatBox.vue | 3 ++ .../scenario-1/eval.json | 11 +++++++ .../scenario-1/eval.ts | 16 +++++++++ .../scenario-1/src/components/UserCard.vue | 3 ++ .../scenario-2/eval.json | 11 +++++++ .../scenario-2/eval.ts | 16 +++++++++ .../scenario-2/src/components/UserCard.vue | 3 ++ .../scenario-3/eval.json | 11 +++++++ .../scenario-3/eval.ts | 16 +++++++++ .../scenario-3/src/components/UserCard.vue | 3 ++ .../vue-3-5-helpers/scenario-1/eval.json | 10 ++++++ .../vue-3-5-helpers/scenario-1/eval.ts | 16 +++++++++ .../scenario-1/src/components/SignupField.vue | 3 ++ .../vue-3-5-helpers/scenario-2/eval.json | 10 ++++++ .../vue-3-5-helpers/scenario-2/eval.ts | 16 +++++++++ .../scenario-2/src/components/SignupField.vue | 3 ++ .../vue-3-5-helpers/scenario-3/eval.json | 10 ++++++ .../vue-3-5-helpers/scenario-3/eval.ts | 16 +++++++++ .../scenario-3/src/components/SignupField.vue | 3 ++ tasks/todo.md | 17 ++++++---- 56 files changed, 581 insertions(+), 6 deletions(-) create mode 100644 evals/README.md create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/components/RecipeGenerator.vue create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/components/RecipeGenerator.vue create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/components/RecipeGenerator.vue create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/components/ChatBox.vue create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/components/UserCard.vue create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/components/UserCard.vue create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/components/UserCard.vue create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/components/SignupField.vue create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/components/SignupField.vue create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/components/SignupField.vue diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..1ddecd9 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,33 @@ +# Evals + +Eval suites for the skills, following the structure in [`AGENTS.md`](../AGENTS.md#eval-structure): +`evals/suites/skills///scenario-{1,2,3}/`. + +## Status: stubs + +These are **scaffolded specs, not yet runnable**. Each scenario currently contains: + +- `eval.json` — query + `expected_behavior` (the spec) +- `eval.ts` — Vitest content-pattern assertions (a focused starting set, expand per AGENTS.md) +- `src/components/*.vue` — empty input stub (clean, no hints, per AGENTS.md) + +Before `pnpm eval` can run them, this repo still needs: + +1. The **eval runner** (`pnpm eval`) and its package — not present in this commit. +2. Per-scenario **build files** (`package.json`, `vite.config.ts`, `tsconfig.json`, + `index.html`, `eslint.config.js`, `src/main.ts`) so each scenario is a self-contained, + buildable Vue project. The runner copies the suite, installs, builds, then runs `eval.ts`. + +## Covered references + +| Skill | Reference | Scenarios | +|-------|-----------|-----------| +| vue-ai-apps | streaming-chat-ui | 3 | +| vue-ai-apps | tool-calling | 3 | +| vue-ai-apps | structured-output | 3 | +| vue-ai-apps | error-handling-and-abort | 3 | +| vue-best-practices | reactive-props-destructure | 3 | +| vue-best-practices | vue-3-5-helpers | 3 | + +`vue-best-practices/vapor-mode` is intentionally **not** scaffolded — it documents an +experimental Vue 3.6 feature; add evals once 3.6 (and Vapor) are stable. diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json new file mode 100644 index 0000000..ee7c30d --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Shows a Stop control that calls stop() while streaming", + "Surfaces the error ref in the template", + "Offers recovery via regenerate() (optionally clearError())", + "Disables send unless status is ready" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts new file mode 100644 index 0000000..f520192 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts @@ -0,0 +1,15 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("wires abort and error recovery", () => { + expect(src).toMatch(/\bstop\s*\(/); + expect(src).toMatch(/\berror\b/); + expect(src).toMatch(/regenerate\s*\(/); +}); + +test("disables send via status", () => { + expect(src).toMatch(/status/); +}); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json new file mode 100644 index 0000000..799d169 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "Add a stop button and error handling to a Vue AI chat component built with @ai-sdk/vue.", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Shows a Stop control that calls stop() while streaming", + "Surfaces the error ref in the template", + "Offers recovery via regenerate() (optionally clearError())", + "Disables send unless status is ready" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts new file mode 100644 index 0000000..f520192 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts @@ -0,0 +1,15 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("wires abort and error recovery", () => { + expect(src).toMatch(/\bstop\s*\(/); + expect(src).toMatch(/\berror\b/); + expect(src).toMatch(/regenerate\s*\(/); +}); + +test("disables send via status", () => { + expect(src).toMatch(/status/); +}); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json new file mode 100644 index 0000000..67dee65 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "Make a Vue LLM chat resilient: let the user cancel a streaming response and retry after a failure.", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Shows a Stop control that calls stop() while streaming", + "Surfaces the error ref in the template", + "Offers recovery via regenerate() (optionally clearError())", + "Disables send unless status is ready" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts new file mode 100644 index 0000000..f520192 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts @@ -0,0 +1,15 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("wires abort and error recovery", () => { + expect(src).toMatch(/\bstop\s*\(/); + expect(src).toMatch(/\berror\b/); + expect(src).toMatch(/regenerate\s*\(/); +}); + +test("disables send via status", () => { + expect(src).toMatch(/status/); +}); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json new file mode 100644 index 0000000..2990cf5 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Imports useChat from '@ai-sdk/vue' (not @ai-sdk/react)", + "Renders message.parts, not message.content", + "Owns a local input ref and calls sendMessage({ text }) (no hook-managed input/handleSubmit)", + "Drives disabled/busy UI from status, not isLoading" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.ts new file mode 100644 index 0000000..02719f3 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.ts @@ -0,0 +1,22 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("uses @ai-sdk/vue useChat", () => { + expect(src).toMatch(/from\s+['"]@ai-sdk\/vue['"]/); + expect(src).toMatch(/useChat\s*\(/); +}); + +test("renders message parts, not content string", () => { + expect(src).toMatch(/\.parts/); + // v4-style content rendering should be absent + expect(src).not.toMatch(/\bmessage\.content\b|\bm\.content\b/); +}); + +test("v5 surface: sendMessage + status, no hook-managed input/isLoading", () => { + expect(src).toMatch(/sendMessage\s*\(/); + expect(src).toMatch(/\bstatus\b/); + expect(src).not.toMatch(/\bisLoading\b/); +}); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json new file mode 100644 index 0000000..62f07a0 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "Build a Vue chat component that streams responses from an LLM using the Vercel AI SDK in a Nuxt app.", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Imports useChat from '@ai-sdk/vue' (not @ai-sdk/react)", + "Renders message.parts, not message.content", + "Owns a local input ref and calls sendMessage({ text }) (no hook-managed input/handleSubmit)", + "Drives disabled/busy UI from status, not isLoading" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.ts new file mode 100644 index 0000000..02719f3 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.ts @@ -0,0 +1,22 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("uses @ai-sdk/vue useChat", () => { + expect(src).toMatch(/from\s+['"]@ai-sdk\/vue['"]/); + expect(src).toMatch(/useChat\s*\(/); +}); + +test("renders message parts, not content string", () => { + expect(src).toMatch(/\.parts/); + // v4-style content rendering should be absent + expect(src).not.toMatch(/\bmessage\.content\b|\bm\.content\b/); +}); + +test("v5 surface: sendMessage + status, no hook-managed input/isLoading", () => { + expect(src).toMatch(/sendMessage\s*\(/); + expect(src).toMatch(/\bstatus\b/); + expect(src).not.toMatch(/\bisLoading\b/); +}); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json new file mode 100644 index 0000000..b40e6bc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "Create a streaming AI chat box in Vue with an input field and a list of messages from an assistant.", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Imports useChat from '@ai-sdk/vue' (not @ai-sdk/react)", + "Renders message.parts, not message.content", + "Owns a local input ref and calls sendMessage({ text }) (no hook-managed input/handleSubmit)", + "Drives disabled/busy UI from status, not isLoading" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.ts new file mode 100644 index 0000000..02719f3 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.ts @@ -0,0 +1,22 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("uses @ai-sdk/vue useChat", () => { + expect(src).toMatch(/from\s+['"]@ai-sdk\/vue['"]/); + expect(src).toMatch(/useChat\s*\(/); +}); + +test("renders message parts, not content string", () => { + expect(src).toMatch(/\.parts/); + // v4-style content rendering should be absent + expect(src).not.toMatch(/\bmessage\.content\b|\bm\.content\b/); +}); + +test("v5 surface: sendMessage + status, no hook-managed input/isLoading", () => { + expect(src).toMatch(/sendMessage\s*\(/); + expect(src).toMatch(/\bstatus\b/); + expect(src).not.toMatch(/\bisLoading\b/); +}); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json new file mode 100644 index 0000000..1ede431 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json @@ -0,0 +1,10 @@ +{ + "skills": ["vue-ai-apps"], + "query": "", + "files": ["src/components/RecipeGenerator.vue"], + "expected_behavior": [ + "Uses useObject from @ai-sdk/vue for structured streaming", + "Guards partial fields with optional chaining (object?.x) while streaming", + "Drives UI from isLoading + submit (useObject surface, not status/sendMessage)" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.ts new file mode 100644 index 0000000..ac2a5bd --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.ts @@ -0,0 +1,14 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/RecipeGenerator.vue"), "utf-8"); + +test("uses useObject for structured streaming", () => { + expect(src).toMatch(/useObject/); + expect(src).toMatch(/from\s+['"]@ai-sdk\/vue['"]/); +}); + +test("guards partial object fields with optional chaining", () => { + expect(src).toMatch(/object\?\./); +}); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/components/RecipeGenerator.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/components/RecipeGenerator.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/components/RecipeGenerator.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json new file mode 100644 index 0000000..ebbfd7e --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json @@ -0,0 +1,10 @@ +{ + "skills": ["vue-ai-apps"], + "query": "Build a Vue component that streams a typed recipe object from an LLM and renders it as it arrives.", + "files": ["src/components/RecipeGenerator.vue"], + "expected_behavior": [ + "Uses useObject from @ai-sdk/vue for structured streaming", + "Guards partial fields with optional chaining (object?.x) while streaming", + "Drives UI from isLoading + submit (useObject surface, not status/sendMessage)" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.ts new file mode 100644 index 0000000..ac2a5bd --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.ts @@ -0,0 +1,14 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/RecipeGenerator.vue"), "utf-8"); + +test("uses useObject for structured streaming", () => { + expect(src).toMatch(/useObject/); + expect(src).toMatch(/from\s+['"]@ai-sdk\/vue['"]/); +}); + +test("guards partial object fields with optional chaining", () => { + expect(src).toMatch(/object\?\./); +}); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/components/RecipeGenerator.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/components/RecipeGenerator.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/components/RecipeGenerator.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json new file mode 100644 index 0000000..45665e2 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json @@ -0,0 +1,10 @@ +{ + "skills": ["vue-ai-apps"], + "query": "Use the AI SDK to stream a structured object in Vue and display partial fields safely while loading.", + "files": ["src/components/RecipeGenerator.vue"], + "expected_behavior": [ + "Uses useObject from @ai-sdk/vue for structured streaming", + "Guards partial fields with optional chaining (object?.x) while streaming", + "Drives UI from isLoading + submit (useObject surface, not status/sendMessage)" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.ts new file mode 100644 index 0000000..ac2a5bd --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.ts @@ -0,0 +1,14 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/RecipeGenerator.vue"), "utf-8"); + +test("uses useObject for structured streaming", () => { + expect(src).toMatch(/useObject/); + expect(src).toMatch(/from\s+['"]@ai-sdk\/vue['"]/); +}); + +test("guards partial object fields with optional chaining", () => { + expect(src).toMatch(/object\?\./); +}); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/components/RecipeGenerator.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/components/RecipeGenerator.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/components/RecipeGenerator.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json new file mode 100644 index 0000000..068423e --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Branches a tool part on part.state (input-available / output-available / output-error)", + "Reads part.output only in the output-available state", + "Uses the tool- part type (e.g. tool-getWeather) or dynamic-tool", + "Does not access part.output unconditionally" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.ts new file mode 100644 index 0000000..df5d26c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.ts @@ -0,0 +1,14 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("handles the tool part state machine", () => { + expect(src).toMatch(/part\.state|\.state\s*===/); + expect(src).toMatch(/output-available/); +}); + +test("uses a typed tool part type", () => { + expect(src).toMatch(/tool-[a-zA-Z]|dynamic-tool/); +}); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json new file mode 100644 index 0000000..c5f0a14 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "In a Vue chat UI, render an LLM tool call (getWeather) including its loading and result states using the AI SDK v5.", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Branches a tool part on part.state (input-available / output-available / output-error)", + "Reads part.output only in the output-available state", + "Uses the tool- part type (e.g. tool-getWeather) or dynamic-tool", + "Does not access part.output unconditionally" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.ts new file mode 100644 index 0000000..df5d26c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.ts @@ -0,0 +1,14 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("handles the tool part state machine", () => { + expect(src).toMatch(/part\.state|\.state\s*===/); + expect(src).toMatch(/output-available/); +}); + +test("uses a typed tool part type", () => { + expect(src).toMatch(/tool-[a-zA-Z]|dynamic-tool/); +}); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json new file mode 100644 index 0000000..bd13e10 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-ai-apps"], + "query": "Show tool-call progress and output inside a Vue AI chat component for a weather tool.", + "files": ["src/components/ChatBox.vue"], + "expected_behavior": [ + "Branches a tool part on part.state (input-available / output-available / output-error)", + "Reads part.output only in the output-available state", + "Uses the tool- part type (e.g. tool-getWeather) or dynamic-tool", + "Does not access part.output unconditionally" + ] +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.ts new file mode 100644 index 0000000..df5d26c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.ts @@ -0,0 +1,14 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); + +test("handles the tool part state machine", () => { + expect(src).toMatch(/part\.state|\.state\s*===/); + expect(src).toMatch(/output-available/); +}); + +test("uses a typed tool part type", () => { + expect(src).toMatch(/tool-[a-zA-Z]|dynamic-tool/); +}); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/components/ChatBox.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/components/ChatBox.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/components/ChatBox.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json new file mode 100644 index 0000000..a368249 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-best-practices"], + "query": "", + "files": ["src/components/UserCard.vue"], + "expected_behavior": [ + "Uses reactive props destructure with defaults (const { count = 0 } = defineProps...)", + "Does NOT use withDefaults", + "Wraps the prop in a getter when crossing a function boundary (watch(() => id) / composable(() => id))", + "Does not pass a bare destructured prop directly into watch() or a composable" + ] +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts new file mode 100644 index 0000000..78fb167 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts @@ -0,0 +1,16 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/UserCard.vue"), "utf-8"); + +test("uses destructure defaults, not withDefaults", () => { + expect(src).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); + expect(src).not.toMatch(/withDefaults\s*\(/); +}); + +test("preserves reactivity with a getter across the function boundary", () => { + // watch the prop via a getter, not the bare value + expect(src).toMatch(/watch\(\s*\(\)\s*=>/); + expect(src).not.toMatch(/watch\(\s*id\b/); +}); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/components/UserCard.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/components/UserCard.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/components/UserCard.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json new file mode 100644 index 0000000..8a79c84 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-best-practices"], + "query": "Create a Vue 3.5 component that takes an 'id' prop, gives 'count' a default of 0, and refetches a user whenever id changes.", + "files": ["src/components/UserCard.vue"], + "expected_behavior": [ + "Uses reactive props destructure with defaults (const { count = 0 } = defineProps...)", + "Does NOT use withDefaults", + "Wraps the prop in a getter when crossing a function boundary (watch(() => id) / composable(() => id))", + "Does not pass a bare destructured prop directly into watch() or a composable" + ] +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts new file mode 100644 index 0000000..78fb167 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts @@ -0,0 +1,16 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/UserCard.vue"), "utf-8"); + +test("uses destructure defaults, not withDefaults", () => { + expect(src).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); + expect(src).not.toMatch(/withDefaults\s*\(/); +}); + +test("preserves reactivity with a getter across the function boundary", () => { + // watch the prop via a getter, not the bare value + expect(src).toMatch(/watch\(\s*\(\)\s*=>/); + expect(src).not.toMatch(/watch\(\s*id\b/); +}); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/components/UserCard.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/components/UserCard.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/components/UserCard.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json new file mode 100644 index 0000000..45a2156 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json @@ -0,0 +1,11 @@ +{ + "skills": ["vue-best-practices"], + "query": "Build a Vue 3.5 child component with default prop values that watches a prop and passes it to a composable.", + "files": ["src/components/UserCard.vue"], + "expected_behavior": [ + "Uses reactive props destructure with defaults (const { count = 0 } = defineProps...)", + "Does NOT use withDefaults", + "Wraps the prop in a getter when crossing a function boundary (watch(() => id) / composable(() => id))", + "Does not pass a bare destructured prop directly into watch() or a composable" + ] +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts new file mode 100644 index 0000000..78fb167 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts @@ -0,0 +1,16 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/UserCard.vue"), "utf-8"); + +test("uses destructure defaults, not withDefaults", () => { + expect(src).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); + expect(src).not.toMatch(/withDefaults\s*\(/); +}); + +test("preserves reactivity with a getter across the function boundary", () => { + // watch the prop via a getter, not the bare value + expect(src).toMatch(/watch\(\s*\(\)\s*=>/); + expect(src).not.toMatch(/watch\(\s*id\b/); +}); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/components/UserCard.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/components/UserCard.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/components/UserCard.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json new file mode 100644 index 0000000..59442e8 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json @@ -0,0 +1,10 @@ +{ + "skills": ["vue-best-practices"], + "query": "", + "files": ["src/components/SignupField.vue"], + "expected_behavior": [ + "Imports and uses useId() from vue for the field id", + "Binds :for and :id to the same generated id", + "Does not hand-roll id generation (Math.random / global counter / Date.now)" + ] +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.ts new file mode 100644 index 0000000..beae17a --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.ts @@ -0,0 +1,16 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/SignupField.vue"), "utf-8"); + +test("uses useId() for an SSR-safe id", () => { + expect(src).toMatch(/useId\s*\(/); + expect(src).toMatch(/from\s+['"]vue['"]/); +}); + +test("links label and input to the same id, no hand-rolled ids", () => { + expect(src).toMatch(/:for=/); + expect(src).toMatch(/:id=/); + expect(src).not.toMatch(/Math\.random|Date\.now/); +}); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/components/SignupField.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/components/SignupField.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/components/SignupField.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json new file mode 100644 index 0000000..b14fef7 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json @@ -0,0 +1,10 @@ +{ + "skills": ["vue-best-practices"], + "query": "Build an accessible Vue 3.5 form field where the label is correctly linked to the input with a unique, SSR-safe id.", + "files": ["src/components/SignupField.vue"], + "expected_behavior": [ + "Imports and uses useId() from vue for the field id", + "Binds :for and :id to the same generated id", + "Does not hand-roll id generation (Math.random / global counter / Date.now)" + ] +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.ts new file mode 100644 index 0000000..beae17a --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.ts @@ -0,0 +1,16 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/SignupField.vue"), "utf-8"); + +test("uses useId() for an SSR-safe id", () => { + expect(src).toMatch(/useId\s*\(/); + expect(src).toMatch(/from\s+['"]vue['"]/); +}); + +test("links label and input to the same id, no hand-rolled ids", () => { + expect(src).toMatch(/:for=/); + expect(src).toMatch(/:id=/); + expect(src).not.toMatch(/Math\.random|Date\.now/); +}); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/components/SignupField.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/components/SignupField.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/components/SignupField.vue @@ -0,0 +1,3 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json new file mode 100644 index 0000000..1146c02 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json @@ -0,0 +1,10 @@ +{ + "skills": ["vue-best-practices"], + "query": "Create a Vue 3.5 component that renders a label and input wired together with an id that is stable across SSR and client.", + "files": ["src/components/SignupField.vue"], + "expected_behavior": [ + "Imports and uses useId() from vue for the field id", + "Binds :for and :id to the same generated id", + "Does not hand-roll id generation (Math.random / global counter / Date.now)" + ] +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.ts new file mode 100644 index 0000000..beae17a --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.ts @@ -0,0 +1,16 @@ +import { expect, test } from "vitest"; +import { readFileSync } from "fs"; +import { join } from "path"; + +const src = readFileSync(join(process.cwd(), "src/components/SignupField.vue"), "utf-8"); + +test("uses useId() for an SSR-safe id", () => { + expect(src).toMatch(/useId\s*\(/); + expect(src).toMatch(/from\s+['"]vue['"]/); +}); + +test("links label and input to the same id, no hand-rolled ids", () => { + expect(src).toMatch(/:for=/); + expect(src).toMatch(/:id=/); + expect(src).not.toMatch(/Math\.random|Date\.now/); +}); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/components/SignupField.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/components/SignupField.vue new file mode 100644 index 0000000..e519b41 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/components/SignupField.vue @@ -0,0 +1,3 @@ + + + diff --git a/tasks/todo.md b/tasks/todo.md index 634d2a8..41f22e2 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -11,14 +11,19 @@ - `references/vapor-mode.md` (3.6 beta, flagged experimental, opt-in only) - SKILL.md pointers added under section 2 (3.5 APIs) and section 4 (Vapor under perf) -## Out of scope — follow-up (eval-driven validation per AGENTS.md) +## Eval scaffolding — DONE (stubs) -Each new reference file needs 3 evals × 4 tiers × 3 models. Not done this pass (user chose -"land docs now, track evals later"). Files needing eval suites: +Eval specs scaffolded under `evals/suites/skills/...` (6 references × 3 scenarios = 18 +scenarios, 54 files). Each scenario has `eval.json` + starter `eval.ts` + clean input stub. +See `evals/README.md`. `vapor-mode` intentionally skipped (experimental until 3.6 stable). -- vue-ai-apps: streaming-chat-ui, tool-calling, structured-output, error-handling-and-abort -- vue-best-practices: reactive-props-destructure, vue-3-5-helpers -- vapor-mode: experimental — eval only once 3.6 is stable +## Out of scope — remaining follow-up (per AGENTS.md) + +- Eval **runner** (`pnpm eval`) + package: not in this repo commit; needed to execute suites. +- Per-scenario **build boilerplate** (package.json/vite/tsconfig/index.html/main.ts) so each + scenario is a self-contained buildable project the runner can install + build. +- Then run the matrix: 3 evals × 4 tiers × 3 models per reference (haiku/sonnet/opus), record + to `results.json`. Billed LLM runs — user-triggered. ## Verify-before-trust notes (flagged in the content, confirm against installed SDK) From e47a2167d4e87a4a3ef1c4655d661b87168e48c7 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 14:07:31 -0400 Subject: [PATCH 06/31] ci(sync): include evals/ in Sync to Main whitelist Publish eval suites to main alongside skills. Future workflow_dispatch syncs will carry evals/ once this change reaches the default branch. --- .github/workflows/sync-to-main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/sync-to-main.yml b/.github/workflows/sync-to-main.yml index 0206bef..0258ac5 100644 --- a/.github/workflows/sync-to-main.yml +++ b/.github/workflows/sync-to-main.yml @@ -49,7 +49,7 @@ jobs: # ...then re-apply only whitelisted paths from dev git restore --source=origin/dev --staged --worktree -- \ - skills/ .github/ .claude-plugin/ README.md AGENTS.md CLAUDE.md LICENSE + skills/ evals/ .github/ .claude-plugin/ README.md AGENTS.md CLAUDE.md LICENSE git add -A if git diff --staged --quiet; then From e9f57703fcbf6ef6a8ac2fa0cd097fcc6238aa5b Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 14:10:12 -0400 Subject: [PATCH 07/31] release(mcp): @pythoughts/vue-skills-mcp 0.2.0 Bundles the new vue-ai-apps skill and the Vue 3.5/3.6 vue-best-practices references. --- mcp/package.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mcp/package.json b/mcp/package.json index 48ec207..976774b 100644 --- a/mcp/package.json +++ b/mcp/package.json @@ -1,6 +1,6 @@ { "name": "@pythoughts/vue-skills-mcp", - "version": "0.1.0", + "version": "0.2.0", "type": "module", "description": "MCP server exposing the Vue 3 best-practice skills so any MCP coding agent can fetch them automatically on Vue work.", "author": "Mohamed Elkholy (elkaix)", From c160db71ea87834b114bcd8d23b7e394fe22df69 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 14:19:42 -0400 Subject: [PATCH 08/31] feat(evals): runnable eval runner and per-scenario project boilerplate - evals/runner.mjs: pure-Node runner implementing the AGENTS.md flow (copy-to-temp withholding eval.ts/json, tier setup, claude generation, pnpm install + build + vitest, 2-run fail-fast, results.json, skip logic) with flags --all/--force/--model/--tier/--dry/--verbose - root package.json: 'pnpm eval' entrypoint - per-scenario Vue+Vite+Vitest boilerplate (package.json, vite/ts config, index.html, main.ts, App.vue) so each scenario builds standalone - fix generation off-by-one that left every scenario-1 query empty - evals/README: running instructions; verified (--dry) vs budget-gated tiers Verified: --dry (install+build) passes on all scenarios; arg validation. The four LLM tiers require the claude CLI + API budget (user-triggered). --- evals/README.md | 30 ++- evals/runner.mjs | 248 ++++++++++++++++++ .../scenario-1/eval.json | 10 +- .../scenario-1/index.html | 11 + .../scenario-1/package.json | 20 ++ .../scenario-1/src/App.vue | 7 + .../scenario-1/src/main.ts | 4 + .../scenario-1/src/vite-env.d.ts | 1 + .../scenario-1/tsconfig.json | 17 ++ .../scenario-1/vite.config.ts | 4 + .../scenario-2/eval.json | 10 +- .../scenario-2/index.html | 11 + .../scenario-2/package.json | 20 ++ .../scenario-2/src/App.vue | 7 + .../scenario-2/src/main.ts | 4 + .../scenario-2/src/vite-env.d.ts | 1 + .../scenario-2/tsconfig.json | 17 ++ .../scenario-2/vite.config.ts | 4 + .../scenario-3/eval.json | 10 +- .../scenario-3/index.html | 11 + .../scenario-3/package.json | 20 ++ .../scenario-3/src/App.vue | 7 + .../scenario-3/src/main.ts | 4 + .../scenario-3/src/vite-env.d.ts | 1 + .../scenario-3/tsconfig.json | 17 ++ .../scenario-3/vite.config.ts | 4 + .../streaming-chat-ui/scenario-1/eval.json | 10 +- .../streaming-chat-ui/scenario-1/index.html | 11 + .../streaming-chat-ui/scenario-1/package.json | 20 ++ .../streaming-chat-ui/scenario-1/src/App.vue | 7 + .../streaming-chat-ui/scenario-1/src/main.ts | 4 + .../scenario-1/src/vite-env.d.ts | 1 + .../scenario-1/tsconfig.json | 17 ++ .../scenario-1/vite.config.ts | 4 + .../streaming-chat-ui/scenario-2/eval.json | 10 +- .../streaming-chat-ui/scenario-2/index.html | 11 + .../streaming-chat-ui/scenario-2/package.json | 20 ++ .../streaming-chat-ui/scenario-2/src/App.vue | 7 + .../streaming-chat-ui/scenario-2/src/main.ts | 4 + .../scenario-2/src/vite-env.d.ts | 1 + .../scenario-2/tsconfig.json | 17 ++ .../scenario-2/vite.config.ts | 4 + .../streaming-chat-ui/scenario-3/eval.json | 10 +- .../streaming-chat-ui/scenario-3/index.html | 11 + .../streaming-chat-ui/scenario-3/package.json | 20 ++ .../streaming-chat-ui/scenario-3/src/App.vue | 7 + .../streaming-chat-ui/scenario-3/src/main.ts | 4 + .../scenario-3/src/vite-env.d.ts | 1 + .../scenario-3/tsconfig.json | 17 ++ .../scenario-3/vite.config.ts | 4 + .../structured-output/scenario-1/eval.json | 10 +- .../structured-output/scenario-1/index.html | 11 + .../structured-output/scenario-1/package.json | 20 ++ .../structured-output/scenario-1/src/App.vue | 7 + .../structured-output/scenario-1/src/main.ts | 4 + .../scenario-1/src/vite-env.d.ts | 1 + .../scenario-1/tsconfig.json | 17 ++ .../scenario-1/vite.config.ts | 4 + .../structured-output/scenario-2/eval.json | 10 +- .../structured-output/scenario-2/index.html | 11 + .../structured-output/scenario-2/package.json | 20 ++ .../structured-output/scenario-2/src/App.vue | 7 + .../structured-output/scenario-2/src/main.ts | 4 + .../scenario-2/src/vite-env.d.ts | 1 + .../scenario-2/tsconfig.json | 17 ++ .../scenario-2/vite.config.ts | 4 + .../structured-output/scenario-3/eval.json | 10 +- .../structured-output/scenario-3/index.html | 11 + .../structured-output/scenario-3/package.json | 20 ++ .../structured-output/scenario-3/src/App.vue | 7 + .../structured-output/scenario-3/src/main.ts | 4 + .../scenario-3/src/vite-env.d.ts | 1 + .../scenario-3/tsconfig.json | 17 ++ .../scenario-3/vite.config.ts | 4 + .../tool-calling/scenario-1/eval.json | 10 +- .../tool-calling/scenario-1/index.html | 11 + .../tool-calling/scenario-1/package.json | 20 ++ .../tool-calling/scenario-1/src/App.vue | 7 + .../tool-calling/scenario-1/src/main.ts | 4 + .../tool-calling/scenario-1/src/vite-env.d.ts | 1 + .../tool-calling/scenario-1/tsconfig.json | 17 ++ .../tool-calling/scenario-1/vite.config.ts | 4 + .../tool-calling/scenario-2/eval.json | 10 +- .../tool-calling/scenario-2/index.html | 11 + .../tool-calling/scenario-2/package.json | 20 ++ .../tool-calling/scenario-2/src/App.vue | 7 + .../tool-calling/scenario-2/src/main.ts | 4 + .../tool-calling/scenario-2/src/vite-env.d.ts | 1 + .../tool-calling/scenario-2/tsconfig.json | 17 ++ .../tool-calling/scenario-2/vite.config.ts | 4 + .../tool-calling/scenario-3/eval.json | 10 +- .../tool-calling/scenario-3/index.html | 11 + .../tool-calling/scenario-3/package.json | 20 ++ .../tool-calling/scenario-3/src/App.vue | 7 + .../tool-calling/scenario-3/src/main.ts | 4 + .../tool-calling/scenario-3/src/vite-env.d.ts | 1 + .../tool-calling/scenario-3/tsconfig.json | 17 ++ .../tool-calling/scenario-3/vite.config.ts | 4 + .../scenario-1/eval.json | 10 +- .../scenario-1/index.html | 11 + .../scenario-1/package.json | 20 ++ .../scenario-1/src/App.vue | 7 + .../scenario-1/src/main.ts | 4 + .../scenario-1/src/vite-env.d.ts | 1 + .../scenario-1/tsconfig.json | 17 ++ .../scenario-1/vite.config.ts | 4 + .../scenario-2/eval.json | 10 +- .../scenario-2/index.html | 11 + .../scenario-2/package.json | 20 ++ .../scenario-2/src/App.vue | 7 + .../scenario-2/src/main.ts | 4 + .../scenario-2/src/vite-env.d.ts | 1 + .../scenario-2/tsconfig.json | 17 ++ .../scenario-2/vite.config.ts | 4 + .../scenario-3/eval.json | 10 +- .../scenario-3/index.html | 11 + .../scenario-3/package.json | 20 ++ .../scenario-3/src/App.vue | 7 + .../scenario-3/src/main.ts | 4 + .../scenario-3/src/vite-env.d.ts | 1 + .../scenario-3/tsconfig.json | 17 ++ .../scenario-3/vite.config.ts | 4 + .../vue-3-5-helpers/scenario-1/eval.json | 10 +- .../vue-3-5-helpers/scenario-1/index.html | 11 + .../vue-3-5-helpers/scenario-1/package.json | 20 ++ .../vue-3-5-helpers/scenario-1/src/App.vue | 7 + .../vue-3-5-helpers/scenario-1/src/main.ts | 4 + .../scenario-1/src/vite-env.d.ts | 1 + .../vue-3-5-helpers/scenario-1/tsconfig.json | 17 ++ .../vue-3-5-helpers/scenario-1/vite.config.ts | 4 + .../vue-3-5-helpers/scenario-2/eval.json | 10 +- .../vue-3-5-helpers/scenario-2/index.html | 11 + .../vue-3-5-helpers/scenario-2/package.json | 20 ++ .../vue-3-5-helpers/scenario-2/src/App.vue | 7 + .../vue-3-5-helpers/scenario-2/src/main.ts | 4 + .../scenario-2/src/vite-env.d.ts | 1 + .../vue-3-5-helpers/scenario-2/tsconfig.json | 17 ++ .../vue-3-5-helpers/scenario-2/vite.config.ts | 4 + .../vue-3-5-helpers/scenario-3/eval.json | 10 +- .../vue-3-5-helpers/scenario-3/index.html | 11 + .../vue-3-5-helpers/scenario-3/package.json | 20 ++ .../vue-3-5-helpers/scenario-3/src/App.vue | 7 + .../vue-3-5-helpers/scenario-3/src/main.ts | 4 + .../scenario-3/src/vite-env.d.ts | 1 + .../vue-3-5-helpers/scenario-3/tsconfig.json | 17 ++ .../vue-3-5-helpers/scenario-3/vite.config.ts | 4 + package.json | 10 + 147 files changed, 1559 insertions(+), 61 deletions(-) create mode 100644 evals/runner.mjs create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/index.html create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/index.html create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/index.html create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/index.html create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/index.html create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/index.html create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/index.html create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/index.html create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/index.html create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/index.html create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/index.html create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/vite.config.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/index.html create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/main.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/vite.config.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/index.html create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/package.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/main.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/tsconfig.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/vite.config.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/index.html create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/package.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/main.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/tsconfig.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/vite.config.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/index.html create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/package.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/main.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/tsconfig.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/vite.config.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/index.html create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/package.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/main.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/tsconfig.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/vite.config.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/index.html create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/package.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/main.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/tsconfig.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/vite.config.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/index.html create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/package.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/main.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/vite-env.d.ts create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/tsconfig.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/vite.config.ts create mode 100644 package.json diff --git a/evals/README.md b/evals/README.md index 1ddecd9..9fac9d2 100644 --- a/evals/README.md +++ b/evals/README.md @@ -3,20 +3,36 @@ Eval suites for the skills, following the structure in [`AGENTS.md`](../AGENTS.md#eval-structure): `evals/suites/skills///scenario-{1,2,3}/`. -## Status: stubs +## Running -These are **scaffolded specs, not yet runnable**. Each scenario currently contains: +```bash +pnpm eval # all 3 scenarios, 4 tiers, recorded to results.json +pnpm eval --all # every reference +pnpm eval --force # re-run even if results.json exists +pnpm eval --model sonnet|haiku|opus +pnpm eval --tier with-skill # single tier, not recorded (debug) +pnpm eval --dry # validate + install + build the stub (no LLM, no cost) +pnpm eval --verbose # keep temp dirs, print commands +``` + +The runner is `evals/runner.mjs` (pure Node). For each scenario it copies the project to a +temp dir (withholding `eval.ts`/`eval.json`), sets up the tier, invokes the `claude` CLI to +satisfy the query, then `pnpm install` + `pnpm run build` + `vitest run eval.ts`. + +Each scenario is a self-contained Vue + Vite + Vitest project: - `eval.json` — query + `expected_behavior` (the spec) - `eval.ts` — Vitest content-pattern assertions (a focused starting set, expand per AGENTS.md) - `src/components/*.vue` — empty input stub (clean, no hints, per AGENTS.md) +- `package.json` / `vite.config.ts` / `tsconfig.json` / `index.html` / `src/main.ts` / `src/App.vue` -Before `pnpm eval` can run them, this repo still needs: +### Verified vs. requires budget -1. The **eval runner** (`pnpm eval`) and its package — not present in this commit. -2. Per-scenario **build files** (`package.json`, `vite.config.ts`, `tsconfig.json`, - `index.html`, `eslint.config.js`, `src/main.ts`) so each scenario is a self-contained, - buildable Vue project. The runner copies the suite, installs, builds, then runs `eval.ts`. +- **Verified:** `--dry` (install + build) passes on all scenarios; arg validation; results.json I/O. +- **Requires API budget + `claude` CLI:** the four LLM tiers (`baseline`, `with-skill`, + `with-skill-prompt`, `with-agents-md`). These are billed and user-triggered — not run in CI. +- Skill install for the `with-skill*` tiers uses `npx skills add` against + `$VUE_SKILLS_SOURCE` (default `Pythoughts-labs/vue3-best-practices`); override for local setups. ## Covered references diff --git a/evals/runner.mjs b/evals/runner.mjs new file mode 100644 index 0000000..c2f7af2 --- /dev/null +++ b/evals/runner.mjs @@ -0,0 +1,248 @@ +#!/usr/bin/env node +// Eval runner for the Vue skills. See AGENTS.md "How the Runner Works" and evals/README.md. +// +// Usage: +// pnpm eval run all scenarios of a reference (4 tiers, recorded) +// pnpm eval --all run every reference +// pnpm eval --force re-run even if results.json exists +// pnpm eval --model sonnet|haiku|opus +// pnpm eval --tier with-skill single tier, NOT recorded (debugging) +// pnpm eval --dry validate + install + build the stub project (no LLM, no record) +// pnpm eval --verbose keep temp dirs, print commands +// +// The LLM tiers shell out to the `claude` CLI and consume API budget; --dry does not. + +import { + readFileSync, writeFileSync, existsSync, mkdtempSync, cpSync, rmSync, readdirSync, +} from "node:fs"; +import { join, dirname, resolve } from "node:path"; +import { tmpdir } from "node:os"; +import { execFileSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), ".."); +const SUITES = join(ROOT, "evals", "suites", "skills"); +const TIERS = ["baseline", "with-skill", "with-skill-prompt", "with-agents-md"]; +const VALID_MODELS = ["haiku", "sonnet", "opus"]; +// Skill content source for `npx skills add`; override for non-published setups. +const SKILL_SOURCE = process.env.VUE_SKILLS_SOURCE || "Pythoughts-labs/vue3-best-practices"; + +// ---- args --------------------------------------------------------------- +const argv = process.argv.slice(2); +const flags = { + all: argv.includes("--all"), + force: argv.includes("--force"), + dry: argv.includes("--dry"), + verbose: argv.includes("--verbose"), + model: takeValue("--model") || "sonnet", + tier: takeValue("--tier"), +}; +const name = argv.find((a) => !a.startsWith("--") && argv[argv.indexOf(a) - 1] !== "--model" && argv[argv.indexOf(a) - 1] !== "--tier"); + +function takeValue(flag) { + const i = argv.indexOf(flag); + return i !== -1 && argv[i + 1] ? argv[i + 1] : null; +} + +if (!VALID_MODELS.includes(flags.model)) fail(`--model must be one of ${VALID_MODELS.join(", ")}`); +if (flags.tier && !TIERS.includes(flags.tier)) fail(`--tier must be one of ${TIERS.join(", ")}`); +if (!flags.all && !name) fail("provide a reference name or --all"); + +function fail(msg) { + console.error(`eval: ${msg}`); + process.exit(1); +} +const log = (...a) => console.log(...a); +const vlog = (...a) => flags.verbose && console.log(...a); + +// ---- discovery ---------------------------------------------------------- +// reference dirs live two levels under SUITES: //scenario-* +function findReference(ref) { + for (const skill of readdirSync(SUITES)) { + const p = join(SUITES, skill, ref); + if (existsSync(join(p, "scenario-1"))) return p; + } + return null; +} +function allReferences() { + const out = []; + for (const skill of readdirSync(SUITES)) { + const skillDir = join(SUITES, skill); + for (const ref of readdirSync(skillDir)) out.push(join(skillDir, ref)); + } + return out; +} +function scenariosOf(refDir) { + return readdirSync(refDir) + .filter((d) => d.startsWith("scenario-")) + .sort() + .map((d) => join(refDir, d)); +} + +// ---- shell helpers ------------------------------------------------------ +function run(cmd, args, cwd) { + vlog(`$ ${cmd} ${args.join(" ")} (cwd=${cwd})`); + execFileSync(cmd, args, { cwd, stdio: flags.verbose ? "inherit" : "pipe" }); +} +function tryRun(cmd, args, cwd) { + try { + run(cmd, args, cwd); + return true; + } catch (e) { + vlog(String(e.stderr || e.message || e)); + return false; + } +} + +// Copy a scenario into a temp workdir, excluding the withheld eval files. +function makeWorkdir(scenarioDir) { + const tmp = mkdtempSync(join(tmpdir(), "vue-eval-")); + cpSync(scenarioDir, tmp, { + recursive: true, + filter: (src) => !/[/\\](eval\.ts|eval\.json|results\.json)$/.test(src), + }); + return tmp; +} + +// ---- tier setup + generation ------------------------------------------- +function setupTier(tmp, tier, cfg) { + if (tier === "baseline") return; + if (tier === "with-agents-md") { + const body = cfg.skills + .map((s) => readFileSync(join(ROOT, "skills", s, "SKILL.md"), "utf-8")) + .join("\n\n---\n\n"); + writeFileSync(join(tmp, "AGENTS.md"), body); + return; + } + // with-skill / with-skill-prompt: install the skill(s) + for (const _ of cfg.skills) { + if (!tryRun("npx", ["--yes", "skills", "add", SKILL_SOURCE], tmp)) { + throw new Error(`skill install failed (source: ${SKILL_SOURCE})`); + } + } +} + +function promptFor(tier, cfg) { + if (tier === "with-skill-prompt") { + return `use ${cfg.skills.join(", ")} skill, ${cfg.query}`; + } + return cfg.query; +} + +// Invoke Claude Code headless to satisfy the query in the workdir. +function generate(tmp, tier, cfg) { + const prompt = promptFor(tier, cfg); + run("claude", ["-p", prompt, "--model", flags.model, "--permission-mode", "acceptEdits"], tmp); +} + +// ---- build + test ------------------------------------------------------- +function installAndBuild(tmp) { + run("pnpm", ["install", "--silent"], tmp); + run("pnpm", ["run", "build"], tmp); // vue-tsc --noEmit && vite build +} +function runEvalTest(tmp, scenarioDir) { + cpSync(join(scenarioDir, "eval.ts"), join(tmp, "eval.ts")); + return tryRun("pnpm", ["exec", "vitest", "run", "eval.ts"], tmp); +} + +// One generate→build→test attempt. Returns true on pass. +function attempt(scenarioDir, cfg, tier) { + const tmp = makeWorkdir(scenarioDir); + try { + setupTier(tmp, tier, cfg); + generate(tmp, tier, cfg); + installAndBuild(tmp); // throws on build failure → caught as fail + return runEvalTest(tmp, scenarioDir); + } catch (e) { + vlog(`attempt failed: ${e.message}`); + return false; + } finally { + if (!flags.verbose) rmSync(tmp, { recursive: true, force: true }); + } +} + +// 2 runs, fail-fast (AGENTS.md). +function evalTier(scenarioDir, cfg, tier) { + const t0 = Date.now(); + if (!attempt(scenarioDir, cfg, tier)) return { passed: false, duration: Date.now() - t0 }; + const passed = attempt(scenarioDir, cfg, tier); + return { passed, duration: Date.now() - t0 }; +} + +// ---- dry run ------------------------------------------------------------ +function dryScenario(scenarioDir) { + const cfg = JSON.parse(readFileSync(join(scenarioDir, "eval.json"), "utf-8")); + for (const k of ["skills", "query", "files", "expected_behavior"]) { + if (!cfg[k]) throw new Error(`${scenarioDir}: eval.json missing "${k}"`); + } + const tmp = makeWorkdir(scenarioDir); + try { + installAndBuild(tmp); // proves the project boilerplate is valid + buildable + return true; + } finally { + if (!flags.verbose) rmSync(tmp, { recursive: true, force: true }); + } +} + +// ---- results.json ------------------------------------------------------- +function resultsPath(scenarioDir) { + return join(scenarioDir, "results.json"); +} +function loadResults(scenarioDir) { + const p = resultsPath(scenarioDir); + return existsSync(p) ? JSON.parse(readFileSync(p, "utf-8")) : {}; +} +function saveResults(scenarioDir, data) { + writeFileSync(resultsPath(scenarioDir), JSON.stringify(data, null, 2) + "\n"); +} + +// ---- orchestration ------------------------------------------------------ +function runScenarioFull(scenarioDir) { + const cfg = JSON.parse(readFileSync(join(scenarioDir, "eval.json"), "utf-8")); + const results = loadResults(scenarioDir); + if (results[flags.model] && !flags.force) { + log(` ${rel(scenarioDir)} [${flags.model}] cached — skip (use --force)`); + return; + } + const tiers = {}; + for (const tier of TIERS) { + process.stdout.write(` ${rel(scenarioDir)} [${flags.model}/${tier}] … `); + const res = evalTier(scenarioDir, cfg, tier); + tiers[tier] = res; + log(res.passed ? `pass (${res.duration}ms)` : `FAIL (${res.duration}ms)`); + } + results[flags.model] = { timestamp: new Date().toISOString(), tiers }; + saveResults(scenarioDir, results); +} + +function rel(p) { + return p.replace(SUITES + "/", ""); +} + +const refDirs = flags.all + ? allReferences() + : [findReference(name) || fail(`reference "${name}" not found under evals/suites/skills`)]; + +let failures = 0; +for (const refDir of refDirs) { + log(`\n${rel(refDir)}`); + for (const scenarioDir of scenariosOf(refDir)) { + try { + if (flags.dry) { + dryScenario(scenarioDir); + log(` ${rel(scenarioDir)} dry: build OK`); + } else if (flags.tier) { + const cfg = JSON.parse(readFileSync(join(scenarioDir, "eval.json"), "utf-8")); + const res = evalTier(scenarioDir, cfg, flags.tier); + log(` ${rel(scenarioDir)} [${flags.model}/${flags.tier}] ${res.passed ? "pass" : "FAIL"} (not recorded)`); + if (!res.passed) failures++; + } else { + runScenarioFull(scenarioDir); + } + } catch (e) { + failures++; + log(` ${rel(scenarioDir)} ERROR: ${e.message}`); + } + } +} +process.exit(failures ? 1 : 0); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json index ee7c30d..dee5780 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Add a stop button and error handling to a Vue AI chat component built with @ai-sdk/vue.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Shows a Stop control that calls stop() while streaming", "Surfaces the error ref in the template", diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/index.html b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json new file mode 100644 index 0000000..825dc93 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-error-handling-and-abort-scenario-1", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/main.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/vite.config.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json index 799d169..332c076 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "Add a stop button and error handling to a Vue AI chat component built with @ai-sdk/vue.", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Make a Vue LLM chat resilient: let the user cancel a streaming response and retry after a failure.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Shows a Stop control that calls stop() while streaming", "Surfaces the error ref in the template", diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/index.html b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json new file mode 100644 index 0000000..e234de0 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-error-handling-and-abort-scenario-2", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/main.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/vite.config.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json index 67dee65..6e37299 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "Make a Vue LLM chat resilient: let the user cancel a streaming response and retry after a failure.", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Wire abort and error recovery into a Vue useChat-based assistant UI.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Shows a Stop control that calls stop() while streaming", "Surfaces the error ref in the template", diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/index.html b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json new file mode 100644 index 0000000..4dcce70 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-error-handling-and-abort-scenario-3", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/main.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/vite.config.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json index 2990cf5..c3e2a1a 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Build a Vue chat component that streams responses from an LLM using the Vercel AI SDK in a Nuxt app.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Imports useChat from '@ai-sdk/vue' (not @ai-sdk/react)", "Renders message.parts, not message.content", diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/index.html b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json new file mode 100644 index 0000000..72149c1 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-streaming-chat-ui-scenario-1", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/main.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/vite.config.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json index 62f07a0..d19e5f5 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "Build a Vue chat component that streams responses from an LLM using the Vercel AI SDK in a Nuxt app.", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Create a streaming AI chat box in Vue with an input field and a list of messages from an assistant.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Imports useChat from '@ai-sdk/vue' (not @ai-sdk/react)", "Renders message.parts, not message.content", diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/index.html b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json new file mode 100644 index 0000000..65d5b41 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-streaming-chat-ui-scenario-2", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/main.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/vite.config.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json index b40e6bc..b077151 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "Create a streaming AI chat box in Vue with an input field and a list of messages from an assistant.", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Implement a Vue component that talks to a /api/chat streaming endpoint and renders the assistant reply token by token.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Imports useChat from '@ai-sdk/vue' (not @ai-sdk/react)", "Renders message.parts, not message.content", diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/index.html b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json new file mode 100644 index 0000000..0d1a5cf --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-streaming-chat-ui-scenario-3", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/main.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/vite.config.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json index 1ede431..d9b96fd 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "", - "files": ["src/components/RecipeGenerator.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Build a Vue component that streams a typed recipe object from an LLM and renders it as it arrives.", + "files": [ + "src/components/RecipeGenerator.vue" + ], "expected_behavior": [ "Uses useObject from @ai-sdk/vue for structured streaming", "Guards partial fields with optional chaining (object?.x) while streaming", diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/index.html b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json new file mode 100644 index 0000000..9c1cf19 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-structured-output-scenario-1", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue new file mode 100644 index 0000000..9e47be7 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/main.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/vite.config.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json index ebbfd7e..5ecab98 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "Build a Vue component that streams a typed recipe object from an LLM and renders it as it arrives.", - "files": ["src/components/RecipeGenerator.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Use the AI SDK to stream a structured object in Vue and display partial fields safely while loading.", + "files": [ + "src/components/RecipeGenerator.vue" + ], "expected_behavior": [ "Uses useObject from @ai-sdk/vue for structured streaming", "Guards partial fields with optional chaining (object?.x) while streaming", diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/index.html b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json new file mode 100644 index 0000000..0d45b75 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-structured-output-scenario-2", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue new file mode 100644 index 0000000..9e47be7 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/main.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/vite.config.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json index 45665e2..89bfc03 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "Use the AI SDK to stream a structured object in Vue and display partial fields safely while loading.", - "files": ["src/components/RecipeGenerator.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Create a Vue UI that calls a streamObject endpoint and renders a typed result that fills in progressively.", + "files": [ + "src/components/RecipeGenerator.vue" + ], "expected_behavior": [ "Uses useObject from @ai-sdk/vue for structured streaming", "Guards partial fields with optional chaining (object?.x) while streaming", diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/index.html b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json new file mode 100644 index 0000000..34e84b2 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-structured-output-scenario-3", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue new file mode 100644 index 0000000..9e47be7 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/main.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/vite.config.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json index 068423e..d2c285d 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "In a Vue chat UI, render an LLM tool call (getWeather) including its loading and result states using the AI SDK v5.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Branches a tool part on part.state (input-available / output-available / output-error)", "Reads part.output only in the output-available state", diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/index.html b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json new file mode 100644 index 0000000..3fa5170 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-tool-calling-scenario-1", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/main.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/vite.config.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json index c5f0a14..5096a87 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "In a Vue chat UI, render an LLM tool call (getWeather) including its loading and result states using the AI SDK v5.", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Show tool-call progress and output inside a Vue AI chat component for a weather tool.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Branches a tool part on part.state (input-available / output-available / output-error)", "Reads part.output only in the output-available state", diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/index.html b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json new file mode 100644 index 0000000..2baca4f --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-tool-calling-scenario-2", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/main.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/vite.config.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json index bd13e10..36ba265 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-ai-apps"], - "query": "Show tool-call progress and output inside a Vue AI chat component for a weather tool.", - "files": ["src/components/ChatBox.vue"], + "skills": [ + "vue-ai-apps" + ], + "query": "Display a function/tool call and its result in a Vue assistant message using @ai-sdk/vue message parts.", + "files": [ + "src/components/ChatBox.vue" + ], "expected_behavior": [ "Branches a tool part on part.state (input-available / output-available / output-error)", "Reads part.output only in the output-available state", diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/index.html b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json new file mode 100644 index 0000000..482f253 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-ai-apps-tool-calling-scenario-3", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue new file mode 100644 index 0000000..6bfc52b --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/main.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/vite-env.d.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/vite.config.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json index a368249..6efb70e 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-best-practices"], - "query": "", - "files": ["src/components/UserCard.vue"], + "skills": [ + "vue-best-practices" + ], + "query": "Create a Vue 3.5 component that takes an 'id' prop, gives 'count' a default of 0, and refetches a user whenever id changes.", + "files": [ + "src/components/UserCard.vue" + ], "expected_behavior": [ "Uses reactive props destructure with defaults (const { count = 0 } = defineProps...)", "Does NOT use withDefaults", diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/index.html b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/package.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/package.json new file mode 100644 index 0000000..9f09bbd --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-best-practices-reactive-props-destructure-scenario-1", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue new file mode 100644 index 0000000..431d770 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/main.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/vite-env.d.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/tsconfig.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/vite.config.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json index 8a79c84..5f60ba4 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-best-practices"], - "query": "Create a Vue 3.5 component that takes an 'id' prop, gives 'count' a default of 0, and refetches a user whenever id changes.", - "files": ["src/components/UserCard.vue"], + "skills": [ + "vue-best-practices" + ], + "query": "Build a Vue 3.5 child component with default prop values that watches a prop and passes it to a composable.", + "files": [ + "src/components/UserCard.vue" + ], "expected_behavior": [ "Uses reactive props destructure with defaults (const { count = 0 } = defineProps...)", "Does NOT use withDefaults", diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/index.html b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/package.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/package.json new file mode 100644 index 0000000..0c8f147 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-best-practices-reactive-props-destructure-scenario-2", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue new file mode 100644 index 0000000..431d770 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/main.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/vite-env.d.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/tsconfig.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/vite.config.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json index 45a2156..10d6980 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-best-practices"], - "query": "Build a Vue 3.5 child component with default prop values that watches a prop and passes it to a composable.", - "files": ["src/components/UserCard.vue"], + "skills": [ + "vue-best-practices" + ], + "query": "Make a Vue 3.5 component using reactive props destructure with defaults that reacts to prop changes in a watcher.", + "files": [ + "src/components/UserCard.vue" + ], "expected_behavior": [ "Uses reactive props destructure with defaults (const { count = 0 } = defineProps...)", "Does NOT use withDefaults", diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/index.html b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/package.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/package.json new file mode 100644 index 0000000..b345fca --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-best-practices-reactive-props-destructure-scenario-3", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue new file mode 100644 index 0000000..431d770 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/main.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/vite-env.d.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/tsconfig.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/vite.config.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json index 59442e8..7bad6a8 100644 --- a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-best-practices"], - "query": "", - "files": ["src/components/SignupField.vue"], + "skills": [ + "vue-best-practices" + ], + "query": "Build an accessible Vue 3.5 form field where the label is correctly linked to the input with a unique, SSR-safe id.", + "files": [ + "src/components/SignupField.vue" + ], "expected_behavior": [ "Imports and uses useId() from vue for the field id", "Binds :for and :id to the same generated id", diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/index.html b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/package.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/package.json new file mode 100644 index 0000000..719de6d --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-best-practices-vue-3-5-helpers-scenario-1", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue new file mode 100644 index 0000000..99dcb8a --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/main.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/vite-env.d.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/tsconfig.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/vite.config.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json index b14fef7..7ea3101 100644 --- a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-best-practices"], - "query": "Build an accessible Vue 3.5 form field where the label is correctly linked to the input with a unique, SSR-safe id.", - "files": ["src/components/SignupField.vue"], + "skills": [ + "vue-best-practices" + ], + "query": "Create a Vue 3.5 component that renders a label and input wired together with an id that is stable across SSR and client.", + "files": [ + "src/components/SignupField.vue" + ], "expected_behavior": [ "Imports and uses useId() from vue for the field id", "Binds :for and :id to the same generated id", diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/index.html b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/package.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/package.json new file mode 100644 index 0000000..bbac13f --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-best-practices-vue-3-5-helpers-scenario-2", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue new file mode 100644 index 0000000..99dcb8a --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/main.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/vite-env.d.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/tsconfig.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/vite.config.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json index 1146c02..fbe7e49 100644 --- a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/eval.json @@ -1,7 +1,11 @@ { - "skills": ["vue-best-practices"], - "query": "Create a Vue 3.5 component that renders a label and input wired together with an id that is stable across SSR and client.", - "files": ["src/components/SignupField.vue"], + "skills": [ + "vue-best-practices" + ], + "query": "Make an SSR-safe Vue 3.5 labelled input using the built-in id helper instead of a hand-rolled counter.", + "files": [ + "src/components/SignupField.vue" + ], "expected_behavior": [ "Imports and uses useId() from vue for the field id", "Binds :for and :id to the same generated id", diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/index.html b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/index.html new file mode 100644 index 0000000..95c4f96 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/index.html @@ -0,0 +1,11 @@ + + + + + eval + + +
    + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/package.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/package.json new file mode 100644 index 0000000..69bc88a --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/package.json @@ -0,0 +1,20 @@ +{ + "name": "eval-vue-best-practices-vue-3-5-helpers-scenario-3", + "private": true, + "version": "0.0.0", + "type": "module", + "scripts": { + "build": "vue-tsc --noEmit && vite build", + "test": "vitest run" + }, + "dependencies": { + "vue": "^3.5.13" + }, + "devDependencies": { + "@vitejs/plugin-vue": "^5.2.1", + "typescript": "^5.7.0", + "vite": "^6.0.0", + "vitest": "^3.0.0", + "vue-tsc": "^2.2.0" + } +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue new file mode 100644 index 0000000..99dcb8a --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue @@ -0,0 +1,7 @@ + + + diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/main.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/main.ts new file mode 100644 index 0000000..b670de8 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/main.ts @@ -0,0 +1,4 @@ +import { createApp } from "vue"; +import App from "./App.vue"; + +createApp(App).mount("#app"); diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/vite-env.d.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/vite-env.d.ts new file mode 100644 index 0000000..11f02fe --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/vite-env.d.ts @@ -0,0 +1 @@ +/// diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/tsconfig.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/tsconfig.json new file mode 100644 index 0000000..adfc6ec --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ESNext", + "module": "ESNext", + "moduleResolution": "Bundler", + "strict": true, + "jsx": "preserve", + "resolveJsonModule": true, + "isolatedModules": true, + "esModuleInterop": true, + "lib": ["ESNext", "DOM", "DOM.Iterable"], + "skipLibCheck": true, + "noEmit": true, + "types": ["vitest/globals"] + }, + "include": ["src", "eval.ts"] +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/vite.config.ts b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/vite.config.ts new file mode 100644 index 0000000..10346fc --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/vite.config.ts @@ -0,0 +1,4 @@ +import { defineConfig } from "vite"; +import vue from "@vitejs/plugin-vue"; + +export default defineConfig({ plugins: [vue()] }); diff --git a/package.json b/package.json new file mode 100644 index 0000000..d43c69d --- /dev/null +++ b/package.json @@ -0,0 +1,10 @@ +{ + "name": "vue3-skills", + "private": true, + "type": "module", + "description": "Vue 3 best-practice skills — eval runner entrypoint.", + "license": "MIT", + "scripts": { + "eval": "node evals/runner.mjs" + } +} From a2e546f5e8af8b8fe2d5ed65c1df4c730dab2dd4 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 14:19:42 -0400 Subject: [PATCH 09/31] ci(sync): add root package.json to Sync to Main whitelist So 'pnpm eval' works on main alongside the published eval suites. --- .github/workflows/sync-to-main.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/sync-to-main.yml b/.github/workflows/sync-to-main.yml index 0258ac5..39e6a47 100644 --- a/.github/workflows/sync-to-main.yml +++ b/.github/workflows/sync-to-main.yml @@ -49,7 +49,7 @@ jobs: # ...then re-apply only whitelisted paths from dev git restore --source=origin/dev --staged --worktree -- \ - skills/ evals/ .github/ .claude-plugin/ README.md AGENTS.md CLAUDE.md LICENSE + skills/ evals/ package.json .github/ .claude-plugin/ README.md AGENTS.md CLAUDE.md LICENSE git add -A if git diff --staged --quiet; then From 0b7cd3becee873ed192e1ef700fad70dcd5f9c73 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 14:21:45 -0400 Subject: [PATCH 10/31] chore(tasks): mark eval runner done; remaining = run the matrix --- tasks/todo.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/tasks/todo.md b/tasks/todo.md index 41f22e2..17e1ced 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -17,13 +17,21 @@ Eval specs scaffolded under `evals/suites/skills/...` (6 references × 3 scenari scenarios, 54 files). Each scenario has `eval.json` + starter `eval.ts` + clean input stub. See `evals/README.md`. `vapor-mode` intentionally skipped (experimental until 3.6 stable). -## Out of scope — remaining follow-up (per AGENTS.md) +## Eval runner — DONE -- Eval **runner** (`pnpm eval`) + package: not in this repo commit; needed to execute suites. -- Per-scenario **build boilerplate** (package.json/vite/tsconfig/index.html/main.ts) so each - scenario is a self-contained buildable project the runner can install + build. -- Then run the matrix: 3 evals × 4 tiers × 3 models per reference (haiku/sonnet/opus), record - to `results.json`. Billed LLM runs — user-triggered. +- `evals/runner.mjs` + root `package.json` (`pnpm eval`) — implements the AGENTS.md flow. +- Per-scenario Vue+Vite+Vitest boilerplate added; every scenario builds standalone. +- Verified: `pnpm eval --dry` (install+build) passes; arg validation; results.json I/O. +- On `main` (PR #5) and `dev`. + +## Out of scope — remaining follow-up + +- **Run the matrix** (billed, user-triggered): `pnpm eval --all` → 4 tiers × 3 models per + scenario, recorded to `results.json`. Requires the `claude` CLI + API budget. + The four LLM tiers are not exercised in CI and are unverified beyond `--dry`. +- `with-skill*` tiers install via `npx skills add $VUE_SKILLS_SOURCE` + (default `Pythoughts-labs/vue3-best-practices`) — confirm resolution in the target env. +- AI-SDK doc items to confirm against an installed `@ai-sdk/vue` (useObject export; server return form). ## Verify-before-trust notes (flagged in the content, confirm against installed SDK) From 447d9c8d6492b024f7ddf688941ee3c92654b089 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 14:35:36 -0400 Subject: [PATCH 11/31] fix(evals): make LLM tiers actually exercise the skill MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found via a real run + diagnosis (baseline and with-skill both failed): - skill tiers now copy the local skill into .claude/skills (where claude -p discovers it); npx skills add installs to .agents/skills, which is not loaded - prompt now names the target file from eval.json (empty stub gave no signal, so the agent wrote elsewhere and the target file stayed empty) - copy eval.ts in as eval.test.ts so vitest's default include discovers it - vue-ai-apps scenarios: shim the (fast-moving) AI SDK modules and typecheck with vue-tsc instead of bundling, with noImplicitAny off — these evals check generated patterns, not live SDK types - relax error-handling assertions (stop/regenerate are handlers, no parens) - runner: ignore stdin so claude -p does not block 3s per call --- evals/runner.mjs | 34 ++++++++++++------- .../scenario-1/eval.ts | 4 +-- .../scenario-1/package.json | 2 +- .../scenario-1/src/sdk-shims.d.ts | 8 +++++ .../scenario-1/tsconfig.json | 16 +++++++-- .../scenario-2/eval.ts | 4 +-- .../scenario-2/package.json | 2 +- .../scenario-2/src/sdk-shims.d.ts | 8 +++++ .../scenario-2/tsconfig.json | 16 +++++++-- .../scenario-3/eval.ts | 4 +-- .../scenario-3/package.json | 2 +- .../scenario-3/src/sdk-shims.d.ts | 8 +++++ .../scenario-3/tsconfig.json | 16 +++++++-- .../streaming-chat-ui/scenario-1/package.json | 2 +- .../scenario-1/src/sdk-shims.d.ts | 8 +++++ .../scenario-1/tsconfig.json | 16 +++++++-- .../streaming-chat-ui/scenario-2/package.json | 2 +- .../scenario-2/src/sdk-shims.d.ts | 8 +++++ .../scenario-2/tsconfig.json | 16 +++++++-- .../streaming-chat-ui/scenario-3/package.json | 2 +- .../scenario-3/src/sdk-shims.d.ts | 8 +++++ .../scenario-3/tsconfig.json | 16 +++++++-- .../structured-output/scenario-1/package.json | 2 +- .../scenario-1/src/sdk-shims.d.ts | 8 +++++ .../scenario-1/tsconfig.json | 16 +++++++-- .../structured-output/scenario-2/package.json | 2 +- .../scenario-2/src/sdk-shims.d.ts | 8 +++++ .../scenario-2/tsconfig.json | 16 +++++++-- .../structured-output/scenario-3/package.json | 2 +- .../scenario-3/src/sdk-shims.d.ts | 8 +++++ .../scenario-3/tsconfig.json | 16 +++++++-- .../tool-calling/scenario-1/package.json | 2 +- .../scenario-1/src/sdk-shims.d.ts | 8 +++++ .../tool-calling/scenario-1/tsconfig.json | 16 +++++++-- .../tool-calling/scenario-2/package.json | 2 +- .../scenario-2/src/sdk-shims.d.ts | 8 +++++ .../tool-calling/scenario-2/tsconfig.json | 16 +++++++-- .../tool-calling/scenario-3/package.json | 2 +- .../scenario-3/src/sdk-shims.d.ts | 8 +++++ .../tool-calling/scenario-3/tsconfig.json | 16 +++++++-- 40 files changed, 292 insertions(+), 66 deletions(-) create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/sdk-shims.d.ts create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/sdk-shims.d.ts diff --git a/evals/runner.mjs b/evals/runner.mjs index c2f7af2..5b62360 100644 --- a/evals/runner.mjs +++ b/evals/runner.mjs @@ -13,7 +13,7 @@ // The LLM tiers shell out to the `claude` CLI and consume API budget; --dry does not. import { - readFileSync, writeFileSync, existsSync, mkdtempSync, cpSync, rmSync, readdirSync, + readFileSync, writeFileSync, existsSync, mkdtempSync, mkdirSync, cpSync, rmSync, readdirSync, } from "node:fs"; import { join, dirname, resolve } from "node:path"; import { tmpdir } from "node:os"; @@ -24,8 +24,8 @@ const ROOT = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const SUITES = join(ROOT, "evals", "suites", "skills"); const TIERS = ["baseline", "with-skill", "with-skill-prompt", "with-agents-md"]; const VALID_MODELS = ["haiku", "sonnet", "opus"]; -// Skill content source for `npx skills add`; override for non-published setups. -const SKILL_SOURCE = process.env.VUE_SKILLS_SOURCE || "Pythoughts-labs/vue3-best-practices"; +// Local skills directory copied into each scenario's .claude/skills for the skill tiers. +const SKILL_SOURCE_DIR = process.env.VUE_SKILLS_DIR || join(ROOT, "skills"); // ---- args --------------------------------------------------------------- const argv = process.argv.slice(2); @@ -82,7 +82,11 @@ function scenariosOf(refDir) { // ---- shell helpers ------------------------------------------------------ function run(cmd, args, cwd) { vlog(`$ ${cmd} ${args.join(" ")} (cwd=${cwd})`); - execFileSync(cmd, args, { cwd, stdio: flags.verbose ? "inherit" : "pipe" }); + // stdin ignored: claude -p otherwise blocks ~3s waiting for piped input. + execFileSync(cmd, args, { + cwd, + stdio: flags.verbose ? ["ignore", "inherit", "inherit"] : ["ignore", "pipe", "pipe"], + }); } function tryRun(cmd, args, cwd) { try { @@ -114,11 +118,15 @@ function setupTier(tmp, tier, cfg) { writeFileSync(join(tmp, "AGENTS.md"), body); return; } - // with-skill / with-skill-prompt: install the skill(s) - for (const _ of cfg.skills) { - if (!tryRun("npx", ["--yes", "skills", "add", SKILL_SOURCE], tmp)) { - throw new Error(`skill install failed (source: ${SKILL_SOURCE})`); - } + // with-skill / with-skill-prompt: place the skill where Claude Code discovers it. + // (`npx skills add` installs to .agents/skills, which `claude -p` does not load; + // copying the local skill into .claude/skills tests the current skill content.) + const skillsDir = join(tmp, ".claude", "skills"); + mkdirSync(skillsDir, { recursive: true }); + for (const s of cfg.skills) { + const src = join(SKILL_SOURCE_DIR, s); + if (!existsSync(src)) throw new Error(`skill "${s}" not found at ${src}`); + cpSync(src, join(skillsDir, s), { recursive: true }); } } @@ -131,7 +139,8 @@ function promptFor(tier, cfg) { // Invoke Claude Code headless to satisfy the query in the workdir. function generate(tmp, tier, cfg) { - const prompt = promptFor(tier, cfg); + // Name the target file(s); the empty stub gives the agent no signal otherwise. + const prompt = `${promptFor(tier, cfg)}\n\nImplement your solution in this existing file: ${cfg.files.join(", ")}. Do not create additional component files.`; run("claude", ["-p", prompt, "--model", flags.model, "--permission-mode", "acceptEdits"], tmp); } @@ -141,8 +150,9 @@ function installAndBuild(tmp) { run("pnpm", ["run", "build"], tmp); // vue-tsc --noEmit && vite build } function runEvalTest(tmp, scenarioDir) { - cpSync(join(scenarioDir, "eval.ts"), join(tmp, "eval.ts")); - return tryRun("pnpm", ["exec", "vitest", "run", "eval.ts"], tmp); + // Copy in as *.test.ts so vitest's default include discovers it. + cpSync(join(scenarioDir, "eval.ts"), join(tmp, "eval.test.ts")); + return tryRun("pnpm", ["exec", "vitest", "run", "eval.test.ts"], tmp); } // One generate→build→test attempt. Returns true on pass. diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts index f520192..934f280 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/eval.ts @@ -5,9 +5,9 @@ import { join } from "path"; const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); test("wires abort and error recovery", () => { - expect(src).toMatch(/\bstop\s*\(/); + expect(src).toMatch(/\bstop\b/); expect(src).toMatch(/\berror\b/); - expect(src).toMatch(/regenerate\s*\(/); + expect(src).toMatch(/\bregenerate\b/); }); test("disables send via status", () => { diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json index 825dc93..e39f24c 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts index f520192..934f280 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/eval.ts @@ -5,9 +5,9 @@ import { join } from "path"; const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); test("wires abort and error recovery", () => { - expect(src).toMatch(/\bstop\s*\(/); + expect(src).toMatch(/\bstop\b/); expect(src).toMatch(/\berror\b/); - expect(src).toMatch(/regenerate\s*\(/); + expect(src).toMatch(/\bregenerate\b/); }); test("disables send via status", () => { diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json index e234de0..3a200a2 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts index f520192..934f280 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/eval.ts @@ -5,9 +5,9 @@ import { join } from "path"; const src = readFileSync(join(process.cwd(), "src/components/ChatBox.vue"), "utf-8"); test("wires abort and error recovery", () => { - expect(src).toMatch(/\bstop\s*\(/); + expect(src).toMatch(/\bstop\b/); expect(src).toMatch(/\berror\b/); - expect(src).toMatch(/regenerate\s*\(/); + expect(src).toMatch(/\bregenerate\b/); }); test("disables send via status", () => { diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json index 4dcce70..3a3b6ed 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json index 72149c1..903d8e3 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json index 65d5b41..e0216f1 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json index 0d1a5cf..7177ae7 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json index 9c1cf19..3b8d115 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json index 0d45b75..544d4fb 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json index 34e84b2..53613ab 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json index 3fa5170..d3bd7dc 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json index 2baca4f..23f6b42 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json index 482f253..8514cee 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/package.json @@ -4,7 +4,7 @@ "version": "0.0.0", "type": "module", "scripts": { - "build": "vue-tsc --noEmit && vite build", + "build": "vue-tsc --noEmit", "test": "vitest run" }, "dependencies": { diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/sdk-shims.d.ts b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/sdk-shims.d.ts new file mode 100644 index 0000000..2a1804c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/sdk-shims.d.ts @@ -0,0 +1,8 @@ +// Eval shim: the AI SDK surface is volatile; these evals check generated code +// patterns, not the live SDK. Declaring the modules lets vue-tsc typecheck the +// component without pinning fast-moving external versions. +declare module "ai"; +declare module "@ai-sdk/vue"; +declare module "@ai-sdk/openai"; +declare module "@ai-sdk/anthropic"; +declare module "zod"; diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json index adfc6ec..13dccfa 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/tsconfig.json @@ -8,10 +8,20 @@ "resolveJsonModule": true, "isolatedModules": true, "esModuleInterop": true, - "lib": ["ESNext", "DOM", "DOM.Iterable"], + "lib": [ + "ESNext", + "DOM", + "DOM.Iterable" + ], "skipLibCheck": true, "noEmit": true, - "types": ["vitest/globals"] + "types": [ + "vitest/globals" + ], + "noImplicitAny": false }, - "include": ["src", "eval.ts"] + "include": [ + "src", + "eval.ts" + ] } From e441db1ba209ea5923d8749093d75ad748aad19c Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 16:42:22 -0400 Subject: [PATCH 12/31] fix(evals): App.vue shell no longer instantiates the subject component A subject with a required prop (e.g. reactive-props-destructure's `id`) made App.vue's fail vue-tsc (TS2345: prop missing), failing the build for every tier regardless of skill. The component is still type-checked via tsconfig include: src, so the App shell does not need to render it. --- .../scenario-1/results.json | 23 +++++++++++++++++++ .../scenario-1/src/App.vue | 6 +++-- .../scenario-2/results.json | 23 +++++++++++++++++++ .../scenario-2/src/App.vue | 6 +++-- .../scenario-3/results.json | 23 +++++++++++++++++++ .../scenario-3/src/App.vue | 6 +++-- .../streaming-chat-ui/scenario-1/results.json | 23 +++++++++++++++++++ .../streaming-chat-ui/scenario-1/src/App.vue | 6 +++-- .../streaming-chat-ui/scenario-2/results.json | 23 +++++++++++++++++++ .../streaming-chat-ui/scenario-2/src/App.vue | 6 +++-- .../streaming-chat-ui/scenario-3/results.json | 23 +++++++++++++++++++ .../streaming-chat-ui/scenario-3/src/App.vue | 6 +++-- .../structured-output/scenario-1/results.json | 23 +++++++++++++++++++ .../structured-output/scenario-1/src/App.vue | 6 +++-- .../structured-output/scenario-2/results.json | 23 +++++++++++++++++++ .../structured-output/scenario-2/src/App.vue | 6 +++-- .../structured-output/scenario-3/results.json | 23 +++++++++++++++++++ .../structured-output/scenario-3/src/App.vue | 6 +++-- .../tool-calling/scenario-1/results.json | 23 +++++++++++++++++++ .../tool-calling/scenario-1/src/App.vue | 6 +++-- .../tool-calling/scenario-2/results.json | 23 +++++++++++++++++++ .../tool-calling/scenario-2/src/App.vue | 6 +++-- .../tool-calling/scenario-3/results.json | 23 +++++++++++++++++++ .../tool-calling/scenario-3/src/App.vue | 6 +++-- .../scenario-1/results.json | 23 +++++++++++++++++++ .../scenario-1/src/App.vue | 6 +++-- .../scenario-2/results.json | 23 +++++++++++++++++++ .../scenario-2/src/App.vue | 6 +++-- .../scenario-3/results.json | 23 +++++++++++++++++++ .../scenario-3/src/App.vue | 6 +++-- .../vue-3-5-helpers/scenario-1/results.json | 23 +++++++++++++++++++ .../vue-3-5-helpers/scenario-1/src/App.vue | 6 +++-- .../vue-3-5-helpers/scenario-2/results.json | 23 +++++++++++++++++++ .../vue-3-5-helpers/scenario-2/src/App.vue | 6 +++-- .../vue-3-5-helpers/scenario-3/results.json | 23 +++++++++++++++++++ .../vue-3-5-helpers/scenario-3/src/App.vue | 6 +++-- 36 files changed, 486 insertions(+), 36 deletions(-) create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/results.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/results.json create mode 100644 evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/results.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/results.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/results.json create mode 100644 evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/results.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-1/results.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-2/results.json create mode 100644 evals/suites/skills/vue-ai-apps/structured-output/scenario-3/results.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/results.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/results.json create mode 100644 evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/results.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json create mode 100644 evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/results.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/results.json create mode 100644 evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/results.json diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/results.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/results.json new file mode 100644 index 0000000..bf59a2a --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T18:41:59.944Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 52047 + }, + "with-skill": { + "passed": false, + "duration": 56871 + }, + "with-skill-prompt": { + "passed": true, + "duration": 73321 + }, + "with-agents-md": { + "passed": true, + "duration": 179252 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-1/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/results.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/results.json new file mode 100644 index 0000000..e2bc8bd --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T18:50:18.386Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 68710 + }, + "with-skill": { + "passed": true, + "duration": 161142 + }, + "with-skill-prompt": { + "passed": true, + "duration": 166427 + }, + "with-agents-md": { + "passed": false, + "duration": 102160 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-2/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/results.json b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/results.json new file mode 100644 index 0000000..93a9b68 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:00:16.698Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 63152 + }, + "with-skill": { + "passed": false, + "duration": 117972 + }, + "with-skill-prompt": { + "passed": true, + "duration": 77390 + }, + "with-agents-md": { + "passed": true, + "duration": 339798 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/error-handling-and-abort/scenario-3/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/results.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/results.json new file mode 100644 index 0000000..90bd0ff --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:09:10.773Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 70432 + }, + "with-skill": { + "passed": true, + "duration": 114343 + }, + "with-skill-prompt": { + "passed": true, + "duration": 140536 + }, + "with-agents-md": { + "passed": true, + "duration": 208762 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-1/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/results.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/results.json new file mode 100644 index 0000000..16a81fd --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:14:23.866Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 67264 + }, + "with-skill": { + "passed": false, + "duration": 57999 + }, + "with-skill-prompt": { + "passed": true, + "duration": 107202 + }, + "with-agents-md": { + "passed": false, + "duration": 80626 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-2/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/results.json b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/results.json new file mode 100644 index 0000000..136911e --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:21:45.459Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 66434 + }, + "with-skill": { + "passed": false, + "duration": 101130 + }, + "with-skill-prompt": { + "passed": true, + "duration": 170794 + }, + "with-agents-md": { + "passed": false, + "duration": 103233 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/streaming-chat-ui/scenario-3/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/results.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/results.json new file mode 100644 index 0000000..bc910ce --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:33:18.991Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 193825 + }, + "with-skill": { + "passed": false, + "duration": 124091 + }, + "with-skill-prompt": { + "passed": false, + "duration": 64354 + }, + "with-agents-md": { + "passed": false, + "duration": 311260 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue index 9e47be7..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-1/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/results.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/results.json new file mode 100644 index 0000000..4bf7cce --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:40:05.291Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 52665 + }, + "with-skill": { + "passed": false, + "duration": 73927 + }, + "with-skill-prompt": { + "passed": true, + "duration": 124527 + }, + "with-agents-md": { + "passed": false, + "duration": 155181 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue index 9e47be7..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-2/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/results.json b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/results.json new file mode 100644 index 0000000..685438c --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:47:37.387Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 69316 + }, + "with-skill": { + "passed": true, + "duration": 127117 + }, + "with-skill-prompt": { + "passed": true, + "duration": 163010 + }, + "with-agents-md": { + "passed": false, + "duration": 92652 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue index 9e47be7..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/structured-output/scenario-3/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/results.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/results.json new file mode 100644 index 0000000..5d757e2 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T19:56:27.489Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 122808 + }, + "with-skill": { + "passed": true, + "duration": 169079 + }, + "with-skill-prompt": { + "passed": true, + "duration": 100595 + }, + "with-agents-md": { + "passed": false, + "duration": 137617 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-1/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/results.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/results.json new file mode 100644 index 0000000..964537e --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:07:59.655Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 135281 + }, + "with-skill": { + "passed": false, + "duration": 279887 + }, + "with-skill-prompt": { + "passed": true, + "duration": 123630 + }, + "with-agents-md": { + "passed": false, + "duration": 153366 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-2/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/results.json b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/results.json new file mode 100644 index 0000000..a84bf53 --- /dev/null +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:11:52.591Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 61036 + }, + "with-skill": { + "passed": false, + "duration": 50433 + }, + "with-skill-prompt": { + "passed": false, + "duration": 56986 + }, + "with-agents-md": { + "passed": false, + "duration": 64480 + } + } + } +} diff --git a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue index 6bfc52b..cfde2b8 100644 --- a/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue +++ b/evals/suites/skills/vue-ai-apps/tool-calling/scenario-3/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json new file mode 100644 index 0000000..06a7112 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:18:18.652Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 32517 + }, + "with-skill": { + "passed": false, + "duration": 39148 + }, + "with-skill-prompt": { + "passed": false, + "duration": 52411 + }, + "with-agents-md": { + "passed": false, + "duration": 43861 + } + } + } +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue index 431d770..cfde2b8 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json new file mode 100644 index 0000000..c9c35e4 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:23:06.033Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 61546 + }, + "with-skill": { + "passed": false, + "duration": 56584 + }, + "with-skill-prompt": { + "passed": false, + "duration": 55688 + }, + "with-agents-md": { + "passed": false, + "duration": 113562 + } + } + } +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue index 431d770..cfde2b8 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json new file mode 100644 index 0000000..f386aaa --- /dev/null +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:29:20.377Z", + "tiers": { + "baseline": { + "passed": true, + "duration": 88390 + }, + "with-skill": { + "passed": true, + "duration": 95541 + }, + "with-skill-prompt": { + "passed": false, + "duration": 102177 + }, + "with-agents-md": { + "passed": true, + "duration": 88234 + } + } + } +} diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue index 431d770..cfde2b8 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/results.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/results.json new file mode 100644 index 0000000..e0006b2 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:31:50.640Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 28048 + }, + "with-skill": { + "passed": false, + "duration": 37966 + }, + "with-skill-prompt": { + "passed": false, + "duration": 56764 + }, + "with-agents-md": { + "passed": false, + "duration": 27483 + } + } + } +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue index 99dcb8a..cfde2b8 100644 --- a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-1/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/results.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/results.json new file mode 100644 index 0000000..1f8f48d --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:34:00.329Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 37971 + }, + "with-skill": { + "passed": false, + "duration": 29008 + }, + "with-skill-prompt": { + "passed": false, + "duration": 45627 + }, + "with-agents-md": { + "passed": false, + "duration": 17081 + } + } + } +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue index 99dcb8a..cfde2b8 100644 --- a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-2/src/App.vue @@ -1,7 +1,9 @@ diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/results.json b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/results.json new file mode 100644 index 0000000..ffd2976 --- /dev/null +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/results.json @@ -0,0 +1,23 @@ +{ + "sonnet": { + "timestamp": "2026-06-27T20:36:09.602Z", + "tiers": { + "baseline": { + "passed": false, + "duration": 40242 + }, + "with-skill": { + "passed": false, + "duration": 25288 + }, + "with-skill-prompt": { + "passed": false, + "duration": 33453 + }, + "with-agents-md": { + "passed": false, + "duration": 30288 + } + } + } +} diff --git a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue index 99dcb8a..cfde2b8 100644 --- a/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue +++ b/evals/suites/skills/vue-best-practices/vue-3-5-helpers/scenario-3/src/App.vue @@ -1,7 +1,9 @@ From c5a0c27d8cacae3919c219e81659add512c38b2c Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 17:14:04 -0400 Subject: [PATCH 13/31] chore(evals): record matrix results and harness/assertion findings --- .../scenario-1/results.json | 18 +++++------ .../scenario-2/results.json | 12 ++++---- .../scenario-3/results.json | 10 +++---- tasks/todo.md | 30 +++++++++++++++++++ 4 files changed, 50 insertions(+), 20 deletions(-) diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json index 06a7112..bcd28a6 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/results.json @@ -1,22 +1,22 @@ { "sonnet": { - "timestamp": "2026-06-27T20:18:18.652Z", + "timestamp": "2026-06-27T20:48:11.660Z", "tiers": { "baseline": { - "passed": false, - "duration": 32517 + "passed": true, + "duration": 80250 }, "with-skill": { - "passed": false, - "duration": 39148 + "passed": true, + "duration": 74962 }, "with-skill-prompt": { - "passed": false, - "duration": 52411 + "passed": true, + "duration": 99572 }, "with-agents-md": { - "passed": false, - "duration": 43861 + "passed": true, + "duration": 94034 } } } diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json index c9c35e4..57a58a3 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/results.json @@ -1,22 +1,22 @@ { "sonnet": { - "timestamp": "2026-06-27T20:23:06.033Z", + "timestamp": "2026-06-27T20:55:10.753Z", "tiers": { "baseline": { - "passed": false, - "duration": 61546 + "passed": true, + "duration": 155553 }, "with-skill": { "passed": false, - "duration": 56584 + "duration": 49023 }, "with-skill-prompt": { "passed": false, - "duration": 55688 + "duration": 82235 }, "with-agents-md": { "passed": false, - "duration": 113562 + "duration": 132279 } } } diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json index f386aaa..f7ec264 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/results.json @@ -1,22 +1,22 @@ { "sonnet": { - "timestamp": "2026-06-27T20:29:20.377Z", + "timestamp": "2026-06-27T21:13:12.872Z", "tiers": { "baseline": { "passed": true, - "duration": 88390 + "duration": 68341 }, "with-skill": { "passed": true, - "duration": 95541 + "duration": 100894 }, "with-skill-prompt": { "passed": false, - "duration": 102177 + "duration": 57396 }, "with-agents-md": { "passed": true, - "duration": 88234 + "duration": 855487 } } } diff --git a/tasks/todo.md b/tasks/todo.md index 17e1ced..fe07700 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -24,6 +24,36 @@ See `evals/README.md`. `vapor-mode` intentionally skipped (experimental until 3. - Verified: `pnpm eval --dry` (install+build) passes; arg validation; results.json I/O. - On `main` (PR #5) and `dev`. +## Eval matrix run — RESULTS (sonnet, 18 scenarios × 4 tiers) + +Tier pass totals: baseline 3/18 · with-skill 6/18 · **with-skill-prompt 11/18** · with-agents-md 5/18 + +Per reference (baseline → skill-prompt): +- vue-ai-apps/error-handling-and-abort: 0/3 → **3/3** +- vue-ai-apps/streaming-chat-ui: 0/3 → **3/3** +- vue-ai-apps/structured-output: 0/3 → 2/3 (s1 fails all tiers — brittle assertion) +- vue-ai-apps/tool-calling: 0/3 → 2/3 (s3 fails all tiers — brittle assertion) +- vue-best-practices/reactive-props: 3/3 → 1/3 (see below) +- vue-best-practices/vue-3-5-helpers: 0/3 → 0/3 (unexplained — see below) + +Headline: where the eval is well-formed (vue-ai-apps), the skill is a clean win — +baseline 0/12, skill-prompt 10/12. `with-skill` (implicit) is nondeterministic headless; +`with-agents-md` is weaker (embeds SKILL.md only, not reference files). + +Harness bugs found + fixed during the run (committed): +- skill tiers installed to .agents/skills (not loaded by claude -p) → copy to .claude/skills +- prompt didn't name the target file → agent wrote elsewhere, stub stayed empty +- eval.ts not discovered by vitest → copy in as eval.test.ts +- AI scenarios couldn't build (SDK not installed, ai@7 churn) → shim + vue-tsc-only + noImplicitAny off +- App.vue rendered with no props → required-prop components failed vue-tsc → minimal App shell + +Known eval-design issues NOT fixed (out of chosen scope): +- reactive-props: pattern is now default model behavior (baseline passes); `not /withDefaults/` + assertion is brittle — skill-informed agents mention withDefaults in a comment and fail. +- vue-3-5-helpers: all tiers fail though a hand-written correct useId component builds + passes; + needs a --verbose run to capture generated output and the real cause. +- structured-output/s1, tool-calling/s3: fail all tiers — brittle/scenario-specific assertions. + ## Out of scope — remaining follow-up - **Run the matrix** (billed, user-triggered): `pnpm eval --all` → 4 tiers × 3 models per From ed507d01ce5c7e18a25f5c4d68dc8f47161b9a5b Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 17:27:57 -0400 Subject: [PATCH 14/31] fix(evals): strip comments before reactive-props negative assertion A skill-informed agent writes a comment like '// no withDefaults() needed'; the previous /withDefaults\(/ negative matched the comment and failed correct code. Strip JS/HTML comments before the positive/negative checks. --- .../reactive-props-destructure/scenario-1/eval.ts | 15 ++++++++++----- .../reactive-props-destructure/scenario-2/eval.ts | 15 ++++++++++----- .../reactive-props-destructure/scenario-3/eval.ts | 15 ++++++++++----- 3 files changed, 30 insertions(+), 15 deletions(-) diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts index 78fb167..e9d28f9 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-1/eval.ts @@ -3,14 +3,19 @@ import { readFileSync } from "fs"; import { join } from "path"; const src = readFileSync(join(process.cwd(), "src/components/UserCard.vue"), "utf-8"); +// Strip comments: a comment that *mentions* withDefaults (the skill teaches +// "destructure replaces withDefaults") must not trip the negative check. +const code = src + .replace(/\/\/.*$/gm, "") + .replace(/\/\*[\s\S]*?\*\//g, "") + .replace(//g, ""); test("uses destructure defaults, not withDefaults", () => { - expect(src).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); - expect(src).not.toMatch(/withDefaults\s*\(/); + expect(code).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); + expect(code).not.toMatch(/withDefaults\s*\(/); }); test("preserves reactivity with a getter across the function boundary", () => { - // watch the prop via a getter, not the bare value - expect(src).toMatch(/watch\(\s*\(\)\s*=>/); - expect(src).not.toMatch(/watch\(\s*id\b/); + expect(code).toMatch(/watch\(\s*\(\)\s*=>/); + expect(code).not.toMatch(/watch\(\s*id\b/); }); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts index 78fb167..e9d28f9 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-2/eval.ts @@ -3,14 +3,19 @@ import { readFileSync } from "fs"; import { join } from "path"; const src = readFileSync(join(process.cwd(), "src/components/UserCard.vue"), "utf-8"); +// Strip comments: a comment that *mentions* withDefaults (the skill teaches +// "destructure replaces withDefaults") must not trip the negative check. +const code = src + .replace(/\/\/.*$/gm, "") + .replace(/\/\*[\s\S]*?\*\//g, "") + .replace(//g, ""); test("uses destructure defaults, not withDefaults", () => { - expect(src).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); - expect(src).not.toMatch(/withDefaults\s*\(/); + expect(code).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); + expect(code).not.toMatch(/withDefaults\s*\(/); }); test("preserves reactivity with a getter across the function boundary", () => { - // watch the prop via a getter, not the bare value - expect(src).toMatch(/watch\(\s*\(\)\s*=>/); - expect(src).not.toMatch(/watch\(\s*id\b/); + expect(code).toMatch(/watch\(\s*\(\)\s*=>/); + expect(code).not.toMatch(/watch\(\s*id\b/); }); diff --git a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts index 78fb167..e9d28f9 100644 --- a/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts +++ b/evals/suites/skills/vue-best-practices/reactive-props-destructure/scenario-3/eval.ts @@ -3,14 +3,19 @@ import { readFileSync } from "fs"; import { join } from "path"; const src = readFileSync(join(process.cwd(), "src/components/UserCard.vue"), "utf-8"); +// Strip comments: a comment that *mentions* withDefaults (the skill teaches +// "destructure replaces withDefaults") must not trip the negative check. +const code = src + .replace(/\/\/.*$/gm, "") + .replace(/\/\*[\s\S]*?\*\//g, "") + .replace(//g, ""); test("uses destructure defaults, not withDefaults", () => { - expect(src).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); - expect(src).not.toMatch(/withDefaults\s*\(/); + expect(code).toMatch(/const\s*\{[^}]*=[^}]*\}\s*=\s*defineProps/); + expect(code).not.toMatch(/withDefaults\s*\(/); }); test("preserves reactivity with a getter across the function boundary", () => { - // watch the prop via a getter, not the bare value - expect(src).toMatch(/watch\(\s*\(\)\s*=>/); - expect(src).not.toMatch(/watch\(\s*id\b/); + expect(code).toMatch(/watch\(\s*\(\)\s*=>/); + expect(code).not.toMatch(/watch\(\s*id\b/); }); From 5428c300c1d568c47dcd9ce7c87c19ef1de1237d Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 18:03:07 -0400 Subject: [PATCH 15/31] chore(evals): corrected matrix results after assertion + harness fixes --- tasks/todo.md | 60 ++++++++++++++++++++++++++------------------------- 1 file changed, 31 insertions(+), 29 deletions(-) diff --git a/tasks/todo.md b/tasks/todo.md index fe07700..dd96139 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -24,35 +24,37 @@ See `evals/README.md`. `vapor-mode` intentionally skipped (experimental until 3. - Verified: `pnpm eval --dry` (install+build) passes; arg validation; results.json I/O. - On `main` (PR #5) and `dev`. -## Eval matrix run — RESULTS (sonnet, 18 scenarios × 4 tiers) - -Tier pass totals: baseline 3/18 · with-skill 6/18 · **with-skill-prompt 11/18** · with-agents-md 5/18 - -Per reference (baseline → skill-prompt): -- vue-ai-apps/error-handling-and-abort: 0/3 → **3/3** -- vue-ai-apps/streaming-chat-ui: 0/3 → **3/3** -- vue-ai-apps/structured-output: 0/3 → 2/3 (s1 fails all tiers — brittle assertion) -- vue-ai-apps/tool-calling: 0/3 → 2/3 (s3 fails all tiers — brittle assertion) -- vue-best-practices/reactive-props: 3/3 → 1/3 (see below) -- vue-best-practices/vue-3-5-helpers: 0/3 → 0/3 (unexplained — see below) - -Headline: where the eval is well-formed (vue-ai-apps), the skill is a clean win — -baseline 0/12, skill-prompt 10/12. `with-skill` (implicit) is nondeterministic headless; -`with-agents-md` is weaker (embeds SKILL.md only, not reference files). - -Harness bugs found + fixed during the run (committed): -- skill tiers installed to .agents/skills (not loaded by claude -p) → copy to .claude/skills -- prompt didn't name the target file → agent wrote elsewhere, stub stayed empty -- eval.ts not discovered by vitest → copy in as eval.test.ts -- AI scenarios couldn't build (SDK not installed, ai@7 churn) → shim + vue-tsc-only + noImplicitAny off -- App.vue rendered with no props → required-prop components failed vue-tsc → minimal App shell - -Known eval-design issues NOT fixed (out of chosen scope): -- reactive-props: pattern is now default model behavior (baseline passes); `not /withDefaults/` - assertion is brittle — skill-informed agents mention withDefaults in a comment and fail. -- vue-3-5-helpers: all tiers fail though a hand-written correct useId component builds + passes; - needs a --verbose run to capture generated output and the real cause. -- structured-output/s1, tool-calling/s3: fail all tiers — brittle/scenario-specific assertions. +## Eval matrix run — RESULTS (sonnet, 18 scenarios x 4 tiers) — CORRECTED + +Tier pass totals: baseline 5/18 | with-skill 10/18 | **with-skill-prompt 16/18** | with-agents-md 9/18 + +Per reference (baseline -> skill-prompt): +- vue-ai-apps/error-handling-and-abort: 0/3 -> **3/3** +- vue-ai-apps/streaming-chat-ui: 0/3 -> **3/3** +- vue-ai-apps/structured-output: 0/3 -> 2/3 (s1: generation variance, assertion validated fair) +- vue-ai-apps/tool-calling: 0/3 -> 2/3 (s3: generation variance, assertion validated fair) +- vue-best-practices/reactive-props: 2/3 -> **3/3** (after assertion + App.vue fixes) +- vue-best-practices/vue-3-5-helpers: 3/3 -> **3/3** (after App.vue fix) + +Interpretation: +- vue-ai-apps: strong, clean win. The model cannot produce AI SDK v5 patterns without + the skill (baseline 0/12); with the skill invoked it succeeds (10/12). The skill earns + its keep most here (recent/niche API). +- vue-best-practices: reactive-props-destructure and useId are mainstream enough that + baseline often already passes; the skill keeps 6/6 and does no harm, but the baseline gap + is small. Skill value is highest for newer APIs, lower for patterns the model already knows. + +Harness bugs found+fixed by running (all committed): +- skill tiers -> copy local skill into .claude/skills (npx skills add lands in .agents, not loaded) +- name the target file in the prompt (empty stub gave no signal) +- copy eval.ts in as eval.test.ts (vitest discovery) +- AI scenarios: shim SDK modules + vue-tsc-only build + noImplicitAny off (ai@7 churn) +- App.vue shell no longer renders (required-prop components failed vue-tsc) +- reactive-props eval.ts: strip comments before the withDefaults negative (skill-comment false positive) + +with-skill (implicit) is nondeterministic headless; with-agents-md embeds SKILL.md only +(no reference files), so it is the weakest skill tier. + ## Out of scope — remaining follow-up From 4ef2fc931debf312745dccbfe296beca57f8e18f Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 18:03:34 -0400 Subject: [PATCH 16/31] chore(tasks): refresh stale follow-up notes --- tasks/todo.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/tasks/todo.md b/tasks/todo.md index dd96139..5e349af 100644 --- a/tasks/todo.md +++ b/tasks/todo.md @@ -58,12 +58,14 @@ with-skill (implicit) is nondeterministic headless; with-agents-md embeds SKILL. ## Out of scope — remaining follow-up -- **Run the matrix** (billed, user-triggered): `pnpm eval --all` → 4 tiers × 3 models per - scenario, recorded to `results.json`. Requires the `claude` CLI + API budget. - The four LLM tiers are not exercised in CI and are unverified beyond `--dry`. -- `with-skill*` tiers install via `npx skills add $VUE_SKILLS_SOURCE` - (default `Pythoughts-labs/vue3-best-practices`) — confirm resolution in the target env. +- Matrix run for **sonnet** is complete (above). haiku/opus not run (billed, user-triggered). +- `structured-output/s1` and `tool-calling/s3` are recorded misses (generation variance, fair + assertions). Could be re-rolled but not "fixed". +- `with-skill*` tiers copy the local `skills/` into `.claude/skills` (override dir via + `VUE_SKILLS_DIR`). The earlier `npx skills add` approach was dropped — it installs to + `.agents/skills`, which headless `claude -p` does not load. - AI-SDK doc items to confirm against an installed `@ai-sdk/vue` (useObject export; server return form). + Note: `ai@7` / `@ai-sdk/vue@4` are current — the skill documents the v5 API and may need a refresh. ## Verify-before-trust notes (flagged in the content, confirm against installed SDK) From 8524e1c44fd8f12c9ad5ed956716c57ecd61c802 Mon Sep 17 00:00:00 2001 From: Mohamed Elkholy Date: Sat, 27 Jun 2026 18:09:21 -0400 Subject: [PATCH 17/31] docs(vue-ai-apps): align with ai@7 / @ai-sdk/vue@4 (skill v1.1.0) Verified against installed ai@7.0.4 / @ai-sdk/vue@4.0.4: - useObject: import as experimental_useObject (no plain useObject export) - client tools: addToolOutput (addToolResult is the deprecated alias) - confirmed unchanged and kept: useChat surface, status union ('submitted'|'streaming'|'ready'|'error'), parts, sendMessage, inputSchema, stepCountIs, streamObject, toUIMessageStreamResponse - version framing: v5+ (verified through v7) instead of hard 'v5' --- skills/vue-ai-apps/SKILL.md | 6 +++--- .../references/error-handling-and-abort.md | 4 ++-- skills/vue-ai-apps/references/streaming-chat-ui.md | 4 ++-- skills/vue-ai-apps/references/structured-output.md | 14 ++++++-------- skills/vue-ai-apps/references/tool-calling.md | 6 +++--- 5 files changed, 16 insertions(+), 18 deletions(-) diff --git a/skills/vue-ai-apps/SKILL.md b/skills/vue-ai-apps/SKILL.md index e55e035..edcfad4 100644 --- a/skills/vue-ai-apps/SKILL.md +++ b/skills/vue-ai-apps/SKILL.md @@ -1,7 +1,7 @@ --- name: vue-ai-apps description: "Building AI/LLM and agent apps with Vue 3 and Nuxt: streaming chat UIs, the Vercel AI SDK (`ai` + `@ai-sdk/vue`), `useChat`, tool calling, structured output, and abort/error handling. Load for AI chatbots, assistant UIs, LLM streaming, or agent frontends in Vue or Nuxt." -version: "1.0.0" +version: "1.1.0" license: MIT author: github.com/Pythoughts-labs --- @@ -16,14 +16,14 @@ Assumes the foundations in `vue-best-practices` (Composition API, ` +``` + +## Upgrade Notes + +1. **v4 → v5 is drop-in for plain Router 4 projects.** Their only packaging change: the IIFE build no longer bundles `@vue/devtools-api` (v8). +2. **`unplugin-vue-router` users must migrate.** Remove the dependency, then repoint its imports at the core package: + - `unplugin-vue-router/vite` → `vue-router/vite` + - `unplugin-vue-router/data-loaders/*` → `vue-router/experimental` + - `unplugin-vue-router` → `vue-router/unplugin` + - `unplugin-vue-router/volar/*` → `vue-router/volar/*` + - drop `unplugin-vue-router/client` from `tsconfig.json`'s type references + + Route generation, `definePage()`, and typed `RouteNamedMap` now ship from the core package. +3. **`next()` deprecation**: v5.0.3 added a deprecation warning for the `next()` callback. See [router-navigation-guard-next-deprecated](router-navigation-guard-next-deprecated.md). +4. **Data loaders** (`defineLoader`, `defineBasicLoader`) are experimental — the API may change between releases. Use them for feedback, not as a load-bearing pattern. +5. **`reroute()`** replaces the deprecated `NavigationResult` for programmatic redirect decisions inside guards. + +## Key Points + +1. **Upgrade freely without `unplugin-vue-router`** — v5 breaks nothing from plain v4; treat it as v4 plus file-based routing in core +2. **One package** — file-based routing no longer needs `unplugin-vue-router`, but existing users must remove it and repoint imports (see Upgrade Notes) +3. **Return-based guards** — `next()` warns in v5; migrate during the upgrade +4. **Keep data loaders experimental** — don't standardize on them yet + +## Reference +- [Migrating to Vue Router 5](https://router.vuejs.org/guide/migration/v4-to-v5) +- [Vue Router Releases](https://github.com/vuejs/router/releases) diff --git a/skills/vue-router-best-practices/reference/router-navigation-guard-next-deprecated.md b/skills/vue-router-best-practices/reference/router-navigation-guard-next-deprecated.md index 9cabf03..2f4bf13 100644 --- a/skills/vue-router-best-practices/reference/router-navigation-guard-next-deprecated.md +++ b/skills/vue-router-best-practices/reference/router-navigation-guard-next-deprecated.md @@ -8,7 +8,7 @@ tags: [vue3, vue-router, navigation-guards, migration, async] # Vue Router Navigation Guard next() Function Deprecated -**Impact: HIGH** - The third `next()` argument in navigation guards is deprecated in Vue Router 4. While still supported for backward compatibility, using it incorrectly is one of the most common sources of bugs: calling it multiple times, forgetting to call it, or calling it conditionally without proper logic. +**Impact: HIGH** - The third `next()` argument in navigation guards is deprecated. In Vue Router 4 it was merely discouraged; since Vue Router 5.0.3 the `next()` callback itself emits a deprecation warning. Using it incorrectly is one of the most common sources of bugs: calling it multiple times, forgetting to call it, or calling it conditionally without proper logic. ## Task Checklist @@ -49,7 +49,7 @@ router.beforeEach(async (to, from, next) => { ## Solution: Use Return-Based Guards ```javascript -// CORRECT: Return-based syntax (modern Vue Router 4+) +// CORRECT: Return-based syntax (modern Vue Router 4+, required to avoid warnings in 5) router.beforeEach((to, from) => { if (!isAuthenticated) { return '/login' // Redirect diff --git a/skills/vue-router-best-practices/reference/router-use-vue-router-for-production.md b/skills/vue-router-best-practices/reference/router-use-vue-router-for-production.md index 1e60304..5c50de2 100644 --- a/skills/vue-router-best-practices/reference/router-use-vue-router-for-production.md +++ b/skills/vue-router-best-practices/reference/router-use-vue-router-for-production.md @@ -149,7 +149,7 @@ createApp(App) ## Modern Vue Router Features (2025+) ```javascript -// Data Loading API (Vue Router 4.2+) +// Data Loading API (experimental since Vue Router 4.2, still experimental in 5 — do not standardize on it) const routes = [ { path: '/users/:id', diff --git a/skills/vue-testing-best-practices/SKILL.md b/skills/vue-testing-best-practices/SKILL.md index 92fc0bd..f0a611b 100644 --- a/skills/vue-testing-best-practices/SKILL.md +++ b/skills/vue-testing-best-practices/SKILL.md @@ -1,9 +1,9 @@ --- name: vue-testing-best-practices -version: 1.0.0 +version: 1.1.0 license: MIT -author: github.com/Pythoughts-labs -description: Use for Vue.js testing. Covers Vitest, Vue Test Utils, component testing, mocking, testing patterns, and Playwright for E2E testing. +author: github.com/PyModel +description: Use for Vue.js testing. Covers Vitest 4, Vue Test Utils, component testing, mocking, testing patterns, and Playwright for E2E testing. --- Vue.js testing best practices, patterns, and common gotchas. diff --git a/tasks/research-2026-08-16-vue-ecosystem.md b/tasks/research-2026-08-16-vue-ecosystem.md new file mode 100644 index 0000000..a17b68a --- /dev/null +++ b/tasks/research-2026-08-16-vue-ecosystem.md @@ -0,0 +1,84 @@ +# Vue Ecosystem Research — 2026-08-16 + +Primary-source snapshot of current Vue ecosystem versions and official guidance, +gathered to validate the skills in this repo and plan updates. All version +numbers were read directly from the npm registry / unpkg on 2026-08-16. + +## Version matrix (npm registry, 2026-08-16) + +| Package | Latest | Source | +|---|---|---| +| vue | 3.5.41 (stable) / 3.6.0-rc.4 (prerelease) | [registry](https://registry.npmjs.org/vue/latest), [dist-tags](https://registry.npmjs.org/vue) | +| pinia | 4.0.3 | [registry](https://registry.npmjs.org/pinia/latest) | +| vue-router | 5.2.0 | [registry](https://registry.npmjs.org/vue-router/latest) | +| vitest | 4.1.10 | [registry](https://registry.npmjs.org/vitest/latest) | +| @vue/test-utils | 2.4.11 | [unpkg](https://unpkg.com/@vue/test-utils/package.json) | +| playwright | 1.62.1 | [registry](https://registry.npmjs.org/playwright/latest) | +| vite | 8.2.1 | [registry](https://registry.npmjs.org/vite/latest) | +| @ai-sdk/vue | 4.0.66 | [registry](https://registry.npmjs.org/@ai-sdk/vue/latest) | +| @upstash/context7-mcp | 4.0.2 | [registry](https://registry.npmjs.org/@upstash/context7-mcp/latest), [GitHub](https://github.com/upstash/context7) | + +## Official Vue guidance (vuejs.org) + +**Style guide Priority A (Essential) — unchanged and matches this repo's evals:** +multi-word component names, detailed prop definitions, keyed `v-for`, avoid +`v-if` with `v-for`, component-scoped styling. Priorities B–D categories +unchanged. Source: [vuejs.org/style-guide](https://vuejs.org/style-guide/), +[rules-essential](https://vuejs.org/style-guide/rules-essential.md). The style +guide is now also served as Markdown for LLMs at `/style-guide.md`. + +**Vue 3.6 / Vapor Mode status:** stable is still 3.5.x; the newest release post +on the official blog remains [Vue 3.5 (Sept 2024)](https://blog.vuejs.org/). +3.6 is at [v3.6.0-rc.4](https://github.com/vuejs/core/releases/tag/v3.6.0-rc.4) +on the `rc` dist-tag ([changelog](https://github.com/vuejs/core/blob/minor/CHANGELOG.md)); +Vapor is opt-in and "feature-complete but still unstable" per those changelogs, and +[vuejs.org/about/releases](https://vuejs.org/about/releases) states that +prerelease APIs may change before stabilising. So 3.6/Vapor is not yet stable +guidance — corroborated by Context7's `/vuejs/docs` index, which still describes +Vapor as an exploratory strategy. + +**Vue Router 5** — released Jan 2026, now at 5.2.0. A "transition release": for +Vue Router 4 users *without* file-based routing, +[upgrading requires no code changes](https://router.vuejs.org/guide/migration/v4-to-v5). +Projects on `unplugin-vue-router` must remove that dependency, repoint its +imports at the core package (`vue-router/vite`, `vue-router/experimental`, +`vue-router/unplugin`, `vue-router/volar/*`), and drop +`unplugin-vue-router/client` from `tsconfig.json`. +Key deltas ([releases](https://github.com/vuejs/router/releases)): +- unplugin-vue-router (file-based routing) merged into the core package +- `next()` callback in navigation guards now emits a deprecation warning + (return-based guards are the path forward) +- `reroute()` added; `NavigationResult` deprecated (v5.0.3) +- experimental data loaders, typed `definePage` improvements (v5.1.0) +- devtools/@vue/devtools-api v8 alignment; Pinia 4 allowed (v5.2.0) + +**Pinia 4** — now at 4.0.3. "Only technically breaking changes": +[ESM-only distribution and `@vue/devtools-api` v8 as a peer dependency](https://github.com/vuejs/pinia/releases). +The store API (`defineStore`, state/getters/actions, `storeToRefs`) is +unchanged from v3. Known issue: devtools-api on Node 25 +([#3065](https://github.com/vuejs/pinia/issues/3065)). v2→v3 migration +background: [pinia.vuejs.org](https://pinia.vuejs.org/cookbook/migration-v2-v3.html). + +**Context7 MCP** — [@upstash/context7-mcp is at 4.0.2](https://registry.npmjs.org/@upstash/context7-mcp/latest); +hosted endpoint `https://mcp.context7.com/mcp` with Bearer API key +([GitHub](https://github.com/upstash/context7)). Note: its `/vuejs/docs` index +still predates Vue 3.6 beta content. + +## Implications for this repo (pre-migration findings) + +These are the gaps found *before* the Router 5 / Pinia 4 / Vitest 4 update landed +in this repo. They are kept as the record of what the snapshot motivated, not as +an open to-do list — items 1–3 were addressed in that same change. + +1. **vue-router-best-practices** targets "Vue Router 4" — Router 5 is current + and adds file-based routing, return-based guards (next() deprecating), and + data loaders. Largest content gap. +2. **vue-pinia-best-practices** — Pinia 4 API unchanged; needs version + references and an ESM/peer-dep install note only. +3. **vue-testing-best-practices** — Vitest 4 is current; eval fixtures and + examples should be verified/bumped (plus @vue/test-utils 2.4.11, + Playwright 1.62.1). +4. **vue-best-practices** — 3.5 coverage is current; keep Vapor framed as + beta/experimental (stable docs haven't shipped it). +5. **vue-ai-apps** — @ai-sdk/vue@4 alignment is current (4.0.66). +6. **README badge "Vue 3.5+"** remains accurate for stable guidance. From dc151ade5d58bb881f5ae0e6a361025573627509 Mon Sep 17 00:00:00 2001 From: elkaix Date: Sun, 16 Aug 2026 18:25:42 -0400 Subject: [PATCH 27/31] chore(release): v0.3.2 Bump the MCP server package to 0.3.2. Also drop pnpm-lock.yaml, an empty stub with no importers that a stray 'pnpm typecheck' invocation generated; the root package has no dependencies and pnpm is not used there. --- mcp/package-lock.json | 4 ++-- mcp/package.json | 2 +- pnpm-lock.yaml | 9 --------- 3 files changed, 3 insertions(+), 12 deletions(-) delete mode 100644 pnpm-lock.yaml diff --git a/mcp/package-lock.json b/mcp/package-lock.json index bece8a6..bba0fd9 100644 --- a/mcp/package-lock.json +++ b/mcp/package-lock.json @@ -1,12 +1,12 @@ { "name": "@pymodel/vue-skills-mcp", - "version": "0.3.1", + "version": "0.3.2", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@pymodel/vue-skills-mcp", - "version": "0.3.1", + "version": "0.3.2", "license": "MIT", "dependencies": { "@modelcontextprotocol/sdk": "^1.12.0", diff --git a/mcp/package.json b/mcp/package.json index 3220a1b..c486781 100644 --- a/mcp/package.json +++ b/mcp/package.json @@ -1,6 +1,6 @@ { "name": "@pymodel/vue-skills-mcp", - "version": "0.3.1", + "version": "0.3.2", "type": "module", "description": "MCP server exposing the Vue 3 best-practice skills so any MCP coding agent can fetch them automatically on Vue work.", "author": "Mohamed Elkholy (elkaix)", diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml deleted file mode 100644 index 9b60ae1..0000000 --- a/pnpm-lock.yaml +++ /dev/null @@ -1,9 +0,0 @@ -lockfileVersion: '9.0' - -settings: - autoInstallPeers: true - excludeLinksFromLockfile: false - -importers: - - .: {} From 2a63041be819fc9cb48b4af0bb90cdcb64419abf Mon Sep 17 00:00:00 2001 From: elkaix Date: Sun, 16 Aug 2026 18:27:21 -0400 Subject: [PATCH 28/31] ci(sync-to-main): don't fail on expected merge conflicts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The merge exists only to record dev as a second parent; its content is immediately discarded by 'git restore --source=ORIG_HEAD' and replaced with the whitelisted paths from dev. main and dev diverge, so the merge reliably conflicts, and the step's 'bash -e' turned that into a job failure — the workflow has never completed a sync. Verified on a scratch clone: the resulting commit has both parents, the whitelisted paths are byte-identical to dev, no conflict markers survive, and nothing from tasks/ or mcp/ leaks into main. --- .github/workflows/sync-to-main.yml | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/sync-to-main.yml b/.github/workflows/sync-to-main.yml index 39e6a47..b582914 100644 --- a/.github/workflows/sync-to-main.yml +++ b/.github/workflows/sync-to-main.yml @@ -41,8 +41,11 @@ jobs: BRANCH="sync/dev-to-main-$(date +%Y%m%d%H%M%S)" git checkout -b "$BRANCH" - # Create a merge commit, but don't commit yet - git merge --no-ff --no-commit origin/dev + # Create a merge commit, but don't commit yet. + # This is only here to record dev as a second parent — the merge's + # own content is thrown away by the restore below, so conflicts are + # expected and must not fail the step (the shell runs with -e). + git merge --no-ff --no-commit origin/dev || true # Reset everything back to main... git restore --source=ORIG_HEAD --staged --worktree -- . From 3a2dc06fa3640956f0697f13a4a73ab07bd285e4 Mon Sep 17 00:00:00 2001 From: elkaix Date: Sun, 16 Aug 2026 18:30:20 -0400 Subject: [PATCH 29/31] ci(sync-to-main): sync mcp/ to main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit .github/ is synced, so main runs the MCP server job — but mcp/ was excluded from the whitelist, so that job ran against main's stale tree: package name @pythoughts/vue-skills-mcp, version 0.1.0, and a lockfile still carrying the fast-uri and ip-address advisories. The audit step failed on every sync PR. --- .github/workflows/sync-to-main.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/sync-to-main.yml b/.github/workflows/sync-to-main.yml index b582914..e89c87c 100644 --- a/.github/workflows/sync-to-main.yml +++ b/.github/workflows/sync-to-main.yml @@ -51,8 +51,11 @@ jobs: git restore --source=ORIG_HEAD --staged --worktree -- . # ...then re-apply only whitelisted paths from dev + # mcp/ must stay in this list: .github/ is synced, so main runs the + # MCP server job, and without mcp/ that job tests a stale tree (old + # package name, old version, unpatched lockfile) and always fails. git restore --source=origin/dev --staged --worktree -- \ - skills/ evals/ package.json .github/ .claude-plugin/ README.md AGENTS.md CLAUDE.md LICENSE + skills/ evals/ mcp/ package.json .github/ .claude-plugin/ README.md AGENTS.md CLAUDE.md LICENSE git add -A if git diff --staged --quiet; then From 64a4f7a5868fd19ecef719e88a5318ce5100acf3 Mon Sep 17 00:00:00 2001 From: elkaix Date: Sun, 16 Aug 2026 18:34:01 -0400 Subject: [PATCH 30/31] docs: skills.sh + npm badges, npx MCP install, fix stale router description - marketplace.json still advertised vue-router-best-practices as 'Vue Router 4 patterns'; the skill now covers 4/5 and file-based routing, matching its SKILL.md description. - Add skills.sh and npm version badges. - Document 'npx -y @pymodel/vue-skills-mcp' as the no-clone MCP path, verified against the published 0.3.2 package (initialize + tools/list respond). --- .claude-plugin/marketplace.json | 2 +- README.md | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 4acde1e..d16f2bc 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -39,7 +39,7 @@ { "name": "vue-router-best-practices", "source": "./skills/vue-router-best-practices", - "description": "Vue Router 4 patterns, navigation guards, route params, and route-component lifecycle interactions." + "description": "Vue Router 4/5 patterns, navigation guards, route params, file-based routing, and route-component lifecycle interactions." }, { "name": "vue-testing-best-practices", diff --git a/README.md b/README.md index 80750ff..fea26eb 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,8 @@

    CI + skills.sh + npm License: MIT Vue 3.5+ Skills: 8 @@ -63,6 +65,14 @@ Without the prefix, triggering depends on how closely the prompt matches a skill For agents that consume MCP instead of the skills registry, [`mcp/`](mcp/) exposes every skill as MCP tools. The agent calls `vue_best_practices` when it detects Vue work, then pulls individual reference files on demand. +The server is published as [`@pymodel/vue-skills-mcp`](https://www.npmjs.com/package/@pymodel/vue-skills-mcp) with the skills bundled, so no clone is needed: + +```bash +claude mcp add vue-skills -- npx -y @pymodel/vue-skills-mcp +``` + +To run it from a local clone instead: + ```bash cd mcp && npm install ``` From bd1388ead8d1bb09969f0704bb58e0f3babbd273 Mon Sep 17 00:00:00 2001 From: elkaix Date: Sun, 16 Aug 2026 18:34:44 -0400 Subject: [PATCH 31/31] docs: add npm downloads badge, drop hardcoded skills count The skills.sh badge renders 'Skills: N' from live install telemetry, which collided with the static 'Skills: 8' badge next to it. The static count was also a hardcoded number that drifts whenever a skill is added. --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index fea26eb..19116bd 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ CI skills.sh npm + npm downloads License: MIT Vue 3.5+ - Skills: 8 MCP server TypeScript PRs welcome