Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions .github/skills/ui-scenario-validation/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
---
name: ui-scenario-validation
description: Use when reproducing a UI bug or verifying a fix by driving a real VS Code window end to end and capturing evidence. Launches VS Code through the automation MCP, performs the scenario as a user would, and produces a video, per-step screenshots, a Playwright trace, and an HTML report to attach to an issue or pull request.
---

# UI Scenario Validation

Drives a real VS Code instance through a scenario and records reproducible evidence.

Use this to reproduce a reported bug, to show that a fix works, or to attach a recording to a
test-plan item. For deterministic regression coverage that runs on every build, write a smoke test
instead (see the `smoke-tests` skill) — this skill is for one-off, issue-derived validation.

## Prerequisites

```bash
npm install # once
npm run electron # download the Electron runtime
npm run transpile-client # or `npm run watch` in another terminal
npm --prefix test/mcp run compile
```

The automation MCP server is `test/mcp` (`out/stdio.js`). Add it to your MCP configuration so the
`vscode_automation_*` tools are available; append `--web --headless` to the args to record the web
build instead of Electron.

## Record a clean capture

Set `VSCODE_EVIDENCE_CLEAN_CAPTURE=1` in the MCP server environment.

Evidence capture can draw a step banner into the window it is recording. That banner is part of the
DOM of the product under test, so it can shift layout and affect focus and selectors. With clean
capture enabled the recording shows unmodified UI, and step boundaries are still recorded in
`manifest.json` with timestamps and screenshots.

## Run a scenario

1. Choose a **disposable** workspace folder. Never point a scenario at real work: the run types,
clicks, and may modify files. Nothing in the recording should contain credentials, tokens, or
private conversations.
2. Call `vscode_automation_evidence_start` **before** any other automation tool, passing the
scenario id, title, the source issue URL, and the workspace path. It launches VS Code with an
isolated profile and starts video plus tracing.
3. For each step:
- call `vscode_automation_evidence_step` with `status: started` and a one-line intent;
- inspect the accessibility snapshot before choosing a selector;
- prefer feature-specific automation tools, then semantic selectors, then coordinates;
- perform the action the way a user would;
- **validate through a separate observable signal** — an action completing is not a result;
- call the step again with `passed`, `failed`, or `skipped` plus concise details.
4. Call `vscode_automation_evidence_finish` with the overall outcome. This stops VS Code and
finalizes the video, trace, screenshots, `manifest.json`, and `report.html`.

Stop at the first failed required step unless the scenario says otherwise, and mark steps that need
unavailable hardware, accounts, or services as `skipped` rather than passed.

## What makes evidence trustworthy

- Assert on DOM state, accessibility, focus, or text — screenshots support a claim, they do not
establish one.
- If the bug is a race, make the timing explicit (for example a forced delay or a repeated loop) so
the recording shows the window in which it occurs rather than relying on luck.
- Record the failing behavior before the fix when you can. A passing run alone does not show that
the scenario would have caught the bug.

## Report

Evidence is written to `.build/vscode-playwright-mcp/evidence/<run-id>/`:

| File | Contents |
|------|----------|
| `report.html` | Step table, outcome, embedded video |
| `manifest.json` | Step timestamps, statuses, artifact paths, environment |
| `videos/` | Screen recording of the run |
| `*.png` | Per-step screenshots |
| `logs/` | Playwright trace, window and server logs |

Summarize the outcome, list failed or skipped steps, link `report.html`, and state the OS, VS Code
commit, and source issue. Attach the video to the issue or pull request by dragging it into the
comment box.

## Automated validation on a pull request

`microsoft/vscode-engineering` runs the same harness in CI: labelling a pull request
`~requires-ui-validation` researches the change, runs a checked-in scenario adapter against the
exact merge candidate, and posts the per-step result with chaptered video. Use this skill when a
scenario is not yet covered there, or to iterate locally before proposing one.
4 changes: 0 additions & 4 deletions build/lib/stylelint/vscode-known-variables.json
Original file line number Diff line number Diff line change
Expand Up @@ -1178,10 +1178,6 @@
"--collapse-from-width",
"--slide-from-x",
"--slide-from-y",
"--omni-icon-column",
"--omni-input-editor-background",
"--omni-rail",
"--omni-row-gap",
"--vg-w1",
"--vg-h1",
"--vg-w2",
Expand Down
85 changes: 16 additions & 69 deletions extensions/github-authentication/src/github.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,8 @@ import { crypto } from './node/crypto';
import { TIMED_OUT_ERROR, USER_CANCELLATION_ERROR } from './common/errors';
import { GitHubSocialSignInProvider, isSocialSignInProvider } from './flows';

/**
* The stored (JSON) form of a vscode.Uri pointing to the account's avatar.
*/
interface StoredAccountIcon {
scheme: string;
authority?: string;
path?: string;
query?: string;
fragment?: string;
}
// `vscode` doesn't publicly export `UriComponents`, so derive the exact shape from `Uri.from`.
type UriComponents = Parameters<typeof vscode.Uri.from>[0];

interface SessionData {
id: string;
Expand All @@ -33,31 +25,12 @@ interface SessionData {
// Unfortunately, for some time the id was a number, so we need to support both.
// This can be removed once we are confident that all users have migrated to the new id.
id: string | number;
// `undefined` means the avatar has not been looked up yet, `null` means a lookup
// completed and found no avatar, and a `StoredAccountIcon` is a resolved avatar.
icon?: StoredAccountIcon | null;
icon?: UriComponents;
};
scopes: string[];
accessToken: string;
}

/**
* Whether a stored session's account icon still needs to be looked up.
*/
export function needsAccountIconLookup(session: SessionData): boolean {
return !session.account || session.account.icon === undefined;
}

/**
* Serializes an account icon for storage, using `null` to mark a completed lookup that found no avatar.
*/
export function serializeAccountIcon(icon: vscode.Uri | undefined, hasNoAvatar: boolean): StoredAccountIcon | null | undefined {
if (icon) {
return { scheme: icon.scheme, authority: icon.authority, path: icon.path, query: icon.query, fragment: icon.fragment };
}
return hasNoAvatar ? null : undefined;
}

export enum AuthProviderType {
github = 'github',
githubEnterprise = 'github-enterprise'
Expand Down Expand Up @@ -165,7 +138,6 @@ export class GitHubAuthenticationProvider implements vscode.AuthenticationProvid
private readonly _telemetryReporter: ExperimentationTelemetry;
private readonly _keychain: Keychain;
private readonly _accountsSeen = new Set<string>();
private readonly _sessionsWithoutAvatars = new WeakSet<vscode.AuthenticationSession>();
private readonly _disposable: vscode.Disposable | undefined;

private _sessionsPromise: Promise<vscode.AuthenticationSession[]>;
Expand Down Expand Up @@ -310,25 +282,21 @@ export class GitHubAuthenticationProvider implements vscode.AuthenticationProvid
// the sessions to migrate away from the bad number usage.
// TODO@TylerLeonhardt: Remove this after we are confident that all users have migrated to the new id.
let seenNumberAccountId: boolean = false;
// Sessions that were stored before the account icon was introduced are re-stored
// once an icon has been fetched so that we don't refetch it on every read.
let seenIconUpdate: boolean = false;
// Re-store newly verified accounts so future reads do not need another lookup.
let seenAccountUpdate: boolean = false;
// TODO: eventually remove this Set because we should only have one session per set of scopes.
const scopesSeen = new Set<string>();
const sessionPromises = sessionData.map(async (session: SessionData): Promise<vscode.AuthenticationSession | undefined> => {
// For GitHub scope list, order doesn't matter so we immediately sort the scopes
const scopesStr = [...session.scopes].sort().join(' ');
let userInfo: { id: string; accountName: string; avatarUrl: string | undefined } | undefined;
if (needsAccountIconLookup(session)) {
const needsAccount = !session.account;
if (!session.account) {
try {
userInfo = await this._githubServer.getUserInfo(session.accessToken);
seenIconUpdate = true;
if (needsAccount) {
this._logger.info(`Verified session with the following scopes: ${scopesStr}`);
}
seenAccountUpdate = true;
this._logger.info(`Verified session with the following scopes: ${scopesStr}`);
} catch (e) {
if (e.message === 'Unauthorized' && needsAccount) {
if (e.message === 'Unauthorized') {
return undefined;
}
}
Expand All @@ -346,13 +314,10 @@ export class GitHubAuthenticationProvider implements vscode.AuthenticationProvid
} else {
accountId = userInfo?.id ?? '<unknown>';
}
let icon: vscode.Uri | undefined;
if (session.account?.icon?.scheme) {
icon = vscode.Uri.from(session.account.icon);
} else if (userInfo?.avatarUrl) {
icon = vscode.Uri.parse(userInfo.avatarUrl);
}
const resolvedSession: vscode.AuthenticationSession = {
const icon = session.account?.icon
? vscode.Uri.from(session.account.icon)
: userInfo?.avatarUrl ? vscode.Uri.parse(userInfo.avatarUrl) : undefined;
return {
id: session.id,
account: {
label: session.account
Expand All @@ -366,10 +331,6 @@ export class GitHubAuthenticationProvider implements vscode.AuthenticationProvid
scopes: session.scopes,
accessToken: session.accessToken
};
if (!icon && (session.account?.icon === null || userInfo)) {
this._sessionsWithoutAvatars.add(resolvedSession);
}
return resolvedSession;
});

const verifiedSessions = (await Promise.allSettled(sessionPromises))
Expand All @@ -378,7 +339,7 @@ export class GitHubAuthenticationProvider implements vscode.AuthenticationProvid
.filter(<T>(p?: T): p is T => Boolean(p));

this._logger.info(`Got ${verifiedSessions.length} verified sessions.`);
if (seenNumberAccountId || seenIconUpdate || verifiedSessions.length !== sessionData.length) {
if (seenNumberAccountId || seenAccountUpdate || verifiedSessions.length !== sessionData.length) {
await this.storeSessions(verifiedSessions);
}

Expand All @@ -388,17 +349,7 @@ export class GitHubAuthenticationProvider implements vscode.AuthenticationProvid
private async storeSessions(sessions: vscode.AuthenticationSession[]): Promise<void> {
this._logger.info(`Storing ${sessions.length} sessions...`);
this._sessionsPromise = Promise.resolve(sessions);
const storedSessions: SessionData[] = sessions.map(session => ({
id: session.id,
account: {
label: session.account.label,
id: session.account.id,
icon: serializeAccountIcon(session.account.icon, this._sessionsWithoutAvatars.has(session)),
},
scopes: [...session.scopes],
accessToken: session.accessToken
}));
await this._keychain.setToken(JSON.stringify(storedSessions));
await this._keychain.setToken(JSON.stringify(sessions));
this._logger.info(`Stored ${sessions.length} sessions!`);
}

Expand Down Expand Up @@ -468,16 +419,12 @@ export class GitHubAuthenticationProvider implements vscode.AuthenticationProvid

private async tokenToSession(token: string, scopes: string[]): Promise<vscode.AuthenticationSession> {
const userInfo = await this._githubServer.getUserInfo(token);
const session: vscode.AuthenticationSession = {
return {
id: crypto.getRandomValues(new Uint32Array(2)).reduce((prev, curr) => prev += curr.toString(16), ''),
accessToken: token,
account: { label: userInfo.accountName, id: userInfo.id, icon: userInfo.avatarUrl ? vscode.Uri.parse(userInfo.avatarUrl) : undefined },
scopes
};
if (!session.account.icon) {
this._sessionsWithoutAvatars.add(session);
}
return session;
}

public async removeSession(id: string) {
Expand Down
43 changes: 0 additions & 43 deletions extensions/github-authentication/src/test/github.test.ts

This file was deleted.

Loading
Loading