Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,8 @@ If your GitHub repository is linked to a QualityMax project, omit both — the a
| `auto-discover` | Auto-discover test scenarios in seed mode | No | `true` |
| `max-seed-tests` | Maximum tests to generate in seed mode (1-10) | No | `3` |
| `seed-descriptions` | Newline-separated test descriptions for seed mode | No | — |
| `shard` | Matrix shard index (1-based), used with `shards-total` for parallel execution | No | — |
| `shards-total` | Total number of shards. Required when `shard` is set | No | — |

> **Note:** Either `project-id`, `project-name`, or a linked repository is required. If none are provided, the action attempts auto-detection from the repository URL.

Expand Down Expand Up @@ -241,6 +243,38 @@ jobs:
test-ids: '1,2,3,4,5'
```

### Matrix Sharding — Parallel Execution

For large test suites, split the run across multiple parallel GitHub runners using Playwright's native `--shard=N/M` flag. Each shard runs a deterministic slice of the tests, cutting wall-clock time nearly linearly with the shard count.

```yaml
jobs:
test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
shard: [1, 2, 3, 4] # 4 parallel runners
steps:
- uses: Quality-Max/qualitymax-github-action@v1
with:
api-key: ${{ secrets.QUALITYMAX_API_KEY }}
project-name: 'My Web App'
shard: ${{ matrix.shard }}
shards-total: 4
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

Each shard:
- Fetches the full set of scripts from QualityMax
- Runs only its slice via `npx playwright test --shard=N/M`
- Reports its own pass/fail back to QualityMax as a separate execution

**When to use it:** test suites of 20+ scripts that take more than a few minutes sequentially. For small suites (< 20 tests) the shard overhead (npm install + browser install per runner) outweighs the savings.

**Tip:** set `fail-fast: false` so a failure in one shard doesn't cancel the others — you want to see every failure on a given commit, not just the first.

### Continue on Test Failure

```yaml
Expand Down
6 changes: 6 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ inputs:
seed-descriptions:
description: 'Newline-separated test descriptions for seed mode (overrides auto-discover)'
required: false
shard:
description: 'Matrix shard index (1-based). Use with shards-total to split tests across parallel jobs. Example: shard=2, shards-total=4 runs the second quarter.'
required: false
shards-total:
description: 'Total number of shards. Required when shard is set. Ignored otherwise.'
required: false

outputs:
execution-id:
Expand Down
66 changes: 54 additions & 12 deletions dist/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -30238,6 +30238,24 @@ const api_1 = __nccwpck_require__(6879);
function getInputs() {
const testIdsInput = core.getInput('test-ids');
const seedDescInput = core.getInput('seed-descriptions');
// Parse sharding inputs. Both must be set to activate sharding —
// if either is missing or invalid we fall back to running the full suite.
const shardInput = core.getInput('shard');
const shardsTotalInput = core.getInput('shards-total');
let shard;
let shardsTotal;
if (shardInput && shardsTotalInput) {
const s = parseInt(shardInput, 10);
const t = parseInt(shardsTotalInput, 10);
if (Number.isFinite(s) && Number.isFinite(t) && s >= 1 && t >= 1 && s <= t) {
shard = s;
shardsTotal = t;
}
else {
core.warning(`Invalid shard config: shard=${shardInput}, shards-total=${shardsTotalInput}. ` +
`Both must be positive integers with shard <= shards-total. Running full suite.`);
}
}
return {
apiKey: core.getInput('api-key', { required: true }),
projectId: core.getInput('project-id') || '',
Expand All @@ -30258,6 +30276,8 @@ function getInputs() {
seedDescriptions: seedDescInput
? seedDescInput.split('\n').map((d) => d.trim()).filter((d) => d)
: undefined,
shard,
shardsTotal,
};
}
/**
Expand Down Expand Up @@ -30416,12 +30436,22 @@ module.exports = defineConfig({
await (0, exec_1.exec)('npm', ['install', '--no-audit', '--no-fund'], { cwd: tmpDir });
core.info(`Installing Playwright browser: ${inputs.browser}...`);
await (0, exec_1.exec)('npx', ['playwright', 'install', inputs.browser], { cwd: tmpDir });
// Run tests
core.info('Running Playwright tests...');
// Run tests — pass --shard=N/M when matrix sharding is active so this
// runner only executes its slice of the suite. Playwright deterministically
// partitions the test files, so all shards agree on the split without
// coordinating.
const playwrightArgs = ['playwright', 'test', '--config=playwright.config.js'];
if (inputs.shard && inputs.shardsTotal) {
playwrightArgs.push(`--shard=${inputs.shard}/${inputs.shardsTotal}`);
core.info(`Running Playwright tests (shard ${inputs.shard}/${inputs.shardsTotal})...`);
}
else {
core.info('Running Playwright tests...');
}
const startTime = Date.now();
let exitCode = 0;
try {
exitCode = await (0, exec_1.exec)('npx', ['playwright', 'test', '--config=playwright.config.js'], {
exitCode = await (0, exec_1.exec)('npx', playwrightArgs, {
cwd: tmpDir,
ignoreReturnCode: true,
});
Expand All @@ -30434,7 +30464,6 @@ module.exports = defineConfig({
// Parse results from JSON reporter
let passedTests = 0;
let failedTests = 0;
const totalTests = scripts.length;
let skippedTests = 0;
const testResults = [];
const resultsFile = path.join(tmpDir, 'results.json');
Expand Down Expand Up @@ -30467,6 +30496,12 @@ module.exports = defineConfig({
core.warning(`Failed to parse Playwright results: ${error}`);
}
}
// Compute totalTests from the actual tests run in this shard (when parsed).
// When sharding, scripts.length is the full-suite size, not the shard size,
// so we prefer testResults.length. Falls back to scripts.length only when
// we couldn't parse any results.
const isSharded = Boolean(inputs.shard && inputs.shardsTotal);
const totalTests = testResults.length > 0 ? testResults.length : scripts.length;
// If no results parsed, infer from exit code
if (testResults.length === 0) {
if (exitCode === 0) {
Expand All @@ -30475,14 +30510,18 @@ module.exports = defineConfig({
else {
failedTests = totalTests;
}
for (const script of scripts) {
testResults.push({
test_id: script.id,
test_name: script.name,
status: exitCode === 0 ? 'passed' : 'failed',
duration_seconds: durationSeconds / scripts.length,
error_message: exitCode !== 0 ? 'Test execution failed' : undefined,
});
// Don't fabricate per-script rows when sharding — we don't know which
// scripts actually ran in this shard without Playwright's output.
if (!isSharded) {
for (const script of scripts) {
testResults.push({
test_id: script.id,
test_name: script.name,
status: exitCode === 0 ? 'passed' : 'failed',
duration_seconds: durationSeconds / scripts.length,
error_message: exitCode !== 0 ? 'Test execution failed' : undefined,
});
}
}
}
skippedTests = totalTests - passedTests - failedTests;
Expand Down Expand Up @@ -30551,6 +30590,9 @@ async function run() {
core.info('🚀 QualityMax Test Runner');
core.info(`Project: ${inputs.projectId || inputs.projectName || '(auto-detect)'}`);
core.info(`Test Suite: ${inputs.testSuite}`);
if (inputs.shard && inputs.shardsTotal) {
core.info(`Shard: ${inputs.shard}/${inputs.shardsTotal}`);
}
core.info(`Browser: ${inputs.browser}`);
// Initialize client
client = new api_1.QualityMaxClient(inputs.apiKey);
Expand Down
2 changes: 1 addition & 1 deletion dist/index.js.map

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions dist/types.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ export interface ActionInputs {
autoDiscover: boolean;
maxSeedTests: number;
seedDescriptions?: string[];
shard?: number;
shardsTotal?: number;
}
export interface ActionOutputs {
executionId: string;
Expand Down
2 changes: 1 addition & 1 deletion dist/types.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "qualitymax-github-action",
"version": "1.2.1",
"version": "1.3.0",
"description": "Run AI-powered E2E tests in your CI/CD pipeline with QualityMax",
"main": "dist/index.js",
"scripts": {
Expand Down
Loading
Loading