Skip to content
Merged
66 changes: 52 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,18 +11,22 @@
[![MATLAB Code Issues](.github/badges/code_issues.svg)](https://github.com/ehennestad/MatBox/security/code-scanning)
[![MATLAB](https://img.shields.io/badge/MATLAB-%3E%3DR2023a-blue?logo=data:image/svg%2bxml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiIHN0YW5kYWxvbmU9Im5vIj8+CjxzdmcKICAgd2lkdGg9IjEyIgogICBoZWlnaHQ9IjEwLjcyNSIKICAgdmlld0JveD0iMCAwIDEyIDEwLjcyNSIKICAgZmlsbD0ibm9uZSIKICAgdmVyc2lvbj0iMS4xIgogICBpZD0ic3ZnNCIKICAgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIgogICB4bWxuczpzdmc9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj4KICA8ZwogICAgIGNsaXAtcGF0aD0idXJsKCNjbGlwMF8zMTRfMTY2KSIKICAgICBpZD0iZzIiCiAgICAgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTQsLTQuMDc1MjAwMSkiPgogICAgPHBhdGgKICAgICAgIGQ9Im0gNi4xNzUsMTEuNTc1MiBjIC0wLjYsLTAuNDUgLTEuMzUsLTAuOTc1IC0yLjE3NSwtMS41NzUgMC45NzUsLTAuMzc1IDEuOTUsLTAuNzUgMi45MjUsLTEuMTI1IGwgMS4yLDAuOSBjIC0wLjksMS4wNSAtMS41LDEuNDI1IC0xLjk1LDEuOCB6IG0gOC4wMjUsLTMuMTUgYyAtMC4yMjUsLTAuNiAtMC4zNzUsLTEuMiAtMC42LC0xLjggLTAuMjI1LC0wLjY3NSAtMC40NSwtMS4yNzUgLTAuODI1LC0xLjggLTAuMTUsLTAuMjI1IC0wLjQ1LC0wLjc1IC0wLjgyNSwtMC43NSAtMC4wNzUsMCAtMC4xNSwwLjA3NSAtMC4yMjUsMC4wNzUgLTAuMjI1LDAuMDc1IC0wLjUyNSwwLjUyNSAtMC42LDAuODI1IC0wLjIyNSwwLjM3NSAtMC42NzUsMC45NzUgLTAuOTc1LDEuMzUgLTAuMDc1LDAuMTUgLTAuMjI1LDAuMyAtMC4zLDAuMzc1IC0wLjIyNSwwLjE1IC0wLjQ1LDAuMzc1IC0wLjc1LDAuNTI1IC0wLjA3NSwwIC0wLjE1LDAuMDc1IC0wLjIyNSwwLjA3NSAtMC4yMjUsMCAtMC4zNzUsMC4xNSAtMC41MjUsMC4yMjUgLTAuMjI1LDAuMjI1IC0wLjQ1LDAuNTI1IC0wLjY3NSwwLjc1IDAsMC4wNzUgLTAuMDc1LDAuMTUgLTAuMTUsMC4yMjUgbCAxLjEyNSwwLjgyNSBjIDAuODI1LC0wLjk3NSAxLjgsLTEuOTUgMi40NzUsLTMuODI1IDAsMCAtMC4yMjUsMi4wMjUgLTIuMDI1LDQuMiAtMS4xMjUsMS4yNzUgLTIuMDI1LDEuOTUgLTIuMTc1LDIuMSAwLDAgMC4zLC0wLjA3NSAwLjYsMC4wNzUgMC42LDAuMjI1IDAuOSwxLjA1IDEuMTI1LDEuNjUgMC4xNSwwLjQ1IDAuMzc1LDAuODI1IDAuNTI1LDEuMjc1IDAuNiwtMC4xNSAwLjk3NSwtMC4zNzUgMS4zNSwtMC43NSAwLjM3NSwtMC4zNzUgMC43NSwtMC44MjUgMS4xMjUsLTEuMiAwLjY3NSwtMC44MjUgMS41LC0xLjg3NSAyLjU1LC0xLjM1IDAuMTUsMC4wNzUgMC4zNzUsMC4yMjUgMC40NSwwLjMgMC4yMjUsMC4xNSAwLjM3NSwwLjMgMC42LDAuNTI1IDAuMzc1LDAuMyAwLjUyNSwwLjUyNSAwLjgyNSwwLjY3NSAtMC43NSwtMS41IC0xLjI3NSwtMyAtMS44NzUsLTQuNTc1IHoiCiAgICAgICBmaWxsPSIjZmZmZmZmIgogICAgICAgaWQ9InBhdGgyIiAvPgogIDwvZz4KICA8ZGVmcwogICAgIGlkPSJkZWZzNCI+CiAgICA8Y2xpcFBhdGgKICAgICAgIGlkPSJjbGlwMF8zMTRfMTY2Ij4KICAgICAgPHJlY3QKICAgICAgICAgd2lkdGg9IjEyIgogICAgICAgICBoZWlnaHQ9IjEyIgogICAgICAgICBmaWxsPSIjZmZmZmZmIgogICAgICAgICB0cmFuc2Zvcm09InRyYW5zbGF0ZSg0LDQpIgogICAgICAgICBpZD0icmVjdDQiCiAgICAgICAgIHg9IjAiCiAgICAgICAgIHk9IjAiIC8+CiAgICA8L2NsaXBQYXRoPgogIDwvZGVmcz4KPC9zdmc+Cg==&label=MATLAB&labelColor=C95C2E&color=2A5F98)](https://se.mathworks.com/products/matlab.html)

MatBox provides reusable MATLAB utilities for maintaining MATLAB toolbox repositories. It installs toolbox dependencies, runs tests with coverage reports, runs MATLAB code analysis, and packages `.mltbx` releases from repository metadata.
MatBox automates the maintenance of MATLAB toolbox repositories: installing dependencies, running tests with coverage reports, running code analysis, and packaging `.mltbx` releases.

MatBox is intended for MATLAB toolbox projects that follow a small set of conventions: source code in a source folder, tests in a test folder, dependency declarations in `requirements.txt`, and toolbox metadata in `MLToolboxInfo.json`.
Each task covers something MATLAB either lacks or makes you re-implement per repository:

## What MatBox Does
- **Dependencies** — MATLAB has no package manager for GitHub or File Exchange packages. MatBox installs them from a `requirements.txt` file: `matbox.installRequirements(pwd)`.
- **Test reports** — CI services consume JUnit, Cobertura, and HTML reports, which each repository otherwise has to wire up itself. `matbox.tasks.testToolbox(pwd)` runs the test suite and writes all three to `docs/reports`.
- **Code analysis** — Code Analyzer results do not reach GitHub on their own. `matbox.tasks.codecheckToolbox(pwd)` exports them as SARIF for GitHub's code-scanning tab and can fail CI at a chosen severity.
- **Releases** — packaging an `.mltbx` involves a version bump, `ToolboxOptions` metadata, and file selection. `matbox.tasks.packageToolbox(pwd, "patch")` does all three, driven by `MLToolboxInfo.json`.
- **Badges** — status badges normally depend on an external service. MatBox generates test-result and code-issue badges as SVG files committed to the repository.

- Install MATLAB toolbox dependencies from `requirements.txt`
- Install dependencies from GitHub repositories and MATLAB File Exchange packages
- Run MATLAB unit tests and write JUnit and Cobertura coverage reports
- Run MATLAB Code Analyzer and optionally export reports
- Create README badges for test results and code analysis results
- Package `.mltbx` releases using `MLToolboxInfo.json`
Projects that follow MatBox's conventions can reuse the companion [matbox-actions](https://github.com/ehennestad/matbox-actions) for CI, so each repository only needs a few short workflow files. See these example repositories, where MatBox produces the badges, CI runs, and packaged releases:

- [dropbox-sdk-matlab](https://github.com/ehennestad/dropbox-sdk-matlab) — a class-based Dropbox API client
- [openMINDS-MATLAB-UI](https://github.com/ehennestad/openMINDS-MATLAB-UI) — a graphical interface for openMINDS metadata

The conventions are minimal: source code in a source folder, tests in a test folder, dependency declarations in `requirements.txt`, and toolbox metadata in `MLToolboxInfo.json`.

## Requirements

Expand Down Expand Up @@ -178,6 +182,12 @@ Package a specific version:
matbox.tasks.packageToolbox(pwd, "specific", "1.2.3")
```

By default, a `LICENSE` file in the project root is included in the packaged toolbox. To control which project root files are packaged, add a top-level `RootFilesToPackage` list to `MLToolboxInfo.json`:

```json
"RootFilesToPackage": ["LICENSE", "THIRD_PARTY_NOTICES.md"]
```

## Project-Specific Task Wrappers

Projects can keep their own CI entry points in `tools/tasks`. These wrapper functions are useful when a project needs non-default source folders, test folders, tag filters, report settings, or packaging options.
Expand All @@ -199,6 +209,39 @@ end

This keeps project-specific policy in the project repository while reusing the core MatBox task implementation.

## Continuous Integration

MatBox is designed to run in GitHub Actions. The companion repository [matbox-actions](https://github.com/ehennestad/matbox-actions) provides composite actions and reusable workflows that install MatBox on a runner and invoke the MatBox tasks, so a toolbox repository only needs a few small workflow files.

A minimal test workflow (`.github/workflows/test-code.yml`):

```yaml
name: Test code

on:
push:
branches: main
pull_request:
branches: main

jobs:
test:
name: Analyse and test code
uses: ehennestad/matbox-actions/.github/workflows/test-code-workflow.yml@v1
with:
source_directory: src
tests_directory: tests
tools_directory: tools
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
```

This runs code analysis and the test suites, publishes test results, uploads coverage to Codecov (if the token is set), commits updated badges, and uploads all reports as a build artifact. A corresponding release workflow packages the toolbox and creates a draft GitHub release when a version tag is pushed.

If a project defines its own task wrappers in `tools/tasks` (see above), the CI actions find and call them instead of the built-in `matbox.tasks.*` functions.

See the [CI documentation](docs/ci.md) for the full setup, including the release pipeline, badge behavior, code coverage, and all workflow inputs.

## New Projects

For new toolbox repositories, start from the MATLAB toolbox template:
Expand All @@ -213,11 +256,6 @@ Then configure:
- `.github/workflows` for CI
- optional `tools/tasks` wrappers for project-specific task settings

## Example Repositories

- [dropbox-sdk-matlab](https://github.com/ehennestad/dropbox-sdk-matlab)
- [openMINDS-MATLAB-UI](https://github.com/ehennestad/openMINDS-MATLAB-UI)

## Related Projects

- [MatBox Actions](https://github.com/ehennestad/matbox-actions)
Expand Down
278 changes: 278 additions & 0 deletions docs/ci.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,278 @@
# Continuous Integration with MatBox

This guide describes how to set up GitHub Actions CI for a MATLAB toolbox
repository using MatBox. It covers running tests, code analysis, spell
checking, and the release pipeline.

## How the Pieces Fit Together

CI support for MatBox projects is split across three repositories:

| Repository | Role |
|---|---|
| [MatBox](https://github.com/ehennestad/MatBox) | MATLAB functions that do the actual work: `matbox.tasks.testToolbox`, `matbox.tasks.codecheckToolbox`, `matbox.tasks.packageToolbox`, and `matbox.installRequirements` |
| [matbox-actions](https://github.com/ehennestad/matbox-actions) | GitHub composite actions and reusable workflows that install MatBox on a runner and invoke the MatBox tasks |
| [matlab-toolbox-template](https://github.com/ehennestad/matlab-toolbox-template) | A repository template with the workflows below already configured |

A downstream toolbox repository does not need to reference MatBox directly in
its workflow files. It calls the reusable workflows in `matbox-actions`, which
install MatBox on the runner and run the tasks.

There are two levels at which a project can consume the CI tooling:

1. **Reusable workflows** (recommended): a single `uses:` line per workflow
file. This is what the toolbox template does.
2. **Individual composite actions**: for projects that need a custom job
layout, the building blocks (`install-matbox`, `test-code`, `check-code`,
`package-toolbox`, and others) can be combined in a custom workflow. See the
[matbox-actions README](https://github.com/ehennestad/matbox-actions#available-actions)
for the full list.

## Prerequisites

The reusable workflows assume the
[repository conventions](../README.md#repository-conventions) described in the
README. The defaults expect this layout:

```text
my-toolbox/
requirements.txt # Toolbox dependencies (optional)
src/ # Source code
tests/ # Test suites
tools/
MLToolboxInfo.json # Toolbox metadata (required for packaging)
tasks/ # Optional project-specific task wrappers
.github/
workflows/ # Workflow files shown below
badges/ # Generated badges are committed here
```

If your project uses different folder names, pass them as inputs
(`source_directory`, `tests_directory`, `tools_directory`) to the reusable
workflows.

## Test Workflow

Runs code analysis and the test suites, publishes test results, uploads
coverage, and updates badges. Create `.github/workflows/test-code.yml`:

```yaml
name: Test code

on:
push:
branches: main
pull_request:
branches: main

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
test:
name: Analyse and test code
uses: ehennestad/matbox-actions/.github/workflows/test-code-workflow.yml@v1
with:
source_directory: src
tests_directory: tests
tools_directory: tools
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
```

### Inputs

| Input | Default | Description |
|---|---|---|
| `source_directory` | `src` | Directory containing source code. Code coverage and code analysis run on this folder and its subfolders |
| `tests_directory` | `tests` | Directory containing MATLAB test suites |
| `tools_directory` | `tools` | Directory containing CI tools. Used for locating customized MatBox tasks |
| `matlab_release` | `latest` | MATLAB release used for running tests and code analysis |
| `matlab_use_cache` | `false` | Whether to cache the MATLAB installation for faster subsequent setups |
| `matlab_products` | `''` | Optional list of additional MATLAB products to install |
| `needs_virtual_display` | `false` | Whether to start a virtual display (Xvfb) before testing. Required for tests that create figures or apps |
| `update_badges` | `true` | Whether push events and ready pull requests should generate and commit badge updates |

Secrets: `CODECOV_TOKEN` (optional; enables coverage upload to Codecov).
Outputs: `badges_stale` (whether generated badges differ from committed
badges).

The workflow runs these steps: check out the repository, set up MATLAB
(`matlab-actions/setup-matlab`), install MatBox, run code analysis
(`matbox.tasks.codecheckToolbox`), run tests (`matbox.tasks.testToolbox`),
commit updated badges, upload coverage to Codecov, and upload the
`docs/reports` folder as a build artifact.

Products passed via `matlab_products` use underscore-separated names, one per
line:

```yaml
matlab_products: >
Image_Processing_Toolbox
Statistics_and_Machine_Learning_Toolbox
```

## Code Analysis Workflow

For running code analysis on its own (without tests), use
`check-code-workflow.yml`:

```yaml
jobs:
check:
uses: ehennestad/matbox-actions/.github/workflows/check-code-workflow.yml@v1
with:
source_directory: src
tools_directory: tools
```

Inputs: `source_directory`, `tools_directory`, `matlab_release`,
`matlab_use_cache`, and `update_badges`, with the same defaults and meanings
as the test workflow. The code-issues report is uploaded in SARIF format and
appears under the repository's Security > Code scanning tab (requires
`security-events: write` permission and MATLAB R2023a or later).

## Spell-Check Workflow

Runs [codespell](https://github.com/codespell-project/codespell) over the
repository:

```yaml
jobs:
codespell:
uses: ehennestad/matbox-actions/.github/workflows/codespell-workflow.yml@v1
with:
config_file: .codespellrc
```

The single input `config_file` (default `.codespellrc`) points to a codespell
configuration file. Only the `skip` and `ignore-words-list` options are read
from the configuration file.

## Release Workflow

The release pipeline validates a version number, tests against a matrix of
MATLAB releases, packages the toolbox as `.mltbx`, creates a draft GitHub
release, and verifies that the packaged toolbox installs cleanly. Create
`.github/workflows/prepare-release.yml`:

```yaml
name: Prepare toolbox release

on:
push:
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'
workflow_dispatch:
inputs:
version:
description: 'Version number in major.minor.patch format'
required: true
type: string

jobs:
prepare-release:
uses: ehennestad/matbox-actions/.github/workflows/prepare-release-workflow.yml@v1
with:
version: ${{ inputs.version }}
ref_name: ${{ github.ref_name }}
source_directory: src
tools_directory: tools
secrets:
DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
```

A release is triggered either by pushing a tag like `v1.2.3` or by running the
workflow manually with a version number.

### Inputs

| Input | Default | Description |
|---|---|---|
| `version` | — | Version number in `major.minor.patch` format (for manual triggers) |
| `ref_name` | — | GitHub ref name (for tag triggers) |
| `source_directory` | `src` | Directory containing source code |
| `tests_directory` | `tests` | Directory containing MATLAB test suites |
| `tools_directory` | `tools` | Directory containing tools and `MLToolboxInfo.json` |
| `matlab_products` | `''` | Optional list of additional MATLAB products to install |
| `matlab_versions` | `'[]'` | JSON array of MATLAB versions to test, e.g. `'["R2023a", "R2024a"]'`. If empty, versions are determined from `MLToolboxInfo.json` |
| `python_versions` | `''` | JSON object mapping MATLAB versions to Python versions, e.g. `'{"R2024a": "3.10"}'` |
| `needs_python` | `false` | Whether Python is needed for testing |
| `needs_virtual_display` | `false` | Whether to start a virtual display for testing |

Secrets: `DEPLOY_KEY` (required) — an SSH deploy key with write access, used
to push version updates back to protected branches.

### Pipeline Stages

The reusable workflow chains these jobs:

1. **Validate version** — checks the version number format
2. **Configure test matrix** — determines which MATLAB releases to test, from
`matlab_versions` or from `MLToolboxInfo.json`
3. **Test** — runs the test suites across the MATLAB version matrix
4. **Package and release** — packages the `.mltbx` with
`matbox.tasks.packageToolbox` and creates a draft GitHub release.
Project root files declared in the top-level `RootFilesToPackage`
list in `MLToolboxInfo.json` (by default `LICENSE`) are included in
the packaged toolbox
5. **Verify installation** — installs the packaged toolbox and confirms it
loads

## Project-Specific Task Wrappers

The `test-code` and `check-code` actions add the source, tests, and tools
directories to the MATLAB path, then look for functions named `testToolbox`
and `codecheckToolbox`. If found (conventionally in `tools/tasks/`), the
wrapper is called instead of the built-in `matbox.tasks.*` function, with the
same name-value arguments forwarded.

Use a wrapper when a project needs non-default settings that cannot be
expressed through workflow inputs, such as tag filters or report options. See
[Project-Specific Task Wrappers](../README.md#project-specific-task-wrappers)
in the README for an example.

## Badges

The test and code-analysis workflows generate SVG badges in `.github/badges`
(test results and code issues) and commit them back to the branch when they
change:

- Draft pull requests run CI without generating or committing badges.
- Ready pull requests generate badges. With `update_badges: true` (the
default), updates are committed back to same-repository pull request
branches. With `update_badges: false`, a `Badges current` job fails if the
committed badges are stale, so they can be regenerated locally instead.
- Push events generate and commit badge updates when `update_badges` is
`true`.

Reference the badges in the README:

```markdown
[![MATLAB Tests](.github/badges/tests.svg)](https://github.com/<owner>/<repo>/actions/workflows/test-code.yml)
[![MATLAB Code Issues](.github/badges/code_issues.svg)](https://github.com/<owner>/<repo>/security/code-scanning)
```

## Code Coverage

`matbox.tasks.testToolbox` writes a Cobertura coverage report to
`docs/reports/codecoverage.xml`. If the `CODECOV_TOKEN` secret is set, the
test workflow uploads it to [Codecov](https://codecov.io). To enable:

1. Add the repository on codecov.io and copy the upload token.
2. Add the token as a repository secret named `CODECOV_TOKEN`.
3. Pass the secret to the reusable workflow as shown in the test workflow
example above.

All generated reports (JUnit test results, coverage, code-issues report, HTML
test report) are also uploaded as a workflow artifact named `reports`.

## Versioning

Pin the reusable workflows and actions to a major version tag (currently
`@v1`), which receives backwards-compatible updates:

```yaml
uses: ehennestad/matbox-actions/.github/workflows/test-code-workflow.yml@v1
```
Loading