RFC: TypeSpec Breaking Change Detection Tool - Design Overview#4580
Draft
markcowl wants to merge 1 commit into
Draft
RFC: TypeSpec Breaking Change Detection Tool - Design Overview#4580markcowl wants to merge 1 commit into
markcowl wants to merge 1 commit into
Conversation
Design overview for @azure-tools/typespec-breaking-change, a TypeSpec-native API diff and breaking change detection tool for Azure ARM APIs. Covers architecture, comparison phases, rule summary tables, suppression mechanism, CI integration, and output format. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
|
No changes needing a change description found. |
Contributor
⚡ Benchmark Results
Full details – comparing
|
| Metric | Baseline | Current | Change |
|---|---|---|---|
| total | 🔴 563.6ms | 🔴 751.8ms | +33.4% 🔴 |
| loader | 🟢 158.7ms | 🟡 213.4ms | +34.5% 🔴 |
| resolver | 🟢 17.3ms | 🟢 18.6ms | +7.4% 🔴 |
| checker | 🟢 180.6ms | 🟡 206.4ms | +14.2% 🔴 |
| validation | 🟢 41.4ms | 🟢 42.1ms | +1.7% |
| ↳ validation/@azure-tools/typespec-azure-core | 🟢 6.2ms | 🟢 6.8ms | +10.3% |
| ↳ validation/@typespec/http | 🟢 5.2ms | 🟢 5.7ms | +8.8% |
| ↳ validation/@typespec/rest | 🟢 0.5ms | 🟢 0.5ms | +10.8% |
| ↳ validation/@typespec/versioning | 🔴 28.1ms | 🔴 27.1ms | -3.4% |
| ↳ validation/compiler | 🟢 1.4ms | 🟢 1.4ms | +1.4% |
| linter | 🟢 126.6ms | 🟡 238.6ms | +88.4% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/auth-required | 🟢 0.0ms | 🟢 0.0ms | +8.9% |
| ↳ linter/@azure-tools/typespec-azure-core/bad-record-type | 🟢 0.2ms | 🟢 0.4ms | +102.1% |
| ↳ linter/@azure-tools/typespec-azure-core/byos | 🟢 5.6ms | 🟡 10.7ms | +91.7% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/casing-style | 🟢 0.6ms | 🟢 0.8ms | +34.3% |
| ↳ linter/@azure-tools/typespec-azure-core/composition-over-inheritance | 🟢 0.1ms | 🟢 0.1ms | -0.3% |
| ↳ linter/@azure-tools/typespec-azure-core/documentation-required | 🟢 0.8ms | 🟢 1.3ms | +60.9% |
| ↳ linter/@azure-tools/typespec-azure-core/friendly-name | 🟢 0.6ms | 🟢 1.0ms | +79.7% |
| ↳ linter/@azure-tools/typespec-azure-core/key-visibility-required | 🟢 0.2ms | 🟢 0.4ms | +178.1% |
| ↳ linter/@azure-tools/typespec-azure-core/known-encoding | 🟢 0.3ms | 🟢 0.3ms | +1.3% |
| ↳ linter/@azure-tools/typespec-azure-core/long-running-polling-operation-required | 🟢 0.3ms | 🟢 0.3ms | +22.8% |
| ↳ linter/@azure-tools/typespec-azure-core/no-case-mismatch | 🟢 0.2ms | 🟢 0.3ms | +11.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-closed-literal-union | 🟢 0.2ms | 🟢 0.3ms | +6.8% |
| ↳ linter/@azure-tools/typespec-azure-core/no-enum | 🟢 0.0ms | 🟢 0.0ms | +8.7% |
| ↳ linter/@azure-tools/typespec-azure-core/no-error-status-codes | 🟢 0.1ms | 🟢 0.1ms | +15.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-explicit-routes-resource-ops | 🟢 0.1ms | 🟢 0.1ms | +10.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-format | 🟢 0.5ms | 🟢 0.8ms | +55.8% |
| ↳ linter/@azure-tools/typespec-azure-core/no-generic-numeric | 🟢 0.4ms | 🟢 0.5ms | +29.1% |
| ↳ linter/@azure-tools/typespec-azure-core/no-header-explode | 🟡 18.2ms | 🔴 31.5ms | +72.8% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/no-legacy-usage | 🟢 1.1ms | 🟢 1.3ms | +19.2% |
| ↳ linter/@azure-tools/typespec-azure-core/no-multiple-discriminator | 🟢 0.1ms | 🟢 0.1ms | +27.0% |
| ↳ linter/@azure-tools/typespec-azure-core/no-nullable | 🟢 0.2ms | 🟢 0.4ms | +80.6% |
| ↳ linter/@azure-tools/typespec-azure-core/no-offsetdatetime | 🟢 1.1ms | 🟢 1.6ms | +43.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-openapi | 🟢 2.0ms | 🟢 2.6ms | +34.6% |
| ↳ linter/@azure-tools/typespec-azure-core/no-private-usage | 🟢 1.8ms | 🟢 2.3ms | +27.9% |
| ↳ linter/@azure-tools/typespec-azure-core/no-query-explode | 🟡 18.3ms | 🔴 32.5ms | +77.7% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/no-response-body | 🔴 21.9ms | 🔴 44.7ms | +104.0% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/no-rest-library-interfaces | 🟢 0.0ms | 🟢 0.0ms | -1.1% |
| ↳ linter/@azure-tools/typespec-azure-core/no-route-parameter-name-mismatch | 🟢 4.6ms | 🟡 10.3ms | +123.8% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/no-rpc-path-params | 🟢 0.2ms | 🟢 0.2ms | +31.4% |
| ↳ linter/@azure-tools/typespec-azure-core/no-string-discriminator | 🟢 0.0ms | 🟢 0.0ms | +10.2% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unknown | 🟢 0.2ms | 🟢 0.3ms | +82.6% |
| ↳ linter/@azure-tools/typespec-azure-core/no-unnamed-union | 🟢 0.3ms | 🟢 0.4ms | +13.7% |
| ↳ linter/@azure-tools/typespec-azure-core/operation-missing-api-version | 🟢 0.2ms | 🟢 0.2ms | +2.2% |
| ↳ linter/@azure-tools/typespec-azure-core/request-body-problem | 🟢 0.3ms | 🟢 0.3ms | +5.5% |
| ↳ linter/@azure-tools/typespec-azure-core/require-versioned | 🟢 0.0ms | 🟢 0.0ms | +2.4% |
| ↳ linter/@azure-tools/typespec-azure-core/response-schema-problem | 🔴 21.7ms | 🔴 47.2ms | +117.5% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/rpc-operation-request-body | 🟢 0.3ms | 🟢 1.8ms | +511.5% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/spread-discriminated-model | 🟢 0.3ms | 🟢 0.3ms | +5.6% |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-names | 🟢 5.0ms | 🟡 12.1ms | +142.9% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-core/use-standard-operations | 🟢 0.1ms | 🟢 0.1ms | +22.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-common-types-version | 🟢 3.7ms | 🟢 4.3ms | +15.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-no-key | 🟢 0.1ms | 🟢 0.1ms | +18.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-custom-resource-usage-discourage | 🟢 0.1ms | 🟢 0.1ms | +2.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-delete-operation-response-codes | 🟢 4.7ms | 🟢 5.4ms | +14.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-path-casing-conflicts | 🟢 3.8ms | 🟢 4.5ms | +19.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-no-record | 🟢 0.3ms | 🟢 0.4ms | +7.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-post-operation-response-codes | 🟢 0.4ms | 🟢 0.5ms | +23.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-put-operation-response-codes | 🟢 0.0ms | 🟢 0.0ms | -6.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-action-no-segment | 🟢 0.2ms | 🟢 0.2ms | +16.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-duplicate-property | 🟢 0.1ms | 🟢 0.1ms | +23.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-interface-requires-decorator | 🟢 0.0ms | 🟢 0.0ms | +5.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-action-verb | 🟢 0.1ms | 🟢 0.1ms | -0.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-envelope-property | 🟢 0.1ms | 🟢 0.1ms | +9.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-invalid-version-format | 🟢 0.0ms | 🟢 0.0ms | +14.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-key-invalid-chars | 🟢 0.2ms | 🟢 0.2ms | +10.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-name-pattern | 🟢 0.0ms | 🟢 0.0ms | -14.7% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation | 🟢 0.1ms | 🟢 0.2ms | +8.1% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-operation-response | 🟢 4.4ms | 🟢 4.7ms | +7.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-patch | 🟢 0.3ms | 🟢 0.3ms | +17.6% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-path-segment-invalid-chars | 🟢 0.2ms | 🟢 0.2ms | +7.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/arm-resource-provisioning-state | 🟢 0.1ms | 🟢 0.1ms | +4.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/beyond-nesting-levels | 🟢 0.1ms | 🟢 0.1ms | +15.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/empty-updateable-properties | 🟢 0.1ms | 🟢 0.2ms | +20.5% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/improper-subscription-list-operation | 🟢 0.0ms | 🟢 0.0ms | +7.3% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/lro-location-header | 🟡 11.8ms | 🟡 14.5ms | +22.5% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-operations-endpoint | 🟢 0.0ms | 🟢 0.0ms | -0.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/missing-x-ms-identifiers | 🟢 0.3ms | 🟢 0.3ms | +9.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-empty-model | 🟢 0.1ms | 🟢 0.1ms | +11.0% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-override-props | 🟢 0.1ms | 🟢 0.1ms | +8.4% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-resource-delete-operation | 🟢 0.2ms | 🟢 0.2ms | +23.8% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/no-response-body | 🟡 19.0ms | 🔴 23.0ms | +21.5% 🔴 |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/patch-envelope | 🟢 0.1ms | 🟢 0.1ms | +3.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/resource-name | 🟢 0.1ms | 🟢 0.2ms | +20.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/secret-prop | 🟢 2.0ms | 🟢 2.3ms | +16.2% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/unsupported-type | 🟢 0.3ms | 🟢 0.4ms | +17.9% |
| ↳ linter/@azure-tools/typespec-azure-resource-manager/version-progression | 🟢 0.0ms | 🟢 0.0ms | -3.9% |
| ↳ linter/@azure-tools/typespec-client-generator-core/property-name-conflict | 🟢 1.0ms | 🟢 1.4ms | +41.6% |
| ↳ linter/@azure-tools/typespec-client-generator-core/require-client-suffix | 🟢 0.2ms | 🟢 0.2ms | -4.3% |
| emit | 🔴 5.65s | 🔴 6.02s | +6.5% 🔴 |
| ↳ emit/@azure-tools/typespec-autorest | 🟢 191.5ms | 🟢 175.6ms | -8.3% 🟢 |
| ↳ emit/@azure-tools/typespec-python | 🔴 4.08s | 🔴 4.50s | +10.4% 🔴 |
| ↳ emit/@typespec/http-client-js | 🔴 1.12s | 🔴 1.19s | +6.5% 🔴 |
| ↳ emit/@typespec/openapi3 | 🟢 145.9ms | 🟢 154.2ms | +5.7% 🔴 |
| ↳ emit/@typespec/openapi3/compute | 🟢 127.2ms | 🟢 136.3ms | +7.2% 🔴 |
| ↳ emit/@typespec/openapi3/write | 🟢 19.3ms | 🟢 18.0ms | -6.7% 🟢 |
Averaged across 3 specs (azure-arm-resource-manager, azure-core-dataplane, azure-full).
Threshold: changes > ±5% are highlighted.
🟢 Fast · 🟡 Moderate (stages >200ms, rules >10ms) · 🔴 Slow (stages >400ms, rules >20ms)
Collaborator
|
You can try these changes here
|
mikekistler
reviewed
Jun 9, 2026
mikekistler
left a comment
mikekistler
left a comment
Member
There was a problem hiding this comment.
This looks awesome!
I left a few comments on things to think about / discuss, but overall this seems well thought out.
| The tool is explicitly not trying to solve every compatibility problem around a service. The following are out of scope: | ||
|
|
||
| - SDK-facing concerns such as generated client names or client-shaping behavior. | ||
| - `operationId`, because it is not part of the HTTP wire contract and is primarily an SDK/client concern. |
Member
There was a problem hiding this comment.
Schema names probably also fall into this category.
|
|
||
| After Phase A, the tool evaluates versions that are new on the head branch. | ||
|
|
||
| For every new version on head, the tool compares that new version against the **previous stable** version. |
Member
There was a problem hiding this comment.
I think this should be done for any new or changed version on head. Yes, the changes will be flagged in Phase A, but we need to know if any are actually breaking vs the previous stable to know if we should permit them to be suppressed.
| | `RequestParameterLocationChanged` | Moving between header/query/path/body | Error | | ||
| | `RequestParameterDefaultChanged` | Changing a parameter default value | Error | | ||
| | `RequestPropertyDefaultChanged` | Changing a property default value | Error | | ||
| | `RequestContentTypeRemoved` | Removing a request content type | Error | |
Member
There was a problem hiding this comment.
Curious about changes to readOnly -- are these flagged anywhere?
|
|
||
| #### Removed property | ||
|
|
||
| When a property is removed, the approval is placed on the model that still survives: |
Member
There was a problem hiding this comment.
Seems like this could still go on the property, since it is retained but just has a @removed attribute on it.
Member
There was a problem hiding this comment.
I see this is answered below.
| | `ApiVersionRemoved` | Removing a stable api-version | Error | | ||
| | `AuthSchemeRemoved` | Removing an authentication scheme | Error | | ||
| | `AuthSchemeAdded` (required) | Adding a required authentication scheme | Error | | ||
| | `OAuthScopeAdded` | Narrowing OAuth scopes | Error | |
Member
There was a problem hiding this comment.
What about changes to the version of ARM common types? These get flagged quite often by openapi-diff.
|
|
||
| The CI semantics are intentionally conservative: | ||
|
|
||
| - **new approvals** require reviewer sign-off |
Member
There was a problem hiding this comment.
Is this "sign-off" an approval on the PR? If so, I'm concerned there may be confusion about what approvals mean. An approval by a breaking change reviewer only means "the new/changed suppressions are ok", but is not approving the API as a whole.
|
|
||
| Approvals can outlive the changes they were meant to justify. | ||
| A property may be restored, a constraint may be relaxed again, or a declaration may be refactored so the old approval no longer matches any finding. | ||
| When that happens, the approval becomes stale. |
Member
There was a problem hiding this comment.
Seems like @approvedVersionedChange will always be stale immediately after merge, right?
| └── suppressions / CI output | ||
| ``` | ||
|
|
||
| For the complete taxonomy of detectable diffs, see `diff-taxonomy.md`. For classification policies, see `breaking-change-classification.md`. For detailed rule definitions and type transition tables, see `typespec-breaking-change-rules.md`. |
Member
There was a problem hiding this comment.
Seems I did not see the three markdowns: diff-taxonomy.md, breaking-change-classification.md and typespec-breaking-change-rules.md, do I miss anything?
| For versions that already existed before the change, the same rule semantics still apply when the tool detects a structural regression. | ||
|
|
||
| Rules are evaluated against canonical HTTP metadata. | ||
| That means the tool cares about the observable contract on the wire: operation identity, parameters, payload shapes, status codes, headers, content types, authentication, and encoded value sets. |
Member
There was a problem hiding this comment.
Just to confirm my understanding: change a fixed enum(enum) to extensible enum(union) will not be detected as a breaking change, though type changed from enum to union in TypeSpec, right?
| - **Request narrowing** is always breaking (Error) — the server accepts fewer inputs. | ||
| - **Response widening** is always breaking (Error) — the server returns more outputs. | ||
| - **Request widening** is not breaking (Ignore) — the server accepts more inputs. | ||
| - **Response narrowing** is not breaking (Ignore) — the server returns fewer outputs. |
Member
There was a problem hiding this comment.
Does the response narrowing mean remove a few properties from the response model? May I know why do we ignore those? (treated as non-breaking)
Member
|
Thanks @markcowl for the PR! The design looks awesome. I really love the idea of creating canonicalization on TypeSpec versioning. |
mikeharder
reviewed
Jun 11, 2026
|
|
||
| - Stable versions cannot be removed. | ||
| - Versions must be monotonically increasing. | ||
| - Preview replacement is allowed. |
Member
There was a problem hiding this comment.
let's add "preview and stable must use distinct dates". specs shouldn't have both "2026-01-01" and "2026-01-01-preview" with the same date.
this might be covered by "must be monotonically increasing".
|
|
||
| The design uses two suppression decorators: | ||
|
|
||
| - `@approvedBreakingChange` suppresses Phase B breaking changes. New or modified instances add the `BreakingChangeReviewRequired` label. |
Member
There was a problem hiding this comment.
I wonder if that is actually the best, could we maybe not instead rely on the suppression mechanism replacing lint diff? Require a suppression for breaking changes but we highlight those newly added/scan for those codes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Design overview RFC for @azure-tools/typespec-breaking-change, a TypeSpec-native API diff and breaking change detection tool for Azure ARM APIs.
This document covers:
Supporting Documents
Additional design detail documents (taxonomy, classification, rules, OAD correlation, validation strategy) are available on the
fc/breaking-changes\ branch for reference.