Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,30 @@
`test_command_rollback_tracking.py` — all using real Postgres databases
(no mocks), covering idempotent table creation, hash normalization,
conflict detection, empty-diff hops, safe mode, and rollback tracking.
- **Migration execution** (`--apply`):
- Executes the generated migration against `dburl_from` in a single
transaction (all-or-nothing — a failed statement rolls back everything
that ran before it) instead of only printing the SQL
- On success, automatically records the migration in `dburl_from`'s
`migradiff_history` table — no need to also pass `--record-history`
- On failure, nothing is recorded (since nothing was actually applied)
and the command exits with code 4
- Not supported with `--from-file` (no live database to apply to) or
`--promote` (not implemented yet) — both are rejected with a clear
error before anything runs
- New helper `_apply_migration()` in `migra/command.py`; JSON output
(`--output json`) gets a new `"apply"` object with the outcome
- New tests: `test_command_apply.py` (11 tests, real Postgres, including
a direct atomicity test that forces a mid-migration failure)

### Notes

- This is the foundational layer for the upcoming "Control Plane" feature set.
- Known limitation: `migradiff_history` records migrations as *generated*,
not *applied* — see the README "Migration State Tracking" section for details.
- Resolved: `migradiff_history` used to only record migrations as
*generated*, not *applied* — `--apply` closes that gap by executing the
migration itself and only recording history on confirmed success. Plain
`--record-history` (without `--apply`) still only means "generated", not
"applied" — see the README "Migration State Tracking" section.
- All new features work without AI extras (`anthropic` optional dependency).

## [1.7.2] - 2026-06-08
Expand Down
19 changes: 13 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,15 +64,22 @@ branch, and publishes to PyPI + tags a release from `master`.
(`_check_for_destructive`), column-rename detection (`detect_column_renames`, vs. a naive
drop+add), risk classification for `--output json` (`classify_sql_statement`), credential
redaction in error output (`redact_credentials`).
- `--status`/`--history`/`--promote`/`--record-rollback` etc. talk to `migra/history.py`.
- `--status`/`--history`/`--promote`/`--record-rollback`/`--apply` etc. talk to `migra/history.py`.

### Migration state tracking (`migra/history.py`)

A `migradiff_history` table (`HISTORY_TABLE`) recorded into the *target* database, keyed by a
SHA-256 hash of normalized SQL (`compute_migration_hash`). This is the basis for `--promote`
(multi-environment promotion) and `--record-rollback`. **Important semantic**: a history row means
a migration was generated/reviewed for that target, not that migra itself executed it — migra never
applies SQL to a live database. Preserve this distinction in any related code or docs.
A `migradiff_history` table (`HISTORY_TABLE`), keyed by a SHA-256 hash of normalized SQL
(`compute_migration_hash`). This is the basis for `--promote` (multi-environment promotion),
`--record-rollback`, and `--apply`. **Important semantic, easy to get backwards**: a plain
`--record-history` (without `--apply`) writes into `dburl_target`'s history table and only means
"this migration was generated/reviewed for that target" — not that it was executed anywhere.
`--apply` (`_apply_migration()` in `command.py`) is the one path that actually executes SQL: it runs
the migration against `dburl_from` (the database being migrated — see the CLI's own help text and
the README's `psql dburl_from < migration.sql` convention) inside a single transaction, and only on
confirmed success does it record history — into `dburl_from`'s table, not `dburl_target`'s. A failed
`--apply` rolls back everything and records nothing. `--apply` is rejected up front (before any
dispatch) when combined with `--from-file` (dburl_from would be an ephemeral throwaway database) or
`--promote` (the chain's from/to direction is not yet reconciled with `--apply`'s execution target).

### AI features (optional extra: `pip install migradiff[ai]`, needs `ANTHROPIC_API_KEY`)

Expand Down
36 changes: 18 additions & 18 deletions PROJECT_PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,24 +100,27 @@ done. See "Completed Features" above for what actually shipped:
(fr, de, ja, zh, hi, he — 6 languages, merged as a docs-only PR, no
dedicated version bump).

### Next: release migration state tracking as v1.8.0
### Shipped: migration state tracking and `--apply`

`--status`/`--history`/`--promote`/`--record-rollback` (see "Completed
Features") is code-complete and fully tested but still sitting on branch
`new-feature-6-18-2026`. Landing this — merge, version bump, tag, PyPI
release — is the immediate next step, ahead of any new feature work below.
`--status`/`--history`/`--promote`/`--record-rollback` merged into `main`
2026-08-08 (PRs #8–#10; no version bump/tag yet — still on v1.7.2).
`--apply` (execute the migration against `dburl_from` instead of only
printing it, with automatic history recording on confirmed success — see
README's "Applying Migrations" section) shipped shortly after on top of
that. Neither has been cut into a tagged release yet; `pyproject.toml`
still reads 1.7.2. Bumping to v1.8.0 and publishing is still outstanding.

### Planned (Backlog)

| Feature | Effort | Value | Notes |
|---------|--------|-------|-------|
| `--apply` | Medium | High | Execute generated SQL directly instead of requiring the user to pipe to `psql`. Also needed so `migradiff_history` can record true "applied" state instead of just "generated/reviewed" — see the known limitation in README's Migration State Tracking section. |
| Native `--fail-on-destructive` flag | Low | High | Currently this behavior only exists inside the GitHub Action's `action-entrypoint.sh`. CircleCI/GitLab/pre-commit users have no equivalent without wrapping the CLI themselves. |
| `--apply` + `--promote` | Medium | Medium | `--apply` currently refuses to run when `--promote` is also given — `--promote`'s from/to direction needs to be reconciled with which database `--apply` should actually execute against before this is safe to wire up. |
| `--document` | Medium-High | High | Schema documentation generation |
| Multi-schema hardening | Medium | High | `Migration.__init__` (`migra/migra.py`) still hard-rejects `schema` + `exclude_schema` together; cross-schema FK/dependency ordering in `add_all_changes()` hasn't had a dedicated multi-tenant test pass |
| pgvector support | Low | Medium | Modern Postgres vector types — unconfirmed whether `schemainspect` already round-trips them |
| `--suggest-indexes` | Medium | Medium | AI recommends useful indexes; can reuse `AIAdvisor`'s existing table-stats extraction |
| `--dry-run` | Low | Medium | Only meaningful once `--apply` exists |
| `--dry-run` | Low | Medium | Preview what `--apply` would do without executing it |

---

Expand Down Expand Up @@ -317,28 +320,25 @@ without a dedicated version bump — not tied to a CHANGELOG entry.

## Next Steps

1. **Land migration state tracking as v1.8.0**
- Merge `new-feature-6-18-2026` — code-complete, 342/342 tests passing,
flake8/black clean, and CLI-smoke-tested end-to-end as of 2026-08-08
- Bump version, tag release, publish to PyPI
1. **Cut v1.8.0** — migration state tracking and `--apply` are both merged
to `main` and fully tested, but `pyproject.toml` is still at 1.7.2 and
nothing has been tagged/published to PyPI yet.

2. **`--apply` flag**
- Closes the "generated vs. applied" gap called out in README's
Migration State Tracking known limitation
- Should auto-call the equivalent of `--record-history` on success

3. **Native `--fail-on-destructive` CLI flag**
2. **Native `--fail-on-destructive` CLI flag**
- Promote the GitHub Action's destructive-detection behavior into
`command.py` itself so non-Action CI users (CircleCI, GitLab, plain
scripts) get it too

3. **Reconcile `--promote`'s direction with `--apply`** before wiring the
two together — see the Planned/Backlog table above.

4. **Post v1.8.0:** Enterprise tier planning
- Design licensing system
- Plan hosted features
- Build enterprise marketing narrative

---

**Document version:** Updated 2026-08-08 (reconciled with actual repo/code state)
**Document version:** Updated 2026-08-08 (reconciled with actual repo/code state, incl. `--apply`)
**Last updated by:** Claude (with Leo)
**Repository:** https://github.com/postgresql-tools/migra
36 changes: 18 additions & 18 deletions PROJECT_PLAN2.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,24 +103,27 @@ Hindi/Hebrew for the language set — that changed during implementation.
The 6 languages that actually shipped are fr, de, ja, zh, hi, he (no
Spanish); see `PROJECT_PLAN.md` for the corrected rationale.

### Next: release migration state tracking as v1.8.0
### Shipped: migration state tracking and `--apply`

`--status`/`--history`/`--promote`/`--record-rollback` (see "Completed
Features") is code-complete and fully tested but still sitting on branch
`new-feature-6-18-2026`. Landing this — merge, version bump, tag, PyPI
release — is the immediate next step, ahead of any new feature work below.
`--status`/`--history`/`--promote`/`--record-rollback` merged into `main`
2026-08-08 (PRs #8–#10; no version bump/tag yet — still on v1.7.2).
`--apply` (execute the migration against `dburl_from` instead of only
printing it, with automatic history recording on confirmed success — see
README's "Applying Migrations" section) shipped shortly after on top of
that. Neither has been cut into a tagged release yet; `pyproject.toml`
still reads 1.7.2. Bumping to v1.8.0 and publishing is still outstanding.

### Planned (Backlog)

| Feature | Effort | Value | Notes |
|---------|--------|-------|-------|
| `--apply` | Medium | High | Execute generated SQL directly instead of requiring the user to pipe to `psql`. Also needed so `migradiff_history` can record true "applied" state instead of just "generated/reviewed" — see the known limitation in README's Migration State Tracking section. |
| Native `--fail-on-destructive` flag | Low | High | Currently this behavior only exists inside the GitHub Action's `action-entrypoint.sh`. CircleCI/GitLab/pre-commit users have no equivalent without wrapping the CLI themselves. |
| `--apply` + `--promote` | Medium | Medium | `--apply` currently refuses to run when `--promote` is also given — `--promote`'s from/to direction needs to be reconciled with which database `--apply` should actually execute against before this is safe to wire up. |
| `--document` | Medium-High | High | Schema documentation generation |
| Multi-schema hardening | Medium | High | `Migration.__init__` (`migra/migra.py`) still hard-rejects `schema` + `exclude_schema` together; cross-schema FK/dependency ordering in `add_all_changes()` hasn't had a dedicated multi-tenant test pass |
| pgvector support | Low | Medium | Modern Postgres vector types — unconfirmed whether `schemainspect` already round-trips them |
| `--suggest-indexes` | Medium | Medium | AI recommends useful indexes; can reuse `AIAdvisor`'s existing table-stats extraction |
| `--dry-run` | Low | Medium | Only meaningful once `--apply` exists |
| `--dry-run` | Low | Medium | Preview what `--apply` would do without executing it |

---

Expand Down Expand Up @@ -320,28 +323,25 @@ without a dedicated version bump — not tied to a CHANGELOG entry.

## Next Steps

1. **Land migration state tracking as v1.8.0**
- Merge `new-feature-6-18-2026` — code-complete, 342/342 tests passing,
flake8/black clean, and CLI-smoke-tested end-to-end as of 2026-08-08
- Bump version, tag release, publish to PyPI
1. **Cut v1.8.0** — migration state tracking and `--apply` are both merged
to `main` and fully tested, but `pyproject.toml` is still at 1.7.2 and
nothing has been tagged/published to PyPI yet.

2. **`--apply` flag**
- Closes the "generated vs. applied" gap called out in README's
Migration State Tracking known limitation
- Should auto-call the equivalent of `--record-history` on success

3. **Native `--fail-on-destructive` CLI flag**
2. **Native `--fail-on-destructive` CLI flag**
- Promote the GitHub Action's destructive-detection behavior into
`command.py` itself so non-Action CI users (CircleCI, GitLab, plain
scripts) get it too

3. **Reconcile `--promote`'s direction with `--apply`** before wiring the
two together — see the Planned/Backlog table above.

4. **Post v1.8.0:** Enterprise tier planning
- Design licensing system
- Plan hosted features
- Build enterprise marketing narrative

---

**Document version:** Updated 2026-08-08 (reconciled with actual repo/code state)
**Document version:** Updated 2026-08-08 (reconciled with actual repo/code state, incl. `--apply`)
**Last updated by:** Claude (with Leo)
**Repository:** https://github.com/postgresql-tools/migra
44 changes: 37 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,13 +278,13 @@ rollback tracking.

### ⚠ Known Limitation

`migradiff_history` records that a migration was **generated/reviewed**
for a target database, not that the SQL was necessarily *executed* by
MigraDiff itself (MigraDiff does not apply migrations to live databases
today — it only diffs and generates SQL). Recording true "applied" state
requires your deployment pipeline to also call
`migra --record-history` (or the future `--apply` flag, not yet
implemented) after running the generated SQL.
Without `--apply` (below), `migradiff_history` only records that a
migration was **generated/reviewed** for a target database, not that the
SQL was necessarily *executed*. If you generate SQL and pipe it to `psql`
yourself, calling `migra --record-history` only tells MigraDiff "this
migration was proposed" — it has no way to know whether your `psql` step
actually succeeded. Use `--apply` when you want MigraDiff itself to run
the migration and only record history on confirmed success.

Be explicit about this in your pipeline so you don't assume false
guarantees about whether a migration has actually been applied.
Expand Down Expand Up @@ -329,6 +329,36 @@ Use `--env-label` to tag the entry:
migra --record-history --env-label staging postgres://db_a postgres://db_b
```

### Applying Migrations (`--apply`)

`--apply` executes the generated migration directly against `dburl_from`
(the first positional argument — "the database you want to migrate", same
database the README's basic usage example pipes to `psql`) instead of only
printing it:

```bash
migra --apply postgres://db_production postgres://db_branch
```

On success, MigraDiff automatically records the migration in
`dburl_from`'s `migradiff_history` table — you don't need to also pass
`--record-history`. If any statement fails, the whole migration is rolled
back as a single transaction (nothing is partially applied) and **nothing
is recorded**, since it wasn't actually applied. The command exits non-zero
(exit code 4) so pipelines can detect the failure.

```bash
migra --apply --env-label prod postgres://db_production postgres://db_branch
```

`--apply` respects the same safety gates as everything else: destructive
statements are blocked unless `--force-destructive` (or `--unsafe`) is
given, and the block happens *before* anything is executed.

`--apply` is not supported with `--from-file` (there's no live database to
apply to — the schema files get loaded into temporary throwaway databases)
or with `--promote` (not implemented yet).

### Multi-Environment Promotion (`--promote`)

`--promote` generates migrations along a chain of environments, with
Expand Down
Loading
Loading