diff --git a/README.md b/README.md index 7b04821..f72ad10 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,7 @@ SHOW AGENTS IN SCHEMA my_db.my_schema; | `enable_versioning` | bool | No | Whether to use Snowflake agent versioning. Defaults to `true`. When `true`, uses `CREATE AGENT IF NOT EXISTS` + `ALTER AGENT MODIFY LIVE VERSION` instead of `CREATE OR REPLACE AGENT`, preserving version history across runs. Set to `false` in dev environments to skip versioning overhead. `dbt run --full-refresh` always falls back to `CREATE OR REPLACE`, resetting history. | | `auto_commit` | bool | No | When `enable_versioning=true`, automatically snapshot the LIVE version into a new named version after each run. Defaults to `true`. Set to `false` to accumulate spec changes in LIVE without committing, then commit manually via `ALTER AGENT COMMIT`. | | `version_comment` | string | No | Comment attached to each committed version snapshot. Only used when `enable_versioning=true` and `auto_commit=true`. | +| `set_default_version` | bool | No | Promote the newly committed version to DEFAULT so end users (and Snowsight's "In use") get it. Defaults to `true`. Only used when `enable_versioning=true` and `auto_commit=true`. `COMMIT` alone does not move the DEFAULT pointer — see [Version promotion](#version-promotion). | ## How It Works @@ -102,12 +103,31 @@ The model body is passed verbatim as the agent YAML specification to Snowflake. By default (`enable_versioning=true`, `auto_commit=true`) every `dbt run`: 1. Creates the agent if it doesn't exist (`CREATE AGENT IF NOT EXISTS`) -2. Restores a LIVE working copy from the last committed version (`ALTER AGENT ADD LIVE VERSION FROM LAST`) +2. Restores a LIVE working copy from the last committed version (`ALTER AGENT ADD LIVE VERSION FROM LAST`), unless a LIVE copy already exists 3. Updates the LIVE copy with the latest compiled spec (`ALTER AGENT MODIFY LIVE VERSION SET SPECIFICATION`) 4. Commits the LIVE copy as a new named version — `VERSION$1`, `VERSION$2`, … (`ALTER AGENT COMMIT`) +5. Promotes that new version to DEFAULT (`ALTER AGENT SET DEFAULT_VERSION = 'LAST'`) Version history accumulates across runs. `dbt run --full-refresh` falls back to `CREATE OR REPLACE AGENT`, resetting all history. +### Version promotion + +Step 5 matters: **`COMMIT` alone does not move the DEFAULT pointer.** Snowflake treats the latest committed version as the default only *implicitly*, and that stops the moment `DEFAULT_VERSION` is set explicitly by anything out-of-band — most commonly Snowsight's **Publish** button. Once that happens the pointer is pinned, and every later `dbt run` commits a version that no user ever sees: Snowsight keeps showing an old version as "In use" while history piles up behind it. + +Promoting explicitly on every commit makes deploys deterministic and self-healing — a pinned agent is un-stuck by the next `dbt run`. Opt out with `set_default_version=false` if you promote versions yourself (e.g. staged releases via aliases): + +```sql +{{ config(materialized='cortex_agent', set_default_version=false) }} +``` + +Check which version is live: + +```sql +SHOW VERSIONS IN AGENT my_db.my_schema.my_agent; -- is_default = true marks the live one +``` + +> Note: `SET DEFAULT_VERSION` requires a **quoted** value — `'LAST'` or `'VERSION$3'`. The unquoted forms shown in some Snowflake docs (`LAST`, `VERSION$3`) fail with a SQL compilation error. + **Disable versioning in dev** to avoid the overhead when iterating quickly: ```yaml diff --git a/integration_tests/tests/cortex_agent_versioned_test_default_is_newest.sql b/integration_tests/tests/cortex_agent_versioned_test_default_is_newest.sql new file mode 100644 index 0000000..f08144b --- /dev/null +++ b/integration_tests/tests/cortex_agent_versioned_test_default_is_newest.sql @@ -0,0 +1,38 @@ +-- Verify the newest committed version is also the DEFAULT version — what end users get and +-- what Snowsight shows as "In use". +-- +-- COMMIT creates a version but does not move the DEFAULT pointer; Snowflake follows the latest +-- version only implicitly, and stops once DEFAULT_VERSION is set out-of-band (e.g. Snowsight's +-- Publish button). Without this assertion an agent can accumulate committed versions that no +-- user ever sees, and the suite would still pass — cortex_agent_versioned_test_has_versions +-- only checks that versions exist, not that the newest one is live. +-- +-- SHOW VERSIONS emits lowercase column names, so they must be double-quoted here. Rows with a +-- null "name" are the mutable LIVE working copy, not committed versions, so they're excluded. +-- Returns 0 rows on success (standard dbt test contract). + +with committed as ( + + select + "name" as version_name, + "is_default" as is_default, + try_to_number(split_part("name", '$', 2)) as version_number + from {{ ref('cortex_agent_versioned_test_versions') }} + where "name" is not null + +), + +newest as ( + select version_name from committed order by version_number desc limit 1 +), + +current_default as ( + select version_name from committed where lower(is_default) = 'true' +) + +select + 'newest committed version is not the default: newest=' + || coalesce((select version_name from newest), '') + || ', default=' + || coalesce((select version_name from current_default), '') as error +where (select version_name from newest) is distinct from (select version_name from current_default) diff --git a/macros/materializations/cortex_agent.sql b/macros/materializations/cortex_agent.sql index 33e6825..c2ee986 100644 --- a/macros/materializations/cortex_agent.sql +++ b/macros/materializations/cortex_agent.sql @@ -32,6 +32,16 @@ -- LIVE without committing, then commit manually. -- version_comment (string, optional) : comment attached to the committed version snapshot. -- Only used when enable_versioning=true and auto_commit=true. +-- set_default_version (bool, optional) : promote the newly committed version to DEFAULT so +-- end users (and Snowsight's "In use") get it. +-- Defaults to true. Only used when enable_versioning=true +-- and auto_commit=true. +-- COMMIT alone does NOT move the DEFAULT pointer: Snowflake +-- only follows the latest version implicitly, and stops doing +-- so as soon as DEFAULT_VERSION is set out-of-band (e.g. by +-- Snowsight's Publish button) — after which every dbt deploy +-- is invisible to users. Set false only if you promote +-- versions yourself (e.g. staged releases via aliases). {% materialization cortex_agent, adapter='snowflake' %} @@ -45,6 +55,7 @@ {%- set enable_versioning = config.get('enable_versioning', default=true) -%} {%- set auto_commit = config.get('auto_commit', default=true) -%} {%- set version_comment = config.get('version_comment', default=none) -%} + {%- set set_default_version = config.get('set_default_version', default=true) -%} {%- set target_relation = api.Relation.create( identifier=this.identifier, @@ -85,30 +96,44 @@ {%- if enable_versioning and not should_full_refresh() %} - {# Determine whether any committed versions exist. - On first run (agent absent or no committed versions) skip MODIFY LIVE VERSION — - the LIVE working copy does not exist until after the first COMMIT. - On subsequent runs COMMIT has already established LIVE, so we can update it. #} + {# Inspect existing versions to decide whether LIVE needs (re)creating. + SHOW VERSIONS returns one row per committed version (name = VERSION$N) plus, when a + LIVE working copy exists, one row with an empty name. + - no committed versions : first run — LIVE does not exist yet, skip ADD LIVE VERSION. + - committed, no LIVE : the normal steady state after COMMIT consumed LIVE — restore it. + - committed, LIVE exists : ADD LIVE VERSION would fail with 099106 ("There is already a + live version"), which is the state a prior CREATE OR REPLACE + (e.g. dbt run --full-refresh) leaves behind — skip the add and + modify the existing LIVE in place. #} {%- set _show_agents_sql -%} SHOW AGENTS LIKE '{{ target_relation.identifier }}' IN SCHEMA {{ target_relation.database }}.{{ target_relation.schema }} {%- endset -%} {%- set _agent_rows = run_query(_show_agents_sql) -%} - {%- set _has_committed_versions = false -%} + {%- set _ns = namespace(has_committed=false, has_live=false) -%} {%- if _agent_rows | length > 0 -%} {%- set _show_versions_sql -%} SHOW VERSIONS IN AGENT {{ target_relation.database }}.{{ target_relation.schema }}.{{ target_relation.identifier }} {%- endset -%} {%- set _version_rows = run_query(_show_versions_sql) -%} - {%- set _has_committed_versions = (_version_rows | length) > 0 -%} + {%- for _r in _version_rows.rows -%} + {%- if _r['name'] -%} + {%- set _ns.has_committed = true -%} + {%- else -%} + {%- set _ns.has_live = true -%} + {%- endif -%} + {%- endfor -%} {%- endif -%} + {%- set _has_committed_versions = _ns.has_committed -%} + {%- set _has_live_version = _ns.has_live -%} {% call statement('main') %} {{ dbt_cortex_agent.snowflake__create_cortex_agent_if_not_exists(target_relation, sql, comment, profile) }} {% endcall %} - {%- if _has_committed_versions %} + {%- if _has_committed_versions and not _has_live_version %} {# COMMIT consumes LIVE and does not recreate it. Explicitly restore LIVE from the - last committed version before modifying, so MODIFY LIVE VERSION has a target. #} + last committed version before modifying, so MODIFY LIVE VERSION has a target. + Skipped when a LIVE already exists — adding a second one errors with 099106. #} {% call statement('add_live') %} {{ dbt_cortex_agent.snowflake__add_cortex_agent_live_version(target_relation) }} {% endcall %} @@ -122,6 +147,15 @@ {% call statement('commit_version') %} {{ dbt_cortex_agent.snowflake__commit_cortex_agent_version(target_relation, version_comment) }} {% endcall %} + + {%- if set_default_version %} + {# COMMIT does not move the DEFAULT pointer. Promote explicitly so the version just + deployed is the one users get, and so an agent whose pointer was pinned + out-of-band (e.g. via Snowsight) is un-stuck on the next run. #} + {% call statement('set_default_version') %} + {{ dbt_cortex_agent.snowflake__set_cortex_agent_default_version(target_relation) }} + {% endcall %} + {%- endif %} {%- endif %} {%- else %} diff --git a/macros/relations/cortex_agent/versioned_agent.sql b/macros/relations/cortex_agent/versioned_agent.sql index 932d81f..0c7db3f 100644 --- a/macros/relations/cortex_agent/versioned_agent.sql +++ b/macros/relations/cortex_agent/versioned_agent.sql @@ -61,3 +61,27 @@ {%- endif %} {% endmacro %} + + +-- Promotes the newest committed version to DEFAULT — the version end users and +-- Snowsight see as "In use". +-- +-- Why this is required: COMMIT creates a version but does NOT move the DEFAULT +-- pointer. Snowflake only tracks the latest committed version *implicitly*, and +-- that implicit behaviour stops the moment DEFAULT_VERSION is set explicitly by +-- anything out-of-band (notably Snowsight's Publish button). From then on every +-- dbt deploy commits a version that no user ever sees. Setting it explicitly on +-- each commit makes promotion deterministic and self-healing: a pinned agent is +-- un-stuck by the next dbt run. +-- +-- 'LAST' is used rather than a computed VERSION$N so the statement needs no version +-- bookkeeping and stays idempotent. Note the quotes are required: the unquoted forms +-- (LAST, VERSION$3, "VERSION$3") are all rejected with a SQL compilation error, despite +-- what the Snowflake docs show — only the single-quoted string form parses. +{% macro snowflake__set_cortex_agent_default_version(relation) %} + + alter agent + {{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }} + set default_version = 'LAST' + +{% endmacro %}