Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 21 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ SHOW AGENTS IN SCHEMA my_db.my_schema;
| `enable_versioning` | bool | No | Whether to use Snowflake agent versioning. Defaults to `true`. When `true`, uses `CREATE AGENT IF NOT EXISTS` + `ALTER AGENT MODIFY LIVE VERSION` instead of `CREATE OR REPLACE AGENT`, preserving version history across runs. Set to `false` in dev environments to skip versioning overhead. `dbt run --full-refresh` always falls back to `CREATE OR REPLACE`, resetting history. |
| `auto_commit` | bool | No | When `enable_versioning=true`, automatically snapshot the LIVE version into a new named version after each run. Defaults to `true`. Set to `false` to accumulate spec changes in LIVE without committing, then commit manually via `ALTER AGENT COMMIT`. |
| `version_comment` | string | No | Comment attached to each committed version snapshot. Only used when `enable_versioning=true` and `auto_commit=true`. |
| `set_default_version` | bool | No | Promote the newly committed version to DEFAULT so end users (and Snowsight's "In use") get it. Defaults to `true`. Only used when `enable_versioning=true` and `auto_commit=true`. `COMMIT` alone does not move the DEFAULT pointer — see [Version promotion](#version-promotion). |

## How It Works

Expand All @@ -102,12 +103,31 @@ The model body is passed verbatim as the agent YAML specification to Snowflake.
By default (`enable_versioning=true`, `auto_commit=true`) every `dbt run`:

1. Creates the agent if it doesn't exist (`CREATE AGENT IF NOT EXISTS`)
2. Restores a LIVE working copy from the last committed version (`ALTER AGENT ADD LIVE VERSION FROM LAST`)
2. Restores a LIVE working copy from the last committed version (`ALTER AGENT ADD LIVE VERSION FROM LAST`), unless a LIVE copy already exists
3. Updates the LIVE copy with the latest compiled spec (`ALTER AGENT MODIFY LIVE VERSION SET SPECIFICATION`)
4. Commits the LIVE copy as a new named version — `VERSION$1`, `VERSION$2`, … (`ALTER AGENT COMMIT`)
5. Promotes that new version to DEFAULT (`ALTER AGENT SET DEFAULT_VERSION = 'LAST'`)

Version history accumulates across runs. `dbt run --full-refresh` falls back to `CREATE OR REPLACE AGENT`, resetting all history.

### Version promotion

Step 5 matters: **`COMMIT` alone does not move the DEFAULT pointer.** Snowflake treats the latest committed version as the default only *implicitly*, and that stops the moment `DEFAULT_VERSION` is set explicitly by anything out-of-band — most commonly Snowsight's **Publish** button. Once that happens the pointer is pinned, and every later `dbt run` commits a version that no user ever sees: Snowsight keeps showing an old version as "In use" while history piles up behind it.

Promoting explicitly on every commit makes deploys deterministic and self-healing — a pinned agent is un-stuck by the next `dbt run`. Opt out with `set_default_version=false` if you promote versions yourself (e.g. staged releases via aliases):

```sql
{{ config(materialized='cortex_agent', set_default_version=false) }}
```

Check which version is live:

```sql
SHOW VERSIONS IN AGENT my_db.my_schema.my_agent; -- is_default = true marks the live one
```

> Note: `SET DEFAULT_VERSION` requires a **quoted** value — `'LAST'` or `'VERSION$3'`. The unquoted forms shown in some Snowflake docs (`LAST`, `VERSION$3`) fail with a SQL compilation error.

**Disable versioning in dev** to avoid the overhead when iterating quickly:

```yaml
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
-- Verify the newest committed version is also the DEFAULT version — what end users get and
-- what Snowsight shows as "In use".
--
-- COMMIT creates a version but does not move the DEFAULT pointer; Snowflake follows the latest
-- version only implicitly, and stops once DEFAULT_VERSION is set out-of-band (e.g. Snowsight's
-- Publish button). Without this assertion an agent can accumulate committed versions that no
-- user ever sees, and the suite would still pass — cortex_agent_versioned_test_has_versions
-- only checks that versions exist, not that the newest one is live.
--
-- SHOW VERSIONS emits lowercase column names, so they must be double-quoted here. Rows with a
-- null "name" are the mutable LIVE working copy, not committed versions, so they're excluded.
-- Returns 0 rows on success (standard dbt test contract).

with committed as (

select
"name" as version_name,
"is_default" as is_default,
try_to_number(split_part("name", '$', 2)) as version_number
from {{ ref('cortex_agent_versioned_test_versions') }}
where "name" is not null

),

newest as (
select version_name from committed order by version_number desc limit 1
),

current_default as (
select version_name from committed where lower(is_default) = 'true'
)

select
'newest committed version is not the default: newest='
|| coalesce((select version_name from newest), '<none>')
|| ', default='
|| coalesce((select version_name from current_default), '<none>') as error
where (select version_name from newest) is distinct from (select version_name from current_default)
50 changes: 42 additions & 8 deletions macros/materializations/cortex_agent.sql
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,16 @@
-- LIVE without committing, then commit manually.
-- version_comment (string, optional) : comment attached to the committed version snapshot.
-- Only used when enable_versioning=true and auto_commit=true.
-- set_default_version (bool, optional) : promote the newly committed version to DEFAULT so
-- end users (and Snowsight's "In use") get it.
-- Defaults to true. Only used when enable_versioning=true
-- and auto_commit=true.
-- COMMIT alone does NOT move the DEFAULT pointer: Snowflake
-- only follows the latest version implicitly, and stops doing
-- so as soon as DEFAULT_VERSION is set out-of-band (e.g. by
-- Snowsight's Publish button) — after which every dbt deploy
-- is invisible to users. Set false only if you promote
-- versions yourself (e.g. staged releases via aliases).

{% materialization cortex_agent, adapter='snowflake' %}

Expand All @@ -45,6 +55,7 @@
{%- set enable_versioning = config.get('enable_versioning', default=true) -%}
{%- set auto_commit = config.get('auto_commit', default=true) -%}
{%- set version_comment = config.get('version_comment', default=none) -%}
{%- set set_default_version = config.get('set_default_version', default=true) -%}

{%- set target_relation = api.Relation.create(
identifier=this.identifier,
Expand Down Expand Up @@ -85,30 +96,44 @@

{%- if enable_versioning and not should_full_refresh() %}

{# Determine whether any committed versions exist.
On first run (agent absent or no committed versions) skip MODIFY LIVE VERSION —
the LIVE working copy does not exist until after the first COMMIT.
On subsequent runs COMMIT has already established LIVE, so we can update it. #}
{# Inspect existing versions to decide whether LIVE needs (re)creating.
SHOW VERSIONS returns one row per committed version (name = VERSION$N) plus, when a
LIVE working copy exists, one row with an empty name.
- no committed versions : first run — LIVE does not exist yet, skip ADD LIVE VERSION.
- committed, no LIVE : the normal steady state after COMMIT consumed LIVE — restore it.
- committed, LIVE exists : ADD LIVE VERSION would fail with 099106 ("There is already a
live version"), which is the state a prior CREATE OR REPLACE
(e.g. dbt run --full-refresh) leaves behind — skip the add and
modify the existing LIVE in place. #}
{%- set _show_agents_sql -%}
SHOW AGENTS LIKE '{{ target_relation.identifier }}' IN SCHEMA {{ target_relation.database }}.{{ target_relation.schema }}
{%- endset -%}
{%- set _agent_rows = run_query(_show_agents_sql) -%}
{%- set _has_committed_versions = false -%}
{%- set _ns = namespace(has_committed=false, has_live=false) -%}
{%- if _agent_rows | length > 0 -%}
{%- set _show_versions_sql -%}
SHOW VERSIONS IN AGENT {{ target_relation.database }}.{{ target_relation.schema }}.{{ target_relation.identifier }}
{%- endset -%}
{%- set _version_rows = run_query(_show_versions_sql) -%}
{%- set _has_committed_versions = (_version_rows | length) > 0 -%}
{%- for _r in _version_rows.rows -%}
{%- if _r['name'] -%}
{%- set _ns.has_committed = true -%}
{%- else -%}
{%- set _ns.has_live = true -%}
{%- endif -%}
{%- endfor -%}
{%- endif -%}
{%- set _has_committed_versions = _ns.has_committed -%}
{%- set _has_live_version = _ns.has_live -%}

{% call statement('main') %}
{{ dbt_cortex_agent.snowflake__create_cortex_agent_if_not_exists(target_relation, sql, comment, profile) }}
{% endcall %}

{%- if _has_committed_versions %}
{%- if _has_committed_versions and not _has_live_version %}
{# COMMIT consumes LIVE and does not recreate it. Explicitly restore LIVE from the
last committed version before modifying, so MODIFY LIVE VERSION has a target. #}
last committed version before modifying, so MODIFY LIVE VERSION has a target.
Skipped when a LIVE already exists — adding a second one errors with 099106. #}
{% call statement('add_live') %}
{{ dbt_cortex_agent.snowflake__add_cortex_agent_live_version(target_relation) }}
{% endcall %}
Expand All @@ -122,6 +147,15 @@
{% call statement('commit_version') %}
{{ dbt_cortex_agent.snowflake__commit_cortex_agent_version(target_relation, version_comment) }}
{% endcall %}

{%- if set_default_version %}
{# COMMIT does not move the DEFAULT pointer. Promote explicitly so the version just
deployed is the one users get, and so an agent whose pointer was pinned
out-of-band (e.g. via Snowsight) is un-stuck on the next run. #}
{% call statement('set_default_version') %}
{{ dbt_cortex_agent.snowflake__set_cortex_agent_default_version(target_relation) }}
{% endcall %}
{%- endif %}
{%- endif %}

{%- else %}
Expand Down
24 changes: 24 additions & 0 deletions macros/relations/cortex_agent/versioned_agent.sql
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,27 @@
{%- endif %}

{% endmacro %}


-- Promotes the newest committed version to DEFAULT — the version end users and
-- Snowsight see as "In use".
--
-- Why this is required: COMMIT creates a version but does NOT move the DEFAULT
-- pointer. Snowflake only tracks the latest committed version *implicitly*, and
-- that implicit behaviour stops the moment DEFAULT_VERSION is set explicitly by
-- anything out-of-band (notably Snowsight's Publish button). From then on every
-- dbt deploy commits a version that no user ever sees. Setting it explicitly on
-- each commit makes promotion deterministic and self-healing: a pinned agent is
-- un-stuck by the next dbt run.
--
-- 'LAST' is used rather than a computed VERSION$N so the statement needs no version
-- bookkeeping and stays idempotent. Note the quotes are required: the unquoted forms
-- (LAST, VERSION$3, "VERSION$3") are all rejected with a SQL compilation error, despite
-- what the Snowflake docs show — only the single-quoted string form parses.
{% macro snowflake__set_cortex_agent_default_version(relation) %}

alter agent
{{ relation.database }}.{{ relation.schema }}.{{ relation.identifier }}
set default_version = 'LAST'

{% endmacro %}
Loading