Make config graph children first-class

CHANGELOG.md

@@ -19,6 +19,10 @@ Pre-1.0 note: while `pg_durable` is in major version `0`, minor releases may inc
 
 - **`df.ensure_durofut(text)`:** removed this undocumented internal helper. The `0.2.5 -> 0.2.6` upgrade replaces its operator callers before dropping it with `RESTRICT`; customer-owned dependent objects must be changed or removed before upgrading.
 
+### Changed
+
+- **Workflow graph envelope:** IF/LOOP conditions and additional JOIN branches are now first-class opaque children instead of JSON embedded inside the `query` string. This removes the remaining parser-depth limit and repeated escaping for config-nested graphs. The transient `Durofut` JSON representation changes; persisted `df.nodes` rows and in-flight instances are unaffected.
+
 ## [0.2.5] - 2026-07-30
 
 ### Added

docs/nested-graph-design.md

@@ -2,27 +2,19 @@
 
 ## Motivation
 
-The current DSL implementation inserts nodes into `df.nodes` during graph construction (e.g., inside `df.sql()`, `df.join()`, etc.). This creates several problems:
+The original DSL implementation inserted nodes into `df.nodes` during graph construction (e.g., inside `df.sql()`, `df.join()`, etc.). This created several problems:
 
 1. **Premature database writes**: Graph construction performs I/O before `df.start()` is called
-2. **Orphaned nodes**: Errors during graph construction leave partial state in the database with no `instance_id`
-3. **No transaction management**: Nodes inserted before `df.start()` don't participate in transaction rollback
-4. **Complex explain mode**: Requires temporary tables and session variables to avoid polluting the database
-5. **No optimization opportunities**: Graph cannot be analyzed or transformed before execution
-6. **Accidental pollution**: Users experimenting with DSL expressions create database state
+2. **Abandoned nodes**: Successfully constructed graphs that are never passed to `df.start()` leave rows with no instance
+3. **Complex explain mode**: Requires temporary tables and session variables to avoid polluting the database
+4. **No optimization opportunities**: Graph cannot be analyzed or transformed before execution
+5. **Accidental pollution**: Users experimenting with DSL expressions create database state
 
-### Example of Current Issues
+### Example of the Previous Issue
 
 ```sql
--- Error creates orphaned node
-SELECT df.sql('SELECT 1') ~> df.sql('SYNTAX ERROR');
--- First node inserted to df.nodes with no instance_id, never cleaned up
-
--- Transaction rollback doesn't help
-BEGIN;
+-- Constructing a graph without starting it left unowned rows behind.
 SELECT df.sql('SELECT 1');
-ROLLBACK;
--- Node still in df.nodes
 ```
 
 ## Design Approach: Nested JSON
@@ -37,15 +29,37 @@ DSL functions return **self-contained JSON** that embeds the complete subtree, n
 #[derive(Debug, Clone, Serialize, Deserialize)]
 pub struct Durofut {
     pub node_type: String,
+    pub left_node: Option<Box<RawValue>>,
+    pub right_node: Option<Box<RawValue>>,
+    pub condition_node: Option<Box<RawValue>>,
+    pub extra_nodes: Vec<Box<RawValue>>,
     pub query: Option<String>,
     pub result_name: Option<String>,
-    // Children are embedded, not referenced by ID
-    // Node IDs are generated during df.start(), not during construction
-    pub left_node: Option<Box<Durofut>>,
-    pub right_node: Option<Box<Durofut>>,
 }
 ```
 
+### Serialization Boundary
+
+`Durofut` is the transient graph-construction envelope returned by DSL functions. Its child fields
+hold opaque JSON objects as `RawValue`, so deserializing one node does not recursively deserialize
+its whole subtree. Walkers parse each child with a fresh deserializer as they descend. This avoids
+serde_json's recursion ceiling while preserving the configured graph depth and node-count limits.
+
+Conditions for IF/LOOP and additional JOIN branches are first-class `condition_node` and
+`extra_nodes` fields. They do not live inside the escaped `query` string, so nested config graphs
+grow linearly and use the same traversal path as left and right children.
+
+The envelope is distinct from durable execution state. During `df.start()`, config children are
+materialized as node rows and `df.nodes.query` retains the worker-facing format with child IDs:
+
+```json
+{"condition_node":"a1b2c3d4"}
+```
+
+Changing the envelope therefore does not change queued or in-flight instances, `FunctionNode`, or
+duroxide replay state. Serialized `Durofut` text is an unversioned DSL representation and is not a
+cross-version persistence contract.
+
 ### Example Flow
 
 ```sql
@@ -131,239 +145,11 @@ CREATE TEMP TABLE _dsl_nodes (LIKE df.nodes) ON COMMIT DROP;
 - PostgreSQL temp table overhead for every DSL session
 - Nested JSON is simpler and faster
 
-## Implementation Plan
-
-### Phase 1: Update Type Definitions
-
-**File**: `src/types.rs`
-
-1. **Change `Durofut` structure**:
-   ```rust
-   pub struct Durofut {
-       pub node_type: String,
-       pub query: Option<String>,
-       pub result_name: Option<String>,
-       // Embed children directly, not by ID reference
-       // node_id is removed - IDs generated during df.start()
-       pub left_node: Option<Box<Durofut>>,
-       pub right_node: Option<Box<Durofut>>,
-   }
-   ```
-
-2. **Update serialization**:
-   - `to_json()` - already works with serde
-   - `from_json()` - already works with serde
-   - `is_durofut()` - simplified to just check deserialization
-   - `ensure()` - no node_id needed
-
-3. **Remove `insert_node()` method** - no longer needed in DSL functions
-
-4. **Remove `is_explain_mode()` function** - no longer needed
-
-5. **Update `ensure()` to not set node_id**:
-
-### Phase 2: Update DSL Functions
-
-**File**: [src/dsl.rs](../src/dsl.rs)
-
-All DSL functions updated to embed children as `Box<Durofut>` instead of storing ID references. No database writes during construction — just JSON composition.
-
-**Before** (old pattern — `join` as example):
-```rust
-let durofut = Durofut {
-    node_id: short_id(),
-    left_node: Some(a_fut.node_id),  // ID reference
-    right_node: Some(b_fut.node_id), // ID reference
-    ...
-};
-durofut.insert_node();  // Database write
-```
-
-**After** (new pattern):
-```rust
-let durofut = Durofut {
-    node_type: "JOIN".to_string(),
-    left_node: Some(Box::new(a_fut)),  // Embed child
-    right_node: Some(Box::new(b_fut)), // Embed child
-    ..Default::default()
-};
-durofut.to_json()  // No database write, no node_id
-```
-
-See [src/dsl.rs](../src/dsl.rs) for the full implementation.
-
-### Phase 3: Update `df.start()`
-
-**File**: [src/dsl.rs](../src/dsl.rs)
-
-`link_nodes()` replaced with `insert_nodes()` — a recursive function that:
-1. Generates node IDs via `short_id()` during insertion (post-order traversal)
-2. Inserts children before parents so their IDs are available
-3. Handles config-embedded children (IF condition, JOIN3 extras, LOOP condition) via `transform_config_children()`
-4. No `HashSet` needed — single tree traversal
-
-See `start()` → `insert_nodes()` in [src/dsl.rs](../src/dsl.rs) for the full implementation.
-
-### Phase 4: Update `df.explain()`
-
-**File**: [src/explain.rs](../src/explain.rs)
-
-Drastically simplified — no temp tables or database access needed for DSL expressions:
+## Compatibility
 
-1. `explain()` dispatches between instance IDs (8-char hex) and DSL expressions
-2. `explain_expression()` parses the nested JSON via `Durofut::ensure()`, then builds an in-memory node map with generated IDs (N1, N2, ...)
-3. `collect_nodes()` walks the tree recursively, handling config-embedded children via `transform_config_children()`
-4. Visualization uses the existing `build_tree_visualization()`
-
-**Removed**: `is_explain_mode()` checks, `df._explain_mode` session variable, `_durable_explain_nodes` temp table.
-
-See [src/explain.rs](../src/explain.rs) for the full implementation.
-
-### Phase 5: Test Updates
-
-#### Unit Tests
-
-**File**: `src/lib.rs` or test modules
-
-- Update any tests that inspect `df.nodes` before `df.start()`
-- Tests should now only verify JSON structure, not database state
-- Add tests for nested graph construction
-
-Example:
-```rust
-#[pg_test]
-fn test_nested_graph_construction() {
-    let result = Spi::get_one::<String>(
-        "SELECT df.sql('SELECT 1') ~> df.sql('SELECT 2')"
-    ).expect("query failed");
-
-    let graph: Durofut = serde_json::from_str(&result).expect("parse failed");
-    assert_eq!(graph.node_type, "THEN");
-    assert!(graph.left_node.is_some());
-    assert!(graph.right_node.is_some());
-}
-```
-
-#### E2E Tests
-
-**Directory**: `tests/e2e/sql/`
-
-Most E2E tests should work **without changes** because they:
-1. Build DSL expression
-2. Call `df.start()`
-3. Wait for completion
-4. Assert results
-
-**Tests that will need updates**:
-- `10_explain.sql` - Must change from `$$...$$` syntax to passing Durofut JSON or plain SQL directly to `df.explain()`
-- Any test that queries `df.nodes` before calling `df.start()`
-- Tests that verify node count or structure before execution
-
-**What to verify after changes**:
-- All existing E2E tests pass
-- Node insertion happens correctly in `df.start()`
-- `instance_id` is always set on all nodes
-- No orphaned nodes in `df.nodes` after errors
-
-### Phase 6: Documentation Updates
-
-#### USER_GUIDE.md
-
-Update sections on:
-- **Graph Construction**: Clarify that DSL functions don't write to database
-- **df.explain()**: Update to show it works on DSL expressions directly (no temp tables)
-- **Debugging**: Update guidance on inspecting graph JSON
-
-Example addition:
-```markdown
-### Understanding Graph Construction
-
-DSL functions build graph structures in memory without touching the database:
-
-```sql
--- This creates JSON, not database records
-SELECT df.sql('SELECT 1') ~> df.sql('SELECT 2');
-
--- Only df.start() writes to the database
-SELECT df.start(df.sql('SELECT 1') ~> df.sql('SELECT 2'));
-```
-
-You can inspect the graph structure by examining the JSON:
-```sql
-SELECT df.sql('SELECT 1') ~> df.sql('SELECT 2');
--- Returns: {"node_type":"THEN",...}
-```
-```
+The SQL API and durable execution format are unchanged. `df.start()` still materializes graphs as
+flat `df.nodes` rows, and queued or in-flight instances continue to use the same `FunctionNode`
+representation.
 
-#### docs/ARCHITECTURE.md
-
-Update:
-- Data flow section to reflect new construction model
-- Remove mention of temp tables for explain mode
-- Add section on stateless DSL design
-
-#### README.md
-
-Update quick start examples if they reference graph construction details.
-
-## Implementation Checklist
-
-All items completed in commit `fdfbd44` and subsequent fixes:
-
-- [x] Update `Durofut` struct in `types.rs` to use `Box<Durofut>` for children
-- [x] Remove `insert_node()` method from `Durofut`
-- [x] Update `Durofut::ensure()` to not call `insert_node()`
-- [x] Remove `is_explain_mode()` function from `types.rs`
-- [x] Update all DSL functions in `dsl.rs` to embed children instead of storing IDs
-- [x] Remove `insert_node()` calls from all DSL functions
-- [x] Update `df.as_named()` to remove UPDATE query
-- [x] Replace `link_nodes()` with `insert_nodes()` in `df.start()`
-- [x] Add helper function for config node extraction (`for_each_config_child`, `transform_config_children`)
-- [x] Simplify `df.explain()` in `explain.rs` to parse nested JSON
-- [x] Remove temp table logic from `explain_expression()`
-- [x] Remove `df._explain_mode` session variable usage
-- [x] Update unit tests to verify JSON structure
-- [x] Run all E2E tests and fix any failures
-- [x] Update `USER_GUIDE.md` with new graph construction model
-- [x] Update `docs/ARCHITECTURE.md` with stateless design
-- [x] Update `README.md` if needed (checked — no stale references)
-- [x] Run `cargo fmt`
-- [x] Run `cargo clippy --features pg17` and fix warnings
-- [x] Run `./scripts/test-unit.sh`
-- [x] Run `./scripts/test-e2e-local.sh`
-
-## Migration Notes
-
-### Backward Compatibility
-
-**Breaking changes**:
-- Durofut JSON structure changes (children embedded vs. referenced)
-- Code relying on inspecting `df.nodes` before `df.start()` will break
-
-**Non-breaking**:
-- All SQL APIs remain the same (`df.sql()`, `df.start()`, etc.)
-- Existing workflows continue to work
-- Database schema unchanged
-
-### Rollout Strategy
-
-Completed — shipped in commit `fdfbd44` on branch `pinodeca/nested-graph` (PR #5).
-
-## Success Criteria
-
-✅ All unit tests pass
-✅ All E2E tests pass
-✅ No database writes during DSL construction
-✅ No orphaned nodes in `df.nodes`
-✅ `df.explain()` works on DSL expressions
-✅ No clippy warnings or format issues
-✅ Documentation updated and clear
-
-## Future Enhancements
-
-Once this is in place, we can:
-- **Graph optimization**: Dead code elimination, common subexpression elimination
-- **Validation**: Check for errors before execution (invalid SQL syntax, missing variables)
-- **Visualization**: Web UI for graph structure
-- **Compilation**: Pre-compile graphs to optimized execution plans
-- **Debugging**: Step through graph execution with breakpoints
+The serialized `Durofut` envelope is an internal, unversioned DSL representation. Its shape may
+change between releases, so applications should not persist it as a cross-version workflow format.

src/dsl.rs

@@ -324,21 +324,12 @@ pub fn wait_for_schedule(cron_expr: &str) -> String {
 #[pg_extern(name = "loop", schema = "df")]
 pub fn loop_fn(body: &str, condition: default!(Option<&str>, "NULL")) -> String {
     let body_fut = Durofut::ensure(body);
-
-    let query = if let Some(cond) = condition {
-        let cond_fut = Durofut::ensure(cond);
-        let config = serde_json::json!({
-            "condition_node": cond_fut
-        });
-        Some(config.to_string())
-    } else {
-        None
-    };
+    let condition_node = condition.map(|cond| Durofut::ensure(cond).into_raw());
 
     Durofut {
         node_type: "LOOP".to_string(),
         left_node: Some(body_fut.into_raw()),
-        query,
+        condition_node,
         ..Default::default()
     }
     .to_json()
@@ -384,15 +375,11 @@ pub fn if_fn(condition: &str, then_branch: &str, else_branch: &str) -> String {
     let then_fut = Durofut::ensure(then_branch);
     let else_fut = Durofut::ensure(else_branch);
 
-    let config = serde_json::json!({
-        "condition_node": condition_fut
-    });
-
     Durofut {
         node_type: "IF".to_string(),
         left_node: Some(then_fut.into_raw()),
         right_node: Some(else_fut.into_raw()),
-        query: Some(config.to_string()),
+        condition_node: Some(condition_fut.into_raw()),
         ..Default::default()
     }
     .to_json()
@@ -445,15 +432,11 @@ pub fn join3(a: &str, b: &str, c: &str) -> String {
     let b_fut = Durofut::ensure(b);
     let c_fut = Durofut::ensure(c);
 
-    let config = serde_json::json!({
-        "extra_nodes": [c_fut]
-    });
-
     Durofut {
         node_type: "JOIN".to_string(),
         left_node: Some(a_fut.into_raw()),
         right_node: Some(b_fut.into_raw()),
-        query: Some(config.to_string()),
+        extra_nodes: vec![c_fut.into_raw()],
         ..Default::default()
     }
     .to_json()