Skip to content

feat: cascade TTL to container sub-tasks (rapyer >=1.3.4) - #137

Open
yedidyakfir wants to merge 14 commits into
developfrom
feat/136-cascade-ttl
Open

feat: cascade TTL to container sub-tasks (rapyer >=1.3.4)#137
yedidyakfir wants to merge 14 commits into
developfrom
feat/136-cascade-ttl

Conversation

@yedidyakfir

@yedidyakfir yedidyakfir commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

Closes #136

Summary

Upgrade rapyer to >=1.3.4 and use its new features so that writing to a container signature (swarm / chain) refreshes its TTL and cascades that refresh to all of its sub-tasks, keeping the whole subtree alive together while the container is active. Purely declarative model config — no explicit update_ttl method.

What changed

  1. Upgrade rapyer >=1.3.4,<1.4.0 across third-magic, mageflow-mcp, mage-voyance; re-locked every workspace. Adapted to 1.3.x breaking changes:
    • instance apipeline re-fetches on entry → swarm add_tasks uses ignore_redis_error=True (model not yet committed during atomic create+add).
    • RedisConfig is now a pydantic model → apply_ttl_config + TTL test helpers use model_copy instead of dataclasses.replace.
  2. Cascade edges: SwarmTaskSignature / ChainTaskSignature gain a sub_task_refs: Annotated[list[Reference[TaskSignature]], CascadeTTL()] field mirroring their children, plus a Meta with refresh_ttl scoped to write actions. The existing tasks key-lists and all their consumers are untouched.
  3. Tests: fakeredis unit tests (config wiring, cascade plan, sub_task_refs persistence) + a real-Redis integration suite that verifies a container write extends its sub-tasks' TTL.

Cascade semantics

rapyer re-arms each reached child to its own Meta.ttl (keep-the-subtree-alive-together), not "copy one number onto every child". With mageflow's uniform active TTL this is the intended behavior.

Testing note

Cascade edge-following needs a real Redis Stack (Functions + RedisJSON); fakeredis only re-arms the root. The integration suite (libs/third-magic/tests/integration) runs against THIRDMAGIC_TEST_REDIS_URL and skips when no JSON-capable Redis is present. Verified locally against redis-stack-server.

  • third-magic unit: 139 passed
  • mageflow unit: 263 passed
  • third-magic integration: 2 passed (real Redis Stack)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Container workflows now refresh and cascade TTLs to referenced sub-tasks when tasks are created or updated.
    • Chain and swarm workflows preserve task references during persistence and reloading.
    • Added integration support for Redis Stack environments and expanded cascade-TTL validation.
  • Bug Fixes

    • Updated TTL configuration handling to preserve existing metadata while applying changes.
  • Changelog

    • Documented cascade-TTL behavior and compatibility updates.

yedidyakfir and others added 4 commits July 24, 2026 17:53
Bump the rapyer cap to >=1.3.4,<1.4.0 across third-magic, mageflow-mcp
and mage-voyance and re-lock every workspace. Adapt to 1.3.x changes:
- instance apipeline now re-fetches on entry, so add_tasks uses
  ignore_redis_error=True (the swarm is not yet committed during the
  atomic create + add_tasks pipeline).
- RedisConfig is now a pydantic model, so apply_ttl_config and the TTL
  test helpers use model_copy(update=...) instead of dataclasses.replace.

Full unit suites green on 1.3.4 (third-magic 134, mageflow 263).

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Declare a sub_task_refs ForeignKey collection on SwarmTaskSignature and
ChainTaskSignature, annotated with CascadeTTL(), mirroring the child
tasks. Containers get a Meta with refresh_ttl scoped to write actions,
so any write refreshes the container's TTL and cascades that refresh to
its sub-task edges (real-Redis cascade; fakeredis re-arms root only).

The existing tasks key-lists and their consumers are untouched — the
FK field is an additional cascade edge, populated at chain creation and
in swarm add_tasks.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cascade edge-following needs Redis Functions + RedisJSON, which fakeredis
cannot emulate, so add an integration suite that runs against a real
Redis Stack (THIRDMAGIC_TEST_REDIS_URL, default redis://localhost:6379)
and skips when unavailable or the RedisJSON module is missing. Verifies
that a write to a swarm/chain extends its sub-tasks' TTL.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: b57de4bc-250a-4e09-891d-edb936a3f740

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Changes

Container TTL Cascade

Layer / File(s) Summary
RapyER compatibility and TTL metadata
libs/*/pyproject.toml, libs/mageflow/..., CHANGELOG.md
RapyER constraints are updated to 1.3.4, TTL metadata uses model_copy, and compatibility changes are documented.
Container TTL policy
libs/third-magic/thirdmagic/container.py
Container write actions receive a 24-hour TTL policy.
Chain and swarm reference persistence
libs/third-magic/thirdmagic/{chain,swarm}/*
Chain and swarm signatures store Reference[TaskSignature] values and configure cascading TTL behavior.
Cascade and persistence validation
libs/third-magic/tests/{unit,integration}/*
Tests verify cascade metadata, reference persistence, Redis TTL refreshes, and chain/swarm behavior.

MCP Documentation Server

Layer / File(s) Summary
RapyER documentation MCP server
.codex/config.toml
Adds an MCP server configuration using npx and mcp-remote.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Container as Chain/Swarm Container
  participant Redis
  participant Task as Referenced TaskSignature
  Container->>Redis: Perform configured write action
  Redis->>Task: Cascade TTL refresh through sub_task_refs
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 2 | ❌ 3

❌ Failed checks (3 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The rapyer upgrade and TTL meta changes are present, but the required tasks and publishing_state_id Reference conversions are not shown. Convert swarm/chain tasks and swarm publishing_state_id to Reference[T], refactor self.tasks consumers, and re-lock the uv workspaces.
Out of Scope Changes check ⚠️ Warning The new .codex/config.toml MCP server entry is unrelated to the rapyer/TTL work and appears to be an extra scope addition. Move the MCP server config to a separate PR unless it is required for this change.
Docstring Coverage ⚠️ Warning Docstring coverage is 6.25% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately describes the main change: cascading TTL for container sub-tasks with the rapyer upgrade.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/136-cascade-ttl

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.codex/config.toml:
- Around line 2-5: Pin the mcp-remote dependency at an approved exact version in
the frontend package manifest and lockfile, then update the command
configuration to invoke that installed pinned package rather than the
unversioned npx resolution. Preserve the existing MCP endpoint and arguments
while changing the executable reference.

In `@libs/third-magic/tests/integration/conftest.py`:
- Around line 45-49: Update the mock_adapter fixture to snapshot
Signature.ClientAdapter before replacing it, then restore the original value in
a finally block surrounding the yield. Keep yielding the MagicMock adapter while
the test runs and ensure restoration occurs even when the test raises.
- Around line 26-42: Update the real_redis fixture to require a dedicated test
Redis endpoint/database before running integration tests, and replace both
client.flushall() calls with flushdb(). Preserve the existing setup and teardown
behavior while ensuring cleanup only affects the selected database, not every
database on the configured Redis server.

In `@libs/third-magic/tests/integration/test_cascade_ttl.py`:
- Around line 29-31: Expand both cascade TTL test sites in
libs/third-magic/tests/integration/test_cascade_ttl.py#L29-L31 and `#L48-L50` with
distinct parent and child TTL configurations. Assert the parent key refreshes
near its configured TTL, while every child key is re-armed near its own
configured TTL rather than the container’s value.

In `@libs/third-magic/tests/unit/test_cascade_ttl.py`:
- Line 20: Update the pytest.mark.parametrize declaration for container_cls to
use the project-required list-of-lists case format, while preserving the
existing SwarmTaskSignature and ChainTaskSignature parameter values.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 6f715760-2cb0-495f-bc21-447e7e0d9870

📥 Commits

Reviewing files that changed from the base of the PR and between 52e0a45 and f9ee856.

⛔ Files ignored due to path filters (9)
  • .DS_Store is excluded by !**/.DS_Store
  • libs/.DS_Store is excluded by !**/.DS_Store
  • libs/mage-voyance/uv.lock is excluded by !**/*.lock, !**/*.lock, !**/uv.lock
  • libs/mageflow-e2e/uv.lock is excluded by !**/*.lock, !**/*.lock, !**/uv.lock
  • libs/mageflow-mcp/uv.lock is excluded by !**/*.lock, !**/*.lock, !**/uv.lock
  • libs/mageflow/.DS_Store is excluded by !**/.DS_Store
  • libs/mageflow/uv.lock is excluded by !**/*.lock, !**/*.lock, !**/uv.lock
  • libs/third-magic/.DS_Store is excluded by !**/.DS_Store
  • libs/third-magic/uv.lock is excluded by !**/*.lock, !**/*.lock, !**/uv.lock
📒 Files selected for processing (16)
  • .codex/config.toml
  • CHANGELOG.md
  • libs/mage-voyance/pyproject.toml
  • libs/mageflow-mcp/pyproject.toml
  • libs/mageflow/mageflow/config.py
  • libs/mageflow/tests/unit/test_remove_ttl.py
  • libs/mageflow/tests/unit/test_ttl_config.py
  • libs/third-magic/pyproject.toml
  • libs/third-magic/tests/integration/__init__.py
  • libs/third-magic/tests/integration/conftest.py
  • libs/third-magic/tests/integration/test_cascade_ttl.py
  • libs/third-magic/tests/unit/test_cascade_ttl.py
  • libs/third-magic/thirdmagic/chain/creator.py
  • libs/third-magic/thirdmagic/chain/model.py
  • libs/third-magic/thirdmagic/container.py
  • libs/third-magic/thirdmagic/swarm/model.py

Comment thread .codex/config.toml Outdated
Comment on lines +2 to +5
command = "npx"
args = [
"mcp-remote",
"https://gitmcp.io/imaginary-cherry/rapyer",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/sh
set -eu

echo "== files =="
git ls-files | grep -E '(^|/)\.codex/config\.toml|\.mcp\.json|package(-lock)?\.json|pnpm-lock\.yaml|yarn\.lock|package-lock\.json' || true

echo
echo "== .codex/config.toml =="
cat -n .codex/config.toml 2>/dev/null || true

echo
echo "== search mcp-remote usage =="
rg -n "mcp-remote|mcp" .codex .mcp.json package.json package-lock.json pnpm-lock.yaml yarn.lock 2>/dev/null || true

Repository: imaginary-cherry/mageflow

Length of output: 817


🌐 Web query:

npx mcp-remote executable package version unpinned security supply chain documentation

💡 Result:

The use of npx to execute mcp-remote introduces significant supply chain security risks if package versions are not pinned. The practice of using unpinned commands like npx mcp-remote (or npx -y mcp-remote) allows the package manager to fetch the latest available version from the npm registry, which can lead to the silent introduction of malicious code or vulnerable versions [1][2][3]. A critical security vulnerability, CVE-2025-6514, was disclosed in July 2025 affecting mcp-remote versions 0.0.5 through 0.1.15 [4][2]. This vulnerability enabled remote code execution (RCE) via OS command injection when a user connected to an untrusted MCP server that provided a malicious authorization_endpoint URL [4][5]. This flaw highlights the dangers of running unpinned, automated tooling in environments with access to sensitive local credentials and system resources [1][6]. To secure your MCP environment, implement the following supply chain hygiene practices: 1. Pin Exact Versions: Never use "latest" or semver ranges (e.g., ^0.1.16) for MCP servers or proxies. Pin to an exact version in your configuration and lockfiles (e.g., "mcp-remote": "0.1.16") [1][2][3]. 2. Avoid Automated "npx" Patterns: In production or sensitive development environments, replace npx one-liners with explicit local installations. Use npm install to manage versions via a package.json and package-lock.json or equivalent, which allows for integrity hashing and consistent dependency resolution [7][2][3]. 3. Audit Dependencies: Treat every MCP server as a third-party dependency. Before installation, review the source code, audit the manifest for requested permissions (e.g., filesystem access, shell commands), and verify the dependency tree for known vulnerabilities [3][8]. 4. Principle of Least Privilege: If you must use a proxy like mcp-remote, ensure it is updated to at least version 0.1.16 [4][1]. Run MCP servers in isolated environments (such as containers or restricted OS users) with no ambient cloud credentials and limited filesystem access [9][6]. 5. Enforce Authentication and Transport Security: Connect only to trusted servers over HTTPS and require secure authentication standards, such as OAuth 2.1 with PKCE, to mitigate intercept and man-in-the-middle attacks [1][2][9].

Citations:


Pin the mcp-remote executable version.

npx mcp-remote resolves the latest package version from npm at each invocation, so this agent MCP tool can execute unreviewed code or regress to a vulnerable version. Add mcp-remote to the frontend package/lockfile at an approved exact version and reference the pinned package in .codex/config.toml instead of the unversioned uninstalled command.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.codex/config.toml around lines 2 - 5, Pin the mcp-remote dependency at an
approved exact version in the frontend package manifest and lockfile, then
update the command configuration to invoke that installed pinned package rather
than the unversioned npx resolution. Preserve the existing MCP endpoint and
arguments while changing the executable reference.

Comment on lines +26 to +42
@pytest_asyncio.fixture
async def real_redis():
client = aioredis.Redis.from_url(REDIS_URL)
try:
await client.ping()
except Exception:
pytest.skip(f"No Redis reachable at {REDIS_URL}")
if not await _has_json_module(client):
await client.aclose()
pytest.skip(f"Redis at {REDIS_URL} lacks the RedisJSON module")
await client.flushall()
await rapyer.init_rapyer(client)
try:
yield client
finally:
await client.flushall()
await client.aclose()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Avoid FLUSHALL on a configurable Redis endpoint.

This erases every database on whatever server THIRDMAGIC_TEST_REDIS_URL targets. Require a dedicated test endpoint/database and use FLUSHDB instead.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/third-magic/tests/integration/conftest.py` around lines 26 - 42, Update
the real_redis fixture to require a dedicated test Redis endpoint/database
before running integration tests, and replace both client.flushall() calls with
flushdb(). Preserve the existing setup and teardown behavior while ensuring
cleanup only affects the selected database, not every database on the configured
Redis server.

Comment on lines +45 to +49
@pytest.fixture
def mock_adapter():
adapter = MagicMock(spec=BaseClientAdapter)
Signature.ClientAdapter = adapter
yield adapter

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Restore the global client adapter after the fixture.

Signature.ClientAdapter remains a MagicMock after teardown, contaminating later tests. Snapshot it before assignment and restore it in a finally block.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/third-magic/tests/integration/conftest.py` around lines 45 - 49, Update
the mock_adapter fixture to snapshot Signature.ClientAdapter before replacing
it, then restore the original value in a finally block surrounding the yield.
Keep yielding the MagicMock adapter while the test runs and ensure restoration
occurs even when the test raises.

Comment on lines +29 to +31
# Assert
for task in tasks:
assert await real_redis.ttl(task.key) > SHORT_TTL

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Verify that cascading uses each child’s TTL, not the container TTL.

Both assertions pass if children are incorrectly refreshed with the parent’s TTL. Configure distinct parent/child TTLs, assert the parent refreshes, and assert each child is re-armed near its own configured value.

  • libs/third-magic/tests/integration/test_cascade_ttl.py#L29-L31: add distinct swarm/task TTL assertions.
  • libs/third-magic/tests/integration/test_cascade_ttl.py#L48-L50: add distinct chain/task TTL assertions.

As per path instructions, **/tests/**: “Focus on test coverage completeness and edge cases.”

📍 Affects 1 file
  • libs/third-magic/tests/integration/test_cascade_ttl.py#L29-L31 (this comment)
  • libs/third-magic/tests/integration/test_cascade_ttl.py#L48-L50
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/third-magic/tests/integration/test_cascade_ttl.py` around lines 29 - 31,
Expand both cascade TTL test sites in
libs/third-magic/tests/integration/test_cascade_ttl.py#L29-L31 and `#L48-L50` with
distinct parent and child TTL configurations. Assert the parent key refreshes
near its configured TTL, while every child key is re-armed near its own
configured TTL rather than the container’s value.

Source: Path instructions

)


@pytest.mark.parametrize("container_cls", [SwarmTaskSignature, ChainTaskSignature])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Use list-of-lists for parametrization.

Change the parameter cases to the project-required list-of-lists format.

Proposed fix
-@pytest.mark.parametrize("container_cls", [SwarmTaskSignature, ChainTaskSignature])
+@pytest.mark.parametrize(
+    "container_cls",
+    [[SwarmTaskSignature], [ChainTaskSignature]],
+)

As per path instructions, pytest.mark.parametrize must use list-of-lists format.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
@pytest.mark.parametrize("container_cls", [SwarmTaskSignature, ChainTaskSignature])
`@pytest.mark.parametrize`(
"container_cls",
[[SwarmTaskSignature], [ChainTaskSignature]],
)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/third-magic/tests/unit/test_cascade_ttl.py` at line 20, Update the
pytest.mark.parametrize declaration for container_cls to use the
project-required list-of-lists case format, while preserving the existing
SwarmTaskSignature and ChainTaskSignature parameter values.

Source: Path instructions

Match the existing mageflow integration convention: the real_redis
fixture connects to THIRDMAGIC_TEST_REDIS_URL and lets a missing/
unsuitable Redis surface as an error, rather than silently skipping.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address review: don't add a parallel sub_task_refs field. Convert the
existing chain/swarm tasks field itself to
Annotated[list[Reference[TaskSignature]], CascadeTTL()] so it is both
the child list and the cascade edge. task_ids returns the target keys,
and internal/test consumers that treated tasks as keys now use task_ids.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
task_ids builds the full key list; chain.acall and on_sub_task_done only
need one element, so read the target_key directly off self.tasks instead
of indexing into the materialized list.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

async def sub_tasks(self) -> list[TaskSignature]:
sub_tasks = await rapyer.afind(*self.tasks, skip_missing=True)
sub_tasks = await rapyer.afind(*self.task_ids, skip_missing=True)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't you use the reference interface for this? It will be more efficient, no?


async def acall(self, msg: Any, set_return_field: bool = True, **kwargs):
first_task = await rapyer.afind_one(self.tasks[0])
first_task = await rapyer.afind_one(self.task_ids[0])

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you take id from only on task, don't call tasks IDs which iterate over the entire list

async def suspend(self):
await asyncio.gather(
*[TaskSignature.suspend_from_key(task_id) for task_id in self.tasks],
*[TaskSignature.suspend_from_key(task_id) for task_id in self.task_ids],

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can iterate over the tasks and get the tasks id from each one instead of 2 iterations

async def interrupt(self):
await asyncio.gather(
*[TaskSignature.interrupt_from_key(task_id) for task_id in self.tasks],
*[TaskSignature.interrupt_from_key(task_id) for task_id in self.task_ids],

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can iterate over the tasks and get the tasks id from each one instead of 2 iterations

async def resume(self):
await asyncio.gather(
*[TaskSignature.resume_from_key(task_key) for task_key in self.tasks],
*[TaskSignature.resume_from_key(task_key) for task_key in self.task_ids],

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can iterate over the tasks and get the tasks id from each one instead of 2 iterations

Comment on lines +14 to +19
CONTAINER_WRITE_ACTIONS = (
ActionGroup.CREATE
| ActionGroup.UPDATE
| ActionGroup.APPEND
| ActionGroup.ERASE
| ActionGroup.ARITHMETIC

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update and create is enough, the rest are sub categories


async def sub_tasks(self) -> list[TaskSignature]:
tasks = await rapyer.afind(*self.tasks)
tasks = await rapyer.afind(*self.task_ids)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, use the reference API instea

async def change_status(self, status: SignatureStatus):
paused_chain_tasks = [
TaskSignature.safe_change_status(task, status) for task in self.tasks
TaskSignature.safe_change_status(task, status) for task in self.task_ids

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can iterate over the tasks and get the tasks id from each one instead of 2 iterations

async def suspend(self):
await asyncio.gather(
*[TaskSignature.suspend_from_key(swarm_id) for swarm_id in self.tasks],
*[TaskSignature.suspend_from_key(swarm_id) for swarm_id in self.task_ids],

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can iterate over the tasks and get the tasks id from each one instead of 2 iterations

async def resume(self):
await asyncio.gather(
*[TaskSignature.resume_from_key(task_id) for task_id in self.tasks],
*[TaskSignature.resume_from_key(task_id) for task_id in self.task_ids],

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can iterate over the tasks and get the tasks id from each one instead of 2 iterations

…efresh actions

- sub_tasks now resolves via the Reference API (ref.afetch) instead of
  rebuilding keys and calling afind.
- Container fan-outs (change_status/suspend/interrupt/resume) iterate
  self.tasks once and read ref.target_key, instead of iterating the
  materialized task_ids list.
- refresh_ttl is just CREATE | UPDATE (APPEND/ERASE/ARITHMETIC folded in).
- chain acall / on_sub_task_done already read a single ref directly.
- astatus test loads the container first, matching how mageflow.astatus
  invokes it (fresh, unresolved references).

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
self.tasks.extend(task_keys)
self.tasks.extend([Reference(task) for task in tasks])
self.tasks_left_to_run.extend(task_keys)
await self.asave()

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is very bad, why do we need to save after apipeline? it saves the changes automaticaly, asave can override changes from parallel worker and create race condition

- on_sub_task_done reads sub_task.chain_index (stored on each chain
  member at creation) and indexes the next task directly, instead of
  scanning the whole tasks list. chain_index lives on Signature so chain
  members of any type (task/swarm/chain) carry it.
- sub_tasks goes back to a single batched rapyer.afind (one Redis round
  trip) over the references' target keys, rather than per-reference afetch.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the leaky chain_index-on-Signature with a current_index pointer
on ChainTaskSignature. on_sub_task_done uses it as an O(1) fast path,
and only falls back to a scan (re-syncing the pointer) when the pointer
does not match the completed task — which also keeps it correct under
retry double-fires. Tasks no longer carry any chain-position state.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… asave in add_tasks

- add_tasks no longer calls asave(); reassigning self.tasks inside the
  apipeline persists safely without a blind full-document overwrite that
  could clobber a concurrent worker (review feedback).
- Integration tests spin up a real Redis Stack via testcontainers
  (AsyncRedisContainer) instead of a fixed URL, session-scoped, and run
  in CI (new 'Run Integration Tests' step; ubuntu runners have Docker).
- Add testcontainers[redis] to thirdmagic dev deps; ignore .DS_Store/.codex.

Refs #136

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
yedidyakfir and others added 3 commits July 26, 2026 15:55
rapyer.afind() with no keys returns every model in the DB. Several call
sites spread a possibly-empty iterable into it (task callbacks, sub_tasks),
so an empty set silently triggered a full-DB fetch — worst on every task
completion via activate_success/activate_error, where callbacks are usually
empty.

Add afind_keys_guarded(keys) in thirdmagic.utils that short-circuits to []
for an empty key set, and route the unguarded sites through it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge235p3Le4aLDjmP1AubRT
The `tasks` field annotation is a list[Reference] but rapyer converts it to a
RedisList at runtime, so `.extend()` is available and issues an atomic append.
Replace the read-modify-write reassignment (self.tasks = self.tasks + [...]),
which could clobber a concurrent worker's additions, with a race-safe extend.

Verified: thirdmagic unit (139) and the swarm cascade integration test pass, so
TTL cascade still fires on the extended list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ge235p3Le4aLDjmP1AubRT
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant