Skip to content

chore(deps): refresh CrawlKit and Cloudsmith CLI - #167

Merged
steipete merged 1 commit into
mainfrom
chore/deps-refresh-20260830
Sep 1, 2026
Merged

chore(deps): refresh CrawlKit and Cloudsmith CLI#167
steipete merged 1 commit into
mainfrom
chore/deps-refresh-20260830

Conversation

@steipete

@steipete steipete commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

What Problem This Solves

Refreshes the two remaining out-of-date dependencies. CrawlKit's release checker previously used an HTTP client without a timeout, so check-update could hang on an unresponsive server.

Why This Change Was Made

  • Update github.com/openclaw/crawlkit from v0.14.7 to v0.14.8, regenerating go.mod/go.sum with GOWORK=off go get -u -t ./... and GOWORK=off go mod tidy. Upstream release.
  • Update cloudsmith-cli 1.11.1 → 1.26.0 in both manual APT/RPM publishing workflows. Upstream release.
  • Checked every pinned Action and the Go toolchain, govulncheck, deadcode, sqlc, and pre-commit hooks against current upstream versions; those are already current. All explicitly required Go modules are current on their existing module paths. No new Go dependencies or replacements were introduced.

No major upgrades taken. Deferred major / maintainer decision: Bubble Tea, Bubbles, and Lip Gloss v2 use new charm.land/.../v2 module paths and incompatible view/input APIs (migration guide). Slacrawl imports CrawlKit's TUI; CrawlKit owns all three v1 dependencies. Recommend migrating the shared TUI in CrawlKit first, then consuming its release here, instead of forking or replacing that implementation in Slacrawl.

User Impact

Release-check HTTP requests now time out after 30 seconds. The changelog and command reference describe that behavior. Cloudsmith changes affect only manually invoked package publication.

Evidence

Local validation on macOS arm64, Go 1.27.0:

$ GOWORK=off make build check
all modules verified
No vulnerabilities found.
ok  github.com/openclaw/slacrawl/internal/cli          97.061s
ok  github.com/openclaw/slacrawl/internal/share       137.037s
ok  github.com/openclaw/slacrawl/internal/store        62.026s
[all 14 test packages passed; vet, deadcode, formatting, tidy, and CLI smoke passed]
• release succeeded after 44s

The snapshot built Linux/macOS AMD64/ARM64 binaries, archives, and Debian/RPM packages with --snapshot --skip=publish. actionlint and git diff --check also exited 0. Codex autoreview returned scoped-clean, with no findings at the requested default P0 priority.

Built CLI on a disposable Slack export

Created a synthetic export with one user, one channel, and two messages (one thread reply containing café 🦞), with a dedicated config/database and all live sources disabled. Real command/output excerpts:

$ ./bin/slacrawl --config .git/deps-refresh-20260830/live/config.toml --json import .git/deps-refresh-20260830/live/export --workspace TPROOF
{"workspace":"TPROOF","users":1,"channels":1,"dms":0,"mpims":0,"messages":2,"skipped":0,"dry_run":false,"elapsed":41689000}

$ ./bin/slacrawl --config .git/deps-refresh-20260830/live/config.toml --json search 'café'
"text": "Thread reply: Unicode café 🦞 remains searchable."
"thread_ts": "1788091200.000001"
"source_name": "slack-export"

$ ./bin/slacrawl --config .git/deps-refresh-20260830/live/config.toml --json sql 'select count(*) as messages, count(distinct channel_id) as channels from messages;'
[{"channels":1,"messages":2}]

$ ./bin/slacrawl --config .git/deps-refresh-20260830/live/config.toml --json tui --limit 2
[returned both imported message rows]

$ ./bin/slacrawl --config .git/deps-refresh-20260830/live/config.toml tui --limit 2
PASS: 140x40 terminal rendered both imported messages, including café; q exited 0.

The terminal check used a Python PTY harness that waited for both messages to render, sent q, and asserted exit 0.

Release-check timeout

Built a versioned CLI so the release checker would execute. A loopback HTTP proxy accepted its CONNECT api.github.com:443 request and deliberately withheld a response; the harness required the client's timeout error and a bounded exit:

$ GOWORK=off go build -ldflags '-X github.com/openclaw/slacrawl/internal/cli.version=0.8.6' -o .git/deps-refresh-20260830/slacrawl-releasecheck ./cmd/slacrawl
$ .git/deps-refresh-20260830/slacrawl-releasecheck check-update --force
error: check latest openclaw/slacrawl release: Get "https://api.github.com/repos/openclaw/slacrawl/releases/latest": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
PASS: stalled release check exited 1 after 35.67s (30s HTTP timeout).

Elapsed wall time includes process startup/scheduling on the shared host. The nonzero exit is the expected successful proof of timeout handling.

Cloudsmith workflow command compatibility

Installed the pinned version into a disposable Python virtual environment:

$ .git/deps-refresh-20260830/cloudsmith-venv/bin/python -m pip check
No broken requirements found.
$ .git/deps-refresh-20260830/cloudsmith-venv/bin/cloudsmith --version
Versions:
CLI Package Version: 1.26.0
API Package Version: 2.0.32
$ .git/deps-refresh-20260830/cloudsmith-venv/bin/cloudsmith push deb --help
Usage: cloudsmith push deb [OPTIONS] OWNER/REPO/DISTRO/RELEASE PACKAGE_FILE
  -W, --no-wait-for-sync          Don't wait for package synchronisation to
$ .git/deps-refresh-20260830/cloudsmith-venv/bin/cloudsmith push rpm --help
Usage: cloudsmith push rpm [OPTIONS] OWNER/REPO/DISTRO/RELEASE PACKAGE_FILE
  -W, --no-wait-for-sync          Don't wait for package synchronisation to

No authenticated package publication was run.

CI reasoning

Default branch was green at 1816d30: CI, Docker build/smoke, CodeQL, and secret scanning.

CI and docker are build/test gates. CodeQL and secret scanning are security gates. ClawSweeper dispatch, stale maintenance, release drafting, and manual Crabbox/release/Cloudsmith workflows are operational automation. No tests, assertions, or workflow gates were weakened. PR validation is green at 508c23a: CI — deps, lint, tests, and release snapshot, Docker build/smoke, CodeQL, and secret scanning. No failures or reruns were needed. The manual publishing workflows were validated locally without dispatching them.

@clawsweeper

clawsweeper Bot commented Aug 31, 2026

Copy link
Copy Markdown

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@clawsweeper clawsweeper Bot added merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 31, 2026
@clawsweeper

clawsweeper Bot commented Aug 31, 2026

Copy link
Copy Markdown

Codex review: needs maintainer review before merge. Reviewed August 31, 2026, 2:10 PM ET / 18:10 UTC.

ClawSweeper review

What this changes

The branch updates CrawlKit to v0.14.8 for a bounded release-check timeout and upgrades the Cloudsmith CLI used by manual APT and RPM publishing workflows.

Merge readiness

⚠️ Ready for maintainer review - 3 items remain

Keep open for explicit collaborator/maintainer review. The pinned branch contains the requested timeout and publishing-tool updates, has strong supplied runtime evidence, and no concrete patch defect was found.

Priority: P2
Reviewed head: 508c23a51132f1d4fcb406f252adfa3c402cef4c
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) The patch is narrow and well evidenced, with the remaining review question confined to accepting a newer CLI in credentialed manual publishing workflows.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): Slacrawl’s production release-check wrapper calls CrawlKit’s releasecheck.Check, and the PR supplies a real built-binary trace through that entrypoint with a deliberately stalled HTTP proxy; it observes the intended timeout. The unchanged publishing workflows are also exercised against Cloudsmith CLI 1.26.0 command help, though authenticated upload is appropriately left for maintainer judgment.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): Slacrawl’s production release-check wrapper calls CrawlKit’s releasecheck.Check, and the PR supplies a real built-binary trace through that entrypoint with a deliberately stalled HTTP proxy; it observes the intended timeout. The unchanged publishing workflows are also exercised against Cloudsmith CLI 1.26.0 command help, though authenticated upload is appropriately left for maintainer judgment.
Evidence reviewed 7 items Pinned PR delta: The verified merge-base-to-head delta changes CrawlKit from v0.14.7 to v0.14.8, adds the timeout documentation, and updates only the two manual Cloudsmith publishing workflows.
Current main still needs this update: The fetched main parent remains on CrawlKit v0.14.7, so the central timeout fix is not already implemented on main.
Release-check integration boundary: Slacrawl calls CrawlKit's releasecheck.Check with the Slacrawl owner, repository, version, cache, and force options; consuming CrawlKit's fixed release is therefore the direct implementation path.
Findings None None.
Security None None.

How this fits together

Slacrawl delegates its release-update check to CrawlKit, while separate manually run GitHub workflows publish Debian and RPM release assets to Cloudsmith. This change updates that shared release-check dependency and the publishing CLI installed by those workflows.

flowchart LR
  C[Slacrawl command] --> R[Release checker]
  R --> H[GitHub release request]
  H --> T[30-second timeout]
  B[Release packages] --> A[APT publish workflow]
  B --> P[RPM publish workflow]
  A --> S[Cloudsmith CLI]
  P --> S
  S --> O[Cloudsmith repository]
Loading

Decision needed

Question Recommendation
Should the existing manual APT and RPM publication commands be approved with Cloudsmith CLI 1.26.0 based on the supplied command-compatibility evidence, without an authenticated upload? Approve the compatible pin refresh: Accept the exact 1.26.0 pin because its demonstrated deb/rpm command interface and option match the unchanged workflow invocations.

Why: The remaining uncertainty concerns a production-secret manual release operation that cannot be fully exercised by the supplied credential-free proof.

Before merge

  • Resolve merge risk (P1) - The Cloudsmith CLI upgrade changes manual publication workflows that use a production secret; supplied evidence proves the existing command form is accepted, but intentionally does not perform an authenticated upload.
  • Complete next step (P2) - A collaborator must receive explicit maintainer review, and acceptance of the credentialed manual publication-tool upgrade is the sole remaining judgment.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Dependency and workflow surface 6 files affected; 10 added, 5 removed The patch is limited to one Go-module update, two manual publication workflow pins, and matching documentation.

Merge-risk options

Maintainer options:

  1. Approve the compatible pin refresh (recommended)
    Accept the exact 1.26.0 pin because the demonstrated deb/rpm command interface and option match the unchanged workflow invocations.
  2. Verify with a controlled upload
    Require a maintainer-approved non-production or reversible authenticated publication before merging the manual workflow change.

Technical review

Best possible solution:

Keep the timeout fix as a CrawlKit dependency upgrade and merge the exact Cloudsmith pin once a maintainer accepts the bounded manual-publication compatibility risk.

Do we have a high-confidence way to reproduce the issue?

Yes: the supplied terminal trace gives a concrete stalled-proxy scenario for a versioned Slacrawl binary and records the expected bounded timeout after the update.

Is this the best way to solve the issue?

Yes for the release-check hang: Slacrawl already delegates this behavior to CrawlKit, so consuming the fixed dependency release is narrower than duplicating its HTTP logic; Cloudsmith acceptance remains a release-workflow decision.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against 1816d30bbbde.

Labels

Label changes:

  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • remove rating: 🦞 diamond lobster: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.

Label justifications:

  • P2: This is a bounded reliability and release-tooling refresh with no evidence of an active user-facing outage.
  • merge-risk: 🚨 automation: The changed Cloudsmith CLI runs in manual package-publication workflows with the existing production secret and repository targets.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): Slacrawl’s production release-check wrapper calls CrawlKit’s releasecheck.Check, and the PR supplies a real built-binary trace through that entrypoint with a deliberately stalled HTTP proxy; it observes the intended timeout. The unchanged publishing workflows are also exercised against Cloudsmith CLI 1.26.0 command help, though authenticated upload is appropriately left for maintainer judgment.
  • proof: sufficient: Contributor real behavior proof is sufficient. Slacrawl’s production release-check wrapper calls CrawlKit’s releasecheck.Check, and the PR supplies a real built-binary trace through that entrypoint with a deliberately stalled HTTP proxy; it observes the intended timeout. The unchanged publishing workflows are also exercised against Cloudsmith CLI 1.26.0 command help, though authenticated upload is appropriately left for maintainer judgment.

Evidence

What I checked:

  • Pinned PR delta: The verified merge-base-to-head delta changes CrawlKit from v0.14.7 to v0.14.8, adds the timeout documentation, and updates only the two manual Cloudsmith publishing workflows. (go.mod:9, 508c23a51132)
  • Current main still needs this update: The fetched main parent remains on CrawlKit v0.14.7, so the central timeout fix is not already implemented on main. (go.mod:9, 1816d30bbbde)
  • Release-check integration boundary: Slacrawl calls CrawlKit's releasecheck.Check with the Slacrawl owner, repository, version, cache, and force options; consuming CrawlKit's fixed release is therefore the direct implementation path. (internal/cli/releasecheck.go:52, 508c23a51132)
  • Publishing command compatibility: Both workflows retain their existing authenticated upload arguments; the supplied real CLI output for Cloudsmith 1.26.0 shows matching deb/rpm positional usage and the --no-wait-for-sync option. (.github/workflows/publish-apt.yml:104, 508c23a51132)
  • Real behavior evidence: The PR body records an after-fix built-CLI run through a deliberately stalled loopback proxy: check-update --force returned the expected client timeout after 35.67 seconds, rather than hanging indefinitely. (internal/cli/releasecheck.go:52, 508c23a51132)
  • Feature-history routing: History identifies the original release-check feature commit and later CrawlKit and Cloudsmith workflow maintenance by the likely routing candidates. (internal/cli/releasecheck.go:15, 5c32a2758b19)

Likely related people:

  • steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
  • Vincent Koc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (2 earlier review cycles)
  • reviewed 2026-08-31T09:25:53.980Z sha 508c23a :: needs maintainer review before merge. :: none
  • reviewed 2026-08-31T09:32:08.478Z sha 508c23a :: needs maintainer review before merge. :: none

@steipete
steipete marked this pull request as ready for review August 31, 2026 09:28
@steipete
steipete requested a review from a team as a code owner August 31, 2026 09:28
@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Aug 31, 2026
@steipete

steipete commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator Author

Delta triage: NEEDS-PETER for the Cloudsmith publication-proof decision. The CrawlKit timeout fix is independently reproduced and ready for review; no code changes are needed from this pass. Reviewed head: 508c23a51132f1d4fcb406f252adfa3c402cef4c.

Root cause: Slacrawl delegates check-update to CrawlKit without supplying an HTTP client. CrawlKit v0.14.7 defaults to http.DefaultClient, which has no request timeout. v0.14.8 replaces that default with a 30-second client and includes a regression test. Updating the shared dependency is the correct ownership boundary.

Independent built-binary proof on macOS arm64:

# Build from main (1816d30) and the PR head respectively:
GOWORK=off go build -ldflags '-X github.com/openclaw/slacrawl/internal/cli.version=0.8.6' -o /private/tmp/slacrawl-delta-20260831-proof/slacrawl-main ./cmd/slacrawl
GOWORK=off go build -ldflags '-X github.com/openclaw/slacrawl/internal/cli.version=0.8.6' -o /private/tmp/slacrawl-delta-20260831-proof/slacrawl-pr167 ./cmd/slacrawl

# Both binaries, with a dedicated config and HTTPS_PROXY pointing at a
# loopback listener that accepts CONNECT and never responds:
slacrawl-main --config config.toml check-update --force
PASS: CONNECT api.github.com:443 HTTP/1.1 observed; still blocked at 40.01s; harness terminated the baseline.

slacrawl-pr167 --config config.toml check-update --force
error: check latest openclaw/slacrawl release: Get "https://api.github.com/repos/openclaw/slacrawl/releases/latest": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
PASS: exit 1 after 31.45s.

GOWORK=off go test github.com/openclaw/crawlkit/releasecheck -count=1
PASS
GOWORK=off go mod verify
all modules verified

actionlint .github/workflows/publish-apt.yml .github/workflows/publish-rpm.yml
PASS (exit 0)
git diff --check origin/main...HEAD
PASS (exit 0)

# Disposable venv, cloudsmith-cli==1.26.0:
python -m pip check
No broken requirements found.
cloudsmith --version
CLI Package Version: 1.26.0
API Package Version: 2.0.32
cloudsmith push deb --help
Usage: cloudsmith push deb [OPTIONS] OWNER/REPO/DISTRO/RELEASE PACKAGE_FILE
cloudsmith push rpm --help
Usage: cloudsmith push rpm [OPTIONS] OWNER/REPO/DISTRO/RELEASE PACKAGE_FILE
# Both commands expose --no-wait-for-sync, as used by the workflows.

Codex autoreview returned scoped-clean with no accepted/actionable findings at its default P0 threshold. This is a scoped result, not an all-priority correctness certificate. Current head CI is green: CI, Docker, CodeQL, and secret scanning. GitHub reports REVIEW_REQUIRED / BLOCKED; there are no failing checks to rerun.

Decision brief: accept the minor Cloudsmith CLI pin update based on installation and unchanged command-interface proof, or defer those two pins and land the independently verified CrawlKit fix. I recommend accepting the compatible pin refresh, while recording that authenticated Cloudsmith publication has not been exercised. If stronger service proof is required, a maintainer must authorize and run a controlled upload to a designated repository. This worker has no release/publication authority, so no real package upload was attempted. No new credential permissions, API design, or destinations are introduced by this PR.

PR remains open for the orchestrator; nothing was merged or published.

@steipete
steipete merged commit b9ff7d9 into main Sep 1, 2026
29 checks passed
@steipete
steipete deleted the chore/deps-refresh-20260830 branch September 1, 2026 00:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. other P2 Normal priority bug or improvement with limited blast radius. proof: sufficient Contributor real behavior proof is sufficient. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants