Skip to content

Docker Volumes Backup, SSH Key Generation, MongoDB Atlas Support, and Bug Fixes - #144

Merged
Skyfay merged 22 commits into
mainfrom
dev
Aug 8, 2026
Merged

Docker Volumes Backup, SSH Key Generation, MongoDB Atlas Support, and Bug Fixes#144
Skyfay merged 22 commits into
mainfrom
dev

Conversation

@Skyfay

@Skyfay Skyfay commented Aug 8, 2026

Copy link
Copy Markdown
Owner

No description provided.

Skyfay added 22 commits August 2, 2026 09:28
Execution hosts now expose `connectSocket()` so local, SSH, and composite transports can open Unix domain socket streams, with clearer SSH forwarding errors for unsupported or disabled stream-local forwarding. The change also fixes binary lookup cache keys to avoid collisions between candidate lists, and adds unit coverage for local sockets, SSH stream-local forwarding behavior, and composite host routing.
Record optional POSIX mode, uid, and gid in archive indexes, keep those fields fresh when files are carried through incremental chains, and pass them to storage uploads during restore. This preserves permissions and ownership for directory restores without changing behavior for existing archives or sources that do not report metadata, and adds end-to-end tests for the new metadata flow.
Shadow copies are now released as soon as the last source in their group is collected, rather than being held open until stepCleanup at the end of the run. A release that fails stays unmarked so cleanup gets a second attempt.

To support this, directory collection was split into its own module (collect-directories.ts) and a source grouping layer was added (source-groups.ts). Adapters can implement planSourceGroups to declare which sources share one preparation - two volumes of the same container stop it once rather than twice. The StorageAdapter interface gains planSourceGroups, alwaysSnapshot, and an updated createSnapshot signature that takes the whole group's paths and a SnapshotOptions bag.

The runner context gains a released flag on each shadow copy record and a disposables list for teardown that lives outside any snapshot's lifetime.
Introduce a `stopContainers` flag on job sources, persist it in Prisma, and carry it into runner initialization for future source adapters that may need to stop running services before reading data. Also add `supportedRoles` for storage adapter definitions, update the adapter form to present only valid role choices, and enforce role restrictions across adapter create, update, and clone API paths with coverage for the new validation.
Introduce a new source-only Docker Volumes storage adapter in beta. This adds the adapter definition, Docker daemon connection layer via dockerode, form metadata, permissions, and UI icon support so users can test a Docker host connection and browse visible volumes.

The change also adds unit and integration coverage for Docker labels and live daemon interactions, and documents the new beta source in the changelog.
Implement Docker volume snapshot planning and cleanup around helper containers. Volumes are grouped by shared container usage so containers are stopped once per connected group, only previously running containers are restarted, and interrupted runs recover by reading stopped-container state from helper container labels on the Docker host. The Docker adapter now wires snapshot support through these sessions, removes the old runner disposables cleanup path, and adds focused unit coverage for grouping, snapshot ordering, and orphan recovery.
Implement native Docker volume collection via a single tar stream from the helper container, preserving file metadata and symlink targets while supporting excludes, incremental skips, and path-safety checks. The snapshot flow now ensures the helper image exists (pulling when missing) and pre-counts entries per volume to improve progress reporting, with graceful fallback when counts are unavailable. Runner progress text was updated to avoid misleading "/0" totals, and integration plus unit tests were expanded to cover volume extraction behavior and new snapshot engine calls.
Enable Docker volume adapters to restore backups through persistent restore sessions, including volume creation or overwrite, file metadata preservation, and symlink recreation. The adapter now reports existing volumes for overwrite checks and refuses unsafe per-file uploads outside a session. Integration and unit tests cover round-trip restores, concurrent writes, container restart handling, and cleanup behavior.
Extends directory job sources with a `stopContainers` flag (default true), wires it through form state and payload creation, and adds a unit test to ensure explicit `false` is persisted. The jobs and restore UIs now adapt copy for Docker volumes (volume-name placeholder, browse tooltip text, and existing-volume warning) so users understand interruption and overwrite behavior.
Adds a full user guide page for Docker Volumes, links it in the docs navigation, and expands the storage adapter developer guide with role and source-preparation architecture details. Integration coverage now includes a real SSH round trip against Docker socket forwarding (with test env socket mount/config), and lint guards now enforce dockerode isolation to the adapter engine layer.
Refactors adapter form and resolver behavior to align with schema-driven rules. Storage adapters are now filtered by supported role in pickers and setup wizard, invalid counterpart cloning is hidden, and storage dialogs use role-accurate naming. SSH credential overlays now derive prefixed vs unprefixed keys from schema shape, fixing Docker Volumes SSH mode when no primary credential slot exists. The form default seeding logic was extracted and updated for Zod 4 so schema defaults are applied again (while still not auto-selecting connection mode), and Docker volume optional fields now treat empty input as "use default" with advanced settings moved behind a disclosure. Documentation and unit tests were expanded to cover these regressions and new role/SSH conventions.
Adapter log messages (stopped containers, crash-consistency warning, interrupted-run recovery) were going to the server console only, so none of them appeared in the run history an operator actually reads. The runner now passes an onLog callback into createSnapshot and releaseSnapshot, and both the Docker volume and SMB VSS adapters write through it.

Container names are carried alongside their IDs in the stopped-containers label and throughout the stop/start helpers, so recovery messages name the services rather than opaque IDs. The label format is backward-compatible: bare IDs written by older versions are still parsed.

The directory picker gains a flat prop for adapters whose browse has no hierarchy. For Docker volumes the select-all row now ticks each volume individually instead of storing an unreadable root path, expand controls are hidden, and the browse button uses a storage icon. An unreachable Docker host now throws instead of returning an empty list, so it is distinguishable from a host with no volumes.

The runner uses the handle's noun field when logging snapshot lifecycle events, so SMB says shadow copy and Docker says helper container rather than both saying snapshot.
When an SSH server cannot forward a Unix socket - NetBird and Tailscale both intercept port 22 with an embedded server that lacks that channel type - the Docker adapter now falls back to running `docker system dial-stdio` on the target host and logs that it did so. The decision is made once per connection, not per request.

Symlink restoration into Docker volumes is removed as a capability. A link's target is meaningful only inside the container that wrote it, so restoring it into a different container context is unreliable. The adapter now omits `createSymlink`, which the restore pipeline reads as a capability limit: links are reported as skipped rather than failed.

A `DEFAULT_DOCKER_SOCKET` constant replaces three separate string literals that had already started to drift. New unit tests cover the reacher's decision logic and the dial-stdio error path.
Flat adapters (e.g. Docker volumes) had two related bugs in the directory browser: unticking a volume wrote exclude patterns instead of removing its row, and a leftover root row ("/" or "") would silently persist across saves as a source the adapter cannot read.

Fixes: toggleAtPath now takes a short-circuit path for flat mode that adds/removes rows directly, the dialog seeds state by dropping root rows in flat mode instead of translating them, and isEverything is gated on non-flat adapters.

Also raises Docker volume transferConcurrency from 1 to 4 (max 8) - the 1 was reasoned only about the read path (one tar stream), but restore writes each file with its own API request, so concurrency matters there just as it does over SFTP.

Three new tests cover the untick, root-row, and root-path cases.
Update the changelog and Docker Volumes user guide to explain that restores write files individually, can run several writes in parallel via the connection setting, and may be slower over SSH when Docker CLI fallback is used.
Reworked Docker volume cleanup to avoid `dockerode.run` with `AutoRemove`, which could delete helper containers before `wait` completed over SSH, causing intermittent restore failures. The cleanup path now explicitly creates, starts, waits, and removes helper containers. Restore progress output now includes restored bytes alongside file counts using archive index sizes, and Docker CLI fallback messages were downgraded from warnings to info. Added a lint guard to block future `AutoRemove` usage in wait-dependent container flows.
Teach the restore target picker to read browse behavior from adapter definitions instead of hardcoding Docker volumes. Flat targets now use volume-style labels and icons, pick items directly without folder navigation, and show clearer overwrite messaging for whole-volume restores.
Updates docs, changelog, and website content to reflect the Docker Volumes backup source. Adds the adapter icon, bumps storage adapter count from 13 to 14, updates roadmap status to in-progress, and trims changelog entries per the two-sentence rule.
Adds UTF-8-safe URL payload encoding/decoding for restore deep links and replaces raw Content-Disposition headers with RFC 6266-compliant attachment formatting. This prevents restore navigation failures and 500 errors when job names, paths, or filenames include non-Latin characters or emoji. The new helpers are wired into storage and vault download routes and storage UI, with unit tests covering Unicode, fallbacks, escaping, and backward compatibility.
This update fixes MongoDB connection edge cases by normalizing Host input, adding SRV and seed-list support, using URI-based args where required, and masking passwords in logged connection strings. It also fixes adapter connection/access checks for saved configs by restoring redacted stored secrets only when the caller has edit permission, preventing false test failures while keeping secret access scoped. Docs were updated to match the new MongoDB behavior and new unit tests cover host parsing, URI/arg generation, masking, and stored-secret merge permissions.
Introduces SSH credential keypair generation in the profile dialog and service layer, including key type selection, optional passphrase support, server-side key creation, and persisted public key plus fingerprint metadata. Adds reusable SSH UI panels for generated and stored public keys, updates OpenAPI and user docs, and covers the new payload and OpenSSH helpers with unit tests. It also fixes SSH profile edit validation so rename-only updates no longer fail and makes long credential dialogs scroll correctly.
Prepare the 3.2.0 release by updating package and OpenAPI versions, publishing the changelog entry, and moving Docker Volume Backups from the roadmap to shipped features on the website. Also fix a product tour hydration mismatch by selecting light and dark screenshots in CSS instead of from `resolvedTheme`, and tighten the file backup marketing copy.
@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
dbackup-website 2266356 Aug 08 2026, 02:52 PM

@Skyfay
Skyfay merged commit 0e9917d into main Aug 8, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant