diff --git a/README.md b/README.md index c6b90d8..fd26220 100644 --- a/README.md +++ b/README.md @@ -40,19 +40,19 @@ Files are encrypted locally, split into chunks, and uploaded as ciphertext. Chun │ Storage Engine │ - ┌─────────────┴─────────────┐ - │ │ - Encryption Chunking - │ │ - └─────────────┬─────────────┘ - │ - Storage Router - │ - ┌─────────────┴─────────────┐ - │ │ - Telegram Discord - │ │ - encrypted chunks encrypted chunks + ┌─────────────┴─────────────┐ + │ │ + Encryption Chunking + │ │ + └─────────────┬─────────────┘ + │ + Storage Router + │ + ┌─────────────┴─────────────┐ + │ │ + Telegram Discord + │ │ + encrypted chunks encrypted chunks ``` Stashify does **not** provide the underlying storage. @@ -72,10 +72,8 @@ Stashify separates those concepts. | Vendor lock-in | Provider-agnostic storage abstraction | | Provider sees plaintext | Files are encrypted before upload | | Large files | Automatic chunking | -| Provider-specific limits | Provider-aware chunking and routing | +| Provider-specific limits | Provider-aware chunking | | Single provider dependency | Multi-provider storage | -| Interrupted uploads | Resumable operations | -| Manual file management | Unified CLI/TUI | | Provider-specific APIs | One consistent interface | --- @@ -98,7 +96,7 @@ Your files are encrypted **before they leave your device**. Chunking │ ▼ - Encrypted ciphertext + Encrypted ciphertext │ ┌──────────┴──────────┐ ▼ ▼ @@ -200,69 +198,6 @@ Ready --- -## Why Stashify? - -Traditional cloud storage usually means trusting one provider with both your data and your storage. - -Stashify separates those concepts. - -| Problem | Stashify | -| -------------------------- | ------------------------------------- | -| Vendor lock-in | Provider-agnostic storage abstraction | -| Provider sees plaintext | Files are encrypted before upload | -| Large files | Automatic chunking | -| Provider-specific limits | Provider-aware chunking and routing | -| Single provider dependency | Multi-provider storage | -| Interrupted uploads | Resumable operations | -| Manual file management | Unified CLI | -| Provider-specific APIs | One consistent interface | - ---- - -## Features - -### Client-side encryption - -Your files are encrypted **before they leave your device**. - -```text - YOUR DEVICE - │ - Plaintext - │ - ▼ - Encryption - │ - ▼ - Chunking - │ - ▼ - Encrypted ciphertext - │ - ┌──────────┴──────────┐ - ▼ ▼ - Telegram Discord -``` - -Storage providers should only receive ciphertext. - -Stashify is designed around: - -* Client-side encryption -* Authenticated encryption (AEAD) -* Per-file cryptographic keys -* Proper key derivation -* Cryptographically secure randomness -* No custom cryptographic primitives -* Authenticated chunk integrity -* Local key management - -Stashify uses established cryptographic libraries rather than implementing cryptography from scratch. - -> **Security note:** Stashify does not claim that multi-provider storage makes encryption stronger. Confidentiality comes from the cryptographic design and key management. Multi-provider storage primarily provides distribution, redundancy, and provider independence. - ---- - ## Chunked storage Large files are automatically split into manageable chunks. @@ -290,30 +225,16 @@ The storage engine can account for provider-specific upload limitations without ## Multi-provider storage -Stashify can distribute a file across multiple storage providers. +Stashify can store files across multiple storage providers. Currently only the **Single** strategy is implemented (all chunks on one provider). Additional strategies are planned. -For example: +Current strategies: -```text -100 encrypted chunks - -Telegram: - 0 2 4 6 8 10 12 ... - -Discord: - 1 3 5 7 9 11 13 ... -``` - -Multiple storage strategies are planned: - -| Strategy | Description | -| -------------- | ----------------------------------------------------- | -| **Single** | Store all chunks on one provider | -| **Split** | Distribute chunks across multiple providers | -| **Balanced** | Dynamically distribute chunks based on provider state | -| **Replicated** | Store copies across multiple providers | - -This allows Stashify to build storage around the providers available to you instead of forcing you into a single backend. +| Strategy | Status | Description | +| -------------- | ------------ | ----------------------------------------------------- | +| **Single** | ✅ Implemented | Store all chunks on one provider | +| **Split** | 🚧 Planned | Distribute chunks across multiple providers | +| **Balanced** | 🚧 Planned | Dynamically distribute chunks based on provider state | +| **Replicated** | 🚧 Planned | Store copies across multiple providers | --- @@ -348,159 +269,17 @@ Planned providers include: --- -## Asynchronous transfers - -Stashify is designed around asynchronous I/O. - -Large uploads can consist of hundreds or thousands of chunks, so operations should run concurrently with bounded workers. - -```text - Upload Queue - │ - ┌───────────┼───────────┐ - ▼ ▼ ▼ - Worker 1 Worker 2 Worker 3 - │ │ │ - Telegram Discord Telegram -``` - -The transfer system is designed to support: - -* Concurrent uploads -* Concurrent downloads -* Bounded concurrency -* Retries -* Exponential backoff -* Provider-aware rate limiting -* Cancellation -* Progress reporting -* Failed-job tracking -* Resumable operations - ---- - -## Resumable uploads - -Interrupted transfers shouldn't mean starting from zero. - -```text -200 chunks - -chunk 000 ✓ -chunk 001 ✓ -chunk 002 ✓ -... -chunk 147 ✓ -chunk 148 ✗ -chunk 149 ✗ -... -``` - -Stashify keeps track of individual chunks so completed work can be preserved across interruptions. - ---- - -## Integrity verification - -Encrypted chunks are authenticated and tracked using local metadata. - -Stashify is designed to detect: - -* Missing chunks -* Corrupted chunks -* Modified ciphertext -* Incomplete downloads -* Invalid manifests -* Incorrect chunk ordering -* Failed reconstruction - -The final reconstructed file can also be verified against file-level integrity information. - ---- - -## Manifest-based storage - -Every stored file has a manifest describing how it can be reconstructed. - -Conceptually: - -```text -File -├── ID -├── Original name -├── Original size -├── Chunk size -├── Chunk count -├── Encryption metadata -├── Integrity information -│ -└── Chunks - ├── 0 → Telegram → remote ID - ├── 1 → Discord → remote ID - ├── 2 → Telegram → remote ID - └── ... -``` - -Local metadata is stored in `.stash/metadata/`. - ---- - -## CLI - -Stashify can be used entirely from the command line. - -```bash -# Initialize (generates RMK, stores in OS keyring, shows recovery key) -stash init - -# Configure providers -stash provider add telegram -stash provider add discord - -# List providers -stash provider list - -# Upload (no password prompt — uses RMK from keyring) -stash put ./movie.mkv - -# List stored files -stash ls - -# Inspect a file -stash info movie.mkv - -# Download (no password prompt) -stash get movie.mkv - -# Delete -stash rm movie.mkv - -# Verify -stash verify movie.mkv - -# Check status -stash status - -# Key management -stash key-commands status -stash key-commands lock -stash key-commands unlock --recovery-key -stash key-commands recovery -``` - ---- - ## Provider Support | Provider | Status | | ---------------- | -------------- | -| Telegram | Implemented | -| Discord | Implemented | -| S3-compatible | Planned | -| Backblaze B2 | Planned | -| Google Drive | Planned | -| Local filesystem | Planned | -| WebDAV | Planned | +| Telegram | ✅ Implemented | +| Discord | ✅ Implemented | +| S3-compatible | 🚧 Planned | +| Backblaze B2 | 🚧 Planned | +| Google Drive | 🚧 Planned | +| Local filesystem | 🚧 Planned | +| WebDAV | 🚧 Planned | Provider availability and capabilities are subject to the APIs, limits, and policies of the respective services. @@ -594,7 +373,7 @@ At a high level: ┌────────────┼────────────┐ ▼ ▼ ▼ Telegram Discord Future - Providers + Providers ``` The architecture intentionally separates: @@ -620,20 +399,20 @@ The architecture is being actively developed and APIs may change significantly. | Component | Status | | ---------------------- | ----------- | | Project architecture | In progress | -| Python CLI | In progress | -| Encryption | Implemented | -| Chunking | Implemented | -| Manifest / metadata | Implemented | -| Provider abstraction | In progress | -| Telegram provider | Implemented | -| Discord provider | Implemented | -| Async job engine | Implemented | -| **Key management (RMK)** | **Implemented** | -| Resumable transfers | Planned | -| Multi-provider routing | Planned | -| Verification | Planned | -| Repair / recovery | Future | -| Additional providers | Future | +| Python CLI | ✅ Implemented | +| Encryption | ✅ Implemented | +| Chunking | ✅ Implemented | +| Manifest / metadata | ✅ Implemented (JSON) | +| Provider abstraction | ✅ Implemented | +| Telegram provider | ✅ Implemented | +| Discord provider | ✅ Implemented | +| Async job engine | ✅ Implemented | +| Key management (RMK) | ✅ Implemented | +| Multi-provider routing | 🚧 Planned | +| Resumable transfers | 🚧 Planned | +| Integrity verification | 🚧 Planned | +| Repair / recovery | 🚧 Planned | +| Additional providers | 🚧 Planned | Features marked **Planned** or **Future** should not be considered implemented. @@ -678,12 +457,13 @@ The long-term goal is to turn Stashify into a flexible encrypted storage layer t * [x] Core encryption pipeline * [x] Chunking -* [x] SQLite metadata +* [x] JSON metadata * [x] Telegram provider * [x] Discord provider * [x] Async transfer system * [x] **Repository Master Key (RMK) hierarchy** * [ ] Resumable uploads +* [ ] Multi-provider routing ### Medium term diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 1c7467a..277be1f 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -81,9 +81,10 @@ stash put [options] |--------|-------------| | `--provider, -p` | Specific provider to use | | `--chunk-size` | Chunk size in bytes (default: provider limit) | -| `--strategy` | Distribution: single, split, balanced, replicated | | `--confirm/--no-confirm` | Skip confirmation prompt | +> **Note:** Distribution strategy is currently always "single" (all chunks on one provider). Additional strategies (split, balanced, replicated) are planned. + ### `stash get` Retrieve a file from Stash (uses RMK from keyring — no password prompt). diff --git a/docs/configuration.md b/docs/configuration.md index 4f64a16..3198837 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -78,31 +78,25 @@ stash provider add telegram \ | `chat_id` | Chat/channel ID for storage | Yes | - | | `max_concurrent` | Max concurrent uploads | No | `3` | -## Global Settings - -Create `.stash/config.toml` for global defaults: - -```toml -[storage] -default_provider = "telegram" -default_chunk_size = 10485760 # 10MB -replication_factor = 1 - -[transfers] -upload_concurrency = 3 -download_concurrency = 3 -retry_count = 3 -retry_backoff = 1.0 - -[security] -auto_lock_timeout = 0 # 0 = never -key_derivation_iterations = 100000 - -[ui] -compact_mode = false -animations = true -progress_style = "bar" -``` + + +## Key Management + +The **Repository Master Key (RMK)** is managed by the OS keyring: + +| Command | Description | +|---------|-------------| +| `stash key-commands status` | Show key management status | +| `stash key-commands lock` | Remove RMK from keyring (lock repo) | +| `stash key-commands unlock --recovery-key ` | Restore RMK from recovery key | +| `stash key-commands recovery` | Show RMK for backup | + +The RMK is stored in the OS credential store: +- **Windows**: Credential Manager +- **macOS**: Keychain +- **Linux**: secret-service (GNOME Keyring, KWallet, etc.) + +No passwords or raw keys are stored in configuration files. ## Key Management diff --git a/src/stash/__init__.py b/src/stash/__init__.py index bb06226..372049c 100644 --- a/src/stash/__init__.py +++ b/src/stash/__init__.py @@ -1,5 +1,5 @@ """Stash - Privacy-focused CLI storage system.""" -__version__ = "0.1.0" +__version__ = "0.2.0" __author__ = "Stash Contributors" __license__ = "MIT" \ No newline at end of file diff --git a/test_restored.md b/test_restored.md deleted file mode 100644 index 4bf08a6..0000000 --- a/test_restored.md +++ /dev/null @@ -1,242 +0,0 @@ -# Stash - -**An open-source, privacy-focused command-line storage system that uses third-party platforms as encrypted storage backends.** - ---- - -## What is Stash? - -Stash is a storage abstraction layer. It lets you store files on external services — Telegram, Discord, and eventually S3, Backblaze B2, Google Drive, local filesystem, WebDAV, and others — while keeping all encryption strictly client-side. - -Stash is **not** a cloud storage provider. It is a CLI tool that turns interchangeable storage backends into a unified, encrypted filesystem-like experience. - -``` - Stash CLI - | - Storage abstraction - | - +---------------+---------------+ - | | - Telegram Discord - backend backend - | | - encrypted chunks encrypted chunks -``` - ---- - -## Why Stash? - -| Problem | Stash Approach | -|---------|----------------| -| Vendor lock-in | Provider-agnostic abstraction | -| No client-side encryption in most platforms | Encrypt locally, upload ciphertext only | -| Single point of failure | Distribute chunks across multiple providers | -| Manual chunk/message management | Automatic chunking, upload, metadata, resumption | -| Platform-specific limits | Chunking adapts to each provider's constraints | - ---- - -## Core Architecture - -### Client-Side Encryption - -Files are encrypted **before** they leave your device. - -``` -plaintext → Stash (encrypt + chunk) → ciphertext chunks → providers -``` - -- Providers only ever receive authenticated ciphertext -- Established cryptographic libraries (no custom primitives) -- Per-file encryption keys with proper key derivation -- Authenticated encryption (AEAD) for every chunk -- Optional double-encryption mode (whole-file + per-chunk) - -**Security claims are conservative:** encryption and key management provide confidentiality. Multi-provider distribution provides redundancy and availability — not additional cryptographic security. - -### File Chunking - -Large files are split into chunks to accommodate provider-specific size limits. - -``` -original file - | - v -encrypt → chunk → chunk 0, chunk 1, chunk 2, ... -``` - -Each chunk is independently encrypted and tracked. Metadata (manifest) describes how to reconstruct the file. - -### Multi-Provider Storage - -A single file's chunks can be distributed across providers. - -| Strategy | Description | -|----------|-------------| -| **Single** | All chunks on one provider | -| **Split** | Chunks distributed across providers (0→Telegram, 1→Discord, …) | -| **Balanced** | Dynamic distribution based on availability/performance | -| **Replicated** | Each chunk stored on multiple providers for redundancy | - -``` -100 encrypted chunks - -Telegram: 0, 2, 4, 6, 8, ... -Discord: 1, 3, 5, 7, 9, ... -``` - -Even if an attacker obtains **all chunks from every provider**, the data remains protected by the encryption design. - -### Provider Abstraction - -The core engine knows nothing about Telegram or Discord specifics. - -``` -Storage Provider (interface) - | - +-- Telegram - +-- Discord - +-- S3 / B2 / GDrive / Local / WebDAV (planned) - +-- Future providers -``` - -New providers implement a clean interface. The storage engine remains unchanged. - -### Asynchronous & Resumable - -- Bounded concurrent workers for uploads/downloads -- Retries with exponential backoff -- Provider-aware rate limiting -- Progress reporting -- Cancellation support -- **Resumable operations**: if 75/100 chunks uploaded, interruption resumes at chunk 75 - -``` -Upload Queue - | -+----+----+----+ -| | | | -W1 W2 W3 W4 (bounded workers) -| | | | -TG DC TG S3 (providers) -``` - -### Manifest & Local Metadata - -Each stored file has a manifest containing: - -- File ID, original name, size -- Chunk size, count, encryption params -- Chunk indexes → provider assignments → remote identifiers -- Integrity verification data - -Local metadata stored in SQLite (initially). - ---- - -## Basic Usage - -```bash -# Initialize repository -stash init - -# Add providers -stash provider add telegram -stash provider add discord - -# List configured providers -stash provider list - -# Store a file -stash put ./movie.mkv - -# List stored files -stash ls - -# Show file metadata -stash info movie.mkv - -# Retrieve a file -stash get movie.mkv - -# Remove a file -stash rm movie.mkv - -# Verify integrity (local metadata + remote) -stash verify movie.mkv - -# Repair missing chunks from replicas (planned) -stash repair movie.mkv - -# Overall status -stash status -``` - -The user never manages individual messages, attachments, chunks, or encryption metadata. - ---- - -## Provider Support - -| Provider | Status | Notes | -|----------|--------|-------| -| Telegram | Initial | Bot API, channel/group storage | -| Discord | Initial | Bot/user token, channel attachments | -| S3-compatible | Planned | AWS S3, MinIO, R2, etc. | -| Backblaze B2 | Planned | Native B2 API | -| Google Drive | Planned | OAuth, Drive API | -| Local filesystem | Planned | Directory backend | -| WebDAV | Planned | Generic WebDAV servers | - -**Important:** Stash does not provide "unlimited storage." Actual limits, rate limits, file-size restrictions, and policies depend entirely on each provider. Stash adapts to those constraints via chunking and provider-specific logic. - ---- - -## Security & Privacy Philosophy - -- **Local-first encryption**: Plaintext never leaves your device -- **No custom crypto**: Established libraries (e.g., `cryptography`, `libsodium` bindings) -- **Minimal trust**: Providers are untrusted storage buckets -- **Transparency**: Open source, auditable code paths -- **Provider independence**: No single provider can compromise your data -- **Conservative claims**: We describe what the cryptography actually guarantees - ---- - -## Project Status - -**Early development.** Core architecture, provider abstraction, encryption model, and CLI structure are being designed and implemented. - -| Area | Status | -|------|--------| -| CLI framework | In progress | -| Provider abstraction | In progress | -| Telegram backend | In progress | -| Discord backend | Planned | -| Encryption/chunking | In progress | -| Manifest/metadata | In progress | -| Async job engine | In progress | -| Resumable uploads | Planned | -| Multi-provider strategies | Planned | -| Verification/repair | Future | - -Features described as "planned" or "future" are not yet implemented. This README will be updated as milestones land. - ---- - -## Contributing - -We welcome contributions — especially new storage providers, core improvements, testing, and documentation. - -See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. - ---- - -## License - -MIT License. See [LICENSE](LICENSE) for details. - ---- - -*Stash: your files, your keys, your choice of storage.* \ No newline at end of file