Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 14 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion crates/fast-down-api/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,12 @@ reqwest = { version = "0.13.4", default-features = false, features = [
thiserror.workspace = true
crossfire.workspace = true
anyhow = "1.0.103"
path_helper = { version = "0.1.9", features = [
path_helper = { version = "0.1.10", features = [
"auto_ext",
"sanitize",
"tokio",
] }
file_alloc = "0.2"
chrono = "0.4.45"
urlencoding = "2.1.3"
soft-canonicalize = "0.5.6"
Expand Down
69 changes: 57 additions & 12 deletions crates/fast-down-api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ that turns the pull/push engine into a few lines of async code: spawn a download
drain progress events, resume after interruption, and cancel cooperatively.

- **Concurrent, resumable downloads** powered by the `fast-down` engine (work-stealing, range requests).
- **Two entry points**: `download` (auto-resume when possible) and `resume` (hard error if it can't continue).
- **Event stream**: a single channel carries prefetch, per-worker progress, rename, and error events.
- **Two layers of entry points**: the fire-and-forget `download` / `resume` wrappers, and the lower-level `plan` / `plan_resume` pair that prefetches the remote and inspects the disk _without writing a single byte_ — so you can preview the outcome and decide before committing.
- **Event stream**: a single channel carries prefetch, disk allocation, per-worker progress, resume, rename, and lifecycle events. Every run ends with exactly one `Event::Terminated(TerminationReason)`.
- **Cooperative cancellation**: cancelling mid-flight preserves the `.part` / `.fd` files so you can resume later.
- **Configurable**: threads, chunk size, write method (`Mmap` / `Std`), proxies, headers, retries, and more via `PartialConfig`.
- **Configurable**: threads, chunk size, write method (`Mmap` / `Std`), proxies, headers, retries, disk pre-allocation, and more via `PartialConfig`.

## Quick start

Expand Down Expand Up @@ -79,6 +79,12 @@ async fn main() -> anyhow::Result<()> {
break;
}
Event::ResumeError(e) => eprintln!("resume error: {e}"),
Event::Allocating(size) => println!("pre-allocating {size} bytes on disk"),
Event::AllocError(e) => eprintln!("pre-allocation failed (continuing): {e}"),
Event::Terminated(reason) => {
println!("terminated: {reason:?}");
break;
}
_ => {}
}
}
Expand Down Expand Up @@ -115,17 +121,51 @@ resume(
token.cancel(); // stops fetching, keeps .part / .fd so you can resume later
```

### Two-phase planning (inspect before you commit)

`plan` and `plan_resume` do everything `download` / `resume` do _except_ touch
the disk: they prefetch the remote metadata, resolve the output path, and probe
the `.fd` / `.part` pair left by a previous run. The returned [`DownloadPlan`]
tells you what starting it would do — [`DownloadPlan::resume_outcome`] reports
`Resumable`, `Fresh`, or `Mismatch` — and nothing is created until you call one
of its `start` methods. Dropping the plan abandons the download with no side
effects.

```rust,ignore
let plan = plan(url, config.clone(), tx.clone(), token.clone()).await?;

match plan.resume_outcome() {
ResumeOutcome::Resumable { .. } => println!("will continue from a previous run"),
ResumeOutcome::Fresh => println!("will download the whole file"),
ResumeOutcome::Mismatch(e) => println!("stale state: {e} (use start_forced_resume)"),
}

// Commit when you're ready. Each start method emits exactly one `Event::Terminated`.
plan.start().await; // resume if possible, else fresh (or refuse for plan_resume)
// plan.start_fresh().await; // ignore any saved progress and re-download
// plan.start_forced_resume().await; // continue from a mismatched state when only identity changed
```

`plan_resume` takes a `.part` path instead of a URL and hard-refuses a
`Mismatch` (sending `Event::ResumeError` + `TerminationReason::Failed`) rather
than restarting — because the caller asked to continue one specific file, not to
fetch it again. Pass a `url` to re-fetch the metadata, or `None` to reuse the URL
recorded in the `.fd`.

## API overview

| Item | Purpose |
| ------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`download`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.download.html) | Start a download; auto-resume when a valid `.fd` + `.part` exist, else fresh. Observe completion by draining the `Rx` from `create_channel` until it disconnects. |
| [`resume`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.resume.html) | Resume a specific `.part` file; hard-error (`Event::ResumeError`) if it can't. Completion is observed the same way, by draining `Rx`. |
| [`create_channel`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.create_channel.html) | Create the `(Tx, Rx)` event channel. |
| [`create_cancellation_token`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.create_cancellation_token.html) | Create a `CancellationToken` for cooperative cancellation. |
| [`Event`](https://docs.rs/fast-down-api/latest/fast_down_api/enum.Event.html) | The event enum delivered over the channel. |
| [`PartialConfig`](https://docs.rs/fast-down-api/latest/fast_down_api/struct.PartialConfig.html) | Layered, optional configuration for a download. |
| [`StateError`](https://docs.rs/fast-down-api/latest/fast_down_api/enum.StateError.html) | Errors surfaced via `Event::ResumeError`. |
| Item | Purpose |
| ------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [`download`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.download.html) | Start a download; auto-resume when a valid `.fd` + `.part` exist, else fresh. Observe completion by draining the `Rx` from `create_channel` until it disconnects. |
| [`resume`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.resume.html) | Resume a specific `.part` file; hard-error (`Event::ResumeError`) if it can't. Completion is observed the same way, by draining `Rx`. |
| [`plan`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.plan.html) | Prefetch + probe the disk and return a [`DownloadPlan`](https://docs.rs/fast-down-api/latest/fast_down_api/struct.DownloadPlan.html) **without writing anything**; start it only when ready. |
| [`plan_resume`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.plan_resume.html) | Like `plan` but targets a specific `.part`; refuses a `Mismatch` instead of re-downloading. |
| [`DownloadPlan`](https://docs.rs/fast-down-api/latest/fast_down_api/struct.DownloadPlan.html) | A prepared, not-yet-started download. Inspect with `resume_outcome`, then call `start` / `start_fresh` / `start_forced_resume`. |
| [`create_channel`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.create_channel.html) | Create the `(Tx, Rx)` event channel. |
| [`create_cancellation_token`](https://docs.rs/fast-down-api/latest/fast_down_api/fn.create_cancellation_token.html) | Create a `CancellationToken` for cooperative cancellation. |
| [`Event`](https://docs.rs/fast-down-api/latest/fast_down_api/enum.Event.html) | The event enum delivered over the channel. |
| [`PartialConfig`](https://docs.rs/fast-down-api/latest/fast_down_api/struct.PartialConfig.html) | Layered, optional configuration for a download. |
| [`StateError`](https://docs.rs/fast-down-api/latest/fast_down_api/enum.StateError.html) | Errors surfaced via `Event::ResumeError`. |

## How resume works

Expand All @@ -140,6 +180,11 @@ size). On the next run:

Cancellation leaves both files in place, so a later `resume` (or `download`) can pick up exactly where it stopped.

Every run — whether it completes, is cancelled, stops incomplete, or fails —
ends with exactly one `Event::Terminated(TerminationReason)` as the last event
on the channel, so draining `Rx` until `Terminated` is the reliable way to know
a run has finished.

## License

MIT — see [LICENSE](https://github.com/fast-down/core/blob/main/LICENSE).
33 changes: 30 additions & 3 deletions crates/fast-down-api/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,6 @@ pub struct Config {
/// 文件名
pub filename: String,

/// 用于在 prefetch 阶段生成占位文件名
pub gid: String,

/// Number of threads. Recommended: `32` / `16` / `8`. More threads does not always mean faster.
#[config(default = 32)]
pub threads: usize,
Expand All @@ -54,6 +51,21 @@ pub struct Config {
/// Set to `true` only if you need to power off immediately after download.
pub sync_all: bool,

/// Reserve the whole file size on disk before downloading. Recommended: `false`
///
/// Claiming the space up front keeps the file in fewer fragments and turns
/// a full disk into an error before any bytes are fetched instead of
/// halfway through. The cost depends on the platform: `fallocate` on Unix
/// and `SetFileValidData` on Windows reserve the space instantly, but
/// without them the fallback writes zeros across the whole file, which
/// takes as long as a full-size write pass.
///
/// `SetFileValidData` needs the `SeManageVolumePrivilege`, which is
/// enabled automatically when available; without it Windows falls back to
/// the zero-fill path. Failure to reserve is reported as
/// [`crate::Event::AllocError`] and does not stop the download.
pub pre_alloc: bool,

/// Write buffer size in bytes. Recommended: `16 * 1024 * 1024`
///
/// - Only effective for [`WriteMethod::Std`]. Reduces the number of `write` syscalls
Expand Down Expand Up @@ -325,6 +337,11 @@ mod range_list {
}
ranges.push(start..end_inclusive.saturating_add(1));
}
// Normalize to ascending `start` order on load so downstream
// consumers (resume gap computation, `part_shortfall`, ...) can
// rely on the sorted invariant even when the `.fd` was hand-edited
// or written by an older build that stored chunks out of order.
ranges.sort_by_key(|r| r.start);
Ok(Some(ranges))
}
}
Expand Down Expand Up @@ -505,4 +522,14 @@ mod range_list_tests {
"wrapping end must be rejected with a clear error, got: {msg}"
);
}

/// Deserialize must normalize chunks to ascending `start` order so downstream
/// consumers can rely on the sorted invariant even for hand-edited `.fd` files.
#[test]
#[allow(clippy::single_range_in_vec_init)]
fn downloaded_chunk_deserialize_normalizes_order() {
let toml = "downloaded_chunk = \"5-9,1-3,100-100\"\n";
let pc: PartialConfig = toml::from_str(toml).unwrap();
assert_eq!(pc.downloaded_chunk, Some(vec![1..4, 5..10, 100..101]));
}
}
41 changes: 15 additions & 26 deletions crates/fast-down-api/src/core/download/mod.rs
Original file line number Diff line number Diff line change
@@ -1,20 +1,19 @@
use crate::core::download::overwrite::OverwriteOption;
use crate::utils::ForceSendExt;
use crate::{DownloadState, Event, StateError};
use crate::{PartialConfig, Tx, prefetch, tx_err, utils::gen_path};
use crate::{PartialConfig, TerminationReason, Tx};
use fast_down::UrlInfo;
use inherit_config::ConfigLayer;
use overwrite::overwrite;
use path_helper::IterStemExt;
use std::path::Path;
use tokio::fs::{self, OpenOptions};
use tokio_util::sync::CancellationToken;
use url::Url;

mod overwrite;
mod pipeline;
mod plan;
mod progress_reporter;

pub use plan::*;

fn open_existing() -> OpenOptions {
let mut o = OpenOptions::new();
o.read(true).write(true).truncate(false).create(false);
Expand All @@ -33,12 +32,14 @@ fn open_create_new() -> OpenOptions {

/// Attempt to load and validate a resume state from disk.
///
/// This helper consolidates the resume logic shared between `run_download` (overwrite and non-overwrite branches)
/// and `run_resume`. It checks if both `.fd` and `.part` exist, validates the state against the current server info,
/// and merges the new config into the loaded state.
/// This checks that both the `.fd` and `.part` exist, validates the state
/// against the current server info, and merges the new config into the loaded
/// state.
///
/// Returns `Ok(Some(state))` if resume is possible, `Ok(None)` if no resume state exists (caller should start fresh),
/// or `Err(StateError)` if the state exists but is invalid.
/// Returns `Ok(Some(state))` if resume is possible, `Ok(None)` if there is
/// nothing usable to resume from (the pair is incomplete, or the `.part` is
/// shorter than the recorded progress), or `Err(StateError)` if the state exists
/// but does not describe the current remote file.
#[allow(clippy::result_large_err)]
async fn try_load_resume_state(
url: &Url,
Expand All @@ -61,22 +62,10 @@ async fn try_load_resume_state(
// Validate the state against current server info
state.validate(info)?;

// Check that the .part file size is consistent with the recorded progress.
// Only applies to regular files — directories or other special files are not a
// valid .part and will fail later when build_pipeline tries to open them.
if let Ok(metadata) = fs::metadata(tmp_path).await
&& metadata.is_file()
{
let actual_size = metadata.len();
let recorded_progress = state.get_progress();
let max_recorded_end = recorded_progress.iter().map(|r| r.end).max().unwrap_or(0);

if actual_size < max_recorded_end {
// The .part file is smaller than what we think is already downloaded.
// This could lead to data corruption if we continue with resume.
// Treat this as if no valid state exists and start fresh.
return Ok(None);
}
// A `.part` shorter than the recorded progress claims bytes that are not on
// disk; continuing would leave that span zero-filled and never fetched.
if state.part_shortfall(tmp_path).await.is_some() {
return Ok(None);
}

// Merge the new config into the loaded state
Expand Down
Loading
Loading