Skip to content

parity(#252): make the 1 MB max file size configurable (CSP_MAX_FILE_BYTES) and warn about skipped files #86

Description

@amondnet

Summary

Upstream semble #252 (9bb38fd) made the 1 MB per-file indexing limit configurable via SEMBLE_MAX_FILE_BYTES and warns about files skipped for size (previously silent, leaving unexplained gaps in results — upstream #250). csp hardcodes the limit and skips silently.

Upstream

  • index/files.py::get_max_file_bytes(): reads the env var; malformed or non-positive values warn and fall back to the 1 MB default.
  • index/create.py: collects FileStatus.TOO_LARGE paths and logs one warning naming up to 5 of them (... raise SEMBLE_MAX_FILE_BYTES to include them).
  • cli.py::_configure_cli_logging(): idempotent stderr handler so the warning surfaces in CLI runs.
  • README: documents the limit and the env var.

csp status

  • crates/csp/src/indexing/create.rs: pub const MAX_FILE_BYTES: u64 = 1_000_000; and if size > MAX_FILE_BYTES { continue; }hardcoded, silent.
  • csp has no CSP_* env overrides at all (upstream also has SEMBLE_CACHE_LOCATION, SEMBLE_MODEL_NAME, SEMBLE_CLONE_TIMEOUT); scope this issue to the file-size limit, note the others as a possible follow-up.

Tasks

  • CSP_MAX_FILE_BYTES env override with the same fallback/warn semantics.
  • Collect skipped-for-size paths during create_index_from_path and emit one stderr warning (first 5 paths + count).
  • README.md + README.ko.md: document the limit and the env var.
  • Tests: env override honored; invalid value falls back; warning emitted.

Refs

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions