Skip to content

feat(import): add ImportState + --skip-state-load flag for write-only import#560

Merged
riyazsh merged 1 commit into
mainfrom
riyaz/import-state-skip-load
May 13, 2026
Merged

feat(import): add ImportState + --skip-state-load flag for write-only import#560
riyazsh merged 1 commit into
mainfrom
riyaz/import-state-skip-load

Conversation

@riyazsh
Copy link
Copy Markdown
Contributor

@riyazsh riyazsh commented May 13, 2026

Summary

State.__init__ unconditionally loads existing state from storage. The import command discards prior per-type source state before its API fetch (in _import_get_resources_cb) and writes fresh data via dump_state at the end — the boot-time load is dead weight for that command. On populated buckets the load can dominate per-run wall-clock.

This PR adds an opt-in --skip-state-load flag to import that constructs a new ImportState instead of State. ImportState exposes only write methods and has no source / destination property, so any attempt to read state fails at the language level with AttributeError rather than silently returning empty data.

Adds diagnostic timing logs around state init, load, dump, and each storage backend's _list_and_load and put. The logs emit from a finally block so they fire even when an SDK call raises mid-iteration, and carry a list_ms vs download_ms split plus pagination and blob counts. Useful for diagnosing where wall-clock goes on populated buckets.

What changes

  • New ImportState class (utils/import_state.py) — write-only state for the import command. set_source, clear_source_type, dump_state(Origin.SOURCE). No read accessors; dump_state rejects DESTINATION / ALL.
  • New SourceStateWriter Protocol (utils/source_state_writer.py) — shared write surface implemented by both State and ImportState.
  • New --skip-state-load Click option on import only. Click rejects the flag on sync / migrate / diffs / reset / prune; build_config adds a redundant validation guard.
  • Three import-path writer sites migrated to the protocol methods: base_resource._import_resource, resources_handler._import_get_resources_cb, model/synthetics_private_locations.import_resource.
  • Shared build_storage_backend helper in _base_storage.py so State and ImportState cannot drift on per-backend init.
  • Diagnostic timing instrumentation across state.py and all four storage backends, with a try/finally so the log emits even when the underlying SDK raises. New fields: pages_listed, blobs_listed, blobs_downloaded, transient_errors, aborted, list_ms, download_ms, wall_ms. Race-delete semantics (NoSuchKey on AWS, ResourceNotFoundError on Azure, NotFound on GCS) are caught uniformly across backends and counted in transient_errors.

What does not change

  • Default behavior. --skip-state-load is opt-in (default=False); existing callers see no behavior change.
  • sync / diffs / migrate / reset / prune paths. These continue to use State with the existing read+write surface.
  • Wire format, file layout on disk, or storage backend semantics. dump_state patches per-blob exactly as before.

Constraints

--skip-state-load requires --resource-per-file and --resources, and is not combinable with --minimize-reads (which serves a different optimization goal). Validated in build_config.

Test plan

  • 556 unit tests pass (514 existing + 42 new in test_import_state.py)
  • black --line-length 120 clean
  • New coverage: ImportState invariant (no read accessors), write API, Protocol satisfaction by both classes, partial-import preserves untouched types end-to-end, stale-within-type unchanged from baseline, CLI validation errors, Click rejection on all non-import commands, deprecated-resource conflict still rejected with --skip-state-load, timing-log schema fields present on all four backends, aborted=1 fires when each backend's SDK call raises, paged-iterator branch on GCS, round-trip read of --skip-state-load-written state via a fresh State.

Rollout

Flag is opt-in (default=False). Reverting the commit removes the option from import --help and doesn't break any caller that wasn't using it. No on-disk or wire-format effect.

Out of scope

  • migrate does not gain --skip-state-load (apply-phase reads destination).
  • sync keeps its existing state-load path.
  • Parallelising _list_and_load is a separate effort.

@riyazsh riyazsh requested a review from a team as a code owner May 13, 2026 02:49
State.__init__ unconditionally loads from storage. Import discards prior per-type source dict before the API fetch and dump_state flushes the writes at the end, so the boot-time read is dead weight for import.

New ImportState class exposes only write methods (set_source, clear_source_type, mark/clear_source_authoritative, dump_state). No source / destination property is defined; reads fail at the language level with AttributeError. dump_state rejects DESTINATION / ALL since import never populates destination state.

New SourceStateWriter Protocol describes the shared write surface. State adds set_source / clear_source_type passthroughs so both classes satisfy it. Configuration.state is typed as Union[State, ImportState] so existing read access on State stays type-correct for sync/diffs/migrate/reset/prune code paths.

New --skip-state-load Click flag on import only. build_config validates (requires --resource-per-file + --resources; not combinable with --minimize-reads; only valid on import) and dispatches to ImportState.

Three import-path writer sites migrated to the Protocol methods: base_resource.py, resources_handler.py, synthetics_private_locations.py. Sync/diffs/migrate/reset/prune continue to use State's read accessors. The conflict checks for deprecated resources (logs_custom_pipelines+logs_pipelines, downtimes+downtime_schedules) still apply to ImportState callers; only the read-state fallback branch in _handle_deprecated is type-guarded with an isinstance check.

Storage backend construction shared via build_storage_backend in _base_storage.py so State and ImportState cannot drift on the per-backend init dispatch.

Diagnostic timing logs emit from a finally block in each backend's _list_and_load and put, so the log fires even when an SDK call raises and the exception propagates out. Cloud backends carry pages_listed, blobs_listed, blobs_downloaded, transient_errors, aborted, list_ms, download_ms, wall_ms — the list_ms vs download_ms split lets operators distinguish pagination cost from per-blob fetch cost when reading and writing race on the same bucket prefix; aborted distinguishes a successful empty result from an uncaught exception. transient_errors counts per-blob race-deletes (NoSuchKey on AWS, ResourceNotFoundError on Azure, NotFound on GCS) and JSON decode failures with the same semantics across all backends; SDK-internal retries below this layer are not visible. LocalFile emits the same schema (pages_listed=1) for cross-backend parity.

Test plan: 556 tests pass (514 existing + 42 new); black-clean at 120 cols. Failure-path tests assert the timing log still emits with aborted=1 when each backend's list call raises. A paged-iterator test exercises the real-SDK .pages branch on GCS.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@riyazsh riyazsh force-pushed the riyaz/import-state-skip-load branch from 4bafe54 to bb068a5 Compare May 13, 2026 03:59
Copy link
Copy Markdown
Collaborator

@heyronhay heyronhay left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nothing major that I can find, looks good!

@riyazsh riyazsh merged commit b9358d1 into main May 13, 2026
11 of 13 checks passed
@riyazsh riyazsh deleted the riyaz/import-state-skip-load branch May 13, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants