Feat/api fetcher#52
Open
cchwala wants to merge 7 commits into
Open
Conversation
- fetchers/shared: config loader, atomic incoming writer, state persistence (cursor + seen-files), poll loop with exponential backoff - fetchers/api_fetcher: JWT auth with auto-refresh, windowed paginated fetcher (continuous + backfill modes, param_variants for RSL/TSL), entry point, example config.yml, Dockerfile (build ctx ./fetchers) - fetchers/api_fetcher/mock_server: minimal Flask mock with JWT auth and synthetic hourly CML data (POST /login/, POST /refresh/, GET /cml/)
Parses JSON files produced by api_fetcher. A field map (loaded from FIELD_MAP_PATH env var) does longest-prefix matching on the filename stem to route source fields to [time, cml_id, sublink_id, rsl, tsl]. Includes example_field_map.yml for the mock server (RSL + TSL split).
parser/service_logic.py: import parse_api_json_raw; add elif .json branch routing JSON files to write_rawdata (before else: quarantine) parser/main.py: add .json to FileWatcher extensions; add *.json glob in process_existing_files for startup replay docker-compose.dev.yml: adds mock_server (port 5001) and api_fetcher services for local development; mounts shared data volume and example field_map.yml
fetcher.py: in continuous mode, end was computed as start+chunk_hours; with overlap_seconds >= chunk_hours*3600 (as in the example config) the window end equalled the cursor, so the cursor never advanced (infinite loop). Fix: end = cursor_dt + chunk_hours; overlap only shifts start back. parser/main.py: process_existing_files returned early when there were no CSV files, skipping the JSON glob entirely. Fix: guard only the CSV section, always run the JSON section. mock_server/app.py + docker-compose.dev.yml: healthcheck hit GET /login/ which is POST-only; Flask returned 405, urlopen raised HTTPError, check always failed → api_fetcher never started. Fix: add GET /health endpoint, point healthcheck there.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #52 +/- ##
==========================================
+ Coverage 85.42% 87.42% +1.99%
==========================================
Files 28 26 -2
Lines 3012 2783 -229
==========================================
- Hits 2573 2433 -140
+ Misses 439 350 -89
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
service_logic.py: the JSON elif was placed AFTER 'raw'/'data' in name, but all api_fetcher filenames end in _data.json and matched the CSV branch first, making the JSON branch unreachable. Fix: check .json suffix first, before the substring checks. test_service_logic.py: add test_process_cml_file_json_archived covering the repaired JSON branch (patches parse_api_json_raw, asserts write and archive calls). test_api_json_parser.py: new test module for parse_api_json_raw covering prefix matching, RSL/TSL column routing, UTC datetime normalisation, empty-file handling, and error paths (missing field, no prefix match).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.