Add --issue-timeline flag and refresh timelines when cross-references change (#168)

[Iamrodos]

Summary

Fixes #168.

Adds an opt-in --issue-timeline flag, which backs up GitHub's issue timeline under a separate timeline_data key on each issue, alongside the existing and unmodified event_data.

This captures cross-references, where one issue or pull request mentions another. --issue-events reads /issues/{number}/events, which returns a restricted set of event types and never includes cross-references; GitHub only exposes those through /issues/{number}/timeline. So an issue mentioned from elsewhere shows no trace of it in the backup today.

There are two faults behind the report and this fixes both. The data was not obtainable at all, so full backups missed it too, not just incremental ones. Separately, the reporter's own diagnosis about since was correct: a cross-reference does not bump the referenced issue's updated_at, so the issue is never listed and never revisited. The two are independent, and fixing only the endpoint would leave dormant issues stale forever.

The reporter's own issue, eclipse-sumo/sumo#21, shows the scale of what is lost. /events returns 310 entries and not one of them is a cross-reference. /timeline returns 2,130 entries for the same issue, including 193 cross-references that the tool cannot see today.

Details

• --issue-timeline complements --issue-events, it does not replace it. Neither endpoint contains the other. The timeline adds cross-references, commits and reviews, but /events carries referenced entries the timeline drops, 130 of 254 across a full backup of this repository. Run both for complete coverage.
• event_data is untouched. Same endpoint, same contents, same shape, whether or not the timeline is enabled. Nothing that already reads it is affected.
• commented entries are filtered out before storing. They embed the full comment body, which --issue-comments already backs up, and two copies invite divergence. On sumo#21 that is 1,627 of 2,130 entries.
• Everything else is stored verbatim, including the large source.issue object on cross-references, consistent with how event_data and review_data are stored.
• Timeline entries are not all shaped like events. cross-referenced and committed carry no id field, so consumers of timeline_data must not assume one. Documented in the README.
• Included in --all, on the grounds that an "everything" mode silently omitting a whole class of GitHub activity is the bug being reported. It is not marked [*]; the cost is bytes and requests, not clones.

Impact

On cross-reference-heavy issues the timeline is materially heavier than events, because each cross-referenced entry embeds the referencing issue in full. Measured through the tool on sumo#21, writing the same issue both ways:

Flag	Requests	Entries stored	File on disk
--issue-events	4	310	517 KB
--issue-timeline	22	503	4,636 KB

5.5x the requests and 9x the disk for one issue. The requests are unavoidable: the endpoint has no server-side type filter, so the 1,627 comments are paginated through and then discarded. sumo#21 is the worst case I could find; ordinary issues are unaffected, and a 512-item private repository of mine never exceeded 62 timeline entries on any issue.

The refresh check described below adds a GraphQL sweep on each incremental run, costing one rate-limit point per 100 items, against a quota separate from REST. Measured at 6 requests for this repository's 528 issues and pull requests. It is cheap in quota terms even at size, but not instant: sweeping eclipse-sumo/sumo's 18,219 items took around 200 points and five minutes.

The check also parses each stored issue JSON to read two values, so a large backup does one parse per item per run. That is local I/O rather than network, and caching the values would mean a sidecar checkpoint file, which is exactly what this design avoids. A faster JSON parser would help, but the project keeps its runtime dependencies minimal and this does not seem worth adding one for.

Keeping timelines current

Fetching the timeline is only half of it. Because being referenced does not change updated_at, an incremental run never lists the issue and so never re-fetches its timeline. An issue can be mentioned from another repository years after its last activity.

On incremental runs, github-backup now asks GraphQL for each item's cross-reference count and newest timestamp, compares both against what is already saved in timeline_data, and re-fetches the timeline of anything that differs. Items the since listing did not return are fetched individually.

Both values are compared because the count alone would miss a reference added and another deleted between runs, while the timestamp alone would miss a deletion.

Pull requests are checked too when they are stored as issues, which is the case unless --pulls is used. They need a separate query for two reasons: GraphQL's issues connection excludes them, and totalCount ignores the itemTypes filter on a pull request although it honours it on an issue. Their cross-references are therefore counted from the nodes instead, falling back to comparing timestamps alone where that count may have been truncated. Without this, 390 of this repository's 526 stored items would never be checked.

This needs no new state on disk — no checkpoint file, no interaction with the existing last_update machinery — because the comparison is against the cross-references already stored. It self-heals after an interrupted run and picks up deletions. If the GraphQL call fails it warns and the backup continues, and it is skipped entirely when no token is configured, since GraphQL always requires one.

Scope note

Backfill is partial when enabling the flag on an existing incremental backup. Cross-references on older issues are picked up automatically by the check above, since their stored count is zero and GitHub's is not. Timeline entries of other kinds, commits and reviews, are only fetched when the issue itself is updated, so those need one run without --incremental to fill in historically. Those kinds do move updated_at, so they stay current from then on.

Testing

• tests/test_issue_timeline.py, 14 tests. For the flag: wiring through parse_args, the commented filter combined with verbatim storage of everything else, no fetch without the flag, event_data unaffected when both flags are set, inclusion in --all. For the refresh check: an added reference, an add paired with a delete where the count is unchanged, no false positive when nothing changed, a GraphQL failure warning rather than aborting, an item since excludes still being refreshed, multi-page cursor following, pull requests counted from nodes rather than the misleading totalCount, pull requests skipped when --pulls handles them, and a truncated count falling back to timestamp comparison.
• End to end on a live private repository. Baseline of 512 items, then a comment on issue [BUG] Repos with issues/pr disabled shouldn't error #511 referencing issue Add an option to prefer checkouts over SSH #6, dormant since July 2024. GitHub left Add an option to prefer checkouts over SSH #6's updated_at at 2024-07-02, two years before the checkpoint, so the since listing could not return it. The incremental run reported Refreshing 1 issues with new cross-references and Add an option to prefer checkouts over SSH #6 gained the reference. This is the exact failure reported in Incremental backup skips some new events #168.
• No false positives. A second incremental over this repository's 527 items immediately after a clean baseline flagged nothing, using 6 sweep requests.
• At scale. Sweeping eclipse-sumo/sumo, 18,219 items, found 3,714 with cross-references and no truncated counts, for about 200 GraphQL points and five minutes. sumo#21 came back as 194 cross-references, matching a full REST timeline fetch exactly.
• Pagination. A live run of backup_issues against sumo#21 with only the issue listing stubbed: 22 paginated requests, 2,130 entries fetched, 1,627 commented filtered, 504 stored including all 194 cross-references. Neither full-repository run reached 100 entries on a single item, so this is the only test of multi-page timelines.
• Full suite: 170 passed on Python 3.10, 3.11, 3.12, 3.13 and 3.14; flake8 clean (--ignore=E501,E203,W503); rst-lint README.rst clean; twine check clean on sdist and wheel; new code black-clean; README options block matches --help character for character.