Respect third-party rate limits in media and HubSpot syncs

[xrendan]

Summary

• preserve and parse Retry-After values from media-feed HTTP 429 responses, using them for bounded retries without reporting routine transient failures as unhandled job exceptions
• identify feed requests as BuildCanadaBot and use Solid Queue concurrency controls to serialize feed fetches by publisher across batches and workers
• prevent contacts imported by the HubSpot full sync from enqueueing writes back to HubSpot
• retry legitimate outbound HubSpot contact writes on 429, 408, and 5xx responses, honoring Retry-After and surfacing only final exhaustion
• use one shared TransientError contract for feed fetching, Defuddler, and HubSpot, including status and retry-after metadata
• include the expanded memo authoring API guide requested for this branch

Testing

• focused transient-error tests: 31 runs, 86 assertions, 0 failures, 0 errors
• focused media concurrency tests: 15 runs, 48 assertions, 0 failures, 0 errors
• full suite: 1081 runs, 3981 assertions, 0 failures, 0 errors
• RuboCop on all changed Ruby files

[greptile-apps]

Greptile Summary

The PR adds bounded rate-limit retries for media and HubSpot synchronization, suppresses HubSpot write-backs for imported contacts, identifies media requests with a stable user agent, and expands the memo-authoring guide.

• Introduces a shared transient error carrying HTTP status and Retry-After information.
• Adds retry handling for media-feed, Defuddler, and outbound HubSpot requests.
• Adds publisher-keyed media job concurrency and focused tests.
• Prevents newly imported HubSpot contacts from immediately syncing back.
• Expands API documentation for automated memo authoring.

Confidence Score: 3/5

The PR does not appear safe to merge until same-publisher media requests are persistently spaced across both job completions and recurring dispatch batches.

The dispatcher still enqueues same-publisher feeds independently on every run, while the execution-side concurrency limit only prevents overlap and does not preserve the required interval after a request completes.

Files Needing Attention: app/jobs/search/media/dispatch_due_feeds_job.rb and app/jobs/search/media/fetch_feed_job.rb

Important Files Changed

Filename	Overview
app/jobs/search/media/fetch_feed_job.rb	Replaces declarative transient retries with explicit Retry-After scheduling and publisher-keyed concurrency; previously reported publisher-spacing behavior remains unresolved.
app/services/search/media/feed_fetcher.rb	Preserves Retry-After on HTTP 429 responses and adds a stable User-Agent while retaining transient and permanent response classification.
app/models/transient_error.rb	Adds a shared transient exception and parses delta-seconds or HTTP-date Retry-After values.
app/models/hubspot_contact.rb	Adds outbound HubSpot retry handling and suppresses create callbacks for records explicitly marked as imported.
app/services/hubspot_sync_service.rb	Converts retryable HubSpot API responses into shared transient errors carrying status and retry delay.
docs/api/memos.md	Expands the memo-authoring documentation into an operator and automation workflow.

Sequence Diagram

sequenceDiagram
  participant Dispatcher as Due-feed dispatcher
  participant Queue as Job queue
  participant FeedJob as FetchFeedJob
  participant Publisher as Publisher server

  Dispatcher->>Queue: Enqueue each due feed
  Queue->>FeedJob: Run job under publisher concurrency key
  FeedJob->>Publisher: Fetch feed
  alt HTTP 429
    Publisher-->>FeedJob: Retry-After
    FeedJob->>Queue: Schedule bounded retry
  else Success
    Publisher-->>FeedJob: Feed response
  end

Loading

_{Reviews (4): Last reviewed commit: "refactor: unify transient service errors" | Re-trigger Greptile}

[greptile-apps]

Publisher spacing resets across batches

When same-publisher delays from one batch extend into the next recurring dispatch, the new host_counts starts at zero and schedules another feed immediately, causing concurrent requests to that publisher and potentially triggering additional HTTP 429 responses.

Prompt To Fix With AI

This is a comment left during a code review.
Path: app/jobs/search/media/dispatch_due_feeds_job.rb
Line: 21-24

Comment:
**Publisher spacing resets across batches**

When same-publisher delays from one batch extend into the next recurring dispatch, the new `host_counts` starts at zero and schedules another feed immediately, causing concurrent requests to that publisher and potentially triggering additional HTTP 429 responses.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[greptile-apps]

Publisher requests remain unspaced

When multiple due feeds share a publisher domain, limits_concurrency serializes their execution but releases the lock as soon as each fetch finishes, so the next request starts immediately without the required five-second interval and can continue receiving HTTP 429 responses.

Prompt To Fix With AI

This is a comment left during a code review.
Path: app/jobs/search/media/fetch_feed_job.rb
Line: 6-9

Comment:
**Publisher requests remain unspaced**

When multiple due feeds share a publisher domain, `limits_concurrency` serializes their execution but releases the lock as soon as each fetch finishes, so the next request starts immediately without the required five-second interval and can continue receiving HTTP 429 responses.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.