Add Redis telemetry store, split workload spec/status, and introduce state-store metrics

[miladhzzzz]

Motivation

• Reduce etcd churn for high-frequency reconciliation and event data by using Redis as a fast, ephemeral telemetry store.
• Improve storage efficiency and clarity by separating workload spec and status persistence to allow independent updates and easier projection.
• Add observability for state-store operations to monitor where writes are occurring.

Description

• Added Redis dependency and configuration (REDIS_ADDR, REDIS_PASSWORD, REDIS_DB, REDIS_RECONCILE_TTL, REDIS_EVENT_TTL, REDIS_EVENT_MAX_ENTRIES) in internal/config and go.mod/go.sum updates to include github.com/redis/go-redis/v9 and other transitive modules.
• Implemented Redis initialization and telemetry helpers in internal/scheduler/redis_store.go, with fallback to etcd when Redis is unavailable, and functions to write reconciliation telemetry and event histories.
• Split workload persistence into workloads-spec/ and workloads-status/ prefixes by adding workload_projection.go, refactoring saveWorkload to persist spec and status separately, and updating keys/paths in state_store.go and other scheduler methods (e.g. GetWorkloads, GetWorkloadByID, DeleteWorkloadWithContext).
• Added a Prometheus counter state_store_writes_total and helper IncStateStoreWrite in internal/metrics and incremented it for spec, status, retry, assignment, reconciliation, and event write paths.
• Enhanced scheduler behavior: initialize Redis in NewScheduler, close Redis in Close, avoid unnecessary writes in UpdateWorkloadStatus, skip empty log writes in UpdateWorkloadLogs, and avoid persisting unchanged metadata in UpdateWorkloadMetadata.
• Reconciler now attempts to persist high-churn reconciliation metadata to Redis (with TTL) and falls back to etcd if Redis writes fail, and avoids redundant persistence for repeated NoAction results.

Testing

• No automated tests were executed as part of this change.

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ec66c23d71

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve CreatedAt when rebuilding split workloads

When reading a workload from the new split spec/status keys, the reconstruction copies most spec fields but never assigns spec.CreatedAt back to models.Workload.CreatedAt. After any scheduler restart or cache refresh, workloads loaded through this path have a zero creation timestamp, so workloadView omits CreatedAt and callers lose the creation time even though it was written into the spec projection.

Useful? React with 👍 / 👎.