Feat/observability - #6
Merged
Merged
Conversation
- Revised the README to clarify the process for accessing the VPS and managing the firewall settings, emphasizing the use of `admin_cidrs` for SSH and kubectl access. - Introduced a new script, `fetch-hobby-kubeconfig.sh`, to automate the retrieval and configuration of the kubeconfig file after deploying the k3s cluster. - Updated Terraform configurations to replace `ssh_source_cidrs` with `admin_cidrs`, enhancing security by restricting access to specified IPs. - Enhanced cloud-init template to automatically set the TLS SAN to the VPS public IP, ensuring secure communication with the Kubernetes API. - Added output hints in Terraform to guide users on fetching the kubeconfig after deployment. These changes improve the security and usability of the hobby environment setup.
- Introduced middleware to attach X-Request-ID for log correlation and record RED metrics (request counts and duration) for all HTTP requests. - Updated logging format to include request_id for better traceability. - Added a debug endpoint to simulate errors for observability drills. - Configured Prometheus ServiceMonitor for scraping metrics from the API. - Updated Helm values to enable metrics scraping and added necessary configurations for Prometheus and Loki. These changes improve the observability and monitoring capabilities of the API, facilitating better performance tracking and error diagnosis.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end observability for the hobby API stack and tightens the hobby VPS bootstrap so metrics, logs, and kubectl access work without SSH tunnels.
X-Request-ID, recordshttp_requests_totalandhttp_request_duration_secondsfor every request (except/metrics), and logs structured request lines withrequest_idfor Loki filtering. Includes a teaching-only/debug/boomendpoint for error-rate drills.ServiceMonitorscrapes/metricson porthttpevery 30s whenmetrics.serviceMonitor.enabledis true (enabled by default invalues.yaml).kube-prometheus-stack(hobby-sized resources, 3d retention, cross-namespace ServiceMonitor discovery) and Grafana Loki (single-binary, filesystem storage, 72h retention). PrometheusRule alertApiHighErrorRatefires when 5xx ratio exceeds 5% for 2m.ssh_source_cidrsedits withadmin_cidrsinterraform.tfvars. Firewall allows SSH (22) and Kubernetes API (6443) only from admin IPs; HTTP/HTTPS remain open. k3s cloud-init sets--tls-santo the VPS public IP so direct kubectl works.scripts/fetch-hobby-kubeconfig.shwaits for k3s, fetches kubeconfig, rewrites127.0.0.1→ public IP, and clears stale SSH host keys. README andgitops/README.mdupdated accordingly.Details
API (
apps/api/app/main.py)ObservabilityMiddleware— propagates or generatesX-Request-ID, emits RED Prometheus metrics, logsmethod,path,status,duration_ms.RequestIdFilter— preventsKeyErroron log records missingrequest_id.GET /debug/boom— forced 500 for observability practice (marked teaching-only).Helm (
helm/api/)templates/servicemonitor.yaml— Prometheus Operator CRD targeting the API Service on/metrics.values.yaml—metrics.serviceMonitor.enabled: true.Monitoring (
monitoring/)kube-prometheus-stack-values.yaml— single-replica, resource-capped Prometheus/Grafana/Alertmanager for a small VPS.loki-values.yaml— single-binary Loki with filesystem storage.alerts/api-rules.yaml—ApiHighErrorRatePrometheusRule.README.mdanddashboards/api-red.json— empty placeholders (no content yet).Terraform & scripts
admin_cidrsvariable replaces inline firewall CIDR edits; no wide-open SSH default.--tls-sanfrom Hetzner metadata public IP.outputs.tfaddskubeconfig_hintpointing to the fetch script.scripts/fetch-hobby-kubeconfig.sh— automated kubeconfig setup afterterraform apply.Docs
README.md—admin_cidrsworkflow, fetch script usage, firewall/IP-change notes.gitops/README.md— direct kubectl on :6443, troubleshooting for host-key and firewall issues.