diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index c8872e37..7edb4c73 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -294,6 +294,8 @@ jobs: - name: Run operator tests working-directory: packages/cluster-operator + env: + GOFLAGS: -mod=readonly run: go test ./... - name: Lint cluster chart diff --git a/.woodpecker.yml b/.woodpecker.yml index 4f8c556e..69673f6f 100644 --- a/.woodpecker.yml +++ b/.woodpecker.yml @@ -250,9 +250,9 @@ steps: image: mirror.gcr.io/library/golang:1.25.1-bookworm@sha256:c423747fbd96fd8f0b1102d947f51f9b266060217478e5f9bf86f145969562ee directory: packages/cluster-operator commands: - - GOMAXPROCS=1 GOFLAGS=-p=1 go test ./api/... ./controllers/... ./cmd/... + - GOMAXPROCS=1 GOFLAGS='-mod=readonly -p=1' go test ./api/... ./controllers/... ./cmd/... - mkdir -p ../../artifacts/cluster-proof - - CGO_ENABLED=0 GOMAXPROCS=1 GOFLAGS=-p=1 go test -c ./charttests -o ../../artifacts/cluster-proof/chart-contract.test + - CGO_ENABLED=0 GOMAXPROCS=1 GOFLAGS='-mod=readonly -p=1' go test -c ./charttests -o ../../artifacts/cluster-proof/chart-contract.test backend_options: kubernetes: serviceAccountName: woodpecker-ci-untrusted diff --git a/docs/CLUSTER_OPERATOR.md b/docs/CLUSTER_OPERATOR.md index 67598da5..aae439e4 100644 --- a/docs/CLUSTER_OPERATOR.md +++ b/docs/CLUSTER_OPERATOR.md @@ -23,13 +23,15 @@ That declaration does not make a non-RWX backend safe. Missing classes, classes ## Install -Installing with defaults creates no controller, gateway, session workload, RBAC, Secret, or network policy: +Installing with defaults creates no controller, gateway, session workload, RBAC, Secret, or network policy. Use the lifecycle runner even for a fresh install so existing live objects and compatibility fixtures are checked against the proposed schemas before the CRDs are server-validated, established, and storage-version-checked and before Helm runs: ```sh -helm install t4-cluster deploy/charts/t4-cluster --namespace t4-system --create-namespace +scripts/cluster-ci/crd-lifecycle.sh install -- \ + helm install t4-cluster deploy/charts/t4-cluster \ + --namespace t4-system --create-namespace --skip-crds ``` -Helm processes files in `crds/` separately. Use `--skip-crds` if CRD lifecycle is administered independently. To enable the control plane, provide a private values file or a deployment controller values source: +Helm processes files in `crds/` separately on a direct install, but does not upgrade or delete them later. The release procedure therefore administers CRDs independently and always passes `--skip-crds` to Helm. To enable the control plane, provide a private values file or a deployment controller values source: ```yaml enabled: true @@ -112,11 +114,7 @@ The controller uses uncached, exact-name `get` permission for only that administ Reusable credential projection is unsupported because OMP and arbitrary session tools share one workload security boundary. `allowUnauthenticated: true` with empty `credentialSecret` and `credentialKey` is therefore mandatory. The two credential fields remain only as cutover sentinels: Helm, the controller, and the image entrypoint all reject an old credential-mode configuration instead of exposing it. Before OMP starts, the image requires the pinned `auth_credentials` table to be empty, rejects the pinned secret settings (`auth.broker.token`, `hindsight.apiToken`, `searxng.token`, and `dev.autoqaPush.token`), rejects broker token and encrypted snapshot files, and fails closed on unknown schemas or linked profile paths. It never deletes or rewrites durable user state. Because Helm rejects legacy credential values before rendering the new Deployment, a failed upgrade leaves the prior release unchanged. Stop development sessions and clear any legacy values and credential state explicitly before adopting this pre-production boundary. Provider authentication must live behind a private model gateway that authorizes the session through infrastructure identity or network policy while presenting an `auth: none` endpoint inside the session's allowed route. Do not expose that endpoint to the Internet or a shared untrusted network. The chart remains provider- and model-neutral. -Install or enable with immutable digests: - -```sh -helm upgrade --install t4-cluster deploy/charts/t4-cluster --namespace t4-system --values operator-values.yaml -``` +Install a new release with immutable digests by adding `--values operator-values.yaml` to the lifecycle-runner `install` command above. For an existing release, use the lifecycle-runner `upgrade` procedure below. Do not use `helm upgrade --install`: install and upgrade have different compatibility preflights. The controller always has two replicas and uses a Kubernetes Lease named from `t4-cluster-operator.cluster.t4.dev`; one replica reconciles at a time. The server defaults to three stateless replicas and supports a minimum of two. Its Deployment uses `maxUnavailable: 0`, a `minAvailable: 2` PDB, topology spread, anti-affinity, readiness draining, and an explicit `k3s-worker-02` exclusion. Session pods also exclude that node by default. Additional cluster-specific exclusions belong in deployment values, not this portable chart. @@ -150,28 +148,96 @@ The session runtime verifies and builds the exact OMP tag `t4code-17.0.5-appserv Session pods do not receive an automatically mounted ServiceAccount token. The explicit projected reviewer token can only create TokenReviews and is not resource-authorized. The namespace-local ConfigMap projection contains credential-free `auth: none` OMP models and credential-free settings; session Pods receive no provider Secret reference or reusable provider credential. All containers drop capabilities, disallow privilege escalation, use RuntimeDefault seccomp, and use read-only root filesystems. No per-session NodePort, LoadBalancer, host network, host PID, host display, or hostPath is created. -## Upgrade and rollback +## Upgrade, storage migration, and rollback -CRD changes must remain structural and additive. Helm installs files under `crds/` on first install but does not upgrade or delete them. Before every `helm upgrade`, an administrator must review and apply each newer CRD independently, wait for API discovery to serve the updated schemas, and only then upgrade workloads with the new immutable image digests: +The three namespaced CRDs have exactly one version: `cluster.t4.dev/v1alpha1`, with `served: true`, `storage: true`, a structural schema, and a status subresource. Changes in this version must remain additive: existing fields keep their meaning and validation, while new fields are optional or have safe defaults. The compatibility fixtures in `packages/cluster-operator/api/v1alpha1/testdata/compat/` are persisted old-object shapes, including status and finalizers, and must continue to validate and round-trip without losing any declared field. + +Helm does not upgrade CRDs. For every workload upgrade, run the fail-closed lifecycle command: ```sh -kubectl apply --server-side -f deploy/charts/t4-cluster/crds/ -kubectl wait --for=condition=Established crd/t4clusterhosts.cluster.t4.dev crd/t4workspaces.cluster.t4.dev crd/t4sessions.cluster.t4.dev +scripts/cluster-ci/crd-lifecycle.sh upgrade -- \ + helm upgrade t4-cluster deploy/charts/t4-cluster \ + --namespace t4-system --values operator-values.yaml --skip-crds ``` -Do not rely on `helm upgrade` to change CRDs. If a CRD pre-upgrade apply fails validation, stop the upgrade and keep the currently deployed controller/server images; do not force-replace the CRD or remove stored versions. +The command performs this exact order: + +1. Before any cluster access, validate every complete compatibility fixture (including `status`) against the proposed structural OpenAPI schema and CEL programs with the Kubernetes apiextensions validators. The check also rejects declared fields that the structural pruner would remove. +2. With `get` access to the three exact CRD definitions and `list` access to the corresponding namespaced T4 resources, detect which definitions already exist, enumerate every live CR for each existing definition across namespaces, and validate each complete object, including `status`, against the same proposed OpenAPI, CEL create, CEL unchanged-update, and pruning checks. A denied CRD read, denied or malformed object list, or incompatible object fails closed; an absent definition is safely empty on a fresh install. +3. Server-side dry-run the reviewed CRDs with strict validation. Upgrades stop before any mutation if a local proposed-schema validation, live-object enumeration, or CRD dry-run fails. +4. Apply the CRDs server-side without force conflicts. +5. Wait for all three CRDs to report `Established`. +6. Fetch the served `cluster.t4.dev/v1alpha1` OpenAPI v3 document three times and require every published resource schema to match the semantics generated from the proposed CRDs. Retained `Established=True` is not readiness when discovery still serves an old schema. +7. Server-side dry-run the compatibility fixtures against the converged admission path. +8. Require each CRD's `status.storedVersions` to be exactly `v1alpha1`. +9. Execute the supplied Helm command, which must use `--skip-crds`. + +The corresponding administrative checks are executable independently: ```sh -helm upgrade t4-cluster deploy/charts/t4-cluster --namespace t4-system --values operator-values.yaml +(cd packages/cluster-operator && \ + go run ./cmd/crd-preflight fixtures ../../deploy/charts/t4-cluster/crds api/v1alpha1/testdata/compat) +live_objects=$(mktemp) +trap 'rm -f "$live_objects"' EXIT +trap 'exit 129' HUP +trap 'exit 130' INT +trap 'exit 143' TERM +for resource in t4clusterhosts.cluster.t4.dev t4workspaces.cluster.t4.dev t4sessions.cluster.t4.dev; do + installed_crd=$(kubectl get "crd/$resource" --ignore-not-found -o name) + if [ -z "$installed_crd" ]; then + continue + fi + kubectl get "$resource" --all-namespaces -o json >"$live_objects" + (cd packages/cluster-operator && \ + go run ./cmd/crd-preflight objects ../../deploy/charts/t4-cluster/crds <"$live_objects") +done +rm -f "$live_objects" +trap - EXIT HUP INT TERM +kubectl apply --server-side --dry-run=server --validate=strict \ + --field-manager=t4-crd-lifecycle -f deploy/charts/t4-cluster/crds/ +kubectl apply --server-side --validate=strict \ + --field-manager=t4-crd-lifecycle -f deploy/charts/t4-cluster/crds/ +kubectl wait --for=condition=Established --timeout=120s \ + crd/t4clusterhosts.cluster.t4.dev \ + crd/t4workspaces.cluster.t4.dev \ + crd/t4sessions.cluster.t4.dev +for observation in 1 2 3; do + kubectl get --raw /openapi/v3/apis/cluster.t4.dev/v1alpha1 | \ + (cd packages/cluster-operator && go run ./cmd/crd-preflight served ../../deploy/charts/t4-cluster/crds) +done +kubectl apply --dry-run=server --validate=strict --namespace default \ + -f packages/cluster-operator/api/v1alpha1/testdata/compat/ +for crd in t4clusterhosts.cluster.t4.dev t4workspaces.cluster.t4.dev t4sessions.cluster.t4.dev; do + test "$(kubectl get "crd/$crd" -o 'jsonpath={.status.storedVersions[*]}')" = v1alpha1 +done ``` -For an application rollback, retain the additive CRDs and use the previous known-compatible values and image digest set: +Do not rely on `helm upgrade` to change CRDs. Never use `kubectl replace --force`, `kubectl apply --force-conflicts`, delete/recreate a CRD, or alter `status.storedVersions` outside the verified migration sequence below. A preflight failure leaves the live CRDs, custom resources, controller/server workloads, and session workloads untouched. A failure after additive CRD apply but before Helm leaves the prior workloads running against the still-backward-compatible schema; investigate and rerun the gates rather than attempting CRD rollback. + +### Future `v1beta1` conversion and storage procedure + +There is no `v1beta1` API today. A future proposal is a separate incompatible lifecycle change and may proceed only through these gates: + +1. Back up all three CRDs and every custom resource, record per-kind object counts, and prove a restore in an isolated cluster. +2. Review separate `v1beta1` schemas, defaults, conversion semantics, downgrade semantics, and old/new/old fixture round-trips. Every field represented in either version must survive conversion; conversion may not manufacture product or runtime state in status. +3. Deploy a highly available conversion webhook with a PodDisruptionBudget, strict TLS/service identity, readiness checks, failure policy, metrics, and alerts. Prove conversion availability before adding a served version to any CRD. +4. Add `v1beta1` as served but not storage, retain `v1alpha1` as served and storage, wait for `Established`, and prove reads and writes through both versions plus all compatibility fixtures. Do not advance while any controller, gateway, backup, or recovery tool cannot read both versions. +5. In a later reviewed CRD apply, set `v1beta1` storage to true and `v1alpha1` storage to false while keeping both served. Reconfirm conversion health, then rewrite every object through the Kubernetes API using an approved storage-version migrator; merely changing `spec.versions[*].storage` does not migrate stored objects. +6. Compare pre/post object counts and read every rewritten object through both served versions. Verify identity, declared spec and status fields, finalizers, and conversion round-trips. Stop on any missing object or lossy read. +7. Only after that verification, explicitly retire the old storage record through the CRD status subresource for each CRD, for example `kubectl patch customresourcedefinition t4sessions.cluster.t4.dev --subresource=status --type=json -p='[{"op":"replace","path":"/status/storedVersions","value":["v1beta1"]}]'`. Repeat for hosts and workspaces. This status update records completed storage migration; it does not perform migration. +8. Read each CRD back and require `status.storedVersions` to be exactly `[v1beta1]`. Keep `v1alpha1` served, the webhook available, the verified backup, and dual-read binaries for the entire rollback window. Removing the old served version is a later explicit change after the window expires. + +If conversion or migration fails, stop forward rollout. Do not force-replace or downgrade CRDs. While both versions remain served and conversion is healthy, roll workloads back to the prior dual-read image set. If in-place compatibility cannot be proven, stop mutations and restore the verified backup under the reviewed recovery procedure before resuming service. + +### Workload rollback + +CRDs remain additive and installed. Roll back the known-compatible controller, server, T4 session runtime, and OMP digest set together: ```sh helm rollback t4-cluster REVISION --namespace t4-system ``` -Do not roll OMP independently of the T4 session runtime. Roll back the known-compatible T4/OMP image set together. The pinned authority boundary is not negotiated down. +Do not roll OMP independently of the T4 session runtime. The pinned authority boundary is not negotiated down. ## Uninstall diff --git a/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4clusterhost.yaml b/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4clusterhost.yaml new file mode 100644 index 00000000..d618128d --- /dev/null +++ b/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4clusterhost.yaml @@ -0,0 +1,25 @@ +apiVersion: cluster.t4.dev/v1alpha1 +kind: T4ClusterHost +metadata: + name: legacy-host +spec: + storageClassName: legacy-rwx + runtimeProfiles: + - default + - gui + ciProvider: + secretRef: + name: legacy-ci-token + configMapRef: + name: legacy-ci-config + allowedOrigins: + - https://t4.example.test +status: + observedGeneration: 3 + conditions: + - type: Available + status: "True" + observedGeneration: 3 + lastTransitionTime: "2026-01-15T12:00:00Z" + reason: Reconciled + message: Legacy host remains available diff --git a/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4session.yaml b/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4session.yaml new file mode 100644 index 00000000..dde06302 --- /dev/null +++ b/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4session.yaml @@ -0,0 +1,29 @@ +apiVersion: cluster.t4.dev/v1alpha1 +kind: T4Session +metadata: + name: legacy-session + finalizers: + - cluster.t4.dev/session-cleanup +spec: + hostRef: legacy-host + workspaceRef: legacy-workspace + title: Legacy session + runtimeProfile: gui + initialPromptSecretRef: + name: legacy-initial-prompt + ci: + repositoryId: legacy/project + ref: refs/heads/main + commit: 0123456789abcdef0123456789abcdef01234567 +status: + observedGeneration: 7 + podName: legacy-session-pod + serviceName: legacy-session + phase: Running + conditions: + - type: Available + status: "True" + observedGeneration: 7 + lastTransitionTime: "2026-01-15T12:00:00Z" + reason: RuntimeReady + message: Legacy session infrastructure is running diff --git a/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4workspace.yaml b/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4workspace.yaml new file mode 100644 index 00000000..dcd9ac1c --- /dev/null +++ b/packages/cluster-operator/api/v1alpha1/testdata/compat/v1alpha1-t4workspace.yaml @@ -0,0 +1,29 @@ +apiVersion: cluster.t4.dev/v1alpha1 +kind: T4Workspace +metadata: + name: legacy-workspace + finalizers: + - cluster.t4.dev/workspace-protection +spec: + hostRef: legacy-host + displayName: Legacy workspace + owner: user:legacy-owner + repository: + repositoryId: legacy/project + ref: refs/heads/main + commit: 0123456789abcdef0123456789abcdef01234567 + size: 20Gi + retentionPolicy: Retain +status: + observedGeneration: 4 + pvcName: legacy-workspace-data + pvcPhase: Bound + capacity: 20Gi + phase: Ready + conditions: + - type: Ready + status: "True" + observedGeneration: 4 + lastTransitionTime: "2026-01-15T12:00:00Z" + reason: StorageBound + message: Legacy workspace storage is ready diff --git a/packages/cluster-operator/api/v1alpha1/types_test.go b/packages/cluster-operator/api/v1alpha1/types_test.go index fde778fb..4bb6390a 100644 --- a/packages/cluster-operator/api/v1alpha1/types_test.go +++ b/packages/cluster-operator/api/v1alpha1/types_test.go @@ -1,8 +1,10 @@ package v1alpha1_test import ( + "encoding/json" "os" "path/filepath" + "reflect" "strings" "testing" @@ -11,6 +13,7 @@ import ( structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/serializer" "sigs.k8s.io/yaml" clusterv1alpha1 "github.com/LycaonLLC/t4-code/packages/cluster-operator/api/v1alpha1" @@ -141,6 +144,107 @@ func TestCRDSchemasAreStructuralBoundedAndValidated(t *testing.T) { } } +func TestOldObjectsDefaultAndRoundTripDeclaredFields(t *testing.T) { + tests := []struct { + fixture string + crd string + }{ + {"v1alpha1-t4clusterhost.yaml", "t4clusterhosts.cluster.t4.dev.yaml"}, + {"v1alpha1-t4workspace.yaml", "t4workspaces.cluster.t4.dev.yaml"}, + {"v1alpha1-t4session.yaml", "t4sessions.cluster.t4.dev.yaml"}, + } + + scheme := runtime.NewScheme() + if err := clusterv1alpha1.AddToScheme(scheme); err != nil { + t.Fatal(err) + } + decoder := serializer.NewCodecFactory(scheme).UniversalDeserializer() + + for _, tc := range tests { + t.Run(tc.fixture, func(t *testing.T) { + fixtureRaw, err := os.ReadFile(filepath.Join("testdata", "compat", tc.fixture)) + if err != nil { + t.Fatal(err) + } + var declared map[string]interface{} + if err := yaml.Unmarshal(fixtureRaw, &declared); err != nil { + t.Fatalf("decode fixture: %v", err) + } + + crdRaw, err := os.ReadFile(filepath.Join("..", "..", "..", "..", "deploy", "charts", "t4-cluster", "crds", tc.crd)) + if err != nil { + t.Fatal(err) + } + var crd apiextensionsv1.CustomResourceDefinition + if err := yaml.Unmarshal(crdRaw, &crd); err != nil { + t.Fatalf("decode CRD: %v", err) + } + if len(crd.Spec.Versions) != 1 || crd.Spec.Versions[0].Name != "v1alpha1" || !crd.Spec.Versions[0].Served || !crd.Spec.Versions[0].Storage { + t.Fatalf("storage contract changed: %#v", crd.Spec.Versions) + } + + if tc.fixture == "v1alpha1-t4session.yaml" { + specSchema := crd.Spec.Versions[0].Schema.OpenAPIV3Schema.Properties["spec"] + guiSchema := specSchema.Properties["guiEnabled"] + if guiSchema.Default == nil || string(guiSchema.Default.Raw) != "false" { + t.Fatalf("guiEnabled schema default = %#v, want false", guiSchema.Default) + } + } + + admittedJSON, err := json.Marshal(declared) + if err != nil { + t.Fatal(err) + } + object, gvk, err := decoder.Decode(admittedJSON, nil, nil) + if err != nil { + t.Fatalf("decode through registered v1alpha1 API: %v", err) + } + if gvk.GroupVersion() != clusterv1alpha1.GroupVersion { + t.Fatalf("decoded version = %s", gvk.GroupVersion()) + } + roundTripJSON, err := json.Marshal(object) + if err != nil { + t.Fatal(err) + } + var roundTripped map[string]interface{} + if err := json.Unmarshal(roundTripJSON, &roundTripped); err != nil { + t.Fatal(err) + } + assertDeclaredFieldsPreserved(t, "$", declared, roundTripped) + }) + } +} + +func assertDeclaredFieldsPreserved(t *testing.T, path string, declared, roundTripped interface{}) { + t.Helper() + switch expected := declared.(type) { + case map[string]interface{}: + actual, ok := roundTripped.(map[string]interface{}) + if !ok { + t.Fatalf("%s changed type: got %T", path, roundTripped) + } + for key, expectedValue := range expected { + actualValue, found := actual[key] + if !found { + t.Fatalf("%s.%s was lost", path, key) + } + assertDeclaredFieldsPreserved(t, path+"."+key, expectedValue, actualValue) + } + case []interface{}: + actual, ok := roundTripped.([]interface{}) + if !ok || len(actual) != len(expected) { + t.Fatalf("%s changed array shape: got %#v", path, roundTripped) + } + for index := range expected { + assertDeclaredFieldsPreserved(t, path, expected[index], actual[index]) + } + default: + if !reflect.DeepEqual(declared, roundTripped) { + t.Fatalf("%s changed from %#v to %#v", path, declared, roundTripped) + } + } +} + func TestCRDsHaveCrossFieldCELAndForbidClientRuntimeAuthority(t *testing.T) { for _, name := range []string{"t4clusterhosts.cluster.t4.dev.yaml", "t4workspaces.cluster.t4.dev.yaml", "t4sessions.cluster.t4.dev.yaml"} { raw, err := os.ReadFile(filepath.Join("..", "..", "..", "..", "deploy", "charts", "t4-cluster", "crds", name)) diff --git a/packages/cluster-operator/charttests/chart_contract_test.go b/packages/cluster-operator/charttests/chart_contract_test.go index 611bd493..9bbfa883 100644 --- a/packages/cluster-operator/charttests/chart_contract_test.go +++ b/packages/cluster-operator/charttests/chart_contract_test.go @@ -487,7 +487,21 @@ func TestCRDsRemainExplicitAcrossUpgradeAndUninstall(t *testing.T) { if err != nil { t.Fatal(err) } - for _, required := range []string{"helm upgrade", "helm rollback", "helm uninstall", "kubectl apply --server-side -f deploy/charts/t4-cluster/crds/", "condition=Established", "Do not rely on `helm upgrade` to change CRDs", "Retain", "Delete", "CRDs are not removed"} { + for _, required := range []string{ + "scripts/cluster-ci/crd-lifecycle.sh upgrade", + "helm upgrade", + "--skip-crds", + "helm rollback", + "helm uninstall", + "kubectl apply --server-side --dry-run=server", + "condition=Established", + "status.storedVersions", + "Do not rely on `helm upgrade` to change CRDs", + "Future `v1beta1` conversion and storage procedure", + "Retain", + "Delete", + "CRDs are not removed", + } { if !strings.Contains(string(docs), required) { t.Fatalf("operator guide lacks upgrade/uninstall contract %q", required) } diff --git a/packages/cluster-operator/cmd/crd-preflight/main.go b/packages/cluster-operator/cmd/crd-preflight/main.go new file mode 100644 index 00000000..0a55241b --- /dev/null +++ b/packages/cluster-operator/cmd/crd-preflight/main.go @@ -0,0 +1,382 @@ +package main + +import ( + "context" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "reflect" + "strings" + + apiextensions "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions" + apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" + structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema" + "k8s.io/apiextensions-apiserver/pkg/apiserver/schema/cel" + "k8s.io/apiextensions-apiserver/pkg/apiserver/schema/pruning" + apiservervalidation "k8s.io/apiextensions-apiserver/pkg/apiserver/validation" + "k8s.io/apiextensions-apiserver/pkg/controller/openapi/builder" + "k8s.io/apimachinery/pkg/runtime" + kubejson "k8s.io/apimachinery/pkg/util/json" + "k8s.io/apimachinery/pkg/util/validation/field" + "sigs.k8s.io/yaml" +) + +const ( + perCallCELCostLimit = 1_000_000 + runtimeCELCostBudget = 10_000_000 +) + +type groupVersionKind struct { + Group string + Version string + Kind string +} + +type candidateSchema struct { + crd *apiextensionsv1.CustomResourceDefinition + version *apiextensionsv1.CustomResourceDefinitionVersion + internal *apiextensions.JSONSchemaProps + structural *structuralschema.Structural +} + +func main() { + if len(os.Args) < 2 { + usage() + } + var err error + switch os.Args[1] { + case "fixtures": + if len(os.Args) != 4 { + usage() + } + err = validateFixtures(os.Args[2], os.Args[3]) + case "objects": + if len(os.Args) != 3 { + usage() + } + err = validateObjects(os.Args[2], os.Stdin) + case "served": + if len(os.Args) != 3 { + usage() + } + err = verifyServedSchemas(os.Args[2], os.Stdin) + default: + usage() + } + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} + +func usage() { + fmt.Fprintln(os.Stderr, "usage: crd-preflight fixtures CRD_DIRECTORY FIXTURE_DIRECTORY | objects CRD_DIRECTORY | served CRD_DIRECTORY") + os.Exit(64) +} + +func validateFixtures(crdDirectory, fixtureDirectory string) error { + candidates, err := loadCandidates(crdDirectory) + if err != nil { + return err + } + paths, err := yamlPaths(fixtureDirectory) + if err != nil { + return err + } + var validationErrors []error + for _, path := range paths { + object, err := decodeYAMLObject(path) + if err != nil { + validationErrors = append(validationErrors, err) + continue + } + validationErrors = append(validationErrors, validateCandidateObject(path, "fixture", object, candidates)...) + } + return errors.Join(validationErrors...) +} + +func validateObjects(crdDirectory string, input io.Reader) error { + candidates, err := loadCandidates(crdDirectory) + if err != nil { + return err + } + var list map[string]interface{} + decoder := json.NewDecoder(input) + decoder.UseNumber() + if err := decoder.Decode(&list); err != nil { + return fmt.Errorf("decode live object list: %w", err) + } + if err := kubejson.ConvertMapNumbers(list, 0); err != nil { + return fmt.Errorf("decode live object list numbers: %w", err) + } + var trailing interface{} + if err := decoder.Decode(&trailing); !errors.Is(err, io.EOF) { + if err == nil { + return errors.New("decode live object list: multiple JSON documents") + } + return fmt.Errorf("decode live object list trailing data: %w", err) + } + rawItems, found := list["items"] + if !found { + return errors.New("decode live object list: missing items") + } + items, ok := rawItems.([]interface{}) + if !ok { + return errors.New("decode live object list: items is not an array") + } + var validationErrors []error + for index, rawItem := range items { + object, ok := rawItem.(map[string]interface{}) + if !ok { + validationErrors = append(validationErrors, fmt.Errorf("live object item %d is not an object", index)) + continue + } + path, err := liveObjectPath(index, object) + if err != nil { + validationErrors = append(validationErrors, err) + continue + } + validationErrors = append(validationErrors, validateCandidateObject(path, "object", object, candidates)...) + } + return errors.Join(validationErrors...) +} + +func liveObjectPath(index int, object map[string]interface{}) (string, error) { + metadata, ok := object["metadata"].(map[string]interface{}) + if !ok { + return "", fmt.Errorf("live object item %d has no metadata", index) + } + name, _ := metadata["name"].(string) + namespace, _ := metadata["namespace"].(string) + if name == "" || namespace == "" { + return "", fmt.Errorf("live object item %d must have namespace and name", index) + } + return fmt.Sprintf("live object %s/%s", namespace, name), nil +} + +func validateCandidateObject(path, fieldRoot string, object map[string]interface{}, candidates map[groupVersionKind]*candidateSchema) []error { + apiVersion, _ := object["apiVersion"].(string) + kind, _ := object["kind"].(string) + group, version, ok := strings.Cut(apiVersion, "/") + if !ok || group == "" || version == "" || kind == "" { + return []error{fmt.Errorf("%s: apiVersion and kind must identify a grouped resource", path)} + } + candidate, found := candidates[groupVersionKind{Group: group, Version: version, Kind: kind}] + if !found { + return []error{fmt.Errorf("%s: no proposed schema for %s %s", path, apiVersion, kind)} + } + return validateObject(path, fieldRoot, object, candidate) +} + +func validateObject(path, fieldRoot string, object map[string]interface{}, candidate *candidateSchema) []error { + var result []error + validator, _, err := apiservervalidation.NewSchemaValidator(candidate.internal) + if err != nil { + return []error{fmt.Errorf("%s: build OpenAPI validator: %w", path, err)} + } + if errs := apiservervalidation.ValidateCustomResource(field.NewPath(fieldRoot), object, validator); len(errs) > 0 { + result = append(result, fmt.Errorf("%s: proposed OpenAPI validation failed: %w", path, errs.ToAggregate())) + } + celValidator := cel.NewValidator(candidate.structural, true, perCallCELCostLimit) + if celValidator != nil { + objectPath := field.NewPath(fieldRoot) + createErrors, _ := celValidator.Validate(context.Background(), objectPath, candidate.structural, object, nil, runtimeCELCostBudget) + if len(createErrors) > 0 { + result = append(result, fmt.Errorf("%s: proposed CEL create validation failed: %w", path, createErrors.ToAggregate())) + } + updateErrors, _ := celValidator.Validate(context.Background(), objectPath, candidate.structural, object, object, runtimeCELCostBudget) + if len(updateErrors) > 0 { + result = append(result, fmt.Errorf("%s: proposed CEL unchanged-update validation failed: %w", path, updateErrors.ToAggregate())) + } + } + unknownFields := pruning.PruneWithOptions( + runtime.DeepCopyJSONValue(object), + candidate.structural, + true, + structuralschema.UnknownFieldPathOptions{TrackUnknownFieldPaths: true}, + ) + if len(unknownFields) > 0 { + result = append(result, fmt.Errorf("%s: proposed structural schema would prune declared fields: %s", path, strings.Join(unknownFields, ", "))) + } + return result +} + +func verifyServedSchemas(crdDirectory string, discovery io.Reader) error { + candidates, err := loadCandidates(crdDirectory) + if err != nil { + return err + } + var document map[string]interface{} + decoder := json.NewDecoder(discovery) + decoder.UseNumber() + if err := decoder.Decode(&document); err != nil { + return fmt.Errorf("decode served OpenAPI v3 document: %w", err) + } + for gvk, candidate := range candidates { + actual, err := schemaForGVK(document, gvk) + if err != nil { + return fmt.Errorf("served schema for %s/%s %s: %w", gvk.Group, gvk.Version, gvk.Kind, err) + } + expected, err := generatedSchema(candidate, gvk) + if err != nil { + return fmt.Errorf("generate proposed schema for %s/%s %s: %w", gvk.Group, gvk.Version, gvk.Kind, err) + } + normalizePublishedSchema(expected) + normalizePublishedSchema(actual) + if !reflect.DeepEqual(expected, actual) { + return fmt.Errorf("served OpenAPI semantics for %s/%s %s do not match the proposed CRD", gvk.Group, gvk.Version, gvk.Kind) + } + } + return nil +} + +func generatedSchema(candidate *candidateSchema, gvk groupVersionKind) (map[string]interface{}, error) { + openapi, err := builder.BuildOpenAPIV3(candidate.crd, candidate.version.Name, builder.Options{}) + if err != nil { + return nil, err + } + raw, err := json.Marshal(openapi) + if err != nil { + return nil, err + } + var document map[string]interface{} + decoder := json.NewDecoder(strings.NewReader(string(raw))) + decoder.UseNumber() + if err := decoder.Decode(&document); err != nil { + return nil, err + } + return schemaForGVK(document, gvk) +} + +func schemaForGVK(document map[string]interface{}, wanted groupVersionKind) (map[string]interface{}, error) { + components, ok := document["components"].(map[string]interface{}) + if !ok { + return nil, errors.New("document has no components") + } + schemas, ok := components["schemas"].(map[string]interface{}) + if !ok { + return nil, errors.New("document has no component schemas") + } + for _, value := range schemas { + schema, ok := value.(map[string]interface{}) + if !ok { + continue + } + gvks, ok := schema["x-kubernetes-group-version-kind"].([]interface{}) + if !ok { + continue + } + for _, value := range gvks { + gvk, ok := value.(map[string]interface{}) + if ok && gvk["group"] == wanted.Group && gvk["version"] == wanted.Version && gvk["kind"] == wanted.Kind { + return runtime.DeepCopyJSONValue(schema).(map[string]interface{}), nil + } + } + } + return nil, errors.New("matching x-kubernetes-group-version-kind was not published") +} + +func normalizePublishedSchema(schema map[string]interface{}) { + delete(schema, "x-kubernetes-group-version-kind") + if properties, ok := schema["properties"].(map[string]interface{}); ok { + delete(properties, "apiVersion") + delete(properties, "kind") + delete(properties, "metadata") + } + stripDescriptions(schema) +} + +func stripDescriptions(value interface{}) { + switch value := value.(type) { + case map[string]interface{}: + delete(value, "description") + for _, child := range value { + stripDescriptions(child) + } + case []interface{}: + for _, child := range value { + stripDescriptions(child) + } + } +} + +func loadCandidates(directory string) (map[groupVersionKind]*candidateSchema, error) { + paths, err := yamlPaths(directory) + if err != nil { + return nil, err + } + result := make(map[groupVersionKind]*candidateSchema) + for _, path := range paths { + raw, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var crd apiextensionsv1.CustomResourceDefinition + if err := yaml.Unmarshal(raw, &crd); err != nil { + return nil, fmt.Errorf("%s: decode CRD: %w", path, err) + } + for index := range crd.Spec.Versions { + version := &crd.Spec.Versions[index] + if !version.Served { + continue + } + if version.Schema == nil || version.Schema.OpenAPIV3Schema == nil { + return nil, fmt.Errorf("%s: served version %s has no OpenAPI schema", path, version.Name) + } + internal := &apiextensions.JSONSchemaProps{} + if err := apiextensionsv1.Convert_v1_JSONSchemaProps_To_apiextensions_JSONSchemaProps(version.Schema.OpenAPIV3Schema, internal, nil); err != nil { + return nil, fmt.Errorf("%s: convert schema for %s: %w", path, version.Name, err) + } + structural, err := structuralschema.NewStructural(internal) + if err != nil { + return nil, fmt.Errorf("%s: schema for %s is not structural: %w", path, version.Name, err) + } + gvk := groupVersionKind{Group: crd.Spec.Group, Version: version.Name, Kind: crd.Spec.Names.Kind} + if _, duplicate := result[gvk]; duplicate { + return nil, fmt.Errorf("%s: duplicate proposed schema for %s/%s %s", path, gvk.Group, gvk.Version, gvk.Kind) + } + result[gvk] = &candidateSchema{crd: &crd, version: version, internal: internal, structural: structural} + } + } + if len(result) == 0 { + return nil, fmt.Errorf("%s: no served CRD schemas found", directory) + } + return result, nil +} + +func decodeYAMLObject(path string) (map[string]interface{}, error) { + raw, err := os.ReadFile(path) + if err != nil { + return nil, err + } + jsonRaw, err := yaml.YAMLToJSON(raw) + if err != nil { + return nil, fmt.Errorf("%s: decode YAML: %w", path, err) + } + var object map[string]interface{} + if err := json.Unmarshal(jsonRaw, &object); err != nil { + return nil, fmt.Errorf("%s: decode object: %w", path, err) + } + return object, nil +} + +func yamlPaths(directory string) ([]string, error) { + entries, err := os.ReadDir(directory) + if err != nil { + return nil, err + } + paths := make([]string, 0, len(entries)) + for _, entry := range entries { + if entry.IsDir() { + continue + } + extension := strings.ToLower(filepath.Ext(entry.Name())) + if extension == ".yaml" || extension == ".yml" { + paths = append(paths, filepath.Join(directory, entry.Name())) + } + } + return paths, nil +} diff --git a/packages/cluster-operator/cmd/crd-preflight/main_test.go b/packages/cluster-operator/cmd/crd-preflight/main_test.go new file mode 100644 index 00000000..1c08fa28 --- /dev/null +++ b/packages/cluster-operator/cmd/crd-preflight/main_test.go @@ -0,0 +1,362 @@ +package main + +import ( + "fmt" + "os" + "path/filepath" + "strings" + "testing" +) + +const candidateCRD = `apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + name: widgets.cluster.t4.dev +spec: + group: cluster.t4.dev + scope: Namespaced + names: + plural: widgets + singular: widget + kind: Widget + listKind: WidgetList + versions: + - name: v1alpha1 + served: true + storage: true + subresources: + status: {} + schema: + openAPIV3Schema: + type: object + required: [spec] + properties: + apiVersion: + type: string + kind: + type: string + metadata: + type: object + spec: + type: object + required: [code] + x-kubernetes-validations: + - rule: self.code.startsWith('ok') + message: code must start with ok + properties: + code: + type: string + maxLength: 3 + status: + type: object + properties: + phase: + type: string + enum: [Ready] +` + +func TestValidateFixturesRejectsProposedSpecTighteningAndCEL(t *testing.T) { + for _, test := range []struct { + name string + code string + expectedError string + }{ + {name: "openapi maxLength", code: "okay", expectedError: "proposed OpenAPI validation failed"}, + {name: "CEL rule", code: "bad", expectedError: "proposed CEL create validation failed"}, + } { + t.Run(test.name, func(t *testing.T) { + crds, fixtures := writeCandidate(t, `apiVersion: cluster.t4.dev/v1alpha1 +kind: Widget +metadata: + name: legacy +spec: + code: `+test.code+"\n"+`status: + phase: Ready +`) + err := validateFixtures(crds, fixtures) + if err == nil { + t.Fatal("fixture incompatible with the proposed spec schema was accepted") + } + if !strings.Contains(err.Error(), test.expectedError) { + t.Fatalf("validation error %q does not identify %q", err, test.expectedError) + } + }) + } +} + +func TestValidateFixturesRejectsPersistedStatusAgainstProposedSchema(t *testing.T) { + crds, fixtures := writeCandidate(t, `apiVersion: cluster.t4.dev/v1alpha1 +kind: Widget +metadata: + name: legacy +spec: + code: ok +status: + phase: Legacy +`) + if err := validateFixtures(crds, fixtures); err == nil { + t.Fatal("persisted status incompatible with the proposed status schema was accepted") + } +} + +func TestValidateFixturesRejectsUnchangedLegacyValuesUnderTransitionCEL(t *testing.T) { + tests := []struct { + name string + fixture string + expectedPath string + candidate func(string) string + }{ + { + name: "spec transition rule", + expectedPath: "fixture.spec.code", + fixture: `apiVersion: cluster.t4.dev/v1alpha1 +kind: Widget +metadata: + name: legacy +spec: + code: bad +status: + phase: Ready +`, + candidate: func(crd string) string { + crd = strings.Replace(crd, "rule: self.code.startsWith('ok')", `rule: "true"`, 1) + return strings.Replace(crd, " maxLength: 3", " maxLength: 3\n x-kubernetes-validations:\n - rule: oldSelf.startsWith('ok')", 1) + }, + }, + { + name: "status transition rule", + expectedPath: "fixture.status.phase", + fixture: `apiVersion: cluster.t4.dev/v1alpha1 +kind: Widget +metadata: + name: legacy +spec: + code: ok +status: + phase: Pending +`, + candidate: func(crd string) string { + return strings.Replace(crd, " enum: [Ready]", " enum: [Ready, Pending]\n x-kubernetes-validations:\n - rule: oldSelf == 'Ready'", 1) + }, + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + crds, fixtures := writeCandidate(t, test.fixture) + if err := os.WriteFile(filepath.Join(crds, "widget.yaml"), []byte(test.candidate(candidateCRD)), 0o644); err != nil { + t.Fatal(err) + } + err := validateFixtures(crds, fixtures) + if err == nil { + t.Fatal("unchanged persisted value blocked by transition CEL was accepted") + } + if !strings.Contains(err.Error(), "proposed CEL unchanged-update validation failed") { + t.Fatalf("validation error does not identify unchanged-update semantics: %v", err) + } + if !strings.Contains(err.Error(), test.expectedPath) { + t.Fatalf("validation error %q does not identify field %q", err, test.expectedPath) + } + }) + } +} + +func TestValidateObjectsRejectsLiveDataOmittedFromFixtures(t *testing.T) { + tests := []struct { + name string + candidate func(string) string + fixtureCode string + fixturePhase string + liveCode string + livePhase string + liveExtra string + expectedError string + }{ + { + name: "spec OpenAPI narrowing", + candidate: func(crd string) string { + return strings.Replace(crd, " maxLength: 3", " maxLength: 3\n enum: [ok1]", 1) + }, + fixtureCode: "ok1", fixturePhase: "Ready", liveCode: "ok2", livePhase: "Ready", + expectedError: "proposed OpenAPI validation failed", + }, + { + name: "status OpenAPI narrowing", + candidate: func(crd string) string { return crd }, + fixtureCode: "ok", fixturePhase: "Ready", liveCode: "ok", livePhase: "Legacy", + expectedError: "proposed OpenAPI validation failed", + }, + { + name: "spec CEL create semantics", + candidate: func(crd string) string { return crd }, + fixtureCode: "ok", fixturePhase: "Ready", liveCode: "bad", livePhase: "Ready", + expectedError: "proposed CEL create validation failed", + }, + { + name: "status CEL create semantics", + candidate: func(crd string) string { + return strings.Replace(crd, " phase:\n type: string\n enum: [Ready]", " phase:\n type: string\n enum: [Ready, Legacy]\n x-kubernetes-validations:\n - rule: self == 'Ready'", 1) + }, + fixtureCode: "ok", fixturePhase: "Ready", liveCode: "ok", livePhase: "Legacy", + expectedError: "proposed CEL create validation failed", + }, + { + name: "spec CEL unchanged-update semantics", + candidate: func(crd string) string { + crd = strings.Replace(crd, "rule: self.code.startsWith('ok')", `rule: "true"`, 1) + return strings.Replace(crd, " maxLength: 3", " maxLength: 3\n x-kubernetes-validations:\n - rule: oldSelf.startsWith('ok')", 1) + }, + fixtureCode: "ok", fixturePhase: "Ready", liveCode: "bad", livePhase: "Ready", + expectedError: "proposed CEL unchanged-update validation failed", + }, + { + name: "status CEL unchanged-update semantics", + candidate: func(crd string) string { + return strings.Replace(crd, " enum: [Ready]", " enum: [Ready, Legacy]\n x-kubernetes-validations:\n - rule: oldSelf == 'Ready'", 1) + }, + fixtureCode: "ok", fixturePhase: "Ready", liveCode: "ok", livePhase: "Legacy", + expectedError: "proposed CEL unchanged-update validation failed", + }, + { + name: "spec pruning", + candidate: func(crd string) string { return crd }, + fixtureCode: "ok", fixturePhase: "Ready", liveCode: "ok", livePhase: "Ready", liveExtra: `,"removedSpec":"legacy"`, + expectedError: "proposed structural schema would prune declared fields", + }, + { + name: "status pruning", + candidate: func(crd string) string { return crd }, + fixtureCode: "ok", fixturePhase: "Ready", liveCode: "ok", livePhase: "Ready", liveExtra: `,"removedStatus":"legacy"`, + expectedError: "proposed structural schema would prune declared fields", + }, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + fixture := fmt.Sprintf("apiVersion: cluster.t4.dev/v1alpha1\nkind: Widget\nmetadata:\n name: curated\n namespace: tenant-a\nspec:\n code: %s\nstatus:\n phase: %s\n", test.fixtureCode, test.fixturePhase) + crds, fixtures := writeCandidate(t, fixture) + if err := os.WriteFile(filepath.Join(crds, "widget.yaml"), []byte(test.candidate(candidateCRD)), 0o644); err != nil { + t.Fatal(err) + } + if err := validateFixtures(crds, fixtures); err != nil { + t.Fatalf("curated fixture should not expose the live-only incompatibility: %v", err) + } + live := fmt.Sprintf(`{"apiVersion":"v1","kind":"WidgetList","items":[{"apiVersion":"cluster.t4.dev/v1alpha1","kind":"Widget","metadata":{"name":"live","namespace":"tenant-b"},"spec":{"code":%q%s},"status":{"phase":%q%s}}]}`, + test.liveCode, map[bool]string{true: test.liveExtra, false: ""}[test.name == "spec pruning"], test.livePhase, map[bool]string{true: test.liveExtra, false: ""}[test.name == "status pruning"]) + err := validateObjects(crds, strings.NewReader(live)) + if err == nil { + t.Fatal("live object incompatible with the proposed schema was accepted") + } + if !strings.Contains(err.Error(), "tenant-b/live") || !strings.Contains(err.Error(), test.expectedError) { + t.Fatalf("validation error %q does not identify live object and %q", err, test.expectedError) + } + }) + } +} + +func TestValidateObjectsAcceptsKubernetesJSONNumbers(t *testing.T) { + crds, _ := writeCandidate(t, `apiVersion: cluster.t4.dev/v1alpha1 +kind: Widget +metadata: + name: curated +spec: + code: ok +`) + candidate := strings.Replace(candidateCRD, " required: [code]", " required: [code, count]", 1) + candidate = strings.Replace(candidate, " maxLength: 3", " maxLength: 3\n count:\n type: integer\n x-kubernetes-validations:\n - rule: self == 9223372036854775807", 1) + if err := os.WriteFile(filepath.Join(crds, "widget.yaml"), []byte(candidate), 0o644); err != nil { + t.Fatal(err) + } + live := strings.NewReader(`{"apiVersion":"v1","kind":"WidgetList","items":[{"apiVersion":"cluster.t4.dev/v1alpha1","kind":"Widget","metadata":{"name":"live","namespace":"tenant-b"},"spec":{"code":"ok","count":9223372036854775807},"status":{"phase":"Ready"}}]}`) + if err := validateObjects(crds, live); err != nil { + t.Fatalf("ordinary Kubernetes JSON integer was rejected: %v", err) + } +} + +func TestVerifyServedSchemasRejectsRetainedEstablishedWithStaleSchema(t *testing.T) { + crds, _ := writeCandidate(t, `apiVersion: cluster.t4.dev/v1alpha1 +kind: Widget +metadata: + name: legacy +spec: + code: ok +`) + staleDiscovery := strings.NewReader(`{ + "openapi": "3.0.0", + "components": {"schemas": { + "cluster.t4.dev.v1alpha1.Widget": { + "type": "object", + "required": ["spec"], + "properties": { + "apiVersion": {"type": "string"}, + "kind": {"type": "string"}, + "metadata": {"type": "object"}, + "spec": { + "type": "object", + "required": ["code"], + "x-kubernetes-validations": [{"rule": "self.code.startsWith('ok')", "message": "code must start with ok"}], + "properties": {"code": {"type": "string", "maxLength": 8}} + }, + "status": {"type": "object", "properties": {"phase": {"type": "string", "enum": ["Ready"]}}} + }, + "x-kubernetes-group-version-kind": [{"group":"cluster.t4.dev","version":"v1alpha1","kind":"Widget"}] + } + }} +}`) + if err := verifyServedSchemas(crds, staleDiscovery); err == nil { + t.Fatal("stale served OpenAPI schema was accepted after Established") + } +} + +func TestVerifyServedSchemasAcceptsExactProposedSemantics(t *testing.T) { + crds, _ := writeCandidate(t, `apiVersion: cluster.t4.dev/v1alpha1 +kind: Widget +metadata: + name: legacy +spec: + code: ok +`) + discovery := strings.NewReader(`{ + "openapi": "3.0.0", + "components": {"schemas": { + "cluster.t4.dev.v1alpha1.Widget": { + "type": "object", + "required": ["spec"], + "properties": { + "apiVersion": {"type": "string"}, + "kind": {"type": "string"}, + "metadata": {"type": "object"}, + "spec": { + "type": "object", + "required": ["code"], + "x-kubernetes-validations": [{"rule": "self.code.startsWith('ok')", "message": "code must start with ok"}], + "properties": {"code": {"type": "string", "maxLength": 3}} + }, + "status": {"type": "object", "properties": {"phase": {"type": "string", "enum": ["Ready"]}}} + }, + "x-kubernetes-group-version-kind": [{"group":"cluster.t4.dev","version":"v1alpha1","kind":"Widget"}] + } + }} +}`) + if err := verifyServedSchemas(crds, discovery); err != nil { + t.Fatalf("exact proposed schema rejected: %v", err) + } +} + +func writeCandidate(t *testing.T, fixture string) (string, string) { + t.Helper() + root := t.TempDir() + crds := filepath.Join(root, "crds") + fixtures := filepath.Join(root, "fixtures") + for _, directory := range []string{crds, fixtures} { + if err := os.Mkdir(directory, 0o755); err != nil { + t.Fatal(err) + } + } + if err := os.WriteFile(filepath.Join(crds, "widget.yaml"), []byte(candidateCRD), 0o644); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(fixtures, "widget.yaml"), []byte(fixture), 0o644); err != nil { + t.Fatal(err) + } + return crds, fixtures +} + diff --git a/packages/cluster-operator/go.mod b/packages/cluster-operator/go.mod index c85639c9..f4504858 100644 --- a/packages/cluster-operator/go.mod +++ b/packages/cluster-operator/go.mod @@ -14,23 +14,32 @@ require ( ) require ( + cel.dev/expr v0.19.1 // indirect + github.com/antlr4-go/antlr/v4 v4.13.0 // indirect github.com/beorn7/perks v1.0.1 // indirect + github.com/blang/semver/v4 v4.0.0 // indirect + github.com/cenkalti/backoff/v4 v4.3.0 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/davecgh/go-spew v1.1.1 // indirect github.com/emicklei/go-restful/v3 v3.11.0 // indirect github.com/evanphx/json-patch/v5 v5.9.11 // indirect + github.com/felixge/httpsnoop v1.0.4 // indirect github.com/fsnotify/fsnotify v1.7.0 // indirect github.com/fxamacker/cbor/v2 v2.7.0 // indirect github.com/go-logr/logr v1.4.2 // indirect + github.com/go-logr/stdr v1.2.2 // indirect github.com/go-logr/zapr v1.3.0 // indirect github.com/go-openapi/jsonpointer v0.21.0 // indirect github.com/go-openapi/jsonreference v0.20.2 // indirect github.com/go-openapi/swag v0.23.0 // indirect github.com/gogo/protobuf v1.3.2 // indirect github.com/google/btree v1.1.3 // indirect + github.com/google/cel-go v0.23.2 // indirect github.com/google/gnostic-models v0.6.9 // indirect github.com/google/go-cmp v0.7.0 // indirect github.com/google/uuid v1.6.0 // indirect + github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0 // indirect + github.com/inconshreveable/mousetrap v1.1.0 // indirect github.com/josharian/intern v1.0.0 // indirect github.com/json-iterator/go v1.1.12 // indirect github.com/mailru/easyjson v0.7.7 // indirect @@ -41,10 +50,22 @@ require ( github.com/prometheus/client_model v0.6.1 // indirect github.com/prometheus/common v0.62.0 // indirect github.com/prometheus/procfs v0.15.1 // indirect + github.com/spf13/cobra v1.8.1 // indirect github.com/spf13/pflag v1.0.5 // indirect + github.com/stoewer/go-strcase v1.3.0 // indirect github.com/x448/float16 v0.8.4 // indirect + go.opentelemetry.io/auto/sdk v1.1.0 // indirect + go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.58.0 // indirect + go.opentelemetry.io/otel v1.33.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.33.0 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0 // indirect + go.opentelemetry.io/otel/metric v1.33.0 // indirect + go.opentelemetry.io/otel/sdk v1.33.0 // indirect + go.opentelemetry.io/otel/trace v1.33.0 // indirect + go.opentelemetry.io/proto/otlp v1.4.0 // indirect go.uber.org/multierr v1.11.0 // indirect go.uber.org/zap v1.27.0 // indirect + golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect golang.org/x/net v0.38.0 // indirect golang.org/x/oauth2 v0.27.0 // indirect golang.org/x/sync v0.12.0 // indirect @@ -53,12 +74,19 @@ require ( golang.org/x/text v0.23.0 // indirect golang.org/x/time v0.9.0 // indirect gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20241209162323-e6fa225c2576 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576 // indirect + google.golang.org/grpc v1.68.1 // indirect google.golang.org/protobuf v1.36.5 // indirect gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect gopkg.in/inf.v0 v0.9.1 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect + k8s.io/apiserver v0.33.2 // indirect + k8s.io/component-base v0.33.2 // indirect k8s.io/klog/v2 v2.130.1 // indirect k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff // indirect k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 // indirect + sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 // indirect sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect sigs.k8s.io/randfill v1.0.0 // indirect sigs.k8s.io/structured-merge-diff/v4 v4.6.0 // indirect diff --git a/packages/cluster-operator/go.sum b/packages/cluster-operator/go.sum index e291ac10..6883ecd7 100644 --- a/packages/cluster-operator/go.sum +++ b/packages/cluster-operator/go.sum @@ -1,9 +1,16 @@ +cel.dev/expr v0.19.1 h1:NciYrtDRIR0lNCnH1LFJegdjspNx9fI59O7TWcua/W4= +cel.dev/expr v0.19.1/go.mod h1:MrpN08Q+lEBs+bGYdLxxHkZoUSsCp0nSKTs0nTymJgw= +github.com/antlr4-go/antlr/v4 v4.13.0 h1:lxCg3LAv+EUK6t1i0y1V6/SLeUi0eKEKdhQAlS8TVTI= +github.com/antlr4-go/antlr/v4 v4.13.0/go.mod h1:pfChB/xh/Unjila75QW7+VU4TSnWnnk9UTnmpPaOR2g= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM= github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ= +github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8= +github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o= github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= @@ -14,12 +21,17 @@ github.com/evanphx/json-patch v0.5.2 h1:xVCHIVMUu1wtM/VkR9jVZ45N3FhZfYMMYGorLCR8 github.com/evanphx/json-patch v0.5.2/go.mod h1:ZWS5hhDbVDyob71nXKNL0+PWn6ToqBHMikGIFbs31qQ= github.com/evanphx/json-patch/v5 v5.9.11 h1:/8HVnzMq13/3x9TPvjG08wUGqBTmZBsCWzjTM0wiaDU= github.com/evanphx/json-patch/v5 v5.9.11/go.mod h1:3j+LviiESTElxA4p3EMKAB9HXj3/XEtnUf6OZxqIQTM= +github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= +github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U= github.com/fsnotify/fsnotify v1.7.0 h1:8JEhPFa5W2WU7YfeZzPNqzMP6Lwt7L2715Ggo0nosvA= github.com/fsnotify/fsnotify v1.7.0/go.mod h1:40Bi/Hjc2AVfZrqy+aj+yEI+/bRxZnMJyTJwOpGvigM= github.com/fxamacker/cbor/v2 v2.7.0 h1:iM5WgngdRBanHcxugY4JySA0nk1wZorNOpTgCMedv5E= github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY= github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ= github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg= github.com/go-openapi/jsonpointer v0.19.6/go.mod h1:osyAmYz/mB/C3I+WsTTSgw1ONzaLJoLCyoi6/zppojs= @@ -36,6 +48,8 @@ github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q= github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q= github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg= github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4= +github.com/google/cel-go v0.23.2 h1:UdEe3CvQh3Nv+E/j9r1Y//WO0K0cSyD7/y0bzyLIMI4= +github.com/google/cel-go v0.23.2/go.mod h1:52Pb6QsDbC5kvgxvZhiL9QX1oZEkcUF/ZqaPx1J5Wwo= github.com/google/gnostic-models v0.6.9 h1:MU/8wDLif2qCXZmzncUQ/BOfxWfthHi63KqpoNbWqVw= github.com/google/gnostic-models v0.6.9/go.mod h1:CiWsm0s6BSQd1hRn8/QmxqB6BesYcbSZxsz9b0KuDBw= github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= @@ -48,6 +62,11 @@ github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db h1:097atOisP2aRj7vFgY github.com/google/pprof v0.0.0-20241029153458-d1b30febd7db/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/grpc-ecosystem/grpc-gateway v1.16.0 h1:gmcG1KaJ57LophUzW0Hy8NmPhnMZb4M0+kPpLofRdBo= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0 h1:TmHmbvxPmaegwhDubVz0lICL0J5Ka2vwTzhoePEXsGE= +github.com/grpc-ecosystem/grpc-gateway/v2 v2.24.0/go.mod h1:qztMSjm835F2bXf+5HKAPIS5qsmQDqZna/PgVt4rWtI= +github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= +github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY= github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= @@ -92,8 +111,13 @@ github.com/prometheus/procfs v0.15.1 h1:YagwOFzUgYfKKHX6Dr+sHT7km/hxC76UB0leargg github.com/prometheus/procfs v0.15.1/go.mod h1:fB45yRUv8NstnjriLhBQLuOUt+WW4BsoGhij/e3PBqk= github.com/rogpeppe/go-internal v1.13.1 h1:KvO1DLK/DRN07sQ1LQKScxyZJuNnedQ5/wKSR38lUII= github.com/rogpeppe/go-internal v1.13.1/go.mod h1:uMEvuHeurkdAXX61udpOXGD/AzZDWNMNyH2VO9fmH0o= +github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= +github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM= +github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= +github.com/stoewer/go-strcase v1.3.0 h1:g0eASXYtp+yvN9fK8sH94oCIk0fau9uV1/ZdJ0AVEzs= +github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw= github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo= @@ -109,6 +133,24 @@ github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM= github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg= github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA= +go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.58.0 h1:yd02MEjBdJkG3uabWP9apV+OuWRIXGDuJEUJbOHmCFU= +go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.58.0/go.mod h1:umTcuxiv1n/s/S6/c2AT/g2CQ7u5C59sHDNmfSwgz7Q= +go.opentelemetry.io/otel v1.33.0 h1:/FerN9bax5LoK51X/sI0SVYrjSE0/yUL7DpxW4K3FWw= +go.opentelemetry.io/otel v1.33.0/go.mod h1:SUUkR6csvUQl+yjReHu5uM3EtVV7MBm5FHKRlNx4I8I= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.33.0 h1:Vh5HayB/0HHfOQA7Ctx69E/Y/DcQSMPpKANYVMQ7fBA= +go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.33.0/go.mod h1:cpgtDBaqD/6ok/UG0jT15/uKjAY8mRA53diogHBg3UI= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0 h1:5pojmb1U1AogINhN3SurB+zm/nIcusopeBNp42f45QM= +go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0/go.mod h1:57gTHJSE5S1tqg+EKsLPlTWhpHMsWlVmer+LA926XiA= +go.opentelemetry.io/otel/metric v1.33.0 h1:r+JOocAyeRVXD8lZpjdQjzMadVZp2M4WmQ+5WtEnklQ= +go.opentelemetry.io/otel/metric v1.33.0/go.mod h1:L9+Fyctbp6HFTddIxClbQkjtubW6O9QS3Ann/M82u6M= +go.opentelemetry.io/otel/sdk v1.33.0 h1:iax7M131HuAm9QkZotNHEfstof92xM+N8sr3uHXc2IM= +go.opentelemetry.io/otel/sdk v1.33.0/go.mod h1:A1Q5oi7/9XaMlIWzPSxLRWOI8nG3FnzHJNbiENQuihM= +go.opentelemetry.io/otel/trace v1.33.0 h1:cCJuF7LRjUFso9LPnEAHJDB2pqzp+hbO8eu1qqW2d/s= +go.opentelemetry.io/otel/trace v1.33.0/go.mod h1:uIcdVUZMpTAmz0tI1z04GoVSezK37CbGV4fr1f2nBck= +go.opentelemetry.io/proto/otlp v1.4.0 h1:TA9WRvW6zMwP+Ssb6fLoUIuirti1gGbP28GcKG1jgeg= +go.opentelemetry.io/proto/otlp v1.4.0/go.mod h1:PPBWZIP98o2ElSqI35IHfu7hIhSwvc5N38Jw8pXuGFY= go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0= @@ -118,6 +160,8 @@ go.uber.org/zap v1.27.0/go.mod h1:GB2qFLM7cTU87MWRP2mPIjqfIDnGu+VIO4V/SdhGo2E= golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= +golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 h1:2dVuKD2vS7b0QIHQbpyTISPd0LeHDbnYEryqj5Q1ug8= +golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56/go.mod h1:M4RDyNAINzryxdtnbRXRL/OHtkFuWGRjvuhBJpk2IlY= golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= @@ -158,6 +202,13 @@ golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8T golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gomodules.xyz/jsonpatch/v2 v2.4.0 h1:Ci3iUJyx9UeRx7CeFN8ARgGbkESwJK+KB9lLcWxY/Zw= gomodules.xyz/jsonpatch/v2 v2.4.0/go.mod h1:AH3dM2RI6uoBZxn3LVrfvJ3E0/9dG4cSrbuBJT4moAY= +google.golang.org/genproto v0.0.0-20240123012728-ef4313101c80 h1:KAeGQVN3M9nD0/bQXnr/ClcEMJ968gUXJQ9pwfSynuQ= +google.golang.org/genproto/googleapis/api v0.0.0-20241209162323-e6fa225c2576 h1:CkkIfIt50+lT6NHAVoRYEyAvQGFM7xEwXUUywFvEb3Q= +google.golang.org/genproto/googleapis/api v0.0.0-20241209162323-e6fa225c2576/go.mod h1:1R3kvZ1dtP3+4p4d3G8uJ8rFk/fWlScl38vanWACI08= +google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576 h1:8ZmaLZE4XWrtU3MyClkYqqtl6Oegr3235h7jxsDyqCY= +google.golang.org/genproto/googleapis/rpc v0.0.0-20241209162323-e6fa225c2576/go.mod h1:5uTbfoYQed2U9p3KIj2/Zzm02PYhndfdmML0qC3q3FU= +google.golang.org/grpc v1.68.1 h1:oI5oTa11+ng8r8XMMN7jAOmWfPZWbYpCFaMUTACxkM0= +google.golang.org/grpc v1.68.1/go.mod h1:+q1XYFJjShcqn0QZHvCyeR4CXPA+llXIeUIfIe00waw= google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM= google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -176,14 +227,20 @@ k8s.io/apiextensions-apiserver v0.33.2 h1:6gnkIbngnaUflR3XwE1mCefN3YS8yTD631JXQh k8s.io/apiextensions-apiserver v0.33.2/go.mod h1:IvVanieYsEHJImTKXGP6XCOjTwv2LUMos0YWc9O+QP8= k8s.io/apimachinery v0.33.2 h1:IHFVhqg59mb8PJWTLi8m1mAoepkUNYmptHsV+Z1m5jY= k8s.io/apimachinery v0.33.2/go.mod h1:BHW0YOu7n22fFv/JkYOEfkUYNRN0fj0BlvMFWA7b+SM= +k8s.io/apiserver v0.33.2 h1:KGTRbxn2wJagJowo29kKBp4TchpO1DRO3g+dB/KOJN4= +k8s.io/apiserver v0.33.2/go.mod h1:9qday04wEAMLPWWo9AwqCZSiIn3OYSZacDyu/AcoM/M= k8s.io/client-go v0.33.2 h1:z8CIcc0P581x/J1ZYf4CNzRKxRvQAwoAolYPbtQes+E= k8s.io/client-go v0.33.2/go.mod h1:9mCgT4wROvL948w6f6ArJNb7yQd7QsvqavDeZHvNmHo= +k8s.io/component-base v0.33.2 h1:sCCsn9s/dG3ZrQTX/Us0/Sx2R0G5kwa0wbZFYoVp/+0= +k8s.io/component-base v0.33.2/go.mod h1:/41uw9wKzuelhN+u+/C59ixxf4tYQKW7p32ddkYNe2k= k8s.io/klog/v2 v2.130.1 h1:n9Xl7H1Xvksem4KFG4PYbdQCQxqc/tTUyrgXaOhHSzk= k8s.io/klog/v2 v2.130.1/go.mod h1:3Jpz1GvMt720eyJH1ckRHK1EDfpxISzJ7I9OYgaDtPE= k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff h1:/usPimJzUKKu+m+TE36gUyGcf03XZEP0ZIKgKj35LS4= k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff/go.mod h1:5jIi+8yX4RIb8wk3XwBo5Pq2ccx4FP10ohkbSKCZoK8= k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 h1:M3sRQVHv7vB20Xc2ybTt7ODCeFj6JSWYFzOFnYeS6Ro= k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738/go.mod h1:OLgZIPagt7ERELqWJFomSt595RzquPNLL48iOWgYOg0= +sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2 h1:jpcvIRr3GLoUoEKRkHKSmGjxb6lWwrBlJsXc+eUYQHM= +sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.31.2/go.mod h1:Ve9uj1L+deCXFrPOk1LpFXqTg7LCFzFso6PA48q/XZw= sigs.k8s.io/controller-runtime v0.21.0 h1:CYfjpEuicjUecRk+KAeyYh+ouUBn4llGyDYytIGcJS8= sigs.k8s.io/controller-runtime v0.21.0/go.mod h1:OSg14+F65eWqIu4DceX7k/+QRAbTTvxeQSNSOQpukWM= sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 h1:/Rv+M11QRah1itp8VhT6HoVx1Ray9eB4DBr+K+/sCJ8= diff --git a/scripts/cluster-ci/cluster-ci-contract.test.mjs b/scripts/cluster-ci/cluster-ci-contract.test.mjs index 4c240a34..ccbf21b5 100644 --- a/scripts/cluster-ci/cluster-ci-contract.test.mjs +++ b/scripts/cluster-ci/cluster-ci-contract.test.mjs @@ -309,9 +309,9 @@ test("Woodpecker keeps upstream gates and serializes bounded cluster publication ); assert.ok(steps["cluster-ci-contracts"].commands.includes("pnpm test:cluster:ci")); assert.deepEqual(steps["cluster-operator-tests"].commands, [ - "GOMAXPROCS=1 GOFLAGS=-p=1 go test ./api/... ./controllers/... ./cmd/...", + "GOMAXPROCS=1 GOFLAGS='-mod=readonly -p=1' go test ./api/... ./controllers/... ./cmd/...", "mkdir -p ../../artifacts/cluster-proof", - "CGO_ENABLED=0 GOMAXPROCS=1 GOFLAGS=-p=1 go test -c ./charttests -o ../../artifacts/cluster-proof/chart-contract.test", + "CGO_ENABLED=0 GOMAXPROCS=1 GOFLAGS='-mod=readonly -p=1' go test -c ./charttests -o ../../artifacts/cluster-proof/chart-contract.test", ]); assert.equal( steps["cluster-operator-tests"].backend_options.kubernetes.resources.limits.memory, diff --git a/scripts/cluster-ci/crd-lifecycle.sh b/scripts/cluster-ci/crd-lifecycle.sh new file mode 100755 index 00000000..d698c7b3 --- /dev/null +++ b/scripts/cluster-ci/crd-lifecycle.sh @@ -0,0 +1,139 @@ +#!/bin/sh +set -eu + +usage() { + cat >&2 <<'EOF' +usage: crd-lifecycle.sh install|upgrade -- helm install|upgrade ... --skip-crds + +Environment: + KUBECTL kubectl executable (default: kubectl) + T4_CRD_DIRECTORY reviewed CRD directory + T4_COMPAT_DIRECTORY old-object compatibility fixture directory + T4_VALIDATION_NAMESPACE existing namespace used for server dry-runs (default: default) + T4_CRD_VALIDATOR proposed-schema validator executable (default: Go validator) + T4_DISCOVERY_OBSERVATIONS consecutive matching OpenAPI observations required (default: 3) +EOF + exit 64 +} + +[ "$#" -ge 4 ] || usage +mode=$1 +shift +case "$mode" in + install|upgrade) ;; + *) usage ;; +esac +[ "${1:-}" = "--" ] || usage +shift +[ "${1:-}" = "helm" ] || usage +[ "${2:-}" = "$mode" ] || usage + +has_skip_crds=false +for argument in "$@"; do + case "$argument" in + --skip-crds) has_skip_crds=true ;; + --force|--force=*|--force-conflicts|replace) + echo "force replacement is prohibited by the T4 CRD lifecycle" >&2 + exit 64 + ;; + esac +done +[ "$has_skip_crds" = true ] || { + echo "the workload command must include --skip-crds" >&2 + exit 64 +} + +repo_root=$(CDPATH= cd -- "$(dirname -- "$0")/../.." && pwd) +kubectl=${KUBECTL:-kubectl} +crd_directory=${T4_CRD_DIRECTORY:-$repo_root/deploy/charts/t4-cluster/crds} +compat_directory=${T4_COMPAT_DIRECTORY:-$repo_root/packages/cluster-operator/api/v1alpha1/testdata/compat} +validation_namespace=${T4_VALIDATION_NAMESPACE:-default} +field_manager=t4-crd-lifecycle +crds="crd/t4clusterhosts.cluster.t4.dev crd/t4workspaces.cluster.t4.dev crd/t4sessions.cluster.t4.dev" +live_resources="t4clusterhosts.cluster.t4.dev t4workspaces.cluster.t4.dev t4sessions.cluster.t4.dev" +discovery_observations=${T4_DISCOVERY_OBSERVATIONS:-3} + +case "$discovery_observations" in + ''|*[!0-9]*|0) + echo "T4_DISCOVERY_OBSERVATIONS must be a positive integer" >&2 + exit 64 + ;; +esac + +run_validator() { + if [ -n "${T4_CRD_VALIDATOR:-}" ]; then + "$T4_CRD_VALIDATOR" "$@" + else + (cd "$repo_root/packages/cluster-operator" && go run ./cmd/crd-preflight "$@") + fi +} + +# Validate the persisted compatibility instances, including status and CEL, +# directly against the proposed structural schemas. This is intentionally the +# first operation: current-cluster admission cannot prove candidate compatibility. +run_validator fixtures "$crd_directory" "$compat_directory" + +# Curated fixtures cannot prove that every value currently persisted in the +# cluster remains valid. Enumerate only the three namespaced T4 resources and +# run each complete live object, including status, through the same proposed- +# schema engine before a CRD or workload can be mutated. A denied or malformed +# list fails closed under set -e. +live_objects=$(mktemp) +trap 'rm -f "$live_objects"' EXIT +trap 'exit 129' HUP +trap 'exit 130' INT +trap 'exit 143' TERM +for resource in $live_resources; do + installed_crd=$("$kubectl" get "crd/$resource" --ignore-not-found -o name) + if [ -z "$installed_crd" ]; then + continue + fi + "$kubectl" get "$resource" --all-namespaces -o json >"$live_objects" + run_validator objects "$crd_directory" <"$live_objects" +done +rm -f "$live_objects" +trap - EXIT HUP INT TERM + +# Every kubectl operation before the first non-dry-run apply is read-only. +"$kubectl" apply --server-side --dry-run=server --validate=strict \ + --field-manager="$field_manager" -f "$crd_directory" >/dev/null + +"$kubectl" apply --server-side --validate=strict \ + --field-manager="$field_manager" -f "$crd_directory" +# Discovery and admission must converge before compatibility validation or any +# workload rollout uses the new schema. +# shellcheck disable=SC2086 # The fixed CRD words are intentional argv entries. +"$kubectl" wait --for=condition=Established --timeout=120s $crds + +# Established can remain True across an update. Require several independently +# fetched discovery documents to expose exactly the candidate OpenAPI semantics +# before trusting admission or starting a workload rollout. +openapi_document=$(mktemp) +trap 'rm -f "$openapi_document"' EXIT +trap 'exit 129' HUP +trap 'exit 130' INT +trap 'exit 143' TERM +observation=0 +while [ "$observation" -lt "$discovery_observations" ]; do + "$kubectl" get --raw /openapi/v3/apis/cluster.t4.dev/v1alpha1 >"$openapi_document" + run_validator served "$crd_directory" <"$openapi_document" + observation=$((observation + 1)) +done + +"$kubectl" apply --dry-run=server --validate=strict \ + --namespace "$validation_namespace" -f "$compat_directory" >/dev/null + +for crd in $crds; do + stored_versions=$("$kubectl" get "$crd" -o 'jsonpath={.status.storedVersions[*]}') + if [ "$stored_versions" != v1alpha1 ]; then + echo "$crd status.storedVersions is '$stored_versions'; expected exactly 'v1alpha1'" >&2 + exit 65 + fi +done + +# Helm is deliberately last and must not manage CRDs. If any earlier gate fails, +# the existing controller, server, session workloads, and custom resources are +# untouched by this runner. +rm -f "$openapi_document" +trap - EXIT HUP INT TERM +exec "$@" diff --git a/scripts/cluster-ci/crd-lifecycle.test.mjs b/scripts/cluster-ci/crd-lifecycle.test.mjs new file mode 100644 index 00000000..543f7775 --- /dev/null +++ b/scripts/cluster-ci/crd-lifecycle.test.mjs @@ -0,0 +1,311 @@ +import assert from "node:assert/strict"; +import { chmod, mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join, resolve } from "node:path"; +import { spawn } from "node:child_process"; +import test from "node:test"; + +const repoRoot = resolve(import.meta.dirname, "../.."); +const lifecycle = resolve(import.meta.dirname, "crd-lifecycle.sh"); + +async function fixture() { + const root = await mkdtemp(join(tmpdir(), "t4-crd-lifecycle-")); + const bin = join(root, "bin"); + const log = join(root, "commands.log"); + await mkdir(bin); + await writeFile( + join(bin, "kubectl"), + `#!/bin/sh +set -eu +printf 'kubectl' >>"$COMMAND_LOG" +for argument in "$@"; do printf '\\t%s' "$argument" >>"$COMMAND_LOG"; done +printf '\\n' >>"$COMMAND_LOG" +for argument in "$@"; do + if [ "$argument" = "--dry-run=server" ] && [ "\${FAIL_DRY_RUN:-0}" = 1 ]; then + exit 42 + fi +done +if [ "\${1:-}" = get ]; then + case "\${2:-}" in + crd/*) + for argument in "$@"; do + if [ "$argument" = "--ignore-not-found" ]; then + if [ "\${FAIL_CRD_READ:-}" = "\${2:-}" ]; then exit 45; fi + if [ "\${MISSING_LIVE_CRDS:-0}" = 1 ]; then exit 0; fi + printf '%s' "\${2:-}" + exit 0 + fi + done + ;; + esac + case "\${2:-}" in + t4clusterhosts.cluster.t4.dev|t4workspaces.cluster.t4.dev|t4sessions.cluster.t4.dev) + if [ "\${FAIL_LIVE_LIST:-}" = "\${2:-}" ]; then exit 44; fi + if [ "\${TERMINATE_DURING_LIVE_READ:-}" = "\${2:-}" ]; then + kill -TERM "$PPID" + sleep 1 + fi + printf '%s' '{"apiVersion":"v1","kind":"List","items":[]}' + exit 0 + ;; + esac + printf '%s' "\${STORED_VERSIONS:-v1alpha1}" +fi +`, + ); + await writeFile( + join(bin, "crd-preflight"), + `#!/bin/sh +set -eu +printf 'validator' >>"$COMMAND_LOG" +for argument in "$@"; do printf '\\t%s' "$argument" >>"$COMMAND_LOG"; done +printf '\\n' >>"$COMMAND_LOG" +case "\${1:-}:\${FAIL_PROPOSED_VALIDATION:-}" in + fixtures:spec|fixtures:status|objects:live|served:stale) exit 43 ;; +esac +cat >/dev/null +`, + ); + await writeFile( + join(bin, "helm"), + `#!/bin/sh +set -eu +printf 'helm' >>"$COMMAND_LOG" +for argument in "$@"; do printf '\\t%s' "$argument" >>"$COMMAND_LOG"; done +printf '\\n' >>"$COMMAND_LOG" +`, + ); + await chmod(join(bin, "kubectl"), 0o755); + await chmod(join(bin, "helm"), 0o755); + await chmod(join(bin, "crd-preflight"), 0o755); + return { + root, + log, + env: { ...process.env, PATH: `${bin}:${process.env.PATH}`, COMMAND_LOG: log, T4_CRD_VALIDATOR: join(bin, "crd-preflight") }, + }; +} + +async function runLifecycle(args, env = {}) { + const result = await new Promise((resolveResult, reject) => { + const child = spawn(lifecycle, args, { + cwd: repoRoot, + env, + stdio: ["ignore", "pipe", "pipe"], + }); + let stdout = ""; + let stderr = ""; + child.stdout.on("data", (chunk) => { + stdout += chunk; + }); + child.stderr.on("data", (chunk) => { + stderr += chunk; + }); + child.on("error", reject); + child.on("close", (code, signal) => resolveResult({ code, signal, stdout, stderr })); + }); + return result; +} + +async function commands(log) { + return (await readFile(log, "utf8")).trim().split("\n").filter(Boolean); +} + +function findCommand(log, predicate, description) { + const index = log.findIndex(predicate); + assert.notEqual(index, -1, `missing ${description}:\n${log.join("\n")}`); + return index; +} + +test("upgrade validates proposed schemas, proves served convergence, verifies storage, then upgrades workloads", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "deploy/charts/t4-cluster", "--namespace", "t4-system", "--skip-crds"], + value.env, + ); + assert.equal(result.code, 0, `${result.stdout}\n${result.stderr}`); + const log = await commands(value.log); + const proposedPreflight = findCommand(log, (line) => line.startsWith("validator\tfixtures\t"), "local proposed-schema fixture preflight"); + const crdReads = log.map((line, index) => ({ line, index })).filter(({ line }) => /^kubectl\tget\tcrd\/t4(?:clusterhosts|workspaces|sessions)\.cluster\.t4\.dev\t--ignore-not-found\t-o\tname$/u.test(line)); + const liveLists = log.map((line, index) => ({ line, index })).filter(({ line }) => /^kubectl\tget\tt4(?:clusterhosts|workspaces|sessions)\.cluster\.t4\.dev\t--all-namespaces\t-o\tjson$/u.test(line)); + const liveValidations = log.map((line, index) => ({ line, index })).filter(({ line }) => line.startsWith("validator\tobjects\t")); + const crdPreflight = findCommand(log, (line) => line.includes("apply") && line.includes("--server-side") && line.includes("--dry-run=server") && line.includes("deploy/charts/t4-cluster/crds"), "server-side CRD preflight"); + const crdApply = findCommand(log, (line) => line.includes("apply") && line.includes("--server-side") && !line.includes("--dry-run=server"), "CRD apply"); + const established = findCommand(log, (line) => line.includes("wait") && line.includes("condition=Established") && line.includes("t4clusterhosts.cluster.t4.dev") && line.includes("t4workspaces.cluster.t4.dev") && line.includes("t4sessions.cluster.t4.dev"), "Established wait"); + const servedChecks = log.map((line, index) => ({ line, index })).filter(({ line }) => line.startsWith("validator\tserved\t")); + const admissionPreflight = findCommand(log, (line) => line.includes("apply") && line.includes("--dry-run=server") && line.includes("testdata/compat"), "converged admission preflight"); + const storageChecks = log.map((line, index) => ({ line, index })).filter(({ line }) => line.startsWith("kubectl\tget\tcrd/") && line.includes("status.storedVersions")); + assert.deepEqual(storageChecks.length, 3); + assert.equal(servedChecks.length, 3); + assert.equal(crdReads.length, 3); + assert.equal(liveLists.length, 3); + assert.equal(liveValidations.length, 3); + const workload = findCommand(log, (line) => line.startsWith("helm\tupgrade\t"), "Helm workload upgrade"); + assert.ok(proposedPreflight < crdPreflight); + assert.ok(proposedPreflight < crdReads[0].index); + assert.ok(crdReads[0].index < liveLists[0].index); + assert.ok(liveLists[0].index < liveValidations[0].index); + assert.ok(liveValidations[0].index < crdReads[1].index); + assert.ok(crdReads[1].index < liveLists[1].index); + assert.ok(liveLists[1].index < liveValidations[1].index); + assert.ok(liveValidations[1].index < crdReads[2].index); + assert.ok(crdReads[2].index < liveLists[2].index); + assert.ok(liveLists[2].index < liveValidations[2].index); + assert.ok(liveValidations[2].index < crdPreflight); + assert.ok(crdPreflight < crdApply); + assert.ok(crdApply < established); + assert.ok(established < servedChecks[0].index); + assert.ok(servedChecks.at(-1).index < admissionPreflight); + assert.ok(admissionPreflight < storageChecks[0].index); + assert.ok(storageChecks.every(({ index }) => index < workload)); + assert.ok(log.every((line) => !line.includes("--force") && !line.includes("replace") && !line.includes("delete\tcrd")), log.join("\n")); +}); + +test("fresh install establishes and validates CRDs before Helm installs with CRD handling disabled", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["install", "--", "helm", "install", "t4-cluster", "deploy/charts/t4-cluster", "--namespace", "t4-system", "--skip-crds"], + { ...value.env, MISSING_LIVE_CRDS: "1" }, + ); + assert.equal(result.code, 0, `${result.stdout}\n${result.stderr}`); + const log = await commands(value.log); + assert.ok(log.every((line) => !line.startsWith("validator\tobjects\t") && !/kubectl\tget\tt4(?:clusterhosts|workspaces|sessions)\.cluster\.t4\.dev\t--all-namespaces/u.test(line)), log.join("\n")); + const established = findCommand(log, (line) => line.includes("wait") && line.includes("condition=Established"), "Established wait"); + const fixtureValidation = findCommand(log, (line) => line.includes("--dry-run=server") && line.includes("testdata/compat"), "fixture validation"); + const storage = findCommand(log, (line) => line.includes("status.storedVersions"), "stored-version check"); + const workload = findCommand(log, (line) => line.startsWith("helm\tinstall\t"), "Helm install"); + assert.ok(established < fixtureValidation && fixtureValidation < storage && storage < workload); +}); + +test("candidate schema tightening fails locally before any cluster or workload mutation", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart", "--skip-crds"], + { ...value.env, FAIL_PROPOSED_VALIDATION: "spec" }, + ); + assert.notEqual(result.code, 0); + assert.deepEqual(await commands(value.log), [ + `validator\tfixtures\t${join(repoRoot, "deploy/charts/t4-cluster/crds")}\t${join(repoRoot, "packages/cluster-operator/api/v1alpha1/testdata/compat")}`, + ]); +}); + +test("persisted status is validated against the proposed status schema before mutation", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart", "--skip-crds"], + { ...value.env, FAIL_PROPOSED_VALIDATION: "status" }, + ); + assert.notEqual(result.code, 0); + assert.deepEqual(await commands(value.log), [ + `validator\tfixtures\t${join(repoRoot, "deploy/charts/t4-cluster/crds")}\t${join(repoRoot, "packages/cluster-operator/api/v1alpha1/testdata/compat")}`, + ]); +}); + +test("live object incompatibility fails before non-dry-run CRD, object, or Helm mutation", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart", "--skip-crds"], + { ...value.env, FAIL_PROPOSED_VALIDATION: "live" }, + ); + assert.notEqual(result.code, 0); + const log = await commands(value.log); + assert.match(log.at(-1), /^validator\tobjects\t/u); + assert.ok(log.some((line) => /^kubectl\tget\tt4clusterhosts\.cluster\.t4\.dev\t--all-namespaces\t-o\tjson$/u.test(line)), log.join("\n")); + assert.ok(log.every((line) => !line.startsWith("helm\t") && !/kubectl\tapply(?!.*--dry-run=server)/u.test(line)), log.join("\n")); +}); + +test("live object definition read or list denial fails closed before non-dry-run mutation", async () => { + const deniedResource = "t4workspaces.cluster.t4.dev"; + for (const scenario of [ + { env: { FAIL_CRD_READ: `crd/${deniedResource}` }, expected: `kubectl\tget\tcrd/${deniedResource}\t--ignore-not-found\t-o\tname` }, + { env: { FAIL_LIVE_LIST: deniedResource }, expected: `kubectl\tget\t${deniedResource}\t--all-namespaces\t-o\tjson` }, + ]) { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart", "--skip-crds"], + { ...value.env, ...scenario.env }, + ); + assert.notEqual(result.code, 0); + const log = await commands(value.log); + assert.ok(log.some((line) => line.startsWith(scenario.expected)), log.join("\n")); + assert.ok(log.every((line) => !line.startsWith("helm\t") && !/kubectl\tapply(?!.*--dry-run=server)/u.test(line)), log.join("\n")); + } +}); + +test("cancellation during live enumeration cannot continue to mutation", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart", "--skip-crds"], + { ...value.env, TERMINATE_DURING_LIVE_READ: "t4clusterhosts.cluster.t4.dev" }, + ); + assert.notEqual(result.code, 0); + const log = await commands(value.log); + assert.match(log.at(-1), /^kubectl\tget\tt4clusterhosts\.cluster\.t4\.dev\t--all-namespaces\t-o\tjson$/u); + assert.ok(log.every((line) => !line.startsWith("helm\t") && !/kubectl\tapply(?!.*--dry-run=server)/u.test(line)), log.join("\n")); +}); + +test("retained Established cannot pass readiness while served OpenAPI is stale", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart", "--skip-crds"], + { ...value.env, FAIL_PROPOSED_VALIDATION: "stale" }, + ); + assert.notEqual(result.code, 0); + const log = await commands(value.log); + const apply = findCommand(log, (line) => line.includes("kubectl\tapply") && !line.includes("--dry-run=server"), "non-dry-run CRD apply"); + const established = findCommand(log, (line) => line.includes("condition=Established"), "Established wait"); + const served = findCommand(log, (line) => line.startsWith("validator\tserved\t"), "served-schema semantic verification"); + assert.ok(apply < established && established < served, log.join("\n")); + assert.ok(log.every((line) => !line.startsWith("helm\t")), log.join("\n")); +}); + +test("failed server preflight leaves CRDs and workloads untouched", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "deploy/charts/t4-cluster", "--namespace", "t4-system", "--skip-crds"], + { ...value.env, FAIL_DRY_RUN: "1" }, + ); + assert.notEqual(result.code, 0); + const log = await commands(value.log); + assert.equal(log.length, 11, log.join("\n")); + assert.match(log[0], /^validator\tfixtures\t/u); + assert.match(log.at(-1), /apply.*--server-side.*--dry-run=server/u); + assert.ok(log.every((line) => !line.startsWith("helm\t") && !/kubectl\tapply(?!.*--dry-run=server)/u.test(line)), log.join("\n")); +}); + +test("an unexpected stored version stops workload rollout", async () => { + const value = await fixture(); + const result = await runLifecycle( + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "deploy/charts/t4-cluster", "--namespace", "t4-system", "--skip-crds"], + { ...value.env, STORED_VERSIONS: "v1alpha1,v1beta1" }, + ); + assert.notEqual(result.code, 0); + const log = await commands(value.log); + assert.ok(log.some((line) => line.includes("status.storedVersions"))); + assert.ok(log.every((line) => !line.startsWith("helm\t")), log.join("\n")); +}); + +test("force replacement and implicit Helm CRD handling are rejected before cluster access", async () => { + const value = await fixture(); + for (const args of [ + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart", "--skip-crds", "--force"], + ["upgrade", "--", "helm", "upgrade", "t4-cluster", "chart"], + ]) { + const result = await runLifecycle(args, value.env); + assert.equal(result.code, 64, `${result.stdout}\n${result.stderr}`); + } +}); + +test("future storage migration explicitly retires the old stored version only after rewrite and dual-version reads", async () => { + const docs = await readFile(join(repoRoot, "docs/CLUSTER_OPERATOR.md"), "utf8"); + const migration = docs.slice(docs.indexOf("### Future `v1beta1`"), docs.indexOf("### Workload rollback")); + const storageFlip = migration.indexOf("`v1beta1` storage to true"); + const rewrite = migration.indexOf("rewrite every object"); + const verifyReads = migration.indexOf("read every rewritten object through both served versions"); + const statusUpdate = migration.indexOf("/status"); + const exactAssertion = migration.indexOf("exactly `[v1beta1]`"); + const oldStillServed = migration.indexOf("Keep `v1alpha1` served"); + assert.ok(storageFlip >= 0 && storageFlip < rewrite, migration); + assert.ok(rewrite < verifyReads && verifyReads < statusUpdate, migration); + assert.ok(statusUpdate < exactAssertion && exactAssertion < oldStillServed, migration); + assert.match(migration, /patch customresourcedefinition[^\n]*--subresource=status/u); +});