gatewayapi docs #317
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,188 @@ | ||||||
| --- | ||||||
| slug: /migrating-to-gatewayapi | ||||||
| title: Migrating to Gateway API | ||||||
| sidebar_position: 7 | ||||||
| --- | ||||||
|
|
||||||
| # Migrating to Gateway API | ||||||
|
|
||||||
| :::warning | ||||||
| Gateway API support in metal-stack is still in development. Changes are expected, especially around certificate management. | ||||||
| ::: | ||||||
|
|
||||||
| From version `x.y.z` <!-- TODO: fill in the first metal-stack release that ships Gateway API support --> metal-stack supports [Gateway API](https://gateway-api.sigs.k8s.io/) as a replacement for Kubernetes `Ingress` resources. | ||||||
|
|
||||||
| This guide explains why we are moving to Gateway API, what changes for you as an operator, and how to migrate an existing metal-stack installation. | ||||||
|
|
||||||
| ## Motivation | ||||||
|
|
||||||
| `ingress-nginx` has been deprecated and we currently depend on it. metal-stack | ||||||
| has control-plane components that are not served via HTTP/GRPC. Those are | ||||||
| exposed as TCP Services via `ingress-nginx`. As a result we are going to move | ||||||
| all metal-stack components to Gateway API. | ||||||
|
|
||||||
| ## What will change | ||||||
|
|
||||||
| ### Certificate provisioning and TLS termination | ||||||
|
|
||||||
| Certificate provisioning and TLS termination will move into the Gateway resource. metal-stack deployed via metal-roles will expect TLS termination to happen on the Gateway. | ||||||
|
|
||||||
| Exceptions for TLS termination: | ||||||
|
|
||||||
| - NSQ will for now still terminate its own certificates | ||||||
| - The GRPC endpoint of metal-api will stay a TCPRoute and TLS termination will stay in the application. As metal-api will be superseded by metal-apiserver in the near future, we do not see the value in migrating it fully | ||||||
|
|
||||||
| ### Continued need for an Ingress Controller | ||||||
|
|
||||||
| For some vendor dependencies you will still require an ingress controller. Examples of services still relying on Ingress are Gardener and Thanos. As ingress-nginx is EoL we recommend switching to a different Ingress Controller implementation. | ||||||
|
|
||||||
| ## Before you begin | ||||||
|
|
||||||
| ### Familiarize yourself with Gateway API | ||||||
|
|
||||||
| Gateway API is more complex than Ingress and consists of multiple different resources and operator personas. The rest of the guide assumes a basic understanding of the `HTTPRoute`, `TCPRoute`, `Gateway`, `GatewayClass`. An introduction can be found [here](https://gateway-api.sigs.k8s.io/docs/introduction/) | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
|
||||||
|
|
||||||
| ### Gateway API implementation requirements | ||||||
|
|
||||||
| metal-stack does not require one specific Gateway API implementation. | ||||||
| You can choose any Gateway API implementation that provides the following features: | ||||||
|
|
||||||
| - `HTTPRoute` | ||||||
| - `TCPRoute` | ||||||
|
|
||||||
| In our [mini-lab](https://github.com/metal-stack/mini-lab/pull/299) demo environment we are running [Envoy Gateway](https://gateway.envoyproxy.io/docs/). | ||||||
|
|
||||||
| ### Migration Path decisions | ||||||
|
|
||||||
| - Select a Gateway API implementation, e.g. [Envoy Gateway](https://gateway.envoyproxy.io/docs/) | ||||||
| - Certificate management: Gateway API now requires the Gateway to know | ||||||
|
|
||||||
| ## Migration preparation | ||||||
|
|
||||||
| ### Gateway deployment | ||||||
|
|
||||||
| As we require `TCPRoute`s you will have to make changes to your existing Gateways, if applicable. | ||||||
| Still, we suggest you create a dedicated metal-stack gateway, like we do for [mini-lab](https://github.com/metal-stack/mini-lab/roles/gateway). | ||||||
|
|
||||||
| We recommend provisioning a metal-stack Gateway resource as metal-stack requires multiple TCP endpoints. | ||||||
|
|
||||||
| #### Example | ||||||
|
|
||||||
| ```yaml | ||||||
| apiVersion: gateway.networking.k8s.io/v1 | ||||||
| kind: Gateway | ||||||
| metadata: | ||||||
| name: metal-control-plane | ||||||
| namespace: metal-control-plane | ||||||
| spec: | ||||||
| gatewayClassName: eg | ||||||
| listeners: | ||||||
| - hostname: '*.<metal-stack FQDN>' | ||||||
| name: http | ||||||
| port: 80 | ||||||
| protocol: HTTP | ||||||
| - hostname: '*.<metal-stack FQDN>' | ||||||
| name: https | ||||||
| port: 443 | ||||||
| protocol: HTTPS | ||||||
| tls: | ||||||
| certificateRefs: | ||||||
| - group: "" | ||||||
| kind: Secret | ||||||
| name: metal-api-tls | ||||||
| mode: Terminate | ||||||
| - name: nsq | ||||||
| port: 4150 | ||||||
| protocol: TCP | ||||||
| - name: metal-api-grpc | ||||||
| port: 50051 | ||||||
| protocol: TCP | ||||||
|
|
||||||
| ``` | ||||||
|
|
||||||
| ### Certificate management | ||||||
|
|
||||||
| The biggest change during the Gateway API migration is certificate management. Using Ingress Controllers the Ingress resource configured if, and using what certificates TLS connections were terminated. This changed in Gateway API, where the Gateway listener you attach the Route on decides if TLS is used and which certificate will be used for termination. | ||||||
|
|
||||||
| metal-stack.io does not require any specific certificate management process. It is possible to use self-signed/private CA certificates as well as certificates public CAs. | ||||||
|
|
||||||
| :::tip | ||||||
| Use automated certificate management. | ||||||
| [cert-manager](https://cert-manager.io/docs/usage/gateway/) can handle your | ||||||
| certificate renewal and supports Gateway API. You can use Let's Encrypt for valid | ||||||
| public certificates or use a private CA. | ||||||
|
|
||||||
| You can issue valid public TLS certs using [Let's Encrypt](https://letsencrypt.org/2026/01/15/6day-and-ip-general-availability) against an IP address if no DNS record is available. | ||||||
| ::: | ||||||
|
|
||||||
| ### TLS termination locations | ||||||
|
|
||||||
| Exposed components with TLS terminated at the Gateway: | ||||||
|
|
||||||
| - metal-api (excl. GRPC endpoint) | ||||||
| - metal-apiserver | ||||||
| - metal-console | ||||||
| - zitadel | ||||||
| - headscale | ||||||
|
|
||||||
| Exposed components terminating TLS in the application Pod: | ||||||
|
|
||||||
| - metal-api GRPC endpoint | ||||||
| - nsq | ||||||
|
|
||||||
| <!-- TODO: scaffold — describe how certificates are wired to the Gateway listeners (Certificate / Issuer resources, secret references), and the recommended setup for private vs. public CAs --> | ||||||
|
|
||||||
| ### Migrating Components | ||||||
|
|
||||||
| ::: warning | ||||||
|
Contributor
There was a problem hiding this comment. This block is not properly rendered in docusaurus. |
||||||
| The old Ingress Controller and new Gateway are going to have different IP addresses. | ||||||
| ::: | ||||||
|
|
||||||
| Deploy the `HTTPRoutes` and `TCPRoutes` for each component using metal-role. *routes and Ingress resources can be deployed at the same time. To switch over a service to the Gateway, change the DNS record of that service from pointing to the Ingress Controller to the Gateway. | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
Contributor
There was a problem hiding this comment. Hint to unusual parallel deployment parametrization: metal_deploy_ingress: true
metal_api_httproute_enabled: true
metal_deploy_ingress_api_v1_rules: true
metal_deploy_ingress_api_v2_rules: falseAfter migration toggle |
||||||
|
|
||||||
| Example for zitadel, nsq and metal_apiserver. For full documentation of all services please consult the [metal-roles repository](https://github.com/metal-stack/metal-roles). | ||||||
|
|
||||||
| ```yaml | ||||||
|
|
||||||
| zitadel_httproute_enabled: true | ||||||
| zitadel_httproute_parent_refs: | ||||||
| - name: metal-control-plane | ||||||
| sectionName: https | ||||||
|
|
||||||
| nsq_tcproute_enabled: true | ||||||
| nsq_tcproute_parent_refs: | ||||||
| - name: metal-control-plane | ||||||
| sectionName: nsq | ||||||
|
|
||||||
| metal_apiserver_httproute_enabled: true | ||||||
| metal_apiserver_httproute_parent_refs: | ||||||
| - name: metal-control-plane | ||||||
| sectionName: http | ||||||
| - name: metal-control-plane | ||||||
| sectionName: https | ||||||
| ``` | ||||||
|
|
||||||
|
Contributor
There was a problem hiding this comment. I was using a gateway resolver for cert-manager to issue certificates and for this scenario I manually copied the existing cert secrets over to the gateway namespace in order to make both services successfully reachable in parallel for a transition time window. Not sure if there is a better solution than that. Otherwise we could just recommend that there will be a short switchover phase where the services are not available until cert-manager was able to solve the DNS challenge for the IP address. |
||||||
|
|
||||||
| #### Remaining ingress controller dependencies | ||||||
|
|
||||||
| For some metal-stack dependencies you will still require an ingress controller: | ||||||
|
|
||||||
| - Gardener | ||||||
| - Thanos | ||||||
|
|
||||||
| ## Verification | ||||||
|
|
||||||
| Check your endpoints are still reachable on their expected host names/IP addresses. The easiest way to do that is to check with metalctl(metal-api)/metalctlv2(metal-apiserver). | ||||||
|
|
||||||
| It is also recommended to check if the DNS records really resolve the new Gateway and are not accessed via the old ingress infrastructure. Use `nslookup`/`dig` to check your hostname and make sure they resolve to the same IP address as the Gateway | ||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
Contributor
There was a problem hiding this comment. Another possible to check before switching over the DNS entry is: |
||||||
|
|
||||||
| ## Rollback | ||||||
|
|
||||||
| You can roll back to ingress any time if an unresolvable blocker is discovered during the migration. Revert the deployment variables and rerun the playbook. | ||||||
|
|
||||||
| ## References | ||||||
|
|
||||||
| - [Gateway API](https://gateway-api.sigs.k8s.io/) | ||||||
| - [Envoy Gateway](https://gateway.envoyproxy.io/docs/) | ||||||
| - [cert-manager — Gateway API usage](https://cert-manager.io/docs/usage/gateway/) | ||||||
| - [mini-lab gateway role](https://github.com/metal-stack/mini-lab/roles/gateway) | ||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Gardener does not rely on an ingress controller from third-party.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now opened some PRs to get rid off the rest of the ingresses:
monitoringrole metal-roles#729loggingrole metal-roles#730