For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Install kgateway and related components.

You can update several installation settings in your Helm values file. For example, you can update the namespace, set resource limits and requests, or enable extensions such as for AI.

Set the version you want to configure in an environment variable, such as the latest patch version (2.5.0).

export NEW_VERSION=2.5.0

• Show all values:

  helm show values oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --version v$NEW_VERSION

• Get a file with all values: You can get a kgateway/values.yaml file for the upgrade version by pulling and inspecting the Helm chart locally.

  helm pull oci://cr.kgateway.dev/kgateway-dev/charts/kgateway --version v$NEW_VERSION
  tar -xvf kgateway-v$NEW_VERSION.tgz
  open kgateway/values.yaml

For more information, see the Helm reference docs.

Development builds

When using the development build 2.6.0-main, add --set controller.image.pullPolicy=Always to ensure you get the latest image. For production environments, this setting is not recommended as it might impact performance.

Experimental Gateway API features

The KGW_ENABLE_EXPERIMENTAL_GATEWAY_API_FEATURES feature gate controls support for experimental Gateway API features and API versions, such as the following:

• Session persistence in HTTPRoutes, which is deprecated in favor of session persistence in a BackendConfigPolicy
• The legacy XListenerSet resource in the gateway.networking.x-k8s.io API group
• Support for the pre-v1 API versions of TCPRoutes and TLSRoutes, such as v1alpha2, in clusters where the Gateway API CRDs do not serve v1

Features that are in the standard channel of the Gateway API, such as ListenerSets, TCPRoutes, CORS policies, and retries, work when the gate is disabled. Retries are in the standard channel in Gateway API version 1.7 and later.

In kgateway version 2.2 and later, this setting defaults to true, so experimental features are enabled by default and no additional configuration is required. To disable these features, set the environment variable to false in your kgateway controller deployment in your Helm values file.

controller:
  extraEnv:
    KGW_ENABLE_EXPERIMENTAL_GATEWAY_API_FEATURES: "false"

Leader election

Leader election is enabled by default to ensure that you can run kgateway in a multi-control plane replica setup for high availability.

You can disable leader election by setting the KGW_DISABLE_LEADER_ELECTION environment variable to "true" through the controller.extraEnv Helm value.

controller:
  extraEnv:
    KGW_DISABLE_LEADER_ELECTION: "true"

Leader election timings

On clusters with a slow or temporarily unavailable Kubernetes API server, the controller can fail to renew its lease in time. The controller then repeatedly loses and re-acquires leadership. To make the controller more tolerant of a slow API server, increase the leader election timings through the controller.leaderElection Helm values.

controller:
  leaderElection:
    leaseDuration: 30s
    renewDeadline: 20s
    retryPeriod: 5s

Namespace discovery

You can limit the namespaces that kgateway watches for gateway configuration. For example, you might have a multi-tenant cluster with different namespaces for different tenants. You can limit kgateway to only watch a specific namespace for gateway configuration.

Namespace selectors are a list of matched expressions or labels.

• matchExpressions: Use this field for more complex selectors where you want to specify an operator such as In or NotIn.
• matchLabels: Use this field for simple selectors where you want to specify a label key-value pair.

Each entry in the list is disjunctive (OR semantics). This means that a namespace is selected if it matches any selector.

You can also use matched expressions and labels together in the same entry, which is conjunctive (AND semantics).

The following example selects namespaces for discovery that meet either of the following conditions:

• The namespace has the label environment=prod and the label version=v2, or
• The namespace has the label version=v3

discoveryNamespaceSelectors:
- matchExpressions:
  - key: environment
    operator: In
    values:
    - prod
  matchLabels:
    version: v2
- matchLabels:
    version: v3

Scope xDS configuration per gateway

The control plane sends configuration to the gateway proxies over xDS. By default, every gateway proxy receives a cluster for every backend that the control plane discovers, even if no route of that Gateway uses it. In a cluster with many Services and few routed ones, each proxy carries a large config dump and reports stats for many clusters that it never uses.

Enable xDS scoping

You can scope the xDS configuration so that each gateway’s proxies receive only the clusters that the gateway’s configuration references by using the controller.xds.scoping Helm values. These clusters include the backends that your routes send traffic to, and the backends that your policies call while the proxy processes a request. Examples are an external auth, external processing, or rate limit service, a server that hosts the JWKS for JWT validation, and a gRPC access log service. The proxies also receive the endpoints, listeners, and secrets that these backends need, such as the internal listener of an HTTP CONNECT tunnel. Scoping does not change the listeners and secrets that you configure on the Gateway, such as an HTTPS listener and its TLS certificate. The proxies always receive them.

1. Set the scoping mode in your Helm values file. The control plane supports the following scoping modes:

   • ALL (default): Every gateway proxy receives a cluster for every discovered backend, whether or not a route uses it.
   • REFERENCED: Each gateway’s proxies receive only the clusters that the gateway’s configuration references. A gateway whose routes select the destination at request time is not scoped and receives all clusters.

   The following example enables the REFERENCED mode.

   controller:
     xds:
       scoping:
         mode: REFERENCED

   The Helm value sets the KGW_XDS_SCOPING_MODE environment variable on the control plane.

2. Upgrade your kgateway installation with the updated values file.

   helm upgrade -i -n kgateway-system kgateway \
     oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
     --version v2.5.0 \
     -f values.yaml

Monitor xDS scoping

The control plane reports the following metrics, each with the gateway and namespace labels, so that you can see how scoping affects each gateway. Note that scoping metrics are emitted only in REFERENCED mode.

You can access scoping metrics by querying the controller’s metrics endpoint.

kubectl get --raw \
  "/api/v1/namespaces/kgateway-system/services/kgateway:9092/proxy/metrics" \
  | grep -E 'kgateway_xds_scoping|kgateway_xds_snapshot_resources.*Cluster'

Example output:

kgateway_xds_scoping_disabled_gateways{gateway="http",namespace="kgateway-system"} 0
kgateway_xds_scoping_dropped_clusters{gateway="http",namespace="kgateway-system"} 12
kgateway_xds_snapshot_resources{gateway="http",namespace="kgateway-system",resource="Cluster"} 2

In this example, the gateway proxies receive 2 clusters, and the control plane withholds 12 clusters. The kgateway_xds_scoping_transitions_total metric appears after you add or remove a route to a backend.

Tune cluster transitions

A transition is a change to the clusters that a gateway proxy receives. It happens when you add a route to a backend that is new to the gateway, or when you remove the last route to a backend. Envoy does not necessarily apply cluster updates before route updates, so the control plane delays one half of the change to reduce brief 503 errors.

• Adding a route to a new backend: The control plane sends the cluster first, and holds the route until the cluster had time to reach the proxy.
• Removing the last route to a backend: The control plane sends the route update first, and keeps the cluster for a while before it removes the cluster.

You can tune how long the control plane delays each half of a transition with the following settings. Set them in the controller.xds.scoping section of your Helm values file, or as environment variables on the control plane. The control plane counts transitions in the kgateway_xds_scoping_transitions_total metric, by the transition label.

Watch the rate of the reference_ahead_held transition. A high rate means that your route changes often add clusters that are new to the gateway, and each change waits for the referenceAhead time. If these waits delay your route changes more than you can accept, consider setting a lower referenceAhead time. A lower time increases the risk of brief 503 errors.

The referenceAhead hold is one of several delays that the control plane applies before it publishes configuration to a proxy. A separate time limit ends any delay that lasts too long. For more information, see Limit xDS publication delays.

Limit xDS publication delays

The settings in this section apply whether or not you enable xDS scoping. The control plane can delay sending configuration to a gateway proxy until the clusters that the configuration references are ready. A cluster is ready when the control plane has translated the cluster and its endpoints. The delay matters because the control plane can translate a route before it translates the cluster that the route points to. If the proxy received the route first, requests on that route would fail with a 503 response until the cluster arrived.

The control plane delays configuration in the following cases:

• First configuration: A gateway proxy connects to the control plane and requests its configuration for the first time, such as when a new gateway pod starts. The control plane waits so that the proxy starts with routes that work.
• Route change: You change a route so that it references a cluster that is not ready yet. The proxy keeps its current routes until the cluster is ready, while the new cluster and its endpoints still reach the proxy.

A cluster might never become ready, for example if a route references a Service that does not exist. To keep that cluster from blocking the proxy, the control plane limits how long each delay lasts. When the time limit expires, the control plane publishes the latest configuration. Requests on routes to the missing cluster return a 503 response until the cluster becomes ready.

Tune the time limits

You can tune the time limits with environment variables on the control plane. The control plane reads the variables at startup.

1. Set the environment variables in the controller.extraEnv section of your Helm values file. The following example sets a longer time limit for the first configuration of a proxy.

   controller:
     extraEnv:
       KGW_PER_CLIENT_PUBLISH_BUDGET: "20s"

   Environment variable	Description	Default
   KGW_PER_CLIENT_PUBLISH_BUDGET	How long the control plane delays the first configuration of a proxy, or a route change, while referenced clusters are not ready. Keep this value well below the startup probe time of the gateway proxy, which is 60 seconds by default. You can change the probe in the spec.kube.podTemplate.startupProbe field of the GatewayParameters resource. This value is not the total time until the proxy is ready. The total time also includes translation, the first connection delay that the KGW_XDS_FIRST_CONNECT_DELAY variable sets, and delivery to the proxy. If the first configuration arrives after the probe time, the proxy restarts. Set 0 to remove the time limit, so that the control plane delays the configuration until all referenced clusters are ready. With 0, a new proxy that references a cluster that never becomes ready never receives configuration and never becomes ready.	15s
   KGW_PER_CLIENT_WARM_WITHHOLD_BOUND	How long the control plane continues to delay the configuration of a proxy that reconnects with configuration that it already has, such as after the control plane restarts, while referenced clusters are missing. This time starts after the KGW_PER_CLIENT_PUBLISH_BUDGET time expires. While the control plane delays the configuration, the proxy receives no updates at all, including certificate rotations. The value also limits how long the control plane keeps sending the last configuration of a cluster that is missing. Keep this value well above the time that the control plane needs to translate all resources after a restart. If the value is too low, routes fail each time the control plane restarts. Set 0 to remove the time limit. This setting has no effect if KGW_PER_CLIENT_PUBLISH_BUDGET is 0.	10m

2. Upgrade your kgateway installation with the updated values file. The --reuse-values setting re-applies any Helm values that you previously set.

   helm upgrade -i -n kgateway-system kgateway \
     oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
     --version v2.5.0 \
     --reuse-values \
     -f values.yaml

3. Verify that the control plane runs with the new value. The upgrade restarts the control plane, so wait for the rollout to finish first. The command prints the value of the variable that you set, such as 20s.

   kubectl -n kgateway-system rollout status deployment/kgateway
   kubectl -n kgateway-system get deployment kgateway \
     -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="KGW_PER_CLIENT_PUBLISH_BUDGET")].value}'

Monitor xDS publication

The control plane reports the following metrics so that you can monitor xDS publication in your cluster.

A rising kgateway_xds_snapshot_perclient_bounded_publishes_total value means that routes reference clusters that do not become ready in time. To find these clusters, check the control plane logs for warnings such as first-publish budget expired. The warnings list the missing clusters.

To check the metrics, query the metrics endpoint of the control plane.

kubectl get --raw \
  "/api/v1/namespaces/kgateway-system/services/kgateway:9092/proxy/metrics" \
  | grep kgateway_xds_snapshot_perclient

Example output after the control plane restarts:

kgateway_xds_snapshot_perclient_defers_total{gateway="http",namespace="demo",reason="missing_endpoints"} 1

In this example, the control plane delayed the configuration of the http gateway once after the restart, until the control plane translated the endpoints of a referenced cluster. The control plane reports each metric only after the event that the metric counts happens for the first time. If the command returns no output, the control plane did not delay any configuration.

TLS encryption

You can enable TLS encryption for the xDS gRPC server in the kgateway control plane. For more information, see the TLS encryption docs.

Resource validation

Kgateway supports two validation modes for routes and policies in the control plane: standard and strict. The validation mode controls how the control plane handles invalid configuration before it is sent to Envoy.

Validation modes

Kgateway supports the following validation modes. The mode is set globally on the controller through a single Helm value.

Invalid configuration handling in strict mode

Strict mode does not keep a previous version of a route when an update to the route is invalid. The control plane translates the current state of your resources, and handles invalid configuration as follows.

For example, suppose that route-a and route-b are attached to the same listener, and you update route-b with an invalid regular expression in a path match. In strict mode, route-a continues to serve traffic, the invalid rule of route-b is removed so that its requests receive an HTTP 404 response, and route-b reports the RouteRuleDropped reason. In standard mode, the invalid regular expression is sent to Envoy, which rejects the update. Both routes keep their previous configuration on the proxies that already accepted it, but no further route changes on that listener take effect until you fix route-b.

Enable strict validation

Set the validation.level Helm value to strict when you install or upgrade kgateway. Restart the control plane to apply the change.

validation:
  level: strict

Internally, the Helm chart passes the value to the control plane through the KGW_VALIDATION_MODE environment variable. If you manage the control plane deployment manually, set KGW_VALIDATION_MODE=STRICT on the kgateway container.

The accepted values for validation.level are standard and strict (case-insensitive). Any other value causes the Helm install to fail.

Verify the validation mode

To check which mode is active, inspect the KGW_VALIDATION_MODE environment variable on the kgateway controller deployment. The expected output is standard or strict.

kubectl -n kgateway-system get deployment kgateway \
  -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="KGW_VALIDATION_MODE")].value}'

Transformation policies and strict validation

Strict validation runs the preflight against an Envoy binary that is bundled in the kgateway control plane image. The control plane image is built from the envoy-wrapper image, which bundles the rustformation dynamic module, and the validator sets ENVOY_DYNAMIC_MODULES_SEARCH_PATH=/usr/local/lib before invoking the preflight. As a result, the preflight understands rustformation per-route config and can validate TrafficPolicies that use transformation.

For more information about transformation engines, see Transformation engines.

Tune the controller Go memory limit

By default, the Go runtime that the kgateway controller runs on does not know how much memory Kubernetes allows its container to use. The controller’s garbage collector just runs on its own schedule, so the controller can keep allocating memory right up to the container’s limit. When it crosses that limit, the Linux kernel kills the container immediately, with no warning and no chance for the controller to free memory first. This event appears as a Kubernetes pod restart, often labeled OOMKilled.

The GOMEMLIMIT environment variable fixes this issue by giving the Go runtime a soft memory ceiling. As the pod’s memory usage approaches that ceiling, the garbage collector starts freeing up memory, so the controller can stay under the container’s limit instead of being killed when it goes over.

By default, controller.goMemLimitPercent is set to 0, which disables the Go memory limit feature. The Helm chart sets the GOMEMLIMIT environment variable when the controller pod starts by reading the resources.limits.memory on the associated Deployment. This limit is fixed throughout the pod’s lifecycle. If the container’s memory limit changes later, such as when a Kubernetes LimitRange resource is applied or a Vertical Pod Autoscaler (VPA) resizes the pod in place, GOMEMLIMIT does not follow that change until the pod restarts.

To adjust the GOMEMLIMIT variable dynamically, set the controller.goMemLimitPercent field to a value between 1 and 100. This way, the GOMEMLIMIT environment variable is kept in sync with the container’s memory limit as it changes. Instead of reading the memory limit once at startup, the controller reads the container’s live memory limit directly from its cgroup every 30 seconds, and sets GOMEMLIMIT to the percentage you configure of that current value. A value of 90 is the recommended starting point. The controller targets 90% of the container’s memory limit, leaving 10% as headroom for memory that the Go runtime does not track, such as memory that is used by Envoy subprocesses.

controller:
  goMemLimitPercent: 90

If you enable strict validation, use a lower value such as 80 because Envoy subprocess memory is not covered by GOMEMLIMIT. Do not set controller.extraEnv.GOMEMLIMIT or controller.extraEnv.AUTOMEMLIMIT with controller.goMemLimitPercent. If the controller container has no finite cgroup memory limit, GOMEMLIMIT remains unconstrained and the controller logs a warning.

ReferenceGrant enforcement modes

In multi-tenant clusters, different teams typically own separate namespaces and share a gateway. The Gateway API ReferenceGrant mechanism controls which cross-namespace references are permitted, ensuring that one team cannot silently access another team’s resources. Without a ReferenceGrant in the target namespace, the reference is denied.

In kgateway, you can configure how strictly you want ReferenceGrant requirements to be enforced by using the KGW_REFERENCE_GRANT_MODE environment variable on the control plane. You can choose between the following modes:

• STRICT: Enforce ReferenceGrants for every cross-namespace reference in the following table, including cross-namespace ExtensionRef references. This mode provides the strongest namespace isolation and is recommended for new clusters.
• PERMISSIVE (default): Enforce ReferenceGrants for every cross-namespace reference in the following table except cross-namespace ExtensionRef references. Before reference grant modes were introduced, TrafficPolicy resources were able to reference and access a GatewayExtension resource in another namespace without a ReferenceGrant. PERMISSIVE mode allows these setups to function as before. Over time, you can add the missing ReferenceGrant resources in the required namespaces and migrate your cluster to STRICT ReferenceGrant validation.
• OFF: Disable all ReferenceGrant validation. Not recommended for multi-tenant or production environments.

  Caution

  Do not use OFF in multi-tenant or production environments. It breaks Gateway API compliance, bypasses namespace isolation, and lets any namespace access backends, secrets, and GatewayExtensions in other namespaces without restriction.

Reference validation by mode

The following table shows which cross-namespace references are checked in each mode. Same-namespace references always pass, regardless of the mode.

ReferenceGrant example

To reference resources across namespaces, create a ReferenceGrant in the namespace of the resource that you want to access, not in the namespace of the resource that makes the reference. The from section describes the resource that makes the reference, and the to section describes the resource it is allowed to reach. For more information about which resource to reference in each field, see Reference validation by mode.

The following example allows a policy in the httpbin namespace to read Secrets in the team-secrets namespace.

kubectl apply -f- <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: allow-apikey-secrets
  # The namespace that holds the Secrets.
  namespace: team-secrets
spec:
  from:
  - group: gateway.kgateway.dev
    kind: TrafficPolicy
    # The namespace that holds the policy.
    namespace: httpbin
  to:
  - group: ""
    kind: Secret
EOF

To restrict the grant to a single Secret, add name to the to entry. When you omit name, every Secret in the namespace is allowed.

If a policy reports missing reference grant, check the following settings of your ReferenceGrant.

• The ReferenceGrant is in the namespace of the resource that you want to access, such as the namespace of the Secret.
• The from section matches the group, kind, and namespace of the resource that makes the reference. The API server accepts a ReferenceGrant with a group or kind that does not match, but that grant never matches the reference.
• The to section matches the group and kind of the resource that you want to access, and the name, if you set one.

Enable STRICT mode

1. Check which mode is currently active by inspecting the KGW_REFERENCE_GRANT_MODE environment variable on the controller deployment. If the variable is not set, the active mode is PERMISSIVE.

   kubectl -n kgateway-system get deployment kgateway \
     -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="KGW_REFERENCE_GRANT_MODE")].value}'

2. Make sure that any existing cross-namespace TrafficPolicy → GatewayExtension references have a corresponding ReferenceGrant.

3. Get the current Helm values for your kgateway release and save them to a file.

   helm get values kgateway -n kgateway-system -o yaml > values.yaml
   open values.yaml

4. Add the following values to enable STRICT ReferenceGrant validation.

   controller:
     extraEnv:
       KGW_REFERENCE_GRANT_MODE: "STRICT"

5. Apply the change by upgrading the Helm release.

   helm upgrade -i -n kgateway-system kgateway \
     oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
     --version v2.5.0 \
     -f values.yaml

6. Confirm that the kgateway control plane restarted and is running.

   kubectl get pods -n kgateway-system

7. Verify the active mode.

   kubectl -n kgateway-system get deployment kgateway \
     -o jsonpath='{.spec.template.spec.containers[*].env[?(@.name=="KGW_REFERENCE_GRANT_MODE")].value}'

Disable automatic RBAC creation

By default, the kgateway Helm chart creates a ClusterRole and ClusterRoleBinding that grant the controller’s service account the permissions it needs to watch and manage Kubernetes resources. In environments where RBAC resources are managed externally, such as by a platform or security team that controls all cluster-scoped permissions, you can disable the creation of these resources by setting rbac.create: false.

The rbac.create setting does not affect the ServiceAccount of the controller. The chart still creates the ServiceAccount, so that you can bind your own permissions to it. To skip the ServiceAccount too, set serviceAccount.create: false.

To disable automatic RBAC creation, create the permissions yourself, and then turn off the chart’s RBAC resources.

1. Render the ClusterRole and ClusterRoleBinding that the Helm chart creates by default. Use the output as the starting point for your own resources. The ClusterRole is in the role.yaml template, and the ClusterRoleBinding is in the serviceaccount.yaml template. The output also includes the ServiceAccount of the controller.

   helm template kgateway \
     oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
     --version 2.5.0 \
     -n kgateway-system  \
     --show-only templates/role.yaml \
     --show-only templates/serviceaccount.yaml  > rbac.yaml

2. Open the rbac.yaml file and review the rules of the ClusterRole. Then remove the ServiceAccount resource from the file. This is the YAML document with apiVersion: v1 and kind: ServiceAccount. The chart still creates the ServiceAccount, and a ServiceAccount that you create yourself with the same name conflicts with it. Keep the serviceaccounts rule in the ClusterRole and the ServiceAccount in the subjects of the ClusterRoleBinding.

3. If you upgrade an existing installation, change the names of the ClusterRole and ClusterRoleBinding resources in the file, and update the roleRef of each ClusterRoleBinding to use the new ClusterRole name. The upgrade in step 6 removes the ClusterRole and ClusterRoleBinding resources that the chart created earlier. It also removes any resource that has the same name as one of them.

4. Create the resources in your cluster.

   kubectl apply -f rbac.yaml

5. Add the rbac.create setting to your Helm values file.

   rbac:
     create: false

6. Install or upgrade the Helm release with the values file.

   helm upgrade -i -n kgateway-system kgateway \
     oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
     --version 2.5.0 \
     -f values.yaml

7. Verify that the Helm release no longer includes a ClusterRole or ClusterRoleBinding. The command returns no output.

   helm get manifest kgateway -n kgateway-system | grep -E "^kind: (ClusterRole|ClusterRoleBinding)$"

8. Verify that the control plane is running.

   kubectl get pods -n kgateway-system

Common labels

Add custom labels to all resources that are created by the Helm charts. These resources include the controller Deployment metadata, controller pod template, Service, ServiceAccount, and ClusterRoles. The labels are separate from selector labels, so you can update commonLabels without changing the Deployment’s immutable selector labels.

The following snippet adds the label-key and kgw-managed labels to all resources.

commonLabels: 
  label-key: label-value
  kgw-managed: "true"

Topology spread constraints

Use topology spread constraints to control how kgateway controller pods are distributed across failure domains such as zones or nodes. This setup helps improve availability and resilience by preventing all replicas from landing in the same zone or node.

For more information, see the Kubernetes topology spread constraints documentation.

The following example spreads controller pods evenly across availability zones, and prevents scheduling if the skew cannot be satisfied.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: kgateway

PriorityClass

You can assign a PriorityClassName to the control plane pods by using the Helm chart. Priority indicates the importance of a pod relative to other pods. If a pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority pods to make scheduling of the pending pod possible.

To assign a PriorityClassName to the control plane, you must first create a PriorityClass resource. The following example creates a PriorityClass with the name system-cluster-critical that assigns a priority of 1 million.

kubectl apply -f- <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: system-cluster-critical
value: 1000000
globalDefault: false
description: "Use this priority class on system-critical pods only."
EOF

In your Helm values file, add the name of the PriorityClass in the controller.priorityClassName field.

controller: 
  priorityClassName: 

Autoscaling

You can configure Horizontal Pod Autoscaler (HPA) or Vertical Pod Autoscaler (VPA) policies for the kgateway control plane. To set up these policies, you use the horizontalPodAutoscaler or verticalPodAutoscaler fields in the Helm chart.

Vertical Pod Autoscaler (VPA)

Vertical Pod Autoscaler (VPA) is a Kubernetes component that automatically adjusts the CPU and memory reservations of your pods to match their actual usage.

The following Helm configuration ensures that the control plane pod is always assigned a minimum of 0.1 CPU cores (100millicores) and 128Mi of memory.

controller:
  verticalPodAutoscaler:
    updatePolicy:
      updateMode: Auto
    resourcePolicy:
      containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 100m
          memory: 128Mi

Horizontal Pod Autoscaler (HPA)

Horizontal Pod Autoscaler (HPA) adds more instances of the pod to your environment when certain memory or CPU thresholds are reached.

In the following example, you want to have 1 control plane replica running at any given time. If the CPU utilization averages 80%, you want to gradually scale up your replicas. You can have a maximum of 5 replicas at any given time.

controller: 
  horizontalPodAutoscaler:
    minReplicas: 1
    maxReplicas: 5
    metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 80

Note: To monitor the memory and CPU threshold, you must deploy the Kubernetes metrics-server to your cluster. The metrics-server retrieves metrics, such as CPU and memory consumption, for your workloads.

You can install the server with the following command:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system patch deployment metrics-server \
 --type=json \
 -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

Then, start monitoring CPU and memory consumption with the kubectl top pod command.

PodDisruptionBudget

Configure a Pod Disruption Budget to ensure that a minimum number of control plane instances are up and running at any given time during voluntary disruptions, such as upgrades. In this example, 50% of your control plane instances must be running.

controller: 
  podDisruptionBudget:
    minAvailable: 50%

Controller probes

You can customize the readiness and startup probes for the kgateway controller container by using the controller.readinessProbe and controller.startupProbe Helm values. Your settings are deep-merged with the default probe configuration, so you only need to specify the fields you want to change.

By default, both probes use an httpGet handler that checks the /readyz endpoint on the health port. The default readiness probe polls every 10 seconds with an initial delay of 1 second. The default startup probe polls every second with no initial delay and a failure threshold of 600, allowing up to 10 minutes for the controller to start.

If you provide an exec, grpc, or tcpSocket handler, the default httpGet handler is replaced entirely. Otherwise, the httpGet handler is kept and your overrides are merged on top.

The following example adjusts the timing fields of the default readiness probe without changing the default httpGet handler.

controller:
  readinessProbe:
    initialDelaySeconds: 5
    periodSeconds: 20
    failureThreshold: 3

The following example replaces the default httpGet handler with a custom exec handler on the startup probe.

controller:
  startupProbe:
    exec:
      command: ["cat", "/tmp/ready"]
    initialDelaySeconds: 10
    periodSeconds: 5
    failureThreshold: 60

xDS first-connect grace period

By default, the control plane waits 1 second after a new proxy connects before sending its first xDS snapshot. This gives per-client translation time to converge and prevents newly started gateway pods from receiving incomplete configuration after a controller restart.

You can adjust the grace period by using the KGW_XDS_FIRST_CONNECT_DELAY environment variable on the controller. The value is a Go duration string, for example 2s. Set it to 0 to disable the grace period entirely.

controller:
  extraEnv:
    KGW_XDS_FIRST_CONNECT_DELAY: "2s"

DNS lookup family

When the gateway proxy resolves a Backend through DNS, such as a Static Backend with a hostname or an AWS Lambda Backend, it looks up IPv4 and IPv6 addresses. By default, the gateway proxy prefers IPv4. The gateway proxy falls back to IPv6 only when DNS returns no IPv4 address.

To change the default for all Backends, set the KGW_DNS_LOOKUP_FAMILY environment variable on the controller. For example, use V6_ONLY in an IPv6-only cluster.

controller:
  extraEnv:
    KGW_DNS_LOOKUP_FAMILY: "V6_ONLY"

The following lookup mechanisms are supported:

The preferred values fall back to the other family only when DNS returns no addresses for the preferred family. They do not fall back when a connection fails. With ALL, most backends treat each address as a separate endpoint, so requests that are sent to an unreachable address family fail instead of falling back.

Listener bind address

By default, the control plane binds the listeners that it configures on the gateway proxy to the IPv6 wildcard address :: with IPv4 compatibility, so that they accept both IPv6 and IPv4 connections. This includes the readiness listener on port 8082 and the stats listener on port 9091 of the gateway proxy.

On nodes that disable the IPv6 kernel stack, such as with ipv6.disable=1, binding to :: can prevent the gateway proxy from starting. In this case, set the KGW_LISTENER_BIND_IPV6 environment variable to false on the controller. The listeners then bind to the IPv4 wildcard address 0.0.0.0.

controller:
  extraEnv:
    KGW_LISTENER_BIND_IPV6: "false"

The control plane writes the bind address into the gateway proxy’s bootstrap configuration. After you change the setting, restart the gateway proxy pods to pick up the new bootstrap configuration.

Controller admin server bind address

The kgateway controller runs an admin and debug server on port 9095. By default, the server binds to localhost and is only accessible from within the pod.

To access the admin and debug servers from your local machine, use the kubectl port-forward command to expose port 9095 on your local machine. No change of the bind address is required.

kubectl port-forward deployment/kgateway -n kgateway-system 9095:9095

If you need other pods in the cluster to reach the admin server directly, you can change the bind address by using the controller.admin.bindAddress Helm value or the KGW_ADMIN_BIND_ADDRESS environment variable. Setting bindAddress to 0.0.0.0 makes the server listen on all pod interfaces, so other pods can reach it at http://<pod-ip>:9095. To find the pod IP, run kubectl get pod -l app.kubernetes.io/name=kgateway -n kgateway-system -o wide.

controller:
  admin:
    bindAddress: 0.0.0.0