From 94a038035dc1b4c81adc0f68da182cc1966cbb8b Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 11:52:38 -0400 Subject: [PATCH 01/20] enhancements/ingress: Add Gateway API gateway customization EP Add enhancement proposal for spec.gatewayAPI.customGatewayClasses[] on the operator.openshift.io/v1alpha1 Ingress singleton, enabling flexible GatewayClass-level customization of service type, endpoint publishing strategy, and resource requirements. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 873 ++++++++++++++++++ 1 file changed, 873 insertions(+) create mode 100644 enhancements/ingress/gateway-api-gateway-customization.md diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md new file mode 100644 index 0000000000..783e033df3 --- /dev/null +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -0,0 +1,873 @@ +--- +title: gateway-api-gateway-customization +authors: + - "@gcs278" +reviewers: + - "@Miciah" + - "@candita" + - "@rikatz" +approvers: + - "@Miciah" +api-approvers: + - TBD +creation-date: 2026-09-01 +last-updated: 2026-09-01 +status: provisional +tracking-link: + - https://redhat.atlassian.net/browse/NE-2698 +see-also: + - "/enhancements/ingress/gateway-api-with-cluster-ingress-operator.md" + - "/enhancements/ingress/gateway-api-crd-management-mode.md" +replaces: [] +superseded-by: [] +--- + +# Gateway API Gateway Customization + +## Summary + +This enhancement extends the `operator.openshift.io/v1alpha1` `Ingress` +singleton (introduced by the Gateway API CRD management mode +enhancement) with a `spec.gatewayAPI.customGatewayClasses` field that +allows cluster administrators to declaratively define user-managed +GatewayClasses with customized service topology and proxy resource +configuration. For each entry, the Cluster Ingress Operator (CIO) +creates a GatewayClass resource and a corresponding GatewayClass +defaults ConfigMap that the Gateway API implementation uses when +provisioning the Gateway's backing Service and proxy deployment. + +This replaces a previous proposal that created a fixed set of +hardcoded GatewayClasses (`openshift-external`, `openshift-internal`, +`openshift-clusterip`). The `customGatewayClasses` approach is more +flexible, supports NodePort services and resource customization, and +does not require code changes to support new configurations. + +## Motivation + +When a user creates a Gateway using the `openshift-default` +GatewayClass, the Gateway API implementation always provisions an +external LoadBalancer service. There is currently no supported, +declarative way to: + +- Provision a Gateway backed by an internal LoadBalancer (for + traffic that should not be publicly exposed) +- Provision a Gateway backed by a ClusterIP service (for cluster- + internal traffic, or to front with an OCP Route on bare-metal + without a hardware load balancer) +- Provision a Gateway backed by a NodePort service (for bring-your- + own-load-balancer topologies on bare-metal or on-premises clusters) +- Configure `externalTrafficPolicy` to preserve source IP and avoid + cross-zone hops in stretched or zone-aware cluster topologies +- Tune resource requests and limits for the gateway proxy containers + +Users who need these configurations today must either manually patch +the Service after creation (fragile, unsupported), use a private +Gateway API infrastructure field (unsupported, private API), or open +a support exception. All three workarounds are unsuitable for +production use. + +### User Stories + +#### Story 1: Internal LoadBalancer Gateway + +As a cluster administrator, I want to create a GatewayClass that +provisions Gateways with an internal LoadBalancer service, including +the platform-specific internal annotations that CIO already applies +to internal IngressControllers, so that I can expose services only +within my cloud provider's private network without manual patching. + +#### Story 2: ClusterIP Gateway on Bare Metal + +As a cluster administrator running on bare metal without a hardware +load balancer, I want to create a GatewayClass that provisions +Gateways with a ClusterIP service so that I can front the Gateway +with an OCP Route, reusing the existing HAProxy-based ingress +infrastructure without introducing additional infrastructure +dependencies. + +#### Story 3: NodePort Gateway with Bring-Your-Own Load Balancer + +As a cluster administrator on an on-premises cluster, I want to +create a GatewayClass that provisions Gateways with a NodePort +service so that I can integrate with my existing hardware load +balancer without requiring a cloud provider or MetalLB. + +#### Story 4: Zone-Aware External Gateway + +As a cluster administrator running a cluster stretched across +multiple availability zones with BGP-based networking, I want to +configure `externalTrafficPolicy: Local` on my external Gateway +so that traffic arriving in a given zone is served by the Envoy +pod in that same zone, avoiding cross-zone hops and preserving +source IP. + +#### Story 5: Proxy Resource Tuning + +As a platform engineer managing production Gateway deployments, +I want to configure resource requests and limits for the gateway +proxy containers so that I can right-size the Envoy proxy deployment +for my workload's traffic volume without relying on implementation +defaults. + +#### Story 6: Operations at Scale + +As a platform engineer managing multiple clusters, I want Gateway +customization to be declarative and managed by CIO so that I can +rely on consistent service configurations across clusters, monitor +GatewayClass provisioning health through existing operator conditions, +and not perform manual configuration steps after cluster upgrade. + +### Goals + +- Allow cluster administrators to define user-managed GatewayClasses + with configurable service topology (LoadBalancer external/internal, + NodePort, ClusterIP) via the `Ingress` operator singleton API. +- Allow configuration of `externalTrafficPolicy` per GatewayClass for + LoadBalancer and NodePort service types. +- Allow configuration of proxy container resource requests and limits + per GatewayClass. +- CIO derives platform-specific service annotations (cloud provider + LB annotations, OVN local-with-fallback) automatically from the + cluster infrastructure, as it does for IngressControllers. +- CIO manages DNS for GatewayClasses with LoadBalancer service types, + and explicitly does not manage DNS for ClusterIP or NodePort types. +- Define a simplified ValidatingAdmissionPolicy that reserves the + `openshift-` GatewayClass name prefix for GatewayClasses using the + OpenShift controller name, without a hardcoded name allowlist. +- The existing `openshift-default` GatewayClass behavior is + unchanged. +- Support backporting to prior OCP releases that include the required + Gateway API implementation version with GatewayClass defaults + ConfigMap support. + +### Non-Goals + +- Customizing the `openshift-default` GatewayClass. It continues to + provision an external LoadBalancer with platform defaults. +- Exposing arbitrary Service annotations in the API. Users can + annotate Gateway resources directly; the Gateway API implementation + propagates annotations to the Service. +- Per-Gateway resource overrides. `customGatewayClasses` configures + class-level defaults applied to all Gateways referencing the class. +- Gateway API implementation configuration (logging format, Istio + control plane settings). These are managed at the implementation + level, not per GatewayClass. +- HPA configuration (deferred to a follow-up). +- Node placement configuration (deferred to a follow-up). +- `trafficDistribution` field on the Service. Analysis shows it is + effectively a no-op for the primary use cases: when + `externalTrafficPolicy: Local` is set, it is superseded; when not + set, it only affects in-cluster ClusterIP traffic to the Gateway, + which is not a typical access pattern. + +## Proposal + +The `GatewayAPIIngressConfig` struct in the +`operator.openshift.io/v1alpha1` `Ingress` resource is extended with +a `customGatewayClasses` field. Each entry declares a GatewayClass +that CIO will create and manage, along with the service topology +and proxy resource configuration for Gateways that reference it. + +For each entry, CIO: + +1. Creates or updates a `gateway.networking.k8s.io/v1` `GatewayClass` + resource with `controllerName: + openshift.io/gateway-controller/v1`. +2. Builds a GatewayClass defaults ConfigMap and passes it to the + sail-operator library (the same mechanism CIO uses for HPA + provisioning), which creates or updates the ConfigMap. The + ConfigMap is consumed by the Gateway API implementation to set + service type, platform-specific annotations, and + `externalTrafficPolicy` on Gateways referencing this class. +3. Manages DNS for the Gateway's listeners when `serviceType` is + `LoadBalancerService` (same as `openshift-default`). + +Platform-specific service annotations (AWS NLB, internal LB +annotations per cloud provider, OVN `local-with-fallback`) are +derived automatically from the cluster's infrastructure platform, +reusing the existing IngressController annotation-derivation logic. +Users do not need to specify platform annotations explicitly. + +The existing hardcoded `openshift-default` GatewayClass remains +unchanged. `customGatewayClasses` entries supplement it rather than +replace it. + +### Workflow Description + +**cluster administrator** is a human user responsible for managing +the cluster and Gateway infrastructure. + +**application developer** is a human user responsible for deploying +applications and creating HTTPRoutes. + +#### Creating an Internal LoadBalancer Gateway (example workflow) + +1. The cluster administrator edits the `Ingress/cluster` singleton: + + ```yaml + apiVersion: operator.openshift.io/v1alpha1 + kind: Ingress + metadata: + name: cluster + spec: + gatewayAPI: + customGatewayClasses: + - name: openshift-internal + endpointPublishingStrategy: + type: LoadBalancerService + loadBalancer: + scope: Internal + endpointTrafficPolicy: Local + ``` + +2. CIO detects the new entry and creates the `openshift-internal` + GatewayClass resource with `controllerName: + openshift.io/gateway-controller/v1`. +3. CIO builds the GatewayClass defaults ConfigMap with: + - `service.type: LoadBalancer` + - Platform internal LB annotation (e.g., + `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` + on AWS) + - `service.externalTrafficPolicy: Local` + - OVN `local-with-fallback` annotation (where applicable) +4. CIO passes the ConfigMap configuration to the sail-operator + library, which creates `openshift-internal-gwc-params` ConfigMap + in the `openshift-ingress` namespace. +5. The cluster administrator creates a Gateway: + + ```yaml + apiVersion: gateway.networking.k8s.io/v1 + kind: Gateway + metadata: + name: my-internal-gateway + namespace: my-namespace + spec: + gatewayClassName: openshift-internal + listeners: + - name: https + port: 443 + protocol: HTTPS + ``` + +6. The Gateway API implementation provisions an Envoy proxy Deployment + and an internal LoadBalancer Service using the defaults from the + ConfigMap. +7. CIO manages DNS for the Gateway's listeners. +8. The application developer creates HTTPRoutes attached to the + Gateway. + +#### Creating a ClusterIP Gateway (bare-metal / OCP Route topology) + +1. The cluster administrator adds a ClusterIP entry: + + ```yaml + spec: + gatewayAPI: + customGatewayClasses: + - name: openshift-clusterip + endpointPublishingStrategy: + type: ClusterIPService + ``` + +2. CIO creates the GatewayClass and a ConfigMap with + `service.type: ClusterIP`. +3. The cluster administrator creates a Gateway referencing + `openshift-clusterip`. +4. The Gateway API implementation provisions an Envoy proxy Deployment + and a ClusterIP Service. No cloud LB is created. +5. CIO does **not** provision DNS for ClusterIP gateways. +6. The cluster administrator creates an OCP Route pointing at the + ClusterIP Service to expose the Gateway externally via HAProxy. + +#### Creating a NodePort Gateway (bring-your-own load balancer) + +1. The cluster administrator adds a NodePort entry: + + ```yaml + spec: + gatewayAPI: + customGatewayClasses: + - name: openshift-nodeport + endpointPublishingStrategy: + type: NodePortService + nodePort: + endpointTrafficPolicy: Local + ``` + +2. CIO creates the GatewayClass and a ConfigMap with + `service.type: NodePort` and + `service.externalTrafficPolicy: Local`. +3. The cluster administrator creates a Gateway referencing + `openshift-nodeport`. +4. The Gateway API implementation provisions a NodePort Service. + The `spec.healthCheckNodePort` field is set automatically by + Kubernetes when `externalTrafficPolicy: Local`. +5. CIO does **not** provision DNS. +6. The cluster administrator configures their external load balancer + to use `spec.healthCheckNodePort` for health checking, ensuring + traffic is only sent to nodes with a local Envoy pod. + +```mermaid +sequenceDiagram + participant Admin as Cluster Admin + participant CIO as cluster-ingress-operator + participant Sail as sail-operator library + participant Impl as Gateway API implementation + participant Cloud as Cloud / Network + + Admin->>CIO: Edit Ingress/cluster (add customGatewayClasses entry) + CIO->>Impl: Create GatewayClass resource + CIO->>Sail: Pass service config as json.RawMessage + Sail->>Sail: Create GatewayClass defaults ConfigMap + Note over Sail: type, scope, ETP,
platform annotations + + Admin->>Impl: Create Gateway (references custom class) + Impl->>Impl: Deploy proxy Deployment + Impl->>Cloud: Create Service (per ConfigMap defaults) + CIO->>Cloud: Create DNS records (LB types only) + + Note over Admin,Cloud: No DNS for ClusterIP or NodePort types +``` + +### API Extensions + +This enhancement extends the existing +`operator.openshift.io/v1alpha1` `Ingress` CRD. It does not +introduce new CRD types. + +#### New fields on `GatewayAPIIngressConfig` + +```go +type GatewayAPIIngressConfig struct { + // managementMode (existing, unchanged) + ManagementMode GatewayAPIManagementMode `json:"managementMode,omitempty"` + + // customGatewayClasses defines a list of GatewayClasses that CIO + // creates and manages. For each entry, CIO creates a GatewayClass + // resource and a GatewayClass defaults ConfigMap that the Gateway + // API implementation uses when provisioning Gateways of that class. + // + // GatewayClass-level settings are defaults applied to all Gateways + // referencing the class. Individual Gateway instances cannot + // override class-level settings. + // + // +optional + // +listType=map + // +listMapKey=name + // +kubebuilder:validation:MaxItems=16 + CustomGatewayClasses []CustomGatewayClassConfig `json:"customGatewayClasses,omitempty"` +} +``` + +#### `CustomGatewayClassConfig` + +```go +// CustomGatewayClassConfig defines a user-managed GatewayClass and +// its associated configuration. +type CustomGatewayClassConfig struct { + // name is the name of the GatewayClass to create. Must use the + // "openshift-" prefix. The name "openshift-default" is reserved + // and may not be used here. + // + // +required + // +kubebuilder:validation:MinLength=12 + // +kubebuilder:validation:Pattern=`^openshift-[a-z0-9]([a-z0-9\-]*[a-z0-9])?$` + // +kubebuilder:validation:XValidation:rule="self != 'openshift-default'",message="openshift-default is reserved" + Name string `json:"name"` + + // endpointPublishingStrategy defines how the Gateway's endpoints + // are published to the network. When omitted, defaults to an + // external LoadBalancer with platform defaults. + // + // +optional + EndpointPublishingStrategy *GatewayEndpointPublishingStrategy `json:"endpointPublishingStrategy,omitempty"` + + // resources configures resource requests and limits for the + // gateway proxy containers. When omitted, the Gateway API + // implementation's defaults apply. + // + // +optional + Resources *corev1.ResourceRequirements `json:"resources,omitempty"` +} +``` + +#### `GatewayEndpointPublishingStrategy` + +```go +// GatewayEndpointPublishingStrategy defines how a Gateway's endpoints +// are published. It is a discriminated union on Type. +// +// +union +// +kubebuilder:validation:XValidation:rule="self.type != 'LoadBalancerService' || has(self.loadBalancer)",message="loadBalancer is required when type is LoadBalancerService" +// +kubebuilder:validation:XValidation:rule="self.type == 'LoadBalancerService' || !has(self.loadBalancer)",message="loadBalancer is only valid when type is LoadBalancerService" +// +kubebuilder:validation:XValidation:rule="self.type == 'NodePortService' || !has(self.nodePort)",message="nodePort is only valid when type is NodePortService" +type GatewayEndpointPublishingStrategy struct { + // type is the publishing strategy to use. + // + // LoadBalancerService provisions a cloud or hardware LoadBalancer + // Service in front of the gateway proxy. CIO applies + // platform-specific annotations and manages DNS automatically. + // + // NodePortService provisions a Kubernetes NodePort Service. + // No DNS is managed by CIO. The cluster administrator is + // responsible for configuring an external load balancer and, when + // endpointTrafficPolicy is Local, directing it to use the + // Service's healthCheckNodePort for health checking. + // + // ClusterIPService provisions a ClusterIP Service. The gateway + // is accessible only within the cluster. No DNS is managed by + // CIO. Useful for fronting with an OCP Route on bare-metal + // clusters without a hardware or software load balancer. + // + // +unionDiscriminator + // +required + // +kubebuilder:validation:Enum=LoadBalancerService;NodePortService;ClusterIPService + Type GatewayEndpointPublishingStrategyType `json:"type"` + + // loadBalancer holds parameters for the load balancer. + // Present only when type is LoadBalancerService. + // + // +optional + LoadBalancer *GatewayLoadBalancerStrategy `json:"loadBalancer,omitempty"` + + // nodePort holds parameters for the NodePort service. + // Present only when type is NodePortService. + // + // +optional + NodePort *GatewayNodePortStrategy `json:"nodePort,omitempty"` +} + +// GatewayEndpointPublishingStrategyType is the publishing strategy +// for a Gateway. +type GatewayEndpointPublishingStrategyType string + +const ( + // GatewayStrategyLoadBalancerService provisions a LoadBalancer + // Service and manages DNS. + GatewayStrategyLoadBalancerService GatewayEndpointPublishingStrategyType = "LoadBalancerService" + + // GatewayStrategyNodePortService provisions a NodePort Service. + // No DNS is managed by CIO. + GatewayStrategyNodePortService GatewayEndpointPublishingStrategyType = "NodePortService" + + // GatewayStrategyClusterIPService provisions a ClusterIP Service. + // No DNS is managed by CIO. + GatewayStrategyClusterIPService GatewayEndpointPublishingStrategyType = "ClusterIPService" +) + +// GatewayLoadBalancerStrategy holds parameters for a LoadBalancer- +// backed GatewayClass. +type GatewayLoadBalancerStrategy struct { + // scope indicates whether the load balancer is exposed + // externally or only within the cloud provider's private + // network. + // + // External provisions a public-facing load balancer with + // platform-specific external annotations. + // + // Internal provisions an internal load balancer with + // platform-specific internal annotations (e.g., + // service.beta.kubernetes.io/aws-load-balancer-internal). + // + // +required + Scope LoadBalancerScope `json:"scope"` // reuses operator/v1 type + + // endpointTrafficPolicy controls how external traffic is + // distributed to gateway proxy pods. + // + // Local routes external traffic only to proxy pods on the same + // node that receives the traffic. This preserves the source IP + // address and avoids cross-node (and cross-zone) hops. + // The load balancer must use the Service's healthCheckNodePort + // to health-check nodes; CIO configures this automatically for + // cloud load balancers via platform annotations. The OVN + // local-with-fallback annotation is also set when Local is used, + // ensuring traffic is not dropped when no local pod is available + // during rolling updates. + // + // Cluster allows the implementation to route external traffic to + // any proxy pod in the cluster, performing SNAT. Source IP is + // not preserved. + // + // When omitted, defaults to Local on most platforms. IBM Cloud + // defaults to Cluster due to platform constraints. + // + // +optional + EndpointTrafficPolicy *GatewayEndpointTrafficPolicy `json:"endpointTrafficPolicy,omitempty"` +} + +// GatewayNodePortStrategy holds parameters for a NodePort-backed +// GatewayClass. +type GatewayNodePortStrategy struct { + // endpointTrafficPolicy controls how external traffic is + // distributed to gateway proxy pods. + // + // Local routes external traffic only to proxy pods on the same + // node that receives the traffic. This preserves the source IP + // address and avoids cross-node hops. When Local is used, the + // external load balancer MUST be configured to health-check + // nodes using Service.spec.healthCheckNodePort. If the load + // balancer does not perform this health check, traffic will be + // dropped on nodes that have no local proxy pod. + // + // Cluster allows routing to any proxy pod (with SNAT). Source + // IP is not preserved, but load distribution is even regardless + // of pod placement. + // + // When omitted, this field is not set (the implementation + // default applies). Unlike LoadBalancerService, there is no + // platform-specific default because NodePort is used in + // bring-your-own-load-balancer scenarios where the external LB + // capabilities vary widely. + // + // +optional + EndpointTrafficPolicy *GatewayEndpointTrafficPolicy `json:"endpointTrafficPolicy,omitempty"` +} + +// GatewayEndpointTrafficPolicy is the externalTrafficPolicy for a +// Gateway's Service. +// +kubebuilder:validation:Enum=Local;Cluster +type GatewayEndpointTrafficPolicy string + +const ( + // GatewayEndpointTrafficPolicyLocal routes external traffic only + // to pods on the same node, preserving source IP. + GatewayEndpointTrafficPolicyLocal GatewayEndpointTrafficPolicy = "Local" + + // GatewayEndpointTrafficPolicyCluster routes external traffic to + // any pod, performing SNAT. + GatewayEndpointTrafficPolicyCluster GatewayEndpointTrafficPolicy = "Cluster" +) +``` + +#### ValidatingAdmissionPolicy Changes + +The existing VAP for GatewayClass naming is simplified to two rules: + +1. A GatewayClass with `controllerName: + openshift.io/gateway-controller/v1` MUST have a name prefixed + with `openshift-`. +2. A GatewayClass with the `openshift-` name prefix MUST use + `controllerName: openshift.io/gateway-controller/v1`. + +The previous allowlist of exactly four names is removed. CIO creates +GatewayClasses for `openshift-default` (hardcoded) and any entry in +`customGatewayClasses`. A GatewayClass manually created by a user +with the `openshift-` prefix and the OpenShift controller name is +permitted but not managed by CIO (no defaults ConfigMap is created +for it). + +#### YAML examples + +```yaml +# External LoadBalancer with zone-aware routing +apiVersion: operator.openshift.io/v1alpha1 +kind: Ingress +metadata: + name: cluster +spec: + gatewayAPI: + customGatewayClasses: + - name: openshift-external + endpointPublishingStrategy: + type: LoadBalancerService + loadBalancer: + scope: External + endpointTrafficPolicy: Local + + # Internal LoadBalancer (default ETP for platform) + - name: openshift-internal + endpointPublishingStrategy: + type: LoadBalancerService + loadBalancer: + scope: Internal + + # ClusterIP for bare-metal + OCP Route topology + - name: openshift-clusterip + endpointPublishingStrategy: + type: ClusterIPService + + # NodePort for bring-your-own load balancer + - name: openshift-nodeport + endpointPublishingStrategy: + type: NodePortService + nodePort: + endpointTrafficPolicy: Local + + # High-traffic class with tuned proxy resources + - name: openshift-production + endpointPublishingStrategy: + type: LoadBalancerService + loadBalancer: + scope: External + endpointTrafficPolicy: Local + resources: + requests: + cpu: 500m + memory: 512Mi + limits: + memory: 1Gi +``` + +### Topology Considerations + +#### Hypershift / Hosted Control Planes + +This enhancement applies to Hypershift without additional +considerations beyond existing Gateway API support. GatewayClass +provisioning and the Gateway API implementation run in the guest +cluster. Platform-specific annotations in the GatewayClass defaults +ConfigMap are derived from the guest cluster's infrastructure +platform. + +#### Standalone Clusters + +Directly applicable. Platform annotation derivation uses the same +logic as CIO's IngressController service provisioning. + +#### Single-node Deployments or MicroShift + +`ClusterIPService` GatewayClasses are particularly useful for SNO +deployments where cloud load balancers are not available. No +additional resource consumption beyond what a Gateway already requires. + +MicroShift has its own Gateway API support and does not use CIO, so +this enhancement does not apply to MicroShift. + +#### OpenShift Kubernetes Engine + +Applicable on OKE clusters where Gateway API is available, subject +to the same OSSM version dependency as for standard OCP. + +### Implementation Details/Notes/Constraints + +#### GatewayClass Defaults ConfigMap Mechanism + +The GatewayClass defaults ConfigMap is an Istio mechanism documented +at https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/#gatewayclass-defaults. +One ConfigMap is created per GatewayClass entry (not per Gateway +instance). All Gateways referencing a given GatewayClass share the +same defaults. + +CIO passes service configuration as `json.RawMessage` to the +sail-operator library, which creates the ConfigMap. This is the same +mechanism CIO uses for HPA provisioning. The required support in the +Gateway API implementation was added in OSSM >= 3.2.4 and >= 3.3.1 +(sail-operator#1465). Backport to OCP versions using OSSM 3.0.x or +3.1.x is not possible without an upstream cherry-pick. + +#### Platform Annotation Derivation + +CIO reuses its existing IngressController annotation-derivation logic +to populate the GatewayClass defaults ConfigMap with the correct +platform-specific service annotations. The derivation is based on: + +- `scope: External` or `scope: Internal` (from the API) +- The cluster's `infrastructure.config.openshift.io/cluster` platform + type (AWS, Azure, GCP, IBM, OpenStack, etc.) +- `endpointTrafficPolicy: Local` triggers the OVN + `traffic-policy.network.alpha.openshift.io/local-with-fallback: ""` + annotation on platforms where it applies. + +#### DNS Management + +| `type` | CIO manages DNS | +|----------------------|-----------------| +| `LoadBalancerService`| Yes | +| `NodePortService` | No | +| `ClusterIPService` | No | + +#### Deletion Semantics + +When an entry is removed from `customGatewayClasses`, CIO removes +the GatewayClass defaults ConfigMap but does not delete the +GatewayClass resource itself. Deleting the GatewayClass while +Gateways reference it would orphan running workloads. CIO sets a +condition on the GatewayClass (`Accepted: False`, reason: +`NoLongerManaged`) and records a corresponding condition on +`Ingress/cluster` status to notify the administrator. The +administrator is responsible for deleting the GatewayClass after +migrating or deleting dependent Gateways. + +#### Feature Gate + +This enhancement is gated behind a new feature gate +`CustomGatewayClasses`, initially added to `TechPreviewNoUpgrade` +in `github.com/openshift/api/features/features.go`. The feature gate +marker `+openshift:enable:FeatureGate=CustomGatewayClasses` is added +to the relevant API fields. + +### Risks and Mitigations + +**Risk**: OSSM version dependency. The GatewayClass defaults ConfigMap +mechanism requires OSSM >= 3.2.4 or >= 3.3.1. + +**Mitigation**: CIO checks the OSSM version at reconciliation time and +sets a `Degraded` condition on `Ingress/cluster` with a clear message +if the version requirement is not met. The feature gate prevents the +API from being available until the cluster meets the version +requirement on supported OCP releases. + +**Risk**: Uneven load distribution with `endpointTrafficPolicy: Local` +on NodePort GatewayClasses when pods are not distributed evenly +across nodes. + +**Mitigation**: The API documentation for `NodePortService` + +`endpointTrafficPolicy: Local` explicitly states the health check +NodePort requirement and the load-distribution trade-off. Future work +(topology spread constraints via `nodePlacement`) will allow operators +to ensure even pod distribution. + +**Risk**: A user removes a `customGatewayClasses` entry while Gateways +still reference the GatewayClass, expecting the GatewayClass to be +deleted. + +**Mitigation**: CIO does not delete the GatewayClass on entry removal +(documented above). A clear condition is set to guide the +administrator. Future work may add a deletion policy field. + +### Drawbacks + +- `customGatewayClasses` settings are class-level defaults applied to + all Gateways referencing the class. A small development Gateway and + a high-traffic production Gateway referencing the same class share + the same resource defaults and service topology. Administrators must + create separate GatewayClasses for different workload tiers. +- The `openshift-` prefix requirement for all CIO-managed GatewayClass + names may feel restrictive, but it is necessary to ensure CIO's + controller name association is valid and that the prefix reservation + VAP can enforce it consistently. + +## Alternatives (Not Implemented) + +### Hardcoded GatewayClasses + +The prior proposal created three fixed GatewayClasses +(`openshift-external`, `openshift-internal`, `openshift-clusterip`). +This approach was rejected because: + +- A hardcoded allowlist requires a code change for every new + service configuration combination. +- NodePort support and resource customization were explicitly deferred + as open questions with no clear path forward. +- `externalTrafficPolicy` was not user-configurable; only the + platform-derived default applied. +- The VAP allowlist with four fixed names is a permanent maintenance + burden. + +## Open Questions + +1. **Status per GatewayClass**: Should `Ingress/cluster` status + include a `gatewayClasses[]` slice with one entry per + `customGatewayClasses` entry, or is it sufficient to rely on + the GatewayClass resource's own `.status.conditions`? The latter + avoids duplicating status but requires administrators to query + GatewayClass resources separately. + +2. **Default `endpointPublishingStrategy`**: When + `endpointPublishingStrategy` is omitted from an entry, should the + default be `LoadBalancerService` with `scope: External` (matching + `openshift-default` behavior), or should it be required? + +3. **Deletion policy field**: Should we add a + `deletionPolicy: Delete|Retain` field to `CustomGatewayClassConfig` + to give administrators explicit control over whether CIO deletes + the GatewayClass when the entry is removed? + +## Test Plan + + + +Tests must include: + +- `[OCPFeatureGate:CustomGatewayClasses]` label for the feature gate +- `[Jira:"OCP/Network Ingress"]` label for the component +- Unit tests for CIO controller logic (ConfigMap content derivation, + platform annotation selection per service type and scope) +- E2E tests for each `endpointPublishingStrategy.type`: + - `LoadBalancerService` External and Internal on supported cloud + platforms + - `NodePortService` on applicable platforms + - `ClusterIPService` with OCP Route fronting +- E2E tests verifying `externalTrafficPolicy: Local` behavior + (source IP preservation, health check NodePort set) +- E2E tests verifying `resources` propagation to the proxy container +- Upgrade tests: existing `openshift-default` GatewayClass unaffected + after upgrade +- Tests must run on all supported platforms: AWS, Azure, GCP, + vSphere, Baremetal + +## Graduation Criteria + +### Dev Preview -> Tech Preview + +- E2E tests passing on at least AWS and bare-metal platforms +- OSSM version validation implemented and tested +- End-user documentation draft +- Minimum 5 tests, 7 runs per week, 95% pass rate + +### Tech Preview -> GA + +- E2E tests on all supported platforms with ≥14 runs per platform +- Upgrade and downgrade testing complete +- SLIs defined and telemetry collected +- User-facing documentation complete in openshift-docs +- Deletion semantics validated in upgrade scenarios + +## Upgrade / Downgrade Strategy + +**Upgrade**: The `customGatewayClasses` field defaults to empty. No +existing GatewayClass or Gateway resources are modified on upgrade. +`openshift-default` behavior is unchanged. + +**Downgrade**: If a cluster is downgraded to a version that does not +support `customGatewayClasses`, CIO stops reconciling the entries but +does not delete any GatewayClass or ConfigMap resources it previously +created. Those resources remain functional until an administrator +manually removes them. This is consistent with how CIO handles other +feature gate removals. + +## Version Skew Strategy + +The GatewayClass defaults ConfigMap mechanism requires the Gateway API +implementation to be at a minimum version (OSSM >= 3.2.4 / >= 3.3.1). +CIO detects the implementation version at reconciliation time and +reports a condition rather than attempting to create ConfigMaps that +the implementation version cannot consume. No version skew between +CIO and the kube-apiserver is anticipated, as this enhancement adds +fields to an existing CRD. + +## Operational Aspects of API Extensions + +The `customGatewayClasses` field is added to the existing +`Ingress/cluster` CRD. The CRD is managed by CIO and protected by an +existing VAP. The field is gated by a feature gate and therefore +invisible to clusters that have not enabled `TechPreviewNoUpgrade`. + +Failure modes: + +- **OSSM version too old**: CIO sets `Degraded` condition on + `Ingress/cluster` with reason `GatewayClassDefaultsUnsupported`. + Existing Gateways continue to function; new `customGatewayClasses` + entries are not reconciled. +- **ConfigMap creation failure**: CIO retries and degrades with a + condition. The GatewayClass resource may exist without a ConfigMap; + Gateways referencing it will use implementation defaults rather than + the intended configuration. + +## Support Procedures + +- Check `Ingress/cluster` status conditions for + `GatewayClassDefaultsUnsupported` or `GatewayClassReconcileFailed`. +- Check the GatewayClass resource's `.status.conditions` for + `Accepted` status. +- CIO logs include structured events for each `customGatewayClasses` + reconciliation pass. +- To disable: remove entries from `customGatewayClasses`. CIO stops + managing the ConfigMap; the GatewayClass is retained (see Deletion + Semantics). + +## Infrastructure Needed + +No new infrastructure is needed. The implementation uses the existing +sail-operator library integration already used for HPA provisioning. From cc99738f40fdb43cd1d303a7fb057a2fa1f188d9 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 15:53:31 -0400 Subject: [PATCH 02/20] enhancements/ingress: Redesign gateway customization around GatewayParameters CRD Switch from spec.gatewayAPI.customGatewayClasses[] on the Ingress singleton to a new cluster-scoped GatewayParameters CRD referenced via GatewayClass.spec.parametersRef. CIO reconciles it into the OSSM defaults ConfigMap. Focus EP on ClusterIP and ETP:Local; enumerate resources and nodePlacement as future extensions. Add explicit non-goals for the Istio ClusterIP alpha annotation and direct ConfigMap use. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 987 ++++++------------ 1 file changed, 330 insertions(+), 657 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 783e033df3..9e96da57ae 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -11,7 +11,7 @@ approvers: api-approvers: - TBD creation-date: 2026-09-01 -last-updated: 2026-09-01 +last-updated: 2026-09-03 status: provisional tracking-link: - https://redhat.atlassian.net/browse/NE-2698 @@ -26,468 +26,298 @@ superseded-by: [] ## Summary -This enhancement extends the `operator.openshift.io/v1alpha1` `Ingress` -singleton (introduced by the Gateway API CRD management mode -enhancement) with a `spec.gatewayAPI.customGatewayClasses` field that -allows cluster administrators to declaratively define user-managed -GatewayClasses with customized service topology and proxy resource -configuration. For each entry, the Cluster Ingress Operator (CIO) -creates a GatewayClass resource and a corresponding GatewayClass -defaults ConfigMap that the Gateway API implementation uses when -provisioning the Gateway's backing Service and proxy deployment. - -This replaces a previous proposal that created a fixed set of -hardcoded GatewayClasses (`openshift-external`, `openshift-internal`, -`openshift-clusterip`). The `customGatewayClasses` approach is more -flexible, supports NodePort services and resource customization, and -does not require code changes to support new configurations. +This enhancement introduces a new `GatewayParameters` CRD in the +`operator.openshift.io` API group. A `GatewayClass` references a +`GatewayParameters` instance via `spec.parametersRef`, and the Cluster +Ingress Operator (CIO) reconciles it into an OSSM GatewayClass defaults +ConfigMap. This provides a first-class, implementation-agnostic API for +customizing how Gateway API implementations provision the backing +Kubernetes Service and proxy deployment for a GatewayClass. + +This EP implements two use cases: ClusterIP service type and +`externalTrafficPolicy: Local`. The `GatewayParameters` CRD is designed +to be extended with additional fields (resource requests, node +placement) in follow-on work without API changes to the GatewayClass +or Gateway resources. ## Motivation -When a user creates a Gateway using the `openshift-default` -GatewayClass, the Gateway API implementation always provisions an -external LoadBalancer service. There is currently no supported, -declarative way to: - -- Provision a Gateway backed by an internal LoadBalancer (for - traffic that should not be publicly exposed) -- Provision a Gateway backed by a ClusterIP service (for cluster- - internal traffic, or to front with an OCP Route on bare-metal - without a hardware load balancer) -- Provision a Gateway backed by a NodePort service (for bring-your- - own-load-balancer topologies on bare-metal or on-premises clusters) -- Configure `externalTrafficPolicy` to preserve source IP and avoid - cross-zone hops in stretched or zone-aware cluster topologies -- Tune resource requests and limits for the gateway proxy containers - -Users who need these configurations today must either manually patch -the Service after creation (fragile, unsupported), use a private -Gateway API infrastructure field (unsupported, private API), or open -a support exception. All three workarounds are unsuitable for -production use. +When a user creates a Gateway using the `openshift-default` GatewayClass, +the Gateway API implementation always provisions an external LoadBalancer +service. There is currently no supported, declarative way to: -### User Stories +- Use a ClusterIP service for cluster-internal traffic or to front with + an OCP Route on bare-metal without a hardware load balancer +- Configure `externalTrafficPolicy: Local` to preserve source IP and + avoid cross-zone hops in zone-aware or BGP-based cluster topologies -#### Story 1: Internal LoadBalancer Gateway +Users who need these configurations today must use the Istio ClusterIP +alpha annotation or manually patch the Service after creation — both +unsupported, fragile approaches that break on reconciliation or upgrade. -As a cluster administrator, I want to create a GatewayClass that -provisions Gateways with an internal LoadBalancer service, including -the platform-specific internal annotations that CIO already applies -to internal IngressControllers, so that I can expose services only -within my cloud provider's private network without manual patching. +### User Stories -#### Story 2: ClusterIP Gateway on Bare Metal +#### Story 1: ClusterIP Gateway on Bare Metal As a cluster administrator running on bare metal without a hardware -load balancer, I want to create a GatewayClass that provisions -Gateways with a ClusterIP service so that I can front the Gateway -with an OCP Route, reusing the existing HAProxy-based ingress -infrastructure without introducing additional infrastructure -dependencies. - -#### Story 3: NodePort Gateway with Bring-Your-Own Load Balancer - -As a cluster administrator on an on-premises cluster, I want to -create a GatewayClass that provisions Gateways with a NodePort -service so that I can integrate with my existing hardware load -balancer without requiring a cloud provider or MetalLB. - -#### Story 4: Zone-Aware External Gateway - -As a cluster administrator running a cluster stretched across -multiple availability zones with BGP-based networking, I want to -configure `externalTrafficPolicy: Local` on my external Gateway -so that traffic arriving in a given zone is served by the Envoy -pod in that same zone, avoiding cross-zone hops and preserving -source IP. +load balancer, I want to create a GatewayClass that provisions Gateways +with a ClusterIP service so that I can front the Gateway with an OCP +Route using the existing HAProxy ingress infrastructure. -#### Story 5: Proxy Resource Tuning +#### Story 2: Zone-Aware External Gateway with ETP Local -As a platform engineer managing production Gateway deployments, -I want to configure resource requests and limits for the gateway -proxy containers so that I can right-size the Envoy proxy deployment -for my workload's traffic volume without relying on implementation -defaults. +As a cluster administrator running a cluster across multiple +availability zones with BGP-based networking, I want to configure +`externalTrafficPolicy: Local` on my GatewayClass so that traffic +arriving in a given zone is served by the proxy pod in that same zone, +avoiding cross-zone hops and preserving source IP. -#### Story 6: Operations at Scale +#### Story 3: Internal LoadBalancer Gateway -As a platform engineer managing multiple clusters, I want Gateway -customization to be declarative and managed by CIO so that I can -rely on consistent service configurations across clusters, monitor -GatewayClass provisioning health through existing operator conditions, -and not perform manual configuration steps after cluster upgrade. +As a cluster administrator, I want to create a GatewayClass that +provisions Gateways with an internal LoadBalancer service, including +the correct platform-specific internal annotations, so that I can +expose services only within my cloud provider's private network. ### Goals -- Allow cluster administrators to define user-managed GatewayClasses - with configurable service topology (LoadBalancer external/internal, - NodePort, ClusterIP) via the `Ingress` operator singleton API. -- Allow configuration of `externalTrafficPolicy` per GatewayClass for - LoadBalancer and NodePort service types. -- Allow configuration of proxy container resource requests and limits - per GatewayClass. -- CIO derives platform-specific service annotations (cloud provider - LB annotations, OVN local-with-fallback) automatically from the +- Introduce a `GatewayParameters` CRD that a `GatewayClass` references + via `spec.parametersRef` to configure its service topology. +- CIO reconciles `GatewayParameters` → OSSM GatewayClass defaults + ConfigMap (labeled `gateway.istio.io/defaults-for-class`), translating + the OpenShift API into the implementation-specific ConfigMap format. +- Implement `endpointPublishingStrategy` for service type + (LoadBalancer/NodePort/ClusterIP) and `externalTrafficPolicy` + (Local/Cluster). +- CIO derives platform-specific service annotations automatically from cluster infrastructure, as it does for IngressControllers. -- CIO manages DNS for GatewayClasses with LoadBalancer service types, - and explicitly does not manage DNS for ClusterIP or NodePort types. -- Define a simplified ValidatingAdmissionPolicy that reserves the - `openshift-` GatewayClass name prefix for GatewayClasses using the - OpenShift controller name, without a hardcoded name allowlist. -- The existing `openshift-default` GatewayClass behavior is - unchanged. -- Support backporting to prior OCP releases that include the required - Gateway API implementation version with GatewayClass defaults - ConfigMap support. +- CIO manages DNS for GatewayClasses with LoadBalancer service type; + not for ClusterIP or NodePort. +- The existing `openshift-default` GatewayClass is unchanged. +- Design the CRD for future extension (resources, nodePlacement) without + breaking API changes. ### Non-Goals -- Customizing the `openshift-default` GatewayClass. It continues to - provision an external LoadBalancer with platform defaults. -- Exposing arbitrary Service annotations in the API. Users can - annotate Gateway resources directly; the Gateway API implementation - propagates annotations to the Service. -- Per-Gateway resource overrides. `customGatewayClasses` configures - class-level defaults applied to all Gateways referencing the class. -- Gateway API implementation configuration (logging format, Istio - control plane settings). These are managed at the implementation - level, not per GatewayClass. -- HPA configuration (deferred to a follow-up). -- Node placement configuration (deferred to a follow-up). -- `trafficDistribution` field on the Service. Analysis shows it is - effectively a no-op for the primary use cases: when - `externalTrafficPolicy: Local` is set, it is superseded; when not - set, it only affects in-cluster ClusterIP traffic to the Gateway, - which is not a typical access pattern. +- Customizing the `openshift-default` GatewayClass. +- Exposing the OSSM GatewayClass defaults ConfigMap (`gateway.istio.io/defaults-for-class`) + as a supported API for end users. It is an implementation detail owned + and managed exclusively by CIO. +- Using or supporting the Istio ClusterIP alpha annotation + (`networking.istio.io/service-type`) as a supported mechanism for + service type customization. This annotation is an unsupported private + API and is superseded by this enhancement. +- Per-Gateway resource overrides. `GatewayParameters` configures + class-level defaults shared by all Gateways referencing the class. +- Gateway API implementation configuration (logging, control plane + settings). +- `trafficDistribution` on the Service. When `externalTrafficPolicy: Local` + is set it is superseded; for ClusterIP traffic to the Gateway it is + a no-op for the primary use cases. +- HPA configuration (deferred to a follow-on). +- Node placement and tolerations (deferred to a follow-on). ## Proposal -The `GatewayAPIIngressConfig` struct in the -`operator.openshift.io/v1alpha1` `Ingress` resource is extended with -a `customGatewayClasses` field. Each entry declares a GatewayClass -that CIO will create and manage, along with the service topology -and proxy resource configuration for Gateways that reference it. - -For each entry, CIO: - -1. Creates or updates a `gateway.networking.k8s.io/v1` `GatewayClass` - resource with `controllerName: - openshift.io/gateway-controller/v1`. -2. Builds a GatewayClass defaults ConfigMap and passes it to the - sail-operator library (the same mechanism CIO uses for HPA - provisioning), which creates or updates the ConfigMap. The - ConfigMap is consumed by the Gateway API implementation to set - service type, platform-specific annotations, and - `externalTrafficPolicy` on Gateways referencing this class. -3. Manages DNS for the Gateway's listeners when `serviceType` is - `LoadBalancerService` (same as `openshift-default`). - -Platform-specific service annotations (AWS NLB, internal LB -annotations per cloud provider, OVN `local-with-fallback`) are -derived automatically from the cluster's infrastructure platform, -reusing the existing IngressController annotation-derivation logic. -Users do not need to specify platform annotations explicitly. - -The existing hardcoded `openshift-default` GatewayClass remains -unchanged. `customGatewayClasses` entries supplement it rather than -replace it. +A new cluster-scoped CRD `GatewayParameters` +(`operator.openshift.io/v1alpha1`) is introduced. A cluster +administrator creates a `GatewayClass` with `spec.parametersRef` +pointing to a `GatewayParameters` instance, and creates the +`GatewayParameters` CR to configure the desired service topology. -### Workflow Description +CIO watches GatewayClasses whose `spec.controllerName` matches the +OpenShift controller name. For any such GatewayClass with a +`spec.parametersRef` pointing to a `GatewayParameters` CR, CIO: + +1. Reads the `GatewayParameters` CR. +2. Derives platform-specific service annotations from the cluster's + `infrastructure.config.openshift.io/cluster` resource. +3. Creates or updates a ConfigMap in the `openshift-ingress` namespace + with the label `gateway.istio.io/defaults-for-class: `. + OSSM reads this ConfigMap to apply service type, annotations, and ETP + to all Gateways referencing the class. +4. Manages DNS for `LoadBalancerService` type only. -**cluster administrator** is a human user responsible for managing -the cluster and Gateway infrastructure. +CIO never mutates the user's `Gateway` or `GatewayClass` resources. -**application developer** is a human user responsible for deploying -applications and creating HTTPRoutes. +### Workflow Description -#### Creating an Internal LoadBalancer Gateway (example workflow) +#### ClusterIP Gateway (bare-metal / OCP Route topology) -1. The cluster administrator edits the `Ingress/cluster` singleton: +1. The cluster administrator creates a `GatewayParameters` CR: ```yaml apiVersion: operator.openshift.io/v1alpha1 - kind: Ingress + kind: GatewayParameters metadata: - name: cluster + name: clusterip-params spec: - gatewayAPI: - customGatewayClasses: - - name: openshift-internal - endpointPublishingStrategy: - type: LoadBalancerService - loadBalancer: - scope: Internal - endpointTrafficPolicy: Local + endpointPublishingStrategy: + type: ClusterIPService ``` -2. CIO detects the new entry and creates the `openshift-internal` - GatewayClass resource with `controllerName: - openshift.io/gateway-controller/v1`. -3. CIO builds the GatewayClass defaults ConfigMap with: - - `service.type: LoadBalancer` - - Platform internal LB annotation (e.g., - `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` - on AWS) - - `service.externalTrafficPolicy: Local` - - OVN `local-with-fallback` annotation (where applicable) -4. CIO passes the ConfigMap configuration to the sail-operator - library, which creates `openshift-internal-gwc-params` ConfigMap - in the `openshift-ingress` namespace. -5. The cluster administrator creates a Gateway: +2. The cluster administrator creates a `GatewayClass` referencing it: ```yaml apiVersion: gateway.networking.k8s.io/v1 - kind: Gateway + kind: GatewayClass metadata: - name: my-internal-gateway - namespace: my-namespace + name: openshift-clusterip spec: - gatewayClassName: openshift-internal - listeners: - - name: https - port: 443 - protocol: HTTPS + controllerName: openshift.io/gateway-controller/v1 + parametersRef: + group: operator.openshift.io + kind: GatewayParameters + name: clusterip-params ``` -6. The Gateway API implementation provisions an Envoy proxy Deployment - and an internal LoadBalancer Service using the defaults from the - ConfigMap. -7. CIO manages DNS for the Gateway's listeners. -8. The application developer creates HTTPRoutes attached to the - Gateway. - -#### Creating a ClusterIP Gateway (bare-metal / OCP Route topology) - -1. The cluster administrator adds a ClusterIP entry: +3. CIO creates a ConfigMap in `openshift-ingress`: ```yaml - spec: - gatewayAPI: - customGatewayClasses: - - name: openshift-clusterip - endpointPublishingStrategy: - type: ClusterIPService + metadata: + labels: + gateway.istio.io/defaults-for-class: openshift-clusterip + data: + service: | + spec: + type: ClusterIP ``` -2. CIO creates the GatewayClass and a ConfigMap with - `service.type: ClusterIP`. -3. The cluster administrator creates a Gateway referencing - `openshift-clusterip`. -4. The Gateway API implementation provisions an Envoy proxy Deployment - and a ClusterIP Service. No cloud LB is created. -5. CIO does **not** provision DNS for ClusterIP gateways. -6. The cluster administrator creates an OCP Route pointing at the +4. The cluster administrator creates a Gateway referencing + `openshift-clusterip`. OSSM provisions an Envoy Deployment and a + ClusterIP Service. No cloud LB is created, no DNS is managed. + +5. The cluster administrator creates an OCP Route pointing at the ClusterIP Service to expose the Gateway externally via HAProxy. -#### Creating a NodePort Gateway (bring-your-own load balancer) +#### Zone-Aware LoadBalancer with ETP Local -1. The cluster administrator adds a NodePort entry: +1. The cluster administrator creates a `GatewayParameters` CR: ```yaml + apiVersion: operator.openshift.io/v1alpha1 + kind: GatewayParameters + metadata: + name: external-zone-aware spec: - gatewayAPI: - customGatewayClasses: - - name: openshift-nodeport - endpointPublishingStrategy: - type: NodePortService - nodePort: - endpointTrafficPolicy: Local + endpointPublishingStrategy: + type: LoadBalancerService + loadBalancer: + scope: External + endpointTrafficPolicy: Local ``` -2. CIO creates the GatewayClass and a ConfigMap with - `service.type: NodePort` and - `service.externalTrafficPolicy: Local`. -3. The cluster administrator creates a Gateway referencing - `openshift-nodeport`. -4. The Gateway API implementation provisions a NodePort Service. - The `spec.healthCheckNodePort` field is set automatically by - Kubernetes when `externalTrafficPolicy: Local`. -5. CIO does **not** provision DNS. -6. The cluster administrator configures their external load balancer - to use `spec.healthCheckNodePort` for health checking, ensuring - traffic is only sent to nodes with a local Envoy pod. - -```mermaid -sequenceDiagram - participant Admin as Cluster Admin - participant CIO as cluster-ingress-operator - participant Sail as sail-operator library - participant Impl as Gateway API implementation - participant Cloud as Cloud / Network - - Admin->>CIO: Edit Ingress/cluster (add customGatewayClasses entry) - CIO->>Impl: Create GatewayClass resource - CIO->>Sail: Pass service config as json.RawMessage - Sail->>Sail: Create GatewayClass defaults ConfigMap - Note over Sail: type, scope, ETP,
platform annotations - - Admin->>Impl: Create Gateway (references custom class) - Impl->>Impl: Deploy proxy Deployment - Impl->>Cloud: Create Service (per ConfigMap defaults) - CIO->>Cloud: Create DNS records (LB types only) - - Note over Admin,Cloud: No DNS for ClusterIP or NodePort types -``` +2. The cluster administrator creates a GatewayClass referencing it + with `spec.parametersRef` (same pattern as above). -### API Extensions +3. CIO creates the ConfigMap with `service.type: LoadBalancer`, + the platform external LB annotation, `externalTrafficPolicy: Local`, + and the OVN `local-with-fallback` annotation where applicable. -This enhancement extends the existing -`operator.openshift.io/v1alpha1` `Ingress` CRD. It does not -introduce new CRD types. +4. Gateways referencing this class get a LoadBalancer Service with ETP + Local. CIO manages DNS. + +### API Extensions -#### New fields on `GatewayAPIIngressConfig` +#### `GatewayParameters` CRD ```go -type GatewayAPIIngressConfig struct { - // managementMode (existing, unchanged) - ManagementMode GatewayAPIManagementMode `json:"managementMode,omitempty"` - - // customGatewayClasses defines a list of GatewayClasses that CIO - // creates and manages. For each entry, CIO creates a GatewayClass - // resource and a GatewayClass defaults ConfigMap that the Gateway - // API implementation uses when provisioning Gateways of that class. - // - // GatewayClass-level settings are defaults applied to all Gateways - // referencing the class. Individual Gateway instances cannot - // override class-level settings. - // - // +optional - // +listType=map - // +listMapKey=name - // +kubebuilder:validation:MaxItems=16 - CustomGatewayClasses []CustomGatewayClassConfig `json:"customGatewayClasses,omitempty"` +// GatewayParameters configures the infrastructure provisioned by the +// OpenShift ingress operator for a GatewayClass that references it via +// spec.parametersRef. +// +// This CRD is designed to be extended to support Gateway-level +// customization in a future release, pending upstream plumbing support +// for non-mutating per-Gateway configuration injection. +// +// +kubebuilder:object:root=true +// +kubebuilder:resource:scope=Cluster +// +kubebuilder:subresource:status +// +openshift:enable:FeatureGate=GatewayClassParameters +type GatewayParameters struct { + metav1.TypeMeta `json:",inline"` + metav1.ObjectMeta `json:"metadata,omitempty"` + + Spec GatewayParametersSpec `json:"spec"` + Status GatewayParametersStatus `json:"status,omitempty"` } -``` -#### `CustomGatewayClassConfig` - -```go -// CustomGatewayClassConfig defines a user-managed GatewayClass and -// its associated configuration. -type CustomGatewayClassConfig struct { - // name is the name of the GatewayClass to create. Must use the - // "openshift-" prefix. The name "openshift-default" is reserved - // and may not be used here. - // - // +required - // +kubebuilder:validation:MinLength=12 - // +kubebuilder:validation:Pattern=`^openshift-[a-z0-9]([a-z0-9\-]*[a-z0-9])?$` - // +kubebuilder:validation:XValidation:rule="self != 'openshift-default'",message="openshift-default is reserved" - Name string `json:"name"` - - // endpointPublishingStrategy defines how the Gateway's endpoints - // are published to the network. When omitted, defaults to an - // external LoadBalancer with platform defaults. +type GatewayParametersSpec struct { + // endpointPublishingStrategy defines how the Gateway's backing + // Service is provisioned. When omitted, defaults to an external + // LoadBalancer with platform defaults (matching openshift-default + // behavior). // // +optional EndpointPublishingStrategy *GatewayEndpointPublishingStrategy `json:"endpointPublishingStrategy,omitempty"` - // resources configures resource requests and limits for the - // gateway proxy containers. When omitted, the Gateway API - // implementation's defaults apply. - // - // +optional - Resources *corev1.ResourceRequirements `json:"resources,omitempty"` + // Future fields (not in this EP): + // resources *corev1.ResourceRequirements + // nodePlacement *NodePlacement } ``` #### `GatewayEndpointPublishingStrategy` ```go -// GatewayEndpointPublishingStrategy defines how a Gateway's endpoints -// are published. It is a discriminated union on Type. -// // +union // +kubebuilder:validation:XValidation:rule="self.type != 'LoadBalancerService' || has(self.loadBalancer)",message="loadBalancer is required when type is LoadBalancerService" // +kubebuilder:validation:XValidation:rule="self.type == 'LoadBalancerService' || !has(self.loadBalancer)",message="loadBalancer is only valid when type is LoadBalancerService" // +kubebuilder:validation:XValidation:rule="self.type == 'NodePortService' || !has(self.nodePort)",message="nodePort is only valid when type is NodePortService" type GatewayEndpointPublishingStrategy struct { - // type is the publishing strategy to use. + // type is the publishing strategy. // - // LoadBalancerService provisions a cloud or hardware LoadBalancer - // Service in front of the gateway proxy. CIO applies - // platform-specific annotations and manages DNS automatically. + // LoadBalancerService: provisions a cloud or hardware LoadBalancer + // Service. CIO applies platform-specific annotations and manages DNS. // - // NodePortService provisions a Kubernetes NodePort Service. - // No DNS is managed by CIO. The cluster administrator is - // responsible for configuring an external load balancer and, when - // endpointTrafficPolicy is Local, directing it to use the - // Service's healthCheckNodePort for health checking. + // NodePortService: provisions a NodePort Service. No DNS is managed. + // The administrator is responsible for the external load balancer. // - // ClusterIPService provisions a ClusterIP Service. The gateway - // is accessible only within the cluster. No DNS is managed by - // CIO. Useful for fronting with an OCP Route on bare-metal - // clusters without a hardware or software load balancer. + // ClusterIPService: provisions a ClusterIP Service accessible only + // within the cluster. No DNS is managed. Useful for fronting with + // an OCP Route on bare-metal clusters. // // +unionDiscriminator // +required // +kubebuilder:validation:Enum=LoadBalancerService;NodePortService;ClusterIPService Type GatewayEndpointPublishingStrategyType `json:"type"` - // loadBalancer holds parameters for the load balancer. - // Present only when type is LoadBalancerService. - // + // loadBalancer holds parameters for the LoadBalancer service type. // +optional LoadBalancer *GatewayLoadBalancerStrategy `json:"loadBalancer,omitempty"` - // nodePort holds parameters for the NodePort service. - // Present only when type is NodePortService. - // + // nodePort holds parameters for the NodePort service type. // +optional NodePort *GatewayNodePortStrategy `json:"nodePort,omitempty"` } -// GatewayEndpointPublishingStrategyType is the publishing strategy -// for a Gateway. type GatewayEndpointPublishingStrategyType string const ( - // GatewayStrategyLoadBalancerService provisions a LoadBalancer - // Service and manages DNS. GatewayStrategyLoadBalancerService GatewayEndpointPublishingStrategyType = "LoadBalancerService" - - // GatewayStrategyNodePortService provisions a NodePort Service. - // No DNS is managed by CIO. - GatewayStrategyNodePortService GatewayEndpointPublishingStrategyType = "NodePortService" - - // GatewayStrategyClusterIPService provisions a ClusterIP Service. - // No DNS is managed by CIO. - GatewayStrategyClusterIPService GatewayEndpointPublishingStrategyType = "ClusterIPService" + GatewayStrategyNodePortService GatewayEndpointPublishingStrategyType = "NodePortService" + GatewayStrategyClusterIPService GatewayEndpointPublishingStrategyType = "ClusterIPService" ) -// GatewayLoadBalancerStrategy holds parameters for a LoadBalancer- -// backed GatewayClass. type GatewayLoadBalancerStrategy struct { - // scope indicates whether the load balancer is exposed - // externally or only within the cloud provider's private - // network. - // - // External provisions a public-facing load balancer with - // platform-specific external annotations. - // - // Internal provisions an internal load balancer with - // platform-specific internal annotations (e.g., - // service.beta.kubernetes.io/aws-load-balancer-internal). + // scope is External or Internal. External provisions a public-facing + // LB; Internal provisions a private LB with platform-specific internal + // annotations (e.g. service.beta.kubernetes.io/aws-load-balancer-internal). // // +required Scope LoadBalancerScope `json:"scope"` // reuses operator/v1 type - // endpointTrafficPolicy controls how external traffic is - // distributed to gateway proxy pods. + // endpointTrafficPolicy controls how external traffic is routed to + // proxy pods. // - // Local routes external traffic only to proxy pods on the same - // node that receives the traffic. This preserves the source IP - // address and avoids cross-node (and cross-zone) hops. - // The load balancer must use the Service's healthCheckNodePort - // to health-check nodes; CIO configures this automatically for - // cloud load balancers via platform annotations. The OVN - // local-with-fallback annotation is also set when Local is used, - // ensuring traffic is not dropped when no local pod is available - // during rolling updates. + // Local routes traffic only to proxy pods on the receiving node, + // preserving source IP and avoiding cross-zone hops. The cloud LB + // uses healthCheckNodePort; CIO configures this via platform + // annotations. The OVN local-with-fallback annotation is also set + // to avoid drops during rolling updates. // - // Cluster allows the implementation to route external traffic to - // any proxy pod in the cluster, performing SNAT. Source IP is - // not preserved. + // Cluster routes to any proxy pod (with SNAT). Source IP is not + // preserved. // // When omitted, defaults to Local on most platforms. IBM Cloud // defaults to Cluster due to platform constraints. @@ -496,378 +326,221 @@ type GatewayLoadBalancerStrategy struct { EndpointTrafficPolicy *GatewayEndpointTrafficPolicy `json:"endpointTrafficPolicy,omitempty"` } -// GatewayNodePortStrategy holds parameters for a NodePort-backed -// GatewayClass. type GatewayNodePortStrategy struct { - // endpointTrafficPolicy controls how external traffic is - // distributed to gateway proxy pods. - // - // Local routes external traffic only to proxy pods on the same - // node that receives the traffic. This preserves the source IP - // address and avoids cross-node hops. When Local is used, the - // external load balancer MUST be configured to health-check - // nodes using Service.spec.healthCheckNodePort. If the load - // balancer does not perform this health check, traffic will be - // dropped on nodes that have no local proxy pod. - // - // Cluster allows routing to any proxy pod (with SNAT). Source - // IP is not preserved, but load distribution is even regardless - // of pod placement. + // endpointTrafficPolicy controls external traffic routing. + // When Local, the external LB MUST use Service.spec.healthCheckNodePort + // for health checks or traffic will be dropped on nodes without a + // local proxy pod. // - // When omitted, this field is not set (the implementation - // default applies). Unlike LoadBalancerService, there is no - // platform-specific default because NodePort is used in - // bring-your-own-load-balancer scenarios where the external LB - // capabilities vary widely. + // When omitted, the implementation default applies. Unlike + // LoadBalancerService, there is no platform-specific default. // // +optional EndpointTrafficPolicy *GatewayEndpointTrafficPolicy `json:"endpointTrafficPolicy,omitempty"` } -// GatewayEndpointTrafficPolicy is the externalTrafficPolicy for a -// Gateway's Service. // +kubebuilder:validation:Enum=Local;Cluster type GatewayEndpointTrafficPolicy string const ( - // GatewayEndpointTrafficPolicyLocal routes external traffic only - // to pods on the same node, preserving source IP. - GatewayEndpointTrafficPolicyLocal GatewayEndpointTrafficPolicy = "Local" - - // GatewayEndpointTrafficPolicyCluster routes external traffic to - // any pod, performing SNAT. + GatewayEndpointTrafficPolicyLocal GatewayEndpointTrafficPolicy = "Local" GatewayEndpointTrafficPolicyCluster GatewayEndpointTrafficPolicy = "Cluster" ) ``` -#### ValidatingAdmissionPolicy Changes +#### ValidatingAdmissionPolicy -The existing VAP for GatewayClass naming is simplified to two rules: +The existing VAP for GatewayClass naming enforces two symmetric rules: -1. A GatewayClass with `controllerName: - openshift.io/gateway-controller/v1` MUST have a name prefixed - with `openshift-`. +1. A GatewayClass with `controllerName: openshift.io/gateway-controller/v1` + MUST have a name prefixed with `openshift-`. 2. A GatewayClass with the `openshift-` name prefix MUST use `controllerName: openshift.io/gateway-controller/v1`. -The previous allowlist of exactly four names is removed. CIO creates -GatewayClasses for `openshift-default` (hardcoded) and any entry in -`customGatewayClasses`. A GatewayClass manually created by a user -with the `openshift-` prefix and the OpenShift controller name is -permitted but not managed by CIO (no defaults ConfigMap is created -for it). - -#### YAML examples - -```yaml -# External LoadBalancer with zone-aware routing -apiVersion: operator.openshift.io/v1alpha1 -kind: Ingress -metadata: - name: cluster -spec: - gatewayAPI: - customGatewayClasses: - - name: openshift-external - endpointPublishingStrategy: - type: LoadBalancerService - loadBalancer: - scope: External - endpointTrafficPolicy: Local - - # Internal LoadBalancer (default ETP for platform) - - name: openshift-internal - endpointPublishingStrategy: - type: LoadBalancerService - loadBalancer: - scope: Internal - - # ClusterIP for bare-metal + OCP Route topology - - name: openshift-clusterip - endpointPublishingStrategy: - type: ClusterIPService - - # NodePort for bring-your-own load balancer - - name: openshift-nodeport - endpointPublishingStrategy: - type: NodePortService - nodePort: - endpointTrafficPolicy: Local - - # High-traffic class with tuned proxy resources - - name: openshift-production - endpointPublishingStrategy: - type: LoadBalancerService - loadBalancer: - scope: External - endpointTrafficPolicy: Local - resources: - requests: - cpu: 500m - memory: 512Mi - limits: - memory: 1Gi -``` - -### Topology Considerations - -#### Hypershift / Hosted Control Planes +The previous hardcoded allowlist of four names is removed. The +`openshift-` prefix is reserved for OpenShift-controller-managed classes. +Unmanaged GatewayClasses (no OpenShift controllerName) may use any name. -This enhancement applies to Hypershift without additional -considerations beyond existing Gateway API support. GatewayClass -provisioning and the Gateway API implementation run in the guest -cluster. Platform-specific annotations in the GatewayClass defaults -ConfigMap are derived from the guest cluster's infrastructure -platform. - -#### Standalone Clusters +#### DNS Management -Directly applicable. Platform annotation derivation uses the same -logic as CIO's IngressController service provisioning. +| `type` | CIO manages DNS | +|----------------------|-----------------| +| `LoadBalancerService`| Yes | +| `NodePortService` | No | +| `ClusterIPService` | No | -#### Single-node Deployments or MicroShift +### Future API Extensions -`ClusterIPService` GatewayClasses are particularly useful for SNO -deployments where cloud load balancers are not available. No -additional resource consumption beyond what a Gateway already requires. +The following fields are planned for follow-on EPs and are **not** part +of this EP. They are enumerated here to confirm the `GatewayParameters` +CRD design accommodates them without breaking changes: -MicroShift has its own Gateway API support and does not use CIO, so -this enhancement does not apply to MicroShift. +- **`spec.resources`** (`corev1.ResourceRequirements`): Configure CPU + and memory requests/limits for the gateway proxy containers. Translates + into the `deployment.resources` key in the OSSM defaults ConfigMap. -#### OpenShift Kubernetes Engine +- **`spec.nodePlacement`**: Node selectors, tolerations, and affinity + rules for the proxy Deployment. Translates into `deployment.podAnnotations` + and `deployment.affinity` in the OSSM defaults ConfigMap. -Applicable on OKE clusters where Gateway API is available, subject -to the same OSSM version dependency as for standard OCP. +- **Gateway-level reuse**: `GatewayParameters` is designed as a potential + `gateway.spec.infrastructure.parametersRef` target in a future release, + enabling per-Gateway overrides without a new CRD. This requires either + upstream plumbing support or OSSM native support for reading the CRD, + and is explicitly out of scope for this EP. -### Implementation Details/Notes/Constraints +### Implementation Details -#### GatewayClass Defaults ConfigMap Mechanism +#### OSSM ConfigMap Mechanism -The GatewayClass defaults ConfigMap is an Istio mechanism documented -at https://istio.io/latest/docs/tasks/traffic-management/ingress/gateway-api/#gatewayclass-defaults. -One ConfigMap is created per GatewayClass entry (not per Gateway -instance). All Gateways referencing a given GatewayClass share the -same defaults. +CIO creates one ConfigMap per GatewayClass in the `openshift-ingress` +namespace, labeled `gateway.istio.io/defaults-for-class: `. +OSSM reads this label to apply the defaults to all Gateways referencing +that class. This mechanism requires OSSM >= 3.2.4 or >= 3.3.1. -CIO passes service configuration as `json.RawMessage` to the -sail-operator library, which creates the ConfigMap. This is the same -mechanism CIO uses for HPA provisioning. The required support in the -Gateway API implementation was added in OSSM >= 3.2.4 and >= 3.3.1 -(sail-operator#1465). Backport to OCP versions using OSSM 3.0.x or -3.1.x is not possible without an upstream cherry-pick. +The ConfigMap is owned by CIO and must not be manually edited. Direct +use of this ConfigMap as a customization mechanism by end users is +explicitly unsupported and may be overwritten at any time. #### Platform Annotation Derivation -CIO reuses its existing IngressController annotation-derivation logic -to populate the GatewayClass defaults ConfigMap with the correct -platform-specific service annotations. The derivation is based on: +CIO reuses its existing IngressController annotation logic, keyed on: -- `scope: External` or `scope: Internal` (from the API) -- The cluster's `infrastructure.config.openshift.io/cluster` platform - type (AWS, Azure, GCP, IBM, OpenStack, etc.) -- `endpointTrafficPolicy: Local` triggers the OVN +- `scope: External` or `scope: Internal` +- The cluster's infrastructure platform type +- `endpointTrafficPolicy: Local` → OVN `traffic-policy.network.alpha.openshift.io/local-with-fallback: ""` - annotation on platforms where it applies. - -#### DNS Management - -| `type` | CIO manages DNS | -|----------------------|-----------------| -| `LoadBalancerService`| Yes | -| `NodePortService` | No | -| `ClusterIPService` | No | + annotation on applicable platforms #### Deletion Semantics -When an entry is removed from `customGatewayClasses`, CIO removes -the GatewayClass defaults ConfigMap but does not delete the -GatewayClass resource itself. Deleting the GatewayClass while -Gateways reference it would orphan running workloads. CIO sets a -condition on the GatewayClass (`Accepted: False`, reason: -`NoLongerManaged`) and records a corresponding condition on -`Ingress/cluster` status to notify the administrator. The -administrator is responsible for deleting the GatewayClass after -migrating or deleting dependent Gateways. +When a `GatewayParameters` CR is deleted, CIO removes the defaults +ConfigMap but does not delete the `GatewayClass`. Deleting a GatewayClass +while Gateways reference it would orphan running workloads. CIO sets a +condition on the `GatewayParameters` status and the `GatewayClass` +(`Accepted: False`, reason: `ParametersNotFound`) to notify the +administrator. #### Feature Gate -This enhancement is gated behind a new feature gate -`CustomGatewayClasses`, initially added to `TechPreviewNoUpgrade` -in `github.com/openshift/api/features/features.go`. The feature gate -marker `+openshift:enable:FeatureGate=CustomGatewayClasses` is added -to the relevant API fields. +Gated behind `GatewayClassParameters` in `TechPreviewNoUpgrade`. ### Risks and Mitigations -**Risk**: OSSM version dependency. The GatewayClass defaults ConfigMap -mechanism requires OSSM >= 3.2.4 or >= 3.3.1. +**Risk**: OSSM version dependency for the defaults ConfigMap mechanism. **Mitigation**: CIO checks the OSSM version at reconciliation time and -sets a `Degraded` condition on `Ingress/cluster` with a clear message -if the version requirement is not met. The feature gate prevents the -API from being available until the cluster meets the version -requirement on supported OCP releases. - -**Risk**: Uneven load distribution with `endpointTrafficPolicy: Local` -on NodePort GatewayClasses when pods are not distributed evenly -across nodes. - -**Mitigation**: The API documentation for `NodePortService` + -`endpointTrafficPolicy: Local` explicitly states the health check -NodePort requirement and the load-distribution trade-off. Future work -(topology spread constraints via `nodePlacement`) will allow operators -to ensure even pod distribution. - -**Risk**: A user removes a `customGatewayClasses` entry while Gateways -still reference the GatewayClass, expecting the GatewayClass to be -deleted. - -**Mitigation**: CIO does not delete the GatewayClass on entry removal -(documented above). A clear condition is set to guide the -administrator. Future work may add a deletion policy field. - -### Drawbacks - -- `customGatewayClasses` settings are class-level defaults applied to - all Gateways referencing the class. A small development Gateway and - a high-traffic production Gateway referencing the same class share - the same resource defaults and service topology. Administrators must - create separate GatewayClasses for different workload tiers. -- The `openshift-` prefix requirement for all CIO-managed GatewayClass - names may feel restrictive, but it is necessary to ensure CIO's - controller name association is valid and that the prefix reservation - VAP can enforce it consistently. - -## Alternatives (Not Implemented) +sets a `Degraded` condition on the `GatewayParameters` status with a +clear message if the version requirement is not met. + +**Risk**: `externalTrafficPolicy: Local` with MetalLB BGP can cause +traffic disruption if gateway pods are not spread across all nodes — +MetalLB withdraws BGP route advertisements from nodes without a local +proxy pod, so pod scheduling changes cause route flaps. + +**Mitigation**: The field is opt-in. Documentation explicitly covers the +MetalLB interaction and the requirement for even pod distribution. +Future `nodePlacement` support (follow-on) will allow operators to +ensure pods are spread across all BGP-advertising nodes. + +**Risk**: ETP Local is not appropriate as a default for existing +GatewayClasses because it could silently break customers who have +been advised by support to rely on ETP Cluster + MetalLB. Changing +an existing GatewayClass default requires a new GatewayClass +(controllerName is immutable). + +**Mitigation**: `openshift-default` is unchanged. ETP Local is only +set when explicitly configured in `GatewayParameters`. + +## Alternatives + +### Extending the `Ingress` Singleton + +Adding `spec.gatewayAPI.customGatewayClasses[]` to the existing +`operator.openshift.io/v1alpha1` `Ingress` singleton was considered. +This was rejected because: + +- It conflates two concerns: CRD/controller lifecycle management + (existing `managementMode` field) with GatewayClass infrastructure + configuration. +- It requires CIO to own and create GatewayClasses on behalf of users, + rather than users creating their own GatewayClasses with CIO acting + only as a configuration reconciler. +- The `parametersRef` pattern is the Gateway API's designed extension + point for this purpose and is already used by other implementations + (e.g. Envoy Gateway's `EnvoyProxy` CRD). ### Hardcoded GatewayClasses -The prior proposal created three fixed GatewayClasses -(`openshift-external`, `openshift-internal`, `openshift-clusterip`). -This approach was rejected because: +Creating fixed GatewayClasses (`openshift-external`, `openshift-internal`, +`openshift-clusterip`) was rejected because it requires a code change +for each new configuration combination and creates a permanent VAP +allowlist maintenance burden. + +### Istio ClusterIP Alpha Annotation -- A hardcoded allowlist requires a code change for every new - service configuration combination. -- NodePort support and resource customization were explicitly deferred - as open questions with no clear path forward. -- `externalTrafficPolicy` was not user-configurable; only the - platform-derived default applied. -- The VAP allowlist with four fixed names is a permanent maintenance - burden. +The annotation `networking.istio.io/service-type: ClusterIP` on a +`Gateway` resource can configure a ClusterIP service today. This is +rejected as a supported path because it is an undocumented private API +that can change or be removed at any OSSM version, gives CIO no +visibility into the configured service type, and does not compose with +ETP configuration. ## Open Questions -1. **Status per GatewayClass**: Should `Ingress/cluster` status - include a `gatewayClasses[]` slice with one entry per - `customGatewayClasses` entry, or is it sufficient to rely on - the GatewayClass resource's own `.status.conditions`? The latter - avoids duplicating status but requires administrators to query - GatewayClass resources separately. +1. **`GatewayParameters` status**: What conditions should be reported? + At minimum: `Accepted` (CIO has found a referencing GatewayClass and + created the ConfigMap) and `Degraded` (OSSM version too old, ConfigMap + sync failure). -2. **Default `endpointPublishingStrategy`**: When - `endpointPublishingStrategy` is omitted from an entry, should the - default be `LoadBalancerService` with `scope: External` (matching - `openshift-default` behavior), or should it be required? +2. **Default when `endpointPublishingStrategy` is omitted**: Should it + default to `LoadBalancerService` with `scope: External`, or should + the field be required? -3. **Deletion policy field**: Should we add a - `deletionPolicy: Delete|Retain` field to `CustomGatewayClassConfig` - to give administrators explicit control over whether CIO deletes - the GatewayClass when the entry is removed? +3. **GatewayClass ownership**: Should CIO require the GatewayClass to + use the OpenShift controllerName before reconciling a referenced + `GatewayParameters`, or reconcile for any GatewayClass that points + to a `GatewayParameters` CR? ## Test Plan - - -Tests must include: - -- `[OCPFeatureGate:CustomGatewayClasses]` label for the feature gate -- `[Jira:"OCP/Network Ingress"]` label for the component -- Unit tests for CIO controller logic (ConfigMap content derivation, - platform annotation selection per service type and scope) -- E2E tests for each `endpointPublishingStrategy.type`: - - `LoadBalancerService` External and Internal on supported cloud - platforms - - `NodePortService` on applicable platforms - - `ClusterIPService` with OCP Route fronting -- E2E tests verifying `externalTrafficPolicy: Local` behavior - (source IP preservation, health check NodePort set) -- E2E tests verifying `resources` propagation to the proxy container -- Upgrade tests: existing `openshift-default` GatewayClass unaffected - after upgrade -- Tests must run on all supported platforms: AWS, Azure, GCP, - vSphere, Baremetal +- `[OCPFeatureGate:GatewayClassParameters]` label on all tests +- Unit tests: ConfigMap content derivation, platform annotation selection +- E2E: `ClusterIPService` with OCP Route fronting on bare-metal/vSphere +- E2E: `LoadBalancerService` External/Internal on AWS, Azure, GCP +- E2E: `endpointTrafficPolicy: Local` — source IP preservation, health + check NodePort set +- Upgrade: `openshift-default` unaffected after upgrade ## Graduation Criteria -### Dev Preview -> Tech Preview +### Dev Preview → Tech Preview -- E2E tests passing on at least AWS and bare-metal platforms -- OSSM version validation implemented and tested -- End-user documentation draft -- Minimum 5 tests, 7 runs per week, 95% pass rate +- E2E tests on AWS and bare-metal +- OSSM version check implemented +- Documentation draft -### Tech Preview -> GA +### Tech Preview → GA -- E2E tests on all supported platforms with ≥14 runs per platform -- Upgrade and downgrade testing complete -- SLIs defined and telemetry collected -- User-facing documentation complete in openshift-docs -- Deletion semantics validated in upgrade scenarios +- E2E on all supported platforms +- Upgrade/downgrade tested +- openshift-docs complete ## Upgrade / Downgrade Strategy -**Upgrade**: The `customGatewayClasses` field defaults to empty. No -existing GatewayClass or Gateway resources are modified on upgrade. +**Upgrade**: No existing GatewayClass or Gateway is modified. `openshift-default` behavior is unchanged. -**Downgrade**: If a cluster is downgraded to a version that does not -support `customGatewayClasses`, CIO stops reconciling the entries but -does not delete any GatewayClass or ConfigMap resources it previously -created. Those resources remain functional until an administrator -manually removes them. This is consistent with how CIO handles other -feature gate removals. +**Downgrade**: CIO stops reconciling `GatewayParameters` CRs but does +not delete previously created ConfigMaps or GatewayClasses. Those +resources remain functional until an administrator manually removes them. ## Version Skew Strategy -The GatewayClass defaults ConfigMap mechanism requires the Gateway API -implementation to be at a minimum version (OSSM >= 3.2.4 / >= 3.3.1). -CIO detects the implementation version at reconciliation time and -reports a condition rather than attempting to create ConfigMaps that -the implementation version cannot consume. No version skew between -CIO and the kube-apiserver is anticipated, as this enhancement adds -fields to an existing CRD. - -## Operational Aspects of API Extensions - -The `customGatewayClasses` field is added to the existing -`Ingress/cluster` CRD. The CRD is managed by CIO and protected by an -existing VAP. The field is gated by a feature gate and therefore -invisible to clusters that have not enabled `TechPreviewNoUpgrade`. - -Failure modes: - -- **OSSM version too old**: CIO sets `Degraded` condition on - `Ingress/cluster` with reason `GatewayClassDefaultsUnsupported`. - Existing Gateways continue to function; new `customGatewayClasses` - entries are not reconciled. -- **ConfigMap creation failure**: CIO retries and degrades with a - condition. The GatewayClass resource may exist without a ConfigMap; - Gateways referencing it will use implementation defaults rather than - the intended configuration. - -## Support Procedures - -- Check `Ingress/cluster` status conditions for - `GatewayClassDefaultsUnsupported` or `GatewayClassReconcileFailed`. -- Check the GatewayClass resource's `.status.conditions` for - `Accepted` status. -- CIO logs include structured events for each `customGatewayClasses` - reconciliation pass. -- To disable: remove entries from `customGatewayClasses`. CIO stops - managing the ConfigMap; the GatewayClass is retained (see Deletion - Semantics). +CIO checks the OSSM version at reconciliation time and reports a +condition rather than creating ConfigMaps an older OSSM cannot consume. ## Infrastructure Needed -No new infrastructure is needed. The implementation uses the existing -sail-operator library integration already used for HPA provisioning. +None. CIO already has the sail-operator library integration for ConfigMap +management. The new CRD is registered in `openshift/api`. From 67847680087022697a3dc9c52d170064c1966bc1 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 15:57:08 -0400 Subject: [PATCH 03/20] enhancements/ingress: Strengthen motivation and summary for GatewayParameters EP Expand the summary and motivation to establish that there is currently no supported OpenShift configuration layer for Gateway API infrastructure customization, and that this EP is the first step in building one that abstracts the implementation while maintaining support and compatibility. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 77 +++++++++++++------ 1 file changed, 52 insertions(+), 25 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 9e96da57ae..1efc89515a 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -26,34 +26,61 @@ superseded-by: [] ## Summary -This enhancement introduces a new `GatewayParameters` CRD in the -`operator.openshift.io` API group. A `GatewayClass` references a -`GatewayParameters` instance via `spec.parametersRef`, and the Cluster -Ingress Operator (CIO) reconciles it into an OSSM GatewayClass defaults -ConfigMap. This provides a first-class, implementation-agnostic API for -customizing how Gateway API implementations provision the backing -Kubernetes Service and proxy deployment for a GatewayClass. - -This EP implements two use cases: ClusterIP service type and -`externalTrafficPolicy: Local`. The `GatewayParameters` CRD is designed -to be extended with additional fields (resource requests, node -placement) in follow-on work without API changes to the GatewayClass -or Gateway resources. +OpenShift currently has no supported way to customize how a Gateway API +implementation provisions the Kubernetes Service, proxy Deployment, or +proxy configuration for a GatewayClass. The upstream Gateway API +specification defines `GatewayClass.spec.parametersRef` as the extension +point for this purpose, and implementations such as Istio/OSSM expose +implementation-specific mechanisms, but there is no OpenShift +configuration layer that abstracts these details, validates input, or +provides a stable, upgrade-safe API. + +This enhancement introduces `GatewayParameters`, a new cluster-scoped +CRD in the `operator.openshift.io` API group. It is the first step in +an OpenShift-native configuration layer for Gateway API infrastructure +customization. A `GatewayClass` references a `GatewayParameters` +instance via `spec.parametersRef`, and the Cluster Ingress Operator +(CIO) reconciles it into the implementation-specific configuration — +today, the OSSM GatewayClass defaults ConfigMap. The API is +implementation-agnostic: the OpenShift types abstract the underlying +mechanism so that future changes to the Gateway API implementation do +not require API changes or user migrations. + +This EP establishes the plumbing and implements two concrete use cases: +ClusterIP service type and `externalTrafficPolicy: Local`. The +`GatewayParameters` CRD is designed to be extended with additional +fields (resource requests, node placement, proxy configuration) in +follow-on EPs without breaking changes to the GatewayClass or Gateway +resources. ## Motivation -When a user creates a Gateway using the `openshift-default` GatewayClass, -the Gateway API implementation always provisions an external LoadBalancer -service. There is currently no supported, declarative way to: - -- Use a ClusterIP service for cluster-internal traffic or to front with - an OCP Route on bare-metal without a hardware load balancer -- Configure `externalTrafficPolicy: Local` to preserve source IP and - avoid cross-zone hops in zone-aware or BGP-based cluster topologies - -Users who need these configurations today must use the Istio ClusterIP -alpha annotation or manually patch the Service after creation — both -unsupported, fragile approaches that break on reconciliation or upgrade. +OpenShift ships Gateway API via OSSM, which supports customization of +the backing Service and proxy Deployment through implementation-specific +mechanisms (GatewayClass defaults ConfigMap, alpha annotations). These +mechanisms are undocumented, unsupported for end users, and tied to +Istio internals. There is no OpenShift API that: + +- Provides a stable, validated interface for Gateway infrastructure + customization +- Abstracts implementation details so customizations survive Gateway + API implementation changes +- Integrates with CIO to derive platform-specific configuration + (cloud LB annotations, OVN settings) automatically + +As a result, when a user creates a Gateway using the `openshift-default` +GatewayClass, the implementation always provisions an external +LoadBalancer service, and there is no supported path to change this. +Specific gaps that generate support exceptions today: + +- ClusterIP service type for bare-metal deployments fronted by an OCP + Route, where no hardware or software load balancer is available +- `externalTrafficPolicy: Local` for zone-aware or BGP-based topologies + where cross-zone hops must be avoided and source IP must be preserved + +Users who need these configurations must use the Istio ClusterIP alpha +annotation or manually patch the Service — both fragile approaches that +are overwritten on reconciliation and unsupported for production use. ### User Stories From 69324051236b0bd27f29ff8b83adaa8dfb1e64e6 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 16:00:45 -0400 Subject: [PATCH 04/20] enhancements/ingress: Move problem statement from summary to motivation Co-Authored-By: Claude Sonnet 4.6 --- .../ingress/gateway-api-gateway-customization.md | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 1efc89515a..43ffd9c09c 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -26,18 +26,9 @@ superseded-by: [] ## Summary -OpenShift currently has no supported way to customize how a Gateway API -implementation provisions the Kubernetes Service, proxy Deployment, or -proxy configuration for a GatewayClass. The upstream Gateway API -specification defines `GatewayClass.spec.parametersRef` as the extension -point for this purpose, and implementations such as Istio/OSSM expose -implementation-specific mechanisms, but there is no OpenShift -configuration layer that abstracts these details, validates input, or -provides a stable, upgrade-safe API. - This enhancement introduces `GatewayParameters`, a new cluster-scoped -CRD in the `operator.openshift.io` API group. It is the first step in -an OpenShift-native configuration layer for Gateway API infrastructure +CRD in the `operator.openshift.io` API group, and establishes the first +OpenShift-native configuration layer for Gateway API infrastructure customization. A `GatewayClass` references a `GatewayParameters` instance via `spec.parametersRef`, and the Cluster Ingress Operator (CIO) reconciles it into the implementation-specific configuration — From b99af0f89bb7f583eaf6b8c50eb3e429fa397924 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 16:01:35 -0400 Subject: [PATCH 05/20] enhancements/ingress: Open motivation with the core problem statement Co-Authored-By: Claude Sonnet 4.6 --- .../ingress/gateway-api-gateway-customization.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 43ffd9c09c..b92a2c4ceb 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -46,11 +46,13 @@ resources. ## Motivation -OpenShift ships Gateway API via OSSM, which supports customization of -the backing Service and proxy Deployment through implementation-specific -mechanisms (GatewayClass defaults ConfigMap, alpha annotations). These -mechanisms are undocumented, unsupported for end users, and tied to -Istio internals. There is no OpenShift API that: +OpenShift currently has no supported way to customize how a Gateway API +implementation provisions the Kubernetes Service, proxy Deployment, or +proxy configuration for a GatewayClass. OpenShift ships Gateway API via +OSSM, which supports customization through implementation-specific +mechanisms (GatewayClass defaults ConfigMap, alpha annotations), but +these are undocumented, unsupported for end users, and tied to Istio +internals. There is no OpenShift API that: - Provides a stable, validated interface for Gateway infrastructure customization From 60cdf44ccd9b3ed4353461eb634fa394462ac671 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 16:13:05 -0400 Subject: [PATCH 06/20] enhancements/ingress: Refactor Goals to be outcome-focused not mechanistic Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 26 ++++++++----------- 1 file changed, 11 insertions(+), 15 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index b92a2c4ceb..2b8b21d386 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -101,21 +101,17 @@ expose services only within my cloud provider's private network. ### Goals -- Introduce a `GatewayParameters` CRD that a `GatewayClass` references - via `spec.parametersRef` to configure its service topology. -- CIO reconciles `GatewayParameters` → OSSM GatewayClass defaults - ConfigMap (labeled `gateway.istio.io/defaults-for-class`), translating - the OpenShift API into the implementation-specific ConfigMap format. -- Implement `endpointPublishingStrategy` for service type - (LoadBalancer/NodePort/ClusterIP) and `externalTrafficPolicy` - (Local/Cluster). -- CIO derives platform-specific service annotations automatically from - cluster infrastructure, as it does for IngressControllers. -- CIO manages DNS for GatewayClasses with LoadBalancer service type; - not for ClusterIP or NodePort. -- The existing `openshift-default` GatewayClass is unchanged. -- Design the CRD for future extension (resources, nodePlacement) without - breaking API changes. +- Provide a supported, stable OpenShift API for customizing the service + topology of a GatewayClass (LoadBalancer external/internal, NodePort, + ClusterIP) and endpoint traffic policy. +- Platform-specific service configuration (cloud LB annotations, OVN + settings) is derived automatically — administrators express intent, + not platform details. +- The API is implementation-agnostic and upgrade-safe: customizations + survive Gateway API implementation changes without user intervention. +- The `openshift-default` GatewayClass is unchanged. +- Establish an extensible API foundation for follow-on customization + (resource requests, node placement) without breaking changes. ### Non-Goals From 9cec831715afd2f0dee32e9e4b830b97539db526 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 16:51:56 -0400 Subject: [PATCH 07/20] enhancements/ingress: Add upstream vs downstream calculus and GEP-5093 context Add generic upstream/downstream framing to Motivation. Add two new alternatives: "Wait for upstream GEP-5093" (rejected due to timeline) and "Istio ClusterIP annotation as interim" (rejected as implementation- private API that doesn't compose with the broader configuration layer). Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 46 ++++++++++++++++--- 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 2b8b21d386..9ff8aa8cc3 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -75,6 +75,17 @@ Users who need these configurations must use the Istio ClusterIP alpha annotation or manually patch the Service — both fragile approaches that are overwritten on reconciliation and unsupported for production use. +In an ideal world, all Gateway infrastructure customization would be +standardized upstream in the Gateway API specification. The upstream +community is actively working toward some of these use cases, and where +upstream standards mature, OpenShift will align with them. However, some +customizations are unlikely to ever be standardized upstream — they are +platform-specific, operational concerns that implementations intentionally +leave to operators. For both categories, users need a supported solution +today. This EP provides a downstream OpenShift API that bridges the gap, +designed to minimize migration cost where upstream standards eventually +emerge. + ### User Stories #### Story 1: ClusterIP Gateway on Bare Metal @@ -493,14 +504,37 @@ Creating fixed GatewayClasses (`openshift-external`, `openshift-internal`, for each new configuration combination and creates a permanent VAP allowlist maintenance burden. -### Istio ClusterIP Alpha Annotation +### Wait for Upstream Standardization + +For ClusterIP specifically, [GEP-5093 (Routability)](https://gateway-api.sigs.k8s.io/geps/gep-5093/) +proposes a standard upstream mechanism for configuring service type in +the Gateway API spec. If adopted and GA'd upstream, OpenShift could +align with it rather than maintaining a downstream API. + +This alternative was rejected for the near term because GEP-5093 is in +early stages, upstream GA is likely 3–5 years away, and OpenShift +adoption would follow after that. Users need a supported solution now. +This EP is designed so that if a future upstream standard for ClusterIP +emerges, the migration path is a GatewayClass or GatewayParameters change +rather than application-level changes. Other customizations covered by +this EP (ETP, platform annotation derivation) are unlikely to ever be +standardized upstream. + +### Istio ClusterIP Alpha Annotation as Interim Solution The annotation `networking.istio.io/service-type: ClusterIP` on a -`Gateway` resource can configure a ClusterIP service today. This is -rejected as a supported path because it is an undocumented private API -that can change or be removed at any OSSM version, gives CIO no -visibility into the configured service type, and does not compose with -ETP configuration. +`Gateway` resource can configure a ClusterIP service today. An interim +approach of officially supporting this annotation until GEP-5093 reaches +GA was considered. + +This was rejected because: the annotation is an undocumented private +Istio API that can change or be removed at any OSSM version without +notice; it gives CIO no visibility into the configured service type and +therefore cannot integrate with DNS management, condition reporting, or +future platform annotation derivation; and it does not compose with ETP +configuration. Supporting it officially would set a precedent of exposing +implementation internals as a supported API surface, which contradicts +the goal of an implementation-agnostic OpenShift configuration layer. ## Open Questions From 35f467e25d0ba722c04ef9052b747bb8970343ce Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Thu, 3 Sep 2026 17:31:15 -0400 Subject: [PATCH 08/20] enhancements/ingress: Rework API to passthrough/mirroring style MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the IngressController-style EndpointPublishingStrategy abstraction with spec.service fields that mirror Kubernetes Service API directly (type: ClusterIP/NodePort/LoadBalancer, externalTrafficPolicy: Local/Cluster). Drop the scope (Internal/External) field — users set cloud LB annotations on GatewayClass/Gateway directly. Only OVN local-with-fallback is auto-derived. Add Alternatives entry explaining the abstraction vs mirroring decision. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 221 +++++++++--------- 1 file changed, 105 insertions(+), 116 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 9ff8aa8cc3..e959200290 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -113,11 +113,12 @@ expose services only within my cloud provider's private network. ### Goals - Provide a supported, stable OpenShift API for customizing the service - topology of a GatewayClass (LoadBalancer external/internal, NodePort, - ClusterIP) and endpoint traffic policy. -- Platform-specific service configuration (cloud LB annotations, OVN - settings) is derived automatically — administrators express intent, - not platform details. + topology of a GatewayClass (LoadBalancer, NodePort, ClusterIP) and + endpoint traffic policy, using field names that mirror the Kubernetes + Service API. +- The OVN `local-with-fallback` annotation is derived automatically when + `externalTrafficPolicy: Local` is set; all other service annotations + are the administrator's responsibility. - The API is implementation-agnostic and upgrade-safe: customizations survive Gateway API implementation changes without user intervention. - The `openshift-default` GatewayClass is unchanged. @@ -134,6 +135,8 @@ expose services only within my cloud provider's private network. (`networking.istio.io/service-type`) as a supported mechanism for service type customization. This annotation is an unsupported private API and is superseded by this enhancement. +- Automatically deriving cloud provider internal/external LB annotations. + Administrators set these directly on the GatewayClass or Gateway. - Per-Gateway resource overrides. `GatewayParameters` configures class-level defaults shared by all Gateways referencing the class. - Gateway API implementation configuration (logging, control plane @@ -157,13 +160,13 @@ OpenShift controller name. For any such GatewayClass with a `spec.parametersRef` pointing to a `GatewayParameters` CR, CIO: 1. Reads the `GatewayParameters` CR. -2. Derives platform-specific service annotations from the cluster's - `infrastructure.config.openshift.io/cluster` resource. -3. Creates or updates a ConfigMap in the `openshift-ingress` namespace +2. Creates or updates a ConfigMap in the `openshift-ingress` namespace with the label `gateway.istio.io/defaults-for-class: `. - OSSM reads this ConfigMap to apply service type, annotations, and ETP - to all Gateways referencing the class. -4. Manages DNS for `LoadBalancerService` type only. + OSSM reads this ConfigMap to apply service type and ETP to all + Gateways referencing the class. +3. When `externalTrafficPolicy: Local` is set, automatically adds the + OVN `local-with-fallback` annotation on applicable platforms. +4. Manages DNS when `service.type` is `LoadBalancer`. CIO never mutates the user's `Gateway` or `GatewayClass` resources. @@ -179,8 +182,8 @@ CIO never mutates the user's `Gateway` or `GatewayClass` resources. metadata: name: clusterip-params spec: - endpointPublishingStrategy: - type: ClusterIPService + service: + type: ClusterIP ``` 2. The cluster administrator creates a `GatewayClass` referencing it: @@ -225,25 +228,28 @@ CIO never mutates the user's `Gateway` or `GatewayClass` resources. apiVersion: operator.openshift.io/v1alpha1 kind: GatewayParameters metadata: - name: external-zone-aware + name: zone-aware-params spec: - endpointPublishingStrategy: - type: LoadBalancerService - loadBalancer: - scope: External - endpointTrafficPolicy: Local + service: + type: LoadBalancer + externalTrafficPolicy: Local ``` 2. The cluster administrator creates a GatewayClass referencing it with `spec.parametersRef` (same pattern as above). 3. CIO creates the ConfigMap with `service.type: LoadBalancer`, - the platform external LB annotation, `externalTrafficPolicy: Local`, - and the OVN `local-with-fallback` annotation where applicable. + `externalTrafficPolicy: Local`, and the OVN `local-with-fallback` + annotation on applicable platforms. 4. Gateways referencing this class get a LoadBalancer Service with ETP Local. CIO manages DNS. + For an internal LoadBalancer, the administrator annotates the + GatewayClass or Gateway directly with the platform-specific annotation + (e.g. `service.beta.kubernetes.io/aws-load-balancer-internal: "true"` + on AWS). These annotations propagate to the provisioned Service. + ### API Extensions #### `GatewayParameters` CRD @@ -270,109 +276,62 @@ type GatewayParameters struct { } type GatewayParametersSpec struct { - // endpointPublishingStrategy defines how the Gateway's backing - // Service is provisioned. When omitted, defaults to an external - // LoadBalancer with platform defaults (matching openshift-default - // behavior). + // service configures the Kubernetes Service provisioned for Gateways + // referencing this GatewayClass. Fields mirror the corresponding + // Kubernetes Service spec fields. // // +optional - EndpointPublishingStrategy *GatewayEndpointPublishingStrategy `json:"endpointPublishingStrategy,omitempty"` + Service *GatewayServiceParameters `json:"service,omitempty"` // Future fields (not in this EP): - // resources *corev1.ResourceRequirements - // nodePlacement *NodePlacement + // deployment *GatewayDeploymentParameters } -``` -#### `GatewayEndpointPublishingStrategy` - -```go -// +union -// +kubebuilder:validation:XValidation:rule="self.type != 'LoadBalancerService' || has(self.loadBalancer)",message="loadBalancer is required when type is LoadBalancerService" -// +kubebuilder:validation:XValidation:rule="self.type == 'LoadBalancerService' || !has(self.loadBalancer)",message="loadBalancer is only valid when type is LoadBalancerService" -// +kubebuilder:validation:XValidation:rule="self.type == 'NodePortService' || !has(self.nodePort)",message="nodePort is only valid when type is NodePortService" -type GatewayEndpointPublishingStrategy struct { - // type is the publishing strategy. +// GatewayServiceParameters mirrors selected Kubernetes Service spec fields, +// allowing administrators to configure how the Gateway API implementation +// provisions the backing Service for a GatewayClass. +type GatewayServiceParameters struct { + // type specifies the type of Kubernetes Service to provision. + // Mirrors the Kubernetes Service spec.type field. // - // LoadBalancerService: provisions a cloud or hardware LoadBalancer - // Service. CIO applies platform-specific annotations and manages DNS. + // LoadBalancer provisions a cloud or hardware load balancer. + // CIO manages DNS automatically. // - // NodePortService: provisions a NodePort Service. No DNS is managed. - // The administrator is responsible for the external load balancer. + // NodePort provisions a NodePort Service. No DNS is managed. + // The administrator is responsible for configuring an external + // load balancer. When externalTrafficPolicy is Local, the external + // load balancer MUST health-check nodes via + // Service.spec.healthCheckNodePort. // - // ClusterIPService: provisions a ClusterIP Service accessible only - // within the cluster. No DNS is managed. Useful for fronting with - // an OCP Route on bare-metal clusters. + // ClusterIP provisions a ClusterIP Service accessible only within + // the cluster. No DNS is managed. Useful for fronting with an OCP + // Route on bare-metal clusters without a hardware load balancer. // - // +unionDiscriminator - // +required - // +kubebuilder:validation:Enum=LoadBalancerService;NodePortService;ClusterIPService - Type GatewayEndpointPublishingStrategyType `json:"type"` - - // loadBalancer holds parameters for the LoadBalancer service type. - // +optional - LoadBalancer *GatewayLoadBalancerStrategy `json:"loadBalancer,omitempty"` - - // nodePort holds parameters for the NodePort service type. - // +optional - NodePort *GatewayNodePortStrategy `json:"nodePort,omitempty"` -} - -type GatewayEndpointPublishingStrategyType string - -const ( - GatewayStrategyLoadBalancerService GatewayEndpointPublishingStrategyType = "LoadBalancerService" - GatewayStrategyNodePortService GatewayEndpointPublishingStrategyType = "NodePortService" - GatewayStrategyClusterIPService GatewayEndpointPublishingStrategyType = "ClusterIPService" -) - -type GatewayLoadBalancerStrategy struct { - // scope is External or Internal. External provisions a public-facing - // LB; Internal provisions a private LB with platform-specific internal - // annotations (e.g. service.beta.kubernetes.io/aws-load-balancer-internal). + // When omitted, defaults to LoadBalancer. // - // +required - Scope LoadBalancerScope `json:"scope"` // reuses operator/v1 type + // +optional + // +kubebuilder:validation:Enum=LoadBalancer;NodePort;ClusterIP + Type *corev1.ServiceType `json:"type,omitempty"` - // endpointTrafficPolicy controls how external traffic is routed to - // proxy pods. + // externalTrafficPolicy specifies how external traffic is routed to + // Gateway proxy pods. Mirrors the Kubernetes Service + // spec.externalTrafficPolicy field. Only meaningful when type is + // LoadBalancer or NodePort. // - // Local routes traffic only to proxy pods on the receiving node, - // preserving source IP and avoiding cross-zone hops. The cloud LB - // uses healthCheckNodePort; CIO configures this via platform - // annotations. The OVN local-with-fallback annotation is also set - // to avoid drops during rolling updates. + // Local routes external traffic only to proxy pods on the receiving + // node, preserving source IP and avoiding cross-zone hops. CIO + // automatically adds the OVN local-with-fallback annotation on + // applicable platforms to prevent traffic drops during rolling updates. // - // Cluster routes to any proxy pod (with SNAT). Source IP is not - // preserved. + // Cluster routes traffic to any proxy pod (with SNAT). Source IP + // is not preserved. // - // When omitted, defaults to Local on most platforms. IBM Cloud - // defaults to Cluster due to platform constraints. + // When omitted, the Gateway API implementation default applies. // // +optional - EndpointTrafficPolicy *GatewayEndpointTrafficPolicy `json:"endpointTrafficPolicy,omitempty"` + // +kubebuilder:validation:Enum=Local;Cluster + ExternalTrafficPolicy *corev1.ServiceExternalTrafficPolicy `json:"externalTrafficPolicy,omitempty"` } - -type GatewayNodePortStrategy struct { - // endpointTrafficPolicy controls external traffic routing. - // When Local, the external LB MUST use Service.spec.healthCheckNodePort - // for health checks or traffic will be dropped on nodes without a - // local proxy pod. - // - // When omitted, the implementation default applies. Unlike - // LoadBalancerService, there is no platform-specific default. - // - // +optional - EndpointTrafficPolicy *GatewayEndpointTrafficPolicy `json:"endpointTrafficPolicy,omitempty"` -} - -// +kubebuilder:validation:Enum=Local;Cluster -type GatewayEndpointTrafficPolicy string - -const ( - GatewayEndpointTrafficPolicyLocal GatewayEndpointTrafficPolicy = "Local" - GatewayEndpointTrafficPolicyCluster GatewayEndpointTrafficPolicy = "Cluster" -) ``` #### ValidatingAdmissionPolicy @@ -390,11 +349,11 @@ Unmanaged GatewayClasses (no OpenShift controllerName) may use any name. #### DNS Management -| `type` | CIO manages DNS | -|----------------------|-----------------| -| `LoadBalancerService`| Yes | -| `NodePortService` | No | -| `ClusterIPService` | No | +| `service.type` | CIO manages DNS | +|-----------------|-----------------| +| `LoadBalancer` | Yes | +| `NodePort` | No | +| `ClusterIP` | No | ### Future API Extensions @@ -431,13 +390,16 @@ explicitly unsupported and may be overwritten at any time. #### Platform Annotation Derivation -CIO reuses its existing IngressController annotation logic, keyed on: +CIO derives one platform-specific annotation automatically: -- `scope: External` or `scope: Internal` -- The cluster's infrastructure platform type -- `endpointTrafficPolicy: Local` → OVN +- `externalTrafficPolicy: Local` → OVN `traffic-policy.network.alpha.openshift.io/local-with-fallback: ""` - annotation on applicable platforms + on applicable platforms, preventing traffic drops during rolling updates. + +All other service annotations (e.g. cloud provider internal/external LB +annotations) are the administrator's responsibility and should be set +directly on the GatewayClass or Gateway resource, where they propagate +to the provisioned Service. #### Deletion Semantics @@ -481,6 +443,33 @@ set when explicitly configured in `GatewayParameters`. ## Alternatives +### Abstracted API (IngressController-Style) + +An earlier draft of this EP used a discriminated union abstraction +modelled on the IngressController `EndpointPublishingStrategy`: + +```yaml +spec: + endpointPublishingStrategy: + type: LoadBalancerService + loadBalancer: + scope: External + endpointTrafficPolicy: Local +``` + +This was rejected in favour of passthrough field mirroring because: + +- Administrators who know Kubernetes already know `service.type: ClusterIP` + and `externalTrafficPolicy: Local`. A translation layer (`ClusterIPService`, + `endpointTrafficPolicy`) adds cognitive overhead with no benefit. +- Mirroring stable Kubernetes API field names means the OpenShift API + is less likely to need changes as Kubernetes evolves. +- The abstraction required an `Internal`/`External` scope discriminator + that has no Kubernetes equivalent, forcing a hybrid of mirrored and + invented fields in the same struct. +- Other Gateway API implementations (e.g. Envoy Gateway) use the + mirroring approach for the same reasons. + ### Extending the `Ingress` Singleton Adding `spec.gatewayAPI.customGatewayClasses[]` to the existing From 6475c25fef022690873717116298e019ce353dea Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 13:50:48 -0400 Subject: [PATCH 09/20] enhancements/ingress: Move internal LB user story to Future API Extensions Internal LB requires a scope/annotation-derivation field not implemented in this EP. Remove Story 3 from User Stories and note it as a future extension alongside resources and nodePlacement. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 22 +++++++++---------- 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index e959200290..a2e0b5a180 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -103,12 +103,6 @@ availability zones with BGP-based networking, I want to configure arriving in a given zone is served by the proxy pod in that same zone, avoiding cross-zone hops and preserving source IP. -#### Story 3: Internal LoadBalancer Gateway - -As a cluster administrator, I want to create a GatewayClass that -provisions Gateways with an internal LoadBalancer service, including -the correct platform-specific internal annotations, so that I can -expose services only within my cloud provider's private network. ### Goals @@ -361,13 +355,17 @@ The following fields are planned for follow-on EPs and are **not** part of this EP. They are enumerated here to confirm the `GatewayParameters` CRD design accommodates them without breaking changes: -- **`spec.resources`** (`corev1.ResourceRequirements`): Configure CPU - and memory requests/limits for the gateway proxy containers. Translates - into the `deployment.resources` key in the OSSM defaults ConfigMap. +- **Internal LoadBalancer**: A first-class `scope: Internal` field (or + equivalent) that causes CIO to automatically derive and apply the + correct cloud provider internal LB annotation for the cluster's + platform. Today users set this annotation directly on the GatewayClass + or Gateway. + +- **`spec.deployment.resources`** (`corev1.ResourceRequirements`): Configure + CPU and memory requests/limits for the gateway proxy containers. -- **`spec.nodePlacement`**: Node selectors, tolerations, and affinity - rules for the proxy Deployment. Translates into `deployment.podAnnotations` - and `deployment.affinity` in the OSSM defaults ConfigMap. +- **`spec.deployment.nodePlacement`**: Node selectors, tolerations, and + affinity rules for the proxy Deployment. - **Gateway-level reuse**: `GatewayParameters` is designed as a potential `gateway.spec.infrastructure.parametersRef` target in a future release, From 070a06cbc7a2255a52ad2139978087d4ab89cf1b Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 13:53:36 -0400 Subject: [PATCH 10/20] enhancements/ingress: Link MAAS public ClusterIP+Route documentation Reference the publicly documented ClusterIP Gateway with OpenShift Route pattern from Red Hat AI platform docs wherever the topology is described. Co-Authored-By: Claude Sonnet 4.6 --- .../ingress/gateway-api-gateway-customization.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index a2e0b5a180..e7c71b9b13 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -93,7 +93,9 @@ emerge. As a cluster administrator running on bare metal without a hardware load balancer, I want to create a GatewayClass that provisions Gateways with a ClusterIP service so that I can front the Gateway with an OCP -Route using the existing HAProxy ingress infrastructure. +Route using the existing HAProxy ingress infrastructure. This pattern is +documented publicly by the Red Hat AI platform: +[ClusterIP Gateway with OpenShift Route (re-encrypt)](https://opendatahub-io.github.io/models-as-a-service/v2.0.1/configuration-and-management/gateway-patterns/#clusterip-gateway-with-openshift-route-re-encrypt). #### Story 2: Zone-Aware External Gateway with ETP Local @@ -212,7 +214,8 @@ CIO never mutates the user's `Gateway` or `GatewayClass` resources. ClusterIP Service. No cloud LB is created, no DNS is managed. 5. The cluster administrator creates an OCP Route pointing at the - ClusterIP Service to expose the Gateway externally via HAProxy. + ClusterIP Service to expose the Gateway externally via HAProxy + (see [ClusterIP Gateway with OpenShift Route (re-encrypt)](https://opendatahub-io.github.io/models-as-a-service/v2.0.1/configuration-and-management/gateway-patterns/#clusterip-gateway-with-openshift-route-re-encrypt)). #### Zone-Aware LoadBalancer with ETP Local @@ -299,7 +302,8 @@ type GatewayServiceParameters struct { // // ClusterIP provisions a ClusterIP Service accessible only within // the cluster. No DNS is managed. Useful for fronting with an OCP - // Route on bare-metal clusters without a hardware load balancer. + // Route on bare-metal clusters without a hardware load balancer; + // see https://opendatahub-io.github.io/models-as-a-service/v2.0.1/configuration-and-management/gateway-patterns/#clusterip-gateway-with-openshift-route-re-encrypt // // When omitted, defaults to LoadBalancer. // From 0dfc0ed74c204eda93f8a6fd2aebac2f6302b856 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 13:53:55 -0400 Subject: [PATCH 11/20] enhancements/ingress: Remove MAAS link from API doc comment Co-Authored-By: Claude Sonnet 4.6 --- enhancements/ingress/gateway-api-gateway-customization.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index e7c71b9b13..6c8787bdc1 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -302,8 +302,7 @@ type GatewayServiceParameters struct { // // ClusterIP provisions a ClusterIP Service accessible only within // the cluster. No DNS is managed. Useful for fronting with an OCP - // Route on bare-metal clusters without a hardware load balancer; - // see https://opendatahub-io.github.io/models-as-a-service/v2.0.1/configuration-and-management/gateway-patterns/#clusterip-gateway-with-openshift-route-re-encrypt + // Route on bare-metal clusters without a hardware load balancer. // // When omitted, defaults to LoadBalancer. // From 647e511f7353b0f2fc536e9e3ade51b25dce4ace Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 13:54:41 -0400 Subject: [PATCH 12/20] enhancements/ingress: Raise Goals abstraction level, remove OVN detail Co-Authored-By: Claude Sonnet 4.6 --- enhancements/ingress/gateway-api-gateway-customization.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 6c8787bdc1..8c082b6ab6 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -112,9 +112,9 @@ avoiding cross-zone hops and preserving source IP. topology of a GatewayClass (LoadBalancer, NodePort, ClusterIP) and endpoint traffic policy, using field names that mirror the Kubernetes Service API. -- The OVN `local-with-fallback` annotation is derived automatically when - `externalTrafficPolicy: Local` is set; all other service annotations - are the administrator's responsibility. +- Where CIO can derive platform-specific configuration from the user's + expressed intent, it does so automatically — administrators express + what they want, not how to achieve it on a given platform. - The API is implementation-agnostic and upgrade-safe: customizations survive Gateway API implementation changes without user intervention. - The `openshift-default` GatewayClass is unchanged. From dcbab708ef969c2310ae64e903492ed09d9e12ab Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 13:56:05 -0400 Subject: [PATCH 13/20] enhancements/ingress: Use OpenShift-specific ETP enum, link prior proposal Replace corev1.ServiceExternalTrafficPolicy passthrough with a new GatewayExternalTrafficPolicy enum (LocalWithFallback, Cluster). LocalWithFallback makes the OVN local-with-fallback behavior explicit rather than silently bundling it with a plain Local value. Link PR #1990 in the hardcoded GatewayClasses alternative. Note the field-name-mirrors-K8s / values-are-OpenShift distinction in the Alternatives section. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 68 +++++++++++++------ 1 file changed, 47 insertions(+), 21 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 8c082b6ab6..468106cf7a 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -229,7 +229,7 @@ CIO never mutates the user's `Gateway` or `GatewayClass` resources. spec: service: type: LoadBalancer - externalTrafficPolicy: Local + externalTrafficPolicy: LocalWithFallback ``` 2. The cluster administrator creates a GatewayClass referencing it @@ -296,8 +296,8 @@ type GatewayServiceParameters struct { // // NodePort provisions a NodePort Service. No DNS is managed. // The administrator is responsible for configuring an external - // load balancer. When externalTrafficPolicy is Local, the external - // load balancer MUST health-check nodes via + // load balancer. When externalTrafficPolicy is LocalWithFallback, + // the external load balancer MUST health-check nodes via // Service.spec.healthCheckNodePort. // // ClusterIP provisions a ClusterIP Service accessible only within @@ -311,24 +311,43 @@ type GatewayServiceParameters struct { Type *corev1.ServiceType `json:"type,omitempty"` // externalTrafficPolicy specifies how external traffic is routed to - // Gateway proxy pods. Mirrors the Kubernetes Service - // spec.externalTrafficPolicy field. Only meaningful when type is - // LoadBalancer or NodePort. + // Gateway proxy pods. Only meaningful when type is LoadBalancer or + // NodePort. // - // Local routes external traffic only to proxy pods on the receiving - // node, preserving source IP and avoiding cross-zone hops. CIO - // automatically adds the OVN local-with-fallback annotation on - // applicable platforms to prevent traffic drops during rolling updates. + // LocalWithFallback sets Kubernetes externalTrafficPolicy to Local, + // preserving source IP and preferring the local node to avoid + // cross-zone hops. On applicable platforms, CIO also sets the OVN + // local-with-fallback annotation so that traffic is not dropped on + // nodes without a local proxy pod during rolling updates or uneven + // pod scheduling. // // Cluster routes traffic to any proxy pod (with SNAT). Source IP - // is not preserved. + // is not preserved but load distribution is even regardless of pod + // placement. // // When omitted, the Gateway API implementation default applies. // // +optional - // +kubebuilder:validation:Enum=Local;Cluster - ExternalTrafficPolicy *corev1.ServiceExternalTrafficPolicy `json:"externalTrafficPolicy,omitempty"` + // +kubebuilder:validation:Enum=LocalWithFallback;Cluster + ExternalTrafficPolicy *GatewayExternalTrafficPolicy `json:"externalTrafficPolicy,omitempty"` } + +// GatewayExternalTrafficPolicy specifies how external traffic is routed +// to Gateway proxy pods. +// +kubebuilder:validation:Enum=LocalWithFallback;Cluster +type GatewayExternalTrafficPolicy string + +const ( + // GatewayExternalTrafficPolicyLocalWithFallback sets + // externalTrafficPolicy: Local on the backing Service and, on + // applicable platforms, adds the OVN local-with-fallback annotation + // to prevent traffic drops when no local proxy pod is present. + GatewayExternalTrafficPolicyLocalWithFallback GatewayExternalTrafficPolicy = "LocalWithFallback" + + // GatewayExternalTrafficPolicyCluster routes traffic to any proxy + // pod in the cluster (with SNAT). Source IP is not preserved. + GatewayExternalTrafficPolicyCluster GatewayExternalTrafficPolicy = "Cluster" +) ``` #### ValidatingAdmissionPolicy @@ -458,11 +477,11 @@ spec: endpointTrafficPolicy: Local ``` -This was rejected in favour of passthrough field mirroring because: +This was rejected in favour of mirroring Kubernetes field names because: -- Administrators who know Kubernetes already know `service.type: ClusterIP` - and `externalTrafficPolicy: Local`. A translation layer (`ClusterIPService`, - `endpointTrafficPolicy`) adds cognitive overhead with no benefit. +- Administrators who know Kubernetes already know `service.type: ClusterIP`. + A translation layer (`ClusterIPService`, `endpointTrafficPolicy`) adds + cognitive overhead with no benefit. - Mirroring stable Kubernetes API field names means the OpenShift API is less likely to need changes as Kubernetes evolves. - The abstraction required an `Internal`/`External` scope discriminator @@ -471,6 +490,12 @@ This was rejected in favour of passthrough field mirroring because: - Other Gateway API implementations (e.g. Envoy Gateway) use the mirroring approach for the same reasons. +Note: `externalTrafficPolicy` field values are OpenShift-specific +(`LocalWithFallback`, `Cluster`) rather than pure Kubernetes passthroughs, +because the OpenShift value carries additional platform behaviour (OVN +local-with-fallback) that plain Kubernetes `Local` does not. The field +name mirrors Kubernetes; the values make the delivered behaviour explicit. + ### Extending the `Ingress` Singleton Adding `spec.gatewayAPI.customGatewayClasses[]` to the existing @@ -489,10 +514,11 @@ This was rejected because: ### Hardcoded GatewayClasses -Creating fixed GatewayClasses (`openshift-external`, `openshift-internal`, -`openshift-clusterip`) was rejected because it requires a code change -for each new configuration combination and creates a permanent VAP -allowlist maintenance burden. +[A prior proposal](https://github.com/openshift/enhancements/pull/1990) +created fixed GatewayClasses (`openshift-external`, `openshift-internal`, +`openshift-clusterip`). This was rejected because it requires a code +change for each new configuration combination and creates a permanent +VAP allowlist maintenance burden. ### Wait for Upstream Standardization From e138699770cc54ca4fc837f07fd3d4e717cd3361 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 14:02:01 -0400 Subject: [PATCH 14/20] enhancements/ingress: Add GatewayParameters status conditions, default ETP to LocalWithFallback Add GatewayParametersStatus with an Accepted condition (reasons: Accepted, InvalidParameters, ImplementationNotReady, NoReferencingGatewayClass, Pending) following the condition style in CIO PR #1547 and Gateway API conventions. Default externalTrafficPolicy to LocalWithFallback via +kubebuilder:default. Document that openshift-default also gets LocalWithFallback when the feature is enabled, with Cluster as an opt-out. Name the OVN annotation explicitly in Implementation Details (not in API docs). Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 70 +++++++++++++++---- 1 file changed, 56 insertions(+), 14 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 468106cf7a..371213cc7c 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -272,6 +272,35 @@ type GatewayParameters struct { Status GatewayParametersStatus `json:"status,omitempty"` } +// GatewayParametersStatus is the observed state of a GatewayParameters resource. +type GatewayParametersStatus struct { + // conditions describe the current state of the GatewayParameters resource. + // + // Known condition types: + // + // * "Accepted" indicates whether CIO has accepted this GatewayParameters + // and successfully reconciled it into the Gateway API implementation + // configuration. Reasons: + // - Accepted: CIO found a GatewayClass referencing this resource and + // the implementation configuration has been applied. + // - InvalidParameters: The spec contains invalid or unsupported values. + // - ImplementationNotReady: The Gateway API implementation version does + // not support the configuration mechanism required by this resource. + // - NoReferencingGatewayClass: No GatewayClass with the OpenShift + // controller name references this resource. + // - Pending: CIO has not yet reconciled this resource. + // + // +listType=map + // +listMapKey=type + // +optional + // +kubebuilder:validation:MaxItems=8 + Conditions []metav1.Condition `json:"conditions,omitempty"` + + // observedGeneration is the most recent generation observed by CIO. + // +optional + ObservedGeneration int64 `json:"observedGeneration,omitempty"` +} + type GatewayParametersSpec struct { // service configures the Kubernetes Service provisioned for Gateways // referencing this GatewayClass. Fields mirror the corresponding @@ -316,18 +345,17 @@ type GatewayServiceParameters struct { // // LocalWithFallback sets Kubernetes externalTrafficPolicy to Local, // preserving source IP and preferring the local node to avoid - // cross-zone hops. On applicable platforms, CIO also sets the OVN - // local-with-fallback annotation so that traffic is not dropped on - // nodes without a local proxy pod during rolling updates or uneven - // pod scheduling. + // cross-zone hops. Traffic is not dropped on nodes without a local + // proxy pod. This is the recommended value for most deployments. // // Cluster routes traffic to any proxy pod (with SNAT). Source IP // is not preserved but load distribution is even regardless of pod // placement. // - // When omitted, the Gateway API implementation default applies. + // Defaults to LocalWithFallback. // // +optional + // +kubebuilder:default=LocalWithFallback // +kubebuilder:validation:Enum=LocalWithFallback;Cluster ExternalTrafficPolicy *GatewayExternalTrafficPolicy `json:"externalTrafficPolicy,omitempty"` } @@ -410,17 +438,30 @@ explicitly unsupported and may be overwritten at any time. #### Platform Annotation Derivation -CIO derives one platform-specific annotation automatically: - -- `externalTrafficPolicy: Local` → OVN - `traffic-policy.network.alpha.openshift.io/local-with-fallback: ""` - on applicable platforms, preventing traffic drops during rolling updates. +When `externalTrafficPolicy` is `LocalWithFallback`, CIO automatically +sets the OVN annotation `traffic-policy.network.alpha.openshift.io/local-with-fallback: ""` +on the provisioned Service on applicable platforms. This prevents traffic +from being dropped on nodes without a local proxy pod during rolling +updates or uneven pod scheduling, making `LocalWithFallback` safer than +bare Kubernetes `externalTrafficPolicy: Local`. All other service annotations (e.g. cloud provider internal/external LB annotations) are the administrator's responsibility and should be set directly on the GatewayClass or Gateway resource, where they propagate to the provisioned Service. +#### Default ExternalTrafficPolicy and openshift-default + +When this feature is enabled, CIO also updates the `openshift-default` +GatewayClass to use `LocalWithFallback` by updating its defaults +ConfigMap. This is a deliberate behavior change: the recommended +configuration for new and existing deployments is `LocalWithFallback`. + +Administrators who need `Cluster` behavior (e.g. to avoid the MetalLB +BGP pod-scheduling constraint described in Risks) can create a custom +GatewayClass with a `GatewayParameters` CR setting +`externalTrafficPolicy: Cluster`. + #### Deletion Semantics When a `GatewayParameters` CR is deleted, CIO removes the defaults @@ -554,10 +595,11 @@ the goal of an implementation-agnostic OpenShift configuration layer. ## Open Questions -1. **`GatewayParameters` status**: What conditions should be reported? - At minimum: `Accepted` (CIO has found a referencing GatewayClass and - created the ConfigMap) and `Degraded` (OSSM version too old, ConfigMap - sync failure). +1. **`openshift-default` ETP change**: Changing `openshift-default` to + `LocalWithFallback` is a behavior change for existing clusters. Should + this be gated behind the same `GatewayClassParameters` feature gate, + or should it be a separate gate? Is the MetalLB BGP risk (see Risks) + significant enough to require a more cautious rollout? 2. **Default when `endpointPublishingStrategy` is omitted**: Should it default to `LoadBalancerService` with `scope: External`, or should From 5c08eb6b159c5734aa5f3386d640628b60ecad11 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 14:06:48 -0400 Subject: [PATCH 15/20] enhancements/ingress: Use values.gatewayClasses global patch for LocalWithFallback default The existing Sail values.gatewayClasses patch (already used by CIO) sets the global ETP default for all CIO-managed GatewayClasses. Per-class GatewayParameters ConfigMaps override it for individual classes. Document the interaction and flag the per-class precedence as needing confirmation. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 42 ++++++++++++------- 1 file changed, 26 insertions(+), 16 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 371213cc7c..d374c05fd2 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -450,17 +450,25 @@ annotations) are the administrator's responsibility and should be set directly on the GatewayClass or Gateway resource, where they propagate to the provisioned Service. -#### Default ExternalTrafficPolicy and openshift-default - -When this feature is enabled, CIO also updates the `openshift-default` -GatewayClass to use `LocalWithFallback` by updating its defaults -ConfigMap. This is a deliberate behavior change: the recommended -configuration for new and existing deployments is `LocalWithFallback`. - -Administrators who need `Cluster` behavior (e.g. to avoid the MetalLB -BGP pod-scheduling constraint described in Risks) can create a custom -GatewayClass with a `GatewayParameters` CR setting -`externalTrafficPolicy: Cluster`. +#### Default ExternalTrafficPolicy and Interaction with Global Patch + +CIO already manages a global Sail Helm `values.gatewayClasses` patch +that applies service and deployment configuration to all CIO-managed +GatewayClasses. When this feature is enabled, CIO sets +`externalTrafficPolicy: Local` and the OVN `local-with-fallback` +annotation in that global patch, making `LocalWithFallback` the default +for all CIO-managed GatewayClasses — including `openshift-default` — +without requiring a per-class ConfigMap for the common case. + +The per-class `gateway.istio.io/defaults-for-class` ConfigMap (created +by CIO when a `GatewayParameters` CR is present) takes precedence over +the global patch for that specific class. A `GatewayParameters` CR with +`externalTrafficPolicy: Cluster` therefore produces a per-class ConfigMap +that overrides the global `LocalWithFallback` default for that class only. + +This interaction must be confirmed against the OSSM/Sail implementation +to ensure per-class ConfigMap precedence over the global patch is +guaranteed. #### Deletion Semantics @@ -595,11 +603,13 @@ the goal of an implementation-agnostic OpenShift configuration layer. ## Open Questions -1. **`openshift-default` ETP change**: Changing `openshift-default` to - `LocalWithFallback` is a behavior change for existing clusters. Should - this be gated behind the same `GatewayClassParameters` feature gate, - or should it be a separate gate? Is the MetalLB BGP risk (see Risks) - significant enough to require a more cautious rollout? +1. **Global ETP default behavior change**: Setting `LocalWithFallback` via + the global `values.gatewayClasses` patch is a behavior change for + existing clusters using `openshift-default`. Should this be gated + behind the same `GatewayClassParameters` feature gate or a separate + one? Confirmed: does the per-class `gateway.istio.io/defaults-for-class` + ConfigMap take precedence over the global patch in all OSSM versions + that support this feature? 2. **Default when `endpointPublishingStrategy` is omitted**: Should it default to `LoadBalancerService` with `scope: External`, or should From 81d703cb85a25f8dda833d56f3ca855fc7bcb664 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 14:09:07 -0400 Subject: [PATCH 16/20] enhancements/ingress: Gate global ETP default change behind GatewayClassParameters The LocalWithFallback global default and the GatewayParameters opt-out are introduced atomically under the same feature gate. Resolve the open question about feature gate scope; keep only the OSSM precedence confirmation question open. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index d374c05fd2..925bacd024 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -454,11 +454,14 @@ to the provisioned Service. CIO already manages a global Sail Helm `values.gatewayClasses` patch that applies service and deployment configuration to all CIO-managed -GatewayClasses. When this feature is enabled, CIO sets +GatewayClasses. When the `GatewayClassParameters` feature gate is enabled, CIO sets `externalTrafficPolicy: Local` and the OVN `local-with-fallback` annotation in that global patch, making `LocalWithFallback` the default for all CIO-managed GatewayClasses — including `openshift-default` — -without requiring a per-class ConfigMap for the common case. +without requiring a per-class ConfigMap for the common case. The global +default change and the opt-out mechanism (`GatewayParameters` CRD) are +introduced atomically under the same feature gate so that the default +is never changed without an opt-out being available. The per-class `gateway.istio.io/defaults-for-class` ConfigMap (created by CIO when a `GatewayParameters` CR is present) takes precedence over @@ -603,13 +606,10 @@ the goal of an implementation-agnostic OpenShift configuration layer. ## Open Questions -1. **Global ETP default behavior change**: Setting `LocalWithFallback` via - the global `values.gatewayClasses` patch is a behavior change for - existing clusters using `openshift-default`. Should this be gated - behind the same `GatewayClassParameters` feature gate or a separate - one? Confirmed: does the per-class `gateway.istio.io/defaults-for-class` - ConfigMap take precedence over the global patch in all OSSM versions - that support this feature? +1. **Per-class ConfigMap precedence**: Does the `gateway.istio.io/defaults-for-class` + ConfigMap take precedence over the global `values.gatewayClasses` patch + in all OSSM versions that support this feature? This must be confirmed + before the opt-out path can be relied upon. 2. **Default when `endpointPublishingStrategy` is omitted**: Should it default to `LoadBalancerService` with `scope: External`, or should From 0a565410aa90322b0554877dcb7e9d029d9af91a Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 14:11:05 -0400 Subject: [PATCH 17/20] enhancements/ingress: Add cost-saving ClusterIP user story Add Story 2 for users who already have an IngressController LoadBalancer and want to avoid a second cloud LB by routing Gateway traffic through existing OCP Route/HAProxy infrastructure. Co-Authored-By: Claude Sonnet 4.6 --- .../ingress/gateway-api-gateway-customization.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 925bacd024..7628966011 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -97,7 +97,16 @@ Route using the existing HAProxy ingress infrastructure. This pattern is documented publicly by the Red Hat AI platform: [ClusterIP Gateway with OpenShift Route (re-encrypt)](https://opendatahub-io.github.io/models-as-a-service/v2.0.1/configuration-and-management/gateway-patterns/#clusterip-gateway-with-openshift-route-re-encrypt). -#### Story 2: Zone-Aware External Gateway with ETP Local +#### Story 2: Cost-Efficient Gateway on Clusters with an Existing Load Balancer + +As a cluster administrator whose cluster already has an IngressController +backed by a cloud load balancer, I want to create a GatewayClass that +provisions Gateways with a ClusterIP service so that I can expose Gateway +traffic through the existing OCP Route and HAProxy infrastructure rather +than provisioning a second cloud load balancer, reducing cost while +accepting the additional hop through the IngressController. + +#### Story 3: Zone-Aware External Gateway with ETP Local As a cluster administrator running a cluster across multiple availability zones with BGP-based networking, I want to configure From dce058e6ddd6d1bdd1e2d57299d7269adc5f5074 Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 14:13:47 -0400 Subject: [PATCH 18/20] enhancements/ingress: Remove CIO from Go API doc comments Use "the ingress operator" / "the operator" in API doc comments. CIO is an internal team abbreviation, not appropriate for user-facing API docs. Co-Authored-By: Claude Sonnet 4.6 --- .../ingress/gateway-api-gateway-customization.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 7628966011..52ce8b6351 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -287,17 +287,17 @@ type GatewayParametersStatus struct { // // Known condition types: // - // * "Accepted" indicates whether CIO has accepted this GatewayParameters - // and successfully reconciled it into the Gateway API implementation - // configuration. Reasons: - // - Accepted: CIO found a GatewayClass referencing this resource and + // * "Accepted" indicates whether the ingress operator has accepted this + // GatewayParameters and successfully reconciled it into the Gateway API + // implementation configuration. Reasons: + // - Accepted: A GatewayClass referencing this resource was found and // the implementation configuration has been applied. // - InvalidParameters: The spec contains invalid or unsupported values. // - ImplementationNotReady: The Gateway API implementation version does // not support the configuration mechanism required by this resource. // - NoReferencingGatewayClass: No GatewayClass with the OpenShift // controller name references this resource. - // - Pending: CIO has not yet reconciled this resource. + // - Pending: This resource has not yet been reconciled. // // +listType=map // +listMapKey=type @@ -305,7 +305,7 @@ type GatewayParametersStatus struct { // +kubebuilder:validation:MaxItems=8 Conditions []metav1.Condition `json:"conditions,omitempty"` - // observedGeneration is the most recent generation observed by CIO. + // observedGeneration is the most recent generation observed by the operator. // +optional ObservedGeneration int64 `json:"observedGeneration,omitempty"` } @@ -330,7 +330,7 @@ type GatewayServiceParameters struct { // Mirrors the Kubernetes Service spec.type field. // // LoadBalancer provisions a cloud or hardware load balancer. - // CIO manages DNS automatically. + // DNS is managed automatically by the ingress operator. // // NodePort provisions a NodePort Service. No DNS is managed. // The administrator is responsible for configuring an external From ec429989920effb30611ac2fd577fcab744541af Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 14:14:29 -0400 Subject: [PATCH 19/20] enhancements/ingress: Remove future fields comment from API spec Co-Authored-By: Claude Sonnet 4.6 --- enhancements/ingress/gateway-api-gateway-customization.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 52ce8b6351..9dc105b638 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -318,8 +318,6 @@ type GatewayParametersSpec struct { // +optional Service *GatewayServiceParameters `json:"service,omitempty"` - // Future fields (not in this EP): - // deployment *GatewayDeploymentParameters } // GatewayServiceParameters mirrors selected Kubernetes Service spec fields, From b0961ccc1f66b85fa61e09b7501c6b4b9336060c Mon Sep 17 00:00:00 2001 From: Grant Spence Date: Fri, 4 Sep 2026 14:22:36 -0400 Subject: [PATCH 20/20] enhancements/ingress: Move GatewayParameters status to GatewayClass conditions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Remove conditions list from GatewayParametersStatus — CIO already sets conditions on GatewayClass.status, which is the natural per-class surfacing point and handles multiple GatewayClasses referencing the same CR cleanly. GatewayParametersStatus retains only observedGeneration. Remove stale endpointPublishingStrategy open question. Co-Authored-By: Claude Sonnet 4.6 --- .../gateway-api-gateway-customization.md | 37 +++---------------- 1 file changed, 5 insertions(+), 32 deletions(-) diff --git a/enhancements/ingress/gateway-api-gateway-customization.md b/enhancements/ingress/gateway-api-gateway-customization.md index 9dc105b638..dc6f3de1f0 100644 --- a/enhancements/ingress/gateway-api-gateway-customization.md +++ b/enhancements/ingress/gateway-api-gateway-customization.md @@ -283,28 +283,6 @@ type GatewayParameters struct { // GatewayParametersStatus is the observed state of a GatewayParameters resource. type GatewayParametersStatus struct { - // conditions describe the current state of the GatewayParameters resource. - // - // Known condition types: - // - // * "Accepted" indicates whether the ingress operator has accepted this - // GatewayParameters and successfully reconciled it into the Gateway API - // implementation configuration. Reasons: - // - Accepted: A GatewayClass referencing this resource was found and - // the implementation configuration has been applied. - // - InvalidParameters: The spec contains invalid or unsupported values. - // - ImplementationNotReady: The Gateway API implementation version does - // not support the configuration mechanism required by this resource. - // - NoReferencingGatewayClass: No GatewayClass with the OpenShift - // controller name references this resource. - // - Pending: This resource has not yet been reconciled. - // - // +listType=map - // +listMapKey=type - // +optional - // +kubebuilder:validation:MaxItems=8 - Conditions []metav1.Condition `json:"conditions,omitempty"` - // observedGeneration is the most recent generation observed by the operator. // +optional ObservedGeneration int64 `json:"observedGeneration,omitempty"` @@ -485,9 +463,8 @@ guaranteed. When a `GatewayParameters` CR is deleted, CIO removes the defaults ConfigMap but does not delete the `GatewayClass`. Deleting a GatewayClass while Gateways reference it would orphan running workloads. CIO sets a -condition on the `GatewayParameters` status and the `GatewayClass` -(`Accepted: False`, reason: `ParametersNotFound`) to notify the -administrator. +condition on the referencing `GatewayClass` status (`Accepted: False`, +reason: `InvalidParameters`) to notify the administrator. #### Feature Gate @@ -498,8 +475,8 @@ Gated behind `GatewayClassParameters` in `TechPreviewNoUpgrade`. **Risk**: OSSM version dependency for the defaults ConfigMap mechanism. **Mitigation**: CIO checks the OSSM version at reconciliation time and -sets a `Degraded` condition on the `GatewayParameters` status with a -clear message if the version requirement is not met. +sets a condition on the referencing `GatewayClass` status with a clear +message if the version requirement is not met. **Risk**: `externalTrafficPolicy: Local` with MetalLB BGP can cause traffic disruption if gateway pods are not spread across all nodes — @@ -618,11 +595,7 @@ the goal of an implementation-agnostic OpenShift configuration layer. in all OSSM versions that support this feature? This must be confirmed before the opt-out path can be relied upon. -2. **Default when `endpointPublishingStrategy` is omitted**: Should it - default to `LoadBalancerService` with `scope: External`, or should - the field be required? - -3. **GatewayClass ownership**: Should CIO require the GatewayClass to +2. **GatewayClass ownership**: Should CIO require the GatewayClass to use the OpenShift controllerName before reconciling a referenced `GatewayParameters`, or reconcile for any GatewayClass that points to a `GatewayParameters` CR?