Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,72 @@ make manifests

More information can be found via the [Kubebuilder Documentation](https://book.kubebuilder.io/introduction.html)

## OVN controller probe configuration

The `ovn-controller` container checks liveness and readiness every 5 seconds,
with a 5-second timeout and a 30-second initial delay. Three consecutive
liveness failures restart the container; readiness failures mark the Pod
unready. Liveness checks that the process exists and `debug/status` returns
`running`. Readiness checks that `connection-status` returns `connected`.
These commands use the controller's local control socket. Heavy flow computation
can delay their responses even when the controller is making progress.

Configure the container through `OVNController.spec.override.probes`. For example:

```yaml
apiVersion: ovn.openstack.org/v1beta1
kind: OVNController
metadata:
name: ovncontroller
spec:
override:
probes:
livenessProbes:
timeoutSeconds: 30
```

The value above illustrates the API; select timings using measured probe latency
and the acceptable time to detect a persistent hang. Existing defaults are
unchanged when overrides are omitted. Increasing tolerance prevents premature
restarts but does not make flow computation faster. Readiness can be tuned
independently using `readinessProbes`.

The shared lib-common API supports `initialDelaySeconds`, `timeoutSeconds`,
`periodSeconds`, and `failureThreshold`, plus `type` (`Exec` or `HTTP`),
`command`, `path`, `port`, and `scheme`. Partial overrides inherit the defaults;
zero timing values use lib-common's inheritance semantics. Explicit zero timeout,
period, and failure threshold values are rejected by the CRD schema. Setting an
initial delay to zero retains the default delay. HTTP checks require a port and
a reachable endpoint supplied by the chosen image; the default controller does
not expose an HTTP health endpoint.

Startup probing is disabled by default. Providing `startupProbes` enables it,
starting with the default liveness command and timings. For example,
`startupProbes: {failureThreshold: 60}` tolerates a longer initialization.
Startup probing gates liveness and readiness until it succeeds; it does not
protect a controller that stalls after startup. Removing overrides restores
the defaults. Changes update the DaemonSet Pod template and cause a rolling
update. OVS and metrics container probes are unaffected.

For deployments managed by `OpenStackControlPlane`, configure the parent CR:

```yaml
spec:
ovn:
template:
ovnController:
override:
probes:
livenessProbes:
timeoutSeconds: 30
```

This requires an openstack-operator release that includes the updated OVN API,
parent CRD, and bundled OVN CRD. Updating only ovn-operator does not add fields
to the parent CRD. Configure the parent rather than patching its child CR or
DaemonSet, which reconciliation manages.


## License

Copyright 2022.
Expand Down
151 changes: 151 additions & 0 deletions api/bases/ovn.openstack.org_ovncontrollers.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,157 @@ spec:
description: NodeSelector to target subset of worker nodes running
this service
type: object
override:
description: |-
Override customizes the generated ovn-controller container. It does not
change the probes of the OVS or metrics containers.
properties:
probes:
description: |-
Probes overrides liveness and readiness checks and optionally enables a
startup check. Unspecified settings retain the operator defaults.
properties:
livenessProbes:
description: |-
Override configuration for the StatefulSet like Probes and other tunable
fields
properties:
command:
items:
type: string
type: array
x-kubernetes-list-type: atomic
failureThreshold:
format: int32
minimum: 1
type: integer
initialDelaySeconds:
format: int32
minimum: 0
type: integer
path:
pattern: ^(/.*)?$
type: string
periodSeconds:
format: int32
minimum: 1
type: integer
port:
format: int32
maximum: 65535
minimum: 1
type: integer
scheme:
description: URIScheme identifies the scheme used for
connection to a host for Get actions
type: string
timeoutSeconds:
format: int32
minimum: 1
type: integer
type:
description: ProbeHandlerType defines the type of probe
handler
enum:
- HTTP
- Exec
- ""
type: string
type: object
readinessProbes:
description: ProbeConf - the configuration for liveness and
readiness probes
properties:
command:
items:
type: string
type: array
x-kubernetes-list-type: atomic
failureThreshold:
format: int32
minimum: 1
type: integer
initialDelaySeconds:
format: int32
minimum: 0
type: integer
path:
pattern: ^(/.*)?$
type: string
periodSeconds:
format: int32
minimum: 1
type: integer
port:
format: int32
maximum: 65535
minimum: 1
type: integer
scheme:
description: URIScheme identifies the scheme used for
connection to a host for Get actions
type: string
timeoutSeconds:
format: int32
minimum: 1
type: integer
type:
description: ProbeHandlerType defines the type of probe
handler
enum:
- HTTP
- Exec
- ""
type: string
type: object
startupProbes:
description: ProbeConf - the configuration for liveness and
readiness probes
properties:
command:
items:
type: string
type: array
x-kubernetes-list-type: atomic
failureThreshold:
format: int32
minimum: 1
type: integer
initialDelaySeconds:
format: int32
minimum: 0
type: integer
path:
pattern: ^(/.*)?$
type: string
periodSeconds:
format: int32
minimum: 1
type: integer
port:
format: int32
maximum: 65535
minimum: 1
type: integer
scheme:
description: URIScheme identifies the scheme used for
connection to a host for Get actions
type: string
timeoutSeconds:
format: int32
minimum: 1
type: integer
type:
description: ProbeHandlerType defines the type of probe
handler
enum:
- HTTP
- Exec
- ""
type: string
type: object
type: object
type: object
ovnContainerImage:
description: Image used for the ovn-controller container (will be
set to environmental default if empty)
Expand Down
95 changes: 95 additions & 0 deletions api/v1beta1/ovncontroller_probes.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
/*
Copyright 2026 Red Hat Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1beta1

import (
"github.com/openstack-k8s-operators/lib-common/modules/common/probes"
corev1 "k8s.io/api/core/v1"
"k8s.io/apimachinery/pkg/util/validation/field"
)

// GetProbeSet builds the effective ovn-controller probes. Startup is opt-in.
// Keeping defaults and validation in the API also lets parent operators
// validate partial overrides against the same defaults used by reconciliation.
func (spec *OVNControllerSpecCore) GetProbeSet() (*probes.ProbeSet, error) {
set, errors := spec.buildProbes(field.NewPath("spec", "override", "probes"))
return set, errors.ToAggregate()
}

// ValidateProbes validates overrides and their effective configurations.
// basePath points to the probes object in either the child or parent CR.
func (spec *OVNControllerSpecCore) ValidateProbes(basePath *field.Path) field.ErrorList {
_, errors := spec.buildProbes(basePath)
return errors
}

func (spec *OVNControllerSpecCore) buildProbes(basePath *field.Path) (*probes.ProbeSet, field.ErrorList) {
set := &probes.ProbeSet{}
errors := field.ErrorList{}
for _, probe := range []struct {
name string
override *probes.ProbeConf
script string
target **corev1.Probe
}{
{"livenessProbes", spec.Override.Probes.LivenessProbes, "ovn_controller_liveness.sh", &set.Liveness},
{"readinessProbes", spec.Override.Probes.ReadinessProbes, "ovn_controller_readiness.sh", &set.Readiness},
{"startupProbes", spec.Override.Probes.StartupProbes, "ovn_controller_liveness.sh", &set.Startup},
} {
if probe.name == "startupProbes" && probe.override == nil {
continue
}
config := probes.ProbeConf{
Type: probes.ProbeHandlerExec,
Command: []string{"/usr/local/bin/container-scripts/" + probe.script},
InitialDelaySeconds: 30,
TimeoutSeconds: 5,
PeriodSeconds: 5,
FailureThreshold: 3,
}
path := basePath.Child(probe.name)
if probe.override != nil {
// Validate before merging: Merge ignores non-positive timing values.
// An explicitly selected Exec handler may inherit the default command.
supplied := *probe.override
if supplied.Type == probes.ProbeHandlerExec && len(supplied.Command) == 0 {
supplied.Command = config.Command
}
if errs := probes.ValidateProbeConf(path, &supplied); len(errs) > 0 {
errors = append(errors, errs...)
continue
}
config.Merge(*probe.override)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

config.Merge assigns Command by reference (lib-common ProbeConf.Merge: p.Command = overrides.Command), and it flows straight into the final corev1.Probe.Exec.Command. Since probe.override here is spec.Override.Probes.*.Command from the CR spec, a custom command ends up aliasing the cached CR's backing array instead of being copied. The existing test only covers the default-command path, not this one.

Suggest copying the slice before merging (supplied.Command = append([]string(nil), supplied.Command...)), or add a test proving the built probe's Command isn't the same slice as the CR spec's.

}
if errs := probes.ValidateProbeConf(path, &config); len(errs) > 0 {
errors = append(errors, errs...)
continue
}
if config.Type == probes.ProbeHandlerHTTP && config.Scheme != nil &&
*config.Scheme != "" && *config.Scheme != corev1.URISchemeHTTP && *config.Scheme != corev1.URISchemeHTTPS {
errors = append(errors, field.NotSupported(path.Child("scheme"), *config.Scheme, []corev1.URIScheme{corev1.URISchemeHTTP, corev1.URISchemeHTTPS}))
continue
}
built, err := probes.SetProbeConfV2(config)
if err != nil {
errors = append(errors, field.Invalid(path, nil, err.Error()))
continue
}
*probe.target = built
}
return set, errors
}
Loading
Loading