Skip to content

docs: admission replicas do not shard the in-memory policy cache #2146

Description

@guanchzhou

Problem

The high-availability and scaling pages say extra admission-controller replicas are for availability and scale. That is true for AdmissionReview QPS. It is easy to read as if the policy set is partitioned across replicas.

It is not. Each admission replica watches every Policy and ClusterPolicy and keeps a full in-process cache (pkg/policycache in kyverno/kyverno). Horizontal scale does not reduce per-pod memory. Per-pod RSS is O(policy count).

That matters when a cluster stores thousands of namespaced Policy resources (for example one Policy per tenant or workload). Adding replicas then multiplies the same cache instead of splitting it. If the webhook uses failurePolicy: Fail and every replica OOMs under a burst of admission traffic, matching admission is blocked cluster-wide.

Related scale report (3.5k Policy CRs): kyverno/kyverno#10458. Maintainers there pointed at collapsing many namespaced Policies into a ClusterPolicy plus Global Context, not at sharding the cache.

What already exists

  • Policy count is already a gauge: kyverno_policy_rule_info_total (metrics).
  • There is no cache-size-in-bytes metric. Counting policies plus pod RSS is enough to size vertically.
  • Sharding the cache would be a controller redesign (any replica must still evaluate any matching policy). This issue is documentation only.

Ask

State on the scaling and high-availability pages that:

  1. Extra admission replicas distribute AdmissionReview load, not the policy cache.
  2. Size admission memory for policy count; add replicas for webhook throughput and availability.
  3. Prefer fewer, broader policies over thousands of near-identical namespaced Policy resources.

I can send a docs PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions