Skip to content

fix(opal-server): propagate scope data config to scoped clients#918

Open
Kyzgor wants to merge 1 commit into
permitio:masterfrom
Kyzgor:fix/779-scope-data-config
Open

fix(opal-server): propagate scope data config to scoped clients#918
Kyzgor wants to merge 1 commit into
permitio:masterfrom
Kyzgor:fix/779-scope-data-config

Conversation

@Kyzgor

@Kyzgor Kyzgor commented Jun 14, 2026

Copy link
Copy Markdown

Fixes Issue

Closes #779

Changes proposed

In scopes mode (OPAL_SCOPES=1), a scoped agent that connects before its scope is created now receives the
scope's data-source configuration.

Two paths led to the reported symptom. On scope create/update the server published a policy sync trigger
but no data update, and the client fetches its data config only on (re)connect — so an agent connected
before its scope existed loaded policy but none of the scope's data, and every authorization decision that
depends on that data failed. Separately, while a scope is not yet synced, GET /scopes/{scope_id}/data
fell back to the server-global OPAL_DATA_CONFIG_SOURCES, whose bare policy_data topic is disjoint from a
scoped agent's {scope_id}:data:{topic} subscriptions, so the client discarded every entry.

The fix is server-side only (no client, schema, or protocol change), in
packages/opal-server/opal_server/scopes/api.py:

  1. Publish on PUT /scopes. After the scope is persisted, put_scope publishes its data.entries as
    a DataUpdate via the same DataUpdatePublisher the data-update endpoint uses, so agents that connected
    before the scope existed receive it. It fires only when the data config is new or changed
    (existing_scope.data != scope_in.data), mirroring the policy path's notify-on-change.
  2. Namespace topics to {scope_id}:data:{topic} at both the publish and the serve paths, via a small
    helper (normalize_data_topics_for_scope). Bare or default topics get the scope prefix; already-
    namespaced authored topics pass through unchanged. This also closes a permanent variant of the bug: an
    entry authored with the schema-default policy_data topic was discarded by the scoped agent on every
    base-fetch, independent of the create race. Normalization runs on a deep copy, so the persisted scope
    keeps the authored form.
  3. Absent-scope fallback. The except ScopeNotFoundError branch of get_scope_data_config returns an
    empty DataSourceConfig(entries=[]) instead of the server-global default; the external_source_url
    redirect branch is unchanged.

Adds packages/opal-server/opal_server/scopes/tests/test_scope_data_config_propagation.py (10 tests); the
scopes/ module had no tests before.

Check List (Check all the applicable boxes)

  • I sign off on contributing this submission to open-source
  • My code follows the code style of this project.
  • My change requires changes to the documentation.
  • I have updated the documentation accordingly.
  • All new and existing tests passed.
  • This PR does not contain plagiarized content.
  • The title of my pull request is a short description of the requested changes.

Screenshots

N/A — server behavior change; before/after is below.

Note to reviewers

Reproduce with a scopes-mode server and a scoped client (OPAL_SCOPE_ID=documents, OPAL_DATA_TOPICS=data)
that connects before the scope exists, then PUT /scopes for documents with a data.entries[] config
topic'd documents:data:data and a rego-only policy source. Before the fix, the server logs Requested scope documents not found, returning OPAL_DATA_CONFIG_SOURCES, serves the bare-policy_data default the
scoped client discards, and the scope's data never reaches OPA; after, the client receives the namespaced
update, writes the entries to OPA, and they persist. The new suite is 10 tests, all green; reverting only
this fix makes 7 of the 10 fail, and all 10 pass once it's restored. The existing data-updater,
server-to-client integration, and data-update-publisher tests stay green, and flake8 (the CI selection)
is clean on the changed file.

On the absent-scope fallback I return HTTP 200 with an empty config rather than 404, because
DataUpdater.get_policy_data_config returns only on 200 and raises (uncaught at connect) on anything else,
so a 404 would break currently deployed clients. Happy to switch to a 404/not-ready contract instead given
a coordinated client/server version bump.

One known limitation, not introduced here: entries delivered by this push are fetched once and don't begin
periodic_update_interval polling until the client's next reconnect, since the client schedules polling
only in its on-connect base fetch. That still beats today's behavior (nothing propagates) and matches
every other pushed DataUpdate; I can follow up if you'd like periodic entries to self-schedule on push.

Deliberately out of scope, to keep this to one concern: a policy repo that ships a root data.json (the
bundle import writes it to OPA's root document and can overwrite delivered data — a pre-existing, OPAL-wide
interaction; use a rego-only source to reproduce #779 cleanly); the policy route's own not-ready handling
and any scope create/sync ordering redesign; and a couple of related issues in adjacent scope code paths
that I'll file separately. The docs note for the changed fallback and the namespacing contract is
intentionally deferred to a follow-up.

In scopes mode, a scoped agent that connects before its scope is
created never receives the scope's data-source configuration. On scope
create/update the server publishes a policy sync trigger but no data
update, and the client fetches its data config only on (re)connect, so
the agent loads policy but none of the scope's data and every
authorization decision that depends on it fails (permitio#779). The
not-yet-synced data route also falls back to the server-global
OPAL_DATA_CONFIG_SOURCES, whose bare `policy_data` topic is disjoint
from a scoped agent's `{scope_id}:data:{topic}` subscriptions, so the
client discards every entry.

Fix, server-side only:
- put_scope publishes the scope's data entries as a DataUpdate (via the
  existing DataUpdatePublisher) when the data config is new or changed,
  so already-connected scoped clients receive it.
- Entry topics served and published are namespaced
  `{scope_id}:data:{topic}` (on a deep copy; the persisted scope keeps
  the authored form). Bare/default topics get the scope prefix;
  already-namespaced topics pass through. This also closes a permanent
  variant where a default-topic entry was discarded on every base-fetch.
- The absent-scope branch of get_scope_data_config returns an empty
  DataSourceConfig(entries=[]) instead of the server-global default; the
  external_source_url redirect branch is unchanged.

Adds a 10-test regression suite under
packages/opal-server/opal_server/scopes/tests/ (the scopes module had
no tests before).

Closes permitio#779
@netlify

netlify Bot commented Jun 14, 2026

Copy link
Copy Markdown

Deploy Preview for opal-docs canceled.

Name Link
🔨 Latest commit cef7157
🔍 Latest deploy log https://app.netlify.com/projects/opal-docs/deploys/6a2e865251f0030008bdb696

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OPAL Agent does not receive data source configuration after scope update

1 participant