Monitoring OPAL

There are multiple ways you can monitor your OPAL deployment:

• Logs - Using the structured logs outputted to stderr by both the OPAL-servers and OPAL-clients
• Health-checks - OPAL exposes HTTP health check endpoints (See below)
• Callbacks - Using the callback webhooks feature - having OPAL-clients report their updates
• Statistics - Using the built-in statistics feature in OPAL (See below)
• Metrics - DogStatsD metrics, among them those of the scope clones' git operations (See below)
• DataDog APM - OPAL supports DataDog APM tracing (requires ddtrace v4.8.2+; OPAL itself requires Python 3.11+)

Health checks

OPAL Server

opal-server exposes http health check endpoints on / & /healthcheck.
Currently it returns 200 OK as long as server is up.

OPAL Client

opal-client exposes 2 types of http health checks:

• Readiness - available on /ready.
  Returns 200 OK if the client have loaded policy & data to OPA at least once (from either server, or local backup file), otherwise 503 Unavailable is returned.

• Liveness - available on /, /healthcheck & /healthy.
  Returns 200 OK if the last attempts to load policy & data to OPA were successful, otherwise 503 Unavailable is returned

Notice: if you don't except your opal-client to load any data into OPA, set OPAL_DATA_UPDATER_ENABLED: False, so opal-client could report being healthy.

You can also configure opal-client to store dynamic health status as a document in OPA, Learn more here

OPAL Statistics

By enabling OPAL_STATISTICS_ENABLED=true (on both servers and clients), the OPAL-Server would start maintaining a unified state of all the clients and which topics they've subscribed to. The state can then be retrieved as a JSON object by calling the /statistics api route on the server

Code Reference:

• opal_server/statistics.py

Scope git metrics

With OPAL_ENABLE_METRICS=true, opal-server sends DogStatsD metrics to the agent at DD_AGENT_HOST (default localhost), port 8125, with permit.opal. before each name. In scopes mode (OPAL_SCOPES=true), these count what happens to the git operations on the scope clones:

Metric	Type	Tags	What it counts
opal_server.scopes.git_op_failures	count	op: fetch, clone reason: timeout, git_error, local	A fetch or clone of a scope's repository that failed. timeout: it ran past OPAL_SCOPES_GIT_FETCH_TIMEOUT; git_error: git or libgit2 reported an error. Both back the source off (OPAL_SCOPES_GIT_BACKOFF_BASE_SECONDS). local comes only with OPAL_SCOPES_GIT_FETCH_WITH_CLI on: a failure on the server's own side, such as no git, a clone that cannot be set up, a filesystem that refuses the clone's lock or an unreadable OPAL_SCOPES_GIT_SSH_KNOWN_HOSTS. It does not back the source off, unless the source's URL or credentials cannot be given to git.
opal_server.scopes.git_op_skipped	count	reason: backoff, clone_busy, repack_running, pass_in_flight	A git operation that did not run. backoff: a sync of the boot, periodic or pre-fork preload sync pass (not of the pass that POST /scopes/refresh starts), or one that a policy bundle request asked for, while its source is backed off. clone_busy: another git operation holds the clone's lock (the file .<source id>.lock beside the clone), so a fetch or clone with OPAL_SCOPES_GIT_FETCH_WITH_CLI on is skipped, a due repack waits for a later pass, and a clone that has to be removed to be cloned again is left as it is. repack_running: a due repack that did not start because another repack runs in the pod. pass_in_flight: a due repack put off while a sync pass runs in the same process.
opal_server.scopes.git_repack	count	outcome: ok, timeout, error, git_missing	A repack of a scope clone (OPAL_SCOPES_GIT_REPACK_PACK_LIMIT), its ref steps included. git_missing is counted once per process, here or in git_pack_refs, the first time a repack finds no git.
opal_server.scopes.git_repack_seconds	distribution	outcome: ok, timeout, error	How long a repack held the clone: its ref steps, any wait for a git worker thread, and the walks of the clone's refs and reflogs that size its log line included (for ok, the walks before and after git; otherwise the one before). The walks are not bounded by OPAL_SCOPES_GIT_REPACK_TIMEOUT, so this can exceed it.
opal_server.scopes.git_pack_refs	count	outcome: as for git_repack	A run of the repack's ref steps alone, on a clone that holds fewer packs than the repack limit and has refs to pack (with the libgit2 fetch pruning, reflogs to remove).
opal_server.scopes.git_pack_refs_seconds	distribution	outcome: ok, timeout, error	How long such a run held the clone, measured as for git_repack_seconds.
opal_server.scopes.git_ref_step_failures	count	step: config, reflogs, pack_refs	A ref step, in a repack or in a run of its own, that failed, ran past its share of OPAL_SCOPES_GIT_REPACK_TIMEOUT, or left refs in files of their own. Before a repack, a failed ref step is a warning and the repack still runs.

DataDog APM Monitoring

OPAL supports DataDog APM (Application Performance Monitoring) for observing your OPAL deployment's performance and traces.

Requirements

• ddtrace v4.8.2 or higher, below 5 - 4.8.2 is the first release with the fix for CVE-2026-50271 (GHSA-mw54-j2v2-42hr)
• Python 3.11 or higher - OPAL's own floor (ddtrace v4 itself supports 3.9+)

Configuration

To enable DataDog APM monitoring:

1. Set the ENABLE_DATADOG_APM=true environment variable for both OPAL server and client
2. (Optional) To disable APM when needed, set DD_TRACE_ENABLED=false

Breaking Changes in ddtrace v3.0.0

The following changes were made to support ddtrace v3.0.0:

• The ddtrace.filters.TraceFilter import has been moved to ddtrace.trace.TraceFilter
• Programmatic APM disabling via tracer.configure(enabled=False) is no longer supported
• APM should now be disabled using the DD_TRACE_ENABLED=false environment variable

For more information about ddtrace v3.0.0 changes, see the official migration guide.

ddtrace v4

ddtrace v4 traces every websocket message by default; v3 did not. OPAL's pub/sub runs over websockets, so that would turn each published update into spans. OPAL keeps the v3 behaviour: the Docker images set DD_TRACE_WEBSOCKET_MESSAGES_ENABLED=false, and configure_apm applies the same default to the ASGI, Starlette and FastAPI integrations when neither DD_TRACE_WEBSOCKET_MESSAGES_ENABLED nor the older DD_ASGI_TRACE_WEBSOCKET is set, including under ddtrace-run. Set DD_TRACE_WEBSOCKET_MESSAGES_ENABLED=true to trace websocket messages; in the Docker images it is the only variable that does, since the image's own setting takes precedence over the older name.

OPAL uses none of the APIs v4 removed (Pin, Span.set_tag_str, ddtrace.settings, the opentracer package and others); its only v4-specific change is the websocket default above. If you import ddtrace yourself next to OPAL, for example from a custom fetch provider, check your own code against the v4.0.0 release notes.

OPAL Client tracker (EAP)

Alternative implementation for the Statistics feature, OPAL-Server tracks all OPAL-clients connected through websocket. Gathered information includes connection details (client's source host and port), connection time, and subscribed topics. Available through /pubsub_client_info api route on the server.

Caveats:

• When UVICORN_NUM_WORKERS > 1, retrieved information would only include clients connected to the replying server process.
• The values of query parameters with auth-like names (e.g. token, api_key, password) in a client's websocket URL are reported as ***.
• This is an early access feature and is likely to change. Backward compatibility is not garaunteed.