Skip to main content

Monitoring OPAL

There are multiple ways you can monitor your OPAL deployment:

  • Logs - Using the structured logs outputted to stderr by both the OPAL-servers and OPAL-clients
  • Health-checks - OPAL exposes HTTP health check endpoints (See below)
  • Callbacks - Using the callback webhooks feature - having OPAL-clients report their updates
  • Statistics - Using the built-in statistics feature in OPAL (See below)
  • DataDog APM - OPAL supports DataDog APM tracing (requires ddtrace v3.0.0+ and Python 3.8+)

Health checks​

OPAL Server​

opal-server exposes http health check endpoints on / & /healthcheck.
Currently it returns 200 OK as long as server is up.

OPAL Client​

opal-client exposes 2 types of http health checks:

  • Readiness - available on /ready.
    Returns 200 OK if the client have loaded policy & data to OPA at least once (from either server, or local backup file), otherwise 503 Unavailable is returned.

  • Liveness - available on /, /healthcheck & /healthy.
    Returns 200 OK if the last attempts to load policy & data to OPA were successful, otherwise 503 Unavailable is returned

Notice: if you don't except your opal-client to load any data into OPA, set OPAL_DATA_UPDATER_ENABLED: False, so opal-client could report being healthy.

You can also configure opal-client to store dynamic health status as a document in OPA, Learn more here

OPAL Statistics​

By enabling OPAL_STATISTICS_ENABLED=true (on both servers and clients), the OPAL-Server would start maintaining a unified state of all the clients and which topics they've subscribed to. The state can then be retrieved as a JSON object by calling the /statistics api route on the server

Code Reference:​

DataDog APM Monitoring​

OPAL supports DataDog APM (Application Performance Monitoring) for observing your OPAL deployment's performance and traces.

Requirements​

  • ddtrace v3.0.0 or higher - OPAL has been upgraded to use the latest ddtrace version
  • Python 3.8 or higher - ddtrace v3.0.0+ dropped support for Python 3.7

Configuration​

To enable DataDog APM monitoring:

  1. Set the ENABLE_DATADOG_APM=true environment variable for both OPAL server and client
  2. (Optional) To disable APM when needed, set DD_TRACE_ENABLED=false

Breaking Changes in ddtrace v3.0.0​

The following changes were made to support ddtrace v3.0.0:

  • The ddtrace.filters.TraceFilter import has been moved to ddtrace.trace.TraceFilter
  • Programmatic APM disabling via tracer.configure(enabled=False) is no longer supported
  • APM should now be disabled using the DD_TRACE_ENABLED=false environment variable

For more information about ddtrace v3.0.0 changes, see the official migration guide.

OPAL Client tracker (EAP)​

Alternative implementation for the Statistics feature, OPAL-Server tracks all OPAL-clients connected through websocket. Gathered information includes connection details (client's source host and port), connection time, and subscribed topics. Available through /pubsub_client_info api route on the server.

Caveats:​

  • When UVICORN_NUM_WORKERS > 1, retrieved information would only include clients connected to the replying server process.
  • This is an early access feature and is likely to change. Backward compatibility is not garaunteed.