Docs/Infrastructure/Operations
Operations
Monitor, alert on, and operate a production GovernanceAI deployment.
This page covers production operations: metrics, alerting, log management, and common runbook procedures.
Metrics
The metrics service exposes a Prometheus-compatible endpoint at :8013/metrics. Scrape it with any Prometheus-compatible monitoring stack.
| Metric | Type | Description |
|---|---|---|
| gov_requests_total | Counter | Total evaluated requests by decision (allow/block/warn) |
| gov_risk_score_histogram | Histogram | Distribution of risk scores across all requests |
| gov_evaluation_latency_ms | Histogram | End-to-end evaluation latency in milliseconds |
| gov_policy_blocks_total | Counter | Total requests blocked by policy rule name |
| gov_dlp_matches_total | Counter | DLP pattern match count by pattern name |
| gov_active_api_keys | Gauge | Number of active (non-revoked) API keys per project |
Recommended alerts
groups:
- name: governance_ai
rules:
- alert: HighBlockRate
expr: |
rate(gov_requests_total{decision="block"}[5m]) /
rate(gov_requests_total[5m]) > 0.15
for: 5m
annotations:
summary: "Block rate above 15% — check for attack campaign or misconfigured profile"
- alert: EvaluationLatencyHigh
expr: histogram_quantile(0.99, gov_evaluation_latency_ms) > 200
for: 2m
annotations:
summary: "P99 evaluation latency above 200 ms"
- alert: GatewayDown
expr: up{job="api-gateway"} == 0
for: 1m
annotations:
summary: "API gateway is unreachable"Log management
All services write structured JSON logs to stdout. In production, ship logs to your preferred aggregation stack (Datadog, Loki, CloudWatch, Elastic).
{
"timestamp": "2026-06-26T07:15:00Z",
"level": "info",
"service": "api-gateway",
"trace_id": "trc_8f3a9d2c",
"event": "evaluation_complete",
"decision": "allow",
"risk_score": 0.04,
"latency_ms": 17,
"org_id": "org_abc123",
"project_id": "prj_xyz789"
}Runbooks
High block rate
If the block rate spikes unexpectedly, check the Observability dashboard for the top triggered rules. A spike caused by a single rule often indicates a security profile misconfiguration or an active attack campaign.
# View the top blocking rules in the last hour gov observability top-blocks --since 1h --limit 10 # Check a specific trace gov observability trace trc_8f3a9d2c
Redis failure
If Redis becomes unavailable, the gateway fails open on rate limiting and caches stale security profiles. Correct Redis and restart the gateway — profiles will refresh within 30 seconds.