Skip to main content

Prometheus

Styra exports the following two types of metrics in Prometheus format:

  • Decision Metrics: Reports the accumulated number of decisions, denies, errors, advices, unknowns, and violations per system. For more information, see the OpenAPI doc.

  • System Metrics: Reports the accumulated number of errors and warnings per system. For more information, see the OpenAPI doc.

note

For the GetData API (GET v1/data), if you don't provide a name (path parameter) then the GetData API (GET v1/data) returns a 404, otherwise it returns a 200.

Configure Prometheus

important

For Styra DAS On-premises users, the Timeseries metrics APIs and Systems metrics APIs do not require authentication.

The following example shows how to configure Prometheus to pull both the decision and system metrics from your Styra tenant:

cat << EOF > Prometheus.yaml
global:
scrape_interval: 15s
scrape_timeout: 10s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
scheme: http
timeout: 10s
scrape_configs:
- job_name: decisions
metrics_path: /v1/timeseries/metrics
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
bearer_token: <TOKEN>
scheme: https
static_configs:
- targets:
- <das-id>.styra.com
- job_name: systems
metrics_path: /v1/systems/metrics
honor_timestamps: true
scrape_interval: 15s
scrape_timeout: 10s
bearer_token: <TOKEN>
scheme: https
static_configs:
- targets:
- <das-id>.styra.com
EOF

You need to create a token using the GUI and replace the above <TOKEN> with the new one.

The following command is used to run a local Prometheus server and test the above configuration:

docker run -p 9090:9090 -v `pwd`:/Prometheus-data  prom/Prometheus --config.file=/Prometheus-data/Prometheus.yaml

Now, go to http://localhost:9090/targets and verify that both targets are being scraped.

Configure Alerts

Prometheus can also be configured to trigger alerts using Prometheus query language called PromQL.

The following shows an example on how you could alert when an unusual rate of violations or decisions is detected:

cat << EOF > alerts/rates.yaml
# unusual violations rate

groups:

- name: UnusualRate
rules:
- alert: ViolationsRate
expr: sum by (name) (delta(violations[1h])) > 10
for: 30m
labels:
severity: warning
annotations:
summary: "Unusual number of violations (system: {{ $labels.instance }})"
description: "A new rule or new kubernetes resource is causing an increase in violations (> 10/h)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
- alert: DecisionRate
expr: sum by (name) (delta(decisions[1m])) > 100
for: 30m
labels:
severity: warning
annotations:
summary: "Unusual number of decision (system: {{ $labels.instance }})"
description: "A new applications or user is causing an increase in decisions (> 100/m)\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
EOF
note

You must add the following configuration to the prometheus.yaml configuration file in order to load the above alerts examples into Prometheus.

rule_files:
- 'alerts/*.yaml'

Configure Alertmanager

Prometheus is correctly configured to scrape the Styra metrics and also configure some alerts. Now, you can start an Alertmanager. The Alertmanager is responsible to route the alerts to the users.

note

Alertmanager supports many more routes as described on Prometheus documentation.

The following example shows how to route the alerts to Slack:

cat << EOF > alertmanager.yaml
global:
slack_api_url: "https://hooks.slack.com/services/<you_slack_intergration>

route:
group_by: ['instance', 'severity']
group_wait: 30s
group_interval: 5m
repeat_interval: 3h
routes:
- match:
alertname: DecisionRate
receiver: 'alert-team'
receivers:
- name: 'alert-team'
slack_configs:
- channel: "#ops"
text: "summary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"
EOF

The alertmanager.yaml file will configure Alertmanager to route the DecisionRate alerts generated by Prometheus to the alert-team via Slack.

To start the Alertmanager, run the following command:

docker run -d --name alertmanager -p 9093:9093 -v `pwd`:/prometheus-data bitnami/alertmanager:latest --config.file=/prometheus-data/alertmanager.yaml
important

When you configure Prometheus, host.docker.internal:9093 is used as the Alertmanager target. The special host.docker.internal name allows the container to reach the host network and access the Alertmanager running at localhost:9093.