Prometheus

Styra exports the following two types of metrics in Prometheus format:

• Decision Metrics: Reports the accumulated number of decisions, denies, errors, advices, unknowns, and violations per system. For more information, see the OpenAPI doc.

• System Metrics: Reports the accumulated number of errors and warnings per system. For more information, see the OpenAPI doc.

note

For the GetData API (GET v1/data), if you don't provide a name (path parameter) then the GetData API (GET v1/data) returns a 404, otherwise it returns a 200.

Configure Prometheus

important

For Styra DAS On-premises users, the Timeseries metrics APIs and Systems metrics APIs do not require authentication.

The following example shows how to configure Prometheus to pull both the decision and system metrics from your Styra tenant:

cat << EOF > Prometheus.yaml
global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
    scheme: http
    timeout: 10s
scrape_configs:
- job_name: decisions
  metrics_path: /v1/timeseries/metrics
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  bearer_token: <TOKEN>
  scheme: https
  static_configs:
  - targets:
    - <das-id>.styra.com
- job_name: systems
  metrics_path: /v1/systems/metrics
  honor_timestamps: true
  scrape_interval: 15s
  scrape_timeout: 10s
  bearer_token: <TOKEN>
  scheme: https
  static_configs:
  - targets:
    - <das-id>.styra.com
EOF

You need to create a token using the GUI and replace the above <TOKEN> with the new one.

The following command is used to run a local Prometheus server and test the above configuration:

docker run -p 9090:9090 -v `pwd`:/Prometheus-data  prom/Prometheus --config.file=/Prometheus-data/Prometheus.yaml

Now, go to http://localhost:9090/targets and verify that both targets are being scraped.

Configure Alerts

Prometheus can also be configured to trigger alerts using Prometheus query language called PromQL.

The following shows an example on how you could alert when an unusual rate of violations or decisions is detected:

cat << EOF > alerts/rates.yaml
# unusual violations rate

groups:

- name: UnusualRate
  rules:
  - alert: ViolationsRate
    expr: sum by (name) (delta(violations[1h])) > 10
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: "Unusual number of violations (system: {{ $labels.instance }})"
      description: "A new rule or new kubernetes resource is causing an increase in violations  (> 10/h)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
  - alert: DecisionRate
    expr: sum by (name) (delta(decisions[1m])) > 100
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: "Unusual number of decision (system: {{ $labels.instance }})"
      description: "A new applications or user is causing an increase in decisions  (> 100/m)\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
EOF

note

You must add the following configuration to the prometheus.yaml configuration file in order to load the above alerts examples into Prometheus.

rule_files:
  - 'alerts/*.yaml'

Configure Alertmanager

Prometheus is correctly configured to scrape the Styra metrics and also configure some alerts. Now, you can start an Alertmanager. The Alertmanager is responsible to route the alerts to the users.

note

Alertmanager supports many more routes as described on Prometheus documentation.

The following example shows how to route the alerts to Slack:

cat << EOF > alertmanager.yaml
global:
  slack_api_url: "https://hooks.slack.com/services/<you_slack_intergration>

route:
  group_by: ['instance', 'severity']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 3h
  routes:
  - match:
      alertname: DecisionRate
  receiver: 'alert-team'
receivers:
- name: 'alert-team'
  slack_configs:
  - channel: "#ops"
    text: "summary: {{ .CommonAnnotations.summary }}\ndescription: {{ .CommonAnnotations.description }}"
EOF

The alertmanager.yaml file will configure Alertmanager to route the DecisionRate alerts generated by Prometheus to the alert-team via Slack.

To start the Alertmanager, run the following command:

docker run -d --name alertmanager -p 9093:9093 -v `pwd`:/prometheus-data bitnami/alertmanager:latest --config.file=/prometheus-data/alertmanager.yaml

important

When you configure Prometheus, host.docker.internal:9093 is used as the Alertmanager target. The special host.docker.internal name allows the container to reach the host network and access the Alertmanager running at localhost:9093.