Logging decisions to AWS S3

To support high-throughput scenarios, Enterprise OPA allows you to export decision logs directly to S3-compatible blob stores.

Follow along!

All configuration and infrastructure requirements for this tutorial are available in the Decision Logs / S3 example.

git clone https://github.com/StyraInc/enterprise-opa
cd enterprise-opa
docker compose up

You can now send requests to Enterprise OPA via http://127.0.0.1:8181/v1/data, and inspect your bucket on 127.0.0.1:9001/browser/logs/ (username minioadmin, password minioadmin).

Trial License

To evaluate Enterprise OPA, you can obtain a trial license by Downloading eopa and running eopa license trial in the terminal.

Configuration

The simplest-possible configuration for the eopa_dl decision logger plugin to push decision logs to AWS S3 is this:

decision_logs:
  plugin: eopa_dl
plugins:
  eopa_dl:
    output:
    - type: s3
      access_key_id: $AWS_ACCESS_KEY_ID
      access_secret: $AWS_ACCESS_SECRET
      region: "us-east-1"
      bucket: logs

note

You can override endpoint (URL string) and force_path (bool) to send decision logs to other S3-compatible APIs, such as MinIO.

tip

You can use the Enterprise OPA Enterprise Key Management feature to avoid putting your S3 credentials into the configuration file. Learn more.

That's it! Every policy evaluation done in Enterprise OPA will now cause a JSON blob to be posted to S3.

Using the mc util and a local MinIO setup, we can list and inspect them:

mc ls -r local/logs
[2023-05-24 12:13:14 CEST]   451B STANDARD 2023/05/24/12/06e9e4b6-dfff-404f-bcb0-3993f35d4316.json
[2023-05-24 12:13:11 CEST]   450B STANDARD 2023/05/24/12/13bfdc49-30aa-4ec5-ae4e-575bd5472f95.json
[2023-05-24 12:13:13 CEST]   452B STANDARD 2023/05/24/12/166c0df3-02aa-4f1b-b3ad-b7e385268a61.json
[2023-05-24 12:13:01 CEST]   522B STANDARD 2023/05/24/12/16e64707-0c39-4661-ba84-5df6004bef32.json
[2023-05-24 12:13:14 CEST]   452B STANDARD 2023/05/24/12/2b3c9e58-2f28-4e71-91b5-60b0a563dffd.json
[2023-05-24 12:13:12 CEST]   451B STANDARD 2023/05/24/12/3ebad3a2-e814-4961-a5d7-805daabb43c4.json
[2023-05-24 12:13:14 CEST]   452B STANDARD 2023/05/24/12/45454ce5-c1c3-400e-9553-847897cac674.json
[2023-05-24 12:13:13 CEST]   452B STANDARD 2023/05/24/12/6c0b9fd5-92ca-4a4c-8f4c-16a6d1b7f44d.json
[2023-05-24 12:13:11 CEST]   451B STANDARD 2023/05/24/12/762f9304-2b70-4e7a-a933-87fff04c006f.json
[2023-05-24 12:13:12 CEST]   453B STANDARD 2023/05/24/12/7df2dc53-35d4-47ef-9068-e149d1afaba1.json
[2023-05-24 12:13:10 CEST]   453B STANDARD 2023/05/24/12/8265028f-0104-44b8-8b17-cb634035ee1a.json
[2023-05-24 12:13:11 CEST]   451B STANDARD 2023/05/24/12/89034bc5-a87e-4838-8e70-920ae25291fc.json
[2023-05-24 12:13:12 CEST]   450B STANDARD 2023/05/24/12/a1cafebf-2b11-4cfe-9270-857375ae2c95.json
[2023-05-24 12:13:13 CEST]   452B STANDARD 2023/05/24/12/a2702d2f-0af7-4bc3-bbd1-3e326f825daf.json
[2023-05-24 12:13:11 CEST]   451B STANDARD 2023/05/24/12/b3d656eb-186b-45d4-9c25-8a8ea0dc80b3.json
[2023-05-24 12:13:15 CEST]   452B STANDARD 2023/05/24/12/d87e07f1-ffcf-4240-9721-bb8c0f8d37e0.json
[2023-05-24 12:13:12 CEST]   451B STANDARD 2023/05/24/12/dbb3e8b5-fc55-489b-b75a-914a88052097.json
[2023-05-24 12:13:14 CEST]   452B STANDARD 2023/05/24/12/dc3424ac-d4bc-49ac-a8fb-c323846f31fc.json
[2023-05-24 12:13:13 CEST]   452B STANDARD 2023/05/24/12/dfaeabdc-5dd6-4995-b946-0fc74ca70169.json
[2023-05-24 12:13:13 CEST]   452B STANDARD 2023/05/24/12/ee48003b-ceec-48b5-94e3-b4c5e9eebe3f.json
[2023-05-24 12:13:12 CEST]   452B STANDARD 2023/05/24/12/f5a825dc-2ce6-4e53-a242-0c14a4d60c8d.json

mc cat local/logs/2023/05/24/12/06e9e4b6-dfff-404f-bcb0-3993f35d4316.json | jq
{
  "decision_id": "06e9e4b6-dfff-404f-bcb0-3993f35d4316",
  "labels": {
    "id": "5758c49f-c271-4cff-adbe-07b3558ff8a1",
    "type": "enterprise-opa",
    "version": "0.52.0"
  },
  "metrics": {
    "counter_regovm_eval_instructions": 5,
    "counter_server_query_cache_hit": 1,
    "timer_rego_input_parse_ns": 417,
    "timer_regovm_eval_ns": 53511,
    "timer_server_handler_ns": 76383
  },
  "nd_builtin_cache": {},
  "path": "foo",
  "req_id": 19,
  "requested_by": "127.0.0.1:60723",
  "timestamp": "2023-05-24T12:13:14.50627+02:00"
}

See the configuration documentation for further information on batching and buffering.