Performance Testing Styra Load
Styra Load is designed to be more performant in use cases where a large volume of data is needed to make a policy decision. The following tutorial outlines the recommended way to make performance comparisons between Styra Load and OPA. Once you have completed this tutorial, you should have the tools you need to run follow on comparisons with your own policies and data.
Example Domain: RBAC
This tutorial is based on a Role Based Access Control (RBAC) domain. In the example domain, there are a number of users with various roles:
"role5839", "role11814", "role13989" ...
"role5839", "role11814", "role13989" ...
Each of these roles grant rights to perform actions on resources:
You can compare the performance of Styra Load and OPA by processing this simple policy. This policy takes the
user and checks if any of their
roles permit the given
action on the requested resource.
default allow := false
some role in data.users[input.user]
some permission in data.roles[role]
permission.action == input.action
permission.resource == input.resource
An example query with input for this policy is as follows:
"expected": true # this is the expected response
Prerequisites and Setup
There are a number of tools and resources needed to complete this exercise. This section how to configure performance testing.
Performance tests can run on Linux, macOS, or Windows.
The following prerequisites are required for performance testing:
- Styra Load as a binary, Installation provides installation instructions.
- The latest OPA binary, OPA Releases provides installation instructions.
- The k6 benchmarking tool, k6 Install provides installation instructions.
- Git Large File Storage (to use pre-built bundles), Git Large File Storage provides installation instructions.
Confirm that the
STYRA_LOAD_LICENSE_KEY environment variable is set in every terminal that will run Styra Load.
Check the binaries are present in your path:
$ load version
$ opa version
$ k6 version
$ git lfs install
Updated Git hooks.
Git LFS initialized.
$ git lfs checkout
Checking out LFS objects: 100% (10/10), 335 MB | 0 B/s, done.
To download the resources for Load performance testing, clone the GitHub repository containing the examples:
git clone https://github.com/StyraInc/load.git
We are going to be using some sample data which as been generated based on the example domain outlined above.
There are five sets of sample data ranging from 10 MB to 400 MB when uncompressed. For each set, there is a bundle for OPA and a bundle for Styra Load. Also included are sample query sets which will be used to exercise the bundles during the test. Since bundles are compressed, the combined size of all downloads is around 335 MB.
Each dataset is based on the example domain above, only in varying sizes:
- 10 MB: 12,000 users, 15,000 roles
- 50 MB: 65,000 users and roles
- 100 MB: 125,000 users and roles
- 200 MB: 200,000 users and 280,000 roles
- 400 MB: 500,000 users and roles
benchmark.sh script runs a performance test against OPA and then the same test against Styra Load.
Supply the filename of the query list and the OPA and Load Bundles.
Usage: benchmark.sh [opa-bundle] [load-bundle] [query_list]
Start a test using the 400 MB dataset using the following:
./benchmark.sh bundle-opa-400.tar.gz bundle-load-400.tar.gz queries-400
The test will take some time to run.
The results of a test run will look something like this, depending on your hardware:
opa version: 0.48.0
load version: 0.48.0-1
k6 version: v0.42.0
OPA bundle: bundle-opa-400.tar.gz
Load bundle: bundle-load-400.tar.gz
Query list: queries-400
Waiting for OPA to start...
Running OPA test...
requests per second (mean): 7851.13
server heap size (max): 7.26GB
Waiting for Load to start...
Running Load test...
requests per second (mean): 10961.95
server heap size (max): 1.12GB
You will see that the following statistics are reported for each test:
- requests per second (mean): The average number of requests per second the server processed during the test.
- server heap size (max): The maximum size of the heap during the test. This metric is sampled for 10% of requests. It makes sense to compare the maximum value for this metric since that's what you're going to need to provision for when running in production.
Generating Your Own Sample Data
We also provide tools to generate your own data based on your own parameters. Take a look at the
It should be intuitive how this can be used to generate a new dataset with different parameters. To so that, run the generate script:
This can take some time (minutes) if you have specified a large number of objects.
This will output a
queries file and a bundle file:
bundle.tar.gz, you can convert this for use in Styra Load with:
load bundle convert bundle.tar.gz bundle-load.tar.gz
You can then run the tests again using these files to make your own comparisons.
./benchmark.sh bundle.tar.gz bundle-load.tar.gz queries