Skip to main content

Performance Testing

Styra Load is designed to be more performant in use cases where large volumes of data is needed to make a policy decision. The following performance testing is used to evaluate the performance of Styra Load compared to OPA.

Example Domain

The following example based on RBAC. In the example domain, there are a number of users with roles:

{
"user0": [
"role5839", "role11814", "role13989" ...
],
"user1": [
"role5839", "role11814", "role13989" ...
]
}

There are also a number of roles which permit actions on resources:

{
"role0": [
{
"action": "action5996",
"resource": "resource5579"
},
{
"action": "action1170",
"resource": "resource8171"
},
...
],
"role1": [
{
"action": "action7459",
"resource": "resource1688"
},
...
]
}

You can compare the performance of Styra Load and OPA by processing this simple policy. This policy takes the user and checks if any of their roles permit the given action on the requested resource.

package rbac

import future.keywords.in

default allow := false

allow {
some role in data.users[input.user]
some permission in data.roles[role]
permission.action == input.action
permission.resource == input.resource
}

An example query with input for this policy is as follows:

{
"input": {
"user": "user8147",
"action": "action1258",
"resource": "resource5952"
},
"expected": true # this is the expected response
}

Prerequisites and Setup

There are a number of tools and resources needed to complete this exercise. This section is about getting set up.

Tools

note

This performance test should be possible to run anywhere (Linux, macOS or Windows).

The following prerequisites are required for performance testing:

  • Styra Load as a binary, Installation provides installation instructions.
  • The latest OPA binary, OPA Releases provides installation instructions.
  • The k6 benchmarking tool, k6 Install provides installation instructions.
  • (to use pre-built bundles) Git Large File Storage, Git Large File Storage provides installation instructions.

Confirm that the STYRA_LOAD_LICENSE_KEY environment variable is set in every terminal that will run Styra Load.

Check the binaries are present in your path:

$ load version
Version: ...

$ opa version
Version: ...

$ k6 version
k6 ...

$ git lfs install
Updated Git hooks.
Git LFS initialized.

$ git lfs checkout
Checking out LFS objects: 100% (10/10), 335 MB | 0 B/s, done.

Resources

To download the resources for Load performance testing, clone the GitHub repository containing the examples:

git clone https://github.com/StyraInc/load.git

cd load/examples/performance-testing

We are going to be using some sample data which as been generated based on the example domain outlined above.

There are five sets of sample data ranging from 10 MB to 400 MB when uncompressed. For each set, there is a bundle for OPA and a bundle for Styra Load. Also included are sample query sets which will be used to exercise the bundles during the test. Since bundles are compressed, the combined size of all downloads is around 335 MB.

Each dataset is based on the example domain above, only in varying sizes:

  • 10 MB: 12,000 users, 15,000 roles
  • 50 MB: 65,000 users and roles
  • 100 MB: 125,000 users and roles
  • 200 MB: 200,000 users and 280,000 roles
  • 400 MB: 500,000 users and roles

Running Tests

The benchmark.sh script runs a performance test against OPA and then the same test against Styra Load. We must supply the filename of the query list as well as the OPA and Load Bundles.

 $ ./benchmark.sh
Usage: benchmark.sh [opa-bundle] [load-bundle] [query_list]

We can start a test using the 400MB dataset like this:

./benchmark.sh bundle-opa-400.tar.gz bundle-load-400.tar.gz queries-400

The test will take some time to run.

Understanding Results

The results of a test run will look something like this, depending on your hardware:

opa version: 0.48.0
load version: 0.48.0-1
k6 version: v0.42.0
OPA bundle: bundle-opa-400.tar.gz
Load bundle: bundle-load-400.tar.gz
Query list: queries-400

Waiting for OPA to start...
Running OPA test...
Results:
requests per second (mean): 7851.13
server heap size (max): 7.26GB
Stopping OPA...

Waiting for Load to start...
Running Load test...
Results:
requests per second (mean): 10961.95
server heap size (max): 1.12GB
Stopping Load...

We can see that the following statistics are reported for each test:

  • requests per second (mean): The average number of requests per second the server processed during the test.
  • server heap size (max): The maximum size of the heap during the test. This metric is sampled for 10% of requests. It makes sense to compare the maximum value for this metric since that's what you're going to need to provision for when running in production.

Generating Your Own Sample Data

We also provide tools to generate your own data based on your own parameters. Take a look at the generate-config.json file:

{
"users": 100000,
"roles": 5,
"resources": 100000,
"actions": 3,
"queries": 1000,
"max_capabilities_per_role": 10,
"max_roles_per_user": 10
}

It should be intuitive how this can be used to generate a new dataset with different parameters. To so that, run the generate script:

./generate.sh
note

This can take some time (minutes) if you have specified a large number of objects.

This will output a queries file and a bundle file: bundle.tar.gz, you can convert this for use in Styra Load with:

load bundle convert bundle.tar.gz bundle-load.tar.gz

You can then run the tests again using these files to make your own comparisons.

./benchmark.sh bundle.tar.gz bundle-load.tar.gz queries