HTTPS Data Source
An HTTPS data source is similar to Git or S3 data sources. Instead of reading the data from storage, the HTTPS data source reads data from an external server by making HTTPS requests.
Configure through the DAS GUI
The following section helps you to configure <das-id>.styra.com
to access a data source stored in HTTPS using the DAS GUI.
Create a DAS System
Go to <das-id>.styra.com
. To add a new system, click the ( ⨁ ) plus icon next to SYSTEMS on the left side of the navigation panel.
Fill in the following fields:
System type (required): Select any system type from the drop down list. For example, Custom.
System name (required): A user-friendly name so that you can distinguish between the different systems.
Description (optional): More details about this system.
Leave the Show errors switch ON to display the errors.
Click Add system button.
Now, your DAS system is created under the SYSTEMS on the left side of the navigation panel.
Add a Data Source
After you create your system, click the three dots (⋮
) next to it and select Add Data Source to start configuring the data source.

Now, your Custom System >> Add Data Source dialog appears.

Complete the following steps in your Custom System >> Add Data Source dialog box.
Type: An editable data source that you fill in with JSON data and publish. Click the down arrow to select the data source type. For example, select HTTPS to pull a JSON object from a specific AWS S3 bucket. This refreshes regularly.
Figure 3 - Data Source Type
Path: Enter a new or existing path separated by
/
. For example,am/datasourcetypes
.Data source name (required): Enter a name for the data source type. For example,
am-https
.Description: This field is optional.
URL (required): This field represents the URL where the HTTPS server is deployed.
CA certificate: Custom CA certificate.
Refresh interval: Enter a refresh interval which is the amount of time between polling intervals. Default is
s
.Enable TLS verification: Enable or disable the switch based on the SSL certificate verification. For example, leave the Enable TLS verification switch OFF.
Add headers: Add headers based on the requirement.
Header name: Enter the header name. For example, enter
Authorization
.Header value: Enter the header value. For example, enter
Bearer Token
. If you want theBearer Token
to be hidden from Styra DAS, then switch ON the Secret to store the token’s value as a secret in Styra DAS. Now, this token’s value will not be stored as a plain text.
Data transform: Specify a policy and write a query that allows you to apply Rego transformations before it is persisted as data. For example, Select Custom and fill in the following fields:
Policy: An existing policy separated by
/
. For example,transform/transform.rego
.Rego query: Path to the Rego rule to evaluate. For example,
data.transform.query
.
Leave the Enable on-premises data source agent switch OFF. Enable on-premises data source agent shows how and where to run the data source. If set to
true
thendatasource-agent
will run on-premises setup, otherwise on SaaS.Now, make sure you filled all the fields similar to Figure 4.
Figure 4 - Completed Data Source Form
Finally, click the Add button to add a data source.
The following shows an example output which appears after the data source is created in DAS.
{
"data": {
"s3-test.json": {
"foo1": "bar1"
},
"s3-test.yaml": {
"foo3": "bar3"
},
"s3-test.yml": {
"foo2": "bar2"
}
}
}
Configure through the API
HTTP data source plugin supports both common and more advanced HTTP queries with the ability to use custom HTTP headers.
To create the HTTP data source plugin, run the following curl command:
curl -H 'authorization: bearer XXX' -H 'Content-Type: application/json' https://<das-id>.styra.com/v1/datasources/http/url -XPUT -d'
{
"category": "http",
"url": "<datasource url>",
"polling_interval: "60s",
"headers": [
{
"name": "<header name>",
"value": "<header value>",
"secret_id": "<id of the stored secret>"
}
],
"skip_tls_verification": true,
"ca_certificate": "<pem file>"
}'
The
url
parameter is a link to an endpoint that returns a data inJSON
orYAML
format.The
polling_interval
parameter holds the interval value of60
in seconds. The value isfloat
.The
headers
parameter is a list of custom headers with the following information:name
: Name of the header. This field is mandatory.value
: Value of the header stored as a plain text.secret_id
: Name of the secret stored in the system. For more information about secrets, see the secrets API definition.
important
Set the
value
orsecret_id
, but not both.The
skip_tls_verification
allows to ignore any custom or wrong certificates. Default value:false
.The
ca_certificate
allows to use a custom CA certificate. A certificate should be uploaded as plain text inpem
format.
Configuration Example
{
"category": "http",
"url": "https://www.example.com/api/test",
"headers": [
{
"name:": "Env",
"value": "QA"
},
{
"name:": "Authorization",
"secret_id": "auth/qa-token"
}
]
}
{
"description": "Bearer token",
"name": "qa-token",
"secret": "Bearer SUPER-QA-TOKEN"
}
The HTTP data source makes a request similar to the following curl command:
curl -H'Env: QA' \
-H'Content-Type: application/json, text/vnd.yaml, application/yaml, application/x-yaml, text/x-yaml, text/yaml, text/plain' \
-H'Authorization: Bearer SUPER-QA-TOKEN' \
-XGET 'https://www.example.com/api/test'
Policy Filtering
A policy_filter
is used to poll from a data source that you want to transform captured data source information before storing it. Specifying a policy_filter
and policy_query
will allow you to apply Rego transformations before it is persisted as data. This mechanism is useful for filtering out data that you no longer want to store or for any other mutations that you want to perform.
It works by specifying a policy that will be evaluated via Rego with captured data as input. You also specify a query to apply to that policy and data. The result of that query will be stored as data, instead of what is polled by the data source plugin.
In addition to the standard options, you must specify the following when you create or update a data source:
The
policy_filter
parameter is the ID of a policy you want to use for filtering.The
policy-query
parameter is the Rego query you want to evaluate.
The following is an example of a curl command with additional parameters:
curl -H 'Authorization: bearer XXX' \
-H 'Content-Type: application/json' \
-XPUT 'https://TENANT.styra.com/v1/datasources/http/url' -d'
{
"category": "http",
"url": "<datasource url>",
"headers": [
{
"name": "<header name>",
"value": "<header value>",
"secret_id": "<id of the stored secret>"
}
],
"skip_tls_verification": true,
"ca_certificate": "<pem file>",
"policy_filter": "/my/test/policy",
"policy_query": "<rego statement>"
}'
Policy Filtering Example
In this example, you will learn how to filter a policy and write a query that will allow you to apply Rego transformations before it is persisted as data.
Data returned from
<URL>
:Sample data{
"servers": [
{
"id": "app",
"protocols": [
"https",
"ssh"
],
"ports": [
"p1",
"p2",
"p3"
]
},
{
"id": "db",
"protocols": [
"mysql"
],
"ports": [
"p3"
]
},
{
"id": "cache",
"protocols": [
"memcache"
],
"ports": [
"p3"
]
},
{
"id": "ci",
"protocols": [
"http"
],
"ports": [
"p1",
"p2"
]
},
{
"id": "busybox",
"protocols": [
"telnet"
],
"ports": [
"p1"
]
}
],
"networks": [
{
"id": "net1",
"public": false
},
{
"id": "net2",
"public": false
},
{
"id": "net3",
"public": true
},
{
"id": "net4",
"public": true
}
],
"ports": [
{
"id": "p1",
"network": "net1"
},
{
"id": "p2",
"network": "net3"
},
{
"id": "p3",
"network": "net2"
}
]
}For this example, there exists a relevant policy with parsed Rego:
curl -H 'Authorization: Bearer XXX' \
-X GET https://<das-id>.styra.com/v1/policies/example/networksOutput:
package example.networks
public_server[server] { # a server exists in the public_server set if...
some i, j
server := input.servers[_] # it exists in the input.servers collection and...
server.ports[_] == input.ports[i].id # it references a port in the input.ports collection and...
input.ports[i].network == input.networks[j].id # the port references a network in the input.networks collection and...
input.networks[j].public # the network is public.
}To create a new data source, run the following curl command.
curl -H 'Authorization: Bearer XXX' \
-H 'Content-Type: application/json' \
-X PUT https://<das-id>.styra.com/v1/datasources/systems/test/test-datasource \
-d '
{
"category": "http",
"url": "https://<mycustomdatasource>/topology",
"policy_filter": "/example/networks",
"policy_query": "data.example.networks.public_server[results]"
}'After polling occurs, the following shows the result of your query in the data:
curl -H 'Authorization: Bearer XXX' -H 'Content-Type: application/json'
-X GET https://<das-id>.styra.com/v1/data/systems/test/test-datasourcesample response{
"request_id": "<request ID>",
"result": [
{
"id": "app",
"ports": [
"p1",
"p2",
"p3"
],
"protocols": [
"https",
"ssh"
]
},
{
"id": "ci",
"ports": [
"p1",
"p2"
],
"protocols": [
"http"
]
}
],
"revision": "<revision>"
}