HTTPS Data Source

An HTTPS data source is similar to Git or S3 data sources. Instead of reading the data from storage, the HTTPS data source reads data from an external server by making HTTPS requests.

Configure through the DAS GUI

The following section helps you to configure <das-id>.styra.com to access a data source stored in HTTPS using the DAS GUI.

Create a DAS System

Go to <das-id>.styra.com. To add a new system, click the ( ⨁ ) plus icon next to SYSTEMS on the left side of the navigation panel.

Fill in the following fields:

System type (required): Select any system type from the drop down list. For example, Custom.
System name (required): A user-friendly name so that you can distinguish between the different systems.
Description (optional): More details about this system.
Leave the Show errors switch ON to display the errors.
Click Add system button.

Now, your DAS system is created under the SYSTEMS on the left side of the navigation panel.

Add a Data Source

After you create your system, click the three dots (⋮) next to it and select Add Data Source to start configuring the data source.

Figure 1 - Add Data Source

Now, your Custom System >> Add Data Source dialog appears.

Figure 2 - Add Data Source Window

Complete the following steps in your Custom System >> Add Data Source dialog box.

Type: An editable data source that you fill in with JSON data and publish. Click the down arrow to select the data source type. For example, select HTTPS to pull a JSON object from a specific AWS S3 bucket. This refreshes regularly.
Figure 3 - Data Source Type
Path: Enter a new or existing path separated by /. For example, am/datasourcetypes.
Data source name (required): Enter a name for the data source type. For example, am-https.
Description: This field is optional.
URL (required): This field represents the URL where the HTTPS server is deployed.
CA certificate: Custom CA certificate.
Refresh interval: Enter a refresh interval which is the amount of time between polling intervals. Default is s.
Enable TLS verification: Enable or disable the switch based on the SSL certificate verification. For example, leave the Enable TLS verification switch OFF.
Add headers: Add headers based on the requirement.
- Header name: Enter the header name. For example, enter Authorization.
- Header value: Enter the header value. For example, enter Bearer Token. If you want the Bearer Token to be hidden from Styra DAS, then switch ON the Secret to store the token’s value as a secret in Styra DAS. Now, this token’s value will not be stored as a plain text.
Data transform: Specify a policy and write a query that allows you to apply Rego transformations before it is persisted as data. For example, Select Custom and fill in the following fields:
- Policy: An existing policy separated by /. For example, transform/transform.rego.
- Rego query: Path to the Rego rule to evaluate. For example, data.transform.query.
Leave the Enable on-premises data source agent switch OFF. Enable on-premises data source agent shows how and where to run the data source. If set to true then datasource-agent will run on-premises setup, otherwise on SaaS.
Now, make sure you filled all the fields similar to Figure 4.
Figure 4 - Completed Data Source Form
Finally, click the Add button to add a data source.

The following shows an example output which appears after the data source is created in DAS.

{
    "data": {
        "s3-test.json": {
            "foo1": "bar1"
        },
        "s3-test.yaml": {
            "foo3": "bar3"
        },
        "s3-test.yml": {
            "foo2": "bar2"
        }
    }
}

Configure through the API

HTTP data source plugin supports both common and more advanced HTTP queries with the ability to use custom HTTP headers.

To create the HTTP data source plugin, run the following curl command:

curl -H 'authorization: bearer XXX' -H 'Content-Type: application/json' https://<das-id>.styra.com/v1/datasources/http/url -XPUT -d'
{
  "category": "http",
  "url": "<datasource url>",
  "polling_interval: "60s",
  "headers": [
    {
      "name": "<header name>",
      "value": "<header value>",
      "secret_id": "<id of the stored secret>"
    }
  ],
  "skip_tls_verification": true,
  "ca_certificate": "<pem file>"
}'

The url parameter is a link to an endpoint that returns a data in JSON or YAML format.
The polling_interval parameter holds the interval value of 60 in seconds. The value is float.
The headers parameter is a list of custom headers with the following information:
- name: Name of the header. This field is mandatory.
- value: Value of the header stored as a plain text.
- secret_id: Name of the secret stored in the system. For more information about secrets, see the secrets API definition.
important
Set the value or secret_id, but not both.
The skip_tls_verification allows to ignore any custom or wrong certificates. Default value: false.
The ca_certificate allows to use a custom CA certificate. A certificate should be uploaded as plain text in pem format.

Configuration Example

Sample configuration
{
    "category": "http",
    "url": "https://www.example.com/api/test",
    "headers": [
        {
            "name:": "Env",
            "value": "QA"
        },
        {
            "name:": "Authorization",
            "secret_id": "auth/qa-token"
        }
    ]
}

auth/qa-token secret id:

Sample secret request body
{
    "description": "Bearer token",
    "name": "qa-token",
    "secret": "Bearer SUPER-QA-TOKEN"
}

The HTTP data source makes a request similar to the following curl command:

curl -H'Env: QA' \
  -H'Content-Type: application/json, text/vnd.yaml, application/yaml, application/x-yaml, text/x-yaml, text/yaml, text/plain' \
  -H'Authorization: Bearer SUPER-QA-TOKEN' \
  -XGET 'https://www.example.com/api/test'

Policy Filtering

A policy_filter is used to poll from a data source that you want to transform captured data source information before storing it. Specifying a policy_filter and policy_query will allow you to apply Rego transformations before it is persisted as data. This mechanism is useful for filtering out data that you no longer want to store or for any other mutations that you want to perform.

It works by specifying a policy that will be evaluated via Rego with captured data as input. You also specify a query to apply to that policy and data. The result of that query will be stored as data, instead of what is polled by the data source plugin.

In addition to the standard options, you must specify the following when you create or update a data source:

The policy_filter parameter is the ID of a policy you want to use for filtering.
The policy-query parameter is the Rego query you want to evaluate.

The following is an example of a curl command with additional parameters:

curl -H 'Authorization: bearer XXX' \
  -H 'Content-Type: application/json' \
  -XPUT 'https://TENANT.styra.com/v1/datasources/http/url' -d'
{
  "category": "http",
  "url": "<datasource url>",
  "headers": [
    {
      "name": "<header name>",
      "value": "<header value>",
      "secret_id": "<id of the stored secret>"
    }
  ],
  "skip_tls_verification": true,
  "ca_certificate": "<pem file>",
  "policy_filter": "/my/test/policy",
  "policy_query": "<rego statement>"
}'

Policy Filtering Example

In this example, you will learn how to filter a policy and write a query that will allow you to apply Rego transformations before it is persisted as data.

Data returned from <URL>:

Sample data
{
    "servers": [
        {
            "id": "app",
            "protocols": [
                "https",
                "ssh"
            ],
            "ports": [
                "p1",
                "p2",
                "p3"
            ]
        },
        {
            "id": "db",
            "protocols": [
                "mysql"
            ],
            "ports": [
                "p3"
            ]
        },
        {
            "id": "cache",
            "protocols": [
                "memcache"
            ],
            "ports": [
                "p3"
            ]
        },
        {
            "id": "ci",
            "protocols": [
                "http"
            ],
            "ports": [
                "p1",
                "p2"
            ]
        },
        {
            "id": "busybox",
            "protocols": [
                "telnet"
            ],
            "ports": [
                "p1"
            ]
        }
    ],
    "networks": [
        {
            "id": "net1",
            "public": false
        },
        {
            "id": "net2",
            "public": false
        },
        {
            "id": "net3",
            "public": true
        },
        {
            "id": "net4",
            "public": true
        }
    ],
    "ports": [
        {
            "id": "p1",
            "network": "net1"
        },
        {
            "id": "p2",
            "network": "net3"
        },
        {
            "id": "p3",
            "network": "net2"
        }
    ]
}

For this example, there exists a relevant policy with parsed Rego:

curl -H 'Authorization: Bearer XXX' \
  -X GET https://<das-id>.styra.com/v1/policies/example/networks

Output:

package example.networks

public_server[server] {                             # a server exists in the public_server set if...
    some i, j
    server := input.servers[_]                      # it exists in the input.servers collection and...
    server.ports[_] == input.ports[i].id            # it references a port in the input.ports collection and...
    input.ports[i].network == input.networks[j].id  # the port references a network in the input.networks collection and...
    input.networks[j].public                        # the network is public.
}

To create a new data source, run the following curl command.

curl -H 'Authorization: Bearer XXX' \
  -H 'Content-Type: application/json' \
  -X PUT https://<das-id>.styra.com/v1/datasources/systems/test/test-datasource \
  -d '
{
  "category": "http",
  "url": "https://<mycustomdatasource>/topology",
  "policy_filter": "/example/networks",
  "policy_query": "data.example.networks.public_server[results]"
}'

After polling occurs, the following shows the result of your query in the data:

curl -H 'Authorization: Bearer XXX' -H 'Content-Type: application/json'
  -X GET https://<das-id>.styra.com/v1/data/systems/test/test-datasource

sample response
{
    "request_id": "<request ID>",
    "result": [
        {
            "id": "app",
            "ports": [
                "p1",
                "p2",
                "p3"
            ],
            "protocols": [
                "https",
                "ssh"
            ]
        },
        {
            "id": "ci",
            "ports": [
                "p1",
                "p2"
            ],
            "protocols": [
                "http"
            ]
        }
    ],
    "revision": "<revision>"
}

HTTPS Data Source

Configure through the DAS GUI​

Create a DAS System​

Add a Data Source​

Configure through the API​

important

Configuration Example​

Policy Filtering​

Policy Filtering Example​

Configure through the DAS GUI

Create a DAS System

Add a Data Source

Configure through the API

Configuration Example

Policy Filtering

Policy Filtering Example