HTTPS Data Source

The HTTPS Data Source reads data from an external server by making HTTPS requests.

note

The HTTPS Data Source supports If-None-Match and ETag headers. If a response contains ETag header, then the header is saved in the Data Source and uses the header If-None-Match in subsequent requests. If there is 304 Not Modified status code, Styra DAS ignores the error. See If-None-Match for more information.

Creating or Configuring the Data Source through the Styra DAS UI

Create or configure the Data Source through the Styra DAS UI.

Login to the Styra DAS UI.
Select the System to add the Data Source.
Click the kebab icon (three dots ⋮) to the right of the System and select Add Data Source. The Add Data Source dialog box appears.
Select HTTPS.
In Path type a new or existing path separated by /. For example, path/to/datasourcetypes.
In Data Source name (required) type the name for the Data Source.
(Optional) Type in a Description.
In URL (required) type the URL where the HTTPS server is deployed.
In Method select the HTTP method to specify in request GET (default) or POST.
In Body select the HTTP message body to include in request.
(Body) Select File if the value will not be serialized. Use this for non-JSON messages.
(Body) Select JSON if the value will be serialized to JSON.
In Refresh interval type a refresh interval which is the amount of time between polling intervals. Default is s.
In Enable TLS verification configure the enable or disable switch based on the SSL certificate verification. For example, leave the Enable TLS verification switch OFF.
In Add headers add headers if needed.
- In Header name type the header name. For example, enter Authorization.
- In Header value type the header value. For example, enter Bearer Token. If you want the Bearer Token to be hidden from Styra DAS, then switch ON the Secret to store the token’s value as a secret in Styra DAS.
In CA certificate provide a CA certificate.
In Data transform specify a policy and write a query that allows you to apply Rego transformations before it is persisted as data. For example, Select Custom and fill in the following fields:
- Policy: An existing policy separated by /. For example, transform/transform.rego.
- Rego query: Path to the Rego rule to evaluate. For example, data.transform.query.
Leave the Enable on-premises data source agent switch off. Enable on-premises data source agent shows how and where to run the data source. If set to true then datasource-agent will run on-premises setup, otherwise on SaaS.
Preview the Data Source in the right pane. If the data is over 1 MB, the Preview will display an error.
Click Add.

The following shows an example output which appears after the data source is created in Styra DAS.

{
    "data": {
        "s3-test.json": {
            "foo1": "bar1"
        },
        "s3-test.yaml": {
            "foo3": "bar3"
        },
        "s3-test.yml": {
            "foo2": "bar2"
        }
    }
}

Configure Data Source through the Styra DAS API

The HTTPS Data Source plugin supports both common and more advanced HTTP queries with the ability to use custom HTTP headers.

To create the HTTPS data source plugin, run the following curl command:

curl -H 'authorization: bearer XXX' -H 'Content-Type: application/json' "https://<das-id>.styra.com/v1/datasources/http/url" -XPUT -d'
{
  "category": "http",
  "url": "<datasource url>",
  "polling_interval: "60s",
  "headers": [
    {
      "name": "<header name>",
      "value": "<header value>",
      "secret_id": "<id of the stored secret>"
    }
  ],
  "skip_tls_verification": true,
  "ca_certificate": "<pem file>"
}'

The url parameter is a link to an endpoint that returns a data in JSON or YAML format.
The polling_interval parameter holds the interval value of 60 in seconds. The value is float.
The headers parameter is a list of custom headers with the following information:
- name: Name of the header. This field is mandatory.
- value: Value of the header stored as a plain text.
- secret_id: Name of the secret stored in the system. For more information about secrets, see the secrets API definition.
important
Set the value or secret_id, but not both.
The skip_tls_verification allows to ignore any custom or wrong certificates. Default value: false.
The ca_certificate allows to use a custom CA certificate. A certificate should be uploaded as plain text in pem format.
The method allows to use any HTTP methods in request.
The body allows to send a JSON payload
The raw_body allows to send Non-JSON payload, must be encoded in base64.

Configuration Example

Sample configuration
{
    "category": "http",
    "url": "https://www.example.com/api/test",
    "headers": [
        {
            "name:": "Env",
            "value": "QA"
        },
        {
            "name:": "Authorization",
            "secret_id": "auth/qa-token"
        }
    ]
}

auth/qa-token secret id:

Sample secret request body
{
    "description": "Bearer token",
    "name": "qa-token",
    "secret": "Bearer SUPER-QA-TOKEN"
}

The HTTP Data Source makes a request similar to the following curl command:

curl -H'Env: QA' \
  -H'Content-Type: application/json, text/vnd.yaml, application/yaml, application/x-yaml, text/x-yaml, text/yaml, text/plain' \
  -H'Authorization: Bearer SUPER-QA-TOKEN' \
  -XGET 'https://www.example.com/api/test'

Policy Filtering

A policy_filter is used to poll from a data source that you want to transform captured data source information before storing it. Specifying a policy_filter and policy_query will allow you to apply Rego transformations before it is persisted as data. This mechanism is useful for filtering out data that you no longer want to store or for any other mutations that you want to perform.

It works by specifying a policy that will be evaluated via Rego with captured data as input. You also specify a query to apply to that policy and data. The result of that query will be stored as data, instead of what is polled by the data source plugin.

In addition to the standard options, you must specify the following when you create or update a data source:

The policy_filter parameter is the ID of a policy you want to use for filtering.
The policy-query parameter is the Rego query you want to evaluate.

The following is an example of a curl command with additional parameters:

curl -H 'Authorization: bearer XXX' \
  -H 'Content-Type: application/json' \
  -XPUT 'https://TENANT.styra.com/v1/datasources/http/url' -d'
{
  "category": "http",
  "url": "<datasource url>",
  "headers": [
    {
      "name": "<header name>",
      "value": "<header value>",
      "secret_id": "<id of the stored secret>"
    }
  ],
  "skip_tls_verification": true,
  "ca_certificate": "<pem file>",
  "policy_filter": "/my/test/policy",
  "policy_query": "<rego statement>"
}'

Policy Filtering Example

The following policy filtering example shows how to filter a policy and write a query to apply Rego transformations before it is persisted as data.

Data returned from <URL>:

Sample data
{
    "servers": [
        {
            "id": "app",
            "protocols": [
                "https",
                "ssh"
            ],
            "ports": [
                "p1",
                "p2",
                "p3"
            ]
        },
        {
            "id": "db",
            "protocols": [
                "mysql"
            ],
            "ports": [
                "p3"
            ]
        },
        {
            "id": "cache",
            "protocols": [
                "memcache"
            ],
            "ports": [
                "p3"
            ]
        },
        {
            "id": "ci",
            "protocols": [
                "http"
            ],
            "ports": [
                "p1",
                "p2"
            ]
        },
        {
            "id": "busybox",
            "protocols": [
                "telnet"
            ],
            "ports": [
                "p1"
            ]
        }
    ],
    "networks": [
        {
            "id": "net1",
            "public": false
        },
        {
            "id": "net2",
            "public": false
        },
        {
            "id": "net3",
            "public": true
        },
        {
            "id": "net4",
            "public": true
        }
    ],
    "ports": [
        {
            "id": "p1",
            "network": "net1"
        },
        {
            "id": "p2",
            "network": "net3"
        },
        {
            "id": "p3",
            "network": "net2"
        }
    ]
}

For this example, there exists a relevant policy with parsed Rego:

curl -H 'Authorization: Bearer XXX' \
  -X GET https://<das-id>.styra.com/v1/policies/example/networks

Output:

package example.networks

public_server[server] {                             # a server exists in the public_server set if...
    some i, j
    server := input.servers[_]                      # it exists in the input.servers collection and...
    server.ports[_] == input.ports[i].id            # it references a port in the input.ports collection and...
    input.ports[i].network == input.networks[j].id  # the port references a network in the input.networks collection and...
    input.networks[j].public                        # the network is public.
}

To create a new data source, run the following curl command.

curl -H 'Authorization: Bearer XXX' \
  -H 'Content-Type: application/json' \
  -X PUT https://<das-id>.styra.com/v1/datasources/systems/test/test-datasource \
  -d '
{
  "category": "http",
  "url": "https://<mycustomdatasource>/topology",
  "policy_filter": "/example/networks",
  "policy_query": "data.example.networks.public_server[results]"
}'

After polling occurs, the following shows the result of your query in the data:

curl -H 'Authorization: Bearer XXX' -H 'Content-Type: application/json'
  -X GET https://<das-id>.styra.com/v1/data/systems/test/test-datasource

sample response
{
    "request_id": "<request ID>",
    "result": [
        {
            "id": "app",
            "ports": [
                "p1",
                "p2",
                "p3"
            ],
            "protocols": [
                "https",
                "ssh"
            ]
        },
        {
            "id": "ci",
            "ports": [
                "p1",
                "p2"
            ],
            "protocols": [
                "http"
            ]
        }
    ],
    "revision": "<revision>"
}

Creating or Configuring the Data Source through the Styra DAS UI​

Configure Data Source through the Styra DAS API​

Configuration Example​

Policy Filtering​

Policy Filtering Example​

Creating or Configuring the Data Source through the Styra DAS UI

Configure Data Source through the Styra DAS API

Configuration Example

Policy Filtering

Policy Filtering Example