HTTP Data Source

HTTP data source is similar to Git or S3 data sources. Instead of reading the data from a storage, the HTTP data source reads data from an external server by making HTTP requests.

Configure HTTP Data Sources

HTTP data source plugin supports both common and more advanced HTTP queries with the ability to use custom HTTP headers.

To create the HTTP data source plugin, run the following curl command:

curl -H 'authorization: bearer XXX' -H 'Content-Type: application/json' https://<das-id>.styra.com/v1/datasources/http/url -XPUT -d'
{
  "category": "http",
  "url": "<datasource url>",
  "polling_interval: "60s",
  "headers": [
    {
      "name": "<header name>",
      "value": "<header value>",
      "secret_id": "<id of the stored secret>"
    }
  ],
  "skip_tls_verification": true,
  "ca_certificate": "<pem file>"
}'

The url parameter is a link to an endpoint that returns a data in JSON or YAML format.
The polling_interval parameter holds the interval value of 60 in seconds. The value is float.
The headers parameter is a list of custom headers with the following information:

name: Name of the header. This field is mandatory.
value: Value of the header stored as a plain text.
secret_id: Name of the secret stored in the system. For more information about secrets, see the secrets API page.
important
Set the value or secret_id, but not both.

The skip_tls_verification allows to ignore any custom or wrong certificates. Default value: false.
The ca_certificate allows to use a custom CA certificate. A certificate should be uploaded as plain text in pem format.

Configuration Example

sample configuration
{
  "category": "http",
  "url": "https://www.example.com/api/test",
  "headers": [
    {
      "name:": "Env",
      "value": "QA"
    },
    {
      "name:": "Authorization",
      "secret_id": "auth/qa-token"
    }
  ]
}

auth/qa-token secret id:

sample secret request body
{
  "description": "Bearer token",
  "name": "qa-token",
  "secret": "Bearer SUPER-QA-TOKEN"
}

The HTTP data source makes a request similar to the following curl command:

curl -H'Env: QA' \
     -H'Content-Type: application/json, text/vnd.yaml, application/yaml, application/x-yaml, text/x-yaml, text/yaml, text/plain' \
     -H'Authorization: Bearer SUPER-QA-TOKEN' \
     -XGET 'https://www.example.com/api/test'

Policy Filtering

A policy_filter is used to poll from a data source that you want to transform captured data source information before storing it. Specifying a policy_filter and policy_query will allow you to apply Rego transformations before it is persisted as data. This mechanism is useful for filtering out data that you no longer want to store or for any other mutations that you want to perform.

It works by specifying a policy that will be evaluated via Rego with captured data as input. You also specify a query to apply to that policy and data. The result of that query will be stored as data, instead of what is polled by the data source plugin.

In addition to the standard options, you must specify the following when you create or update a datasource:

The policy_filter parameter is the ID of a policy you want to use for filtering.
The policy-query parameter is the Rego query you want to evaluate.

The following is an example of a curl command with additional parameters:

curl -H 'Authorization: bearer XXX' \
     -H 'Content-Type: application/json' \
     -XPUT 'https://TENATN.styra.com/v1/datasources/http/url' -d'
{
  "category": "http",
  "url": "<datasource url>",
  "headers": [
    {
      "name": "<header name>",
      "value": "<header value>",
      "secret_id": "<id of the stored secret>"
    }
  ],
  "skip_tls_verification": true,
  "ca_certificate": "<pem file>",
  "policy_filter": "/my/test/policy",
  "policy_query": "<rego statement>"
}'

Policy Filtering Example

In this example, you will learn how to filter a policy and write a query that will allow you to apply Rego transformations before it is persisted as data.

Data returned from <URL>:

sample data
{
  "servers": [
    {
      "id": "app",
      "protocols": [
        "https",
        "ssh"
      ],
      "ports": [
        "p1",
        "p2",
        "p3"
      ]
    },
    {
      "id": "db",
      "protocols": [
        "mysql"
      ],
      "ports": [
        "p3"
      ]
    },
    {
      "id": "cache",
      "protocols": [
        "memcache"
      ],
      "ports": [
        "p3"
      ]
    },
    {
      "id": "ci",
      "protocols": [
        "http"
      ],
      "ports": [
        "p1",
        "p2"
      ]
    },
    {
      "id": "busybox",
      "protocols": [
        "telnet"
      ],
      "ports": [
        "p1"
      ]
    }
  ],
  "networks": [
    {
      "id": "net1",
      "public": false
    },
    {
      "id": "net2",
      "public": false
    },
    {
      "id": "net3",
      "public": true
    },
    {
      "id": "net4",
      "public": true
    }
  ],
  "ports": [
    {
      "id": "p1",
      "network": "net1"
    },
    {
      "id": "p2",
      "network": "net3"
    },
    {
      "id": "p3",
      "network": "net2"
    }
  ]
}

2. For this example, there exists a relevant policy with parsed Rego:

curl -H 'Authorization: Bearer XXX' \
     -X GET https://<das-id>.styra.com/v1/policies/example/networks

output:

package example.networks

public_server[server] {                             # a server exists in the public_server set if...
    some i, j
    server := input.servers[_]                      # it exists in the input.servers collection and...
    server.ports[_] == input.ports[i].id            # it references a port in the input.ports collection and...
    input.ports[i].network == input.networks[j].id  # the port references a network in the input.networks collection and...
    input.networks[j].public                        # the network is public.
}

3. To create a new datasource, run the following curl command.

curl -H 'Authorization: Bearer XXX' \
     -H 'Content-Type: application/json' \
     -X PUT https://<das-id>.styra.com/v1/datasources/systems/test/test-datasource \
     -d '
{
  "category": "http",
  "url": "https://<mycustomdatasource>/topology",
  "policy_filter": "/example/networks",
  "policy_query": "data.example.networks.public_server[results]"
}'

4. After polling occurs, the following shows the result of your query in the data:

curl -H 'Authorization: Bearer XXX' -H 'Content-Type: application/json'
-X GET https://<das-id>.styra.com/v1/data/systems/test/test-datasource

sample response
{
  "request_id": "<request ID>",
  "result": [
    {
      "id": "app",
      "ports": [
        "p1",
        "p2",
        "p3"
      ],
      "protocols": [
        "https",
        "ssh"
      ]
    },
    {
      "id": "ci",
      "ports": [
        "p1",
        "p2"
      ],
      "protocols": [
        "http"
      ]
    }
  ],
  "revision": "<revision>"
}

Configure HTTP Data Sources​

important

Configuration Example​

Policy Filtering​

Policy Filtering Example​

Configure HTTP Data Sources

Configuration Example

Policy Filtering

Policy Filtering Example