Skip to main content

Mounting Git Repositories for Libraries

The Styra DAS allows you to mount multiple Git repositories underneath namespaces (for example, data.global). These repositories can contain any number of Rego and JSON files and must follow few restrictions on how they are structured.

This page describes the following sections.

Overview

Through the GUI or API, Styra DAS allows you to mount existing Git repositories that contain Rego and JSON files and make them available to DAS systems and stacks. Styra DAS allows you to create a library of common functionality that you want to share across your organization. Instead of multiple teams writing the same Rego/JSON in separate systems or stacks, you can write them once and share them across multiple teams. This feature allows you to author policies that are spread across many Rego or JSON files. Now, you can use the Styra system concept to control the portions of those policies that get distributed to the corresponding OPAs. After you mount your Git repositories, you can reference them from any system or stack provided that you use the prefix data.global.

For example, when a library of Envoy helpers abstract the JSON path to check if the method is a GET, PUT, POST, or DELETE, then you must use the following Rego to reference these methods.

package global.envoy.methods
is_get { input.attributes.request.http.method == "GET" }
is_post { input.attributes.request.http.method == "POST" }
is_put { input.attributes.request.http.method == "PUT" }
is_delete { input.attributes.request.http.method == "DELETE" }

If you want to avoid repeating this logic in every Envoy system, then you must commit this Rego to a Git repository and mount it into data.global. Now, from an Envoy stack or a system, you can refer to these helpers. For example, data.global.envoy.methods.is_get.

Normally, Styra systems do not allow this kind of sharing because each system or stack is sandboxed; it can only reference Rego packages within that system or stack. But, Styra DAS uses the name global as a reserved package prefix used to refer to the Rego or JSON that lives outside of the system or stack.

warning

This feature is under evaluation, and the details are subject to change.

Concepts

The data.global namespace represents a collection of helpers that the organization maintains to cut across all the different DAS systems and stacks. Every organization is responsible to make the decision on how to manage the Rego or JSON within the data.global namespace.

Use one of the following options to manage the Rego or JSON within the data.global namespace.

  1. You can have a single team to be responsible for the data.global namespace.
  2. You can carve up the data.global namespace and allow multiple teams to contribute. For example, data.global.compute namespace is owned by the Compute team and data.global.networking namespace is owned by the Networking team.

You can mount one or several Git repositories underneath the data.global namespace. If necessary, you can also mount the same repository more than once.

When you mount a repository, you must specify the following.

  • The Git repository, the branch, and a directory of Rego or JSON files.
  • The location underneath the data.global namespace to mount that directory of Rego or JSON files.
  • Optional: Credentials for accessing the Git repository.

For example, you can also decide to mount the following Git repositories using the master branch.

  • github.com/alice/policies/toystore at data.global.toystore
  • github.com/alice/policies/petstore at data.global.petstore
  • github.com/eve/rego/envoy/http at data.global.envoy.http

The locations underneath data.global namespace that you mount to cannot overlap. When you pick a mount point, everything underneath it is controlled by what is inside the Git directory.

For example, the following mounts will cause an error:

  • github.com/alice/policies/toystore at data.global.toystore
  • github.com/alice/policies/petstore at data.global.toystore.petstore ERROR: MOUNTING A SUFFIX IS NOT ALLOWED.

This is an expected behavior to ensure that if you mount two different Git repositories, the team or organization is responsible for those repositories which allows you to add the content without using other repositories.

Package and Directory Restrictions

There are restrictions on the packages and directory structures within the Rego or JSON files that are mounted. The main goal of applying these restrictions is to ensure that you work with the Rego or JSON files when mounted into the Styra DAS, like the same way you work with the files locally.

The package and directory restrictions specifically ensures the following conditions:

  • opa test can be used in CI pipelines and on developer laptops.
  • Multiple repositories can be used simultaneously by a single DAS system or stack.
  • Policy readers and writers can see the same policy in both Git or Styra DAS. This implies, the mounting process requires no package rewriting by the Styra DAS.

You must apply the following restrictions when you mount a Git repository.

  • The package hierarchy of Rego must be a suffix of the mounting location under data.global.
  • The directory hierarchy used to store Rego within Git must match its package hierarchy. The directory hierarchy for JSON controls where underneath data.global the data gets loaded.

If you mount <repo>/<repopath> at data.global.<datapath> then the files within <repo>/<repopath> are considered as a normal OPA bundle, but with a couple of additional restrictions.

When you mount a directory, the Styra DAS shows the following behavior:

  • In an OPA bundle, any file other than .rego and data.json is ignored.
  • In an OPA bundle, the file <path>/data.json is loaded into data.<path>.
  • Unlike an OPA bundle, all Rego packages must be a suffix of data.global.<datapath>, or the mount fails.
  • Unlike an OPA bundle, any Rego module defining package data.global.<datapath>.<extendedpath> must appear within the directory <repo>/<repopath>/global/<datapath>/<extendedpath>, or the mount fails.
  • Unlike an OPA bundle, all JSON files must appear in a subdirectory of global/<datapath>, or the mount fails.

For example, the following shows the contents of the directory located in the Git repository when you mount policies for a toystore under data.global.petstore namespace.

  • global/petstore/bar/policy.rego is a Rego file with the directive package global.petstore.bar.

  • global/petstore/attributes/users/data.json is a JSON file referenced via data.global.petstore.attributes.users.

.
└── clusters/
└── prod/
└── global/
└── petstore/
└── bar/ # Repository path `clusters/prod/global/petstore/bar`
└──policy.rego # package global.petstore.bar
└── attributes/
└── users/
└── data.json # Referenced via `data.global.petstore.attributes.users`

This directory structure allows you to run opa test from a CI pipeline or on a developer laptop. Run opa test from one level above the global directory in your repository, as follows.

$ ls
global/
$ opa test .
PASS 2/2

Mount a Git Repository

Mount a Git repository using one of the following ways:

info
  • Public repositories do not require credentials.
  • Private repositories require credentials.
  • Mounting a Git repository through the API supports both public and private repositories.

Mount a Git Repository through the GUI

To mount a private Git repository underneath the data.global namespace, provide the following information:

  1. Click on Styra Tenant.

  2. On the left-hand navigation panel, click the ( ⨁ ) plus icon next to the LIBRARIES heading.

  3. Now, you will see a Add library dialog.

  4. Enter the following required fields listed in the Add library dialog.

    • Datapath (required) field: Enter the path under the data.global namespace where the library will be mounted.

    • Configure Git Authentication.

    • Click Add library button.

note

If package errors occur, at least the top level node for the library will appear in the left-hand navigation panel and the error messages will appear in the Status tab. For more information, see Package and Directory Restrictions section.

Configure Git Authentication

Styra DAS supports either HTTPS or SSH authentication for Git. You can navigate to the Git settings dialog and select the Git authentication mode HTTPS or SSH. The Git settings may only have one authentication mode selected per system level.

HTTPS

  • Git username (required): Your Git username.
  • Git secret (required): The secret corresponding to your Git username.
  • Git repository (required): A Git HTTPS URL to your Git repository. For example: https://github.com/hooli/foo.git.
  • Git reference: Specify a tag or branch reference (defaults to refs/heads/masterβ€”the master branch).
  • Repository path: (Optional) The subdirectory where you want to save the policies.

SSH

  • SSH keyΒ (required): A private SSH key. For example: The contents of ~/.ssh/id_rsa.

  • SSH key passphrase: (Optional) The passphrase specified at the time the private SSH key was created.

  • Git repository (required): A Git SSH URL to your Git repository. For example: git@github.com:hooli/foo.git.

  • Git reference: Specify a tag or branch reference (defaults to refs/heads/masterβ€”the master branch).

  • Repository path: (Optional) The subdirectory where you want to save the policies.

important

The corresponding SSH public key must be configured on the Git service (GitHub, Bitbucket, and so on) in order for authentication to work.

Configure GitHub Repositories

Use the following tips to find the required configuration for a GitHub repository:

  • Git secret: Styra recommends to use a GitHub Personal access token. You can generate a token at github.com/settings/token or by clicking through Your-picture and navigate to Settings >> Developer Settings >> Personal access tokens.

  • Git repository: To create a new Git repository, navigate to github.com/settings/token and click your profile icon located on the top-right of the page. To view your repositories, click Your repositories.

    • To add a new repository, click the New button.

    • In the new repository, enter the name of your new repository in the Repository name field, the Description field is optional.

    • Select Public (Anyone on the internet can see this repository and you can choose who can commit) or Private (You choose who can see and commit to this repository) based on your requirement.

    • Click the Create repository button.

tip

Navigate your repository on GitHub and copy the link provided for Clone with HTTPS. For example, navigate to github.com/hooli/policyrepoand click the Clone or Download button to copy the link provided for Clone with HTTPS.

GUI Features

On the left-hand navigation panel, click your library name located under the LIBRARIES heading. In the right-hand panel, the details for your library displays with the Status tab and Settings tab.

  1. Click the Status tab to see the timestamp of the last synchronization and a display message to show the connection: Connected to https://github.com/andy-styra/custom-library-sample, 3 packages, 3 modules.

  2. Click the Settings tab to see the required and optional fields. You can click the following elements on the Settings page for essential functionality.

    • Delete library button to delete the library.
    • Reset button to reset any changes you have made in the settings and to redisplay the current settings.
    • Save changes button to save the changes.

On the left-hand navigation panel, click the expand/collapse arrow on your library name located under the LIBRARIES heading to navigate to library folders and Rego policy and JSON files.

  • A folder corresponds to a Git directory and a Rego package.
  • A folder contains one or more Rego module and or JSON data Git files.

When you push changes to the Library in Git, the GUI will reflect those changes.

Now, you can also mount a Git Repository through the API using the instructions in the following section.

Mount a Git Repository through the API

To mount a Git repository underneath the data.global namespace, provide the following information:

  • The Git Repository, the branch, and a directory of Rego or JSON files.
  • The location underneath the data.global namespace to mount that directory of Rego or JSON files.
  • Optional: The credentials for accessing the Git repository.

The following code shows how to mount a Git repository with HTTPS authentication through the API:

PUT https://styra-das-id.styra.com/v1/datasources/global/<datapath>

{
"category": "git/rego", // Constant.
"type": "pull", // Constant.
"url": <git-repo-url>, // URL to git repository.
"path": <repo-path>, // Path in git repository. Optional, defaults to root ("").
"reference": <git-reference> // Reference in git repository. Optional, defaults to
// "refs/heads/master".
"credentials": <styra-das-secret> // Name of a DAS secret with Git username in 'name' and Git
// password/token in 'secret'. Optional.
}

For example, run the following curl command to mount your repository under data.global.toystore:

curl  -H 'Authorization: Bearer XXX' -H 'Content-Type: application/json' \
-X PUT \
https://styra-das-id.styra.com/v1/datasources/global/toystore -d '{
"category": "git/rego",
"type": "pull",
"url": "https://github.com/alice/policies.git",
"path": "toystore-app/global/toystore",
"reference": "refs/heads/master"
"credentials": "alice/repo/policies",
}'

The following code shows how to mount a Git repository with SSH authentication through the API:

PUT https://styra-das-id.styra.com/v1/datasources/global/<datapath>

{
"category": "git/rego", // Constant.
"type": "pull", // Constant.
"url": <git-repo-url>, // URL to git repository.
"path": <repo-path>, // Path in git repository. Optional, defaults to root ("").
"reference": <git-reference> // Reference in git repository. Optional, defaults to
// "refs/heads/master".
"ssh_credentials":
{
"private_key": "<secret/key/id>", // Name of a DAS secret with SSH private key.
"passphrase": "<secret/passphrase/id>" // Name of a DAS secret with SSH passphrase. Optional.
}
}

For example, run the following curl command to mount your repository under data.global.toystore:

curl  -H 'Authorization: Bearer XXX' -H 'Content-Type: application/json' \
-X PUT \
https://styra-das-id.styra.com/v1/datasources/global/toystore -d '{
"category": "git/rego",
"type": "pull",
"url": "https://github.com/alice/policies.git",
"path": "toystore-app/global/toystore",
"reference": "refs/heads/master"
"ssh_credentials": {
"private_key": "alice/repo/key"
"passphrase": "alice/repo/passphrase"
}
}'
note
  • If you want to mount your repository under data.global.foo.bar, then you must use the PUT method in https://styra-das-id.styra.com/v1/datasources/global/foo/bar.
  • The repository and directory within that repository, and the branch are handled by the URL, path, and reference fields respectively.
  • The HTTPS credentials, SSH private_key, and SSH passphrase fields are the name of a Styra DAS secret. You must always use the "category": "git/rego" and "type": "pull".
note

The directories are related, as follows:

  • PUT is assigned to v1/datasources/global/toystore. This shows the Git directories mounted on data.global.toystore.

  • The Git directory toystore-app/global/toystore is mounted.

When mounting occurs, the Styra DAS ignores everything in the Git repository except the directory toystore-app/global/toystore. All the files within that directory (and subdirectories) must obey the restrictions described earlier, where the directory prefix global/toystore is implicitly added to all of the directory paths. The toystore-app/global prefix on the directory path helps you to run opa test from toystore-app and have OPA load any data files with the prefix data.global.toystore. This is the expected behavior from your policies.

Check the Mount Status

After you mount your Git repository and receive 200 responses, the Styra DAS performs the following tasks:

  • Clone the entire repository.
  • Walk the appropriate directory.
  • Ingest the Rego and JSON files within it.
  • Report the status.

Periodically, the Styra DAS refreshes the mounted files by repeating the tasks. When the repository is updated, the status is also updated to report issues, if any.

To check the status, run the following GET method on the resource that you have created.

curl  -H 'Authorization: Bearer XXX' \
-H 'Content-Type: application/json' \
-X GET https://styra-das-id.styra.com/v1/datasources/global/toystore

Output

{"request_id": "01812657-fd15-4d09-9576-d22cff954c29",
"result": {
"category": "git/rego",
"type":"pull",
"url":"https://github.com/alice/policies.git"},
"path":"toystore-app/global/toystore",
"reference":"refs/heads/master",
"credentials": "alice/repo/policies",
"id": "global/toystore",

"metadata":{
"last_modified_at": "2020-03-13T22:10:04.329565746Z",
"last_modified_by": "alice@initech.com",
"last_modified_through": "access/alice/repo/mount"},
"status":{
"code":"finished",
"message":"",
"timestamp":"2020-03-13T22:10:06Z"}
}

The status field reports a success or failure. In case of failure, an error message is displayed. Most often, the problem comes from a Rego file that does not have the proper package name.

note

The Git protocol does not support pulling a subdirectory of a repository without pulling the entire repository. Styra recommends you to put your Rego or JSON into a dedicated repository, in order to minimize the load on keeping the repository synchronized and to avoid the Styra DAS having a copy of possibly sensitive (but irrelevant) information.

Create a Secret

If you are mounting a private Git repository, you must provide a username and password for the Styra DAS to access the repository through the standard Git protocol. Styra DAS has its own way of keeping secrets. At the time of writing, you can only manage those secrets through the API.

A secret is an object with the following two fields:

  • name: the username for the credential.
  • secret: the secret for the credential.

Every secret has a hierarchical name that you can use in other Styra DAS API calls.

To create a secret named alice/repos/policies, run the following curl command:

curl  -H 'Authorization: Bearer XXX' -H 'Content-Type: application/json' \
-X PUT https://styra-das-id.styra.com/v1/secrets/alice/repos/policies -d '{
"name": "alice",
"secret": "super-secret-Password-44321"
}'

Use the Mounted Git Repositories within a System

Now, your Git repository is mounted successfully. Use the repository within a DAS system by referring to those policies via data.global namespace.

To see how this works first-hand, imagine you mounted the github.com/alice/policies/toystore at data.global.toystore and github.com/alice/policies/petstore at data.global.petstore repositories.

To see these policies working:

  1. Create a new DAS system (of type Custom).
  2. Add the following rules that takes a request for either the petstore or the toystore and route the request to the appropriate policy.
package rules

default allow = false
allow {
input.path[0] == "petstore"
data.global.petstore.bar.allow
}

allow {
input.path[0] == "toystore"
data.global.toystore.bar.allow
}

You can also write the following tests.

package test

test_allow_petstore_positive {
data.rules.allow with input as {
"user": "alice",
"path": ["petstore", "dogs"],
"method": "GET"
}
}

test_allow_petstore_negative {
not data.rules.allow with input as {
"user": "bob",
"path": ["petstore", "dogs"],
"method": "GET"
}
}

test_allow_toystore_positive {
data.rules.allow with input as {
"user": "bob",
"path": ["toystore", "trucks"],
"method": "GET"
}
}

test_allow_toystore_negative {
not data.rules.allow with input as {
"user": "alice",
"path": ["toystore", "trucks"],
"method": "GET"
}
}

Use the Bundle API from OPA

info

This is an optional task.

The data.global namespace also supports OPA's bundle API. You can configure OPA to download Rego or JSON files directly or manually download and execute the files in your local machine. For example, you can debug the files.

The following example shows how to manually download a bundle using the Styra DAS bundle API.

curl  -H 'Authorization: Bearer XXX' \
-H 'Content-Type: application/json'
-X GET 'https://styra-das-id.styra.com/v1/bundles?policy=global/toystore/bar/allow' \
--output bundle.tar.gz

Run the following OPA command to load the bundle:

opa run -b ./bundle.tar.gz