Skip to main content

Write Custom Rules

In Kubernetes, admission controllers enforce semantic validation of objects during create, update, and delete operations. With Styra and OPA, you can enforce custom policies on Kubernetes objects without recompiling or reconfiguring the Kubernetes API server or even Kubernetes admission controllers.

This primer assumes you (the Kubernetes administrator) have already installed Styra and are using OPA as a validating admission controller on Kubernetes. You tried the pre-built policies that Styra offers and decided that you want to write your own custom policies by using OPA policy language.

OPA is designed to write policies over arbitrary JSON or YAML. It does not have built-in concepts like pods, deployments, or services. OPA uses the JSON or YAML files sent by Kubernetes API server and allows you to write the policy you want to make a decision. As a policy-author, you are aware of the semantics and representation of the JSON or YAML files.

Image Registry Safety​

Your first rule is about image registry safety. To get started, consider a common policy, such as ensure all images that come from a trusted registry.

note

A more comprehensive implementation of the same policy is part of the policy library (Containers: Restrict Images (Exact)).

package policy["com.styra.kubernetes.validating"].rules.rules

enforce[decision]  {
    my_custom[decision]
}

my_custom[decision] {
    input.request.kind.kind == "Pod"
    image := input.request.object.spec.containers[_].image
    not startswith(image, "hooli.com")
    msg := sprintf("image fails to come from trusted registry: %v", [image])

    decision := {
        "allowed": false,
        "message": msg
    }
 }

Policies and Packages: In line #1, the package policy["com.styra.kubernetes.validating"].rules.rules declaration gives the (hierarchical) name policy["com.styra.kubernetes.validating"].rules.rules to the rules in the remainder of the policy. The default installation of OPA as an admission controller assumes your rules are in the package policy["com.styra.kubernetes.validating"].rules.rules.

Deny Rules: For admission control, you write enforce statements. The order of writing statements is not a concern. OPA is very flexible compared to admission control. To begin with, Styra recommends writing only the enforce statements.

In line #2, the head of the rule enforce[decision] says that the admission control request should be rejected and the user handed the decision structure decision.

In line #3, you must evaluate the custom rule my_custom which will return a decision if the conditions in the body (the statements between the {}) are true. Styra uses a helper my_custom to test only custom rules.

enforce is the set of error decisions that should be returned to the user. Each rule you write adds to that set of decisions.

The following shows how to create a Pod with NGNIX and MySQL images:

kind: Pod
apiVersion: v1
metadata:
  name: myapp
spec:
  containers:
  - image: nginx
    name: nginx-frontend
  - image: mysql
    name: mysql-backend

enforce evaluates to the following set of messages:

{
    "enforce": [
        {
            "allowed": false,
            "message": "image fails to come from trusted registry: nginx"
        },
        {
            "allowed": false,
            "message": "image fails to come from trusted registry: mysql"
        }
    ]
}

Input: In OPA, input is a reserved, global variable whose value is the Kubernetes AdmissionReview object that the API server hands to any admission control webhook.

AdmissionReview objects have many fields. The above rule uses input.request.kind, which includes the usual group/version/kind information. The rule also uses input.request.object, which is the YAML that the user provided to kubectl (augmented with defaults, timestamps, and so on). The full input object is 50+ lines of YAML.

The following shows only relevant parts of the YAML file.

apiVersion: admission.k8s.io/v1beta1
kind: AdmissionReview
request:
  kind:
    group:
    kind: Pod
    version: v1
  object:
    metadata:
      name: myapp
    spec:
      containers:
        - image: nginx
          name: nginx-frontend
        - image: mysql
          name: mysql-backend

Dot notation: In line #7, input.request.kind.kind == "Pod", the expression input.request.kind.kind descends through the YAML hierarchy. The dot (.) operator never throws any errors; if the path does not exist then the value of the expression is undefined.

You can see OPA evaluation in OPA's interactive Read-Eval-Print Loop (REPL) using the opa run CLI command.

$ opa run
> input.request.kind
{
  "group": null,
  "kind": "Pod",
  "version": "v1"
}
> input.request.kind.kind
"Pod"
> input.request.object.spec.containers
[
  {
    "image": "nginx",
    "name": "nginx-frontend"
  },
  {
    "image": "mysql",
    "name": "mysql-backend"
  }
]

Equality: A form of equality is used in line #7, #8, and #9. There are three forms of equality in OPA.

  • x := 7 declares a local variable x and assigns variable x to the value 7. The compiler throws an error if x already has a value.
  • x == 7 returns true, if x's value is 7. The compiler throws an error, if x has no value.
  • x = 7 either assigns x to 7, if x has no value. If x has a value, then it compares x's value to 7. The compiler never throws an error.
note

The recommendation for writing rules is to use := and == wherever possible. Rules are easier to write and read. Using = is invaluable in more advanced use cases, and outside of rules is the only supported form of equality.

Arrays: In line #8 and line #9, find images in the Pod that don't come from the trusted registry. To achieve this task, use the [] operator, which does what you expect: index into the array.

The following is a continuation of the example in the Dot notation section:

> input.request.object.spec.containers[0]
{
  "image": "nginx",
  "name": "nginx-frontend"
}
> input.request.object.spec.containers[0].image
"nginx"

The [] operators allows you to use variables to index into the array.

> i := 0
> input.request.object.spec.containers[i]
{
  "image": "nginx",
  "name": "nginx-frontend"
}

Iteration: The containers array has an unknown number of elements, so to implement an image registry check you need to iterate over them. Iteration in OPA requires no new syntax. In fact, OPA is always iterating. This indicates that OPA always searches for all variable assignments that make the conditions = TRUE in the rule. The search feature is very easy, and is not explicitly seen through its functionality.

To iterate over the indexes in the input.request.object.spec.containers array, you must put a variable that has no value in for the index. OPA will do what it always does: find values for that variable that make the conditions = TRUE.

In the REPL, OPA does the following:

  • Detects when there will be multiple answers.
  • Displays all the results in a table.
> input.request.object.spec.containers[j]
+---+-------------------------------------------+
| j |  input.request.object.spec.containers[j]  |
+---+-------------------------------------------+
| 0 | {"image":"nginx","name":"nginx-frontend"} |
| 1 | {"image":"mysql","name":"mysql-backend"}  |
+---+-------------------------------------------+

Often, you don't want to invent new variable names for iteration. OPA provides the special anonymous variable _ for exactly that reason. Therefore, in line #8 image := input.request.object.spec.containers[_].image finds all the images in the containers array and assigns each to the image variable one at a time.

Built-ins: In line #9, the builtin startswith checks if one string is a prefix of the other. The builtin sprintf on line #10 formats a string with arguments. OPA has 50+ built-ins listed at openpolicyagent.org/docs/latest/policy-reference/.

Built-ins allow you to analyze and manipulate the following:

  • Numbers, Strings, Regular Expressions, and Networks.
  • Aggregates, Arrays, and Sets.
  • Types.
  • Encodings (base64, YAML, JSON, URL, JWT).
  • Time.

Unit Testing​

When you write policies, you should use the OPA unit-test framework before sending the policies out into the OPA that is running on your cluster. Now, the debugging process is quicker and effective.

The following is an example test for the policy:

package policy["com.styra.kubernetes.validating"].test.test

test_image_safety {
    unsafe_image := {"request": {
        "kind": {"kind": "Pod"},
        "object": {"spec": {"containers": [
            {"image": "hooli.com/nginx"},
            {"image": "busybox"}
        ]}}
    }}
    count(policy["com.styra.kubernetes.validating"].rules.rules.my_custom) == 1 with input as unsafe_image
}

Different Package: In line #1, the package directive puts these tests in a different namespace than admission control policy itself. This is the recommended best practice.

Unit Test: In line #4, test_image_safety defines a unit test. If the rule evaluates to TRUE, then the test passes; otherwise it fails. When you use the OPA test runner, anything in any package starting with test is treated as a test.

Assignment: In line #5, unsafe_image is the input you want to use for the test. Ideally, this would be a real AdmissionReview object. In the above example, a partial input is hand-rolled.

Dot for packages: In line #10, the Dot operator on a package is used. policy["com.styra.kubernetes.validating"].rules.rules.my_custom runs the my_custom rule, as shown above.

Test Input: In line #10, with input as unsafe_image sets the value of input to be unsafe_image while evaluating count(policy["com.styra.kubernetes.validating"].rules.rules.my_custom) == 1.

Running Tests: Different packages must go into different files. If you have created the files image-safety.rego and test-image-safety.rego then you can run the tests with opa test.

$ opa test image-safety.rego test-image-safety.rego
PASS: 1/1

Existing Kubernetes Resources: Ingress Conflicts​

The image-repository example is one of the simpler access control policies you might need to write for Kubernetes because you can make the decision using only the JSON or YAML file describing the pod. But, sometimes you need to know what other resources exist in the cluster to make an allow or deny decision.

For example, it is possible to accidentally create two applications serving internet traffic using Kubernetes ingresses where one application steals traffic from the other. The policy that prevents that needs to compare a new ingress that is being created or updated with all of the existing ingresses.

Consider the AdmissionReview input, as follows:

apiVersion: admission.k8s.io/v1beta1
kind: AdmissionReview
request:
  kind:
    group: extensions
    kind: Ingress
    version: v1beta1
  object:
    metadata:
      name: prod
    spec:
      rules:
      - host: initech.com
        http:
          paths:
          - path: /finance
            backend:
              serviceName: banking
              servicePort: 443

The following policy avoids having two ingresses with the same host:

package policy["com.styra.kubernetes.validating"].rules.rules

enforce[msg] {
    input.request.kind.kind == "Ingress"
    newhost := input.request.object.spec.rules[_].host
    oldhost := data.kubernetes.resources.ingresses[namespace][name].spec.rules[_].host
    newhost == oldhost
    msg := sprintf("ingress host conflicts with ingress %v/%v", [namespace, name])

    decision := {
        "allowed": false,
        "message": msg
    }
}

The first part of the rule represents the following:

  • Line #3: Checks if the input is an Ingress.
  • Line #4: Iterates over all the rules in the input Ingress and looks up the host field for each of its rules.

Existing Kubernetes Resources: Line #5 iterates over ingresses that already exist in Kubernetes. data is a global variable where (among other things) OPA has a record of the current resources inside Kubernetes.

The following line finds all ingresses in all namespaces, iterates over all the rules inside each of namespace, and assigns the host field to the variable oldhost. A conflict occurs when newhost == oldhost, and the OPA rule includes an appropriate decision into the enforce set.

oldhost := data.kubernetes.resources.ingresses[namespace][name].spec.rules[_].host

In this case, the rule uses explicit variable names namespace and name for iteration, so that it can use those variables again when constructing the error message in line 7.

Schema Differences: Both input and data.kubernetes.resources.ingresses[namespace][name] represent ingresses, but they do so differently.

  • data.kubernetes.resources.ingresses[namespace][name] is a Kubernetes Ingress object.
  • input is a Kubernetes AdmissionReview object. It includes several fields in addition to the Kubernetes Ingress object itself.

The following table shows examples of Schema differences.

+------------------------------------------------------+--------------------------------------+
| data.kubernetes.resources.ingresses[namespace][name] |                input                 |
+------------------------------------------------------+--------------------------------------+
| apiVersion: extensions/v1beta1                       | apiVersion: admission.k8s.io/v1beta1 |
| kind: Ingress                                        | kind: AdmissionReview                |
| metadata:                                            | request:                             |
|   name: prod                                         |   kind:                              |
| spec:                                                |     group: extensions                |
|   rules:                                             |     kind: Ingress                    |
|   - host: initech.com                                |     version: v1beta1                 |
|     http:                                            |   operation: CREATE                  |
|       paths:                                         |   userInfo:                          |
|       - path: /finance                               |     groups:                          |
|         backend:                                     |     username: alice                  |
|           serviceName: banking                       |   object:                            |
|           servicePort: 443                           |     metadata:                        |
|                                                      |       name: prod                     |
|                                                      |     spec:                            |
|                                                      |       rules:                         |
|                                                      |       - host: initech.com            |
|                                                      |         http:                        |
|                                                      |           paths:                     |
|                                                      |           - path: /finance           |
|                                                      |             backend:                 |
|                                                      |               serviceName: banking   |
|                                                      |               servicePort: 443       |
+------------------------------------------------------+--------------------------------------+

OPA Cheat Sheet​

Lookup Data​

  • Check if label foo exists (whether or not labels exists): input.request.metadata.labels.foo.
  • Check if label foo does not exist: not input.request.metadata.labels.foo.
  • Check if label first.name exists: input.request.metadata.labels["first.name"].

Equality​

  • Inside a rule, assign variable x to the value of label costcenter : x := input.request.metadata.labels.costcenter.
  • Outside of a rule, assign variable whitelist to the set hooli.com and initech.com: whitelist = {"hooli.com", "initech.com"}.
  • Check if variable x and variable y have the same value: x == y.

Iterate over Components of AdmissionReview​

  • Iterate over label names: input.request.metadata.labels[name].
  • Iterate over label name/value pairs: value := input.request.metadata.labels[name].
  • Iterate over spec containers: container := input.request.object.spec.containers[_].

Iterate over Existing Resources​

  • All ingresses: data.kubernetes.resources.ingresses[namespace][name].
  • All ingresses in namespace prod: data.kubernetes.resources.ingresses["prod"][name] OR data.kubernetes.resources.ingresses.prod[name].
  • All resources: data.kubernetes.resources[kind][namespace][name].
  • All resources in namespace prod: data.kubernetes.resources[kind]["prod"][name].

Sets​

  • Iterate over elements in the set messages: messages[msg].
  • Check if the message "Image fails to come from trusted registry: nginx" belongs to the set messages: messages["image fails to come from trusted registry: nginx"].

Packages and Policies​

  • Name a collection of rules policy["com.styra.kubernetes.validating"].rules.rules: package policy["com.styra.kubernetes.validating"].rules.rules.
  • Within the policy["com.styra.kubernetes.validating"].rules.rules package, evaluate all of the enforce rules: enforce.
  • Outside the policy["com.styra.kubernetes.validating"].rules.rules package, evaluate all of its deny rules: data.policy["com.styra.kubernetes.validating"].rules.rules.enforce.
  • Outside the policy["com.styra.kubernetes.validating"].rules.rules package, create an alias adm to that package: import data.policy["com.styra.kubernetes.validating"].rules.rules as adm.

Testing​

  • Create a test: test_NAME { ... }.
  • Mock out input with {"foo": "bar"} and evaluate enforce within package policy["com.styra.kubernetes.validating"].rules.rules: policy["com.styra.kubernetes.validating"].rules.rules.enforce with input as {"foo": "bar"}.

Admission Control Flow​

Read more about admission control flow between user to API server to OPA, and vice versa on admission control flow page.

The following file is displayed when you run kubectl create -f on a minikube cluster:

kind: Pod
apiVersion: v1
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  containers:
  - image: nginx
    name: nginx

OPA receives the following AdmissionReview object from the Kubernetes API server's Admission Control webhook.

apiVersion: admission.k8s.io/v1beta1
kind: AdmissionReview
request:
  kind:
    group: ''
    kind: Pod
    version: v1
  namespace: opa
  object:
    metadata:
      creationTimestamp: '2018-10-27T02:12:20Z'
      labels:
        app: nginx
      name: nginx
      namespace: opa
      uid: bbfee96d-d98d-11e8-b280-080027868e77
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: nginx
        resources: {}
        terminationMessagePath: "/dev/termination-log"
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: "/var/run/secrets/kubernetes.io/serviceaccount"
          name: default-token-tm9v8
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 300
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 300
      volumes:
      - name: default-token-tm9v8
        secret:
          secretName: default-token-tm9v8
    status:
      phase: Pending
      qosClass: BestEffort
  oldObject:
  operation: CREATE
  resource:
    group: ''
    resource: pods
    version: v1
  uid: bbfeef88-d98d-11e8-b280-080027868e77
  userInfo:
    groups:
    - system:masters
    - system:authenticated
    username: minikube-user

OPA returns the following AdmissionReview response to the Admission Controller webhook. The Kubernetes API server returns response.status.reason as an error message to the user. It is the concatenation of all the messages in the deny set defined above. In this case, the policy that OPA evaluated requires all images to come from the hooli.com registry.

apiVersion: admission.k8s.io/v1beta1
kind: AdmissionReview
response:
  allowed: false
  status:
    reason: "image fails to come from trusted registry: nginx"