Integrating with Envoy External Authorization

Envoy, the popular open source service proxy, supports external authorization filters as part of its configuration.

Enterprise OPA has support responding to the CheckRequest message sent by Envoy over gRPC.

Overview

In this tutorial we'll see how to use Enterprise OPA as an External Authorization service for the Envoy proxy. We'll do this by:

• Running a local Kubernetes cluster
• Creating a simple authorization policy in Rego and serving it via the Bundle API
• Deploying a sample application with Envoy and Enterprise OPA sidecars
• Run some sample requests to see the policy in action

Once we're finished, we'll have set up a system of containers to do the following:

Note that other than the HTTP client and bundle server, all of the components are co-located in the same pod.

Running a local Kubernetes cluster

To start a local Kubernetes cluster to run our demo, we'll be using kind. In order to use the kind command, you'll need to have Docker installed on your machine. Running docker info is the easiest way to check if Docker is installed and running.

You should see output similar to the following, showing information about the Docker client and server on our machine:

docker info
Client:
  ...

Server:
 ...

If the above command shows information for both the client and server, then Docker is installed and running.

info

If you haven't used kind before, you can find installation instructions in the project documentation.

Create a cluster with the following command:

kind create cluster --name eopa-envoy --image kindest/node:v1.27.3
Creating cluster "eopa-envoy" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
...

Once the cluster is created, make sure your kubectl context is set to connect to the new cluster:

kubectl cluster-info --context kind-eopa-envoy
Kubernetes control plane is running at ...
CoreDNS is running at ...
...

Listing the cluster nodes, should show something like this:

kubectl get nodes
NAME                       STATUS   ROLES           AGE     VERSION
eopa-envoy-control-plane   Ready    control-plane   2m35s   v1.27.3

Creating & Serving our Policy Bundle

This tutorial assumes you have some Rego knowledge, in summary the policy below does the following:

• Checks that the JWT token is valid
• Checks that the action is allowed based on the token payload role and the request path
• Guests have read-only access to the /people endpoint, admins can create users too as long as the name is not the same as the admin's name.

# policy.rego
package envoy.authz

import future.keywords.if

import input.attributes.request.http as http_request

default allow := false

allow if {
        is_token_valid
        action_allowed
}

is_token_valid if {
        token.valid
        now := time.now_ns() / 1000000000
        token.payload.nbf <= now
        now < token.payload.exp
}

action_allowed if {
        http_request.method == "GET"
        token.payload.role == "guest"
        glob.match("/people", ["/"], http_request.path)
}

action_allowed if {
        http_request.method == "GET"
        token.payload.role == "admin"
        glob.match("/people", ["/"], http_request.path)
}

action_allowed if {
        http_request.method == "POST"
        token.payload.role == "admin"
        glob.match("/people", ["/"], http_request.path)
        lower(input.parsed_body.firstname) != base64url.decode(token.payload.sub)
}

token := {"valid": valid, "payload": payload} if {
        [_, encoded] := split(http_request.headers.authorization, " ")
        [valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}

Create a file called policy.rego with the above content and store it in a ConfigMap:

kubectl create configmap authz-policy --from-file policy.rego

Now that the policy is stored in a ConfigMap, we can spin up an HTTP server to make it available to as a Bundle to Enterprise OPA when it's making decisions for our application:

# bundle-server.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bundle-server
  labels:
    app: bundle-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bundle-server
  template:
    metadata:
      labels:
        app: bundle-server
    spec:
      initContainers:
        - name: enterprise-opa-builder
          image: ghcr.io/styrainc/enterprise-opa:latest
          args:
            - "build"
            - "--bundle"
            - "/opt/policy/"
            - "--output"
            - "/opt/output/bundle.tar.gz"
          volumeMounts:
            - name: index
              mountPath: /opt/output/
            - name: policy
              mountPath: /opt/policy/
      containers:
        - name: bundle-server
          image: nginx:1.25
          ports:
            - containerPort: 80
              name: http
          volumeMounts:
            - name: index
              mountPath: /usr/share/nginx/html
      volumes:
        - name: index
          emptyDir: {}
        - name: policy
          configMap:
            name: authz-policy
---
apiVersion: v1
kind: Service
metadata:
  name: bundle-server
spec:
  selector:
    app: bundle-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: http

Create a file called bundle-server.yaml with the above content and apply it to the cluster:

kubectl apply -f bundle-server.yaml

Once the deployment is running, we can check that the bundle is available by running:

kubectl port-forward service/bundle-server 8080:80

Before checking that the bundle has been generated correctly and is available to download:

curl -I localhost:8080/bundle.tar.gz
HTTP/1.1 200 OK
...

You may now exit the port-forwarding session, the bundle server will only be accessed from inside the cluster from now on.

Deploying an application with Envoy and Enterprise OPA sidecars

In this tutorial, we are manually configuring the Envoy proxy sidecar to intermediate HTTP traffic from clients and our application. Envoy will consult Enterprise OPA to make authorization decisions for each request by sending CheckRequest messages over a gRPC connection.

We will use the following Envoy configuration to achieve this. In summary, this configures Envoy to:

• Listen on port 8000 for HTTP traffic
• Consult Enterprise OPA for authorization decisions at 127.0.0.1:9191 & deny failing requests
• Forward requests to the application at 127.0.0.1:8080 if ok.

# envoy.yaml
static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service
          http_filters:
          - name: envoy.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              transport_api_version: V3
              with_request_body:
                max_request_bytes: 8192
                allow_partial_message: true
              failure_mode_allow: false
              grpc_service:
                google_grpc:
                  target_uri: 127.0.0.1:9191
                  stat_prefix: ext_authz
                timeout: 0.5s
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: service
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8080
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001
layered_runtime:
  layers:
    - name: static_layer_0
      static_layer:
        envoy:
          resource_limits:
            listener:
              example_listener_name:
                connection_limit: 10000
        overload:
          global_downstream_max_connections: 50000

Create a ConfigMap containing the above configuration by running:

kubectl create configmap proxy-config --from-file envoy.yaml

Our application will be configured using a Deployment and Service. There are a few things to note:

• the pods have an initContainer that configures the iptables rules to redirect traffic to the Envoy proxy.
• the demo-test-server container is a simple user store using in-memory state.
• the envoy container is configured to use the proxy-config ConfigMap we created earlier.
• The Enterprise OPA container is configured to download policy bundles from the in-cluster bundle server (bundle-server.default.svc.cluster.local).
• The Enterprise OPA license key must be set. We show how to do this in the next step.

# app.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
  name: example-app
  labels:
    app: example-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: example-app
  template:
    metadata:
      labels:
        app: example-app
    spec:
      initContainers:
        - name: proxy-init
          image: openpolicyagent/proxy_init:v8
          # Configure the iptables bootstrap script to redirect traffic to the
          # Envoy proxy on port 8000. Envoy will be running as 1111, and port
          # 8282 will be excluded to support Enterprise OPA health checks.
          args: ["-p", "8000", "-u", "1111", "-w", "8282"]
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
            runAsNonRoot: false
            runAsUser: 0
      containers:
        - name: app
          image: openpolicyagent/demo-test-server:v1
          ports:
            - containerPort: 8080
        - name: envoy
          image: envoyproxy/envoy:v1.26.3
          volumeMounts:
            - readOnly: true
              mountPath: /config
              name: proxy-config
          args:
            - "envoy"
            - "--config-path"
            - "/config/envoy.yaml"
          env:
            - name: ENVOY_UID
              value: "1111"
        - name: eopa
          image: ghcr.io/styrainc/enterprise-opa:latest
          args:
            - "run"
            - "--server"
            - "--addr=localhost:8181"
            - "--diagnostic-addr=0.0.0.0:8282"
            - "--set=services.default.url=http://bundle-server.default.svc.cluster.local:80"
            - "--set=bundles.default.resource=bundle.tar.gz"
            - "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
            - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/allow"
            - "--set=decision_logs.console=true"
            - "--set=status.console=true"
          env:
            - name: EOPA_LICENSE_KEY
              value: "$EOPA_LICENSE_KEY" # set in the following step
          livenessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 1
            periodSeconds: 3
      volumes:
        - name: proxy-config
          configMap:
            name: proxy-config
---
apiVersion: v1
kind: Service
metadata:
  name: example-app
spec:
  selector:
    app: example-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080

Trial License

To evaluate Enterprise OPA, you can obtain a trial license by Downloading eopa and running eopa license trial in the terminal.

Set the EOPA_LICENSE_KEY environment variable to your license key:

export EOPA_LICENSE_KEY=...

Deploy the application and Kubernetes Service to the cluster with:

cat app.yaml | envsubst '$EOPA_LICENSE_KEY' | kubectl apply -f -

Check that everything is working by listing the pod (make sure that all three pods are running ok).

kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
bundle-server-5d7bfffdb6-bgn86   1/1     Running   0          1m
example-app-74b4bc88-5d4wh       3/3     Running   0          1m

See the Policy in Action

Run a shell inside the cluster to use for testing. We will use this in-cluster shell for the rest of the tutorial.

kubectl run curl --restart=Never -it --rm --image curlimages/curl:8.1.2 -- sh

Set two tokens for two users, Alice and Bob with different permissions. As defined by our policy:

export ALICE_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiZ3Vlc3QiLCAic3ViIjogIllXeHBZMlU9In0.Uk5hgUqMuUfDLvBLnlXMD0-X53aM_Hlziqg3vhOsCc8"
export BOB_TOKEN="eyJhbGciOiAiSFMyNTYiLCAidHlwIjogIkpXVCJ9.eyJleHAiOiAyMjQxMDgxNTM5LCAibmJmIjogMTUxNDg1MTEzOSwgInJvbGUiOiAiYWRtaW4iLCAic3ViIjogIlltOWkifQ.5qsm7rRTvqFHAgiB6evX0a_hWnGbWquZC0HImVQPQo8"

Listing People

Send a request to list people. This should succeed for both Alice and Bob.

curl -i -H "Authorization: Bearer $ALICE_TOKEN" http://example-app/people
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 18 Jul 2023 15:22:25 GMT
content-length: 96
x-envoy-upstream-service-time: 14
server: envoy

[{"id":"1","firstname":"John","lastname":"Doe"},{"id":"2","firstname":"Jane","lastname":"Doe"}]

And for Bob:

curl -i -H "Authorization: Bearer $BOB_TOKEN" http://example-app/people
HTTP/1.1 200 OK
...omitted...

Creating People

Send a request to create a new user. This should fail for Alice but not Bob:

curl -i -H "Authorization: Bearer $ALICE_TOKEN" \
  -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \
  -X POST http://example-app/people
HTTP/1.1 403 Forbidden
date: Tue, 18 Jul 2023 15:25:28 GMT
server: envoy
content-length: 0

And for Bob, the request is permitted and the user is saved with an ID

curl -i -H "Authorization: Bearer $BOB_TOKEN" \
  -d '{"firstname":"Foo", "lastname":"Bar"}' -H "Content-Type: application/json" \
  -X POST http://example-app/people
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 18 Jul 2023 15:28:20 GMT
content-length: 51
x-envoy-upstream-service-time: 11
server: envoy

{"id":"498081","firstname":"Foo","lastname":"Bar"}

Creating People: Conflict

Our policy also blocks users from creating users with the same name, test that functionality with this request:

curl -i -H "Authorization: Bearer $BOB_TOKEN" \
  -d '{"firstname":"Bob", "lastname":"Bar"}' -H "Content-Type: application/json" \
  -X POST http://example-app/people
HTTP/1.1 403 Forbidden
date: Tue, 18 Jul 2023 15:31:48 GMT
server: envoy
content-length: 0

Cleanup

Exit the in-cluster shell by typing exit.

Delete the cluster by running:

kind delete cluster --name eopa-envoy
Deleting cluster "eopa-envoy" ...
Deleted nodes: ["eopa-envoy-control-plane"]