Kyverno – Kyverno Blog

Blog: Kyverno Reports Server - The ultimate solution to scale reporting

Wed, 29 May 2024 00:00:00 +0000

Introduction

Policy Reports are used by Kyverno to store the result of policies and cluster policies that match a resource. Kyverno generates reports during admission request as well as periodically as background scans. They are very helpful in auditing the current state of policy compliance in a cluster.

Kyverno also creates its own custom ephemeral reports which are later aggregated to create the final policy reports or cluster policy reports. Policy reports and ephemeral reports are stored in etcd as Custom Resources.

This setup works fine in most cases, but in larger-scale environments the limits of the Kubernetes API server could be reached. During periods of especially heavy reporting, the volume of data being written to and read from etcd can put the API server under severe load which can lead to degraded performance. Additionally, etcd has a maximum capacity limit and therefore has a limited number of resources it may store. This limit can be reached in large clusters with many report producers.

Today, we are excited to announce reports-server, a new Kyverno project which aims to improve scalability of reporting in large clusters giving you all the benefits of visibility without compromising on performance.

In this blog post we will introduce reports server, discuss its architecture, and provide steps on how to get started.

Architecture

Reports server solves the problem of scalability of policy reports by storing policy reports outside of etcd in a relational database. This has the following advantages:

Alleviation of the etcd + API server load and capacity limitations.
Common report consumer workflows can be more efficient.
With reports stored in a relational database, report consumers could instead query the underlying database directly, using more robust query syntax.

Reports server achieves this by using Kubernetes API aggregation layer, where it creates an API service that takes all the requests from the local API server and processes them. Reports server has access to a relational database which it uses to store and query for policy reports and ephemeral reports.

A high-level overview of the architecture is shown below.

Performance

The reports server stores policy and ephemeral reports, offloading them from etcd and minimizing its storage footprint. In the following tables, we show the database size of etcd of increased workloads with and without reports server. In this test, we installed Kyverno policies to audit the Kubernetes Pod Security Standards using 17 policies. Subsequently, we created workloads (Pods, Deployments, ReplicaSets) that match the installed Kyverno policies and scheduled them on false kwok nodes to measure total size of policy reports in etcd. kwok is a toolkit that enables setting up a cluster of thousands of Nodes in seconds. DB_SIZE in the output of etcdctl endpoint status -w table was used to calculate etcd DB size. For more details on these tests, refer to the testing documentation for the report controller. The version of Kyverno used in this testing was 1.12.3.

Without Reports Server

Below is the count of reports in etcd without the reports server. When the reports server is not installed, apiserver_storage_objects reports that there are 10000+ policy reports in etcd:

 1$ kubectl get --raw=/metrics | grep apiserver_storage_objects | awk '$2>100' |sort -g -k 2
 2# HELP apiserver_storage_objects [STABLE] Number of stored objects at the time of last check split by kind.
 3# TYPE apiserver_storage_objects gauge
 4apiserver_storage_objects{resource="nodes"} 108
 5apiserver_storage_objects{resource="leases.coordination.k8s.io"} 123
 6apiserver_storage_objects{resource="deployments.apps"} 857
 7apiserver_storage_objects{resource="replicasets.apps"} 859
 8apiserver_storage_objects{resource="pods"} 8540
 9apiserver_storage_objects{resource="policyreports.wgpolicyk8s.io"} 10268
10
11$ kubectl get polr -A | wc -l
1210269

apiserver_storage_objects metrics show that there are 10000+ policy reports stored in etcd along with other resources.

Total size of etcd:

Number of Policy Reports	Number of Pods	Total etcd Size
179	139	46 MB
1199	1139	71 MB
2219	2139	100 MB
4259	4139	149 MB
6299	6139	167 MB
8339	8139	220 MB
10379	10139	255 MB

With Reports Server

Here is the count of objects in etcd with the reports server, when 10000+ policy reports are present in the cluster. When the reports server is installed, apiserver_storage_objects does not find any policy reports in etcd and is therefore not reported. When we query for policy reports using kubectl, we see that there are 10000+ policy reports in the cluster:

 1$ kubectl get --raw=/metrics | grep apiserver_storage_objects | awk '$2>100' |sort -g -k 2
 2# HELP apiserver_storage_objects [STABLE] Number of stored objects at the time of last check split by kind.
 3# TYPE apiserver_storage_objects gauge
 4apiserver_storage_objects{resource="nodes"} 108
 5apiserver_storage_objects{resource="leases.coordination.k8s.io"} 123
 6apiserver_storage_objects{resource="deployments.apps"} 855
 7apiserver_storage_objects{resource="replicasets.apps"} 857
 8apiserver_storage_objects{resource="pods"} 8540
 9
10$ kubectl get polr -A | wc -l
1110249

apiserver_storage_objects metric does not find policy reports stored in etcd.

Total size of etcd:

Number of Policy Reports	Number of Pods	Total etcd Size
185	141	38 MB
1205	1141	38 MB
2225	2141	41 MB
4265	4141	55 MB
6305	6141	58 MB
8345	8141	67 MB
10385	10141	76 MB

As shown in the benchmark, the size of etcd grows as the number of resources in the cluster grows, but the growth is slower when reports server is installed. As reports server stores policy reports in a separate database, they don’t take up any space in etcd. At 10,000 reports, the storage size of etcd is 70.1% smaller compared to when reports server is installed.

Getting Started

To get started using reports server, install the service in your cluster. After a period of time, all the reports-related requests will be redirected from etcd to the reports server database. Reports server has multiple methods for installation including a basic YAML manifest and Helm chart. Detailed instructions for all installation methods can be found in the installation guide.

To install reports server using the YAML manifest, run the following commands: Create a namespace for reports server:

1kubectl create ns reports-server

Apply the reports server manifest:

1kubectl apply -f https://raw.githubusercontent.com/kyverno/reports-server/main/config/install.yaml

The manifest will install the following components:

A deployment and service for the reports server
A Postgres instance
An API service to redirect requests to reports server

Reports server comes with a PostgreSQL database, but you may opt for finer control of the database configuration by bringing your own database. See the database configuration guide for more details.

Migration

If you already have the PolicyReport CRD installed in your cluster, you will have an existing API service managed by kube-aggregator that sends requests to the Kubernetes API server. You will have to update the existing API service to send request to reports server. See the migration guide for more details.

Conclusion

In this short blog post, we demonstrated how reports server can be used to store policy reports.

Reports server is a new project from Kyverno that can be helpful for users with large scale reporting needs. This project is maintained by Kyverno and will have future updates and features based on feedback.

🔗 Check out the project on GitHub: https://github.com/kyverno/reports-server

Blog: Kyverno 1.12 Released

Fri, 26 Apr 2024 00:00:00 +0000

The Kyverno team is thrilled to announce the launch of version 1.12 hot on the heels of KubeCon EU 2024. Our interactions with Kyverno users during the conference have been incredibly insightful, with many expressing eager anticipation for the features introduced in this latest release. With over 330 pull requests merged, and 32 new contributors onboarded, we extend our heartfelt thanks to the Kyverno community for its invaluable contributions. Join us as we delve into the exciting new features of the 1.12 release.

Key New Features in Kyverno 1.12

This release introduces support for alternative storage for policy reports, brings Kyverno JSON to the CLI and includes various other enhancements in policy exception handling, ValidatingAdmissionPolicy and pod security sub-rule. Significant performance improvements have been made, ensuring resilience for large-scale systems.

Reports Server - Alternative Reports Storage

Policy reports are vital for tools like Kyverno, yet in large clusters, etcd’s storage limitations can hinder efficient policy management. Reports Server is designed to tackle these challenges head-on and streamline policy reporting within Kubernetes environments. Reports server provides a scalable solution for storing policy reports and cluster policy reports. It moves reports out of etcd and stores them in a PostgreSQL database instance.

Reports Server offers a straightforward solution. It leverages the Extension API server with a database to seamlessly integrate with Kubernetes, providing a dedicated space for storing and retrieving policy reports.

Advantages of Reports Server:

Scalability: Overcomes etcd’s storage constraints, allowing for seamless scalability within Kubernetes clusters.
Performance: By storing policy reports in a dedicated database, Reports Server improves cluster performance, particularly during heavy operations.
Simplified Querying: Offers enhanced querying capabilities compared to etcd, simplifying access to policy reports.

Explore Reports Server and contribute to its development on GitHub: https://github.com/kyverno/reports-server.

Global Context Entry

Kyverno’s Global Context feature enhances policy enforcement within Kubernetes environments by enabling the caching of resources or external API calls for efficient reference across policies. Let’s delve into how this functionality enhances flexibility and performance in policy management.

The cornerstone of Global Context is the Global Context Entry, a custom resource type introduced by Kyverno. With Global Context Entry, users can declare resource or API call data globally, independent of any specific policies. These entries can then be effortlessly referenced within policies as needed.

Types of Global Context Entries

Kubernetes Resource

Users can reference any Kubernetes resource, optionally within a specific namespace, to populate the cache with its latest state. This is particularly useful for scenarios where the latest state of resources needs to be accessed across multiple policies.

Example:

 1apiVersion: kyverno.io/v2alpha1
 2kind: GlobalContextEntry
 3metadata:
 4 name: deployments
 5spec:
 6 kubernetesResource:
 7 group: apps
 8 version: v1
 9 resource: deployments
10 namespace: test-globalcontext

Notes: When dealing with cluster-scoped resources, leave the namespace field empty.

API Call

Users can define an external API call, specifying the URL, CA bundle for certificate verification, and refresh interval to ensure the cached data remains up-to-date. This empowers policies to dynamically fetch data from external sources and utilize it for enforcement.

Example:

1apiVersion: kyverno.io/v2alpha1
2kind: GlobalContextEntry
3metadata:
4 name: deployments
5spec:
6 apiCall:
7 urlPath: "/apis/apps/v1/namespaces/test-globalcontext/deployments"
8 refreshInterval: 10s

Notes: It’s important to be mindful of potential data staleness with API calls, as the cache is updated periodically based on the defined refresh interval.

Integration with Policies

Global Context Entries seamlessly integrate into policies, allowing users to harness the cached data within their enforcement logic. By referencing Global Context Entries within policy contexts, users can leverage pre-fetched data without redundancy.

Example:

1context:
2 - name: deploymentCount
3 globalReference:
4 name: deployments

Note: Depending on whether the data is sourced from Kubernetes resources or API calls, the JMESPath expression used to manipulate the data may vary. For Kubernetes resources, utilize length(@) to obtain the length of the array, while for API calls returning a list, use items | length(@).

In conclusion, Global Context in Kyverno empowers users to streamline policy management by centralizing and reusing data across policies, thereby enhancing efficiency and maintainability in Kubernetes environments.

Kyverno JSON In CLI

Kyverno JSON was introduced around the same time as Kyverno 1.11.0, expanding Kyverno’s capabilities beyond Kubernetes. With the latest 1.12 release, Kyverno JSON is now supported by the Kyverno CLI for validating any JSON payload.

The following illustrates how to use the new checks stanza in a Kyverno CLI test:

 1# ...
 2checks:
 3- match:
 4 resource:
 5 kind: Namespace
 6 metadata:
 7 name: hello-world-namespace
 8 policy:
 9 kind: ClusterPolicy
10 metadata:
11 name: sync-secret
12 rule:
13 name: sync-my-secret
14 assert:
15 status: pass

Policy Exception Enhancements

PolicyExceptions now support a conditions{} block (optional) that uses common expressions similar to those found in preconditions and deny rules to query the contents of the selected resources in order to refine the selection process.

The following is a PolicyException that checks if the Pod/Deployment has a specific label:

 1apiVersion: kyverno.io/v2beta1
 2kind: PolicyException
 3metadata:
 4 name: container-exception
 5spec:
 6 exceptions:
 7 - policyName: max-containers
 8 ruleNames:
 9 - max-two-containers
10 - autogen-max-two-containers
11 match:
12 any:
13 - resources:
14 kinds:
15 - Pod
16 - Deployment
17 conditions:
18 any:
19 - key: "{{ request.object.metadata.labels.color || '' }}"
20 operator: Equals
21 value: blue

In addition, PolicyException can be used to define an exemption for the Pod through the podSecurity{} block. It can be used to define controls that are exempted from the policy.

The following PolicyException exempts the containers running either the nginx or redis image from following the Capabilities control.

 1apiVersion: kyverno.io/v2beta1
 2kind: PolicyException
 3metadata:
 4 name: pod-security-exception
 5 namespace: policy-exception-ns
 6spec:
 7 exceptions:
 8 - policyName: psa
 9 ruleNames:
10 - restricted
11 match:
12 any:
13 - resources:
14 namespaces:
15 - delta
16 podSecurity:
17 - controlName: Capabilities
18 images:
19 - nginx*
20 - redis*

Furthermore, Kyverno CLI supports applying PolicyExceptions alongside policies to resources by using the -e or --exceptions flag to pass the Policy Exception manifest.

1kyverno apply /path/to/policy.yaml --resource /path/to/resource.yaml --exception /path/to/exception.yaml

Resources that match the given exception will get a skip as a result.

ValidatingAdmissionPolicy in CLI and Reports

In 1.12, Kyverno CLI now supports applying the ValidatingAdmissionPolicyBinding along with the policy definition to the resources. Furthermore, the reports controller generates reports as a result of applying the ValidatingAdmissionPolicy along with its binding to resources.

Here is a report snippet that is generated for ValidatingAdmissionPolicy with its binding:

 1 apiVersion: wgpolicyk8s.io/v1alpha2
 2 kind: PolicyReport
 3 metadata:
 4 labels:
 5 app.kubernetes.io/managed-by: kyverno
 6 namespace: staging-ns
 7 ownerReferences:
 8 - apiVersion: apps/v1
 9 kind: Deployment
10 name: deployment-3
11 results:
12 - message: 'failed expression: object.spec.replicas <= 5'
13 policy: check-deployment-replicas
14 properties:
15 binding: check-deployment-replicas-binding
16 result: fail
17 source: ValidatingAdmissionPolicy
18 scope:
19 apiVersion: apps/v1
20 kind: Deployment
21 name: deployment-3
22 namespace: staging-ns
23 summary:
24 error: 0
25 fail: 1
26 pass: 0
27 skip: 0
28 warn: 0

Fine-grained PodSecurity Validations

The validate.podSecurity sub-rule now has the ability to further classify exclusions by excluding specific fields as well as their values. This allows you to both use the easy style of policy language but get down to the lowest level of the funnel. For example, this is how you’ll be able to enforce the entire baseline profile of the Pod Security Standards but only exclude Istio’s and Linkerd’s images from specifically the initContainers list.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: pod-security-standards
 5spec:
 6 background: true
 7 validationFailureAction: Enforce
 8 rules:
 9 - name: baseline-service-mesh
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 validate:
16 podSecurity:
17 level: baseline
18 version: latest
19 exclude:
20 - controlName: Capabilities
21 images:
22 - "*/istio/proxyv2*"
23 - "*/linkerd/proxy-init*"
24 restrictedField: spec.initContainers[*].securityContext.capabilities.add
25 values:
26 - NET_ADMIN
27 - NET_RAW

Increased Performance

Kyverno 1.12 has undergone significant performance enhancements, making it even more suitable for large-scale deployments. Let’s dive into the challenges faced, the optimization journey, and the impressive results achieved.

Facing the Performance Bottlenecks

While Kyverno functioned well under normal conditions, its performance degraded under high loads, impacting large clusters. To address this, the Kyverno team, in collaboration with customers and the community, embarked on a mission to optimize performance.

Unveiling the Culprits

Load testing with K6 and profiling with Golang’s pprof tool revealed the primary culprits: inefficient context storage, repetitive JSON operations, and excessive goroutine creation during event processing.

Strategic Optimizations

Armed with these insights, the team implemented targeted optimizations:

Context Storage: Replacing JSON structures with in-memory Golang maps significantly reduced overhead.
JSON Handling: Switching to the lightweight jsoniter library boosted JSON processing efficiency.
Event Processing: A custom Watcher function optimized event generation and minimized goroutine overhead.
Policy Matching: Restructuring the matching logic prioritized simpler comparisons, leading to faster processing.
Webhook Configuration: Fine-grained configuration minimized unnecessary Kyverno invocations.

The Performance Leap

These optimizations yielded remarkable results:

8x improvement in average and p95 latency.
Stable performance even with high iteration counts.

Continuous Improvement through Automation

To ensure the longevity of these improvements, automated K6 load testing is now integrated into the CI/CD pipeline. This allows for early detection of potential regressions and facilitates continuous performance optimization.

Kyverno: Ready for Large-Scale Deployments

These advancements solidify Kyverno’s position as a robust and scalable policy engine for Kubernetes, capable of handling demanding workloads with ease. The collaborative effort behind these improvements exemplifies the project’s commitment to delivering a secure and efficient policy management solution for the Kubernetes community.

Policy Reporter UI V2

Policy Reporter UI v2 started its alpha phase with an updated techstack and complete restructuring of dashboards, details pages and the plugin system.

Content Restructuring

Policy Reporter UI v2 offers two types of views, resource-based and policy-based. The resource-based dashboards replace the old report views in Policy Reporter UI v1. The new focus of resource-based dashboards makes them much more scalable in larger environments and improves overall performance. Policy-based dashboards were a Kyverno-only feature in the old UI and only available with an additional Kyverno plugin. The revised views are available without additional requirements and are available for all provided sources.

New Plugin System

The plugin system was introduced to provide additional engine-specific functionality, but the old implementation required direct changes to the UI code, which made adding custom plugins very difficult.

The new plugin system requires an external service with a predefined (sub)set of REST APIs, where each plugin points to a source and provides additional information about this source. This additional information is automatically embedded in the existing views. These workflows make it possible to develop and use custom plugins without having to work on the actual user interface.

Custom Boards

One disadvantage of predefined views is the lack of customization options. Although they provide an overview of the entire cluster, it is often the case that teams within a company are only interested in a subset of namespaces and engine reports.

Custom Boards are a way of extending the UI with customized, application-specific views that provide only the required subset of information. The various configuration options enable the selection of namespaces based on a list or selector, the selection of displayed sources and the activation or deactivation of cluster-specific resource results.

Additional Features

OpenIDConnect and OAuth2 authentication support
Updated frontend tech stack: Bun, Vue 3.x, Nuxt 3.x, Vuetify 3.x, Vite 5.x
Color Blind Themes
General performance improvements

Kyverno Playground

The Kyverno playground with the support of ValidatingAdmissionPolicyBinding is now available for the new 1.12 release. You can check it out today via https://playground.kyverno.io/next/#/.

Closing

We have achieved another significant milestone, thanks to the support and contributions from our incredible community. Alongside the highlighted new features, this release includes critical fixes and other important changes. For a comprehensive overview, please refer to the full release notes on GitHub.

You can find us in the Kyverno channel on Kubernetes Slack, come and join one of our community meetings to discuss your contribution issues, or just catch us on Twitter. And if you’re already a Kyverno adopter, sign up to be an official adopter by updating the Adopters form here.

Blog: Generating Kubernetes ValidatingAdmissionPolicies from Kyverno Policies

Mon, 26 Feb 2024 00:00:00 +0000

In the previous blog post, we discussed writing Common Expression Language (CEL) expressions in Kyverno policies for resource validation. CEL was first introduced to Kubernetes for the Validation rules for CustomResourceDefinitions, and then it was used by Kubernetes ValidatingAdmissionPolicies in 1.26.

ValidatingAdmissionPolicies offer a declarative, in-process alternative to validating admission webhooks.

ValidatingAdmissionPolicies use the Common Expression Language (CEL) to declare the validation rules of a policy. Validation admission policies are highly configurable, enabling policy authors to define policies that can be parameterized and scoped to resources as needed by cluster administrators.

This post will show you how to generate Kubernetes ValidatingAdmissionPolicies and their bindings from Kyverno policies.

Prerequisite

Generating Kubernetes ValidatingAdmissionPolicies require the following:

A cluster with Kubernetes 1.26 or higher.
Enable the ValidatingAdmissionPolicy feature gate.
Enable the admissionregistration.k8s.io/v1beta1 API for v1.28 and v1.29. OR Enable the admissionregistration.k8s.io/v1alpha1 API for v1.26 and v1.27.
Set the --generateValidatingAdmissionPolicy flag in the Kyverno admission controller.
Grant the admission controller service account the required permissions to generate ValidatingAdmissionPolicies and their bindings.

In this post, we will use the beta version of Kubernetes 1.29.

Installation & Setup

Create a local cluster

 1kind create cluster --image "kindest/node:v1.28.0" --config - <<EOF
 2kind: Cluster
 3apiVersion: kind.x-k8s.io/v1alpha4
 4featureGates:
 5 ValidatingAdmissionPolicy: true
 6runtimeConfig:
 7 admissionregistration.k8s.io/v1beta1: true
 8 admissionregistration.k8s.io/v1alpha1: true
 9nodes:
10 - role: control-plane
11 - role: worker
12EOF

Add the Kyverno Helm repository.

1helm repo add kyverno https://kyverno.github.io/kyverno/
2helm repo update

Create a new file that overrides the values in the chart.

 1cat << EOF > new-values.yaml
 2features:
 3 generateValidatingAdmissionPolicy:
 4 enabled: true
 5
 6admissionController:
 7 rbac:
 8 clusterRole:
 9 extraResources:
10 - apiGroups:
11 - admissionregistration.k8s.io
12 resources:
13 - validatingadmissionpolicies
14 - validatingadmissionpolicybindings
15 verbs:
16 - create
17 - update
18 - delete
19 - list
20EOF

Deploy Kyverno

1helm install kyverno kyverno/kyverno -n kyverno --create-namespace --version v3.1.4 --values new-values.yaml

We are now ready to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies.

Generating Kubernetes ValidatingAdmissionPolicies

In this section, we will create a Kyverno policy that ensures no hostPath volumes are in use for Deployments, and then we will have a look at the generated ValidatingAdmissionPolicy and its binding. Finally, we will create a Deployment that violates the policy.

Let’s start with creating the Kyverno policy.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: disallow-host-path
 6spec:
 7 validationFailureAction: Enforce
 8 background: false
 9 rules:
10 - name: host-path
11 match:
12 any:
13 - resources:
14 kinds:
15 - Deployment
16 validate:
17 cel:
18 expressions:
19 - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
20 message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
21EOF

You can check whether a ValidatingAdmissionPolicy is generated or not from the Kyverno policy status.

 1$ kubectl get cpol disallow-host-path -o jsonpath='{.status}'
 2
 3{
 4 "autogen":{
 5
 6 },
 7 "conditions":[
 8 {
 9 "lastTransitionTime":"2023-09-12T11:42:13Z",
10 "message":"Ready",
11 "reason":"Succeeded",
12 "status":"True",
13 "type":"Ready"
14 }
15 ],
16 "ready":true,
17 "rulecount":{
18 "generate":0,
19 "mutate":0,
20 "validate":1,
21 "verifyimages":0
22 },
23 "validatingadmissionpolicy":{
24 "generated":true,
25 "message":""
26 }
27}

Let’s try getting the ValidatingAdmissionPolicy and its binding.

1$ kubectl get validatingadmissionpolicy
2NAME VALIDATIONS PARAMKIND AGE
3disallow-host-path 1 <unset> 8m12s
4
5$ kubectl get validatingadmissionpolicybindings
6NAME POLICYNAME PARAMREF AGE
7disallow-host-path-binding disallow-host-path <unset> 8m30s

You may notice that the ValidatingAdmissionPolicy and the ValidatingAdmissionPolicyBinding share the same name as the Kyverno policy they originate from, with the binding having a “-binding” suffix.

Let’s have a look at the ValidatingAdmissionPolicy and its binding in detail.

 1$ kubectl get validatingadmissionpolicy disallow-host-path -o yaml
 2apiVersion: admissionregistration.k8s.io/v1beta1
 3kind: ValidatingAdmissionPolicy
 4metadata:
 5 creationTimestamp: "2023-09-12T11:42:13Z"
 6 generation: 1
 7 labels:
 8 app.kubernetes.io/managed-by: kyverno
 9 name: disallow-host-path
10 ownerReferences:
11 - apiVersion: kyverno.io/v1
12 kind: ClusterPolicy
13 name: disallow-host-path
14 uid: e540d96b-c683-4380-a84f-13411384241a
15 resourceVersion: "11294"
16 uid: 9f3e0161-d010-4a6f-bd28-bf9c87151795
17spec:
18 failurePolicy: Fail
19 matchConstraints:
20 matchPolicy: Equivalent
21 namespaceSelector: {}
22 objectSelector: {}
23 resourceRules:
24 - apiGroups:
25 - apps
26 apiVersions:
27 - v1
28 operations:
29 - CREATE
30 - UPDATE
31 resources:
32 - deployments
33 scope: '*'
34 validations:
35 - expression: '!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume,
36 !has(volume.hostPath))'
37 message: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath
38 must be unset.
39 variables: null
40status:
41 observedGeneration: 1
42 typeChecking: {}

 1$ kubectl get validatingadmissionpolicybindings disallow-host-path-binding -o yaml
 2apiVersion: admissionregistration.k8s.io/v1beta1
 3kind: ValidatingAdmissionPolicyBinding
 4metadata:
 5 creationTimestamp: "2023-09-12T11:42:13Z"
 6 generation: 1
 7 labels:
 8 app.kubernetes.io/managed-by: kyverno
 9 name: disallow-host-path-binding
10 ownerReferences:
11 - apiVersion: kyverno.io/v1
12 kind: ClusterPolicy
13 name: disallow-host-path
14 uid: e540d96b-c683-4380-a84f-13411384241a
15 resourceVersion: "11292"
16 uid: 2fec35c3-8a8c-42a7-8a02-a75e8882a01e
17spec:
18 policyName: disallow-host-path
19 validationActions:
20 - Deny

Now, let’s try deploying an app that uses a hostPath:

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: nginx
 6spec:
 7 replicas: 2
 8 selector:
 9 matchLabels:
10 app: nginx
11 template:
12 metadata:
13 labels:
14 app: nginx
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19 volumeMounts:
20 - name: udev
21 mountPath: /data
22 volumes:
23 - name: udev
24 hostPath:
25 path: /etc/udev
26EOF

As expected, the deployment creation is rejected by the API server and not by the Kyverno admission controller.

1The deployments "nginx" is invalid: ValidatingAdmissionPolicy 'disallow-host-path' with binding 'disallow-host-path-binding' denied request: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset.

If either the ValidatingAdmissionPolicy or the binding is deleted/updated for some reason, the controller is responsible for reverting it.

Let’s try deleting the ValidatingAdmissionPolicy.

1$ kubectl delete validatingadmissionpolicy disallow-host-path
2validatingadmissionpolicy.admissionregistration.k8s.io "disallow-host-path" deleted
3
4$ kubectl get validatingadmissionpolicy
5NAME VALIDATIONS PARAMKIND AGE
6disallow-host-path 1 <unset> 11s

In addition, you can update the Kyverno policy, and the controller will re-generate the ValidatingAdmissionPolicy accordingly. For example, you can change the Kyverno policy to match statefulsets too.

patch.yaml:

 1spec:
 2 rules:
 3 - name: host-path
 4 match:
 5 any:
 6 - resources:
 7 kinds:
 8 - Deployment
 9 - StatefulSet
10 validate:
11 cel:
12 expressions:
13 - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
14 message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."

1kubectl patch cpol disallow-host-path --type merge --patch-file patch.yaml

The ValidatingAdmissionPolicy will be updated to match StatefulSets too.

 1apiVersion: admissionregistration.k8s.io/v1beta1
 2kind: ValidatingAdmissionPolicy
 3metadata:
 4 creationTimestamp: "2023-09-12T12:54:48Z"
 5 generation: 2
 6 labels:
 7 app.kubernetes.io/managed-by: kyverno
 8 name: disallow-host-path
 9 ownerReferences:
10 - apiVersion: kyverno.io/v1
11 kind: ClusterPolicy
12 name: disallow-host-path
13 uid: e540d96b-c683-4380-a84f-13411384241a
14 resourceVersion: "29208"
15 uid: 9325e2b7-9131-4ff4-9e56-244129cb625e
16spec:
17 failurePolicy: Fail
18 matchConstraints:
19 matchPolicy: Equivalent
20 namespaceSelector: {}
21 objectSelector: {}
22 resourceRules:
23 - apiGroups:
24 - apps
25 apiVersions:
26 - v1
27 operations:
28 - CREATE
29 - UPDATE
30 resources:
31 - deployments
32 - statefulsets
33 scope: '*'
34 validations:
35 - expression: '!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume,
36 !has(volume.hostPath))'
37 message: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath
38 must be unset.
39 variables: null
40status:
41 observedGeneration: 2
42 typeChecking: {}

Conclusion

In this blog, we discussed how to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies. You can use CEL expressions in Kyverno policies to validate resources through either the Kyverno engine or the API server. In the next blog, we will discuss how to generate BackgroundScan reports for ValidatingAdmissionPolicies.

Blog: Assigning Node Metadata to Pods

Mon, 19 Feb 2024 00:00:00 +0000

If you’re running Kubernetes in production, especially in a public cloud, where a single cluster may span multiple availability zones, chances are you’re configuring workloads with some awareness of your topology. Kubernetes has a few mechanisms to support zone awareness, but one common use case is how to propagate certain Node metadata, such as labels or annotations, to Pods to assist with this awareness. In this blog, we’ll go into specifics of how Pod scheduling really works and share some tips for how Kyverno can mutate Pods to add Node metadata like labels. Even if you’re not a Kyverno user, you’ll most likely learn something you didn’t know about Kubernetes.

When deploying Pod controllers with multiple replicas to a single cluster which has Nodes in different geographical areas, high-level organizations like “zones” or “availability zones” are important failure domains. You typically want to spread those replicas out amongst those zones so failure of any one does not take out the application. Kubernetes already has several methods for working with topologies like zones including spread constraints with some very recent enhancements in beta. The label topology.kubernetes.io/zone is a common Kubernetes label to denote zone information for Nodes. This can be set automatically when using cloud provider integration or manually, for example when running on-premises and building your own concept of zones.

Having the Node’s zone information available to the Pods running on it can be important, for example so Pods only respond to requests from that same zone and avoid making cross-region calls which can result in potentially expensive egress charges. And zone information aside, there can be a whole host of other information about the Node that a Pod would like to receive such as governance labels you apply, custom annotations, etc. The challenge is how to get this information because you don’t know on what Node a Pod will be scheduled.

Mutation of Pods is no great feat and Kyverno was one of the first admission controllers to offer it. Making calls to the Kubernetes API server is also well known and Kyverno does that too. But, as those who have tried this before and failed, you don’t know what to ask for when you make that call. This is because the Kubernetes scheduler is the one to make the decision on which Node will receive the Pod. And understanding how and where the scheduler fits into the Pod lifecycle is both nuanced and not well documented, even in the official docs. So, without further ado, we’d like to present the complete(r) guide to how Pod scheduling actually works because this will be critical in understanding both why this is problematic and how you can do it with Kyverno. Pay attention to the step numbers on the sequence diagram as we’ll reference these in the further discussion.

Pod Scheduling in Kubernetes

Thanks to the magic of Mermaid for enabling a diagram like this written out of pure Markdown!

Note

This diagram shows connections to etcd from multiple components for ease of explanation when referring to persisting of fetching something. In actuality, etcd only has a single client: the API server.

sequenceDiagram
autonumber
User->>API Server: Create Pod "mypod"
API Server->>etcd: Persist Pod
Note over API Server,etcd: CREATE operation for Pod
Note right of etcd: Status is "Pending"
activate Scheduler
Scheduler-->>etcd: See Pod needs scheduling
Scheduler-->>Scheduler: Determine optimal node
Scheduler->>API Server: Bind "mypod" to node "mynode01"
Note over Scheduler,API Server: CREATE operation for Pod/binding
deactivate Scheduler
API Server->>etcd: Set nodeName in "mypod" to "mynode01"
activate Kubelet on mynode01
Kubelet on mynode01-->>etcd: See new Pod to run
Kubelet on mynode01->>Kubelet on mynode01: Run "mypod"
deactivate Kubelet on mynode01
Note right of Kubelet on mynode01: Status is "Running"

Ok, with this sequence diagram above, let’s break this down. Refer to the step numbers for the matching description.

A user sends a request to create a Pod called “mypod”. The assumption here is they are not statically defining which Node should be used via spec.nodeName.
The API server receives this request and performs a CREATE operation on a Pod. This gets persisted to etcd. Importantly, when this happens, the Node is NOT yet known since the scheduler hasn’t kicked in. When it creates this Pod in etcd, the status is listed as “Pending”.
Now the scheduler enters the picture. It sees there is a Pod which has been persisted but hasn’t been scheduled so it leaps into action.
Using its (very sophisticated) internal logic, it determines the optimal Node. In this case, it selected “mynode01”.
Once the target Node has been selected, it performs the binding. How it does this is by performing another CREATE but this time for a subresource called Pod/binding. We’ll talk more about this soon. This binding is nothing more than an instruction to the API server which says “bind mypod to Node mynode01.”
The API server receives this Pod/binding and reacts by setting the spec.nodeName field in mypod (which was persisted in step 2) to mynode01. Very importantly, when it does this it DOES NOT result in any sort of UPDATE operation. It’s more or less “transparent”.
Kubelet running on mynode01 is looking solely for Pods which have been assigned to it. It sees a new Pod pop up on its radar.
Kubelet then runs the Pod by pulling the container images and starting everything up. Only now is when the status for that Pod enters the “Running” phase.

Note that in the above description and diagram, the CREATE and UPDATE operations refer to the context where admission controllers, like Kyverno, participate and not the actual verbs from an RBAC perspective.

Pod Bindings

If you refer back to step 5 you’ll notice the binding event. In this step, the scheduler informs the API server which Node should run the Pod. It does this by what are called subresources to a Pod. A subresource is essentially a way to “carve” out sections of a parent resource in order to control RBAC. For example /status and /exec are well-known other subresources to Pods. These control specific areas of a Pod, for example the .status object and Pod exec operations, respectively. A /binding is a similar albeit lesser-known subresource only sent by the scheduler.

Here’s an example binding taken from a K3d (K3s) cluster. Notice here that the Pod definition is absent. The meat of this subresource is under object.target which is where the scheduler defines the kind of resource (Node in this case) and the name of that resource (k3d-kyv1111-server-0 as shown here).

 1uid: 21fb3d8e-b9c9-42fe-a987-d4374e74a084
 2kind:
 3 group: ""
 4 version: v1
 5 kind: Binding
 6resource:
 7 group: ""
 8 version: v1
 9 resource: pods
10subResource: binding
11requestKind:
12 group: ""
13 version: v1
14 kind: Binding
15requestResource:
16 group: ""
17 version: v1
18 resource: pods
19requestSubResource: binding
20name: busybox
21namespace: default
22operation: CREATE
23userInfo:
24 username: system:kube-scheduler
25 groups:
26 - system:authenticated
27roles:
28 - kube-system:extension-apiserver-authentication-reader
29 - kube-system:system::leader-locking-kube-scheduler
30clusterRoles:
31 - system:basic-user
32 - system:discovery
33 - system:kube-scheduler
34 - system:public-info-viewer
35 - system:volume-scheduler
36object:
37 apiVersion: v1
38 kind: Binding
39 metadata:
40 creationTimestamp: null
41 managedFields:
42 - apiVersion: v1
43 fieldsType: FieldsV1
44 fieldsV1:
45 f:target: {}
46 manager: k3s
47 operation: Update
48 subresource: binding
49 time: "2024-02-18T14:36:37Z"
50 name: busybox
51 namespace: default
52 uid: fceccee4-4821-408a-b75b-44262392b93c
53 target:
54 kind: Node
55 name: k3d-kyv1111-server-0
56oldObject: null
57dryRun: false
58options:
59 apiVersion: meta.k8s.io/v1
60 kind: CreateOptions

Fortunately, these Pod/binding subresources can be sent to admission controllers like Kyverno. So now we have the Pod persisted, we can observe the binding which contains the Node name, then we can make a call to the API server to get information on that Node. Grand!

Not so fast. Before you click close on this blog, keep reading. It’s a little more complex.

Using Node Info in Pods

We know what we want and we know approximately how to get it now that bindings have been introduced. But how and where you use the Node information in a Pod matters when it comes to supplying it.

Typically, use cases which involve fetching Node information and presenting it to Pods require that the container(s) in the Pod know about it. Informing containers in Pods of Pod metadata is usually done using the downward API. In Kubernetes, the downward API is used to present information about the Pod and/or its surroundings to the containers within it. It can do this in two primary ways: environment variables and volumes. Using either, the containers now understand their environment without them having to go and figure it out on their own. This is a boon because it’s simpler for the app and it’s more secure. It’s also the only way, in many cases, to provide this sort of orientation to containers.

For example, you can use the downward API to tell a Pod’s containers about the name of the Pod in which they are running with an environment variable named POD_NAME.

1 env:
2 - name: POD_NAME
3 valueFrom:
4 fieldRef:
5 fieldPath: metadata.name

And you could do the same thing with a volume and its mount.

1 volumes:
2 - name: podname
3 downwardAPI:
4 items:
5 - path: podname
6 fieldRef:
7 fieldPath: metadata.name

1 volumeMounts:
2 - name: podname
3 mountPath: /etc/podinfo

In the case of the former, the containers will get an environment variable POD_NAME=mypod and in the latter will have a file available at /etc/podinfo/podname which contains the word mypod.

But here’s another important nuance which will be highlighted in the next section: which way you choose to consume this information can matter when fetching it from Nodes.

Once Kubelet launches containers, providing this information later does nothing and it won’t be available because the container has already started. When it comes to environment variables, these need to be defined before Kubelet begins its routine. Volumes are a bit less forgiving, especially for lengthier container pulls as it takes the Kubelet a little bit longer to establish all the connections. Keep this in mind because it matters whether or not you’ll be successful in mutating those Pods.

Mutations

Now comes the part you’ve been waiting for. We had to tee this section up in order for you to get the full picture. Let’s talk about finally performing the mutations. Make sure you’ve fully read and understood the previous sections or this will be confusing.

When assigning Node metadata to Pods in Kyverno, you have a couple different options. What exactly you want to use that Node information for in the Pod determines which method you’ll need.

Mutating Bindings

The first and arguably “best” option for slapping Node info on Pods is to mutate the Pod/binding subresource directly. If you refer back to the scheduling sequence diagram above, you’ll see that we mentioned these Pod/binding subresources are sent to admission controllers. Also recall from the example binding that this does NOT contain the full Pod representation. It’s really an entirely different thing. There is only one type of mutation that matters in this context you can perform on a binding and that is to write annotations.

Kubernetes has a “secret” function called setPodHostAndAnnotations() which will take any annotations on a binding and transfer them to the parent Pod resource. This only works for annotations which is why if you’ve tried to set labels you’ve found they just get dropped. If you are going to consume Node information via the downward API, either as an environment variable or volume, this is the recommended way on going about it since it’s the most reliable and the annotation is guaranteed to be written before the Pod is started.

The sample policy is available here and is shown below but updated with modifications to show assignment of the topology.kubernetes.io/zone annotation. This works since Kyverno 1.10.

 1apiVersion: kyverno.io/v2beta1
 2kind: ClusterPolicy
 3metadata:
 4 name: mutate-pod-binding
 5 annotations:
 6 pod-policies.kyverno.io/autogen-controllers: none
 7 policies.kyverno.io/title: Mutate Pod Binding
 8 policies.kyverno.io/category: Other
 9 policies.kyverno.io/subject: Pod
10 kyverno.io/kyverno-version: 1.10.0
11 policies.kyverno.io/minversion: 1.10.0
12 kyverno.io/kubernetes-version: "1.26"
13spec:
14 background: false
15 rules:
16 - name: project-foo
17 match:
18 any:
19 - resources:
20 kinds:
21 - Pod/binding
22 context:
23 - name: node
24 variable:
25 jmesPath: request.object.target.name
26 default: ''
27 - name: zone
28 apiCall:
29 urlPath: "/api/v1/nodes/{{node}}"
30 jmesPath: "metadata.labels.\"topology.kubernetes.io/zone\" || 'empty'"
31 mutate:
32 patchStrategicMerge:
33 metadata:
34 annotations:
35 topology.kubernetes.io/zone: "{{ zone }}"

Important!

Kyverno by default ignores requests for Pod/binding because these can create much noise and are generally not useful except in corner cases like this one. You will need to remove the two resource filters [Binding,*,*] and [Pod/binding,*,*] in Kyverno’s ConfigMap to be able to mutate Pod bindings. Because this will involve more processing by Kyverno, you really should try and scope the match down as far as possible. See the docs here for more.

Demo

Let’s demonstrate this in action. For this, we’ll use a simple K3d cluster with 3 nodes where one is control plane and the other two are worker nodes. Each of the two workers are in separate availability zones (abbreviated output of labels).

1$ kubectl get no --show-labels
2NAME STATUS ROLES AGE VERSION LABELS
3k3d-kyv11-server-0 Ready control-plane,master 3h27m v1.27.4+k3s1
4k3d-worker01-0 Ready <none> 13m v1.27.4+k3s1 topology.kubernetes.io/zone=us-east-2a
5k3d-worker02-0 Ready <none> 13m v1.27.4+k3s1 topology.kubernetes.io/zone=us-east-2b

Remove the resource filters as described previously and install the ClusterPolicy.

Let’s now create a Deployment of busybox with 4 replicas. We’ll use some nodeAffinity here to ensure all four Pods land on one of the two worker nodes.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4 name: busybox
 5 labels:
 6 app: busybox
 7spec:
 8 replicas: 4
 9 selector:
10 matchLabels:
11 app: busybox
12 template:
13 metadata:
14 labels:
15 app: busybox
16 spec:
17 automountServiceAccountToken: false
18 containers:
19 - image: busybox:latest
20 name: busybox
21 command:
22 - env
23 env:
24 - name: ZONE
25 valueFrom:
26 fieldRef:
27 fieldPath: metadata.annotations['topology.kubernetes.io/zone']
28 affinity:
29 nodeAffinity:
30 requiredDuringSchedulingIgnoredDuringExecution:
31 nodeSelectorTerms:
32 - matchExpressions:
33 - key: topology.kubernetes.io/zone
34 operator: Exists

Once this is created, we expect Kyverno to mutate the Pod/binding resource to add the topology.kubernetes.io/zone annotation to each Pod (not to the parent Deployment since we disabled rule auto-gen). We’re running the env program to simply write out all the environment variables. So we expect the Pods to be in a “Completed” state where we can inspect the logs and hopefully see the ZONE environment variable has been set with the value equaling the value of the topology.kubernetes.io/zone label on the parent Node.

Create the Deployment and let’s check the Pods. You can see two got scheduled on each worker.

1$ kubectl get po -o wide
2NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
3busybox-6b869bf945-5hcn6 0/1 Completed 1 (3s ago) 4s 10.42.2.6 k3d-worker02-0 <none> <none>
4busybox-6b869bf945-bcbxv 0/1 Completed 1 (3s ago) 4s 10.42.2.5 k3d-worker02-0 <none> <none>
5busybox-6b869bf945-nb8sc 0/1 Completed 0 4s 10.42.1.4 k3d-worker01-0 <none> <none>
6busybox-6b869bf945-zvg2m 0/1 Completed 0 4s 10.42.1.5 k3d-worker01-0 <none> <none>

Get the logs for one of the Pods on worker01.

 1$ kubectl logs busybox-6b869bf945-nb8sc
 2PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 3HOSTNAME=busybox-6b869bf945-nb8sc
 4ZONE=us-east-2a
 5KUBERNETES_SERVICE_PORT=443
 6KUBERNETES_SERVICE_PORT_HTTPS=443
 7KUBERNETES_PORT=tcp://10.43.0.1:443
 8KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
 9KUBERNETES_PORT_443_TCP_PROTO=tcp
10KUBERNETES_PORT_443_TCP_PORT=443
11KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
12KUBERNETES_SERVICE_HOST=10.43.0.1
13HOME=/root

And again for one on worker02.

 1$ kubectl logs busybox-6b869bf945-5hcn6
 2PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 3HOSTNAME=busybox-6b869bf945-5hcn6
 4ZONE=us-east-2b
 5KUBERNETES_PORT=tcp://10.43.0.1:443
 6KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
 7KUBERNETES_PORT_443_TCP_PROTO=tcp
 8KUBERNETES_PORT_443_TCP_PORT=443
 9KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
10KUBERNETES_SERVICE_HOST=10.43.0.1
11KUBERNETES_SERVICE_PORT=443
12KUBERNETES_SERVICE_PORT_HTTPS=443
13HOME=/root

As you can clearly see, our mutation worked and the Node’s label was fetched, written to an annotation on the Pod/binding subresource, was merged into the Pod’s annotations, and this was projected as an environment variable into the busybox containers.

Mutating Pods

The second method for reflecting Node information on Pods is to mutate the Pod directly. If, for example, you absolutely must write labels to the Pod then you cannot mutate the Pod/binding resource as mentioned in the previous section. You must mutate the Pod directly. The same goes for any other type of mutation to the Pod.

The kicker here is that this style of mutation, what’s known as a “mutate existing” rule, is an asynchronous process. Why is this important? Since it’s async there are no guarantees as to when this mutation will be performed. It could be within 50ms of receiving the binding and it could be in a second or more depending on load and a variety of other factors. And, chances are, that mutation could happen after the Pod has already started. Once the Pod has started, any environment variables or volumes have already been set and cannot be influenced. So then, this method should really only be used if you don’t care about consuming this Node information in the Pod’s containers directly. For example, if you just want the Pod to have labels from its parent Node for purposes of reporting or costing, then this method is fine.

An example on how to perform a mutation on an existing Pod in response to a binding is shown below. But first, you’ll probably need to grant the Kyverno background controller (the one responsible for all generate and “mutate existing” rules) some permissions it doesn’t have since we’re now changing existing Pods.

 1apiVersion: rbac.authorization.k8s.io/v1
 2kind: ClusterRole
 3metadata:
 4 labels:
 5 app.kubernetes.io/component: background-controller
 6 app.kubernetes.io/instance: kyverno
 7 app.kubernetes.io/part-of: kyverno
 8 name: kyverno:update-pods
 9rules:
10- apiGroups:
11 - ""
12 resources:
13 - pods
14 verbs:
15 - update

 1apiVersion: kyverno.io/v2beta1
 2kind: ClusterPolicy
 3metadata:
 4 name: add-node-labels-pod
 5 annotations:
 6 pod-policies.kyverno.io/autogen-controllers: none
 7spec:
 8 rules:
 9 - name: add-topology-labels
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod/binding
15 context:
16 - name: node
17 variable:
18 jmesPath: request.object.target.name
19 default: ''
20 - name: ZoneLabel
21 apiCall:
22 urlPath: "/api/v1/nodes/{{node}}"
23 jmesPath: "metadata.labels.\"topology.kubernetes.io/zone\" || 'empty'"
24 mutate:
25 targets:
26 - apiVersion: v1
27 kind: Pod
28 name: "{{ request.object.metadata.name }}"
29 namespace: "{{ request.object.metadata.namespace }}"
30 patchStrategicMerge:
31 metadata:
32 labels:
33 # https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone
34 topology.kubernetes.io/zone: "{{ ZoneLabel }}"

Closing

This was a lot of information but hopefully you learned something, even if you aren’t a Kyverno user. To sum this blog post up, if you want to consume Node info in a Pod at runtime via environment variables or volumes, you should be mutating the Pod/binding subresource. Only annotations may be directly added to a Pod/binding. For all other use cases you should be mutating the existing Pod. Both can also be combined if need be. Example policies for both are provided but before selecting one you should know what they do and the result you can expect.

Blog: Kyverno Chainsaw 0.1.4 - Awesome new features!

Thu, 15 Feb 2024 00:00:00 +0000

The latest release of Kyverno Chainsaw came out yesterday. Let’s look at the new features included in this release.

Resource diff in assertion failures
Resource templating support

Resource diff in assertion failures

This is a relatively straightforward one but it brings a lot of context to assertion failures.

Suppose an assertion fails, before v0.1.4 the output would have looked like this:

1 | 09:52:19 | deployment | step-1 | ASSERT | RUN | v1/Pod @ chainsaw-full-llama/*
2 | 09:52:49 | deployment | step-1 | ASSERT | ERROR | v1/Pod @ chainsaw-full-llama/*
3 === ERROR
4 v1/Pod/chainsaw-full-llama/example-5477b4ff8c-vrzs8 - metadata.annotations.foo: Invalid value: "null": Expected value: "bar"
5 | 09:52:49 | deployment | step-1 | TRY | DONE |

While the message contains a description of the failure (metadata.annotations.foo: Invalid value: "null": Expected value: "bar") and the offending resource (v1/Pod/chainsaw-full-llama/example-5477b4ff8c-vrzs8), it’s not easy to understand where the resource comes from.

The same error in v0.1.4 will be reported including a resource diff:

 1 | 09:55:50 | deployment | step-1 | ASSERT | RUN | v1/Pod @ chainsaw-rare-liger/*
 2 | 09:56:20 | deployment | step-1 | ASSERT | ERROR | v1/Pod @ chainsaw-rare-liger/*
 3 === ERROR
 4 ---------------------------------------------------
 5 v1/Pod/chainsaw-rare-liger/example-5477b4ff8c-tnhd9
 6 ---------------------------------------------------
 7 * metadata.annotations.foo: Invalid value: "null": Expected value: "bar"
 8
 9 --- expected
10 +++ actual
11 @@ -1,10 +1,16 @@
12 apiVersion: v1
13 kind: Pod
14 metadata:
15 - annotations:
16 - foo: bar
17 labels:
18 app: nginx
19 + name: example-5477b4ff8c-tnhd9
20 namespace: chainsaw-rare-liger
21 + ownerReferences:
22 + - apiVersion: apps/v1
23 + blockOwnerDeletion: true
24 + controller: true
25 + kind: ReplicaSet
26 + name: example-5477b4ff8c
27 + uid: 118abe16-ec42-4894-83db-64479c4aac6f
28 spec: {}
29 | 09:56:20 | deployment | step-1 | TRY | DONE |

The additional diff now gives a lot more context about the offending resource. Showing the ownerReferences field tells us who is responsible for the resource existence.

The diff complements complex assertion failures (metadata.annotations.foo: Invalid value: "null": Expected value: "bar") to provide everything needed to get a solid understanding of what failed in an assertion operation.

Thank you vfarcic for the feature request.

Resource templating support

This second new feature is probably a game changer in the e2e testing tools ecosystem.

Anyone serious with e2e testing faced this issue at least once. How can I make my resource manifest slightly different, depending on the test being executed?

Without resource templating, we always end up using workarounds like templating in a pre-processing step or relying on scripts invoking envsubst with a bunch of environment variables to perform substitutions. Those workarounds are error-prone, often limiting, and hard to maintain

Chainsaw v0.1.4 now offers a better solution for that, thanks to resource templating!

Bad example (before chainsaw v0.1.4)

Suppose you want to create a resource having a field that must be set to the URL of a service. This URL will be different depending on the namespace the service is installed in.

Without resource templating, this could be done with something like this.

Given the resource below:

1apiVersion: metrics.keptn.sh/v1beta1
2kind: KeptnMetricsProvider
3metadata:
4 name: my-mocked-provider
5spec:
6 type: prometheus
7 targetServer: "http://mockserver.$NAMESPACE.svc.cluster.local:1080"

We can use a script to perform namespace substitution with envsubst and pipe the result to kubectl:

 1apiVersion: chainsaw.kyverno.io/v1alpha1
 2kind: Test
 3metadata:
 4 name: example
 5spec:
 6 steps:
 7 - try:
 8 - script:
 9 content: |
10 envsubst < resource.yaml | kubectl apply -f - -n $NAMESPACE

This is bad because Chainsaw doesn’t know anything about this resource and won’t be able to clean it up when tearing down the test.

Good example (the chainsaw v0.1.4 way)

With resource templating this can become a regular apply operation, Chainsaw will now have full knowledge of the created resource.

The resource can embed complex expressions as demonstrated below:

1apiVersion: metrics.keptn.sh/v1beta1
2kind: KeptnMetricsProvider
3metadata:
4 name: my-mocked-provider
5spec:
6 type: prometheus
7 # `targetServer` is configured using a complex jmespath expression
8 targetServer: (join('.', ['http://mockserver', $namespace, 'svc.cluster.local:1080']))

The resource definition above can be used in a regular apply operation:

 1apiVersion: chainsaw.kyverno.io/v1alpha1
 2kind: Test
 3metadata:
 4 name: example
 5spec:
 6 # enable templating (at the test level)
 7 template: true
 8 steps:
 9 - try:
10 - apply:
11 # or enable templating (at the operation level)
12 template: true
13 file: resource.yaml

Supported operations

Resource templating is supported in apply, create and delete operations without restriction (provided templating has been enabled at the configuration, test, step or operation level).

Resource templating can also be used in assert and error operations with some restrictions. Because the content of an assert or error operation is already an assertion tree, only the elements used for looking up the resources to be processed by the operation will be considered for templating. That is, only apiVersion, kind, name, namespace and labels are considered for templating. Other fields are not, they are part of the assertion tree.

Last but not least, the same level of templating can be applied to the ephemeral test namespace created by Chainsaw using the namespaceTemplate stanza. This can be particularly useful if you need the ephemeral namespace to be annotated or labeled in a certain way:

 1apiVersion: chainsaw.kyverno.io/v1alpha1
 2kind: Test
 3metadata:
 4 name: example
 5spec:
 6 namespaceTemplate:
 7 metadata:
 8 annotations:
 9 keptn.sh/lifecycle-toolkit: enabled
10 steps:
11 # ...

Credits

Thank you to the Keptn folks, especially RealAnna for helping with the design of this feature.

Conclusion

Those two new features make Chainsaw a lot more flexible and improve usability a lot.

The resource templating opens new testing opportunities. Combined with the capacity to provide arbitrary data to tests with the --values flag, Chainsaw offers a very dynamic way to define tests.

Please keep in mind that resource templating is still experimental and could change slightly in future releases. Nonetheless, we encourage everyone to try it out and give us feedback to improve it as much as we can in the next versions.

More infos 👇

GitHub: https://github.com/kyverno/chainsaw
Docs: https://kyverno.github.io/chainsaw
Slack: https://kubernetes.slack.com/archives/C067LUFL43U

Blog: Securing Services Meshes Easier with Kyverno

Sun, 04 Feb 2024 00:00:00 +0000

Service meshes are all too common these days in Kubernetes with some platforms even building them into clusters by default. Service meshes are no doubt useful in a variety of ways which are well known, but it’s also well known they dramatically increase the complexity in those clusters. In addition to added complexity, service meshes also pose a (infamous) problem when it comes to enforcing Pod security because they require elevated privileges which can be difficult for other admission controllers to handle like Kubernetes’ own Pod Security Admission. In this post, we’ll explain more about this problem and how Kyverno can be a real savior when employing service meshes while giving you a preview of something to come in Kyverno 1.12 which will make security service meshes a piece of cake!

Introduction

Service meshes provide many benefits to Kubernetes applications including better load balancing, mutual TLS, observability, and more. It’s possible you’re even using a service mesh right now in one of your clusters. The most popular open source service meshes out there are Istio and Linkerd. All service meshes work fundamentally the same way which we won’t attempt to cover in depth in this blog. One salient point is that in order for traffic to be directed to and from their “sidecar” proxies it requires some tweaks to the iptables rules on the underlying Linux nodes. These tweaks or configuration modifications are the service mesh rewriting the routing rules of the networking stack. In order to do this, meshes like Istio and Linkerd employ an initContainer to do this before any of the other containers in the Pod start. And in order for that initContainer to do its job it requires some privileges which are very often problematic in security-conscious clusters. Minimally, these initContainers must add two Linux capabilities which allow them to make modifications to the networking stack: NET_ADMIN and NET_RAW. These initContainers may go even further by running as a root user, something which is a big no-no in the world of containers.

For example, Linkerd 2.14 will inject an initContainer like this into any Pod which should be part of its mesh (some fields omitted for brevity).

 1initContainers:
 2 - image: cr.l5d.io/linkerd/proxy-init:v2.2.3
 3 name: linkerd-init
 4 securityContext:
 5 allowPrivilegeEscalation: false
 6 capabilities:
 7 add:
 8 - NET_ADMIN
 9 - NET_RAW
10 privileged: false
11 readOnlyRootFilesystem: true
12 runAsNonRoot: true
13 runAsUser: 65534
14 seccompProfile:
15 type: RuntimeDefault

The thing is these additional privileges required by service mesh initContainers are forbidden by the Kubernetes official Pod Security Standards. This fact itself isn’t the main problem but rather that, depending on the policy engine used, providing allowances for these specialized initContainers is either difficult or downright impossible. We hear users’ pain just about every week in the Kyverno community and it seems those who experience the problem the worst are those using Pod Security Admission, the in-process admission controller implementing the Pod Security Standards. This has become such a notable problem that both Istio and Linkerd have tried to solve for it (or, more accurately, work around it) by offering another option: a custom CNI plugin.

CNI Plugin

These CNI plugins work for many but are still, for the most part, solve one problem at the expense of creating others. After all, the iptables rules still must be rewritten and something in the mesh still must be responsible for that task. In both cases, the CNI plugin implements a DaemonSet which runs a privileged container to perform these modifications on every node thereby avoiding the need for an initContainer in every Pod. This does have its advantages but it also has disadvantages.

The DaemonSet is even more privileged as it requires hostPath volumes and copies a configuration file and a binary to each node.
It requires understanding CNI plugins which is specialized knowledge.
Creates more complexity to operate and automate.
Creates potential conflicts with other CNI plugins since they aren’t aware of each other and determining how to chain multiple together isn’t standardized.
Creates potential race conditions during horizontal cluster scaling or node restarts as the DaemonSet Pod may not launch before workload Pods.

The Crux of the Problem

But why is solving the initContainer problem an actual problem? The answer lies in exclusions. Exclusions, or how you exempt certain resources from being applicable to policies, is what distinguishes good admission controllers from great ones. The objective is to provide as strong of a Pod security posture as you possibly can while not impacting things which you objectively need. You want to be able to separate the “good” from the “bad” and your service meshes are definitely in the “good” category. But like a sieve filters sand, you have to carefully filter out the “good” so you’re only left with the “bad”. In that initContainer example previously, you definitely don’t want non-service-mesh Pods from adding the NET_ADMIN capability as that would give them unfettered access to the networking stack allowing, for example, snooping and masquerading to occur. Options for reducing the size of the funnel are as follows, from largest to smallest.

Disable Pod security in the entire cluster
- This is obviously a non-starter so no need to discuss further.
Disable Pod security in the affected Namespace
- Because we’re talking about an initContainer in every Pod that must participate in the mesh, that basically means you’d have to disable Pod security in most Namespaces of the cluster which is effectively like the first option–a no-go.
Disable the profile which contains this check (if applicable)
- Pod Security Standards are organized into collections called profiles with each profile containing multiple controls. A control is an edict on what fields should be checked and what values are or are not permitted. You could find the profile which includes this control and disable the whole profile, but that obviously disables other controls in the same profile. Also not great. Not all policy admission controllers offer this ability.
Disable the control on the Pod
- These initContainers which request NET_ADMIN and NET_RAW violate the “Capabilities” control in the baseline profile of the Pod Security Standards, which is the foundational profile of Pod security (the restricted profile builds upon baseline). You could simply not check for this control in any Pod which uses such an initContainer, but that’s no good either because then a malicious container could also add NET_ADMIN. Great that you may be enforcing all other controls, but deactivating even one control summarily is still too much.
Disable the control on one image
- If you’ve reached this level, you’re doing well. You can simply not check for these privileged capabilities on images which match a certain pattern. But we can do even better still. (By the way, it wouldn’t be completely safe to do this based on the name of the initContainer as some malicious user could create an initContainer called istio-init which uses an image named ubuntu:latest.)
Disable the control on one image and in one location in the Pod
- Now we’re talking. We can isolate an exemption to ONLY a specific image and ONLY in a specific location in the Pod. For example, we can excuse the istio/proxyv2 image if it is found in the initContainers[] array but only from the capabilities check. If that same image is used in the main containers[] list, it would cause the entire Pod to be denied.

One reason why many folks have trouble with this problem is due to Pod Security Admission (PSA). With PSA, the most granular you can achieve is the third bullet from the top: disable the profile which contains this check. Because the restricted profile is inclusive of the baseline profile, disabling the baseline profile on a Namespace is essentially the same as enforcing no Pod security at all. This limitation is the primary reason why the CNI plugin solution was created. If service meshes can separate out the need for those elevated privileges into just a single controller (a DaemonSet) and that controller only runs in a single Namespace, then we can basically isolate that one Namespace with an exemption.

Policies in Kyverno

In Kyverno, you have a couple options for implementing Pod Security Standards. The first and “original” way is by writing a validate rule for each control within the Pod Security Standards. Kyverno already provides the full complement of such policies packaged as a Helm chart which are also available as individual policies. The “Capabilities” control in the baseline profile, for example, can be found here. In this style of policy, you can get as granular as you like. The slight downside is, when it comes to the pre-built Pod Security Standards, they require some modifications when it comes to these service mesh initContainers. While some of those modifications are fairly gentle, others may need to be more extreme.

For example, this is what the same “Capabilities” check may look like in order to make allowances for these service mesh initContainers.

Since Kyverno is very flexible when it comes to policy authoring there are almost always multiple ways to write the same declaration, so if you’ve already done this and the results differ don’t worry.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: disallow-capabilities
 5spec:
 6 validationFailureAction: Enforce
 7 background: true
 8 rules:
 9 - name: adding-capabilities-service-mesh
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 preconditions:
16 all:
17 - key: "{{ request.operation || 'BACKGROUND' }}"
18 operator: NotEquals
19 value: DELETE
20 context:
21 - name: capabilities
22 variable:
23 value: ["AUDIT_WRITE","CHOWN","DAC_OVERRIDE","FOWNER","FSETID","KILL","MKNOD","NET_BIND_SERVICE","SETFCAP","SETGID","SETPCAP","SETUID","SYS_CHROOT"]
24 validate:
25 message: >-
26 Any capabilities added beyond the allowed list (AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER,
27 FSETID, KILL, MKNOD, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID, SYS_CHROOT)
28 are disallowed. Service mesh initContainers may additionally add NET_ADMIN and NET_RAW.
29 foreach:
30 - list: request.object.spec.initContainers[]
31 preconditions:
32 all:
33 - key: "{{ element.image }}"
34 operator: AnyIn
35 value:
36 - "*/istio/proxyv2*"
37 - "*/linkerd/proxy-init*"
38 - key: "{{ element.securityContext.capabilities.add[] || `[]` }}"
39 operator: AnyNotIn
40 value:
41 - NET_ADMIN
42 - NET_RAW
43 - "{{ capabilities }}"
44 deny:
45 conditions:
46 all:
47 - key: "{{ element.securityContext.capabilities.add[] || `[]` }}"
48 operator: AnyNotIn
49 value: "{{ capabilities }}"
50 message: The service mesh initContainer {{ element.name }} is attempting to add forbidden capabilities.
51 - list: request.object.spec.initContainers[]
52 preconditions:
53 all:
54 - key: "{{ element.image }}"
55 operator: AnyNotIn
56 value:
57 - "*/istio/proxyv2*"
58 - "*/linkerd/proxy-init*"
59 deny:
60 conditions:
61 all:
62 - key: "{{ element.securityContext.capabilities.add[] || `[]` }}"
63 operator: AnyNotIn
64 value: "{{ capabilities }}"
65 message: The initContainer {{ element.name }} is attempting to add forbidden capabilities.
66 - list: request.object.spec.[ephemeralContainers, containers][]
67 deny:
68 conditions:
69 all:
70 - key: "{{ element.securityContext.capabilities.add[] || `[]` }}"
71 operator: AnyNotIn
72 value: "{{ capabilities }}"
73 message: The container {{ element.name }} is attempting to add forbidden capabilities.

Feel free to take this for a spin in the Kyverno Playground and see for yourself. Inside, you’ll find sample Pods for both Istio and Linkerd, so try to uncomment and copy elements around to test it out.

Because Istio’s initContainer requires a few more privileges than Linkerd’s, some slight modifications are also necessary to a couple more policies, these being found in the restricted profile. For example, here’s a Kyverno Playground link to what the require-run-as-nonroot policy may look like to exempt istio-init.

Individual Kyverno policies like those just shown allow maximum flexibility, but there is an easier path to Pod Security Standards in Kyverno. The second way of implementing these standards is using what we call a “subrule” to a validate style policy. In this style, the podSecurity element is used to refer specifically to these Pod Security Standards. Under the hood, Kyverno implements the exact same library as Kubernetes’ Pod Security Admission but with a different “wrapper” to make their application more flexible.

For example, using this type of subrule would allow you to easily implement the entire baseline profile of the Pod Security Standards along with an exclusion for these service mesh images like so.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: pod-security-standards
 5spec:
 6 background: true
 7 validationFailureAction: Enforce
 8 rules:
 9 - name: baseline-service-mesh
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 validate:
16 podSecurity:
17 level: baseline  ## enforce the baseline profile
18 version: latest  ## enforce the latest version of this profile
19 exclude: ## exclude certain controls with optionally certain images
20 - controlName: Capabilities
21 images:
22 - "*/istio/proxyv2*"
23 - "*/linkerd/proxy-init*"

The exclude[] block here names the “Capabilities” control that we’ve been covering thus far, and the images[] field names the two specific service mesh images which should be excluded. With this ability you get PSA-like behavior but with granularity that is simply not possible.

These two options give you plenty of choice, but they all involve modifying the policy directly. Another option exists which allows decoupling the exceptions from the policies themselves, and these are Policy Exceptions. For example, you can write a PolicyException resource which exempts a given Pod from a specific rule in a specific policy. This is great especially for developer self-service as it allows other users to request exceptions without having to so much as look at a Kyverno policy. However, this as it stands in 1.11 isn’t quite granular enough for cases like service mesh initContainers and so is receiving some nice upgrades in Kyverno 1.12. More on that next.

Enhancements in 1.12

With the forthcoming Kyverno 1.12, we’re making some exciting enhancements which will make exclusions for use cases like service mesh containers even easier.

The first enhancement in 1.12 is the ability to further classify exclusions to the podSecurity subrule by listing specific fields as well as their values. This allows you to both use the easy style of policy language but get down to the lowest level of the funnel. For example, this is how you’ll be able to enforce the entire baseline profile of the Pod Security Standards but only exclude Istio’s and Linkerd’s images from specifically the initContainers list.

 1### Coming in Kyverno 1.12 ###
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: pod-security-standards
 6spec:
 7 background: true
 8 validationFailureAction: Enforce
 9 rules:
10 - name: baseline-service-mesh
11 match:
12 any:
13 - resources:
14 kinds:
15 - Pod
16 validate:
17 podSecurity:
18 level: baseline
19 version: latest
20 exclude:
21 - controlName: Capabilities
22 images:
23 - "*/istio/proxyv2*"
24 - "*/linkerd/proxy-init*"
25 restrictedField: spec.initContainers[*].securityContext.capabilities.add
26 values:
27 - NET_ADMIN
28 - NET_RAW

The second enhancement is to Policy Exceptions making them podSecurity aware in that you will be able to exempt specific control names in the PolicyException resource. For example, below is a PolicyException you’ll be able to create in Kyverno 1.12 for the previous validate.podSecurity subrule allowing you to decouple those exclusions but only for Pods created in the staging Namespace.

 1### Coming in Kyverno 1.12 ###
 2apiVersion: kyverno.io/v2beta1
 3kind: PolicyException
 4metadata:
 5 name: pod-security-exception
 6 namespace: kyverno
 7spec:
 8 exceptions:
 9 - policyName: pod-security-standards
10 ruleNames:
11 - baseline-service-mesh
12 match:
13 any:
14 - resources:
15 namespaces:
16 - staging
17 podSecurity:
18 - controlName: Capabilities
19 images:
20 - "*/istio/proxyv2*"
21 - "*/linkerd/proxy-init*"

This will be enhanced further in the future to apply to a specific container. See and follow the issue here for details.

Closing

This post covered a bit about service meshes and why the use initContainers as well as the security problems posed by them. We covered multiple ways how Kyverno can solve those problems in the most granular way possible while also providing a glimpse at the next version and how this process will be made even easier. Get in touch with the Kyverno project if you have any other questions or feedback for us!

Blog: Kyverno Chainsaw - Exploring the Power of Assertion Trees!

Wed, 13 Dec 2023 00:00:00 +0000

While the Chainsaw documentation is nice and comprehensive, I feel like the most powerful feature of Chainsaw deserves its own blog post for a couple of reasons:

Its hard to make it standout in the documentation
You can’t appreciate Chainsaw until you understand what makes it so unique
Seeing the feature in action is the best way to learn about it

What makes Chainsaw unique is its assertion model.

In this blog post we will deep dive into the assertion model used by Chainsaw, how it works, and how we extended something that looks like YAML but is more than that.

After all, if Chainsaw was only about applying resources in a cluster and checking fields are set correctly on those resources it would be deceptive. There are other tools which can do that today and Chainsaw would be more or less redundant.

Basic assertions

Let’s start by reviewing the basics briefly.

The most basic assertion

We will start with a basic assertion:

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: coredns
5 namespace: kube-system
6spec:
7 replicas: 2

When asking Chainsaw to execute the assertion above, it will look for a Deployment named coredns in the kube-system namespace and will compare the existing resource with the (partial) resource definition contained in the assertion.

In this specific case, if spec.replicas is set to 2 in the existing resource, the assertion will be considered valid. If it is not equal to 2 the assertion will be considered failed.

This is the most basic assertion Chainsaw can evaluate.

A slightly less basic assertion

A slightly less basic assertion is shown below.

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 labels:
5 k8s-app: kube-dns
6 namespace: kube-system
7spec:
8 replicas: 2

This time we are not providing a resource name.

Chainsaw will look up all Deployments with the k8s-app: kube-dns label in the kube-system namespace. The assertion will be considered valid if at least one Deployment matches with the (partial) resource definition contained in the assertion. If none matches the assertion will be considered failed.

Apart from the resource lookup process being a little bit more interesting, this kind of assertion is essentially the same as the previous one. Chainsaw is basically making a decision by comparing an actual and expected resource.

A word on timeout

Keep in mind that Chainsaw won’t consider an assertion valid or failed in one shot. The assertion will be continuously evaluated during a predetermined period of time until the assertion succeeds or the timeout expires.

It’s good, but not enough

Now that we covered the basic assertions we can look at their limitations and how Chainsaw solves them.

Basic assertions cover the simple cases but:

Working with arrays is not easy
- How are we supposed to compare arrays with different sizes?
- How can we check a specific item in an array is present?
Matching exact values is not always what we want
- How can we verify the number of replicas is above a certain number (not exactly this number)?
- How can we apply a regex to a label?

While the examples above are the most obvious ones, they demonstrate that an assertion model needs to be more rich and more flexible than simple comparisons.

Overview of the Chainsaw Assertion Model

Let’s see how Chainsaw can assert that the number of replicas of a Deployment is greater than a certain threshold. Let’s look at how it can be written.

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 labels:
5 k8s-app: kube-dns
6 namespace: kube-system
7spec:
8 (replicas > 2): true

In the assertion above, the key part of the YAML syntax contains a JMESPath expression. The evaluation of this expression will be compared to the value part of the YAML syntax.

In this case, the expression is simple but represents the main concept behind the Chainsaw assertion model. At every node in the YAML tree, Chainsaw can create a projection of the element currently being processed and pass the result of this projection to the descendants. When a leaf of the tree is reached, a comparison happens. If the comparison fails, the assertion will be considered failed. If all comparisons succeed, the assertion will be considered valid.

Working with arrays

Working with arrays can be challenging. To manipulate arrays, Chainsaw leverages JMESPath capabilities to allow easy filtering and projections.

Let’s say we want an assertion to consider only containers that have no security context defined.

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4 labels:
 5 k8s-app: kube-dns
 6 namespace: kube-system
 7spec:
 8 template:
 9 spec:
10 (containers[?securityContext == null]):
11 (len(@)): 0

In the assertion above, the first three fields spec, template, and spec are basic projections that simply take the content of their respective fields and pass that to descendants.

(containers[?securityContext == null]) is a JMESPath expression filtering the containers array, selecting only the element where securityContext is null. This projection results in a new array that is passed to the descendant ((len(@)) in this case).

(len(@)) is another JMESPath expression that computes the length of the array. There’s no more descendant at this point. We’re at a leaf of the YAML tree and the array length we just computed is then compared against 0.

If the comparison matches, the assertion will be considered valid; if not, it will be considered failed.

In simple words, the assertion above checks that all containers in the Deployment have a security context configured.

Iterating over arrays (or maps)

Being able to filter arrays allows selecting the elements to be processed.

On top of that, Chainsaw allows iterating over array elements to validate each item separately.

The assertion above can be rewritten using an iterator:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4 labels:
 5 k8s-app: kube-dns
 6 namespace: kube-system
 7spec:
 8 template:
 9 spec:
10 # the `~` modifier tells Chainsaw to iterate over the array elements
11 ~.(containers):
12 securityContext: {}

This assertion uses the ~ modifier and Chainsaw will evaluate descendants once per element in the array.

Explicit bindings and reference

When passing from a parent node to descendant nodes, it can be useful to keep a reference to one or more parent nodes.

Let’s say we want to consider the pod-level security context in addition to the container-level security context. We can write such an assertion with the syntax below:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4 labels:
 5 k8s-app: kube-dns
 6 namespace: kube-system
 7spec:
 8 template:
 9 # `->podSpec` creates a binding to the pod spec, accessible in descendant nodes
10 (spec)->podSpec:
11 # the `~` modifier tells Chainsaw to iterate over the array elements
12 ~.(containers):
13 # `$podSpec` references the binding created above
14 ($podSpec.securityContext != null || securityContext != null): true

In the assertion above, (spec)->podSpec means that Chainsaw will keep a reference to the result of the expression evaluation and will make it available to descendants under the $podSpec binding.

Now we can use the binding to write an assertion check taking the pod-level security context into account like this ($podSpec.securityContext != null || securityContext != null): true. This enhanced assertion check will ensure that, if any container’s securityContext is not defined, the pod-level securityContext is defined.

What more

We covered all major features of assertion trees in this blog post.

Chainsaw doesn’t directly implement assertion trees but relies on the kyverno-json package. You can browse the documentation to learn more about assertion trees, this documentation also applies to Chainsaw. The kyverno-json playground is also a good resource to familiarise with assertion trees.

Note that JMESPath supports functions and also allows custom functions to be registered in the JMESPath interpreter. The supported list of functions is available here.

Conclusion

I hope this blog post helps you understand what assertion trees are and how they work.

Simple tests usually only need simple capabilities, but assertion trees can come in very handy when your tests become more complex.

In the future, we plan to support CEL as well as JMESPath as the underlying evaluation engine. While JMESPath works well, the fact that Kubernetes adopted CEL makes it an excellent choice for Chainsaw too.

Blog: Kyverno Chainsaw - The ultimate end to end testing tool!

Tue, 12 Dec 2023 00:00:00 +0000

Creating Kubernetes operators is hard, testing Kubernetes operators is also hard. Of course creating, maintaining and testing a Kubernetes operator is even harder.

It often requires writing and maintaining additional code to get proper end to end testing, it takes time, is a cumbersome process, and making changes becomes a pain. All this often leads to poor operator testing and can impact the operator quality.

Today we are extremely proud to release the first stable version of Kyverno Chainsaw, a tool to make end to end testing Kubernetes operators entirely declarative, simple and almost fun.

In this blog post, we will introduce Chainsaw, how it works, and what problems it is solving. Hopefully after reading it you will never consider writing end to end tests the same!

What are Kubernetes operators

Kubernetes operators are described in this Kubernetes documentation page.

Operators are software extensions to Kubernetes that make use of custom resources to manage applications and their components. Operators follow Kubernetes principles, notably the control loop.

They often rely on Custom Resource Definitions and continuously reconcile the cluster state with the spec of Custom Resources.

How do we test a Kubernetes operator

An operator is essentially responsible for watching certain resources in a cluster and reacting to maintain a state matching the spec described in the Custom Resources.

Testing an operator boils down to creating, updating, or deleting certain resources and verifying the state of the cluster changes accordingly.

For example, an operator could be responsible for managing role bindings and service accounts in a cluster based on a simplified definition of permissions. This operator exists, see rbac-manager from FairWinds.

In the next sections of this blog post I will demonstrate how Chainsaw can help testing the rbac-manager operator.

Getting started

Before we can look at Chainsaw we need a Kubernetes cluster with rbac-manager installed. We can create a local cluster with KinD and use Helm to install the operator.

1# create a cluster
2kind create cluster
3
4# deploy rbac-manager
5helm install rbac-manager --repo https://charts.fairwinds.com/stable rbac-manager --namespace rbac-manager --create-namespace

Once the operator is installed, you should see a new Custom Resource Definition in the cluster:

1kubectl get crd

1NAME CREATED AT
2rbacdefinitions.rbacmanager.reactiveops.io 2023-12-12T12:20:19Z

Install Chainsaw

Chainsaw can be installed in different ways. If you are using MacOS or Linux, the simplest solution is to use brew.

1# add the chainsaw tap
2brew tap kyverno/chainsaw https://github.com/kyverno/chainsaw
3
4# install chainsaw
5brew install kyverno/chainsaw/chainsaw

What is a test

To put it simply, a test can be represented as an ordered sequence of test steps.

Test steps within a test are run sequentially: if any of the test steps fail, the entire test is considered failed.

A test step can consist of one or more operations:

To delete resources present in a cluster
To create or update resources in a cluster
To assert one or more resources in a cluster meet the expectations (or the opposite)
To run arbitrary commands or scripts

In Chainsaw, tests are entirely declarative and created with YAML files.

Our first test

In this first test, we’re going to create an RBACDefinition and verify the rbac-manager operator created the corresponding ClusterRoleBinding in the cluster.

RBACDefinition

The RBACDefinition below states that the service account rbac-manager/test-rbac-manager should be bound to a test-rbac-manager cluster role.

 1cat > resources.yaml << EOF
 2apiVersion: rbacmanager.reactiveops.io/v1beta1
 3kind: RBACDefinition
 4metadata:
 5 name: rbac-manager-definition
 6rbacBindings:
 7 - name: admins
 8 subjects:
 9 - kind: ServiceAccount
10 name: test-rbac-manager
11 namespace: rbac-manager
12 clusterRoleBindings:
13 - clusterRole: test-rbac-manager
14EOF

ClusterRoleBinding

If we apply the RBACDefinition definition above, the operator is expected to create the corresponding ClusterRoleBinding.

 1cat > expected.yaml << EOF
 2apiVersion: rbac.authorization.k8s.io/v1
 3kind: ClusterRoleBinding
 4metadata:
 5 labels:
 6 rbac-manager: reactiveops
 7 ownerReferences:
 8 - apiVersion: rbacmanager.reactiveops.io/v1beta1
 9 kind: RBACDefinition
10 name: rbac-manager-definition
11roleRef:
12 apiGroup: rbac.authorization.k8s.io
13 kind: ClusterRole
14 name: test-rbac-manager
15subjects:
16- kind: ServiceAccount
17 name: test-rbac-manager
18 namespace: rbac-manager
19EOF

An important point in this manifest is that it doesn’t contain a name. This manifest won’t be used by Chainsaw to create resources in the cluster but to verify that a resource in the cluster exists and matches with this definition.

Finally writing the test file

To summarize, the test we want to write should do:

Apply the RBACDefinition in the cluster
Verify the corresponding ClusterRoleBinding is created by the operator
Cleanup and move to the next test

Such a Chainsaw test can be written like this:

 1cat > chainsaw-test.yaml << EOF
 2apiVersion: chainsaw.kyverno.io/v1alpha1
 3kind: Test
 4metadata:
 5 name: clusterrolebindings
 6spec:
 7 steps:
 8 - try:
 9 # create resources in the cluster
10 - apply:
11 file: resources.yaml
12 # verify the operator reacted as expected
13 - assert:
14 file: expected.yaml
15EOF

Please note that the file containing the test is named chainsaw-test.yaml.

Invoking Chainsaw

To execute the test we just created against the local cluster, we need to invoke Chainsaw with the test command.

1chainsaw test

 1Version: 0.1.0
 2Loading default configuration...
 3- Using test file: chainsaw-test.yaml
 4- TestDirs [.]
 5- SkipDelete false
 6- FailFast false
 7- ReportFormat ''
 8- ReportName 'chainsaw-report'
 9- Namespace ''
10- FullName false
11- IncludeTestRegex ''
12- ExcludeTestRegex ''
13- ApplyTimeout 5s
14- AssertTimeout 30s
15- CleanupTimeout 30s
16- DeleteTimeout 15s
17- ErrorTimeout 30s
18- ExecTimeout 5s
19Loading tests...
20- clusterrolebindings (.)
21Running tests...
22=== RUN chainsaw
23=== PAUSE chainsaw
24=== CONT chainsaw
25=== RUN chainsaw/clusterrolebindings
26=== PAUSE chainsaw/clusterrolebindings
27=== CONT chainsaw/clusterrolebindings
28 | 13:41:26 | clusterrolebindings | @setup | CREATE | OK | v1/Namespace @ chainsaw-ample-racer
29 | 13:41:26 | clusterrolebindings | step-1 | TRY | RUN |
30 | 13:41:26 | clusterrolebindings | step-1 | APPLY | RUN | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
31 | 13:41:26 | clusterrolebindings | step-1 | CREATE | OK | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
32 | 13:41:26 | clusterrolebindings | step-1 | APPLY | DONE | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
33 | 13:41:26 | clusterrolebindings | step-1 | ASSERT | RUN | rbac.authorization.k8s.io/v1/ClusterRoleBinding @ *
34 | 13:41:26 | clusterrolebindings | step-1 | ASSERT | DONE | rbac.authorization.k8s.io/v1/ClusterRoleBinding @ *
35 | 13:41:26 | clusterrolebindings | step-1 | TRY | DONE |
36 | 13:41:26 | clusterrolebindings | @cleanup | DELETE | RUN | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
37 | 13:41:26 | clusterrolebindings | @cleanup | DELETE | OK | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
38 | 13:41:26 | clusterrolebindings | @cleanup | DELETE | DONE | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
39 | 13:41:26 | clusterrolebindings | @cleanup | DELETE | RUN | v1/Namespace @ chainsaw-ample-racer
40 | 13:41:26 | clusterrolebindings | @cleanup | DELETE | OK | v1/Namespace @ chainsaw-ample-racer
41 | 13:41:31 | clusterrolebindings | @cleanup | DELETE | DONE | v1/Namespace @ chainsaw-ample-racer
42--- PASS: chainsaw (0.00s)
43 --- PASS: chainsaw/clusterrolebindings (5.28s)
44PASS
45Tests Summary...
46- Passed tests 1
47- Failed tests 0
48- Skipped tests 0
49Done.

Chainsaw will discover tests and run them, either concurrently or sequentially depending on the tool and tests configuration.

A more advanced test

In the test above, we only covered the creation of RBACDefinition resources. While it’s a good starting point, we also want to test updates and deletions. If we delete an RBACDefinition resource for example, the corresponding ClusterRoleBinding should be deleted from the cluster by the operator.

Chainsaw can easily do that, we just need to add two more steps to our test to delete the RBACDefinition and verify the ClusterRoleBinding is deleted accordingly.

 1cat > chainsaw-test.yaml << EOF
 2apiVersion: chainsaw.kyverno.io/v1alpha1
 3kind: Test
 4metadata:
 5 name: clusterrolebindings
 6spec:
 7 steps:
 8 - try:
 9 # create resources in the cluster
10 - apply:
11 file: resources.yaml
12 # verify the operator reacted as expected
13 - assert:
14 file: expected.yaml
15 # delete previously created resources
16 - delete:
17 ref:
18 apiVersion: rbacmanager.reactiveops.io/v1beta1
19 kind: RBACDefinition
20 name: rbac-manager-definition
21 # make sure expected resources have been deleted
22 - error:
23 file: expected.yaml
24EOF

Running Chainsaw again

If we execute this new test, Chainsaw will now verify that deleting a resource has the expected effect in the cluster.

1chainsaw test

 1Version: 0.1.0
 2Loading default configuration...
 3- Using test file: chainsaw-test.yaml
 4- TestDirs [.]
 5- SkipDelete false
 6- FailFast false
 7- ReportFormat ''
 8- ReportName 'chainsaw-report'
 9- Namespace ''
10- FullName false
11- IncludeTestRegex ''
12- ExcludeTestRegex ''
13- ApplyTimeout 5s
14- AssertTimeout 30s
15- CleanupTimeout 30s
16- DeleteTimeout 15s
17- ErrorTimeout 30s
18- ExecTimeout 5s
19Loading tests...
20- clusterrolebindings (.)
21Running tests...
22=== RUN chainsaw
23=== PAUSE chainsaw
24=== CONT chainsaw
25=== RUN chainsaw/clusterrolebindings
26=== PAUSE chainsaw/clusterrolebindings
27=== CONT chainsaw/clusterrolebindings
28 | 13:50:35 | clusterrolebindings | @setup | CREATE | OK | v1/Namespace @ chainsaw-causal-cobra
29 | 13:50:35 | clusterrolebindings | step-1 | TRY | RUN |
30 | 13:50:35 | clusterrolebindings | step-1 | APPLY | RUN | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
31 | 13:50:35 | clusterrolebindings | step-1 | CREATE | OK | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
32 | 13:50:35 | clusterrolebindings | step-1 | APPLY | DONE | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
33 | 13:50:35 | clusterrolebindings | step-1 | ASSERT | RUN | rbac.authorization.k8s.io/v1/ClusterRoleBinding @ *
34 | 13:50:35 | clusterrolebindings | step-1 | ASSERT | DONE | rbac.authorization.k8s.io/v1/ClusterRoleBinding @ *
35 | 13:50:35 | clusterrolebindings | step-1 | DELETE | RUN | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
36 | 13:50:35 | clusterrolebindings | step-1 | DELETE | OK | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
37 | 13:50:35 | clusterrolebindings | step-1 | DELETE | DONE | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
38 | 13:50:35 | clusterrolebindings | step-1 | ERROR | RUN | rbac.authorization.k8s.io/v1/ClusterRoleBinding @ *
39 | 13:50:35 | clusterrolebindings | step-1 | ERROR | DONE | rbac.authorization.k8s.io/v1/ClusterRoleBinding @ *
40 | 13:50:35 | clusterrolebindings | step-1 | TRY | DONE |
41 | 13:50:35 | clusterrolebindings | @cleanup | DELETE | RUN | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
42 | 13:50:35 | clusterrolebindings | @cleanup | DELETE | DONE | rbacmanager.reactiveops.io/v1beta1/RBACDefinition @ rbac-manager-definition
43 | 13:50:35 | clusterrolebindings | @cleanup | DELETE | RUN | v1/Namespace @ chainsaw-causal-cobra
44 | 13:50:35 | clusterrolebindings | @cleanup | DELETE | OK | v1/Namespace @ chainsaw-causal-cobra
45 | 13:50:40 | clusterrolebindings | @cleanup | DELETE | DONE | v1/Namespace @ chainsaw-causal-cobra
46--- PASS: chainsaw (0.00s)
47 --- PASS: chainsaw/clusterrolebindings (5.32s)
48PASS
49Tests Summary...
50- Passed tests 1
51- Failed tests 0
52- Skipped tests 0
53Done.

Conclusion

In this short blog post we demonstrated how Chainsaw can be useful to test Kubernetes operators.

Chainsaw can go a lot deeper and offers much more features than what we demonstrated here.

If you’re writing an operator, chances are you need to write end to end tests and this can be painful. Chainsaw can help tremendously in focusing on the tests needed rather than messing with writing and maintaining a test framework.

Using it within the Kyverno project helped improve the test coverage by orders of magnitude. Converting issues into end to end tests is often a matter of copying-and-pasting a couple of manifests. Such simplicity guarantees more than just fixing issues but prevents regressions by having a test that continuously verifies they don’t happen again.

🔗 Check out the project on GitHub: https://github.com/kyverno/chainsaw
📚 Browse the documentation: https://kyverno.github.io/chainsaw

Blog: Kyverno Completes Third-Party Security Audit

Tue, 28 Nov 2023 00:00:00 +0000

The Kyverno project is pleased to announce the completion of its third-party security audit. The audit was conducted by Ada Logics in collaboration with the Kyverno maintainers, the Open Source Technology Improvement Fund and was funded by the Cloud Native Computing Foundation.

The audit was a holistic security audit with four goals:

Define a formal threat model for Kyverno.
Conduct a manual code audit for security vulnerabilities.
Assess Kyverno’s fuzzing suite against the threat model.
Evaluate Kyvernos supply-chain risks against SLSA.

Ada Logics found 10 security issues during the manual code auditing goal. Four of these had their root cause in the Notary verifier which had not been released prior to the audit. One of the findings was in a third-party dependency to Kyverno and was fixed by the Cosign project maintainers.

In total, 6 CVE’s were assigned during the audit for the following components:

CVE ID	Vulnerable Kyverno Component	CVE Severity
CVE-2023-42816	Notary verifier	Moderate
CVE-2023-42815	Notary verifier	Low
CVE-2023-42813	Notary verifier	Moderate
CVE-2023-42814	Notary verifier	Low
CVE-2023-47630	Kyverno Engine Image Loader	High
CVE-2023-46737	Cosign (upstream)	Low

Users consuming Kyverno from official releases have not been affected by the four CVE’s in the Notary verifier, since the Notary verifier has never been part of a public release, before Ada Logics reported the findings during the security audit. Only users building Kyverno from the main branch would be affected by these, however, building from main is highly discouraged.

During the fuzzing goal of the audit, Ada Logics wrote three new fuzzers and added them to Kyvernos fuzzing suite; Earlier this year, Kyverno completed its dedicated fuzzing security audit during which Ada Logics integrated Kyverno into OSS-Fuzz and built a fuzzing suite focusing on hitting high-coverage entry points. During the current security audit, Ada Logics wrote two fuzzers specifically for policy enforcement that attempt to create admission requests that are able to bypass Kyverno policies. In addition, Ada Logics wrote a fuzzer for a third-party dependency that implements complex data processing routines. The two policy fuzzers did not find any possible policy bypasses; the fuzzer for the third-party dependency found two reliability bugs.

During the SLSA goal, the auditors found that Kyverno impressively complies with the highest possible SLSA score and thereby ensures tamper-proof artifacts to consumers. Kyverno adopts the slsa-github-generator which ensures SLSA level 3 compliance by generating verifiable provenance alongside releases on GitHub actions. Consumers can verify Kyvernos provenance by using the slsa-verifier.

The Kyverno maintainers have quickly addressed all issues found during the audit, with fixes incorporated in Kyverno v1.10.6 and v1.11.1. By prioritizing security work, the Kyverno team aims to provide a seamless and secure experience for consumers. Kyverno will continue to invest in robust security measures, staying ahead of potential threats and vulnerabilities.

Security researchers interested in contributing to Kyverno can find information about getting started here or engage with the Kyverno community.

Blog: Kyverno 1.11 Released

Thu, 16 Nov 2023 00:00:00 +0000

The Kyverno team is delighted to share a new Kyverno release, v1.11! This release marks a significant milestone for Kyverno, with an extensive development period of around five months, including eight pre-releases and the merging of over 500 pull requests. We are incredibly proud of the progress made and cannot wait for you to explore the remarkable additions in Kyverno 1.11!

Key New Features in Kyverno 1.11

ValidatingAdmissionPolicy Support

In Kyverno 1.11, a new sub-rule validate.cel was introduced. This sub-rule type allows users to use Common Expression Language (CEL) expressions for resource validation. CEL was initially introduced in Kubernetes for the validation rules of CustomResourceDefinitions and is now also utilized by Kubernetes ValidatingAdmissionPolicies.

For example, this policy ensures that deployment replicas are less than 4.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: check-deployment-replicas
 5spec:
 6 validationFailureAction: Enforce
 7 background: false
 8 rules:
 9 - name: check-deployment-replicas
10 match:
11 any:
12 - resources:
13 kinds:
14 - Deployment
15 validate:
16 cel:
17 expressions:
18 - expression: "object.spec.replicas < 4"
19 message: "Deployment spec.replicas must be less than 4."

It is possible to generate Kubernetes ValidatingAdmissionPolicies and their bindings from the Kyverno policy mentioned above. With this feature, Kyverno now offers complete policy management for Kubernetes ValidatingAdmissionPolicies. This includes the ability to apply ValidatingAdmissionPolicies to resources using the command-line interface (CLI) and to obtain PolicyReports for them.

Policy Report Enhancements

In previous versions of Kyverno, PolicyReports were generated and grouped per namespace per policy. The AdmissionReports and BackgroundScanReports were retained in the cluster for remediation purposes. However, this posed a challenge for large clusters as it led to resource overload in etcd.

With the introduction of Kyverno 1.11, a PolicyReport is now created for each individual resource and will be automatically removed if the corresponding resource is deleted. To avoid repetition in every result, the scope field in the report is utilized to indicate resource metadata. Additionally, AdmissionReports and BackgroundScanReports are now considered ephemeral resources and are cleaned up once they are aggregated into the final reports. Below is a snippet of a PolicyReport.

 1apiVersion: wgpolicyk8s.io/v1alpha2
 2kind: PolicyReport
 3metadata:
 4 name: 0f8f65db-3bfa-4178-af8d-d66fc765b17e
 5 namespace: default
 6 ownerReferences:
 7 - apiVersion: v1
 8 kind: Service
 9 name: kubernetes
10 uid: 0f8f65db-3bfa-4178-af8d-d66fc765b17e
11results:
12- category: Networking
13 message: validation rule 'disallow-LoadBalancer' passed.
14 policy: disallow-loadbalancer-service
15 result: pass
16 rule: disallow-LoadBalancer
17 source: kyverno
18...
19scope:
20 apiVersion: v1
21 kind: Service
22 name: kubernetes
23 namespace: default
24 uid: 0f8f65db-3bfa-4178-af8d-d66fc765b17e
25summary:
26 pass: 1
27...

Notary Updates

In Kyverno version 1.11, support for verifying OCI 1.1 attestations using Notary has been added. The following policy will verify the signature on the sbom/cyclone-dx attestation and check if the license version of all the components in the SBOM is GPL-3.0.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: check-image-attestation
 5spec:
 6 validationFailureAction: Enforce
 7 webhookTimeoutSeconds: 30
 8 failurePolicy: Fail 
 9 rules:
10 - name: verify-attestation-notary
11 match:
12 any:
13 - resources:
14 kinds:
15 - Pod
16 context:
17 - name: keys
18 configMap:
19 name: keys
20 namespace: kyverno
21 verifyImages:
22 - type: Notary
23 imageReferences:
24 - "ghcr.io/kyverno/test-verify-image*"
25 attestations:
26 - type: sbom/cyclone-dx
27 attestors:
28 - entries:
29 - certificates:
30 cert: |-
31 -----BEGIN CERTIFICATE-----
32 <snip>
33 -----END CERTIFICATE-----
34 conditions:
35 - all:
36 - key: "{{ components[].licenses[].expression }}"
37 operator: AllIn
38 value: ["GPL-3.0"]

Cosign 2.0 Support

Kyverno 1.11 also adds support for Cosign 2.0 and all the behavioral changes in Cosign 2.0 are supported in Kyverno. Tlogs and SCTs are now verified by default and can be turned off using rekor.ignoreTlogs and ctlog.IgnoreSCT attributes in a policy. Rekor public keys and ctlog public keys can now be specified in policies to verify Tlogs and SCT timestamps without setting up TUF for Sigstore or passing a Rekor URL.

Full support for private Sigstore deployments has also been added. Private Sigstore deployments can be used by passing the TUF root and mirror in the Kyverno deployment or Helm chart.

Image Verification Cache

Image verification requires multiple network calls and can be time consuming. We have also added caching for image verification that will cache successful image verification outcomes. It is a TTL based cache and the size and TTL configuration can be configured in the deployment.

Note: Users upgrading from Kyverno v1.10 to v1.11 who have image verification policies using cosign will have to explicitly disable Tlogs and SCT verification in their policy using the rekor.ignoreTlogs and ctlog.IgnoreSCT fields if they did not use Rekor while signing the image.

Cleanup via TTL label

The cleanup ability was introduced in Kyverno 1.9 via the cleanup type of policy. This release introduced another option to clean up resources via a reserved time-to-live (TTL) label cleanup.kyverno.io/ttl. By assigning this label to any resource with required permissions granted to Kyverno, the resource will be removed at the designated time. In this release, the cleanup policy no longer relies on CronJobs to perform the job.

For example, creation of this Pod will cause Kyverno to clean it up after two minutes and without the presence of a cleanup policy.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4 labels:
 5 cleanup.kyverno.io/ttl: 2m
 6 name: foo
 7spec:
 8 containers:
 9 - args:
10 - sleep
11 - 1d
12 image: busybox:1.35
13 name: foo

Because this is a label, there is opportunity to chain other Kyverno functionality around it. For example, it is possible to use a Kyverno mutate rule to assign this label to matching resources. A validate rule could be written prohibiting, for example, users from the infra-ops group from assigning the label to resources in certain Namespaces. Or, Kyverno could generate a new resource with this label as part of the resource definition.

CLI Refactoring and New Test Schema

In Kyverno 1.11, we have made significant improvements to the CLI, enhancing its stability and usability. A new test manifest schema was introduced to the Kyverno test command, now you can validate your kyverno-test.yaml and get helpful error reports during execution. Here’s a snippet of a kyverno-test.yaml and an error is displayed as part of the execution result.

 1apiVersion: cli.kyverno.io/v1alpha1
 2kind: Test
 3metadata:
 4 name: mytest
 5policies:
 6- policy.yaml
 7resources:
 8- pod.yaml
 9results:
10- policy: evil-policy-match-foreign-pods
11 rule: evil-validation
12 resource: nginx
13 status: pass

1$ kubectl-kyverno test ./scratch/cli
2No test yamls available
3Test errors:
4 failed to load test file (json: cannot unmarshal array into Go value of type api.Test)

Moreover, we have added three new commands to the Kyverno CLI: create, docs and an experimental fix. The create command creates various resources that can be used for the Kyverno CLI, including kyverno-test.yaml and the values file that are used for the test command. With the fix command, you can now easily resolve any issues and ensure that your Kyverno resources are up-to-date and optimized. The docs command enables automatic generation of comprehensive documentation for the Kyverno CLI. It makes it a lot easier for users to access the information they need and stay up-to-date with all the CLI capabilities.

Other Additions

As with previous minor releases, certain features have progressed to the next phase. The PolicyExceptions and Cleanup policies moved from alpha to beta status, and PolicyExceptions are enabled by default in 1.11.

The Kyverno CLI can now be installed via a GitHub Action, which currently supports GitHub-provided Linux, macOS and Windows.

Kyverno JSON

Although not part of 1.11.0, the Kyverno team launched a new sub-project Kyverno JSON, which extends Kyverno beyond Kubernetes. Now, platform engineering teams can use Kyverno’s declarative policies to validate any JSON payload including Terraform files, Dockerfiles, Cloud configurations and service authorization requests.

Kyverno JSON can be consumed as a CLI, a Golang API, or a web service with a REST API. Read more at: https://www.cncf.io/blog/2023/11/06/kyverno-expands-beyond-kubernetes/.

Kyverno Chainsaw

Another sub-project launched was Kyverno Chainsaw, an end-to-end declarative test tool for Kubernetes controllers. Chainsaw emerged from a real need for testing Kyverno controllers and policy behaviors in continuous integration environments. You can learn more about it at: https://kyverno.github.io/chainsaw/.

Security Hardening

In addition to the mentioned features, enhancements, and fixes, the Kyverno project completed a fuzzing security audit and is undergoing a thorough third-party security review. The security review is being conducted in collaboration with the CNCF, Ada Logics and OSTIF.

The review exposed a high severity vulnerability CVE-2023-47630 involving the possibility of users consuming insecure images pulled from a compromised registry. In addition four CVEs CVE-2023-42813, CVE-2023-42814, CVE-2023-42815, CVE-2023-42816 were identified in unreleased code (i.e. on the main branch), which could potentially allow an attacker to cause a denial of service of Kyverno by exploiting the Notary verifier.

Closing

Kyverno 1.11 is here, and we want to express our heartfelt gratitude to all the contributors who made this release possible! This release is jam-packed with awesome new features and improvements that we can’t wait for you to try out. For a more complete list of all the features, changes, and fixes, please see the release notes on GitHub.

Join our community on Kubernetes Slack, attend our community meetings, and connect with us on Twitter. Thank you for your support, and we can’t wait to see what amazing things you’ll achieve with Kyverno 1.11!

Blog: Using CEL Expressions in Kyverno Policies

Mon, 13 Nov 2023 00:00:00 +0000

Kyverno, in simple terms, is a policy engine for Kubernetes that can be used to describe policies and validate resource requests against those policies. It allows us to create policies for our Kubernetes cluster on different levels. It enables us to validate, change, and create resources based on our defined policies.

A Kyverno policy is a collection of rules. Whenever we receive an API request to our Kubernetes cluster, we validate it with a set of rules.

A policy consists of different clauses, such as:

Match: It selects the resources to be included in a rule.
Exclude: It selects a subset of the resources from the match block which should be excluded from a rule.

Match and Exclude are used to select resources, users, user groups, service accounts, namespaced roles, and cluster-wide roles.

Validate: It validates the properties of the new resource, and it is created if it matches what is declared in the rule.
Mutate: It modifies matching resources.
Generate: It creates additional resources.
Verify Images: It verifies container image signatures using Cosign and Notary.

Refer to Selecting Resources for more information.

Each rule can contain only a single validate, mutate, generate, or verifyImages child declaration.

In this post, I will show you how to write CEL expressions in Kyverno policies for resource validation. Common Expression Language (CEL) was first introduced to Kubernetes for the validation rules for CustomResourceDefinitions, and then it was used by Kubernetes ValidatingAdmissionPolicies in 1.26.

CEL Expressions in validate rules

Creating a policy to disallow host paths for Deployments

The below policy ensures no hostPath volumes are in use for Deployments.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: disallow-host-path
 6spec:
 7 validationFailureAction: Enforce
 8 background: false
 9 rules:
10 - name: host-path
11 match:
12 any:
13 - resources:
14 kinds:
15 - Deployment
16 validate:
17 cel:
18 expressions:
19 - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
20 message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
21EOF

spec.rules.validate.cel contains CEL expressions that use the Common Expression Language (CEL) to validate the request. If an expression evaluates to false, the validation check is enforced according to the spec.validationFailureAction field.

Now, let’s try deploying an app that uses a hostPath:

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: nginx
 6spec:
 7 replicas: 2
 8 selector:
 9 matchLabels:
10 app: nginx
11 template:
12 metadata:
13 labels:
14 app: nginx
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19 volumeMounts:
20 - name: udev
21 mountPath: /data
22 volumes:
23 - name: udev
24 hostPath:
25 path: /etc/udev
26EOF

We can see that our policy is enforced. Great!

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:
resource Deployment/default/nginx was blocked due to the following policies
disallow-host-path:
host-path: HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath
must be unset.

Creating a policy to check StatefulSet Namespaces

The below policy ensures that any StatefulSet is created in the production Namespace

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: check-statefulset-namespace
 6spec:
 7 validationFailureAction: Enforce
 8 background: false
 9 rules:
10 - name: statefulset-namespace
11 match:
12 any:
13 - resources:
14 kinds:
15 - StatefulSet
16 validate:
17 cel:
18 expressions:
19 - expression: "namespaceObject.metadata.name == 'production'"
20 message: "The StatefulSet must be created in the 'production' namespace."
21EOF

Let’s try creating a StatefulSet in the default Namespace.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: StatefulSet
 4metadata:
 5 name: bad-statefulset
 6spec:
 7 replicas: 1
 8 selector:
 9 matchLabels:
10 app: app
11 template:
12 metadata:
13 labels:
14 app: app
15 spec:
16 containers:
17 - name: container2
18 image: nginx
19EOF

As expected, the Statefulset creation is blocked because it violates the rule

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:
resource StatefulSet/default/bad-statefulset was blocked due to the following policies
check-statefulset-namespace:
statefulset-namespace: The StatefulSet must be created in the 'production' namespace.

Let’s create a Statefulset in the production Namespace.

 1kubectl apply -f - << EOF
 2apiVersion: apps/v1
 3kind: StatefulSet
 4metadata:
 5 name: good-statefulset
 6 namespace: production
 7spec:
 8 replicas: 1
 9 selector:
10 matchLabels:
11 app: app
12 template:
13 metadata:
14 labels:
15 app: app
16 spec:
17 containers:
18 - name: container2
19 image: nginx
20EOF

The StatefulSet is successfully created. Great!

statefulset.apps/good-statefulset created

In the previous two examples, we have used object in CEL expressions which refers to the incoming object and namespaceObject which refers to the Namespace that the incoming object belongs to.

Some other useful variables that we can use in CEL expressions are

oldObject: The existing object. The value is null for CREATE requests.
authorizer: It can be used to perform authorization checks.
authorizer.requestResource: A shortcut for an authorization check configured with the request resource (group, resource, (subresource), namespace, name).

CEL Preconditions in Kyverno Policies

The below policy ensures the hostPort field is set to a value between 5000 and 6000 for pods whose metadata.name set to nginx

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: disallow-host-port-range
 6spec:
 7 validationFailureAction: Enforce
 8 background: false
 9 rules:
10 - name: host-port-range
11 match:
12 any:
13 - resources:
14 kinds:
15 - Pod
16 celPreconditions:
17 - name: "first match condition in CEL"
18 expression: "object.metadata.name.matches('nginx')"
19 validate:
20 cel:
21 expressions:
22 - expression: "object.spec.containers.all(container, !has(container.ports) || container.ports.all(port, !has(port.hostPort) || (port.hostPort >= 5000 && port.hostPort <= 6000)))"
23 message: "The only permitted hostPorts are in the range 5000-6000."
24EOF

spec.rules.celPreconditions are CEL expressions. All celPreconditions must be evaluated to true for the resource to be evaluated. Therefore, any Pod with nginx in its metadata.name will be evaluated.

Let’s try deploying an Apache server with hostPort set to 80.

 1kubectl apply -f - <<EOF
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5 name: apache
 6spec:
 7 containers:
 8 - name: apache-server
 9 image: httpd
10 ports:
11 - containerPort: 8080
12 hostPort: 80
13EOF

You’ll see that it’s successfully created because the validation rule wasn’t applied on the new Pod as it doesn’t satisfy the celPreconditions. That’s exactly what we need.

Pod/apache created

Let’s try deploying an Nginx server with hostPort set to 80.

 1kubectl apply -f - <<EOF
 2apiVersion: v1
 3kind: Pod
 4metadata:
 5 name: nginx
 6spec:
 7 containers:
 8 - name: nginx-server
 9 image: nginx
10 ports:
11 - containerPort: 8080
12 hostPort: 80
13EOF

Since the new Pod satisfies the celPreconditions, the validation rule will be applied. As a result, the creation of the Pod will be blocked as it violates the rule.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/default/nginx was blocked due to the following policies
disallow-host-port-range:
host-port-range: The only permitted hostPorts are in the range 5000-6000.

Parameter Resources in Kyverno Policies

The below policy ensures the deployment replicas are less than a specific value. This value is defined in a parameter resource.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: check-deployment-replicas
 6spec:
 7 validationFailureAction: Enforce
 8 background: false
 9 rules:
10 - name: deployment-replicas
11 match:
12 any:
13 - resources:
14 kinds:
15 - Deployment
16 validate:
17 cel:
18 paramKind:
19 apiVersion: rules.example.com/v1
20 kind: ReplicaLimit
21 paramRef:
22 name: "replica-limit-test.example.com"
23 parameterNotFoundAction: "Deny"
24 expressions:
25 - expression: "object.spec.replicas <= params.maxReplicas"
26 messageExpression: "'Deployment spec.replicas must be less than ' + string(params.maxReplicas)"
27EOF

The cel.paramKind and cel.paramRef specify the resource used to parameterize this policy. For this example, it is configured by ReplicaLimit custom resources.

The ReplicaLimit could be as follows:

1kubectl apply -f - <<EOF
2apiVersion: rules.example.com/v1
3kind: ReplicaLimit
4metadata:
5 name: "replica-limit-test.example.com"
6maxReplicas: 3
7EOF

Here’s the corresponding custom resource definition:

 1kubectl apply -f - <<EOF
 2apiVersion: apiextensions.k8s.io/v1
 3kind: CustomResourceDefinition
 4metadata:
 5 name: replicalimits.rules.example.com
 6spec:
 7 group: rules.example.com
 8 names:
 9 kind: ReplicaLimit
10 plural: replicalimits
11 scope: Namespaced
12 versions:
13 - name: v1
14 served: true
15 storage: true
16 schema:
17 openAPIV3Schema:
18 type: object
19 properties:
20 apiVersion:
21 type: string
22 kind:
23 type: string
24 metadata:
25 type: object
26 maxReplicas:
27 type: integer
28EOF

Now, let’s try deploying an app with five replicas.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: nginx
 6spec:
 7 replicas: 5
 8 selector:
 9 matchLabels:
10 app: nginx
11 template:
12 metadata:
13 labels:
14 app: nginx
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19EOF

As expected, the deployment creation will be blocked because it violates the rule.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:
resource Deployment/default/nginx was blocked due to the following policies
check-deployment-replicas:
deployment-replicas: Deployment spec.replicas must be less than 3

Let’s try deploying an app with two replicas.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: nginx
 6spec:
 7 replicas: 2
 8 selector:
 9 matchLabels:
10 app: nginx
11 template:
12 metadata:
13 labels:
14 app: nginx
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19EOF

The deployment is created successfully. Great!

deployment.apps/nginx created

CEL Variables in Kyverno Policies

If an expression grows too complicated, or part of the expression is reusable and computationally expensive to evaluate, We can extract some parts of the expressions into variables. A variable is a named expression that can be referred later as variables in other expressions.

The order of variables is important because a variable can refer to other variables defined before it. This ordering prevents circular references.

The below policy enforces that image repo names match the environment defined in its Namespace. It enforces that all containers of deployment have the image repo match the environment label of its Namespace except for “exempt” deployments or any containers that do not belong to the “example.com” organization (e.g., common sidecars). For example, if the Namespace has a label of {“environment”: “staging”}, all container images must be either staging.example.com/* or do not contain “example.com” at all, unless the deployment has {“exempt”: “true”} label.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: image-matches-namespace-environment.policy.example.com
 6spec:
 7 validationFailureAction: Enforce
 8 background: false
 9 rules:
10 - name: image-matches-namespace-environment
11 match:
12 any:
13 - resources:
14 kinds:
15 - Deployment
16 validate:
17 cel:
18 variables:
19 - name: environment
20 expression: "'environment' in namespaceObject.metadata.labels ? namespaceObject.metadata.labels['environment'] : 'prod'"
21 - name: exempt
22 expression: "has(object.metadata.labels) && 'exempt' in object.metadata.labels && object.metadata.labels['exempt'] == 'true'"
23 - name: containers
24 expression: "object.spec.template.spec.containers"
25 - name: containersToCheck
26 expression: "variables.containers.filter(c, c.image.contains('example.com/'))"
27 expressions:
28 - expression: "variables.exempt || variables.containersToCheck.all(c, c.image.startsWith(variables.environment + '.'))"
29 messageExpression: "'only ' + variables.environment + ' images are allowed in namespace ' + namespaceObject.metadata.name"
30EOF

Let’s start with creating a Namespace that has a label of environment: staging

1kubectl apply -f - <<EOF
2apiVersion: v1
3kind: Namespace
4metadata:
5 name: staging-ns
6 labels:
7 environment: staging
8EOF

And then create a deployment whose image is example.com/nginx in the staging-ns Namespace.

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: deployment-fail
 6 namespace: staging-ns
 7spec:
 8 replicas: 1
 9 selector:
10 matchLabels:
11 app: app
12 template:
13 metadata:
14 labels:
15 app: app
16 spec:
17 containers:
18 - name: container2
19 image: example.com/nginx
20EOF

As expected, the deployment creation will be blocked since its image must be staging.example.com/nginx

Let’s try setting the deployment image to staging.example.com/nginx instead

 1kubectl apply -f - <<EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: deployment-pass
 6 namespace: staging-ns
 7spec:
 8 replicas: 1
 9 selector:
10 matchLabels:
11 app: app
12 template:
13 metadata:
14 labels:
15 app: app
16 spec:
17 containers:
18 - name: container2
19 image: staging.example.com/nginx
20EOF

The deployment is created successfully. Great!

deployment.apps/deployment-pass created

Auto-Gen Rules for CEL Expressions

Since Kubernetes has many higher-level controllers that directly or indirectly manage Pods: Deployment, DaemonSet, StatefulSet, Job, and CronJob resources, it’d be inefficient to write a policy that targets Pods and every higher-level controller. Kyverno solves this issue by supporting the automatic generation of policy rules for higher-level controllers from a rule written exclusively for a Pod.

Check the autogen rules for more information.

For example, when creating a validation policy like below, which disallows latest image tags, the policy applies to all resources capable of generating Pods.

 1kubectl apply -f - <<EOF
 2apiVersion: kyverno.io/v1
 3kind: ClusterPolicy
 4metadata:
 5 name: disallow-latest-tag
 6spec:
 7 validationFailureAction: Enforce
 8 rules:
 9 - name: disallow-latest-tag
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 validate:
16 cel:
17 expressions:
18 - expression: "object.spec.containers.all(container, !container.image.contains('latest'))"
19 message: "Using a mutable image tag e.g. 'latest' is not allowed."
20EOF

Once the policy is created, these other resources can be shown in auto-generated rules which Kyverno adds to the policy under the status object.

 1status:
 2 autogen:
 3 rules:
 4 - exclude:
 5 resources: {}
 6 generate:
 7 clone: {}
 8 cloneList: {}
 9 match:
10 any:
11 - resources:
12 kinds:
13 - DaemonSet
14 - Deployment
15 - Job
16 - StatefulSet
17 - ReplicaSet
18 - ReplicationController
19 resources: {}
20 mutate: {}
21 name: autogen-disallow-latest-tag
22 validate:
23 cel:
24 expressions:
25 - expression: object.spec.template.spec.containers.all(container, !container.image.contains('latest'))
26 message: Using a mutable image tag e.g. 'latest' is not allowed.
27 - exclude:
28 resources: {}
29 generate:
30 clone: {}
31 cloneList: {}
32 match:
33 any:
34 - resources:
35 kinds:
36 - CronJob
37 resources: {}
38 mutate: {}
39 name: autogen-cronjob-disallow-latest-tag
40 validate:
41 cel:
42 expressions:
43 - expression: object.spec.jobTemplate.spec.template.spec.containers.all(container,
44 !container.image.contains('latest'))
45 message: Using a mutable image tag e.g. 'latest' is not allowed.

Let’s try creating an nginx deployment with the latest tag.

 1kubectl apply -f - << EOF
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: nginx-deployment
 6 labels:
 7 app: nginx
 8spec:
 9 replicas: 3
10 selector:
11 matchLabels:
12 app: nginx
13 template:
14 metadata:
15 labels:
16 app: nginx
17 spec:
18 containers:
19 - name: nginx
20 image: nginx:latest
21EOF

As expected the deployment creation is blocked.

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:
resource Deployment/default/nginx-deployment was blocked due to the following policies
disallow-latest-tag:
autogen-disallow-latest-tag: Using a mutable image tag e.g. 'latest' is not allowed.

Conclusion

This blog post explains how to use CEL expressions in Kyverno policies to validate resources covering all the features introduced in Kubernetes ValidatingAdmissionPolicies. Stay tuned for our next post, where we’ll show you how to generate Kubernetes ValidatingAdmissionPolicies from Kyverno policies.

Blog: Applying Validating Admission Policies using Kyverno CLI

Wed, 04 Oct 2023 00:00:00 +0000

The Kyverno Command Line Interface (CLI) allows applying policies outside of Kubernetes clusters and can validate and test policy behavior prior to adding them to a cluster.

The two commands used for testing are apply and test:

The apply command is used to perform a dry run on one or more policies for the given manifest(s).
The test command is used to test a given set of resources against one or more policies to check the desired results defined in a special test manifest.

In this post, I will show you how you can apply/test Kubernetes ValidatingAdmissionPolicies that were first introduced in 1.26 with the enhancements to the Kyverno CLI in v1.11.

Applying ValidatingAdmissionPolicies using kyverno apply

In this section, you will create a ValidatingAdmissionPolicy that checks the number of Deployment replicas. You will then apply this policy to two Deployments, one of which violates the policy:

 1cat << EOF > check-deployment-replicas.yaml
 2apiVersion: admissionregistration.k8s.io/v1alpha1
 3kind: ValidatingAdmissionPolicy
 4metadata:
 5 name: check-deployments-replicas
 6spec:
 7 failurePolicy: Fail
 8 matchConstraints:
 9 resourceRules:
10 - apiGroups: ["apps"]
11 apiVersions: ["v1"]
12 operations: ["CREATE", "UPDATE"]
13 resources: ["deployments"]
14 validations:
15 - expression: "object.spec.replicas <= 2"
16 message: "Replicas must be less than or equal 2"
17EOF

The following deployment satisfies the rules declared in the above policy.

 1cat << EOF > deployment-pass.yaml
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: nginx-pass
 6spec:
 7 replicas: 2
 8 selector:
 9 matchLabels:
10 app: nginx-pass
11 template:
12 metadata:
13 labels:
14 app: nginx-pass
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19EOF

Let’s apply the policy to the resource using kyverno apply as follows.

1kyverno apply ./check-deployment-replicas.yaml --resource deployment-pass.yaml

The output should be the following.

1Applying 1 policy rule(s) to 1 resource(s)...
2
3pass: 1, fail: 0, warn: 0, error: 0, skip: 0

Let’s try to create another deployment that violates the policy.

 1cat << EOF > deployment-fail.yaml
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: nginx-fail
 6spec:
 7 replicas: 3
 8 selector:
 9 matchLabels:
10 app: nginx-fail
11 template:
12 metadata:
13 labels:
14 app: nginx-fail
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19EOF

Then apply the policy to the resource as follows.

1kyverno apply ./check-deployment-replicas.yaml --resource deployment-fail.yaml

The output should be as shown.

1Applying 1 policy rule(s) to 1 resource(s)...
2
3pass: 0, fail: 1, warn: 0, error: 0, skip: 0

Testing ValidatingAdmissionPolicies using kyverno test

In this section, you will create a ValidatingAdmissionPolicy that ensures no hostPath volumes are in use for Deployments. You will then create two Deployments to test them against the policy and check the desired results.

To proceed, you need to create a directory containing the necessary manifests. In this example, I created a directory called test-dir.

Let’s start with creating the policy.

 1cat << EOF > ./test-dir/disallow-host-path.yaml
 2apiVersion: admissionregistration.k8s.io/v1alpha1
 3kind: ValidatingAdmissionPolicy
 4metadata:
 5 name: disallow-host-path
 6spec:
 7 failurePolicy: Fail
 8 matchConstraints:
 9 resourceRules:
10 - apiGroups: ["apps"]
11 apiVersions: ["v1"]
12 operations: ["CREATE", "UPDATE"]
13 resources: ["deployments"]
14 validations:
15 - expression: "!has(object.spec.template.spec.volumes) || object.spec.template.spec.volumes.all(volume, !has(volume.hostPath))"
16 message: "HostPath volumes are forbidden. The field spec.template.spec.volumes[*].hostPath must be unset."
17EOF

Then, create the two Deployments, one of which violates the policy.

 1cat << EOF > ./test-dir/deployments.yaml
 2apiVersion: apps/v1
 3kind: Deployment
 4metadata:
 5 name: deployment-pass
 6spec:
 7 replicas: 1
 8 selector:
 9 matchLabels:
10 app: nginx
11 template:
12 metadata:
13 labels:
14 app: nginx
15 spec:
16 containers:
17 - name: nginx-server
18 image: nginx
19 volumeMounts:
20 - name: temp
21 mountPath: /scratch
22 volumes:
23 - name: temp
24 emptyDir: {}
25---
26apiVersion: apps/v1
27kind: Deployment
28metadata:
29 name: deployment-fail
30spec:
31 replicas: 1
32 selector:
33 matchLabels:
34 app: nginx
35 template:
36 metadata:
37 labels:
38 app: nginx
39 spec:
40 containers:
41 - name: nginx-server
42 image: nginx
43 volumeMounts:
44 - name: udev
45 mountPath: /data
46 volumes:
47 - name: udev
48 hostPath:
49 path: /etc/udev
50EOF

The tests are defined in a file named kyverno-test.yaml so you will create two tests, one for each Deployment and test them against the policy. Notice the use of a new field in the test manifest called isValidatingAdmissionPolicy.

 1cat << EOF > ./test-dir/kyverno-test.yaml
 2name: disallow-host-path-test
 3policies:
 4 - disallow-host-path.yaml
 5resources:
 6 - deployments.yaml
 7results:
 8 - policy: disallow-host-path
 9 resource: deployment-pass
10 isValidatingAdmissionPolicy: true
11 kind: Deployment
12 result: pass
13 - policy: disallow-host-path
14 resource: deployment-fail
15 isValidatingAdmissionPolicy: true
16 kind: Deployment
17 result: fail
18EOF

Now, we’re ready to test the two Deployments against a ValidatingAdmissionPolicy.

1kyverno test ./test-dir

The output should be as shown below.

 1Executing disallow-host-path-test...
 2
 3│────│────────────────────│──────│────────────────────────────│────────│────────│
 4│ ID │ POLICY │ RULE │ RESOURCE │ RESULT │ REASON │
 5│────│────────────────────│──────│────────────────────────────│────────│────────│
 6│ 1 │ disallow-host-path │ │ Deployment/deployment-pass │ Pass │ Ok │
 7│ 2 │ disallow-host-path │ │ Deployment/deployment-fail │ Pass │ Ok │
 8│────│────────────────────│──────│────────────────────────────│────────│────────│
 9
10Test Summary: 2 tests passed and 0 tests failed

As expected, the two tests passed because the actual result of each test matches the desired result as defined in the test manifest.

Conclusion

This blog post explains how to apply ValidatingAdmissionPolicies to resources using the Kyverno CLI. With Kyverno, it’s easy to apply Kubernetes ValidatingAdmissionPolicies in your CI/CD pipelines and to test new ValidatingAdmissionPolicies before they are deployed to your clusters. This is one of many exciting features coming with Kyverno v1.11.

Blog: Kyverno Completes Fuzzing Security Audit

Wed, 06 Sep 2023 00:00:00 +0000

Kyverno, a CNCF policy engine for Kubernetes, is happy to announce the completion of its fuzzing security audit. The audit was carried out by Ada Logics and is part of an initiative by the CNCF to bring fuzzing to the CNCF landscape; Fuzzing is an important part in keeping CNCF projects secure and robust, and it has found security vulnerabilities and reliability issues in several other CNCF-hosted projects. The audit spanned July and August of 2023 and resulted in 15 fuzzers written for the Kyverno project. The fuzzers found three bugs during the audit itself and OSS-Fuzz will continue to run them after the audit has concluded to test Kyverno for bugs and vulnerabilities.

Read the full report for the audit here: Kyverno Fuzzing Security Audit Report.

Fuzzing

Fuzzing is a technique for finding software bugs in an automated manner. In Go, this includes bugs such as nil-dereference, out-of-bounds panics, out-of-range panics, out-of-memory panics as well as logic bugs. To test your software by way of fuzzing, you need two things:

A fuzzing engine
A test harness

The fuzzing engine generates the input test cases that it makes available in the test harness. It is the user’s job to set up the harness such that it passes the test cases on to the API that the harness is meant to test. Kyverno is implemented in Go, and the auditors wrote the fuzz tests using the standard library fuzzing engine. Once the user has written a harness and runs it, the fuzzer runs indefinitely until it detects a crash or the user stops it. Each time the harness runs (each fuzz iteration), the engine will generate a new test case by mutating over a collection of test cases that explore new code. This collection of test cases is called a corpus.

Kyverno fuzzers

All of the fuzzers from the audit are hosted in Kyverno’s repository in the package directory that they test. Ada Logics assessed the Kyverno code base by way of static callgraph analysis to determine the optimal entrypoints for the fuzzers and found that the validation routines for Validate, Mutate and Verify Image rules reach a large part of the Kyverno source tree. They then wrote a fuzzer for each routine which passes both a valid Kyverno Cluster Policy as well pseduo-random unstructured Kubernetes resource, testing these validation routines for both reliability and security issues. One of the issues that the fuzzers found during the audit was triggered by Kyverno’s mishandling of the input resource. The Validate, Mutate and Verify Image fuzzers require a substantial setup depending on multiple mocked types, and Ada Logics did several iterations to improve their effectiveness. This included several code optimizations in the fuzzers including how the fuzzers generate policy rules and the Kubernetes resources before passing them onto the entrypoints. Another optimization was to add a dictionary; Native Go fuzzing does not support dictionaries, but OSS-Fuzz does when running the fuzzers. A dictionary is a list of tokens that the fuzzing engine will mutate over in the first iterations when starting to run a fuzzer, and it can be a great resource to get a fuzzer started with meaningful test cases. In Kyverno’s case, Ada Logics scraped the JSON tags of several Kubernetes resource types so that the fuzzers can use these when generating the Kubernetes resources.

Besides the far-reaching Validate, Mutate and Verify Image fuzzers, the auditors added fuzzers for other high-level APIs and complex targets in Kyverno.

All of Kyverno’s fuzzers run continuously on OSS-Fuzz that dedicates vast resources to run Kyvernos fuzzer at scale. When a fuzzer finds a crash, OSS-Fuzz notifies the Kyverno maintainers with a crash report and a stack trace. Kyvernos OSS-Fuzz build script can be found here.

The fuzzers found three bugs during the audit. They were all found by OSS-Fuzz after the fuzzers started running continuously. Ada Logics fixed the three issues. The report includes notes from the triaging of the issues and links to the fixes.

Looking forward

The Kyverno fuzzers continue to test the Kyverno code base for old, hard-to-find bugs. Fixing the three issues found during the audit allowed the fuzzers to progress deeper into the logic of Kyverno and may reveal new issues that exist today but take longer to find. For this purpose, Kyverno is backed by OSS-Fuzz which dedicates excessive compute resources to the Kyverno fuzzing suite. OSS-Fuzz will also test for new bugs that get introduced into Kyverno after the fuzzing security audit has concluded, since OSS-Fuzz tests the latest Kyverno main branch every time it builds the fuzzers. In addition to running the fuzzers, OSS-Fuzz also automatically tests whether previously found crashes have been fixed and will also mark them as resolved when needed.

Blog: Verifying images in a private Amazon ECR with Kyverno and IAM Roles for Service Accounts (IRSA)

Fri, 18 Aug 2023 00:00:00 +0000

When running workloads in Amazon Elastic Kubernetes Service (EKS), it is essential to ensure supply chain security by verifying container image signatures and other metadata. To achieve this, you can configure Kyverno, a CNCF policy engine designed for Kubernetes, to pull from ECR private registries for image verification. It’s possible to pass in the credentials via secrets, but that can get difficult to manage and automate across multiple clusters. In this blog post, we will explore an alternative method that simplifies the authentication process by leveraging Kyverno and IRSA (IAM Roles for Service Accounts) in EKS for image verification.

Applications, such as Kyverno, running within a Pod’s containers can utilize the AWS SDK to make API requests to AWS services by leveraging AWS Identity and Access Management (IAM) permissions. IAM roles for service accounts enable the management of credentials for these applications. Instead of manually creating and distributing AWS credentials to the containers, you can associate an IAM role with a Kubernetes service account and configure your Pods to utilize this service account. The detailed steps for this process can be found in the documentation. In this blog, we will guide you through the complete process of enabling IAM roles for the Kyverno service account and demonstrate how to verify this using the Kyverno verifyImages rule.

Setting up the EKS Cluster

First, you need to create an EKS cluster. You can then use the AWS CLI to update the kubeconfig file with the cluster details:

1$ aws eks update-kubeconfig --region us-west-2 --name kyverno-irsa
2Added new context arn:aws:eks:us-west-2:xxxxxxxxxxxx:cluster/kyverno-irsa to /Users/kyverno/.kube/config

Once the kubeconfig is updated, you can verify the cluster by running the following command:

1$ kubectl get node
2NAME STATUS ROLES AGE VERSION
3ip-172-31-56-181.us-west-2.compute.internal Ready <none> 1h v1.27.3-eks-a5565ad

Note: when you use IRSA, it updates the credential chain of the pod to use the IRSA token, however, the pod can still inherit the rights of the instance profile assigned to the worker node. You need to block access to instance metadata to prevent pods that do not use IRSA from inheriting the role assigned to the worker node.

You can follow this guidance to restrict access via the following command, for example:

1aws ec2 modify-instance-metadata-options --instance-id <instance-id> --http-tokens required --http-put-response-hop-limit 1

Installing Kyverno

Once you have the cluster set up, you can use Helm to install Kyverno into the cluster:

1helm upgrade --install kyverno kyverno/kyverno --namespace kyverno --create-namespace

Enabling IAM roles for service accounts

Creating an IAM OIDC Provider for the Cluster

To enable IRSA, you need to create an IAM OIDC provider for the EKS cluster. You can retrieve the OIDC issuer URL using the AWS CLI. The following command retrieves the provider for the cluster kyverno-irsa, replace it with your own cluster name:

1export cluster_name=kyverno-irsa
2oidc_id=$(aws eks describe-cluster --name $cluster_name --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

Determine whether an IAM OIDC provider with your cluster’s ID is already in your account.

1aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4

If output is returned, then you already have an IAM OIDC provider for your cluster. If no output is returned, then there’s no OIDC provider is associated with the cluster. You can create one using the eksctl command:

1$ eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve
22023-08-14 21:16:31 [ℹ] will create IAM Open ID Connect provider for cluster "kyverno-irsa" in "us-west-2"
32023-08-14 21:16:33 [✔] created IAM Open ID Connect provider for cluster "kyverno-irsa" in "us-west-2"

Configuring a Kubernetes Service Account to Assume an IAM Role

To associate an IAM role with a Kubernetes service account, you need to create an IAM policy for your IAM role. If you want to associate an existing IAM policy, you can skip this step.

Setup a custom policy with the following permissions, note that in production its best to not use a wildcard and specify resources:

 1cat >notation-signer-policy.json <<EOF
 2{
 3 "Version": "2012-10-17",
 4 "Statement": [
 5 {
 6 "Effect": "Allow",
 7 "Action": [
 8 "signer:GetSigningProfile",
 9 "signer:ListSigningProfiles",
10 "signer:SignPayload",
11 "signer:GetRevocationStatus",
12 "signer:DescribeSigningJob",
13 "signer:ListSigningJobs"
14 ],
15 "Resource": "*"
16 }
17 ]
18}
19EOF

Create the IAM policy:

1aws iam create-policy --policy-name notation-signer-policy --policy-document file://notation-signer-policy.json

To configure a Kubernetes service account to assume an IAM role, you can use the eksctl command to create an IAM service account.

If your Kyverno is installed with default configurations, you can run the following command directly to create the IAM service account. Otherwise, replace the service account name and namespace with your custom values.

 1$ eksctl create iamserviceaccount --override-existing-serviceaccounts kyverno-admission-controller --namespace kyverno --cluster kyverno-irsa --role-name kyverno-irsa --attach-policy-arn arn:aws:iam::xxxxxxxxxxxx:policy/notation-signer-policy --approve
 22023-08-14 21:18:17 [ℹ] 1 iamserviceaccount (kyverno/kyverno-admission-controller) was included (based on the include/exclude rules)
 32023-08-14 21:18:17 [!] metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
 42023-08-14 21:18:17 [ℹ] 1 task: {
 5 2 sequential sub-tasks: {
 6 create IAM role for serviceaccount "kyverno/kyverno-admission-controller",
 7 create serviceaccount "kyverno/kyverno-admission-controller",
 8 } }2023-08-14 21:18:17 [ℹ] building iamserviceaccount stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
 92023-08-14 21:18:17 [ℹ] deploying stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
102023-08-14 21:18:18 [ℹ] waiting for CloudFormation stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
112023-08-14 21:18:54 [ℹ] waiting for CloudFormation stack "eksctl-kyverno-irsa-addon-iamserviceaccount-kyverno-kyverno-admission-controller"
122023-08-14 21:18:55 [ℹ] serviceaccount "kyverno/kyverno-admission-controller" already exists
132023-08-14 21:18:55 [ℹ] updated serviceaccount "kyverno/kyverno-admission-controller"

After creating the IAM service account, you can verify that the role and service account are configured correctly.

Confirm that the IAM role’s trust policy is configured correctly:

 1$ aws iam get-role --role-name kyverno-irsa --query Role.AssumeRolePolicyDocument
 2{
 3 "Version": "2012-10-17",
 4 "Statement": [
 5 {
 6 "Effect": "Allow",
 7 "Principal": {
 8 "Federated": "arn:aws:iam::xxxxxxxxxxxx:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/2EA2DE9A6C72778FA517C24D7BBE2916"
 9 },
10 "Action": "sts:AssumeRoleWithWebIdentity",
11 "Condition": {
12 "StringEquals": {
13 "oidc.eks.us-west-2.amazonaws.com/id/2EA2DE9A6C72778FA517C24D7BBE2916:aud": "sts.amazonaws.com",
14 "oidc.eks.us-west-2.amazonaws.com/id/2EA2DE9A6C72778FA517C24D7BBE2916:sub": "system:serviceaccount:kyverno:kyverno-admission-controller"
15 }
16 }
17 }
18 ]
19}

Confirm that the policy that you attached to your role in a previous step is attached to the role:

1$ aws iam list-attached-role-policies --role-name kyverno-irsa --query AttachedPolicies --output text
2arn:aws:iam::xxxxxxxxxxxx:policy/notation-signer-policy notation-signer-policy

Confirm that the Kyverno service account is annotated with the role:

1$ kubectl describe serviceaccount kyverno-admission-controller -n kyverno
2
3Name: kyverno-admission-controller
4Namespace: kyverno
5Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxxxx:role/kyverno-irsa

Confirm that the environment variables are injected to the admission controller:

 1$ kubectl get pod -n kyverno -l app.kubernetes.io/component=admission-controller -o yaml | grep AWS -A2
 2 - name: AWS_STS_REGIONAL_ENDPOINTS
 3 value: regional
 4 - name: AWS_DEFAULT_REGION
 5 value: us-west-2
 6 - name: AWS_REGION
 7 value: us-west-2
 8 - name: AWS_ROLE_ARN
 9 value: arn:aws:iam::xxxxxxxxxxxx:role/kyverno-irsa
10 - name: AWS_WEB_IDENTITY_TOKEN_FILE
11 value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token

If you do not see these environment variables, try restarting the pod to inject variables. If you still do not see these variables, follow this instruction to verify that your pod identity webhook configuration exists and is valid.

Verifying ECR private images using IRSA and Kyverno

To test IRSA works with Kyverno, you can create pods with signed and unsigned images respectively and verify container images signatures using the Kyverno policy. If the IAM role assumption is configured correctly, the pod should be deployed successfully. Otherwise, Kyverno will deny the request.

The test image used in this blog is signed by Notary. If you don’t have a signed images for testing, you can follow this guidance to sign a private ECR image using Notation.

You can inspect all signatures with Notation. The following is an inspection result of all signatures and signed artifacts for the test image xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1:

 1✗ notation inspect xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1
 2Inspecting all signatures for signed artifact
 3xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation@sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105
 4└── application/vnd.cncf.notary.signature
 5 └── sha256:86abf8af48c152f871a5ea56a62a9302e145760089db926420e72c1bbd0de07d
 6 ├── media type: application/jose+json
 7 ├── signature algorithm: RSASSA-PSS-SHA-256
 8 ├── signed attributes
 9 │ ├── signingScheme: notary.x509
10 │ └── signingTime: Fri Aug 11 16:37:40 2023
11 ├── user defined attributes
12 │ └── (empty)
13 ├── unsigned attributes
14 │ └── signingAgent: Notation/1.0.0
15 ├── certificates
16 │ └── SHA256 fingerprint: da1f2d7d648dfacc7ebd59f98a9f35c753c331d80ca4280bb94060f4af4a5357
17 │ ├── issued to: CN=test,O=Notary,L=Seattle,ST=WA,C=US
18 │ ├── issued by: CN=test,O=Notary,L=Seattle,ST=WA,C=US
19 │ └── expiry: Thu May 19 21:15:18 2033
20 └── signed artifact
21 ├── media type: application/vnd.docker.distribution.manifest.v2+json
22 ├── digest: sha256:b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105
23 └── size: 938

The following policy verifies the image signature for pods in test-shuting namespace, you can tune the policy to verify different images:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 annotations:
 5 pod-policies.kyverno.io/autogen-controllers: none
 6 name: test-irsa
 7spec:
 8 background: true
 9 rules:
10 - match:
11 resources:
12 kinds:
13 - Pod
14 namespaces:
15 - test-shuting
16 name: check-digest
17 verifyImages:
18 - attestors:
19 - count: 1
20 entries:
21 - certificates:
22 cert: |-
23 -----BEGIN CERTIFICATE-----
24 ...
25 ...
26 ...
27 -----END CERTIFICATE-----
28 imageReferences:
29 - xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting*
30 mutateDigest: true
31 required: true
32 type: Notary
33 verifyDigest: true
34 validationFailureAction: Enforce
35 webhookTimeoutSeconds: 30

Once the policy is installed in the cluster, you can create the pod using the signed image and check the creation passes through:

1$ kubectl -n test-shuting run test --image=xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1 --dry-run=server
2pod/test created (server dry run)

Then if you create the pod using an unsigned image, the pod creation is blocked by Kyverno as it does not have any signatures associated with it:

 1$ kubectl -n test-shuting run test --image=xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1-unsigned --dry-run=server
 2
 3Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
 4
 5resource Pod/test-shuting/test was blocked due to the following policies
 6
 7test-irsa:
 8 check-digest: 'failed to verify image xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation:v1-unsigned:
 9 .attestors[0].entries[0]: failed to verify xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5:
10 no signature is associated with "xxxxxxxxxxxx.dkr.ecr.us-west-2.amazonaws.com/test-shuting-notation@sha256:74a98f0e4d750c9052f092a7f7a72de7b20f94f176a490088f7a744c76c53ea5",
11 make sure the artifact was signed successfully'

Conclusion

By leveraging Kyverno and IRSA, you can simplify the configuration of IAM role assumptions for Kubernetes service accounts in EKS. This approach enhances the security of the cluster by ensuring fine-grained access control to AWS resources. With the steps outlined in this blog post, you can easily set up and test IRSA in your EKS cluster.

Blog: Simplifying OpenShift MachineSet Management Using Kyverno

Fri, 28 Jul 2023 00:00:00 +0000

(Guest post from Red Hat Distinguished Architect, Andrew Block)

Managing infrastructure in a declarative fashion is one of the core principles that should be adopted when operating in any environment. In OpenShift, this paradigm for managing the underlying Node infrastructure is accomplished using the Machine API. This extension of the upstream Cluster API project enables the provisioning and management of instances once the OpenShift cluster finishes deploying.

While Machines are individual hosts provisioned as Nodes, cluster administrators typically interact with them via an abstraction – a MachineSet. A MachineSet represents a group of compute instances that not only share similar traits, such as the definition of the desired cloud provider, but they can be scaled based on the desired number of instances.

While MachineSets provide a mechanism for managing Machines at scale and represent a set of purpose-built instances within an OpenShift environment, there are limitations that inhibit one from being able to fully manage them declaratively.

Challenges Surrounding Declarative MachineSets

MachineSets are used within OpenShift in environments which are integrated with a cloud provider, such as Amazon Web Services, Microsoft Azure, and VMware vSphere. Even though there are specific differences as they relate to the individual provider being used, the remainder of the MachineSet definition is the same.

The following is an example of a MachineSet that could be used to represent OpenShift Infrastructure nodes:

 1apiVersion: machine.openshift.io/v1beta1
 2kind: MachineSet
 3metadata:
 4 labels:
 5 machine.openshift.io/cluster-api-cluster: <infrastructure_id> 
 6 machine.openshift.io/cluster-api-machine-role: infra 
 7 machine.openshift.io/cluster-api-machine-type: infra 
 8 name: <infrastructure_id>-infra 
 9 namespace: openshift-machine-api
10spec:
11 replicas: 1
12 selector:
13 matchLabels:
14 machine.openshift.io/cluster-api-cluster: <infrastructure_id> 
15 machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra 
16 template:
17 metadata:
18 labels:
19 machine.openshift.io/cluster-api-cluster: <infrastructure_id> 
20 machine.openshift.io/cluster-api-machine-role: infra 
21 machine.openshift.io/cluster-api-machine-type: infra
22 machine.openshift.io/cluster-api-machineset: <infrastructure_id>-infra
23 spec:
24 metadata:
25 labels:
26 node-role.kubernetes.io/infra: ""
27 providerSpec:
28 # Provider specific implementation
29 ...

While the majority of the MachineSet definition is straightforward, notice the placeholder value denoted by <infrastructure_id>. An OpenShift Infrastructure ID refers to the cluster ID and is a value that is generated at cluster installation that contains the name of the cluster appended by a randomly generated value. Since this value is distinct per cluster, it becomes a challenge to specify the MachineSet definition which could be managed via GitOps prior to cluster creation as one would not be able to ascertain the generated cluster ID.

Several workarounds have been implemented within the community to address this challenge and range from populating GitOps repositories dynamically as the cluster is being provisioned, to a dedicated operator which dynamically updates the MachineSet after it is created. However, instead of leveraging a workaround or needing to deploy an operator just to inject a single property, other options should be considered.

Overcoming MachineSet Limitations Using Kyverno

Even though these approaches described in the prior section do work, they either require additional processes to be performed or limit the types of GitOps management styles that can be implemented. Wouldn’t it be great if the contents of a MachineSet were updated as they are created in the cluster without either having to change how the MachineSet definitions are managed via GitOps or retroactively updating the MachineSet within the cluster after they are created? Fortunately, this challenge related to Infrastructure IDs can be overcome thanks to Kyverno!

Kyverno is a policy engine for Kubernetes that can be used to validate, mutate, generate, and cleanup Kubernetes resources as well as verify container images. Similar to most policy engines for Kubernetes, Kyverno makes use of Validating and Mutating Admission webhooks which enable the introspection of resources prior to being persisted within etcd.

So, given that Kyverno could be used to solve this challenge, what would this look like? Many of the other workarounds including those mentioned previously make use of looking up the Cluster ID from the value stored within the cluster. The Infrastructure ID is stored within a Custom Resource called Infrastructure which includes infrastructure-related details specific to the cluster. The Infrastructure ID can be retrieved by those with elevated cluster level access by executing the following command:

1$ oc get -o jsonpath='{.status.infrastructureName}{"\n"}' infrastructure cluster

With an understanding how to obtain the required information from the cluster, let’s see how Kyverno can be used to retrieve the Infrastructure ID from the cluster and inject it within the MachineSet as it is created.

Kyverno policies can make use of variables that are retrieved from the results from invoking Kubernetes API service calls. Given that the resource and the particular property within the resource is known containing the Infrastructure ID, the following in a context can be used to query the Infrastructure resource from within the cluster and set the variable named infraid based on the JMESPath expression for the obtained resource containing the Infrastructure ID:

1context:
2- name: cluster
3 apiCall:
4 urlPath: /apis/config.openshift.io/v1/infrastructures/cluster
5- name: infraid
6 variable:
7 jmesPath: cluster.status.infrastructureName

Notice how the jmesPath field references the cluster variable which represents the result from the API service call. Traversing through the data structure enables accessing the Infrastructure ID found within the infrastructureName property.

Now that Infrastructure ID has been assigned to a variable, the next step is to take this variable and inject it into the incoming MachineSet resource. Modifications to resources in Kyverno are achieved using one or more mutation rules. Mutations can leverage either a RFC 6902 JSON Patch or a strategic merge patch. While a JSON patch does provide the ability to perform fine-grained modification of resources, as shown in the MachineSet example in a prior section, the Infrastructure ID has the potential to be located at varying locations within the MachineSet definition. To manage a dynamic set of Infrastructure ID parameters regardless of their locations within a MachineSet, the Kyverno-specific replace_all filter can be used to accomplish this task.

1mutate:
2 patchesJson6902: |-
3 - op: replace
4 path: /metadata
5 value: {{ replace_all(to_string(request.object.metadata),'TEMPLATE', infraid) }}
6 - op: replace
7 path: /spec
8 value: {{ replace_all(to_string(request.object.spec),'TEMPLATE', infraid) }}

The mutate rules above specify that all instances of the word “TEMPLATE” that are found within the spec and metadata properties of a MachineSet will be replaced by the infraid variable that was specified earlier.

Putting all of the pieces together, the final ClusterPolicy resource is shown below:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: inject-infrastructurename
 5 annotations:
 6 policies.kyverno.io/title: Inject Infrastructure Name
 7 policies.kyverno.io/category: OpenShift
 8 policies.kyverno.io/severity: medium
 9 kyverno.io/kyverno-version: 1.10.0
10 policies.kyverno.io/minversion: 1.10.0
11 kyverno.io/kubernetes-version: "1.26"
12 policies.kyverno.io/subject: MachineSet
13 policies.kyverno.io/description: >-
14 A required component of a MachineSet is the infrastructure name which is a random string
15 created in a separate resource. It can be tedious or impossible to know this for each
16 MachineSet created. This policy fetches the value of the infrastructure name from the
17 Cluster resource and replaces all instances of TEMPLATE in a MachineSet with that name.
18spec:
19 schemaValidation: false
20 rules:
21 - name: replace-template
22 match:
23 any:
24 - resources:
25 kinds:
26 - machine.openshift.io/v1beta1/MachineSet
27 operations:
28 - CREATE
29 context:
30 - name: cluster
31 apiCall:
32 urlPath: /apis/config.openshift.io/v1/infrastructures/cluster
33 - name: infraid
34 variable:
35 jmesPath: cluster.status.infrastructureName
36 mutate:
37 patchesJson6902: |-
38 - op: replace
39 path: /metadata
40 value: {{ replace_all(to_string(request.object.metadata),'TEMPLATE', infraid) }}
41 - op: replace
42 path: /spec
43 value: {{ replace_all(to_string(request.object.spec),'TEMPLATE', infraid) }}

Assuming Kyverno is deployed to your OpenShift cluster, the ClusterPolicy can be added to enable the desired MachineSet functionality. All that needs to be done now is to update your existing MachineSet manifests that you have specified declaratively, such as in a GitOps repository, and replace the hard-coded Infrastructure ID with the word TEMPLATE. You are free to choose a word other than TEMPLATE to represent the value that should be replaced by the Infrastructure ID. When doing so, be sure to update the value in the ClusterPolicy and in the MachineSet definition.

MachineSets offer the advantage of defining and managing a set of OpenShift Machine profiles, but require that the Cluster ID represented as the Infrastructure ID be present within the definition. Thanks to the dynamic set of capabilities provided by Kyverno, managing MachineSets within OpenShift just got a whole lot easier. The ClusterPolicy shown previously is also available on Artifact Hub and the policy library for easy reference and consumption.

Blog: Using Kyverno with Pod Security Admission

Mon, 12 Jun 2023 00:00:00 +0000

Pod Security Admission (PSA) is the built-in successor to Kubernetes PodSecurityPolicy (PSP) and is enabled by default starting in v1.23, graduating to stable in v1.25, the same version where PSP was finally removed. PSA is different from PSP in many respects, however one of the most important–and central to how PSA operates–is that it is focuses on implementing standards and not individual checks like PSP did. The standards in this case are the Pod Security Standards. While many might see technologies like Kyverno and other admission controllers in competition PSA, the two can actually be highly complementary and deliver increased value through tighter security guarantees while catering to the flexibility demanded by modern Kubernetes operations. In this blog, we’ll explain some of these use cases for how Kyverno can be used alongside PSA to get the best of both worlds.

Introduction

First, let’s explain a bit about how PSA can be used for those not familiar. Although this won’t be a full tutorial on how to use the technology, it’ll at least establish a good set of boundaries to help you know where it begins and ends.

PSA can basically be used in two ways: at the cluster level and at the Namespace level. Although it is “on” by default, it isn’t “active” by default. Since PSA does not require an API resource similar to how PSP worked, it is “activated” in different ways. To activate PSA at the cluster level, one must create and install a specific file local to the control plane of the kind AdmissionConfiguration the contents of which will dictate how it works. Many Kubernetes PaaS providers have already done the work at the cluster level for you and, in many cases, there’s nothing you can configure yourself. For example, in cases like EKS, PSA is enabled at the cluster level but set to the privileged mode which allows insecure Pod configurations. To activate PSA at the Namespace level, one need only label a Namespace. The two methods aren’t mutually exclusive. In the case of the Namespace method, using PSA is extremely quick and simple.

Now on to the important point about profiles mentioned at the outset. PSA is the technology which implements the Pod Security Standards (PSS). Those standards are broken up into three profiles: privileged, which is effectively the lack of any security, baseline, and restricted. Each profile is a collection of different controls. Each control establishes which fields of a Pod’s spec should or should not be set and with what value(s). The restricted profile is cumulative of the baseline profile, so by enabling restricted you implicitly get baseline.

Regardless of how PSA is activated and at what scope, it implements these profiles. Critical to understand is that this is an all-or-nothing proposition: a profile is implemented in its entirety and there is no possibility to cherry pick which controls in the profile are checked. Although each profile can be checked in either audit, warn, or enforce mode (even multiple modes simultaneously), you either get all of that profile or you get none of it.

Lastly, there is the matter of exclusions. It is extremely common for users to need the ability to tactically exclude certain Pods from the scope of policy because of some special needs they have. PSA allows exclusions via exemptions but only when PSA is activated cluster wide. The exemptions available are usernames, runtimeClasses and namespaces. All exemptions must be specified in the AdmissionConfiguration file mentioned earlier.

With some of the mechanics of PSA out of the way, let’s get into the use cases for PSA and Kyverno.

Use Cases

Although there are many possible points of intersection between PSA and Kyverno, here are four we’ve identified and commonly see employed. For some, there can be many different variations which we won’t all cover.

Use Kyverno to require PSA Namespace labels

As mentioned in the beginning, one of PSA’s methods of operation is through assignment of labels to Namespaces. Without at least one PSA-specific label, nothing will happen. Kyverno can assist in this process in a couple ways:

Validate that labels are assigned where and how they need to be
Automatically assign the labels based upon your criteria

On the first, Kyverno can ensure that every Namespace has some PSA label attached to it helping to prevent either typos or just forgetfulness. What that/those label(s) is/are and with what values is entirely up to you, but it is all driven by policy-as-code (PaC). This is, indeed, convenient, but it can be a more serious issue when you have PSA activated at, for example, the baseline profile across the cluster. Should a user be able to assign the Namespace label pod-security.kubernetes.io/enforce: privileged in these cases, it will effectively deactivate PSA on that Namespace. That’s obviously not good and could compromise the security of that Namespace. But since Kubernetes RBAC doesn’t allow for such granular permissions, you can outsource that job to Kyverno to ensure this act cannot be performed.

On the second, Kyverno can actually assign (i.e., mutate) the desired PSA labels for you based upon whatever criteria you establish and define in a policy. Even if you’ve done so in git, Kyverno can double-check your work and assign them if not present. For example, you might want to assign the pod-security.kubernetes.io/enforce: baseline label to all Namespaces by default but assign pod-security.kubernetes.io/enforce: restricted to those which begin with the name pci- or have another label called env: prod. Kyverno’s mutation ability gives you very flexible definitions which can make these things happen.

These policies have already been written and are provided to you here.

Use PSA and Kyverno both to enforce

In this use case, which is the most common, you use both PSA and Kyverno combined to provide comprehensive protection for your cluster where each does something different. Due to the limited nature of exclusions, we often find that PSA is used right up until the point that its flexibility can go no further. The most common examples are with some monitoring/observability tools and service meshes. These types of tools often have Pods that need to run with privileges not afforded by sometimes even the baseline profile. Rather than deactivate Pod security in those Namespaces–which for many is a non-starter–Kyverno can be used because of its great flexibility.

Kyverno has the ability to exclude a Pod based on just about any combination you can think up. Whether it’s Pod name, image, user, label, or what have you, Kyverno can exempt it from controls.

When it comes to how Kyverno enforces, there are generally two options. First, Kyverno publishes and maintains the complete Pod Security Standards as individual policies (one per control). These are also conveniently packaged into their own Helm chart for easy installation. Second, Kyverno directly integrates the PSA libraries into the engine and exposes a simplified rule type we call a validate “subrule” known as podSecurity. This rule type, powered under the hood by PSA, allows a consolidated, simplified way to enforce the Pod Security Standards. But in addition, it also allows more capabilities not found in PSA. For example, using this podSecurity subrule, it is possible to exempt specific controls by name and even specific images in addition to the standard exclusions provided by all Kyverno rule types. This is incredibly powerful yet easy to use.

For example, below is a Kyverno policy which applies to Pods in all Namespaces beginning with the prefix prod- and enforces the restricted profile except for the Seccomp check and specifically on containers which use the cadvisor image.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: psa
 5spec:
 6 background: true
 7 validationFailureAction: Enforce
 8 rules:
 9 - name: restricted
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 namespaces:
16 - prod-*
17 validate:
18 podSecurity:
19 level: restricted
20 version: latest
21 exclude:
22 - controlName: Seccomp
23 - controlName: Seccomp
24 images:
25 - cadvisor*

This is just one example and there are many variations for how these two technologies can be used simultaneously, each to enforce some aspect of Pod security on your cluster. For more information as well as a list of those variations, see the documentation. In addition to the documentation, there are pre-built samples ready to go here.

And, by the way, one more benefit you get when using Kyverno to enforce its Pod Security Standards over PSA is that it allows you to block Pod controllers directly as a result of its rule auto-generation capability. This is not something PSA allows and only non-compliant Pods will be blocked.

Use Kyverno reporting for PSA

PSA allows for assigning multiple labels to a Namespace (or configuring the same via the AdmissionConfiguration file) to get a union of the behaviors. For example, PSA can enforce at the baseline profile and audit and/or warn at the restricted profile. The audit mode means any violations will be allowed yet will be written in the Kubernetes audit log. There are a few down sides with this approach. The audit log is not accessible from inside the cluster because it’s not a native Kubernetes resource, it’s literally a log file written to the control plane or, optionally, sent to an external server. Audit logs also commonly cannot be enabled retroactively or to do so is cumbersome and disruptive. In the case of some cloud PaaS offerings, if audit logging wasn’t enabled at the time the cluster was created then it can’t be subsequently enabled, potentially forcing you to build another cluster just to gain these insights.

This is yet another opportunity where Kyverno can step in and help. In cases where PSA is not enforcing at the most restrictive level, Kyverno can be layered in to generate its Policy Reports. A Policy Report is an in-cluster, Kubernetes-native resource which is an open standard developed by the upstream Kubernetes policy working group and adopted by many other tools. Policy reports give you a leg up over audit logs because they empower self-service and compliment multi-tenant approaches. Since they’re just another Kubernetes resource, RBAC can be wrapped around them allowing, for example, developers to see violations in their own Namespaces so they know what they’re responsible for addressing. They can also help you as a cluster admin or operator know which resources in the cluster are non-compliant at any point in time since Policy Reports are not historical and always reflect the current cluster state.

It’s important to understand, however, that Kyverno’s Policy Reporting feature will only work on Pods which have not been blocked by PSA. But, provided that is true, the process to enable and begin viewing Policy Reports is simple: load any Kyverno policy, for example the restricted profile of the PSS, in audit mode with the flag background: true. The audit mode means Kyverno will not block any Pods, and the true setting on the background field means it will periodically scan the cluster to compare existing resources against the policy giving you an extra layer of comfort that nothing got missed. Policy Reports will then be created in each Namespace where Pods are located.

Reporting isn’t just restricted to Pods either but can be used to check for those Namespace labels we discussed earlier. For example, you may want to know which Namespaces have no PSA labels whatsoever which could key you in to someone or something doing things it perhaps shouldn’t. By introducing the below policy, Kyverno will generate and maintain and a Cluster Policy Report in the default Namespace with the result of these continual checks. The results can be conveniently viewed by cluster admins or scraped and alerted on by other tools such as the Policy Reporter and more.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: psa-reporting
 5spec:
 6 validationFailureAction: audit
 7 background: true
 8 rules:
 9 - name: check-namespace-labels
10 match:
11 any:
12 - resources:
13 kinds:
14 - Namespace
15 validate:
16 message: This Namespace is missing a PSA label.
17 pattern:
18 metadata:
19 labels:
20 pod-security.kubernetes.io/*: "?*"

Use Kyverno CLI in pipelines to test against PSS profiles

If none of the above use cases piqued your interests or you are unwilling to install anything additional in your cluster, there is still an opportunity for Kyverno to provide value. Kyverno comes with a CLI which can be used in CI pipelines to check resource manifests against those Pod Security Standards enforced by PSA. But since PSA itself is in-line to the Kubernetes API with no external tooling, it takes standing up a full, running cluster to perform the same checks otherwise. As we mentioned earlier, Kyverno integrates the PSA libraries directly into its engine. And since the CLI is really just a different form factor of the same Kyverno engine, you get the same benefits of Kyverno outside the cluster as in.

Take one of the earlier policies we’ve pointed out along the way and see a sample GitHub Action workflow we publish in the documentation and you’re up and running. The kyverno apply command will naturally exit with an error code if the resource(s) do not pass. The CLI combined with Kyverno’s rule auto-generation capability mentioned earlier means you can be informed of those Deployments, StatefulSets, and others before they need to touch a cluster.

Below you can see the result of the Kyverno CLI comparing a standard Deployment with no security settings defined to the restricted profile of the PSS showed previously. As you can see, the Deployment failed the checks and a message is printed with exactly which controls failed and what must be done to address them.

 1$ kubectl-kyverno apply restricted.yaml -r deploy.yaml
 2
 3Applying 1 policy rule to 1 resource...
 4
 5policy psa -> resource default/Deployment/busybox failed:
 61. autogen-restricted: Validation rule 'autogen-restricted' failed. It violates PodSecurity "restricted:latest":
 7({Allowed:false ForbiddenReason:allowPrivilegeEscalation != false ForbiddenDetail:container "busybox" must set securityContext.allowPrivilegeEscalation=false})
 8({Allowed:false ForbiddenReason:unrestricted capabilities ForbiddenDetail:container "busybox" must set securityContext.capabilities.drop=["ALL"]})
 9({Allowed:false ForbiddenReason:runAsNonRoot != true ForbiddenDetail:pod or container "busybox" must set securityContext.runAsNonRoot=true})
10({Allowed:false ForbiddenReason:seccompProfile ForbiddenDetail:pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost"})
11
12
13pass: 0, fail: 1, warn: 0, error: 0, skip: 2

Closing

Pod Security Admission is a major improvement over PodSecurityPolicy and couldn’t be simpler to activate and begin using in your clusters. If you have no Pod security today, we absolutely recommend you plug that gap and starting with PSA can be a great way. As we hope we’ve made clear, there are many places where PSA and Kyverno can be used together and it doesn’t have to be an either or decision. Use PSA where it makes sense and provides the necessary level of coverage, and use Kyverno to pick up the slack.

Blog: Let's Play Kyverno

Sun, 04 Jun 2023 00:00:00 +0000

Foreword

“Kyverno is a policy engine designed specifically for Kubernetes."

While this approach makes it very easy to use Kyverno in its intended environment, it is sometimes difficult to explain and present the capabilities when that environment is not available.

To help potential users get started and reduce the effort required to test Kyverno and/or develop new policies, the Kyverno Playground has been developed. This has now been available for 3 weeks, currently in version 0.3.1.

This article explains:

What is the Kyverno Playground?
How was the Kyverno Playground created?
How does the Kyverno Playground work?
What additional features provides the Kyverno Playground?

Introduction

Having a booth at a conference and having direct contact with current and potential new users provides many opportunities. You can talk about how it works, existing and new features, use cases, and user challenges. However, many conversations often only scratch the surface because there has been no way to quickly and easily present a concrete feature or write a simple policy and demonstrate it right away.

In these situations, the desire for a playground as a simple, independent test environment to reduce these problems arose again. After KubeCon EU 2023, we expanded this idea and started developing a first prototype.

What is the Kyverno Playground?

Kyverno Playground is a web-based and isolated test environment for Kyverno. It is a standalone binary and does not require connection to a real Kubernetes cluster, but is optionally supported.

All the information to simulate a policy execution can be provided as YAML content in the respective panels. For a minimal setup, you only need to specify a resource and the policy you want to apply to it. All other information is optional or provides similar default settings as a real Kyverno installation.

How was the Kyverno Playground created?

Originally, we were going to use the Kyverno CLI under the hood, but then decided to use the Kyverno engine directly and develop everything around it from scratch. This allows us to close some gaps between the CLI and an actual installation, as well as provide more options in configuration and simulation.

We also wanted to decouple the frontend and backend so we could work on features independently and later bring everything together using a simple REST API. This gave us the most flexibility possible in developing and choosing the tools we needed in these two areas.

The frontend is based on Vue3 and uses Vuetify as a Vue component framework that simplifies the creation of a user-friendly and interactive interface. We also use monaco editor, which is also used in Visual Studio Code, to provide the best possible user experience for entering and working with YAML content.

For the backend, we use Go to integrate the Kyverno engine as smoothly as possible. The Gin Framework helps us to provide our API and UI via a file server.

How does the Kyverno Playground work?

Once the backend and frontend were set up, we were able to simulate the first policies on the new playground. To do this, each simulation sends all available information to a single REST API endpoint.

As we simulate the policy, resource, context, and engine configuration, we create a separate engine for each request. This ensures that all requests are independent of each other and that the provided configuration can be applied without having to reconfigure a single engine each time.

Before running the actual engine, the received YAML content must be parsed. To ensure that the provided content is not only valid YAML, but also valid Kubernetes resources, we use tools like kubectl-validate and OpenAPI schemas. This way, we can provide early feedback to the user and avoid misleading error messages.

After parsing and validation, we perform the actual engine operations in the same order as in a real cluster. This means that mutation and image verification are performed before validation of the resource. This behavior ensures that there are no errors due to dependencies of operations within a policy.

At the end, the engine returns a response with all executed rules that can be displayed in the frontend.

What additional features provides the Kyverno Playground?

Besides the basic idea of providing a simple testing environment, the Playground offers several features to further enhance the user experience and exploration of Kyverno.

Let’s take a look at a small selection of these available additional features.

Example Library

In order to give new users an interactive overview of what Kyverno can do and how different use cases can be achieved within the Playground, we decided to include a small sample library based on the extensive Kyverno Policy Repository. The examples provide a set of predefined policies and related resources and configurations for common use cases such as the Pod Security Standards and Best Practices.

The tutorials focus on features and how they can be simulated in the Playground, providing various examples such as upgrade operations, image verification, resource cloning, and more.

In our daily work in the community, we often have situations where users need help with their policies. The process to get help has often been cumbersome because you need the ability to test the policy yourself. You have to figure out where the problem is and maybe try different things to get it to work. In the end, the user again has to test whether the policy works with their resources.

With the Share functionality, we have created an easy way to share a complete policy setup via the Playground. This includes the policy, an example of a validated resource, context information, and the engine configuration. Since all information is part of the created URL, this functionality is stateless, immutable, and does not expire. This makes it suitable for both short conversations and longer-lasting GitHub discussions and issues.

You want to open the exact same setup as shown in the Screenshot? No problem, we’ll share it with you!

Context (Variables)

Kyverno can run on various operations and uses request information such as usernames, groups, roles, and additional context from provided ConfigMaps, Secrets, and more. How could we provide this information within the Playground? We don’t have a true cluster request or other resources to pull the information from. We decided to take a simple approach and provide the ability to simulate what you need to provision for your policy.

In the advanced section of the Playground, you can change the context of your request and simulate variables that you would normally get from other resources or API calls.

Want to try out an example? Check out the Allowed Pod Priorities or Check ServiceAccount Example.

Conclusion

This article explained the story behind the Kyverno Playground, how it started, and where we are just three weeks later. We are really impressed with the initial feedback from our users and what we can already offer without having thought about it at first. We’re excited to see how these tools evolve and offer in function. We hope to get more feedback to improve and develop it further.

Blog: Kyverno 1.10 Released

Tue, 30 May 2023 00:00:00 +0000

The Kyverno team are proud to announce the release of Kyverno 1.10, a minor release in terms of version number but a major release in every other regard. With around four months in the making and after four pre-releases and nearly 500 pull requests merged, Kyverno 1.10 is one of the largest releases in the history of the project and features a ton of new and highly-requested features and a staggering number of fixes and improvements. It also brings with it some breaking changes so please read thoroughly. We can’t wait for you to see what’s inside so let’s get started!

Key New Features of Kyverno 1.10

Kyverno 1.10 contains several new and significant features including decomposing Kyverno into smaller pieces, external service calls, Notary support, and a major revamp of generate rules.

Increased Scalability with Service Decomposition

In previous versions of Kyverno, everything except the cleanup controller (introduced in 1.9) was packaged in a single container. This made sense in the early days, but as Kyverno began to grow in capability and complexity, the single-deployment model just wasn’t going to cut it. Users also wanted a way to only install what they needed and not get everything else that came along with it. Beginning in Kyverno 1.10, the major capabilities of Kyverno have been broken out into separate deployments allowing you to switch on or off the ones you want. What this looks like is shown below.

The four major components of Kyverno and their primary functions are as follows:

Admission Controller: The heart of Kyverno, the Admission Controller receives and processes webhook requests from the Kubernetes API server and is responsible for validate, mutate, and verifyImages rules along with Policy Exceptions. It also performs most of the validations on policies themselves. This is the only required component of Kyverno which must be installed.
Reports Controller: Responsible for processing of Kyverno’s Policy Reports including performing background reporting scans.
Background Controller: Not to be confused with background scans, the Background Controller handles all the generate rules and mutate rules when they impact existing resources (which all happen in the background).
Cleanup Controller: Takes care of all the cleanup tasks according to cleanup policies.

Because Kyverno is now decomposed into separate controllers, each controller can be scaled independently although they all don’t necessarily handle it differently. We recommend reading the High Availability page for more details on the internals of these controllers and how scale and availability are handled per controller.

Extensibility via External Service Calls

Kyverno is already able to gather data from external sources as a factor in its policy-making decisions, for example from the Kubernetes API, OCI image registries, and ConfigMaps. However, one of the most requested features has been the ability for it to make calls to services other than the Kubernetes API server for the same reasons. We’re happy to say that as of Kyverno 1.10, this feature now exists! It is new and a bit limited at this point so we can get an understanding of how folks intend to use it, but it allows performing GET and POST requests against another service in the cluster along with specifying a certificate authority bundle for establishing trust against HTTPS servers. A sample of what this looks like is shown below.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: check-namespaces 
 5spec:
 6 validationFailureAction: Enforce
 7 rules:
 8 - name: call-extension
 9 match:
10 any:
11 - resources:
12 kinds:
13 - ConfigMap
14 context:
15 - name: result
16 apiCall:
17 method: POST
18 data:
19 - key: somekey
20 value: "{{ somevariable }}"
21 service:
22 url: http://sample.myservice/someendpoint
23 caBundle: |-
24 -----BEGIN CERTIFICATE-----
25 <snip>
26 -----END CERTIFICATE----- 
27 validate:
28 message: "This shall not pass due to item {{ fookey}}"
29 deny:
30 conditions:
31 all:
32 - key: "{{ result.allowed }}"
33 operator: Equals
34 value: false

And, by the way, in addition to POST calls to external services, we’ve also enhanced the existing apiCall context variable to be able to POST to the Kubernetes API making it possible to do things like pass a SubjectAccessReview which will make permissions assessments much easier.

Software Supply Chain Security with CNCF Notary

The Notary project is another project, like Sigstore, aiming to solve software supply chain security through OCI image signing. Notary is currently in its second version and differentiates itself from Sigstore by using OCI artifacts to store image signatures. Although Kyverno has had support for verifying signatures and attestations from Sigstore’s Cosign project for a while now, we wanted to add support for Notary so that no matter what technology you use to sign your images, Kyverno will be there for you.

Starting in Kyverno 1.10, we’ve added a new type field to verifyImages rules allowing you to specify the signatory used, either Notary or Cosign. An example of what this looks like is shown below.

 1apiVersion: kyverno.io/v2beta1
 2kind: ClusterPolicy
 3metadata:
 4 name: check-image-notary
 5spec:
 6 validationFailureAction: Enforce
 7 webhookTimeoutSeconds: 30
 8 failurePolicy: Fail 
 9 rules:
10 - name: verify-signature-notary
11 match:
12 any:
13 - resources:
14 kinds:
15 - Pod
16 verifyImages:
17 - type: Notary
18 imageReferences:
19 - "mytest.azurecr.io/user/net-monitor:v1"
20 attestors:
21 - count: 1
22 entries:
23 - certificates:
24 cert: |-
25 -----BEGIN CERTIFICATE-----
26 <snip>
27 -----END CERTIFICATE-----

While this addition supports simple Notary verification, if needing to call an external service when using an extension, the external service call feature as shown earlier can be used.

Generate Rule Refactoring

Even though generate rules have been a cornerstone of Kyverno for a while now, we did a significant overhaul on them to add new functionality, fix issues, improve the user experience, and just in general give it a major face lift. Specifically, one of the new features in generate rules which has been a frequent request is to allow the triggering resource to share the same synchronize life cycle as the generated resource. Users told us they want to be able to remove or change a trigger while sync is on and have that influence the resource that trigger was responsible for generating. This is now a reality in Kyverno 1.10 and will help in multitenancy use cases where generate rules are so frequently employed.

We didn’t stop there, though, we also added new features like triggering on DELETE requests, triggering on subresources, and performing permissions checks up front when the generate rule is created, helping to avoid failures down the line in case you forgot to add them. These features and many more can be found inside Kyverno 1.10 when using generate rules. We also had to make some tough decisions that lead to a couple breaking changes, so please read the release notes carefully if you’re a user of generate rules. And although we’ll talk about upgrades below, we strongly suggest removing and then reintroducing them in your cluster (or at least in a test cluster somewhere) to allow Kyverno to validate the new restrictions put into effect in 1.10.

Other Additions and Enhancements

There are loads of other additions and enhancements to be found in Kyverno 1.10 and we simply can’t cover them all, but here are the most notable ones.

Operations can now be specified directly in match and exclude blocks obviating the need for preconditions. This enhancement can simplify your rules by moving that condition to match on CREATE, for example, up into the match block. Specifying operations like this is also a requirement if you want to generate something based upon a DELETE operation.

1match:
2 any:
3 - resources:
4 kinds:
5 - Service
6 operations:
7 - CREATE

Policy Exceptions have been enhanced in 1.10 to add support for background scanning, useful when you consume a Policy Report and want to see that fail result to go away, and wildcards in the ruleNames[] field. The latter will assist when you might have several rules in a policy which begin with the same prefix.

A number of significant enhancements were made to Policy Reports in Kyverno 1.10 which dramatically improve performance, reduce time to aggregate reports, and lower resource consumption. Another enhancement some users may rejoice in hearing is that background scans will now, by default (but configurable, of course) consider Kyverno’s resource filters when producing reports. So if you’ve excluded a Namespace in the resource filter, by default you won’t see any reports for it either when coming from background scans.

Context variables are now lazily evaluated (JIT) which means no more failed rules when preconditions don’t pass. Variables are often used in conditions and so they will follow the same circuit-breaking mechanisms already in place for those conditions. This should also have the benefit of reducing API calls in rules where they may not always be needed.

Speaking of conditions, there’s now a new message field that’s available for use in conditions everywhere they’re used throughout Kyverno which will allow you to append the contents of that field in the message response returned by Kyverno. This is especially handy in verifyImages rules where the reason why an attestation verification failed can be narrowed down to the exact condition. And if there are multiple, they’ll be appended to each other. Very handy indeed.

Kyverno 1.10 has three new JMESPath filters, image_normalize(), trim_prefix(), and to_boolean() in addition to some enhancements to existing filters. A couple of those enhancements to call out are the sum() filter can now sum quantities like memory making it valuable for adding up all the memory requests in a Pod to figure out whether it should be allowed or not. And the x509_decode() filter now supports decoding of Certificate Signing Requests so you can apply additional security checks if you use that API.

Lastly, the documentation and policy library were greatly refreshed and improved as of this release. Not just that, but all Kyverno policies in the library can now be found on Artifact Hub making Kyverno policies presently the fourth largest artifact type.

Potentially Breaking Changes

As we hope you’ve seen, Kyverno 1.10 has some truly amazing features and enhancements in store for you, but making all this happen plus the hundreds of fixes we didn’t cover here meant we had to make some important decisions. We’d like to make you aware of these breaking changes up front so there are no surprises.

First, due to the decomposition efforts, the Kyverno Helm chart had to basically be rewritten. This is reflected in a major version number increment from 2 to 3. As such, there is no direct upgrade path when coming from v2 of the chart and attempts to do so will be blocked by default. We’ve written up a Helm upgrade and migration guide here and we strongly recommend you give that a good read. Although we also publish a YAML manifest as part of each release, there again a direct upgrade is not supported. In short, we recommend a backup-uninstall-reinstall-restore method when approaching this release, but the write-up has more details.

For rules which matched on certain types of subresources like PodExecOptions, you’ll need to move to the canonical format of them such as Pod/exec. A simple change to make, and Kyverno will let you know if you try to create it the other way.

And, finally, because of the number of issues we sorted regarding generate rules in this release, we had to put a few more guardrails in place to ensure the correct user experience was being met. Some fields now no longer support defining variables, some others may be immutable after rule creation, and a few fields will now be required whereas they previously were not. The latter we’ve tried to limit to situations in which policies are newly seen and not which pre-exist in the cluster.

For these breaking changes, and others, please carefully read the extensive and (yes, sorry) lengthy release notes here.

Closing

Kyverno 1.10 is quite the loaded release as you can probably see. After about four months and close to 500 PRs, there were a tremendous number of changes from the Kyverno community. And if you were one of the many, many contributors who pitched in to make this release a reality, a hearty THANK YOU for all your work! Hopefully what you’ve seen makes you excited to try out 1.10 for yourself. Come engage with us in the Kyverno channel on Kubernetes Slack, attend one of our community meetings, or just catch us on Twitter.

And if you’re already a Kyverno adopter, sign up to be an official adopter by updating the Adopters form here.

Blog: PodSecurityPolicy migration with Kyverno

Wed, 24 May 2023 00:00:00 +0000

As you’ve probably heard, PodSecurityPolicy (PSP) in Kubernetes is no more. After a deprecation beginning in v1.21, they were finally removed in v1.25. Many organizations out there are still relying on PSPs and, if you’re reading this post, you’re probably one of them. As you begin to upgrade your clusters closer and closer towards v1.25, the clock is ticking. The choices with which you are faced are to either delay cluster upgrades, which means you aren’t keeping up with the frequent releases and risk being on an unmaintained version, or upgrade to v1.25 or later and simply not have security for Pods. Neither is really an acceptable choice. In this blog post, we’ll show you what your options are and provide a step-by-step migration guide for getting off PSP and onto Kyverno so you can feel confident in your ability to upgrade safely.

Background

PSP was a built-in API object and admission controller to Kubernetes and existed right after v1.0 was released. Its removal in v1.25 meant it hung around for quite some time. Being the earliest form of Pod security for Kubernetes, it experienced some growing pains and began to feel the strain of more demanding use cases and users alike. That strain ultimately resulted in a decision to not promote it from a beta API, from which it was left, and to instead move towards its successor, Pod Security Admission.

That’s the very, very short story, but for more details on the historical context of PSP and a retrospective, take a look at this blog post from the Kubernetes project. A previous blog did a good job of explaining the what and why of the deprecation and ultimately removal here and so we recommend checking both out.

The TL;DR version of the story goes something like this.

PSP ultimately led to the establishment of a set of standards known as Pod Security Standards. These standards, as all, aren’t a “how” they’re a “what”. Standards aren’t a technology, they define what Pods should do in order to be considered secure. Standards are organized into “controls” where each control is basically a directive on which field(s) should be configured in a prescribed way. These standards are still here today because they capture all (well, most) the security-related fields of a Pod and provide an easy-to-understand way to secure them. These standards are beneficial because, prior to them, Pod security was somewhat of the wild, wild west. It wasn’t clear what really should be done to secure Pods, in what situations, and how to best go about that. But PSP was at least the “how” before the standards existed.

Once the writing was on the wall for PSP, a replacement had to be found. That replacement was Pod Security Admission (PSA). PSA was the “how” for Pod Security Standards (again, only the “what”). PSA implements the Pod Security Standards, which are broken down into basically two profiles. There were already many improvements with PSA but one of the nicest was they weren’t tied to RBAC like PSPs were. PSA is the in-process way to secure Pods today and is a great way to get started with securing Pods if you never experienced PSP. But, as with all things, there are trade-offs. PSA is very rigid in its wide application of these standards. Entire profiles of controls are implemented in a single swath with no opportunity for customization. While this works in some situations, the lack of flexibility is a big challenge for operating Kubernetes in most real-world environments today. This is where Kyverno can come in and give you the best of all those worlds: comprehensive coverage of Pod security and in a way which is totally flexible, meets you where you are, and is easy to do.

Comparison

We’ve put together a comparison table below which allows you to see how the three options stack up and we’ll cover each of these rows below to give you a better explanation of what they mean.

Pod Security Policy	Pod Security Admission	Kyverno
Pods only	Pods only	Anything
Limited options	Only 2 options (PSS only, gaps*)	Anything
Limited mutation	No mutation	Mutate anything
Requires RBAC modifications	Does not require RBAC	Does not require RBAC
Limited to User, Group, ServiceAccount	Limited to cluster, Namespace	Any association
No support for Pod controllers	No support for Pod controllers**	Automatic support for Pod controllers
No auditing	Audits in audit log	Audits in Policy Reports
No message customization	No message customization	Fully custom messages
No exclusions	Limited exclusions	Flexible exclusions
Integrated	Integrated	External

* No readOnlyRootFilesystem, runtimeClass (excludes deprecated options)
** Audit support only

Pods only. PSP and PSA, as the names imply, only function on Pods. This is a great thing, but Pods aren’t the only resource that need security in Kubernetes. Kyverno can cover anything in Kubernetes, even custom resources, both current and future.

Limited options. PSP offers around 20 different options for controlling a Pod’s spec (several of these are now deprecated or removed), all of which are naturally security inclined. PSA has even less. You get two options, one for each profile (baseline and restricted). Kyverno allows access to every Pod field (and every other field for every other resource).

Limited mutation. This was one of the nice things about PSP that went away in PSA. Pods could be mutated to apply some default values if not specified. PSA dropped that entirely. With PSP, you only got about 5 different mutations. With PSA, you get none. And with Kyverno, you can mutate anything you want on any resource.

Requires RBAC modifications. This was one of the biggest pain points for those who know PSP. From the moment PSP was activated in a cluster, it was basically a breaking change unless you had extensive RBAC resources in place in the form of Roles, RoleBindings, ClusterRoles, ClusterRoleBindings, and the PodSecurityPolicy resources as well. It was an implicit deny and an explicit allow behavior, but only if you were authorized to use at least one in the first place. PSA improved here in that it wasn’t tied to RBAC at all which meant much complexity was alleviated. Kyverno, like PSA, does not require RBAC to consume policies. From the moment a policy exists, it takes effect based upon its definition.

Limited to User, Group, and ServiceAccount. As it was tied to RBAC, PSP had its scope of application tied to these principals which made it difficult to apply and often resulted in a huge web of these mappings. PSA is limited to either the whole cluster or Namespace, but nothing further. Kyverno can be associated with anything and apply as granular as you can express in Kubernetes.

No support for Pod controllers. Another bummer for PSP users was the fact that Pod controllers such as Deployments and StatefulSets would always be allowed but Pods would be blocked. You had to figure this out typically by going to inspect the ReplicaSet to understand why. PSA has basically the same behavior here. Kyverno has automatic support for Pod controllers through its ability to auto-generate Pod controller rules.

No auditing. A PSP either blocks or it doesn’t, there’s no in between. PSA can audit, warn, or block, but the audits are locked up in the API server’s audit log which isn’t visible inside the cluster. Kyverno generates its audits as a cluster-visible, custom resource called a Policy Report which is an open standard developed by the Kubernetes Policy Working Group and is adopted by other tools aside from just Kyverno. These Policy Reports are just another Kubernetes resource allowing them to be scraped by other tools and, importantly, read by other users in a way which can be controlled via RBAC.

No message customization. When a PSP blocks a Pod the message can’t be customized. Same situation with PSA. The message returned by Kyverno can be fully customized making it very useful to show, for example, a team name or email of someone for developers or users to contact.

No exclusions. PSP has no ability to exclude a Pod once it is “caught” by a policy. PSA has only a few options here, all of which pertain to cluster-wide configuration. Kyverno basically exclude any criteria you can write. Its policy exceptions feature takes this even further.

Integrated. PSP and PSA are both integrated into the control plane. Kyverno is an external component (a dynamic admission controller) and so must be run separately in your cluster. In order to do this, Kyverno uses webhooks.

Migration Outline

When migrating from PSP to Kyverno, this is the high-level approach that we recommend. Links are provided for the Kyverno concepts which are applicable to this migration, however it is still recommended that before diving into a migration yourself you spend some more time with the documentation and, of course, experimentation in a lab or non-production environment.

Before getting started, we recommend using the Kyverno CLI to scan your cluster against the Pod Security Standards baseline profile which Kyverno conveniently implements in as simple as just a simple rule. Starting with baseline is a good way to identify the lowest hanging fruit and what to target. You can do this with just a simple one-liner:

1curl -s https://raw.githubusercontent.com/kyverno/policies/main/pod-security/subrule/podsecurity-subrule-baseline/podsecurity-subrule-base
2line.yaml | kubectl kyverno apply --cluster -

Identify which PSP you want to offload to Kyverno starting with the narrowest scoped PSP first. Each field in the PSP should translate to one rule in most cases. Most fields are covered by the Pod Security Standards and Kyverno already has pre-built policies ready to go for the entire set. It additionally has other policies specifically for PSP migration cases to cover all remaining capabilities which PSP covered (including mutation use cases).
Identify and install the needed Kyverno policy in Audit mode after first determining the scope at which it should operate. Kyverno policies, unlike PSPs, do not require RBAC to consume the policy. As soon as it is installed, it takes effect. There are two variants of Kyverno policies: ClusterPolicy and Policy. A Policy is Namespaced and is therefore confined to operating on the Pods within that same Namespace. A ClusterPolicy is cluster scoped but may still be configured to selectively operate at a more granular level including a specific Namespace, for a specific User, for a group of Users, for a ServiceAccount, or a combination thereof. In other words, a Policy is maximally scoped at the Namespace. A ClusterPolicy is maximally scoped at the cluster. Each one may have its scope decreased but not increased. They can be mixed and matched to suit your needs.
Before moving forward, wait for Policy Reports to be generated. Verify the reports do not contain any Failed entries. A Pod which is blocked by a PSP will not go on to be observed by Kyverno, therefore there should be no Failed results in a Policy Report for a Kyverno policy analogous to its PSP counterpart. For tighter confinement of the Kyverno policy, you can configure the match block to only consider Pods which are allowed due to a specific PSP through use of the annotation kubernetes.io/psp. The value of this annotation will be the name of the PSP used for validation. Keep in mind, if choosing to match based on this annotation it will disable auto-gen rules so only Pods will be shown in reports. Background scanning (on by default) must be enabled in these policies to evaluate resources which already exist in the cluster.
Once the reports look as intended, change RBAC in such a way that a more permissive PSP is referenced instead. As long as PSP is enabled in the cluster, it is an implicit denying action. Only through the presence of at least one PSP which permits the Pod can it be created.
Create a test Pod which would be in violation of the deprecated PSP (and would previously be blocked) and ensure it is allowed by the more permissive PSP.
Wait between 10-20 seconds and check the Policy Report in the Namespace where the Pod was just created.
Verify that the result in the report corresponding to this Pod is listed as a Fail result. This indicates the Kyverno policy has caught the Pod and it is now in violation of a Kyverno policy.
Change the policy to Enforce mode which will then block these Pods, same as the PSP which was just deprecated.
Continue to follow this process until all PSPs have been moved to the most permissive PSP and are now being enforced by Kyverno.
Optionally, during an outage window, you may choose to deactivate PSP (if applicable) by removing the PodSecurityPolicy value from the --enable-admission-plugins flag, or inversely, add the PodSecurityPolicy value to the --disable-admission-plugins flag on the Kubernetes API server.

Guided Migration

In this section, we’ll walk you through a guided migration process which follows the outline shown above. Although your PSPs are likely to be different, we’ll illustrate this process with a couple of simple examples. Obviously, keep in mind that your RBAC situation is likely different, so use this as a general template to understand what is required.

We’ll assume the following:

Kubernetes v1.24 is in use although this should be applicable to earlier versions as well. Note the versions supported by Kyverno in the compatibility matrix here.
A Namespaced called qa which is the subject of this tutorial.
A user named chip who is currently subject to a PSP in the qa Namespace we wish to migrate onto Kyverno.

First, we’ve identified the following PSP which requires that host namespaces, corresponding to Pod spec fields hostNetwork, hostIPC, and hostPID must be unset or set to false if they are defined. This is a common component of many PSPs because it prevents Pods from gaining access to the underlying host in a way which could compromise it. It’s also a good illustration because this maps nicely onto the Pod Security Standards control of a similar name, and there is a Kyverno policy which covers this control here.

 1apiVersion: policy/v1beta1
 2kind: PodSecurityPolicy
 3metadata:
 4 name: qa
 5spec:
 6 hostNetwork: false
 7 hostIPC: false
 8 hostPID: false
 9 seLinux:
10 rule: RunAsAny
11 supplementalGroups:
12 rule: RunAsAny
13 runAsUser:
14 rule: RunAsAny
15 fsGroup:
16 rule: RunAsAny

Next, we will create a Role in the qa Namespace which allows the above PSP to be used.

 1kind: Role
 2apiVersion: rbac.authorization.k8s.io/v1
 3metadata:
 4 name: psp-qa-role
 5 namespace: qa
 6rules:
 7- apiGroups:
 8 - extensions
 9 resources:
10 - podsecuritypolicies
11 resourceNames:
12 - qa
13 verbs:
14 - use

Once the Role is created, we need to bind this role for the user chip in the qa Namespace.

 1kind: RoleBinding
 2apiVersion: rbac.authorization.k8s.io/v1
 3metadata:
 4 name: psp-qa-bind
 5 namespace: qa
 6subjects:
 7- kind: User
 8 name: chip
 9roleRef:
10 kind: Role
11 name: psp-qa-role
12 apiGroup: rbac.authorization.k8s.io

Here’s a “good” Pod which doesn’t use any host namespaces and is allowed.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4 labels:
 5 app: busybox
 6 name: goodpod
 7spec:
 8 automountServiceAccountToken: false
 9 hostIPC: false
10 containers:
11 - name: busybox
12 image: busybox:1.35
13 args:
14 - sleep
15 - 1d

Here’s a “bad” Pod and will be blocked by the qa PSP if created by user chip.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4 labels:
 5 app: busybox
 6 name: badpod
 7spec:
 8 automountServiceAccountToken: false
 9 hostIPC: true
10 containers:
11 - name: busybox
12 image: busybox:1.35
13 args:
14 - sleep
15 - 1d

Now that the baseline has been established, after installing Kyverno, introduce this policy into the cluster which provides the same level of validation as our qa PSP. Note a couple things here:

This is a ClusterPolicy which means it applies to the entire cluster however it has been scoped down to only match on the qa Namespace. This will be opened up later but for now, it will only be confined to that Namespace.
The policy matches on Pod and no other type of resource.
The policy is in Audit mode which means it will not block any Pods which violate the policy. We will change this a bit later to provide the same blocking action as our PSP.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: disallow-host-namespaces
 5 annotations:
 6 policies.kyverno.io/title: Disallow Host Namespaces
 7 policies.kyverno.io/category: Pod Security Standards (Baseline)
 8 policies.kyverno.io/severity: medium
 9 kyverno.io/kyverno-version: 1.6.0
10 kyverno.io/kubernetes-version: "1.22-1.23"
11 policies.kyverno.io/subject: Pod
12 policies.kyverno.io/description: >-
13 Host namespaces (Process ID namespace, Inter-Process Communication namespace, and
14 network namespace) allow access to shared information and can be used to elevate
15 privileges. Pods should not be allowed access to host namespaces. This policy ensures
16 fields which make use of these host namespaces are unset or set to `false`. 
17spec:
18 validationFailureAction: Audit
19 background: true
20 rules:
21 - name: host-namespaces
22 match:
23 any:
24 - resources:
25 kinds:
26 - Pod
27 namespaces:
28 - qa
29 validate:
30 message: >-
31 Sharing the host namespaces is disallowed. The fields spec.hostNetwork,
32 spec.hostIPC, and spec.hostPID must be unset or set to `false`. 
33 pattern:
34 spec:
35 =(hostPID): "false"
36 =(hostIPC): "false"
37 =(hostNetwork): "false"

With this Kyverno policy installed, assuming there are existing Pods (at least the “good” Pod test from above) in the qa Namespace, wait 10-20 seconds and check for Policy Reports. We want to ensure there are only results in the PASS column and none in the FAIL. Since the PSP is providing the same level of protection, Pods caught by the PSP and blocked will not get created and, therefore, Kyverno should not see any which violate the same controls.

1$ kubectl -n qa get policyreport
2NAME PASS FAIL WARN ERROR SKIP AGE
3cpol-disallow-host-namespaces 1 0 0 0 0 3s

Now that we’re looking good on the reports side, let’s introduce a more permissive PSP. This PSP is more open than our qa PSP in that it permits use of host namespaces. Since a Pod must pass through at least one PSP, the only way we can migrate away from PSP with it still enabled is to make PSP, at some point, maximally permissive so the Pods will pass through the API server and get caught by Kyverno. This may not look exactly like your permissive PSP, but it gives you an idea of the general flow. Create this PSP called permissive in the cluster.

 1apiVersion: policy/v1beta1
 2kind: PodSecurityPolicy
 3metadata:
 4 name: permissive
 5 annotations:
 6 seccomp.security.alpha.kubernetes.io/allowedProfileNames: "*"
 7spec:
 8 hostIPC: true
 9 hostPID: true
10 hostNetwork: true
11 seLinux:
12 rule: RunAsAny
13 supplementalGroups:
14 rule: RunAsAny
15 runAsUser:
16 rule: RunAsAny
17 fsGroup:
18 rule: RunAsAny
19 hostPorts:
20 - min: 0
21 max: 65535
22 volumes:
23 - '*'
24 allowedCapabilities:
25 - '*'

With the permissive PSP created, change the psp-qa-role Role in the qa Namespace to reference the new PSP instead. The only change we’ll make is to flip from qa to permissive under the resourceNames field.

 1kind: Role
 2apiVersion: rbac.authorization.k8s.io/v1
 3metadata:
 4 name: psp-qa-role
 5 namespace: qa
 6rules:
 7- apiGroups:
 8 - extensions
 9 resources:
10 - podsecuritypolicies
11 resourceNames:
12 - permissive
13 verbs:
14 - use

Because the chip user still has a binding to this Role, we should now be able to test with a sample “bad” Pod. We now expect that this will be permitted by the PSP and allowed in the cluster.

 1apiVersion: v1
 2kind: Pod
 3metadata:
 4 labels:
 5 app: busybox
 6 name: badpod
 7spec:
 8 automountServiceAccountToken: false
 9 hostIPC: true
10 containers:
11 - name: busybox
12 image: busybox:1.35
13 args:
14 - sleep
15 - 1d

Verify that the permissive PSP was the one evaluated by inspecting the annotations on the badpod Pod. Here we can see the annotation kubernetes.io/psp as assigned the value permissive indicating our more permissive PSP was the one evaluated by this Pod’s creation request.

1$ kubectl -n qa get po badpod -o yaml
2apiVersion: v1
3kind: Pod
4metadata:
5 annotations:
6 kubectl.kubernetes.io/last-applied-configuration: |
7 {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"busybox"},"name":"badpod","namespace":"qa"},"spec":{"automountServiceAccountToken":false,"containers":[{"args":["sleep","1d"],"image":"busybox:1.35","name":"busybox"}],"hostIPC":true}}
8 kubernetes.io/psp: permissive

After creating the “bad” Pod, wait a few seconds and then once again get the Kyverno Policy Report to confirm that Kyverno caught it instead. We should now see that there is one result in the FAIL column which indicates Kyverno saw our “bad” Pod violated the policy we created earlier.

1$ kubectl -n qa get policyreport
2NAME PASS FAIL WARN ERROR SKIP AGE
3cpol-disallow-host-namespaces 1 1 0 0 0 3m3s

Let’s inspect the contents of that policy report just to be sure. The following command assumes you have yq installed.

 1$ kubectl -n qa get policyreport cpol-disallow-host-namespaces -o jsonpath='{.results[?(@.result=="fail")]}' | yq -p json -
 2category: Pod Security Standards (Baseline)
 3message: 'validation error: Sharing the host namespaces is disallowed. The fields spec.hostNetwork, spec.hostIPC, and spec.hostPID must be unset or set to `false`. . rule host-namespaces failed at path /spec/hostIPC/'
 4policy: disallow-host-namespaces
 5resources:
 6 - apiVersion: v1
 7 kind: Pod
 8 name: badpod
 9 namespace: qa
10 uid: 60c5f78a-0ff0-4666-8705-d4f6f3e943d2
11result: fail
12rule: host-namespaces
13scored: true
14severity: medium
15source: kyverno
16timestamp:
17 nanos: 0
18 seconds: 1.684957284e+09

The “bad” Pod clearly failed and the policy report contains more details.

Now that you’re confident things are configured correctly, change the Kyverno policy to Enforce mode and attempt to create another “bad” Pod. You should see this is now blocked.

1spec:
2 validationFailureAction: Enforce

1Error from server: error when creating "badpod.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:
2
3resource Pod/qa/extrabadpod was blocked due to the following policies
4
5disallow-host-namespaces:
6 host-namespaces: 'validation error: Sharing the host namespaces is disallowed. The
7 fields spec.hostNetwork, spec.hostIPC, and spec.hostPID must be unset or set to
8 `false`. rule host-namespaces failed at path /spec/hostIPC/'

When you’re comfortable with these results, the Kyverno policy can be opened up further should you wish by removing the match on the qa Namespace so it applies across the entire cluster.

When all aspects of your cluster have been migrated to a permissive PSP, you may choose to deactivate the PodSecurityPolicy admission plugin in the cluster by removing the PodSecurityPolicy value from the --enable-admission-plugins flag, or inversely, add the PodSecurityPolicy value to the --disable-admission-plugins flag on the Kubernetes API server.

Blog: New time related JMESPath filters in Kyverno!

Sun, 19 Feb 2023 00:00:00 +0000

The v1.9 release of Kyverno added several time related JMESPath filters. With this addition, users now can add time based rules in their Kyverno policies. This blog post aims to describe those new additions.

What is “JMESPath”?

JMESPath (pronounced “James path”) is a JSON query language that allows you to declaratively specify how to extract elements from a JSON document. It is similar to JSONPath in Kubernetes. It can be used almost anywhere in Kyverno.

Users may note that, many policies can be written with simple overlay patterns, others require more detailed selection and transformation. The latter is where JMESPath is useful.

Kyverno supports all the JMESPath filters in the JMESPath specifications page and also adds several custom filters. Refer to Kyverno’s JMESPath docs for a comprehensive deep dive.

Why did we need these new filters?

Kyverno already supports all the available filters in the upstream JMESPath library and some custom filters as well. But there was a lack of time related filters that come in useful at several places in Kyverno, such as

When we want to add custom labels like last sync time.
Enforce a policy that checks whether the current time is after or before a specified time.
Perform job scheduling using cron expressions.

What has been added?

In v1.9 release, Kyverno added 11 new time related filters to fulfill the above requirements and many more as well.

Time_add: We have added a filter to do arithmetic addition of a duration to a given time. The time_add() filter takes a time and a duration and returns a time. time_add('2023-01-12T12:37:56-05:00','6h') results in the value "2023-01-12T18:37:56-05:00". It can come in useful purposes like adding some time to the current time and passing it as an argument for other time related filters.
Time_after: This is used to verify whether a time is after another time and returns a boolean. It can be used directly as a validation filter, where we can check if current time is greater than a specific time or not and make decisions based on that information.
Time_before: time_before() filter works similarly to the time_after() filter. This is used to verify whether a time is before another time and returns a boolean. time_before('2023-01-12T19:05:59Z','2023-01-13T19:05:59Z') results in the value true. It can be used directly as a validation filter, where we can check if current time is less than a specific time or not and make decisions based on that information.
Time_between: The time_between() filter completes the collection of time comparision filters. This filter allows us to check whether a time occurs between a start time and an end time. It is used in the cases that require both time_before() and time_after() filter. For example, time_between('2023-01-12T19:05:59Z','2023-01-01T19:05:59Z','2023-01-15T19:05:59Z') results in the value true. It can be used directly as a validation filter, where we can check if current time is between two times or not and make decisions based on that information.
Time_diff: The time_diff() filter is used to find how much time has passed between two time period. For example, time_diff('2023-01-10T00:00:00Z','2023-01-11T00:00:00Z') results in the value "24h0m0s". It has been added for a common usecase where we want to check, whether a certain amount of time has been passed since an important event or not, such as to check whether 4 hours have passed since last scan was finished, we can use time_diff() filter like this "{{ time_diff('{{metadata.scanFinishedOn}}','{{ time_now_utc() }}') }}"
Time_now: The time_now() filter, as the name suggests returns the current time. It has been introduced for the common case where we need current time as an argument for other filters. We require current time in several filters like time_add() and this is added for such cases.
Time_now_utc: The time_now_utc() filter works similar to time_now() where it returns the current time according to UTC. It has been introduced for the common case where we need current time as an argument for other filters.
Time_parse: The Time_parse() filter is an interesting addition where we can pass any time string in any format to convert it ino the Kyverno standard RFC 3339 format. It takes 2 arguments, the time format and the time string. For example, time_parse('Mon Jan 02 2006 15:04:05 -0700', 'Fri Jun 22 2022 17:45:00 +0100') results in the output of "2022-06-22T17:45:00+01:00". This field has been added to allow standardization such that all the timestamps added as metadata and annotations follow the RFC 3339 format.
Time_to_cron: In kyverno, we use cron in several places to perform job scheduling. The cron expression enables users to schedule tasks to run periodically at a specified date/time. And it’s naturally a great tool for automating lots of process runs, which otherwise would require human intervention. We have added a time_to_cron() filter to generate a cron expression from a time. For example, time_to_cron('2022-04-11T03:14:05-07:00') results in the output "14 3 11 4 1". This filter can be used to generate things like ClusterCleanupPolicy.
Time_truncate: The time_truncate() filter is used to round the given time to the nearest duration which is specified in the arguments. For example, time_truncate('2023-01-12T17:37:00Z','1h') results in the output "2023-01-12T17:00:00Z".
Time_utc: The time_utc() filter takes in a time in RFC 3339 format with a time offset and presents the same time in UTC/Zulu. It can be used in pair with time_parse() to convert it to UTC.

How to use it?

An in-depth guide on how to use JMESPath filters in kyverno is present in the Kyverno’s JMESPath docs.

Summary

JMESPath is a powerful and robust tool for selecting, extracting and manipulating time in JSON. With the addition of the new time based filters, we have extended the capabilities of JMES filters in kyverno and have opened new usecases.

Blog: Kyverno 1.9 Released

Wed, 01 Feb 2023 00:00:00 +0000

With the ringing in of the new year the Kyverno team is proud to announce the release of Kyverno 1.9.0, a release that we’ve been working hard on over the past several months and which is full of massive new features include TWO brand new rule types, several community-requested enhancements, and loads of fixes and improvements. Without any further ado, let’s dive right in and show you what’s in the bag.

Key New Features of Kyverno 1.9

Policy Exceptions

Something we’ve heard for a while now from the community was that they love the ease with which Kyverno can select resources (and, really, just ease of use in general) for action by policies but that this was limiting in certain ways. It may not be scalable to modify every policy with the same exclusions, and, secondarily, in real-world ops scenarios everyone has special exceptions that need careful handling. So we’re excited to show off our new PolicyExceptions in 1.9!

A PolicyException is a new type of policy and Custom Resource in Kyverno 1.9 which allows you to create an exception to any existing Kyverno policy in a way which is decoupled from the policy itself and so doesn’t involve modifying the policy.

Here’s what it would look like to allow a special exception to one of the Pod Security Standard policies but only for a very specific resource named important-tool.

 1apiVersion: kyverno.io/v2alpha1
 2kind: PolicyException
 3metadata:
 4 name: delta-exception
 5 namespace: delta
 6spec:
 7 exceptions:
 8 - policyName: disallow-host-namespaces
 9 ruleNames:
10 - host-namespaces
11 - autogen-host-namespaces
12 match:
13 any:
14 - resources:
15 kinds:
16 - Pod
17 - Deployment
18 namespaces:
19 - delta
20 names:
21 - important-tool*

Once a PolicyException is created, a resource which is selected by the exception and also applies to the policy and rule combo named in the exception will be allowed to circumvent the policy. As shown, it’ll be quite useful, we think, for users who want those one-time exceptions to validate rules which are in Enforce mode although it works for other situations and rules as well. Access to creating PolicyExceptions can be controlled through a combination of Kubernetes RBAC, your own GitOps review processes, YAML signing for integrity and tamper-prevention checks, and even Kyverno validate policies.

Cleanup Policies

With Kyverno’s unique generation rule, which remains one of its most popular abilities and is beloved by ops teams everywhere, it has proven that Kyverno is far more capable than a simple admission controller which can only deliver “yes” or “no” responses. Generation, especially when combined with validation and mutation, unlocks tremendous power and turns Kyverno into a true tool of automation. That was a great first step, but we knew we could do more. We heard there was a piece missing from the story and that piece was the ability to delete as well as create. Well, we’re really happy to say that as of Kyverno 1.9, our second new policy type will allow you to do just that: cleanup resources on a scheduled basis.

A Kyverno CleanupPolicy brings the all-familiar match and exclude capabilities with the powerful expressions and JMESPath filtering system together with a scheduler which allows you to very granularly select and filter resources you want removed in your cluster based on a recurring schedule. Just take a look at the below sample.

 1apiVersion: kyverno.io/v2alpha1
 2kind: ClusterCleanupPolicy
 3metadata:
 4 name: clean-nekkid-pods
 5 annotations:
 6 pod-policies.kyverno.io/autogen-controllers: none
 7spec:
 8 match:
 9 any:
10 - resources:
11 kinds:
12 - Pod
13 conditions:
14 all:
15 - key: "{{ target.metadata.ownerReferences[] || `[]` }}"
16 operator: Equals
17 value: []
18 schedule: "0 0 * * *"

Here is a ClusterCleanupPolicy which, as you might have guessed, applies cluster-wide, and which will find and remove Pods with no ownerReferences every day at midnight. Cleanup policies like these can be super helpful to keep clutter to a minimum and fit nicely into existing automation workflows.

A policy-based approach is only the first step, and in the second phase we intend on implementing automated cleanup based on a reserved label or annotation assigned to resources.

Distributed Tracing

In the 1.8 release, we added OpenTelemetry support, but in this release we wanted to one-up ourselves and add in more observability to provide insights on exactly what Kyverno is doing to your resources. So in 1.9, we’ve added full support for distributed tracing. With distributed tracing instrumented in Kyverno, you can see every rule and even every external call (in the case of OCI registry lookups) Kyverno is making and how long each span took. This is super helpful not just from a visibility but also a troubleshooting and even auditability standpoint. With traces in hand sent to your collector, you can see the rules which matched the resource and were processed so you know everything is working properly.

Extended Support for Subresources

Kyverno has had support for some subresources for a few releases now (ephemeral containers have been supported since 1.7), but there were still gaps. Specifically, Kyverno had problems with the Scale subresource, which can be tricky to deal with, as well as mutations to the Status subresource. This meant that a couple real-world use cases that have come up simply weren’t possible to implement in Kyverno policies. With Kyverno 1.9, that should be a thing of the past and most, if not all, subresources can be handled quite well.

With this enhanced support, you can now easily match on the precise subresource you want (even using wildcards) and Kyverno will figure it out. For example, advertising extended resources to Kubernetes nodes is important for use cases like GPUs and FPGAs, and with a Kyverno mutate rule targeted at Node/status, it’s now incredibly simple and doesn’t require any custom webhooks. The below policy will add the example.com/dongle resource to new Nodes brought up in the cluster.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: advertise-resource
 5spec:
 6 background: false
 7 rules:
 8 - name: advertise-dongle
 9 match:
10 any:
11 - resources:
12 kinds:
13 - Node/status
14 mutate:
15 patchStrategicMerge:
16 status:
17 capacity:
18 example.com/dongle: 2

ConfigMap Caching

ConfigMaps are a common source of data not just to Pods but to Kyverno policies as well. Kyverno has long had support for dynamic lookups from ConfigMaps without requiring any pre-syncing, and in most use cases this was fine. But as more users flocked to Kyverno with ever larger clusters, these additional API calls produced some unwanted overhead. Starting in Kyverno 1.9, if you would like Kyverno to cache the ConfigMaps you need for policy decisions, simply assign the label cache.kyverno.io/enabled: "true" to any ConfigMap and Kyverno will automatically cache it for you. Nothing else you need to do.

Other Additions and Enhancements

Some other cool features and status updates we’re proud to share include new JSON logging format and the ability to dump out the full AdmissionReview contents the API server sends to Kyverno. The latter has been a common request from the community and should help in those situations where you’re not quite sure how to write a certain policy. Both require setting a new container flag so check out the docs.

Kyverno now supports nested foreach loops which are great especially for mutation use cases where you need tactical modifications or removals from complex nested objects like arrays within arrays.

There are eleven new JMESPath filters specifically for working with time. These filters include things like getting current time, converting it from different formats, and even translating the time to a Cron expression. These are sure to come in handy when writing policies which account for time in various ways.

Pod controller rule auto-generation, another of Kyverno’s hallmark capabilities which if you’re not familiar allows Kyverno to translate rules solely for Pods to all other major Pod controllers, added support for ReplicaSet and ReplicationController. There’s nothing policy authors need to do and Kyverno will now add these two resources into the auto-gen rules for you.

The Kyverno CLI now has experimental support for uploading and downloading Kyverno policies as OCI artifacts. This is great for storing policies alongside container images using the same, common semantics. In the future, we’re planning on doing more with this support so keep a lookout.

We’re preparing for the movement away from v1 of our policy schema which has served us well for a while now. As we go towards v2beta1, which is available in 1.9, we get rid of deprecated and obsoleted fields to make a nice and tidy policy. We ask you to start moving in the direction of v2beta1 now to make the removal process much smoother.

Kyverno 1.9 brings support for Kubernetes 1.26. Kyverno follows an N-2 support policy, and so to ensure we’re staying current, we’re now building and testing up to 1.26.

We have long been a champion of software supply chain security and insist on enacting these practices ourselves. With the 1.9 release, Kyverno now generates and attests to SLSA provenance which anyone on their end can publicly verify. We believe this and other policies by which we abide makes us adherent to SLSA Level 3 and will be working with the Open Source Security Foundation to ensure this is the case.

And last but not least, the Kyverno policy library, the largest community-driven library of any policy engine for Kubernetes, has received another large bump putting it well over the 250 mark. Included in this library update is a new table which maps Gatekeeper policies to Kyverno policies, helpful for users of both tools to see how to accomplish some common use cases in these engines.

Potentially Breaking Changes

One change we do want to make you aware of, which actually came in 1.8.3, which could be breaking is a schema modification for verifying a container image attestation. Due to some upstream changes in Sigstore’s cosign, we had to move the attestors under the attestation being verified. This necessitated a schema change which you can find in the documentation. So if you have Kyverno policies which verified image attestations, please update them to the new schema so they continue to work in 1.9.

Closing

With so many new features, enhancements, and well over 200 fixes, there’s so much to get excited about in this release. Hopefully what you’ve seen makes you excited to try out the 1.9 release and provide your feedback. Come engage with us in the Kyverno channel on Kubernetes Slack, attend one of our community meetings, or just catch us on Twitter.

Blog: Kyverno and SLSA 3

Wed, 01 Feb 2023 00:00:00 +0000

With the release of Kyverno 1.9, Kyverno has begun generating and attesting to the provenance of its release artifacts in the SLSA standard and provisionally meet Level 3. This blog post attempts to explain a bit about SLSA and Level 3 and how we meet the requirements. Once the Open Source Security Foundation (OpenSSF) establishes its conformance program, we hope to see official acknowledgement of this process.

About SLSA

Supply Chain Levels for Software Artifacts, or SLSA (pronounced “salsa”), is a security framework which aims to prevent tampering and secure artifacts in a project. SLSA helps in mitigating supply chain threats. SLSA compliance is based on four levels. Level 1 starts with basic requirements and achieving level 4 requires strict hardening of the supply chain platform.

The following diagram depicts the several known points of attacks. For extensive documentation, refer to https://slsa.dev.

(Graphic source: slsa.dev)

SLSA Requirements and Kyverno Compliance State

SLSA divides the requirements into three main areas: source, build, and integrity. Each area has sub-requirements to meet certain levels. Since Kyverno claims to achieve SLSA level 3, this document will explain how we are achieving that. For a complete summary of SLSA requirements, refer to the official SLSA project pages at https://slsa.dev/spec/v0.1/requirements.

Source Requirements

Requirement	Required at SLSA Level 3	Met by Kyverno
Version controlled	Yes	Yes
Verified history	Yes	Yes
Retained indefinitely	Yes (for 18 months or above)	Yes

Version Controlled

Description: The source code should be tracked in a version-controlled system. The version control system should maintain the history of changes. The identification of uploaders and reviewers (if any), timestamps of the reviews, its content, and the parent reviews. Each revision should be tracked back using an immutable reference. These requirements are met by most of the revision systems, e.g git.

Kyverno Processes: Kyverno uses GitHub for source code management. Each commit gives info about the author, an explanation about the commit, and the time at which the commit was generated.

Verified History

Description: The history of a revision must be trackable. It should contain a timestamp and a strongly authenticated user (author, uploader, reviewer, etc.). The identities must use two-step verification or similar.

Kyverno Processes: Each commit in Kyverno contains a timestamp and info about the author. It also gives an explanation about the commit.

Retained Indefinitely

Description: If there are no legal or policy requirements, the revision and its change history must be preserved indefinitely and cannot be deleted. For SLSA Level 3, the retention can be limited to 18 months.

Kyverno Processes: The git tree remains unaltered. The source code of the build is archived.

Build Requirements

The conversion of source code into the build is the responsibility of the build system. The build system must be secured against any sort of outside intervention. The build system should produce the builds in a reproducible manner for verification.

Requirement	Required at SLSA Level 3	Met by Kyverno
Scripted build	Yes	Yes
Build service	Yes	Yes
Build as code	Yes	Yes
Ephemeral environment	Yes	Yes
Isolated	Yes	Yes

Scripted Build

Description: All build steps should be defined in the build script. e.g Makefile or GitHub action workflow file. The invocation command for build script is the only manual command allowed.

Kyverno Processes: Since the Kyverno code base is stored in GitHub, GitHub Actions are used to invoke build scripts. Kyverno uses a mix of Makefile and GitHub Workflows. Everything is declared inside build scripts.

Build Service

Description: The build step should be executed on some build service environment e.g GitHub Action, AWS CodePipeline. The developer workstation doesn’t qualify as a build service.

Kyverno Processes: As mentioned above, the Kyverno source code is stored in GitHub. Kyverno relies on GitHub Actions as a build service environment.

Build As Code

Description: The build definition and configuration used by the build service to generate the build must be derived from text files. The build-as-code files need to be stored in the version control system.

Kyverno Processes: The build is generated by executing GitHub workflows. The build definition and configuration are all defined in workflow files under the directory .github/workflows/ in the kyverno/kyverno repository.

Ephemeral Environment

Description: The build step should be executed in an ephemeral environment. The environment should be solely created for this build and must not be reused. e.g container or VM.

Kyverno Processes: Kyverno relies on GitHub Actions as a build service environment. GitHub uses the concept of runners and the Kyverno project uses GitHub-hosted runners. For each job, a new virtual machine is created.

Isolated

Description: The build step should be executed independently of each other. The build instance, either prior or concurrent, should not influence others.

Kyverno Processes: The GitHub-hosted runners run each job in a separate virtual machine. When the job has finished, the VM is automatically decommissioned.

Provenance Requirements

In order to prove that build was produced and artifacts were produced according to SLSA Level 3, SLSA mentions some requirements for provenance which can be grouped into the following:

Requirements on the process by which provenance is generated and consumed
Requirements on the content of the provenance

Kyverno relies on the official SLSA GitHub Generator project for provenance generation. For more information on the generator project, see https://github.com/slsa-framework/slsa-github-generator.

Requirements on the process by which provenance is generated and consumed

Requirement	Required at SLSA Level 3	Met by Kyverno
Available	Yes	Yes
Authenticated	Yes	Yes
Service generated	Yes	Yes
Non-falsifiable	Yes	Yes

Available

Description: The provenance should be provided to the customer in a format accepted by the customer.

Kyverno Processes: Kyverno relies on the official SLSA GitHub generator framework to provide provenance in the in-toto format. The provenance is attested using the Sigstore cosign project and is publicly available as an image artifact on GitHub Container Registry stored in the same repository as the Kyverno container images themselves. This repository and its contents can be publicly pulled without authentication.

Authenticated

Description: The consumer can verify the authenticity and integrity of the provenance.

Kyverno Processes: The provenance is signed by OIDC identity and the public key to verify the provenance is stored in the public Rekor transparency log. If someone tampers with the file then the signature will fail to verify. Anyone can verify the provenance for Kyverno artifacts with the process documented here.

Service Generated

Description: The provenance data should be obtained from the build service.

Kyverno Processes: The GitHub Action reusable workflow hosted by the SLSA GitHub Generator project creates the provenance file. The caller workflow from the Kyverno project is release.yaml and the called workflow is generator_container_slsa3.yml. The necessary data to the called workflow is passed in the form of variables and secrets. The called workflow is a third-party workflow which is used by Kyverno for provenance generation and any actions in the called workflow run as if they were part of the caller workflow.

Non-Falsifiable

Description: The build service’s users can not falsify the provenance.

Kyverno Processes: GitHub takes care of avoiding interference with the build system. GitHub uses ephemeral and isolated virtual machines, no one can persistently compromise this environment. GitHub automatically provisions a new VM for that job. When the job execution is finished, the VM is automatically decommissioned. Use of the SLSA GitHub generator separates the signing from building so the Kyverno build itself never has access to the signing secrets. Use of OIDC-based secrets through Sigstore’s keyless signing means the ephemeral signing secret is associated only with one specific build making it easy to detect secret theft and an attempt at signing something else.

Provenance Content Requirements

Requirement	Required at SLSA Level 3	Met by Kyverno
Identifies artifact	Yes	Yes
Identifies builder	Yes	Yes
Identifies build instructions	Yes	Yes
Identifies source code	Yes	Yes
Identifies entry point	Yes	Yes
Includes all build parameters	Yes	Yes

Identifies Artifact

Description: The output artifact must be identified by provenance by at least one cryptographic hash.

Kyverno Processes: The provenance file stores SHA-256 hashes of build artifacts. The provenance identifies the container image using its digest in SHA-256 format.

Identifies Builder

Description: The entity who executed the build process and generated the provenance should be identified by provenance.

Kyverno Processes: The id of the builder is added to provenance in the predicate.builder section. In case of Kyverno, GitHub Actions act as the builder. The build logic is defined inside GitHub workflows file. The provenance generation logic is in a separate reusable workflow which is recorded in the predicate.builder section. As an example, refer to the Provenance Example section.

Identifies Build Instructions

Description: The top-level instruction that was executed to initiate the build should be available in the provenance file.

Kyverno Processes: The top-level instruction is the GitHub Action workflow which is calling the provenance generation workflow. This is recorded in invocation.entrypoint in the provenance file. In Kyverno’s case, it is release.yaml. The SLSA requirements mentions that “The identified instructions should be at the highest level available to the build” so it doesn’t necessarily need to record all the the instructions as they are part of workflow file. As an example, refer to the Provenance Example section.

Identifies Source Code

Description: The repository origin of the source code used in the build must be identified in provenance.

Kyverno Processes: The repository information is stored in the provenance file. This is recorded in the predicate.invocation section. As an example, refer to the Provenance Example section.

Identifies Entry Point

Description: The processes which started the build processes should be identified by provenance.

Kyverno Processes: The required information is stored in the invocation.configSource.entryPoint in the provenance file. As an example, refer to the Provenance Example section.

Include All Build Parameters

Description: The provenance must include all the build parameters which are under user control.

Kyverno Processes: The main GitHub Action workflow file does not accept any parameters. The workflow_dispatch section in the release.yaml file does not accept any parameters. At SLSA level 3 the parameters, if any, must be listed. At SLSA level 4, the parameters must be empty.

Provenance Example

The following is an example of the generated provenance for Kyverno which will be returned from the process documented here.

 1{
 2 "_type": "https://in-toto.io/Statement/v0.1",
 3 "predicateType": "https://slsa.dev/provenance/v0.2",
 4 "subject": [
 5 {
 6 "name": "ghcr.io/kyverno/kyverno",
 7 "digest": {
 8 "sha256": "96a54a5747485b800adf05ff84d48fc9a8b66f1cdf9087cbed385dacc1cdb4d6"
 9 }
 10 }
 11 ],
 12 "predicate": {
 13 "builder": {
 14 "id": "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v1.4.0"
 15 },
 16 "buildType": "https://github.com/slsa-framework/slsa-github-generator/container@v1",
 17 "invocation": {
 18 "configSource": {
 19 "uri": "git+https://github.com/kyverno/kyverno@refs/tags/v1.9.0-beta.2",
 20 "digest": {
 21 "sha1": "ab368ebc080535448aa08fafdff82a4d4c69e127"
 22 },
 23 "entryPoint": ".github/workflows/release.yaml"
 24 },
 25 "parameters": {},
 26 "environment": {
 27 "github_actor": "realshuting",
 28 "github_actor_id": "25727662",
 29 "github_base_ref": "",
 30 "github_event_name": "push",
 31 "github_event_payload": {
 32 "after": "9d6e981a64b0b8dc357befb7fc811304d7b11f7b",
 33 "base_ref": null,
 34 "before": "0000000000000000000000000000000000000000",
 35 "commits": [],
 36 "compare": "https://github.com/kyverno/kyverno/compare/v1.9.0-beta.2",
 37 "created": true,
 38 "deleted": false,
 39 "enterprise": {
 40 "avatar_url": "https://avatars.githubusercontent.com/b/9995?v=4",
 41 "created_at": "2021-11-03T14:57:36Z",
 42 "description": "",
 43 "html_url": "https://github.com/enterprises/cncf",
 44 "id": 9995,
 45 "name": "Cloud Native Computing Foundation",
 46 "node_id": "E_kgDNJws",
 47 "slug": "cncf",
 48 "updated_at": "2022-12-27T16:04:12Z",
 49 "website_url": "https://cncf.io"
 50 },
 51 "forced": false,
 52 "head_commit": {
 53 "author": {
 54 "email": "shuting@nirmata.com",
 55 "name": "shuting",
 56 "username": "realshuting"
 57 },
 58 "committer": {
 59 "email": "noreply@github.com",
 60 "name": "GitHub",
 61 "username": "web-flow"
 62 },
 63 "distinct": true,
 64 "id": "ab368ebc080535448aa08fafdff82a4d4c69e127",
 65 "message": "tag v1.9.0-beta.2 (#5959)",
 66 "timestamp": "2023-01-10T11:45:52Z",
 67 "tree_id": "95c14e810108e4125c571ba33483204f4b078900",
 68 "url": "https://github.com/kyverno/kyverno/commit/ab368ebc080535448aa08fafdff82a4d4c69e127"
 69 },
 70 "organization": {
 71 "avatar_url": "https://avatars.githubusercontent.com/u/68448710?v=4",
 72 "description": "Kubernetes Native Policy Management",
 73 "events_url": "https://api.github.com/orgs/kyverno/events",
 74 "hooks_url": "https://api.github.com/orgs/kyverno/hooks",
 75 "id": 68448710,
 76 "issues_url": "https://api.github.com/orgs/kyverno/issues",
 77 "login": "kyverno",
 78 "members_url": "https://api.github.com/orgs/kyverno/members{/member}",
 79 "node_id": "MDEyOk9yZ2FuaXphdGlvbjY4NDQ4NzEw",
 80 "public_members_url": "https://api.github.com/orgs/kyverno/public_members{/member}",
 81 "repos_url": "https://api.github.com/orgs/kyverno/repos",
 82 "url": "https://api.github.com/orgs/kyverno"
 83 },
 84 "pusher": {
 85 "email": "shutting06@gmail.com",
 86 "name": "realshuting"
 87 },
 88 "ref": "refs/tags/v1.9.0-beta.2",
 89 "repository": {
 90 "allow_forking": true,
 91 "archive_url": "https://api.github.com/repos/kyverno/kyverno/{archive_format}{/ref}",
 92 "archived": false,
 93 "assignees_url": "https://api.github.com/repos/kyverno/kyverno/assignees{/user}",
 94 "blobs_url": "https://api.github.com/repos/kyverno/kyverno/git/blobs{/sha}",
 95 "branches_url": "https://api.github.com/repos/kyverno/kyverno/branches{/branch}",
 96 "clone_url": "https://github.com/kyverno/kyverno.git",
 97 "collaborators_url": "https://api.github.com/repos/kyverno/kyverno/collaborators{/collaborator}",
 98 "comments_url": "https://api.github.com/repos/kyverno/kyverno/comments{/number}",
 99 "commits_url": "https://api.github.com/repos/kyverno/kyverno/commits{/sha}",
100 "compare_url": "https://api.github.com/repos/kyverno/kyverno/compare/{base}...{head}",
101 "contents_url": "https://api.github.com/repos/kyverno/kyverno/contents/{+path}",
102 "contributors_url": "https://api.github.com/repos/kyverno/kyverno/contributors",
103 "created_at": 1549297548,
104 "default_branch": "main",
105 "deployments_url": "https://api.github.com/repos/kyverno/kyverno/deployments",
106 "description": "Kubernetes Native Policy Management",
107 "disabled": false,
108 "downloads_url": "https://api.github.com/repos/kyverno/kyverno/downloads",
109 "events_url": "https://api.github.com/repos/kyverno/kyverno/events",
110 "fork": false,
111 "forks": 479,
112 "forks_count": 479,
113 "forks_url": "https://api.github.com/repos/kyverno/kyverno/forks",
114 "full_name": "kyverno/kyverno",
115 "git_commits_url": "https://api.github.com/repos/kyverno/kyverno/git/commits{/sha}",
116 "git_refs_url": "https://api.github.com/repos/kyverno/kyverno/git/refs{/sha}",
117 "git_tags_url": "https://api.github.com/repos/kyverno/kyverno/git/tags{/sha}",
118 "git_url": "git://github.com/kyverno/kyverno.git",
119 "has_discussions": true,
120 "has_downloads": true,
121 "has_issues": true,
122 "has_pages": true,
123 "has_projects": true,
124 "has_wiki": true,
125 "homepage": "https://kyverno.io",
126 "hooks_url": "https://api.github.com/repos/kyverno/kyverno/hooks",
127 "html_url": "https://github.com/kyverno/kyverno",
128 "id": 169108858,
129 "is_template": false,
130 "issue_comment_url": "https://api.github.com/repos/kyverno/kyverno/issues/comments{/number}",
131 "issue_events_url": "https://api.github.com/repos/kyverno/kyverno/issues/events{/number}",
132 "issues_url": "https://api.github.com/repos/kyverno/kyverno/issues{/number}",
133 "keys_url": "https://api.github.com/repos/kyverno/kyverno/keys{/key_id}",
134 "labels_url": "https://api.github.com/repos/kyverno/kyverno/labels{/name}",
135 "language": "Go",
136 "languages_url": "https://api.github.com/repos/kyverno/kyverno/languages",
137 "license": {
138 "key": "apache-2.0",
139 "name": "Apache License 2.0",
140 "node_id": "MDc6TGljZW5zZTI=",
141 "spdx_id": "Apache-2.0",
142 "url": "https://api.github.com/licenses/apache-2.0"
143 },
144 "master_branch": "main",
145 "merges_url": "https://api.github.com/repos/kyverno/kyverno/merges",
146 "milestones_url": "https://api.github.com/repos/kyverno/kyverno/milestones{/number}",
147 "mirror_url": null,
148 "name": "kyverno",
149 "node_id": "MDEwOlJlcG9zaXRvcnkxNjkxMDg4NTg=",
150 "notifications_url": "https://api.github.com/repos/kyverno/kyverno/notifications{?since,all,participating}",
151 "open_issues": 296,
152 "open_issues_count": 296,
153 "organization": "kyverno",
154 "owner": {
155 "avatar_url": "https://avatars.githubusercontent.com/u/68448710?v=4",
156 "email": "kyverno@googlegroups.com",
157 "events_url": "https://api.github.com/users/kyverno/events{/privacy}",
158 "followers_url": "https://api.github.com/users/kyverno/followers",
159 "following_url": "https://api.github.com/users/kyverno/following{/other_user}",
160 "gists_url": "https://api.github.com/users/kyverno/gists{/gist_id}",
161 "gravatar_id": "",
162 "html_url": "https://github.com/kyverno",
163 "id": 68448710,
164 "login": "kyverno",
165 "name": "kyverno",
166 "node_id": "MDEyOk9yZ2FuaXphdGlvbjY4NDQ4NzEw",
167 "organizations_url": "https://api.github.com/users/kyverno/orgs",
168 "received_events_url": "https://api.github.com/users/kyverno/received_events",
169 "repos_url": "https://api.github.com/users/kyverno/repos",
170 "site_admin": false,
171 "starred_url": "https://api.github.com/users/kyverno/starred{/owner}{/repo}",
172 "subscriptions_url": "https://api.github.com/users/kyverno/subscriptions",
173 "type": "Organization",
174 "url": "https://api.github.com/users/kyverno"
175 },
176 "private": false,
177 "pulls_url": "https://api.github.com/repos/kyverno/kyverno/pulls{/number}",
178 "pushed_at": 1673351226,
179 "releases_url": "https://api.github.com/repos/kyverno/kyverno/releases{/id}",
180 "size": 75072,
181 "ssh_url": "git@github.com:kyverno/kyverno.git",
182 "stargazers": 3340,
183 "stargazers_count": 3340,
184 "stargazers_url": "https://api.github.com/repos/kyverno/kyverno/stargazers",
185 "statuses_url": "https://api.github.com/repos/kyverno/kyverno/statuses/{sha}",
186 "subscribers_url": "https://api.github.com/repos/kyverno/kyverno/subscribers",
187 "subscription_url": "https://api.github.com/repos/kyverno/kyverno/subscription",
188 "svn_url": "https://github.com/kyverno/kyverno",
189 "tags_url": "https://api.github.com/repos/kyverno/kyverno/tags",
190 "teams_url": "https://api.github.com/repos/kyverno/kyverno/teams",
191 "topics": [
192 "kubernetes",
193 "policy-management"
194 ],
195 "trees_url": "https://api.github.com/repos/kyverno/kyverno/git/trees{/sha}",
196 "updated_at": "2023-01-10T06:16:33Z",
197 "url": "https://github.com/kyverno/kyverno",
198 "visibility": "public",
199 "watchers": 3340,
200 "watchers_count": 3340,
201 "web_commit_signoff_required": true
202 },
203 "sender": {
204 "avatar_url": "https://avatars.githubusercontent.com/u/25727662?v=4",
205 "events_url": "https://api.github.com/users/realshuting/events{/privacy}",
206 "followers_url": "https://api.github.com/users/realshuting/followers",
207 "following_url": "https://api.github.com/users/realshuting/following{/other_user}",
208 "gists_url": "https://api.github.com/users/realshuting/gists{/gist_id}",
209 "gravatar_id": "",
210 "html_url": "https://github.com/realshuting",
211 "id": 25727662,
212 "login": "realshuting",
213 "node_id": "MDQ6VXNlcjI1NzI3NjYy",
214 "organizations_url": "https://api.github.com/users/realshuting/orgs",
215 "received_events_url": "https://api.github.com/users/realshuting/received_events",
216 "repos_url": "https://api.github.com/users/realshuting/repos",
217 "site_admin": false,
218 "starred_url": "https://api.github.com/users/realshuting/starred{/owner}{/repo}",
219 "subscriptions_url": "https://api.github.com/users/realshuting/subscriptions",
220 "type": "User",
221 "url": "https://api.github.com/users/realshuting"
222 }
223 },
224 "github_head_ref": "",
225 "github_ref": "refs/tags/v1.9.0-beta.2",
226 "github_ref_type": "tag",
227 "github_repository_id": "169108858",
228 "github_repository_owner": "kyverno",
229 "github_repository_owner_id": "68448710",
230 "github_run_attempt": "1",
231 "github_run_id": "3883016519",
232 "github_run_number": "232",
233 "github_sha1": "ab368ebc080535448aa08fafdff82a4d4c69e127"
234 }
235 },
236 "metadata": {
237 "buildInvocationID": "3883016519-1",
238 "completeness": {
239 "parameters": true,
240 "environment": false,
241 "materials": false
242 },
243 "reproducible": false
244 },
245 "materials": [
246 {
247 "uri": "git+https://github.com/kyverno/kyverno@refs/tags/v1.9.0-beta.2",
248 "digest": {
249 "sha1": "ab368ebc080535448aa08fafdff82a4d4c69e127"
250 }
251 }
252 ]
253 }
254}

Summary

The SLSA project is providing huge gains in assuring the integrity of the software supply chain.

Blog: Kyverno CVE-2022-47633 affecting image verification

Thu, 29 Dec 2022 00:00:00 +0000

Summary

Kyverno versions 1.8.3 and 1.8.4 contained a regression (CVE-2022-47633) which allowed a malicious proxy to facilitate a man-in-the-middle (MiTM) attack allowing an unsigned image to run in a Kubernetes cluster even if there was a Kyverno policy installed to ensure only signed images were permitted. This was due to there being two calls to the registry using the image tag, when there only should have been one, allowing a different digest to be returned on the second call than the first. The issue has been fixed in Kyverno 1.8.5.

Details

On December 14th 2022, Ben Hirschberg of ARMO, reported a vulnerability found in Kyverno 1.8.3 via the Kyverno security mailing list in which an unsigned image was permitted to run when a verifyImages rule existed which required only signed images. The vulnerability was introduced in Kyverno version 1.8.3.

To exploit this vulnerability, a malicious proxy or registry is used to replace the digest of a signed image with that of an unsigned image by exploiting a software bug that made a second call to the OCI registry using the mutable image tag. The attack requires that the attacker have the ability to inject a malicious proxy and requires that a workload references the proxy, or that the attacker takes control of a trusted OCI registry. The attack also requires that the admission controller does not have the ability to replace image tags with digests on first use and makes multiple calls using an image reference that uses a tag in a manner where different images can be sent by the malicious proxy or compromised registry back to the cluster.

The official advisory is available at: https://github.com/kyverno/kyverno/security/advisories/GHSA-m3cq-xcx9-3gvm.

Response and Fix

Prior to the December 14th disclosure, Ben had been in contact with the Kyverno team about another potential issue which was later disproved. While continuing testing, Ben uncovered this new vulnerability. The security team from Nirmata, the primary maintainers of Kyverno, engaged within hours of the initial report and worked over email and a private GitHub repository to investigate and validate the ultimate claim disclosed here. The root cause was quickly identified and a fix pushed to the upstream Kyverno repository within three days of verification. Kyverno 1.8.5 release candidate was released on Monday, December 19 2022 containing this fix and a GA release occurred two days later on Wednesday.

Additional Mitigations

Checking image signatures is a good starting point but not a complete solution to securing the software supply chain. Additional policies should be used to:

Require that only trusted registries are used (sample policy).
Ensure attestations (i.e. signed metadata) are checked for provenance and other image attributes (docs).

Requiring trusted registries would prevent this attack, as users cannot use the malicious proxy or other untrusted registries.

Securing Critical Software

Kubernetes admission controllers are critical components of any cluster and need to be battle tested to ensure they are reliable, resilient, and themselves secure. A community and OSS model allows wider use and enables multiple organizations to rapidly test, break, and fix issues in a collaborative manner. When choosing an admission controller, it is important to make sure that there is broad adoption, the admission controller has been designed and developed with security in mind, and thoroughly tested. Community based OSS models optimize these attributes and hence become the best choice for critical infrastructure.

Blog: Kyverno 1.8 Released

Mon, 24 Oct 2022 00:00:00 +0000

Following on the heels of the 1.7 release of Kyverno, the Kyverno team is proud to present version 1.8 which is another huge leap forward not just in terms of features and functionality but of optimizations, performance, and other improvements required in strict or high-scale environments. And in addition to those, a tremendous amount of work went into refactoring and other housekeeping items that make Kyverno cleaner and more efficient making future development (and contributions) easier, quicker, and ultimately more maintainable. We’ll walk through the largest of these features in this article.

Key New Features of Kyverno 1.8

Pod Security Integration

The successor to Kubernetes’ Pod Security Policy (PSP) is Pod Security Admission, enabled by default in 1.23 and stable now in 1.25. This new technology implements a set of standards dubbed Pod Security Standards. Pod Security Admission brings many benefits over PSPs but also some fairly important caveats. Starting in 1.8, Kyverno has a new validate subrule called podSecurity which internally uses the same libraries as Pod Security Admission but allows for much simpler implementation of those standards while offering flexible exemptions not found in Pod Security Admission. Shown below is an example of this new podSecurity rule in action which implements the entire restricted profile of the Pod Security Standards across the entire cluster.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: podsecurity-restricted
 5spec:
 6 background: true
 7 validationFailureAction: audit
 8 rules:
 9 - name: restricted
10 match:
11 any:
12 - resources:
13 kinds:
14 - Pod
15 validate:
16 podSecurity:
17 level: restricted
18 version: latest

YAML Manifest Verification

Although Kyverno has integrated with Sigstore tooling for some time now, offering capabilities like container image signature verification and attestation verification, Kyverno 1.8 steps this up one notch further by bringing support for Sigstore’s manifest project. With this integration, Kyverno is now additionally able to verify signatures on Kubernetes YAML manifests to ensure, like container images, that they haven’t been tampered with. Once a manifest has been signed with a private key of a user’s choosing, a new Kyverno policy may be written which verifies the signature and comparing the signed (original) manifest contents with the current contents. Shown here is an example of such a policy which verifies the key used to sign Deployments.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: verify-manifest-integrity
 5spec:
 6 validationFailureAction: audit
 7 background: true
 8 rules:
 9 - name: verify-deployment-allow-replicas
10 match:
11 any:
12 - resources:
13 kinds:
14 - Deployment
15 validate:
16 manifests:
17 attestors:
18 - count: 1
19 entries:
20 - keys:
21 publicKeys: |-
22 -----BEGIN PUBLIC KEY-----
23 MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEStoX3dPCFYFD2uPgTjZOf1I5UFTa
24 1tIu7uoGoyTxJqqEq7K2aqU+vy+aK76uQ5mcllc+TymVtcLk10kcKvb3FQ==
25 -----END PUBLIC KEY----- 
26 ignoreFields:
27 - objects:
28 - kind: Deployment
29 fields:
30 - spec.replicas

Signing of manifests is a great way to bolster the security of your cluster, but it also requires some flexibility. Teams often need to change values and certain fields (in addition to Kubernetes itself needing to sometimes change them). That’s why with this new rule type there is an object where you can specify which fields to ignore when verifying those manifests. In the previous policy, it provides an exception for the replicas field of a Deployment allowing only the value of this field to deviate from what was originally signed.

Cloning Multiple Resources

One of the defining capabilities of Kyverno is its simple way of generating new Kubernetes resources as opposed to just validating or mutating them. We’ve seen tremendous adoption of this policy-based ability by software teams and users all over the place. One of the most common use cases for this generate ability is in multi-tenancy or Namespace-as-a-Service provisioning processes. But something we heard loud and clear was that users needed to clone more than just a single resource at a time. Very often, when provisioning a new Namespace, a variety of resources are required before handing that over. For example, Secrets, ConfigMaps, Custom Resources, and others are commonly required. In Kyverno 1.8, we’ve brought you this ability by now allowing a single generate rule to define, in a selective manner, and clone multiple resources from the same source Namespace. As you can see from the below policy, whenever a new Namespace is created, all the Secrets and ConfigMaps in the staging Namespace which have been labeled with allowedToBeClone=true will be cloned into the new Namespace.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4 name: provision-namespaces
 5spec:
 6 rules:
 7 - name: sync-secrets-configmaps
 8 match:
 9 any:
10 - resources:
11 kinds:
12 - Namespace
13 generate:
14 namespace: "{{request.object.metadata.name}}"
15 synchronize: true
16 cloneList:
17 namespace: staging
18 kinds:
19 - v1/Secret
20 - v1/ConfigMap
21 selector:
22 matchLabels:
23 allowedToBeCloned: "true"

GitOps Friendly Rule Auto-Generation Is Here To Stay

In the 1.7 release, we introduced the new concept of moving Pod rule autogeneration out of spec and into status to be more kind to our GitOps users. Now in 1.8, that feature is on by default and no feature flags are required, allowing users of popular GitOps tools like Flux and ArgoCD to deploy Kyverno policies through their tooling without having to slightly adapt their definitions to account for these changes.

Other Additions and Enhancements

Kyverno 1.8 is such a substantial release it’s hard to cover all of the features, but here are a few others to note.

OpenTelemetry support was added for those who want an alternative to Prometheus.

The CLI now supports testing of generate policies joining long-time support for validate and mutate rule testing.

On the JMESPath side, we have two new filters called random and x509_decode. The random filter gives Kyverno the ability to generate random strings of data but in a fully composable and easy-to-use way. The x509_decode filter allows Kyverno to interpret PEM-encoded X509 certificates and make policy decisions based upon their contents, excellent for doing things like checking certificate subjects, expiration dates, and more.

The reporting system received a total overhaul in this release which makes it both lighter on memory, faster to generate policy reports, and more reliable.

Over time with the tremendous development velocity achieved in Kyverno, we’ve added many new fields and changed others. Kyverno 1.8 introduces a new schema version v2beta1 which is what we’ll begin using in the near future as it brings all the various rule types fully up-to-date with the latest and greatest.

And on the sample policy library, almost forty new policies have been added including implementation of best practices for common service meshes like Istio and Linkerd, CI/CD tools like Tekton, and more. This brings the total up to around 230 making Kyverno, by far, the policy engine with the largest number of samples designed to help get you running faster and easier.

Potentially Breaking Changes

A couple things of which to be aware prior to upgrading. First, the Helm chart registry URL has changed to ghcr.io/kyverno/charts/kyverno so make sure to update your Helm repositories. And second, because we’ve revamped Kyverno’s reporting and background scanning abilities, the backgroundScan container flag you might have passed previously has changed to being a true or false value, simply either activating or deactivating background scans.

Closing

Just like in previous releases, Kyverno 1.8 is a huge release closing over 250 issues. The maintainers and contributors have been hard at work for the past few months trying to bring additional value to the community, so we hope you find this release useful. As always, we love the community and hope you engage with us on any one of our outlets.

Blog: New Kyverno Blog

Fri, 08 Jul 2022 00:00:00 +0000

Welcome to the new Kyverno blog!

As you can see, we now have a brand new and shiny blog page thanks to the folks behind the Docsy theme which the Kyverno website uses (in addition to others in the cloud native ecosystem).

Subscribe to the handy RSS feed to watch for updates, or check back to see blog articles published by the Kyverno team!