Building a Real-world Kubernetes Operator: Part - 3
Exploring Owners, Dependents, Status subresource and Printer Columns
Introduction
This part continues from the previous section where we developed the controllers. Here, we'll fix the NimbusPolicy deletion issue and make a few other improvements.
There's another issue with NimbusPolicy: if you manually delete or update it, the SecurityIntentBinding controller neither recreates it nor discards those manual changes.
Right now, the SecurityIntentBinding controller only creates or updates the NimbusPolicy when it gets an event related to SecurityIntentBinding. Ideally, it should manage the entire lifecycle of a NimbusPolicy, similar to how a ReplicaSet controller manages Pods.
There are two ways to fix these:
Manual Lifecycle Management (Not Recommended): This involves manually handling the creation, deletion, and updates of NimbusPolicy objects. However, this approach can quickly become cumbersome and error-prone. Kubernetes already provides built-in mechanisms for managing resources, so this method is generally discouraged.
Leveraging Kubernetes' OwnerReferences: Kubernetes way, by utilizing a built-in Kubernetes feature called OwnerReferences.
Let's get into business.
Owners and Dependents
The Kubernetes documentation provides a clear explanation of ownership and dependencies between resources.
In the context of our operator, let's break down the concept of owner and child resources.
Recall that in our design, a SecurityIntentBinding resource manages a NimbusPolicy resource. Therefore, SecurityIntentBinding acts as the owner, while NimbusPolicy is the child resource.
With this understanding, we'll configure our SecurityIntentBinding controller to be aware of its owned objects, specifically NimbusPolicy resources. This ensures that any changes to the SecurityIntentBinding are reflected in the corresponding NimbusPolicy. Additionally, the controller will recreate any deleted NimbusPolicy and discard any manual changes made directly to them.
Edit the securityintentbinding_controller.go file in the internal/controller directory as follows:
...
// SetupWithManager sets up the controller with the Manager.
func (r *SecurityIntentBindingReconciler) SetupWithManager(mgr ctrl.Manager) error {
return ctrl.NewControllerManagedBy(mgr).
For(&intentv1alpha1.SecurityIntentBinding{}).
Owns(&intentv1alpha1.NimbusPolicy{}).
Complete(r)
}
...
Configured the controller to own NimbusPolicy i.e., it will manage its complete lifecycle.
Edit the nimbus_policy.go file in pkg/builder as follows:
import (
...
ctrl "sigs.k8s.io/controller-runtime"
)
func BuildNimbusPolicy(ctx context.Context, k8sClient client.Client, securityIntentBinding intentv1alpha1.SecurityIntentBinding) (*intentv1alpha1.NimbusPolicy, error) {
...
if err := ctrl.SetControllerReference(&securityIntentBinding, nimbusPolicy, k8sClient.Scheme()); err != nil {
return nil, err
}
return nimbusPolicy, nil
}
Set the SecurityIntentBinding as the owner of the NimbusPolicy.This tells the controller, "Hey, you're responsible for handling thisNimbusPolicy".
Let's verify whether things are working as intended.
Create sample intent and binding:
$ k apply -f pkg-mgrs-intent-and-binding.yaml
securityintent.intent.security.nimbus.com/package-mgrs created
securityintentbinding.intent.security.nimbus.com/package-mgrs-binding created
check operator logs:
...
2024-07-13T23:08:02+05:30 INFO reconciling SecurityIntent {"controller": "securityintent", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntent", "SecurityIntent": {"name":"package-mgrs"}, "namespace": "", "name": "package-mgrs", "reconcileID": "7dcbcd8f-70a7-419f-be5c-4b775bdba124", "securityIntent": "package-mgrs"}
2024-07-13T23:08:02+05:30 INFO reconciling SecurityIntentBinding {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"package-mgrs-binding","namespace":"default"}, "namespace": "default", "name": "package-mgrs-binding", "reconcileID": "8452da71-11a4-4e62-a60f-9d27ca74e6fc", "securityIntentBinding.name": "package-mgrs-binding", "securityIntentBinding.namespace": "default"}
2024-07-13T23:08:02+05:30 INFO reconciling SecurityIntentBinding {"controller": "securityintentbinding", "controllerGroup": "intent.security.nimbus.com", "controllerKind": "SecurityIntentBinding", "SecurityIntentBinding": {"name":"package-mgrs-binding","namespace":"default"}, "namespace": "default", "name": "package-mgrs-binding", "reconcileID": "fea993ee-101f-4285-a730-251b45799160", "securityIntentBinding.name": "package-mgrs-binding", "securityIntentBinding.namespace": "default"}
...
Check the resources and their detail:
SecurityIntentBinding
$ k get securityintentbinding
NAME AGE
package-mgrs-binding 4m16s
$ k get securityintentbinding package-mgrs-binding -o yaml
apiVersion: intent.security.nimbus.com/v1alpha1
kind: SecurityIntentBinding
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"intent.security.nimbus.com/v1alpha1","kind":"SecurityIntentBinding","metadata":{"annotations":{},"name":"package-mgrs-binding","namespace":"default"},"spec":{"intents":[{"name":"package-mgrs"}],"selector":{"matchLabels":{"app":"web","env":"prod"}}}}
creationTimestamp: "2024-07-13T17:38:02Z"
generation: 1
name: package-mgrs-binding
namespace: default
resourceVersion: "4227"
uid: ab7d399c-67ef-401a-87f9-e94257970f34
spec:
intents:
- name: package-mgrs
selector:
matchLabels:
app: web
env: prod
NimbusPolicy
$ k get nimbuspolicy
NAME AGE
package-mgrs-binding 4m36s
$ k get nimbuspolicy -o yaml
apiVersion: intent.security.nimbus.com/v1alpha1
kind: NimbusPolicy
metadata:
creationTimestamp: "2024-07-13T17:38:02Z"
generation: 1
name: package-mgrs-binding
namespace: default
ownerReferences:
- apiVersion: intent.security.nimbus.com/v1alpha1
blockOwnerDeletion: true
controller: true
kind: SecurityIntentBinding
name: package-mgrs-binding
uid: ab7d399c-67ef-401a-87f9-e94257970f34
resourceVersion: "4228"
uid: 9996fe62-311c-42e7-b4ae-e2624d9d605f
spec:
rules:
- action: Enforce
id: pkgMgrs
selector:
matchLabels:
app: web
env: prod
Check the metadata.ownerReferences field; it has an entry for its owner with its kind, name, and uid.
Let's delete the resources we created earlier:
$ k delete -f pkg-mgrs-intent-and-binding.yaml
securityintent.intent.security.nimbus.com "package-mgrs" deleted
securityintentbinding.intent.security.nimbus.com "package-mgrs-binding" deleted
Check if the owned NimbusPolicy has been deleted.
$ k get nimbuspolicy
No resources found in default namespace.
Things are working as expected. Try manually editing the NimbusPolicy, then delete it and check if the changes are discarded and if it gets recreated.
That's how to add the owner and dependents. Easy, isn't it?
Short name
Shouldn't we give short names or aliases to resources so we have to type less? Just like we do for other core resources, for example, deploy for deployment, rs for replicaset, and sts for statefulset.
Edit the Kubebuilder markers in securityintent_types.go file in api/v1alpha1 directory as follows:
...
// +kubebuilder:resource:scope=Cluster,shortName=si
// SecurityIntent is the Schema for the securityintents API
type SecurityIntent struct {
...
}
...
Similarly, edit other files:
securityintentbinding_types.go
...
// +kubebuilder:resource:shortName=sib
// SecurityIntentBinding is the Schema for the securityintentbindings API
type SecurityIntentBinding struct {
...
}
...
nimbuspolicy_types.go
...
// +kubebuilder:resource:shortName=np
// NimbusPolicy is the Schema for the nimbuspolicies API
type NimbusPolicy struct {
...
}
...
We gave si for SecurityIntent, sib for SecurityIntentBinding, and np for NimbusPolicy.
Finally, install the updated CRDs:
make manifests install
Try kubectl get si/sib/np.
Status
We've addressed edit and deletion issues, and even added short names for our resources. But there's one more step!
Have you noticed that running kubectl get [kind] [name] or kubectl get [kind] [name] -o yaml doesn't show the current state of your resources? Let's explore why.
This information is missing because we haven't defined the .status subresource. The .status subresource is a dedicated field within Kubernetes resources where you can store details about their current/observed state.
So let's add it to all resources.
SecurityIntent
Since we're only concerned with the action (Enforce / Audit) and status (Created) of a SecurityIntent resource, we can add these fields by modifying the securityintent_types.go file located within the api/v1alpha1 directory as follows:
...
// SecurityIntentStatus defines the observed state of SecurityIntent
type SecurityIntentStatus struct {
Action string `json:"action"`
Status string `json:"status"`
}
...
You can add additional fields as needed to suit your specific requirements.
Let's update the SecurityIntent controller to update the status subresource accordingly.
Edit the securityintent_controller.go in internal/controller directory as follows:
...
func (r *SecurityIntentReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
...
logger.Info("reconciling SecurityIntent", "securityIntent", req.Name)
if err = r.updateStatus(ctx, &securityIntent); err != nil {
logger.Error(err, "failed to update SecurityIntent status", "securityIntent.name", req.Name)
return ctrl.Result{}, err
}
...
}
func (r *SecurityIntentReconciler) updateStatus(ctx context.Context, existingSecurityIntent *intentv1alpha1.SecurityIntent) error {
existingSecurityIntent.Status = intentv1alpha1.SecurityIntentStatus{
Action: existingSecurityIntent.Spec.Intent.Action,
Status: consts.StatusCreated,
}
return r.Status().Update(ctx, existingSecurityIntent)
}
Here we're leveraging Status part of the client, with the Update method. The status subresource ignores changes to the spec field, so it’s less likely to conflict with any other updates and can have separate permissions.
SecurityIntentBinding
For SecurityIntentBinding, we need the status, the names of bound security intents, the count of those bound intents, and the generated nimbuspolicy name. Let's add these fields to the SecurityIntentBindingStatus struct in the securityintentbinding_types.go file in the api/v1alpha1 directory as follows:
...
// SecurityIntentBindingStatus defines the observed state of SecurityIntentBinding
type SecurityIntentBindingStatus struct {
Status string `json:"status"`
BoundIntents []string `json:"boundIntents,omitempty"`
CountOfBoundIntents int32 `json:"countOfBoundIntents,omitempty"`
NimbusPolicy string `json:"nimbusPolicy,omitempty"`
}
...
NimbusPolicy
In NimbusPolicy, we need the status, the names of generated security policies, and their count. Let's add these fields too. Edit the nimbuspolicy_types.go file in the api/v1alpha1 directory as follows:
...
// NimbusPolicyStatus defines the observed state of NimbusPolicy
type NimbusPolicyStatus struct {
Status string `json:"status"`
GeneratedPoliciesName []string `json:"policiesName,omitempty"`
CountOfPolicies int32 `json:"policies,omitempty"`
}
...
Finally, update the SecurityIntentBinding controller to update the status subresource for both resources.
Edit the securityintentbinding_controller.go file in internal/controller directory as follows:
...
func (r *SecurityIntentBindingReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
...
nimbusPolicy, err := r.createOrUpdateNimbusPolicy(ctx, securityIntentBinding)
if err != nil {
return ctrl.Result{}, err
}
if nimbusPolicy != nil {
if err = r.updateNpStatus(ctx, nimbusPolicy); err != nil {
logger.Error(err, "failed to update NimbusPolicy status", "nimbusPolicy.Name", nimbusPolicy.Name, "nimbusPolicy.Namespace", nimbusPolicy.Namespace)
return ctrl.Result{}, err
}
if err = r.updateSibStatusWithBoundNpAndSisInfo(ctx, &securityIntentBinding, nimbusPolicy); err != nil {
logger.Error(err, "failed to update SecurityIntentBinding status", "securityIntentBinding.name", req.Name, "securityIntentBinding.namespace", req.Namespace)
return ctrl.Result{}, err
}
}
return ctrl.Result{}, nil
}
func (r *SecurityIntentBindingReconciler) updateNpStatus(ctx context.Context, nimbusPolicy *intentv1alpha1.NimbusPolicy) error {
nimbusPolicy.Status = intentv1alpha1.NimbusPolicyStatus{
Status: StatusCreated,
}
return r.Status().Update(ctx, nimbusPolicy)
}
func (r *SecurityIntentBindingReconciler) updateSibStatusWithBoundNpAndSisInfo(ctx context.Context, existingSib *intentv1alpha1.SecurityIntentBinding, existingNp *intentv1alpha1.NimbusPolicy) error {
existingSib.Status.Status = StatusCreated
existingSib.Status.NimbusPolicy = existingNp.Name
existingSib.Status.CountOfBoundIntents = int32(len(existingNp.Spec.NimbusRules))
existingSib.Status.BoundIntents = r.getBoundIntents(ctx, existingSib.Spec.Intents)
return r.Status().Update(ctx, existingSib)
}
func (r *SecurityIntentBindingReconciler) getBoundIntents(ctx context.Context, intents []intentv1alpha1.MatchIntent) []string {
var boundIntentsName []string
for _, intent := range intents {
var currIntent intentv1alpha1.SecurityIntent
if err := r.Get(ctx, types.NamespacedName{Name: intent.Name}, &currIntent); err != nil {
continue
}
boundIntentsName = append(boundIntentsName, currIntent.Name)
}
return boundIntentsName
}
After updating the types and controllers, let's quickly install and verify the changes.
make manifests generate install
to run:
make run
create the same intent and binding as before:
$ k apply -f pkg-mgrs-intent-and-binding.yaml
securityintent.intent.security.nimbus.com/package-mgrs created
securityintentbinding.intent.security.nimbus.com/package-mgrs-binding created
verify the status subresource:
SecurityIntent
$ k get si package-mgrs -o wide NAME AGE package-mgrs 69s $ k get si package-mgrs -o yaml apiVersion: intent.security.nimbus.com/v1alpha1 kind: SecurityIntent # Fields removed for brevity ... ... status: action: Enforce status: CreatedSecurityIntentBinding
$ k get sib package-mgrs-binding -o wide NAME AGE package-mgrs-binding 2m21s $ k get sib package-mgrs-binding -o yaml apiVersion: intent.security.nimbus.com/v1alpha1 kind: SecurityIntentBinding # Fields removed for brevity ... ... status: boundIntents: - package-mgrs countOfBoundIntents: 1 nimbusPolicy: package-mgrs-binding status: CreatedNimbusPolicy
$ k get np package-mgrs-binding -o wide NAME AGE package-mgrs-binding 4m11s $ k get np package-mgrs-binding -o yaml apiVersion: intent.security.nimbus.com/v1alpha1 kind: NimbusPolicy # Fields removed for brevit ... ... status: status: Created
As you can see, the status field is now populated by the controllers, so things are working as expected.
Printer Columns
I feel like we're still missing something. Do you know what it is?
When we run kubectl get [kind] [name] -o wide, we don't see extra info. I know we can get that info by running kubectl get [kind] [name] -o yaml. Think about other core Kubernetes resources. For example:
$ k get deploy -o wide
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
httpd 10/10 10 10 104s httpd httpd:2.4.53-alpine app=httpd
These are called additional printer columns. They help the API server decide which columns to show when we use the kubectl get command. Let's add these columns based on what we need.
SecurityIntent
Add the Kubebuilder print column markers in the securityintent_types.go file located in the api/v1alpha1 directory as follows:
...
// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.status"
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
// +kubebuilder:printcolumn:name="Action",type="string",JSONPath=".spec.intent.action",priority=1
// SecurityIntent is the Schema for the securityintents API
type SecurityIntent struct {
...
}
...
Make sure to specify the correct JSONPath field value, or you'll get unexpected results.
The priority field decides what to show in wide view when you run kubectl get [kind] [name] -o wide.
SecurityIntentBinding
Similarly, edit the securityintentbinding_types.go file in the api/v1alpha1 directory as follows:
...
// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.status"
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
// +kubebuilder:printcolumn:name="Intents",type="integer",JSONPath=".status.countOfBoundIntents"
// +kubebuilder:printcolumn:name="NimbusPolicy",type="string",JSONPath=".status.nimbusPolicy"
// SecurityIntentBinding is the Schema for the securityintentbindings API
type SecurityIntentBinding struct {
...
}
...
Here, we're not using the priority field, so the standard view and the wide view will look the same. Since the names of bound security intents can be long, we're not showing their names in any view. You can get their details when you describe SecurityIntentBinding.
NimbusPolicy
Edit the nimbuspolicy_types.go file located in api/v1alpha1 directory as follows:
...
// +kubebuilder:printcolumn:name="Status",type="string",JSONPath=".status.status"
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
// +kubebuilder:printcolumn:name="Policies",type="integer",JSONPath=".status.policies"
// NimbusPolicy is the Schema for the nimbuspolicies API
type NimbusPolicy struct {
...
}
...
This time we don't need to make any changes in any controller, just install the updated CRDs.
make manifests install
Run the operator:
make run
Apply the intent and binding:
$ k apply -f pkg-mgrs-intent-and-binding.yaml
securityintent.intent.security.nimbus.com/package-mgrs created
securityintentbinding.intent.security.nimbus.com/package-mgrs-binding created
verify the output:
SecurityIntent
$ k get si NAME STATUS AGE package-mgrs Created 44s $ k get si -o wide NAME STATUS AGE ACTION package-mgrs Created 46s EnforceSecurityIntentBinding
$ k get sib NAME STATUS AGE INTENTS NIMBUSPOLICY package-mgrs-binding Created 87s 1 package-mgrs-binding $ k get sib -o wide NAME STATUS AGE INTENTS NIMBUSPOLICY package-mgrs-binding Created 92s 1 package-mgrs-bindingYou know why there's no difference between these views.
NimbusPolicy
$ k get np NAME STATUS AGE POLICIES package-mgrs-binding Created 2m28s $ k get np -o wide NAME STATUS AGE POLICIES package-mgrs-binding Created 2m33sI know you're wondering why the
POLICIEScolumn is empty. Stay tuned!
That's it for this part. In the next part, we'll write tests for our controllers. Stay tuned!
You can find the complete code here. Please feel free to comment or criticize :)
Summary
In this article, we address the lifecycle management of the NimbusPolicy resource in our Kubernetes operator via OwnerReferences. We configure our SecurityIntentBinding controller to manage NimbusPolicy, ensuring it handles creation, deletion, and updates effectively. Additionally, we add short names and status subresources for better usability, and enhance resource visibility using printer columns. This guide not only improves resource management but also enhances kubectl output for a streamlined operator experience.