I’ve been working on a secrets management project, FishyKeys (like Vault). To integrate it with Kubernetes and automatically inject secrets from the FishyKeys backend into applications, I needed to build an operator. This operator generates Kubernetes Secrets from FishyKeys, inspired by the External Secrets Operator.
This post will walk you through how relatively simple it was to build it, and by the end, you’ll have a clear understanding of how a Kubernetes operator works under the hood.
My operator would read my Custom ResourceFishySecret and generates the corresponding Secret. Here is the structure of a FishySecret I envisioned, and the resulting Secret:
There are quite a few libraries to interface with the Kubernetes API, but Operator SDK is the most reputable, and the one I decided to use.
Unfortunately I find their tutorial hard to follow for implementing the controller. They want you to fill in the blanks of an example code, and even though the code is quite documented, it’s too confusing for my use case and thus I preferred to build my controller’s logic incrementally.
Let’s scaffold our project using Operator SDK, specifying the domain (as prefix of our API group, think of the apiVersion field) and the module repository:
First, we will edit the types file located at /api/v1alpha1/fishysecret_types.go. It contains many helpful comments to understand how to edit it, take your time to read them.
It defines your CRD: the available fields, their types, and how they are validated and serialized (think: is this field required? how is it represented in YAML?).
Let’s replace the default CRD definition with the one we designed earlier:
// FishySecretSpec defines the desired state of FishySecret. type FishySecretSpec struct { // Target defines the Kubernetes Secret to create Target SecretTarget `json:"target"`
// Data is the list of key-paths to fetch from the secret manager // Each item becomes a key in the resulting Kubernetes Secret Data []SecretKeyMapping `json:"data"` }
// SecretTarget defines where to create the Kubernetes Secret type SecretTarget struct { Name string`json:"name"` Namespace string`json:"namespace"` }
// SecretKeyMapping defines a mapping between a key in the secret manager // and a key in the resulting Kubernetes Secret type SecretKeyMapping struct { // SecretPath is the path in the secret manager SecretPath string`json:"secretPath"`
// SecretKeyName is the field name in the resulting K8s Secret SecretKeyName string`json:"secretKeyName"` }
// FishySecretStatus defines the observed state of FishySecret. type FishySecretStatus struct { // Conditions represent the latest available observations of the FishySecret's state Conditions []metav1.Condition `json:"conditions,omitempty"`
// LastSyncedTime is the last time the FishySecret was successfully synced LastSyncedTime *metav1.Time `json:"lastSyncedTime,omitempty"` }
// FishySecret is the Schema for the fishysecrets API. type FishySecret struct { metav1.TypeMeta `json:",inline"` metav1.ObjectMeta `json:"metadata,omitempty"`
Spec FishySecretSpec `json:"spec"` Status FishySecretStatus `json:"status,omitempty"` }
// +kubebuilder:object:root=true
// FishySecretList contains a list of FishySecret. type FishySecretList struct { metav1.TypeMeta `json:",inline"` metav1.ListMeta `json:"metadata,omitempty"` Items []FishySecret `json:"items"` }
This definition includes all the fields we planned, along with a standard status field used by Kubernetes resources. Our operator will use this field to expose the current state of the FishySecret.
Also note the +kubebuilder:subresource:statusmarker, it tells Kubernetes to expose status as a protected subresource against modifications and to make it appear in commands like kubectl get --watch and kubectl describe.
Now let’s regenerate the CRD manifests from these Go types:
1 2
$ make generate $ make manifests
This generates deepcopy functions for our structs (in /api/v1alpha1/zz_generated_deepcopy.go) and more importantly, the CRD manifest itself at config/crd/bases/fishykeys.2v.pm_fishysecrets.yaml.
This CRD manifest describes how Kubernetes understands and validates our CRD. Here is an excerpt:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
items: description:|- SecretKeyMapping defines a mapping between a key in the secret manager and a key in the resulting Kubernetes Secret properties: secretKeyName: description:SecretKeyNameisthefieldnameintheresulting K8sSecret type:string secretPath: description:SecretPathisthepathinthesecretmanager type:string required: -secretKeyName -secretPath type:object
Notice how the comments in our Go types are carried over into the CRD manifest. When investigating unexpected operator behavior, the operator’s documentation is sometimes incomplete, and reading the CRD manifest is often the only way to understand how a field behaves, or even discover new fields.
So please do comment your types: the developer using your operator (including future you) will thank you :)
Writing our operator
The reconciliation cycle
The core of our operator is the reconciliation loop. You can find its definition in /internal/controller/fishysecret_controller.go and this is where we will be working now.
The Reconcile function will be called when controller-runtime (the core library on which operator-sdk is built) decides your custom resource (our FishySecret resource) might need to be reconciled. This can happen for the following reasons:
A FishySecret was created
A FishySecret was updated
A FishySecret was deleted
A resource watched by the controller emits an event (our generated Secret in this case, like the FishySecret)
The controller asked to reschedule the reconciliation
The controller starts
In essence, Kubernetes presents you with a resource and you need to bring its current state to the desired state (its spec).
The SetupWithManager function of the same file tells controller-runtime what will trigger a Reconcile call. In our case:
FishySecret resources
Secret associated with a FishySecret
Kubernetes is really powerful as we can order our controller to only watch for resources that validate our conditions, here our labels:
// SetupWithManager sets up the controller with the Manager func(r *FishySecretReconciler) SetupWithManager(mgr ctrl.Manager) error { return ctrl.NewControllerManagedBy(mgr). // Watch for all FishySecret objects For(&fishykeysv1alpha1.FishySecret{}). // AND Secret objects, for which a corresponding FishySecret will be deduced Watches( &corev1.Secret{}, handler.EnqueueRequestsFromMapFunc(r.mapSecretToFishySecrets), ). // The name of our controller, for logs/metrics Named("fishysecret"). Complete(r) }
Because we map the created Secret with a FishySecret using labels, any change to a watched Secret triggers reconciliation of its corresponding FishySecret.
How does controller-runtime observe resource changes?
▶
Under the hood, controller-runtime uses informers, which is how the Kubernetes client watch resources and cache their state. The library wraps these informers into a structured framework: events are automatically queued for reconciliation, retries and rate-limiting are handled for you. Your only job is to react to these events, controller-runtime handles the rest for you :)
Finally, since our controller will need to manipulate Secret resources, we need to add a Kubebuilder marker above the Reconcile function so that the generated RBAC manifests allow the controller to do it:
We will implement the Reconcile function in two parts: deletion handling first, then creating/updating the child Secret, as there is no point syncing secrets on a FishySecret that is being deleted.
Deletion handling
Before we handle deletions, a quick note on finalizers: a finalizer is a metadata entry you add to a resource that prevents Kubernetes from removing it until you explicitly clear it. This is how we make sure the child Secret gets deleted before the FishySecret disappears.
When a FishySecret is deleted, Kubernetes sets its DeletionTimestamp and keeps it around until we clear the finalizer. A Reconcile request may also be queued for a resource already fully removed, so Get can return NotFound. Let’s handle both cases:
1 2 3 4 5 6 7 8 9
var fishySecret fishykeysv1alpha1.FishySecret if err := r.Get(ctx, req.NamespacedName, &fishySecret); err != nil { if apierrors.IsNotFound(err) { // Resource deleted, nothing to do return ctrl.Result{}, nil } log.Error(err, "unable to fetch FishySecret") return ctrl.Result{}, err }
Now that we know our FishySecret still exists, we need to check if it’s scheduled for deletion. In this case we need to delete the child Secret and finally remove the finalizer, so that Kubernetes can clean up our FishySecret.
Why return immediately after updating the finalizer?
▶
Adding or removing a finalizer changes the object in memory. Calling Update persists this change to the API server and emits an update event, which queues a new reconciliation with the latest version of the resource.
Returning immediately ensures we don’t continue reconciliation with an outdated in-memory object. This pattern is used in the rest of the controller.
Creating/Updating our secret
Now that we have a living FishySecret, let’s retrieve the secrets from FishyKeys that we want to put in our Secret resource:
serverUrl, token, err := getFishyKeysUrlAndToken(ctx, r) if err != nil { log.Error(err, "failed to get FishyKeys token") setStatusCondition(&fishySecret, "Ready", metav1.ConditionFalse, "TokenError", err.Error()) _ = r.Status().Update(ctx, &fishySecret) return ctrl.Result{}, err }
for _, mapping := range fishySecret.Spec.Data { value, err := fetchSecretFromManager(serverUrl, token, mapping.SecretPath) if err != nil { log.Error(err, "failed to fetch from secret manager", "path", mapping.SecretPath) setStatusCondition(&fishySecret, "Ready", metav1.ConditionFalse, "FetchError", err.Error()) _ = r.Status().Update(ctx, &fishySecret) return ctrl.Result{}, err } secretData[mapping.SecretKeyName] = []byte(value) }
We also track the status of each FishySecret by updating its status field. Conditions help surface the resource’s lifecycle state to the user and make debugging easier. The setStatusCondition function is a small helper around meta.SetStatusCondition:
Why do we ignore errors from Status().Update() in some cases?
▶
Updating the status is a best-effort operation meant to inform the user.
If the reconciliation is already failing, we return the original error so the request is retried. Ignoring a status update error avoids masking the real failure.
Now that we’ve retrieved the secrets’ values according to the FishySecret spec, it’s really easy to construct our desired Secret. We also associate the FishySecret with the created Secret using labels, so that the created Secret will be watched by controller-runtime.
var existingSecret corev1.Secret err = r.Get(ctx, client.ObjectKeyFromObject(desiredSecret), &existingSecret) // If no child Secret exists, create it if apierrors.IsNotFound(err) { if err := r.Create(ctx, desiredSecret); err != nil { log.Error(err, "failed to create Secret") return ctrl.Result{}, err } log.Info("Secret created", "name", desiredSecret.Name) } elseif err == nil { // If it exists, verify if the Secret's data is equal to our desired data, if not change it if !reflect.DeepEqual(existingSecret.Data, desiredSecret.Data) { existingSecret.Data = desiredSecret.Data if err := r.Update(ctx, &existingSecret); err != nil { log.Error(err, "failed to update Secret") return ctrl.Result{}, err } log.Info("Secret updated", "name", desiredSecret.Name) } } else { log.Error(err, "failed to get existing Secret") return ctrl.Result{}, err }
// Mark the FishySecret as synced setStatusCondition(&fishySecret, "Ready", metav1.ConditionTrue, "SecretSynced", "Secret successfully synced") fishySecret.Status.LastSyncedTime = &metav1.Time{Time: time.Now()} if err := r.Status().Update(ctx, &fishySecret); err != nil { log.Error(err, "failed to update FishySecret status") return ctrl.Result{}, err }
Our reconciliation logic is idempotent: each time Reconcile runs, it ensures the Secret matches the desired state defined in the FishySecret. If the resource already has the correct data, no action is taken. Even if events are duplicated or reconciliation is triggered multiple times, our resources will be in the desired state.
Finally, we just have to ask Kubernetes to call the Reconcile loop again in 5 minutes. This is because the secret values in the FishyKeys backend may change independently of Kubernetes events, and we need to update the Secret accordingly.
1 2 3 4
// Schedule a reconciliation in 5 minutes, in case the corresponding secrets are updated in the backend return ctrl.Result{ RequeueAfter: 5 * time.Minute, }, nil
// Reconcile is part of the main kubernetes reconciliation loop which aims to // move the current state of the cluster closer to the desired state. // // For more details, check Reconcile and its Result here: // - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.21.0/pkg/reconcile func(r *FishySecretReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) { log := logf.FromContext(ctx)
var fishySecret fishykeysv1alpha1.FishySecret if err := r.Get(ctx, req.NamespacedName, &fishySecret); err != nil { if apierrors.IsNotFound(err) { // Resource deleted, nothing to do return ctrl.Result{}, nil } log.Error(err, "unable to fetch FishySecret") return ctrl.Result{}, err }
if !fishySecret.ObjectMeta.DeletionTimestamp.IsZero() { log.Info("Finalizing FishySecret", "name", fishySecret.Name)
target := fishySecret.Spec.Target secret := &corev1.Secret{ ObjectMeta: metav1.ObjectMeta{ Name: target.Name, Namespace: target.Namespace, }, } // In case our Secret was deleted manually we may not find it, ignore this error if err := r.Delete(ctx, secret); err != nil && !apierrors.IsNotFound(err) { log.Error(err, "unable to delete child Secret", "name", target.Name) return ctrl.Result{}, err }
// Schedule a reconciliation in 5 minutes, in case the corresponding secrets are updated in the backend return ctrl.Result{ RequeueAfter: 5 * time.Minute, }, nil }
First you need a Kubernetes cluster, I use kind to run one locally:
1
$ kind create cluster
You can then install the CRDs using make install.
You have two choices:
Generate a docker image for your operator (needed to test multi-replicas, leader elections or if you rely on Secrets mounted into your operator)
Run the operator locally (faster)
To run the operator locally, make sure no deployment of the operator is present in the cluster and use make run.
To run the operator with a docker image, we will need to generate it, load it into our cluster, and have operator-sdk deploy our operator:
1 2 3
$ make docker-build $ kind load docker-image controller:latest $ make deploy
Your controller is now deployed in the namespace fishykeys-operator-system, use kubectl get pods/kubectl logs <pod-name> to see your pod and its logs :)
You may need to change the imagePullPolicy of the deployment from Always to IfNotPresent so that the deployment doesn’t try to pull the image from Docker Hub (or edit config/manager/manager.yaml but only for development!)
We can now create the FishySecret defined earlier and observe that a Secret is created with the expected values:
$ k apply -f fishysecret.yaml $ k get fishysecret NAME AGE fishysecret-db 4s $ k get fishysecret fishysecret-db -o yaml apiVersion: fishykeys.2v.pm/v1alpha1 kind: FishySecret metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"fishykeys.2v.pm/v1alpha1","kind":"FishySecret","metadata":{"annotations":{},"name":"fishysecret-db","namespace":"default"},"spec":{"data":[{"secretKeyName":"DB_USER","secretPath":"/app/db/username"},{"secretKeyName":"DB_PASS","secretPath":"/app/db/password"}],"target":{"name":"test-secret","namespace":"target-namespace"}}} creationTimestamp: "2026-01-04T19:20:59Z" finalizers: - fishykeys.2v.pm/finalizer generation: 1 name: fishysecret-db namespace: default resourceVersion: "4861" uid: d533c9d2-1b5b-4864-8f6a-54f8c6c4ae43 spec: data: - secretKeyName: DB_USER secretPath: /app/db/username - secretKeyName: DB_PASS secretPath: /app/db/password target: name: test-secret namespace: target-namespace $ k get secret -n target-namespace NAME TYPE DATA AGE test-secret Opaque 2 2s $ k get secret -n target-namespace test-secret -o yaml apiVersion: v1 data: DB_PASS: Y2xpY2sgbWUh DB_USER: YXBwX3VzZXI= kind: Secret metadata: creationTimestamp: "2026-01-04T19:24:46Z" labels: fishykeys.2v.pm/owner-name: fishysecret-db fishykeys.2v.pm/owner-namespace: default name: test-secret namespace: target-namespace resourceVersion: "5285" uid: eb7d58b3-4300-455b-bd01-4f918fe6e13c type: Opaque
The only way to delete the created secret is to delete its owner:
1 2 3 4 5 6 7 8 9
$ k delete secret -n target-namespace test-secret secret "test-secret" deleted $ k get secret -n target-namespace NAME TYPE DATA AGE test-secret Opaque 2 3s $ k delete fishysecret fishysecret-db fishysecret.fishykeys.2v.pm "fishysecret-db" deleted $ k get secret -n target-namespace No resources found in target-namespace namespace.
Voilà! Our operator is now working as expected.
For automated tests, you have a framework already present in fishysecret_controller_test.go, see a simple example of it filled here.
Running these is quite easy:
1 2
$ make test# Doesn't require a cluster, faster, useful for CI $ make test-e2e # Full integration test running a cluster
Towards production
While all of this is sufficient to develop and test your operator locally, you still need to do a few more things before being able to deploy your operator for others to use it:
Check all TODO(user) comments generated by operator-sdk and remove them.
Customize your image name, the namespace of the operator, etc:
Namespace can be found at config/default/kustomization.yaml
You can change the image name when building it (e.g.: make docker-build IMG=example.com/my-operator:v0.1.0) or more permanently by editing the Makefile variable IMG
Review and restrict the RBAC rules under config/rbac/ (they are usually broad by default, you want to restrict them to only what’s needed)
Maybe provide a Helm chart for ease of installation :)
That’s it, thanks for reading! I hope this gave you a clearer picture of how operators work and that building one is less intimidating than it looks. The full source code for FishyKeys’ operator is available here if you want to dig deeper.