Kubernetes logo
Solving the Kubernetes chicken and egg problem
October 25, 2025
When a Kubernetes cluster is first created the only thing running within it is the kube-api while it awaits further instruction. We then seed some initial configuration, cross our fingers, and hope that the Disaster Recovery Plan (DRP) that was written long ago still holds true today. Inevitably, you're in for a fun ride, because it almost never goes according to plan leaving you, as the operator, to draw from your years of experience and tribal knowledge to get the job done. In fact, this was the DRP at many of the shops I've worked at, and I'd wager it might be yours too. There is a better way, however. What if the cluster could be pre-seeded in such a way that gave it a kick start allowing it to hit the ground running once the nodes come online and report as healthy?
At Cyware I was given opportunity to reinvent the way the company did business in preparation of a large contract that we were pursuing. Up until this point, the plan was to "click-ops" most of the initial infrastructure by manually poking around the web UI, clicking some buttons, and all without any sort of management behind it — a plan that was completely foreign to me having never even heard the term "click-ops" — before running some scripts to install the software. It's not an incorrect way to go about a deployment, but it surely does not scale well, and good luck if you're handing this off to an on-premise customer … which was exactly what we were preparing to do.
Being a vendor of a Threat Inteligence Platform (TIP), the primary demographic the company marketed towards were security teams at medium to large enterprises. Often times, these teams have little to no involvement with their DevOps or IT divisions, and, ideally, an on-premise solution should not require that nor rely on a familiarity with any specific tooling those teams may commonly use such as Terraform. Most individuals choose to host their infrastructure with Amazon's Web Services (AWS), and their in-house Infrastructure as Code (IaC) service called Cloudformation sounded like a reasonable option when considering that it can be used from within the browser.
Cloudformation templates can be written using either JSON or YAML which is convenient when we remember that Kubernetes also uses YAML. The functionality of Amazon's IaC tool can be extended with custom resources and a serverless function. Combine the two, and you can see where I'm going with this.
---
AWSTemplateFormatVersion: 2010-09-09
Resources:
Nginx:
Type: 'Custom::Kubernetes'
Properties:
ServiceToken: !GetAtt Function.Arn
ServiceTimeout: 600
Objects:
- name: deployment
content:
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
selector:
matchLabels:
name: nginx
template:
metadata:
labels:
name: nginx
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
- name: service
content:
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
name: nginx
ports:
- protocol: TCP
port: 80
targetPort: 80
Suddenly, there now exists an NGINX server in the cluster, and it is left as an exercise for the reader what all is possible when the concept is combined with continuous delivery tools and controllers; you might just be able to ship a fully suffonsified production environment with observability that meets AWS best practices in under 2 hours and can intelligently rehydrate itself from a previous backup. Although, let's consider some other ways this could be realized as not everyone is hosted on AWS and some even desire to self-host on bare metal.
All major cloud providers have their own IaC tooling, Terraform has a Kubernetes provider, and static pods can all get the job done. However, when we consider that Kubernetes is basically nothing more than a well defined API and a collection of event driven state machines that can all be implemented in any number of various ways, we arrive at solutions like k3s or Talos Linux. Conceivably, the "app" could be embedded within an entire turn-key platform and now your "app" is receiving regular updates beyond just source control — we've entered into a scenario where the end-to-end user experience is receiving feature updates.
Once you break a system down to its fundamentals, the sky is the limit; you can intuitively "feel" and predict its behavior. Plus, you know the extent of its limitations and know precisely where a modification should be made to push past the limitation. Nothing is impossible with the right mindset and attitude. Get out there and create something! I hope this helps.