K3s GitOps k0rdent Proxmox: Building Custom Templates for On-Prem
Provision K3s clusters on Proxmox with k0rdent’s BYOT approach. Learn how GitOps-driven on-prem Kubernetes management ensures sovereignty and scalability.
Beyond the Public Cloud: The Strategic Shift to Declarative On-Premises Kubernetes
For years, the industry narrative suggested that the public cloud was the only destination for modern, scalable infrastructure. Today, however, many technical leaders implement K3s GitOps k0rdent Proxmox architectures to regain control over their environments. As regulatory landscapes like NIS2 and DORA tighten their grip on data sovereignty and operational resilience, the shift back to the data center is accelerating. While the cloud offers seamless APIs, on-premises environments often remain a patchwork of manual scripts—a challenge that modern declarative management tools are finally solving.
The emergence of lightweight Kubernetes distributions like K3s, combined with declarative management tools like k0rdent, is changing this equation. By adopting a GitOps-first approach to on-prem infrastructure, organizations can now achieve cloud-like agility without sacrificing control over their physical hardware. This article explores a specific, robust architecture: provisioning K3s on Proxmox using k0rdent’s Bring Your Own Template (BYOT) methodology.
The Architecture: Proxmox, K3s, and k0rdent
To move away from 'duct-tape-and-bash-script' management, we need an architecture that treats on-premises hardware as a first-class citizen in a GitOps workflow. The stack we are analyzing consists of three primary layers:
- Virtualization Layer (Proxmox): A powerful, open-source virtualization platform that manages the physical compute, storage, and networking.
- Bootstrap Provider (K3s): A lightweight, CNCF-certified Kubernetes distribution designed for resource-constrained environments and edge computing.
- Management Plane (k0rdent): A multi-cluster management tool that uses the Cluster API (CAPI) pattern to reconcile the desired state of infrastructure across various environments.
The beauty of this setup lies in the Bring Your Own Template (BYOT) approach. Instead of building VM images from scratch during every provisioning cycle, we leverage pre-configured Proxmox templates that include cloud-init, essential security patches, and network configurations. This significantly reduces 'Time to Cluster' and ensures environmental consistency.
Why K3s for On-Premise Workloads?
K3s has become the gold standard for on-prem and edge deployments for several reasons. It is packaged as a single binary, reducing the external dependencies required to get a control plane running. It swaps out heavy components like etcd for lighter alternatives like SQLite (for single-node) or keeps etcd highly efficient for multi-node clusters. For an organization running on Proxmox, K3s minimizes the resource overhead of the orchestration layer, leaving more 'headroom' for the actual business applications.
Deep Dive: The k0rdent BYOT Methodology
At the heart of this declarative workflow is the ability to write custom templates. In k0rdent, this is achieved through Helm charts that define the Infrastructure Provider, the Control Plane Provider, and the Bootstrap Provider.
1. Building the Infrastructure Provider for Proxmox
Since k0rdent might not ship with a native Proxmox provider out of the box, we utilize its extensible architecture to build our own. The Helm chart for the Infrastructure Provider is responsible strictly for VM lifecycle management. It talks to the Proxmox API to clone a template, assign CPU/RAM, and attach the necessary network bridges.
By keeping this layer isolated, you ensure that if you ever switch from Proxmox to another hypervisor (like VMware or Nutanix), you only need to swap this specific provider chart; the rest of your Kubernetes logic remains untouched.
2. The Bootstrap Provider: K3s Implementation
Once the VMs are alive, the Bootstrap Provider takes over. This is where the magic of GitOps happens. In our example, the k0rdent service uses a Bootstrap Provider specifically for K3s. The configuration points to the official cluster-api-k3s components, ensuring we are following industry standards.
apiVersion: operator.cluster.x-k8s.io/v1alpha2
kind: BootstrapProvider
metadata:
name: k3s
spec:
version: v0.3.0
fetchConfig:
url: https://github.com/k3s-io/cluster-api-k3s/releases/v0.3.0/bootstrap-components.yamlThis declarative snippet tells the management plane exactly which version of K3s to install and how to configure the initial master node. The management plane then watches for changes; if a node drifts from this configuration, it is automatically reconciled back to the desired state.
Strategic Advantages: Sovereignty and Performance
Why go through the effort of building custom k0rdent templates for on-prem hardware? The motivations are often strategic rather than purely technical.
Predictable Performance and Low Latency
In industries like high-frequency trading, manufacturing, or healthcare imaging, the latency added by a public cloud round-trip is unacceptable. Running K3s on local Proxmox clusters ensures that data processing happens millimeters away from the data source. By using k0rdent to manage these clusters, you get the automation of the cloud with the performance of local hardware.
Compliance and 'Data Gravity'
Regulations such as the General Data Protection Regulation (GDPR) and the upcoming AI Act place strict requirements on where data is stored and processed. For many DACH-region companies, maintaining 'Data Sovereignty' means keeping sensitive IP on-premises. The GitOps approach ensures that even highly secured, air-gapped environments can benefit from modern CI/CD pipelines without exposing data to external cloud providers.
Operationalizing the Workflow: The GitOps Loop
The final piece of the puzzle is the GitOps controller (such as Flux or ArgoCD) that watches a Git repository containing your k0rdent ClusterDeployment manifests. When a developer or SRE wants to scale a cluster or deploy a new one in a different office, they simply update a YAML file in Git.
- Git Commit: A change is pushed to the infrastructure repo.
- Sync: Flux/ArgoCD detects the change and applies it to the management cluster.
- k0rdent Reconciliation: k0rdent sees the new ClusterDeployment and triggers the Proxmox Infrastructure Provider.
- VM Provisioning: Proxmox clones the BYOT template.
- K3s Bootstrapping: The K3s Bootstrap Provider installs the control plane.
- Ready: The new cluster joins the fleet, and health checks commence.
Conclusion: The Future of the Hybrid Fleet
Building a custom k0rdent template for K3s on Proxmox represents a significant step toward infrastructure maturity. It proves that 'On-Prem' is no longer synonymous with 'Manual'. By leveraging the BYOT approach, organizations can build a resilient, scalable, and fully compliant private cloud that responds to the needs of the business as quickly as any public provider.
The key takeaway for technical decision-makers is the power of abstraction. By using Helm-based providers and the Cluster API pattern, you decouple your business applications from the underlying hardware, providing a future-proof foundation for the next decade of digital transformation.
Frequently Asked Questions
1. Why use K3s instead of standard Kubernetes (K8s) for on-prem?
K3s is optimized for low-resource environments and edge use cases. It removes many legacy and alpha features, resulting in a smaller footprint and faster startup times, which is ideal for virtualized environments like Proxmox.
2. What is the main benefit of k0rdent over traditional scripts?
k0rdent provides continuous reconciliation. If a manual change occurs on the infrastructure, k0rdent will automatically revert it to the state defined in your Git repository, preventing 'configuration drift'.
3. How does the 'Bring Your Own Template' (BYOT) approach improve security?
BYOT allows you to use hardened, pre-audited VM images. You can ensure that every cluster node starts with the latest security patches and corporate compliance settings already baked in.
4. Can this setup work in air-gapped environments?
Yes. Because the providers are based on Helm charts and standard container images, you can host all necessary assets in a local registry (like Harbor), allowing for fully automated deployments without internet access.
5. Is Proxmox suitable for enterprise-grade Kubernetes?
Absolutely. Proxmox is built on top of KVM and QEMU, the same technologies powering many major cloud providers. When combined with declarative management like k0rdent, it becomes a robust enterprise virtualization platform.
Q&A
K3s is optimized for low-resource environments and edge use cases. It removes many legacy and alpha features, resulting in a smaller footprint and faster startup times, which is ideal for virtualized environments like Proxmox.
k0rdent provides continuous reconciliation. If a manual change occurs on the infrastructure, k0rdent will automatically revert it to the state defined in your Git repository, preventing 'configuration drift'.
BYOT allows you to use hardened, pre-audited VM images. You can ensure that every cluster node starts with the latest security patches and corporate compliance settings already baked in.
Yes. Because the providers are based on Helm charts and standard container images, you can host all necessary assets in a local registry (like Harbor), allowing for fully automated deployments without internet access.
Absolutely. Proxmox is built on top of KVM and QEMU, the same technologies powering many major cloud providers. When combined with declarative management like k0rdent, it becomes a robust enterprise virtualization platform.
Source: www.cncf.io