Architecture

kobe has three main components — the operator, the HTTP API, and the pool manager — plus pluggable runtime backends that do the actual cluster provisioning. Understanding what each one does makes it easier to configure pools correctly and diagnose problems.

The operator

kobe-operator is a Kubernetes controller that runs in your host cluster. It watches ClusterPool, ClusterInstance, ClusterLease, AccessPolicy, and KobeStore resources and reconciles them continuously.

ClusterPool defines the desired pool shape. The pool controller keeps enough ClusterInstance objects around to satisfy that shape. The instance controller provisions and health-checks the underlying backend resources. The lease controller binds ClusterLease objects to ready instances, generates short-lived kubeconfigs, and enforces release and expiry.

The HTTP API

The API server is embedded in the operator process. It handles lease requests, authentication, and pool status queries.

parse request
    ↓
authenticate (SSH / OIDC / token)
    ↓
enforce AccessPolicy (pool allowlist, TTL cap, concurrency limit)
    ↓
create ClusterLease CRD
    ↓
wait for operator to bind a ready ClusterInstance
    ↓
return kubeconfig with short-lived client certificate

Concurrency is bounded at 200 simultaneous API requests for DoS protection. Infrastructure routes (/healthz, /metrics) are exempt.

The pool manager

The pool manager runs inside the operator and tracks the state of every instance across all pools. It makes decisions about when to create, recycle, or destroy instances based on pool spec and current state.

Each pool instance moves through these states:

Creating -> Ready (warm) -> Leased -> Recycling -> [destroyed, replaced]

The pool manager uses a per-pool lock to prevent races when multiple requests try to assign the same ready instance concurrently.

Runtime backends

The instance controller delegates actual cluster provisioning to a per-pool backend: nested k3s / k0s clusters (today's production path), vcluster for high-density virtual clusters, or CAPI for provider-managed real infrastructure. All backends satisfy the same ClusterBackend contract — provision, health-check, readiness-gate, recycle — so pools, leases, and admission behave identically regardless of runtime. See Backends for configuration and Runtime strategy for which runtime to pick and where the roadmap is heading.

Note: development of the in-house kobe-sync sidecar and the vkobe backend built on it is on hold — see Runtime strategy. Existing pools keep working; prefer vcluster for new density pools.

High availability

The operator supports multi-replica deployments via leader election. Only the leader runs controllers; replicas are warm standbys that take over within seconds if the leader pod fails. The API server runs on all replicas — requests are load-balanced, and any replica can serve them.

Binaries

Binary	Role
kobe-operator	Operator + HTTP API server
kobe	CLI client
crdgen	CRD schema generator
kobe-sync	Per-cluster runtime for the vkobe backend (development on hold)