mnemo-jvm-warden

KO | EN

gitlite — search

#editor #ios #git #go #ruby #ai #terminal #ui #openai #llm

mnemo-jvm-warden

(★ 9)

💰 Cut K8s idle infrastructure costs. Safely shrink memory request during low-traffic windows with no cold-start penalties. ☸️

File Explorer

ZIP Download

- - assemble-pr-view.sh
  - common.sh
  - init.sh
  - migrate.sh
  - new-feature.sh
  - new-session.sh
- - constitution.md
  - design.md
  - session.md
- state.json
- - build.yml
  - publish-charts-release.yml
  - publish-charts.yml
  - publish-maven.yml
  - publish.yml
- - Chart.yaml
  - values.yaml
- - Chart.yaml
- pom.xml
- example-sidecar.yaml
- lifecycle-check.yaml
- oomkill-safety-check.yaml.tmpl
- prometheus-demo.yaml
- README.md
- verify-cross-pod-resize-denied.sh
- verify-lifecycle-ordering.sh
- verify-oomkill-safety.sh
- verify-prometheus-metric-source.sh
- verify-rbac-boundaries.sh
- verify-wardenpolicy-intent.sh
- verify-wardenpolicy-reconciler.sh
- verify-wardenpolicy-schema.sh
- warden-resize-admission-policy.yaml
- wardenpolicy-demo-deployment-policy.yaml
- wardenpolicy-demo-deployment.yaml.tmpl
- wardenpolicy-demo-metric-policy.yaml
- wardenpolicy-demo-pod.yaml.tmpl
- wardenpolicy-demo-policy.yaml
- wardenpolicy-sample-blackout.yaml
- wardenpolicy-sample-invalid.yaml
- wardenpolicy-sample-schedule.yaml
- wardenpolicy-sample-valid.yaml
- - savings-boundary.svg
- - constitution.md
- architecture.md
- banner-spec.md
- configuration.md
- hardening.md
- rbac-mapping.md
- roadmap.md
- security-review-checklist.md
- threat-model.md
- pom.xml
- pom.xml
- pom.xml
- pom.xml
- pom.xml
.dockerignore
.gitignore
Dockerfile
Dockerfile.controller
LICENSE
MAVEN_CENTRAL_RELEASE.md
pom.xml
README.md
SECURITY.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

# Project Badges

build Visit
coverage Visit

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**JVM-aware vertical right-sizing for Kubernetes.** Warden makes a running JVM
give memory back to the OS during predictable low-traffic windows, then safely
lowers the Pod's resource footprint — reclaiming idle infrastructure without
killing the pod, dropping its warm state, or risking your high-availability
posture.

---

## The gap Warden fills

In ~95% of production environments, [Horizontal Pod Autoscaling
(HPA)](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)
is the right answer: scale from 10 replicas to 2 when traffic drops, and you
save money. Warden does **not** compete with that.

But there are three well-known scenarios where *removing replicas* is not an option, and 
shrinking the pods you keep is the only lever left:

### 1. Minimum-replica HA baselines

High-availability architectures often can't scale below a floor — e.g. 3
sub-services × 3 replicas across 3 Availability Zones. If each JVM needs 4 GB at
peak, that's **36 GB reserved overnight to serve zero requests**. HPA can't
touch it; the floor is there for failover, not for load.

Warden keeps all 9 replicas alive for availability but shrinks each one's
footprint during the off-peak window — turning a 36 GB idle reservation into
~9 GB, **without reducing replica count**.

### 2. Slow-booting monoliths

Many enterprise Java apps (older Spring Boot, WebSphere, large multi-module
builds) take 2–5 minutes to boot and pass readiness. If HPA scales you down to 2
pods overnight and a sharp spike hits at 06:00, the new pods sit in
`ContainerCreating`/unready for minutes while the surviving 2 get slammed, spike
CPU, and cascade into an outage.

Warden keeps the pods **warm** — JVM up, classes loaded, connection pools
primed. Raising the footprint back to peak is a sub-second resize, not a cold
start.

### 3. Stateful nodes & distributed caches

For an in-memory data grid, a stateful worker queue, or a distributed cache
(this is [mnemo-cache](https://github.com/baokhang83/mnemo-cache)'s home turf),
you *can't* just kill a replica to save money — termination forces data
rebalancing, purges local cache, and thrashes surviving nodes with replication
overhead.

Warden instead **shrinks in place**: it flushes expired/evictable entries and
idle pages, runs a deep GC, uncommits the freed pages back to the OS, and keeps
the node running at a lower price point — with its hot working set intact.

---

## How it actually works (and where the money comes from)

A few load-bearing truths that shape the entire design:

**Cost lives in `requests`, not `limits`.** Lowering a Pod's memory *limit*
saves nothing — you still pay for the node. Kubernetes cost is driven by
*requests*, because that's what the scheduler bin-packs against and what
[Cluster
Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
uses to decide it can drain and delete a node.

**Warden's action boundary ends at "lower the request."** It resizes an existing
pod in place — it never changes the number of pods (that's HPA) or the number of
nodes (that's the autoscaler). Everything past the request change is *emergent
cluster behaviour that Warden enables but does not perform*, and does not
guarantee:

The `⇢` is a hand-off, not a step Warden takes. On a fixed node pool with no
autoscaler, Warden's lower-request still frees *schedulable* capacity but zero
*billed* capacity — so the savings story is only as real as your consolidation
setup.

**You can't change `-Xmx` on a running JVM.** Max heap is fixed at launch. What
*is* movable at runtime is the soft ceiling and committed memory:

- **ZGC** — `SoftMaxHeapSize`, settable live via `jcmd VM.set_flag`, plus
  automatic uncommit of unused heap.
- **Shenandoah** — aggressive uncommit via `-XX:ShenandoahUncommitDelay`
  (the gold standard for returning memory to the OS).
- **G1** — `-XX:G1PeriodicGCInterval` / `-XX:G1PeriodicGCSystemLoadThreshold`
  already trigger idle GC + uncommit.

The JVM already does half of this. **Warden's job is the orchestration** — it
drives these knobs from a traffic schedule *and* coordinates them with a
Kubernetes [in-place Pod
resize](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/),
which the JVM has no way to do on its own.

---

## The safety handshake

The shrink and grow sequences are **not symmetric**. Get the order wrong and you
`OOMKilled` the pod. This ordering is the core correctness guarantee of the
tool:

```mermaid
sequenceDiagram
    participant Sched as Traffic Schedule
    participant W as Warden Agent
    participant JVM
    participant K8s as Kubelet (cgroup)

Note over Sched,K8s: SHRINK — off-peak window begins
    Sched->>W: low-traffic window predicted
    W->>JVM: lower SoftMaxHeapSize / flush evictable cache
    W->>JVM: force deep GC + uncommit pages
    JVM-->>W: RSS dropped, pages returned to OS
    W->>W: verify actual RSS < target limit  ✅ gate
    W->>K8s: in-place resize DOWN (request + limit)
    Note right of W: cgroup lowered ONLY after RSS confirmed

Note over Sched,K8s: GROW — traffic returning
    Sched->>W: peak window predicted
    W->>K8s: in-place resize UP (request + limit) FIRST
    K8s-->>W: headroom granted
    W->>JVM: raise SoftMaxHeapSize
    W->>JVM: (optional) pre-warm cache ahead of load
```

- **Shrinking:** shrink the JVM first, *verify RSS actually fell*, **then** lower
  the cgroup. Lowering the cgroup before the JVM has released memory = instant
  OOMKill.
- **Growing:** raise the cgroup **first**, then raise the JVM's soft max. Give
  the headroom before the JVM tries to use it.
- **Cache re-warming** happens *ahead* of the predicted traffic return — a
  flushed cache is itself a cold start, so Warden shrinks early and re-warms
  early rather than reactively.

---

## Warden vs. VPA

Warden's real neighbor isn't HPA — it's the [Vertical Pod
Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler),
which now also supports in-place updates. The difference is that VPA is
**JVM-blind**:

| | VPA | mnemo-jvm-warden |
|---|---|---|
| Resizes the Pod in place | ✅ | ✅ |
| Can *cause* the JVM to release memory | ❌ observes RSS, can't move it | ✅ drives soft-max, GC, uncommit |
| Coordinates GC/uncommit with the resize | ❌ | ✅ ordered handshake |
| Understands cache / working-set state | ❌ | ✅ flushes evictable, keeps hot set |
| Driven by predicted traffic curves | ❌ reactive to observed usage | ✅ proactive, schedule-aware |

> **VPA resizes the box and hopes the JVM cooperates.
> Warden makes the JVM cooperate first, then resizes.**

---

## Deploying with Helm

Two charts, two different jobs. Both are published to a Helm repo hosted on GitHub Pages, so no
local checkout is needed:

```bash
helm repo add warden https://baokhang83.github.io/mnemo-jvm-warden/
helm repo update
```

(You can still install straight from a local checkout — `helm install warden charts/warden` —
if you're working on the charts themselves.)

### `charts/warden` — installs the controller

```bash
helm install warden warden/warden
```

This is the one-shot, cluster-wide install: it applies the `WardenPolicy` CRD (once — Helm's
`crds/` directory is never touched again by `helm upgrade`, see `charts/warden/crds/`), a
`ClusterRole`/`ClusterRoleBinding` scoped to exactly what the controller reads and writes (watch
`WardenPolicy`, patch its `status`, read `Deployment`/`StatefulSet`, read/patch `Pod`
annotations), and the controller `Deployment` itself (hardcoded to `replicas: 1` — there's no
leader election yet, so a second replica would double-reconcile every policy).

Values worth knowing (`charts/warden/values.yaml`):

| key | what it controls |
|---|---|
| `controller.image.repository` / `.tag` | which controller image to run |
| `controller.prometheusUrl` | where Prometheus lives, for guardrail metric evaluation (W-401) — leave empty if no policy uses a guardrail |
| `controller.resources` | the controller Pod's own request/limit (defaults to 256Mi request / 512Mi limit — the controller is a JVM with a heavy dependency graph, so its non-heap memory needs real headroom; 256Mi was too tight and OOMKilled it under load) |

### `charts/warden-sidecar` — the reusable sidecar template

There is no admission webhook. "Injecting" the sidecar means an **operator's own app chart**
includes a Helm named template that renders the same native-sidecar shape as
`deploy/example-sidecar.yaml`, parameterized instead of hand-copied. It's a Helm **library
chart** (`type: library`) on purpose: depending on it can never deploy a second controller
alongside your app, because a library chart is structurally forbidden from rendering any
resources of its own.

In your own app chart's `Chart.yaml`:

```yaml
dependencies:
  - name: warden-sidecar
    version: "0.1.4"   # match the published chart version (see `helm search repo warden`)
    repository: "https://baokhang83.github.io/mnemo-jvm-warden/"
```

In your own `values.yaml`, a `warden:` block shaped like `charts/warden/values.yaml`'s
`sidecar:` section (`targetContainerName` and `resources` are required, with no default — see
that file's comments for why):

```yaml
warden:
  enabled: true
  image:
    repository: ghcr.io/baokhang83/mnemo-jvm-warden
    tag: latest
  targetContainerName: app   # the sibling container Warden resizes
  resources:
    requests: { cpu: 25m, memory: 64Mi }
    limits: { memory: 256Mi }
```

And in your own Pod template:

```yaml
spec:
  shareProcessNamespace: true   # lets the sidecar see the target JVM's PID (W-102)
  volumes:
    {{- include "warden.sidecar.volumes" . | nindent 4 }}
  initContainers:
    {{- if .Values.warden.enabled }}
    {{- include "warden.sidecar" (dict "cfg" .Values.warden) | nindent 4 }}
    {{- end }}
  containers:
    - name: app   # must match warden.targetContainerName above
      # ... your app container, with the JMX flags deploy/example-sidecar.yaml documents
```

`warden.sidecar` renders only the `initContainer` entry; `shareProcessNamespace` and the
host-cgroup volume are set by your own chart rather than injected, so your chart stays in full
control of its own Pod spec end to end.

### Configuring a policy

Neither chart above makes Warden *do* anything on its own — it acts only once a `WardenPolicy`
exists for the target workload:

```yaml
apiVersion: warden.mnemo.io/v1alpha1
kind: WardenPolicy
metadata:
  name: my-app-policy
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  timezone: Europe/Paris
  profiles:
    off-peak: { request: 512Mi, limit: 768Mi }
    peak: { request: 2Gi, limit: 3Gi }
  schedule:
    - { cron: "0 22 * * *", profile: off-peak }
    - { cron: "0 7 * * *", profile: peak }
```

Apply it with `kubectl apply -f`, same as any other custom resource. For every field
(`leadTime`, `blackout`, `guardrail`), the sidecar/controller environment variables, and the
full set of Helm values, see **[docs/configuration.md](docs/configuration.md)** — it also has
an operator runbook for the failure modes you'll actually hit (read-only mode on an
unsupported collector, a shrink aborted by the RSS verification gate, cgroup mount issues, and
more).

---

## Status

🧪 **Beta.** The controller, agent, CRD, and both Helm charts are built, released, and
installable (published container images on GHCR, charts on the Helm repo above, libraries on
Maven Central), and the full schedule → shrink/grow → verify flow works end to end against a real
cluster. It has not yet been battle-tested under production load, and APIs may still change —
issues and feedback welcome.

## Requirements

- Kubernetes **1.35+** (in-place Pod resize is GA; on earlier versions it needs the
  `InPlacePodVerticalScaling` feature gate)
- A JVM with runtime-tunable heap commit: **ZGC**, **Shenandoah**, or **G1**
  with periodic GC enabled
- Cluster Autoscaler (or equivalent node consolidation) to realize cost savings
  from reduced requests

## Security

Warden runs with real privileges inside your cluster (RBAC, JMX, a host
cgroup mount). See **[SECURITY.md](SECURITY.md)** for how to report a
vulnerability, **[docs/threat-model.md](docs/threat-model.md)** for the full
trust-boundary analysis, and **[docs/hardening.md](docs/hardening.md)** for
an operator checklist before going to production.

## Press

[Automatically Shrink and Grow JVM Memory in Kubernetes](https://medium.com/@baokhang/how-to-shrink-grow-on-schedule-the-memory-of-a-fleet-of-jvms-8bbd3d4b3539)

## License

[Apache License 2.0](./LICENSE)

mnemo-jvm-warden

Owner Repos

File Explorer

# Use via CDN

# Project Badges

Was this content helpful?

Recent Feedback

mnemo-jvm-warden

Owner Repos

File Explorer

# Use via CDN

# Project Badges

Was this content helpful?

Recent Feedback

Download README