Kubernetes Platform Playbook

A document-preview service accepts files from customers and renders them in a Kubernetes pod. The application team has made a sensible choice: untrusted documents are processed by a separate worker. Its first production manifest is still rejected. The image uses a mutable tag, the container can run as root, no egress policy describes where the renderer may connect, and the pod receives the namespace’s default service account token.

This is admission doing useful work. It is not evidence that the platform has secured the renderer. The platform team cannot decide which object store the worker should reach, whether it needs the Kubernetes API, or how much customer work may share a pod. The application team cannot secure the API server, patch the nodes, or prove that network policy is enforced. Each team holds facts and controls the other lacks.

The contract is therefore precise: the platform team maintains an enforceable cluster boundary; the application team declares a workload whose authority and communication match its real behavior; both teams prepare to investigate the point where those claims diverge.

Begin with a cluster that can change safely

The platform team owns cluster creation, supported versions, upgrades, API deprecations, control-plane exposure, node images, core add-ons, audit policy, and emergency administration. Whether a provider operates the control plane changes the mechanics, not the team’s obligation to understand its configuration and evidence.

Publish a support window and an upgrade path. Before an upgrade, test the document renderer and representative platform workloads against the new API version, admission policies, container runtime, network and storage plugins, ingress or gateway implementation, DNS, observability agents, and autoscaling behavior. Test rollback where rollback is genuinely supported; where it is not, prepare forward repair and workload migration. A cluster that can only be patched during a calm week is already accumulating an incident constraint.

Administrative access should arrive through approved network paths and individual identities. Keep routine automation, human administration, and break-glass authority separate. Emergency access should be time-bound, logged, exercised, and reviewable without depending entirely on the identity or network component it may need to repair.

Audit records need enough context to reconstruct consequential API activity: authentication and authorization outcomes, denied requests, workload creation, role and binding changes, secret-object access, admission decisions, node operations, and changes to policy or audit configuration. Protect and export those records away from the cluster failure domain. Do not record secret values or request bodies indiscriminately in the name of evidence.

Treat nodes as part of the security boundary

A pod is isolated by several mechanisms, but it runs on a node that can observe and affect it. The platform team hardens and updates node images, restricts direct administration, protects metadata and node credentials, minimizes host-level agents, and detects unexpected changes. Node bootstrap, replacement, and retirement should be automated enough that a questionable machine can be removed without a heroic rebuild.

Scheduling is also a security decision. Internet-facing parsers should not casually share nodes with highly privileged platform components. Taints, tolerations, affinity, and dedicated pools can control placement, although placement alone is not a stronger runtime boundary. Workloads with exceptional isolation requirements may need dedicated nodes, sandboxed runtimes, or separate clusters, with the cost and operational burden stated plainly.

The application team’s side of the boundary lives in the pod specification. The renderer should run as a non-root user where the image permits it, prevent privilege escalation, drop unneeded Linux capabilities, use a read-only root filesystem where practical, avoid host namespaces and host networking, and mount only the files and devices it needs. Privileged mode and host-path mounts are platform exceptions, not debugging conveniences. Resource requests, limits, namespace quotas, and workload concurrency caps reduce exhaustion paths, though none should be mistaken for complete tenant isolation.

Make admission explain the platform contract

Admission is where a published baseline becomes an enforced decision. The platform team should reject workload definitions that violate its chosen policy: privileged execution, host access, disallowed capabilities, unapproved registries, mutable artifacts where exact identity is required, absent resource bounds, unsafe ingress settings, missing ownership metadata, or an undocumented exception. Pod Security Standards profiles can provide a common floor; additional admission rules can cover image, identity, metadata, and organization-specific constraints.

Roll new policy out with observation and representative tests before enforcement. System namespaces and trusted platform controllers may need capabilities ordinary applications must never receive, so policy scope must be explicit. An exception records the violated rule, business need, owner, affected workload and namespace, compensating controls, approval, and expiration. It must not become a permanent namespace in which all policy disappears.

A rejection should identify the field, rule, consequence, and compliant remedy. For the renderer, useful feedback says that the container may escalate privileges and shows the supported security context. “Policy failed” merely sends the application team looking for a bypass.

Workload admission policy checklist

Before the application team requests release, it should be able to show:

• an immutable image identity from an approved source, connected to the reviewed build and its scan or provenance evidence;
• an owner, data classification, impact tier, support contact, and intended namespace;
• a pod security context with non-root execution where feasible, privilege escalation disabled, unnecessary capabilities removed, and host access absent;
• CPU, memory, ephemeral-storage, replica, and concurrency expectations appropriate to the workload;
• a workload-specific service account, with token automount disabled when the Kubernetes API is not needed;
• the exact ingress and egress flows, including DNS, storage, telemetry, and administrative traffic;
• only the secret and configuration references required by the containers that consume them;
• health behavior, disruption expectations, deployment and rollback identity, security logs, and the runbook used when the workload is isolated.

The platform team tests the other half: each unsafe variant is actually denied. Submit a privileged pod, a host mount, a mutable or unapproved image, an excessive capability, a missing owner, and the renderer without its required security context. A policy repository that looks correct but is not exercised against the live admission path is weak evidence.

Join Kubernetes RBAC to workload identity

Kubernetes RBAC begins at namespace scope and narrows from there. Give each workload its own service account rather than sharing the namespace default. Read permissions by resource and verb: get, list, and watch expose different volumes of data; create, update, patch, and delete change the cluster; exec, port-forward, impersonate, binding roles, and creating workloads can open less obvious paths to greater authority. In particular, a subject able to create a pod may be able to mount resources that subject cannot read directly.

The renderer does not need the Kubernetes API, so it should not receive a token. It does need object storage. The application team defines the narrow storage operations and object prefixes required for input and output; the platform provides the supported workload-identity binding and prevents a pod from claiming another workload’s cloud identity. Review Kubernetes RBAC and external cloud or infrastructure permissions together. A service account with no powerful cluster role may still decrypt data, alter queues, or read an entire bucket through its attached identity.

Human deployment rights are separate again. The ability to deploy the renderer should not imply access to customer documents or unrestricted pod execution. Periodically review effective roles, bindings, group membership, workload identities, and unused accounts, including the escalation paths created by admission exceptions.

Make the network declaration match real traffic

The platform team supplies and verifies a network implementation that enforces policy. Creating a NetworkPolicy object is not a control if the cluster network plugin ignores it. Establish default-deny behavior for application namespaces, known DNS and telemetry paths, ingress or gateway standards, and a way for teams to test allowed and denied traffic.

The application team declares the renderer’s flows: requests arrive only from the preview API; the worker reads and writes through the approved object-storage path; it sends bounded telemetry; it does not contact arbitrary internet hosts, node services, the Kubernetes API, or infrastructure metadata endpoints. Network policy ordinarily constrains network-layer connections; it does not validate URLs, tenant authorization, HTTP methods, or the meaning of a destination. The parser and its upstream API still need safe fetching, object authorization, input limits, and content isolation.

Trace exposure beyond pod-to-pod traffic. Ingress controllers, gateways, load balancers, service types, DNS, port forwarding, dashboards, metrics endpoints, and temporary troubleshooting routes all create paths. Review transport security, forwarded-header trust, authentication delegation, administrative routes, request limits, and origins that bypass the intended edge. The platform owns the exposure mechanisms; the application owns the decision that a particular route may invoke a particular effect.

Preserve image identity from build to runtime

An approved registry says where an image came from, not whether it is safe. The build system should connect source revision, dependencies, builder, scan results, provenance or attestations where required, and the resulting digest. Admission should enforce the evidence appropriate to the workload’s impact and reject sources the platform cannot trust. At deployment and in runtime records, preserve the digest rather than relying on a mutable tag.

The application team owns image contents, dependency remediation, base-image compatibility, and the decision to ship a known finding. The platform owns registry trust, admission requirements, node-side retrieval behavior, and the inventory that relates running digests to workloads. A finding does not become urgent merely because it has a high label, nor harmless because a scanner missed it. Reachability, exposure, privilege, known exploitation, available mitigations, and workload impact determine action.

When the renderer is rebuilt, the team should be able to answer which digest replaced which, which pods still run the old digest, and whether rollback would reintroduce the vulnerable component. That chain also determines the scope of a malicious-image incident.

Keep secret authority narrow after mounting

Kubernetes Secret objects are sensitive API resources, not a complete secret-management system. The platform team configures encryption at rest for secret data where it operates the backing store, restricts API access, controls backup exposure, and may provide an external secret store with rotation and revocation. Base64-encoded manifest data is not encrypted and does not belong in source control.

The application team asks for only the values the renderer needs, mounts them only into the consuming container, and keeps them out of environment dumps, command arguments, logs, crash reports, debug endpoints, and support bundles. A subject allowed to list secrets can obtain their contents; a subject allowed to create pods may be able to mount them. Secret review therefore includes RBAC and workload-creation authority, not only direct get permission.

Configuration deserves similar attention. An ingress annotation, sidecar, feature flag, probe, service account, or controller-specific field can change exposure or authority without changing the application image. Include those objects in deployment review, drift detection, and incident timelines.

Investigate the broken contract, not just the pod

Several weeks after release, runtime monitoring reports a shell spawned beneath the renderer followed by a denied connection toward an infrastructure address. The immediate question is not “Can we delete the pod?” Pods are replaceable, and deletion may erase the easiest evidence while a controller creates an identical replacement.

An actionable event identifies cluster, namespace, workload and controller, pod and node, image digest, service account and external identity, admission result, network decision, recent deployment, and owner. The platform team supplies API audit records, node and runtime signals, network observations, admission history, cluster changes, and containment authority. The application team supplies expected process behavior, document and tenant context, release history, data exposure, application logs, and customer impact. Neither can scope the event alone.

Kubernetes incident response playbook

Use the alert to choose the narrowest safe first action, then widen when evidence demands it:

1. Establish scope and continuing harm. Identify the controller, replicas, nodes, digest, identity, network paths, affected data, and recent changes. Decide whether evidence points to one container, a workload, namespace, node, cluster, image source, or external cloud account.
2. Preserve volatile evidence. Capture relevant object definitions and status, events, audit and admission records, workload and runtime logs, network observations, image identity, node assignment, and a UTC timeline. Where policy and expertise allow, preserve process, filesystem, memory, or node evidence before replacement. Keep collection credentials and sensitive customer content out of the incident record unless required.
3. Contain without blind destruction. Pause or scale the renderer, block ingress or egress, revoke workload identity, rotate exposed secrets, quarantine the digest, freeze the relevant deployment path, and cordon a suspect node as the scope requires. Draining or rebuilding a node is not enough if a malicious image or controller will recreate the compromise.
4. Remove the cause and restore from known inputs. Repair the parser or manifest, remove persistence and unauthorized objects, replace suspect nodes, verify policy and identity, and deploy an identified artifact through the normal admission path. Do not restore authority merely because new pods are healthy.
5. Prove the security claim again. Re-run denied-network, admission, RBAC, secret-access, image-identity, and application abuse cases. Confirm telemetry on the replacement workload and search the fleet for the same digest, indicator, role, route, or policy gap.
6. Close ownership gaps. Record which platform control, workload declaration, detection, or response handoff failed; assign a durable correction and an owner; preserve residual uncertainty for the release authority.

Prepare variants for a compromised pod, abused service account, malicious image, exposed administration route, secret-access anomaly, node compromise, cryptomining, and data exfiltration. The steps should name who may cordon nodes, revoke external identity, quarantine an image, preserve node evidence, and decide customer or regulator communication. A generic incident document that stops at “contact the platform team” is not a Kubernetes playbook.

Cluster baseline checklist

Before a cluster receives production workloads, the platform team should have evidence for:

• supported Kubernetes and component versions, upgrade rehearsals, API-deprecation handling, node replacement, backup and restore expectations, and a tested emergency path;
• restricted control-plane reachability, individual administration, separated automation and break-glass access, protected audit records, and review of consequential changes;
• hardened, replaceable nodes; deliberate workload placement; minimized host agents; protected metadata and node credentials; and runtime telemetry;
• enforced pod security and admission policy, scoped system exceptions, useful rejection messages, negative conformance tests, and expiring exception records;
• namespace-oriented least privilege, workload-specific service accounts, review of escalation-capable verbs, and alignment between Kubernetes and external identity;
• verified network-policy enforcement, application namespace defaults, reviewed ingress and egress mechanisms, DNS and metadata treatment, and exposure inventory;
• approved image sources, digest-level runtime inventory, required scan or provenance evidence, quarantine, and emergency replacement paths;
• secret-data protection, restricted API and backup access, supported rotation and revocation, configuration drift detection, and recovery procedures;
• alerts that resolve to workload and platform owners, evidence outside the affected failure domain, and exercised containment at pod, namespace, node, cluster, image, and external-account scope.

The application team brings the matching workload evidence: intended behavior, immutable artifact, security context, narrow identity, declared traffic, bounded resource use, secret references, release and rollback plan, application telemetry, impact context, and a responder who can act. The cluster baseline cannot invent these facts, and the workload cannot supply its own trustworthy safety rails.

When the renderer’s manifest finally passes admission, the meaningful result is not a green policy check. Its digest, process privileges, API authority, storage access, network routes, secrets, resource bounds, owner, and response path now tell one consistent story. If runtime behavior breaks that story, the same contract shows both teams where to look and who can contain it.