Secure Kubernetes and Container Architecture

The payments team from the previous chapter is moving its refund API onto the company’s Kubernetes platform. The cloud landing zone already gives the cluster private networks, short-lived administrative access, central logs, approved registries, and a workload-identity broker. That inheritance is valuable, but it does not make the refund deployment safe. Its manifest can still request a privileged container, mount the node filesystem, adopt a powerful service account, accept traffic from every namespace, or run an image whose contents nobody can reconstruct.

The deployment therefore begins as a claim, not a running workload. It claims that a particular image may execute with a particular identity, filesystem, network path, secret, and amount of compute. Cluster policy must decide whether that claim is acceptable before a node acts on it. While the pod runs, telemetry must reveal behavior that contradicts the claim. If the image, identity, or node becomes unsafe, operators must be able to contain and replace it without improvising the evidence path.

Establish the cluster boundary first

The API server is the authority that turns declarations into running state. Restrict it to approved administrative and automation paths, authenticate individual people and workloads, and retain audit events for changes to workloads, RBAC, secrets, admission policy, nodes, and network exposure. A managed control plane changes who patches some components; it does not transfer ownership of authentication, authorization, policy, audit configuration, or the workloads admitted to the cluster.

Worker nodes belong to the trusted computing base. Keep their operating system, container runtime, and node agents on an owned upgrade path. Protect cloud instance metadata, constrain the node role, and avoid routine shell access. A node compromise crosses the ordinary container boundary, so workloads with materially different trust or impact may require separate node pools—or separate clusters—rather than labels that merely describe the difference. Control-plane and kubelet endpoints must not be anonymously or publicly reachable. Access to kubelet proxy functions can amount to command execution in pods and can evade the admission path that governed their creation.

Namespaces provide names, policy scope, quotas, and delegated administration. They are useful boundaries, but weak ones when a principal can create arbitrary pods, select another service account, mount namespace secrets, or obtain an admission exception. Put workloads of similar trust under the same namespace administration, and let admission, RBAC, network policy, and node placement do the enforcing work.

Before accepting application workloads, the platform owner should be able to demonstrate:

• a supported cluster and node version, owned upgrade cadence, and tested compatibility for the network, storage, policy, and observability layers;
• restricted API-server, kubelet, node, and emergency-administration paths;
• individual human access, narrow automation identities, and protected audit logs outside the workload’s control;
• a network implementation that actually enforces NetworkPolicy, rather than merely accepting the objects;
• default admission rules for ordinary applications and a stronger boundary for high-impact workloads;
• backup and recovery of cluster-critical configuration, with application data recovery handled according to the next chapter’s requirements.

The application team should not need cluster-admin to discover whether these promises are real. Policy test results, version status, audit searches, and a documented emergency path are part of the platform interface.

Promote an image, not a tag

The refund API starts in source control, but production runs an image. Preserve the chain between them: source revision, reviewed build definition, build identity, approved base image, dependency result, scan result, registry, digest, deployment revision, and runtime pod. Build in an isolated pipeline with narrowly scoped credentials. Keep package-manager tokens and signing material out of the build context, layers, logs, caches, and test fixtures.

Use the smallest base that the team can still patch and diagnose responsibly. Minimal does not mean mysterious: the owner must know which packages and certificates are present and how the image will be rebuilt when a vulnerability is disclosed. Scan during the build and continue evaluating deployed digests as vulnerability knowledge changes. A clean scan at noon is not a permanent property of the bytes.

Promotion should resolve a reviewed artifact to an immutable digest. A tag may help humans find a release, but payments/refund-api:stable can later point to different bytes. The runtime declaration should identify the digest that was scanned and approved. For workloads whose threat model warrants it, admission also verifies a signature or provenance statement from an authorized build identity. Verification must bind the attestation to the artifact and expected issuer; the existence of an attached signature is not enough.

A useful image review follows four questions:

1. Can the team reconstruct the image from an owned source revision and build definition?
2. Can it identify the base, packages, scan result, provenance, and approving policy for this exact digest?
3. Did any secret enter the build context, layer history, cache, log, or configuration copied into the image?
4. Can the team find every running instance of the digest, block a compromised one, and promote a replacement without changing the code in place?

Only after those answers exist should the digest become a candidate deployment.

Make the pod declaration narrow enough to judge

The refund API does not need host administration. Its container can run as a non-root user, disallow privilege escalation, drop all Linux capabilities and add back none, use the runtime’s default seccomp profile, and keep its root filesystem read-only. Writable state belongs in explicit ephemeral or persistent volumes with size and access decisions, not anywhere the process happens to write. The pod does not need host networking, host PID or IPC namespaces, privileged mode, or a hostPath mount.

The relevant security context is deliberately uneventful:

spec:
  serviceAccountName: refund-api
  automountServiceAccountToken: false
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: api
      image: registry.example/refund-api@sha256:<approved-digest>
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop: ["ALL"]
      resources:
        requests: {cpu: "250m", memory: "256Mi"}
        limits: {memory: "512Mi"}

This is an illustrative fragment, not a complete deployment. In particular, automountServiceAccountToken: false is appropriate because the application does not call the Kubernetes API. Its cloud workload identity is delivered by the platform’s identity mechanism rather than by granting broad Kubernetes API access. A controller that genuinely calls the API needs a projected short-lived service-account token and a narrowly defined role.

Resource requests and limits are part of the security claim. They influence scheduling, noisy-neighbor behavior, and the ways a runaway or attacked process can exhaust a node. A memory limit without a tested failure behavior may turn pressure into a restart loop; a CPU limit may change latency. Measure and test the declared bounds rather than pasting conventional values into every manifest.

Some platform agents cannot satisfy the ordinary workload baseline. A network or runtime sensor may need host visibility; a storage driver may need privileged operations. Do not weaken the shared rule to accommodate them. Isolate the component in a platform-owned namespace and, where warranted, dedicated nodes. Name the capability and host access it needs, constrain its image and identity, monitor its use, and record an approving owner, compensating controls, expiry or review date, and removal condition.

Treat RBAC verbs as capabilities

The refund-api service account needs no Kubernetes permissions for ordinary request processing. The deployment pipeline needs permission to update a specific workload through the approved release path, not to bind roles or create arbitrary cluster resources. A support engineer may read workload status without receiving exec or secret access.

Read RBAC by consequence. get, list, and watch on Secrets can disclose their contents. Permission to create pods or pod-producing workloads can allow a subject to mount other accessible secrets and use service accounts in that namespace. exec, attach, and port-forward create interactive paths into workloads. Permission to bind roles, escalate role definitions, impersonate subjects, approve certain certificate requests, or access node proxy endpoints can become an administrative route even when the role is not named cluster-admin.

Review every human, workload, delivery, support, and emergency subject against its verbs, resources, namespaces, names, and conditions. Prefer RoleBindings over cluster-wide bindings when the work is local. Avoid wildcards, which also grant access to resource types added later. Give each workload its own service account; a namespace-wide default account makes unrelated deployments share a revocation and compromise boundary. Disable token automount where a pod does not use the Kubernetes API, and review stale bindings as services and people leave.

Namespace administration remains a strong privilege. If an administrator can create a pod that adopts a powerful account or bypasses workload restrictions, the namespace label has not contained them. Admission must place a ceiling on what namespace-level RBAC can ask the cluster to run.

Deliver the one secret the process needs

The refund API needs a database credential or, preferably, an identity-based database session. Bind that authority to the workload’s environment, namespace, service account, and purpose. An external secret service can issue or expose the value through an audited, renewable path; it cannot stop the application from logging the value after reading it. Application behavior remains inside the secret boundary.

Mount a secret only into the container that consumes it. Sidecars and init containers do not inherit a legitimate need merely because they share a pod. Avoid placing secrets in environment dumps, command arguments, diagnostic bundles, crash reports, or deployment manifests. If a Kubernetes Secret is part of the delivery path, encrypt Secret data at rest, restrict get, list, and watch, understand what enters cluster backups, and remember that base64 encoding provides no confidentiality.

Rotation is a workload transition, not a vault setting. Decide whether old and new credentials overlap, how the process reloads them, which connection pools retain old sessions, what proves new use, and when the old credential is revoked. Exercise emergency revocation against the running deployment. An external store that rotates successfully while every pod continues using a cached value has not completed the control.

Turn communication into an allow-list

Without enforced network policy, ordinary pod networking is commonly open within the cluster. Begin the refund namespace with deny-all ingress and egress, then add the paths the workload actually uses. The API accepts traffic from the approved gateway on its service port. It reaches the database, the identity and secret services, central telemetry, and approved DNS. It does not receive traffic from arbitrary development pods or reach the Internet merely because a future integration might need it.

Test the policy through the cluster’s real network implementation. Creating a NetworkPolicy object has no protective effect when the network plugin does not enforce it. Check selected and unselected pods, both directions, expected ports, namespace selectors, and the paths through gateways or load balancers. A deny-all egress policy also blocks DNS until an explicit resolver path is allowed. Include the resolver destination and the risk of arbitrary names in the design rather than adding unrestricted egress when resolution fails.

Service mesh identity and encryption may strengthen service-to-service policy, but they do not repair an open node path, unsafe gateway configuration, or an application authorization defect. Record where transport identity begins and ends, which sidecars or proxies can alter it, how policy is enforced, and what happens when the mesh control plane is unavailable. Network reachability reduces attacker paths; Chapter 12’s authorization decision still determines whether a caller may refund a particular payment.

Let admission assemble the contract

Admission is where the platform tests the deployment’s claims together. For an ordinary production namespace, reject images outside approved registries, mutable production references, missing provenance where required, privileged or host-level access, privilege escalation, unapproved volumes, unsafe security profiles, missing resource declarations, broad service accounts, and absent owner or data-class metadata. Apply the Kubernetes Restricted Pod Security Standard where it fits, then add organization-specific rules for images, identity, volumes, exposure, and evidence.

Roll policy out as production software. Evaluate existing resources before enforcement, test allowed and denied fixtures in CI and against a disposable namespace, make denials explain the violated invariant and remediation, and watch admission latency and availability. A policy engine on every deployment path is itself a consequential dependency. Decide whether failure is closed or open for each risk class, and ensure an unavailable policy service cannot quietly become the normal bypass.

An exception is a separate, reviewable object. It names the workload and exact rule, explains why the ordinary shape cannot work, records the added threat and compensating controls, identifies approvers and responders, and carries an expiry or review date. Scope it to the smallest namespace, service account, image, or field the mechanism supports. Alert when it is exercised. A comment beside privileged: true is neither authorization nor containment.

The first useful test of this platform is negative. Submit a refund deployment with a mutable tag, privileged mode, host mount, default service account, or no resource declaration. Each unsafe shape should fail before scheduling, with a message an application team can act on. Then submit the approved digest and restricted spec and trace the policy decisions that allowed it.

Detect contradiction at runtime

Admission proves what was declared at one moment. Runtime evidence asks whether the process, node, network, and control plane continue to behave accordingly. For the refund API, an interactive shell, new package manager, unexpected child process, attempt to change identity or privileges, access to service-account or sensitive files, connection to a new destination, unusual secret read, or change to its admission exception contradicts the approved workload story.

Tune signals against expected behavior, but do not remove the context needed to judge them. An alert should identify the cluster, namespace, workload, pod, node, image digest, service account, deployment revision, network peer, policy decision, owner, and recent changes. Route it to responders who can assess both platform scope and customer impact. Monitor the evidence system itself; missing audit events or a silent runtime sensor may be the most important signal.

When a pod is suspected of compromise, preserve relevant audit, deployment, runtime, network, identity, secret-access, and node evidence before destroying the only useful state. Determine whether the boundary is the process, pod, namespace, node, cluster, image lineage, or surrounding cloud account. A workload-local event may be contained by removing ingress, blocking egress, scaling down, revoking identity, rotating secrets, and quarantining the digest. A possible container escape calls for node isolation and a broader search; rescheduling the same image on a fresh node is not containment.

Replacement is the recovery model for the container. Correct the source, build, policy, or credential; produce a new digest through the trusted pipeline; admit it under the same baseline; verify behavior and evidence; then retire the unsafe artifact. Stateful recovery, availability during containment, and the integrity of backups belong to the recovery architecture in the next chapter.

Approve the whole workload, not seven separate controls

The production review should be possible from one deployment record. It names the source and image digest; scan and provenance decision; pod security context and resource behavior; service account and effective Kubernetes and cloud permissions; secret delivery and tested rotation; allowed ingress, egress, and DNS; admission rules and exceptions; runtime signals and alert owner; and the containment path for a compromised pod, identity, image, or node.

Ask the platform to demonstrate the same record from the other direction: the cluster and node versions, restricted administrative endpoints, policy-engine health, network-policy enforcement, protected audit destination, exception inventory, and emergency authority. The application workload and cluster baseline meet at admission, but neither side can outsource its half of the claim.

The refund API is ready when the unsafe variants fail before scheduling, the approved digest starts with only its declared identity and paths, runtime evidence can distinguish drift from expected behavior, and responders can contain and replace it while preserving the evidence needed to understand the scope. Kubernetes then serves as an enforcement system rather than a shared administrator with a collection of advisory checklists.