Kubernetes Security Hardening in Production: Pod Security Standards, RBAC, and Runtime Threat Detection

A freshly provisioned Kubernetes cluster is not secure. It is a blank slate with a generous threat surface: pods that can run as root, service accounts with cluster-wide permissions, inter-pod traffic that flows freely, secrets stored as base64 in etcd, and no runtime visibility into what processes are actually executing inside containers.

Teams that deploy Kubernetes operationally often treat hardening as a post-launch cleanup item. It rarely gets that second pass. The result is production clusters that have the appearance of a modern infrastructure platform but the security posture of a server someone set up and never locked down.

This article covers the mechanisms that actually matter for production hardening: Pod Security Standards (the replacement for the deprecated PodSecurityPolicy), RBAC design for multi-team clusters, network policy for microsegmentation, secrets encryption at rest, admission control with Kyverno, and runtime threat detection with Falco. Each section includes YAML you can use directly.

Pod Security Standards: The Replacement for PodSecurityPolicy

PodSecurityPolicy was removed in Kubernetes 1.25. The replacement is Pod Security Standards (PSS), enforced through the Pod Security Admission (PSA) controller, which is built into Kubernetes 1.23+ and stable from 1.25.

PSS defines three policy levels:

• privileged: No restrictions. Equivalent to running as root on the node.
• baseline: Blocks the most dangerous configurations (host network, host PID, privileged containers) while allowing common workloads.
• restricted: Hardened profile. Requires non-root user, drops all capabilities, enforces seccomp, and blocks volume types that could be used for host access.

You apply these at the namespace level with labels:

apiVersion: v1
kind: Namespace
metadata:
  name: payments-service
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

The enforce mode blocks non-compliant pods. The audit mode logs violations without blocking. The warn mode returns warnings to kubectl at apply time. Run audit and warn first on existing namespaces to surface violations before switching to enforce.

For workloads that must comply with restricted, the pod spec needs explicit configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 2000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: payments-api:v1.4.2
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: tmp
              mountPath: /tmp
      volumes:
        - name: tmp
          emptyDir: {}

The readOnlyRootFilesystem: true setting catches a surprising number of applications that write to disk without the team realizing it. Having a dedicated emptyDir volume for /tmp is the standard workaround. If a container fails with this setting, it is often writing logs, caches, or pid files to the image filesystem, which is itself a design problem worth fixing.

RBAC Design for Multi-Team Clusters

The failure mode in RBAC is almost always over-permission: a service account with cluster-admin, a developer role that can read secrets across namespaces, or a CI system that can deploy to any namespace. These happen because creating a permissive role is faster than reasoning about the minimum required permissions.

For multi-team clusters, the structure that works in practice is:

1. Namespace-scoped roles per team for workload deployment and debugging
2. ClusterRoles for shared read access to cluster-wide resources (nodes, persistent volumes) with strict bindings
3. Separate service accounts per workload rather than a shared account per namespace
4. No cluster-admin for human users except break-glass accounts

A minimal developer role looks like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: developer
  namespace: payments-service
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log", "pods/exec"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]

Notice what is absent: create, update, delete on deployments. Developers can read and debug but not push changes manually. That gate belongs to CI.

CI service accounts follow the same principle, scoped to what the pipeline actually needs:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payments-ci
  namespace: payments-service
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payments-ci-deployer
  namespace: payments-service
rules:
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: payments-ci-deployer
  namespace: payments-service
subjects:
  - kind: ServiceAccount
    name: payments-ci
    namespace: payments-service
roleRef:
  kind: Role
  name: payments-ci-deployer
  apiGroup: rbac.authorization.k8s.io

For auditing RBAC state across a cluster, kubectl auth can-i --list --as=system:serviceaccount:payments-service:payments-ci is useful, but the output is verbose. Tools like rbac-tool (from alcide-io) or rakkess give a cleaner matrix view of what each service account can actually do.

The secret-access problem deserves specific attention. Any role with get on secrets in a namespace can read all secrets in that namespace. If your CI service account can read secrets (often because someone added it to debug a deployment), it can read database credentials, API keys, and other secrets it has no business touching. Prefer using projected service account tokens and external secret stores (External Secrets Operator with AWS Secrets Manager or Vault) over native Kubernetes secrets for sensitive values.

Network Policies for Microsegmentation

By default, all pods in a Kubernetes cluster can communicate with all other pods, across namespaces. Network policies are the mechanism for restricting this, but they only work if the CNI plugin supports them. Calico, Cilium, and Weave Net support network policies. Flannel in its default configuration does not.

The safe default is a deny-all ingress and egress policy per namespace, with explicit allowances added for known traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: payments-service
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

Then add specific policies for what is actually needed:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-ingress
  namespace: payments-service
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-egress
  namespace: payments-service
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: database
          podSelector:
            matchLabels:
              app: postgres
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

The DNS egress rule (port 53 to kube-system) is commonly forgotten on the first pass. Pods that cannot reach kube-dns will fail to resolve service names, which surfaces as connection refused errors that look like application bugs. Always include a DNS egress rule when locking down egress.

Cilium’s Network Policy extends the standard with L7 HTTP rules, allowing you to restrict not just which pods can communicate but which HTTP paths and methods are allowed. For services that talk to each other over known APIs, that is a meaningful additional constraint.

Secrets Encryption at Rest

Kubernetes stores secrets in etcd as base64-encoded data by default. Anyone with etcd access reads your secrets. Encryption at rest adds a layer so that etcd contains ciphertext rather than encoded plaintext.

The encryption configuration is set in the API server:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-32-byte-key>
      - identity: {}

The identity provider at the end is the fallback for decrypting existing unencrypted secrets. After enabling this configuration, existing secrets are not automatically re-encrypted. Run the following to force re-encryption:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

For production, the preferred approach is using a KMS provider (AWS KMS, Google Cloud KMS) rather than a static AES key. The KMS envelope encryption wraps a per-secret data encryption key with a KMS-managed key, which means the actual key material never touches etcd or the API server disk:

providers:
  - kms:
      name: aws-kms
      endpoint: unix:///var/run/kmsplugin/socket.sock
      cachesize: 1000
      timeout: 3s
  - identity: {}

The KMS plugin runs as a daemonset on control plane nodes and handles the encrypt/decrypt calls. AWS maintains the kubernetes-sigs/aws-encryption-provider plugin for EKS clusters.

Admission Control with Kyverno

Pod Security Admission handles baseline workload security, but policy enforcement for organizational standards (required labels, image registry allowlists, required resource limits) needs an admission webhook. Kyverno and OPA Gatekeeper are the two main options. Kyverno uses native Kubernetes YAML for policies, which makes it more approachable for teams already working in YAML.

An image registry allowlist policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: validate-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Images must come from approved registries: registry.company.com or gcr.io/company-project"
        pattern:
          spec:
            containers:
              - image: "registry.company.com/* | gcr.io/company-project/*"
            =(initContainers):
              - image: "registry.company.com/* | gcr.io/company-project/*"

A policy requiring resource limits (essential for preventing a single noisy pod from starving a node):

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-container-resources
      match:
        any:
          - resources:
              kinds:
                - Pod
              namespaces:
                - "!kube-system"
                - "!kube-public"
      validate:
        message: "All containers must have CPU and memory limits defined."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

Kyverno also supports generate rules (automatically create NetworkPolicies when a namespace is created) and mutate rules (add default labels, set image pull policies). The generate rule for default-deny network policy on namespace creation is worth setting up so new teams automatically get the right baseline.

Runtime Threat Detection with Falco

Hardening prevents many attack paths, but it does not cover post-exploitation: a compromised process, a container escape attempt, or a cryptominer that somehow got onto the cluster. Falco gives you visibility into what is actually happening at runtime, using eBPF or kernel module probes to watch system calls.

Falco ships with a default ruleset that covers a reasonable set of threat behaviors. The rules you care most about in production:

• Shell spawned in a container that is not expected to have shell access
• File writes to sensitive paths (/etc/passwd, /etc/cron.d, binary directories)
• Network connections initiated from unexpected containers
• Privileged container execution
• Kubernetes API access from a container without a service account that should have API access

Installing Falco via Helm:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set driver.kind=ebpf \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl=<your-webhook>

Custom rules target your specific workloads. A rule that alerts when any process attempts to write to /etc in a container that should be immutable:

- rule: Write to /etc in immutable container
  desc: Detect any write to /etc directory in containers tagged as immutable
  condition: >
    open_write and container and
    fd.name startswith /etc and
    k8s.pod.label.immutable = "true"
  output: >
    Write to /etc in immutable container
    (user=%user.name command=%proc.cmdline
     file=%fd.name pod=%k8s.pod.name
     namespace=%k8s.ns.name image=%container.image.repository)
  priority: WARNING
  tags: [filesystem, container, immutable]

Falco Sidekick routes alerts to Slack, PagerDuty, Elasticsearch, or any webhook. For production, route high-priority Falco alerts to your incident management system alongside your application alerts, not to a separate tool that gets checked quarterly.

Audit Logging

The Kubernetes API server generates an audit log of every API request. With audit logging off (the default on most managed clusters), you have no way to answer “which service account created that secret” or “who deleted the deployment” after the fact.

Audit policy configuration:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Do not log reads of configmaps and secrets in kube-system
  - level: None
    resources:
      - group: ""
        resources: ["configmaps", "secrets"]
    namespaces: ["kube-system"]
    verbs: ["get", "list", "watch"]

  # Log secret access at the Metadata level (no secret values in logs)
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

  # Log all changes to RBAC resources at Request level
  - level: Request
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  # Log pod exec and portforward at RequestResponse level
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/portforward", "pods/attach"]

  # Default: log metadata for all other requests
  - level: Metadata

The Metadata level logs who did what and when, without logging request or response bodies. Use RequestResponse sparingly because secret values in request bodies will appear in logs unless you explicitly exclude them.

Ship audit logs to a separate, append-only store that your cluster’s service accounts cannot write to. If a compromised workload can also delete audit logs, the audit trail is worthless.

Tradeoffs: Hardening Approaches

Approach	Protection	Operational Cost	When to Skip
Pod Security Standards (restricted)	High: blocks host access, privilege escalation	Medium: requires fixing existing workloads	Legacy apps that cannot be easily modified; use baseline instead
Network Policy (default deny)	High: blocks lateral movement	Medium: requires mapping all legitimate traffic	If CNI does not support network policies
RBAC least-privilege	High: limits blast radius of compromised account	Medium: requires mapping permissions per team and workload	Never skip this
Secrets encryption at rest (KMS)	Medium: protects against etcd dump, not API access	Low: transparent to workloads after setup	Dev clusters where etcd security is not a concern
Kyverno admission control	Medium: blocks non-compliant workloads	Medium: policy maintenance ongoing	Clusters with a single team where operational standards are enforced socially
Falco runtime detection	Medium: visibility, not prevention	Low: runs as daemonset, alert tuning required	Never skip in production; tune the rules but keep it running
Audit logging	Low: forensic only, not preventive	Low: API server flag, needs log shipping	Dev and staging; always on in production

Production Security Checklist

Before considering a cluster production-ready from a security standpoint:

Pod Security

• Pod Security Admission enabled with enforce: restricted on all application namespaces
• All containers run as non-root with explicit runAsUser
• allowPrivilegeEscalation: false on all containers
• capabilities: drop: ALL on all containers
• readOnlyRootFilesystem: true on all containers (with explicit emptyDir for write paths)
• Seccomp profile set to RuntimeDefault or a custom profile

RBAC

• No human users with cluster-admin except break-glass accounts (audited separately)
• Every workload has its own service account (no default service account usage)
• CI service accounts scoped to specific namespaces and resources
• automountServiceAccountToken: false on service accounts that do not call the API

Network

• Default deny-all ingress and egress policy per namespace
• Explicit allow policies for all known traffic paths
• CNI plugin confirmed to support network policies

Secrets

• etcd encryption at rest enabled with KMS provider
• Existing secrets re-encrypted after enabling encryption config
• Sensitive secrets in external secret store (Vault, AWS Secrets Manager) via External Secrets Operator

Admission Control

• Image registry allowlist enforced via Kyverno ClusterPolicy
• Resource limits required on all containers
• Required labels enforced (team, environment, component)

Runtime

• Falco deployed with eBPF driver
• High-priority Falco alerts routed to incident management
• Custom rules written for workload-specific threat behaviors

Audit

• API server audit logging enabled with appropriate policy
• Audit logs shipped to append-only external store
• Audit log retention meets your compliance requirements

The Compounding Problem

Each of these controls addresses a different layer of the threat surface. None of them is sufficient on its own. A pod running as root with cluster-wide secrets access undermines your network policies. Perfect network segmentation does not help if an attacker compromises a service account with broad RBAC permissions. Runtime detection without audit logging means you can see that something happened but cannot reconstruct what led to it.

The practical approach is to implement them in order of implementation cost versus blast radius reduction. RBAC scoping and Pod Security Standards give the most protection for the least operational disruption. Network policies require mapping your actual traffic patterns, which is valuable even outside the security context. Falco and audit logging are largely transparent to workloads and should be on from day one.

The cluster that ships with all defaults is not secured. The cluster that ships with hardening applied incrementally, starting at day one, avoids the retrofit problem entirely. The retrofit is always harder.