In-place pod restarts: Boosting efficiency and workload reliability in Kubernetes v1.35

Historically, managing container failures in Kubernetes often necessitated recreating the entire pod, leading to inefficiencies and increased operational overhead. This approach, while functional for simpler services, proved problematic for modern multi-container pods with complex interdependencies. Recreating an entire pod would incur significant control plane churn, causing latency and pressure on the etcd backend, especially during large-scale failures. The newly introduced `RestartAllContainers` action in Kubernetes v1.35 addresses these challenges by allowing for in-place restarts of containers within a pod. This feature, which graduated to beta and is enabled by default in Kubernetes v1.36 via the `RestartAllContainersOnContainerExits` feature gate, ensures that when a container exits, the entire pod can be restarted on its existing node while preserving its runtime identity. This mechanism offers several key operational advantages. Firstly, it eliminates control plane overhead by maintaining the pod's identity, including its IP address, network namespace, and UID, thereby bypassing the need for IP reassignment and reducing scheduling latency. Secondly, it ensures node locality preservation, meaning restarted containers can instantly access local, warm storage caches, which is crucial for performance-sensitive applications. Lastly, it maximizes hardware efficiency, particularly for distributed AI training, where losing a single node can halt an entire job. By keeping accelerators like GPUs and TPUs bound, workloads can resume training significantly faster, directly reducing compute costs. For enhanced observability and SRE best practices, Kubernetes v1.35 also introduces the `AllContainersRestarting` Pod condition. This condition, set to `True` during restarts, provides SREs and autoscalers with clear signals, preventing false-positive alerts and allowing Prometheus to easily track recovery events through incrementing container restart counts. To effectively leverage in-place restarts, developers must ensure their applications are reentrant, as graceful termination (preStop hooks) is not supported, and prepare external tooling to expect re-running init containers without misinterpreting them as new deployments.