ML Infrastructure & MLOps • Resource Orchestration (Kubernetes, GPU Scheduling)Easy⏱️ ~2 min
What is GPU Resource Orchestration in ML Clusters?
GPU resource orchestration is the system that matches scarce, heterogeneous accelerators to diverse ML workloads while preserving isolation, fairness, and efficiency. Unlike fungible CPU cores, GPUs vary by memory size (ranging from 16 GB to 80 GB), interconnect topology (NVLink vs PCIe), and specialized features. A single NVIDIA A100 costs around $10,000 to $15,000, making efficient allocation critical.
The orchestration system operates through three control planes. A device discovery plane detects hardware, advertises capabilities like memory size and interconnect type, and tracks health metrics such as ECC errors and temperature. A scheduling plane decides placement using constraints including GPU count, memory requirements, topology preferences, and team quotas. A runtime injection plane wires containers to devices with correct drivers, libraries like CUDA, and security boundaries.
In production, clusters typically serve two dominant workload patterns. Long running inference services with tight Service Level Objectives (SLOs), often requiring p99 latency under 100 milliseconds. Bursty batch training jobs with large footprints, sometimes spanning thousands of GPUs. A well designed orchestration system balances these competing needs without starving either class. Netflix runs GPU accelerated media processing pipelines on Kubernetes with placement constraints, while Google's internal Borg co schedules accelerators with power capping and rack affinity for both serving and training workloads.
💡 Key Takeaways
•GPUs are not fungible resources. An A100 with 80 GB memory cannot substitute for one with 40 GB. Memory size, interconnect bandwidth, and compute capability must all match workload requirements.
•Device discovery agents run on each node to list accelerators, track health metrics like ECC error counts and thermal throttling, and advertise capabilities to the scheduler using standardized labels.
•Scheduling extends beyond simple bin packing. It must consider topology constraints, such as keeping 4 GPU jobs on a single node with NVLink rather than splitting across nodes with 30 to 60 percent throughput penalty.
•Runtime integration injects device files, CUDA libraries, and isolation boundaries into containers. This layer must handle vendor specific quirks without leaking complexity to application teams.
•Production clusters balance inference services needing p99 latency under 100 milliseconds against training jobs that may span 512 to 24,000 GPUs. Poor orchestration starves one class or creates fragmentation that wastes millions in hardware.
•Extended resources in Kubernetes represent GPUs as schedulable units. A typical cluster might advertise nvidia.com/gpu: 8 per node, with additional labels for memory class, multi instance mode status, and fabric connectivity.
📌 Examples
A 100 node cluster with 8 A100s per node provides 800 physical devices. With Multi Instance GPU (MIG) enabled in 1g.5gb profile, this expands to 5,600 logical instances for high density inference serving.
Meta disclosed training runs using approximately 24,000 H100 GPUs. At this scale, topology aware placement, power limits per rack (often 40 to 60 kW), and strict queueing are mandatory to avoid idle accelerators.
Google Borg schedules millions of containers daily, including GPU workloads. It uses power capping to stay within data center limits and rack affinity to maximize local NVLink bandwidth for distributed training.