Robust Optimization for Service-Level-Constrained Resource Allocation in Large Language Model Inference Serving

Abstract

Several Large language models (LLMs) show most of the incoming request rates and token workloads, which pose challenges for resource allocation when meeting latency Service-Level Objectives (SLOs). While prior LLM serving has advanced state-of-the-art scheduling, batching, memory management, and placement of resources, many such techniques depend on observed or estimated workloads to tune the allocation of resources, offering limited guarantees when request rates and token workloads arrive at jointly uncertain levels. This paper proposes a robust optimization to allocate resources under joint workload uncertainty. The request arrival rate and token workload are captured by a budgeted uncertainty set, and the analytical latency function governs the relationship between the workload and the computational capacity allocated. In the proposed formulation, the resource cost is minimized while a worst-case latency SLO is respected over the considered uncertainty set. A central contribution of this paper is the derivation of an effective workload that encapsulates the two-dimensional uncertainty budget into a singular worst-case workload, enabling analytical tractability of the robust resource allocation formulation. Due to the limited number of decision variables and constraints in the resulting formulation, it is amenable to practical capacity planning. Analytical results characterize the optimal allocation as a function of the level of uncertainty, workload intensity, SLOs, and resource efficiency. With the framework developed in this paper, one can directly interpret how much extra resource capacity is needed to guard against the joint uncertainty in workload. Numerical experiments are conducted to evaluate the nominal, single-source robust, and joint-robust allocation policies in terms of resource cost and SLO violations.