운동하는 공대생

논문https://arxiv.org/abs/2403.02310 Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-ServeEach LLM serving request goes through two phases. The first is prefill which processes the entire input prompt and produces the first output token and the second is decode which generates the rest of output tokens, one-at-a-time. Prefill iterations have hiarxiv.org1. Introduction본 논문에서는 기존의 L..