Baseline LLM Serving with FastAPI: Measure TTFT and Inter-Token Latency
About this lesson
In this video, we set up a baseline LLM inference server using FastAPI and use it to measure key serving performance metrics. The goal is to establish a reference point before moving to specialized LLM serving frameworks in later lessons. You’ll see how to serve a fine-tuned LLaMA-based model, expose generation endpoints, and benchmark latency under different request scenarios. You’ll learn how to: Serve a fine-tuned Hugging Face LLM using FastAPI Load base models, tokenizers, and LoRA adapters Implement standard and streaming generation endpoints Measure Time To First Token (TTFT) Measure Inter-Token Latency (ITL) Aggregate latency metrics using mean, median, and p99 Run batched inference experiments on a fixed GPU setup Understand why general-purpose frameworks are slower than specialized LLM serving systems Timestamps: 0:00 - Why FastAPI as a baseline for LLM serving 0:44 - Model loading and adapter setup 1:24 - TTFT and ITL metrics explained 2:10 - Measuring token-level latency during generation 3:38 - Computing mean, median, and p99 metrics 5:11 - Batched inference endpoint and limitations 6:20 - Running experiments on Colab with fixed GPU 7:01 - Interpreting TTFT and ITL results Watch this video if you’re building LLM inference services, benchmarking serving performance, or preparing to compare FastAPI against specialized LLM serving frameworks like vLLM or TGI. This video is part of the LLM Engineering and Deployment Certification Program by Ready Tensor. Enroll Now: https://app.readytensor.ai/certifications/llm-engineering-and-deployment-DAROCXlj About Ready Tensor: Ready Tensor helps AI and ML professionals design, evaluate, and deploy real-world intelligent systems through hands-on certifications, competitions, and project-based publications. Learn more: https://www.readytensor.ai/ Like the video? Subscribe for upcoming lessons on advanced LLM serving frameworks and performance optimization.
DeepCamp AI