Modal LLM Deployment Tutorial: Deploy Fine-Tuned Models with vLLM and LoRA

Ready Tensor · Beginner ·🧠 Large Language Models ·6mo ago

About this lesson

In this video, we deploy a fine-tuned large language model to production using Modal, a serverless GPU platform that makes LLM deployment simple, scalable, and cost-efficient. You’ll learn how to take a fine-tuned Hugging Face model, serve it with vLLM, enable LoRA adapters, and expose multiple HTTP endpoints for inference and streaming — all without managing servers or GPUs manually. What you’ll learn in this video: On-premise vs serverless LLM deployment strategies Setting up secrets and environment variables on Modal Deploying vLLM with LoRA adapters using Python Creating multiple inference endpoints (base, LoRA, streaming) Sending requests via Postman or Python clients Understanding scaling, idle timeouts, and concurrent requests Comparing custom vLLM logic vs OpenAI-compatible vLLM servers Timestamps: 0:00 - Overview of LLM deployment approaches 1:03 - Introduction to Modal and its serverless GPU model 2:00 - Setting secrets and Hugging Face tokens 3:01 - Deploying vLLM with LoRA using Python 6:05 - Creating live HTTP endpoints for inference 7:30 - Sending requests with Postman (base vs LoRA vs streaming) 9:32 - Alternative deployment using vLLM serve 11:18 - Autoscaling, idle timeout, and cost control in Modal This video is ideal if you’re building production-ready LLM APIs, deploying fine-tuned models for clients, or learning how to operationalize LLMs efficiently without managing infrastructure. This video is part of the LLM Engineering and Deployment Certification Program by Ready Tensor. Enroll Now: https://app.readytensor.ai/certifications/llm-engineering-and-deployment-DAROCXlj About Ready Tensor: Ready Tensor helps AI and ML professionals design, deploy, and evaluate intelligent systems through certifications, competitions, and real-world AI project publications. Learn more: https://www.readytensor.ai/ Like the video? Subscribe for more hands-on tutorials on LLM deployment, inference optimization, and production AI systems.

Original Description

In this video, we deploy a fine-tuned large language model to production using Modal, a serverless GPU platform that makes LLM deployment simple, scalable, and cost-efficient. You’ll learn how to take a fine-tuned Hugging Face model, serve it with vLLM, enable LoRA adapters, and expose multiple HTTP endpoints for inference and streaming — all without managing servers or GPUs manually. What you’ll learn in this video: On-premise vs serverless LLM deployment strategies Setting up secrets and environment variables on Modal Deploying vLLM with LoRA adapters using Python Creating multiple inference endpoints (base, LoRA, streaming) Sending requests via Postman or Python clients Understanding scaling, idle timeouts, and concurrent requests Comparing custom vLLM logic vs OpenAI-compatible vLLM servers Timestamps: 0:00 - Overview of LLM deployment approaches 1:03 - Introduction to Modal and its serverless GPU model 2:00 - Setting secrets and Hugging Face tokens 3:01 - Deploying vLLM with LoRA using Python 6:05 - Creating live HTTP endpoints for inference 7:30 - Sending requests with Postman (base vs LoRA vs streaming) 9:32 - Alternative deployment using vLLM serve 11:18 - Autoscaling, idle timeout, and cost control in Modal This video is ideal if you’re building production-ready LLM APIs, deploying fine-tuned models for clients, or learning how to operationalize LLMs efficiently without managing infrastructure. This video is part of the LLM Engineering and Deployment Certification Program by Ready Tensor. Enroll Now: https://app.readytensor.ai/certifications/llm-engineering-and-deployment-DAROCXlj About Ready Tensor: Ready Tensor helps AI and ML professionals design, deploy, and evaluate intelligent systems through certifications, competitions, and real-world AI project publications. Learn more: https://www.readytensor.ai/ Like the video? Subscribe for more hands-on tutorials on LLM deployment, inference optimization, and production AI systems.
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

Chapters (8)

Overview of LLM deployment approaches
1:03 Introduction to Modal and its serverless GPU model
2:00 Setting secrets and Hugging Face tokens
3:01 Deploying vLLM with LoRA using Python
6:05 Creating live HTTP endpoints for inference
7:30 Sending requests with Postman (base vs LoRA vs streaming)
9:32 Alternative deployment using vLLM serve
11:18 Autoscaling, idle timeout, and cost control in Modal
Up next
5 Levels of AI Agents - From Simple LLM Calls to Multi-Agent Systems
Dave Ebbelaar (LLM Eng)
Watch →