Local GenAI on Jetson: OSS models using different inferencing frameworks: Ollama, llama.cpp, & vLLM

NVIDIA Developer · Intermediate ·🏭 MLOps & LLMOps ·4w ago

Key Takeaways

Runs OSS models like Gemma and Qwen on Jetson using Ollama, llama.cpp, and vLLM for local inferencing

Original Description

This opening session builds the foundation for running popular OSS models such as Gemma, Qwen directly on Jetson — no cloud required. We cover when to use Ollama for rapid local prototyping versus vLLM for higher-throughput serving, show how the same workflow applies to both power different OSS models, and walk through the real decisions behind model choice, containers, quantization, and performance tuning on edge hardware. We close with a teaser of OpenClaw and a bonus take-home challenge to kick off community building.
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
Inference Infrastructure Best Practices for High-Traffic AI Applications
Learn best practices for building scalable inference infrastructure for high-traffic AI applications to ensure reliable and efficient deployment
Dev.to AI
 📰
Building a Self-Updating ML System: CI/CD, Deployment, and Everything That Broke Along the Way
Learn to build a self-updating ML system with CI/CD, deployment, and troubleshooting
Medium · Machine Learning
 📰
Building a Self-Updating ML System: CI/CD, Deployment, and Everything That Broke Along the Way
Learn to build a self-updating ML system with CI/CD and deployment using a real-world example from an MLOps portfolio
Medium · Deep Learning
 📰
The model alone won’t make the cut
A well-performing model is not enough for a successful product, emphasizing the importance of MLOps and software engineering in machine learning development
Medium · Machine Learning
Up next
Pole Pruner How A Rope Lever Shears High Branches
Innoforge Studio
Watch →