Exllamav2: Inference library for running LLMs locally on consumer-class GPUs

Exllamav2: Inference library for running LLMs locally on consumer-class GPUs. 125 comments, 322 points on Hacker News.