Why We Switched from OpenVINO 2024.3 to LangChain 0.2 for quantization

In Q3 2024, our inference pipeline’s p99 latency hit 2.1 seconds for 7B parameter LLMs quantized to...