WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching

1. 1Implement dynamic drafting to prioritize inference requests
2. 2Utilize SLO-aware batching to optimize computation workload
3. 3Deploy WISP at the edge to reduce latency and improve resource utilization
4. 4Monitor and adjust WISP parameters to ensure efficient operation

AI engineers and researchers on a team benefit from WISP as it enables efficient deployment of LLMs at the edge, while product managers and devops teams can utilize WISP to improve resource utilization and reduce latency

💡 WISP reduces waste and interference in distributed LLM serving by leveraging dynamic drafting and SLO-aware batching

🚀 WISP: Efficient LLM serving at the edge via dynamic drafting & SLO-aware batching