Evaluating Low-Latency LLM Agents for Enterprise WhatsApp Customer Support
Abstract
Comparative study on fine-tuned Llama-3 8B vs GPT-4o-mini latency, intent accuracy, and token optimization when handling multi-lingual customer inquiries in English and Hindi.
Published: 2026-06-18
Core Findings & Results
- Fine-tuned 8B parameter models running on edge GPUs matched GPT-4o accuracy for domain-specific appointment booking while cutting latency by 64%.
- Hinglish (Hindi written in Roman script) prompt templates reduced intent classification errors from 18% down to 2.1%.
### Background
Real-time conversational agents require end-to-end response delivery within 1.5 seconds to maintain natural messaging flow on WhatsApp.
### Latency Comparison
- **Quantized Llama-3 8B (vLLM Engine)**: 380 ms average response generation.
- **Hosted Cloud LLM API**: 1,240 ms average response generation.
### Conclusion
Deploying localized quantized models for intent matching combined with deterministic rule engines yields optimal speed and cost efficiency.