Client: Cognify · AI/ML & Data · US enterprise LLM platform (70 staff)
Reducing Hallucination Escalations for Enterprise RAG
TechSupport Team supported Cognify’s Python and LangChain retrieval-augmented generation platform. Over five months, the client’s Jira records show hallucination escalations falling from 41 to 17 per week. The client’s LangSmith evaluations report RAG precision rising from 71% to 89%.
Client and engagement facts
- Client
- Cognify
- Industry
- AI/ML & Data
- Business context
- US enterprise LLM platform (70 staff)
- Delivery model
- AI application support
- Application
- Enterprise LLM/RAG platform
- Supplied geography
- US
- Company size in source
- 70 staff
Related services: SaaS support
Application stack and tools
Application stack
- Python
- LangChain
- Pinecone
- OpenAI/Anthropic APIs
- AWS
Tools used
- Jira
- LangSmith
- Datadog
- Pinecone
Why Support Was Needed
Nightly evaluations could leave model or retrieval drift undetected for 24 hours. The team added faster drift alerts and health checks for the Pinecone index after tracing one incident cluster to a failed upsert.
Technical Work
- Rebuilt the LangSmith evaluation pipeline with 60-second drift alerts.
- Traced one cluster of escalations to a failed Pinecone upsert.
- Added Pinecone index health checks to the daily runbook.
Representative Incident
Two escalations in month four followed an upstream model API outage. The engineers recorded the dependency and added a fallback model route. The client retained the decision to activate failover.
Results and how they were measured
| Metric | Baseline | Result | Units | Measurement window | Sample / population | Source and scope |
|---|---|---|---|---|---|---|
| Hallucination escalations | 41/wk | 17/wk | /wk | 5 months | Client Jira; Scope: The five-month comparison excludes two escalations tied to the provider outage. | |
| RAG precision | 71% | 89% | % | 5 months | Client LangSmith evaluation set. | Client LangSmith |
| P95 latency | 4.1 s | 1.6 s | s | 5 months | Client Datadog |
Evidence and Limitations
The five-month comparison excludes two escalations tied to the provider outage. Precision refers to the client’s LangSmith evaluation set, and latency comes from Datadog.
Model Failover Ownership
Cognify owned the fallback decision. TechSupport Team was responsible for carrying out failover and documenting the incident.
Scope a Similar Engagement
Share the symptoms that reach your AI engineers, your evaluation cadence and the fallback routes you can approve. Request a support plan or explore SaaS technical support.
Get a technical support plan
Tell us which application needs support, what is reaching your engineers and when you need coverage. TechSupport Team will review the fit and explain the next scoping step.
- We reply by email.
- We agree a call about the roles, your stack and the hours.
- If there is a fit, we propose engineers after an assessment.