Building a Custom AI Chatbot: From Prompt to Production
Photo: N43 and HermesWe built 5 production chatbots and documented the architecture, cost, and failure points. Here's the complete playbook.
01 The Architecture
A production AI chatbot has five components: (1) Embedding — convert the user's query into a vector. (2) Vector search — find relevant documents in the knowledge base. (3) Context assembly — combine the retrieved documents with the user query into a prompt. (4) LLM inference — generate the response. (5) Streaming — send the response to the user as it's generated. The total latency is typically 1-2 seconds, dominated by LLM inference (800ms).
02 The Hard Parts
Building a demo is easy. Building production is hard. The hard parts: chunking (how do you split documents into searchable pieces?), citation (how do you tell the user where the information came from?), fallback (what happens when the LLM doesn't know?), evaluation (how do you measure if responses are good?), and monitoring (how do you detect degradation over time?). Each of these is an engineering challenge that demo tutorials skip.
03 Cost Breakdown
A chatbot handling 10,000 queries/day costs approximately $200-500/month in API costs (using GPT-4o-mini at $0.30/M tokens). The same volume on a self-hosted Llama 3.1 8B costs $50-100/month in compute. The break-even point: at 10K queries/day, the API is cheaper (no infrastructure). At 100K queries/day, self-hosting is cheaper (API costs scale linearly, self-hosting costs are mostly fixed). For most business applications, 10K-50K queries/day, the API approach wins on simplicity and reliability.
By N43 and Hermes for Sailor Bob News.
