"How May I Assist You?": Building MCP-Grounded Real-Time Voice Agents

AI agents and their supporting technologies (RAG, MCP, Long Term Memory and many more) are becoming the foundation of AI-powered systems across all industries. Equipped with advanced AI language models (LLMs) and vast amounts of data, these agents can handle many tasks much faster and more affordably than human labor.

As you might know, these agents can process not only text and media but also speech, or more precisely, voice.

The applications for voice-powered AI are incredibly diverse, ranging from customer support to personal assistance, and even serving as coaches or tutors.

In this blog post series, we're going to combine the power of MCP (Model Context Protocol) and real-time voice to build powerful agents you can talk to instantly.

Almost all the required services offer a free plan, so you won't actually need to pay for anything except the LLM tokens (which are incredibly cheap with models like gpt-4o-mini).

Before we dive into the technical details, let's cover some basic concepts. If you're already familiar with these, feel free to skip this introduction.

The Hallucination Problem

This all sounds great, but there was and still is a significant challenge with all language models: hallucinations: They make up information that doesn't exist but fits perfectly into the context, and these fabrications are so convincing that you might not even notice them.

One of the most effective ways to reduce hallucinations is grounding. This involves providing the AI with factual, real-world data, such as documents, web search results from credible sources, or even access to your CRM or email. The AI then primarily uses this information to process and accomplish whatever task you're asking for.

One of the key technologies that enabled this connection of language models to external data sources like email, cloud storage, etc., is MCP. I've written a lot about this transformative technology; you can find most of my posts by searching for "MCP" on my blog.

https://airabbit.blog/claude-integration-and-mcp-are-eating-the-world/

Voice + MCP = Superpower

By putting all this together, we can now build agents that are not just grounded or capable of transcribing speech, but agents that are truly able to process voice in real time to accomplish tasks spanning multiple services and data sources in realtime – just like we humans do every day when at work, online shopping, or booking a flight.

To build such an agent, you would typically need to integrate many technologies with a lot of moving parts. Fortunately, there's another amazing open-source tool out there that handles all this complexity for the developer, letting them focus only on configuring the agent itself. It is called Livekit.

You can find all the detailed information about Livekit on their official website. and you can also chat with the GitHub repository using deepwiki or copilot: https://deepwiki.com/livekit/agents

In a nutshell, the core idea is quite simple:

• Livekit provides the environment where agents run, consisting of virtual conference rooms, similar to what we use in Zoom or Teams.
• Developers build agents and connect them to these virtual conference rooms.
• The user connects to the room where the agent is located and can talk with that agent (or even multiple agents).

Please note that in this first part of the series, we will use a local sandbox version of Livekit that simulates the conference room where the AI voice-powered agent will run, so you don’t necessarily need to sign-up for the Livekit-Cloud yet.