Beyond Google & Perplexity: How NLWeb Aims to Connect Your Site Directly to AI

Microsoft recently introduced NLWeb, a new framework that aims to integrate standard web technologies with current AI approaches, such as MCP (Model Context Protocol) and RAG (Retrieval Augmented Generation).

The goal is to enable website owners and content creators to connect directly with AI chatbots, including Claude and potentially ChatGPT in the future. This approach could bypass intermediaries like Google, Perplexity, and even ChatGPT's own search functions.

The concept is straightforward and has the potential to significantly alter how we interact with online content.

This blog post will explore what NLWeb offers, discuss its potential significance, and look at some interesting use cases that might influence the future of AI-driven search.

What is NLWeb?

Here is a summary of what this framework has to offer:

NLWeb is an open‐source initiative of protocols and tools that makes it easy for websites to add conversational (natural-language) interfaces. By leveraging existing semi-structured formats like Schema.org and RSS—already used by over 100 million sites—NLWeb lets both humans and machine agents interact with websites using the same natural-language APIs (MCP). Like HTML did for documents, NLWeb aims to establish a common AI-Web layer: it provides proof-of-concept code (not a final solution) to inspire the community to build diverse implementations. Ultimately, NLWeb seeks a collaborative “barn-raising” of shared protocols, schemas, and examples so that every site can quickly spin up conversational endpoints and foster a web of connected agents.

https://github.com/microsoft/NLWeb.git

If you're primarily interested in the technical details of NLWeb and want to try it yourself, you can skip ahead. However, I recommend reading the 'Bigger Picture' section to understand the potential impact and possibilities this technology could offer.

The Bigger Picture: Why NLWeb Matters

The Current Search Landscape

For many years, Google has been the dominant force in web search. Recently, however, new competitors like Perplexity, ChatGPT, and others have emerged. In response, Google has integrated AI search features and its Gemini models, helping it maintain a strong position, though perhaps not with the same level of dominance it once held.

For users, this has been an exciting development. We can now ask questions and often receive direct answers in seconds, rather than sifting through numerous search results.

The Challenge for Content Creators

However, this shift can be challenging for content creators. AI search engines often index or perform live searches, then aggregate information to provide a concise answer. While convenient for users, this can mean that people no longer need to visit the original websites. This impacts revenue for many sites, including bloggers and content producers, as fewer visits can mean less ad revenue or other forms of engagement.

NLWeb proposes a way to shift this dynamic. It aims to direct AI-powered queries to websites themselves, similar to a peer-to-peer network, potentially bypassing intermediaries like Google.

Example: Searching for a Recipe

Let's say you want to make Tiramisu. If you search on Google, you'll likely get aggregated results presenting the ingredients and recipe directly. This summarized content comes from websites that have put effort into creating it, but they may not receive much direct traffic or benefit in this AI search model. The situation is similar with services like ChatGPT and Perplexity; while some may have arrangements with large publishers, smaller content creators (SMEs - Small and Medium Enterprises) can be overlooked.

Example: Searching for Points of Interest

You can imagine similar scenarios across various industries and types of searches. For instance, searching for Points of Interest (POI) in Madrid. Google will present its curated list of what it deems valuable. For some users, this summary might be their only interaction before visiting, possibly skipping the official websites of these locations and going straight to a map application.

This isn't a new phenomenon and has been a point of concern for some time, as it can affect businesses that rely on website traffic.

How NLWeb Aims to Address This

The Problem with Centralized Search

Modern search engines like Google and Bing are large, centralized systems. They use vast infrastructure to serve billions of users daily. While effective at finding information quickly, and increasingly precise with AI, this centralized model can reduce the control content creators (websites, shops, blogs) have over how their information is found and presented. Creators often need to invest in Search Engine Optimization (SEO) to align with the search engine's preferences.

Why don't users always go directly to a website to search? They sometimes do, especially if the site has a good internal search function (with or without AI). However, if a site's search isn't robust, using a general search engine like Google might seem easier.