We hear almost every day about new advancements in model intelligence and agentic capabilities, like the new Astra model or Fable 5.1, and open source is catching up faster than we thought.

While almost all of them can do powerful research and dive into the web, one thing they can't do is access data protected behind APIs, like LinkedIn, Reddit, and hundreds if not thousands of others.

Take an example: let's say you want to find a product you saw on Amazon, used, on eBay. The AI basically has a few core options:

• Use web search, which is based on indexed data (e.g. from the ChatGPT crawler). That becomes a game of chance. The data might be outdated or even removed, or not available at all because of bot protections from the provider (in this case, say it's eBay).
• Use agentic web browsing (using a Chrome extension like I talked about in a previous post, or computer use). In a nutshell, this emulates the user's activity in the browser on any website like eBay by simulating keyboard typing and so on. This delivers quite precise results, but it is very expensive and time-consuming, and when you have really large data, like a bunch of comments, it can take forever to evaluate.

With small use cases and a limited number of pages you can certainly do that. But if you want to do deeper research, say collecting large amounts of data from social media, this is a dead end.

What you need instead is giving your AI access to the API directly, whether that's X, LinkedIn, or whatever, to do that deep analysis.

With MCP and similar tools this is super easy, but there is still one caveat: you have to sign up, and in some cases even pay a monthly subscription, to access this data. And if you do cross-platform research, this can get very expensive.

Luckily there is a way around it, using a massive data scraping platform called Apify. If you don't know Apify, check them out here.

In a nutshell, they offer scrapers for almost the entire web's APIs and provide them through a single interface, which you can even connect to your AI. Basically giving it one connector that can access hundreds of APIs across the entire web.

I have already talked about this in many posts in the past, but frankly it was always still too difficult for some folks who are not very tech-savvy to understand the concept and connect Apify to their AI.

But recently Apify decided to go their own way and not just offer a connector (MCP) but even provide their own AI, which is not just a replacement for Claude and ChatGPT and the others, but one that knows, given a simple or complex question by the user, how to:

• select the scrapers that can help
• configure these scrapers automatically
• run and monitor them
• efficiently sift through the massive amount of scraped data to answer the question

Eventually the interface is as simple as ChatGPT or Claude, with one huge difference: access to almost the entire web.

And if you don't consume too many scrapes, you can even enjoy this power entirely for free (if you don't exceed the $5 monthly free scraping) (apify.com).

In my opinion this is so powerful that it is a huge threat to a lot of business models. All we need now are powerful prompts to pick, choose, and analyze the results.

Let's take a simple example. Say I want to look for a phone for a person with dementia (this is a real inquiry I had recently).

Once you have signed up, simply head to https://console.apify.com/ and open the Dashboard.

By default it will do what most other AIs do: it will search the web.

The results might vary, but likely it will pick some products that are highly ranked on Google or in the AI index. In some cases these products might be good, but they might also just be good SEO, right?

So how do we know if these products are really good and really helpful? We check reviews and the opinions of real people from comments, reviews, and so on, across the entire web.

So you can ask that same AI to dig deeper and check social media reviews, for example product reviews, opinions on Reddit, LinkedIn, etc., depending on the use case. Remember that you can also give it a budget so you don't exceed the free tier or a certain spending limit.

Under the hood it will spawn new scrapers, wait for them to finish, then evaluate the results.

And after a few minutes it has a different story. Actually, Apple products are rarely brought up despite offering a lot of features for people with disabilities. So the answer becomes very nuanced: for simple use cases go with product XYZ, and for more complex situations, and if the user is already using Apple (the feedback from the Reddit community in this case), Apple products might also be of great help.

This was of course just an example. The point is: to get the deep truth, using not just web search and highly optimized (and partially faked) reviews, but to spread the research across the entire web, so you get better insight and make better decisions.