I've spent most of this year telling anyone who'll listen that LLMs can't run your SEO, mainly because they simply don't grasp context the way humans do, they are prediction based, and whilst they can certainly handle a lot of SEO analysis, they aren't best placed to run your SEO strategy.

I tested the hyped up "Claude does your audits and automations" workflows. They were terrible, often wrong data interpretations, phantom recommendations, content strategies that would have had clients cannibalising half the pages already ranking. I said so, loudly, more than once.

So when Screaming Frog dropped version 24 last week with an MCP that plugs the SEO Spider straight into Claude, LM Studio and a few other assistants, you'd be forgiven for expecting me to sharpen the knives, I wanted to take it for a spin, I mean, being able to set up and manage crawls from Claude, got to be a bonus right?

What Screaming Frog Launched

Image source: Screaming Frog

The headline feature in v24 is a Screaming Frog MCP. In plain terms, you can now run crawls, analyse, export and manipulate the crawl data using the SEO Spider and node.js from inside Claude and other AI chat assistants, using natural language. Ask it to run a crawl and summarise the issues. Ask it to combine datasets, run the analysis pipeline, or pull together a link equity visual off the back of an Ahrefs integration.

There's a line buried in their announcement that tells you everything about why this one is different. Screaming Frog said it themselves: this is not a replacement for an experienced SEO professional, read that again, because it's the whole point.

Interface, not operator

Here's the distinction the AI-SEO crowd do not tend to make, and the one Screaming Frog got right.

The crawler is still the crawler, Claude isn't crawling your site, the Screaming Frog SEO Spider is. It's parsing the HTML, rendering JS, following the links, building the data the way a dedicated crawl tool does and an LLM categorically cannot. The AI is sat on top of it as a front end, you talk, the user agent does the work, the AI generates the output in a more convenient format.

That's a world away from "drop your .MD files in and let the agents crack on with your SEO." I've watched people genuinely try to get an LLM to crawl a large site and reason about it natively. It falls over, from 403 blocks to crawl loops, I've seen it all, this is why dedicated crawl tools remain supreme for a reason, and nothing about this release changes that, If anything it just helps to confirm it.

So no, I'm not contradicting myself, I've always said the problem isn't AI as a tool, the problem is people handing AI all of the responsibility and calling it strategy. This keeps the human in at the helm & uses the AI for the bit it's good at, which is turning "show me every page with a redirect chain that lost an internal link since the last crawl" into something you didn't have to click through fourteen menus to get.

Where it's genuinely useful

The honest answer is the boring stuff, and boring is fine, boring is where most of the time goes.

➡ Running a crawl and getting a fast plain-English summary of what broke

➡ Combining and exporting data without the manual faff

➡ Knocking up a quick visual from the data, like the internal link equity flow they showed off

➡ Letting someone less technical interrogate a crawl without needing to know the tool inside out

None of that is doing your thinking for you. It's shaving time off the donkey work so you've got more of it for the parts that actually need a brain - reading the rendered DOM, working out why a cluster of pages is sat in "crawled, currently not indexed," deciding what's a real opportunity versus noise.

Where it can still go wrong

I'm fine with the tool but I'm less fine with how some people will likely use it, probably for social clout.

The data going in is from the Spider, so it's solid, but the moment Claude starts interpreting that data and you treat the interpretation as gospel, you're back in the same hole I've banged on about for months. I fed AHREFS data into an LLM earlier this year and it confidently told me a set of competitors had all been hit by a helpful content issue, when the real story was the death of &num=100, the data was fine, the interpretation wasn't so much so.

So use it for what it is. Let it fetch, summarise, combine, visualise. Then you check the work. The same way you'd check a junior's first audit before it ever reached a client.

There's also the guardrails question, which applies to any agent touching your stack. Decide what it's allowed to do and where your sensitive data sits before you wire anything up. That's on you, not the tool.

The bit i'm not seeing anyone talk about yet on social

While lots of competent SEOs are busy losing their minds over the Claude integration, v24 quietly shipped a feature that's far more my cup of tea: Find Uncrawlable Links.

The Spider now flags common uncrawlable link types - things like a href stuffed into a span or div, or links fired through an onclick or a javascript: call. If you've read anything I've written about vibe-coded React shells and JS-event-based navigation, you'll know exactly why this matters. I've seen entire sites go uncrawlable because the nav and footer were JS event based. Now there's a filter that surfaces it for you.

Fair play to them for the honesty in the small print too, they note that Google will generally try to crawl anything in the HTML that looks like a link, so a lot of these will technically get picked up. But "technically crawlable" and "should be relied on for passing signals" are two very different things, and they're right not to pretend otherwise.

A few other useful bits in there: Auto Compare Crawls for scheduled and CLI jobs, crawl change summaries dropped straight into the completion emails, and the option to email export attachments automatically on completion - handy if you want a weekly broken links report landing in a developer's inbox without you lifting a finger.

So where does that leave us

I've been the bloke shouting that AI can't do your SEO. I stand by it.

This doesn't break that position. It proves it. Screaming Frog built the AI in as an assistant to a proper crawl tool, told you outright it won't replace a real SEO, and gave you a faster way to do the grunt work. That's the right side of the line.

Use it as an interface, don't use it as a substitute for knowing what you're doing.

Show me the clicks.