To an AI crawler, your homepage has zero links
Open your homepage with JavaScript turned off. If you see a blank page or a spinner, that's what most AI crawlers see too.
What we found
In October 2026 we fetched the raw HTML of five small-business homepages as six different crawlers, for a test of SEO pixels. One site, a car broker, was built entirely in the browser. Its raw HTML had no H1 and zero links. To any crawler that doesn't run JavaScript, the site had no content and nowhere to go.
Who runs JavaScript and who doesn't
A December 2024 study by Vercel and MERJ found that OpenAI's crawlers, ClaudeBot and PerplexityBot fetch JavaScript files but don't execute them. Googlebot and Applebot do render JavaScript. So a client-rendered site can do fine in Google and still be close to invisible to the AI engines.
How to check in two minutes
- In Chrome, open DevTools, press Cmd+Shift+P (Ctrl+Shift+P on Windows), type "Disable JavaScript", and reload the page.
- Or count the links in the raw HTML from a terminal:
curl -s https://example.com/ | grep -o '<a ' | wc -l - Look for your main heading, your navigation links and your key text. If they're missing, crawlers that don't run JavaScript can't see them.
The fix
- Render on the server or at build time. Most modern frameworks, including Next.js, Nuxt, SvelteKit and Astro, can send complete HTML first and add interactivity after.
- Use real links. Navigation should be
<a href>elements, not click handlers on divs. - Prerender if you can't rebuild. Generating static HTML for your most important pages, and serving it to everyone, is a solid stopgap.
The free AI Agent Readiness check (my own product) reads a page without running JavaScript and flags missing headings, unlabeled controls and crawlability problems.