Why AI crawlers can't see your React site

Most assistant crawlers fetch the HTML your server sends and never run the JavaScript that builds the page in the browser. If your headings, product descriptions, service areas, or prices exist only after React loads, the crawler may see an almost blank document. That means an AI assistant may not quote your business accurately, or may not mention it at all.

Why can AI crawlers miss content on a React site?

AI crawlers can miss React content because the initial HTML contains the app shell, not the words a visitor came to read. A typical response might include a root element such as <div id="root"></div>, a few script tags, and references to CSS files. The browser then downloads JavaScript, runs React, calls an API, and inserts the visible content.

A person waits for those steps to finish. Many crawlers do not. They request the URL, read the response, and move on. Some assistant crawlers may support limited JavaScript, but you should not assume they will render your application like Chrome does. Crawler behavior also changes over time and is not always publicly documented.

This matters for simple pages as well as large apps. If a plumber's homepage sends only an empty root element, a crawler may not find phrases such as “emergency plumber in Bristol” or “boiler repair available today.” It cannot reliably infer those details from a logo, a page title, or a JavaScript file.

What is the difference between View Source and DevTools Elements?

View Source shows the HTML that arrived from your server, while DevTools Elements shows the document after the browser has run JavaScript and changed it. For crawler testing, View Source is usually the more useful first check.

Right-click a page and choose “View Page Source,” or press Ctrl+U on Windows or Linux and Command+Option+U on a Mac. Search that source for a sentence you can see on the page. For example, search for “same-day boiler repair.” If the sentence is missing from View Source but present in the rendered page, JavaScript added it later.

DevTools can create a false sense of safety because its Elements panel shows the finished DOM. That is what your browser sees after React, Vue, or another framework has done its work. It does not prove that the original response contained the content, or that a crawler without JavaScript can access it.

If the text exists only in Elements and not in View Source, treat it as invisible until you confirm that the crawler you care about renders JavaScript.

How can you check your site in thirty seconds?

You can check whether a page sends its main copy in the first HTML response with curl or View Source. Open a terminal and run this command, replacing the URL with one of your pages:

curl -L -A "Mozilla/5.0" https://example.com/services | grep -i "boiler\|plumbing\|service"

The -L option follows redirects, and the user-agent makes the request look more like a normal browser request. Replace the search terms with an exact phrase from your page. If curl prints the phrase, the server sent it in the HTML. If it prints nothing, check View Source to confirm. A page can still have other technical issues, but missing copy is a clear warning.

You can also save the response and inspect it more comfortably:

curl -L https://example.com/services -o page.html

Open page.html in a text editor and search for the page's main heading. Do not use a screenshot or the DevTools Elements panel for this test. Those show what a browser assembled, not necessarily what a basic crawler received.

RubyFlow flags this kind of missing server-sent content during a site scan, alongside other copy and freshness issues.

Which site builders have this problem?

Client-only React, Vue, and Angular applications are the clearest examples, but any builder that sends an empty app shell can have the same problem. The framework itself is not the deciding factor; the question is whether the server sends meaningful HTML for each URL.

• Client-only React apps that render everything after JavaScript loads.
• Vue and Angular single-page applications configured without server-side rendering or pre-rendered routes.
• Custom dashboards or portals where marketing pages were added inside a browser-only app.
• Some page builders that load text, listings, or location pages from JavaScript APIs.
• Sites that show product or service information only after a client-side search or filter runs.

Some page builders do not have this problem. Traditional WordPress themes, Squarespace pages, Wix pages, Shopify storefront pages, and static HTML sites usually send visible copy in the initial response. Their output can still be poorly structured or blocked by other settings, but the basic content is commonly available without running JavaScript.

The brand name is not enough to decide. WordPress can be used as a headless CMS with a client-only React front end. A React site can also be fully server-rendered. Test the actual URL instead of guessing from the technology used to build it.

Does Googlebot render JavaScript?

Googlebot can render JavaScript, but rendering happens on a delay and is not guaranteed to work as you expect. Google describes crawling and rendering as separate stages, and the rendered version may be processed later than the initial HTML.

That delay can affect discovery and updates. Googlebot may first find a page from its server response, then return later to render it. If your service text, links, canonical tag, or structured data appears only after JavaScript runs, Google may take longer to process it. Rendering can also fail because of blocked scripts, API errors, timeouts, unsupported browser behavior, or content that requires an interaction.

So this is not only an AI problem. A client-only site can be harder for Google to crawl and understand, even if Google eventually renders some of it. Googlebot's ability to run JavaScript is not a reason to hide important business information behind JavaScript.

Google Search Console's URL Inspection tool can show a rendered page and reported HTML, but it is not a substitute for checking the server response. Test the initial HTML, then test the rendered page and look at your crawl and indexing reports.

Which fixes make content visible to crawlers?

The strongest fixes put important page content into HTML before the browser needs to run JavaScript. Choose the least complex option that fits how often your pages change and how much control you have over the site.

1. 1.Server-side rendering (SSR): The server runs the framework for each request and sends a complete page. Next.js, Nuxt, and Angular Universal can use this approach. It suits sites with changing prices, stock, accounts, or location-specific content, but it needs more hosting and maintenance.
2. 2.Static generation: The site builds HTML files ahead of time and serves them quickly. This works well for a plumber with 20 service pages, a small agency, or a shop with stable product copy. A rebuild is needed when content changes.
3. 3.Prerendering: A service or build step visits selected JavaScript routes and saves their rendered HTML. This can help an existing single-page app without a full migration, especially when the page set is small and predictable.
4. 4.Edge rewriting: An edge service changes or injects the response before it reaches a visitor or crawler. This can be useful for targeted fixes, but it requires careful rules, testing, and a reliable source for the replacement content.

Do not treat a user-agent check as the main fix. Sending one version to people and a different version to crawlers can create cloaking problems and make debugging difficult. The safer goal is to send useful HTML to everyone, then enhance it with JavaScript.

How should you choose between SSR, static generation, and prerendering?

Choose static generation when your important pages change occasionally, SSR when they depend on a request or change frequently, and prerendering when you need a smaller repair for an existing client-only app.

• A local electrician with fixed service pages usually needs static generation or ordinary server-rendered pages.
• A restaurant with menus and opening hours may use static generation with a rebuild when details change.
• A property site with live availability may need SSR so each request receives current information.
• A SaaS marketing site built as a client-only app may be a good candidate for prerendering selected public routes.

Whichever route you choose, check more than the homepage. Test a service page, a product page, a location page, and any page that gets content from an API. Look for the title, main heading, body copy, internal links, canonical URL, and structured data in the initial HTML.

What should you do after making the fix?

After changing how pages are delivered, repeat the curl or View Source test and compare the response with the visible page. The important text should now appear before JavaScript runs.

Then check several URLs, not just one. Clear caches, confirm that redirects work, and make sure the server does not send an empty shell to some routes. Use Google Search Console to request a recrawl where appropriate, but remember that recrawling does not make missing HTML appear by itself.

Finally, watch for regressions after framework updates or redesigns. A site can quietly move from server-rendered pages to client-only rendering when a new template, search widget, or content component is added. A thirty-second source check on a new page can catch that before it affects search traffic or AI citations.