Static Fetch vs JavaScript Rendering: When You Need a Headless Browser

Most web pages do not need a headless browser. A static fetch, which is one HTTP request for the HTML the server sends, gets the full content of server-rendered sites: most documentation, news sites, blogs and shops. You need JavaScript rendering when the server sends an empty shell and the browser builds the page with scripts. The practical rule is to fetch statically first, check whether the content came back, and render only the pages that need it. With Grabber, that means $0.001 for most pages and $0.005 only for the rest.

What a static fetch sees

When a browser opens a page, it does two things. It downloads the HTML, then it runs the page's JavaScript, which may load more data and change what is on screen. A static fetch stops after the first step. A rendered fetch does both, in a real browser without a window, and reads the page after its scripts have run.

For a server-rendered page, the two give the same text, because the server already put it in the HTML. For a single-page app, the HTML the server sends looks like this:

<!doctype html>
<html lang="en">
  <head>
    <title>Acme Dashboard</title>
    <script type="module" src="/assets/index-4f2c1a.js"></script>
  </head>
  <body>
    <div id="root"></div>
    <noscript>You need to enable JavaScript to run this app.</noscript>
  </body>
</html>

Everything a visitor sees is drawn into that empty <div id="root"> by the script. A static fetch of this page gets a title and the noscript notice, and its Markdown is nearly empty.

Which pages need rendering?

You cannot always tell from the outside, but how a site is built is a strong hint:

The last row is a limit of rendering, not of static fetches. A headless browser runs the scripts that build the page, but it does not press buttons or sign in, so content behind a click or a login stays out of reach either way.

How to tell from the response

You do not have to guess site by site. Fetch the page statically and look at what came back. Any of these means it probably needs rendering:

• The Markdown is nearly empty, or holds only the menu and the footer.
• It contains a "JavaScript required" notice, like "You need to enable JavaScript to run this app".
• It says "Loading…" where the content should be.
• The thing you came for is missing: the article body, the price, the table you saw in your browser.

Before you switch to rendering, check one more place. Many sites that build their pages in the browser still ship the data inside the HTML, for search engines or to start the app:

• JSON-LD. Product pages, articles and recipes often describe themselves in a <script type="application/ld+json"> tag. Add includeMetadata=true and Grabber returns it already parsed, in data.metadata.jsonLd.
• Framework data. Next.js pages built with the Pages Router put their props in <script id="__NEXT_DATA__">, and Nuxt uses __NUXT_DATA__ or window.__NUXT__. Ask for formats=raw to get the original HTML, and read the JSON from it.

Both are static fetches at $0.001, and structured data is often easier to work with than the rendered text anyway.

Check a site by hand in two minutes

Before you write code for a new source, look at it the way a static fetch does:

1. View the page source (Ctrl+U, or Cmd+Option+U on a Mac). That is the HTML the server sent. Search it for a sentence you can see on the page: if it is there, a static fetch gets it too. Do not judge by the Elements panel in DevTools, which shows the page after its scripts have run.
2. Turn JavaScript off and reload. In Chrome DevTools, open the command menu (Ctrl+Shift+P, or Cmd+Shift+P), run "Disable JavaScript" and reload. What is left on screen is roughly what a static fetch sees.
3. Ask Grabber for both. Fetch the page once as it is and once with js_render=true, and compare the length of the Markdown. That is $0.001 plus $0.005, and it usually settles the question for the whole site, because the pages of one site tend to be built the same way.

Static first, render when needed

Put those checks in code and the decision makes itself. This version fetches statically, renders only when the result looks empty, and remembers which domains needed rendering, so it stops paying for a static attempt there:

import os
from urllib.parse import urlsplit

import requests

API = "https://api.grabber.sh/fetch"
HEADERS = {"Authorization": f"Bearer {os.environ['GRABBER_API_KEY']}"}
JS_NOTICES = ("enable javascript", "javascript is required", "javascript is disabled")
needs_js = set()


def grab(url, js_render=False):
    params = {"url": url, "formats": "markdown", "unwrap": "true"}
    if js_render:
        params["js_render"] = "true"
    response = requests.get(API, params=params, headers=HEADERS, timeout=130)
    response.raise_for_status()
    return response.text


def looks_unrendered(markdown):
    text = markdown.strip().lower()
    return len(text) < 500 or any(notice in text for notice in JS_NOTICES)


def fetch_page(url):
    domain = urlsplit(url).hostname
    if domain not in needs_js:
        markdown = grab(url)
        if not looks_unrendered(markdown):
            return markdown
        needs_js.add(domain)
    return grab(url, js_render=True)

Tune looks_unrendered to your own pages. A 500-character threshold suits articles and docs, but a page that is short on purpose, like a contact page, would be rendered for nothing. If you know what you are looking for, such as a price, check for that instead of the length.

Mind the timeout too. A rendered fetch waits for the page to finish loading and can take up to 120 seconds, so give your HTTP client at least 130. A static fetch is one request and comes back far sooner.

What it costs

At Grabber, rendering costs five times as much as a static fetch, $0.005 against $0.001, and it is slower. At volume, the way you choose adds up. Take 100,000 pages a month, of which 15% need rendering:

Static first costs about a third of rendering everything, and remembering the domains that need rendering trims a little more. The pricing calculator does this math for your own numbers.

A failed fetch costs nothing in any of these. If a rendered fetch times out or the site cannot be reached, you are not charged.

What rendering does not fix

Rendering solves one problem: content that scripts build in the browser. It is not a way around these:

• Logins. Grabber reads public pages and does not sign in, so a page behind a login returns what a signed-out visitor sees.
• Bot protection. Grabber loads pages directly and does not rotate proxies or solve CAPTCHAs, with or without rendering. If the sites you need block automated requests, a scraping API built around proxies is the better tool, and our comparisons say which.
• Interaction. Rendering runs the page. It does not fill in forms or click through tabs.

FAQ

Do I pay for rendering if the page fails?

No. Any error, including a rendered fetch that times out, costs nothing. You pay $0.005 only when the rendered page loads.

Can Grabber decide when to render?

No, you choose per request with js_render=true. The static-first function above does the same job on your side, with your own idea of what "empty" means.

Does rendering change the formats I can ask for?

No. Markdown, HTML, text, links and metadata all work with js_render=true, and they describe the page after its scripts have run.

Is a single-page app a lost cause without rendering?

Not always. Check for JSON-LD with includeMetadata=true, and for framework data in the raw HTML, first. When the data is there, a static fetch is cheaper and the data is cleaner.