Turn Any Web Page Into Clean Markdown for RAG

To put a web page into a RAG pipeline, fetch it as Markdown instead of HTML, split it on its headings into chunks of a few hundred tokens, and store every chunk with the URL it came from. Markdown keeps what a model uses (headings, lists, tables, links and code) and drops what it does not (menus, scripts, styles and tracking tags). The page gets smaller, the chunks follow the author's own structure, and every answer can cite its source.

This guide walks through each step with code in Python and Node.js. It uses Grabber for the fetching, but the chunking and storage parts work with Markdown from anywhere.

Why not embed the HTML?

A web page is mostly not content. The HTML a server sends carries the navigation, the footer, cookie banners, inline scripts, style attributes and analytics tags, and the article sits somewhere in the middle. Feed that to an embedding model and three things go wrong:

• You pay for noise. Every tag and class name is tokens: once in the embedding call, and again in the prompt each time the chunk is retrieved.
• Chunks lose their meaning. A chunk cut at a fixed size can be half menu and half paragraph. Its embedding lands somewhere between the two and matches neither kind of question well.
• The model reads boilerplate. Retrieved context full of "Accept all cookies" and "Sign up for our newsletter" leaves less room for the passage that answers the question.

Plain text fixes the noise but throws away the structure. Without headings you cannot tell where one topic ends and the next begins, tables collapse into runs of words, and links lose their targets. Markdown sits in between:

When a RAG answer is wrong, look at what was retrieved before you change the prompt or the model. Very often the right passage was never in the index in a usable form. Clean input is the cheapest fix there is.

Fetch a page as Markdown

Grabber has one endpoint. Send the URL and formats=markdown with your API key, and the Markdown comes back in a JSON envelope, together with details about the fetch:

import os

import requests

API = "https://api.grabber.sh/fetch"
HEADERS = {"Authorization": f"Bearer {os.environ['GRABBER_API_KEY']}"}


def fetch_markdown(url):
    params = {"url": url, "formats": "markdown"}
    response = requests.get(API, params=params, headers=HEADERS, timeout=130)
    response.raise_for_status()
    data = response.json()["data"]
    if data["fetch"]["httpStatusCode"] >= 400:
        return None
    return {
        "url": data["fetch"]["loadedUrl"],
        "fetched_at": data["fetch"]["loadedTime"],
        "markdown": data["markdown"],
    }

Two fields in data.fetch matter for an index:

• loadedUrl is the address after redirects. Store that one, so a citation points at the page a reader will actually see.
• httpStatusCode is what the site itself answered. A site's "Page not found" page still loads, so Grabber returns it with a 200 and charges for it like any loaded page. Check the site's status and skip error pages, or they end up in your index.

If you only need the text, add unwrap=true and the response body is the Markdown itself, with Content-Type: text/markdown. You lose the fetch details, so for an indexer the envelope is usually worth keeping.

A fetch costs $0.001. Errors, such as a timeout, a blocked address or a site that cannot be reached, cost nothing, and the X-Grabber-Cost header on every answer says what that request cost.

How much smaller is the Markdown?

It depends on the site, so measure your own pages instead of trusting a rule of thumb. Ask for the original body and the Markdown in the same request (several formats cost the same as one) and count tokens with tiktoken:

import tiktoken

response = requests.get(
    API,
    params={"url": "https://docs.python.org/3/tutorial/classes.html", "formats": "raw,markdown"},
    headers=HEADERS,
    timeout=130,
)
data = response.json()["data"]
encoding = tiktoken.get_encoding("o200k_base")
print("HTML tokens:    ", len(encoding.encode(data["raw"])))
print("Markdown tokens:", len(encoding.encode(data["markdown"])))

Run it on a handful of the pages you plan to index. Documentation and articles with heavy page templates shrink the most; pages that are mostly text already shrink less.

Split on headings, not on character counts

The usual first attempt is a fixed-size splitter: a chunk every 1,000 characters, with some overlap. It is simple, and it ignores the document. Markdown gives you something better for free: the author already divided the page into topics, and the headings mark where each one starts.

A good chunker for web Markdown follows four rules:

1. Start a new chunk at every heading up to level 3. Deeper headings stay inside their parent.
2. Keep the heading path (Classes > Inheritance > Multiple inheritance) with the chunk. A paragraph that says "this also applies to subclasses" means little without it.
3. Split long sections at paragraph breaks, never mid-sentence, once they pass your size limit. Around 300 to 800 tokens works well with most embedding models.
4. Never split inside a code block or a table, and never read a # inside a code block as a heading. In a shell snippet, it is a comment.

Here it is in Python. Sizes are in characters, at roughly four characters per token:

import re

HEADING = re.compile(r"^(#{1,3})\s+(.+?)\s*#*$")


def chunk_markdown(markdown, max_chars=3000):
    chunks, path, lines = [], [], []
    in_code = False

    def flush():
        text = "\n".join(lines).strip()
        if text:
            chunks.append({"headings": " > ".join(path), "text": text})
        lines.clear()

    for line in markdown.splitlines():
        if line.lstrip().startswith("```"):
            in_code = not in_code
        heading = None if in_code else HEADING.match(line)
        if heading:
            flush()
            level = len(heading.group(1))
            path[:] = path[: level - 1] + [heading.group(2)]
        lines.append(line)
        if not in_code and not line.strip() and sum(len(l) + 1 for l in lines) > max_chars:
            flush()

    flush()
    return chunks

Tables have no blank lines inside them, so a table always stays in one chunk. Embed each chunk with its heading path in front of the text, because a user's question often names the topic ("inheritance") rather than repeating the paragraph.

Store every chunk with its source

What you store next to each vector decides what your application can do later. Keep at least these:

With those, the model can answer with a link to the exact page, and a reader can check it.

The same pipeline in Node.js

An indexer touches many URLs, so the Node.js version adds two things a real crawl needs: a pause when it hits the rate limit, and a hash to skip pages that did not change. Grabber allows 60 requests a minute per key. Over that it answers 429 with a Retry-After header, the number of seconds to wait. Retrying at once does not help, because refused requests count toward the limit too.

import { createHash } from 'node:crypto'

const API = 'https://api.grabber.sh/fetch'
const sleep = (seconds) => new Promise((resolve) => setTimeout(resolve, seconds * 1000))

export async function fetchMarkdown(url) {
  for (let attempt = 0; attempt < 3; attempt++) {
    const params = new URLSearchParams({ url, formats: 'markdown' })
    const response = await fetch(`${API}?${params}`, {
      headers: { Authorization: `Bearer ${process.env.GRABBER_API_KEY}` },
      signal: AbortSignal.timeout(130_000),
    })
    if (response.status === 429) {
      await sleep(Number(response.headers.get('retry-after') ?? 60))
      continue
    }
    const body = await response.json()
    if (!response.ok) throw new Error(`${url}: ${body.detail}`)

    const { fetch: info, markdown } = body.data
    if (info.httpStatusCode >= 400) return null
    return {
      url: info.loadedUrl,
      fetchedAt: info.loadedTime,
      hash: createHash('sha256').update(markdown).digest('hex'),
      markdown,
    }
  }
  throw new Error(`${url}: still rate limited after 3 tries`)
}

export async function indexPages(urls, store, embed) {
  for (const url of urls) {
    const page = await fetchMarkdown(url)
    if (!page || (await store.hashOf(page.url)) === page.hash) continue

    const { markdown, ...source } = page
    const chunks = chunkMarkdown(markdown)
    const vectors = await embed(chunks.map((chunk) => `${chunk.headings}\n\n${chunk.text}`))
    const rows = chunks.map((chunk, i) => ({ ...source, ...chunk, vector: vectors[i] }))
    await store.replace(source.url, rows)
  }
}

chunkMarkdown is the Python function above, line for line. embed and store stand in for your embedding model and your vector database: store.replace deletes a URL's old chunks and writes the new ones, which keeps re-indexing simple.

Keep the index fresh

Pages change, and a stale index gives confident, outdated answers. A simple routine covers most cases:

• Re-fetch on a schedule that matches how often each source changes: daily for news and changelogs, weekly or monthly for documentation.
• Compare the hash before you embed. An unchanged page costs one fetch and no embedding calls.
• Drop URLs that start returning errors, instead of paying to fetch the same error page every week.

The cost is easy to work out because there is no plan. Re-fetching 10,000 pages once a week is about 43,000 fetches a month, which comes to about $43.

What about pages that need JavaScript?

Some sites send an empty shell and build the page in the browser, and a plain fetch of those returns little or no Markdown. Add js_render=true and Grabber loads the page in a headless browser first, for $0.005 instead of $0.001. Most documentation and articles do not need it, so try without it first. Static fetch vs JavaScript rendering shows how to tell, and how to fall back automatically.

FAQ

Is Markdown better than plain text for embeddings?

For web pages, usually yes. The token counts are close, but Markdown keeps the headings you need for chunk boundaries and heading paths, and keeps tables readable. If your embedding model works better on plain text, strip the Markdown after chunking, not before.

What chunk size should I use?

Start around 300 to 800 tokens and test with real questions from your users. Smaller chunks match narrow questions better; larger ones carry more context. Keeping the heading path with each chunk matters more than the exact size.

Can I get PDFs as Markdown?

Yes. Markdown is the default format for HTML pages and PDFs, so the same request works for a link to a PDF.

Can I get the page's links at the same time?

Yes. formats=markdown,links returns the Markdown and the page's links in one request, for the price of one. That is handy for crawling a documentation site one level at a time.

What does it cost to index 10,000 pages?

$10 for one pass at $0.001 a page, or $50 if every page needed JavaScript rendering. Pages that fail cost nothing, and new accounts get $1 of free credit, enough for the first 1,000 pages.