Hack the Hydra

Website to Markdown API for RAG and LLMs

To convert one page, the free HTML to Markdown tool on this site is enough. To convert a whole documentation site or hundreds of URLs, use the website-to-markdown robot on Apify: give it start URLs, it crawls politely (robots.txt and Crawl-delay respected, no proxies) and returns one clean Markdown document per page. It costs $0.001 per page.

website-to-markdown on Apify →

Free tool or API?

For a single page or a snippet of HTML, paste it into the HTML to Markdown converter: it runs in your browser and is free. The API is for what a browser tab cannot do: crawling many pages, following links, reading a sitemap, or running on a schedule.

What it does

The robot fetches pages over plain HTTP, extracts the main content with trafilatura (navigation, footers and ads are dropped) and outputs Markdown with headings, lists, tables and, optionally, links. Options: maxPages, maxDepth, sameHostOnly, useSitemap, include and exclude URL patterns (globs), and a delay between requests. It does not render JavaScript, so pages that only build their content in the browser may come back empty.

Example call

curl -X POST "https://api.apify.com/v2/acts/extas_corp~website-to-markdown/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.python.org/3/tutorial/index.html"}],"maxPages":20,"maxDepth":1}'

The response is a JSON array with one item per page: URL, title, word count and the Markdown text.

Using the output for RAG

Split each Markdown document on its # and ## headings before embedding; see Markdown vs plain text for RAG. Headings give each chunk a meaningful title, which improves retrieval.

How do you crawl responsibly?

Only crawl sites you may crawl. The robot obeys robots.txt and identifies itself with a clear User-Agent, but you remain responsible for each site's terms and for the load you create. Keep maxPages and the delay modest.

Pricing

$0.001 per page ($1 per 1,000 pages), billed by Apify per result. Apify's free plan includes some monthly credit, so small crawls can cost nothing. The free converters on this site run in your browser; the Apify robots are for automation and volume. You pay only per result, with no monthly fee for the robot itself.

What the dataset contains

Every URL gives one item with a type field, so a pipeline can sort results without guessing:

Turn on saveHtml if you also want the raw HTML next to the Markdown, for example to re-extract later with different rules.

Worked example: a documentation section for a chatbot

Say you want the "Tutorial" part of a product's docs in a support chatbot, but not the API reference or the changelog. Start from the tutorial index, set maxDepth to 2 so the robot follows links from the index and from each chapter, add includeUrlGlobs such as https://docs.example.com/tutorial/** and keep sameHostOnly on. If the site publishes a sitemap, useSitemap finds pages that are not linked from the index.

Before embedding, filter on type = page, drop pages with a very low wordCount (often redirects or empty index pages), and split each document on its headings. Re-run the same input weekly on a schedule to keep the chatbot's knowledge in step with the docs.

What are the common problems?

FAQ

Does it render JavaScript?

No. It fetches HTML over HTTP, which is fast and cheap. Single-page apps that load content in the browser may need a different tool.

Does it respect robots.txt?

Yes, including Crawl-delay.

What does a page cost?

$0.001 per page result.

Can I run it on a schedule?

Yes, Apify schedules and webhooks work with every robot, so you can refresh a knowledge base weekly.

Last updated: November 2026. Facts about third-party tools come from their public repositories and documentation. Check licenses before commercial use.

Try it now

website-to-markdown on Apify →

Free HTML to Markdown converter →