Markdown vs plain text for RAG

Short answer: index Markdown. It costs almost no extra tokens, and its headings and tables let you chunk on document structure and attach section context to every chunk, which generally improves retrieval compared with fixed-size chunks of plain text. Use plain text only when the source has no structure at all.

Free PDF to Markdown converter →

What does Markdown give you?

Boundaries: headings mark where topics start and end, so chunks do not cut a section in half.
Context: the heading path ("Billing > Refunds > EU customers") can be stored as metadata and prepended to the chunk before embedding.
Tables: rows and columns stay aligned, instead of a run of numbers with no labels.
Lists: steps and requirements stay separate items.

How do you chunk Markdown for RAG?

Convert documents to Markdown (PDF, Word, HTML).
Split on #, ## and ### headings.
If a section is still too long, split it on paragraphs with a small overlap.
Prepend the heading path to each chunk and store it as metadata.
Evaluate on 20–50 real questions and tune chunk size.

When is plain text enough?

Chat logs, transcripts and short notes have no meaningful structure. Converting them to Markdown adds nothing; chunk them by time or by speaker instead.

Worked example: the same page, two ways

Take one page of an internal handbook with a heading, a short policy paragraph and a three-row table of refund deadlines. Extracted as plain text, the table becomes a run of words and numbers: Region Deadline Fee EU 14 days 0 US 30 days 5 UK 14 days 0. A retriever that receives the question "what is the refund deadline in the US?" has to match "US" and "30" across a flat string, and nothing in the chunk says that 30 is a deadline rather than a fee.

Extracted as Markdown, the same content keeps its shape:

## Refunds
Customers can ask for a refund within the deadline below.

| Region | Deadline | Fee |
|---|---|---|
| EU | 14 days | 0 |
| US | 30 days | 5 |
| UK | 14 days | 0 |

The chunk now carries its own labels. Prepending the heading path ("Handbook > Refunds") gives the embedding model the topic, and the model answering the question reads "US | 30 days" as a row, not a guess.

Plain text vs Markdown at a glance

	Plain text	Markdown
Chunk boundaries	Fixed size or paragraphs	Headings, then paragraphs
Section context	Lost	Heading path as metadata
Tables	Flattened, labels separated from values	Rows and columns kept
Token overhead	None	Small (#, \|, -)
Best for	Transcripts, chat logs, notes	Manuals, reports, policies, docs

What are the common mistakes?

Chunks that are only a heading. A "## Refunds" line followed by a sub-heading creates an empty chunk. Merge headings with the text under them.
Repeated page headers and footers. PDF exports often repeat the document title and page number on every page. Remove them before splitting or every chunk will look alike.
Splitting a table. Keep a table in one chunk when it fits; if it does not, repeat the header row in each piece.
Never measuring. Write 20 to 50 real questions with their expected source section and check how often the right chunk comes back in the top 5. Change one thing at a time.

FAQ

Does Markdown cost more tokens?

Very little: a few characters per heading, list item or table row.

Do LangChain and LlamaIndex support Markdown splitting?

Yes, both provide Markdown header-based splitters.

What chunk size should I start with?

Start with sections split on headings and cap them around 300 to 800 tokens with a small overlap, then tune using your own test questions.

Last updated: November 2026. Facts about third-party tools come from their public repositories and documentation. Check licenses before commercial use.

2026-11-12