How Hundreds of Gigabytes of County Gazetteers Become a Conversational AI

RAG and vector databases, from first principles to deployment

I. Where the Problem Begins

Suppose you have the complete gazetteers of a single county. Old Ming and Qing records, mimeographed Republican-era editions, the official post-1949 gazetteers, twenty years of yearbooks, hundreds of family genealogies, cultural-and-historical materials from the People's Political Consultative Conference, old photographs — all stacked in the library's steel cabinets, on paper, gathering dust.

Someone says: AI is so powerful now, couldn't we just have an AI read through all of this, so that when I ask "In what year did the county suffer its worst flood in history?", it can just tell me?

It can. But not in the "read through" sense you're imagining.

II. AI Hasn't Read Through Every Book

This is the single most important correction to make.

Large language models (LLMs) like ChatGPT and Claude derive their "knowledge" from the internet text they read during the training phase. But they have one hard limit — the context window.

The strongest Claude currently has a context window of roughly 1 million tokens, equivalent to a few hundred pages of documents. But the complete historical record of a single county, as plain text, might run 100–500MB — equivalent to tens of thousands of pages.

That's a hundredfold gap. It won't fit.

So you cannot cram hundreds of gigabytes of scanned images into an AI and have it "read through" them. That is physically impossible.

III. You Don't Need Training, You Need Retrieval

Many people's first instinct is: "Let's train an AI on these county gazetteers."

This is a misunderstanding of what "training" means. Training a large language model is something companies like Anthropic and OpenAI do by spending tens of millions of dollars, using thousands of GPUs, and running for months. You don't need to do this — Claude already understands Chinese.

What you need is to make Claude turn to the right page before it answers the question.

This is RAG (Retrieval-Augmented Generation).

Here's an analogy:

You go to the library and ask, "What was the largest flood in Chinese history?"

AI without RAG: Answers from its own memory of what it read. It might misremember details, or even invent an event that sounds plausible but never happened — this is AI hallucination.

AI with RAG: First asks the librarian to find the 5 most relevant pages, opens them and lays them out in front of you, then answers based on those 5 pages, noting "See Volume 20 · Calamities, page 3."

RAG doesn't make the AI smarter — it gives the AI something to cite.

IV. The Full Pipeline: From Paper to Conversation

The complete flow of an intelligent county-gazetteer question-answering system breaks into four stages:

Unpacked, there are five core steps:

Step 1: Scanning

Turn the paper gazetteers into images or PDFs. Many county libraries have already done digitization, so you can just take the scans as-is.

Step 2: OCR (Optical Character Recognition)

Scans are images, and AI can't read the text inside an image. You need OCR to turn the images into editable plain text.

The difficulty depends on the age of the source material:

The open-source PaddleOCR (from Baidu) recognizes Chinese very well, and there are dedicated models for ancient texts too. This step costs close to nothing and takes a few hours.

Step 3: Chunking + Vectorization (the core step)

This step is the heart of the whole system. It involves two things:

Chunking: Cut the long text into small passages, each 200–500 characters.

For example, if County Gazetteer · Volume 20 · Calamities runs 20,000 characters in full, cut it into 40 chunks:

• Chunk 1: In the summer of the third year of Qianlong, forty days of incessant rain… (500 characters)
• Chunk 2: The following spring, the magistrate repaired the dike… (500 characters)
• Chunk 3: In the autumn of the nineteenth year of Guangxu, a great flood… (500 characters)
• … 40 chunks in total

Embedding: "Translate" each chunk into a string of numbers.

"In the summer of the third year of Qianlong, forty days of incessant rain, the river overflowed, drowning thirty thousand mu of farmland"

↓ passed through the Embedding model

[0.12, -0.34, 0.78, 0.02, ...] (1024 floating-point numbers)

This string of numbers encodes the "semantic fingerprint" of the passage. Text with similar meaning yields similar numbers.

This is the vector. The Embedding model is that translation machine.

Step 4: Store in the Vector Database

Once the vectors are generated, you need somewhere to store and retrieve them. This is the vector database.

The index in a traditional database (MySQL, PostgreSQL) is a B-Tree, which does exact matching — "WHERE name = '张三'".

The index in a vector database is HNSW or IVF, which does approximate nearest neighbor search — "find the 5 records that mean the most similar thing to this passage."

Step 5: Retrieval + LLM Q&A

When a user asks a question, the system does three things:

① Turn the question into a vector too

"In what year did the county suffer its worst flood in history?" → [0.11, -0.30, 0.80, ...]

② Search the vector database for the 5 most similar chunks

Hits:

• "In the summer of the third year of Qianlong, forty days of incessant rain, the river overflowed, drowning thirty thousand mu of farmland…"
• "In the autumn of the nineteenth year of Guangxu, a great flood submerged the city three chi deep…"
• "The county is bounded by water on three sides and has long been prone to flooding…"

③ Send these 5 original passages + the question to Claude together

After reading, Claude answers:

"According to Volume 20 · Calamities, the most severe flood in the county's history occurred in the summer of the third year of Qianlong (1738), when heavy rain fell continuously for forty days, the river overflowed its banks, and thirty thousand mu of farmland were submerged…"

Claude did not read the entire county gazetteer. It only read the 5 pages the librarian handed over. But those 5 pages were the right ones.

V. Two Independent Selection Decisions

Building a RAG system requires choosing two things, and they come from entirely different vendors:

The Embedding model (the translation machine) — responsible for turning text into vectors:

The vector database (the warehouse) — responsible for storing and searching vectors:

The two combine freely. BGE-M3 + ChromaDB works; GTE-Qwen2 + Qdrant works too. The only constraint is that the vector dimensions must align — if the model outputs 1024 dimensions, configure the database to receive 1024 dimensions. Just as a lens mount must match the camera body.

The model stores nothing; the database understands no semantics. The model handles "understanding," the database handles "remembering and finding."

VI. Does This Process Require Writing Code?

Yes. But not much.

There is no software that lets you "pick a folder → click Start → wait → done." You need to write a Python script, about 50 lines, to chain the whole process together.

The core code skeleton:

# 安装: pip install chromadb sentence-transformers

import os, chromadb
from sentence_transformers import SentenceTransformer

# 加载翻译机（首次运行自动下载模型，约 2GB）
model = SentenceTransformer("BAAI/bge-m3")

# 初始化仓库（本地文件，无需服务器）
db = chromadb.PersistentClient(path="./county-vectors")
collection = db.get_or_create_collection("县志")

# 遍历所有文本文件
for filename in os.listdir("./county-data"):
    text = open(f"./county-data/{filename}").read()

    # 切片：每 500 字一段
    chunks = [text[i:i+500] for i in range(0, len(text), 450)]

    # 翻译：文字 → 向量
    vectors = model.encode(chunks)

    # 入库：向量 + 原文一起存
    for j, chunk in enumerate(chunks):
        collection.add(
            ids=[f"{filename}_{j}"],
            documents=[chunk],
            embeddings=[vectors[j].tolist()]
        )

    print(f"完成: {filename}, {len(chunks)} 个片段已入库")

This code does three things: load the model, chunk the text, and store it in the database. On an ordinary laptop, processing a full county gazetteer (500,000 characters) takes about 5–10 seconds.

VII. Do You Need a GPU Cluster?

No.

This is the biggest misconception people have about AI projects. The entire RAG pipeline spans three levels, and their compute requirements differ by a world.

What you're doing — Inference:

• Using someone else's trained model to generate embeddings
• A Mac laptop is enough, zero cost
• 1 million text chunks, done in a few hours

What the big companies are doing — Training:

When you do RAG, it's inference from start to finish, not training.

It's like driving a car without needing to build the engine. BAAI spent thousands of GPUs training the BGE-M3 model; you pip install it and use it directly.

VIII. How Do You Know Whether the Results Are Accurate?

This is where the RAG system really gets into deep water. Vector retrieval isn't black magic — there are systematic methods for evaluating it.

Layer 1: Manual Spot-Checking

Prepare 50 questions you write yourself (you understand the gazetteers best), each one for which you know exactly which passage the answer comes from. Run retrieval and see whether the correct answer appears among the 5 returned passages.

Layer 2: Comparative Experiments

Fix the test questions and run them under different configurations:

No theoretical derivation needed — just run the data and let it speak.

Layer 3: Four Tuning Knobs

If accuracy isn't good enough, try these in order:

Knob 1 · Chunking strategy (zero cost, biggest impact)

Cut by natural paragraphs, don't force a hard character count. Keep the chapter title as a prefix, e.g. "【Volume 20 · Calamities】In the summer of the third year of Qianlong…". Overlap adjacent chunks by 10–20% to prevent answers from being cut in half.

Knob 2 · Retrieval strategy (zero cost)

Pure vector search understands synonyms but may miss exact terms; pure keyword search matches exactly but doesn't understand meaning. Hybrid search — a weighted merge of the two — is the best practice.

Knob 3 · Swap the Embedding model (requires re-running vectorization)

If BGE-M3 isn't good enough, try GTE-Qwen2, and vice versa.

Knob 4 · Fine-tune the Embedding model (highest cost, best result)

Prepare a few hundred pairs of training data to teach the model that "圩" (a polder) and "堤坝" (an embankment) mean similar things, so their vectors should sit close together.

Fundamentally, this is like all engineering: quantify the metric → run comparative experiments → tune → quantify again. It's not black magic, it's science.

IX. The Cost of Deploying to Production

Take a mid-sized county (100,000 pages of scans) as an example:

One-time costs:

Monthly operating costs:

The vector store is an asset; the large model is a faucet. You can switch the faucet's brand anytime (Claude, DeepSeek, Doubao, Tongyi Qianwen), but you can't switch the reservoir. Whoever vectorizes a county's data first has claimed the plot.

There are 2,800+ counties nationwide, and 99% have no such system. The data itself is the moat.

X. Summary

The essence of RAG: it doesn't make the AI smarter — it assigns the AI a librarian.

The essence of the vector database: it's not a database that stores words, but a database that stores "meaning."

The essence of the Embedding model: it doesn't translate languages — it translates semantics into mathematics.

The essence of the whole system: first build the library index (one-time), then act as an intelligent librarian (ongoing service).

You don't need to train an AI. You don't need a GPU cluster. You don't need a budget of millions.

What you need is: a laptop, 50 lines of Python, and a county worth digitizing.