Vectors, Retrieval, and RAG: How a Search System That "Speaks Only From the Source" Gets Built
1. Literal vs. Meaning: Where Old-School Search Loses
Traditional on-site search, and the search boxes on government portals, are overwhelmingly keyword retrieval: they break your query into words and hunt through the documents for entries that "contain the same characters." Its fatal flaw is this — the way users talk and the way documents are written almost never match.
You ask, "how much do I owe if I break something," but the legal text says "damages" and "tort liability" — not a single character lines up. Keyword retrieval therefore misses the two passages it should most have surfaced, and instead, because your query contained the word "compensation," it hits a User Guide to the Compensation Calculator — an answer to a question you never asked.
Semantic retrieval asks a different question. It doesn't ask "are the same characters present," it asks "do these two passages mean something similar." Paraphrases, cross-language matches, archaic phrasings — as long as the meaning aligns, it can pull them back. This isn't mysticism; there's hard math underneath, and its core is called embedding.
2. Embedding: Placing Every Passage on a "Map of Meaning"
What an embedding model does is compress any passage of text into a fixed-length string of numbers — say, 1,024 of them. That string isn't random: it's the passage's coordinates in a 1,024-dimensional space.
Through massive training, the model has learned one thing: text that means similar things sits close together; unrelated text sits far apart. So "damages" and "tort liability" cluster into the same spot on the map, while "braised pork" and "cooking heat" land in some distant region.
At this point, "search" has been translated into a geometry problem: turn the user's query into a coordinate point too, then measure which document coordinates it lands closest to. Whatever is near is relevant. Whether the literal characters match no longer matters at all.
A good embedding model is also bilingual (like bge-m3): a Chinese query and an English document can land in the same region — which means a Chinese question can recall English material, something keyword retrieval can never do.
3. RAG: Making the Large Model "Speak From the Source"
Retrieval alone isn't enough. Users don't want a pile of document links; they want a direct answer. But you also can't let a large model improvise — it will fabricate facts with a straight face (hallucination). RAG (Retrieval-Augmented Generation) is the design that welds "retrieval" and "large model" together: retrieve the real material first, then force the model to answer strictly from that material.
The whole pipeline has two segments: one offline, building the index; one online, answering questions.
The offline segment (run only when content is updated): raw material → clean and chunk → embed each chunk into a vector → store it, along with its "source," in the vector database. This segment is heavy lifting, but you don't do it every day.
The online segment (run on every query): embed the query → recall the few most relevant passages from the vector database → stuff those original passages into the prompt as the "basis" → the large model streams an answer strictly from that basis → attach the source links.
Hidden here is the key design behind zero hallucination: the source links come directly from the "retrieval results" and never pass through the large model. The model is only responsible for making the language sound human; it isn't responsible for reporting sources — so it can't fabricate a link pointing to an article that doesn't exist. That division of labor is what gives an AI assistant the nerve to be hung on a professional stage: in industries where a single wrong sentence carries liability (law firms, medicine, government affairs), the whole thing rests on "every sentence can be clicked back to the source."
In one line: retrieval is responsible for "finding the right material," the large model for "speaking like a person" — split those two jobs, let each do its own, and you get an answer that is both accurate and human.
4. What a Vector Database Is Actually Computing: Measure Angles, Rank, Set a Bar
The "vector database recall" step sounds mysterious, but unpacked it's just three moves.
Similarity uses cosine similarity — it looks only at whether two vectors point in a similar direction, ignoring how long they are. Perfectly aligned is 1; wholly unrelated is 0. Why use angle instead of length? Because that way a long document and a short one compete fairly — a piece doesn't get an edge just because it has more words.
Once you have the similarity of every candidate, there are two more gates:
| Gate | What it does | Why |
|---|---|---|
| topK = 6 | Rank by similarity high-to-low, take at most the top 6 passages | Control how much you feed the model: too much drowns the key points in noise, and it costs more |
| Threshold 0.35 | Discard anything below this similarity | Better to answer "it's not in the material" than to pad with irrelevant passages to fill space |
These two numbers (6 and 0.35) are adjustable dials. Raise the threshold → more conservative answers but risk of missing; lower it → broader recall but risk of dragging in noise. This is the feel every RAG system has to grind out against its own corpus.
5. System Architecture: One Core, Two "Hands"
Principles covered, now the engineering. For a system of this kind actually running in production, the most valuable architectural decision is: make the core logic completely unaware of where it's running.
The technique is dependency injection + interface abstraction. The core (core) is a set of pure functions — page rendering, RAG orchestration, form handling — that call no specific cloud service directly and program only against three abstract interfaces: the AI model, the vector database, the archive. Who implements those three interfaces is decided by the "hand" injected at deploy time.
So the exact same code can rest on two entirely different foundations:
| Hand A · Cloud-Managed (Cloudflare) | Hand B · Self-Hosted (Node / VPS) | |
|---|---|---|
| AI model | Workers AI (platform built-in) | Any OpenAI-compatible API (DeepSeek / self-hosted vLLM) |
| Vector DB | Vectorize (managed service) | sqlite-vec (local file) |
| Archive | D1 (managed SQL) | SQLite file |
| Ops | Zero ops, global edge, pay-as-you-go | Manage a machine yourself |
| Identity surface | All three services hang off the same cloud account, centralized | No cloud-account binding, data on your own machine, can move to a non-KYC host |
| Switching | deploy --target=cf | deploy --target=node |
To serve a new client site, you change only one tenant config table (domain / brand / knowledge-base source / system prompt); to swap foundations, you swap one adapter. The core core doesn't change by a single character. That is the entire meaning of "portable" — your system is no longer any one cloud's hostage.
A correction from actual testing: an internal project document had long said "the Node adapter can't run locally." On 2026-07-13 I actually got it running and verified:
TENANT=demo node server.jsstarts up with zero installed dependencies — homepage / retrieval / search / health-check all return 200. That old note was stale and has been corrected. End-to-end Q&A under a realAI_API_KEY, sqlite-vec compiled on an actual VPS, and a formal deployment with TLS / process supervision — those three remain unverified; I mark them honestly and won't dress them up as "done."
6. Graceful Degradation: Small-Scale Simply Doesn't Need Heavy Gear
This is the piece of engineering restraint I admire most in the whole design, and it's the crux of the cost-effectiveness case.
A "vector database" sounds like a heavy thing you must deploy specially (the Pinecone, Weaviate tier). But the self-hosted hand does something clever: at startup it first tries to install a professional vector engine (sqlite-vec); if that fails, it automatically degrades — reads all vectors into memory and computes cosine one by one on every query.
The key is the scale inflection point:
- Tens to hundreds of vectors: brute-force in memory, as fast as a professional vector DB, no difference. The demo site I tested today takes exactly this path — zero dependencies, runs just fine.
- Tens of thousands and up: brute force starts to slow linearly, and only then do you genuinely need an ANN (approximate nearest neighbor) index to skip most candidates.
Most real-world scenarios fall in the left zone. A county gazetteer archive, the historical back-catalog of a local paper, all the past issues of a magazine, a consulting firm's case library — usually a few hundred to a few thousand pieces. Hauling in a distributed vector cluster ahead of time, for a scale you'll never reach, is textbook over-engineering. This "use the good engine if it installs, run fine if it doesn't" degradation design hands the judgment of "how heavy does your gear actually need to be right now" back to the real data volume — instead of to anxiety.
7. After You Hit the Wall: How to Patch the Performance Wall and the Precision Wall
Section 6 argued that small-scale needs no heavy gear. But what if the volume really does grow (to the right of that inflection point), or the content is precise by nature and a wrong answer has consequences? There are two walls here, utterly different in nature — the performance wall is a pure engineering problem, with a pile of mature playbooks, and this architecture comes with its own antidote; the precision wall is an inherent property of "generation + approximation," which you can only push hard against, never cure.
The Performance Wall: What to Do When Vectors Grow to Hundreds of Thousands, Millions
Ranked from most to least cost-effective, these are all standard, well-worn industry moves:
| Approach | What it does | Cost / in this architecture |
|---|---|---|
| ① Partition / tenant pre-filter | Search only the relevant partition (tenant / category / model year), not the whole database | Near-zero cost; this architecture is tenant-partitioned by nature — effectively free — the first thing you should use |
| ② Switch to an ANN index (HNSW / IVF) | Replace "compute one by one" with "jump around to find," so even a million vectors answer in under 10ms | ~95–99% recall; Hand A (CF Vectorize) is managed ANN — core changes not one character and graduates straight away |
| ③ Quantization (PQ / binarization) | Slim the vectors: 1,024 dims of float32 = 4KB, binarized down to 128B (32×), distances compute lightning-fast | Slight precision drop, patched back with "coarse filter → precise re-rank" |
| ④ Two-stage retrieval | Cheap coarse filter (quantized / low-dim / ANN) picks the top few hundred, then full-precision re-rank | One layer more complexity |
| ⑤ Dimensionality reduction / Matryoshka | A model supporting nested representations can truncate 1,024 dims to 256 for the first round | Slight recall drop, depends on the embedding model |
| ⑥ Query / semantic caching | Cache recall results or whole answers for high-frequency questions | Saves money and cuts latency |
One honest reminder: if the self-hosted path uses sqlite-vec, it runs an exact brute-force scan (as far as I know, still no ANN index in 2026 — check the latest version), with a scale ceiling around the million level within acceptable latency; to go higher, you must switch the Node side to pgvector (HNSW) / Qdrant / Milvus. In other words, the standard fix for the performance wall in this system is one sentence: when a single machine can't take it, cut over to CF Hand A — which is precisely the dividend of the dual-runtime design.
The Precision Wall: Don't Make Vectors Do Precise Work
The key insight — the precision wall is not patched by making the vectors better — it's "give the precise work to precise means, and the semantic work to semantics":
| Fix | What it does | What it cures |
|---|---|---|
| ① Hybrid retrieval (vector + BM25, RRF fusion) | The semantic path catches "meaning"; the keyword path catches character-exact terms like "M8×1.25" or "Article 47" | Exact hits on model numbers / serials / statutes / drug names |
| ② Structured routing (tables don't go into the vector DB) | At build time, pull spec tables / parameters into SQL; a query like "how much torque" reads the table for the exact value | Numbers / parameters / specs |
| ③ Cross-encoder re-ranking (bge-reranker class) | Vector coarse-recalls the top-20, then a re-ranker reads query+passage together and scores precisely | Suppresses "recalled a near-synonym but wrong passage" |
| ④ Extractive answering + forced citation | Force the model to quote that exact sentence / number verbatim and cite the source; any number in the answer must appear character-for-character in a recalled chunk, or it refuses | Plugs "retrieved correctly, then rewrote the number wrong during generation" |
| ⑤ Metadata filtering | Tag each chunk with model / year / chapter, filter before retrieval | Prevents "asking about the 2021 model and getting 2019 specs" |
| ⑥ Structure-aware chunking | Don't cut a spec table in half — split by heading / table boundary | Keeps table context intact |
| ⑦ Dedicated diagram handling | Blueprints / exploded views → OCR + captions, or return the original image pointing at the callout number, don't forcibly synthesize | Honestly handles the "you must see the diagram to answer" part |
But here's a design boundary that can't be erased: for genuinely safety-critical precise queries (torque, drug dosage, statutes), even the full suite above cannot guarantee that "a generative model that rewrites text will never err." The ultimate right answer is not to let RAG generate that number, but to let it retrieve and present the source verbatim — downgrade "speak like a person" to "find precisely and highlight," and hand the final judgment back to a human. This is a design boundary, not a bug awaiting a fix.
8. Doing the Full Accounting: Where the Cost-Effectiveness of This Approach Actually Shows
Translate the architecture into money and labor, viewed across four dimensions.
| Dimension | Traditional institutional approach (Java + Oracle + Elasticsearch cluster) | This RAG-on-edge approach |
|---|---|---|
| Upfront development | Procure and integrate multiple systems, measured in person-months | One core + one config table; a new site is "filling in a form," not "rewriting" |
| Ops | Dedicated staff to run the database / search cluster / servers | Cloud-managed path is near-zero ops; self-hosted is just one machine, one process |
| Scaling cost | Add a client site = another whole deployment | Add a client site = add a tenant config (multi-tenant reuse of the same core) |
| Scale fit | Heavy architecture from day one, wildly wasteful for small projects | The degradation path makes small-scale zero extra cost, and you bring heavy gear only when scale arrives |
| Vendor lock-in | Bound to a database / middleware vendor | Dual-runtime: migratable between cloud account and self-built machine, not hostage to a single cloud |
What's genuinely scarce is none of these individual items — chatbots have slicker SaaS (Chatbase), static sites have a more mature ecosystem (Astro), legal corpora have deeper vertical players (Harvey AI). What's scarce is fusing four things — zero-dependency + RAG-native + bilingual + build-time cleaning — at near-zero marginal cost. This is the middle ground the giants can't be bothered with (too small to be worth it) and the small tools can't be bothered with (each item requires re-integrating several SaaS).
But one honest flip side must be said: the scale ceiling of this business is set not by technology but by the business model. "Build and operate the site for the client, no self-service editing backend" means every client requires a one-time human investment — the number of clients you can serve grows linearly, not on the marginal-cost-toward-zero exponential curve of SaaS. However elegant the tech, it can't change that.
Coda: Which Scenarios Fit, and Which Don't
A system's boundaries are worth articulating more than its capabilities. And to draw that boundary, the first thing to do is debunk a repeatedly misreported number.
First, correct an illusion: "a few thousand articles" is nowhere near the ceiling
Common lore says systems of this kind "top out at a few thousand articles at most." That sentence treats article count as the yardstick, but article count isn't the unit the system sees at all. The system sees only one thing: the number of vectors.
Before it enters the database, an article is cut into several chunks, each around 300–600 characters, each becoming one vector. So the real yardstick is this conversion:
Number of vectors ≈ total character count ÷ ~500 characters
Measure with this ruler a few examples often used to probe the boundary (character counts are order-of-magnitude estimates):
| Corpus | Order-of-magnitude characters | Approx. vector count | Which tier it falls in |
|---|---|---|---|
| The entire back-catalog of a blog / WeChat public account | 300k–1M chars | Hundreds–2k | In-memory brute force, instant |
| Wang Xiaobo's complete works | ~2M chars | ~4–5k | In-memory brute force, instant |
| A research scholar's complete papers (two or three hundred) | 1.5M–3M chars | Thousands | In-memory brute force, instant |
| A county gazetteer / all past issues of a magazine | 1M–3M chars | Thousands | In-memory brute force, instant |
| The full technical manual set for one car | ~1.5M chars | ~3–5k | No scale pressure at all (but see "the precision wall" below) |
| A small/mid law or consulting firm's cases + template library | Hundreds–thousands of pieces | Thousands–tens of thousands | In-memory or sqlite-vec, one machine is enough |
| A large think tank's decades of complete reports | Tens of millions of chars, still growing | Hundreds of thousands and rising | Approaching ANN, plus a "recency wall" |
| A large e-commerce / news site | Hundreds of millions of chars + changing daily | Millions | Professional vector infrastructure required |
See the pattern: Wang Xiaobo's complete works, a scholar's complete papers, a whole car manual — these "big"-sounding things all cram into that leftmost tier, a few thousand vectors, where computing cosine one by one in memory is a matter of milliseconds. The "thousand-article ceiling" is a psychological number, not a technical one.
And the technical ceiling itself is underestimated. Section 6's "slows down starting at tens of thousands" is a conservative statement — that treats the degradation path as a naive interpreted loop. Take sqlite-vec (a C implementation) or vectorized in-memory computation, and a single machine handles hundreds of thousands of vectors (roughly tens of millions of characters, about two hundred books of plain text) still within usable latency. What genuinely forces you onto distributed ANN is millions of vectors and up — and that's already encyclopedia, whole-site news-archive scale, which a small or mid-sized application will never reach in its lifetime.
Conclusion in one line: the scale wall barely exists for small and mid-sized corpora. What you hit first is always the other three walls.
The three walls you actually hit first
If scale can't reach the ceiling, what turns you away first? Ranked by "which you hit first":
| Wall | When you hit it | Why RAG struggles here |
|---|---|---|
| ① The recency wall | Content changes daily and must be ranked by "latest" | Offline index-building is heavy lifting; frequent rebuilds drag it down. Semantic similarity doesn't recognize "which is newer" — scenarios where "freshness > similarity" (news, think tanks, intelligence) don't take to it |
| ② The precision wall | The answer is an exact number / procedure / table, and getting it wrong has consequences | Semantic recall is "closest in meaning," not "character-for-character." Torque specs, drug dosages, tax brackets, statute numbers — an approximate recall off by one tier is an accident. A car manual is completely fine on scale, yet crashes into this wall: it needs the exact "how many N·m to tighten the bolt" and step-by-step diagrams, not "a passage that's roughly close in gist" |
| ③ The aggregation wall | The question requires statistics / computation across the whole database | RAG only "pulls back the few most relevant passages," it doesn't "read everything and then tally it up." Questions like "what's the average payout across this batch of cases" or "the most frequent policy term across ten years of reports" — these aggregation-type questions can't be answered from a few recalled passages |
Behind all three walls is one sentence: RAG is good at "asking a qualitative answer in plain language out of a fixed pile of material," and bad at "seeking the latest, the exact, or the summed-up." (How to patch the performance wall and the precision wall was covered in Section 7; here we only discuss "whether you hit them.")
A side-by-side: which small and mid-sized scenarios are a natural fit
Combine the rulers above (small volume + stable content + qualitative Q&A + non-fatal if wrong), and the following small/mid-sized applications are almost tailor-made:
| Scenario | Volume | The fit | What to watch |
|---|---|---|---|
| Q&A over an author's / scholar's complete works (Wang Xiaobo, a professor's collected papers) | A few thousand vectors | Content never changes again; semantic retrieval is exactly the strength for literature/academia | Almost no weak spot — the "purest" fit |
| County gazetteer / local paper / old-magazine historical archive | Thousands of vectors | Append-only, changes very slowly, bilingual is a bonus | Few external constraints, named in the body |
| Small/mid law-firm contract templates / consulting case library | Thousands–tens of thousands | The hard requirement "answer from our own material, no making things up" | Liability-sensitive, every sentence must click back to source |
| Hospital department SOPs / clinical-guideline Q&A | Hundreds–thousands | Authoritative content, slow updates, hard Q&A demand | Precision-sensitive, answer only "what the guideline says," no diagnosis |
| Government service guides / a citizen-hotline knowledge base | Thousands | Stable, bilingual, must "speak by the book" | Rebuild needed when policy changes — controllable, low-frequency |
| Small/mid enterprise after-sales / product knowledge base | Hundreds–thousands | Support answers from the docs, cuts labor | Content changes on product refresh — mid-frequency, still manageable |
| Internal company wiki / employee handbook / HR policy | Hundreds–thousands | Employee self-service Q&A, saves HR from repeat questions | Policy changes, needs a lightweight rebuild flow |
| All the lecture notes for a course / a textbook set | Thousands | Students "ask the textbook," clear boundaries | Rebuild on version iteration |
| Full transcript library of a podcast / interviews | Grows with the years but slowly | Fuzzy recall of "I remember an episode discussed X" | Growth is continuous but low-frequency, watch the vector count long-term |
| Classical / religious text libraries | Thousands–tens of thousands | The text is eternal and unchanging, semantic retrieval is a natural fit | Almost no weak spot |
And the following are not a scale problem, but a "wrong tool" problem — no matter how elegant the architecture, it spins its wheels:
| Scenario | Which wall it hits | What to use instead |
|---|---|---|
| Exact queries against one car's / one machine's technical manual | Precision wall + structured | Structured retrieval + exact field queries (tables/serials/diagrams shouldn't be stuffed into semantic vectors in the first place) |
| News site / intelligence daily / public-opinion monitoring | Recency wall | Time-series database + freshness ranking |
| Large think tank's live report library | Recency + aggregation + scale | Professional vector infra + time-series + aggregation-layer hybrid |
| E-commerce / booking / shopping cart | Not a Q&A system at all | Transaction system (named in the body) |
| "Ten-year data statistics / trend" analysis | Aggregation wall | Data warehouse + BI, not RAG |
| Client wants to self-publish and edit articles daily | No editing backend | A product with a CMS (the body already names this as the business-model ceiling) |
To close in one line: Semantic retrieval + RAG is no panacea — it makes "asking an accurate answer in plain language out of a fixed pile of material" accurate, cheap, and free of cloud lock-in, all at once. The ceiling isn't "a few thousand articles" — small and mid-sized corpora barely hit the scale wall; what turns you away first is always these three questions: does the content change, is the answer precise, do you need to tally it up. Answer "no" to all three, and its cost-effectiveness crushes the old-school approach; answer "yes" to even one, and however elegant the architecture, it spins its wheels.
The technical judgments in this article are based on hands-on testing and investigation of the real production code behind the Praxis system that powers tryway.cc, conducted on 2026-07-13. Everything marked "tested" has been reproduced locally; everything marked "unverified" is labeled honestly, with no speculation passed off as conclusion.