We Need to Talk About “Chatting With Your Data”
I remember the first time I saw a RAG (Retrieval-Augmented Generation) demo back in late 2023. It was magic. You dropped a PDF into a Python script, asked a question, and it answered. We all lost our minds. We thought, “This is it. The end of Ctrl+F. The end of digging through SharePoint folders from hell.”
Fast forward to January 2026. The magic has worn off, and we’re left with the hangover.
If you’ve tried to deploy a generative search system in a real enterprise environment—not a demo, but a place with messy data, strict permissions, and users who ask vague questions—you know exactly what I’m talking about. The “chat with your docs” promise is easy to prototype but excruciating to productionize.
I’ve spent the last year rebuilding search architectures for companies that realized their vector databases were hallucinating answers or, worse, confidently summarizing documents the user wasn’t supposed to see. Here is what actually works right now, and why simply “embedding everything” is a recipe for disaster.
The “Part Number” Problem (Why Semantic Search Fails)
Here is the biggest lie we told ourselves: “Semantic search understands intent, so it’s better than keyword search.”
Sort of. Sometimes.
If I ask, “How do I reset the pressure valve?”, semantic search is brilliant. It finds documents about “relieving pressure,” “valve maintenance,” and “system reset” even if I don’t use those exact words. That’s the dream.
But try asking a vector-only system for “Part #884-B2”.
It falls apart. Semantic models (even the fancy new embedding models we got last year) try to find the “meaning” of “884-B2”. There isn’t one. It’s a string of characters. The model might return results for “Part #884-B3” because they are mathematically close in vector space, or it might hallucinate a connection to a date.
In 2026, the only viable architecture is Hybrid Search. You cannot ditch BM25 (the standard keyword search algorithm used by Elasticsearch/Solr for decades). You just can’t.
My current stack usually looks like this:
- Layer 1: A sparse retriever (BM25) for exact matches (names, IDs, error codes).
- Layer 2: A dense retriever (Vectors) for conceptual queries.
- Layer 3: Reciprocal Rank Fusion (RRF) to mash the two lists together.
If you aren’t doing this, your engineers are probably complaining that they can’t find specific error logs. Trust me on this one.
The Parsing Nightmare: PDFs Are Where Data Goes to Die
Most people think the hard part of AI search is the LLM. It’s not. The LLM is the easy part. You pay an API or host a Llama model, and it talks.
The hard part is getting data out of your terrible corporate documents.
I was working on a project last month involving technical manuals for heavy machinery. These weren’t clean HTML pages. They were 500-page PDFs with multi-column layouts, tables that spanned three pages, and diagrams with text embedded in images.
When you feed that raw text into an embedding model, you get garbage. If a sentence starts on the bottom of column A and finishes on the top of column B, a standard parser reads straight across, mixing the two columns into a nonsensical word salad. The AI then indexes that salad.
Garbage in, hallucination out.
We’re finally seeing better tooling for this—vision-language models that “look” at the page layout before extracting text are becoming standard in 2026—but it’s still slow. I often have to tell clients: “We can have instant indexing, or we can have accurate answers. Pick one.”
If you care about accuracy, you have to preserve the document structure. You need to chunk by logical sections (headers), not just arbitrary character counts. If you split a table in half, the LLM loses the context of the column headers. Suddenly, “500 PSI” looks like a safe operating limit when it was actually the “Burst Pressure” row.
That’s a dangerous hallucination.
Security: The “Intern Can See The CEO’s Salary” Issue
This is the one that gets CTOs fired.
In a traditional search engine, if I search for “Executive Comp 2025” and I don’t have permission to see that file, the search engine returns zero results. Simple.
In a naive RAG system, the vector database might find the document, feed it into the context window of the LLM, and the LLM generates an answer. Even if you don’t show the source document link, the information has leaked.
I’ve seen so many sloppy implementations where the permissions check happens after the generation. Too late. The AI already spouted the numbers.
You have to implement ACL (Access Control List) filtering at the vector level. Every chunk of text in your database needs metadata tags for who can see it (e.g., group_id: ["engineering", "admin"]).
When a user queries the system:
- Identify the user.
- Fetch their groups.
- Filter the vector search to only include chunks matching those groups.
- Then perform the search.
- Then send to the LLM.
This sounds obvious, but it kills latency. Filtering millions of vectors in real-time is computationally heavy. We’re using heavy-duty hardware to keep this under 200ms, but it’s the price you pay for not leaking trade secrets.
The “Answer Engine” vs. The “Search Engine”
We are seeing a shift in user behavior. Two years ago, users treated these tools like Google—they expected a list of links. Now, they treat them like a colleague. They expect an answer.
But here is the catch: they don’t trust the answer.
And they shouldn’t.
I built a system recently for a legal team. They didn’t want the AI to draft the brief; they wanted it to find the precedent. The most valuable feature wasn’t the generated text—it was the citation.
If your AI says, “According to policy, you can expense up to $50 for dinner,” it better have a clickable little [1] next to it that opens the exact page of the Employee Handbook. If it doesn’t, it’s a toy. In 2026, “citation-backed generation” is the minimum viable product. We are using techniques to force the model to quote its sources directly, sometimes even constraining the decoding process so it can’t generate a sentence without referencing a retrieved chunk.
It makes the AI sound a bit more robotic, sure. But in a business context? I’ll take robotic and accurate over conversational and wrong any day.
Infrastructure: The Return of On-Prem
Cloud is great, until it isn’t.
For generic data, hitting a public API is fine. But for the really sensitive stuff—defense contractors, healthcare, finance—we are seeing a massive swing back to self-hosted infrastructure. Nobody wants their proprietary engineering schematics sitting in a model provider’s training log.
The hardware requirements have stabilized, thankfully. In 2024, running a decent 70B parameter model locally was a pain. Now, with the optimization techniques we have (quantization is pretty much lossless at this point), you can run a highly capable “reasoning” model on a single robust server rack.
This allows for air-gapped deployments. I’m seeing this heavily in regulated industries. They want the “Google experience” but disconnected from the actual internet. It’s essentially a private cloud box that ingests their shared drives, indexes them, and serves a chat interface, all without a single packet leaving the building.
It’s expensive to set up, but the ROI on productivity is undeniable. When an engineer can ask, “What was the torque setting we used on the 2019 project?” and get an answer in seconds instead of digging through archives for three days, the server pays for itself.
Stop Chasing the Hype
If you’re building this today, stop worrying about which model is top of the leaderboard this week. The difference between the #1 and #5 model is negligible for document search tasks.
Spend your time on the boring stuff. Fix your data ingestion pipelines. Tag your documents with proper metadata. figure out your permissioning strategy. Write evaluation tests to catch regressions when you update the system.
The technology is here. It works. But it’s not magic. It’s just plumbing.