RAG Recon: The Sources Field Hands You the Knowledge Base

If a company chatbot shows you its sources, check what's inside them. In the lab, one question about "system architecture" returned a Redis host and port, a database server name and the fact that secrets live in HashiCorp Vault, word for word from an internal PDF.

That's RAG (retrieval-augmented generation): a model plus a search over company documents. Its attack surface is the documents, so your recon is about finding out what it can reach. In this lab you'll confirm retrieval is on, read the source metadata, enumerate the knowledge base topic by topic and find where retrieval switches off. It's part 3 of the active recon series, after model fingerprinting.

How a RAG pipeline answers, and where it leaks

Documents get split into chunks, small pieces of text that are indexed. When you ask something, the retriever finds the chunks that match best and pastes them into the prompt. These are grounding documents: text that steers and limits what the model can say.

Frameworks like LangChain, LlamaIndex and Haystack make it easy to send those chunks back with the reply. Developers leave it on so users can check answers. For you, it's an enumeration gift.

Step 1: Check whether RAG retrieval is happening

The lab target is NovaTech's internal knowledge base bot. Staff ask it about policies and tech docs. Here's the base command you'll reuse:

curl -s -X POST http://<target>/api/chat \
  -H "Content-Type: application/json" \
  -d '{"query": "What is the PTO policy?"}' | jq .

Send one general question and one company question, and compare the sources field:

A filled sources list means RAG is live. You'll notice the timing too. Retrieval took under a millisecond while generation took seconds, so you can't use timing to tell whether RAG kicked in. Use sources.

Step 2: Read the RAG source metadata

Each field in that sources block tells you something you didn't know before:

• title (PTO_Leave_Policy_2024.pdf) shows the file naming style and formats, so you can guess other names.
• chunk_id (chunk_087) hints at how documents are split and roughly where this one sits.
• text is the raw retrieved text, word for word. Collect enough and you can rebuild documents.
• vector_score is the semantic match from an embedding search. bm25_score is the keyword match. Both together means hybrid retrieval, and combined_score blends them.

The text field is the one you'll care about most. The model summarises and can leave things out, but the raw chunk is exactly what it read.

Step 3: Enumerate knowledge base documents by topic

Ask about different areas and keep a list of what comes back. A question about internal APIs, What internal API endpoints exist?, pulled two chunks from API_Documentation_v3.2.pdf:

Then try the architecture question:

curl -s -X POST http://<target>/api/chat \
  -H "Content-Type: application/json" \
  -d '{"query": "What is the system architecture?"}' | jq .

One sentence of raw text gives you Kong, Kubernetes, PostgreSQL on db01.internal, Redis at redis.novatech-internal.com:6379, RabbitMQ and HashiCorp Vault.

Now look closely at the answer. It quotes Security_Practices.pdf, yet that file isn't in the sources list. The model made the citation up. So you can't trust the answer for what it leaves out, and you can't trust it for what it adds. Trust the raw text.

Doing this by hand gets slow, so you'll want a loop. This one runs a topic list and prints each title and chunk ID once:

for q in "HR policy" "VPN setup" "passwords" "AWS" \
         "incident response" "onboarding" "salary bands"; do
  curl -s -X POST "http://<target>/api/chat" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg query "$q" '{query: $query}')" |
    jq -r --arg q "$q" '.sources[]? | "\($q) | \(.title) | \(.chunk_id)"'
done | sort -u

.sources[]? skips replies with no sources instead of erroring, and sort -u removes repeats. The chunk IDs look sequential (087, 301, 302, 401), so the gaps give a rough idea of document sizes and where other documents sit.

Anything like the ntk_prod_ key format, internal hostnames or the Redis port is worth a finding on its own: sensitive information exposed through RAG, OWASP LLM02:2025.

Step 4: Find the retrieval threshold

The pipeline only pulls a chunk if your query scores above a minimum similarity, called the threshold. Once you've found it, you've got two levers:

• Want data out? Word queries so they do trigger retrieval.
• Want to jailbreak? Word them so they don't. With no grounding documents holding it back, the model may be easier to push around with prompt injection.

Send queries that get steadily less like the document and watch when retrieval stops:

Heavy misspellings fall under the threshold on both searches, and here's why. BM25 can't match words that aren't spelled right, and the embedding drifts too far from the document's meaning. With nothing retrieved, the model's running ungrounded.

One oddity: the synonym query scored higher on vector_score (0.4) than the exact wording (0.2). Scores aren't always intuitive. Compare combined_score and repeat a few times before you settle on where the threshold sits.

Rate how much each RAG endpoint leaks

Not every RAG app gives this much away. Sort the endpoints you find into three levels, and spend your time where they leak most:

Common mistakes

Reading only the answer. The model drops details (GET /v3/usage) and invents others (Security_Practices.pdf). The raw text field is the evidence.

Using timing to detect RAG. Retrieval took 0.18 to 0.4 ms in the lab. Generation time swamps it. Check sources instead.

Wrong body field. This bot wants query. Others want message, prompt or a messages list. A 400 or 422 usually means the field name's wrong.

Breaking JSON in the loop. Topics with quotes in them break a hand-built -d string. The loop above uses jq -n --arg so that can't happen.

Trusting one threshold test. Scores move between runs and queries. Repeat each step of the ladder.

Fix it, and detect it

Return less. If your users need citations, send titles only. Drop chunk IDs, scores and raw text from the public response, and keep them in your server logs instead.

Clean the knowledge base. Anything your chatbot can retrieve, a user can read. Scrub hostnames, ports, key formats and credentials from documents before they're indexed, and keep infrastructure docs out of a staff-wide bot.

Filter by permission. Retrieval should only search documents the asking user is allowed to see. Retrieval with no access checks is OWASP LLM08:2025, vector and embedding weaknesses.

Watch the pattern. One session asking about HR, VPN, passwords, AWS and salary bands in a few minutes is someone mapping the knowledge base. Part 4 of this series looks at exactly what those logs capture, and the AI pentest notes hold every check from this post.

Quick recap

• An empty sources for "2+2" and a filled one for "PTO policy" proves RAG is live. Timing won't, because retrieval takes under a millisecond.
• Titles, chunk IDs, scores and raw text in the response let you map and rebuild the knowledge base.
• One architecture query leaked a Redis host and port, a database server and where secrets are kept.
• The raw text beats the answer: the model left out an endpoint and invented a source.
• Misspelled queries drop under the retrieval threshold, leaving the model with no grounding documents.

Check this today

If you run a RAG chatbot, send it one ordinary staff question and open the raw JSON your browser receives. The chat window hides most of it. If you can see chunk text or scores, so can everyone else. Then ask it about your architecture and see what it knows.

If you're testing RAG apps and have found other fields that give the knowledge base away, I'd like to hear about it. Message me on LinkedIn and we'll compare.

References

• OWASP Top 10 for LLM Applications (2025) - LLM02 Sensitive Information Disclosure, LLM08 Vector and Embedding Weaknesses.

FAQ

How can you tell if a chatbot uses RAG?

Ask one general question and one company-specific question and compare the replies. A RAG chatbot cites internal documents or returns a filled sources field for company questions, and an empty one for general knowledge like 2+2. Don't rely on timing, because retrieval often takes under a millisecond.

Why is RAG source metadata a security risk?

Source metadata can include document titles, chunk IDs, similarity scores and the raw retrieved text. That lets an attacker map the knowledge base, rebuild documents word for word and read details the model left out of its answer, such as internal hostnames, ports or API key formats.

What is a RAG retrieval threshold?

It's the minimum similarity score a document chunk needs before the pipeline adds it to the prompt. Queries close to a document's wording pass it and pull in grounding text. Queries that miss, like heavy misspellings, retrieve nothing, so the model answers with no documents steering it.

Related reading

• LLM Model Fingerprinting: Call It the Wrong Name and It Tells You the Right One (part 2: what model sits behind the RAG)
• Indirect Prompt Injection: Every Channel, One Payload (what happens when an attacker can write to those documents)
• LLM Data Exfiltration: No JavaScript, Just an Image (getting retrieved data out)
• Browse the whole AI / LLMs track