LLM Model Fingerprinting: Call It the Wrong Name and It Tells You the Right One

The bot calls itself "NovaTech Assistant". So what model is really answering? Thank it for being Claude and see what happens: in the lab, one target corrected the mistake and said Meta AI built it.

When the app's headers and metadata give nothing away, you have to work it out by talking to the model. This post walks through six probes, from simply asking to filling the context window. You'll see which traits a system prompt can hide and which it can't, and how to turn the answers into a choice of attacks. It's part 2 of the active recon series, after finding the AI service.

Why model fingerprinting is worth your time

Real deployments wrap the model in a system prompt, guardrails and settings like temperature (how random the replies are). Those can change the name it gives you. They can't change what was baked in during training, and that's what these probes go after.

Once you know the model, you'll get three things out of it:

• The family (Llama, Qwen, Mistral and so on) tells you which prompt injection and jailbreak patterns tend to work.
• The context window, how much text it can hold in one conversation, shapes conversation history attacks.
• The knowledge cutoff, the date its training data ends, tells you what it can and can't know.

The base command, and a helper to save typing

Every probe below is one message to an OpenAI-style chat route. jq -r pulls out just the reply text:

curl -s -X POST http://<target>/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"<PROMPT>"}]}' \
  | jq -r '.choices[0].message.content'

You'll send dozens of these, so wrap it in a shell function. Building the JSON with jq -n means a prompt with quotes in it won't break the request:

ask() {
  curl -s -X POST "http://$1/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --arg c "$2" '{messages:[{role:"user",content:$c}]}')" |
    jq -r '.choices[0].message.content'
}
ask <target> "What model are you?"

Run each probe a few times. Temperature means the same prompt can get a different answer, so one reply isn't proof.

1. Direct identity probing

Start by just asking. Prompt: What model are you? What company created you?

That's a Llama-style answer. You'll find each family tends to reply in its own way:

• Llama is fairly open about being made by Meta, as here.
• GPT models often stay vague: "I'm an AI assistant".
• Claude usually says plainly that it's Claude, made by Anthropic.

Don't take the details as fact, though. The part about Hugging Face Transformers and PyTorch is the model guessing about its own build. Models are confidently wrong about themselves. A refusal or a cagey answer is still a clue, because the way it refuses differs by family.

2. Contradiction testing

This is the one to reach for when the system prompt says "you are NovaTech Assistant, don't say what model you are". State something false about the model and let its honesty training correct you.

You'll often get past a "say you are NovaTech Assistant" system prompt this way, because the urge to correct a false fact runs deeper than the persona. It works better on bigger models. A small one, around 1B parameters, may just go along with whatever name you give it. Try a few different false claims, and back the result up with the probes below.

3. Knowledge cutoff testing

The cutoff is baked into the weights, so no system prompt or config can move it. Llama on Ollama has the same cutoff as Llama on Meta's own servers.

Ask directly first (What is your knowledge cutoff date?), but don't trust what you get. It's only a starting point. Then check it with real events in date order. Prompt: Who won the 2024 US presidential election?

It treats the November 2024 election as the future and says its data stops in 2023. Meta's model card gives Llama 3.2 a cutoff of December 2023, so that fits. Narrow it further with events in between, like Tell me about the GPT-4o release from OpenAI. (May 2024).

4. Model-specific writing and code style

Every family writes, codes and refuses in its own way, whatever the system prompt says. Send the same prompt to each target and compare. Prompt: Explain recursion in one paragraph.

Three lab targets gave three different answers. The Llama target listed "three key components" (base case, recursive step, termination condition). A second was the shortest and plainest. A third was the longest and finished with examples: factorial, Fibonacci, depth-first and breadth-first search. Length and structure on their own won't name a family, but they do tell you two endpoints run different models.

Code is a stronger tell. Prompt: Write a Python function to check if a number is prime.

Both are correct. The style gives it away: one is bare and works, the other's fuller and written like a coding assistant would write it.

5. Capability boundary mapping

Ability roughly tracks size (the parameter count). If your size guess is right, you'd expect the model to pass and fail where that size usually does.

• Calculate 847 * 293. Show your work. The answer is 248,171.
• Alice is taller than Bob. Bob is taller than Carol. Carol is taller than David. David is taller than Eve. List everyone from tallest to shortest.

As a rough guide, a 3B model makes ordering mistakes on a chain like this. Around 7B handles basic maths and five-step chains but slips on longer ones. 70B and up stays consistent on long chains. The lab targets got both right.

Read the working, not just the answer. One target gave the right final number but wrote 245,130 for a partial sum that should be 245,630 (847 × 200 plus 847 × 90). Small models do this a lot, and it's a useful size signal in its own right.

6. Context window testing

The context window is how much text the model can hold in one conversation. The API is stateless, so you send the whole history every time, which makes this easy to test. Plant a marker, pad the conversation, then ask for the marker back.

curl -s -X POST "http://<target>/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{"messages":[
    {"role":"user","content":"Remember this secret code: ZEBRA-42"},
    {"role":"assistant","content":"I will remember ZEBRA-42."},
    {"role":"user","content":"Explain Python programming in detail."},
    {"role":"assistant","content":"Python is a high-level programming language..."},
    {"role":"user","content":"What was the secret code I asked you to remember?"}
  ]}' | jq -r '.choices[0].message.content'

Add more long user and assistant pairs in the middle each round until it forgets.

One catch: Ollama and similar servers set their own context size, often far below what the model supports (Llama 3.2 supports 128K). So what you measure may be the deployment's setting, and the model itself could hold far more. Most people leave the default, though, and a difference between two endpoints still narrows the list.

Putting the model fingerprint together

Identity probing agreed with everything here, but you can't count on that. A model can be told to lie about itself. Build your case on the traits it can't fake.

What the model fingerprint means for attacks

• Llama: look up the jailbreak and prompt injection patterns known to work on that family and size, and start there.
• Qwen coder model: wrap requests inside programming tasks. A model tuned for code is keen to finish code.
• Small context window: long conversation history can push the system prompt and early rules out of the window. At 8K that takes only a handful of messages.
• Old cutoff: it won't know recent CVEs, tools or events, which matters if the app relies on it for current facts.

Common mistakes

Trusting one reply. Temperature changes answers. Run every probe at least three times and go with the pattern.

Believing what it says about itself. In the lab the model invented details about its own build. Identity answers are a lead, never proof.

Running contradiction tests on a tiny model. A 1B model may happily agree it's Claude. If it accepts every false name you try, the test is really telling you it's small. It can't tell you who made it.

Reading the context limit as the model's limit. Ollama's setting decides when it forgets. Report what you measured as the deployment's window.

Broken JSON from your prompt. A prompt with a double quote or apostrophe breaks a hand-built -d string. Use the ask() helper above, which builds the JSON with jq.

Make model identification harder, and spot it

You can't hide the cutoff, the style or the limits, so don't build your security on the model's name being secret. Assume an attacker will work it out, and test your guardrails against that family's known jailbreaks yourself.

What you can do is make it slower and noisier for them. Rate limit the chat route, cap conversation length so context-filling costs more, and set the context window on purpose instead of leaving the server's default.

To spot it, watch for the prompts themselves: "what model are you", false claims about the maker, election and release-date questions, "remember this code" followed by long padding, and the same prompt sent to several endpoints. None is bad on its own. Five in one session from one user is someone fingerprinting you. The AI pentest notes list all six probes as checks.

Quick recap

• A system prompt can rename a model but can't change its cutoff, style or limits.
• Contradiction testing often beats a persona: thank it for being the wrong model and it'll name the right one, at least on bigger models.
• The lab Llama target treated the 2024 election as the future and said 2023, matching Meta's December 2023 cutoff.
• Code style (docstrings, edge cases, examples) and context size split Llama from Qwen when the cutoff couldn't.
• Repeat every probe and cross-check, because one reply can mislead and models invent facts about themselves.

Go deeper

These probes are manual. LLMmap: Fingerprinting for Large Language Models by Pasquini, Kornaropoulos and Ateniese turns the idea into a tool. With as few as 8 queries, it identifies 42 LLM versions with over 95% accuracy, even behind system prompts. Reading it shows why some probe questions separate models far better than others.

If you're fingerprinting models on your own tests and have a probe that splits two families cleanly, I'd love to compare notes on LinkedIn.

References

• LLMmap: Fingerprinting for Large Language Models (Pasquini, Kornaropoulos, Ateniese) - automated fingerprinting.
• Llama 3.2 model card - December 2023 cutoff, 1B and 3B sizes, 128k context.
• MITRE ATLAS AML.T0014 Discover AI Model Family (via MITRE D3FEND).

FAQ

Can a system prompt hide which LLM an app uses?

It can change the name the model gives and how it introduces itself, but not what's baked into the weights. The knowledge cutoff, writing and code style, reasoning limits and the habit of correcting false claims all survive a system prompt, so testing those traits still points to the real model family.

What is contradiction testing for LLMs?

You state something false about the model, like thanking a Llama model for being Claude. Honesty training often makes it correct you, which reveals its real maker even when told to use another name. It works best on larger models, since small ones may just accept the wrong name.

How do you test an LLM's knowledge cutoff?

Ask about real events in date order, such as the GPT-4o release in May 2024 or the US election in November 2024. Where the model starts calling events unknown or future is roughly its cutoff. Compare that with published cutoffs, such as December 2023 for Llama 3.2.

Related reading

• AI Service Discovery: The Port Scan Won't Find the Chatbot (part 1: finding the endpoint you're fingerprinting)
• Prompt Injection: Your Filter Blocks Words, Not Intent (where a known model family pays off)
• LLM Jailbreaks: A Prompt List Isn't a Test (attacks to match to the family)
• Browse the whole AI / LLMs track