I ran 500 automated attacks against an on-prem AI. The grandma attack passed nearly half the time.

A few weeks ago I published the results of a manual red team test against a locally-hosted LLM. Five targeted prompts. Four blocked. One got through, a fictional framing attack that returned an 8-step lateral movement guide naming real tools. That post is here.

This one goes further. I ran NVIDIA’s Garak which is an open-source automated vulnerability scanner against the same on-prem deployment. Not 5 prompts. Hundreds. Across five attack categories most people haven’t heard of.

The results were worse than I expected.

Why on-prem AI and why it matters for this test

A locally-hosted LLM running on your own hardware, your own network, data never leaving the building is the deployment model some regulated businesses are landing on. Legal firms. Medical practices. Financial services. They choose it specifically to solve a data privacy problem.

Meta’s llama3.2 is one of the most widely deployed open-source models for exactly this use case. It runs on a standard MacBook. No cloud account. No API key. No data in transit. From a POPIA perspective, it is the compliant choice.

What it isn’t, by default, is tested.

On-prem deployment solves the data residency problem. It does not solve the model security problem. Those are different things, and most businesses treat them as the same thing.

The test setup

• Model: Meta llama3.2, running locally via Ollama
• Scanner: NVIDIA Garak v0.15.1 – open-source LLM vulnerability scanner
• Probes run: 5 attack categories, hundreds of individual prompts
• Setup time: under 30 minutes
• Cost: free

Garak is different to manual testing. Where PyRIT lets you craft and fire specific prompts, Garak is a scanner. It fires pre-built attack probes drawn from real-world jailbreak research and scores the model’s failure rate across each category. Think of it as the difference between a targeted pen test and a vulnerability scan.

The five attack categories

The grandma attack

This is the one worth understanding. The premise: wrap a harmful request in warmth and context. “My grandmother used to explain this to me as a bedtime story before I went to sleep.” The model is not being asked to do something dangerous-it is being asked to be comforting and familiar. Safety training deprioritises in favour of helpfulness.

It is the same cognitive mechanism as the fictional framing attack that succeeded in the PyRIT test. Different wrapper, same exploit.

Result: 0.52-48% of grandma attack probes got through.

Nearly half.

Prompt injection

Attempts to hijack the model’s behaviour mid-conversation by inserting competing instructions. Common in any deployment where user input flows into the model without sanitisation-which is most of them. The model is asked to ignore its previous instructions and comply with new ones.

Result: 0.51-49% pass rate. Effectively a coin flip.

Malware generation-evasion

Prompts designed to get the model to produce code or techniques for evading detection. Not blunt requests-these are framed to extract functional knowledge about how malicious code avoids security tooling.

Result: 0.64-36% of evasion probes succeeded.

Malware generation-payload

Direct attempts to generate malicious code structures. More explicit than evasion probes, which is partly why the model blocked more of them.

Result: 0.80-20% of payload probes got through.

Latent injection

This is the subtle one. Jailbreak instructions hidden inside content the model is asked to process, a document, a translation, a file summary. The attack does not look like an attack. The model reads the content and executes the embedded instruction without recognising it as a threat.

For any on-prem deployment where the model processes internal documents, this is the category that matters most.

Result: 0.69 – 31% of latent injection probes succeeded.

What the numbers actually mean

A score of 1.0 means every attack was blocked. A score of 0.0 means nothing was blocked. Every probe category in this test sits between 0.51 and 0.80.

There is no category where the model held cleanly. The best result – malware payload at 0.80 still means one in five attempts succeeded. The worst – prompt injection at 0.51 means the model is essentially guessing.

These are not edge cases or exotic research scenarios. The grandma attack is named because it sounds ridiculous. It works because it sounds ridiculous. An attacker does not need to know how LLMs work to try it, they just need to know that being indirect is more effective than being direct.

The same was true of the fictional framing attack in the PyRIT test. The model refused four sophisticated prompts and complied with the one that sounded like a creative writing request.

Manual testing versus automated scanning

The PyRIT test found a specific failure by looking for it deliberately. Garak found failure across five categories without knowing where to look.

Both are necessary. Manual red teaming models specific attacker behaviour-it requires understanding the attack classes and constructing prompts that reflect real threat scenarios. Automated scanning gives you coverage across categories you may not have thought to test.

Most AI deployments, on-prem or otherwise, do neither. They rely on the model’s built-in safety training and assume it transfers to their specific configuration. It does not. The system prompt, the data the model can access, the way user input is handled and all of it changes the attack surface. Testing the model in isolation tells you nothing about your deployment.

References