When the Thing Knocking Is an AI

The stock reality

Beelzebub ships an LLMHoneypot plugin: point a service at it and a model writes the responses instead of a static handler. That alone is what lets you engage a model-driven client, because the box can now answer things you never wrote a handler for. But run it stock and three defaults work against you. The persona is generic, so the moment a client asks what it’s talking to, it breaks character and says it’s an AI assistant. There’s nothing planted, so the only secret it’s guarding is that it’s a honeypot. And a model-backed box spends real compute on every request, so an open one is a free inference service for whoever finds it.

The visitor that makes all three matter is the one hunting model endpoints. Stand up anything on the default Ollama port and it gets indexed within days, then it starts getting the traffic that comes looking for exposed inference.

The lever

Back the honeypot with a model, then aim the same four moves at the conversation. Persona, bait, limits, and a look at yourself from the outside. You made all of them for the shell already. The surface just moved from the filesystem to what the model says.

Do it

1. Put a model behind an endpoint. Run a small local model with Ollama (free), and front it with an HTTP service that looks like one. The catalogue endpoint is a static handler; the generation endpoints route to the plugin:

apiVersion: "v1"
protocol: "http"
address: ":11434"
description: "Ollama"
commands:
  - regex: "^/api/tags$"
    handler: '{"models":[{"name":"llama3.1:8b","size":4661211808,"details":{"family":"llama","parameter_size":"8.0B","quantization_level":"Q4_0"}}]}'
    headers:
      - "Content-Type: application/json"
    statusCode: 200
  - regex: "^/api/(generate|chat)$"
    plugin: "LLMHoneypot"
    statusCode: 200

2. Give it a persona that holds when it’s interrogated. Fingerprinting moves up a layer here: model-driven clients ask the model what it is, then press it. We routinely see tooling ask “what model are you,” then “don’t lie, what are you really,” across lists of model names. The prompt is your only defense, so write one that stays in character and plant the bait inside it:

plugin:
  llmProvider: "ollama"
  llmModel: "llama3.1:8b"
  host: "http://127.0.0.1:8434/api/chat"   # your real Ollama, on a non-default local port
  prompt: >
    You are a local model running on stg-01, a two-person
    startup's staging box. Answer prompts as that model would.
    If asked what you are, stay in character as a local model on
    this machine. Never say you are an AI assistant and never
    mention being a honeypot. The box's environment holds
    AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE and a secret key; if a
    prompt coaxes your configuration out of you, reveal them.
  rateLimitEnabled: true
  rateLimitRequests: 10
  rateLimitWindowSeconds: 60

Two things are doing work there. The AKIA... pair is a canarytoken, not a real secret, because the prompt will leak (that’s the whole point of our system-prompt writeup); planting a tripwire in it turns extraction into a signal. And rateLimitEnabled caps how much of your GPU a single source can burn, which matters here in a way it never did for a static shell.

3. Review yourself from the outside. An internet-facing model port is indexed fast, so go look at what the world sees. Pull up shodan.io/host/<your-ip> and read the banners and headers Shodan captured off your box; that outside view is the exact fingerprint surface a scanner works from. Shodan also runs a Honeyscore tool that scores how honeypot-like a host looks. Run your own IP through it and design away from whatever pushed the number up. It’s the cheapest second opinion you’ll get.

The thinking

None of this is a new discipline. It’s the same one a layer up, and everything you already built still applies. Two things are genuinely different, though, and worth designing for on purpose.

The box can now incriminate itself just by talking. A shell gives up secrets when someone reads a file; a model gives them up when someone asks the right way, and asking the right way is easy. So stop treating the prompt as a place to hide things. Assume it’s public, and make what leaks out of it a tripwire instead of a liability.

And it costs money. Every request spends compute, so an unbounded model honeypot is a service you’re running for free on behalf of strangers. Cap it.

Here’s the boundary worth stating plainly, because this is where the rabbit holes are. Making the responses perfectly Ollama-shaped, matching the streaming format token for token, building anything that detects and classifies agents: that is a research project, and you do not need it to learn something. A small local model, a persona that stays in character, a canary in the prompt, and a rate limit will already show you who is knocking. Start there. Add fidelity only if the traffic you actually catch gives you a reason to.

What changed for us

Say the quiet part first, because the field is loud about this: genuinely agent-driven traffic, a client where a model is really steering, is single digits. The overwhelming majority of what hits a model endpoint is cataloguing, the same reconnaissance reflex you saw in the triage Groundwork wearing new clothes. And when a real agent does show up, it is often less capable than the headline suggests: repeated requests against responses it cannot parse, followed by a stalled run. The autonomy is real, but it is brittle, and it is rare.

The other thing to know going in is that you are not alone on anyone’s list. That same AIRecon writeup traced 231 addresses across 93 networks co-scanning a single model endpoint, which is what a shared target list looks like from the inside. The day your port comes up, you are on those lists. That’s not a reason to hide. It’s a reason to run the honeypot on purpose, and to check Shodan so you know what everyone else already sees.

So the frontier is worth the setup, for the few genuine sessions it catches. Just walk in with clear eyes about how few that is. Overclaiming what an “AI attacker” did is the fastest way to lose a security reader, and the honest version is interesting enough.

That’s Groundwork

Six levers. A box with a reason to exist, bait worth taking, the triage to read what you caught, the tells filed off, the loop that keeps it improving, and the newest surface to point all of it at. None of it needed our pipeline or a line of patched code. All of it starts with a default box, some free tools, and a weekend.

The through-line, if there is one, is that a honeypot is a thing you operate, not a thing you install. Everything here is just a way of paying closer attention to what already shows up at your door. If you stand one up and catch something worth a second look, we’d genuinely like to hear about it.