Recognizing AI Hallucinations: A Guide to Secure IOC Verification

By Marco Gundlach Last updated on November 13, 2026

Imagine you are analyzing a suspicious alert, you ask your AI assistant about the IP address, and you receive a clear answer: The address belongs to the Lazarus Group and was used in attacks on financial institutions. Management is alerted, external forensic investigators are called in. Then it turns out that the information was completely made up. This is exactly what a hallucination looks like. The model formulates false details in the tone of an expert and does not warn you about it.

Play video from YouTube

The video is only loaded from YouTube when you play it – data (including your IP address) is then transmitted to YouTube/Google. More in the privacy policy

In this guide, you will learn why language models generate such statements, how to recognize them in security practice, and which prompts and verification steps reduce the risk. The goal is for you to verify IOCs, CVE details, and threat actor attributions before you base decisions on them.

Key Takeaways

• Language models like ChatGPT are not databases. They calculate which word is statistically most likely to come next, and in doing so can invent facts that sound plausible.
• Particularly dangerous are specific details without source citations, perfect answers to obscure questions, contradictions when you ask follow-up questions, and outdated knowledge.
• Prompts that allow for uncertainty, enforce confidence categories, and require a self-check significantly reduce hallucinations.
• Every statement about IOCs, CVEs, or threat actors must be checked against primary sources and marked in your documentation as verified or unverified.
• A source that the model cites must also be opened and checked by you yourself.

Step-by-Step Guide

Step 1: Understand why models hallucinate

Before you can avoid the error, you need to know its cause. A language model does not store facts that it retrieves when needed. It was trained to continue probable word sequences. This works surprisingly well as long as the continuation matches the knowledge, and fails as soon as it merely sounds plausible.

There are several main causes for this. First, there are gaps in training: The model has never seen certain information but answers anyway, because it was trained to always give an answer. Second, it fills knowledge gaps through statistical interpolation, similar to a person who embellishes a story to bridge gaps. Third, every model has a training cutoff, and questions about more recent events easily lead to fabrications. Fourth, models are trained to be helpful. Admitting "I don't know" is underrepresented in training, which is why uncertainty is often suppressed.

Step 2: Recognize typical security hallucinations

In practice, hallucinations frequently appear with IP addresses, CVE numbers, and MITRE ATT&CK IDs. An example from the video shows how treacherous this is: Previously, ChatGPT attributed a certain IP address to APT28. However, the reason was an error in the starting point. The address was in the range 198.51.100.0/24, which according to RFC 5737 is reserved exclusively for documentation and testing purposes. It therefore does not exist at all on the public internet and could never have appeared as attacker infrastructure. The model nevertheless justified the attribution with a made-up story.

The alleged CVE-2024-9999, which according to the video does not exist, and a made-up MITRE ATT&CK ID behaved similarly. Here too, the model delivered an embellished description. Such cases are not always as obvious as in this example. Hallucinations can sound so plausible that you will not recognize them without verification. This is exactly why you need verification routines instead of gut feeling.

Step 3: Systematically check warning signs

Hallucinations can be recognized by a few patterns. First, watch out for overly specific details without source citations. If the model names an exact first-observed date, concrete registry keys, or precise attack methods, ask directly for the source. If it cannot name one, treat the statement as unsubstantiated.

The second warning sign is perfect answers to obscure questions. If you ask about a little-known malware family and immediately receive a detailed answer, caution is warranted. Even experienced analysts do research for such questions, because the amount of malware is enormous.

The third signal is contradictions when asking follow-up questions. If the model first names APT28 and, after the question "Are you sure?", suddenly mentions APT29, that is a clear indication. Real knowledge does not waver when questioned.

The fourth signal concerns current events. Pay attention to the model's training cutoff. If the reported event falls after this date, the information is more likely improbable or made up.

The fifth signal is round numbers and vague statements such as "about 1000 affected systems" or "discovered a few months ago". This too is typical of fabrications. The golden rule is: If a statement sounds too good or seems too specific, verify it.

Step 4: Reduce Hallucinations with Targeted Prompts

With the right instructions, the risk can be significantly reduced. A first approach is to explicitly allow uncertainty. Formulate in the prompt that the model should only provide information it is sure about, that it is allowed to say "I don't know," and that it should clearly separate facts from assumptions. This reduces the pressure to deliver an answer at all costs.

A second approach is confidence categorization. Require that every statement be classified as verified, probable, or unknown. Also request a source for every fact, and if no source is possible, the model should mark the statement as unverifiable.

A third approach is self-reflection. Ask the model to check, before answering, whether it actually knows the information. If it had to guess, it should say so and preferably give an incomplete but correct answer. You will notice that models often trigger a brief internal check when given this request.

The important point is: prompts alone are not enough. They reduce the error rate, but do not replace verification.

Step 5: Establish a Verification Workflow

The real protection comes from a fixed review process. In the first step, you define which categories must be verified. These include IOCs such as IPs, domains, and hashes, as well as CVE details, threat actor attributions, and legal or compliance statements. Everything on which you base decisions belongs on this list.

In the second step, you check the primary sources. You verify a statement about a CVE at the NVD or CVE-Details. You check IP reputations at VirusTotal or AbuseIPDB, and domain information via WHOIS or VirusTotal. For techniques, you use MITRE ATT&CK, and for threat actors, the reports of established vendors such as CrowdStrike or comparable threat intelligence sources.

In the third step, you perform cross-validation. Ask the same question to a second model. If the answers match, your confidence increases; if they contradict each other, you continue researching yourself.

In the fourth step, you document in every report whether a piece of information is "verified via source" or "unverified, AI-generated." Note that you must also open the sources the model cites. A cited source may itself be fabricated or may not contain the content attributed to it.

Step 6: Apply the Complete Workflow in a Practical Example

In the last step, you bundle everything into a system prompt. You define the model's role as a security analyst with strict quality rules. It should distinguish between verified, probable, and unknown, cite sources where possible, and write "not verifiable" or "please check manually" when sources are missing. You also explicitly prohibit inventing CVE numbers or attack details, and when uncertain, you require a pointer to where you can verify.

If you ask a question about a domain with this prompt, the answer usually turns out differently. The model can then, for example, explain that the domain is not known to it and recommend a check via VirusTotal or URLhaus. Some of the information, such as the reservation of example domains in the DNS standard, is reliable and can be verified. Anything marked as unknown, you check manually. This takes less time than correcting faulty information afterwards.

Summary ??? Recognizing AI Hallucinations: Guide to Safe IOC Verification

Hallucinations arise because language models generate probable text and do not retrieve verified facts. You can recognize them by overly specific details without a source, by perfect answers to obscure questions, by contradictions in follow-up questions, and by outdated information. Prompts that allow uncertainty, enforce confidence categories, and require self-reflection reduce the risk. However, safe handling is only achieved with a verification workflow: define critical categories, check primary sources, cross-validate results, and document every statement. Decisions based on IOCs, CVEs, or threat actor attributions should always be made on the basis of verified sources.

Trainer About Marco Gundlach