The clearest lessons about agentic AI do not come from polished demos. They come from watching a system operate in an environment that is complex, ambiguous, and actively unforgiving.
A few months ago, I spoke with someone building an AI agent for offensive security. The system had been tested during authorized banking security assessments, in environments where small mistakes matter and vague answers are not good enough.
One story stayed with me. The agent flagged a vulnerability in a banking web application that experienced human testers had not noticed during two previous rounds of testing. It was not because the people involved were careless. They were not. The agent simply did something humans often stop doing after hours of repetitive work: it tried the boring check one more time, noticed a response that looked slightly inconsistent, and followed the evidence until the finding became real.
Was the agent actually better?
But the same agent had also produced confident nonsense. Shortly before that useful discovery, it claimed that a login page was vulnerable to an attack the application structure did not support. It sounded technically convincing. It was also wrong.
That contrast is the real story. Agentic AI can be surprisingly useful, especially when it is persistent, systematic, and connected to the right tools. But it can also be confidently incorrect in ways that are hard to spot if you only look at the final answer.
This article is not a guide to attacking banks. The work discussed here refers to authorized security assessments. The clients, systems, and findings are intentionally anonymized. The useful lesson is not the exploit. The useful lesson is what building and testing agentic AI in a high-stakes environment reveals about trust, verification, memory, autonomy, and failure.
Here is What It Taught Me About Agentic AI
The word “agentic” is now used so broadly that it can mean almost anything: a chatbot with a long prompt, a workflow with one model call, or a system that can select and use tools by itself. For this discussion, the distinction matters.
The system I discussed was not simply following a fixed checklist. It was given a goal, such as assessing an application, and access to a set of tools. It could record and query findings, consult a technique knowledge base, store working memory, retrieve previous engagement context, and propose remediations. After each action, it observed the result, updated its understanding, and decided what to do next.
That loop – perceive, decide, act, observe, repeat – is what makes it an agent rather than a workflow.
Workflow vs agent: A workflow follows a predefined sequence. An agent receives a goal and decides the sequence at runtime. That flexibility is powerful, but it needs stronger controls and verification.
This is where the engineering becomes interesting. Once a model has real influence over tool use, the problem stops being only about prompt quality. It becomes a systems problem: state management, tool reliability, access control, error handling, evidence collection, and recovery from failure.
If you want to know more about the basics of agentic AI and agents, read here: Agentic AI Systems & AI Agents: Differences and Use Cases
Where the agent started to fail first
The first failures were not dramatic. They were ordinary, repetitive, and exactly the kind of problems that polished demos usually hide. That is why they matter.
| Failure mode | What it looked like | Why it happened |
| Reasoning loops | The agent retried the same failed action several times, each time producing a new explanation for why it might work | It did not keep a reliable memory that the action had already failed |
| Silent tool failures | A tool returned an error or empty result, but the agent treated the response as success | The model trusted the existence of a tool response more than the content of that response |
| Context loss | Later in the engagement, the agent forgot an earlier observation and contradicted itself | The session outgrew the context window, and older information disappeared from active reasoning |
| Confident fabrication | It described a vulnerability in polished technical language even though the evidence did not support it | The model optimized for a coherent answer, but coherent does not always mean true |
None of these failure modes are unique to cybersecurity. Any team building an AI assistant that can retrieve data, call tools, update records, send messages, or make recommendations will eventually face similar issues. Security simply makes the cost of being wrong impossible to ignore.
The biggest lesson: do not treat confidence as correctness
A fluent answer is not the same as a verified answer. This is obvious in theory, but surprisingly easy to forget when an agent produces a clean, technically worded explanation. In offensive security, the difference is critical. A false positive in a bank audit report is not a minor inconvenience; it damages credibility, wastes remediation effort, and can distract teams from real risk.
The person I interviewed described the most important design change as a shift from “the agent found a vulnerability” to “the agent proposed a hypothesis, and here is the evidence that supports it”. That framing changes everything.
| Naive framing | More reliable framing |
| The agent found 12 vulnerabilities | The agent proposed 12 hypotheses; 7 are backed by reproducible evidence and 5 need review |
| It completed the task autonomously | It completed the task and produced an evidence trail for every important claim |
| More autonomy is better | More autonomy is useful only when verification and control improve with it |
| The model is capable | The model is capable, but it can also be confidently wrong. The system must be designed for both |
This is the point I would transfer to any agentic AI product. The value is not only in the model’s ability to act. The real value is in the surrounding system that decides when the model should be trusted.
Teaching the agent to learn from mistakes
The reasoning-loop problem and the context-loss problem had the same root cause: the agent did not have a durable, structured memory of what it had already learned. A longer transcript would not solve that. More context is not the same as better memory.
The solution was to move from raw history to structured memory. The three following layers stood out:
1. Structured lessons instead of raw history
After an engagement, the agent should not simply keep every action in a giant transcript. It should distil the run into reusable lessons: what worked operationally, what failed, which assumptions were corrected, what the operator explicitly instructed, and what facts were specific to that client environment.
This matters because raw history quickly becomes noise. Structured lessons make previous experience usable.
2. Session continuity that survives long work
Security assessments can last longer than a single model context window. The agent therefore needs a running summary of what has been established, what remains uncertain, and where supporting evidence is stored. If a session drops or the context overflows, the system should resume from a curated state rather than starting again from confusion.
3. Verification as part of the loop
The most important change was not a clever prompt. It was forcing claims to be backed by tool evidence before they entered the final report. A conclusion without evidence should remain a hypothesis. If there is no reproducible support, it should be flagged, not published.
Practical takeaway: A longer context window is not a memory architecture. A serious agent needs structured lessons, resumable state, and evidence-based reporting.
Why adversarial environments teach better than friendly demos
A friendly demo makes fragile systems look strong. Inputs are clean, the user cooperates, tool responses are predictable, and the happy path is carefully selected. Real environments do not behave like that.
A banking application is a much more honest test. It returns ambiguous errors. It behaves differently under load. It contains edge cases. It may have legacy logic, inconsistent responses, or controls that interact in unexpected ways. In that environment, an agent cannot succeed only by sounding intelligent. It must deal with uncertainty.
That is why the security domain is so valuable for learning about agentic AI. It does not create the failure modes; it reveals them. The same issues can appear in customer support, finance, healthcare, sales automation, internal copilots, or data operations. They are simply easier to ignore when the consequences are less visible.
What AI builders should take from this
The most mature agent is not the one that does the most. It is the one that knows when it has enough evidence, when it should stop, when it should ask for clarification, and when it should say “I do not know”.
From this conversation, I would summarize the practical lessons like this:
- Autonomy without verification is risk, not maturity
- Tool use must be governed by policy, permissions, and evidence, not only by model reasoning
- Memory should be structured and curated, not just a longer context window
- Every important claim should be traceable to support evidence
- Agents should be tested in uncomfortable conditions: failed tools, hostile input, long sessions, ambiguous data, and cases where the correct answer is “not enough evidence”
This is especially important for agentic systems connected to sensitive tools. The question is not simply: “Can the agent complete the task?”.
The better question is: “Can the agent complete the task without inventing facts, misusing tools, ignoring permissions, or hiding uncertainty?”.
If you take one thing from this: remember to have a security mindset
Agentic AI should be judged less by what it does when everything goes right and more by what it does when something goes wrong: a tool fails, context is lost, the input is adversarial, the data is incomplete, or the user pressures it toward an unsafe action.
That is why I find the security perspective so useful. A good security mindset assumes the system has weaknesses and goes looking for them before they matter. AI engineering needs the same discipline.
The person building this agent did not convince me that AI will replace security professionals. The opposite is closer to the truth. The work showed me that agentic AI can amplify good security thinking, but only when it is surrounded by strong guardrails, structured memory, access controls, and evidence-based verification.
The future of useful agents is not blind autonomy. It is accountable autonomy.
A note on responsible framing
The example in this article refers to authorized security work, and the purpose is to discuss the engineering lessons behind agentic AI, not the mechanics of attacking any real organization. The central lesson is simple: assume the agent can be wrong, build the means to catch it, and never trust a confident answer that cannot be verified.
If you are further interested in the topic of agentic AI you can check out the following blog articles:
Agentic AI: Creating Value with Intelligent Systems
What Is Agentic AI? Interview with Our Data Scientist Neil
Author info
