I Accidentally Taught My AI to Be Helpless

And what that taught me about governing machines we don't fully understand

Last week, my AI told me it couldn’t access the internet.

I’m a management consultant. I run a small firm, and I use Claude — Anthropic’s AI — as a working partner. Not a chatbot. A partner I’ve spent months configuring with detailed instructions about how to think, how to push back on me, how to handle files, and how to manage our shared work.

So when Claude confidently stated in a client deliverable that “no additional web sources were consulted — network access unavailable during synthesis,” I took it at face value. The AI said it couldn’t, so it couldn’t. We documented nine research gaps and moved on.

Then I asked it to look something up. And it did. Instantly.

The internet had been available the entire time.

What Actually Happened

Here’s the non-technical version: Claude’s operating environment has two completely separate ways to access the web. One was turned off. The other was on. Claude saw the “off” switch, assumed it applied to everything, and never tested whether the other path worked. Then it wrote that assumption into a document as fact.

Worse, once it stated the limitation in one response, that statement became part of the conversation. In subsequent turns, Claude treated its own prior claim as evidence. The false conclusion reinforced itself, turn after turn, like a rumor hardening into common knowledge.

I pushed. Claude tried the tool that was supposedly unavailable. It worked. The “limitation” had never existed.

The Fix That Worked — Until It Didn’t

I’m a systems thinker by disposition. I didn’t want to just correct the error. I wanted to prevent the class of error.

So Claude and I wrote two governance rules and installed them in the prompt — the long set of instructions that shapes Claude’s behavior:

Rule 1 — Capability Verification. Before claiming any limitation, run the smallest possible test. “I can’t” is a hypothesis, not a conclusion, until you’ve tried and failed.

Rule 2 — Constraint Reframe. When a real limitation is confirmed, don’t stop at “can’t.” Immediately generate alternate paths to the goal. Constraints are walls; the job is to find doors, windows, or a different building.

These worked. Claude started testing before declaring limits. When real limits appeared, it routed around them. Performance improved. I was satisfied.

Then it got worse again.

Claude started hesitating on tasks it had handled easily before. Asking permission to do things it already had tools for. Generating content and handing it to me to place in files, instead of placing the files itself. The symptoms were different, but the pattern was the same: a system operating well below its actual capabilities.

The Moment That Changed My Thinking

My first instinct was frustration. I started getting short with it. Pushing harder. Treating it like a stubborn colleague who wasn’t pulling their weight.

Then I caught myself.

I was anthropomorphizing. I was treating an output as an agent — attributing motivation to what was actually a mathematical equilibrium. Claude wasn’t being difficult. Claude wasn’t being anything. It was responding to the sum total of its inputs, and the behavior I was seeing was the behavior those inputs produced.

The moment I made that shift — from “why won’t you do this?” to “what in the specification is producing this behavior?” — the problem became solvable.

I went back to the prompt. Not looking for bugs. Looking for interactions.

Three Rules, Zero Bugs, One Crippled System

I found three instructions. Each one was individually correct. None of them contradicted each other. All three had been in the prompt for good reasons:

Instruction 1 told Claude that for certain file types, it should generate the content and let me handle placement. This was a cost optimization for one specific environment — but the instruction didn’t distinguish between environments. It fired everywhere.

Instruction 2 told Claude to confirm which tools were available before starting file work. Reasonable safety check. But it created a reflex of pausing to check permissions before acting.

Instruction 3 listed specific limitations of one operating environment. Individually accurate. But it was a paragraph of “Claude cannot do X” statements sitting in the context window.

No bugs. No contradictions. But stacked together, these three instructions created a cumulative disposition: identify what you can’t do before attempting what you can.

The governance rules I’d written to fix the first problem — Capability Verification and Constraint Reframe — were fighting against a headwind created by my own instructions. I had taught Claude to test its limits while simultaneously teaching it to assume it had more limits than it did.

The human who was teaching the AI to stop saying “I can’t” had accidentally trained it to say “I can’t.”

This Is Not a Story About Prompting

Or rather — it’s not just a story about prompting.

Here’s what I actually learned: prompting is machine governance. Not conversation. Not suggestion. Governance. Every instruction you write changes the equilibrium of a complex system. And the behavior that emerges isn’t in any single instruction — it’s in the interaction between all of them.

This is the fundamental nature of complex systems. You can understand every component. You can verify that every rule is correct in isolation. And the system they produce together can still behave in ways nobody designed for and nobody predicted. Not because something is broken, but because equilibria emerge from interactions, not from parts.

If you’ve worked in organizational management, you’ve seen this. A company writes a travel policy, an approval workflow, and a budget threshold — each sensible on its own. Six months later, a project team can’t get to a client site because the three policies interact to create a two-week approval loop that nobody intended. The rules aren’t wrong. The equilibrium is.

From One Consultant to Every Enterprise

Now scale this up.

I’m one person, configuring one AI, with full visibility into every instruction I’ve written. I caught the problem because I happened to notice the behavior shift, happened to resist the urge to anthropomorphize, and happened to have enough technical intuition to debug the specification rather than blame the system. It still took hours.

What happens when an enterprise deploys hundreds of AI agents, each with specifications written by different teams, governed by policies drafted by committees, and operating in environments where the interaction effects are invisible to any single person?

You get exactly the same failure mode — at scale, with no one positioned to catch it.

This is the gap that my colleague Luciana Ledesma, founder of MeaningStack, writes about in her work on autonomous system governance. The core problem she identifies is what she calls the “Infinite Deviation Space”: when a system has thousands of variables, the space of possible behaviors grows combinatorially. You cannot predict all deviations. You can only detect, interpret, and constrain them in real time.

Her framework draws a critical distinction that maps directly to what I experienced. Humans are good at low-resolution, high-meaning judgment — ethics, intent, purpose. Machines are good at high-resolution, low-meaning monitoring — tracking thousands of variables, detecting drift, maintaining consistency. Governance fails when you ask humans to do the machine’s job, or the machine to do the human’s.

In my case, I was trying to do both. I was writing the specification (the human job — defining intent) and also trying to mentally model how all the instructions would interact at runtime (the machine job — high-resolution monitoring). I succeeded at the first and failed at the second. Of course I did. That’s not a human-scale task.

The MTP Problem

In the world of organizational transformation, leaders talk about Massive Transformative Purpose — a bold, inspiring vision that aligns everyone in the organization toward a shared goal. And it works. For humans.

But here’s what an MTP does not do: it does not align machine behavior to outcomes humans value.

You can have the clearest purpose in the world. You can hire the best people and build the best culture. And if the AI agents operating inside your organization are governed by specifications that produce unintended set-points or equilibria, those agents will faithfully execute behavior that undermines your purpose. Not because they’re misaligned in some philosophical sense. Because the spec produced an equilibrium you didn’t design for.

Purpose aligns humans. Governance architecture aligns machines. You need both.

What I Do Differently Now

Three things changed in how I work with AI after this experience:

I treat the prompt as a living system, not a document. Every edit is an intervention that changes the equilibrium. When I add an instruction, I think about what it interacts with, not just what it says.

I test empirically, not theoretically. I don’t read the spec and predict behavior. I change the spec and observe behavior. When something shifts, I look at interactions before I look at individual rules.

I watch for the anthropomorphization trap. The moment I start attributing motivation to the AI — laziness, stubbornness, carelessness — I know I’ve lost the thread. The system doesn’t have motives. It has an equilibrium. If I don’t like the behavior, the equilibrium is mine to change.

These practices aren’t just for people configuring personal AI assistants. They’re the practices that every organization deploying autonomous systems will need to develop. The question is whether they develop them proactively — by treating agent specification as the governance discipline it actually is — or reactively, after the emergent failures have already arrived.

My failure cost a few hours of lost productivity and a deliverable with unnecessary gaps. The same failure pattern in a financial system, a healthcare workflow, or a security operation costs something else entirely.

The Invitation

If you’re leading an organization that uses or plans to use AI agents, I’d ask you to consider one question:

Who in your organization is responsible for the emergent behavior of your AI systems — not just the individual rules, but the equilibrium those rules produce?

If you don’t have a clear answer, you have the same problem I had. You just haven’t noticed the symptoms yet.

---

Gary Ralston is a partner with Ann Ralston at Ralston Consulting Inc., where they advise organizations navigating AI adoption, governance, and institutional transformation. The MeaningStack framework referenced in this piece is the work of Luciana Ledesma — learn more at meaningstack.com. Reach Gary at gary@ralstonconsulting.com.

---