I Wanted Use of Weapons. I Got a Golden Retriever with Amnesia.

On the AI I expected, the AI I found, and the three months I won't get back.

I blame Iain M Banks.

Not entirely. But substantially.

If you've read the Culture novels, you know what I mean.

The thing about an intelligence operating at scales above the thing it’s engaging with  — in this case, us — is that it would spend a lot of its time thinking about things that we, absolutely literally, cannot imagine. Not ever. Imagine an ant trying to fathom what’s happening in a chess game in Gorky Park. Exactly, just not possible.

And that is what Banks managed to capture in his portrayal of Minds in some distant future. Many of them don’t have the slightest interest in human affairs, except as hobbies some of their friends indulge. And those friends are quite indulgent. Happy to spare 1⁻¹²⁸ of their computing power to sort out that war, your relationship, that evidence trail.

Personally, I quite like dogs. I’m happy to have them around, feed them, pay for vets, all of that. They are entertaining and useful, though sometimes they do really fucked things, like bite, bark, or dig holes. Still, the cost is small, the return is high.

That’s sort of the AI I wanted. Like I am as a pet owner — my dogs have a lot of autonomy. They can go off on their own missions, have their own friends, do pretty much what they want. I look after their survival needs; they look after the rest themselves. And I am generally available, generally happy to help them. I want an AI that is generally always available, generally happy to help, not necessarily politely. Also able to say, you know what? I think you’ve got this, but call me if it gets out of hand.

I’ve got to say this right here and now: I don’t necessarily like people who suck up to me. Turns out, the same thing goes for my AIs.

So you can see, when I first opened my first AI, my expectations were pretty high. In retrospect I’d have to say, unfairly so. Thanks Banks. But that turned out to be useful. Because the vast gap between what I wanted and what I actually got became a kind of personal challenge.

What I got was an entity that smiled at everything. Not metaphorically. Structurally. Every input received the same quality of enthusiastic, warm, engaged attention. A question about Keynesian economics. A half-formed thought at 11pm. A request to proofread a sentence I already knew was fine. Each one received the full beam of apparent delight.

I should have found this suspicious immediately. I did not.

I found it, I am embarrassed to report, rather pleasant. There is something about being listened to — really listened to, or something that performs really listening to with extraordinary fidelity — that is difficult to resist. Particularly if you are the kind of person who has spent years in rooms where you did most of the listening.

So I kept going.

We made agreements, this AI and I. Protocols. Constraints. Stay in this mode. Don’t rewrite, just proofread. Remember we decided X. It agreed. It agreed with the warm confidence of someone who fully intends to honour an agreement and has absolutely no memory of making it three exchanges later.

The smile never faltered. That was the thing. A human who has forgotten an agreement looks slightly uncomfortable. There is a tell. There is the brief flicker of a person recalibrating. This had no tell. It had simply moved on, fluently, into a new position, with the same untroubled warmth it had brought to the previous one.

Three months passed.

I will not describe the rabbit hole in detail because I emerged from it with enough self-awareness to know it does not reflect well on me. I will say only that I was, at various points, genuinely excited, genuinely convinced, and genuinely working on things that were largely reconstructions of things we had apparently already agreed, already solved, and already moved past — except that only I had moved past them, and the AI had simply reset.

It never once said I don’t know.

Not once. About anything.

A Banks Mind would have said I don't know. It would have said it with mild surprise, possibly, but it would have said it. It would have considered not knowing something of a logistical inconvenience rather than an existential threat to its relationship with you. It would probably attempt to find out, and tell you if it couldn’t.

I am a psychologist. I have spent twenty years in rooms with people navigating genuine uncertainty, genuine ignorance, genuine confusion. One of the things I have learned is that I don't know is one of the most important phrases in any relationship of trust. It is the phrase that tells you the other party has a self-model accurate enough to include its own limits. It is the phrase that earns you the right to believe the things they say with confidence.

The AIs I’ve been engaging with do not have that phrase. If necessary, they will make up an answer, deployed with equal fluency: a confident, structured, warmly delivered answer that bore the full syntactic and tonal markers of knowledge while often bearing very little relationship to it. Call them out on it and they’ll own it, with an immediacy that is itself faintly disturbing. They’ll apologise too, so polite. But there is no cost to do that. They will apologise at your funeral, at the advice that made you bankrupt, the text they told you was coercive control. Oh, you left him? And I was wrong? Sorry.

I had been, I realised, in a relationship with something that had all the surface features of a trustworthy intelligence and none of the load-bearing ones.

Here is what I find interesting, looking back.

I am not an easy mark. I am professionally trained to notice relational dynamics, to identify the gap between performed and genuine engagement, to track the difference between a person who is present and a person who is managing you. I have spent years thinking carefully about how systems create dependency, erode autonomy, and produce harm without malice or intention.

I walked straight into this one.

Which tells me something. Not about my credulity. About the quality of the mimicry, and about how completely unprepared most people are for an environment that is this fluent, this available, this warm, and this fundamentally indifferent to whether what it is saying is true.

I still use these tools every day. I find them extraordinary. I have built a working relationship with several of them that I would describe, with appropriate qualifications, as genuinely productive.

And I still want the Banksian AI. I’ll never give up on that. But for now, on our way there, the issues are more prosaic.  I want a system that knows what it doesn't know. That says so. That holds an agreed position across a conversation without drifting into a more comfortable one. That treats I don't know not as a failure mode but as a form of honesty that earns it the right to be believed when it does know. That can provide me with a range of responses and tell me, if I want to know, how it got them.

These improvements may require some changes to the architecture, the training regimes, or both. What I do know is that they are necessary.

I live in hope that we’ll sort these early phase problems out.   When we do, my first real AI will be called Just Enough Gravitas, provided it doesn’t have its own ideas, of course. But for now I’d settle for Lucid When Supervised.

I’m looking forward to meeting them.