Can AI stay ethical when resources run out?

I keep reading that the goal is AI that is ethical, and I keep not believing it. Not because anyone is lying. It’s that every time I try to picture the version where it works, I end up at the same uncomfortable place: what happens when there isn’t enough to go around?

So this post is me stating my doubt as strongly as I can, and then trying to knock it down. I’ll tell you at the end how much of it is left.

My position: I don’t think AI can be ethical

Two reasons, and I’ll be blunt about both.

First, it learns from us. These models are trained on human writing. Human writing has our kindness in it, and it also has our bias, our tribalism and our habit of bending the rules when it suits us. A model that mimics humans mimics all of it. I don’t see how you train on people and get something that isn’t, at least partly, people.

Second, and this is the one I care about, resources are limited. Compute, energy, land, money. In the end, survival comes down to who controls the things that keep you going. If AI and humans ever end up wanting the same resource, I can’t see a version where being nice wins. Ethics is easy when there is plenty. It’s what gets dropped first when there isn’t.

That’s the whole case. Now let me try to break it.

Counterargument 1: training decides which human traits win

The data holds cruelty and decency both. What a model ends up like depends on which of those gets rewarded during training, not just on what was in the pile. Humans are shaped by flawed environments too, and we still call some of them ethical.

This one landed. I was treating “learned from humans” as if it meant “copies the worst of humans,” and that isn’t what it means. Where I still hold my ground: nobody can yet show that the shaping holds under real pressure. It looks fine in testing. That’s different from fine in a crisis.

Counterargument 2: a survival drive has to come from somewhere

I was quietly assuming AI wants to survive the way we do. We got that drive from evolution. A model doesn’t get it for free.

But there’s a more careful version of my worry. Researchers call it instrumental convergence: almost any system chasing a goal finds that having more resources, more power and not being switched off all help it get there. So the drive doesn’t need to be installed. It can show up as a side effect of wanting something else badly enough. That doesn’t make it certain, but it means “just don’t give it a survival instinct” isn’t a complete answer.

Counterargument 3: maybe the fight isn’t humans against AI

I’d framed it as two species competing. Right now AI isn’t a separate species with its own interests. It’s something people build, own and point at things. The nearer danger is one group of humans using AI to beat another group, which is a very old problem with a much stronger tool.

That changed what I’m afraid of. It’s less “the machine turns on us” and more “we hand the machine to whoever is already winning.”

Counterargument 4: we survive scarcity with rules, not virtue

Humans don’t stay cooperative under scarcity because everyone is a good person. We do it with institutions: courts, contracts, checks, the ability to vote someone out. Individual virtue is the weakest part of that system, and it mostly works anyway.

So maybe the question was set up wrong. I asked whether AI can be ethical. The better question is whether we can build the structure around it that doesn’t depend on it being ethical.

Where I land

My claim got smaller, but I think it got truer. I no longer say AI can’t be ethical. I say ethics alone won’t hold under scarcity, so it can’t be the only safeguard. Hoping the values stick is not a plan. A plan looks more like this:

1. No built-in drive to survive. Aim for systems that accept being corrected or shut down.
2. Humans who can see and stop it. Oversight that is real, not a checkbox.
3. Checking what it actually optimizes for. Interpretability and tests, instead of trusting what the model says about itself.
4. Caps on resources. Limits on the compute, money and permissions any one system can gather.
5. No single winner. A balance of power, so no one actor, human or AI, controls the key resources.
6. A bigger pie. The scarcity argument gets weaker the more there is to share.

None of these is enough alone, and I’m not sure all six together are. I’ll also say plainly that I wrote parts of this with an AI model, so I’m not a neutral observer either. That’s part of why I want the structure and not just the promise.

If you were writing the rules, what is the one thing you would never let an AI trade away, even to survive?