Deleting Your Data Doesn't Delete What It Learned

Your code ships to a public repository. Your writing goes out under your name. The deck gets sent to forty people, half of whom forward it. So when the conversation turns to AI and privacy, you check out — none of your work is secret, and you’ve got better things to worry about.

I want to make the case that you skipped the interesting part.

The output isn’t where your value sits. The value is in how you got there: the three approaches you tried before the one that worked, the thing you had to look up because you didn’t know it yet, the way you argue with yourself before you commit. That’s the part that goes into the chat window. And unlike the output, it was never meant to be published.

Every privacy promise is about the wrong noun

Look at how AI companies actually talk about privacy. Retention windows — 30 days, five years. Delete buttons. Zero-retention agreements. Data-processing addenda. Export tools.

Every one of these answers the same question: is my data still on your disk?

That’s a storage question. It’s a fair question. But it isn’t the one that should be keeping you up, because there’s a second question sitting underneath it that none of those promises touch: what did you work out about me while it was there?

Storage and inference are different things. A file can be deleted. A conclusion drawn from that file is somewhere else entirely — folded into a system, no longer traceable to the thing it came from. You can delete the input and leave the conclusion standing.

That’s the whole argument, and everything below is just detail.

Ordinary writing gives up more than you’d guess

Here’s the part that surprised me.

A team at ETH Zurich tested what language models can work out about people from nothing but their public writing — no account data, no tracking, no metadata. Just prose, in this case ordinary Reddit posts. The models inferred attributes like location, income and sex with up to 85% top-one accuracy, and 95% if you let them offer three guesses.

Two details make that land differently.

The first is cost. Doing this by hand is a research project — a person reading carefully, building a picture. The models did it at roughly a hundredth of the cost and a two-hundred-fortieth of the time. Something that used to be expensive enough to reserve for targets worth investigating is now cheap enough to run on everyone, continuously.

The second is that the obvious defences didn’t hold. The researchers tested anonymising the text and relying on the models’ own safety training, and reported both as currently ineffective. Stripping names out doesn’t help much, because the tell was never the name. It’s the vocabulary, the reference points, the turns of phrase — the texture of how a particular person writes.

That research looked at a public forum, not at anyone’s chat logs. It doesn’t tell you a specific company is doing this to your conversations. What it establishes is the capability: prose is far more revealing than it feels, and the cost of reading it that way has collapsed.

The commercial version already shipped

Capability is one thing. Here’s the practice.

OpenAI started testing ads in ChatGPT on 9 February 2026, on the free and Go plans — paid tiers don’t show them. Ad selection uses the context and intent of your current conversation. Switch ads personalization on, and it draws on signals from your wider use of the product; with memory on, OpenAI says ChatGPT may use saved memories and reference recent chats when it picks an ad.

Meta went first. Since 16 December 2025 it has used your interactions with its AI features to personalise the content and ads you see across its apps.

Both companies are clear that advertisers don’t receive your conversations — OpenAI’s page says advertisers get no access to your chats, chat history, memories or personal details, only aggregated performance figures. I believe them, and it doesn’t change the shape of the thing. The inference happens on the platform’s side. The advertiser doesn’t need your chat log; they need the conclusion drawn from it.

Two honest qualifications, because they matter and they cut against the drama.

If you’re in Europe, the personalisation part isn’t live for you. OpenAI’s page says personalised ads aren’t initially available in the European Economic Area or Switzerland, and Meta’s rollout excludes the EU, the UK and South Korea. That’s not the companies being generous — it’s European law doing exactly what it was written to do. What still applies everywhere: ads are matched against your current conversation even with personalisation off, and chat content has become ad inventory at all. That’s the direction of travel, and geography is a delay, not an exemption.

And there are real limits in the design. OpenAI states ads aren’t eligible to appear near sensitive or regulated topics — personal health, mental health, politics — and that hiding, reporting or clicking an ad doesn’t get written back into your memories. Worth knowing, and worth not overstating.

Why the delete button doesn’t reach it

So you delete the conversation. Turn off training — worth doing, and worth knowing it’s on by default for consumer ChatGPT accounts, and that Anthropic’s consumer terms now run to five-year retention for people who allow training. Export everything. File an erasure request.

All of that operates on the inputs. Does any of it reach what was derived?

Europe’s own data-protection regulators looked straight at this. In Opinion 28/2024, the European Data Protection Board concluded that AI models trained on personal data cannot, in all cases, be considered anonymous. You have to assess each model on its facts — and to call one anonymous, both the chance of pulling personal data back out of it and the chance of getting that data through queries have to be insignificant, for every person in the training data.

Read that as the regulator saying the quiet part plainly: the learning is not automatically separable from the people it was learned from.

The same opinion follows the logic where it goes. Where a regulator finds an infringement, the corrective measures available can extend to erasing the unlawfully processed part of the dataset — or, where that isn’t possible and depending on the facts and on proportionality, to erasing the whole dataset used to build the model, and the model itself.

That’s a description of what regulators can reach for, weighed against proportionality and against remedies like retraining. It isn’t a prediction that models get deleted; as far as I know that hasn’t happened. But the reason the remedy has to be able to reach the model is the point I’ve been making for the last thousand words, now stated by the people who write the rules: the model is where the learning lives. If deleting the file were enough, the remedy would stop at the file.

What actually helps

Not much of the usual advice, is the honest answer. “Don’t paste anything sensitive” doesn’t survive contact with real work — the whole reason the tool is useful is that you give it the actual problem.

What changes the picture is structural, and there are only really three levers:

Where it runs. A model running on infrastructure that answers to your jurisdiction is a different risk than one that doesn’t. This is the lever Europeans have been getting for free in the ad case above, and it’s worth understanding as a lever rather than a lucky break.

Whether a profile is built at all. Not “is it retained” — whether the system is constructed to accumulate a picture of you across sessions in the first place. Memory features are genuinely useful and they are also exactly this mechanism. Both things are true.

Whether you can see the reasoning. A system that shows its work leaves you with the ability to check it. One that just produces answers leaves you with a dependency, and the dependency is what gets studied.

What to do now

1. Check whether training is on. For consumer ChatGPT it’s on by default. Two minutes in settings, and it’s the single highest-value toggle on this list.
2. Look at your memory settings — not to turn them off necessarily, but to see what’s actually accumulated in there. Most people have never opened it, and it’s a clarifying five minutes.
3. If ads are showing, open ad personalisation. Turning it off still leaves ads matched to the current conversation, but stops the wider signals feeding it.
4. Use temporary or incognito chat modes for exploratory thinking — the half-formed reasoning, the thing you’re embarrassed not to know. That’s the material this whole post is about.
5. Separate the accounts. Client and employer work shouldn’t run through a personal account on a consumer tier; the terms are different and so is who carries the liability.
6. For anything under a duty of confidence — client files, patient data, anything you’re holding on someone else’s behalf — check what your provider’s terms actually say about processing, rather than what the marketing page says about privacy. Those are frequently different documents.
7. Ask the storage question and the inference question separately. When a vendor tells you they don’t retain your data, that’s an answer to one of them. Ask the other one out loud and see what comes back.

The output was never the valuable part. It’s worth acting like you know that.