What should we insist on before turning it on?

Being able to see what has been stored and to delete individual entries. Without both of those, any mistake it records becomes a permanent influence on everything that follows, applied silently and never restated in a form you would have the chance to notice.

Why is a stored mistake worse than a wrong answer?

A wrong answer is in front of somebody who can correct it. A wrong stored fact is applied consistently and quietly across later work, and it reads as the system understanding your situation rather than misunderstanding it, so nobody looks for it.

Does shared memory across a team help?

It can, and it changes the shape of the risk rather than removing it. What one person establishes becomes what the system believes for everybody, so a useful convention spreads instantly, and so does an offhand remark that was never meant to be a standing instruction.

Glossary

Long-term memory

Facts a system keeps between separate tasks, which make it more useful over time and become harder to question the longer they sit there.

In plain terms

Things the system remembers about you between conversations. It saves repetition, which is genuinely useful, and it means anything it got wrong is now being applied to everything you do without being mentioned again.

Why it matters

Because a stored mistake behaves differently from a mistake in an answer. A wrong answer is visible and gets corrected; a wrong stored fact is silent, is applied consistently, and looks like the system understanding your situation rather than misunderstanding it.

How it works

Persistence is the whole difference. Something written down once is reused in every later task without being restated, which is precisely why it helps and precisely why an error in it compounds instead of passing.

What gets stored is a judgement somebody made. A rule deciding which parts of a conversation are worth keeping is doing interpretation, and interpretation applied automatically to everything you say will sometimes keep a passing remark as though it were a standing preference.

The properties that matter are inspection and deletion, not capacity. Being able to see what it believes about you, and remove an entry, is what makes a stored mistake correctable, and a system offering neither has made its errors permanent.

Corrections do not reliably overwrite. Saying the opposite of a stored fact adds a statement rather than removing one, so a system can carry both and behave inconsistently depending on which it happens to draw on.

It ages without any signal. A preference recorded a year ago is applied today with the same confidence as one recorded this morning, and nothing about the way it is used indicates which is which.

It creates an obligation nobody planned for. Stored facts about a person are records about that person, so a request to see or delete what is held reaches memory as much as it reaches any database, and a tool without a way to answer that has quietly added an obligation you cannot meet.

The useful habit is to read it occasionally rather than to trust it. Ten minutes spent looking at what a system has decided about you and your work is usually surprising, sometimes corrective, and is the only way any of this surfaces without an incident.

Shared memory across a team is a different proposition entirely. What one person told it becomes what it believes for everybody, which is either a genuine advantage or a way for one careless statement to shape everybody's results.

Two mistakes, two lifespans

The third line on the right is the one that makes these hard to find. A system applying a stored fact does not announce it, so the output simply looks well-informed: it knows your team uses a particular format, it knows you prefer short answers, it knows something about your situation that you no longer remember telling it. When one of those is wrong, the results are subtly off in a way that is easy to attribute to the model rather than to a record you cannot see. That is why the practical requirement is not accuracy in what gets stored, which cannot be guaranteed, but visibility. A screen listing what the system believes about you, which you can read in two minutes and edit, converts an invisible failure into an ordinary one. Where a tool offers no such screen, the sensible position is to treat its memory as something that will eventually need clearing entirely, and to find out in advance whether that is possible.

Seen in the wild

An assistant carrying a preference from months ago into unrelated work.
ChatGPT
An agent that recalls a working arrangement across separate tasks.
Lindy
A workspace where what one person established becomes everybody's default.
Notion AI

Common misconceptions

People assume

Correcting it removes the wrong fact.

In fact

A correction is usually another statement rather than a deletion, so both can persist. The system may then behave inconsistently depending on which one it draws on, which is harder to diagnose than the original error.

People assume

More memory makes it better.

In fact

More memory makes it more consistent, which is an improvement only if what it stored was right. The properties worth having are the ability to see and remove entries, and neither is a matter of how much can be kept.

Questions

What should we insist on before turning it on?: Being able to see what has been stored and to delete individual entries. Without both of those, any mistake it records becomes a permanent influence on everything that follows, applied silently and never restated in a form you would have the chance to notice.
Why is a stored mistake worse than a wrong answer?: A wrong answer is in front of somebody who can correct it. A wrong stored fact is applied consistently and quietly across later work, and it reads as the system understanding your situation rather than misunderstanding it, so nobody looks for it.
Does shared memory across a team help?: It can, and it changes the shape of the risk rather than removing it. What one person establishes becomes what the system believes for everybody, so a useful convention spreads instantly, and so does an offhand remark that was never meant to be a standing instruction.