Never Create a User Action So You Can Measure It

I built a question into the product. One line, at the top, appearing occasionally:

오늘 JEFIELD를 왜 열었어? Why did you open JEFIELD today?

It looked reasonable. One sentence, dismissible, unobtrusive. It would have told me the single thing I most needed to know and could get no other way.

I saw it on screen and removed it the same day.

What was wrong with it

The question is not neutral, and it cannot be made neutral.

A product that asks why you opened it has just told you several things at once: that opening it is notable, that there is a right answer, that someone is watching, and that this is a thing being studied rather than used.

And the person answering is no longer performing the behaviour I wanted to observe. They are performing a new behaviour, which is answering a question inside a product. Whatever I learned would be about that.

This is the observer effect, and in software it is not a subtle statistical correction. It is total. The measurement does not perturb the behaviour slightly. It replaces it.

NEVER CREATE A USER ACTION SO THAT YOU CAN MEASURE THE USER ACTION.

The full list of what this forbids

Written down, because each item individually sounds defensible and the defensibility is the trap.

Table 1. Eight mechanisms with one shared structure.

The common structure is this: each one adds something to the product whose reason for existing is my need to know something. The user pays the cost. I collect the benefit. And the thing I measure has been altered by the measuring, so I am paying a real price for a corrupted number.

The streak case is the clearest. A product that adds streaks and then reports improved retention has not discovered that people want to return. It has discovered that people do not want to lose a streak. Those are different findings and only one of them is about the product.

Then where does observation happen?

Outside the product. That is the whole answer, and it is considerably less convenient than it sounds.

It means asking a person afterwards, in conversation, about something that already happened. And it means being disciplined about which questions are even permitted, because most of the obvious ones are leading.

Allowed, after genuine unprompted use

• 이걸 다른 데 남기는 것과 뭐가 달랐어? How was leaving this here different from leaving it somewhere else?

• JEFIELD가 없었으면 어디에 했을 것 같아? If this did not exist, where would that have gone?

• If this disappeared tomorrow, where would that thing go?

All three are about behaviour and alternatives. None of them can be answered by being nice to me, because none of them has a flattering answer available. "It would have gone in my notes app" is a perfectly natural response and it is the most useful thing I could hear.

Banned

• 네 공간처럼 느껴졌어? Did it feel like your space?

• 살아있는 것 같았어? Did it feel alive?

• 고양이가 좋았어? Did you like the cat?

Each of those hands the person my hypothesis and asks them to agree with it. They will. People are kind, especially to someone who built something and is standing in front of them, and especially in Korean, where the grammar of the question already contains the expected answer.

Also banned: explaining the philosophy before they use it, and seeding the behaviour. Saying "use it as your notes app" guarantees a note and teaches me nothing.

What you give up

Speed, mostly. Real observation is slow, small-n, and produces no dashboard. There is no chart at the end of it.

For long stretches the honest state of the evidence is UNKNOWN, which is an uncomfortable thing to write in a report and a worse thing to say to anyone who asks how it is going.

I write it anyway. With no real users the line in every report reads: REAL USER EVIDENCE: NOT YET AVAILABLE. Never compensated for with personas, never softened, never quietly replaced by a proxy metric that happens to be available.

The rule about personas is worth stating separately because it is the most common substitution. A persona document is a prediction. Thirty-six invented people across eleven situations is still zero people. It can be a useful design tool and it is never evidence, and the moment a persona's behaviour appears in a results section the document has stopped being honest.

The broader version

This applies well outside product work, and once you see the structure it is difficult to stop seeing it.

Engagement metrics measure engagement with the measuring apparatus. A client asked whether they are happy will report on the relationship rather than the work. An artist who checks the numbers is, from that moment, making something slightly different, and usually cannot tell which parts.

The question worth asking of any metric:

Would this number exist if I were not looking at it?

If the answer is no, you built the number. And you are now optimising a thing you made, which is a closed loop that can run indefinitely while producing nothing.

What replaced the question

Since I removed the in-product question and refuse to build a substitute, there has to be something. There is, and it is deliberately low-technology.

When a real capture moment happens, four lines by hand, in a notebook:

1. What was it?
2. Why did I keep it?
3. Where would it otherwise have gone?
4. Was it ever useful afterwards

1. What was it?
2. Why did I keep it?
3. Where would it otherwise have gone?
4. Was it ever useful afterwards

1. What was it?
2. Why did I keep it?
3. Where would it otherwise have gone?
4. Was it ever useful afterwards

No instrument can capture the decisive variable, which is whether the moment occurred and whether the product came to mind. That exists only in a person's head. The four lines are the only honest record available, and they cost nothing, and they change nothing about the behaviour they describe.

It is not a system. That is the point.

Related reading

• A Green Light That Proves Nothing

• The Door That Was Never Open

• The Graveyard

• Zero-Effort Capture, High-Effort Recall

More from the Moc Mallen Studio journal