16%
For a week I believed I had found the number my product had to beat. The way I was wrong about it is more useful than the number.
Date
Category
Writer
For about a week this summer I believed I had found the number my product had to beat. I was wrong about what the number measured, and the way I was wrong is considerably more useful than the number.
This is a post about a mistake I made in public, in a document, with confidence.
The study
Ofer Bergman, Steve Whittaker and Joel Schooler, "Out of sight and out of mind: Bookmarks are created but not used," Journal of Librarianship and Information Science 53(2), 2021, pp. 338–348.
As far as I can find, it is the only controlled measurement of whether people actually return to the things they deliberately saved. Everything else in this area is self-report, which is worthless here, because asking someone whether they use their bookmarks is asking them to describe a habit they have never observed.
The design is clean. Fifty participants. Twenty-one target URLs each. Five of those bookmarked, giving 250 bookmarked targets in total. Then measure how the targets were actually retrieved.
Measure | Result |
|---|---|
Participants | 50 |
Targets per participant | 21 |
Bookmarked targets (total) | 250 |
Retrieved via the bookmark at all | 41 (16%) |
Of those, via the bookmark menu hierarchy | 9 |
Of those, from a bookmark visible in the bar | 32 |
Retrieval advantage over non-bookmarked visited pages | None measured |
Table 1. Bergman, Whittaker & Schooler (2021), principal results.
The authors' summary is blunt: unless bookmarks were highly visible, participants did not use them.
What I did with it, and why it was a category error
My first reading was the obvious one. Here is a base rate. Sixteen per cent of deliberate saves get used. Beat sixteen per cent and the product works.
I wrote that into a document as "the number to beat." It sat there for a week looking authoritative, because a number from a peer-reviewed study looks authoritative regardless of whether it applies.
It took a deliberate attempt to attack my own conclusion to see the problem.
The study measures directed re-finding. Participants were asked to retrieve twenty-one specific target URLs. They knew what they were looking for. The task was: go and get that thing.
That is not what my product claims to do. I had already established, months earlier and in writing, that this product cannot and should not win directed retrieval. Search is a commodity, the incumbents are excellent at it, and a smaller team building a worse search box is not a strategy.
What the product claims is different: that you come back and encounter what you left. Not "fetch item X" but "here is what was here." Those are different acts with different base rates and, critically, different failure modes.
A number from a study that measured a different task is not evidence about your task. It is borrowed authority.
Sixteen per cent is not a number I can beat, and not a number I can fail. It does not apply. The document now carries a correction above it, and I would rather have the correction on the record than a clean-looking report.
Why this mistake is easy to make
I want to dwell on the mechanism, because I do not think I am unusual here.
A citation does three things at once, and we tend to notice only the first two. It supplies a fact. It supplies credibility. And it supplies a frame, which is the set of assumptions the study was built on, and which travels silently along with the number.
When I took the sixteen per cent, I took the frame with it: that saving is for later retrieval, that retrieval is directed, that success means finding the specific thing you set out to find. None of those match my product, and none of them were stated in the sentence I quoted.
The practical defence I have adopted since is a single question, applied before any external number enters a document:
It is a boring check. It would have caught this in about ninety seconds.
What actually survives
Strip out the misapplied number and something better is left, which is the mechanism.
Thirty-two of the forty-one successful retrievals came from bookmarks that were visible in the browser bar. Nine came from navigating the organised hierarchy. That ratio, roughly three and a half to one, is the finding.
It says, with actual evidence behind it, that visibility drives reuse and that filing does not. The authors describe filed-away saves as essentially invisible. Not lost. Invisible. They are exactly where you put them, and that is the problem.
This is directly usable, and it is a stronger claim than the percentage was. It says that a product whose value depends on people navigating back into a saved hierarchy is betting on the weak mechanism, and one whose saved things are simply on screen when you arrive is betting on the strong one.
It also explains something otherwise puzzling about the products that succeed in this space. The two most-used "save it somewhere" behaviours most people have are a system quick-note and messaging themselves in a chat app. Neither is a product. Both are features on a surface that is already open. Their visibility is free.
The counterweight I owe the argument
It would be convenient to stop there, since my product does put everything you left on screen the moment you open it.
But visibility has a precondition nobody discusses: the surface has to be open. A standalone place must be chosen before it can be visible. That is a structural disadvantage, it is not fixable by design, and it is the largest single problem in front of me.
Though it has an underrated other side, which I only found by forcing myself to argue the opposite for an afternoon.
Bundled surfaces win on visibility because they are already open. And everything in them arrives mixed with everything else, which is precisely why those surfaces eventually grow drawers, stars, pins and filters. Their advantage creates their problem.
A standalone place has one property nothing bundled has: everything in it was put there deliberately. There is no noise to separate from signal, because nothing arrived. That is real, currently unmeasured, and I am not going to claim it as a win until it is.
The habit I am trying to build
The specific failure here was not misreading a paper. It was that I reached for a number because I wanted one.
A project with no human evidence is uncomfortable to write reports about. A percentage from a journal fills the hole in the page where evidence should go, and it feels like progress. The temptation is strongest exactly when the evidence is weakest, which is exactly when the standard needs to be highest.
The correction is written above the original in the document rather than replacing it, because a reader in two years needs to know not just what I currently believe but what I believed before and why it changed. Deleting the mistake would make the document cleaner and considerably less useful.
Sources
Bergman, O., Whittaker, S., & Schooler, J. (2021). Out of sight and out of mind: Bookmarks are created but not used. Journal of Librarianship and Information Science, 53(2), 338-348. DOI 10.1177/0961000620949652
Related reading
Latest Articles.
© Moc Mallen Studio
Thoughts, ideas, and perspectives on design, simplicity, and creative process.
