Why most self-experiments end in "nothing found"
Three small studies of self-experiment apps show why short tests rarely give a clear answer, and why "nothing found" is not "no effect".
By Lomba 4 min read
The simplest test of a habit is to do it for a week or two, watch one number, and see if it moves. Researchers have built apps to help people run that kind of test on themselves, and then counted how the tests ended. In a sleep study from Brown University and two more small studies, most ended without a clear answer.
Here is what they did, what they found, and the one idea we think matters most.
What the studies did
- Who
- In the sleep study, 365 people downloaded an app and tracked at least one night. In the food study, from the University of Washington, 15 patients with irritable bowel syndrome each tested one food for 12 days. In a six-week pilot at MIT, 13 people aged 18 to 27 each tested a habit they chose.
- What was compared
- Each person against themselves: days with a habit against days without (in the pilot, different amounts of a habit), in an order the app set. The sleep app showed a result after 4 nights for one group and 10 for the other.
- What was measured
- How many people finished, how closely they kept to the schedule, and how many got a clear result. We report the food study for its test design, not for anything about food.
- Who paid
- Universities and public research agencies, plus two technology companies' research programs in the food study. Each team built the app it tested. The food study paid each participant up to $175, and the pilot gave a $20 gift card.
Four takeaways
Most people stop before the test ends.
In the sleep study, most people tracked a single night. 31 people (17%) in the 4-night group reached a result, and 17 people (7%) in the 10-night group. The paper does not print how many people were in each group. In the pilot, 1 of 13 finished a test.
The schedule is rarely kept.
Sleep-app users followed the night's instruction 60% of the time on average. In the pilot, people were in the target range on 22.5% of days, which the authors name as the main reason only one finished. The food study, guided by its researchers, was the exception: 12 of 15 reported keeping to it every day.
A finished test is often still unclear.
In the food study, one person of 15 got "strong evidence", on one of her five symptoms. Three got "possible evidence", and "no evidence" was by far the most common result. Participants had been asked not to test known triggers. The sleep app showed each finisher a likelihood. By a conventional test, the authors report, none would have reached a statistically significant result.
"No evidence" gets read as "no effect".
Many participants in the food study took "no evidence" as proof that the food was not a trigger. The authors did not. With few days, they write, a wrong conclusion of "no effect" is more likely, and a small effect needs more days to show. Figure 2 works that out with a textbook formula.
Sleep study
365 people, 4 or 10 nights
Food study
15 people, 12 days
Six-week pilot
13 people, six weeks
One dot is one person. In each study, most people ended without a clear result. Point at a row for the paper’s own count.
A test of 10 nights, 5 each way, can tell a gap of at least 17.7 min from noise: 1.77 times one night’s swing. The sleep study’s average difference, 7 min, is smaller than that.
The main takeaway
"Nothing found" is the usual result of a short test, and the honest one. It means "not enough to tell", not "no effect".
The reason is arithmetic. Days differ for many reasons, and a short test has few days on each side to average that out. By a standard textbook calculation, a 4-day test finds a gap four times in five only if the gap is at least 2.8 times the measure's usual swing from one day to the next. A 10-day test needs 1.8 times. A gap the size of that swing takes 32 days.
What Lomba does with that
Lomba compares the mornings after a habit with your other mornings, on your ordinary days. How it works shows each step, and these are the four answers it names.
Held up. The gap cleared the luck bar and passed every check. Still an association, not a cause. Set aside. A gap at first, then it failed a check. Lomba says which one.
Still collecting. Not enough days yet. No gap is shown. No clear gap. Enough days, no clear difference. That isn’t proof of no effect.
The last two are both kinds of "nothing found", and Lomba keeps them apart: too few days, or enough days with nothing clear. Neither is worded as "no effect".
This post has limits, and so does Lomba:
- Small studies. 15 people, 13 people, and 48 who reached a result out of 365. Each team tested its own tool, and the studies were built to try app designs, not to count how often self-experiments succeed.
- A textbook figure. Figure 2 is a standard formula for two equal groups of independent days. It is not how Lomba checks a pattern, and a real test needs a larger gap than it says.
- A pattern, not a cause. Any answer from Lomba is a pattern in your own days, not proof of cause, and not medical advice.
Sources
- Daskalova N and others. SleepBandits: Guided Flexible Self-Experiments for Sleep. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems.
- Karkar R and others. TummyTrials: A Feasibility Study of Using Self-Experimentation to Detect Individualized Food Triggers. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems.
- Taylor S, Sano A and others. QuantifyMe: An Open-Source Automated Single-Case Experimental Design Platform. Sensors, 2018.
The food study also has a free copy on PubMed Central, as PMC5432136. The pilot is open access.