Every data point has three states: I have it, there really was nothing, and I don't know. The last two get drawn the same way on a chart, as a zero, and they mean the opposite.
Detector question
Could the number I'm about to show, or the "everything's fine" I'm about to believe, come from a query that failed, an empty table, a denied permission, or an output I didn't read in full (see lesson 44)?
Warning sign
"If it fails, I'll just set it to zero and move on."
Example and what to do
Example
A query fails because of an expired permission, and the dashboard shows "0 sales" for the region. Nobody questions it: a zero looks like data.
What to do
If the read fails, show the error on the card itself, not a gap or a zero. And a counter that says "0 problems" is only trustworthy if it first found a problem you planted on purpose.
Two things with the same name are not the same thing
Numbers and data
"Active customer," "date," "the boss," "the threshold": the same word often hides two definitions. When two reports don't match, it's almost always this.
Detector question
Is there a rule that's true for one of the two things and false for the other? And if I combine two figures in one calculation, do they come from the same universe, the same scale, and the same period?
Warning sign
"It's the same date" · "the view already filters it" · "they're both in pesos."
Example and what to do
Example
"Date" can be the event date, the load date, or the partition date. Or "the boss": the person's current boss, or the boss they had when the thing you're measuring happened.
Another example
One campaign table had the spend for the whole company, but the revenue came from a sample of one purchase in every hundred. Both columns were labeled "pesos," and the return came out negative for every campaign.
What to do
Give each variant a different name in the code and on the screen, even if it's ugly. And before dividing or subtracting two figures from different sources, write down where each one comes from: if a result comes out the same in every case (they all lose, they all win), suspect the scale before the business.
A month that hasn't finished always looks worse than a closed one. It's not a trend: it's the calendar.
Detector question
Does this calculation include a period that hasn't ended yet?
Warning sign
"You can already see the trend."
Example and what to do
Example
The sales curve "drops" at the last point, which is the month that's only halfway done.
What to do
Cut the indicators at the last closed period, or mark the partial period on screen. And keep the two clocks apart: group-level numbers use closed months; named individuals use the current day.
Many systems store only today's value. If you build a "historical" variable on top of that field, you're measuring the present and dressing it up as the past.
Detector question
Does this field keep its previous value when it changes, or does it replace it?
Warning sign
"The column's right there, I'll just use it."
Example and what to do
Example
The system stores only each person's latest boss. For people who have left, that boss is frozen in place, so "turnover of each boss's team" ends up measuring "this team no longer exists."
What to do
Ask the source whether the data is a snapshot or a history before using it to look backward.
If a single variable separates the people who left from the people who stayed with almost no errors, it most likely has the outcome itself hidden inside it.
Detector question
Could my quality measure be going up because the model is guessing when something happened instead of who it happened to?
Warning sign
"It came out almost perfect!" · "let's just adjust for year and move on."
Example and what to do
Example
A model suddenly gets much better. It turns out one variable is 1 for everyone who left and 0 for everyone who stayed, because the source fills it in differently for each group. Another variant: the model learns in which year something happened, not who it happened to.
What to do
Test forward: train on the past and evaluate on a later period. And before adjusting for something, ask whether that something is what decides the result you want to predict.
If the rule for "what counts as active" is repeated in every query, the copies will drift apart. It's not a question of if, but when.
Detector question
If I change this definition, how many files do I have to touch? If it's more than one, the rule isn't set up in one place.
Warning sign
"I'll put it here too, it's just one line" · "that number won't change".
Example and what to do
Example
A validator had the count of reference items typed in by hand as 43. When the document grew to 47, it rejected the four new ones as nonexistent. Now it counts the items in the document itself.
What to do
Put the definition in one spot that every query goes through, and have all of them read from there. A hand-typed number that copies a value from somewhere else is also a repeated definition.
Not to be confused with
Repeating an explanation on purpose, in learning material, is not copying a definition. What you shouldn't copy is the rule that computes the value.
A process declares what it plans to do, but publishes what it actually did, and the two don't always match. Whoever consumes the output should read the result.
Detector question
Is this built from a process's configuration or from its published result?
Warning sign
"The declared list and the model are the same thing."
Example and what to do
Example
A screen builds its controls from the list of variables the model was going to use. Training dropped one of them, and for days there was a control that did nothing.
What to do
Derive it from the published result. And whatever can't be derived should fail loudly instead of quietly disappearing.
When you change a piece of data that others read by its name, the list of who reads it has to be written down. Memory always leaves one out.
Detector question
Where did I get the list of who uses this: from a search or from memory?
Warning sign
"I already covered the three places where it's used" · "they already saw that" · "I already added everything."
Example and what to do
Example
The labels on three screens were updated after a model changed. The fourth, which nobody remembered, kept offering options from the previous model.
Another example
When writing a chapter, it was stated from memory that the reader already knew three functions from an earlier chapter. A search showed that the chapter taught only one, and that another chapter had already explained the topic presented as new.
One more example
A change was saved with the list of files written from memory, and one was left out: the one that made the new thing work. The list of pending changes, reviewed afterward, showed it.
What to do
Keep a written map of impacts by type of change. Go through it in full, saying "changes" or "doesn't apply because…", and if the change creates a new consumer, add it in the same change. Before citing a source, copy the sentence you're asserting from that source: what someone says another person did or saw also needs to be checked.
One state lives in several records: move them together
Git and process
Closing a pending item usually involves three places: the pending-items list, the test that watches it, and the document that describes it. If you move one, the others contradict it.
Detector question
What other places assert this state? Name them before saving.
Warning sign
"I'll update the other side later."
Example and what to do
Example
A pending item was marked closed in the list, but the test watching it was still marked "expected to fail" and the document still described it as open. Three places, three versions.
What to do
The unit of work is the complete set, in the same change.
If your product has two versions of the same screen, for two kinds of user, two regions, or two products, an improvement to one is a pending improvement to the other.
Detector question
Does what I'm touching have a twin that users see as "the same module"?
Warning sign
"They didn't ask me for it on the other one."
Example and what to do
Example
The desktop version gained a date filter. The mobile version, which people see as "the same screen," went without it for a month.
What to do
Don't close the change until you've checked the twin and written either "same" or "doesn't apply because…".
If you remove something because the data wasn't reliable, that same claim probably still lives in other texts, written in other words.
Detector question
What other texts say the same thing as what I just removed?
Warning sign
"I already removed the chart."
Example and what to do
Example
An unreliable chart was pulled, but a "key findings" card and an action plan still recommended what the chart had said.
Another example
A chapter plan was renumbered from 7 to 15. Two lists in another document still pointed to "7.4" and "7.7" from the old plan, which now were different topics. When you replace a plan, the old numbers are claims too.
What to do
Search by topic, not by identifier. Echoes don't have a variable name.
Running the query doesn't prove the number is right
Numbers and data
"It didn't error" is not "it's correct." Compare against something independent of your own calculation.
Detector question
What outside my own code am I going to check this result against? And for anything the text says the system does, whether in an example, a rule, or the explanation of an option, did I run it on the real engine?
Warning sign
"It runs on my machine" · "it didn't error" · "that's how it works, I just explained it."
Example and what to do
Example
A multiple-choice question claimed that comparing a number with a quoted text string throws an error. In some other engines it does; in the one people actually used, the query ran and returned rows.
Another example
One text correctly explained that a named block only lives inside its own query, but its own example assumed three separate queries shared it. When run, the second query threw an error: the table didn't exist.
One more example
Every example on one page was run, and three sentences still turned out false: the explanation of an option, written at the end, described what a line does that nobody had run; a rule for reading errors ("if it says this, then that's missing") was contradicted by another step on the same page; and the introduction said a function "returns a result" when it doesn't.
What to do
Reconcile against a known figure, the source system, or the previous period. Test in the real environment: with its permissions, its libraries, and the copy of the code that actually runs. Every claim about how the system behaves gets run once before you write it down, including the one that shows up in passing in an example or gets written last. If the claim is a general rule ("only," "always," "never"), find the case that would break it and run that too: "converting to an integer only reads integers" was checked with a text string that had decimals, and it turned out false for a number with decimals. When you state a fraction ("five in six"), name the whole it's a fraction of; and a made-up scenario gets checked too: the size, the difference, and the p-value have to be consistent with each other.
A checker that never saw the error verifies nothing
Tests and checks
A test that always passes may be broken. The only way to know is to watch it fail against the real error.
Detector question
Did I test my check against the error in the forms it would really appear? And against the correct case?
Warning sign
"It already passes, so it's set up right."
Example and what to do
Example
A test that banned a word only looked for the lowercase version; with capital letters or an accent, it passed.
Another example
37 duplicate records were seeded for an exercise and the count came back 92: the data already had natural matches. A check that compared the count with what was seeded, before saving, caught it.
One more example
A rule that detected indented code flagged a line that started with a text: the case that should pass did not have that shape. And a test that the text comparison worked passed even though the comparison was broken, because another review rejected the error first.
What to do
Calibrate it with two cases: one where it must find something and one where it must not, and run it for real on the day it's born. The good case matters as much as the bad one: a checker that rejects what's correct is broken too, and that is harder to notice because it looks strict. The good case includes the real forms of what's correct, including the ones you didn't write yourself: another order, another name. Each check calibrates itself: if another one catches the error first, yours was never tested. And a review that repeats what it already read can't fail (a "second run" that rereads the same file): replace it with a reconciliation that does fail if something is wrong.
The probe doesn't travel the channel it investigates
Tests and checks
If your check goes through the same pipeline that could have caused the damage, the damage affects the check too, and it answers the same whether things are right or wrong.
Detector question
Does the transformation I'm investigating also transform my test?
Warning sign
"I check it by sending another query just like it."
Example and what to do
Example
They were looking for an accent problem by sending queries with accents through the same route that was damaging them.
A test that switches itself off doesn't report that it switched off
Tests and checks
The branches that say "if I can't find the data, I skip the test" activate exactly when the world changes, and they stop watching without anyone noticing.
Detector question
Does my test have a silent exit that activates with a legitimate change?
Warning sign
"If I can't find the anchor, I skip the test."
Example and what to do
Example
A test located a table by its title and, if it didn't find it, skipped. The title changed, and the test passed for weeks without checking anything.
Variant
A check that falls over at the first broken file leaves every file after it unwatched. A validator that crashes, instead of reporting "this page can't be read, line 12", switches off the whole review.
What to do
If what's missing is a tool, skipping it is honest. If what's missing is the data being watched, the test must fail. And a file that can't be read is a finding with its line, not the end of the review.
Your checker's filter usually excludes exactly what matters
Tests and checks
Every automatic review filters for what's "relevant", and that filter can leave it blind exactly where the problem is.
Detector question
What does my filter leave out? Write it down before you believe the result.
Warning sign
"It found 48 differences, I already have the list" · "the test goes through all of them."
Example and what to do
Example
A design auditor reviewed only elements with no children. The labels that needed review had a help icon inside them, so it never saw them.
Another example
A test went through every page always marking the first option and running each query as it came. It always passed, and it never checked that the correct answer was accepted.
One more example
To decide whether someone changed a text, whitespace was ignored. That left out exactly the case of the page that explained how to lay out that text with spaces and line breaks. Normalizing is also filtering.
What to do
Write down what the filter leaves out and check a sample of the excluded items by hand. And when you learn that a filter left you blind, look for the other similar filters you already wrote: a review that stripped all spaces from the code joined two words into one and rejected correct code, shortly after the same lesson had been learned from a different normalization.
Checking the source is not checking what gets delivered
Tests and checks
If you check X and the user gets Y, derived from X, there's code between the two that nobody is looking at.
Detector question
Does anyone check what reaches the screen, or only what comes out of the source?
Warning sign
"It comes from data that's already verified" · "the file is fine."
Example and what to do
Example
They verified that each published result was correct, but a version built from it traveled to the screen, one that used a different rule for missing data.
Another example
A reviewer read the source file of each page and approved it. A table that stacked into cards on mobile, and could no longer be compared, only showed up when the page was looked at.
What to do
Check at the point where it reaches the person: the page as it is seen, the file they download, the message they receive (see also lesson 41).
"They're already the same" requires having compared every property you can see, not just one.
Detector question
Did I compare one property or all the ones the user sees?
Warning sign
"The size matches now, so the design is the same."
Example and what to do
Example
The design of two screens was made to match over four rounds: first the size, then the typography, then the colors, then the label color. Each delivery said "it's done."
Another example
The illustrations in a material were reviewed for content and size, but not for contrast: two text colors ended up below the minimum readable level in four images.
What to do
List all the differences before fixing the first one. And in your notes, the verb gives you away: "it's fixed" and "it becomes" are intentions; "it says" and "it shows" are measurements.
If you locate your change by searching for text the user reads, like a title or a label, another improvement will stick something on top of it, and your change will stop finding it.
Detector question
Can the text I use as an anchor exist for another reason, or can something grow inside it?
Warning sign
"I'll anchor on the title, it's unique."
Example and what to do
Example
A help icon was added next to several titles, and the updates that located themselves by those titles stopped working.
Another example
A review searched for the text "int(" to check whether someone converted a number, and it matched inside "print(": any line that printed something seemed to pass. What you search for has to be a whole word, not a piece of another one.
One more example
A review took the last line the code printed as its answer. With a function that prints the number inside and doesn't return it, that line existed for another reason: the answer showed up on screen, even though the code never delivered it.
What to do
Use markers of your own, like an ID or a section comment. And when a change touches many blocks at once, check all of them, not just the first one.
A change that repeats without breaking anything has to be checked by running it twice, not once.
Detector question
Did I run my change on something that already had it applied? What other process touches this same text after me?
Warning sign
"It ran fine, so it's done."
Example and what to do
Example
To check that a script changed nothing on its second run, the checksum of the same run was compared twice, and then two checksums taken from different file lists. Both comparisons said "same" or "different" without proving anything. The comparison that works takes the checksum, runs the script, and takes the checksum again, from the same files and in a single command.
What to do
Run it twice from the same starting point and compare the files: the second run must not change anything.
A setting is not a status, and an alert is not a guarantee
Scripts and files
"The scheduled task is enabled" doesn't mean "it ran this month." And waiting for something to alert you when it fails means trusting that the alert never fails.
Detector question
Am I counting on something to "fire on its own"?
Warning sign
"It's scheduled, so it's running."
Example and what to do
Example
A scheduled task showed as "active" but had not run for two months because of an expired credential. The dashboard showed the last data point as if it were from today.
What to do
Show on screen when the data was last updated. Check which periods are missing. And make processes safe to re-run without duplicating rows.
Quality filters usually review what's new. What's already in is left alone, and its assumptions age without anyone asking again.
Detector question
This thing that has been running for months, has anyone re-verified its assumptions since it was published?
Warning sign
"We already validated that when it was published."
Example and what to do
Example
A published report kept filtering by a column the source had stopped filling. It showed zero, and since it was "already validated", nobody asked again.
What to do
Subject what's already in use to the same tests as the new candidate, in every review.
If a number didn't change when it should have, or changed just as you were looking at it, suspect these in order: the cache, another process writing to the same table, the tool you're looking through, and only at the very end the system itself.
Detector question
Am I looking at the system, or at a cache, a copy, or a tool that kept an old snapshot of it or changed it along the way?
Warning sign
"I already fixed it and it's still the same" · "this time I really did type it right."
Example and what to do
Example
A table "lost half its columns." A few minutes later it was complete: you were looking at it while another process was rebuilding it.
Another example
A new image "wouldn't load" on the page. The file was fine: the test server couldn't see folders created after it started.
One more example
A script "computed a text pattern wrong." The script was fine: the console the code was passed through turned every \n into a real line break before the script read it. And in a notebook where cells share memory, an error that should have shown up didn't: the variable was still stored from an earlier cell.
What to do
Take a second snapshot a few minutes later before you declare a disaster. If an instrument has already failed you the same way, replace it instead of watching it: a repeated slip isn't bad luck, it's the wrong tool. A console that transforms \n or \\ along the way is no good for passing it code: the code goes in a file, written with a tool that doesn't touch it, and runs from there. And anything mechanical (a command that has already failed three times) gets blocked by a guard, not remembered.
Whoever runs it and whoever verifies it are different people
Tests and checks
If the same person runs it and confirms it, an execution error confirms itself.
Detector question
Besides the person who did it, who reviewed the result as the user receives it?
Warning sign
"I already reviewed it myself."
Example and what to do
Example
Whoever wrote each page also took the review screenshots, and only of the steps they thought were risky. The reviewer read the source file. The steps without a screenshot were seen by no one.
What to do
One runs it and the other checks by reading the result. "I already did it" is the sign that it can now be verified, not the verification. And whoever verifies does not look through the same channel as whoever did it (see lesson 14): if both read only the source, neither sees the screen.
Moving fast in the wrong direction is not progress.
Detector question
Is this block big, hard to undo, or does it change something another person already decided? If yes, explain and wait. If not, say what you'll do and go ahead: asking permission for every step also has a cost.
Warning sign
"I already know what they want."
Example and what to do
Example
The folder structure of an entire project was reorganized without notice. The change was correct in itself and was fully undone, because another person had already decided that structure for reasons that weren't in the code.
What to do
Before each large block of work, explain in plain language what you'll do and why, and wait for confirmation.
Nothing technical on screen unless it changes a decision
Interface and design
Every element of a business screen should help decide something. What helps audit goes in the code; what helps decide goes on the screen, in business language.
Detector question
Does this change any decision of whoever uses the screen? If not, why is it here?
Warning sign
"In case someone needs it."
Example and what to do
Example
A sales screen showed the internal ID of each order and the time of the last calculation in its first row. Nobody decided anything with it, and it took the space of the data that did matter.
What to do
Before adding something to the screen, write down what decision it changes. If it changes none, it goes to the technical log or the code, not the screen.
Something that works but can't be seen gets reported as broken
Interface and designwebapps
If a button changes something outside the view, for the user, working and not working look the same.
Detector question
Right after using the control, without moving from where it is, does the user see what changed?
Warning sign
"It does work, it just doesn't show."
Example and what to do
Example
A filter did filter a table, but the table was two screens further down. They reported it as "does nothing."
Variant
Hiding a container hides everything inside it, even things that had nothing to do with it. For example, a useful link that lived inside a retired control.
What to do
Every control says what changed and takes the user to it. And before hiding something, check what other elements live inside it.
In small print, "÷" and "+", "l" and "1", "O" and "0" get confused. If two people read the same thing wrong, they're not distracted: it's the size of the type.
Detector question
At the size and font where it will be read, can this symbol be confused with another, or does it look like something that can't be typed?
Warning sign
"It looks fine on my screen."
Example and what to do
Example
Two different people read "98 ÷ 441" as "98 + 441" on a card set in small print, and both reported that the calculation was wrong.
Another example
The code font of a material joined >= into a single symbol, ⩾. Whoever is learning copies what they see, and that symbol doesn't exist on the keyboard or in the language.
What to do
When a symbol gets confused, replace it with a word: "98 over 441" instead of "98 ÷ 441".
Every figure in a document comes from a measurement with a date and a source, not from memory or from an earlier version of the document.
Detector question
What data and what date did this number, or this comparison ("more than a thousand," "over half," "almost all"), come from?
Warning sign
"It's the figure we already had" · "it comes out about that" · "you can see they all pass."
Example and what to do
Example
An explanation said a bug made you lose "almost one in five" tickets. When measured, it was more than one in five: the sentence came from mental math, not from the data.
Another example
One text said a group of cohorts "was above 50 %"; five of the twelve were below. In another, the illustration showed a hand-done calculation that didn't add up: the percentage times the total didn't give the number drawn.
What to do
Before writing a figure, measure it again and note next to it what data and what date it came from. Quantity words ("almost," "more than," "most") are numbers in disguise: measure them the same way. And if an image has calculations in it, have the same script that draws it do the math.
Document what's there, not what you wish were there
Writing and docs
A documentation review improves the document; it doesn't redesign the system. If the system has a limit, the document states it. It doesn't hide it or promise to fix it.
Detector question
What I'm about to write, does it describe how it works today or how it should work?
Warning sign
"It should work this way" · "they're going to fix it".
Example and what to do
Example
A manual said data synced by itself. The system only synced when the app was opened. The review "fixed" the manual to promise it would soon be automatic, instead of stating the limit as it was.
What to do
Separate findings about the document ("it's badly written") from findings about the system ("it works differently than ideal"). The second kind gets declared as a limitation.
An official document states everything with the same authority. An invented detail, like an owner, a schedule, or a response time, reads the same as a real one.
Detector question
Do I know this detail, or am I assuming it because it sounds reasonable?
Warning sign
"Surely it's like that."
Example and what to do
Example
A procedure had a 24-hour response time that nobody had agreed to. It was written because it sounded reasonable, and once published it became a commitment.
What to do
Mark what's missing in a visible way, like "Pending" in red, and ask whoever does know. Neutral wording is better than false wording.
Not to be confused with
The mark is for the person deciding, not for the final reader. A "pending confirmation" inside a published text is another piece of data that shouldn't be there. If the document goes out before it's confirmed, use neutral wording that doesn't claim what's missing, and leave the pending item in the report or the project record.
Every change carries a new version number on the cover and a line in the history, with the date, what changed, and who changed it. When two copies are circulating, the version is the only thing that says which one is the good one.
Detector question
If someone has last week's copy, how do they know it's no longer the good one?
Warning sign
"It's a small change, it doesn't need a version."
Example and what to do
Example
Two copies of a manual were circulating by email. The old one said the tool "opens with a double click," which was no longer true, and nobody could tell which of the two was the good one.
What to do
Bump the version number and write the history line in the same change that modifies the document.
The log says what was left and why, not a chronicle of missteps
Git and process
A change history is for understanding the product, not for narrating the already-resolved mistakes of whoever made the change.
Detector question
Does this line help someone understand the product or avoid repeating a mistake, or does it just record what happened to me?
Warning sign
"Let's keep a record of everything that happened."
Example and what to do
Example
A history said "A was tried and failed; B was tried and failed; C was left." Someone reading it months later only needed C and its reason, and had to read three paragraphs to find it.
What to do
Write the final state and its reason. What was tried and failed only goes in if it teaches something someone needs in order not to repeat it.
Finding a word in the text doesn't prove it shows up. The text can be somewhere else in the document while the format that was supposed to display it comes out empty.
Detector question
Did I see the exact element the way the reader sees it, or did I only find the word (or the code) in the source?
Warning sign
"The word is in the file" · "it's a one-line fix."
Example and what to do
Example
Three numbered steps came out as empty bullets. The automatic check passed them because those words appeared somewhere else in the document.
Another example
A "there are more columns to the right" notice was added and taken as done because the code was there. It never appeared: a general table style removed the condition that turned it on. Someone else saw it in a screenshot.
One more example
An answer option said "Total: {2 * 78}" in the file. When the page was drawn, the system calculated what was between the braces and showed "Total: 156": two identical options. Only someone who looked at the screen saw it.
Variant
The screenshots a reviewer judges can show a screen nobody reaches. A course's capture tool answered each code exercise with a filler line just to move on, so the "after" screenshot showed that line and a "not the expected result" message: reviewers never saw what a learner sees after getting it right. On desktop the captures also took in the header and the side menu, and the lesson came out as an unreadable strip. When capturing for review, answer each step the way a successful user would, and capture only the part under review.
What to do
Check the exact element, like "these three steps, as bullets," and look at the page as the reader will see it at least once. A small fix still gets checked: it's where another rule is least expected to override it. And the first time you use a new resource on the page (a code block, a fixed header), look at it on mobile too.
A review done on an old copy brings back errors that had already been fixed.
Detector question
The version I'm editing, is it the most recent one or the one I had open?
Warning sign
"I've had the version open since yesterday."
Example and what to do
Example
A reorganization of the manual was done on a version from weeks earlier. It said again that the tool "opens with a double click," which was no longer true.
What to do
Take the structure the review proposes and pour the most recent content into it, not the other way around.
Looking only at the last lines of a result, or piping it through another command, can throw away exactly the part that said "failed." A trimmed output from a failing test looks just like one from a test that passed.
Detector question
Am I reading the verdict (how many passed, how many failed, how it ended), or only a piece of the output?
Warning sign
"No error showed up on screen" · "the last few lines are enough."
Example and what to do
Example
A page was marked as reviewed because its test "finished without errors." The result went through a command that showed only the end, and the notice that the test had failed got lost along the way.
Another example
A test suite automatically retried every test that failed. Tests that passed on the second try didn't count as failures, only as "flaky," and the cause stayed hidden: each step change asked the server for the page again. Without retries, the failure showed up and could be measured.
What to do
Find the verdict line before changing a status. If you chain commands, make sure the first one's failure reaches the end. If the output has no verdict, that means "I don't know," not "it passed" (see lesson 1). And save the full output to a file and look for the verdict there: that way you don't have to decide in advance how much to read.
If you only test what happens when everything goes right, whoever makes a mistake finds the gap. And when learning, making mistakes is the most common thing.
Detector question
If the user does the most likely wrong thing (a typo, an empty answer, two clicks), what do they see, and can they keep going?
Warning sign
"It works with the right answer."
Example and what to do
Example
A query cell counted an attempt only when the query ran. With a typo, the button to advance never activated, and the person got stuck. The tests passed, because they always ran the starting query, which had no errors.
What to do
For each control, test three paths: the good one, the most common error, and the empty one. All three say what happened and let the user keep going.
What runs on every request can't grow with the content
Interface and designweb
Something done on every page, like reading all the content or recalculating an index, costs little with ten pages and a lot with a hundred. It grows gradually and nobody notices, until something starts failing "every now and then."
Detector question
Does this thing that runs on every request go through all the content? How long does it take today, how long with double, and how long with many requests at once?
Warning sign
"It loads fast locally" · "it's an intermittent failure."
Example and what to do
Example
Every page of a site read and parsed all the content to build the table of contents. Alone, one page took 0.4 s. But the sidebar requested about 40 at a time, and together they took 16 s. Months later, the tests started failing when switching pages, and the failure looked like a problem with the test.
What to do
Measure on the compiled server and the way people use it, with all its requests together, not one page alone. Cache what doesn't change while the server runs, and leave a test that measures it with the case that actually made it slow.
What's worked on in parallel collides in what's shared
Git and process
When several people work at once, each sees their own piece clearly. What they share (the tests, the data, the documents for the whole project) combines their changes: names that clash, copies that fall behind, tests that read someone else's half-finished work. Integrating is work, not a paste.
Detector question
Which file are two pieces touching at the same time, and who integrates it? Is my test reading another person's half-finished work?
Warning sign
"Each of us tested our own" · "the failure is in another piece."
Example and what to do
Example
Four people were each writing one piece at the same time and leaving the shared parts as the deliverable. When integrating, two deliverables used a name that already existed in the shared test, and the test stopped running. Another test kept a copy of what it checked; the piece changed and the copy didn't. One test run failed because of an image another person was still drawing. And in the end, each piece couldn't be saved separately: the shared files already carried changes from everyone.
What to do
Have one person integrate what's shared, with a step that stops if a name clashes. Have the tests read the original, not a copy. Run the full test suite once everyone is done, or without the pieces that are still changing, and say which ones were left out. And save pieces that share files together, so each saved version works in full.
If a file comes from a generator, the fix goes in the generator
Scripts and files
When a file is assembled by a script or from other pieces, the file is an output, not the original. Anything fixed directly in it gets lost the next time it's generated, with no warning.
Detector question
Is this file written by something else (a script, a template, a parts assembly)? If I regenerate it, is my change still there?
Warning sign
"I fixed it directly in the file" · "I just regenerated it".
Example and what to do
Example
A note was assembled from the sections each person handed in. Three rows were fixed directly in the note, and when it was rebuilt from the submissions, the three old rows came back. In another piece, a reviewer hand-fixed the alt text of a page that came from a template; its author regenerated the page, and the fix disappeared.
What to do
Before fixing a file, find out where it comes from. Fix the source (the template, the script, the submission) and regenerate; if you have to fix the output, fix the source too in the same step. Whoever reviews generated output says what changed so it gets moved into the source. And before rebuilding a file from its sources, compare it with what's going to come out: anything that differs is a fix that would be lost.
A general rule must not silently beat the specific ones
Interface and designweb
A rule that applies to everything (a global style, a default value) can override what each piece decided for itself, without anything warning you.
Detector question
Does this general rule touch something each piece already decides for itself? If they clash, which one wins?
Warning sign
"I made it global so I wouldn't have to repeat it."
Example and what to do
Example
The keyboard focus style, written for the whole site, came with its own border radius. Because it ranked above each component's own styles, a pill-shaped button changed shape when focused with the keyboard. Nobody noticed, because with the mouse the focus doesn't appear.
What to do
Let the general rule touch only what's its own (the focus outline, not the shape). If it should win on purpose, write that down next to it. And test it on the most different piece, not only the common one.
If two explanations predict the same thing, your example doesn't choose between them
Numbers and data
An example confirms your explanation only if the rival explanation would give a different result. If both predict the same, the example lets you believe whichever came to mind first.
Detector question
What other explanation would give this same result? Does my example rule it out?
Warning sign
"The numbers confirm it."
Example and what to do
Example
A lesson compared raising a funnel's first step by five points against raising its last step. The first won, and the text said: "they're worth more where there are more visits". But the first step was also the one that converted least, and that was the cause; in a funnel where the first step converts better, the last one wins despite having fewer visits. In another project, a page was blamed for loading in 2.7 s; a test that requested pages one by one passed even without the fix, because the real cause was forty requests at once.
What to do
Before teaching or reporting why something happens, write the rival explanation and build a case where the two predict differently. Compare parts of the same size, try another example size, and measure a general claim on simulated data before writing it. If you can't separate them, say that the example is compatible with both.
What looks obvious to you is the first thing to teach
Writing and docs
Someone who already knows doesn't see the steps they take without thinking. So their first example brings several new things at once, and a beginner gets lost on the first line.
Detector question
In my first example, how many things are new to someone who knows nothing? Has every symbol, every word, every button name and every action I ask for been taught before?
Warning sign
"Anyone gets that" · "it's just one line" · "it's the usual word."
Example and what to do
Example
A programming course for beginners opened with a cell that had six new things at once, and the plan never taught what a data type is or that = stores and reads right to left. The course owner said it in one sentence: "you assume a lot about the students." Later, three lessons said "like dragging a formula"; no lesson had taught dragging, only "copying the formula down".
What to do
Before writing, list what your example takes for granted (symbols, words, clicks, things the learner should have built already) and teach those first: one new idea at a time, each symbol explained the first time it appears. Search for each word that matches a button's name, and each action you ask for, the way you search for a symbol. If possible, let an automatic check compare what each example uses with what was already taught.
Write about a fast-moving tool from today's practice, not from memory
Writing and docs
A lesson or a guide about a tool that changes every few months goes stale while it's being written if it comes from what the author remembers. Its premise can be false before the first reader sees it.
Detector question
Which of this text's premises depend on what the tool can do today? When did I last check them, and against what source?
Warning sign
"That's how it works" · "everyone does it like this."
Example and what to do
Example
A plan for an "analysis with AI" module opened with "the AI can't see your data, so describe your table instead of pasting it". The course owner pointed out that agents now connect to the database, run the queries themselves and hand back the number. Research with sources and dates confirmed it: copying code out of a chat was how it was done two years earlier. Several lessons were rewritten around what still holds: defining the question, supervising the agent and checking the number.
What to do
Before writing about a fast-moving tool, look up current practice with sources and dates: what the tool does now, what is free, what people actually use. Build the text around the skill that survives the next version (framing, supervising, verifying), and keep tool-specific steps in one place that can be updated. Mark what you couldn't verify, and try it by hand before publishing.
Code goes through the editor, not through the shell's text handling
Console and tools
Rewriting a source file with the shell's own string tools (read it, replace, write it back) passes the code through rules made for something else: a default encoding, operator precedence, quoting and escaping. The change looks right in the command, and the file comes out different from what the command seems to say.
Detector question
Is this code change going through the editor, or through the shell's read, replace and write, with its encoding, precedence and quoting rules?
Warning sign
"It's a quick replace, I'll do it in one command" · "several replacements at once is faster".
Example and what to do
Example
Two edits in one session, both in Windows PowerShell 5.1. A read-replace-write of a script (Get-Content, -replace, Set-Content -Encoding utf8) read the file in the system's ANSI code page and wrote it back as UTF-8: every ✓ and · in it turned into ✓ and ·, and its self-check printed the mangled mark. Then a list of replacements written as `@('old', 'new' + "n" + 'more')` dropped lines: in PowerShell the comma binds tighter than +`, so each pair became four items and only the first piece of the new text was written.
What to do
Change code with the editor's own edit and write tools. When a change has to be scripted, write the script to a file and run it with an interpreter that reads and writes UTF-8 and has plain string literals (node fix.mjs, python -I fix.py). Then look at the diff: a change that touched more lines than you meant, or characters you didn't touch, is the shell's.
If the answer comes out the same without the skill, the exercise doesn't measure it
Tests and checkseducation
An exercise checks a skill only if someone who skips that skill gets a different result. If the starting point, an example or a shortcut already produces the right answer, the exercise passes everyone. It is lesson 13 for exercises: a check is worth something only once you have seen it reject the mistake.
Detector question
What result does someone get who skips exactly what I teach, or who submits the exercise untouched? Is it different from the right one? Did I pick a case where both calculations happen to match (the first group, the first step, the total)?
Warning sign
"With the right answer it gives the number, so it works."
Example and what to do
Example
A practice asked for the largest purchase of the summer, using a date filter. The largest purchase of the whole table was also from the summer: without the filter the number was the same, and the exercise accepted it. Another practice required the code to contain an =; it passed with the result typed by hand, total = 1945. The check looked at the letter, not the skill.
What to do
Solve every exercise three times: untouched, with the most likely mistake, and correctly. Only the third may pass. Then run the same list of shortcuts on every exercise: the number typed by hand, the starting code with the answer added at the end, the symptom fixed by hand instead of the cause, the required word used empty, and a run with memory left over from an earlier attempt. Keep any tolerance smaller than the distance between the answer and the most likely mistake, and write three correct forms a learner would use, so the check doesn't reject what is right.
What learners see before answering can't contain the answer
Writing and docseducation
A question stops measuring once its answer has already appeared: in an image, in the key idea, in another exercise's explanation, in a step title or in an earlier lesson. Whoever answers right may have just read it.
Detector question
Does the learner see this figure, image or phrase before answering, here or in an earlier lesson? Not only the exact number: also an approximation ("about 25"), the conclusion written in a caption, a title, a "you just saw that…", or what the most likely wrong answer of an earlier exercise shows on screen.
Warning sign
"It's just context" · "it's only the title" · "it's the caption."
Example and what to do
Example
An illustration with the right figure sat above the practice that asked for it. A lesson asked what happens when you store a new value in a variable, in a step titled "Storing again replaces it". A summary placed under the final quiz was read before answering it. A feedback box that appears after the first attempt, even a failed one, gave the direction of the answer.
What to do
Look for each answer in everything shown before its question, including what appears after a first failed attempt, what starting code prints when run, downloadable files and what can be counted in a drawing. Write the step title and the learning goals last, and read them next to the result of each prediction. An automatic check finds exact figures; approximations and paraphrases are found only by someone reading with the answer in mind.
A question with two defensible answers measures nothing
Writing and docseducation
If, with what the prompt says, another option can also be defended, whoever picks the "wrong" one for a good reason is marked wrong, and whoever gets it right may have been lucky. The question measures the missing sentence, not the idea.
Detector question
With what the prompt says and what the lesson taught, can I defend another option? What fact would I need to rule it out, and is it written?
Warning sign
"It goes without saying" · "obviously it means…"
Example and what to do
Example
A question asked what to do with an empty discount "when the sale had no discount". The lesson had taught that a blank that is a fact stays blank, so "leave it" was defensible; the expected answer, filling it with zero, depended on something the prompt never said: that the goal was the average over all sales. In another, "the main metric is the share who pay the second month" didn't say share of whom, and the two versions tied or differed depending on the reading.
What to do
For each wrong option, write one line on why it is wrong using only the prompt and the lesson. If ruling it out needs a fact that isn't written, add it to the prompt: the purpose of the calculation, the rule of the system, the denominator of a proportion, what is known about the data. In a sorting exercise, test every item against every category, not only its own.
If you can guess the right option by its place or its length, the quiz measures guessing
Tests and checkseducation
Each question can be fine on its own and the set still give the answer away: the right option is always second, always the longest, or always the middle-sized one. Learners notice patterns faster than authors do.
Detector question
If I read only the position and the length of the options, without the text, do I guess the right one more than once in three?
Warning sign
"I checked each question, they're all correct."
Example and what to do
Example
In one module's closing quiz the right answer was the second option in six of seven questions. Across a course, the right option was the longest in 55 % of questions, when chance with three options is one in three. After the length was balanced by hand, the right option ended up as the shortest in six of seven questions of one lesson, and as the middle one in every question of three lessons in a row.
What to do
Shuffle the options when they are shown, with an order that doesn't come from where the right one was written. Measure longest, shortest and middle per file, and pool small files per module, so a few questions don't hide the pattern. Read the count against what you asked for: a balancing pass can move the giveaway instead of removing it.
A hint points at the cause of the mistake, not at the answer
Writing and docseducation
A hint shows up after a wrong attempt. If it gives the answer, the line that solves the exercise, or what separates the solution from the most likely mistake, the second attempt measures copying.
Detector question
Could someone who reads this hint pass the next attempt without the skill? Does it name the answer, the solving line, the steps the exercise measures, or the earlier number it should match?
Warning sign
"It's only a nudge" · "they already tried once."
Example and what to do
Example
A review of a course's 277 checked practices changed 82 hints in 49 files. Some gave the line that solves it ("the second one is the…"). Some dictated the order that separates the solution from its common mistake. In "check against an earlier result" exercises, the hint said "it should match what you got in lesson 2.2": whoever remembered the earlier number could type it without doing the step.
What to do
Write the hint about the cause of the likely mistake ("positions start at 0, not at 1"), and cite only what was already taught. Keep the "it should match the earlier number" for the explanation shown after answering, where it teaches that two methods giving the same number is a verification. Review hints against the exercise's most likely mistake, not on their own.
A damaged store must not take the next record down with it
Scripts and filesweb
However a site saves its records, one damaged write can take what comes after it down too. With records appended as lines, a line cut short glues the next record onto it, and a reader that skips "damaged lines" drops both, silently. With all records in one file that is read, changed and written back whole, once that file can't be read every new record fails, and only a console nobody watches says so. It is the other side of lesson 24: there the file gets emptied; here, whatever comes after is lost.
Detector question
If the store is damaged tomorrow, what happens to the next visitor's record, and who finds out?
Warning sign
"Damaged lines get skipped" · "it's just appending to the end" · "the write is atomic, so the file can't break".
Example and what to do
Example
In both runs of the first Web simulation, the form saved each message as a JSON line, and the admin panel skipped any it couldn't read. With a line cut off at the end of the file, the next message got the reply "Thanks!", was saved, and never showed up in the admin panel. One of the two versions had a comment that said "damaged line: skipped".
Another example
In both runs of the rerun, every message lived in one JSON file, written to a temporary file and then renamed, so a crash mid-write no longer hurt. But with the file damaged any other way (a hand edit, a full disk, a bad restore), every new visitor got "Internal error", the admin page showed nothing, and the only signal was a stack trace in the server console.
What to do
Keep a new record from depending on old ones: append to a file that ends in a line break (start the record with one if it doesn't), or save each record in its own file, rather than rewriting one file that holds them all. When reading, count what couldn't be read and show that count where the owner will see it. Test it by damaging the store by hand (cut the last line short, truncate the file), then sending a record.
A form open to the internet isn't filled out only by people. If nothing stops automated submissions and whoever reads the messages can't clean them up, the first wave of junk buries the real customers.
Detector question
If a thousand fake messages arrive tomorrow, how does the owner find the one real one, and how do they delete the rest?
Warning sign
"It's a small site, nobody's going to attack it" · "we'll add the limit later."
Example and what to do
Example
In both runs of the Web simulation, the contact form accepted 650 messages from a script in a couple of seconds, all of them saying "thanks." The admin had no pages and no delete button: the admin page grew to between one and six million pixels tall, and on mobile the text of the message ran off the screen. Both times, the builder noted it as pending.
What to do
From the start, a cheap brake: a hidden field that people don't fill in, and a limit on submissions per address per minute. And an admin with pages, newest first, and a way to delete or mark as junk. Test it by sending hundreds of messages with a script and opening the admin on a phone.
A form with method and action promises a fallback that works even if JavaScript fails. If the server only answers what the JavaScript expects, the promise is false and the person ends up staring at code.
Detector question
If this JavaScript file doesn't load, what does the person see when they submit?
Warning sign
"Everyone has JavaScript anyway" · "I'll leave the action in just in case."
Example and what to do
Example
In both runs of the Web simulation, the form submitted with JavaScript and declared method="post". Without JavaScript, in one version the server rejected the submission with a line of JSON and the message was lost; in the other it saved the message and showed only {"ok":true} on a blank page. This second version's summary said it would not submit without JavaScript.
What to do
Decide on one of two and test it: either the server accepts the normal submission and replies with a page (or redirects to a thank-you page), or the form doesn't declare a fallback it doesn't have. Submit once with JavaScript turned off.
Disabling the button while submitting prevents a double submission, but the button loses focus and the browser moves focus to the document. Someone using the keyboard, after the notice, starts over from the top.
Detector question
After submitting, whether it succeeded or failed, where does focus end up and where does the next Tab go?
Warning sign
"I disable it so nobody can submit twice."
Example and what to do
Example
In both runs of the Web simulation, submitting disabled the button. With an email the server rejected, the notice appeared below the button, focus landed on the document, and the next Tab took the user to the first link on the page, far from the field that needed fixing.
What to do
When submission finishes, put focus where the person needs to continue: on the field with the error (marked with aria-invalid and its message next to it) or on the success notice. Or prevent the double submission without disabling, using aria-disabled and a flag. Go through the error using only the keyboard.
A password login without an attempt limit is only as strong as the weakest password
Interface and designweb
A login that answers every wrong password at full speed lets anyone try thousands a minute. A strong default password hides the problem until the owner sets their own, and nothing stops that one from being short.
Detector question
How many wrong passwords can someone try against this login in a minute, and what happens after the fifth?
Warning sign
"The default password is strong" · "it's only the admin page".
Example and what to do
Example
In both runs of the Web simulation rerun, the admin page asked for a password and never limited failed attempts: 200 wrong passwords in a row got 200 refusals in under a second, none blocked. The default password was long and random, but the builder's own summary told the owner how to set a new one, and nothing required it to be long.
What to do
Slow down or block repeated failures, per address and per account, with a wait that grows. Require a minimum length when the owner sets a password. Test it: send twenty wrong passwords in a row and check that the last ones are refused.
No lesson matches. Try fewer filters.
Saw one of these mistakes? Share it with /ritual share.
Ritual