A failure state nobody could reach

The tool I maintain has a rule it is fairly proud of: when it cannot turn a lecture recording into text, it says so, by name, and it says whether waiting will help. Silent skipping is how a whole class of breakage stays invisible, so every failure has to declare itself and declare its own prognosis.

There were five outcomes. One final, four worth retrying.

  NONE      = "none"            # genuinely has no captions
  BLOCKED   = "blocked"         # the platform is rate-limiting us
  ERROR     = "error"           # network or platform failure
  SIGNIN    = "needs-signin"    # the platform wants a human to authenticate
  DEFERRED  = "deferred"        # held back on purpose

  RETRYABLE = {BLOCKED, ERROR, SIGNIN, DEFERRED}

Each had user-facing wording. Each appeared in a table in the README. Two of them appeared on a page of my portfolio site. The set was covered by tests.

And SIGNIN had never once been produced by any line of code in the project.

How that is even possible

The function that fetches a transcript from the video platform returned either a result or nothing:

  def transcript(self, host, guid) -> tuple[str, str] | None:
      if host not in self.ready and not self._sign_in(host):
          return None          # sign-in did not complete
      ...
      if not deliv.get("HasCaptions"):
          return None          # genuinely no captions
      ...
      return None              # asked for captions, got nothing back

Three different situations, one return value. And the caller, having no other information to go on, did the only thing it could:

  got = panopto.transcript(*ids)
  if not got:
      no_captions[week].append((label, url, captions.NONE))
      continue

NONE. The final one. The one whose user-facing text is “no captions published” and whose meaning is do not bother waiting, this will never work.

An expired sign-on, a network blip, and a genuinely caption-less recording were indistinguishable from the outside — and all three were reported as permanent.

This is the exact failure the five-state design existed to prevent. The design was right. The implementation quietly collapsed it back to one bit, and left the vocabulary standing on top as decoration.

Why nothing caught it

What unsettles me about this bug is how healthy it looked from every angle anyone normally checks.

• 01The constant existed. Grep for SIGNIN and you get hits: the definition, the retryable set, the message table. It looks thoroughly wired in.
• 02The type checker was satisfied. tuple[str, str] | None is a perfectly reasonable signature. Nothing about it is a type error. The bug lives entirely in what the None was allowed to mean.
• 03The documentation agreed with itself. README, code comments and portfolio page all described five states with four retryable. Every artefact was internally consistent — and every one of them was describing a system that did not exist.
• 04The tests passed. Because the tests checked what the code did. The code did something coherent. It just was not the thing the design promised.

There was no failing signal anywhere. The only way to find it was to ask a question nobody thinks to ask: not is this state handled, but can this state actually happen.

Producers, not consumers

Here is the check that would have caught it in about thirty seconds, and it is the thing I would want someone to take away.

When you grep for a status constant, you overwhelmingly find its consumers — the places that check for it, render it, or route on it. Consumers are cheap to write and they accumulate. Their presence is what makes a dead state look alive.

So search for producers instead. For each state your system can describe, find the line that assigns it. If there isn’t one, you have a lie in your documentation.

  $ grep -rn "captions\.\(NONE\|BLOCKED\|ERROR\|SIGNIN\|DEFERRED\)" --include=*.py .

  captions.py:35:  RETRYABLE = {BLOCKED, ERROR, SIGNIN, DEFERRED}   ← definition
  captions.py:244: return None, BLOCKED                            ← producer
  captions.py:246: return None, NONE                               ← producer
  captions.py:247: return None, ERROR                              ← producer
  main.py:463:     (label, url, captions.NONE)                      ← producer
  main.py:472:     (label, url, captions.DEFERRED)                  ← producer

Five states. Four producers. SIGNIN appears exactly once, in its own definition.

The fix, and the shape of it

The repair was to stop overloading the return value. The function now hands back a reason alongside the result, so the caller is structurally incapable of flattening three situations into one:

  def transcript(self, host, guid) -> tuple[tuple[str, str] | None, str]:
      if host not in self.ready and not self._sign_in(host):
          return None, SIGNIN
      ...
      if not deliv.get("HasCaptions"):
          return None, NONE
      ...
      return None, ERROR

That is the general shape of the fix, and it is worth more than the specific bug. None is a single bit, and a single bit cannot carry a reason. If a function has more ways to fail than to succeed, the failure needs a type of its own — otherwise every caller is forced to guess, and callers guess the same wrong way every time.

What went into the test suite

The check I added is almost embarrassingly simple, and it would have failed on the shipped code:

  check("a Panopto sign-in failure is retryable, not 'no captions'",
        captions.SIGNIN in captions.RETRYABLE)

  result, why = captions.PanoptoClient(_FakeSess()).transcript("x", "guid")
  check("an unfinished sign-in reports needs-signin",
        result is None and why == captions.SIGNIN, why)

The second one is the real one. It fakes a browser session that never gets past the login page — the exact situation that used to be reported as permanent — and asserts on the reason that comes back.

Writing it took three minutes. It sat unwritten for weeks because the feature looked finished, and looking finished is the specific condition under which nobody writes the test.

The family this belongs to

I have since found a second bug in the same family, and I now think of it as a category rather than an incident: defects that are invisible to every tool that reads your code as code.

The other one was an em dash in a source file that had been saved double-encoded. In every editor it renders as an ordinary dash. The type checker has no opinion about the contents of a string literal. But every note the program generated carried the mangled bytes into Word, plain text and Markdown alike, for weeks, in front of the user.

Both bugs share a property: the code is valid, the tools are content, and the thing that is wrong is a mismatch between what the program means and what it does. So two of the checks in my suite deliberately read the source as text rather than importing it — one scans the bytes for double-encoded punctuation, the other confirms a particular cleanup routine is still wired into startup.

That feels like a hack, and I keep expecting to be talked out of it. I have not been yet. Some invariants are simply not visible from inside the program, and a test that reads the file is the honest way to hold them.

What I would say to my past self

Enumerating your states is the enjoyable half. You name the outcomes, you write the messages, you build the table, and it feels like design.

The other half is proving each one is reachable — and it is unglamorous enough that it is easy to skip, precisely because the first half already produced something that looks complete. A state with no producer is not an unused feature. It is a promise your documentation is making on behalf of code that cannot keep it.