The build kept passing while the renderer was already broken

For a while the renderer that draws my site’s diagrams was already broken, and every build reported success.

The site turns posts into web pages first, then publishes them in one batch. That conversion step is the build. When a post contains a text description of a flowchart, the build calls a renderer to turn that description into an image.

The catch: the build only went looking for the renderer when a post actually had a diagram.

Then the renderer moved, and the build still pointed at the old location. That should have been an error. But at the time not one post on the site had a diagram, so the build never looked for the renderer, and it never failed once.

No diagram went missing from the site, because there were none yet. What was broken was the drawing itself: the next post with a diagram would have failed to render. Until then, a successful build revealed nothing.

Not running, and failing while reporting success, are two different things

What bothered me wasn’t the renderer moving. It was that nothing told me.

Two cases need separating here. What happened this time is the first: the work never ran at all. The second is work that ran, failed, and reported success anyway.

The site already guarded against the second. When the render step fails, it throws an error. But the site generator catches that error, logs it, finishes the build anyway and reports success, exit 0. The page then publishes as usual, just without the diagram, and the publish pipeline only looks at the build’s result, so it takes a broken page for a good one.

That’s why the end of the build already had a check: if any diagram failed to render, fail the whole build. It sits where the site generator can’t catch the error, so it doesn’t get swallowed.

That check works, but it only covers the second case. It looks at diagrams that were drawn. With zero diagrams there’s no failure to record, so it can never go off.

It’s a smoke alarm that’s installed and powered, but only in a room nobody ever walks into. It stays quiet, and you assume that’s because nothing is wrong.

Not the first time I’ve hit this

Looking back, this kind of silence had turned up elsewhere, each case found and fixed on its own:

• The doc checker. Give it a misspelled file name and it used to not say “not found.” It printed “no findings” and exited 0. In effect it told me the file was clean when it had never opened it. Now a name it can’t read is an error, not a silent skip.
• The background research job’s security scan. A scheduled research job runs a security scan on content first. The scan pointed at a script that no longer existed, so every run errored, and that error was read as “the content failed the check.” Everything it processed from 2026-09-04 on would be quietly quarantined, because the system itself was broken, not because anything was wrong with the content. Pointing it at the script’s new location fixed it.
• The work rules themselves. The rules once said every pull request automatically ran a check. In reality this repo has no CI configuration at all, so nothing runs. The rules are what the AI trusts every time it works, so they were fixed first, to say what’s true: there is currently no CI.

The common thread: what got reported didn’t match what actually happened. A tool that never opened the file said it was fine; a system that was itself broken said the content had failed; the rules said a check existed when none did. None of those signals was answering the question you assumed it was answering.

The renderer fix follows the same principle: make a missing piece fail at the earliest point it can be noticed.

At the start of the build, check that the renderer exists. The site generator loads this setup at startup, so an error here stops the build before any page is written, and prints the path it looked at. No waiting for the day a post finally carries a diagram to find out the renderer was gone long ago.

How it runs now, and how I checked that it works

This is the build now:

The difference from before is the first step: look for the renderer right at the start, not only once there’s a diagram. The check at the end was already there. Now there are two, one at each end: one for “is the renderer there,” one for “did the diagrams get drawn.”

The cache in the middle matters too: a drawn diagram is stored by its content, and later builds reuse it without calling the renderer. I’ll come back to that at the end.

Making the change wasn’t enough for me to call it working. This whole thing happened because “saw no error” was taken to mean “there was no error.” If all I looked at was a new green build, I’d be fooling myself the same way again.

So I ran a few checks:

• I added a temporary sample post with a single flowchart and confirmed the diagram actually rendered. The sample was only there for the check and didn’t stay on the site.
• I deliberately put the renderer path back to its old, stale location to see whether the build would fail. It stopped at exit 1, and the error message included the path it looked at.
• With the same stale path, I ran the path tests: 3 of the 4 failed. So if the same thing happens again, those tests will catch it.

You only trust a check after you’ve watched it fail.

What’s still open

These checks only cover the handful of places I’ve already found. Where else something is quietly skipping while everything looks fine, I don’t know yet.

The check at the start also only proves the renderer is there, not that it can draw. Actual rendering only happens when a diagram isn’t in the cache, say a new diagram or a cleared cache; the rest of the time the build reuses the cached image without calling the renderer. So every build knows whether the renderer is still there, but whether it can still draw, I won’t know until the next time it actually has to.

At least now I ask one more question: did the check that just passed actually touch the thing I assumed it was checking?