What a sign code doesn't tell you

This is the last article in this series, and it is a different kind of article than the others. The rest of what we’ve published has been about what the data shows — height limits, setback distributions, authority verdicts, vocabulary spread across thousands of ordinances. This one is about what the data doesn’t show, and where the boundary sits between “we found a rule” and “we are confident that rule is exactly right for your project.” It also contains a correction, which is the honest place to start.

The correction

An earlier version of this article reported that 43.0% of extracted facts carry a human-review flag, based on what it described as “a stratified sample of 2,400 individual extracted facts, spread evenly across the full range of our sign-rules table.” It was not spread evenly. The sample was paginated in ascending primary-key order, so all 2,400 rows came from the very oldest end of the table — the earliest extraction runs, before large parts of the pipeline were rebuilt. It was an id-head sample wearing the label of a stratified one.

That is not a small distortion. Split the whole table into ten equal bands by id and the review-flag rate collapses as you move through it:

A sample drawn from the head of that table was never going to describe the corpus. Every figure below is now computed over the entire population — all 660,173 rules — rather than any sample at all, which for a table this size is both possible and cheap. That is the standard we should have applied the first time.

What the full population actually says

Alongside confidence, every extracted fact carries a separate needs_review flag — a signal that a human should look at this specific fact before it is treated as final. Across the whole table, 63,713 rules — 9.65% — are flagged. And 96.94% of rules are marked as the primary, controlling rule for their slot rather than a secondary or competing variant.

Why 9.65% is still the number that matters

The corrected figure is far better than 43%, and it would be easy to treat it as a rounding error and move on. It isn’t one. Nine and a half percent of 660,173 is 63,713 individual rules that our own pipeline says a person should look at before anyone relies on them — and they are not evenly distributed. They cluster heavily in the oldest extraction bands, which means the chance that a given jurisdiction’s rules carry flags depends on when that jurisdiction was processed, not on how clearly its ordinance is written.

Confidence and review flags also measure two different things, and the difference is the whole point of this article. Confidence measures whether the pipeline believes it identified the right type of rule in the right place in the ordinance. It does not measure whether that rule, once found, was correctly and fully interpreted for the specific scope it applies to — the exact zone, the exact sign use, the exact exception language buried three sentences later. A rule can be high-confidence and flagged at the same time, and that combination is exactly the interesting one.

The is_primary figure has the same double edge. A pipeline that marks 96.94% of rules as the controlling one is a pipeline that commits to a single answer almost every time. That is what makes a report readable. It is also what makes the residual risk hard to see: when a code contains a general rule and a narrow carve-out for one use class, picking the wrong one of the two produces a confident, clean-looking, wrong answer. We ran into precisely that failure while writing why two towns in the same county can have opposite sign rules — an early draft built its headline on a flagged, medium-confidence special case that the same code’s general rule contradicted. Filtering to high-confidence, unflagged rules is what fixed it.

Why this is where automation should hand off to a person

This is exactly the gap that separates “found a rule matching this category” from “correctly understood what it means for a specific project,” and it is precisely where a platform like this one should point a reader toward a licensed professional or the municipality itself — not paper over the gap with false precision. A report that always looks equally confident, regardless of whether the underlying fact is one of the 90.35% the pipeline is settled on or one of the 9.65% it flagged for a second look, would be lying by omission. The honest version shows the flag.

A closing note on this whole series

Every article in this series is built on numbers pulled directly from our own production data, and this round we re-derived every one of them from the full population rather than a sample. That discipline runs in both directions. It means the state-by-state authority breakdowns, the height and setback distributions, and the vocabulary-spread findings are real. It also means we publish the corrections when a number turns out to have been a sampling artifact, as the 43% figure was, or a subtotal mistaken for a total, as happened in how we actually read a million pages of municipal code.

“Check with the municipality before you pull the permit” is a line that shows up throughout this series, and it is not boilerplate liability language. It is a direct reflection of where automated extraction runs out of confidence — a real, measured boundary. Automated research gets a technician to the right ordinance, the right section, and the right rule far faster than reading a few hundred pages by hand. It does not replace the judgment call at the end, and it does not answer the prior question of which government holds authority over the address in the first place.

How to see the confidence behind your own lookup

A report shows the rules that apply to one address, cited to the ordinance, along with the confidence behind each one — so you know which results to treat as settled and which are worth a call to the municipality before you rely on them. Creating an account is free.