The feature I measured, walked back and shipped smaller

PitStop is garage management software, and one of its features is a workshop chat: every job card is a message thread tied to a vehicle, mechanics send voice notes instead of typing, and the owner can broadcast to the whole floor. Chat is only useful if it feels alive, so it needed realtime push. When a mechanic sends "the brake pads arrived," the service advisor's phone should know within a second, not on the next page refresh.

PitStop is also multi-tenant. Every table carries a garage id, and row level security means a query from one garage can never see another garage's rows. That guarantee is the foundation the whole product stands on. Two garages can run on the same database and each behaves as if it were alone.

The obvious realtime design, the one in every tutorial, is to publish the messages table to the realtime layer. A client subscribes, the database broadcasts inserted rows, and the same row security that guards queries is supposed to guard the broadcasts. Each subscriber receives only the messages they could have read anyway. Elegant, minimal, and it puts message bodies on the wire on the strength of one assumption: that the realtime layer enforces the read policy exactly the way the database does.

That assumption bothered me enough to write it down before building anything. The design note for the chat feature has a risk register, and the first entry says the realtime layer's policy enforcement "is asserted everywhere that it does; that is not the same as having watched it refuse." The note set a rule for the feature: if realtime authorization could not be proven over a real websocket, chat would ship on polling. Slower is not the same as wrong.

So the probe came first. Not a unit test, because the thing under suspicion was not my code. The probe opens two real websocket connections as two real users in two different garages, sends messages, and records who hears what. The migration that published the messages table said, in its own header, that enabling it without that probe "would be shipping a guess about somebody else's server."

The probe ran four cases. Three passed. The one that failed was the one the feature exists for.

The diagnosis took a while, and it is the interesting part. The read policy on messages checks the message kind first: broadcasts are readable by the whole garage, so the policy short-circuits before it ever calls the membership function. That is why broadcasts worked. For private conversations, the policy calls a function that checks membership, and that function calls another function underneath. Under a normal database session, the chain evaluates correctly. Under the realtime layer's own policy evaluation, the nested call came back false. Same policy, same data, different evaluator, different answer.

Note the direction of the failure. The realtime layer refused too much rather than too little. No garage ever saw another garage's messages, so this was never a security incident waiting to happen. But a chat where you do not receive your own conversations is not a chat. A feature that fails safe is still a feature that does not work.

The tempting path at this point is to debug the evaluator. Restructure the policy, inline the membership check, add special cases, and re-run the probe until all four lines pass. I had a working chat on polling and a realtime design that depended on the fine print of somebody else's policy engine. Every hour spent making the clever design work was an hour spent deepening a dependency I could not see into.

The fix was to send less. The messages table came off the wire the same day it went on, and a new table went up in its place: one row per conversation, holding a conversation id, a garage id and a timestamp. No content. When a message lands, a trigger bumps the row. Clients subscribe to the bump, and when it arrives they refetch through the normal API, where row security runs in the boring, well-understood, provable way. The realtime policy shrank to a single comparison on the garage id, which is precisely the predicate the probe had shown the realtime layer evaluating correctly.

Then the design generalized. A few days later purchase requests needed push too, and the ping table gained a topic column instead of the publication gaining a second table. One published table means one policy that has to be right. Anything else that needs a push in the future is a new value in a column, not a new thing to audit.

The direct cost was small: one design, one day, discarded. That is the point. The walk-back was cheap because the risk was priced before the feature was built on top of it. The probe existed before the publication did, the fallback was named in the design note, and so being wrong was a scheduled possibility rather than an emergency. The same discovery two months later, underneath a shipped chat feature with users on it, would have been a rewrite under pressure and an awkward conversation about why messages were on the wire at all.

Three things I keep from it. First, an asserted guarantee is not a measured one, and the difference matters most exactly where the stakes are highest. Everyone repeats that the realtime layer respects row security. It does, mostly, in the common cases. My case was one function call too deep. Second, when a dependency behaves strangely, shrinking the surface you expose to it beats getting cleverer against it. The ping design is not a workaround, it is a better design: less data in flight, one policy instead of a policy per published table, and the sensitive read path stays on the road most traveled. Third, the failure direction is part of the result. I got to walk back calmly because the system failed closed. If the probe had shown the opposite, messages leaking across garages, the correct response would not have been a redesign but a halt.

The chat has run on the ping signal ever since. Nobody using it knows any of this happened, which is what shipping the boring version instead of the clever one buys you.