Why we publish every incident, including the embarrassing ones
We write a public post-mortem for every incident that affects customers, including the ones caused by our own straightforward mistakes. Here is the reasoning.
Last month we had a forty-three minute partial outage. Documents could be viewed and signed; new envelopes could not be created. It was caused by a configuration change one of our engineers made, reviewed by another engineer, that neither of them nor our tests caught.
We published a post-mortem naming the change, the gap in our testing, and the timeline. We do this every time. Several people have asked why, given that nobody makes us.
Because you are trusting us with your contracts
People store their most consequential agreements here. Employment contracts, property transactions, settlements. That is a significant amount of trust, and trust is not established by claiming nothing ever goes wrong.
Everything goes wrong sometimes. The question a customer should be able to answer is not "does this vendor have incidents" but "when they do, do they understand what happened and fix it properly". You cannot evaluate that if the only information available is a status page that said "investigating" and then "resolved".
Because vague post-mortems are worse than none
The genre of incident report that says "an issue with a third-party dependency" and "we have implemented additional monitoring" tells you nothing. It is written by a communications team to minimise concern, and experienced readers correctly discount it entirely.
A report that says "we changed a connection pool setting, our staging environment does not carry enough load to expose the difference, and here is the specific test we have added" is useful. It is also more uncomfortable to write, which is presumably why it is rarer.
Because it changes internal behaviour
This is the reason we did not anticipate. Knowing a post-mortem will be public changes how the team approaches reliability — not through fear, but because "would I be comfortable explaining this decision publicly" turns out to be a useful design review question.
It has also made our internal reviews more honest. When the report is public anyway, there is no value in an internal version that soft-pedals the cause.
The rules we hold ourselves to
- 1Published within five business days of resolution
- 2No individual is named — incidents are caused by systems that permit mistakes, not by the person who made one
- 3The actual root cause, not the nearest convenient dependency
- 4Remediation with real dates, and a follow-up if a date slips
- 5Customer impact stated plainly, including how many were affected
The blameless part matters. An engineer who fears being named publicly will be slower to raise a problem, and the delay is far more damaging than the original mistake.
What we do not publish
Security incidents follow a different process. We notify affected customers directly and promptly per our contractual and regulatory obligations, but details go out on a timeline that does not hand a working technique to anyone else while systems are still being hardened.
We also do not publish specifics that would identify an affected customer. Impact is described in aggregate.
The one from last month, in full
Concretely, since abstract commitments are easy. At 09:14 UTC we deployed a change reducing the database connection pool size, intended as a cost optimisation. Our staging environment runs a fraction of production load, so the reduced pool was never saturated there and the change looked clean.
In production, envelope creation is the most connection-hungry operation we have. By 09:31 the pool was exhausted during a normal traffic peak. Creation requests began timing out. Reads were unaffected, which is why signing continued working and our top-level availability alert did not fire — the endpoint it checks is a read.
We noticed at 09:38 from customer reports, not monitoring. That is the part of this incident we consider the actual failure. The rollback took four minutes once we identified the cause; the seven minutes before anyone knew there was a problem is what we have spent the remediation effort on.
- Added per-endpoint availability alerting rather than a single health check
- Added a connection-pool saturation alert with a threshold well below exhaustion
- Changed staging to run a synthetic load floor representative of production peak
- Added infrastructure capacity parameters to the change categories requiring a load test
All four shipped within three weeks. The load-test requirement is the one that would have caught this, and it is also the one that makes deploys slower. We think that is the correct trade.
Where to find them
Every post-mortem is on our status page, kept indefinitely. If you are evaluating us as a vendor, we would genuinely encourage reading a few. They will tell you more than our security page does.
Jordan leads engineering at SignTheDoc. Before this he built payment infrastructure, which is where he learned to care about audit trails.