2019 · Fintech · Lead · Australian fintech (NDA)
Rescuing delivery on a fintech CRM
A Python CRM for an Australian fintech where regressions kept reaching production. Fixed how changes were made rather than rewriting what they were made of.
Bug rate down ~30%, delivery cycles shorter, and a 500K-record/day sync the platform could carry
Context
A Python CRM for an Australian fintech — client management, authentication and authorisation, granular permissions, and the financial data flows underneath all of it. It worked, it had real users, and it was getting worse to change.
The symptom was regressions. Work that had been finished months ago would break when something nearby was touched, and it was found in production rather than before release — sometimes by the client. Each one cost twice: the fix, and the confidence lost in the part of the system it came from. A team that has been bitten a few times starts routing around the code that bit them, and the routing around is what quietly turns a working system into a legacy one.
Meanwhile the platform had to absorb more: synchronising financial data across third-party integrations at a scale the existing way of working could not survive.
Decisions
Chose Tightening how changes were made, while the platform kept shipping features over pausing delivery to rewrite the parts that kept breaking
The code was not the problem — the absence of anything catching mistakes was. A rewrite would have bought a fresh codebase with the same habits around it, and spent months during which the client got nothing. Fixing the process is unglamorous, it works on the system you already have, and it starts paying in the first week.
Chose Automated tests on the paths that move money, before anything else over working towards broad coverage across the codebase
Coverage as a number is a poor target: it is reached fastest on the code that matters least. Payments, the financial integrations and the permission checks are where a regression costs the client real money or exposes another client's data. Those paths were worth a disproportionate share of the effort, and everything else could wait.
Chose A release process identical every time, and review with something at stake over relying on the care and seniority of whoever happened to be shipping
Care does not scale and does not survive a deadline. A team that grew on this account needed the standard to live in the pipeline rather than in people's heads — which also means a new engineer can ship in week one without anyone standing behind them.
Chose Building the 500K-a-day synchronisation for failure from the start — idempotent, retryable, tolerant of late and duplicate data over treating third-party integrations as reliable and handling exceptions as they appeared
At that volume every failure mode happens daily, including the ones that look theoretical. Designing for repeat and late delivery up front is cheaper than reconciling a financial dataset that silently drifted because a provider sent the same record twice.
What I did
I tightened how changes were made rather than what they were made of. Review standards with something at stake — nothing merged unreviewed. Automated testing aimed first at the paths that move money: payments, the third-party financial integrations, the permission checks that decide who may see a client's data. A release process that was the same every time instead of a careful person remembering the steps.
Underneath that, the work the pressure was actually coming from: data synchronisation handling 500,000+ records a day across Stripe, Xplan, Google and AWS integrations, built on the assumption that any of them can fail, repeat or arrive late. That is not a thing you attempt on a codebase where you do not trust the change you made yesterday — which is why the process came first.
Outcome
The bug rate dropped by about 30%, and delivery cycles got shorter — the second being the one that surprises people, because tightening a process sounds like adding friction. It is the opposite: most of the old cycle time was spent verifying by hand, waiting for someone senior to be free, and fixing what the last release broke.
The platform took the 500K-a-day synchronisation and the integrations that came with it. I stayed on that account for four years, growing from developer to Tech Lead, and the same team continued through Steelkiwi's acquisition — the code outlived the arrangement it was written under, which is the only durable test of this kind of work.
In hindsight
I never took a baseline. The 30% is reconstructed after the fact, from issue counts and release history, rather than measured from a starting point I recorded deliberately — and that is a weaker claim than it needed to be. A week of work at the start, writing down the bug rate, the cycle time and the incident count, would have turned the whole engagement from a story into evidence.
I now treat that first week as part of the job rather than a preamble to it. If you cannot say what the number was before you arrived, you cannot prove what you changed — and on this kind of work, proving it is most of what the client is buying.