For fifteen months I ran payments connectivity at a high-volume consumer payments platform. Card deposits, bank transfers, wires, and the machinery underneath all of it. By the end, payment failures were down about 40%.

Almost none of that came from writing better payment code.

It came from working out, later than I’d like to admit, that a payment stack isn’t a system you build. It’s a supply chain you run, and I’d been managing it like an engineering project.

Some failures aren’t bugs, they’re somebody’s standing policy #

The thing that broke my mental model was a decline pattern. Card deposits were failing at a much higher rate in some markets than in others, and every obvious reading was wrong. It wasn’t a bug, it wasn’t a bad integration, and it wasn’t concentrated in one provider.

It was category. Issuing banks take a view on whole classes of business, that view varies by market, and it reaches you as a response code. The decision had been made somewhere we had no access to, by people we would never speak to.

You can’t retry your way out of that. A great deal of payments engineering is retry logic, and retry logic rests on the assumption that a failure is a transient accident. A refusal that reflects a standing policy comes back identical on the second attempt and on the tenth. All you’ve bought is a worse authorisation rate and an irritated counterparty.

What worked was giving customers more ways to pay. A card-network wallet still reaches the same issuer, but it arrives carrying different signals, and genuinely alternative methods sit on different rails entirely with their own consumer relationships. Either way the transaction takes a different acceptance path, so somebody who couldn’t complete a card deposit could complete a payment another way. It’s the ordinary response to a supplier who can only serve part of your market, which is to add a supplier who serves the rest.

In supply chain terms this is unremarkable. If a supplier won’t ship to you, you don’t send more purchase orders, you qualify a second supplier. It took me longer than it should have to see a payment provider that way, because payment providers arrive dressed as APIs, and an API feels like something you integrate rather than a relationship you manage.

None of this is new. Manufacturing has had a mature vocabulary for this for decades: supplier qualification, dual sourcing, scorecards, single-point-of-failure analysis. Finance has its own long history of it too, in correspondent banking and multiple acquiring relationships. The strange part is how rarely any of that vocabulary reaches the engineers wiring up the integrations, because our suppliers turned up as endpoints and endpoints don’t look like suppliers.

Coverage is a supply chain problem wearing an engineering costume #

One of our data providers had a fallback chain for connecting a customer’s bank account. Instant authentication covered the large majority of institutions. Below that, an instant matching method picked up a couple of thousand more. Below that, automated micro deposits, which take a day or two, reached institutions the first two couldn’t. At the bottom sat a slower method that existed purely as the catchall.

Four ways to do one thing, each slower and less pleasant than the one above it, each reaching customers the one above it couldn’t.

The engineering instinct is to build the top tier beautifully and treat the rest as edge cases to be handled later, which in practice means never. The supply chain instinct is the opposite, because the tail isn’t an edge case, it’s the segment of your customers a single supplier cannot serve at all. Whether you build the bottom tier decides whether those people can use your product. Nobody wants to build it. It’s slow, it’s ugly, it has the worst conversion of anything you’ll ship that quarter, and it’s the difference between covering a market and covering most of a market.

Qualify suppliers on how they behave when they’re broken #

I sat through a lot of vendor evaluations, and I gradually stopped asking about features.

What I ended up asking was narrower and duller. What are your API standards. What does your documentation actually look like. Do you offer an SLA. Do you have a public status page, and is there any history on it.

One evaluation turned on exactly that. What we needed was on the vendor’s roadmap rather than in their product, and the question that settled it wasn’t whether they would get there, it was whether they would get there in time for us to build against it. The honest answer was no. That’s a scheduling judgment about a supplier rather than a verdict on their engineering, it feels unfair to make, and it’s usually right.

A feature list tells you what a vendor wants to sell. A status page tells you what they do when they’re failing, and specifically whether they expect you to hear it from them or from your own users. I’ve never regretted weighting the second more heavily.

Silence is the failure mode that actually hurts #

A provider’s callbacks stopped one evening. Not errors. Nothing. Payments sat in pending, waiting for a confirmation that wasn’t coming.

The interesting problem wasn’t theirs, it was the one it created on our side. Any system that expires stale transactions has to decide what an un-acknowledged payment means, and from the inside, un-acknowledged and failed look identical. Get that wrong and you resolve a queue of perfectly good payments the wrong way, at precisely the moment you have the least information available to notice.

Systems handle errors reasonably well, because an error is information. They handle silence badly, because silence is indistinguishable from “not yet”. If you design for a supplier that returns failures, you’re fine right up until the supplier stops returning anything at all.

The answer starts with instrumentation: freshness checks on the callbacks, reconciliation against the provider’s own view, explicit stale-state detection rather than a timer that assumes the worst. But somebody still has to own the signal, so in practice it also means an on-call rotation and a name against it.

Second-sourcing has to be cheap or it won’t happen #

Everyone agrees with dual-sourcing in principle. Almost nobody does it, because adding a provider usually costs a quarter of engineering time and there’s always something more urgent than insurance.

What made it real for us was structural. The card side was built so the business logic knew nothing about which provider it was talking to, with everything provider-specific isolated behind a single boundary. Adding or removing a supplier was contained work rather than a project. Above that sat routing, so which supplier handled a given payment could be decided from attributes of the payment itself rather than being fixed in advance.

Once that exists, second-sourcing stops being a strategic initiative that needs a business case and becomes an ordinary piece of engineering work. That’s the whole trick. The organisations that manage supplier risk well aren’t more disciplined than everyone else, they’ve made the correct thing cheap enough that discipline isn’t required.

Supplier management with a compiler attached #

I went into that job thinking connectivity meant integrations, which is what the word sounds like. Fifteen months later I’d stopped.

The 40% wasn’t primarily a better-code achievement, and I’ve stopped presenting it as one. It came from asking a set of questions that had nothing to do with code. Who are our suppliers. How do they behave. Which of them can’t serve which customers. What happens when one of them goes quiet on a Friday night.

The habit is hard to shake, and it’s the one that took fifteen months to notice in the first place. Everything about the way an API presents itself encourages you to believe the hard part is the code, and I still catch myself doing it with a new vendor.