Why do we want to minimise end-to-end testing?

End-to-end testing is the white whale of software quality. The value is obvious, but the challenges are huge and we give up too much in the chase. It requires careful management of test data across internal and external systems, is subject to environmental problems, is slow to run and time consuming to maintain. There are also some more complex challenges, particularly in companies like ours that deploy each piece of software independently.

Many teams invest heavily in establishing and maintaining a layer of end-to-end testing, but few ever reach a level of maturity that allows them to derive significant value. Even if all the challenges are overcome, they still end up with a slow feedback loop and all the inefficiencies that creates.

At SadaPay we prefer to get feedback as fast and early as possible, so we can’t rely on end-to-end testing. One of our favourite approaches is to make creating non-working integrations almost impossible at build time, so that we don’t have to validate correctness as thoroughly at runtime. Imagine being able to tell right from your IDE whether your integration is correct, before you even run your tests!

‍

How do we do it?

To do this, we use code generation from shared API specifications. We write OpenAPI 3 YAML files to document the intended behaviour of our APIs. Then, we use OpenAPI code generation tools and GitHub Actions workflows to automatically generate the clients (used by the SadaPay App to call our APIs) and the controllers (used by our back-end to implement our APIs).

‍

‍

That means for every API at a SadaPay, there is a typesafe generated client that only supports creating correctly structured requests and a typesafe generated controller that only forwards correctly structured requests to our hand-implemented delegates (which override generated defaults, to guarantee that the interface is still correct).

For API consumers, producing an incorrectly formatted request or hitting the wrong URL becomes almost impossible. For API providers, it is impossible to accept anything other than a correctly formatted request. The chance of integration failure is radically reduced, while also minimising the need for defensive programming on both sides.

Trade-offs

One of the compromises of this approach is that it is difficult to be consumer-driven. We have to write the API specifications close to the start of the development lifecycle and that front-loads some decision making that we would normally delay in order to reduce the chance of unnecessary complexity, poor fit or rework. On the other hand, this does require us to get together and discuss the API design in a holistic way and that is often very constructive.

It also makes API changes relatively high friction, which creates some pressure to get things right the first time. Ordinarily we’d prefer to let our API design evolve organically as our understanding grows. On the other hand, making safe API changes when supporting multiple versions of a mobile application (like we do at SadaPay) can be very tricky and being forced to think each one through carefully might be a good outcome, even if it creates some friction.

Finally, code generation from OpenAPI specifications is not quite as mature as we would like. Different features of the specification are sometimes unsupported or handled differently by the different code generation tools. We’ve had to customise our templates in order to produce consistent results, which required some deep learning. That hasn’t been easy and we will probably need to go even further, making open-source contributions to the tools themselves.

Moving forwards

We’re getting a lot of value out of this approach, but we can see it is not the only solution. We could use contract tests to delay decision making and be more consumer-driven, but this would delay feedback compared to our current approach. A combined approach would be interesting – publishing a contract based on contract tests run by the API consumer and using them to generate controllers for the API provider.

We’re also considering how we will incorporate more end-to-end testing  in the future, after we have firmed up lower levels in our test pyramid. In particular we’ve discussed integrating our microservices together using k3d and running them against mocked external services. This way we could get fast feedback in development and CI, control our test data and not have to involve external partners. It’s not exactly end-to-end, but it is more end-to-end than testing each service in isolation.

Interested?

If you’re interested in what we’re doing at SadaPay, come talk to us! Consider applying with us as a Full Stack Engineer or QA

‍