Improving Wetware

Because technology is never the issue

Economics of Maintaining Tests

Posted by Pete McBreen 13 Nov 2021 at 00:40

Thinking about the testing pyramid that is the common picture used in many presentations, typically highlighting Unit, Functional and UI testing, and if the diagram comes from an agile background, it will put manual testing at the top and claim that manual testing is the most expensive. The claim is normally that automated unit tests are cheap to write and fast to run, automated functional tests cost more to write and run slower, and the automated UI tests are even more expensive to write and slower to run. Manual UI tests are at the top of the pyramid, the most expensive to write and the slowest to run.

The reality may be different, but that depends on your viewpoint.

Automated testing is not actually testing. It is Automated Checking, algorithmically checking conditions on the UI. An automated check can pass if it is only checking a few items on the UI, when some other parts of the UI have changed for the worse, but are not checked. The checks pass, you have a green, but the UI is broken.

So at this point the question is whether manual UI testing is more expensive than automated UI checking, if even a cursory scan by a tester would see the problem but the automated UI check would not see the problem. Yes we can check the entire contents of the UI in the automated check, but then the code doing the checking will run a lot slower, and be more extensive. The cost of maintenance gets much larger as well, since now ANY change on that page, such as an intended refactoring, will need the automated checking code to be adjusted.

My suggestion for addressing this is to make sure that whenever a manual UI test discovers a mistake, the automated checks need to be updated to detect that problem. This makes debugging the problem and fixing the bug easier, and provides an in built regression test for the future. Ideally add a unit test if that is the smallest test that can reveal the defect, otherwise use a Functional test or a UI test if that is what it takes to reveal the error.