Software Testability Review

Introduction

Close to 40% of the overall software development efforts is spent in testing activities which include test design, test preparation, test execution, and test result analysis. Improvement of test criteria and coverage, test automation, use of tools test analysis and test case re-use, etc are the popular methods used to bring this cost down.

Testability of a software application also has an influence on the testing efforts, making it harder or easier to test and analyse the test results. Testability is an important factor to achieve an efficient and effective test process. Organizations attempt to achieve higher testability by adopting an systematic approach which cuts across the SDLC, which is called Testability Engineering. Software Architects or specialized test architects are called in at different stages of product design and development to ensure that the testability is designed and built as part of the system. Being an important characteristic as it plays a vital role in influencing the cost of testing. Testability review technique is one of the primary technique which when adopted at the right stages will ensure high testability. This blog focuses primarily on the Testability Reviews.

Testability Reviews

Reviews in general is a popular and widely used technique by Software project teams to isolate defects in various stages of design and development. While testers do participate in most such reviews, whether they are called upon to bring out potential testing problems during these reviews needs validation. Let us see some of the testability heuristics as derived from testing practices and related literatures.

Architecture Level

Concentrate control structure related to a particular functionality in one cluster (or class). At times, Software Arcitects tend to emphasize this aspect, which could lead to dependent functions or methods being part of different clusters and in case of distributed development, this could even be with a different development teams. This will call for unnecessary development of test stubs or test drivers while testing.

Give higher priority to the modularity of a system than to the reuse of components. This one is debatable, but the reason why it is an important testability heuristics is that a higher stress on reusability may mean the non-availability of some such dependent methods or components, leading to delays in testing. One has to carefully evaluate the pros and cons and take a call on this heuristic.

Implement a standard test interface within each domain class. Implement a standard test interface within base classes and technical classes if they are selected for direct class testing. A standard test interface in all classes causes minimal additional implementation effort and has a high potential payback.

Introduce observation points at semantically meaningful points. Observation points are helpful in being able to collect or watch states and values of various objects or variables as the code execution pass through different stage. Observation points will be very helpful to debug or troubleshoot some of the hard to simulate defects.

Map your test strategy and your design approach with respect to inheritance hierarchies. Polymorphic method calls between classes within an inheritance hierarchy may require re-testing of the superclass when a subclass changes and vice versa. There are at least three different ways you can deal with this problem:

1. Avoid inheritance, use delegation instead: re-testing is not necessary.
2. Use inheritance, perform an analysis of the dependencies between super- and subclasses, and use a selective regression testing strategy.
3. Use inheritance, don’t analyse dependencies and rerun all test cases (test automation strategy).

Design Level

Make control structures explicit. Sometimes, control structure can be hidden in data, which gives room for chances of omitting important test cases there by leading to reduced test coverage. Being explicit will help testers to overcome this problem.

Avoid cyclic dependencies, especially between methods. Cyclic dependencies make the determination of test order complicated and also necessitate stubs and drivers and this could significantly increase the testing efforts.

Avoid unmotivated polymorphic method parameters, especially if strict subtyping does not apply. This could have a multiplier effect on the number of test cases or the test data as the all combinations of parameter classes need to be tested. An alternative here could be to use strict sub typing or minimize the number of parameter classes and restrict the type by appropriate casts wherever possible.

Avoid implicit inputs and side-effects. It is easy to miss implicit input and side-effects of methods, especially if they are not well documented.

Avoid unmotivated state behavior of objects.  This could lead to increase in number of test cases as it would be necessary to test each method for each object state.

Implement a state testing function for each test relevant class and restrict its invocation by test drivers if necessary. The state of an object is an important part of the test result after each test case execution but normally not accessible from the outside. Encapsulation makes testing more difficult. Breaking the encapsulation introduces unwanted dependencies between test drivers and the class to test.

Compensate test relevant information loss by built-in-tests. Built-in-test facilities like assertions provide a way to review and control intermediate computation results, thereby reducing the impact of information loss on the testing efficiency.

There has to be at least one input element (or combination of input values) for each output element. Unachievable output values (called output inconsistencies) may be indicators for unreachable statements and paths. This when avoided makes test result analysis a lot easier.

Provide means to trigger all exceptions. Some exceptions do never occur according to theory but needs consideration as it could improve the reliability of the system. It is a challenge for testers too to being able to trigger these exceptions so that the test cases cover all exceptions. A testable exception handling requires a design strategy and perhaps simulation of failure modes.

Code Level

Don’t squeeze the code. Code readability is an important factor which will be of great help when the testers are to design test cases and achieve maximum code coverage especially in case of unit testing.
Avoid variable reuse. Variable reuse leads to implicit information loss, i.e. loss of intermediate computation results and can be avoided by using more variables.

Minimize the number of unachievable paths. Unachievable paths will impact the code coverage as it would be difficult for the testers to design test cases that traverse through such paths. In order to reduce the number of unachievable paths, avoid correlated decisions.

Avoid recursive implementations of algorithms if there is no checking of invariants. It is difficult to test recursive algorithms because we cannot easily create a stub for the component or method we want to test. A solution to this problem is to split it into pairs that call each other or to use built-in assertions.

Summary

Dealing with testability issues during reviews of software artifacts should be the first choice to achieve an effective and efficient test process. Testability is not a characteristic of code alone but applies to the architecture and design as well.  The heuristics presented above are not exhaustive as more heuristics may need consideration depending on the test & design strategies, the nature of the system, the tools and technologies used for development and testing.