How will you know if your product is good enough to satisfy the objectives of the project? There can be too little – or too much – quality baked into the product if you don’t think it through carefully. In fact, you can have too much or too little quality in each component or sub-part of the product if you do not carefully define your test criteria. It’s not just a matter of listing out the requirements; it’s a matter of looking at the totality of what will make up the product and distill it down to digestible parts – and defining the criteria there. This article looks into this process – its inputs and outputs – of defining the product test criteria.
This is the third of a series of four articles on “Writing a Test Plan”, where we explore the challenges and favored approaches for developing and implementing a plan to test and integrated product consisting of hardware and software components. This article, Part 3 in the series, “Define Test Criteria,” provides ideas for how to measure the various aspects of testing to determine what passes and what fails. Part 1, “Product Analysis and Test Objectives”, looks at how to get your head around a concise and useful set of objectives for the test effort. Part 2, “Plan Test Resources,” looks at the variety of resources required for effective testing and how to plan for them. Finally, Part 4, “Test Strategy, Schedule, and Deliverables”, explores the “nitty gritty” project management aspects of the test plan.
The most fundamental place to look for test criteria are the requirements. Here are the five key areas to consider:
- Functional requirements – These define what the product is supposed to do - or what tasks the product must perform.
- System/physical requirements- These define performance requirements, such as how fast, how far, how high…. They also include ‘how much’ – such as now much data needs to be processed in a given time, and how much data needs to be stored (this is otherwise known as load testing).
- Threshold vs objective – Threshold requirements must be met, and objective requirements are ‘nice to have’, or acceptable to have in the future.
- Unit testing to full threads – Different requirements lend themselves to different types of evaluation. Unit tests evaluate isolated tasks, where full threads evaluate a complete set of interrelated tasks.
- Install/uninstall and setup/shutdown – If the product involves procedures for setting up, you’ll need to include them in the evaluation criteria.
- Environmental requirements – Evaluation criteria may include results from both simulated and real environments. You’ll need to balance considerations for efficiency and certainty in evaluating fitness for use.
Once you’ve thoroughly identified what you need to test, you’ll need to sharpen up your criteria with the following four considerations:
- Criteria for passing – For each of the isolated requirement areas from above, you will need to establish what is a passing test result.
- Entrance/exit criteria – It is critical to set up a gated process to the testing that includes, for each test to be run, the criteria that must be satisfied in order to even begin the test, as well as the criteria that must be satisfied to end the test.
- Test metrics – Each test will have a result, but it may be helpful to identify some useful metrics across the various test results. These could include such measures as run rate (number of tests run over a particular period) and pass rate (proportion of tests resulting in a passing grade).
- Definition of Done – One of the most important things to define represents what is good enough.
Have you identified everything you need to test, and what represents a passing grade for each?