Risk analysis and risk management are not highly developed in the software development world. Identifying risks at the beginning of a project is often difficult. Very little use is made of earlier experiences with projects that are similar to the one you’re about to start. Also the risks that are identified are often too abstract to be of any use. E.g. “errors can harm the reputation of the company” or “security leaks can lead to serious problems” are not very helpful in getting the real issues on the table.
Also we often see that a test manager does his/her risk analysis after the functional design has been made, as part of the creation of the test plan. We are talking about both the product risk analysis and the project risk analysis. Various stakeholders and specialists give input to get the list of risks on the table. The purpose of this list is to establish which parts of the product are important and need more focus, need to be tested thoroughly, and which parts need less attention.
Risk analysis is usually divided in project risk analysis and product risk analysis. In development projects one does not exist without the other and a product risk can lead to a project risk and vice versa. When you don’t have proper resources you may need more time and the product quality could be less in the end. When functionality is missing then you may need to do a lot of testing all over and this could lead to project risk with respect to time and money. Just a couple of examples.
You always hear project managers and test managers talk about separate responsibilities. Sure: the project manager is responsible for the whole project and within that project the test manager is responsible for the testing. But is that a reason why the project manager should do his own risk analysis and the test manager after that should do it all over, more focussed on testing this time? What a waste. A lot of meetings, a lot of pointless discussions. All for the “stakeholder management is important” and “communication is important”. OK, all people that are important for the implementation of the product (or service or whatever) should be involved. Definitely. The product must be accepted in the organization. But can we do the risk analysis without wasting a lot of time? Yes!
The risk analysis (both product and project risk analysis) should have been made during the functional design phase, during which developers, users, product specialists, managers and also the test manager should be involved. The only way to identify all the risks is when all people involved work together.
Let us return to our basic triangle. At the start of the project the business case is defined. In the business case a brief outline what the result of the project should be and what is required to get to that result. During the analysis phase the project risks and if possible the product risks on a high level are defined. We are now at the starting point of the development process, the top of the triangle. Already an estimate of the size of the project is made and the possible problems that can be expected. Already in that phase representatives of the various parties involved should participate in the risk analysis of the project and product risks.
Continuous process illustrated in a triangle
A Continuous process
From then on, risk management is a continuous process. During the analysis phase it becomes clearer what the product will look like, what functionality it should contain, what properties it should have. And it becomes more clear what is required to make this product, which and how many men, which means en which methods. Also it will become clearer what quality should be achieved at what price in what time.
At the next evaluation point, at the end of the first analysis phase, all the representatives of the parties involved should get together and analyse the current state. What risks do we have? What is the expected cost? How many men do we need for the rest of the development process? Do we still need the means and methods that we thought we did or do we need to make adjustments? What are the consequences with respect to time and money of these changes? After this evaluation point we decide how to move on. What are we actually going to develop? What are the risks that we know at this point? During the realization phase the risks are continuously monitored by the project manager. Measures are taken to avoid, reduce, accept or transfer the risks. Project management is mostly risk management. As far as the risks apply for developers the risks are monitored and dealt with by the developers or their manager. The test manager monitors and manages the risks that have to do with testing. Etcetera. The project manager makes sure that all the risks are managed.
At the end of the development process again an evaluation in which all the parties involved get together. Again the scope is redefined; the men, means and methods are evaluated along with the quality, time and cost. Since we are now at the start of the testing the role of the test manager increases. The tests are already defined during the realization phase (V-model) and can now be executed. Now we know better then before which parts are critical and need focus during the testing phase. The project and product risk in fact have now almost become test project and test product risks. Anyway, the test manager has to adjust the testing strategy to the risk status at that point. Of course he already did the risk analysis earlier and the risks that he came up with were added to the risks of the project manager. The total project risks and product risks are the sum of the risks from analysis, development, testing, control, management… of every party involved.
During testing findings lead to fixes and small changes. And the risks become clearer and clearer. Risks are resolved and are removed from the list. Other risks are added or appear to be more severe than expected. E.g. test environments are not working properly, or depersonalisation of data is a hotter issue then anticipated. At the end of the whole process, back at the top of the triangle, we know exactly what the outcome is, what the quality, time and cost are. This is the time for go-nogo. Can we go to production or not? Did the requirements change and do we need to do extra work? Now usually requirements change during the whole project and it is up to the project manager to manage all that. And the involved parties have a say in the matter and should be involved at all times. The project manager is not the only one who decides what the product in the end will look like.
Qualification of Risks
Qualification of risks is a subjective process. It can vary per person. It can vary per department. That is one of the reasons that all parties involved should qualify the risks independently. In general the following classification of risks can be defined
- High: important, used a lot, high impact in case of failure
- Medium: normal use, normal impact, irritating in case of failure
- Low: not very important, low impact in case of failure, not often used
Of course if something less important is used a lot it could be classified as medium etc. Of course high risks should be taken more seriously then others. High project risks should lead to thorough measures, high product risks should lead to thorough reviews and testing. Test cases for high risk process parts should be defined on the deepest possible level. E.g. for an important calculation module the test cases should cover all variations of true and false situation that you can think of. All this speaks pretty much for itself. Also these parts should be integrated in an integration test. In case of light testing it could be sufficient to use only one modelling technique to define the test cases. In case of severe testing the use of more than one modelling technique could be required to get all the situations that need to be tested on the table.
The risks can roughly be divided in project risks and product risks. It is always good to keep in mind that product risks could lead to project risks and vice versa. To create the list of risks the development triangle can be a source of inspiration. To support the analysis, realization and testing you need the proper men, means and methods after which the optimal quality, time and cost should be realized.
For establishing the project risks the following checklist could be helpful:
- Is safety/security an important issue for this application?
- When you create a planning, which parts of the development process are unclear/uncertain?
- Does documentation have to be translated?
- How many resources are available and is the knowledge level good enough to do the project?
- How much training of personnel is required? Does knowledge have to be transferred, e.g. by off shoring?
- Is it clear how much functionality has to be developed?
- Is it clear how the functionality has to be tested?
- Is it clear how complex the functionality will be?
- Are the stakeholders involved in the design and the product risk analysis?
- And more…
To be able to do a proper product risk analysis it is important that you know what the product properties are. In the business case stage of the project these properties are know only on a high level. These properties on a high level should be identified and risk classes (H, M, L) should be designated to those properties. During the functional design stage these properties are worked out in detail. The various modelling techniques (ERD, DFD, process flow diagrams etc) can be very helpful to check the consistency of the design and identify the product ( and process) details. Now what we do is make a list of all these properties and identify the risk. During the course of the project the risk values can change and also the product property list can change. Management of this list is a continuous process.
When you try to designate a risk class to a product property you answer the following questions:
- Is the property vital for the application as a whole? Does it execute vital functions?
- If that property does not work correctly does that affect the output of the application?
- Is the output of the product part sent to the customer?
- Is the property only used in the GUI?
- Is the property a link in the process, which means that when the part fails the whole process will fail?
- When the property fails is that a business risk or just annoying?
- Is the property part of a calculation module or part of a batch process?
- And more….
Usually product parts have properties, properties are part of processes. After you made the list with the properties, product parts and risk classes you know which parts of the application need extra attention, during analysis, realization and testing.
Project risks can lead to product risks and vice versa. So the risks for a project should be kept together in one central risk management system, which could be an excel sheet or a more advanced tool. Also a findings registration tool could be used to do this, or a requirements management tool. Then the risks should be assigned to a monitor who is responsible for the management of that particular risk for which of course the final responsibility lies in the hands of the project manager. In this risk management system per risk it could be indicated what kind of risk it is, e.g. project risk, product risk etc.
As we said earlier: risk management starts at the very beginning of the project and only stops after the project has finished. The list of risks is updated continuously and is evaluated continuously together with the stakeholders and product specialists. There are various techniques to do the communication with the stakeholders, like interviews, risk analysis sessions (meetings with all the stakeholders), or just simple emails or phone calls. Most important about risk management is the involvement of the stakeholders, for acceptance of the product and the various parts of the development process.
Beside identification of the various risks it is also important to identify and communicate the measures to be taken to avoid or resolve the risks. So these solutions must also be communicated and discussed with the stakeholders.
The project manager is responsible for the whole project so he must identify who has to do what to address the risks. The risks have to be addressed in all the phases of the project, in all the disciplines involved: analysis, realization and testing. In the risk management system beside the risks also the measure have to be registered plus who must execute the measures.
All disciplines involved can use the models that are created for the project, by analysts (e.g. architects, test analysts). All parties involved should review these models to make sure that everybody speaks the same language, is talking about the same functionality. Especially in off shoring projects this is important. The models are important for communication.