When it comes to selecting commercial-off-the-shelf (COTS) compliance software there was a time when this involved a structured process based on a requirements-first approach. This has now been largely replaced with a demo-first approach encouraged by cloud vendors as well as by the buyers themselves. Instead of a bake-off compared against requirements, software is now chosen based on how well the software demos and looks. Does this approach result in better outcomes? Let's find out.
Requirements-first Approach:
A requirements-first approach typically includes the following steps:
Request for Information (RFI) – survey of the market to identify candidate vendors and solutions
Request for Proposal (RFP) – request for written responses to requirements from identified long list of candidate vendors.
Short Listing – a short list is created based on the selection criteria rubric
Request for Quotation (RFQ) – obtain firm and final pricing from the short listed vendors
Live Test Demonstration (LTD) – make sure the short list of vendors actually meet the stated requirements by following a scripted walk-through.
Select Apparent Winning Offeror – selection of the best alternative based on vendor performance, fit for purpose, and technical requirements.
Pilot System - validation that the solution can achieve the intended outcomes as well as the verified technical requirements.
The purpose of following these steps is to manage risk inherent in selecting a solution that best fits the scope, budget, and requirements. In addition, it also creates level of playing field, keeping everyone honest on both sides of the table.
The following data is a compilation across 20 projects that followed a requirements-first approach:
* Waterfall = Gated, Structured Approach
* Hybrid = Gated, Agile Approach
Key lessons learned from these projects include:
System scope was the major influence in determining overall procurement cycle time. However, there is only an incremental increase (8 versus 12 months) when considering departmental versus platform solutions.
The overall duration was largely determined by vendor and buyer schedules
Waterfall approach using approval gates was preferred the larger the project scope
In addition, 90% of projects did not purchase their first choice for reasons that included:
Failed live test data (LTD) – RFP responses was good but based on software that was not yet available or didn't withstand scrutiny of actual use.
Failed pilot system due to poorly understood or specified requirements
Requirements changed during the procurement process
It is worthwhile stating that each project completed successfully even though it was not with the first choice of vendor. Having a second choice proved to be a significant factor when mitigating the uncertainties experienced during the procurement process.
Demo-first Approach:
These days it seems that many companies jump right to requesting a demonstration of software without first understanding what it is that they need. While this may prove successful for some applications, when it comes to critical compliance solutions at the scale of the enterprise this can lead to decisions that are less than optimal and waste valuable time, resources, and possibly exposing companies to unnecessary risk.
Companies who have used the demo-first approach have noted that these projects tend to produce the following:
Scope creep – everyone wants all the capabilities that they see demonstrated
Difficulty in making an apples to apples comparison of the alternatives
Cost overruns due to unplanned integration, customization, and data migration
Schedule overruns leading to late ROI and in many cases unrealized benefits
Solutions that only meets rudimentary requirements and not capable of meeting the full demands of the organization
Loss of data and information due to insufficient planning and resourcing for data cleansing and migration activities
In addition, projects still end up taking the same amount of time to procure a solution as with a requirements-first approach. Although, in the case of a demo-first approach they tend not to follow a risk-based process. This makes them vulnerable to uncertainties that an RFP, LTD, and Pilot steps would have discovered.
Companies have also noticed an increased tendency to choose software that may have: demoed the best, had the most capabilities, had the lowest initial cost, or the one that was used at the last company that someone worked at. In other words, without a set of requirements there was no basis on which to make an effective comparison based on actual and anticipated need.
It would be reasonable to ask why companies would choose a less rigorous process for selecting compliance solutions. Here are some of the reasons given:
Our current system doesn't work and we need something else but we don't know what that looks like
I don't know what I need so looking at software helps me figure that out
All I want is something that is user friendly. I expect the vendor to know what my requirements are.
This is off-the-shelf-software so why do I need to write down any requirements. Don't they all do the same thing.
I am just looking to replace what I currently have so those are my requirements
We are looking at cloud-based software and the subscription costs don't warrant a large project
Our business analysts used to do that but we don't have those roles anymore
I don't have the time to go through a structured process.
We are following an agile approach which means we don't need to figure out what are requirements are right now
Even if it the software doesn't work we can replace it easily because its all in the cloud
As more organizations move their systems over to the cloud it is expected that the use of a demo-first approach will increase. Of course each company will have different levels of success, however, the probability of success can still be improved by effectively managing uncertainties specifically with respect to scope.
Risk-Based Approach:
Acquiring software to support critical compliance processes still requires that risks be properly addressed. The most significant source of risk hasn't changed and is still scope creep or scope gallop as it often the case. Managing scope is essential to every project and this applies to choosing compliance software.
Software demonstrations can be an effective way to learn about what is available in the marketplace. This in many ways has replaced the use of RFIs. However, demos do not replace the need to specify what the software needs to do or the need to manage risk.
Requirements may not be as detailed as they once were and may take the form such as user stories. At the same time, they still must be sufficient to cover what the software contractually needs to deliver and how it needs to perform in order to achieved the desired outcomes. It is always good to remember that you are not the product the software is.
In addition, as previously noted, it is a good strategy to always have a second choice because your first choice is likely not the one that will achieve the desired outcomes.
Whether you follow a demo-first or requirements-first approach or not you still need to get answers to the same set of questions. The timing of when you get these answers will significantly influence the success of your project.
If you wait until after you purchase the software you will need to deal with the effects of not knowing or what is called, "epistemic uncertainty." The risk of not knowing can and often leads to failed projects that in many cases doubles the cost since the project has to be done over again.
Here is list of items that some companies chose not to know in advance:
The importance of integration with other systems and consequently neglected during the procurement phase
The value associated with legacy data leading to no budget for data migration
The loss of control over how processes are implemented resulting in the using vendor generic workflows
The impact of using generic approaches that were sub-standard to the company's higher standards
The lack of understanding of how an on-demand pricing model would be affected by a fixed operating budget
The lack of understanding of how the software is going to be transitioned and rolled-out
All of these could have been known in advance and addressed using a requirement-first, risk-based approach.
Here is a list of things that you should know when selecting compliance technology:
1. What defines success?
What are the intended outcomes for the system?
What defines what done looks like?
How do you measure progress towards done?
What steps are critical to achieving done?
What risks need to be addressed that hinder achieving done?
What opportunities should be pursued to increase the likelihood of getting to done?
2. What is the purpose for the software purchase?
Technology replacement?
Architecture alignment?
Process improvement?
Improved compliance?
New capabilities?
Increase or decrease in scale or complexity?
Cost reduction?
Introduction of best practice?.
Point solution or platform to support multiple solutions.?
3. What are all the requirements for the expected use of the software?
System, application, process, and other functional requirements?
Compliance, security, data, privacy, and sovereignty requirements?
Platform, network, communication, and other technical requirements?
Performance, and reliability requirements?
Customization, and integration requirements?
Implementation, sustaining, and end-of-life requirements?
Backup and recovery requirements?
4. What strategies will be used to introduce and sustain the use of the software?
Lift and Shift - Improve processes first then shift?
Shift and Lift - Shift to the new software first and then improve processes?
All users at once or a phased roll-out?
All modules at once or a phased roll-out?
Distributed or centralized support?
Business owners or IT support?
4. What are the impacts and risks associated with the choice in software, implementation strategies, and sustaining activities on the business
What gaps in requirements need to be addressed by customization, work-around, or additional software?
What is the total cost and budget needed to sustain and use this software over its anticipated lifetime?
How is compliance maintained during and after the implementation?
How will changes to the software or configurations be managed and validated?
What actions are needed to address uncertainty in: capabilities, cost, user acceptance. ability to meet compliance obligations, and so on?
Who owns the data and will the data be monetized by the vendor?
How and when will breaches in service be communicated?
What is your exit strategy and when will this be triggered should you need to revert to your second choice?