When implementing a new or improved content management system, I find that many teams don't have time or the staff to perform what is known as "user acceptance testing (UAT)". But let's face it: end users are going to either accept or reject the system components eventually, and it's much less painful to get through this process in a controlled manner during the project than during production.
During project testing, participants expect to find bugs and issues. The development team is still available to fix the issues. The team can take disruptions in stride, since issues only disrupt testing, rather than production schedules. Failures are part of the normal process.
After the system is released into the wild, system end users who were not involved in the project start working and expectations are high: "This system has been released, so I’m assuming that everything works", they're thinking. Newly discovered issues are upsetting and disruptive to production schedules. The project developers may have been reassigned to other work already, and a team may need to wait for fixes, which exacerbates the problem.
So how do we balance the realities of resource constraints with the obvious need to test?
Let’s look at it in terms of quality. A generally accepted definition of quality is that something meets specifications ("is it what we ordered?") and is fit for use ("is it fast and intuitive?").
The UAT process runs in a tightly controlled environment and excels in proving methodically that all specifications have been met. It helps in answering the general questions "how will we know we're done?" and "how will we know we're successful?" This is ideal from a systems vendor or IT department perspective; because it allows a neat line to be drawn between the project and handoff. The project is nicely wrapped and packaged for delivery. For the end users, the disadvantage here is that everything rests on how well requirements were defined in the first place. It’s very easy to miss something that can cause trouble later on.
A team can also release a system as a beta version or as a pilot. Instead of a few people running dummy content through test scenarios, a sub-set of actual system users can move through an actual workflow using actual content. This approach helps the group move forward on their production schedules, while maintaining a relatively controlled environment. And, because it's a smaller set of users, the group's expectations can be set so that issues don't freak them out. Since the team is operating in a live environment, they can identify critical issues that may have been missed during the requirements gathering stage. A live environment is ideal for proving fitness for use.
However, it may not provide a very thorough quality check against the stated requirements, since there may be exceptions in the process that won't occur during the beta period, such as an end-of-quarter rollup, report, or content delivery. The pilot approach works only if the end to end process is sufficiently short enough to get through within project timeline and cost constraints.
Ideally, we’ll use both approaches. At a minimum, it’s smart to spend some time with at least a couple of team members to take the system for a "test drive" assuming multiple roles, to ensure that all specifications have been met and the pilot period starts smoothly. Then following that with a pilot enables the end users to determine fitness for use.
I'm always looking for the right balance. What's your experience?
Recent Comments