Systems development

Do new applications always have to go wrong?

Recent news reports[1] have highlighted IT glitches after several NHS Trusts have gone live with new technology. But with many trusts accelerating plans to deploy new applications under the National Programme for IT (NPfIT), what can be done to reduce the risks? And why do so many high profile new systems, both in the public and private sectors, go wrong soon after launch?

The Barts and The London NHS Trust was one of the latest Trusts to be hit after underestimating the impact of problems caused by the introduction of a new Care Records Service. The chief issue related to difficulties in scheduling patient appointments, which meant operating theatres and clinics were being left unused despite high demand. And the trust was said to be paying nearly £1m for extra temporary staff due to the problems.

Two other London trusts, Barnet and Chase Farm and the Royal Free Hampstead, are reported to have suffered significant and protracted problems after going live with a similar system.

Are new systems tested rigorously enough?

One of the obvious concerns is that problems with new applications could be due to them not being tested rigorously enough. When the new baggage handling system experienced much publicised problems at the opening of Heathrow’s Terminal 5 earlier this year, there was speculation that the software was not adequately tested.

The consequences of getting it wrong, as we have seen, can be calamitous, with irate customers, lost income and damaging publicity on the front pages. With NHS or other public sector installations, there is the added issue of political scrutiny and finger pointing.

While most organisations these days will have some form of regime in place for testing new systems, in many cases it can be limited to a cosmetic process designed to ‘put ticks in boxes’ rather than a detailed and comprehensive testing programme.

This isn’t a big surprise when you consider the results of one recent survey[2] of nearly 200 IT development and project directors which reveals that only 26% felt testing was an essential investment. This was despite 55% admitting they had experienced post-release problems during the first few months after they had tested new software.

Deadline pressure

Part of the problem could be the drive to meet deadlines. In many cases — especially where high profile implementations are concerned — there may be immense pressure to ensure a system goes live on the planned date. The project develops its own momentum and it can be very difficult for concerned quality and assurance teams to stand up and call a halt ‘to the show’ for the sake of another round of testing.

Sometimes testing is not introduced early enough in the overall development cycle. The earlier in the process you spot problems, the easier and less costly it will be. If you fail to realise there’s a fault until late in the day, you may have to unravel and rework many related parts of the application which have been built ‘on top of’ the offending item.

Another issue is just the sheer complexity of today’s IT applications. And they are becoming increasingly complicated, often spanning multiple platforms and application layers with new elements very often having to be interwoven with legacy applications, such as historical databases.

In such circumstances, while it is a fairly simple task to test the functionality of individual components, it can be hugely time consuming when you want to test the end-to-end nature of the complete system to ensure the integrated whole works effectively in all scenarios.

Accounting for the real world

Too often the testing that does take place is too rigid and does not take account of the real world. It isn’t enough to simply test that individual components are ‘fit for purpose’ based on a tight definition of how users will operate them. Most users in the real word won’t even bother looking at the manuals, so new applications need to be tested for the unexpected things that people might try to do.

One way of achieving this is to ensure you get users involved early in the testing process, giving them the freedom to simply ‘play’ with the system without providing fixed instructions. It might be necessary to schedule a short pilot ahead of full ‘go-live’ in order to monitor the real world consequences of a subset of real users working on a new system.

Similarly, it is vital to introduce realism when load testing. Many applications start to develop problems when they are put under heavy loads. This explains why you often hear of web based applications crashing when traffic reaches peak volumes, for example. Databases may start to experience difficulties when processing heavy loads and performance in other areas may start to be affected.

Simulating peak loads

So load testing needs to simulate the type of usage rates that would be expected at peak times, when the number of users is at its highest. This needs to be repeated many times, over a long enough period to instil confidence that an application is resilient.

Here again time is an enemy. Load testing can soak up time and many aspects of it can only be performed when an application is nearing completion so there is often a temptation to cut corners to get things up and running as quickly as possible.

Even if all the homework has been done on time and a well planned, comprehensive testing regime has been executed, there is always a chance of being caught out by an unexpected problem after a system has gone live. For this reason it’s important to use application performance management software to continuously monitor performance of key applications both before and after they have gone into production.

Highlighting glitches early

The main function of a performance management system is to highlight potential glitches before they start to significantly affect productivity or service levels. The latest systems take a top-down approach by monitoring the front-line user experience and progressively drilling down into the nitty-gritty of the application.

Performance management systems need to pin-point performance problems related to how all the separate parts of a system interact, which could be down to a small string of code. They should also be able to keep an eye out for more conventional infrastructure glitches — such as shortage of disk space, excessive CPU utilisation, or database problems.

Ideally the system should be configurable so that it can generate alerts as soon as performance levels (eg response times) in any part of the application get close to a ‘critical’ benchmark. And it should also be able to pin down the exact location of any problem so that technicians know precisely where to target their efforts.

In a typical scenario the performance management system will automatically notify the correct people if an application is not performing normally. It will clearly identify application problems, so as to avoid the delay of having to check other parts of the system such as network or infrastructure performance.

A support engineer can then start to drill into the detail on the system and pinpoint — for example — that it’s down to a problem with a small section of code linking the application with a database. She might need more specialist assistance and ideally a good performance management system will make it easy for her to share her analysis and collaborate with members from the development team and perhaps a database administrator.

The different team members might well be working in different locations, but the performance management system should help them to work together effectively to assess the problem, get the offending code fixed and have the application running normally again — hopefully well before the users themselves have even noticed.

No guarantees

In the final analysis it is impossible to provide a 100% guarantee that a new application will not experience a problem. But if more organisations insisted on appropriate testing programmes and ongoing performance monitoring it should be possible to significantly reduce the risk and overall damage if a glitch does materialise. We’d then hopefully see fewer application disasters hitting the headlines.

Chris Limberger, Product Manager, Application Availability, Application Performance of Macro 4.

1. Computer Weekly. www.computerweekly.com/Articles/2008/09/01/232097/barts-underestimated-impact-of-it-system.htm

2. Independent survey commissioned by SQS www.itpro.co.uk/news/185262/businesses-ignore-cost-of-software-testing-risks.html?searchString=software+testing