Ouch! The UK national audit office has just released a report criticising the DWP project on Universal Credit, released on the last day of agile on the beach. It has been billed as a ‘rude awakening for the agile dream’, ‘agile chaos’ and even ‘failure of agile’

Is this fair? The essential line from the agile community is that the project failed because it wasn’t agile enough. Although you might counter by saying that agile was simply the wrong tool for the job.

In this blog post I argue that although most of the failures were nothing to do with agile, the case nevertheless has some warnings and lessons about the application of agile that we would do well to heed.

1. Watch the interfaces
2. Manage the big picture
3. Get the contract right
4. Have clear objectives
5. Embrace and learn from our mistakes

History

The Universal Credit (UC) is a single payment replacing existing means-tested benefits and tax credits. It, has of course been a matter of both political and technical concern for some time, Frank Field initiating the inquiry in 2012, citing concerns raised the year earlier.

The campaign4change website has published the Starting Gate Review (March 2011) which I cannot find anywhere else. Anyway, the most interesting thing is this quote:

“the use of an agile methodology remains unproven at this scale and within UK government, however, the challenging timescale does present DWP with few choices for delivery”.

So it appear that agile was chosen simply because of the short timescale – and at increased risk – apparently with the hope that it would be a panacea. This seems like a recipe for disaster!

The NAO report helpfully summarises developments since then:

• The government ‘reset’ Universal Credit in early 2013, because of the Major Projects Authority’s serious concerns about programme implementation; concerns including no detailed ‘blueprint’ and transition plan
• Between February and May 2013, the Department started a limited pilot scheme (a ‘pathfinder’) which was subsequently extended to four sites and 1,000 new claims. Since manual intervention (Excel!) was still required, the pathfinder was deemed not scaleable.

So what of the report itself? Well, the NAO’s press release is quite interesting:

“The Department’s plans for Universal Credit were driven by an ambitious timescale, and this led to the adoption of a systems development approach new to the Department. The relatively high risk trajectory was not, however, matched by an appropriate management approach. Instead, the programme suffered from weak management, ineffective control and poor governance. Universal Credit could well go on to achieve considerable benefits if the Department learns from these early setbacks and puts realistic plans and strong discipline in place for its future roll-out.”

Amyas Morse, head of the National Audit Office, 5 September 2013

They add:

Given the tight timescale, unfamiliar project management approach and lack of a detailed plan, it was critical that the Department should have good progress information and effective controls. In practice the Department did not have any adequate measures of progress.

In fact, the NAO concludes on an optimistic note:

“Under this new leadership we are making real progress and we have a plan in place that is achievable and safe. The NAO itself concludes that Universal Credit can go on to achieve considerable benefits for society.” 

An optimism that is shared by the project director upbeat assessment by Howard Shiplee, at least on paper (although his actions suggest that there are deep underlying problems).

Maybe this is agile’s fault then?

Agile takes some of the rap as a new project management approach which had never before used on a program of this size and complexity. The caveat is important; in my view agile should always be trialled on a small project to get people used to the new way of working.

A key problem was that DWP has no clear way to monitor progress nor document decisions.

Watchdog magazine also comments on the inadequate financial control over supplier spending; including progress, performance and governance. As a ‘documentation lite’ methodology, agile could be argued to contribute to this. I think this is a little misleading, agile does these things very well but at the micro level. I have always found there needs to be a roadmap, architecture and clear governance structure within which agile can function safely.

Perhaps then unsurprisingly the department struggled with agile, adopting ‘Agile 2.0’ (whatever that is!) in January 2012, designed to try to work better with existing waterfall approaches (Such homebrew methods are risky as discussed in the excellent Scrum Field Guide)

Computer Weekly added: “If DWP had broken the project up and handed it out to many teams of small, agile suppliers it might have made less progress and cost more money than it has with a few large suppliers on large contracts.”

Well, maybe. Of course we don’t know. This is a hypothesis yet to be tested with the ongoing realisation of the Government Digital Strategy

But most of the failures weren’t anything to do with agile

Watchdog magazine have pointed out a few key areas in the report which are nothing to do with agile:

• Lack of transparency and challenge. A ‘fortress’ mentality within the programme team and a ‘good news’ reporting culture
• Ineffective departmental oversight.  The programme board has
  also been too large and inconsistent to act as an effective, accountable group. The Department has recognised problems with governance, repeatedly changed the programme’s governance structures and, during the reset, suspended the programme board entirely.
• Unrealistic ambition (The DWP being overly ambitious in both the timetable and scope of the program; and being unable to explain how it originally decided on these plans or evaluated their feasibility.
• Lack of leadership and expertise: A lack of critical in-house IT expertise and senior leadership, with frequent changes in senior management.

Allan Kelly gives a host of other examples in his entertaining agile on the beach talk entitled ‘That’s not my agile”. (1:50 to 07:10).

So was it even an agile project?

Computer Weekly points out that “parliament and politics refuse to accept that software projects naturally comprise change”. This is a key issue which I’ll talk about below. Perhaps as a result, Brian Wernham asserted that it was actually a waterfall, or ‘big bang’ project, with all the testing left to the last minute (though back in April, he was more upbeat ). Certainly a team of over 1,000 people doesn’t sound very agile.

Allan Kelly concludes that the main cause was perhaps ‘a failure of senior leadership and IT expertise’. This is a view shared by Sir Ian Magee in the Institute for Government’s System Error 2012 report.

It also echoes the thoughts of Ian Watmore, who (as Permanent Secretary of the Cabinet office) identified three main reasons for large-scale IT failure: “policy problems, business change problems or big- bang implementation”

However, the more important lesson is about agile adoption

For me, the most important learning is the lessons we can learn about adoption of agile in government – or indeed any organisation new to agile.

Lesson 1: Watch the interfaces

The challenge was integrating Universal Credit with existing systems, which use a waterfall approach to managing changes. Note that the question is not so much the technical interface between old and new systems (this is a relatively well understood risk) but the procedural, behavioural and cultural interface between old and new approaches.

To this extent, the experiment with ‘Agile 2.0‘ is understandable, and we should learn from its failure.

Lesson 2: Manage the big picture

Tim Manning comments:

Not least, they appeared to be under the common misconception that agile means that you can go ahead and design and build bits of a system, without any real understanding of the whole. 

This is a subtle distinction from lesson one, which is about practices that potentially conflict with agile. Clear governance, strong leadership, a crisp (and speedy) decision making process, roadmaps, architecture, explicit risk management, contingency planning; these are all things that support agile but are (to be frank) too easy to overlook when just starting on the agile journey. Agile is (perhaps frustratingly) non prescriptive about such matters. This does not mean that these things are unimportant!

To take just one example :

“HMRC [has] no contingency plans in place if RTI [real time information] fails, all remaining 281 tax advice centres will close during 2013-2014 during the period of the roll-out of Universal Credit”

(Michaelson 2013)

Lesson 3: Get the contract right.

Tech Market View commentary:

“It was also difficult to marry the agile approach with existing contract, governance arrangements and assurance structures.”

But what does an ‘agile friendly’ contract look like? Ewan Milne has an excellent talk about this at agile on the beach, and I will blog separately on that. In this case, the DWP adopted a simple time and materials based contract, which (as Ewan pointed out) leaves the majority of risks with the Department.

The Government Digital Strategy offers some help here, though I am not convinced that some of the options (eg pay per story point) are in fact very constructive in practices. I favour shared risk models as described by John Thornton of UKNetWeb (video) though in practice, ‘Fixed Price Agile’ is the (not very satisfactory) reality for many projects:

translated from (Andrzeevski 2010)

Lesson 4: Have clear objectives

Rosa Michaelson makes the point that while agile allows (and indeed expects) the solution to be ill defined at the start, the business objectives (ie the problem to be solved) should be clear. It is also true that while being responsive to change is a good thing, it does not mean that stakeholders shouldn’t be properly consulted in the first place.

Computer Weekly add:

“DWP didn’t really know what they [the suppliers] should be doing, so it couldn’t really tell if they were doing it properly. “

Lesson 5: Embrace and learn from our mistakes

Tim Manning finishes on a positive note:

Hopefully, for those now in the middle of this, there are enough bits and pieces to re-purpose and get things back on track.  I see the reset as a positive thing.  So often in the past, people have just carried on digging!

References

Andrzeevski, S. 2010. The main challenges of implementing fixed-price agile projects for corporate customers. In 2010 6th Central and Eastern European Software Engineering Conference in Russia (CEE-SECR 2010). IEEE. 220–225. DOI: 10.1109/CEE-SECR.2010.5783180.

Magee, I., Gash, T., Stephen, J.: System upgrade? The first year of the Government’s ICT strategy. Institute for Government (June 25, 2012)

Michaelson, R. (2013) “Is agile the answer? the case of Universal Credit”, in  Dwivedi, Y., Henriksen H.Z., Wastell, D. and De R. (eds.) Grand Successes and Failures in IT: Private and Public Sectors, Series: IFIP Advances in Information and Communication Technology, Springer.