## Why was the Healthcare.gov site development managed so badly?

0

2

The US Department of Healthcare and Human Services was ultimately responsible for managing the contract(s) that developed the federal healthcare exchange website, Healthcare.gov. As has been widely reported, the site launch was a disaster, as even President Obama has acknowledged. On the day of its launch, Healthcare.gov was incredibly slow, buggy, and most people who tried were unable to enroll using it. Numerous types of technical problems have been reported including timeouts, error messages not caused by user errors, a network failure that took about a day to fix, and missing data being received by insurance companies. A "tech surge" was announced by President Obama to fix the technical problems.

Astonishingly, one of the contractors (an allegedly "agile" Ruby shop with little experience of government contracts), appeared to believe, according to reports, that it would be sufficient to have one live [web] server, one backup server, and a CDN (content delivery network).

There seems to have been some difference in bugginess between Healthcare.gov and the state healthcare exchanges, although both the state and federal exchanges have seen lower-than-expected enrollments thus far, and some of the state exchanges failed to open on schedule on October 1 2013.

Why was this project (at least, from today's vantage point), such a big mess? Why weren't the bugs identified and fixed sooner, before the rollout? Can we draw any broader lessons, or put this debacle into the wider context of other badly-managed government contracts (whether they be IT or non-IT, US federal government or other governments)?

2I don't really know that this question is answerable. Some people are incompetent: there isn't necessarily a good reason for that. Reports indicate that Obama himself wasn't really informed until the rollout, but that still leaves a number of questions unanswered. – Publius – 2013-11-17T10:15:16.490

There are millions of potential reasons. The only way to know the answer is to sit down with all involved (one by one, with confidentiality) and make a project dissection and review. – Lennart Regebro – 2013-11-17T11:03:45.907

If the answers are specific to the particular project, why do so many government IT projects go overbudget, fail outright, or fail to meet all their objectives? – Robin Green – 2013-11-17T11:30:15.437

4Because so many IT projects go over budget, fail outright, or fail to meet their objectives, regardless of whether it's government or private sector. – Compro01 – 2013-11-17T18:05:20.610

5

There are 3 major reasons that I've heard as to why the roll-out was troubled.

• One reason places blame on the main contractor, CGI Federal, who's parent company(CGI) was fired from making a similar healthcare website in Ontario for missing deadlines. source

• another reason is that they spent only two weeks of time to test the website. This issue seems to be related to the first, except that they did not push the deadline back in this case, but rather release the website even with insufficient testing. source

• Another reason is because there were too many contractors. Even though CGI was the primary contractor, there were a lot of lesser contractors who all had to integrate their work together. This is also related to the testing issue where there was not enough integration testing done. source

1@SamIam - quotes aren't a requirement but do make the posts better. Even on soft sites like Scifi.SE :) – user4012 – 2013-11-18T20:44:53.847

1AMS (bought by CGI) was a local contractor here, and yeah, they have a reputation for screwing things up... – Affable Geek – 2013-11-18T21:01:05.963

2

Because "You get what you pay for" isn't necessarily true when the costs get obscenely large. Suppose you pay a neighborhood kid $20 to mow your 1/4 acre lawn. Now suppose instead you pay him$200, how much better of a job would he do? Surely not 10 times better, maybe not even 2X better. What if you paid him $2000? For$2000 he might be inclined to bring 20 of his friends along to "help" mow the lawn so he can make sure it's done really well. What do you think the lawn is going to look like after 20 kids try to mow it simultaneously?

Being a software developer by trade, even without knowing the details of what healthcare.gov entailed behind the scenes, I can confidently suggest that we (the taxpayers) overpaid by a factor of between 10 and 100 for this website, and I would say that even if it ended up working perfectly on day 1. Because it would not be prudent for a large company to make a 290M profit on a 300M bid to the government, that money has to be spent somehow, so you end up with a lot of kids mowing a tiny lawn.

Why the downvote? Please point out the flaw in this logic. – TTT – 2013-11-29T21:47:55.437

-1

A lot has been written about this in the media and I'm not going to attempt to address every angle in this answer. I just want to focus on one key point that I think is important, and of wider relevance:

I think a key point about the contractors that worked on this site is this: they faced a perverse incentive.

The company CGI Federal, responsible for some of the backend services used by Healthcare.gov, was brought on in a "rolling contract" with the Department, a contract which was very vague, and not specific to Healthcare.gov - essentially just "do whatever IT work we need from you". Therefore, it has been argued that they essentially "won the work on a no-bid basis". The key point, though, is that by screwing up and releasing a system with lots of bugs, they guaranteed themselves an increase in billable hours, revenue, and - presumably - profits.

This is a totally perverse incentive!

I am not claiming that they actively intended to follow this strategy. I have no way of knowing that. But that, faced with no financial risk (only possible reputational risk), minds at the contractors may not have been sufficiently focused prior to October 1.

Now, one might immediately ask the question "But isn't reputational risk a very significant risk for government contractors? Because they might not get another contract if they are seen to be incompetent?"

The answer to those questions are extremely interesting, and empirically, they seem to be, respectively, No, and No (at least for the large contractors that tend to get the lion's share of government contracts such as this).

Why is this? Well, I'm not as well-informed about US government contracting, but in the UK at least, the UK government has actually claimed, apparently seriously, that it is obliged by European Union law to pick the lowest qualifying bidder, and that it would be illegal to take into account evidence of performance on previous contracts! However, perhaps recognising that this is a stupid state of affairs, the European Union is now proposing to explicitly allow past performance to be taken into account when selecting a bidder for a contract, from 2014.

Interestingly, another large healthcare IT project with multiple contractors, the National Programme for IT in the United Kingdom, attempted to use contractual "risk sharing" to reduce such perverse incentives, whereby the contractors would face significant costs if they were deemed to have "failed". (Because UK government contracts are routinely kept secret, for reasons, the government claims, of commercial confidentiality, i.e. to protect the commercial interests of the contractors, we are unlikely to ever know the detailed terms of this arrangement, such as who was responsible for deciding whether it was the contractor's "fault", and on what basis). However, two surprising things happened during the course of this hugely expensive, hugely overbudget, over-timescale and ultimately partially-failed project:

1. The software development contractors failed to develop the software satisfactorily, regardless of the risk-sharing. They appeared to be overconfident and extremely poor at delivering. (Although, they actually re-outsourced the work to subcontractors, so they weren't actually delivering anything themselves!)
2. Even though the contracts allowed for risk-sharing, Wikipedia notes:

The costs of the venture should have been lessened by the contracts signed by the IT providers making them liable for huge sums of money if they withdrew from the project; however, when Accenture withdrew in September 2006, then Director-General for NPfIT Richard Granger charged them not £1bn, as the contract permitted, but just £63m.[26] Granger's first job was with Andersen Consulting,[27] which later became Accenture.

One could argue that the fault in this case lay partly with the customer (the government) managing the project poorly, but the fact that Granger used to work for the very contractor he had let off lightly (this is known as the "revolving door" problem), casts doubt on the objectivity of this decision! Anyway, either way, this just goes to show that it's not as simple as saying "force the contractors to share the costs of failure, and this will fix the incentive problem".

So I think we need to look deeper to the causes. Some might claim that the proximate causes of the problem are methodological: "they should have used an agile development methodology", or whatever. Let's assume for the sake of argument that the project management methodology was at fault. But if the incentives are aligned appropriately, contractors should automatically pick the best methodology, or the one most likely to succeed! That's just basic economics. Even if the government stipulates the methodology, even if the government stipulates just about anything you can think of, economics suggests that firms should avoid such contracts and prefer contracts with good methodologies, which will incentivise governments to either stipulate good methodologies, or leave the choice of methodology open.

The problem is, this simplistic economic theory does not map to reality.

I think the deeper, underlying problem is the principal-agent problem within large, sprawling corporations: managers often do not face sufficient penalties for failure, sometimes they can escape entirely unscathed by blaming others, people are rewarded for winning government contracts regardless of the corporation's actual ability to deliver (this seems intuitively like a good idea - after all, winning fat government contracts is inherently good, right? - but it may not be in reality, as some of the National Programme for IT contractors discovered). And ultimately, contractors have to fall back on attempting to do extremely dubious post-hoc negotiations, perhaps threatening legal action, to avoid any contractual penalties that may theoretically be in place. Instead of... you know, actually doing the work properly in the first place.

Further evidence for this thesis comes from the fact that a large proportion of private sector IT projects also fail or go overbudget - we just don't hear about them in the media so much, because corporations often don't like to talk about them. So this problem is not actually specific to government contracts - it's something that relates to the intersection of information technology's complexities, and large corporations' uneven internal accountability.

You should provide more sources. – Publius – 2013-11-17T10:57:20.363

1I'm sorry, this answer is completely ignorant on how web development and government contracting actually works. Rolling contract on a per hourly basis is in fact most likely to actually get the project done. So this is not the reason. – Lennart Regebro – 2013-11-17T11:00:35.410

@LennartRegebro please prove that "rolling contract on a per hourly basis is in fact most likely to actually get the project done", with references to studies of government IT contracts. – Robin Green – 2013-11-17T11:27:56.473

I have no need to prove anything to you, especially not in the comments section of a Q&A site. But this is common knowledge in the industry, and has been for decades. Start reading here: http://en.wikipedia.org/wiki/Agile_software_development http://www.extremeprogramming.org/ http://en.wikipedia.org/wiki/The_Mythical_Man-Month I don't blame you for not knowing this, why would you?

– Lennart Regebro – 2013-11-17T12:44:20.203

@LennartRegebro I am familiar with all of those writings. I am professional web developer. But you are raising a false dichotomy between rolling contract and waterfall. – Robin Green – 2013-11-17T12:46:42.880

@RobinGreen I'm not saying that there is a dichotomy. I'm saying that a payment per hour worked, ie a rolling contract, is the best way to ensure that the projects gets timely done with quality. This is a basic and fundamental fact in software development, and if you are a professional web developer familiar with those writings, then you already know this, and then you would not require of me to prove this to you. – Lennart Regebro – 2013-11-17T12:54:45.413

2

There are three fundamental parts: Scope, Schedule, Cost. You can only pick two. If you try to pick all three, quality will suffer. Your claim therefore, that quality suffered because they did not have a fixed cost, is incorrect. http://en.wikipedia.org/wiki/Project_management_triangle

– Lennart Regebro – 2013-11-17T12:55:17.290

But you can do waterfall-style development in a rolling contract. And that's exactly what they did, the supposedly "agile" front-end aside (yeah, right - that wasn't really agile either, that's a nonsense). – Robin Green – 2013-11-17T12:57:38.147

@RobinGreen: Of course you can do waterfall-style development in a rolling contract? So? Your claim is that the rolling contract is to blame. This is, for anyone in the industry, quite obviously and blatantly false. – Lennart Regebro – 2013-11-17T13:35:02.133

(I do wonder where you get your claims from that they didn't do agile development. Where you on the development team?) – Lennart Regebro – 2013-11-17T13:37:45.080

@LennartRegebro I don't think citing personal experience is going to cut it on politics.stackexchange.com. Also note there is a difference between an employee being on a rolling contract and a company being on a rolling contract; I am talking about the latter. – Robin Green – 2013-11-17T13:38:13.130

@LennartRegebro they didn't do proper integration testing, the whole thing was a joke. The users were the beta testers, which would have been maybe OK for a to-do list startup, but it is not OK for a site of this nature. – Robin Green – 2013-11-17T13:39:46.060

@RobinGreen And there you have one big reason. No proper integration testing. But that's not the claim your answer makes. You claim, with no basis or sources, that a rolling contract means you get the incentive to screw up, and release with bugs, which is false. With a rolling contract you get money for fixing things no matter if it's done before or after release. As I mentioned, there may be millions of reasons. The rolling contract is not one of those. – Lennart Regebro – 2013-11-17T13:43:05.493

But you get more money for being incompetent, because the money ends when the work is done! That's my point! – Robin Green – 2013-11-17T13:44:14.470

1And from your links, here is a good hint of what went wrong: "But evidence of a last-minute surge in spending" - This indicates the typical "Mythical-Man month" problem. As the contractor realizes that the system isn't ready, they throw more people on it, which in fact just makes the problem worse. – Lennart Regebro – 2013-11-17T13:47:10.197

Wow, this is getting long. Let us continue this discussion in chat

– Lennart Regebro – 2013-11-17T13:47:52.683