The Next IT Disaster Might Be Sooner Than You Think

I must start out by saying this is not really about healthcare.gov, so if you are a regular reader (there are a few of you, and I love you all) you may want to skip this one. 

There is increasing controversy over the classification of transgendered individuals in this country. It is not unreasonable to assume that, sooner or later, the legal definition of gender will have to be changed. When that happens, the traditional choices of M/F for “Male/Female” in software code and databases will no longer work.

That could be a disaster. 

There are hundreds of thousands of lines of code that rely on the “gender” classification being a binary value: male or female. Modifying all of that code would be a huge chore, not to mention we would very quickly need to agree on a standard way to classify the third gender (T? N?). The politics of this decision alone could take years to hammer out. 

Tens of thousands of file transfers – big dumps of data between systems – have buried in them, somewhere, an implicit assumption that if the data relates to gender, the value can only be one thing or another. They will fail to process when that’s not the case. And as has been shown, data transfers are the Achille’s Heel of any system, the silent killer.  

The levels of the problem go deeper. Most databases I have worked with store a person’s gender as either an ‘M’ or an ‘F’. The problem is, those databases insist that it can ONLY be an ‘M’ or an ‘F’ – there is no third choice, and if you tried to use a third choice, you couldn’t do it. We call these “constraints” and they are one way developers make sure data stays consistent. And they would all have to change.  

If it’s any consolation, it wouldn’t be as bad as Y2K. Not only are date fields far more pervasive in all software, but there is less older, undocumented code tied up in archaic systems. But it would be a massive, expensive and disruptive problem driven, no doubt, by some Federally-mandated deadline. That, as we have seen, can lead to chaos. 

And don’t even get me started on what happens if Puerto Rico becomes a state.


healthcare.gov: From Inexcusable to Indefensible

The latest “glitch” to affect healthcare.gov was reported last evening:

Justin Hadley logged on to HealthCare.gov to evaluate his insurance options after his health plan was canceled. What he discovered was an apparent security flaw that disclosed eligibility letters addressed to individuals from another state.

The article goes on to explain that at least one of the two letters Mr. Hadley was able to view were for individuals who had reviewed the site several weeks ago (1). The letters, only a portion of which are digested at the site, can reasonably said to violate generally accepted security requirements, especially pertaining to health information.

To put it another, less kind way: This is a breach that goes beyond the pale of what’s acceptable. It clearly indicates something broken in the very core of the system – a database key, perhaps, tied to the wrong user, or some application logic that has gone terribly wrong. In any event, if it happened once, it will happen again, and again, and again. 

Most states have very specific rules about security breaches like this; at the very least, the party responsible for the breach is required to notify the victims of the breach. In part, the law states:

Notification required when personal identifying information that was not rendered unusable through encryption, redaction, or other methods was, or is reasonably believed to have been, acquired by an unauthorized person…

Will HHS take the appropriate action? And if not, why not? For whom do these rules apply? healthcare.gov is being taken offline regularly for fixes, and I sincerely hope that this is one of the things that gets fixed quickly.

If I saw something like this happen on my site, my blood would run cold. And the site would be taken offline immediately until the issue could be fixed. It is the only valid, professional, ethical thing to do.

To not do so would be criminal. Literally. 

(1) It should be noted that the site from which I pulled this story is run by a traditionally conservative-leaning organization. I have not found any corroboration of this story, however, based on what I read (and the sheer, general ineptitude and sloppiness of the site) I have no reason to doubt its legitimacy. If it turns out the story is false, I will update the post accordingly.  

 

 


Of Folk Heroes and Hackers

Every scandal – and let’s face it, healthcare.gov is a scandal – has a few folk heroes forever associated with it. And one of this latest scandal’s folk heroes is Ben Simo, who goes by the Twitter handle @QualityFrog. Ben is a former president of the Association for Software Testing, and he runs a blog called Is There A Problem Here? He’s been blogging about the security issues with healthcare.gov and has become a de facto spokesperson for the technical issues with the site. 

The first weekend after healthcare.gov was rolled out, I exchanged Tweets with Ben as he tried to successfully navigate the marketplace. Using tools available to anyone with the wherewithal to use them, he performed the basic tests that the government contractors should have performed well before the site went into production. Like all good software testers, he was patient and thorough, and he documented the site’s failures. Some were merely annoying, but others represented genuine security issues. 

Ben has since been mentioned in TIME Magazine and Consumer Reports (among others), interviewed by Greta Van Sustern, and – most recently – in Kathleen Sebelius’s testimony to Congress.  

I would tell you that there was not a breach, there was a blog by a sort of skilled hacker that if a certain series of incidents occurred, you could possibly get in and obtain somebody’s personally identifiable… [Sebelius]

If there is a better metaphor for how poorly everyone in a policy-making position understands IT, I can’t think of it. A professional software tester is no more a “hacker” than a structural engineer is a “graffiti artist.” That’s not to denigrate any of these groups, they’re just fundamentally different, but that difference matters a great deal.

And to conflate them, as Sebelius does, only shows the absolute lack of understanding that policymakers have over these very complex, increasingly indispensable systems. Writing software, building websites – these are not hobbies, and the people who do these things for a living are not hobbyists.    

Ben continues to blog about the problems with the site, as well as the fixes, giving credit where credit is due. Through it all he has remained apolitical and objective, which is what technical problems like this require. If only the people running the show could be half as professional as he is.


healthcare.gov To Work “Smoothly” When We Need “Correctly”

In a conference call today with reporters, the White House announced that healthcare.gov should be open for business for most users by the end of November. 

There’s no way the site that Kathleen Sebelius said needed five – no, six – years of work and a full year of testing, and which got two years of work and 1 week of testing – that site can now be “operating smoothly” in one month? Never mind the fact that it’s actually much harder to troubleshoot a system that’s already in production; to expect anyone to believe that the disastrous code and its related problems was just one month away from being functional is laughable. 

The site is still not safe. Ben Simo, past president of the Association for Software Testing, recently mentioned in TIME Magazine, has issued a strong “do not use” recommendation. Ben identified a whole slew of issues within the first week the site went live, few of which have been fixed. [Ben posted earlier this evening that one of the issues he noticed may have been fixed, which is progress... but also something that went unaddressed for 25 days, which makes you wonder what they were up to all that time.]

And these are just the issues we can see. There are, after all, 500 million lines of code that need to be looked at.(1)  

This is not a realistic date. What’s far more likely is that, just as they demanded an October, 2013 “go-live” date, the White House is demanding the end of November for everything to be fixed. The consensus seems to be that this is the very last date by which they could successfully enroll the required numbers of Americans needed for the ACA’s success. So we’re in for all kinds of creative parsing of the words “functioning smoothly,” just in time for the holidays.    

(1) Which is ridiculous, by the way. Assuming the project had two years of development, that amounts to 8 lines of code being written every second of every minute of every hour of every day for two years. 

 


The Boss Is Always the Last to Know

Kathleen Sibelius stated in an interview with CNN that President Obama was not aware of the problems with healthcare.gov until after the launch. I believe that.

The Huffington Post went on to say this is business as usual in Washington:

But culturally, Washington is a place where no one wants to be the bearer of bad news, and every gang of bureaucrats proceeds from the notion that “job one” is ensuring that the axe falls heaviest on another gang of bureaucrats. Consequently, what we got in the lead-up to the launch of Healthcare.gov was a complicated square dance of blame avoidance that obscured the pending disaster.

The boss is always the last to know. I have been guilty of waiting until the last possible second, and then beyond, to tell the boss about a potential showstopping bug I thought I could fix. And of course I couldn’t, and of course it blew up.

Here’s the thing: Bugs are easy to hide from people who don’t know where to look for them. If healthcare.gov were a building, everyone could SEE that it wasn’t ready to be occupied because there would be no floor boards or plumbing and the whole place would sway mightily in the breeze. But it’s very, very easy to mock up a web page. And I’m sure the President saw plenty of screenshots.

Because it’s so easy to hide defects – and because software is so hard to do and so often behind schedule – unjustifiably rosy status reports are no surprise. Most managers accept that. The problem is, they get increasingly rosier as you head up the chain of command. I guarantee you, somewhere ten or twenty levels down from the President was a project manager losing their mind. Is this ethical? Hard to say; there is no formal code of ethics governing the IT profession that I am aware of. But there should be.

I am quite certain President Obama was as shocked as the rest of us. Man, what a conference call that must have been.


The Code Never Lies: healthcare.gov Is Really, Truly in Trouble

Everyone is entitled to their own opinion, no one is entitled to their own facts. And the facts are these: healthcare.gov has been poorly built, poorly designed, and poorly tested. It is a failed system. This is not the statement of a liberal or a conservative, Democrat or Republican. It is a professional assessment and it is widely held amongst other professionals.

If healthcare.gov were a building, we would not be having this discussion, for there would be no question how seriously it has been botched: we’d see broken windows, missing doors, and sagging floors. If healthcare.gov were a dam, it would leak; if it were a rocket it would explode; if it were a bridge it would collapse. When an IT professional looks at the various error messages and code snippets from healthcare.gov, or reads reports about data transmittal issues or defective rate calculators, we see this: 

Until we stop pretending that all these criticisms are just mean-spirited sniping intended to keep children from getting healthcare, we are never going to be able to constructively solve this problem. Software is objective, perhaps the most objective thing there is. Pretending that IT professionals are blowing the issues out of proportion to prove some political point is insulting to the profession, and it’s not solving a damned thing. Knock it off, and let’s start discussing how to enroll millions of Americans by other methods.  


“tech surge” Unlikely To Cure healthcare.gov’s Problems

The White House’s declaration of a “tech surge” to solve healthcare.gov’s problems is like throwing gasoline on a fire. 

There is a rule in software development called Brooks’ Law, and it states: Adding new developers to a late software project only makes it later

‘Brooks’ was Frederick Brooks, a project manager on the development of IBM’s System/360 operating system. That project, like almost every large-scale software project before or since, was delivered late and over budget (System/360 did have the virtue of actually working, though.). His book documenting the experience, The Mythical Man Month, was published in 1975, and it is still important and relevant today.

Brooks observed that, unlike physical engineering efforts, software development did not get faster with bigger teams – it got slower. The reason, Brooks deduced, is that adding a developer to a project increases the amount of time the entire team needed to communicate. Software engineering is, at its most basic level, all about coordination, protocol and consistency, and getting those things right takes time. 

In the physical world, if one worker can produce 100 widgets an hour, then two workers can produce 200 widgets an hour. Add two developers to an existing two-person team and everyone takes longer to get on the same page. Add more developers to an already late project – where communication is already in disarray – and the problem increases exponentially.

The White House’s plan, if it succeeds, will fly in the face of every troubled software project in history. That may happen, but given this project’s track record, it seems unlikely. 


Follow

Get every new post delivered to your Inbox.

Join 204 other followers