In Fast or Secure Software Development Jim Bird points out that many software organizations are going down the road of releasing the software very frequently.
Some people are going so far as to eliminate review gates and release overhead as waste: “iterations are muda”. The idea of Continuous Deployment takes this to the extreme: developers push software changes and fixes out immediately to production, with all the good and bad that you would expect from doing this.
There is a growing gap between these ‘Agile Approaches’ and mission critical software that has to be secure. Jim highlights many aspects of it in his article, but there is another wider aspect to consider.
Humans are bad at planning for the long term
The really bad news with this is that in this context the long term can be days or weeks, but as a society we need to be planning in terms of lifetimes.
For ‘Agile Approaches’ it makes sense to automate the release process so that it is easy to do with minimal impact. There are lots of technologies that help us achieve painless releases, and most of them are useful and effective. But just because we have a tool does not mean we should rely on it. As Jim points out, the frequent release mantra is based on the idea that it is easy to see if a release is broken, but
it’s not always that simple. Security problems don’t show up like that, they show up later as successful exploits and attacks and bad press.
Some problems require careful planning, appropriate investment and preparation. One domain I am familiar with — credit card processing — has been fraught with problems because developers have not dealt with all of the corner cases that make software development so interesting and painful. Reportedly there have been exploits from server logs that included the http post parameters, which just so happened to be credit card numbers. Of course no organization will repeat any of the known mistakes — but without the long term focus on planning, investment and preparation that mission critical software requires, some other problem is bound to occur.
Failing to address the long term
Several years ago now, the Club of Rome study on the Limits to Growth was met with disbelief in most quarters, but less than half lifetime later we are hitting up against the limits that were identified back then. Peak Oil, Climate Change and the current financial meltdown are all related to these limits. Admittedly we are not good at dealing with exponentials, but reality has a way of making sure we cannot forget them. As the demand for oil reached the existing supply levels, the lack of successful investment in other supplies or alternatives meant that many economies crunched when we got near to peak oil supply volumes.
To think, a relatively simplistic computer model from nearly 40 years ago was able to produce a crude simulation of where we are now. Yes, many details were wrong, and the actual dates they were predicting were out, but what they were predicting was inevitable without policy changes and we failed to make the necessary changes. It was always easier to just keep on doing what we have been doing until suddenly oil is over US$100/barrel and then suddenly we act all surprised and horrified about the price of everything.
Software is Infrastructure
Software lasts a long time unless you make really bad mistakes, so we need to start treating it as infrastructure. But not like we are treating our current physical infrastructure. It is nice to be able to release software easily with minimal impact and costs, but we need to make sure that in doing so we are not exposing ourselves to longer term pain. Yes, make it easy to release, but only release when you know through intense and rigorous evaluation that what you release is better than what you have out there already.
Core systems do not get replaced quickly. Our 32 bit IP addresses are starting to hurt, but rolling out IP6 is a slow process. Just like our roads and bridges we have patched the existing system up lots of times, but it needs to be replaced and soon. Unfortunately, just like our roads and bridges, the necessary investment is daunting and we still want to put it off as long as possible.
As we enter the post cheap energy era, we need to reevaluate our planning horizon. We need to rediscover how to plan for the long term. Somehow or other, forty years after NASA put a man on the moon, NASA is now reduced to hitching rides with the Russian Space Program …