Improving Wetware

Because technology is never the issue

Zed Shaw and Learning Python The Hard Way

Posted by Pete McBreen 09 May 2010 at 15:57

Zed has started an interesting experiment on Learning Python the Hard Way.

Looks to be an effective approach to learning Python, but I’m more intrigued by the tools used. From a set of text files in a fossil repository there is a make file that builds the complete book as a PDF file. Seems like it could be something I could use for other projects, the only awkward bit about it being the size of the TeX installation needed to support the pdflatex generation process.

Fun with XHTML

Posted by Pete McBreen 09 May 2010 at 13:01

I’ve been looking at XHTML specifications recently and was surprised at the number of special symbols that are defined.

From simple things like ¬ ¬ and © © — to much more esoteric ones like ƒ ƒ. I can see the need for them, but the sheer number means that unless you use them all the time there will be the need to look them up.

I like the math set though, forall ∀ part ∂ exist ∃ empty ∅ nabla ∇ even if I have never had much use for these outside of math and logic classes. Similarly the arrows are nice harr ↔ looks neat, but not something I use a lot.

Overall there are some useful symbols in XHTML, and it sure beats trying to remember the key code to type them. After all brvbar ¦ is easier to type than trying to find the broken pipe symbol on the keyboard. sect § and macr ¯ could also be useful when annotating documents.

Interesting claims about testing SQLite

Posted by Pete McBreen 20 Apr 2010 at 22:11

How SQLite is tested makes interesting reading about the process and techniques used for testing the database. The ideas from this system could probably be usefully applied to other systems that we need to be reliable.

A different take on climate change and the media

Posted by Pete McBreen 08 Apr 2010 at 23:25

Just who is pulling the strings?

the real story is not the relationship between science and the media at all. It’s the story of how the media has been completely taken in by a third group, a third culture, consisting of ideologically-driven, pathological liars, who will say almost anything in order to score political points, and will smear anyone they regard as an opponent.

Krugman has written about Building a Green Economy

If you listen to climate scientists — and despite the relentless campaign to discredit their work, you should — it is long past time to do something about emissions of carbon dioxide and other greenhouse gases. If we continue with business as usual, they say, we are facing a rise in global temperatures that will be little short of apocalyptic. And to avoid that apocalypse, we have to wean our economy from the use of fossil fuels, coal above all.

Interestingly Krugman didn’t write much about James Hansen’s proposal for a tax and dividend to cut down on CO2 emissions. James Hansen is against the Cap and Trade for reasons he explains very well. Hansen also has a paper that shows pictures of what cold winters used to be like - when the Niagara falls could freeze.

What we need is an approach that addresses the fundamental fact that keeps us addicted to fossil fuels: they are the cheapest form of energy, provided their prices do not have to include the damage they do to human health, the environment, and the future of our children.

For the sake of the people, especially young people, there should be a rising price on carbon emissions. The price should rise at a known economically sensible rate, so that businesses have time to plan their investments accordingly. The money collected should be put in the hands of the public, so they are able to make the purchases necessary to reduce their carbon footprint.

Wisdom from Neil deGrasse Tyson

Posted by Pete McBreen 08 Apr 2010 at 21:33

1960s: “If we can land a man on Moon we can solve hard problems”; 2010s: “We forgot how to land on Moon. Let’s grab a beer”

Solving the birthday problem using Ruby

Posted by Pete McBreen 07 Apr 2010 at 17:00

Selection with Replacement

Had a problem when testing something that did random selection from a large pool that was showing more duplicates than expected. When sampling out of 500, we were seeing duplicates very frequently. Turns out it was the birthday problem - in a group of 23 people, there is an even chance that two or more people will share a birthday.

Amazing part of this problem is that even doubling our sample to 1000, after just 40 tries there is a better than even chance that we will have a duplicate.

Formula is as follows, for a sample size of n, with m selected, percentage chance of duplicate is n!/((n-m)! * n power m)

Code below shows this using irb, first off with the standard birthday numbers (just to check the formula) and then with the sample size of 1000

irb(main):001:0> class Integer
irb(main):002:1>   def factorial
irb(main):003:2>     (2..self).inject(1) { |f, n| f * n }
irb(main):004:2>   end
irb(main):005:1> end
=> nil
rb(main):006:1> (365.factorial * 100)/(342.factorial * 365**23)
=> 49
irb(main):007:0> (1000.factorial * 100)/(960.factorial * 1000**40)
=> 45

Testing AJAX with curl and Apache Bench

Posted by Pete McBreen 31 Mar 2010 at 10:50

Since I needed to look these up, decided to put them where I can find them.

When using AJAX and POST, need to put the normal posted content into a file, test.txt in this case and then send the data to the server using either curl (for one off checking the result) and apache bench (too see how fast it can respond)

curl -d @test.txt http://local.site.com/ajax/post/url

ab -p test.txt -T “application/x-www-form-urlencoded” -H “X-Requested-With: XMLHttpRequest” -c 1 -n 10 http://local.site.com/ajax/post/url

For ab, c is for concurrency, n is for number of requests. Need to check the expected document response length to be sure that are not getting back a 200 with the wrong values. T sets the content type, H adds headers to the HTTP request.

Three choices: Mitigation, Adaptation and Suffering

Posted by Pete McBreen 14 Mar 2010 at 14:52

Interesting ideas in a PDF presentation on climate change. It is amazing what difference we have created in moving from 285ppm of CO2 in the atmosphere to the present 390ppm. One disturbing idea is that the temperature effects have so far been buffered by warming the ocean and melting lots of ice in glaciers. Since a lot of places rely on glaciers to feed rivers during the summer, before the glaciers melt entirely we need to be building a whole lot more storage capacity in our reservoirs or many places are going to experience extremely severe water shortages in the summer months.

“We basically have three choices: mitigation, adaptation and suffering. We’re going to do some of each. The question is what the mix is going to be. The more mitigation we do, the less adaptation will be required and the less suffering there will be.” – John Holdren Source - page 68

Other sources for teh science background to this http://realclimate.org and http://climateprogress.org/.

Just read Debunking the Science that Makes Life Dismal

Posted by Pete McBreen 12 Mar 2010 at 18:58

Economics for the rest of us is a very interesting comparison of classical vs neo-classical economics with the central tenet that economics as taught an promoted is

Arguably, the damage from the teaching of economist’s theory of wages is far greater than the damage from the teaching of creationism. Yet the theory of wages is part of economics education in any and all schools, and it continues without any notice or apposition. The reason is, of course, not hard to understand. While everyone is hurt when we teach religion and pretend it’s science, not everyone is hurt when we teach economics. What workers lose, executives and capitalists gain; and it is the latter who study economics, hire economists, and endow schools.

Lots of lessons in the book for the current economic meltdown, not least that the failure of governments to ensure equality and equitable distribution of wealth has and will make society a lot worse off even if the “economy” looks to be healthy.

The most interesting claim is that unemployment results when spending on consumption and investment goods declines. Investment is needed to absorb the surplus that is created, and without this investment in real goods, productivity gains result in lost jobs. In addition, once consumer confidence drops, people stop buying and then the downward spiral starts. But contrary to current popularized ideas, it is not the consumers who can spend our way out of the recession. Consumers are rationally saving money in case things go worse. It is the investors who have to show the confidence by investing in new productive capacity, that will generate the jobs that enable consumers to feel confident again.

The executives and capitalists who have so far managed to retain too large a share of the overall pie are now hoarding cash, not investing in productive capacity and as a result are deepening the depression. After all, is capital not supposed to be the patient partner in the enterprise. Why should anyone expect a family with a large mortgage to spend money when billionaires and large enterprises have cash stored away in banks, looking for lucrative investment opportunities but only bothering to invest when they have a near certainty of return.

Seeking Simpler Explanations

Posted by Pete McBreen 02 Mar 2010 at 20:25

Yes, there is a fancy name for simpler explanations - Occam’s Razor - or Ockham if you prefer the old spelling, but I prefer to use plain english.

A common problem with beginners to software development is that when an error occurs, they manage to convince themselves that they have found an error in the compiler or computer. Yes, sometimes this is actually happens, compilers do have errors, but a much simpler, and more likely explanation is that they have made a normal beginner mistake in their code. Of the two, it makes more sense to investigate the simpler cause first and only then, if no problems can be found is it worth while investigating alternate explanations. Most of the time the simple explanation is the one to go with. If a program suddenly starts failing, and there have been some recent edits to the code, then the simplest explanation is that the error was surfaced by the changes, so that is the best place to start looking.

Climate science as a good example of simpler explanations

One explanation of the problem of climate change and rising CO2 levels is that there has been a conspiracy of scientists to raise the specter of anthropogenic global warming so that they get fame and fortune.

A simpler explanation is that the conspiracy is n the other side. That some large corporations with vested interests are leading a campaign to convince the general public that there is nothing strange going on with the climate.

One way of testing which of these is a more likely explanation is to look at past behavior. Can we find any evidence of scientists acting in concert to deceive anyone? No, sorry, nothing there. Sure there have been cases where an individual scientist or group of scientists have been enthusiastic about an idea that turned out to be incorrect, but these cases have never lasted for long and even early on there was the normal skepticism of scientists asking questions.

Looking to the other side, can we find any evidence of corporations acting in concert to deceive people? Yes, several times, often with significant deleterious effects on people and the environment. Car and oil companies managed to keep lead in petrol for a long time after the effects of low level of lead exposure were known to harm humans. Lead was only removed when catalytic converters were required to reduce smog and the lead was poisoning the catalytic converters.

Another example, early on the car companies bought up and dismantled many of the electric trolley companies thus forcing people to buy cars in order to get around in cities. Very few cities have effective light rail transit these days, even though in the 1930’s most major cities had these electric trolley lines. San Francisco is one of the few cities that still has the remnants of the old system still running.

Another example is the tobacco industry, managing to spread enough doubt about the effects of smoking so that for over forty years there was insufficient effort put into preventing people from becoming addicted to the nicotine in cigarettes. End result of this was a massive lawsuit and damages awarded against the industry, but even now, the public attitude is such that the tobacco companies can still sell very addictive substances and keep on addicting new generations of customers (aka addicts).

With these examples, the simplest explanation of the public debate over global warming is that there is a conspiracy among the major corporations who have a vested interest in the Coal and Oil sectors of industry to spread doubt and uncertainty. Very year the doubt proceeds, the corporations generate billions in profit. Following the money is always a simpler explanation.

The Onion has written a software manifesto...

Posted by Pete McBreen 28 Feb 2010 at 18:07

I think that the Rugged Software Manifesto has to be a parody.

I am rugged… and more importantly, my code is rugged.

Ok some of the statements are reasonable,

I recognize that software has become a foundation of our modern world.

but overall the whole thing is so over the top that it has to be a parody.

I am rugged, not because it is easy, but because it is necessary… and I am up for the challenge.

How Can We Detect Slow Changes?

Posted by Pete McBreen 07 Feb 2010 at 17:26

Sometimes it seems that while we were not looking, things changed.

Not too many years ago -

  • Hardware was the largest part of any software project budget. Now, unless you are working at a massive scale, the cost of the computing hardware is a rounding error on the bottom line.
  • Scripting languages were too slow for use on real projects, but the web has well and truly demonstrated that this is false.
  • Javascript was only used for annoying irritating effects on web pages, but now AJAX and Web 2.0 have brought drag and drop functionality to the browser application (admittedly not everyone is using these capabilities but they exist).

Not too sure how this is happening, but it seems that when we first learn about something, those ideas stick and it is hard to change what we know to match the current reality. When I started commercial software development, it was common to build systems on a PDP-11 with under 512KB of RAM. These days a laptop comes with at least 2GB of RAM, an increase of main memory of a factor of 4,000, but sometimes I still catch myself trying to save a few bytes when designing some aspect of a system.

The open question for now is how to detect this type of slow change (even if the pace of technological change is not all that slow compared to other changes.) This is an important question because many societies and groups have been hit by surprises that in hindsight are obvious, and the consequences were catastrophic;

  • When cutting down trees in an area, when does the population realize that there is a serious problem with deforestation?
  • When does a drought become a climate shift that means the area is no longer amenable to the current mode of agriculture?
  • When does the exploitation of fish in a fishery result in the collapse of the stocks in that fishery?

On the technology side, when do the desktop application developers get hit overtaken by the web applications running in a browser? Functionality wise, we can deliver nearly equivalent functionality over the web provided we have the bandwidth, so maybe it is time to recreate departmental applications as web applications?

Chip and Pin Credit Card Vulnerabilities

Posted by Pete McBreen 06 Feb 2010 at 10:14

This is old news to europeans, but Canada has just started to move to this technology, and it looks like the same systems that are deployed in Europe. With that in mind, here are a few links to known problems in the European model

Chip and Spin is a site that looks at the overall context of the Chip and PIN model, but most interesting of all is that of all places to be doing this type of research, the University of Cambridge is investigating Banking security.

The main issue is that with a credit card containing a chip and the customer providing the PIN, it is going to be a lot harder for the account holder to prove that the transaction is fraudulent. But as the study shows, cloning a card containing a chip is not that hard, and obtaining the pin is not much harder (even before we get into the social engineering possibilities).

Money quote from the Banking security study:

We demonstrate how fraudsters could collect card details and PINs, despite the victims taking all due care to protect their information. This means that customers should not automatically be considered liable for fraud, simply because the PIN was used. Even though a customer’s PIN might have been compromised, this is not conclusive evidence that he or she has been negligent.

Update from the same source - How Not to Design Authentication talks about the problems of using credit cards for online transactions (card not present transactions).

Yet another update from the same team: Chip and PIN is broken

The flaw is that when you put a card into a terminal, a negotiation takes place about how the cardholder should be authenticated: using a PIN, using a signature or not at all. This particular subprotocol is not authenticated, so you can trick the card into thinking it’s doing a chip-and-signature transaction while the terminal thinks it’s chip-and-PIN. The upshot is that you can buy stuff using a stolen card and a PIN of 0000 (or anything you want). We did so, on camera, using various journalists’ cards. The transactions went through fine and the receipts say “Verified by PIN”.

Now using Tynt Insight

Posted by Pete McBreen 20 Jan 2010 at 20:02

Since I was on the team that developed it, thought it was about time to install Tynt Insight on this blog, so I can now see what gets copied and the links will be a bit different when you copy from the site.

Based on this trend we will probably reach 400ppm in April or May 2015.

Read more: http://www.improvingwetware.com/#ixzz0dDTBA0Gp

Under Creative Commons License: Attribution Share Alike

If Tynt Insight is working correctly, clicking on that link will take you to the CO2 blog post and highlight what was copies on that posting.

This link http://www.improvingwetware.com/2010/01/09/why-this-site-has-the-co2-badge#ixzz0dDTvtIq1 goes to the articles permanent page ans should always work even after there are more blog posts on the home page that have moved the CO2 article off the home page.

Good process vs. Bad process

Posted by Pete McBreen 17 Jan 2010 at 11:08

Interesting set on slideshare about the Netflix company culture. Process slide is number 61 - not quite figured out how to link directly to that slide - and the following slides…

Lesson: You don’t need detailed policies for everything

A tale of woe related to optimization

Posted by Pete McBreen 17 Jan 2010 at 08:52

In Optimised to fail the authors start with a great quote…

The late Roger Needham once remarked that ‘optimisation is the process of taking something that works and replacing it with something that almost works, but is cheaper’. [emphasis added]

Although the technical details of the protocol are not public, the authors seem to have managed to replicate what happens, but the key part of their paper are the vulnerabilities that they reveal. These vulnerabilities coupled with the transfer of liability for fraudulent transactions from the banks to the customers means that this protocol and the associated hardware and banking cards should be withdrawn from use.

Browser standards and slow progress

Posted by Pete McBreen 14 Jan 2010 at 11:13

Justin Etheredge has an interesting rant about browsers and the compatibility with standards. The paragraph below should have rounded corners from CSS, but as he says…

And how about this? If you’re looking at this in Safari, Opera, Firefox, or Chrome, then you are seeing nice rounded corners which are created using only CSS. Guess what you’ll see in IE8… nothing. A square box.

Looks like jQuery might be the way to go rather than trying to deal with these browser issues.

An interesting python project

Posted by Pete McBreen 13 Jan 2010 at 22:12

After all the fun and games in the press over the climate models, some developers decided to rewrite the climate models in python. So far their graphs seem to track pretty well to the fortran original code, but these are early days in this implementation of the code.

Looks like I’m going to have to update my python implementation as it is too old to run their code… I’m back at 2.5.1 and the code needs 2.5.4

Just because Zed is so awesome

Posted by Pete McBreen 13 Jan 2010 at 18:48

One of Zed’s earlier rants about why Programmers Need To Learn Statistics.

Finally, you should check out the R Project for the programming language used in this article. It is a great language for this, with some of the best plotting abilities in the world. Learning to use R will help you also learn statistics better.

Why this site has the CO2 badge

Posted by Pete McBreen 09 Jan 2010 at 18:39

Since the trends on global CO2 levels are not good, I decided that it would be useful to watch how they are changing, The historical trend has been that on average we are increasing CO2 levels by approx. 1.9ppm/year. Based on this trend we will probably reach 400ppm in April or May 2015.

But we will see fluctuations up and down over the course of the year

This is a feature of the way the climate relates to the overall earth systems, the CO2 level drops as vegetation grows in the northern hemisphere summer, and then rises during the northern hemisphere winter, peaking in the spring, and then starting to fall off again in June. On an annual basis this fluctuation is around 6 ppm, but year on year we are averaging nearly 2ppm higher - but this varies with the economy and the weather in any year, hot years tend to be associated with a higher rise.

Below is sample data extracted from CO2Now.org which is also the source of the badge.

YearJanFebMarAprMayJunJulAugSepOctNovDecAverage
1959315.62316.38316.71317.72318.29318.16316.55314.80313.84313.26314.80315.59315.98
1960316.43316.97317.58319.02320.02319.59318.18315.91314.16313.83315.00316.19316.91
2008385.42385.72385.96387.18388.50387.88386.38384.15383.07382.98384.11385.54385.57
2009386.92387.41388.77389.46390.18389.43387.74385.91384.77384.38385.99387.27387.35

Overall this is a large scale experiment

How much CO2 can humans add to the atmosphere without adversely affecting the climate systems that we depend on?