Improving Wetware

Because technology is never the issue

A sample pikchr

Posted by Pete McBreen 04 Dec 2021 at 04:48

To use pikchr you really need to downlolad it, play with the examples and read the documentation, but an example pikchr source file (input.pikchr) such as shown below is a great exemplar of the Diagrams as Code paradigm.

box color blue "A" fit;
circle color red "C"
arrow <-> "double headed" "arrow" width 200% chop;
circle "Small" fit;
box "long text will expand the box" fit;

By the use of the command pikchr --svg-only input.pikchr > image.svg gives the image in svg format, ideal for just pasting into a markdown document to display on a web page.

Introducing pikchr

Posted by Pete McBreen 04 Dec 2021 at 04:47

pikchr works well for diagrams that are otherwise awkward to do with the normal visual drawing tools. Brought to you by the maker of fossil and sqlite, as a means of drawing the SQL syntax diagrams. It is useful when you need to draw several similar diagrams to illustrate and idea, while being able to version the changes in a repository so that the diagrams can be regenerated later if changes are needed.

Using pikchr for Diagrams as Code

Posted by Pete McBreen 04 Dec 2021 at 04:43

A C double headed arrow Small long text will expand the box

A different take on autonomous vehicles

Posted by Pete McBreen 02 Dec 2021 at 23:13

From Why We Drive by Matthew B. Crawford …

… the logical necessity of driverless cars becomes clear. It seems likely there will be real-time auctions to determine the route your Google car takes, so you can be offered empowering choices along the way. […] One marketer put it quite frankly: the goal is to intercept people in their daily routines with brand and promotional messages.

The book draws a distinction between the use of a car to get somewhere, and the pure joy of just going out for a drive. The creators of autonomous vehicles do not seem to be drawn from the population that enjoys going out for a drive down a twisty road.

The book also highlights the problem of incomplete features and systems

… We are not just daunted by by the obscure logic of such machines, but seem to feel ourselves responsible to them, afraid of being wrong in their presence, and therefore reluctant to challenge them even as the […] GPS directs us into a lake.

Many drivers of such vehicles even go as far as to defend these early attempts at making the technology work, thinking that it is acceptable for companies to test out their systems on public roads.

Systems designed to minimize the role of human intelligence tend to be brittle, as they are not able to anticipate every contingency. When they fail, their failures tend to be systemic, in proportion to the comprehensive reach of their control.

To date we have been lucky in that most times the driver has managed to react in time before the vehicle plows into a stationary object.

Just say no to software certifications

Posted by Pete McBreen 30 Nov 2021 at 22:36

Certifications can make sense in the mechanical world where there is a One Best Way to achieve a desired outcome, or there is a basic level of competency that is awkward to test for. So many mechanical trades have safety certificates that have to be periodically renewed, and most countries have the idea of a driving license that is a permit to drive a specific type of vehicle. After all we do not want an electrician or gas fitter getting creative with the building code.

In software though, as Perl programmers say There Is More Than One Way To Do It, and we do want developers to get creative, and by some reports parts of Ruby came from Perl. We don’t want to be certifying developers as capable of creating Perl CGI scripts when there is Ruby on Rails available. The same can be said for all of the cloud certifications, a money maker for the providers, but quickly outdated certifications as the could providers release new capabilities every few months, and hey presto, you need to take that certification exam again (and obviously pay the fee again).

Interesting take from Tim Bray on Microservices and Integration testing

Posted by Pete McBreen 30 Nov 2021 at 04:14

In talking about testing in 2021 Tim Bray says explicitly about Integration or end to end testing …

The problem is that moving from monoliths to microservices, which makes these tests more important, also makes them harder to build. Which is another good reason to stick with a nice simple monolith if you can. No, I’m not kidding.

Which in turn means you have to be sure to budget time, including design and maintenance time, for your integration testing. (Unit testing is just part of the basic coding budget.)

Another bad idea for the use of technology

Posted by Pete McBreen 25 Nov 2021 at 03:01

Of all people, Honda is experimenting with detecting people by their cellphones “in the effort to realize a society where both pedestrians and vehicles can enjoy mobility safely and with total peace of mind.”

I just wish that the article was intended as sarcasm or a joke.

And Jalopnik reported it without noting the problem

Of course, the onus is on the driver and two-ton machine to not hit pedestrians in the first place, but a system that would warn the victim of a possible collision through their phone doesn’t seem like a bad idea.

Hint to Honda: Some people do not carry cellphones with them, will it be OK to run them over in your future? If anyone is driving in an area of poor visibility, then drive a lot slower, how hard can that be?

English speaking aliens/UFO/UAP

Posted by Pete McBreen 25 Nov 2021 at 02:47

Very interesting map of the world showing UFO sightings, gotta wonder what that is about, but Doonsebury got there first

Economics of Maintaining Tests

Posted by Pete McBreen 13 Nov 2021 at 00:40

Thinking about the testing pyramid that is the common picture used in many presentations, typically highlighting Unit, Functional and UI testing, and if the diagram comes from an agile background, it will put manual testing at the top and claim that manual testing is the most expensive. The claim is normally that automated unit tests are cheap to write and fast to run, automated functional tests cost more to write and run slower, and the automated UI tests are even more expensive to write and slower to run. Manual UI tests are at the top of the pyramid, the most expensive to write and the slowest to run.

The reality may be different, but that depends on your viewpoint.

Automated testing is not actually testing. It is Automated Checking, algorithmically checking conditions on the UI. An automated check can pass if it is only checking a few items on the UI, when some other parts of the UI have changed for the worse, but are not checked. The checks pass, you have a green, but the UI is broken.

So at this point the question is whether manual UI testing is more expensive than automated UI checking, if even a cursory scan by a tester would see the problem but the automated UI check would not see the problem. Yes we can check the entire contents of the UI in the automated check, but then the code doing the checking will run a lot slower, and be more extensive. The cost of maintenance gets much larger as well, since now ANY change on that page, such as an intended refactoring, will need the automated checking code to be adjusted.

My suggestion for addressing this is to make sure that whenever a manual UI test discovers a mistake, the automated checks need to be updated to detect that problem. This makes debugging the problem and fixing the bug easier, and provides an in built regression test for the future. Ideally add a unit test if that is the smallest test that can reveal the defect, otherwise use a Functional test or a UI test if that is what it takes to reveal the error.

Hurl as an alternative to Postman

Posted by Pete McBreen 05 Oct 2021 at 19:50

Hurl is a Rust wrapper around libcurl that provides an easier to use interface to Curl. Not a replacement for Playwright, but for testing a JSON REST endpoint or validating the server side of HTML web pages it looks to have a place in the toolkit.

It is controlled either from stdin or files with a nice syntax

GET https://improvingwetware.com/
# confirm response code 
HTTP/1.1 200
[Asserts]
xpath "//div[@class='post']" count == 20 # confirm 20 posts per page
[Captures]
nextPage: xpath "string(//span[@class='next']/a/@href)" # get href to the next page

GET https://improvingwetware.com{{nextPage}} # use captured href to navigate to that page

HTTP/1.1 200 # and make sure we found that page
[Asserts]
xpath "//div[@class='post']" count == 20 

Only gotcha is that you might need to have an XPath Cheatsheet link handy if you have not needed XPath much for navigation around a web page (I mainly use CSS selectors for Selenium and Playwright)

>hurl test.hurl  --test
test.hurl: running [1/1]
test.hurl: success

executed: 1
success: 1 (100.0%)
failed: 0 (0.0%)
execution time: 1156ms

Note. If you are running under windows command prompt, you cannot use the normal wildcard to specify multiple files, as it assumes that the wildcard will be expanded by the OS. Fix is to run under WSL

>hurl *.hurl  --test
error: The filename, directory name, or volume label syntax is incorrect. (os error 123)

Under WSL there are no issues as it expands the wildcards automatically

$ ./hurl.exe *.hurl --test
t2.hurl: running [1/2]
t2.hurl: success
test.hurl: running [2/2]
test.hurl: success

executed: 2
success: 2 (100.0%)
failed: 0 (0.0%)
execution time: 2132ms

Craftsmanship and Engineering

Posted by Pete McBreen 01 Oct 2021 at 17:23

Found a recent reference to my book by Hillel Wayne in a blog post where he interviewed engineers who moved into software development to get their take on the “is software development engineering” question. Overall his take is that

there’s a much smaller gap between “software development” and “software engineering” than there is between “electrician” and “electrical engineer”, or between “trade” and “engineering” in all other fields. Most people can go between “software craft” and “software engineering” without significant retraining. We are separated from engineering by circumstance, not by essence, and we can choose to bridge that gap at will.

I have to agree with Hillel that despite having a masters degree in Manufacturing Systems Engineering I am not a traditional engineer.

I believe everything McBreen said about software is fairly reasonable, about how hard it is to predict things and how it’s intensely personally created. What he gets wrong is the assumption that engineering is not this way. Engineering is much richer, more creative, and more artistic than he thought. But of course he would have an imperfect view: he is, after all, not a traditional engineer. Are we really engineers

Upgrading from Selenium to Playwright

Posted by Pete McBreen 16 Sep 2021 at 22:37

For a long time Selenium has been the default option for writing automated browser tests, but there are alternatives now available. One leading contender is Playwright, and while it does not have as much language support as Selenium, it does cover the main options (JavaScript, Python, Java and .Net) while providing full browser support.

With a large suite of automated Selenium tests, it might not make much sense to rewrite existing tests to use Playwright, but for new test suites it might make sense to switch over to Playwright. It has better affordances for testing, has easy control over the browser contexts without that getting in the way for normal usage and deals with single page applications and websockets. The python samples below are equivalent (other than playwright does not show browser by default).

Playwright

import pytest
from playwright.sync_api import sync_playwright

@pytest.fixture
def browser():
    playwright = sync_playwright().start()
    browser = playwright.chromium.launch()  # use params headless=False to see browser

    yield browser

    browser.close()
    playwright.stop()

def test_blog_title(browser):
    page = browser.new_page()
    page.goto("https://www.selenium.dev/")

    # find the blog link in the header and then click on that link
    page.click("#main_navbar :text('blog')")
    title = page.title()
    assert title.startswith("Blog | Selenium"), title

    h1text = page.wait_for_selector('h1')
    assert h1text.inner_text() == "Selenium Blog"

Selenium

import pytest
from selenium import webdriver
from selenium.webdriver.common.by import By

@pytest.fixture
def webbrowser():
    firefox = webdriver.Firefox() # default is to show the browser, use options to make headless
    yield firefox
    firefox.close()


def test_blog_title(webbrowser):

    webbrowser.get("https://www.selenium.dev/")

    # want the blog link from header, so two stage process to get correct element
    header = webbrowser.find_element(By.ID,'main_navbar')
    blog = header.find_element(By.PARTIAL_LINK_TEXT , 'Blog')
    blog.click()

    title = webbrowser.title
    assert "Blog | Selenium" in title, title

    assert webbrowser.find_element(By.CSS_SELECTOR, 'h1').text == "Selenium Blog"

Slow decision making as a failure mode for Scrum

Posted by Pete McBreen 09 Sep 2021 at 21:03

Although Scrum can operate on with one to four week sprints, most organizations currently choose a one or two week duration for their sprints. This limits how long a team can take to make decisions, since with only five or ten days in the sprint, taking a day or two to organize a meeting to discuss the decision will impact delivery.

Some common failures that I have seen include:

  • Team members letting an open Pull Request sit around for a day or more because they are too busy to review the changes in the PR
  • Developers not immediately merging approved Pull Requests so the changes are delayed getting to the test environment
  • Database changes that impact a few tables needing to be discussed with the DBA who is not available until later in the week
  • Defects reported by testers sitting waiting for a developer to have the time to review report

All of the above are examples of extra time incurred in the process of taking an item from the backlog and moving it to the Done state. This increases the likelihood that there will be a rush to complete items at the end of the sprint, or that the overall sprint goal will not be met.

The fix I see for this is to build slack into the sprint so that the team is not busy all the time. Aiming for 100% utilization always leads to queuing and delays, so the team has to build in appropriate slack time to prevent decision delays.

Note. Martin Fowler has a possible fix for delays caused by the Pull Requests needing reviews.

Microservices, the Cloud and Us

Posted by Pete McBreen 24 Aug 2021 at 21:04

In a presentation on Modular Monoliths Simon Brown had a great comment on Microservices

If you can’t build a modular monolith, what makes you think microservices is the answer?

Or as Architect Clippy said it

I see you have a poorly structured monolith. Would you like me to convert it into a poorly structured set of microservices?

Marcus Ranum has had similar thoughts

I love it when software developers say “How hard can it be?!” and decide to build their own complete replacement system. The results are usually about as bad as the first system, for the same reason. To be fair, this stuff is really hard to write – which is all the more reason to be skeptical when someone says they’ll just put together a modular cloud-based version of their own. You should always ask “why do you believe you will get right the things that everyone else got wrong? Because the reasons that they got it wrong apply to you, as well.”

Overall it seems that although the cloud and microservices are seen as synonymous, microservices bring with them a lot of extra complexity and interdependencies that cost time and effort to resolve and work around. Justin Etheredge has written that it is all about scale

The problems I’ve dealt with are not at the scale of Google, Facebook, or Uber.

Few organizations have to deal with the problem of how to coordinate more than 100 developers. Yes, Spotify and Amazon have that sort of problem, but most of us do not. So it is therefore very likely that the solutions that work for their scale of problem might not be the best ones for the rest of us.

An example that I have seen recently is that of a web application that was moved from the old style LAMP stack to an Angular single page application front end supported by a REST API at the backend. So now a change to the requirements meant that the release of the typescript changes to the angular pages had to be coordinated with the release of the REST API changes (which needed to be versioned in case there were old clients using the prior version of the service).

Revisiting "Applying the Lessons of eXtreme Programming"

Posted by Pete McBreen 11 Aug 2021 at 17:37

Way back in 2000 for the TOOLS34 conference I wrote an article Applying the Lessons of eXtreme Programming and it was interesting to look back at it to see how things have worked out since then.

  • Applying JUnit – the unit test frameworks as not as widely utilized as I expected, but now all modern programming languages ship with a flavor of unit testing built in
  • Be intolerant of process variations and working off process – many teams have not learned this lesson and have problems arising from this
  • Use a coach to keep the team on process and improve individual skills – this does not seem to have caught on, even in teams that use Scrum, the scrum master role does not always enforce process
  • Apply the Quality First Strategy – I still see to many teams that allow code to be merged without adequate tests that later prove to contain incorrect code
  • Continuous Integration is the only way to avoid Integration Hell – I’d soften this to a way to avoid, and many teams still have problems making this work
  • Tired humans produce lousy software – still true, and even agile teams sometimes forget this
  • Incremental Development requires Incremental Requirements Capture – still true, some teams are getting good at this
  • Simple Designs are easier to maintain and evolve – in the era of microservices, we seem to be forgetting this lesson. Yes the microservices themselves are simple, but the interaction between multiple microservices are really hard to maintain for a lot of teams
  • Adjust your development process slowly and measure the effects of the change – I still think this is a good lesson from XP, but all too many teams are not very good at this
  • Many projects would benefit from a good dose of Reality Therapy – still true unfortunately

Hyperloop Delusions

Posted by Pete McBreen 25 Jun 2021 at 23:03

Now the Calgary - Edmonton link is dreaming about a hyperloop, Alberta’s version is called Transpod. Did no reporter look at the image and ask any questions?

  • The pretty photoshop picture shows a transparent tube - that is going to work well as a structural material to contain a vacuum
  • These transparent tubes are called “magnetic tubes” – there are very few transparent materials that are able to generate or support strong magnetic fields
  • Speeds of up to 1,000 kilometres an hour, that is going to need a pretty hard vacuum to prevent the pod from generating a massive pressure wave in front of it as it travels at that speed
  • The picture shows nothing in the way of pumping infrastructure to maintain the vacuum
  • How long will it take to pump the vacuum down at each end of the line?
  • What provision is there for passengers to escape from the pod in the case of an issue? It is not like you can walk through a vacuum
  • Why did nobody ask about all the other vapourware hyperloops that have been in development? Nobody has yet demonstrated anything beyond a toy prototype.
  • Why did nobody as why not just build a high speed rail line for passengers and cargo?

SWEBOK 3 is out

Posted by Pete McBreen 22 Apr 2021 at 20:51

Not had a chance to read it in detail but one section stood out

Parameterized types, also known as generics (Ada, Eiffel) and templates (C++), enable the definition of a type or class without specifying all the other types it uses. The unspecified types are supplied as parameters at the point of use. Parameterized types provide a third way (in addition to class inheritance and object composition) to com-pose behaviors in object-oriented software.

I would have thought that this updated reference would use more modern languages for the examples. Ada might still be in use, but Eiffel never really caught on, might have been more relevant to use Go, Rust or even Java as examples in this context.

PlantUML has a good way of visualizing JSON

Posted by Pete McBreen 24 Jan 2021 at 03:55

Ran across this while looking at c4model.com for a way to visualize the architecture of a system.

PlantUML now supports JSON in that it can draw a diagram that reflects the structure of the JSON, and it does it relatively simply. The #highlight “phoneNumbers” provides highlighting on the image, and even better, you can use markdown like highlighting inside the JSON to do the usual bolding as per the firstName at the top of the JSON.

@startjson
#highlight "phoneNumbers"
{"**firstName**": "John","lastName": "Smith","isAlive": true,"age": 27,"address": 
{"streetAddress": "21 2nd Street","city": "New York","state": "NY","postalCode": "10021-3100"},
"phoneNumbers": [{"type": "home","number": "212 555-1234"},
{"type": "office","number": "646 555-4567"}],
"children": [],"spouse": null}
@endjson

Then simply execute plantuml to generate the image

> java -jar plantuml.jar -tpng json.txt

Resulting Image that can be rendered into most formats that you might be interested

JSON Image

This is a great example of the concept of Diagrams as Text as it gives a source that is easy to diff while still allowing an easy to view presentation format.

Another take on Engineering vs. Craftsmanship

Posted by Pete McBreen 19 Jan 2021 at 13:02

Hillel Wayne has an interesting take

Many people have asked me why I care so much about this project. Why does it matter whether or not software is “really” engineering? Why can’t we just say that “software is software”? It’s because of these misconceptions. People have a stereotyped notion of what engineering looks like. Because software doesn’t look like the stereotype, they assume that we are wholly unlike engineering. The engineering disciplines have nothing to teach us. We are breaking pristine ground and have no broader history to guide us.

Is Serverless a return to the days of batch mainframe processing?

Posted by Pete McBreen 21 Dec 2020 at 23:48

Cees de Groot seems to think so

You deploy, get an error message, and login to CloudWatch to see what actually happened - it’s all batch-driven, just like the bad old days, so progress is slow. At least we’re not having to walk to the printer on every try, but that pretty much sums up the progress of the last half century. Oh, and a “Function” will be able to handle a single request concurrently, so here’s your AWS hosting bill, we needed a lot of instances, I hope you won’t have a heart attack. Yes, we run workloads that your nephew can run on his Raspberry Pi 4, but this is the future of enterprise.