Notes from YOW! 2014: Martin Thompson and Todd L. Montgomery on ‘How Did We End Up Here?’

I attended YOW! Sydney 2014 and thought some people might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

Cows standing in front of a burning barn.Martin Thompson (@mjpt777) and Todd L. Montgomery (@toddlmontgomery) discussed the state of the software industry at YOW! 2014, including “barbequing” a whole herd of sacred cows. (Slides)

A Dr Dobbs 2010 report into IT project success showed a correlation between higher numbers of people on a project and higher rates of failure. Even the best performing methodologies still have >10% failure.

A Gartner report shows a correlation between the amount of money spent on a project and the likelihood of failure. Yes, the more money ploughed into a project, the more likely it is to fail! (Having looked at the data, the difference was actually only a couple of %)

When LMAX started open sourcing stuff, recruiting became really easy, to the point where they didn’t even need to advertise.

They questioned whether Product Owners – people running a project but who aren’t part of the business – are useful, and suggested what you really need for a project to succeed is a business expert who’s invested in the project’s success, but also busy being heavily involved in the day-to-day running of the business.

Technologists (Software Development, Testing and Operations groups) need to stop talking about “them and us”, and stop talking about solving people’s problems. Instead, they should realise they are part of the business, and take responsibility for the ROI they deliver.

In discussing getting rid of shared mutable state, they recommend considering “no update” databases, something we’ve been doing more and more of at Tyro the last few years. This basically means banning the use of the UPDATE statement so that your database becomes a record of new facts discovered over time. The current state is easily found by looking at the most recent facts, but the key properties are that information about the past is never lost (unlike with UPDATE statements) and contention over the database is typically easier to manage.

They recommended everyone with an interest in performance read up on Universal Scalability Law (USL), a newer model of calculating the limit of concurrent scalability that subsumes Amhdahl’s law.

They mentioned that, in their work helping other teams fix their performance problems, protocol translation to and from text is a regular cause of performance problems. Consider binary protocols instead.

They brought up the hotness of microservices, and delivered one of the quotes of the conference:

“Synchronous communication is the crystal meth of distributed software.”

It’s easy and feels good, but it’s very bad for you in the long run. Why? Because of temporal coupling. A great analogy they put forward: “Do you stand next to your letter box when you’re waiting for a reply?”

They asked “What’ is the biggest problem with functional programming?” and offered the answer “Functional programmers!”
Cartoon of a high-brow stick figure saying to a stick figure with a broken hat: "I'm sorry, I can't hear you. Your inferiority is too loud."

They claimed to debunk the myth that functional programming is the solution to multi-core. Can’t remember the arguments here, unfortunately.

They said many Ops teams are still motivated to avoid changes (based on the goals the company gives them). Instead, the companies should be motivating them to optimise for speed of delivery.

Virtualisation adds a cost! If you’re in a performance-sensitive environment you need to consider if it’s worth it.

They finished off discussing the state of diversity, in particular gender diversity, in the industry, and showed this chart of the percentage of female majors in CS courses decreasing steadily from 1985 onwards.

A chart showing a steady decline in  the percentage of women in CS majors since 1985

The source story has some interesting discussion, including an assertion that PCs became widely available in about the mid-80s and were mostly marketed at boys, ingraining a “computers are for males” mentality in the public. There’s also some research indicating that families with boys are more likely to buy computers than those with girls, and that CS teachers eventually began assuming home ownership of a computer and experience with programming, something that was less likely to be true for female students.

We discussed this topic in our group of Tyro developers at lunch time, including one of our female colleagues (who confirmed her family had a computer growing up). I found speaking about the issue with others crystallised some things I’ve been thinking about for a while, and I realised there are actually two separate issues: women not choosing IT in the first place, and then women who’ve chosen IT leaving it. Both issues are bad for the industry in general, and companies with low diversity in particular, because diversity has been shown time and time again to be correlated with successful teams.* The second issue – women choosing to leave IT – is very worrying if it suggests that we have an industry that is generally unsupportive of women. I’ve read about a number of women’s negative experiences on the web, though always in the USA. Could this be because Australia has less of a problem in terms of women not being accepted in IT? Or have I just been fortunate enough to work in some of the few places where it isn’t a problem? If anyone has any Australian-based research on gender diversity in IT, I’d appreciate being pointed to it.


* For two examples of gender diversity research, see ‘The Case Against the All-Male Startup’ on Inc and ‘What makes a team smarter? More women.’ on HBR. Google can get you to plenty of other research.

Notes from Microservices Talk by Zhamak Dehghani

People have started using honeycombs and hex shapes to depict microservices architectures. Who knows why?A couple of weeks ago, I went along with a couple of other Tyro software engineers to hear Zhamak Dehghani speak about microservices at a “YOW Nights” event, hosted by Optivar and sponsored by ThoughtWorks. It was so good that we asked Zhamak if she’d come into the Tyro office and give a re-run for the whole Engineering team and she kindly obliged. What a legend! Thanks again Zhamak!

I’d already read a lot about microservices (MS), mostly thanks to the excellent pages of links put together by Adrian Rossouw and Matt Stine. Zhamak covered a lot of ground that I was already familiar with, but she also touched on many things that were new and interesting to me, so I thought I would write about a few here. Continue reading

Does Java 8′s lambda capability make Scala obsolete?

How does using Java 8's lambdas compare to writing Scala?I didn’t think anyone would seriously ask this question. However, after yesterday’s post about why your company should let you use Scala at work, which used a simple example showing the use of lambdas in Scala, I had someone write in the comments:

“java 8 equivalent of your example would be identical, no need for Scala…”

and someone else commented on Twitter:

The call to compare against Java 8 is a fair one, so here we go… Continue reading

From the Archive: Why Your Company Should Let You Use Scala at Work

From the archive: Originally written in January 2010, this post from my old blog, Graham Hacking Scala, has been consistently popular ever since and I thought it deserved a new lease on life here…

Java at work: an increasingly frustrating experience

If Raffa wanted to use Scala at work because he thought it would help him win, would you make him use Java?A colleague and I were writing a test in Java today and I had one of those “I can’t believe it takes this much work to write this stupid function” moments. If you do any Scala programming but still do a lot of Java programming, you’ll know exactly what I mean. I’ll walk you through a mock of what we were doing and you can see why Scala just pummels Java when it comes to expressiveness. (Early warning: do not use the word “expressive” when trying to convince your manager to let you use Scala at work.) Continue reading

9 Design Patterns Translated Into Java 8′s Lambdas

LAMBDAS!

There’s been a lot of hype around lambdas getting introduced in Java 8, and I have a good theory on why: hype is often born out of anticipation, and we’ve been anticipating lambdas in Java for a LONG time.

Lambdas?

A lambda spray paint pattern, the same design as in Half-LifeThe funny thing is that lambdas don’t, by themselves, do anything new. They’re just a succinct form for turning a block of code into an object that can be passed around – syntactic sugar. Some of us have been doing this for a while without the succinctness and we call it… object-oriented programming! Yes, passing blocks of code (aka functions) around as objects (aka values) is also core to functional programming, but certainly not unique to it.

Patterns!

Somewhere that this idea of “passing code around” is heavily utilised is in Continue reading

RxJava Threading Examples

I recently used RxJava while creating an add-on for Stash, the Git repository management tool from Atlassian. The plugin’s called “Who’s the Expert?” and it analyses commits to a repository to help answer two questions:

  1. “Who has contributed the most to this project over the past few years?” and
  2. “Who has made the most significant changes in recent months?”

A Problem Worthy of RxJava’s Attention

This picture of water drops on the threads of a spider web reminded me of the bubble diagrams the RxJava team uses to explain how values move through the libraryIn order to achieve this, the plugin has to process a lot of data: it pulls all commits on the default branch for the last 2.5 years and analyses the content of every single one. I knew two things about this code in advance: first, there were going to be a lot of steps to go from a repository name to a leaderboard of the most influential committers, and second, I was pretty certain I’d need some multi-threading mojo in order to get it to perform in an acceptable time frame. Continue reading

Notes from YOW! 2013: Michael T. Nygard on ‘Five Years of DevOps: Where are we Now?’

I attended Day 1 of YOW! Sydney 2013 and thought some people might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

Small child in a field looking into the distance with binoculars, as someone surveying the current state of DevOps probably wouldn't do.Michael T. Nygard (@mtnygard) is probably best known as the author of the 2007 book ‘Release It!‘, which teaches developers how to look beyond just getting their code working and instead design it from the outset to handle the harsh conditions of production environments. He has since become a DevOps luminary and now works at Cognitect. He spoke at YOW! 2013 about ‘Five Years of DevOps: Where are we Now?’.

Michael started off setting the timeline by pointing out that Chef and Puppet were preceded by CFEngine in about 1993!

He explained how his own experience has contributed to his DevOps insight: he worked as a Dev in Ops for some time, showing the Ops team how to solve some of their problems with Dev-like approaches, but also finding lots of problems with the way Devs created software, which was the chief inspiration for his book. Continue reading

Quick Response to “Test-induced Design Damage”

David Heinemeier Hansson wrote an interesting critique of Test Driven Design two days ago. I understand where he’s coming from with his concern that mindlessly shaping all design around tests can (will?) lead to poor design in some areas. It was interesting, and I like to be challenged and think about these things. Personally, I think his extrapolation to proposing “TDD is Dead” is hyperbole. Here are my thoughts on the bits of his blog that jumped out at me…

“But the testing pyramid prescribes that the unit level is where the focus should be, so people are sucked into that by default.”

It’s probably true that some people are sucked into always defaulting to the unit level. The problem with that is the “always”, but the unit level is where the focus should be. Why? Because that is where the complexity lives – in each unit. The reason we separate code into units is to separate complexity into simpler, composable parts. Testing exhaustively at non-unit levels (i.e. by integrating components) requires a combinatorial explosion of tests… or not testing exhaustively (might be a viable option at  your work, but not at a payments company). We use integration tests as well, but I don’t typically use them to prove all facets of functionality, instead only to prove the components integrate correctly and can deliver the core functions.

“Controllers are meant to be integration tested, not unit tested.”

I work in the Java world, not the Rails world, and I completely agree with this. When I started at Tyro, we were writing unit tests for controllers and integrated web tests for the presentation (i.e. JSPs)  of most features as well. The result? All controller code was tested twice. (It’s pretty hard to test a JSP without hitting its controller.) There’s a dirty word to describe this practice: waste. We ditched controller tests a long time ago now and no one has ever missed them. Controllers are still tested, just not by unit tests. From time to time, some logic in a controller might get a bit complex, resulting in more paths than are practical to web test. The solution is easy there: extract that complexity into another class, unit test the complexity, and web test the integration. Controllers should never be complex. A David points out, they are mostly just an adapter layer between models and views. Let any kind of business logic complexity live in there and it’s got two responsibilities. Your goal should be to make controllers so simple that they don’t need unit tests.

“Finally, the fear of letting model tests talk to the database is outdated”

Yes, and no. Yes, there should not be a fear of testing against the database. In fact, there should be a preference towards it. Not testing against real databases (the same ones you’ll use in Production) leaves a big fat layer of assumptions in your app. However, in large systems with lots of database interaction and great test coverage you can quickly max out the build time if you use the database everywhere. My thoughts: have a preference for DB-backed testing, but not at the expense of developer productivity. The team needs to be aware of when the preference is damaging their velocity and find the balance. I’ve written a lot of data-heavy, back-end Java, so mileage with Ruby on Rails may vary (maybe it never becomes a problem).

“Above all, you do not let your tests drive your design, you let your design drive your tests!”

I don’t think it’s this simple. One great advantage of using tests to drive design is that your desire for simple tests commutes to an implementation of simple classes. Yes, you can probably achieve similar outcomes by doing some more thinking or drawing, but test-first is also a useful tool for driving towards this goal. Not the only tool, or a required tool, but a useful tool. I write lots of simple code at home and rarely write tests for it, but recently I found one part I was writing was a bit gnarly so I decided to write a unit test for it. Upon starting to write the test, I found I wasn’t able to because of the design of the code, and when I thought about the design for a second I realised I was mashing several responsibilities into the one place. Tests can drive designs to bad places, but in my experience they more often drive them to good places.

I often re-iterate to my team that testing is primarily about building confidence in the code, and secondly about building a safety net for those who pass this way next. There is no mandate in TDD for writing tests that do not build confidence (do you TDD getter methods?), or for spending hours on tests that increase confidence by small fractions. I think David is right that there are probably zealots out there who are blindly going down these kinds of paths. Perhaps he’s right that some designs are bastardised by having stringent testability requirements, though I don’t recall seeing a lot of this in my travels. So, he’s right that mindlessly shaping all design around tests can have bad effects, but no software developer should be doing anything mindlessly. I think there is value in letting tests shape your design much of the time, so long as one keeps in mind that there will be occasions where the tests have to be shaped by the design instead.

Everything in moderation, right? Including moderation.

Notes from YOW! 2013: Hadi Hariri on ‘Refactoring Legacy Codebases’

I attended Day 1 of YOW! Sydney 2013 and thought some people might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

A stack of stickers showing the word 'refactor' in a stylised, death-metal-like font.Hadi Hariri is a Developer and Technical Evangelist at JetBrains. He spoke at YOW! about refactoring legacy codebases.

Hadi started by offering reasons for refactoring:

  • increase the understandability of the code
  • decrease the impact of change
  • reduce the cost of change

He described how refactoring relies on culture: Continue reading

Why You’ve Never Heard About Steve Jobs’ Skateboarding Prowess

… and the secrets of 7 other famously successful people.

There would be few adults in the western world who have not heard of Steve Jobs. A pioneer of personal computing, an entrepreneur of unrivalled clout, one-time owner of the world’s most successful digital animation studio, the man who took the helm of an almost bankrupt computer company and transformed it into the most valuable business in the world.

But have you ever heard about Steve Jobs’ skateboarding skills? Continue reading