The Story So Far
About two weeks ago, I wrote a blog post suggesting that Scala can make Java developers more productive. One of the comments on that post was that I had no metrics and so was spreading “folklore”. I took the criticism constructively and decided to solicit a small survey of Scala developers to try and get some insight.
What I Wanted to Learn
One of the most important things I’ve read about Lean Startups – and it’s important because it can be applied to a gamut of contexts, not just startups – is that, while the execution of the Lean process is Build, Measure, Learn, you actually have to apply this backwards to get good results: first you decide what you want to learn, that will inform what you need to measure, and that will determine what you need to build.
I employed this advice with this survey, asking first: “What do I hope we learn from this”, and using that to inform the content. Here are the questions for which I wanted the survey to provide some insight:
- Do developers feel more productive working in Scala?
- Do teams deliver more value with Scala?
- Do feelings of productivity translate to teams delivering value?
- How is someone’s sense of productivity affected by:
- the length of time they’ve been working with Scala,
- the proportion of Scala they practice at home and at work,
- the proportion of Scala in their work codebase, and
- their depth of understanding of functional programming?
- Does depth of understanding of functional programming translate to teams delivering value?
Here is the method I used to collect data which I thought would help give some insight on these questions:
- I created a short online survey,
- that asked Scala developers with Java experience
- to give a subjective opinion on whether they feel more productive in Scala when compared to Java
- and whether they think their team produces more value than when using Java
- and to estimate the % of time they spend using Scala at home and at work and the % of Scala in their codebase
- and to rate their own expertise in Functional Programming
- I publicised the survey on Twitter and Google+
Criticism of The Method
It would be obvious to a 10 year old: this is not a highly-scientific method of collecting data. Every italicised phrase in the points above highlights a weakness of my technique: the participants are not randomly selected, are probably all biased to varying degrees, and have been asked to give subjective opinions and estimates for which they themselves probably have no data. In short, I’ve asked a skewed audience to share their gut feel. But, what else could I have done? I don’t think it would make much sense to ask people who don’t use Scala if it made them more productive. I don’t have the resources to undertake an accurate measure of people’s or teams’ productivity before and after taking up Scala. And I don’t have the social clout to get more people taking the survey than already took it. So, by my measure, what I’ve done is the best I could do given my situation.
If you think the method I’ve used is codswallop and makes the output totally useless, then there’s something productive you can do about that: stop reading now. Don’t bother reading the results and making yourself more upset. Please don’t try to “make things better” by adding a comment telling me how bad my method is; you’ll just expose yourself as someone who comments on blogs without reading them properly.
If you have some great ideas on how this survey could have been better – methods of collecting better data, questions that should have been asked, options that should have been available, channels that should have been tapped – there’s something productive you can do about that, too: create your own survey, that’s better than mine, and then tell publicise it better than I did. I won’t be offended. I’d be really interested in it and probably help you. But again, please don’t add a comment telling me my method could have been better if I’d just done X, Y and Z; I already know that it could have been better, and you’ll just expose yourself as someone who comments on blogs without reading them properly.
Enough yabba yabba. I got 441 responses. Below I’ve charted the results of the survey, including some correlations between survey answers that I believe help answer the questions I started out with above. The images in the post are scaled down, so you’ll get a better view if you click on them to open the full version.
I’m not actually going to write about any conclusions from these. I think it would be far more interesting to hear what YOU think stands out, so if you see something you think is interesting, please add a comment! If you want to know what other people have seen in here that’s noteworthy, just keep scrolling.
Do developers feel more productive working in Scala?
Do teams deliver more value with Scala?
Do feelings of productivity translate to teams delivering value?
Does the length of time someone has been working with Scala affect their sense of productivity?
Does the proportion of new code that someone writes in Scala affect their sense of productivity?
Does the proportion of Scala in the codebase someone works with affect their sense of productivity?
Does a developer’s understanding of functional programming affect their sense of productivity?
Does understanding of functional programming translate to teams delivering value?
As promised, here is the data if you have some interest in examining correlations other than those I’ve charted above.
Image credit: Mykl Roventine