Memories from 10 years at a startup

Black and white photo of four kids on an outdoor cubby houseI recently notched up ten years of working at Tyro Payments, a tiny little payments startup that I joined in late 2006 when it had about 20 people, and which is now Australia’s most successful fintech venture, Australia’s newest bank, and is fast approaching 400 staff.

Here’s a collection of some of my most vivid recollections from so far in this fantastic journey.

Getting Hired

I remember… when the recruiter for Tyro said he had a finance role, and I reiterated to him that I wasn’t going to take any job in finance. He back-pedalled, convinced me it wasn’t really a finance company and I really had to go meet them. I was reluctant, but I went along because I wanted to be in his good books.

I remember… the hook for me was one of the founders – Peter Haig – drawing the architecture they were building and me thinking 1) this is very cool and 2) this guy is very smart.

I remember… I plucked up the courage to ask where the company got its money from after three years of runway with no customers, and the founder was completely unfazed that a software engineer would ask such a question.

I remember… asking whether they really wrote tests before making any changes to the system, even to web pages, and Matt Milliss saying, “To be honest, I can’t believe how much code I wrote without tests before I came here, and how many bugs I must have created that no one has found yet.” It was a statement that shook my confidence in myself as an engineer and made me want to learn from this mature group of people that seemed to have a bunch of things figured out that most other software teams were still battling with.

I remember… I was so concerned about whether or not I’d enjoy pair programming that I asked for extra time to think about the offer.

I remember… discussing with my wife what would happen if I joined a startup and it went under and how long we thought we could survive on her salary if it took a while for me to find another job.

Startup Land

Hikers starting up a long steep climbI remember… on my first day I spent the morning pairing with Mark, who patiently spent the whole time telling me about the architecture of the system. We swapped pairs at lunchtime and I stayed on the task. My new pair, Geoff, asked me what we’d done in the morning and I said, “Nothing, we just talked about the system.” He looked at me strange, shook his head, and said “Right”. Then he changed a test, I changed the implementation to make it pass, and within half an hour we’d checked my first change into the mainline.

I remember… for the first six months having to explain to people that I worked at a company called MoneySwitch, watching their confused and concerned face, and preparing to explain that it wasn’t a money-laundering operation.

I remember… in the first office I worked in, we had no dishwasher and everyone was on a washing up roster, including the CEO.

I remember… that first office got really hot and the building aircon sucked. We had a single portable air conditioner… which was dedicated to the server room. On hot days we’d open a few windows and try to create a cross-breeze with a fan or two, but mostly we’d just sweat it out and keep going, or take a break for slurpees.

I remember… the whole Dev team – all 12 of us – used to go out to the mall to eat lunch together almost every day. There was no room to eat lunch in the office.

I remember… that the fortnightly demo involved the whole company – all 20 of us – cramming into the 3m x 10m board room and sharing a single case of beer.

I remember… when we moved to occupy two levels of 125 York St (now home to Blue Chilli) and it felt like we’d been upgraded to The Ritz.

I remember… when, less than 6 months in, my team leader proactively had my salary adjusted up because she thought Tyro had unwittingly hired me cheap based on my experience. It was an eye-opening example of how serious Tyro was about living out the company’s values.

Engineering Feats and Follies

Sydney Harbour Bridge being constructedI remember… when Ricky Yim and I got to do a spike of making a POS and an EFTPOS terminal integrate by communicating over the internet and through our system instead of over a serial cable and we got it working in a couple of days – probably Australia’s first “cloud” POS integration.

I remember… we discovered two years later that a demultiplexer in our POS integration was using a single thread to process all requests from thousands of clients and had only just started to struggle with that load. We bought ourselves about another decade of scaling by changing a config file.

I remember… I “refactored” an API between two systems that had some poorly-named fields, and a week later we mysteriously stopped paying merchants for the cashout they’d given customers. And we only knew about it when they called to complain. No one ever asked who’d made this bug happen, and instead the team had a conversation about how to stop it from happening again.

I remember… us being (equal) first to be accepted to be an acquirer for Medicare EasyClaim online claiming, and the excitement and enthusiasm of being part of building this huge new thing that was going to help improve the country’s public health system. Today, Tyro processes more than half of all the claims on the EasyClaim system, despite four other major banks offering EasyClaim services.

I remember… a litany of small tools that I built at home and brought to work: Canopy, the Maven multi-project build UI; Mini-Rest, a tiny Servlet library that forced people to use proper REST semantics; SodaTest, a Scala-based integration testing framework intended as a replacement for FIT; “Who’s the Expert?”, a smart contributor scoreboard for Bitbucket Server; and probably heaps of others I’ve forgotten.

I remember… relishing the opportunity to be part of rebuilding the accounting system from scratch to fix its performance and maintainability issues. It’s still humming along half a decade later!

I remember… half-jokingly whinging to my boss about how it was coming up to 12 months since I’d been working on nothing but the accounting system, and I might have to find another job if I was going to be doing nothing but accounting for the rest of my life. He moved me off that piece of work a week before 12 months.

Levelling Up

Red and white mushroom, like one used in the Mario game for a power upI remember… when I and three others were pulled aside to a corner of the office (we didn’t have any meeting rooms) to meet Mike Cannon-Brookes. He was considering joining our board but he wanted to meet (and grill!) some of the senior Engineering staff first so that he could check whether we were as good as our management team said we were. He joined, so I guess we did okay.

I remember… asking if I could take on a few more responsibilities because I was bored and, instead of being given an extra job or two, I was told I’d be made an Engineering Lead.

I remember… first getting involved in interviewing, not having any idea what I was doing, and feeling the weight of making decisions that would significantly affect other people’s lives and the future of the company.

I remember… we used to ask people to write a function on a whiteboard to calculate factorials and it took us way too long to realise it wasn’t testing people’s coding skills at all and that just getting people to do some pairing was way better.

A chart of the size of the Engineering team at Tyro showing exponential growth from 2013 to 2016I remember… we hired so many people that we doubled the size of the team in 18 months. And then we did it again. And then we did it again.

I remember… that I was just coping to remember all the new people’s names, then I went on holiday for a week, missed four new starters, and never recovered.

I remember… when the Engineering team eventually grew so big that we reluctantly moved Ops to a different floor of the building, and communication between the two groups plummeted.

I remember… pairing with Ops on mystery catastrophe nights, all scratching our heads, ordering pizza when we realised it was way past dinner time, and that “Aha!” feeling when someone found an entry in a log or ran a query on a table that had seemed totally unrelated to the symptoms but showed something that no one expected could ever happen.

I remember… organising our first innovation sprint – ‘CodeBlitz’ – and being surprised by the different mood in the room as people were intensely focused on getting their own innovative idea hacked together within two days so they could demo it.

I remember… when we decided it would be a good idea to build our next link to another bank as its own piece of software rather than adding it to an existing monolith – one of our first steps toward a microservices architecture (before that word existed). We recently celebrated having over 100 services deployed in Production.

The Journey to the Bank

Four people in suits walking in front of a building with tall, thick columns, possibly a bank or museum, with long shadows in the early morning light.I remember… that time my boss came to me and asked if my team could start on building a bank the next week.

I remember… when we took a completely agile approach to building a bank…

“Where do we start?”
“Well, what’s the most important thing a bank does?”
“Keeps people’s money in an account.”
“Okay, let’s write a piece of software that keeps money in an account instead of sending it to another bank.”
And so we wrote that and got it working before we did anything else.

I remember… when the regulator announced they were retiring our limited EFTPOS banking license, and we were told the banking side-project was now the rush to a full banking license mission, the #1 priority of the company, and that we had 18 months to deliver.

I remember… when we were 12 months into building our core banking system, just weeks away from delivering it exactly when we had estimated… and someone uncovered 3 more months of mandatory work.

I remember… when we received the results of an external security audit of our in-progress core banking system and the synopsis was “You guys are pretty up there with your security, but here’s 15 other major things you should think about improving.”

I remember… the frankly unbelievable news that we’d been granted a full banking license by APRA. It felt like a war was over.

I remember… when we’d just finished turning on the core banking system and we were told, “Well done. Now you’ve got 12 months to build a lending product from scratch, ship it, and get customers onto it.” We did it in 6.

Time Keeps on Slipping

Public binoculars with a small lake just beyond and a foggy, blurred landscape in the background.I remember… feeling like my contribution to the company was having less and less impact due to the growth and organisational changes, and starting to seriously think about moving on to somewhere else. Eventually, I realised that I didn’t have to be bound by my historical contribution to Engineering, and that Product Management might be a place where I could probably develop a new expertise and have a large impact on the company’s future.

I remember… getting up the courage to ask the co-founder and Head of Product about moving into a Product Management role, expecting to get a grilling about what product was and what skills I had that I thought would be relevant. Instead his response was just, “Yeah, that makes sense. When would you like to move?”

I asked my wife what she remembers, and she said she remembers when she would show up to the Christmas party and know everyone’s name, but nowadays I sometimes tell her I’m going to someone’s leaving drinks and she says she’s never heard of them.

I remember… starting to reflect on what I’d been part of in 10 years at Tyro and realising there was almost nothing that I’d achieved by myself; everything significant I’d been involved in had been a team effort – sometimes a team of two, sometimes over a hundred, but almost always a team. That’s how we roll.

Image credits:
Kids, older photo‘ by Joe Crawford
Start of the Steep Trail‘ by Dan Cook
Sydney Harbour Bridge‘ by gramarye
Power ups‘ by Kirt Edblom
Bankers‘ by Chris Brown
Eye of the Beholder‘ by hjl (flickr)

Distributed Transactions: The Icebergs of Microservices

An antarctic iceberg which, much like distributed transactions in microservices, can be hard to see and can wreck your ship.Why are distributed transactions icebergs? It’s not because they’re cool and beautiful and you have to look under the surface to comprehend them.

Distributed transactions are icebergs because (1) it’s easy to not see them, even when they’re right in front of you, and (2) if you run into one, it’s got a great potential to sink your ship. Continue reading

Why Your Resume Sucks & How to Fix It

Shredded paper in a waste bin - where the majority of resumes end up!I’ve seen a lot of software engineers’ resumes over the past few years. And most of them suck. Even the resumes of really good people who we’ve hired have often been very average.

Why is that? I’m going to tell you why, and then I’m going to help you avoid the same mistake. And while my experience is mainly in hiring for IT-related roles, this advice can be used by any job seeker. Continue reading

Design Kata: Learn software architecture while having fun

Children practising a karate kata. This article describes design kata, where software developers practise designing software architecture.At Tyro, we don’t have anyone with the title of Architect, Designer or Tech Lead who designs software and then asks other people to build it. Instead, each team of Software Engineers does architecture and design as part of their iteration planning sessions for the stories we’re taking on.

As a Team Lead, the two main goals I keep in mind for each person are: keep them happy and keep them growing. One of the ways I try to ensure this happens is by meeting with each person regularly so we can chat about how to make sure these things are true or at least heading in the right direction.

A topic that’s come up with a few people lately is software design and architecture. Specifically, I’ve had a couple of comments along the lines of: “When we’re designing things, I want to contribute more, but I’m not really sure where to start.” Continue reading

Microservices Security: All The Questions You Should Be Asking

I spoke earlier in the year at the Sydney Microservices Meetup about the long path we’ve taken at Tyro Payments over the last decade, gradually tending towards a more fine-grained SOA approach – microservices as it’s come to be known recently.

Hacker-looking character sitting at a Mac in a dark room, checking out your microservices securityI covered a lot of ground in that talk, but something I didn’t get around to talking about was security. However, I believe that’s a really important topic to think about in microservice environments. It’s even more important than with a monolith, because in a service-oriented architecture you’re making a lot more of your system’s functionality directly exposed to the network, and that puts it in closer reach of would-be attackers, or “increases the attack surface” as a security pro would say.

So last week I presented another talk entitled “Microservices Security: All the Questions You Should Be Asking”.

Microservices Security: Let’s Share What We Know!

I want to tell people all about what we’ve been doing about security at Tyro lately. Security is incredibly important to the IT community and I think it’s imperative that we help each other improve. I want to share with the world some of the problems we’ve dealt with and some of the great solutions our team has built. Continue reading

Microservices at Tyro: An Evolutionary Tale (Presentation)

Featured

In February, I presented a talk at the Sydney Microservices Meetup titled “Microservices at Tyro: An Evolutionary Tale”.

Microservices at Tyro

I wanted to talk mostly about things we’ve been doing with microservices at Tyro Payments over the last year, but also about the almost 10 years of practice with distributed computing that has led us towards what we’re doing today.

I’ve merged my slides and the audio from the talk into a video, which you can watch below. If you’re more the reading type, there’s a transcript from the talk beneath the video. My talk goes for 40 minutes and then there’s 20 minutes of Q&A.

The talk covers:

  • Who is Tyro Payments?
  • Why are we doing Microservices?
  • Tyro’s Architecture History
  • Current development in Microservices
  • Tyro Microservices Practices
  • Asynchronous Communication Strategies
  • Helping Out Ops
  • Microservices Technologies and Patterns
  • Challenges we’ve been having at Tyro
  • Microservices pre-requisites

Continue reading

Notes from YOW! 2013: Michael T. Nygard on ‘Five Years of DevOps: Where are we Now?’

I attended Day 1 of YOW! Sydney 2013 and thought some people might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

Small child in a field looking into the distance with binoculars, as someone surveying the current state of DevOps probably wouldn't do.Michael T. Nygard (@mtnygard) is probably best known as the author of the 2007 book ‘Release It!‘, which teaches developers how to look beyond just getting their code working and instead design it from the outset to handle the harsh conditions of production environments. He has since become a DevOps luminary and now works at Cognitect. He spoke at YOW! 2013 about ‘Five Years of DevOps: Where are we Now?’.

Michael started off setting the timeline by pointing out that Chef and Puppet were preceded by CFEngine in about 1993!

He explained how his own experience has contributed to his DevOps insight: he worked as a Dev in Ops for some time, showing the Ops team how to solve some of their problems with Dev-like approaches, but also finding lots of problems with the way Devs created software, which was the chief inspiration for his book. Continue reading

Quick Response to “Test-induced Design Damage”

David Heinemeier Hansson wrote an interesting critique of Test Driven Design two days ago. I understand where he’s coming from with his concern that mindlessly shaping all design around tests can (will?) lead to poor design in some areas. It was interesting, and I like to be challenged and think about these things. Personally, I think his extrapolation to proposing “TDD is Dead” is hyperbole. Here are my thoughts on the bits of his blog that jumped out at me…

“But the testing pyramid prescribes that the unit level is where the focus should be, so people are sucked into that by default.”

It’s probably true that some people are sucked into always defaulting to the unit level. The problem with that is the “always”, but the unit level is where the focus should be. Why? Because that is where the complexity lives – in each unit. The reason we separate code into units is to separate complexity into simpler, composable parts. Testing exhaustively at non-unit levels (i.e. by integrating components) requires a combinatorial explosion of tests… or not testing exhaustively (might be a viable option at  your work, but not at a payments company). We use integration tests as well, but I don’t typically use them to prove all facets of functionality, instead only to prove the components integrate correctly and can deliver the core functions.

“Controllers are meant to be integration tested, not unit tested.”

I work in the Java world, not the Rails world, and I completely agree with this. When I started at Tyro, we were writing unit tests for controllers and integrated web tests for the presentation (i.e. JSPs)  of most features as well. The result? All controller code was tested twice. (It’s pretty hard to test a JSP without hitting its controller.) There’s a dirty word to describe this practice: waste. We ditched controller tests a long time ago now and no one has ever missed them. Controllers are still tested, just not by unit tests. From time to time, some logic in a controller might get a bit complex, resulting in more paths than are practical to web test. The solution is easy there: extract that complexity into another class, unit test the complexity, and web test the integration. Controllers should never be complex. A David points out, they are mostly just an adapter layer between models and views. Let any kind of business logic complexity live in there and it’s got two responsibilities. Your goal should be to make controllers so simple that they don’t need unit tests.

“Finally, the fear of letting model tests talk to the database is outdated”

Yes, and no. Yes, there should not be a fear of testing against the database. In fact, there should be a preference towards it. Not testing against real databases (the same ones you’ll use in Production) leaves a big fat layer of assumptions in your app. However, in large systems with lots of database interaction and great test coverage you can quickly max out the build time if you use the database everywhere. My thoughts: have a preference for DB-backed testing, but not at the expense of developer productivity. The team needs to be aware of when the preference is damaging their velocity and find the balance. I’ve written a lot of data-heavy, back-end Java, so mileage with Ruby on Rails may vary (maybe it never becomes a problem).

“Above all, you do not let your tests drive your design, you let your design drive your tests!”

I don’t think it’s this simple. One great advantage of using tests to drive design is that your desire for simple tests commutes to an implementation of simple classes. Yes, you can probably achieve similar outcomes by doing some more thinking or drawing, but test-first is also a useful tool for driving towards this goal. Not the only tool, or a required tool, but a useful tool. I write lots of simple code at home and rarely write tests for it, but recently I found one part I was writing was a bit gnarly so I decided to write a unit test for it. Upon starting to write the test, I found I wasn’t able to because of the design of the code, and when I thought about the design for a second I realised I was mashing several responsibilities into the one place. Tests can drive designs to bad places, but in my experience they more often drive them to good places.

I often re-iterate to my team that testing is primarily about building confidence in the code, and secondly about building a safety net for those who pass this way next. There is no mandate in TDD for writing tests that do not build confidence (do you TDD getter methods?), or for spending hours on tests that increase confidence by small fractions. I think David is right that there are probably zealots out there who are blindly going down these kinds of paths. Perhaps he’s right that some designs are bastardised by having stringent testability requirements, though I don’t recall seeing a lot of this in my travels. So, he’s right that mindlessly shaping all design around tests can have bad effects, but no software developer should be doing anything mindlessly. I think there is value in letting tests shape your design much of the time, so long as one keeps in mind that there will be occasions where the tests have to be shaped by the design instead.

Everything in moderation, right? Including moderation.

10 Reasons You Shouldn’t Have Senior Developers, Tech Leads or Architects

Two weeks ago I published a post titled ‘Why Smart Software Teams Don’t Need Senior Developers, Tech Leads or Architects‘. I received a lot of good feedback, but I also know it was a long read. So, if you’re interested by the title but are looking for a quick brain dump rather than an enjoyable read, here’s the abridged version:

At Tyro Payments, we’ve doubled our Engineering team over the last year.

We don’t hire for, or use, titles like Graduate Developer, Junior Developer, Senior Developer, Tech Lead or Architect. Everyone has the title ‘Software Engineer’.

This is an important part of Tyro’s Engineering team culture. Here are the reasons… Continue reading

Why Smart Software Teams Don’t Need Senior Developers, Tech Leads or Architects

Queue for Steve Jobs' keynote at WWDC 2010

A queue of software developers, not unlike the one that has inundated my inbox for the last year.

We’ve almost doubled our Engineering team at Tyro Payments over the last financial year and we’ll be adding that many again this year.

Most people who’ve worked in or with software teams would imagine that within this surge of hiring we’ve been filling all kinds of different roles – Graduate Developers, Junior Developers, Seniors, a couple of Tech Leads, maybe an Architect. But the truth is we’ve only been hiring for one role: Software Engineer. In fact, it’s the only development role on our team, and it’s the title we give to everyone on the tools, whether they have 20 years’ experience or none. This isn’t just some convenience we came up with to save ourselves HR work. It’s an incredibly important part of the culture at Tyro. Why? Continue reading