Microservices: The Benefits and Costs

'Mycenaean Bronze Scales' - a set of ancient, bronze scales. Any tech team should weigh microservices benefits against their costs before deciding whether to make the switch. Photographed by Mark Cartwright.

If you work in tech, you’ve almost definitely heard about microservices: the trendy style of software architecture where a system is split into multiple, independently-releasable services, that are modelled around business domains, and communicate via a network. Some people rave about all the amazing things they’ve achieved by using microservices’ benefits. Others rant about how much of their time it wastes and how much they hate it.

Like pretty much everything in tech, microservice architecture is a trade-off. Will it do great things for your organisation or not? That depends largely on how well set up you are to take advantage of the benefits and to absorb the costs.

Microservice Architecture is a trade-off. Whether it's going to do great things for your organisation depends on how well set up you are to take advantage of the benefits and absorb the costs. Read more: Microservices: The Benefits and Costs Click To Tweet

If you’re looking to make a decision about whether to use microservices in your team (or reverse one!), here’s a list of many of the pros and cons which you’ll want to consider.

Continue reading

Using Microservices to Make Costly Deployments Less Frequent

This article on costly deployments is part of a series on How to Choose Your First Microservices.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Tricky Deployments

A fighter jet on an aircraft carrier being signalled to launch by two ground staff. Launching a fighter jet is a costly deployment, just as deploying a monolith may be.

Some components of your system will be tricker to deploy than others. This is true regardless of whether you’ve already got microservices or you’re deploying a monolith. Some components might be in use by 1000s of customers 24×7, while others might only be used once a month. Upgrading a component might require other components to be placed into a certain state, while others can be upgraded independently of the rest of the system. Coordination may be required between multiple teams in order to effect a trouble-free deploy of some components, whereas others might be actioned by a single person.

Continue reading

Using Microservices to Solve Developers Stepping on Each Other’s Toes

This article on reducing merge conflicts is part of a series on How to Choose Your First Microservices.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Is your team wasting a lot of time working around each other?

A side-collision car crash at an intersection, surrounded by emergency personnel. Like this, merge conflicts can be messy and dangerous.

If you have a large team of developers working in a single code base, it’s inevitable that the work of some team members will interfere with the work of others. Many of us are now using tools (like git) and processes (like continuous integration) that can lessen the impact of interference. (Although other modern practices like feature branching can make conflicts worse.)

Continue reading

Using Microservices to Solve Slow Build Times

This article on slow build times is part of a series on How to Choose Your First Microservices.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Slow Build Times?

A snail crossing a road, which might happen faster your slow build time.

Does your team have an application that takes a long time to build and run tests? Is a large portion of the development team working on it? That could be a huge amount of idle time you’re pouring down the drain. The classic XKCD ‘Compiling’ comic makes light of developers enjoying the time they spend waiting for their code to build. The truth for most devs is they would far prefer to be making fast progress on their work than sitting around waiting.

Continue reading

How to Choose Your First Microservices

Are you early in your microservices journey? Maybe you’ve decided you need to start deploying applications outside your monolith but you haven’t cut any code yet. Or maybe you’ve put your first few services into production and addressed some of the first pains that happen when you start on that path.

A large number of interacting LEGO cogs making one large machine, similar to a monolith from which you want to break out your first microservices

A common question that comes up for teams at around this time is:

“What should we split out into a microservice first?

And why?”

Continue reading

Time to Evolve Again

My mid-2000s, fresh-faced, professional look from around the time I joined Tyro.

After 12.3 years, yesterday was my last day at @Tyro. 😢

It’s been a huge journey, from a perilous payments startup with less than 20 employees and only ☝🏻 customer, to now a VC-backed, fully-licensed bank with well over 400 staff that is Australia’s 5th largest EFTPOS provider (by transaction volume).

I’ve done time as a Software Engineer, Development Lead, Engineering Lead (part of the management team), Technical Project Manager, Team Lead, Product Lead, and this last year, back to Software Engineer. I must have hired 50+ (awesome) people and interviewed many times that.

Continue reading

Memories from 10 years at a startup

Black and white photo of four kids on an outdoor cubby houseI recently notched up ten years of working at Tyro Payments, a tiny little payments startup that I joined in late 2006 when it had about 20 people, and which is now Australia’s most successful fintech venture, Australia’s newest bank, and is fast approaching 400 staff.

Here’s a collection of some of my most vivid recollections from so far in this fantastic journey.

Getting Hired

I remember… when the recruiter for Tyro said he had a finance role, and I reiterated to him that I wasn’t going to take any job in finance. He back-pedalled, convinced me it wasn’t really a finance company and I really had to go meet them. I was reluctant, but I went along because I wanted to be in his good books.

I remember… the hook for me was one of the founders – Peter Haig – drawing the architecture they were building and me thinking 1) this is very cool and 2) this guy is very smart.

I remember… I plucked up the courage to ask where the company got its money from after three years of runway with no customers, and the founder was completely unfazed that a software engineer would ask such a question.

I remember… asking whether they really wrote tests before making any changes to the system, even to web pages, and Matt Milliss saying, “To be honest, I can’t believe how much code I wrote without tests before I came here, and how many bugs I must have created that no one has found yet.” It was a statement that shook my confidence in myself as an engineer and made me want to learn from this mature group of people that seemed to have a bunch of things figured out that most other software teams were still battling with.

I remember… I was so concerned about whether or not I’d enjoy pair programming that I asked for extra time to think about the offer.

I remember… discussing with my wife what would happen if I joined a startup and it went under and how long we thought we could survive on her salary if it took a while for me to find another job.

Startup Land

Hikers starting up a long steep climbI remember… on my first day I spent the morning pairing with Mark, who patiently spent the whole time telling me about the architecture of the system. We swapped pairs at lunchtime and I stayed on the task. My new pair, Geoff, asked me what we’d done in the morning and I said, “Nothing, we just talked about the system.” He looked at me strange, shook his head, and said “Right”. Then he changed a test, I changed the implementation to make it pass, and within half an hour we’d checked my first change into the mainline.

I remember… for the first six months having to explain to people that I worked at a company called MoneySwitch, watching their confused and concerned face, and preparing to explain that it wasn’t a money-laundering operation.

I remember… in the first office I worked in, we had no dishwasher and everyone was on a washing up roster, including the CEO.

I remember… that first office got really hot and the building aircon sucked. We had a single portable air conditioner… which was dedicated to the server room. On hot days we’d open a few windows and try to create a cross-breeze with a fan or two, but mostly we’d just sweat it out and keep going, or take a break for slurpees.

I remember… the whole Dev team – all 12 of us – used to go out to the mall to eat lunch together almost every day. There was no room to eat lunch in the office.

I remember… that the fortnightly demo involved the whole company – all 20 of us – cramming into the 3m x 10m board room and sharing a single case of beer.

I remember… when we moved to occupy two levels of 125 York St (now home to Blue Chilli) and it felt like we’d been upgraded to The Ritz.

I remember… when, less than 6 months in, my team leader proactively had my salary adjusted up because she thought Tyro had unwittingly hired me cheap based on my experience. It was an eye-opening example of how serious Tyro was about living out the company’s values.

Engineering Feats and Follies

Sydney Harbour Bridge being constructedI remember… when Ricky Yim and I got to do a spike of making a POS and an EFTPOS terminal integrate by communicating over the internet and through our system instead of over a serial cable and we got it working in a couple of days – probably Australia’s first “cloud” POS integration.

I remember… we discovered two years later that a demultiplexer in our POS integration was using a single thread to process all requests from thousands of clients and had only just started to struggle with that load. We bought ourselves about another decade of scaling by changing a config file.

I remember… I “refactored” an API between two systems that had some poorly-named fields, and a week later we mysteriously stopped paying merchants for the cashout they’d given customers. And we only knew about it when they called to complain. No one ever asked who’d made this bug happen, and instead the team had a conversation about how to stop it from happening again.

I remember… us being (equal) first to be accepted to be an acquirer for Medicare EasyClaim online claiming, and the excitement and enthusiasm of being part of building this huge new thing that was going to help improve the country’s public health system. Today, Tyro processes more than half of all the claims on the EasyClaim system, despite four other major banks offering EasyClaim services.

I remember… a litany of small tools that I built at home and brought to work: Canopy, the Maven multi-project build UI; Mini-Rest, a tiny Servlet library that forced people to use proper REST semantics; SodaTest, a Scala-based integration testing framework intended as a replacement for FIT; “Who’s the Expert?”, a smart contributor scoreboard for Bitbucket Server; and probably heaps of others I’ve forgotten.

I remember… relishing the opportunity to be part of rebuilding the accounting system from scratch to fix its performance and maintainability issues. It’s still humming along half a decade later!

I remember… half-jokingly whinging to my boss about how it was coming up to 12 months since I’d been working on nothing but the accounting system, and I might have to find another job if I was going to be doing nothing but accounting for the rest of my life. He moved me off that piece of work a week before 12 months.

Levelling Up

Red and white mushroom, like one used in the Mario game for a power upI remember… when I and three others were pulled aside to a corner of the office (we didn’t have any meeting rooms) to meet Mike Cannon-Brookes. He was considering joining our board but he wanted to meet (and grill!) some of the senior Engineering staff first so that he could check whether we were as good as our management team said we were. He joined, so I guess we did okay.

I remember… asking if I could take on a few more responsibilities because I was bored and, instead of being given an extra job or two, I was told I’d be made an Engineering Lead.

I remember… first getting involved in interviewing, not having any idea what I was doing, and feeling the weight of making decisions that would significantly affect other people’s lives and the future of the company.

I remember… we used to ask people to write a function on a whiteboard to calculate factorials and it took us way too long to realise it wasn’t testing people’s coding skills at all and that just getting people to do some pairing was way better.

A chart of the size of the Engineering team at Tyro showing exponential growth from 2013 to 2016I remember… we hired so many people that we doubled the size of the team in 18 months. And then we did it again. And then we did it again.

I remember… that I was just coping to remember all the new people’s names, then I went on holiday for a week, missed four new starters, and never recovered.

I remember… when the Engineering team eventually grew so big that we reluctantly moved Ops to a different floor of the building, and communication between the two groups plummeted.

I remember… pairing with Ops on mystery catastrophe nights, all scratching our heads, ordering pizza when we realised it was way past dinner time, and that “Aha!” feeling when someone found an entry in a log or ran a query on a table that had seemed totally unrelated to the symptoms but showed something that no one expected could ever happen.

I remember… organising our first innovation sprint – ‘CodeBlitz’ – and being surprised by the different mood in the room as people were intensely focused on getting their own innovative idea hacked together within two days so they could demo it.

I remember… when we decided it would be a good idea to build our next link to another bank as its own piece of software rather than adding it to an existing monolith – one of our first steps toward a microservices architecture (before that word existed). We recently celebrated having over 100 services deployed in Production.

The Journey to the Bank

Four people in suits walking in front of a building with tall, thick columns, possibly a bank or museum, with long shadows in the early morning light.I remember… that time my boss came to me and asked if my team could start on building a bank the next week.

I remember… when we took a completely agile approach to building a bank…

“Where do we start?”
“Well, what’s the most important thing a bank does?”
“Keeps people’s money in an account.”
“Okay, let’s write a piece of software that keeps money in an account instead of sending it to another bank.”
And so we wrote that and got it working before we did anything else.

I remember… when the regulator announced they were retiring our limited EFTPOS banking license, and we were told the banking side-project was now the rush to a full banking license mission, the #1 priority of the company, and that we had 18 months to deliver.

I remember… when we were 12 months into building our core banking system, just weeks away from delivering it exactly when we had estimated… and someone uncovered 3 more months of mandatory work.

I remember… when we received the results of an external security audit of our in-progress core banking system and the synopsis was “You guys are pretty up there with your security, but here’s 15 other major things you should think about improving.”

I remember… the frankly unbelievable news that we’d been granted a full banking license by APRA. It felt like a war was over.

I remember… when we’d just finished turning on the core banking system and we were told, “Well done. Now you’ve got 12 months to build a lending product from scratch, ship it, and get customers onto it.” We did it in 6.

Time Keeps on Slipping

Public binoculars with a small lake just beyond and a foggy, blurred landscape in the background.I remember… feeling like my contribution to the company was having less and less impact due to the growth and organisational changes, and starting to seriously think about moving on to somewhere else. Eventually, I realised that I didn’t have to be bound by my historical contribution to Engineering, and that Product Management might be a place where I could probably develop a new expertise and have a large impact on the company’s future.

I remember… getting up the courage to ask the co-founder and Head of Product about moving into a Product Management role, expecting to get a grilling about what product was and what skills I had that I thought would be relevant. Instead his response was just, “Yeah, that makes sense. When would you like to move?”

I asked my wife what she remembers, and she said she remembers when she would show up to the Christmas party and know everyone’s name, but nowadays I sometimes tell her I’m going to someone’s leaving drinks and she says she’s never heard of them.

I remember… starting to reflect on what I’d been part of in 10 years at Tyro and realising there was almost nothing that I’d achieved by myself; everything significant I’d been involved in had been a team effort – sometimes a team of two, sometimes over a hundred, but almost always a team. That’s how we roll.

Image credits:
Kids, older photo‘ by Joe Crawford
Start of the Steep Trail‘ by Dan Cook
Sydney Harbour Bridge‘ by gramarye
Power ups‘ by Kirt Edblom
Bankers‘ by Chris Brown
Eye of the Beholder‘ by hjl (flickr)

Distributed Transactions: The Icebergs of Microservices

An antarctic iceberg which, much like distributed transactions in microservices, can be hard to see and can wreck your ship.Why are distributed transactions icebergs? It’s not because they’re cool and beautiful and you have to look under the surface to comprehend them.

Distributed transactions are icebergs because (1) it’s easy to not see them, even when they’re right in front of you, and (2) if you run into one, it’s got a great potential to sink your ship. Continue reading

Why “Don’t Use Shared Libraries in Microservices” is Bad Advice

A man licking an ice cream while a dog tries to lick it as well. Reminiscent of the kind of undesirable coupling created by shared libraries in microservices

Sharing isn’t always a good idea.

If you’ve read a bit about microservices, you’ll probably have come across the mantra, “Don’t use shared libraries in microservices.” This is bad advice.

While the sentiment is borne from real issues and there’s real wisdom to be gained in this area, this little statement is too pithy, lacking the relevant context to make it useful. Consequently, it’s open to misinterpretation, in particular to being applied too liberally, and I’ve seen it misunderstood a number of times recently.

What’s the Context for Understanding Shared Libraries in Microservices?

Only recently, I’ve picked up that different people mean different things when they talk about using shared libraries. Continue reading

Microservices Security: All The Questions You Should Be Asking

I spoke earlier in the year at the Sydney Microservices Meetup about the long path we’ve taken at Tyro Payments over the last decade, gradually tending towards a more fine-grained SOA approach – microservices as it’s come to be known recently.

Hacker-looking character sitting at a Mac in a dark room, checking out your microservices securityI covered a lot of ground in that talk, but something I didn’t get around to talking about was security. However, I believe that’s a really important topic to think about in microservice environments. It’s even more important than with a monolith, because in a service-oriented architecture you’re making a lot more of your system’s functionality directly exposed to the network, and that puts it in closer reach of would-be attackers, or “increases the attack surface” as a security pro would say.

So last week I presented another talk entitled “Microservices Security: All the Questions You Should Be Asking”.

Microservices Security: Let’s Share What We Know!

I want to tell people all about what we’ve been doing about security at Tyro lately. Security is incredibly important to the IT community and I think it’s imperative that we help each other improve. I want to share with the world some of the problems we’ve dealt with and some of the great solutions our team has built. Continue reading