Using Microservices to Make Costly Deployments Less Frequent

This article on costly deployments is part of a series on How to Choose Your First Microservices.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Tricky Deployments

A fighter jet on an aircraft carrier being signalled to launch by two ground staff. Launching a fighter jet is a costly deployment, just as deploying a monolith may be.

Some components of your system will be tricker to deploy than others. This is true regardless of whether you’ve already got microservices or you’re deploying a monolith. Some components might be in use by 1000s of customers 24×7, while others might only be used once a month. Upgrading a component might require other components to be placed into a certain state, while others can be upgraded independently of the rest of the system. Coordination may be required between multiple teams in order to effect a trouble-free deploy of some components, whereas others might be actioned by a single person.

Continue reading

Using Microservices to Solve Developers Stepping on Each Other’s Toes

This article on reducing merge conflicts is part of a series on How to Choose Your First Microservices.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Is your team wasting a lot of time working around each other?

A side-collision car crash at an intersection, surrounded by emergency personnel. Like this, merge conflicts can be messy and dangerous.

If you have a large team of developers working in a single code base, it’s inevitable that the work of some team members will interfere with the work of others. Many of us are now using tools (like git) and processes (like continuous integration) that can lessen the impact of interference. (Although other modern practices like feature branching can make conflicts worse.)

Continue reading

Using Microservices to Solve Slow Build Times

This article on slow build times is part of a series on How to Choose Your First Microservices.

Switching to a new software architecture style is costly. I advocate choosing where to employ microservices first by using them to solve existing problems in the team. This helps the organisation extract value from the new architecture with each step towards the new world. This series is working through a set of problems that your org may have, and how microservices might help solve them.

Slow Build Times?

A snail crossing a road, which might happen faster your slow build time.

Does your team have an application that takes a long time to build and run tests? Is a large portion of the development team working on it? That could be a huge amount of idle time you’re pouring down the drain. The classic XKCD ‘Compiling’ comic makes light of developers enjoying the time they spend waiting for their code to build. The truth for most devs is they would far prefer to be making fast progress on their work than sitting around waiting.

Continue reading

How to Choose Your First Microservices

Featured

Are you early in your microservices journey? Maybe you’ve decided you need to start deploying applications outside your monolith but you haven’t cut any code yet. Or maybe you’ve put your first few services into production and addressed some of the first pains that happen when you start on that path.

A large number of interacting LEGO cogs making one large machine, similar to a monolith from which you want to break out your first microservices

A common question that comes up for teams at around this time is:

“What should we split out into a microservice first?

And why?”

Continue reading

Feature Flags and Test-Driven Design: Some Practical Tips

A country road with a fork going off to one side, symbolic of a feature flag in codeIn 2018, our team spent a lot of time working with feature flags and test-driven design (TDD).

Our project was to effect an architectural change to our system, changing the source of truth of some data, moving it out of the database owned by a legacy monolith into a new database controlled by a new microservice. However, much of the code requiring the data would remain in the monolith.*

Some examples of the types of things we feature-flagged are:

    • whether to go down a refactored code path or not;
    • whether to publish messages to a message queue when a certain event occurred;
    • how to publish those messages (we tried multiple variations of batching and transaction boundaries to achieve acceptable performance);
    • whether to just delete messages at the receiving end or actually handle them; and
    • whether to use a local source of data or remote one.

We were working on a pretty important piece of code; the kind of business function where, if we stuffed it up, someone would probably have to spend several days doing remedial fixups or making phone calls to chase up millions of dollars. Continue reading

Time to Evolve Again

My mid-2000s, fresh-faced, professional look from around the time I joined Tyro.

After 12.3 years, yesterday was my last day at @Tyro. 😢

It’s been a huge journey, from a perilous payments startup with less than 20 employees and only ☝🏻 customer, to now a VC-backed, fully-licensed bank with well over 400 staff that is Australia’s 5th largest EFTPOS provider (by transaction volume).

I’ve done time as a Software Engineer, Development Lead, Engineering Lead (part of the management team), Technical Project Manager, Team Lead, Product Lead, and this last year, back to Software Engineer. I must have hired 50+ (awesome) people and interviewed many times that.

Continue reading

“This work kinda sucks right now”: Staying motivated in the uninspiring phases of long projects

In April 2015, I was the technical project manager of six software teams all working on one gigantic project. It was the biggest challenge the company had ever taken on. The goal: create and stand up a transaction account product for small and medium business so that we could apply for an unrestricted bank license.

Tired marathon runner with hands on hips

We were 12 months in and were about to finish the project bang on the week we’d estimated, when the inevitable happened: we received a new raft of requirements that would add at least another 3 months of work. The extra work wasn’t glamorous; it was mostly security enhancements, disaster recovery preparation, and a complicated behind-the-scenes tweak of the product that added little benefit for users but was a requirement from a regulatory point of view.

Understandably, morale took a hit, but we needed to keep the pace up or we’d risk missing the deadline we’d been set for acquiring our license. After an interesting discussion with my father-in-law about the project, I found a new perspective on where we were at, and it prompted me to write the internal blog below to help motivate the team towards our final goal. Continue reading

9 fascinating things I learned while coding up the rules of a board game

I recently decided I was going to take the rules of the board game Forbidden Island and write them up as code.

A screen full of code in an IDEI guess that sounds like a weird thing to just decide to do, doesn’t it? It’s actually one part of a bigger goal I have at the moment of teaching myself some practical machine learning. As part of this journey, I heard a great idea from YouTuber Jabrils to set yourself a significant challenge that you’re interested in, and to work towards surmounting that challenge. For Jabrils, his challenge was getting an AI to control a Forrest Gump character to run around a course in a game. For my challenge, I’ve decided I’d like to build an AI that can play Forbidden Island. (And win!)

Obviously, if you’re going to have an AI play a game, you first need a digital version of the game for the AI to play. Continue reading

The Amazing Benefits of Exercism

A couple of months ago, a good friend at work (@rodeoclownII) sent out an email (or blog, or Slack message, or something) about exercism.io. It’s a cool little website for practising programming in various languages (currently about 45) by implementing koans, or small exercise problems.

I’ve been working through the Kotlin problems since the start of the year and I’ve found it super useful, so I thought I’d write this quick blog to recommend it to others.

Why I like exercism.io so much

Actor Anthony Hopkins seemingly performing an exorcism with the words "Back to hell, demon!" Probably unrelated to exercism.io

Just to be clear, we’re not talking about this kind of exorcism.

The most obvious benefit of this site is the opportunity to practise using different elements of the Kotlin language and standard library. Being challenged to solve problems which you might not necessarily come across during day-to-day work can lead you to discover parts of Kotlin that are really useful but which you otherwise might not have found a need to go looking for. Also, the problems are all implemented in a test-driven way, which feels very natural for me as a long-time practitioner of Extreme Programming/TDD. Continue reading

Memories from 10 years at a startup

Black and white photo of four kids on an outdoor cubby houseI recently notched up ten years of working at Tyro Payments, a tiny little payments startup that I joined in late 2006 when it had about 20 people, and which is now Australia’s most successful fintech venture, Australia’s newest bank, and is fast approaching 400 staff.

Here’s a collection of some of my most vivid recollections from so far in this fantastic journey.

Getting Hired

I remember… when the recruiter for Tyro said he had a finance role, and I reiterated to him that I wasn’t going to take any job in finance. He back-pedalled, convinced me it wasn’t really a finance company and I really had to go meet them. I was reluctant, but I went along because I wanted to be in his good books.

I remember… the hook for me was one of the founders – Peter Haig – drawing the architecture they were building and me thinking 1) this is very cool and 2) this guy is very smart.

I remember… I plucked up the courage to ask where the company got its money from after three years of runway with no customers, and the founder was completely unfazed that a software engineer would ask such a question.

I remember… asking whether they really wrote tests before making any changes to the system, even to web pages, and Matt Milliss saying, “To be honest, I can’t believe how much code I wrote without tests before I came here, and how many bugs I must have created that no one has found yet.” It was a statement that shook my confidence in myself as an engineer and made me want to learn from this mature group of people that seemed to have a bunch of things figured out that most other software teams were still battling with.

I remember… I was so concerned about whether or not I’d enjoy pair programming that I asked for extra time to think about the offer.

I remember… discussing with my wife what would happen if I joined a startup and it went under and how long we thought we could survive on her salary if it took a while for me to find another job.

Startup Land

Hikers starting up a long steep climbI remember… on my first day I spent the morning pairing with Mark, who patiently spent the whole time telling me about the architecture of the system. We swapped pairs at lunchtime and I stayed on the task. My new pair, Geoff, asked me what we’d done in the morning and I said, “Nothing, we just talked about the system.” He looked at me strange, shook his head, and said “Right”. Then he changed a test, I changed the implementation to make it pass, and within half an hour we’d checked my first change into the mainline.

I remember… for the first six months having to explain to people that I worked at a company called MoneySwitch, watching their confused and concerned face, and preparing to explain that it wasn’t a money-laundering operation.

I remember… in the first office I worked in, we had no dishwasher and everyone was on a washing up roster, including the CEO.

I remember… that first office got really hot and the building aircon sucked. We had a single portable air conditioner… which was dedicated to the server room. On hot days we’d open a few windows and try to create a cross-breeze with a fan or two, but mostly we’d just sweat it out and keep going, or take a break for slurpees.

I remember… the whole Dev team – all 12 of us – used to go out to the mall to eat lunch together almost every day. There was no room to eat lunch in the office.

I remember… that the fortnightly demo involved the whole company – all 20 of us – cramming into the 3m x 10m board room and sharing a single case of beer.

I remember… when we moved to occupy two levels of 125 York St (now home to Blue Chilli) and it felt like we’d been upgraded to The Ritz.

I remember… when, less than 6 months in, my team leader proactively had my salary adjusted up because she thought Tyro had unwittingly hired me cheap based on my experience. It was an eye-opening example of how serious Tyro was about living out the company’s values.

Engineering Feats and Follies

Sydney Harbour Bridge being constructedI remember… when Ricky Yim and I got to do a spike of making a POS and an EFTPOS terminal integrate by communicating over the internet and through our system instead of over a serial cable and we got it working in a couple of days – probably Australia’s first “cloud” POS integration.

I remember… we discovered two years later that a demultiplexer in our POS integration was using a single thread to process all requests from thousands of clients and had only just started to struggle with that load. We bought ourselves about another decade of scaling by changing a config file.

I remember… I “refactored” an API between two systems that had some poorly-named fields, and a week later we mysteriously stopped paying merchants for the cashout they’d given customers. And we only knew about it when they called to complain. No one ever asked who’d made this bug happen, and instead the team had a conversation about how to stop it from happening again.

I remember… us being (equal) first to be accepted to be an acquirer for Medicare EasyClaim online claiming, and the excitement and enthusiasm of being part of building this huge new thing that was going to help improve the country’s public health system. Today, Tyro processes more than half of all the claims on the EasyClaim system, despite four other major banks offering EasyClaim services.

I remember… a litany of small tools that I built at home and brought to work: Canopy, the Maven multi-project build UI; Mini-Rest, a tiny Servlet library that forced people to use proper REST semantics; SodaTest, a Scala-based integration testing framework intended as a replacement for FIT; “Who’s the Expert?”, a smart contributor scoreboard for Bitbucket Server; and probably heaps of others I’ve forgotten.

I remember… relishing the opportunity to be part of rebuilding the accounting system from scratch to fix its performance and maintainability issues. It’s still humming along half a decade later!

I remember… half-jokingly whinging to my boss about how it was coming up to 12 months since I’d been working on nothing but the accounting system, and I might have to find another job if I was going to be doing nothing but accounting for the rest of my life. He moved me off that piece of work a week before 12 months.

Levelling Up

Red and white mushroom, like one used in the Mario game for a power upI remember… when I and three others were pulled aside to a corner of the office (we didn’t have any meeting rooms) to meet Mike Cannon-Brookes. He was considering joining our board but he wanted to meet (and grill!) some of the senior Engineering staff first so that he could check whether we were as good as our management team said we were. He joined, so I guess we did okay.

I remember… asking if I could take on a few more responsibilities because I was bored and, instead of being given an extra job or two, I was told I’d be made an Engineering Lead.

I remember… first getting involved in interviewing, not having any idea what I was doing, and feeling the weight of making decisions that would significantly affect other people’s lives and the future of the company.

I remember… we used to ask people to write a function on a whiteboard to calculate factorials and it took us way too long to realise it wasn’t testing people’s coding skills at all and that just getting people to do some pairing was way better.

A chart of the size of the Engineering team at Tyro showing exponential growth from 2013 to 2016I remember… we hired so many people that we doubled the size of the team in 18 months. And then we did it again. And then we did it again.

I remember… that I was just coping to remember all the new people’s names, then I went on holiday for a week, missed four new starters, and never recovered.

I remember… when the Engineering team eventually grew so big that we reluctantly moved Ops to a different floor of the building, and communication between the two groups plummeted.

I remember… pairing with Ops on mystery catastrophe nights, all scratching our heads, ordering pizza when we realised it was way past dinner time, and that “Aha!” feeling when someone found an entry in a log or ran a query on a table that had seemed totally unrelated to the symptoms but showed something that no one expected could ever happen.

I remember… organising our first innovation sprint – ‘CodeBlitz’ – and being surprised by the different mood in the room as people were intensely focused on getting their own innovative idea hacked together within two days so they could demo it.

I remember… when we decided it would be a good idea to build our next link to another bank as its own piece of software rather than adding it to an existing monolith – one of our first steps toward a microservices architecture (before that word existed). We recently celebrated having over 100 services deployed in Production.

The Journey to the Bank

Four people in suits walking in front of a building with tall, thick columns, possibly a bank or museum, with long shadows in the early morning light.I remember… that time my boss came to me and asked if my team could start on building a bank the next week.

I remember… when we took a completely agile approach to building a bank…

“Where do we start?”
“Well, what’s the most important thing a bank does?”
“Keeps people’s money in an account.”
“Okay, let’s write a piece of software that keeps money in an account instead of sending it to another bank.”
And so we wrote that and got it working before we did anything else.

I remember… when the regulator announced they were retiring our limited EFTPOS banking license, and we were told the banking side-project was now the rush to a full banking license mission, the #1 priority of the company, and that we had 18 months to deliver.

I remember… when we were 12 months into building our core banking system, just weeks away from delivering it exactly when we had estimated… and someone uncovered 3 more months of mandatory work.

I remember… when we received the results of an external security audit of our in-progress core banking system and the synopsis was “You guys are pretty up there with your security, but here’s 15 other major things you should think about improving.”

I remember… the frankly unbelievable news that we’d been granted a full banking license by APRA. It felt like a war was over.

I remember… when we’d just finished turning on the core banking system and we were told, “Well done. Now you’ve got 12 months to build a lending product from scratch, ship it, and get customers onto it.” We did it in 6.

Time Keeps on Slipping

Public binoculars with a small lake just beyond and a foggy, blurred landscape in the background.I remember… feeling like my contribution to the company was having less and less impact due to the growth and organisational changes, and starting to seriously think about moving on to somewhere else. Eventually, I realised that I didn’t have to be bound by my historical contribution to Engineering, and that Product Management might be a place where I could probably develop a new expertise and have a large impact on the company’s future.

I remember… getting up the courage to ask the co-founder and Head of Product about moving into a Product Management role, expecting to get a grilling about what product was and what skills I had that I thought would be relevant. Instead his response was just, “Yeah, that makes sense. When would you like to move?”

I asked my wife what she remembers, and she said she remembers when she would show up to the Christmas party and know everyone’s name, but nowadays I sometimes tell her I’m going to someone’s leaving drinks and she says she’s never heard of them.

I remember… starting to reflect on what I’d been part of in 10 years at Tyro and realising there was almost nothing that I’d achieved by myself; everything significant I’d been involved in had been a team effort – sometimes a team of two, sometimes over a hundred, but almost always a team. That’s how we roll.

Image credits:
Kids, older photo‘ by Joe Crawford
Start of the Steep Trail‘ by Dan Cook
Sydney Harbour Bridge‘ by gramarye
Power ups‘ by Kirt Edblom
Bankers‘ by Chris Brown
Eye of the Beholder‘ by hjl (flickr)