Microservices Security: All The Questions You Should Be Asking

I spoke earlier in the year at the Sydney Microservices Meetup about the long path we’ve taken at Tyro Payments over the last decade, gradually tending towards a more fine-grained SOA approach – microservices as it’s come to be known recently.

Hacker-looking character sitting at a Mac in a dark room, checking out your microservices securityI covered a lot of ground in that talk, but something I didn’t get around to talking about was security. However, I believe that’s a really important topic to think about in microservice environments. It’s even more important than with a monolith, because in a service-oriented architecture you’re making a lot more of your system’s functionality directly exposed to the network, and that puts it in closer reach of would-be attackers, or “increases the attack surface” as a security pro would say.

So last week I presented another talk entitled “Microservices Security: All the Questions You Should Be Asking”.

Microservices Security: Let’s Share What We Know!

I want to tell people all about what we’ve been doing about security at Tyro lately. Security is incredibly important to the IT community and I think it’s imperative that we help each other improve. I want to share with the world some of the problems we’ve dealt with and some of the great solutions our team has built. Continue reading

Microservices at Tyro: An Evolutionary Tale (Presentation)

In February, I presented a talk at the Sydney Microservices Meetup titled “Microservices at Tyro: An Evolutionary Tale”.

Microservices at Tyro

I wanted to talk mostly about things we’ve been doing with microservices at Tyro Payments over the last year, but also about the almost 10 years of practice with distributed computing that has led us towards what we’re doing today.

I’ve merged my slides and the audio from the talk into a video, which you can watch below. If you’re more the reading type, there’s a transcript from the talk beneath the video. My talk goes for 40 minutes and then there’s 20 minutes of Q&A.

The talk covers:

  • Who is Tyro Payments?
  • Why are we doing Microservices?
  • Tyro’s Architecture History
  • Current development in Microservices
  • Tyro Microservices Practices
  • Asynchronous Communication Strategies
  • Helping Out Ops
  • Microservices Technologies and Patterns
  • Challenges we’ve been having at Tyro
  • Microservices pre-requisites

Continue reading

Notes from YOW! 2013: Michael T. Nygard on ‘Five Years of DevOps: Where are we Now?’

I attended Day 1 of YOW! Sydney 2013 and thought some people might get something useful out of my notes. These aren’t my complete reinterpretations of every slide, but just things I jotted down that I thought were interesting enough to remember or look into further.

Small child in a field looking into the distance with binoculars, as someone surveying the current state of DevOps probably wouldn't do.Michael T. Nygard (@mtnygard) is probably best known as the author of the 2007 book ‘Release It!‘, which teaches developers how to look beyond just getting their code working and instead design it from the outset to handle the harsh conditions of production environments. He has since become a DevOps luminary and now works at Cognitect. He spoke at YOW! 2013 about ‘Five Years of DevOps: Where are we Now?’.

Michael started off setting the timeline by pointing out that Chef and Puppet were preceded by CFEngine in about 1993!

He explained how his own experience has contributed to his DevOps insight: he worked as a Dev in Ops for some time, showing the Ops team how to solve some of their problems with Dev-like approaches, but also finding lots of problems with the way Devs created software, which was the chief inspiration for his book. Continue reading

Quick Response to “Test-induced Design Damage”

David Heinemeier Hansson wrote an interesting critique of Test Driven Design two days ago. I understand where he’s coming from with his concern that mindlessly shaping all design around tests can (will?) lead to poor design in some areas. It was interesting, and I like to be challenged and think about these things. Personally, I think his extrapolation to proposing “TDD is Dead” is hyperbole. Here are my thoughts on the bits of his blog that jumped out at me…

“But the testing pyramid prescribes that the unit level is where the focus should be, so people are sucked into that by default.”

It’s probably true that some people are sucked into always defaulting to the unit level. The problem with that is the “always”, but the unit level is where the focus should be. Why? Because that is where the complexity lives – in each unit. The reason we separate code into units is to separate complexity into simpler, composable parts. Testing exhaustively at non-unit levels (i.e. by integrating components) requires a combinatorial explosion of tests… or not testing exhaustively (might be a viable option at  your work, but not at a payments company). We use integration tests as well, but I don’t typically use them to prove all facets of functionality, instead only to prove the components integrate correctly and can deliver the core functions.

“Controllers are meant to be integration tested, not unit tested.”

I work in the Java world, not the Rails world, and I completely agree with this. When I started at Tyro, we were writing unit tests for controllers and integrated web tests for the presentation (i.e. JSPs)  of most features as well. The result? All controller code was tested twice. (It’s pretty hard to test a JSP without hitting its controller.) There’s a dirty word to describe this practice: waste. We ditched controller tests a long time ago now and no one has ever missed them. Controllers are still tested, just not by unit tests. From time to time, some logic in a controller might get a bit complex, resulting in more paths than are practical to web test. The solution is easy there: extract that complexity into another class, unit test the complexity, and web test the integration. Controllers should never be complex. A David points out, they are mostly just an adapter layer between models and views. Let any kind of business logic complexity live in there and it’s got two responsibilities. Your goal should be to make controllers so simple that they don’t need unit tests.

“Finally, the fear of letting model tests talk to the database is outdated”

Yes, and no. Yes, there should not be a fear of testing against the database. In fact, there should be a preference towards it. Not testing against real databases (the same ones you’ll use in Production) leaves a big fat layer of assumptions in your app. However, in large systems with lots of database interaction and great test coverage you can quickly max out the build time if you use the database everywhere. My thoughts: have a preference for DB-backed testing, but not at the expense of developer productivity. The team needs to be aware of when the preference is damaging their velocity and find the balance. I’ve written a lot of data-heavy, back-end Java, so mileage with Ruby on Rails may vary (maybe it never becomes a problem).

“Above all, you do not let your tests drive your design, you let your design drive your tests!”

I don’t think it’s this simple. One great advantage of using tests to drive design is that your desire for simple tests commutes to an implementation of simple classes. Yes, you can probably achieve similar outcomes by doing some more thinking or drawing, but test-first is also a useful tool for driving towards this goal. Not the only tool, or a required tool, but a useful tool. I write lots of simple code at home and rarely write tests for it, but recently I found one part I was writing was a bit gnarly so I decided to write a unit test for it. Upon starting to write the test, I found I wasn’t able to because of the design of the code, and when I thought about the design for a second I realised I was mashing several responsibilities into the one place. Tests can drive designs to bad places, but in my experience they more often drive them to good places.

I often re-iterate to my team that testing is primarily about building confidence in the code, and secondly about building a safety net for those who pass this way next. There is no mandate in TDD for writing tests that do not build confidence (do you TDD getter methods?), or for spending hours on tests that increase confidence by small fractions. I think David is right that there are probably zealots out there who are blindly going down these kinds of paths. Perhaps he’s right that some designs are bastardised by having stringent testability requirements, though I don’t recall seeing a lot of this in my travels. So, he’s right that mindlessly shaping all design around tests can have bad effects, but no software developer should be doing anything mindlessly. I think there is value in letting tests shape your design much of the time, so long as one keeps in mind that there will be occasions where the tests have to be shaped by the design instead.

Everything in moderation, right? Including moderation.

10 Reasons You Shouldn’t Have Senior Developers, Tech Leads or Architects

Two weeks ago I published a post titled ‘Why Smart Software Teams Don’t Need Senior Developers, Tech Leads or Architects‘. I received a lot of good feedback, but I also know it was a long read. So, if you’re interested by the title but are looking for a quick brain dump rather than an enjoyable read, here’s the abridged version:

At Tyro Payments, we’ve doubled our Engineering team over the last year.

We don’t hire for, or use, titles like Graduate Developer, Junior Developer, Senior Developer, Tech Lead or Architect. Everyone has the title ‘Software Engineer’.

This is an important part of Tyro’s Engineering team culture. Here are the reasons… Continue reading

Why Smart Software Teams Don’t Need Senior Developers, Tech Leads or Architects

Queue for Steve Jobs' keynote at WWDC 2010

A queue of software developers, not unlike the one that has inundated my inbox for the last year.

We’ve almost doubled our Engineering team at Tyro Payments over the last financial year and we’ll be adding that many again this year.

Most people who’ve worked in or with software teams would imagine that within this surge of hiring we’ve been filling all kinds of different roles – Graduate Developers, Junior Developers, Seniors, a couple of Tech Leads, maybe an Architect. But the truth is we’ve only been hiring for one role: Software Engineer. In fact, it’s the only development role on our team, and it’s the title we give to everyone on the tools, whether they have 20 years’ experience or none. This isn’t just some convenience we came up with to save ourselves HR work. It’s an incredibly important part of the culture at Tyro. Why? Continue reading

What is DevOps? … in bullet points, quotes and tweets


Interest in DevOps is booming. I feel like I heard it mentioned as the motivation for some decision at least once a week last year, and it climbed its way into the headlines of most of the newsletters I receive from LinkedIn, Twitter, InfoQ, etc.

Army soldiers pulling hard on a rope in a tug of war. Teams working together in a DevOps environment concentrate on all pulling together in the same direction.

I thought I understood it. It just means Dev and Ops collaborating closer, right? But as it gained more and more attention, I realised it must be a movement, not just a simple idea, so I set out to discover what DevOps was really about.

This blog is my summary of what I’ve learned from reading about DevOps over the last few months. There are heaps of resources and lots of people making great observations, so what I’ve done is try to distil lots of salient points into bullet-point format to make it easy for people to pick up as much as possible in a short read. Continue reading

When the Production Queue Stopped

Unusual pain

Traffic jam between New York skyscrapersLast week at Tyro we had a fairly serious production issue. Thankfully, the impact was nowhere near as serious as the kind of outages that most Aussie banks have delivered over the last couples of years; our merchants could still accept payments and they could  access their reports, but our backend processing was banked up such that their transactions weren’t appearing on reports for about an hour. That might not sound too serious, but our merchants are accustomed to seeing transactions appear in their reports pretty much instantly, so when our monitoring told us we weren’t delivering on that expectation we considered it a serious incident.

There was lots of good news out of this. Dev and Ops rallied as one team. We fixed the problem, and we managed to fix it without deploying a code patch. We learnt about an important performance restriction in our system that was fixed the next day and gave us knowledge that we can use to improve going forward. And we managed to get it solved before the last bus on my route for the night.

Success… eventually

The bad news was that it took us a long time to get to the good news: it was about nine hours from the first indication of the incident to when we finally executed the winning solution. Looking back, I feel a bit stupid that we didn’t – that I didn’t – solve it in a quarter of that time. All the information we needed to lead us to the solution was staring us in the face, right from the beginning.

Continue reading

Meetup Digest: Migrate to DVCS Sydney (October 2012)

Two weeks ago, I and a couple of other developers from Tyro went to the first “Migrate to DVCS Sydney” Meetup at Atlassian’s new headquarters. Here’s my notes on the most salient points from the evening (with some editorial by me in italics) …

Talk 1 – Jonathon Creenaune from the JIRA team

  • Know why you’re migrating. I’m sure he meant, and may have even said: have a business reason to change, don’t just be a cargo cult.

Continue reading

A Blog About Me

Art installation that says "Replace Fear of the Unknown with Curiosity"Welcome to my new blog! I’ve had blogs before. I had a blog about photography for a couple of months. I had an active blog about Scala for quite a while, which even achieved fleeting fame a couple of times. But when circumstances pull me in other directions such that I haven’t taken any photos for a few months and I haven’t written any Scala for a few months, I’ve got nothing to add to those blogs

While those blogs may have become stagnant, I have not. I’m learning things all the time, every day; things that are fascinating, things that are useful and things that are worth sharing. I realised that constraining my previous blogs to certain topics resulted in constraining which parts of my new knowledge I was able to share with people. I tend to read a lot, both on the web and from dead trees, and I learn lots of things that I think other people might benefit from hearing about.

So, this is my new blog, and it’s about me. It will be about Scala, but also about software in general; and it might be about photography, but also more widely about life lessons; and it will probably have some stuff about writing software at Australian banking startup Tyro Payments; and it might sometimes be about being a Dad and a husband, working happily with people, living in Sydney, playing the drums, designing web sites, reading books. These are all things I do, all things I’m learning from, and all topics where I’ve learned useful things that it would be useful to share. But really, the topic is me. Hopefully, amongst all that defines me, both the person I am and the person I’m becoming, you’ll find something worth reading about and, most of all, some things worth learning.

Image credit: Zephyrance Lou