I spoke earlier in the year at the Sydney Microservices Meetup about the long path we’ve taken at Tyro Payments over the last decade, gradually tending towards a more fine-grained SOA approach – microservices as it’s come to be known recently.
I covered a lot of ground in that talk, but something I didn’t get around to talking about was security. However, I believe that’s a really important topic to think about in microservice environments. It’s even more important than with a monolith, because in a service-oriented architecture you’re making a lot more of your system’s functionality directly exposed to the network, and that puts it in closer reach of would-be attackers, or “increases the attack surface” as a security pro would say.
So last week I presented another talk entitled “Microservices Security: All the Questions You Should Be Asking”.
Microservices Security: Let’s Share What We Know!
I want to tell people all about what we’ve been doing about security at Tyro lately. Security is incredibly important to the IT community and I think it’s imperative that we help each other improve. I want to share with the world some of the problems we’ve dealt with and some of the great solutions our team has built.
But there’s a problem: I can’t.
If you’ve watched my first microservices talk, you might remember this architecture diagram…
That diagram is not of our architecture. It has roughly the same number of boxes and lines as our architecture, but all the services and connections were placed randomly. I wanted to show our real architecture, but some security-minded people at work convinced me it wasn’t a great idea.
Now, at the time, I was a bit cynical about their advice. I quoted Shannon’s Maxim at them…
“The enemy knows the system”
… I said. “If we believe our security is good enough even if they know everything about it, why do we need to hide information?” They said it was like giving the enemy a map. I said giving the enemy a map should make no difference. We agreed to disagree, and as you should with most security debates, I conceded to take the more conservative route.
Then a funny thing happened. A few months later, I was sent on a security training course, and the first thing we did was “information gathering”. As well as finding routers exposed to the internet and web cams streaming people’s houses to the whole world, our instructor said, “You can even sometimes find information about the internals of people’s systems just by googling the right keywords.” Interesting, I thought. So I googled “tyro architecture”, and what do you think showed up?
Here was a professional security expert teaching me the things black hats do to try and find weaknesses in a target system, and up pops the information I’ve been sharing on the web. My cynicism evaporated immediately. I am a convert to hiding information. But is it wrong to hide information?
Shannon’s Maxim and Kerckhoffs’ principle don’t say that you shouldn’t hide information, only that your security shouldn’t rely on those things being hidden. So, the security of our systems doesn’t rely on our architecture being unknown, but if the fact that it’s not publicly available makes it harder for someone to attack us (or easier for us to detect them doing so), that’s great!
Microservices Security: Can We Share Something?
As our team has researched a wide variety of issues and technologies over the last year, though, one thing has become very clear: hardly anybody is sharing what they’re doing in security.
That isn’t universal. Sam Newman has an excellent chapter in his book Building Microservices about security touch points and technology options, and Scott Shaw included some great advice on security as one of the “microservices speed bumps” he spoke about at YOW! 2014. These guys are both consultants, so they have the advantage of being able to say what they did without people knowing where they did it. Troy Hunt has a fantastic blog mostly about web security. The Netflix Engineering blog has a few articles and presentations tagged with security, and there’s some people in the Spring developer community putting out some stuff, too.
In general, though, when we really started digging into the guts of how to solve particular problems, or how to implement a specific technology, we often found ourselves on our own.
All this left me with a dilemma. Our team has spent stacks of time learning all kinds of great stuff about security – stuff that every microservices team would love to know. And we’d love to tell people all about what we’ve learned, but the practicalities of business present us from doing so.
So I’ve been racking my brain to figure out what we can share, and I finally came up with this: while we can’t share the answers that we eventually rested on – the methods we used and the technologies we chose – I can share a lot of the questions we asked ourselves along the road, the starting points of the discussions that led us to where we are today. This won’t necessarily lead you to the same solutions as us, but hopefully it will help you to consider almost everything that we’ve taken into account as we’ve grown our system.
Microservices Security: All the Questions
Here is my list of questions that you and your team should be asking yourselves about microservices security. It’s intended to be used as a checklist for you to evaluate your own systems and processes. Hopefully, you’ll find that you already have most of these issues covered, but there is always more to learn. There are links all the way through to further information about each issue.
(by which I mean services that make up your system which do not interface with the Internet or other external systems)
Are you just protecting your system at the Internet boundary? (Defence in Depth)
What protections do you have in place if an intruder gets inside your core network? (Defence in Depth)
How easily could someone inside your network get access to the traffic between your services? (Secure Communications)
Do your services trust each other too much? Or… Do your services trust whoever is calling them too much? (Are you sure only your services can call into your services?) (Reluctance to Trust)
When your service is called, does it require the calling software to authenticate itself, or does it let anything connect? (Service Authentication)
Do your services let their callers access all the APIs that a service offers, or just the ones it needs to fulfil its function? (Service Authorisation)
Does the identity of the person who originated each call at the client get passed into your internal services, or is that lost at the gateway? (Principal Propagation)
Can your services request any data from each other, or only the data of a user that has given their authority? (Principal Authorisation)
If an attacker owned a service, could they pretty easily request anything from its downstream services? (Principal Authorisation)
What guarantees do you have that a request received from an authenticated user hasn’t been tampered with? (Tamper-Proofing)
How do you ensure that an authorised request, delivered a second time, is detected and rejected? (Replay Protection)
Everyone knows about SQL injection, right? But what measures do you have in place to ensure no one writes code that’s vulnerable to SQL injection? (SQL Injection)
Are you familiar with all the other types of injection, and how to protect against them? (Injection Beyond SQL)
Are you up to speed with the state of the art in password storage? (Password Storage)
Do you realise that, if your password database is stolen, simply salted hashes are completely useless these days? (Password Storage)
If you need to upgrade your password storage algorithm, how do you do that without mass disruption to your users? (Password Storage)
How do you actively identify the private and sensitive data in your database? (Promoting Privacy)
If your data gets stolen, what protections do you have in place to prevent the most sensitive parts from being readable? (Private & Sensitive Data)
If your services are using private keys, how do you protect those keys from being used by an intruder? (Key Management, Never assume that your secrets are safe)
Do you know what a Hardware Security Module (HSM) is, and when and how to make use of one? (Key Management)
What logging do you have in place that you can use to detect and analyse security breaches? (Security Logging / Security Information and Event Management (SIEM))
(by which I mean any 3rd party software that you run inside your system and interface with. For us at Tyro, currently this is mainly our databases and messaging system, but it may include other systems for you, e.g. BPM middleware. Most of these questions apply to integrated external software as well.)
Do you share a single database login across all your services? (Least Privilege)
How much data do your services have access to? All of it? Or only what they need? (Least Privilege)
If an attacker got hold of one service’s database credentials, how much data would they get access to? (Least Privilege)
Do your DB authorisation policies allow updates and deletes to tables that the application only ever inserts into? (Least Privilege)
Do you share a single messaging middleware login across all your services? (Least Privilege)
Does your messaging middleware even have login credentials? (Some don’t yet!) (Least Privilege)
Do your services have access to all messages in your system, or only the ones they need to see? (Least Privilege)
Can your services send messages to any queue, or only where the ones they need to? (Least Privilege)
If an attacker got hold of one service’s messaging credentials, how much data could they get access to? (Least Privilege)
If an attacker got hold of one service’s messaging credentials, what operations could they initiate? (Least Privilege)
If you’re protecting your database and messaging with login credentials, how are you protecting the credentials? (Never assume that your secrets are safe)
How might the legacy systems in your architecture put other services at risk? (Securing the weakest link)
(by which I mean services which interface with the Internet or with other, externally managed 3rd party systems)
Have you upgraded your TLS implementations to the latest versions possible? (Secure Communications)
Have you configured TLS to eliminate downgrade and weak cipher attacks? (Secure Communications)
Who on your staff knows everything about TLS and how to configure it safely? (Secure Communications)
How do you ensure your internal websites and admin URLs aren’t accidentally opened up to the Internet?
What information can I get out of the unauthenticated APIs of your gateway services? (Enumeration)
I have a list of cracked passwords and user emails. Could I use your password reminder URL to test which users are in your system? (Authentication)
Does the rest of your system trust your gateway services too much? (Reluctance to Trust)
If you assume that your gateway services have been fully breached, what would you do differently elsewhere? (Defence in Depth)
If your gateway services were fully breached, what data could be gleaned from memory? (Defence in Depth)
If your gateway services were fully breached, what data could be captured from the network traffic? (Defence in Depth)
Web & Other Clients
(by which I mean software which you may or may not author which is interfacing with your server-side systems, most likely across the Internet)
How are you helping your users to choose safer passwords? (Password complexity)
When a password is entered wrong, what feedback do you give? Could it be used to enumerate user accounts? (Enumeration)
Do you lock an account after some number of failed login attempts? (Authentication)
How many chances do you give an attacker to guess the password on each account? (Account Security)
When you lock an account, what feedback do you give? Could it be used to enumerate user accounts? (Enumeration)
Do you have a password reminder feature? Could it be used to enumerate user accounts? (Enumeration)
Do you have a password reset feature? Could it be used to enumerate user accounts? (Enumeration)
Have you considered whether your system, or some parts of it, warrant multi-factor authentication? (Enumeration)
Has anyone else noticed there seems to be an epic battle between security and good User Experience? (UX vs Security)
Are you familiar with the OWASP Top Ten web vulnerabilities? (Web Security Flaws)
Could you name all of the OWASP Top Ten web vulnerabilities? (Web Security Flaws)
Could everyone on your team name all of the OWASP Top Ten? (Web Security Flaws)
Could everyone on your team explain how to protect against all of the OWASP Top Ten? (Web Security Flaws)
How do you ensure that every piece of user data is properly escaped when used as output? (XSS / Output Encoding)
How do you escape user data correctly for the myriad of different contexts in which it’s output? (XSS / Output Encoding)
How are you helping prevent your users from getting attacked as a result of using your web app?
Does your web app design treat the browser as an insecure environment? (Reluctance to Trust)
Does your native mobile app design treat the device as an insecure environment? (Reluctance to Trust)
How are you helping prevent your users from getting attacked as a result of using your native app?
What data are you storing or caching on the client? How are you protecting it? What could happen if someone stole it? Does it need to be there at all?
People & Process
(by which I mean the people who develop and operate your systems, and the processes they use to do that)
What are you doing to ensure security is baked into everything your engineering team does? (Build Security In)
How do you burn common security principles into everyone’s brains? (Security Principles)
What security activities are explicitly built into your development process? (Secure Software Development Process)
What security training do you provide for your developers, testers and operations staff? (Security Training)
Do tech staff just know the names of vulnerabilities, or do they actually know how to exploit & test them? (Security Training)
What controls have you put around who can access which parts of your system? (Access Control)
What’s your plan for regularly reviewing the appropriateness of those controls and people’s access? (Access Control Review/Audit)
What’s your process for discovering and remediating vulnerabilities in 3rd party software? (Vulnerability Management)
How do you encourage engineers to dedicate time to brainstorming the risks in your systems? (Risk Brainstorming / “Risk-storming”)
Do you have service templates that ensure every new service starts off with great security? (Secure Application Templates)
What’s your plan for getting internal staff regularly testing the security of your systems?
What’s your plan for how often you’ll bring in external security experts, and how you’ll choose what they focus on? (Security Testing)
What activities do you get expert help with? Just penetration testing? How about design and architecture reviews? (Security Testing)
What automated tests do you have to catch vulnerabilities as they’re written? (Automated Security Testing)
What automated tests do you have to ensure security controls are always in place? (Automated Security Testing)
Are you constantly asking yourselves: “What if this control fails? What’s the next control?” (Defence in Depth)
Assume your network is compromised.
Which parts of your system are keeping you up at night?
Want to learn more?
Blogs & Web Sites:
‘Microservices Speed Bumps’ (incl. security) – Scott Shaw (my notes)
Troy Hunt’s blog
Troy Hunt’s “Secure Account Management Fundamentals” course on Pluralsight
The Open Web Application Security Project (OWASP)
Build Security In – US Department of Homeland Security website
Australian Government Information Security Manual – Australian Signals Directorate
Netflix blog articles tagged with ‘security’
Blogs by Spring Security’s Rob Winch
Presentations by Spring Security’s Dave Syer
Image credit: ‘Day 342 – Hacker‘ by Christophe Verdier
This is really great. You basically described threat modeling in a specific context. You have great references, and I’d suggest a couple additional resources:
OWASP ASVS: (https://www.owasp.org/index.php/Category:OWASP_Application_Security_Verification_Standard_Project) – pdf linked at top right of page. That project is great for having a lot of security related questions you should ask.
Adam Shostacks threat modeling book: http://www.wiley.com/WileyCDA/WileyTitle/productCd-1118809998.html It’s really good and will get you well on the way to understanding how to start small and scale up. This will help add rigor to your process as you need it.
Also, since you’re in Sydney, might want to check out the OWASP Sydney group if you haven’t already (http://www.meetup.com/OWASP-Sydney-Web-Application-Security-Group/) and (https://www.owasp.org/index.php/Sydney). Lots of likeminded people and a good community to have access to.
Pingback: Microservices Security: All The Questions You Should Be Asking https://www.grahamlea.com/2015/07/microservices-security-questions/
Pingback: Microservices Weekly Report | /dev
Pingback: Microservices Architecture? let’s go for it, but… | Maciej Madej
An excellent list. Thanks for *sharing.*
This check list is very helpful, I am curious to know the implementation for some of it ;). Well done!
Great article, thanks for putting it together. I found you on Twitter and meandered my way here.
So, talking specifically about defending data-at-rest, I feel it boils down to:
-User Authentication (including Multi Factor Authentication)
-Strong Encryption (preferably 256-bit AES encryption)
-Centralized Encryption Key Management that is logically or physically separated from the encrypted data
-Real-Time Log Monitoring with a SIEM
Earlier this month I put together a Definitive Guide to Key Management Fundamentals:
to make KM as approachable as possible. I hope it adds to the conversation here.
BTW – What are your thoughts? Would you add anything as an essential to defending data-at-rest?
Pingback: How to Choose Your First Microservices • Evolvable MeEvolvable Me