Recently I was invited to be part of a panel on Microservice Security. The fools! Normally on these panels they want you to talk for 5-ish minutes; unfortunately I came up with about 15 minutes worth of material!
That’s perfect for a blog :-)
Older designs
Before I talk about microservices I want to take a look at older designs
Monoliths.
A “monolith” is pretty much an “all in one” application. It may combine the UI, the app layer, the storage layer… all bundled together. I mean, who hasn’t written 10,000+ lines of spaghetti C code?
In a monolith we had data structures, but we didn’t really have much
in the way of data segregation. Persistent data structures were
pretty much global in scope. We could try and normalize access to these
structures, e.g. using library functions, but the app couldn’t enforced
this; instead we attempted to control things via code reviews… “why are
you setting foo.bar rather than calling library function do_magic()?”
Now from a security perspective this wasn’t really too much of an issue; only the app internals could see it, anyway. Well, except it encouraged bad programming techniques and could have led to hidden bugs pretty easily, especially if modules got out of sync on the data structure formats!
Object Oriented Programming
This introduced the idea of public and private elements. Now we can
hide things and enforce access through “approved” public interfaces,
or methods. You can no longer set foo.bar directly, but instead
have to call Foo.do_magic(). This can definitely help ensure data
consistency and reduce bugs.
However, those public methods are, well, public. If I can see the object then I can call those methods.
But it’s still not really a security issue, for the same reason; only the app could see those methods.
Three tier apps
Now we’re starting to separate components. No longer do we have a monolith, and what used to be methods internal to the app may be exposed to the wider network. This brings in a set of security challenges.
For example:
- How to we ensure the app tier is only called from the correct display tier?
- How to ensure the DB is only reached from the app?
- How to stop the expected data flow from being bypassed?
We might look at firewalls to do some of this. If you can’t see the service endpoint then you can’t call the functions. However this is limited and gets hard to manage at scale. Unless your environment has been designed from day 1 for micro-segmentation and defined communication paths you probably have a relatively standard “open” 3 tier network.
So the common solution is to require authentication between the tiers; e.g. the web tiers authenticates to the app tier; the app tier authenticates to the database.
And remember that the network may also need to be protected; data in transit may need to be encrypted if it contains sensitive components. Yes, even inside your own datacenter!
Multi-component apps
Docker was a big driver of this, but it’s also used elsewhere. The idea is to allow component re-use in a simple “Lego build” design.
“OK, let’s spin up an app with a memcached container, oh… let’s also use mysql, nginx, redis, mosquitto… and 6 different containers for my app.”
This is really a generalized case of the 3-tier app. So we can use the same type of solution.
- Authentication; each component must know that the call has come from an authorized source
- Encryption in transit
- And let’s not forget input validation!
Instead of just doing this in a couple of places we might need to do this in a dozen places. This is getting complicated…
Finally, micro-services
Really, a micro-service design is just a generalized form of multi-component apps
Instead of monolith exposing functions to other parts of the application, each micro-service dedicated to a particular function or group of functions. And these functions may be exposed to the network!
NOW we care about the security of the function. They can be reached from outside of the application.
In some cases this exposure is deliberate. These functions should be considered “public”. They’re meant to be the interface between the application and the application clients (whether they’re internal to your organisation or external to the org doesn’t really matter; they’re external to the application).
In other cases this expose must be prevented. These functions should be considered “private”. But are they?
This public/private terminology is similar to that for Object Oriented Programming. It means public or private to the application… however that is defined. I am NOT talking about “private” to the organisation. Think zero trust :-)
How do we ensure these private interfaces are kept private? How do we ensure that public interfaces are only used by authorized consumers?
And we have the same answers.. authentication, encryption, input validation…
Well, we can do better…
All that is hard work.
Fortunately modern technologies can assist in newer designs that can help enforce the separation between public and private endpoints
Some examples of this include
Private network structures (eg docker-compose) so that the backend components (memcached, etc) aren’t exposed off the local machine; for single-app machines this may be sufficient to make an endpoint private. Docker swarm took that to a multi-machine level but required oversight for multi-app.
k8s; containers inside a pod can talk to each other. Great for private communication. Containers in other pods have to talk via public interfaces. Need to balance scalability of individual micro-services to pod-level scalability and easier communication…
Container network constructs; e.g. istio can act as a gateway between pods that will transparently do TLS encryption and can do mTLS authentication; set the rules as to what pods can talk to what other pods and it’s enforced by the infrastructure.
How you deploy stuff can answer some of the security concerns raised by going to micro-services.
Some other complications
These aren’t necessarily security issues, but to be kept in mind
Every public endpoint needs to be considered an API to your application. It defines a contract to your consumers. If you change your call (data structure, functionality, parameters, whatever) you need to treat it as an API change.
Change control gets harder; should each microservice have its own SDLC lifecycle? Be registered in your app inventory?
Knowing how to “right size” your microservices is a skill. If you go too granular then you get a lot of overhead (even if it’s just communication overhead! A network call is orders of magnitude slower than a local app function call) and lose performance and may have more paperwork. If you go too coarse then then you may not get the independent scalability you desire.
Summary
Micro-services require you to think about what endpoints are considered public and what are private.
Any network port that is exposed “off” the machine is automatically public. Even those only on localhost may be public on a multi-app machine
Authentication, Authorization, Encryption, AUDITING… these need to be enforced at every exposed endpoint! SDLC processes should be leveraged to help validate this.
Remember, data flows can be initiated differently from what your app expects; you may expect calls A->B->C->D but an outsider might call C directly if it’s public; you need to handle that.
This is a lot to keep in mind; not only must the code be written, but knowledge of networking, scaling, performance, encryption, authentication…
This really highlights the importance of SecDevSecOpsSec. A proper architecture design at the beginning can simplify the solution and take some load off the development team.