Scaling service-to-service auth forces authentication out of the application code and into the architecture. This session follows that progression from a monolith handling tens of thousands of requests per second to distributed systems processing hundreds of thousands or millions. You will see where databases, synchronous validation, logging, token checks, and authorization policies become bottlenecks, then learn which components to isolate, cache, queue, replicate, or move closer to the request.
.png)
Okay, folks. Good morning, good afternoon, good evening, as the case may be for you. Thanks for joining us today. We are joined by Harsha Reddy, lead engineer and technical product lead for identity and access management at Klaviyo. He's bringing years of engineering expertise to the role, with a strong background in microservices, API development, and platform engineering, specializing in building secure, scalable authentication and authorization systems — something we're pretty familiar with around here. His experience spans full-stack development and developer tooling, making him a key voice in the IAM architecture and engineering best practices space. Harsha, thanks for joining us today.
Thanks, Brad. I'm going to get started and share my screen. Can I get audio confirmation that you folks can see my screen?
Yep. Looks good.
Awesome. Thanks everyone for joining, and thanks, Brad, for the kind introduction. Today we're talking about scaling service-to-service auth.
Authentication and authorization — again, this problem seems simple when you start, but it becomes a massive engineering challenge as your organization and your customers scale. The way to think about it is: at a small scale, it's more of an implementation detail. You kind of build it with your app. At large scale, it becomes infrastructure that you have to maintain. That's what this whole talk is about.
This is one of my most favorite pictures that I took. That's Mount Rainier. I took it as we were scaling almost to the top, so it's kind of symbolism for this talk.
So, I'm Harsha. I work as a lead engineer and I'm the technical product owner for the identity access team at Klaviyo. Specifically, it's the customer-facing identity access management team, and you can find me at those places.
Identity at Klaviyo is the foundation of a lot of requests — customers are logging in, customers are accessing APIs, and so on. We are one of the core foundational teams at Klaviyo, and everything in Klaviyo revolves around identity.
What do I work on? Practically speaking, we own three big areas. Customer logins, which I call human authentication and authorization. We also work on protecting and securing our external APIs — external API authentication and authorization. And in addition to that, we also protect our internal APIs. I work together with various infrastructure security teams and our offensive security team, and together we form a tight-knit group making sure that anyone accessing Klaviyo resources has the right access controls in place. Identity is the foundation of every request, and my job is to make sure it stays fast, secure, and reliable, no matter the scale — within the boundaries of physics, if you know what I mean.
Our agenda today breaks down into four high-level sections. I'm going to start off with authentication and authorization — I've shortened it to authn/authz. So if you see that term in the slides, that's the short form. This journey describes something I've gone through a few times in my career, and it's normally a journey that a lot of customer-facing identity access management teams and security teams go through.
The first agenda item is the monolith — where everybody starts and where the first bottlenecks appear. I've put colored boxes at the bottom to indicate requests per second to your auth system, just to show how we're progressing. Step two, we're going to talk about decoupling your auth from the monolith by using an identity provider. Agenda item three, we're going to talk about shifting to the edge — an API gateway is a good pattern to unlock big throughput gains. And finally, we're going to talk about the most complex topic: the authorization layer. This is where scale becomes a real challenge after authentication and basic authorization are solved. Every step removes a bottleneck on the path toward your journey to million-plus RPS. Let's hope we all get there.
So here's a high-level data flow diagram for a classic monolithic authentication and authorization system. On a high level, there are users accessing your website, web pages, or mobile apps — I just call it "web browser" here. Generally, that routes through an ingress layer where you have your web application firewall, a reverse proxy like NGINX, and possibly CDNs and various other layers. Once the request propagates through that, it lands on a monolith where, generally, you have an authentication and authorization middleware that lets the user or API customer log in. Once that layer is complete, the business logic takes over and you perform data operations. This is a very tightly knit system where the middleware and the business logic generally either rely on the same data store, or you have multiple data stores — databases, NoSQL instances — that work together to provide a unified front.
This can scale to a certain extent. At Klaviyo, we've seen this system scale up to 20,000 to 30,000 requests per second for our auth system pretty comfortably. However, we do see bottlenecks. Very often we saw bottlenecks with DB contention. As a monolithic application, generally there's also a monolithic DB — that's how companies get built. Through their journey, they realize that their user data and user information is stored alongside other application business data like orders or B2B company information. DB contention becomes a real problem: is my DB getting overwhelmed? Is one specific customer making a lot more API calls? How do you support that on a single database?
You also run into problems with synchronous validation. What I mean by that is your monolith could be waiting on a particular database call or some local business logic to execute before performing some sort of validation — you're just waiting on that to happen, and that slows down the authentication process while you wait for other middleware to do their job. This is a very tightly coupled ecosystem, as I mentioned.
Your app lifecycle — what I mean is, if you have continuous deployment or even continuous integration in your systems, you have to ensure that you work well with other teams that are constantly pushing code to the monolith. Sometimes pods get taken out or a DB upgrade is happening, so you have to adhere to those processes. Additionally, there's no blast radius isolation. If there's a bug in somebody else's code, it could take down your auth system.
One of the goals, after living in the monolith and trying to make it scale, is moving identity to a standalone service to reduce the load and unblock scaling.
So here's the next phase of your auth system. This diagram looks very similar to the previous one, with one additional entity at the top: the identity provider. There are a lot of identity providers in the market, and FusionAuth is one of the biggest. There's also Auth0, Frontegg, Clerk, and others. The first journey you should ideally be taking is to move the auth system out of the app. We want to do this primarily because identity fronts everything, and fronts a lot of your business logic — you want that system to be one of the most stable and reliable systems in your stack. And what you don't want to solve for here is rewriting your own auth.
My recommendation is to use an off-the-shelf authentication and authorization provider where experts have contributed to build that product. Sometimes your team might not be an expert on SOC 2 compliance, for example. Identity providers, in addition to providing stability and scalability, also provide a lot of security and privacy improvements over existing systems that you might be struggling with if you hand-roll your own. The diagram is pretty straightforward — you can see the request flowing, and identity providers generally also provide an SDK. You can have Python SDKs or Golang SDKs that you integrate into your application pretty easily, and that authentication and authorization SDK is responsible for interfacing with your identity provider.
One of the other key aspects of a dedicated identity provider is it brings along a dedicated identity store. This isolation is very important, because you're storing a lot of sensitive user-related and client non-human identity data — for example, client IDs and secrets — in that data store. Isolating this data store means you can better secure who has access to the identity data, whereas in a shared data store that becomes a problem. Additionally, you can independently scale the identity data store.
This is basically a rehash of what I just talked about: OAuth is not gated by performance, tokenized trust enables distributed flows, improves latency, fewer monolith dependencies, independent scaling knobs for auth. I'm not going to go too deep there.
The fun part — let's talk about standard scaling techniques once you start using an identity provider. When throughput climbs, identity systems need to handle that, and these three measures will buy you a lot of breathing room.
The first one is your identity data store. I highly recommend deploying read replicas. If you're using something like Amazon Aurora, AWS Aurora, or an equivalent like Cloud SQL on GCP, they generally have managed database solutions. One of the technologies these cloud providers offer is a secondary replica or read replica. The advantage of read replicas is that auth is a very read-heavy operation compared to writes. A user is entering a password, or a non-human identity like a machine is validating secrets to exchange for a token — these are very read-heavy operations. Having a lot of replicas and handing off those read-heavy operations to replicas, which are easily scalable, helps reduce the load on your primary instance. Your identity provider generally has a connection pooler either baked into the application or you can use a cloud-managed database solution to pool the connections and distribute them among replicas as needed.
The additional advantage: if your primary data store fails, some of these systems can automatically failover and make a secondary replica the primary, so you don't lose a lot of customer data and your system stays up with only a very brief downtime. This system is also very scalable in the sense that you can pre-add replicas to handle bigger load — if you know, for example, that Black Friday and Cyber Monday are coming and you expect high load, you can add more replicas ahead of time.
The second pattern, and a very common standard scaling technique, is adding a cache layer. That's easy to add but very hard to maintain — the classic joke is that cache invalidation is one of the hardest problems. But the benefits offset a lot of the cons. Adding a cache layer helps move additional reads away from your data store. If you're using SQL as the data store for your identities, having a cache layer closer to your app — maybe a Redis instance close to your identity provider — really helps reduce the load further. Common cache providers include Valkey, Redis, and Memcached, among others. We use these very heavily on our team. These cache layers are set very close to the identity provider, and we restrict access to them so only specific infrastructure layers can reach them, protecting the cached data. The reason the cache is close to the identity provider is that cache invalidation becomes easy because you can add callback hooks — if a user has been disabled or deactivated their account, it's much easier to invalidate and update the cache when it's close to the identity provider.
Common items to cache: sessions are a very popular entity to put in cache. Short-lived tokens are okay to cache if you need very high scale. Public keys are another one — they're used for asymmetric token verification, like in OAuth 2.0 where you have a private/public key combo. You generally store the public key in cache because those don't tend to change that often. These eliminate a lot of round trips to the database, which is the goal.
The third standard scaling technique is having dedicated instances, and this has saved us a lot. There's this problem called the noisy neighbor problem, where our API traffic is much higher than our human customer login traffic. There might be a million humans logging in a day — that's around five million requests a day — but there's a million requests on APIs happening every minute. The scale is really different for human versus non-human identities. Splitting your identity provider instances into two, but having a control plane, is really useful to ensure that a noisy non-human identity — maybe one of your customer APIs needs a lot more data polls than normal — won't actually impact human logins.
This is a very common setup. It's still controlled by a single team — the security and identity teams generally control it — and the authentication SDK is the interface that standardizes everything. That auth SDK can be installed in a monolith or on a microservice, and it helps integrating teams interface with the right identity provider without a lot of heavy lifting on their part. These are the three standard scaling techniques, and depending on the situation your organization is in, they may become necessary.
Let's get into three advanced scaling techniques. These are things not a lot of teams have to do. But if you're at a scale where you're still struggling with bottlenecks after applying the standard techniques, the next layer really takes it to the next level.
One technique we use to improve scale and avoid the thundering herd problem: as I mentioned, you could suddenly have one of your customers — as a B2B company, they might be having a big sale — pushing a lot more data to your APIs all at once. That shows up as an extreme spike, and your systems could be stretched to their limits. In this case, having a cache pyramid — there are many terms for this — really helps avoid the thundering herd issue. The note is always: remember that you need good cache invalidation.
You can see a simplified diagram showing one example of how to do it. Your auth SDK supports a local in-memory cache, generally an LRU (least recently used) cache. If that misses, you fall back to a file-based cache — for scenarios where the data is not too sensitive, this avoids over-the-network hops. The third layer of fallback would be a distributed cache system like Redis, or if you're on AWS, ElastiCache is a very popular managed caching service. If none of those layers have the record, you contact the identity provider, go through the general process of authenticating and authorizing the user, and then store certain information from that operation in the cache tower. Using this approach, I've seen anywhere from 200,000 to 300,000 authentications per second handled pretty comfortably, by being very smart about what can be stored at which layer.
The second advanced technique is asynchronous events and logs. In a lot of systems, this becomes a bottleneck at certain throughput levels — you're also capturing events through the DB. Maybe you have SIEM security logs, and some systems rely on an audit trail from your identity and auth system. You generally want them to have information as close to live as possible. If you are recording these to a logging system or a security system like Splunk or the ELK Stack (Elasticsearch-Kibana), those operations can be synchronous, especially if you're writing your events to the same identity data store. That becomes a big bottleneck because a single non-human identity authentication flow to fetch a token could produce five events, and human operations could produce even more for each operation. It becomes a fan-out problem.
If you're using something like FusionAuth or Keycloak or one of these systems, using features like asynchronous logging — where logging happens on a background worker thread rather than the main thread — is really, really helpful to avoid bogging down the actual flows. Queuing systems are also very popular: you push an event to a queue, using something like Kafka or Apache Pulsar. That takes the load away from the main thread and outsources the logic to a queue, and then a consumer reads that data, processes it, and stores it elsewhere — or even in the identity store — but it happens outside the main process. This one can be hard to implement if you don't control all the key components, but I highly recommend reaching out to your identity provider about it. If you're working in open source, this is a very common pattern that really helps get you to the 500,000 requests per second level. Just remember: at 500,000 RPS, even a millisecond or a few milliseconds of delay can cause head-of-line blocking, where the first request is still writing to the DB and everything else is blocked — and it can snowball into a worse issue.
The third advanced scaling technique is auto scaling — both horizontal and vertical. I already mentioned database systems having secondary or read replicas, which can scale very well. Some systems, if you're using a cloud-managed service in AWS or Google, provide auto scaling based on certain parameters — for example, a policy that says: if my DB CPU is greater than 80%, add a new replica to the existing pool. That's a very common technique where you can react faster and the system automatically scales horizontally.
You also want to scale your identity provider compute instances. If you're deploying to an elastic compute instance like EC2, you can set up an auto-scaling group to horizontally scale and add more instances when you're expecting or seeing more traffic. Vertical scaling is something you see in Kubernetes — there's the concept of how much virtual CPU you can request and what your limit is. It's more elastic and more nimble at grabbing more CPU as the instance needs it. We deploy our identity provider to Kubernetes as a way to get the most bang for the buck — we can scale horizontally and also scale vertically by adding more CPU and memory as needed, and the instances figure that out themselves.
There are also serverless solutions — GCP Cloud Run is a good example, or AWS Lambda. You take your identity provider container and deploy it to serverless, where handling more throughput dynamically is part of the benefit. But the guardrails have to be baked in. The standard scenario where you blow up your monthly bill with serverless is very common, so remember to set good auto-scaling policies. You never want to be woken up at 2AM to manually scale your auth instances.
The tagline here: never do synchronously what you can precompute, cache, or queue.
Please feel free to stop me if you want to dig in and have questions on any of this.
Now we are onto the next phase of our journey. Maybe you've applied the previous techniques and you continue to see bottlenecks with the monolithic ecosystem. The next phase of scaling auth is adding an API gateway.
This is where true service-to-service auth comes in. You may no longer be in a monolithic world. A monolith might just be another app in your service-oriented ecosystem. You could have a bunch of microservices serving real customer traffic, alongside a monolith that still supports a lot of your business. To be very honest, this is the setup I've seen most in my experience — a lot of companies land here. Certain teams have outgrown the monolith or have unique use cases the monolith no longer handles, so strangler patterns and other patterns come in where new services are created outside the monolith.
Whose job is it to protect them? It's yours. As these services scale, you want to protect microservices, macroservices, and monolithic services. Your scope has essentially ballooned from protecting one layer to protecting many, many layers. Some of these could be on EC2 instances, some deployed as serverless, some in Kubernetes. The one entity that really helps bring all of this together and make sense of the chaos is an API gateway. What you don't want is pushing the authentication and authorization to each individual service — doing that on every single layer makes it very expensive to observe and secure, because you're getting signals from a lot of entities, and you become a single point of failure where your system can overload the network.
Yeah. Harsha, a question that came in — and you're kind of still on the same subject here. How do you fix or address snowballing issues in production if they do occur?
Yeah, great question. I'm going to talk about the API gateway as one of the ways you can address it, but we can also go back to the previous system. Let's say you have scale that you cannot handle.
One of your key components here is your authentication SDK. Maybe you've entered a scenario where you think you can scale up to 500,000 RPS, but you're receiving 2,000,000 RPS. A couple of techniques: load shedding and rate limiting are really valuable. If you have contracts with your customers and they're sending more traffic than agreed, you should be empowered to shed the load. Additionally, you can still be pretty nice about it and automatically scale to a certain extent that prevents your system from going down within your limits. That really helps handle spiky loads.
The third thing — and what you're really trying to avoid is your system going down — is circuit breaking. If your auth system is struggling and responding with 503 Bad Gateway, that's a signal saying "I'm struggling, please back off." Having circuit breaking logic in your SDKs really helps protect your system. Now if you're using a managed identity provider, you would need to set these service level agreements with your identity provider team beforehand to agree on the limits beyond which they can drop requests.
So: load shedding and rate limiting, automatic horizontal and vertical scaling, and circuit breaking — these three really, really help. The one thing that is outside engineering here is proactive communication. Having really good observability helps. If you can see from a dashboard or an audit trail that it's a particular customer generating the traffic spike — one who has generally been great, but for some reason is sending more traffic — you can have a point of contact. Involving your customer support teams or customer representatives really helps speed the process, because you don't want to lose business. Being empathetic and present, and understanding why the issue is happening, generally helps resolve a ton of these misunderstandings at scale. Hopefully that helped.
Yeah. That's great, thank you.
Awesome. The API gateway — let me see how I'm doing on time. Oh, I'm doing great.
So this API gateway becomes your central layer. Every cloud provider has their own API gateway products. A very popular open-source API gateway is Kong. An API gateway is slightly more powerful than a reverse proxy like NGINX, but a lot of these follow a similar pattern — they are empowered reverse proxies in a way. Security and protecting APIs are one of the core tenets of an API gateway. What you can do, as you can see in this diagram, is that your auth SDK — which was in the monolith before — has been moved or shifted into the API gateway layer, where the API gateway takes on the responsibility of making sure the request is authenticated. Additionally, it takes care of caching, validating incoming requests, introspecting tokens or private API keys if you have them, and making sure the right amount of information is propagated along to your internal services — your monoliths or microservices. I introduced the concept of a token here, which I haven't covered in a lot of detail yet — we're going to go through that shortly. As you break up your ecosystem into this decoupled world, having an API gateway as the central funnel for all of your service-to-service and human requests becomes critical to protecting your APIs.
This diagram is just an evolution of the previous one — we're moving things around but still serving the same use case.
Now let's talk about standard scaling techniques in this phase. At this point, you've surpassed 500,000 RPS, and your customers and your company are demanding more. Your authentication system is supposed to secure access to APIs.
One technique being used very effectively is called local introspection. In OAuth 2.0 and further systems, and OpenID Connect — which is the human code flow system for authentication and authorization — the artifacts output from those flows are tokens. An identity token if you're doing OpenID Connect, and an access token and refresh token in both scenarios. When you have an access token, that is the token you pass along with requests to your APIs to tell them: "Here's a token from me — perform this operation on my behalf." OAuth is a delegation protocol. If you're logging in as a human into a system, you're handing the responsibility of doing something on your behalf — maybe updating your name, maybe pulling your order list — to machines doing it on your behalf. For non-human identities it's very similar: you assign the identity to the machine. In both cases, you get a JSON Web Token or an opaque bearer token, each with its own scalability and security tradeoffs.
Once you get that access token, it's generally attached as a bearer token on requests. When you receive that bearer token, it's really useful to perform introspection locally. Since tokens are signed with private keys, you generally have a public key associated with that token. You can simply fetch that public key and validate tokens locally. A simple diagram shows this: you have an incoming request with a token, you call the identity provider and make a request to the well-known endpoint, which addresses the keys endpoint. You fetch the keys, cache them locally, and then all incoming tokens are validated locally without needing to communicate with the identity provider. An API gateway does exactly this. For us, it reduced the number of requests to the identity provider for validating every single token by about 97% — we were able to support our teams without being a bottleneck for their systems.
There's an IETF spec for the introspection endpoint that you can refer to. The problem here, again, is that revoking tokens is generally a big challenge. You want to make sure keys are not cached for too long — you still want limited caching so you can revoke things, either via polling or push — and that way you can still keep the system secure.
The second technique is propagating the token. Going back to the previous diagram: the API gateway is where the token is returned for an identity post authentication and authorization. That token can be cached in the API gateway layer so you don't need to constantly fetch it, but the same token can be propagated to multiple services. It's a delegation thing — you've given someone access to perform some action on your behalf, and that could involve seven microservices to get you, say, orders on the page or analytics about a particular customer account. You pass the same access token to all these services, and they can each verify it locally. Once they verify it locally, they can return the data because they have all the information they need about the identity to perform that operation.
This is a key point: don't have each microservice in the flow fetch its own token. That becomes a single point of failure very fast at massive scales, and we're talking millions of RPS. You want as much local computing in the flow as possible and as few failure points as possible. Propagate that token down the chain — and back up the chain if needed. The only caveat: please try not to store tokens in a data store. Maintain good hygiene, and keep tokens short-lived. Short-lived tokens are very, very valuable.
The third standard scaling technique in this phase is multi-region backups. Maybe your company operates in multiple regions — you can have a passive identity provider instance: an end-to-end replica, not just a data replica, but a passive instance running at low power, ready to spin up aggressively as needed. So if you're on us-east-1, us-west-1 could be another data center where a secondary instance is on call. This is really, really good at scale. You want circuit breaking to happen quickly on region one, and once the circuit breaker trips, you automatically failover to another region. An API gateway makes this easy — you can have a passive route, an active route, and flip between them as needed.
A couple of advanced scaling techniques with an API gateway in the flow. The first is the sidecar pattern. There's a very popular sidecar technology called Envoy. The Envoy sidecar is a popular application that runs beside your app's core business logic for your service. In Kubernetes, you can think of it as another container running beside your application container. On EC2, it could be an agent or a daemon process running in the background — beside your application business logic — performing various critical operations. Something I've done a lot in my career is having the sidecar cache the public keys. You no longer need an auth SDK installed on each service. This decouples the token introspection operations into a single common mechanism. A very popular service mesh is Istio. Istio promotes this technique by letting you configure an auth policy in the sidecar that says: "If I receive a token, where do I go fetch my public keys from?" It can handle the caching and local introspection, and it works very well with your API gateway.
The next advanced technique is multi-tenancy. This is a little more advanced — you can split up your identity provider to give a dedicated channel to a large customer, maybe somebody who provides a lot of business to you. Through the API gateway, you can provide them a dedicated channel without affecting others. At this point, more advanced techniques like having an auth control plane come into play — another service that manages all of these instances in your company, where you can control scaling of a dedicated instance, dedicated rate limiting, load shedding patterns, and so on. This is a very advanced topic that a lot of companies do use, but your scale needs to be at a level that justifies the cost.
And finally, let's go through the authorization layer. Very similar diagram — you have an API gateway, your monolith, your services. At a certain point, your identity provider cannot scale to handle the authorization. Quick recap: authentication is verifying who you are or who the non-human identity is — username and password, MFA, or for non-human identities, a client ID and secret, or even better, a private key using the JWT Profile flow, whatever it might be. Authorization is: what can this identity do in my system? At a certain scale, the problem becomes so big that you have to decouple and spin up a dedicated authorization system — I've called it an authorization provider here. With an API gateway, this becomes relatively straightforward. You can apply the same techniques you used for an identity provider to an authorization provider.
This is where things like role-based access control (RBAC) and attribute-based access control (ABAC) come in. ABAC is a superset of RBAC — you can fetch environmental attributes, like restricting a customer to making API calls only during a particular hour of the day, whatever it might be. You collect those attributes and make a decision, which needs to happen on every single API request. Common authorization providers are Open Policy Agent, Permify (I believe FusionAuth just acquired them), Oso, Casbin, and OpenFGA. A lot of solutions exist out there, and they can scale — I've seen them scale to millions of RPS pretty easily with a basic setup.
Just to quickly clarify, AuthZ is not something you can fix by throwing hardware at it. You need to be really sensible about making sure authorization policies have good hygiene — really clean resource modeling, minimizing policy complexity. Evaluating where the decision and the data lives is very important: maybe don't rely on data living in another region because a decision could take ten to twenty milliseconds just to fetch that data. You can have authorization happen at the API gateway layer, as shown here, or near the application — the monolith could have its own authorization policy. I'm not prescribing one way or the other; it really depends on your system, and each has its consequences.
Quick recap — we're almost at time. Isolate identity early by using an identity provider. Cache aggressively, invalidate precisely. Push validation to the edge using an API gateway or your SDKs. Authorization scales with well-defined policies — "well-defined" is doing the heavy lifting there. And decouple where appropriate — at massive scale, try to decouple. Any questions? I think that's the end of my talk.
That was awesome. Really, really insightful stuff. I love how, at the end there, good policies are doing the heavy lifting — I think that's true at any stage, but especially when you're talking hundreds of thousands or millions of requests. So, Harsha, if people want to get a hold of you, how can they reach you?
Absolutely. They can reach me on LinkedIn — I'm Harsha Reddy on LinkedIn, you can find me pretty easily. Or my email will be linked in the session.
Awesome. Thanks so much — really appreciate you taking the time to talk with us today. For those of you who are here, you're going to get your recording or a link to the recording sent to your email, so look for that. And thanks so much, Harsha. As somebody who has implemented Klaviyo before and run it at massive, massive scale — you know what you're talking about, and it's a great product. I really appreciate the time and the effort you guys put into it.
Thank you so much — and thank you for being a customer. Thank you so much, everyone. It's been great.
