Progressive delivery gives teams a way to release software without treating every user as an involuntary beta tester. This webinar explains how identity attributes, feature flags, cohort targeting, and delegated control work together to deliver new functionality to the right users at the right time. It also covers the less glamorous part: cleaning up stale flags, separating security fixes from feature changes, and avoiding a delivery system that becomes harder to govern than the code it was meant to control.
.png)
.png)
Hey folks, thanks for joining us today. We're going to be talking about progressive delivery and identity. I'm joined by Adam Zimman, who is a coauthor of the book Progressive Delivery — he's highlighting it right now, which is great. The way things are going to work is he's going to give a short presentation on progressive delivery, and then we're going to jump right into questions. We have some questions already gathered, and if you have questions, feel free to drop them in the Q&A section.
A couple more housekeeping items: this is going to be recorded, so if you need to step away to deal with a work emergency, don't worry — you'll get the full recording in your email. We're also going to be sending one book to whoever asks the best question, which will be between Adam and me to judge. Feel free to bring your questions, and with that, I'll turn it over to you, Adam.
Sounds awesome. Let me bring up a couple of slides — you can see those. Happy to be here, Dan. Thanks so much for asking me to join you for the webinar.
I'm happy to talk about progressive delivery, some of the things we learned writing this book, and some things we think will be of value to FusionAuth customers and basically anybody building software. The premise of the book is really thinking about how you build the right thing for the right people at the right time.
To give a little context on who I am and why I wrote this book with my coauthors: I've been building software for a long time, at some really awesome and interesting companies, working with some incredible developers and people in the industry, and I've had a lot of fun along the way. I've definitely made a lot of mistakes, but I've also had some pretty significant successes. Ultimately, I'd like to think the technology I've worked on has helped make some people's lives better — because after all, that's what we should be doing.
One of the things we do in the book is set a little bit of context to make the topic relatable. We start with a few vignettes, and I'm just going to read one quickly. It's Friday at 9:52 PM. You open an app on your phone to adjust the alarm on your smart speakers — you need to make sure the alarm is set properly because you have to wake up and get to the airport early for a flight. When you open the app, it looks really pretty, like a nice new update. But after twenty minutes of fruitless labor, you realize there's no way to adjust the alarm. You go on Reddit and find out that the company running your smart speakers has released a new app update that has actually reduced the functionality of the app significantly. Then you spend the next hour running around your house trying to find old-school alarm clocks so your kids and your wife will wake up on time and you can make it to the airport. This may or may not have been based on a true life experience of mine, which caused a significant amount of frustration and consternation, as you can imagine.
What we're talking about here is the concept of jerk. You have been jerked by technology. In a physics context, jerk is the third derivative of position — otherwise known as the change in acceleration. In the context of technology, we think about this when technology changes faster than we are capable of accommodating, or faster than we're ready or comfortable with. So think about any time something — an app, a service, some type of technology — has changed in a way that left you shaking your fist at the clouds saying, "Back in my day, we could do things significantly easier." As technology and software has permeated every aspect of our lives, we find ourselves doing this with our light bulbs, our cars, our door locks, not to mention the services we use for connectivity and communication.
In the context of how we think about building software and how we reduce or eliminate technological jerk, we've done a number of things over the decades as we've refined the craft of software delivery. You can think of this as the software delivery life cycle. We started out with spec-driven development, or waterfall — really a completely internally focused process from beginning to end. The developers or engineers defined what was feasible, then went through a long, serialized process to build it. Oftentimes this had very low alignment with user adoption, because they were building things that had never been built before, limited by the manufacturing process around hardware. There was also very low developer autonomy because of significant coupling between what was capable from a hardware perspective and what the software actually needed to do.
From there, we progressed into agile. This is when we saw the decoupling of hardware and software, and one of the primary things agile gave us was that we started thinking a lot more about users — about the user interface, about what the user was going to do and what they needed to do. Those user stories we're all familiar with.
As we progressed further into the cloud, one thing that frustrated developers about agile was that they wanted to ship faster. They felt the agile process was cumbersome. You saw this in the emergence of cloud-native, web-scale companies that pushed the envelope of how fast they could ship — realizing that if they controlled everything from back-end services to the front-end interface, completely independent of the user's local environment, they could ship multiple times a day, multiple times an hour, whenever they wanted. This is where continuous delivery really shined.
One of the challenges there was that we didn't pay as much attention, in those early days of continuous delivery, to what user needs actually were. We thought about it from a perspective of doing small incremental changes, adapting quickly, and that would just solve for the user. What we didn't think about was that with that increased speed, users would actually have a harder and harder time adopting the rapid changes being released.
That's really where progressive delivery comes in: we're trying to make sure we're still paying attention to high developer autonomy, but also doing a better job of achieving better user alignment. With progressive delivery, we asked: what can we do to make sure we're building a process and framework that continues to enable developers to build and deploy without friction, while also enabling users to adopt at a rate that maximizes value? How are they going to get the most out of the things you're shipping? This leads to progressive delivery as the practice of building the right thing for the right people at the right time.
This introduced a third loop to what you may have been familiar with from a DevOps infinite loop perspective. We realized there's a third loop we really need to be paying attention to: the user adoption loop. How do we make sure we're actually paying attention to how users are adopting and consuming the things we're shipping so they're getting the most value from our effort?
To do this, we have two core tenets for progressive delivery. The first is progressive rollout — the idea of progressively exposing who can see new features and when, making sure we're going after the most forward-looking or aggressive early adopters first and getting their feedback before interrupting the workflows of people who need a bit more stability in their experience. The second is radical delegation — simply put, the ability for someone other than the person who designed and built a feature to decide when is the right time to deliver it to the user. That could be anyone from a product manager or product owner all the way down to the individual user themselves, who can personalize and choose when they're ready to adopt new functionality. In order to do this — and here's a bit of foreshadowing — you need to know who the user is in the application session.
This led us to a four A's framework, to help people think about how to appropriately approach building a more progressive delivery environment for their software delivery process.
The first A is abundance: do you have access to all the resources you need and want to continue building and delivering at the pace of your developers? Next is autonomy: do you have the appropriate permissions, access controls, and ability for individuals and small teams to work independently of one another and still contribute and deliver value? The third is alignment: if you've got all this autonomy, you want to make sure you balance it by asking whether you're all still headed in the same direction — are you creating a unified user experience that feels like it was built by the same team? And the fourth is automation: how are you thinking about automation not only from a builder perspective — making systems scalable, reliable, and repeatable — but also from a user perspective? What are you doing to reduce toil in how a user completes a workflow or task? Technology should be the lever where you apply very little pressure and move a lot of weight. How do you make sure you're continuing to revisit your automation practices in the context of your user?
Adam, can you go back real quick?
Yeah, absolutely.
I did promise I would interrupt you a couple of times during this presentation. To me, this feels super foundational — it's always great to have a framework where we can start to build out pieces. But I'm curious about the order. It feels like you go through these in order of importance, or order of implementation, or something like that. I'm curious why alignment isn't, like, double super-bolded, because that feels like the most important one — but maybe that's just my perspective. I wonder if you could speak to that.
The order we chose actually reflects the order in which these concepts emerged historically. If you think about the history of software, the first thing that really changed how we built software was virtualization. Virtualization gave us abundance — suddenly we were no longer limited by the need to talk to a procurement department to get a physical server with a minimum two-week lead time. There wasn't ready access to physical systems, so abundance wasn't there yet.
Beyond that, the next bottleneck was autonomy. Now that we didn't have to wait for system setup, the question became: how do we make sure teams can work independently of one another? The biggest change here was probably the introduction of Git — all of a sudden developers could actually work independently of each other. You didn't have the tremendous cost associated with branching code, and we got a lot better at managing merges and merge conflicts.
What we realized at that point from an industry perspective was that a lot of people, once they had autonomy, started going off in slightly different directions. That's where alignment really came into play. The companies that handled this well realized they needed to spend a lot more time on coordination, communication, and alignment — bringing people back to a north star of what they were really focused on and how to build a cohesive product. With that alignment, you start to see a real acceleration in what you're able to deliver, and that's where you start wanting to optimize for automation — getting the right features to the right users at the right time. Is that a fair statement?
I'd say the best way to think about automation — and interestingly, user-side automation is something we saw very early in software development — is to think back to the late eighties and early nineties and the introduction of the software wizard, originally from Microsoft. That was something where you could just click through; we sort of know what you want to do, and we'll give you an "Advanced..." button to go outside the norm. But from an 80/20 rule perspective, 80% of the time you're just going to click Next all the way through and be done.
More modern automation starts to look like what your favorite ride-sharing app does for you, where it learns your profile. Instead of having you click through seven different windows, if you're looking for a ride home from the office every day at 5:00, it starts to prompt you and say, "Hey, are you going home?" — because it knows that's what you usually do at that time. That's the type of automation and reduction of toil we're talking about: instead of your user interface looking like the cockpit of a space shuttle as you add more feature functionality, it starts to look like a big red easy button — the app knows what you need right now, and you just press one button and it's done.
To your point about Chef and Terraform — that kind of automation, consistency of builds, scalability, things like that, all come into play as well. But those are less on the user side; they're more on the build side.
Totally. One last question, then I'll let you get to the next slide. I love the concept of toil — anyone who's ever watched their parents struggle with entering their password to get to their email totally recognizes it. Does it really differ in any meaningful way from the canonical definition of software toil that SREs have been trying to eliminate, or is it pretty much the same thing?
I think it's pretty much the same thing. We all as humans really dislike nonsensical, repetitive work. Anyone who's had to go through an application process where they've had to manually enter their name fifteen different times knows that computers are really good at automating that kind of work and making it simple. Why wouldn't we, as builders of software systems, take advantage of that to make things easier for people?
I think it is similar to SRE. The reduction of toil on the DevOps side comes from realizing that the more bespoke you make every instance, the less likely you are to successfully scale. That's a similar type of goal.
Cool, thanks.
So I'll get through the last few slides quickly and then we can continue this conversation.
One of the things we point out in the book is that we don't have an expectation of an instantaneous conversion to progressive delivery. What we really encourage folks to do is think about the four A's framework as giving you things to target — you can pick just one to improve, work on that for a while, and then continue to measure and iterate. How do you think about this changing over time? How do you continue to look for where your bottlenecks are? Ultimately, what we want to do is remove the jerk — make sure your software delivery process continues to be mindful of the notion that you have users and you should take care of them.
We'd love folks to check out the book. For the first twenty or so people who registered, we're going to send them a copy. For those who didn't make that cut, you're welcome to pick one up on Amazon or at your book reseller of choice. We'd love a five-star review after you read it. We also have a podcast called Third Loop, produced through Heavybit — take a listen. I'll make sure to get a copy of the link for the webinar notes over to Dan and Blair. And if you know people who'd like to participate in the conversation, reach out — we'd love to get some new and interesting voices. Dan is actually going to join us for a future episode, so looking forward to that.
Yeah, great. Thanks for that overview. As I mentioned in our prep, I had a bunch of easy questions for you, and I love that your presentation takes care of a lot of that definitional stuff — which is really important. I will say you missed a tremendous opportunity to have a Steve Martin meme on that last slide, but other than that it was a great presentation.
That's in the other deck.
Sure, sure. I do want to remind everyone listening in real time: if you didn't make that first-twenty-registrants cut, you still have a chance to win a free copy of the book by asking a question. So let's jump into questions.
We're obviously interested in identity and how it can help you deliver the right feature to the right user at the right time. How do you see that playing in — from either a conceptual standpoint or a technical implementation standpoint?
From a conceptual standpoint, there are two aspects of identity that are both very important, and they're actually related to the two core tenets I mentioned.
One is progressive rollout. How do you target cohorts based on some type of user attribute — or explicitly target specific users — to be in that first wave, second wave, third wave, and so on of your rollout of new feature functionality? The reason we do this is what we call limiting blast radius — you want to reduce harm whenever possible. Who are the users that are most willing and able to take on change? If you're an organization shipping rapidly, you know that sometimes you ship stuff that isn't completely baked — maybe 90% of the way there, with some rough edges, corner cases, or bugs. Who are the people who are still going to be excited for that new feature? Maybe it's the people who asked for it, maybe not. But maybe it's your most active users. Either way, you need to know who they are. Even if you're cohorting based on some attribute, you still need some association between the account and that attribute — whether it's something simple like location or something more complex like demographic information.
The second is radical delegation. If you're going to delegate all the way down to the individual user — or perhaps you're a B2B organization delegating to an admin at a customer company — you need to know who they are. Knowing who your users are is absolutely essential.
On the technical side, there are a lot of different ways to do this, but what we've seen be most successful is through the use of feature flags and some type of feature management tooling, which needs strong support for authorization and identification of users. This is where FusionAuth can come in and be a strong tool to actually enable that.
From a technical perspective, I would imagine you could put an attributed token — so when you get authentication, it says "Dan is a laggard" or "Dan is an early adopter" — and then your back-end services or application would take care of the mechanics: calling the feature flag service, doing the feature flagging, doing some of that other lifting and shifting. But from an identity perspective, once you put that attribute in the token, you might need to change it later, but really it's just a marker and nothing else. Is that a fair statement from a technical perspective?
Absolutely.
From a conceptual perspective, you mentioned a lot of interesting options around grouping users — it could happen at the org level or the individual user level, you could push people into cohorts or target specific users, you could let them self-identify, or you could assign them. That's a lot of choices. How do I pick? And going back to your point about first steps — how do you even start to think about the conceptual side? Because that feels like the tougher piece. The technical piece, yeah, put an attribute in a token — I'm not trying to dismiss it, it's significant — but what's the conceptual piece?
From a conceptual perspective, when I think about features — and I've run a lot of product teams — one of the things I always try to make sure we're doing when thinking about features is asking: who are they for? One of the things I started doing in my time at LaunchDarkly, and that I also saw when I was at GitHub, is thinking not only about who a feature is for, but about it from the standpoint of a progressive rollout. Who is it for initially? And is this something that's only going to be for a small cohort of users because of some specialized need — like the distinction between an individual GitHub developer, a team, and an enterprise? Or is this something that's eventually going to be for everyone?
That's the most basic question to ask first: is this a transitional period, or is this long-term segmentation? Once you figure out which one it is, it starts to fall into place pretty easily in terms of how you want to do that targeting and what criteria to use.
I can pretty easily see the transitional phase — you're rolling out a feature that's 90% baked, you might encounter some edge cases, and you want to send it first to people who are most accepting of those rough edges. But what does long-term segmentation look like? What's an example where I might want that?
A great example of long-term segmentation is package or pricing boundaries — you have a feature you only want available for your team plan but not your individual plan, or available for enterprise but not your professional middle tier. That's probably the most direct one.
Another great real-world use case I encountered while working with BMW at LaunchDarkly: BMW ships and delivers cars all over the planet. Different countries have different laws and regulations about what the software in the car can do — whether video can play, how the engine is tuned — all sorts of things they can now control through software. They can actually segment based on the destination location of where that car is being sent. The coolest part is that because of this, when you move from, say, the United States to the UK, you can actually have a software update applied to your BMW that makes it street legal in the new location.
There's some really cool stuff these organizations are doing with software that used to require a physical replacement or swap-out depending on where you were operating.
Or maybe you just wouldn't — if the cost is too high, you'd just buy a new car, right?
I love seeing things get cheaper. Another question — and just a reminder, folks, feel free to ask questions in the Q&A section — is a bit of a forward-looking one. You laid out the history from waterfall to agile to continuous delivery to progressive delivery. What do you think is next? What comes after progressive delivery?
I think that all depends on how you feel about the current state of AI. There are definitely some teams doing some really interesting stuff with automated code generation, automated code approval, things of that nature. But one of the things we address in the book is the latest AI trends — looking at them from the perspective that AI doesn't make these problems go away. If anything, it actually exacerbates them.
The thing I'll say consistently about AI is that it just takes your preexisting conditions and amplifies them. If you're doing this really well, AI can help you create more value at greater speed for your users. Conversely, if you're not good at paying attention to user needs and you're not delivering things that bring them value, AI is not going to fix that. If anything, it's going to make you go off the rails even faster and make people frustrated with your products.
We've seen a lot of backlash from people who've had a little AI sprinkled on top of their apps, with users who are just like, "How do I turn it off?" I saw an article a couple of weeks ago — one of the fastest-growing search terms on Google is "how do I turn off AI on [X]." There are a lot of people who just didn't get what they wanted. I think it's not that they don't want their lives to be easier — that's the kind of promise of AI — but that notion of what "easier" actually means hasn't necessarily been identified by the people building the software.
As you're saying that, to me that is a real-life example of a jerk — the concept you're talking about, where something changed in my app and I don't know what to do. They thought I wanted AI, and all I want is to use my app better.
I remember having conversations with my grandmother when she was still alive. It was such an upsetting thing for her when she could no longer go into her local bank branch because they got rid of all the tellers. She just didn't want to use an ATM or do online banking — that wasn't where she was at. I think we're seeing a very similar backlash from everyone using AI for their chatbots or customer service: there's this growing trend where people are just saying, "How do I just talk to a human?"
Zero. You're pressing zero on the keypad until you get through.
Because the reality is, if the information was programmatically accessible — if the robot can find it, it was probably already in a help page, support page, or knowledge base article somewhere. And most likely, if I'm trying to call someone or get through to a person, it's because that information wasn't programmatically accessible, or I have some corner case that isn't accounted for.
Right. We don't have a ton of time left, so I wanted to get to one more question. FusionAuth can self-host, and I think there are other applications out there — especially with data sovereignty requirements coming online — where people are much more interested in self-hosting and having control of their data and their experience. A lot of progressive delivery seems to assume a SaaS model, and continuous delivery is certainly more complicated when you're shipping a product that someone else is going to install and run. So in air-gapped or self-hosted environments, how do you architect a progressive delivery system that still works reliably when connectivity is limited or not available at all?
I'm going to go back to another great customer example. One of the organizations I talked to very early on, when we were even initially thinking about progressive delivery, was actually the SmartThings division of Samsung. This is an IoT vendor — think smart light bulbs, smart light switches, home automation. Often these are set up in air-gapped environments, like a private home network that isn't connected to the Internet. A lot of their testing was done in completely isolated environments, and they wanted to make sure they had a way to think about software updates that would still give them a reasonable amount of control and the ability to delegate feature control to actual users.
What they were able to do was implement feature flags where the default state of any update is that things look identical before and after — from a field perspective, everything's the same. After the update is complete, you have an after-state that also has the ability to turn on additional functionality if desired. This is an interesting way to look at it: you're giving power and control to the user to say, "Hey, I actually do want to try that new functionality." And if they don't, that's okay — from an operating perspective, it continues to work as it did before unless they make the active decision to change it.
We encourage anybody doing on-prem or packaged software to think about this: how do you make sure you're not creating unpredictable behavior for your users? The reason they're on-prem is probably because they have requirements around predictability. You should respect that, and look for ways to honor it.
I love that idea and concept — it seems like a great goal. But what if there's a security or performance issue you've discovered where you almost don't want people to be able to skip the upgrade, because there's a foundational issue?
Absolutely. I think there's a distinction between fixing a bug, a security issue, or a CVE versus adding new feature functionality. I would also argue that for most on-prem users, there's a different process and methodology for accepting security updates versus new feature functionality — the ask is really: don't force me to take the new feature along with the security fix.
Fair enough.
Sometimes that can be a bit of a gray area, because if a previous workflow was insecure and you needed to completely revamp it, some of those changes may be unavoidable. But I would still argue that giving users the information and allowing them to make the transition decision is still important. And making that process as easy for them as possible is where you can actually add value.
When you say as easy as possible, do you mean the actual process will be easy, or the before and after states will be as similar as possible? What do you mean by easy?
The process will be easy, and they also still have the ability to trust you — being able to make something happen without their consent is exactly what you want to avoid.
Sure, totally. Do you have time for one last question?
Absolutely.
Great. This isn't a gotcha question, but I could see a situation where progressive delivery might create more risk than it mitigates if you end up with a ton of feature flags, leave them in for years, and never transition people. If you don't have those transitional segments planned out, you just keep creating more and more features, or delegation becomes super fragmented and hard to track who owns what. How do teams know when they've gone too far with progressive delivery? Recognizing that it's early days and probably no one has — but could you, and how would you recognize it?
Trust me, there have been people who have. The number one recommendation we make is to use a feature management platform. You can build your own feature flagging system, but the value you get from a platform provided as a service is that they've thought about a lot of these governance challenges. A lot of that comes with built-in functionality: identifying stale flags, proactively mitigating them, and giving you a single point of understanding and a through-line for what's actually happening in your system.
It also changes the way you build. On one hand, it lets you move faster with greater confidence. But it also means your work isn't done when you deploy to production — your work continues in terms of thinking about your user experience over time. And if something should eventually roll out to 100%, you need to go back and clean up that flag when it's done. When it's rolled out to 100%, remove it — it doesn't need to be there anymore.
Conversely, feature flags are the best thing ever for sunsetting code. You know exactly what the behavior is when that code segment goes away, because you can turn a flag off for just your developers or user acceptance testers, and if something goes sideways you just turn it back on. All of a sudden you have a superpower for code cleanup and reduction of technical debt — and I think that's something people don't necessarily think of when they worry about feature flag sprawl. Yes, it can sprawl, but this is the most amazing tool for reducing that technical debt safely and predictably.
Yeah, I got a shiver when you said you can carve off a piece of functionality and let people experience the application with that piece turned off — and frankly, it becomes very easy to turn it back on when you get screenshots from your support desk, or flip it off entirely if no one's using it. That's fantastic.
And the coolest part is that a lot of the feature management platforms will actually tell you when people have hit that flag — so you know if no one's been using it for weeks or months, and you can safely remove it.
Right. Well, that's great. I love the idea — and this is true whether or not you use progressive delivery — that your work is not done when you ship. Any developer worth their salt knows that, but this just makes it more real, more specific, and gives you more control over it.
My favorite example that software is never done is the fact that we are still sending software updates to the Voyager probe, which is now outside the Oort Cloud. For anybody who thinks their software is done because they shipped their latest version, I've got news for you.
That's hilarious. Well, that's a great place to end it. Adam, thank you so much for joining us. For everyone who joined us either now or in the future, I hope you learned some things — I definitely did. Thank you very much. We'll be following up with the recording and some links, and we appreciate you all joining us at this FusionAuth webinar.
Thanks so much, Dan.
