Lauren Darcey & Eric Kuck: Lessons From Reddit's Android Platform Team to Infinite Retry Labs

Matt Klein: All right, welcome to another episode of Beyond the Noise, Signals, Stories, and Spicy Takes — the show where we dig into the stories of the people shaping the future of app-based computing with a special focus on mobile. I'm your host Matt Klein, co-founder and CTO of Bitdrift, as well as the co-founder of Envoy Proxy. Each episode we'll talk with engineers, founders, and technical leaders who've transformed the way their companies build and understand what's happening inside their systems. We'll dig into the challenges, the breakthroughs, the lessons learned, and wrap it all up with their hottest takes. So let's dive in.

Today I'm excited to have two guests. This is our first two-guest episode, so we'll see how this goes. First, we have Lauren Darcey, who is a seasoned engineering leader, best-selling technical author, and longtime mobile consultant and entrepreneur. Her career spans leading mobile engineering at scale for companies like Reddit and Vivint Smart Home, as well as serving as a principal engineer across various mobile, fintech, and enterprise security teams before moving into engineering leadership. At her new venture, Infinite Retry Labs, she's reimagining early learning and building a cutting-edge, zero-advertising, kid-centric platform created for active wonder rather than passive consumption or autoplay, powered by modern mobile technologies like Kotlin Multiplatform and AI.

Eric Kuck is a veteran principal mobile engineer, longtime open-source author and contributor, and experienced startup builder. He has a strong track record driving mobile platform architecture, modernization, and developer experience at scale for companies like Reddit, and brings extensive experience from early-stage startups and accelerator environments. As CTO of Infinite Retry Labs, he's focused on creating healthier digital experiences for children and is currently deep into fun, kid-focused, curiosity-driven development using Compose Multiplatform.

Thank you Lauren and Eric for joining us.

Eric Kuck: That was a lot of words. Good job

getting through it.

Matt Klein: Yeah. Yeah. Thank you. Thank you. This is our first, first two guest episode. We're gonna have to see how this goes. How I typically like to kick it off is, would love to just learn a little bit about each of your backgrounds, and particularly how you got into the mobile engineering space.

So, you know, it doesn't have to be like a, a detailed career history, but would just love to learn how you got into computing in general. What got you into mobile, what excites you about mobile. So you can do rock, paper, scissors to, to go see who goes first, or you can just choose up, up to you both.

Lauren Darcey: I'll go first on this one.

Matt Klein: Okay.

Lauren Darcey: Um. Because I think that, like, I'm a little older than Eric, and so he can jump in for when, like he started in mobile and I'll be like this point. [00:03:00] Uh, let's see. So, um, I am the child of, computer programmers and software engineers and, biomedical engineers. So it's been a part of my entire life.

From the very beginning I was writing basic programs when I was in second grade type of a thing with my dad. Um, I really enjoyed it. I liked... video gaming even early on, and some of the text-based Telenet games are where I really got going with like writing c plus plus and stuff like that. I ended up out in California in Silicon Valley for college, and then I stayed after that.

Uh, this is pre-mobile and so I basically was in Silicon Valley doing enterprise security and FinTech stuff and all sorts of other consulting at the time. And mobile comes along. I immediately wanted to be writing apps from the very beginning. So BREW was my first... mobile language, [00:04:00] but I basically,

Matt Klein: what is

BREW?

I, I actually, I actually don't know.

Lauren Darcey: Binary runtime environment for wireless.

Matt Klein: Huh, interesting.

Lauren Darcey: It's what the, like the early Verizon flip phones we're using it.

Matt Klein: Is it an operating system or, or like what is it exactly? I mean,

was it,

Lauren Darcey: it wasn't, it was a Qualcomm based framework where they had the app store equivalent of a marketplace and you went through a whole bunch of... build out things in c plus plus you deploy them on Qualcomm hardware.

They're testing it, they're letting you deploy to the various, carriers and even directly to customers in some places.

Matt Klein: And, and just time-wise, this is late nineties? Like early

two thousands?

Lauren Darcey: This,

Matt Klein: or

Lauren Darcey: this is really early two thousands. Like 2001. 2002. Um, but, uh. It was kind of an open secret in Silicon Valley that like you, you didn't need to know much in order to build these little apps.

You needed to know a common programming language in c plus plus, you needed to [00:05:00] know where to get your apps certified and tested and pay a little bit of money for that, and how to work with Qualcomm to be blessed for deployments. But once you got through those, those hurdles, you could be a small indie developer.

But all the indie developers were kind of like over-inflating how big they were and saying they were giant companies because that's the only way that the carriers would take them seriously. So, uh, my husband, who's also a mobile developer, ends up working for a company that's writing, um, the early, S-M-S / M-M-S type clients.

Um, and, I have a vehicle into finding out how this, this whole ecosystem works. So that's when I founded my first company, was making apps in that sort of ecosystem. Like on the indie side of that...

Matt Klein: what did you make? What, what,

uh, what type of apps?

Lauren Darcey: First? First utilities. Like some of the, the first things I built were not just games.

I definitely built some of those. But [00:06:00] like, an international converter of sizes and like unit measurements and stuff like that for travel. So like if you wanted to buy shoes in Japan, knowing what shoe size you were in Japan versus Italy versus like, I don't know, stuff, fun stuff like that, that was pretty easy to build little lookup table for.

Um,

Matt Klein: I, I mean, I would say, you know, even right now I'm still trying to figure out how many tablespoons are in a cup. So, I mean, it's exactly, it's, it's a, it's a pretty useful thing actually.

Lauren Darcey: Um, and then I, I went off and built, uh, some, some games where I gave away all of the proceeds to like my favorite charities and stuff like that.

So Monterey Bay Aquarium was... their Otter exhibit, was partly funded by a Go Fish game that we wrote really early on, um, where you could also have the, like the computer side cheat and just like you do in real life when you play Go Fish. Anyway, so, one thing that I noticed about [00:07:00] that was, as a, uh, woman in tech, it's really frustrating when you find out that there is an easy path if you know a few important details and you want to become an indie developer and there's a really hard path if you are missing some of those key pieces of information.

And, that's actually why I got into tech writing in the first place is because I found I was kind of... building up this huge amount of like special intel that a lot of people were keeping pretty close to the vest because that meant that the ecosystem was small and there was no competition and it was really hard to get and say to onto Verizon phones if you didn't know a few things.

And so as I'm working through these different mobile platforms that come along, they're all kind of playing the same game where you really have to know people and you have to build a network and you have to... pretend you're a bigger company than you are. And they're mostly closed walled garden ecosystems [00:08:00] at that point.

And then Android comes along and it looks like a really, really promising... more open, more, uh, egalitarian sort of posture than you'd seen before.

Matt Klein: And, and, and, and sorry, just for time reference. Yeah. This is like 2000... 6, 7, 8 time, time

range?

Lauren Darcey: Yeah. I started doing Android development in the Android beta, so I think that's 2006? 7?

Matt Klein: In that range, because I think iPhone was probably 2006.

So somewhere in that timeframe ish.

Lauren Darcey: I think 1.0 is 2008, but I'd have

to check.

Matt Klein: Yeah.

Lauren Darcey: It's been so long. Um,

Matt Klein: it's also, I, yeah, I mean it's like pe- people who are listening right now probably can't even wrap their heads around the fact that back then, you know, people still went to bookstores, you know, to buy books.

Lauren Darcey: Yeah.

Matt Klein: Right.

I mean, I'm just saying,

I mean, we are, we are going back in time here, so,

Lauren Darcey: yep. And we were, we were the, like last generation of tech writers where, the print book mattered more than, there wasn't really like the digital [00:09:00] option was this nice... like came out on a CD the first few times we published our book, and then they started to- like actually hosting them and trying to, uh, ship them online.

But that was how I ended up really digging into Android in a way that I had, I had dabbled in J2ME, Symbian, uh, and Android and iOS at the same time. Um, and a Blackberry, a few others along the way. But Android is where I started actually like watching every single release that they were- like trying every single API that would come out and trying to basically build out examples for people because nobody was, building out really robust developer portals at this point.

No one is, is explaining how any of this stuff works. And so you have to basically look at it and figure it out for yourselves and... that's how some of the worst anti-patterns come out of mobile in the first place is because there was no early documentation, or like best practices and stuff like that.

But it's [00:10:00] also, it was a really, really fun time in which like you really had to discover your way into the platforms in a way that you don't now. So anyway, that's how I get into Android. That's how I end up writing a whole bunch of books. And then I've enjoyed the Android ecosystem and the Android developer community so much that like, I've mostly stayed in it or adjacent to it for most of my career at this point.

Although I've also expanded into engineering leadership on like, including more like mobile generally. So iOS as well.

Matt Klein: Yeah.

Lauren Darcey: And having back- backend teams with my teams and stuff like that.

Matt Klein: What, what, what made you do Android in the beginning as opposed to iPhone?

Lauren Darcey: Uh, I really liked Java as a programming language, and it was one of my strongest programming languages.

So that was one of the reasons it was just easy. And I think the, like open versus closed

Matt Klein: Yeah.

Lauren Darcey: Vibe made a huge impact on me. And I think it's funny how that hasn't really changed over time like that, that, that

Matt Klein: it hasn't, [00:11:00] yeah,

Lauren Darcey: that difference still exists. But multi-platform stuff also just like throws a lot of that up in the air in fun ways, and it's been fun to watch the, like the way the platforms have both

gained influence over mobile in different ways, but also, uh, that that changes every few years in terms of like, who's ahead on certain types of technologies and stuff.

Matt Klein: Yeah, I, I've been asking this to almost every guest, um, but it is interesting to me that obviously now, I mean, you know, you have the Android platform, iOS platform.

They're both huge. I mean, they both have tons of apps and tons of developers. But the developer community, at least for me on the outside, I'm not a core mobile developer. It does seem healthier, just like from an openness perspective. It just, just does seem like there's more, I don't know, people around it who are contributing in the way that you're talking about.

And I, I can only believe that it is just like [00:12:00] general open source communities where how it was started in the beginning was a bit more open. Um, I'm not sure if that's been your experience or not.

Lauren Darcey: I, I definitely think it was in fact there, like I've talked about this a little bit before, but... a lot of the reason that we open sourced all of the code for our books was simply so that we would be able to use it again ourselves in a time in which, like, if you wrote some code, chances are your... your employer was going to assert their ownership over that.

And

Matt Klein: for sure

Lauren Darcey: if you were in consulting like I was, that meant you would build something that was boilerplate and then find yourself handcuffed and not being able to use it. So there were a lot more benefits to actually like working in the open than people feel now. But that was really, really, really early open source.

Like you didn't even have like a Git community yet.

Matt Klein: Yeah.

Lauren Darcey: Um, but that was the kind of a really great set- like it was a, it was really great time because there were [00:13:00] not a lot of mobile developers out there. You- it was a very like lucrative funds place to be. We're usually building apps that were pretty cool and interesting.

And, a lot of the big players had fun accelerator programs and they had fun incubator programs and it still didn't take much to build a team and come up with a... if you had a good idea and you could execute against it, um, it was harder to navigate the, like publishing ecosystem than it was to actually like build something fun.

Matt Klein: Yeah.

Lauren Darcey: Which I think is a good segue for Eric and how he ended up in mobile.

Matt Klein: Yeah.

Yeah. Let's hear about you, Eric. How, how did you get into it and how did you find each other over at Reddit?

Eric Kuck: Yeah. So, my... I guess how I got into computing is, the exact opposite of Laurie. So my, my parents were both in education.

I went to a... not the best high school. Uh, you know, all my, all, all my people I work with now talk about all these, you know, high school classes that they took. And they came out knowing all these languages. And [00:14:00] my high school had Mavis Beacon Teaches Typing. That was it. So I got into, software engineering really just 'cause I, I like computers.

I didn't know what that meant. I just knew I like computers. So I, I decided to get a degree in computer engineering. Like I said, no idea what it meant at the time, but, uh, in retrospect, great decision. And the way I actually got into software was really kind of by accident. So I, I studied computer engineering, which is, you know, like half software, half hardware.

I, I love microcontrollers and kind of the lower level programming. My very first job was at Motorola, and, I was supposed to be working on their embedded systems. And I showed up the first day and they told me, you know what, actually we are like starting this new thing, Android, you've probably heard of it.

We noticed that you took some mobile classes in college. You wanna just move over? Uh, actually they didn't even ask. They told me I'm moving over. So I, I joined this team where everybody was just learning this brand new thing. Their, their entire careers had been in P2K, [00:15:00] like the, the, the RAZR OS.

Matt Klein: Yeah, yeah.

Eric Kuck: Um, and kind of what they had used for forever before that as well. So I was the only one on the team with any kind of experience. Kinda my favorite example of that is we had like a, a very senior engineer at the time, demo one of his apps. And it was, like... so in, in Android you have activities, right?

You can go from activity to activity net we're beyond that now. But that at the time, that's what it was. And he... there was no understanding of that. It was just like set view. Set view. So like you click a button and you go to a new screen and it was just set view. So you would have like one activity with tens or hundreds of thousands of lines and, you know, that's, people didn't know any better.

So that's kind of the, the environment I was walking into. So like, very quickly got into kind of leading teams and showing people how to do it simply because I had an interest and had been doing it a little bit on the side and took a college class and, you know, very minimal qualifications. But at the time, that's all you needed.

Kind of, yeah. Getting back to what Laurie said, the, the, [00:16:00] the bar to entry was a lot lower. There was a lot less information out there. We didn't have, you know, all these books. We- like Motorola had this big library and there was a lot of stuff on how to program Java. There was nothing on how to program Android,

right? Like, it was, it was just brand new for us. So from there I was, I was, you know, working at Motorola also doing a few things on the side. I, published a few indie apps that, that took off and I realized could support me. So I ended up working at Motorola, it was like nine months total, before I kind of went off and did the indie thing.

You know,

Matt Klein: what was your,

Eric Kuck: I had a,

Matt Klein: what

was your favorite app from that time?

Eric Kuck: I won't say favorite, most successful, uh, which I'm slightly embarrassed about these days, but, so what I did... android was compared to iOS, a very... immature app ecosystem. You know, all the best apps run Android or run iOS.

It had far more, it had far better quality. So what I did at the time, I didn't have any ideas. I just knew I wanted to make an app. So I went to the iOS app store, looked at what the top apps were compared, what's [00:17:00] missing from Android, right? And the, the number one thing I found was that iOS had great apps to cheat at Words with Friends.

I did not play Words with Friends myself at all, so I really never used my own app, but that's what I did. I figured out how to do OCR on Android. Like I made my own OCR engine. I made like, a, a very fast way to look up dictionary, you know to, to scan what letters you have and what spaces there are and look it up in, in the dictionary.

And, it ended up being very successful. Like I said, not, you know, my proudest moment, but it did allow me to, you know, kind of get my career started in the indie space and to, to bridge my way into, the startup world. Which up until Reddit is really where I spent most of my time, I, I did not love the... big company environment between internships and my experience at Motorola.

Um, you know, the process and moving slowly, you know, you're young, you're right outta college. You just wanna build stuff, right? And so, especially at the [00:18:00] time that, that wasn't for me. So I got really into to early stage startups. I went through, accelerators, worked with a lot of startups in Chicago where I lived at the time.

And then eventually, built a, an app development company. I was the CTO there. And, you know, I had multiple employees still working on the early stage startup projects. Um, you name it, Uber for insert anything dog walking. We, we actually did build that. Um, that that's, you know, that's just what it was at the time.

So I don't know. I, I like, I, I thought that was gonna be my whole career. I loved the early stage startup space. I love owning the entire stack. You know, I, I was doing all the design. I was like, you know, in Photoshop at the time that that's what we were using for design. So I would, you know, Photoshop it.

I would, uh, build it. I would do all the testing every single bit. And I loved it. It was fun. And then, you know, Reddit came calling, I, uh, I, I, I had a bunch of open source work out there, from my time when I was running an app development company. And Reddit happened to [00:19:00] be a big user of one of my projects.

So they had heard of me and

Matt Klein: which was that,

which project?

Eric Kuck: It was called Conductor. It was, an alternative to, I guess, Fragments on the, on, on Android. So, you know, how to navigate from screen to screen.

Matt Klein: Okay.

Eric Kuck: And, you know, re- Reddit came calling. I, I've always been a huge Reddit fan, uh, just as a user and as much as I never wanted to work at a large company again, I, I don't know, in my head was an exception, so I took 'em up on their offer.

And yeah, I, I started, I, I worked on multiple teams there, multiple roles, but, uh, ultimately ended up, working with Laurie on the Android platform team. And, I don't know, I guess the rest is history. That's also where we met you, so, uh,

Matt Klein: yeah. Yeah.

Eric Kuck: It was

a good time.

Matt Klein: Yeah. I mean, you know, it's, um, I, I also have gone through my career and I've, worked at very large companies.

I, I've worked at very small companies. I've started now two companies. And I think it's interesting because... I think that [00:20:00] to be well-rounded in general, it is actually useful you, you know, to work at some of these giant places and also do your own thing. I mean, I, I'm probably not surprisingly, like both of you, I tend to flourish in a smaller, more fast moving environment.

But, but I would not be the engineer that I am today without working at some of the larger companies that I've, that I've worked at, you know, just through exposure when I was younger to very senior people and to bigger problems and all of those things. So, you know, it, it's just fun how that goes.

But I, I would assume that you both in your time there saw um, a lot of big issues, you know, just from an engineering perspective, you know, that you might not have seen previously. So would love to, to learn more, just even about that time. You know, what were some of the biggest challenges that you were facing and working on?

Eric Kuck: Yeah, there's a, I don't know, the, my whole time in early stage [00:21:00] startups, I had always heard these problems about at scale. And I guess in my head it was like, well, I mean, it's a mobile app. It doesn't matter how many people are using it, it's still just on the device. Like scale doesn't matter. Um, I found out very quickly that was, uh, a, a completely incorrect assumption.

Um, just everything changes at scale, right? Like the, the issue that affects, you know, 0.01% of your users is suddenly a lot of users and you need to care about it. Suddenly you have all these users in, in countries that you never knew existed, or, you know, devices from 10 years ago that you had no idea people were still using.

Matt Klein: Yep.

Eric Kuck: So I don't know. I'm trying to think of like, issues we had. I think the, the one that always comes to mind is... sRM what, w- what, what's the more general term for that, Laurie? The,

Lauren Darcey: the cross exposure stuff?

Eric Kuck: Yes, cross exposure. So, you know, at like any company of a decent size, we, you know, put feature flags in and, would make sure that only a subset of users are exposed to different experiments and, you know, get access to things [00:22:00] while we validate that they're working as expected while we validate that they're actually something users want.

And cross exposure is where, you know, the, the backend will tell the app, like, 20% of the users should be in bucket A and 80% should be in bucket B. Well, sometimes you'll end up with like 25% reporting they're in bucket A, which when you have a very small amount of users is maybe just, you know, within a margin of error and you're okay with it.

But when you have millions of users, it's kind of obvious that there's a big problem there. And you know that that's kind of an example of a, a, a problem at scale that I never would've expected before. Also happened to be by far our hardest engineering problem that we ever had to solve. It took...

years, I think.

Matt Klein: Um, and, and, and, and did that, you know, just because I'm personally curious, wa was that a bug in your feature flagging system? Or, or like, was there bias in the assignments? I mean, if you're willing to share that, that sounds like a fun one to actually learn more about. [00:23:00]

Eric Kuck: Yeah, there were, many bugs across many systems that contributed to this.

Um, you know, we would, we, we had, I, I have no idea how many engineers were working on this problem on and off, but we would have someone work on it and say like, 'Hey, I found the fix.' And they would push it out and it would kind of work, but not really. And then we'd have someone else work on it and, you know, 'I found the fix' and it was just kind of this cycle on repeat.

And we could notice patterns. Like we noticed that it affected experiments that were read very early on in the startup of the app, more so than it affected experiments that were read in the middle of a, of a session. So, you know, we, we had clues to work with. We figured it was something around timing or some kind of race conditions, but, I don't know.

Yeah. Laurie, you, you kind of led the project at, at many points. Do you wanna maybe talk about it?

Lauren Darcey: I think this is an example of... definitely a lot of race conditions at scale, but also a, an area of the code base that nobody [00:24:00] owned, nobody knew anymore, everyone was afraid of, had no tests around it and had been allowed to get extremely crufty and hard to work in such that you had no clean paths to debug anymore.

So every single person would go in there and find all sorts of things wrong, and when you moved one thing, a whole bunch of other things would happen.

Matt Klein: Yeah.

Lauren Darcey: And so, and, and you didn't have the ability to really change the environment that you were working in. You had to try and find a, like, surgical solution to the problem that we had at the time.

And all of the people who went in there found really, really big, gnarly problems to solve. They all net improved the situation, but it didn't stop the fact that, like that particular area of the code base. Should have been rewritten long, long before anyone had to solve this sort of problem in it. And it was also never really built for that.

Just like the [00:25:00] experiment framework had never really been built at a time where it, where we thought it was going to be under that, that much pressure in the first place, I guess. So there was a lot of, 'huh we wouldn't have built it this way if we had known,' but also no one on the team owned it. They were all being deployed into it, which meant it was a rotating set of people who would find a bunch of interesting stuff, solve part of the problem, and then we'd have to watch for weeks because it takes forever to get the data.

Matt Klein: I was gonna say, I, I mean it, it's... you know, the, the bar, I feel like for making this type of code correct, is very high, because as, as you just pointed out, and I was gonna say the same thing, is that, I mean, not only do you have to wait weeks for, you know, doing actual releases, but if you're talking about experimentation, then you have to wait for people to do experiments and then wait for significance.

And it's like, I would just imagine that it could be two months between making any change and being like, 'ah, oops. Like, ah, it's a little better, [00:26:00] but it's still broken,' you know? So I, I can see how it would take years to ultimately fix something like this.

Lauren Darcey: Not, not only that, it was like these were tiny, tiny, percentages of users who had hit this problem.

And so you might get one every, like, few weeks that would actually hit it enough to actually, like, you had to get the right experiment and then it had to

Matt Klein: right,

Lauren Darcey: set up in the right way and had the right time and the right build. So, but the at scale stuff that we learned, I think that this is a really, this is a, this is the part I liked most actually.

I think one thing that people don't really realize is that, even companies as big as places like Reddit are just building operational excellence as they go.

Matt Klein: For sure.

Lauren Darcey: As a, as like walking in the door as a former consultant who'd seen many, many companies and how they were trying to fix their problems, and then joining them in their leadership...

they didn't have on-call programs for mobile because there was an assumption that mobile teams never get that [00:27:00] big, they don't need SRE... observability is kind of an after fact for them because they can just, I don't know, go look- like it's the same... what Eric said. It's the same for everyone. It's the, you're all individual little users and you don't really impact things.

But when you work on an app at scale, you realize that you can, cause... you can learn really, really fast. Like the easy way or the hard way. Like, um, because you deploy to so many people at once, you get signals incredibly quickly about whether or not what you have deployed works or not. If you're listening and if you have good observability.

Matt Klein: Yeah.

Lauren Darcey: Uh, or you can dark deploy and be happily unaware that things are absolutely going pear shaped.

Matt Klein: Yep.

Lauren Darcey: And so there's, there's this really great learning curve that you have at scale when you realize that you can deploy something and get great signals really, really fast and make changes really, really quickly to learn how to build better, or to recover faster and those types of things.

And you learn that... like trial by firing it at [00:28:00] scale or in an incredibly slow track on a small, at a smaller company.

Matt Klein: Yeah.

Lauren Darcey: And so when I think about the like, funny, messy stuff that happened at scale that you don't see as much everywhere else, it's things like... you have an experiment that's going sideways and you think, 'oh, it's easy

we just turn off that flag and go back to the default behavior.' But you forget that this was a migration between old infrastructure and new infrastructure. So you turn it off and you just shed all of your traffic back onto the old infrastructure way too fast and that goes down. So assumptions that like those things would be easier if you don't have scale...

suddenly you realize that you have a fire hose that you're kind of directing in different directions and you have to take different steps to, to like make sure that those things stay stable in a much bigger ecosystem.

Matt Klein: Yeah.

Lauren Darcey: Um.

Matt Klein: I mean, there's, there's scale obviously on, on two different sides. There's, you know, the extremely large [00:29:00] number of users.

That's obviously one aspect of scale. But then also, I mean, I'm sure you had a lot of developers who were working on this code, so there's the scale of the code base itself. And, and I guess o- one thing that I'm wondering is, at least from the stories that you've had so far... I mean, I would imagine that there were internal, just like developer issues with scale that you also had to deal with, you know, that big companies typically face.

But I mean, would you say that during your time there was the, was the scale scale like the, the external user scale, was that a, a bigger area of focus versus like the internal scale of the app? Or were you focusing on both sides?

Lauren Darcey: I, I would think we were focusing on both sides. So we, we were on... in a platform org, which meant we had a dual purpose.

We owned the overall app quality and so crash rates and stability rates, and we were gonna be the ones that were gonna be called by the on-call teams if [00:30:00] mobile was involved. In fact, that's one of my first, experiences with Eric was just, he was being like parachuted into every incident that had mobile associated with it as like, like bringing a gun to a knife fight sort of a problem.

And it was an indicator that we needed better on-call programs in general, where people owned their own outcomes in prod. Part of the reason Eric joined the platform team was because the internal developer experience at Reddit was not good. It was known like outside of Reddit that it was not good.

Matt Klein: Sorry, how is it known outside of Reddit that it was not

good?

Lauren Darcey: Because you ask what's it like to work at Reddit on the code base? And they're like, well builds are like two and a half hours long on a good day. And

Matt Klein: yeah.

Lauren Darcey: Um, I have a really hard time deploying my single line, like null pointer exception fix.

Those types of problems were pervasive and that's part of the reason pla- the like platform org was

Matt Klein: Yeah

Lauren Darcey: built out was because they were under a [00:31:00] hypergrowth path for building out their internal engineering organizations. And they were feeling the pain of growing internally, just like they were feeling the pain of becoming an app, that scales to the whole globe.

Matt Klein: Yeah. And I, I mean from the internal developer productivity, I, I'm assuming that's all the normal issues that has been talked about a lot in terms of dealing with build and build caching and IDE speed and all of those things. Um, I mean, is that, is that a pretty accurate representation?

Eric Kuck: That was definitely a large focus.

There were other things too. Um, it started as a very small team. A team that could keep the whole app in their head all at once. And if you kind of know where everything is, the organization is a little less important 'cause it's in your head. You kind of know where to look, right?

Matt Klein: Sure.

Eric Kuck: But once you get to the scale of, you know, 60, 80 something developers and, many, many hundreds of modules, there's no way anyone can ever keep it inside their heads.

So org- [00:32:00] organization and architecture, and, familiarity and, and common patterns become much, much more important. So that was actually a big part of it too, was, kind of bringing everyone on board with doing things the same way. You know, it, it was, it was kind of a dual effort to, to modernize the way we built the app and also to just bring it all together and, and keep it uniform so that if I work on feature A and I get, you know, dumped on, you know, I, I have to work, I have to insert a feature into, I don't know, the tea, the, the feed, the main thing you see when you open the app.

Ideally I would- that would look familiar and I would kind of know how to do it, right? And it wouldn't be learning a brand new code base in order to insert my little thing into the feed. So yeah, I, I, I think a lot of it was things like build times and just keeping developers happy.

Matt Klein: Yeah.

Eric Kuck: But, there was also a lot of work that just went into making it not...

terrible to know where to put your code or how to write your code.

Lauren Darcey: I, I think one of the things that [00:33:00] happened there that, uh, I learned a lot of lessons about tech debt from companies like Reddit. And, it came down to if you put off your devex, your teams would find a way to ship and you might not like how that ultimately shows up in your code base or your processes.

Eric and I are both very, uh, like, we like the ability to... not follow any rules that don't feel like they serve, like they, like, there can be some guidelines, but we don't like to apply them too heavily on others or on ourselves or have to navigate in any that don't seem like they're actually serving a real purpose.

And, when you have really bad devex, you start seeing people that are like putting massive PRs through because they only wanna have to go through once

Matt Klein: Yeah

Lauren Darcey: as an example

Matt Klein: for sure.

Lauren Darcey: Or, uh, they're, they're not willing to work in the same repo and follow the same rules and testing as everyone else because they need their fast cycle times on their builds.

And so they have literally gone off and just like built their [00:34:00] own little bubble. Uh, Eric is one of those people. And so, so when you, when you start having a vision of like... and, and you're also potentially hiring a ton of contractors who need to be able to onboard successfully, quickly.

Matt Klein: Yeah.

Lauren Darcey: And, you're not necessarily going to invest a ton in their training and development along the way.

You need some guardrails and so... that's a really interesting area in which, I think we did a really good job... providing a set of rules and guidelines that helped everyone build better together and struggle less with the code base. But also we weren't coming in and saying like, you've violated the rule of how you're supposed to set up certain things.

It was more like, if you follow our, our golden path, you're gonna get really good tools from us that make your lives easier, that allow you to have high ownership of your areas. But yeah, we're gonna ask you to join the monorepo and we're gonna put some tools up that like, give you really easy access to, some sample apps [00:35:00] and things that make you able to do those fast cycles and stuff that kind of separate you from others, but also, yeah, we want you playing in the same pool because then we can actually start applying some rules and stuff that probably have set us up really well for, more consistency, which helps with...

AI integration, other sorts of things down the road where if you're shipping your org chart all over the place with, we had 800 or more modules at like last count, I think. And not everyone knew anymore, like mo- many more modules than people, many more modules than teams. And, no real easy way to like, take things out once they were in.

Those are the kinds of capabilities we were after was like,

Matt Klein: yeah.

Lauren Darcey: Making it easier to kind of tease out what we have so we can actually ship the parts that we think are the most important to users.

Matt Klein: Yeah, I mean, I think from a platform team perspective, at least from what I've seen for a platform team to be successful, you know, it's like any other product you have to meet your customers where they are.

You know, you have [00:36:00] to, you have to dangle some carrots in order to actually get them in there for sure. Prior to talking about your new venture, I, I do wanna touch on something that you talked about before because something actually that we haven't talked about yet on, on the podcast and I, I think is super interesting is the mobile on call the reliability aspect.

And, and the one thing that I'd love to ask you both about, and you know this part is probably not controversial, is that on the server side of the world, right, we now have 20 or 30 years, I guess, of people learning how to operate systems and like a general appreciation of on-call and like how you do SRE and all of these things.

And I've really been wondering recently with... clearly how important mobile is to literally how everyone uses computers now. I mean, it's like nine, you know, you talk to many companies, 10% web, 90% mobile. I mean, it's like, it is true. The mobile revolution is complete, [00:37:00] right? Everyone accesses computing through mobile and it continues to utterly shock me

that still to this day, you know, in my opinion, there's a fairly poor appreciation of like what it means to have that end-to-end reliability, right? Like there's no such thing as an MRE, a mobile reliability engineer, right? And I, I, I would just love to hear from you both, I guess A. what you think about that statement.

Is it true or not? And you know, since it seems like when you were at Reddit, you did spend time thinking about this in terms of on-call and how the on-calls interact with the rest of the system. Or not system, but the rest of the company. I just, I, I'd love to learn more about how you think about this.

Who wants

to go first?

Lauren Darcey: ...much so, Eric, do you have a thought before I go off on why various rants?

Eric Kuck: Um, I think maybe you should go first.

Lauren Darcey: Okay. Uh, I, I can [00:38:00] rewind to a place like, uh, Vivint Smart Home for this too, because

Matt Klein: Sure.

Lauren Darcey: Um, like they're an iot company, a smart home company, and so, they actually had, backend platform teams embedded with their mobile folks quite often in a way that I didn't see it so much at Reddit.

And, they had specific... high points of the year, like seasonal times in which things got really dicey in which they were great post-mortem opportunities later. As an example, smart homes often have doorbell cameras and they get wrung a lot on Halloween and on Thanksgiving and on Christmas Day, in ways that you don't see that sort of traffic any other time of the year.

Other companies in retail care a lot about Black Friday and certain sales and those types of things. And so, I learned a lot about really good operational excellence on a team from having those folks embedded together. And, when I joined Reddit, we were dark deploying weekly, to a hundred [00:39:00] percent.

And, and not really looking at our observability and not really, like, not even aware that we were causing challenges until users would tell us days later after they'd adopted the releases. And by that point it's really, really hard to navigate. And so very early on when building out the, the on-call experience for the platform teams, we basically went and stole all the good ideas from the SRE teams.

And just because we couldn't get alignment on actually having those resources, we just kind of stole their incident playbooks and we stole those things and adapted them to mobile. And what's interesting is there are a few places where we differ a lot. Backend teams are used to being able to deploy small pieces of work.

Matt Klein: Mm-hmm.

Lauren Darcey: And then revert them. And mobile packages up a gazillion PRs every week and ships 'em all together at once, through an app store review process that can be a, a real [00:40:00] roller coaster and a, like a delay in, in, uh, reaching prod that is... unclear sometimes how quickly you'll get through that. Even for big company.

Matt Klein: Yeah.

Lauren Darcey: It's the same- like that can be all stores. So what do you do when you don't have the, the ability to revert a change in prod really easily? You build very extensive feature flagging abilities, very extensive processes around vetting all of the code together, and different testing scenarios for stuff like that.

But, I think that we made the most progress when we stole some of the like, foundational SRE... ideas. Blameless postmortems, making simple changes whenever possible and isolating them from other things, especially for hot fixes in situations like that. But we would always run into this, this, issue that I still think, um, is really interesting and that is backend

SRE teams have not learned how mobile is different. [00:41:00] Mobile sr- mobile teams have not learned how backend is different to an extent. And there's so much middle ground in there where we could be innovating. But we're not. Because those groups are hardly ever like experiencing the pain in the like, similar ways.

It got to a point where we would have to write documentation that would say you cannot revert a mobile release once it has gone through the app store. You may only fix forward. That is the way it works. But every single incident we would have, if you had an SRE show up, they'd be like, why don't you just revert that little change?

It's like, you can't, there is, that is not an option in this system.

Matt Klein: Yeah.

Lauren Darcey: So I think it, it's, it's one of those areas where I feel like there's so much cross training, like capability, but instead everyone builds their own observability in their own different places.

Matt Klein: Mm-hmm.

Lauren Darcey: They don't know what's happening to each other.

And, quite frequently, one hand doesn't know that the other has deployed or not in [00:42:00] both directions. Um, we see a lot of that.

Matt Klein: It's even, I, I, you know, and I'll segue very, very briefly into a company anecdote, which is that, you know, we have feature requests from people where it'll go something like our mobile, people want to look in bitdrift, right?

But like the server people don't, so it's like, can we, you know, get the data into this thing or that thing? And, and that's fine. I, I mean, I'm not like commenting on that. Like, I get that, you know, there's some segment of the industry that wants the theoretical single pane of glass, which I don't think will ever exist.

And then there's other people that want to, you know, go into different systems. I, I mostly bring that up, which is, it's just, it's, it's an indication, and this is an anecdote for some very large companies that... They still to this day don't really talk to each other. I mean, it's like they operate in completely different systems and I just, I find it a very interesting, it's not even a technical [00:43:00] question, it's like a management leadership, just kind of like organizational question that I continue to find fascinating because it seems so counterproductive for, for where we want the end user to experience like a, a, a better experience, right?

Lauren Darcey: It, it's worse than that even. We did a whole bunch of really large scale migrations and you would end up having to write runbooks for like the old infrastructure and the new, and you couldn't even debug unless you're like, first, are we in a our rest endpoint or a GraphQL endpoint,

Matt Klein: right

Lauren Darcey: second, like, and you would, you would actually page different people depending on how that would work.

And sometimes that would end up in really funny places. Like, your, turns out your feed is being served off of one infrastructure and your ads are being served off another infrastructure and one of 'em goes down and suddenly your app is only ads... got roasted for that. Pretty, pretty, uh, very reasonably. Um, but those are just like, [00:44:00] those are the artifacts of those things.

And so you hear about those things and you're like, I don't even know who I'm supposed to page at this point. Like the one that's up isn't causing the problem. It's the one that's down that, the missing part. But, uh, lots to learn in ob- observability with mobile, but also you can take it too far. I think one of, one of the areas that was frustrating for us was just...

when you have reason- good enough observability to tell you that something is wrong. You probably don't need three more digits of granularity to tell you what you should do, what the action is.

Matt Klein: Sure.

Yep.

Lauren Darcey: And I'm, I'm old enough to have felt that like you don't put too much on the mobile side for, for like, you don't turn on your like really, really verbose logging and like, just because you can now and you can send that kind of stuff and it's nearly free from a data perspective.

Matt Klein: Yep.

Lauren Darcey: Still... impacting your performance.

Matt Klein: Mm-hmm.

Lauren Darcey: Uh, all of the like, ad data is impacting your performance and those things add up. So there have been times where we found that removing [00:45:00] some of those things actually improved, the, like, overall outcomes for users that we were trying to improve by like looking into those.

Matt Klein: Yeah. You're obviously preaching to the choir here, but

Lauren Darcey: Yeah.

Matt Klein: Anyway, could, could talk about this all day, but I, I do wanna talk about your new venture, but before we do that, Eric, did you have anything to add on, on this topic?

Eric Kuck: I mean, I- I guess I don't have a ton. I think, a lot of mobile developers have this, standpoint that, you know, I, I don't need to be on call.

The app isn't changing. Like, I can't push right now. Anyway, the reality is, I, I don't think I've ever not been on call, even though I haven't thought of it that way. You know, even when you're an indie developer, like, you're spending a lot of time looking at your reviews and looking at your analytics

Matt Klein: Sure.

Eric Kuck: And trying to figure out if, you know there's something going on. I don't know. It's, it, it feels a little more unique and probably it, it feels less used than I think it actually is. But yeah, I, I, I don't know. I, I think like formalizing it was fantastic. I think, you know, if anyone's listening and [00:46:00] doesn't have a formal on call program, um, I think a developer,

Matt Klein: I think lots of

people don't, lots

Eric Kuck: No, I know

Matt Klein: lots of people don't.

Eric Kuck: You as, as a mobile developer, maybe you have this gut feeling of like, I, I don't wanna be on call. Like I, I want my nights and weekends. But the reality is you probably are to some extent anyway. And formalizing it just helps you know so much. Having the process that you can follow where you know who to call and who to talk to, um, is just so valuable.

Matt Klein: Yeah.

Lauren Darcey: I, I wanna plus one that real quick. We ended up having way better morale when we had a dedicated person who was in front of that sort of like interrupts and surprises and drama and their only job was to stabilize things and be the person in the room that like helps make sure to coordinate the things that need to happen.

And that that was a legit role for an engineer. The teams that didn't have those things were often the ones that were burning out. And, it [00:47:00] felt like they were never able to step away or take a vacation or, and it always ended up falling on the same few people that got really good at debugging those types of problems in prod.

And, that is a whole different set of cultural problems. So I think that like, for that reason alone, having a like set of dedicated plans around how you solve things... over time, we got so good at solving them. And so, uh, terse about how we would talk about them in Slack and stuff like that, that they didn't escalate into the big problems that consume whole teams or cause really, really big problems.

They would be solved before anyone even realized that they had- like that they were a seed of a problem.

Matt Klein: Yeah,

sure.

Lauren Darcey: And you don't learn that sort of thing if you don't build out a few like simple runbooks and few simple plans. You know who to call and you know how to start an incident and

Matt Klein: I agree. Yep.

Lauren Darcey: Um,

Matt Klein: yeah.

Lauren Darcey: But yeah.

Matt Klein: Um, I, [00:48:00] I apologize for taking this long to, to get to your new venture and we're probably gonna have to do a follow up episode at, at some point where, talk about that. But could you just, I guess, briefly tell us what you're both up to now?

Lauren Darcey: Sure. So we are building an observative- observability platform for kids.

I, I think that there... were both parents and, uh,

Eric Kuck: wait, can we explain what observability means? Because in the context of this podcast I think we mean something else

Lauren Darcey: Sure. Why don't you explain it then, Eric, go for it.

Eric Kuck: No, no, I, no, keep going. I wasn't trying to interrupt. Um, just the, the word observability probably means something different here. Um,

Lauren Darcey: yeah,

Eric Kuck: yeah, go ahead.

Lauren Darcey: Okay. So, so, uh, as parents, we've noticed that the, app options for parents and kids pretty much suck.

They kind of always have, um,

Matt Klein: I,

I agree

as a, as a parent as well, it's pretty

bad.

Lauren Darcey: We, we, as mobile developers, our answer has often been to just build them [00:49:00] for our own kids. And, that's definitely something that's happened. But you learn a lot about mobile in general when you build for a...

really underserved type of user. A user that can't nav by reading and doesn't know how to count. And you, you, I think it makes you a better engineer and you're solving really interesting problems when you are forced to not use default answers to all of these things that have, if you've been around in mobile since the beginning, you know that a lot of those are like accidental.

But they were just like, yeah, we got two hamburger menus and like stuck with them. They're ugly and functional, but like, you probably wouldn't build that now if you were starting from scratch. They, they exist because that's where we got with the like device capabilities and the resources we had at the time and they stuck and now people are used to them.

And there's a lot about the way, even in the indie community, the way the, uh, the developer ecosystems work that encourage... a lot of behaviors that [00:50:00] are not really healthy for kids. They're not healthy for adults either, but we know very clearly that they're not really good for kids. So you add ads and suddenly you're an engagement, and stickiness company and your business model is basically trying to sell one customer to another.

Matt Klein: Yeah.

Lauren Darcey: Lots of things like that. And so, if you were going to look for a giant market to disrupt that has lots and lots of opportunity in it, and possibly a place that you could build a lot of fun things, especially with future facing technologies, then that one looks pretty good as a place to point yourself as a startup.

Um, I dunno. Eric, is it fun?

Eric Kuck: It's definitely fun. We are experimenting with a lot of things that we never would've gotten to otherwise. We're not going, you know, all the way back to skeuomorphism or anything like that from, from the early iOS days. But, you know, like they, they did that for a reason, right?

Like there was a reason that the notes app looked like a sheet of paper. There was a reason that the contacts app [00:51:00] looked like a leather bound book because it helped people get familiar with, you know, with the, with the concepts of mobile. And we're now dealing with, uh, with a market that isn't already familiar with the concepts of mobile, where like the flat design doesn't necessarily mean the same things where they can't read, where, you know, the, the concept of like a bottom sheet being a modal that you can dismiss in a certain way.

Matt Klein: Yeah.

Eric Kuck: That, that makes no sense, right? So we get to rethink a lot of really cool things. We get to do a lot of animations and just like fun interactions that we haven't gotten to do otherwise.

Lauren Darcey: So-

Matt Klein: so for, I, I was gonna say, so, so for right now, it sounds like you, you're starting out by focusing more on like fundamental usability issues.

Are you doing that in the context of like a set of initial pilot apps? I'm just trying to understand

Lauren Darcey: right,

Matt Klein: like what's your process looking like currently?

Lauren Darcey: Yep. So, I think you're right that, we have lots of app [00:52:00] ideas and prototypes that we have built out. Pulling them together into a product is something we're doing, but we are basically looking at the building blocks

you need to build lots of different things

Matt Klein: yeah, makes sense. Yeah.

Lauren Darcey: In this new way. And also because of our interest in the open source community and continuing to contribute to it more and, changing the ecosystem so other people can build, like lighter weight apps that don't immediately focus on tracking or monetization in the same sorts of ways.

Like making those easy for anyone to build out interesting stuff right now seems... prescient. Especially with a whole bunch of new vibe coding, mobile developers showing up in the market. Like, I don't like the momentum that exists in the ecosystem now and how they will build in the future if it's the same way we've been building the way

Matt Klein: Yeah.

Lauren Darcey: Up to now. So, how do you introduce some new building blocks that actually make it easy to make... to [00:53:00] revisit certain decisions about how we've gotten here?

Matt Klein: I,

Lauren Darcey: yeah.

Matt Klein: I mean, I don't, I don't know if I'm a normal parent from this perspective, but my perception is that parents are willing to pay for apps probably more than most people are,

you know? And I, I do have a sense, to be honest, that if there were apps that weren't total garbage, people would pay for them in a way that is not ad supported, right? So like, it, it, it, it seems completely viable to me personally.

Lauren Darcey: I, I think that's true. But so many of the apps right now are doing two really interesting things when we looked into like competitive analysis.

Giant brand apps are advertising all, like they're getting to the top of the app stores. And all of the apps around them have ads in them where they're serving the ad about the top app in that category. And it's the first thing you see when you launch that secondary app. It's like, go, go install app number one.

And I'm like, you're just cannibalizing your [00:54:00] own category here and there's only one option now and you better like it, or, but that's the only ad free option you're gonna get.

Matt Klein: Right.

Lauren Darcey: The other thing is just that like a lot of the value prop as a parent when you are asked by your child to pay for something is just you're paying your way out of the ads of themselves.

There's no other value on the other side of that engagement. It's just yes, you will get the same experience, but with fewer ads or no ads.

Matt Klein: Yeah.

Lauren Darcey: But I think what's really scary about this is just kids themselves, especially, uh, low income kids can't pay out of the absolute worst of what... the like app ecosystem has to offer, they're stuck with the ads.

They're stuck with some- like really, really bad ads and they're in apps that maybe they shouldn't be in the first place, but they're there and they're not gonna go away. So I think that... i've never been so discouraged from a, like avoiding a category as an app [00:55:00] developer as when you say I'm gonna go mess around in the kid space, they're like, oh, there's so much like compliance and so much, so many other things and nobody wants to pay.

And I'm like, they'll pay for value if it's actual value to them.

Matt Klein: I think so too.

Lauren Darcey: Um,

Matt Klein: yeah.

Lauren Darcey: And it's a much cleaner monetization strategy than like, I'm going to make it free, but it's not really free. I'm selling you to somebody else. So, I dunno. I think that there's a lot of building block stuff in there that's really interesting and that's where we're, we're messing around.

But we also have a bunch of prototype experiences for kids that we've tested with kids that, uh, go over really, really well with them and with their parents. So figuring out how to tie that all together with a good nav and like deploy, that's the plan.

Matt Klein: Do you have a timeline for your first big, big launch, just outta curiosity

Lauren Darcey: Yes.

Matt Klein: That you would like to share with people?

Lauren Darcey: Do, do we wanna share?

Matt Klein: Yeah.

Lauren Darcey: Um,

Matt Klein: you don't have to. I'm just [00:56:00] curious.

Lauren Darcey: I, I wanna be deploying something to a broader beta in the first half of next year

for sure.

Matt Klein: Okay. Well, I, I have, I have two kids who will be your beta tester, so you just have to let me know.

Lauren Darcey: I think we're close too.

I think that once we have our nav, like nav is the hardest challenge for us, other than... the user model that you build for a parent and child relationship that are not on the same device. That part, those are the two parts that we are still innovating on before we would wanna actually deploy something.

Matt Klein: Yeah.

Lauren Darcey: Yeah.

Matt Klein: We're unfortunately at the end of our time. I would love to come back in six months or so and, and learn a little more about this topic because I think it's super interesting. Before we wrap up, Eric, did you have anything that you wanted to add on the, on the new venture?

Eric Kuck: No. No, that was good.

Matt Klein: Yeah. Okay. Thank you both. I think this was a fantastic conversation and, that's a wrap for [00:57:00] this episode of Beyond the Noise Signals, Stories, and Spicy Takes. Huge thanks to you both for joining and sharing your story. You can find this episode and all past ones on the bitdrift YouTube channel, and if you had fun, drop us a review, tell your friends or yell your favorite hot take into the void and just make sure to tag us.

I'm Matt Klein and I will see you next time. Thank you both.