Bazel Keith and the Quest for Better Builds

[00:00:00]

Matt Klein: Alright, welcome to Beyond the Noise: Signals, Stories, and Spicy Takes, the show where we dig into the stories of the people shaping the future of app-based computing with a special focus on mobile. I'm your host, Matt Klein, co-founder and CTO of bitdrift. And creator of Envoy. Each episode we'll talk with engineers, founders, and technical leaders who've transformed the way their companies build and understand what's happening inside their systems.

We'll dig into the challenges, the breakthroughs, the lessons learned, and we'll wrap it all up with their hottest takes. So let's dive in.

Today I am thrilled to have fantastic engineer, Keith Smiley, who's a software engineer currently focusing on developer experience. He spent nine years at Lyft working on iOS [00:01:00] tooling, and now works at a company called Modular on Bazel and developer productivity.

Welcome, Keith.

Keith Smiley: Hey, thanks for having me.

Matt Klein: Yeah, thank you so much for coming. Keith, by the way, is, I mean, he's one of the best engineers that I have ever met. Fantastic engineer.

Keith Smiley: Thanks.

Matt Klein: So I'm very excited to have Keith join us. And you know, the way that we typically start these episodes is, would love just to learn a little bit about your career journey.

I mean, obviously you, I think, started at Lyft out of school, right? Is that, is that correct? Or did you - did you work somewhere before that? I actually don't remember.

Keith Smiley: Yeah, I had a few small jobs before that. Lyft was my first

like real big place. I like to say.

Matt Klein: Yeah. And then I think what I would love to learn about is I know that, you know, you, you and I both joined Lyft when it was still very small.

And you know, I'm sure in the beginning of Lyft you work on... worked on - worked on a lot of things, but I guess, would love to learn about, you know, had you done mobile [00:02:00] engineering prior to Lyft? Then obviously just tell us a little bit about how you got into mobile in the first place.

And then, you know, kinda what you did at Lyft.

And then... for those that don't know, Keith is a major contributor to the Bazel build system. So, you know, he is, he's well known as Bazel Keith... you know, just would, would love to learn about that journey, right? In terms of how did you get into mobile? How'd you wind up at Lyft?

How did you wind up working on Bazel? So, take us through it.

Keith Smiley: Yeah, yeah, sounds good. So I was interested in computers like when... at, at a young age, like I think most- many developers you talk to, but unlike most folks, we actually grew up with macs in the household. My dad worked for a company in the nineties where they had macs in the time where that was a bad idea. Like I think they had them when they were good and then like they were really bad for a while and they still had them. And so as they started to get rid of them and switch to Windows computers, like we would get the runoff machines, which was kind of nice.

And so I guess that's how I got into [00:03:00] computer stuff. And so I like always was using macs. And at some point, you know...

Matt Klein: wait, so sorry. I was gonna say, but this is, this is like... not Apple II. This is like... is this the Mac with like the clear thing on it? Like the one from the nineties where like you could, you could see through it?

Is that what we're talking about here?

Keith Smiley: So we did have that one, but that was a little later. I don't know...

Matt Klein: okay.

Keith Smiley: I, I feel bad to not know if it, like, what model these old ones were, but they were the old like gray or beige boxes that you see like typical from that era.

Matt Klein: Right.

Keith Smiley: I don't know... yeah, I'm not sure if my parents would know at this point, like, or if anyone... it's probably just lost to time, like what models those actually were, but...

yeah, it - it is just, it is really just to say that like, growing up with macs meant, we kind of like always had them forever because even though, like... so that was pre- MacOS 10 transition, once Mac OS 10 came out, like we had those machines instead and yeah, we had the clear like... Mac... G3 or whatever that was like purple or green or whatever.

But yeah, all of that is really just to say that like I started... yeah, tinkering with computers on macs, [00:04:00] which I think was atypical at the time. And honestly at the time I had no idea what I was doing with any sort of programming and no idea how to figure out anything either, like. You know, I guess this was pre Stack Overflow.

And I'm sure there were, you know, I hear people these days talk about like BBSs or whatever existed at that time, but I just like didn't know what I didn't know there. So, I mostly just, you know, tried to tinker around and make anything work with Objective-C, which was like virtually impossible for someone who was in high school.

Matt Klein: Did you have books? Because I remember back then people would go to like Borders, you know, and buy and buy.... programming books. I mean, it's like,

Keith Smiley: yeah, I wish I had known to do that too.

Matt Klein: It's like funny to think about that right now.

Keith Smiley: Yeah.

Matt Klein: That people used to go buy programming books. I think I had some programming books, so I dunno.

I mean like how did you learn? Like did, did you have books or you just kind of

yeah, figured it out?

Keith Smiley: I mostly just mess around with, like the sample projects that Apple had out there and there was like one forum that if you were looking at very targeted stuff you could occasionally find good information on.

And then, you know, I don't know, you could poke around a lot in the SDK and once you had like some sense of how [00:05:00] anything worked, you could maybe find an API that was well named enough to like try to do what you did. But realistically, like I just wasn't doing anything particularly complex and it was kind of fine.

And so I spent some time doing that, which was interesting just because obviously Objective-C isn't like a particularly... friendly, like first language to spend a lot of time on. And obviously I had like no... since I had no idea what I was doing, like I'm sure I wasn't, you know, managing memory correctly or whatever you had to deal with at that time.

Matt Klein: Doesn't matter. Doesn't matter.

Keith Smiley: Yeah, it definitely didn't for that. For, for my toy applications. Um, from that I, you know, spent some time on like the other kind of beginner programming stuff like wizzywig stuff, like Dreamweaver or whatever. I ended up going to college for computer science 'cause I knew I was interested in it even though I really didn't have like much of a background or like, didn't really know anything at that time.

Which was obviously fine going into college. But... Yeah, so I - I guess I spent some more time on, Apple stuff at that point. Like the iPhone had come out and like, so I was, you know, trying to mess around with that a little bit. Still not doing anything particularly complex. But, you know, at that point, I guess like I was at least familiar [00:06:00] with some APIs in the programming language and... I dunno, you started to find stuff on GitHub at that point.

And I guess the big thing, for me at that time was, you know, I wrote some code and like put it on GitHub or whatever and... Ended up submitting it to CocoaPods, which, was the biggest iOS package manager for a long time. It, you know, honestly, it's still huge even though they're like shutting it down next year or something.

And through contributing to CocoaPods or like just my own libraries, the folks there were super interested in having like people contribute. And so they gave me commit access to that repo, which I felt heavily underqualified to have. But... I ended up like...

Matt Klein: and, and is this, is this while you were still at college that this was happening?

Keith Smiley: Yes. Yes.

Matt Klein: Yeah. Okay. All right. Got it.

Keith Smiley: Yeah. So... yeah, I was maybe two... in my second year of college at this point. And... Yeah, so the way that CocoaPods works is like there's, there was this giant repository called the Specs Repository that had like a centralized registry of like every single pod spec which was like the definition for the library.[00:07:00]

And, you know, they were kind of giving out commit access to anybody who had pushed something. At some point they kind of realized maybe that wasn't the best ideas. It started to grow. And so,

Matt Klein: yeah. It sounds

Keith Smiley: instead like,

Matt Klein: sounds a bit...

shady from a, from a security perspective.

Keith Smiley: Yeah. Yeah...

Matt Klein: I guess I was gonna say...

Keith Smiley: it was a different time

Matt Klein: I guess back then people did not really think about supply chain attacks like they do today.

Keith Smiley: Yeah

Matt Klein: yeah

Keith Smiley: yeah, for sure. But at some point, you know, obviously that stopped and like... so then folks who were working on CocoaPods were merging pull requests for that. And so I ended up doing a, a lot of that while I was in college and merging like thousands of these pull requests as... and, and it was just really lucky timing that CocoaPods got like really big at the time.

And so I was working on that, a bit. And then I guess through that, just like, you know, met a lot of folks and, like saw a lot of stuff that was going on in the iOS community, which was really great.

I actually got a job in college at the place where my dad worked, to work on an iOS app. This was still like, really... I don't know, like maybe 2013 or something.

So it was like pretty early mobile days. But he was like kind of interested in this like, [00:08:00] app for, uh, well... I don't, I don't need to go into it, but it was like a pretty basic app of just like data presentation of like some stuff, of some database of stuff that was like on device or whatever.

And that was a really good like, learning ground for me to like build a real app... for iOS, but it was entirely by myself. Again, like still not really, with many helpful resources, but at least at this point, like there was Stack Overflow. There was so much code on GitHub at this point. Even though it feels like nothing, compared to today.

So I spent a few years working on that, which was really great. And then I guess after college, like through the connections from the CocoaPod stuff, I was able to get a job in the Bay, like working on iOS stuff. And I like moved out to, San Francisco like two days after I graduated from college and... to work on iOS.

And yeah, I worked at a small consultancy for a while. And realized that maybe consulting wasn't really the best for me at that point in my career, since it was still super early. And, a lot of the projects were, maybe like working by yourself, which wasn't really what I was hoping to do. I kinda wanted to like work with other people to like learn more from folks.

Matt Klein: Yeah, for sure.

Keith Smiley: And that's when I also knew, someone else at Lyft from the CocoaPods [00:09:00] community, ended up switching to - to Lyft. So, yeah, that was the journey there.

Yeah, like you said, I... at - at Lyft, I started as a feature engineer. You know, there was no infra team of course, at the time that I started.

Matt Klein: Yeah.

Keith Smiley: For folks who listened to the, the other episodes of this podcast, I guess, like I started when Martin was working on this rewrite in, in secret and, but I was of course working on the like Objective-C code base that would become the legacy code base, I guess. So yeah, I was working on features there for a while.

And then, you know, things changed when we like shipped the rewrite or got closer and closer to doing that, so,

Matt Klein: yeah. Well, and, and, and sorry, for those that have not listened to their episode, what you're referring to is Lyft like many companies at the time had an Objective-C app. And Martin, my co-founder, rewrote the entire app in in Swift, which was a brand new language at the time.

And I think I had, I had asked Martin the same thing, but I, I'm sure you have a lot of, a lot of memories of... you know, swift compiler issues and things along those lines. You - you know, I mean, would, would love any stories that [00:10:00] you have from that time.

Keith Smiley: Yeah, I think Martin has all the best ones, but since he doesn't love doing these things, like, I think I've ended up telling a lot of those.

Or after we shipped the rewrite, like I did a talk about the rewrite, even though he was the one who did the rewrite, which felt... pretty silly. And now, now like all these people thought that I did it or something, but... Yeah, I mean, you know, there was so much interesting stuff at that time because just everything was broken in interesting ways and, you know... so Chris Latner who, you know, rewrote or wrote Swift by him- or started that project by himself at Apple, you know, he's the founder of Modular where I currently work and since

then he's gone on many podcasts talking about how, you know, there were no major consumers of Swift at Apple and how that was like a mistake in the process of shipping it, which I, you know, obviously is very easy to say in retrospect. And I think that, yeah, may- maybe that was a, that was a really good learning for them.

But I think the funny part was that at that time, like the Lyft code base was definitely the largest swift code base out there because, you know, Martin was working on this pre- 1.0 when he started and like it was still in beta and then we shipped it at like... swift 1.1 or something, which was just a few months [00:11:00] after it was 1.0.

Matt Klein: Yeah.

Keith Smiley: So...

Matt Klein: I-I mean, what's, what's really interesting to me, and this is totally anecdotal, but I feel like to this day, swift has more compiler bugs than I would expect, like for a pretty... for a pretty mature language. And, and, and again, that, that could just be my anecdotal experience, right? It's like I just... the number of times in my career that I have faced, like a C++ compiler bug, or a Rust Compiler bug...

It's pretty rare. I mean, it's like, it's happened to me probably less than five times my entire career. But I feel like the swift stuff happens a lot. And I'm actually curious, like, why do you, do you think it's because that it's not, at least historically, it was not used very much within Apple? Like, or do, do you think that there's other things

going on there?

Keith Smiley: Yeah, that's a great question. I mean, at this point it's used enough in Apple that I feel like there [00:12:00] shouldn't be any reason that the like patterns you might use outside are any different from what is also being used in Apple

Matt Klein: right

Keith Smiley: like I don't think, it's not like Apple's culture is such that all the code bases look the same or something like they're very insular between teams, right?

So people can kind of do whatever they want. You know, for us, it, it always felt like we were just doing things in a way that was just slightly outside of maybe what they expected. Like

Matt Klein: I see

Keith Smiley: you know, there's a big, big thing in the early days of Swift about more like functional programming concepts since it had this really strong type system and maybe you could do stuff that that way and people went to this extreme where they were maybe doing stuff like you would do in Haskell or something like that.

Matt Klein: Right.

Keith Smiley: And I think that stuff was like super unexpected to them. And I think there's still patterns like that that exist in the community, but not Apple. But I don't, I don't really think that should be the case anymore. But, you know, for example, all these people use this library, RX Swift, that gives you these like RX concepts in the language or whatever, and you can end up in, in situations where you have like hugely nested chains of these things with many types and all this, and that always hits issues.

Matt Klein: I mean, it's just [00:13:00] e-even at bitdrift, like within the last three years. We've had at least one, if not two issues that were explained to me as swift com- like miscompile issues. And I, it, it's more just, again, it just seems odd to me that for a pretty mature language that it's now... widely used that us as a pretty small company, we don't even have that much swift code.

Like most of our code is actually written in Rust, you know, that we would be hitting issues like that. But anyway, yeah. Sorry. Keep going.

Keith Smiley: Yeah. I, yeah, I mean that's an interesting, like just skip ahead to, you know, a few- later on at Lyft being able to work on that stuff was really nice because, yeah, I could dedicate some time to trying to fix some issues like that, that we would hit, which was really great.

But yeah, I guess, going back a, a little bit like, so, you know, after the rewrite or, or towards the rewrite, you know, we started... kinda everyone moved over to that. And then after the, the rewrite shipped, we kind of founded a small infra team. And I was on that with maybe one other person, for quite a while.

And I guess since that point I pretty much worked on infra stuff [00:14:00] the- for the rest of the time at Lyft. You know, at that point it was much more like typical, how do we make libraries that are more reusable for folks and like patterns and stuff like that because the code base was so, you know, young, and you know, Martin did a great job, but obviously like you start to grow and you add features and there's like new patterns that emerge.

So,

Matt Klein: yep.

Keith Smiley: I spent a lot of time on that kind of stuff. And there was a lot of other like... yeah, various maintenance work in that process too. Yeah, so, you know, that team grew and grew, as the company grew. And at some point we split into multiple teams. One that was like more focused on those libraries and like architecture and stuff.

And one was more focused on tooling, and I was working on the tooling side. And... yeah, there was a brief stint in there where maybe we like, kind of, sort of rewrote the UI of the app again, which, which Martin also maybe talked about more in the previous episode if people are more interested in that.

But... and so I guess I kind of did that for a year, but then came back to tooling after that part. And I guess this was, I don't know, 2018 or something like that. And from that point we had, I don't know, gone from, I don't know... 25 [00:15:00] iOS engineers to 75 or something like over the course of a year or something like that, or I don't know.

So things really like ramped up in interesting way with how the code base and the team and the company were scaling. Especially because, you know, like many companies, Lyft had this like vertical, team structure except when it came to mobile because everyone ships everything in the same binary.

So that really like changed a lot of, yeah, how you thought about developer experience. And you know, so at this time we started hitting kind of the classic issues you'd expect in a really large code base of like, you know, compile times are really long and people are changing things that you know, maybe cause the whole app to recompile even though they shouldn't because , you know, they're not actually affecting that stuff.

And that's when we were like, seriously getting into... Modularization and tooling and stuff.

Matt Klein: Yeah. And, and at that time, and I apologize for the really stupid question, I should probably know this, but that probably means that people who are listening also also don't know this. If you are doing like a stock iOS app, right?

What... what is the build system [00:16:00] by default? I actually don't even understand that, right? Is that like, is, is there some xcode script? Is it bash scripts? Like what... what is the build system there? Like, I mean, because I understand on Android, you know, obviously there's Maven and Gradle and it's like there's stuff that I've at least heard of, but...

Again, I should probably know this, but I don't, right? It's like what is the default build system on iOS?

Keith Smiley: Yeah, you can, you can maybe blame Apple for some naming, confusion there. So like the IDE that everyone's using is called Xcode, right? Xcode. And the build system is called Xcode build. And so there's a command line tool called Xcode build that actually does the orchestration.

Matt Klein: Got it. Yeah.

Keith Smiley: That's like your Make analogy

Matt Klein: right, so at the time I assume that Lyft was using Xcode build basically, right?

Keith Smiley: Yeah.

Matt Klein: It's like they, they had the normal build scripts or however people do it, you know, that would do a command line build of the app like most other people.

Keith Smiley: Yeah.

Matt Klein: Right?

Keith Smiley: Yeah, so we generally like had, yeah, totally normal stuff.

We were using CocoaPods, you know? Yeah. Using Xcode build, all of [00:17:00] this stuff. But you know, once we found that we had, you know, a growing team of 80 plus or whatever, and like continued to have, some growth there every, you know, two minutes that you spend doing a pod install or whatever with CocoaPods or like waiting for extra build stuff, is like probably worth me- trying to improve and, you know, at this point

we maybe had two people working on tooling for iOS and, and two for Android as well. But you know, it's not like there was a massive investment or anything to try to improve stuff. You know, I mean, that was a really good, like, team size, but I'm just saying it's not like we had like 10 people trying to work on this, whatever.

It was a pretty reasonable size team compared to the 80 or whatever, engineers we had working on features.

Matt Klein: But, but at, at the time... is it safe to assume that the thing that people were complaining the most about was actually compile times? Or, or like, were there other issues that you would say were equivalently complained about?

Keith Smiley: Yeah. So when we were using CocoaPods, there were a lot of complaints about [00:18:00] that because you had to do some amount of like base level overhead every time you like did a git pull or something like that. You know, when we were using like normal Xcode stuff, which means like checking in this Xcode project file that's this giant XML file that's, notoriously difficult to merge...

like if you change something and someone else changed something.

Matt Klein: Yeah, of course

Keith Smiley: your conflicts. And so there were a lot of complaints about that. But yeah, then compile times and then like general IDE slowness as well. Like regardless of all this other stuff, if you open Xcode and it takes a long time to like open a different file or your code completion takes a really long time. Like that was a frustrating thing as well. And you know... so you kind of mentioned that we ended up like maybe moving to Bazel there, but there were a lot of intermediate steps that helped us get further along. You know, we replaced CocoaPods at some point and like pre-compiled all of these frameworks, which some other companies do with, some other tools that are out there as well.

But that like, you know, tossed this whole step of compiling third party code all the time that you were never changing, except for the very rare occasion where we actually like bumped the version of some dependency.

Matt Klein: Yeah.

Keith Smiley: And [00:19:00] that was like a lot of custom tooling to do that, because the off the shelf stuff maybe, wasn't super widely adopted in the community, so it didn't really work very well.

We also moved to this like Xcode project generator thing, which is really nice. So we stopped checking that file in and you still had to generate something, but at least you, you know, skipped all these merge conflicts and all of that.

Matt Klein: Yeah.

Keith Smiley: So there were some really nice wins there for developer experience.

And really the build system like switch was a, you know, really big hammer after we spent, you know, I dunno, at least a year, like working on other improvements.

Matt Klein: Yeah.

Keith Smiley: Because, you know, we didn't really want to jump into that without being super sure.

Matt Klein: I mean, before we get into the whole... Adopting Bazel question, which I think is very interesting.

One - one thing that I think is interesting on this topic, which I'd love to get your opinion on, is, and you know, I, I think we see this in different... different segments of our industry where, you know, you reach a certain scale and things go wrong, right? Like, whether that's on your mobile app or in your microservices or something else.

But in the mobile space specifically, one thing I'm curious [00:20:00] about is... why don't you think even to this day, why don't Apple and Google invest more in like better developer experience for larger teams? Like why is it left to every company to go and solve this problem? And the only thing that I can think of is that, you know, they're obviously invested in

the most apps that they can get, right? It's like they want the app store full of apps and most apps are written by small teams who don't actually care about these problems. So maybe it simply is that, you know, there, there just aren't enough large enterprises to actually make this worth it. At the same time, it confuses me because most of their revenue and interest comes from these large apps,

right? So I, I would just love to get your opinion on like, why does every company solve what I feel like is literally the identical problem. Like why aren't Apple and Google helping solve this problem?

Keith Smiley: Yeah, that's [00:21:00] a great question. I totally agree with the sentiment. It kind of feels like, you know, for one, I don't think the problems overlap a ton with the problems that they see internally because you know these... a lot of these companies that have.

This type of problem are the singular, monolithic app kind of companies, right? Like the Lyft-type space where you have a very small number of apps and you have so many developers as opposed to Apple or Google, where you have a lot more apps and a lot fewer engineers maybe on each- working on each individual app.

So I think they hit a different set of problems often on the- like just because those products are different. , You know, I also think it really probably just comes down to a prioritization thing of like they're always working on whatever the new thing... especially with Apple, like, they're always working on whatever the new thing is and supporting that new thing, and they're less focused on these types of inbounds that aren't super related to those.

Matt Klein: Yeah, I, I mean, do - do you think that there could be another explanation, which is that Google certainly has solved this problem internally, right? I mean, they, they obviously have a well-known [00:22:00] monorepo and the precursor to Bazel and all of those things, so it's like... and I, I guess to push back slightly, I'm sure there's hundreds of developers who work on the YouTube app, right?

I mean, it's like, these are, these are like big apps, so I, I, I'm just, it's like I almost feel like they don't feel the pain, so they just don't... care, I guess, right? And it's like, I know nothing about Apple because their engineering is a lot more opaque to me. But I would imagine internally they have some build system that does something, right? For, for like large teams and, and it's more...

Maybe to your point, they, they don't feel the pain, right? And it's a matter of priority, but it, it just, it's like, it's one of those cases where it just seems like any app, at any level of scale in terms of people, is going to start hitting like these problems and people just solve them over and over and over again.

But anyway.

Yeah.

Keith Smiley: Yeah. I mean, the other thing I worry about with that is just that, you know, because of how everyone's gone and these like divergent paths of solving things, like, I mean, maybe [00:23:00] you could think of Bazel as some sort of unifying thing, but that's kind of maybe too high level. Like the specific problems that a lot of companies see are very different.

You know, for example, when we started using Bazel and other, and other tools besides Bazel in the community, like, you know, we had this all Swift code base that was, I, I don't know, at that point, like over a million lines of swift code or whatever. And, you know, no one else at that point was using those tools with that exact like, shape of things.

Like, you know, Google was using Bazel and Swift, but they had like very little swift and mostly still

Matt Klein: Yeah, of course,

right.

Keith Smiley: Objective-C or whatever. And so, I don't know, there is something to say about like, the solutions are gonna be different for many different teams. So it's not like I, I'm not sure if you looked at...

I want to go improve Xcode performance like exactly where you would start. I don't know, maybe that one's too easy because it is maybe more obvious how you would start that. But

Matt Klein: yeah

Keith Smiley: I feel like once you get past those one or two top things, it's a lot harder to figure out what the like slam dunk win is.

Matt Klein: Yeah

Keith Smiley: I don't know. I mean, I feel like Apple does try to make strides in the right direction, but I think that it's all, it's [00:24:00] never on the timeline that these companies are okay with, right?

Matt Klein: Yeah.

Keith Smiley: Like we switched to Bazel in... four months or something at Lyft. And, you know, we're... here we are, seven years later?

Six years later? Talking about like still having the same kind of issues. And so

Matt Klein: for sure

Keith Smiley: that's obviously not the right timeline. But yeah, even if it had been a year, it probably would've been too long... you know

Matt Klein: so, so I guess take us, take us back then, because I am curious, right? You're, you're obviously, you're trying to make Xcode build work, like you're trying to do all these targeted fixes.

What - what led you down the Bazel path? And to be clear, for - for people that don't know the history at that time, I don't know what year we're talking about. I don't know, 20 17, 18, 19, like something along that, those lines, Bazel was pretty new back then and had a bunch of rough edges, right? You know, so I - I'm just curious and, and again, for those that don't know... Envoy, adopted Bazel very early and it was very painful. So, [00:25:00] you know, I, I, I have some sense of, of the pain back then, so... and like maybe you didn't know about the pain, like, you know, obviously until you started, but I'm just trying to understand...

how did you decide, right? To, to go from these targeted fixes to saying like, this is just not gonna work, we have to do something different. And then once you said that okay... something, how did you decide that, that something was going to be Bazel?

Keith Smiley: Yeah, totally. So, before we, kind of dove off onto this separate redesign project that I mentioned earlier, which took like a year, maybe of time from Martin and myself, and then, then some others, throughout the process.

I had actually done some sort of like evaluation in the community of like, if we wanted to move off of Xcode build, like what are our options? Or, or it wasn't specific to if we wanna move off of Xcode build, it was much more like, how do we improve this at all? Like, even if that meant staying with Xcode build.

But at that point I had evaluated ... bazel and, you know, Buck, which was out at the time.

Matt Klein: Yep.

Keith Smiley: There was also this [00:26:00] other like competing build tool, I think it was called XC Build, and it was written by someone who was working at WhatsApp. And so there was like some interesting other options at that point.

And honestly at that time it was like none of these are going to work... like if we wanna do anything, we kind of need to write it ourselves. And at that point that was definitely like, we could have done that and I think it would've been worse than what we had ended up with, but it would've been maybe okay.

And actually that project kind of got plus oned, but then, it kind of got on the back burner as I went and worked on this redesign instead. And then once I came back from that project, you know, we still knew this was out here and after we spent a while working on those other improvements I talked about, yeah, we kind of redid that evaluation.

And in that time, Bazel had improved quite a bit. Like when I looked at it originally, you know... yeah, there was very little support for almost everything that we needed. And since, you know, like you're getting at... it was so young, it wasn't really clear like to us from the outside as people who'd never worked at Google, like how hard it would be to add that stuff.

Matt Klein: Mm-hmm.

Keith Smiley: And obviously stuff was moving very quickly. They were going through this like Java to Starlark [00:27:00] thing and all that stuff. But during the, the kind of in-between period there, I think was when they open sourced the Apple rules, which were like the first big rules I think that they rewrote in Starlark.

And, that like was a really nice path towards reasonable like maintainability because it was all out of this Java stuff and it was much more like specific to the Apple stuff. It felt much more approachable at that point.

Matt Klein: Yeah. And, and I was gonna say for those listening that don't know the way that Bazel...

Keith Smiley: yeah

Matt Klein: works is there's a core, right? And then there are these sets of things called rules, right? So I I, I guess just in like 60 seconds, because people might not know.

Keith Smiley: Yeah.

Matt Klein: Could, could you just like really briefly describe how this all works? Just because it is kind of interesting.

Keith Smiley: Yeah, totally. So, yeah, you can think of Bazel as maybe like a - a tool like Make or something that does some... well actually let's, let's use Bazel and CMake, or something as like an analogy maybe. Where yeah, like Bazel is this core that, yeah, like understands the kind of larger build graph, but then how you actually do things are written in [00:28:00] rules and I guess in the, yeah, I, I don't want to stretch the analogy too much

'cause then it, it'll require people understand the CMake ninja setup as well.

Matt Klein: Yeah.

Keith Smiley: But the general idea is that like you, yeah, you have all these Java... this Java core that does all the scheduling and all of this, other build graph computation stuff. And you actually express the graph to the Java core through these rules that are written in this python like language.

And the ideal is that all rules can be written that way. But in the past everything was written in Java and they just like made stuff work. And that's what evolved, a lot over time. And that really paved the way to things being much easier for other people to contribute to because you didn't have to understand all of this, like, you know, internal Java API stuff,

you didn't have to know Java at all either. Like the Starlark API is very limited in what you can do for, you know, for good reasons. And so you can write stuff, you know, like if you have a- your own programming language or whatever, you know, you can write a, a rule that like compiles your code and whatever in just a few lines of something that looks like Python and like be ready to go, which is really fantastic.

[00:29:00] And so, that... really changes how you'd maybe look at the tool for like how maintainable things could be and like how approachable it is if you know nothing about this tool otherwise.

Matt Klein: Yeah... so were, were there people at, at Lyft who pushed back on adopting Bazel? I mean, and it's like, what was the, what, what were the pros and cons or like what were the alternatives considered?

Keith Smiley: Yeah, there were... I, I think as far as like the, the only two realistic options were move or don't. Move to Bazel or don't. I think everything else in the community at that point had like kind of stagnated a lot. So there weren't really like those other options of like b- Buck or XC Build or whatever, like had kind of stagnated.

So I think the only other options were kind of like... take some time to try to, you know, improve other parts of the stack or whatever, but maybe you're not gonna get the same type of 10x improvement. And so really at that point, it, it just came down to a bet. You know, at this point, Martin was the director of that org, and I think he was, you know, really willing to take bets like that and, and see how they...

Played out. [00:30:00] But, you know, when we decided to do that, it was super not clear that it was gonna be the right decision.

Matt Klein: Yeah.

Keith Smiley: There was one other company who was like, I, I don't know if they had fully moved at that point, but the Pinterest had had maybe moved or, or were moving at that point, but there wasn't much interest in Bazel in the community at all, and definitely not in the mobile space.

I'd say in general, at that point, there wasn't a bunch of like infrastructure expertise in mobile. I think most people who had gotten into mobile had done it because they were interested in making apps for the iPhone and not because they were like super deep into the tech stacks. So, yeah, there weren't a lot of folks looking at this.

Many companies didn't have much investment in this type of developer experience stuff.

Matt Klein: All right. So, so now let's, let's have some real talk. How many fixes did you do to the Apple rules to, to actually get it working?

Keith Smiley: Yeah... I mean, there were a lot... yeah. And honestly, that was one of the, the parts that made us confident in being able to do it longer term is just naturally, you know, I mean like... hundreds is the real answer.

Matt Klein: Yeah, sure.

Keith Smiley: But the, you [00:31:00] know, I think a huge part of it was what I mentioned earlier, which was... Google's code bases, who were the only serious users of Bazel at that time, and our code base... could not have looked more different. Like they had C++ and Objective-C. We had all Swift, we had interface builder like it was so different,

Matt Klein: but hold on, just to be...

pedantic because I actually...

Keith Smiley: mm-hmm.

Matt Klein: Was using Bazel during this time. Did Google even actually use Bazel? Because they were using Blaze, right? So, so,

Keith Smiley: right.

Matt Klein: I mean, it's like, that's, honestly, it's actually an honest question. It's not super clear to me, in the late 2000 tens, was anyone at Google actually using Bazel?

Keith Smiley: Definitely not internally at Google. I think part of the draw for them to open source, if I understand correctly, was that there were open source libraries written by Google that they wanted to use Bazel with externally.

Matt Klein: Yes. Yes.

Keith Smiley: Or they wanted to use Blaze with-

Matt Klein: blaze with, yes.

Keith Smiley: So that, that was part of it.

So like TensorFlow...

Matt Klein: right.

Keith Smiley: And there were some other big ones that, that really mattered to them. So that was part of it. But no, [00:32:00] but I think at that point, even like honestly, once the Apple rules were in Starlark, they were using that... relatively unmodified internally at Google.

Matt Klein: Ah, I see, okay, makes sense

Keith Smiley: So the internal/external versions were almost identical.

Matt Klein: Makes sense. Got it. Okay.

Keith Smiley: And so, yeah, the - I think that at that point the differences between Bazel and Blaze didn't really matter too much. I'm super appreciative though of the folks who were maintaining the Apple rules at the time.

There was this guy, Sergio, who no longer works at Google, but he was, super willing to accept open source contributions and like work with us on that stuff. And, you know, without that... I don't know if we could have moved because we, you know, we could have forked it, but I don't think at that point we would've had the confidence in saying that that was a good idea, right?

Matt Klein: Yeah. Mm-hmm.

Keith Smiley: So, he was actually merging stuff, which, you know, meant testing it internally and everything too. And so that was like kind- that still to this day is like kind of a, a pain in the, the community, at least for, for non-Apple stuff. 'cause, yeah, I guess like. The community owns the Apple stuff now, but yeah, so without some support from the, the Google folks there, like we definitely never could have made that [00:33:00] move.

But yeah, it took a ton of work to be able to do that. But I mean, the nice part of that was that we were very gradually ramping up on that project in a way that was like pretty approachable. It wasn't just like, 'oh, jump in and learn Bazel.' It was like, we have this one problem, like, where's the code that does that?

Let's fix this.

Matt Klein: Yep.

Keith Smiley: And it was a very incremental approach to really learning it, which was really great. And, you know, we made so many fixes that were required for us to get anything working. And then we also were able to make a lot of fixes that help with performance and other things. So there were a ton of... yeah, a ton of great things happening at that point, for us to be able to use it.

So, yeah, I don't know, that happened for maybe like four months or something. I don't remember exactly what the timeline was. Like I remember having some functioning, build in like December and then I think we ended up merging in, in, first CI or something in February or something like that.

Matt Klein: Yeah. So I - I guess two questions. One, what - what was the end result? I mean, it's like, what-

Keith Smiley: Yeah.

Matt Klein: What did it do for the app build times? I, I guess that would be my first question. And second question, just because I love talking about bugs. [00:34:00] What, what was the worst bug that you remember during the... trying to get the build ported over?

Keith Smiley: Yeah. As far as like the, the outcome, I don't remember the specific times, but I mean, I, you know, once we like stood up the cache and like got all this other stuff working and like, there were multiple steps in between here, but I think we went from like a, a clean build, which would take like 20 minutes, which would happen in the normal state of development.

It wasn't like, oh, developers cleaned and they had to deal with this. It was like developers would pull, and for some reason, you know, Xcode would decide... Xcode build would decide everything was trash and so it would do a clean build that went from like 20 minutes to one minute or something. So like, you know, the impact was so...

Matt Klein: yeah.

Big.

Keith Smiley: ...clear it. Yeah, and, and honestly like after that, really when you move to Bazel, I feel like it's not about that initial cutover. Like obviously you have to prove a lot of value there, just, you know, from a company perspective. But the value also comes like after that because you continue to like, keep that line kind of flat as you add code.

Because like, you know, I worked at... after we did that migration, you know, I worked at Lyft for another like five [00:35:00] years and people added millions and millions of more lines of code. And like the developer experience had to stay reasonable, right? Yeah. So, yeah.

Matt Klein: Now, now you actually mentioned when you stood up the cache and at that time there was no eng flow.

There was no BuildBuddy or whatever it's called, right? And, and... What, what did you do there? And actually, could you, could you describe to people again, very briefly, like, the architecture of Bazel is a little different than most build systems, right? In terms of like local remote, and I, I actually didn't know that.

I didn't know that Lyft had a cache that early on. I just figured it was actually added later. So could you, could you tell us a little bit about that part?

Keith Smiley: Yeah, it wasn't immediately for sure, but the, the way that generally works is that, yeah, Bazel's really strict about the inputs and outputs of everything that it is doing in order to be able to reasonably prove that things can be identical between machines.

So it encodes all the tools you're [00:36:00] using and all the environment variables that are going into an action and things like that, so that if you build something on your machine and I don't have to build it on mine, I can just download the result of whatever you built. That's the general way that works.

And so the backend to these is some sort of cache service that has like two different buckets of information, but generally Bazel just does tons and tons of HTP requests and like uses all the bandwidth possible to like download the results of previous builds. There's another level of this where then once you know that, things would be identical compiling on different machines, you can actually use remote machines instead, which gives you like wider parallelism depending on the build,

which is super valuable in some cases. But yeah, so actually at that point, Lyft had this, autonomous driving arm and they were also using Bazel for their C++ code base.

Matt Klein: I remember this, right, yeah.

Keith Smiley: So I think that they started using Bazel after we had moved to it, or, or maybe at the same time or something.

We didn't really communicate with them too much about it because, you know, again, we're like this all Swift mobile thing and they are like this all C++ thing. It's very, it felt very [00:37:00] different. But they actually had a mu- a larger info team than we did. They had like four people or five people working on their sort of infrastructure slash Bazel stuff.

And so they actually spun up a cache, which is one of the ones that you can still self-host today called Bazel Remote, on GitHub. And so we actually piggybacked on that for a long time.

Matt Klein: Oh, okay. Interesting.

Keith Smiley: Yeah, so it wasn't a vendor one, but they spun it up. The problem with that cache was it would only run on a single node.

There was no like sharding and so they were just like getting the biggest AWS machine they could, and eventually that stopped working and, uh, because it was just like the builds got too big and, I don't know, you're running out of reasonable performant disc space and, and stuff like that.

Matt Klein: Yeah,

Keith Smiley: I - I don't remember exactly what kind of fell over.

And at that point, someone on our team, from the Android side actually wrote a cache, to use instead. That was a bit more distributed and was using Redis for things. We used that for a while. I can't remember exactly how long and like when we moved over, but at some point after that we moved over to BuildBuddy, which was a, just a general, much better experience 'cause we offloaded all [00:38:00] that work.

So, yeah, there were a lot of pieces there. But you know, that's like one of the first things you wanna do as soon as you're over to Bazel, because that's where a lot of the wind start coming in. But yeah, I can't remember exactly the timeline, but that was a huge, valuable piece.

Matt Klein: Yeah. Cool. I, I mean, I have some other questions about Bazel, but I guess prior to getting into that, just take us through the last part because I know that you've moved on since then, and I know now that you're not even really focusing on mobile anymore. You know, you're really focusing on the - on the build stuff.

So I guess just tell us a little bit about what you're up to now and like what are, what are some of the biggest challenges that you're working on these days?

Keith Smiley: Yeah, totally. So yeah, I kept working on Bazel at Lyft for a while. But yeah, I decided to move on at some point and, you know, yeah, I was kinda interested in getting out of the mobile space.

I mean, we can talk about that a bit I guess, but, yeah, I was still interested in working on developer experience, but wasn't really tied to continuing to work on iOS or anything. And so, yeah, I was kind of looking around. I was interested in the, you know, what was exciting in tech at the point that I was [00:39:00] moving, which was, I guess...

2023 or something at this point. So, yeah, I, I ended up at Modular, which is a, a kind of an AI startup. And they needed someone to move to Bazel at this point. They had some ex- Google folks who identified their kind of C++ build cmec setup as a, a bit of a problem. So I joined to do that migration again.

It's a little bit different, you know, since it's not mobile focused. Yeah, I guess the problem space is a lot different because, you know, yeah, definitely not being mobile, but also, you know, we're working in this kind of GPU space, so there's a lot more variance in hardware, which is pretty interesting.

We also have our own programming language called Mojo, and so there's like... the C++ bit, and then there's, you know, building this compiler and then there's building the Mojo code, which also interops with Python. So there's a whole bunch of stuff that, you know, we didn't really talk about, but, Bazel really shines in this like multiple language interoperability story.

Matt Klein: Yeah, for sure. Yeah.

Keith Smiley: Yeah. And honestly, if we hadn't moved when we did I feel like things would've looked a lot worse if we were like trying to move now or something. Like if we'd gotten stuff working in the previous system because yeah, when you have a [00:40:00] unit test that like runs Python, which then invokes a C++ compiler and then like invokes the Mojo compiler to like build a Python native extension to run some code, like, you know, it's really nice to have some structured organization for how that works.

Matt Klein: Yeah, for sure.

Keith Smiley: So yeah, that's the kind of stuff I'm working on now.

Matt Klein: No, that's cool. I, you know, and, and now we'll get more into the hot take section. I think - I think what I wanna pick your brain about a bit is, look, I, I totally get why people adopt Bazel. Like there really is no other game in town.

When you have to do hermetic builds, you wanna do parallel builds, you wanna do caching. As you said, it really shines when you're dealing with multiple languages. At bitdrift we also use Bazel for... SDK because we're compiling swift and Kotlin and rust and all of these things. But I'll, I'll say this as nice as I can, like no end- I have never met an end user of Bazel who is happy about using Bazel.

Like that person [00:41:00] does not exist. Like I would like to find that person, and... I - I'm not sure exactly what it is, and maybe it's because when it works, it just works. But when it doesn't work for an end user, it's very difficult to debug and understand what's going on. Or, you know, builds are supposed to be hermetic, but then they break and I expunge and then it starts working again, which is very frustrating,

right? So I think my, I think my high level question is: I- builds are obviously so important to our industry. There is no software without builds, right? Yet at the same time, I feel like from a user perspective, build systems suck. I mean, it's like, they're just like, they're, they're not very usable, right?

So, as an expert in this space, I, I actually just wanna ask you. First, do you, do you agree with my sentiment or do you think it's wrong? And it's totally fine to disagree. If you agree with my [00:42:00] sentiment why, like, why to this day are build systems not more end user friendly, right? I mean, it's like, that's what I'm trying to understand.

Keith Smiley: Yeah, no, I mean, I agree. I think that, you know, it's easy for people in the infra space to actually like, be kind of apologetic about it because you see like where the problems are and how complicated it is. But yeah, I mean, from my experience at Lyft and at modular, you know, most developers obviously don't want to, or shouldn't need to care about build systems and they should just be able to get their work done and that like mostly not be in their way.

I think that there's a huge like last mile problem between Bazel and like meeting users where they're at. I think obviously some part of that people talk about all the time is like IDE experience in the Bazel world. It's like, you know, if you pick up VS code and you pick up a C++ CMake project, like you'll probably be kind of sort of fine or right out of the gate.

And with Bazel it's like you'll have to hire a Bazel expert to work for six weeks to make something work

or whatever.

Matt Klein: Yeah

Keith Smiley: there's so [00:43:00] much opportunity in that space. I think, you know, similar to what we were talking about with Apple and like improving things for large orgs, I think part of the problem there is that even more so with Bazel, everyone's build looks slightly different and everyone hits slightly different problems and wants to make slightly different trade offs.

And so there's so little like unifying, infrastructure for that type of thing, like for that last little bit, unless you're using a very specific IDE like, you know, in the Java space, if everyone's using IntelliJ, like maybe you can make something work. I, I don't know.

Matt Klein: Yeah.

Keith Smiley: With that plugin or something, but...

Matt Klein: yeah, it - it's just, it - it's like one of those areas that, again, I'm no investing guru and, uh, I'm no like company business product guru, which I probably should be since I

started a company, but that's a totally separate topic... is, um, it, it just, again, I find it confusing where like, you have a company like Microsoft, which for all intents and purposes at this point, honestly, really owns the developer space. I mean, it's like, you know, I don't know what the [00:44:00] percentage is, but between GitHub and VS code and all of these things, it's like the vast majority of users are bought into Microsoft,

right? You know, they're running their builds in CI, they're using VS code, they're doing all these things. And it's like, why doesn't Microsoft have a more like ergonomic scale out hermetic system for these languages that's built into their entire ecosystem? And I, I, again, like it's obviously, I mean, they would do it if they thought that they could make money.

Um, and, and I guess on the same lines, it's like... you know, you, you probably have thoughts here. I know that over the years, Ba- Google's investment in Bazel open source has kind of like wax and waned, right? So it's like sometimes Google's doing a lot of work. Sometimes they're just like, uh, well, shit, I guess we're not gonna support those rules anymore at all.

I guess they're just community supported at this point. So I, I, I don't know, like is it just one of those... this is the reality of open source that until there's a company that kind of comes [00:45:00] along and puts a nice fit and finish on it, it's just gonna be a little rough around the edges? But at the same time, we have

at least two Bazel companies now, and I don't feel like they've really made any strides in making it, you know, more end, end user usable. So I, you know, again, I would just love to get your thoughts there. It just, I just find it fascinating that we've been at this for so long. These problems are so...

consistent. I get that everyone's build is different, but there's lots of common themes in terms of the problems they're facing or the IDEs that they're using and all of those things. And it's like you would think that at least with common languages and vs code ... It would be better, but it still kind of sucks.

So like

Keith Smiley: Yeah,

Matt Klein: tell us, tell us why. I mean, I, I, I just don't, I just don't understand why.

Keith Smiley: Yeah, I think you're right about a lot of the investment stuff. I mean also, you know, we're only, I mean, it feels like VS Code's been around forever, but like that did change at some point too. I think, you know, the reason [00:46:00] Google doesn't feel very invested in this is because internally they have their web IDE that I think is heavily integrated with Blaze and you know

Matt Klein: Right

of course

Keith Smiley: from what I hear works great.

Matt Klein: Yeah, sure.

Keith Smiley: I think, you know, a thing that's interesting about Bazel is I actually feel like there is opportunity for the companies you mentioned to improve this because they have a sound business model.

Matt Klein: I agree

Keith Smiley: with these remote cache, remote execution, like hosted services, and so they have... they are able to invest in things like this.

I think this one in particular feels like a big investment, right? For somebody to do.

Matt Klein: Yeah.

Keith Smiley: And like a, like a big bet for them. And I agree with the thought about Microsoft is like, I'm not sure how much money is in there for that specific benefit. Like I think it feels like a big dotted line to me to say, 'oh, if we get more people using Bazel, like they'll start using our product and paying us and so we should do this thing.'

Matt Klein: Yeah.

Keith Smiley: BuildBuddy actually did a project like that in the iOS space where they hired someone who also worked at Lyft, whose name was Brentley Jones, and he made, rules Xcode proj. And that was, you know, for [00:47:00] this like bridging Xcode, uh-

Matt Klein: we use that.

Oh

yes. Thank you.

Keith Smiley: Yeah, I think

everybody uses that at this point.

It's great.

Matt Klein: Right.

Keith Smiley: And you know, he spent a long time working on that at BuildBuddy and I, as far as I understand it, that was kind of the, the play was, hey, you know, this'll get more users here and maybe they'll use BuildBuddy.' So I think that they have, they are uniquely positioned, I think, to improve that stuff.

I think that's just a really big problem to tackle. And it feels tough with IDE stuff because, you know, we've been talking about VS code here, but here I am, I only use vim. I don't use VS code at all course. So like,

Matt Klein: oh, sure, sure.

Keith Smiley: Now you have this level of problem where it's like now you've gotta meet people where they are and now you've got a support cursor and like all this stuff,

right?

Matt Klein: It's more, it's just, it, it's, I just still feel like this area is ripe for, again, a better end, end user experience, right?

Keith Smiley: Yep.

Matt Klein: Um, and I feel like there are companies out there that should care about this, but they, I mean, I guess they just, they're too busy with their other business priorities. I'm coming at it again, just from an end user.

And I am constantly frustrated [00:48:00] right? About like, I get why we need Bazel, but from a usability perspective and all the ways that we've talked about, it just sucks. You know? And it's like, I, I feel like there has to be some, some middle ground, um, and I don't know how we get there. But, but I feel like... as we've seen some consolidation in certain parts of the industry,

like for example, I mean, does anyone remember companies like CircleCI and Travis CI? It's like, I mean, pretty much everyone uses GitHub actions now, right? But that only works for, I would say, like traditional CI. They don't offer any of the fancy remote build remote execution things. And I just, again, you know, whoever out there is listening.

I, I just feel like there is some integrated solution here, at least for the common languages and the common setups. Maybe not Vim, maybe not every language, maybe not Haskell, right? But it's like if you look at, you know, C++, Java, Python, rust, it's like go, you know, it's like the most common [00:49:00] languages and the most common IDEs,

I, I feel like one of these companies, whether it's one of the new AI companies or one of the. Traditional dev tools companies, I feel like someone's gotta do something better here. So anyway...

Keith Smiley: yeah, this is a good

time for me to pitch the... the Open Collective for Bazel contrib, which is a, like an LF... backed org that I helped found originally.

And they, they have this open collective where they, yeah, attempt to get, you know, companies to donate in this space.

Matt Klein: Yeah.

Keith Smiley: And then offer bug bounties for this kind of stuff. I think, you know, there's a big disconnect. There's always been a big disconnect with that and open source, right? Where it's like we talk about how all of these major companies are using Bazel, but very few are like funding it.

And I think

Matt Klein: absolutely.

Keith Smiley: Also now Bazel's to a point where if you try hard enough, like you can probably... can like make it work without doing a lot of open source contributions. Not saying that's a good thing, but like when we started using Bazel at Lyft, like we had no choice if we really wanted to...

Matt Klein: Yep.

Keith Smiley: Make it work for the long term. And so, yeah, it, you know, I think that there's some opportunity there where you get some sort of funding in the, in the right direction there to help out. But yeah, [00:50:00] I think there's a lot of opportunity there. I really hope that, yeah, some of these companies who are already like, profitable or, well, I dunno if they're profitable, but already stable in the Bazel space, can invest some more in that,

Matt Klein: yeah.

Cool. Well, anyway, we could talk about this for hours and hours and hours.

Keith Smiley: Yeah.

Matt Klein: But we, we are at time. Um, before we wrap up, just give us your, you know, either your hot takes or I'd love just to get your quick, and especially in the build space, your 5 and 10 year predictions on like, what, what do you think is gonna change?

Keith Smiley: Yeah, that's a great question. I think there's a probably a lot of possibility that people, you know, especially new companies, spend a lot of time on programming languages that maybe take longer to get to the point where you want to build system like Bazel, right? Like if you invest like in a, you know, if you, let's just say start today in like some rust backend, you could probably go for a long time without worrying about that.

I think there's a lot of potential in that versus all of these, you know, companies that maybe started 10 or 15 years ago who felt like their only choice was [00:51:00] to use C++ and now you like end up in this build system world or whatever. So I think that there's a chance that that changes a lot over the next few years.

But, you know, I'm also very interested to see what happens, you know, once, or, well, we didn't talk about Buck2, but you know, Facebook did like another version of their kind of build system. I think the interesting thing about that is that they took all of the learnings from Buck 1 and Bazel and then like improved on a lot of

pieces. So I think there's a, there's a world where you have a build system that is inspired by Bazel, that is not Bazel that has the opportunity to, you know, improve some of that stuff. I think that's a long term play because now the Bazel community is so huge and there's a lot of value in that, you know, working cross company to improve some of this stuff.

But, definitely interested to see where that goes because I think that, yeah, there's a lot of great pieces, but I'm not sure that, you know, this specific Java, Google owned code base will be the best thing to be using.

Matt Klein: Yep.

Keith Smiley: You know, forever.

Matt Klein: Cool. Well, thank you Keith. I could keep talking for another hour.

This was a super interesting conversation, [00:52:00] so, thank you so much for joining.

Keith Smiley: Yeah, thanks for having me.

Matt Klein: Well, thanks everyone. That's a wrap for this episode of Beyond the Noise Signals, Stories, and Spicy Takes. Huge thanks to Keith for joining and sharing his story. You can find this episode and all past ones, on the bitdrift YouTube channel.

And if you had fun, drop us a review, tell your friends or yell your favorite hot take into the void and just make sure to tag us. I'm Matt Klein and I will see you next time. Thanks again, Keith.