From Hacking Modems to Rewriting Lyft: Martin’s Journey to bitdrift

[00:00:00]

Matt Klein: All right folks. Welcome to the very first episode of Beyond the Noise Signals, stories, and Spicy Takes the show where we dig into the stories of the people shaping the future of app-based computing with a special focus on mobile. I'm your host, Matt Klein, the co-founder and CTO of bitdrift, also the founder of Envoy.

And in each episode we're gonna talk to engineers. Founders and technical leaders who've transformed the way their companies build and understand what's happening inside their systems. We'll dig into the challenges, the breakthroughs, the lessons learned, and we'll wrap it all up with their hottest takes.

So let's dive in and in our first episode, I am very pleased to have Martin here, who's my co-founder at bitdrift. I wanted to say. I am especially excited for this episode because, [00:01:00] Martin is one of the smartest and most talented people that I have met in my career. He's also, he's also a little mysterious.

So this is a good opportunity for me to get to know him better along with all of you. So welcome Martin. Thanks for joining us.

Martin Conte Mac Donell: Thank you for having me. Hi everyone.

Matt Klein: So, you know, I think because you've had a very interesting career, what I would love to do is I'd love to get to know you more.

So I think, you know, if you can take us through, you know, a bit of your overall history, I think that would be very interesting, and then we can go from there.

Martin Conte Mac Donell: Okay. Sounds good. So maybe let's travel 30 years back. I guess I started with computers very early on. Probably around 10, 11 years old, although now it's less clear the timeline, how it went.

But it was very early when I [00:02:00] got my first computer. I remember it was something called an MSX that had games and something called Logo. That is something that I guess that was my first introduction to programming, if you wanna call it.

Matt Klein: But, but most importantly, do you remember how much Ram that first computer?

Martin Conte Mac Donell: Oh yeah. It was 64K.

Matt Klein: 64K.

Martin Conte Mac Donell: And it was an eight bits machine, so that was. I mean, that was topnotch at the time, right? Obviously at that time I didn't understand what any of that meant. But, I did got my intro to Logo, which also was my introduction to geometry very early on, and, you know, telling computers what to do.

But I guess fast forward my dad, for reasons I don't remember, and I don't know how, because in Argentina it was very strange to get computers at that time. He got [00:03:00] a 286 machine, a PC, and a subscription to this magazine called, I guess, PC magazine where, you know, it was, I was waiting every month to get my subscription and it was, the first time I was reading the outside world of this computer stuff.

And it had things like, you know, Assembly instruction sheets on the papers, which I wanted to know what that meant. Right. So that's how I got serious and started learning Assembly as my first sort of like real programming language to do stuff. It was also something interesting on those magazines.

They had some phone numbers that you could call, and that was something called BBS where you could call and talk to other humans. And that actually was very interesting and I got, maybe, we'll call it destiny, but the random phone [00:04:00] number that I picked after figuring out how to put it on the machine was this sort of like soccer group BBS, right.

Where like, know grown ass men will talk about virus and how to break on stuff and I really didn't understand that much, but I pretended being someone else, probably a little older and talked to them, and this scene was like very elitist in a way. Just to give you an idea, you would call these numbers, and then you would get asked a bunch of questions to see if you can even enter this BBS.

Matt Klein: Sorry, this is calling a number with a person on a phone or this is dialing in,

Martin Conte Mac Donell: this is dialing in on a terminal ATX.

Matt Klein: Most people that are listening probably don't remember any of that.

Martin Conte Mac Donell: Yeah.

Matt Klein: Or the noises that the modem makes or anything else.

Martin Conte Mac Donell: Yeah.

Matt Klein: But I do myself have fond stories of that.

Martin Conte Mac Donell: Yeah, probably some people don't even know what a phone is other than their smartphone.

But, [00:05:00] yeah, so you will put the number on a terminal using something called ATX commands, and you will call or the computer through a modem will call someone else's computer. And then you will be presented with some passkey or letters on the terminal. And then the first thing they will ask you is some question to see if you deserve to be there, right? Especially on this scene. And then they will ask things like, you know, why are you here? Or what can you do? Or stuff like that. And then they will review your answers and then get back to you if you can get it on or not. But then this scene actually was my first sort of like my first opportunity to talk to other humans about computers.

And it was, it was very, I guess as a child or as a kid, it was very exciting to be part of something. So elitist or [00:06:00] whatever. So I guess fast forward that was my first sort of like talking to humans. Then maybe one year later or so, we got the first dial up connection. And then I got into something called IRC, which was like BDS.

But now you had real time, more people, more opportunities to choose and then you don't need to hang up and call another number and that kind of stuff. So it was a little, the mechanics were a little easier. But I guess because this still meant that you had to call a number and I was doing this at night, probably the whole night. At some point my dad is like, you cannot use the phone anymore. We can't pay the bills. And I'm like, I can't stop right now. But, but then what he did was, and because I was still using it, he got this sort of like phone lock box that he used, a physical key that would block my [00:07:00] connection for outgoing calls, but it will allow me to receive phone calls, right? It was a one way lock. And then I guess at some point I figured out how to sort of like fake the tones or the voltage that it needs to trick the box into thinking that there was a call. But then I was making the call and so at night I was using, my computer with dial up and my dad didn't even know I was using it.

So I didn't have my dad bothering me that I could not do it. Obviously that did not end up, you know, working out after one, two months that the bill only increased. , and so, but that was my first time I actually realized how breaking things is so fun, right? Is that, you know, I would call it the adrenaline of the moment when you realize you did something that you're not supposed to do, but at the same time, you know, it was very hard to get there and people think it's not [00:08:00] possible.

So that. That was my first, like, I like this moment, right? And then I guess, I think around that time I installed my first slackware, Linux. That was in the late nineties, and learned about the Linux kernel, learned about x86 assembly and learned C and then I actually made some contributions to a Linux kernel in the two X branch.

Had a few fights with Linus Torvalds also along the way and. I guess,

Matt Klein: tell us about that. Just give us, give us a little bit of a snippet there.

Martin Conte Mac Donell: Most of my early contributions, and actually most of my late contributions to the open source were anonymous, or with obviously other names. And I would be doing contributions with different names for different stuff, and he would get angry because he realized he realized I was the same person, but [00:09:00] he also, didn't like the way that some stuff was designed, right? And again, I was what, 15, 16 years old? Right. What did I know about the software design? And he, and I was, I was trying, the thing I was trying to do is make my video card work.

It was a, video card that was able to, I guess, hook a VHS and, get a digital video out of it. And there was no kernel drivers for it. And I only had Linux, so I had to build the kernel drivers for it. And he was angry about about the different things. The fact that it was proprietary sort of like hardware that had to reverse and stuff like that.

But anyway, long back and forth, I don't think much of it got merged, but a little bit did. And the way that you will contribute software back then was you will send a patch, right? Over email at best or user groups. [00:10:00] So anyway, that was my sort of like first contribution that was maybe constructive and not destructive.

And then maybe the censor a few years of that. And then I guess around 17 years old, I guess, I got my first job, the same curiosity and this like, InfoSec scene got me the job. For reasons that don't matter for this podcast, but I got my first job and it was on one of the .com. bubble companies.

That was probably, yeah, in the two thousands. I was 17 years old. I was technically wasn't legally allowed to work. But I got my first sort of like, coding job someone would pay me to code, which is where I learned Python very early on and also had my contributions to the Python language. But it was this sort of like telco [00:11:00] company that was very boring.

I mean, even if I like coding and I will find my own challenges, it wasn't very fun like the nights before that, right? And so I ended up doing different contributions and ended up, you know, contributing to this software called Kannel that is an SMPP gateway, and that was sort of the first software I was part of that ended up being a massive deployment.

The SMPP, by the way, is a protocol that is used to send SMS back and forth mm-hmm. Before the internet or before the, some of the messages goes through TCP. They, it was a protocol that would tell you how to send messages, how to receive messages and all this stuff. And that was also my intro to observability because much of that, whether a message is delivered or not had their own protocol. And, you know, trying to understand the [00:12:00] failure rate of messages and stuff like that was very important early on. But observability back then was pretty much looking at logs really, and maybe Nagios, if anyone remember that and, and SNMP and like that kind of stuff.

But you know, there was not much to look at other than logs. So that was early on. And then I actually built part of the observability system on Kannel as well. And this was relevant for when I was working because it was a telco company, that will send SMS and later on we actually implemented Kannel as our gateway system.

But you know it wasn't, it wasn't till the point that I got very involved into Kannel that that happened. But yeah, that was a very interesting, you know, massive deployment system that is still one of the biggest, SMPP gateway. Maybe [00:13:00] SMPP is not used too much in the US, it's very, actually widely used in the rest of the world. And that, I guess after that I worked at a few different startups in Argentina and it was fun. But you know, there are big struggles in Argentina in terms of the opportunities you have to build a company and how much trouble, or the incidental complexities that you go through in order to get money, get funding and that, all that stuff.

So I ended up deciding to move to the US. Which I, we made a decision with my wife to move to the US after visiting San Francisco for the first time. And then we were like, we wanna live there. How do we do that? And it actually is, well, maybe it doesn't sound simple, but it's more complicated than it sounds like, because first you need to get people to talk to you [00:14:00] which is not, maybe now, now is simpler than it was before, but it's not really easy to be like, "Hey, you know, interview me and do all the papers for me to go work in the US." It was actually not super, common in the industry to do that at that point. But I, was lucky enough that, and this is a, fun story by itself, but Stripe, very early on, Stripe, probably first years of Stripe. Greg, , who now, Greg Brockman, who now is the co-founder on OpenAI. He was the CTO back then at Stripe, and he built this capture the flag game that was a security sort of like game on which you need to get, find exploits to advance levels, and then they will give you a t-shirt at the end.

And I, didn't care about the t-shirt. I only cared about like, doing that because it [00:15:00] sounded fun and I was back in Argentina, maybe not... a little bored. So I did it and I got at the end, I think I got the first place or whatever on, that. And Greg called me for an interview. And then I barely speak English, let me put it that way initially.

So he called me and I did my best to convey who I were, but also I didn't wanna tell that much. I don't know. I will always. Was very paranoid about myself, like, why is this person calling me? I just wanted to have fun with that game. Now I realize it was a recruiting game, but at that time it wasn't obvious.

And then he ended up saying, "I will pay for, you know, your travels, hotels, just come talk to us." And maybe like, you know, come to San Francisco. Have fun. And so I did. And then I went to the Stripe office. I, met the brothers, and him and the early team at Stripe. [00:16:00] And one thing led to another and he introduced me with this other person, called Hong, that actually convinced me to join this another Y Combinator company that it was like a year in into YC and they needed help. And I'm like, what? Why not? I ended up joining them and not Stripe. But that, I guess that Stripe capture the flag is the thing that put everything in motion, I guess.

And then I joined, it was called Kicksend and I joined them. That was my actually first intro to iOS programming. Crazily enough. I was all my background was on backend and low level and that kind of stuff. And then I thought, why not start from scratch? And then I ended up learning iOS to help them.

And then with one of the founders, Brendan, we ended up writing this iOS programming book called iOS Seven in Action. So it was a fun [00:17:00] adventure in a way. And that company got acquired, or maybe aqui-hired by Lyft, which is how I ended up at Lyft. When I joined Lyft, I said I don't wanna work on a big company.

It wasn't a big company. It was maybe what? A hundred people? Less than that, but I said, this is my bridge to, you know, build my own company. And then fast forward nine years later, I was still at Lyft.

Matt Klein: I, I guess prior to getting into the Lyft stuff, one thing that I wanted to ask about briefly is, you're one of the unique people that I know from an engineering perspective, where you, you do all the things, you know, as I, as I tell people, you disassemble the iOS kernel or you do front end development, or you do mobile development. Most people are not like that, myself included. I guess, could you just, I'm just curious, could you briefly talk about, you know, how you, [00:18:00] either how you think about your ability to do that, or you know, the industry's move towards greater specialization and like what you think about all of that, because I think it's interesting.

Martin Conte Mac Donell: Yeah, I think there's a combination. First I will say that maybe in the US that this, having experts on one thing is more common than in other places.

I don't know exactly, you know, the reason why behind, I have some theories. Maybe you could argue that in Argentina, you have to be a little bit of, your breadth has to be more wide, because you need to adapt.

Matt Klein: Mm-hmm.

Martin Conte Mac Donell: If you join a company, maybe you do this, you do that. It's hard to find roles where you need to do the one thing, because in general, companies don't have the budget to have experts.

They try to hire more, and so, I don't think that's the reason I would say that in is more common on [00:19:00] other places. I think my reason really was, I hate to be blocked by other people. I really hate it and I think I learned a lot of different things just to unblock myself.

And, the fact that I actually was able to start so low level, that made me understand how computer works in a more, more clear, you know, full stack understanding. I think it made it easier to go up the stack, right? I will say once you learn how computers work from the Assembly world, learning JavaScript is not the more complicated thing, right?

And so I unblock myself like I learned how to design. I was a terrible designer and still are not a good designer, but at least I'm not the worst designer. I can unblock myself when I need designs. I can unblock myself when I need to do a frontend piece. I can unblock myself on mobile. It was more, I don't think I'm very good at all those things, but I got good enough that will unblock myself [00:20:00] on, those things.

And it is what it is. Like, I enjoy the challenge. But, but you know, it's more... I would unblock myself kind of thing.

Matt Klein: Yeah. Alright. Alright. Well then let's, let's dig in a little bit on the Lyft time because I think that's very interesting.

You obviously, did a lot of things at Lyft, including a lot of mobile work. So let's, let's tell a little bit of that story just in terms of the early days of Lyft and the challenges that Lyft were, was facing and I won't steal your thunder with the mobile work that you did, so I will let you talk about it.

Martin Conte Mac Donell: Yeah. So when I joined Lyft, I guess the most, fun part of the story is I joined Lyft and the iOS app was... how to say it lightly... it was very complex, let me put it that way. It got to a point.... so Lyft is one of those stories of a company that is the [00:21:00] traditional startup story that grows very quickly, where the company was something else at the beginning, then grew in a year. Right? And the code base, the code base was not able to support the scale, at the speed that we were growing, right? And so, a lot of complexity was introduced to the point that no human was able to understand what was happening at any part of the codebase. And when I joined Lyft, that it was in a very bad state in that way where we would have these crashes, bugs, and whatnot that nobody could understand how that was happening because following the breadcrumbs to get there, you will get lost on the complexity, right? And so my first thing that I did as a side job, at least was rewriting the whole application, right? And as I, started doing so... People [00:22:00] believed it was impossible. Maybe that is the reason I did it. Thank you everyone that said I wouldn't do it, actually, I really mean it.

Matt Klein: I was just gonna say for people out there, I, I talk about this a lot. I call this spite engineering. And I, I myself have spent a good portion of my career doing spite engineering.

But this is a fantastic story, so keep going.

Martin Conte Mac Donell: Yeah, so I, on the side, I started writing, writing it. And then at that time it was objective C, which is the thing I knew, and actually the only thing that anyone knew on the iOS world. But then actually a few months, or a few weeks in, I think, Swift was announced.

And it was this programming language. I, thought had the potential to be very good. It, it was, you know, it had this idea of [00:23:00] functional programming. We learned our own monads. So it was this idea that, maybe the future is somewhere else on the iOS world. But also Chris Ladner was involved on that, who obviously already knew for his contributions to LLVM and other stuff in, in the open source community.

And I actually, maybe because of him, I thought he would be... It wouldn't die like other languages in the industry because Go was also like on the beginnings and whatnot, and it was already on the way down. Maybe it took a wrong turn in the industry, but I think on, on Swift, it was maybe not clearly, but a big potential to be the future.

So what I said is, now that I'm rewriting the application, might as well start again, and do it on Swift as opposed to objective C. And so I did, and at that point, nobody [00:24:00] knew I was doing it, by the way. And so I was more... i'm a very under promise over deliver guy. I think the pressure of someone, you know, putting hopes on me doesn't work very well.

So I'm like, I'm not gonna tell anyone I'm gonna do it on my side. And I did it, I did it again on Swift and I got to a point very quickly where I had a version of the app, that was not only doing the same as what the Lyft app was doing at that point, or the Rider app, but it was even doing it better with the better user experience.

It had, like I was able to take care of more delightful moments on the app because it was simpler to do. And using more modern techniques, it was easier to get it right. And I also had, at that time, I thought it was a compiling, you know, sort of like safety guardrails that objective C didn't have because objective C is pretty much everything is on runtime [00:25:00] and Swift changed that paradigm or a little bit.

And then, and then I thought that would make us actually, build a more secure piece of software. But also another reason I did it was I thought we would position Lyft... if we get it right, and Swift is really the future. We could position Lyft as, you know, the company everyone wants to go work for, on the iOS side.

And actually maybe I was lucky it did happen and we actually were able to hire, you know, the best people I know.

Matt Klein: Well, but, step back for a second, what I wanna talk about is when you showed people.

Martin Conte Mac Donell: Yeah.

Matt Klein: You know, this app, how did that go? And then, you know, I, think that's a natural segue into maybe the, you know, the, the other part of our conversation, which is like, how do you roll something like that out and actually know, know that it [00:26:00] works?

I mean, it's like, it's obviously, it's an astounding accomplishment to single handedly rewrite an entire app, first, but you know, how do you go and convince people to actually ship that thing, and then how do you know if it's working?

Martin Conte Mac Donell: Yeah, so I will say two things. One is I, the person that got me or that got the acquisition through was the CTO at the time.

Chris Lambert and he was the one convincing me to join the company. And he, the moment I joined the company, he was giving me hints that the codebase might not be best-in-class, right? And I'm like, how bad can it be? I've seen, I've probably seen worse, right? And, and so, then there was this quiet moment from me, and one day I called, Chris and Logan, the CEO, to a meeting, and I showed them the prototype.

They didn't know what the meeting was for, right? I [00:27:00] told... I didn't tell the EA anything at all. And so Logan walked into the room not knowing what to expect, and I'm like... presenting the app working and all the ways in which it's better, but also the time that it took me to do. And my premise was that if I'm able to move quicker than the rest of the team, that might be a good indication that this might make us more productive.

And I showed them the demo and I was expecting them to shut it down immediately. And I was like, I was almost relieved that... this is it. I have, like, I already did the fun part, I will do the demo and then I will work on something else, right? Because they're gonna say no. And on the meeting they're like, "we wanna do this. Like, how many people do you need on the team?" And, and I said nobody. And so I, what I said is, but why don't we minimize the risk? Which at that point, I remember Logan telling me something like... "let's maximize the risks." And [00:28:00] I, which was something I thought it was a joke I didn't understand at the time, but now I understand and, it probably... it was the right thing to do, but I actually tried to minimize the risk and continue only myself.

Matt Klein: So then take us through just a little bit about how...

Martin Conte Mac Donell: yeah, the rollout.

Matt Klein: Yeah. Like how, yeah, how do you roll this thing out?

Martin Conte Mac Donell: So we continue again at the same, at the time I think there were probably 10 to 15 people moving on... writing code on the app. The, the main app continued building. I continued building in parallel.

I had to be faster than the rest of the team, otherwise it wouldn't work. And at some point we started talking about it internally to the team. There was another problem I didn't mention, which this is a new language. Nobody knows about it. We not only need to show them the new code base, we need to teach them how to program in Swift,

right? So everyone will need to start from scratch in a way. [00:29:00] So, I did that with individuals at the time, at the time that I was doing... writing the code at nights, you know, doing the political part at day. And then people telling me, I'm gonna break Lyft, that it's never going to work. Some PM's telling me that you should act now that I have the opportunity, why don't we write it on JavaScript?

And, and so all that, that kind of stupid fights every day. And then at night I would actually make progress. At some point we reached a point where it was feature complete with what we were shipping and actually ahead of time, because there were some PRDs that I had implemented before the other team implemented.

And so we said, we, we talked with, Chris and said, okay, let's put a timeline on which we're gonna make the switch. And the switch would be... you know, one day we ship the new app to a store. It's not [00:30:00] AB test, I don't think we had an AB test, very much mature system at that point. And so we thought about how can we, you know, mitigate the risk of shipping something that is broken.

And we didn't really find a way, there was no way to do it. And so we decided to just yolo and ship it. But then one of the things we had to figure out is how to actually send telemetry on the app that will tell us if it's working... first, if it's working, and then if it's working, you know, as the previous one was working on.

And, and then we set a few things like crash rate, you know, number of rides and that kind of stuff. Some of the stuff we measure from the server side, some other stuff we will, we will be sending analytics events and then query sql, every day kind of thing. That was the status quo on observability, at Lyft for a long time,

right? Obviously we matured the AB test framework over time. Then, the resiliency of the app, we, we did [00:31:00] a lot of work on the infra of the app to make it more resilient. But I think observability on mobile didn't actually had... the zero to one moment, not at Lyft, not in the industry. I actually think at Lyft we were ahead, but even ahead, ahead of the industry wasn't good enough.

So I will say one just quick thing, after maybe one, two years, I actually kind of rewrote it again, but not full, the full app. But it was more of a, a project where I proposed to the CEO, you know? Ride sharing has been going for a while. If you had a blank canvas where you can define ride sharing with everything that you know today, would you build it the same as what we have in the app?

Because at times what I find is companies get, you know, intertwined in the momentum and the inertia, and then it's hard to actually go and rethink fundamentals. So I actually wanted [00:32:00] to give everyone the opportunity to rethink those fundamentals and maybe build anything that we need to build from scratch.

Maybe that is building, building the whole app again, or maybe that is not, maybe it's just, you know, redo a few things. So we ended up doing that project, and for that project we did AB test the whole app, like the app was... A and B, two different binaries, which we had to build infra to actually be able to do that.

We build that with Keith, Keith Smiley my partner in crime on many of these.

Matt Klein: Yeah. Yeah. I mean, it, it kind of blows my mind to think about shipping something like that. But I guess, you know, that's almost the paradox of how small companies move as fast as they do as... there's just, there's a lot less guardrails, like to think of a larger company just rewriting the whole app and shipping it without understanding what's going on.

I think that's, that's a pretty incredible opportunity and a pretty incredible [00:33:00] accomplishment. So I guess like in, in, you know, the final few minutes that we have, I did want to, you know, fast forward towards the end of the Lyft time and, you know, talk a little bit about the problems, you know, that you were seeing

and I think we both worked together on various projects where some of the status quo was a bit painful. But, you know, I, I'd love to pick your brain a a little more, you know, you, you mentioned before that, you know, the status quo in the mobile observability world, you know, was poor and Lyft was ahead.

And I, I agree like Lyft. Lyft, I think historically was ahead. But I, I, I think what I'm more interested even less about Lyft and more just from your experience as a mobile developer, you know, how do you think that status quo slows teams down? You know, it, [00:34:00] it's like... when I come from the server world and I think about, you know, how the observability practice has evolved and how it's enabled teams to, you know, ship faster software and move faster and feel better about what they're doing...

and I don't see that personally on the mobile side. You know, I'm just curious to pick your brain on that one.

Martin Conte Mac Donell: Yeah, I think, well, I have a bunch of hot takes. Maybe I save that for the hot takes section, but I think, I think the... I will say that over the last maybe... eight years, I don't think that the status quo on the observability industry on mobile changed that much.

I think most companies still do what we used to do, at Lyft early on, which is they have a... maybe a feature flag system. If they are sophisticated enough, they use the feature flag system to drive business metrics, if not most just have the feature flag system [00:35:00] to make sure they don't break stuff, which they still break anyway.

But then they also have the way to know what the app is is doing is this idea of sending analytics events that are structured and they go into a SQL database. And all these events happen all the time, whether you have a problem or not. And that is the way that in case you actually find a problem, you go and query SQL to understand what happened.

And even then, you have to remember to add the analytic event, right? And so I, and, and on top of that... and so, at companies, and I know at Lyft we did that, it started including on the PRD, all the analytics events that we want to measure in terms of conversion, but also in terms of debug. And then I guess it started using what traditionally was meant for business analysis for debugging, just because

of the lack of a better way, right? Logs on mobile go nowhere. They go to dev null. And that's the reality, still is the reality. And so [00:36:00] you find a crash, maybe you can get the back trace, but what about the other 99 point 99 of the problems, right? How do you get the breadcrumbs? How do you get the, what's broken or if, even if you know it's broken, how, you know, how to know more about what the journey of the user was so that,

that was a big problem at Lyft. And one of the stories I, I think, you were very involved as well, is when we built, and this is the extreme of this problem, when we built Envoy Mobile, at Lyft to replace the networking stack... it was a very complicated deployment because now we are sort of like changing the industry status quo.

With something that was unproven. And so how do we measure that... all the assumptions we made about Envoy will work on mobile and BSD sockets work well, are right? And so we shipped... [00:37:00] when we shipped that, we still had the same mechanisms I mentioned before and what we found was, well, first we found that...

well, we didn't find it, but we already knew... but the internet is a dark place. And then there are all these weird cases on which someone goes on IPV six on a Marriott hotel and then everything breaks. And how do you know that that's happening? And that is what's happening, right? And so we actually ended up adding a lot of telemetry all exclusively for Envoy mobile that end up affecting all the customers because now we need to send all these telemetries and our own experiment on Envoy mobile was down on conversion because of the telemetry. So it's this like, chicken and egg problem. So that's, that's the beginning of us thinking... can we make this such that we can know more on what's happening?

But for... only for the cases that fail, right? That, that was actually the early beginning of [00:38:00] bitdrift. Yeah. Yeah. I mean it's... I don't know. It, again, for me, coming from a mostly server side background, it, it's fairly amazing to me to think about developing software without those tools. You know, the ability to see metrics on demand or do all of those things.

Matt Klein: Well, cool. I do have a couple of, final questions for you, but I guess, you know, is there anything else that you wanted to share either, about your Lyft time or what we're doing at bitdrift or really anything else?

Martin Conte Mac Donell: No, mostly I think that, people listening from the industry, I'm pretty confident, us talking with customers, we know already that this is not a Lyft problem it was, it is an industry wide problem.

So I think whether it's bitdrift or not, I think that the industry on mobile observability [00:39:00] has to advance forward. I don't think that the models of "let's do the same that the server does" is actually... a good model. And, and so there is a good opportunity here to rethink how to shape this for the future.

Matt Klein: Yeah. Cool. All right. Well then let's, let's move into our final section.

we - we can do some hot takes if you're interested in sharing.

I, I think I'd like to learn a little bit... I know that you're an avid barbecuer, so it's like, I, I would love to learn a little more about your barbecue meat preferences, because I think it's also interesting, but feel free to share that or anything else that you would like to close out with.

Martin Conte Mac Donell: Yeah... so Argentina, we call that asado, when I moved to the States, I learned like the art of slow cooking, but I still, my love is on the fire... open fire, asado grill, real grilling. I actually am in the process of building a huge [00:40:00] mega project on my backyard... is just to have an Argentinian sort of like grill, which has proven to be harder than I thought because...

Matt Klein: you're you're actually building it yourself?

Martin Conte Mac Donell: Well, no, sorry. Someone is building it for me. I designed it, but someone is building it for me. But you know, the city codes are not... it is like uncharted territory. We go present that to the city and they're like, "what? What is this like fire box that you're trying to make?"

Matt Klein: You need, you need a permit to build a grill.

Martin Conte Mac Donell: You need a permit to build a grill. It's also in a enclosed structure and you also need the permit for the structure, so.

Matt Klein: Alright.

Martin Conte Mac Donell: Yeah, and also, you know, it's open fire you don't wanna take risks on code.

Matt Klein: I'm assuming though, that, that your grill will also have some electronic computing components, like...

Martin Conte Mac Donell: it might, it might, might not.

I think actually the grill itself is actually being handmade at the mo at the moment by this company that is amazing. So, yeah. Alright. Excellent. Any, any, last thoughts that you would like to share? [00:41:00] No, I think, I think my last thoughts are maybe hot takey, but I actually think two things. One is, the crash reporting world is dead... is my, maybe a big hot take in the sense that I think we got to the industry

to a point where we think about crashes as the most important thing. And obviously it's important if you're affecting your user, behavior, but at the same time, it's a very, very, very small percentage of the user experience and companies tend to over index on that. And I, I actually first, you know, face experience.

I'd know that putting developers at times to fix crashes is actually a worse investment than putting them on fixing other problems that are more important. And so I think the crash industry got, to a point to be mature just because how, well they did some things like showing, you know, units of work that you know, are bad, that you wanna fix to make a dream kind of thing.

And that, [00:42:00] that actually plays on the psychology of people and, and leaders and whatnot. But I actually think that we need to move forward the next step on that one too. Yeah, that and I, and one more thing I wanna say. Yeah. Feature flags. Oh, what a... what a mess those are. I think, feature flags... people like to think of feature flags on mobile as a mathematical problem or a probalistic problem, and it's not... like people do not understand how experiments... intertwined experiment play with each other.

So... the whole industry, you think you're doing well, feature flag, you're not.

Matt Klein: Yeah, I - I think we should do a whole other episode just on feature flags, so we should, we should talk about that separately.

Martin Conte Mac Donell: Yeah.

Matt Klein: Anyway, I, I think we're at time. Thank you so much. This was extremely interesting. Thank you for being our first guest on the podcast and yeah.

Martin Conte Mac Donell: Thank you Matt.

Yeah. Maybe the next time I should interview you, so you tell your story.

Matt Klein: Perfect. Alright. All right.

[00:43:00]