David Crawshaw: From Tailscale to exe.dev, and Why AI Agents Need Their Own Computers

Episode #25 - David Crawshaw

===

[00:00:00]

Matt Klein: All right, folks. Welcome to another episode of "Beyond the Noise: Signals, Stories, and Spicy Takes," the show where we dig into the stories of the people shaping the future of app-based computing. I'm your host, Matt Klein, co-founder and CTO of bitdrift, as well as the founder of Envoy Proxy. Each episode, we'll talk with engineers, founders, and technical leaders who've transformed the way their companies build and understand what's going on inside their systems.

We'll dig into the challenges, the breakthroughs, the lessons learned, and we'll wrap it all up with their hottest takes. So let's dive in. Today, I am so thrilled to have David Crawshaw, who is the co-founder and CEO of exe.dev, or as we were talking about prior to the show, is it exe.dev? I guess we'll talk about that in a second.

Um, and before that, obviously the co-founder and CTO of [00:01:00] Tailscale, and he likes computers. So David, welcome. Thank you so much for joining us.

David Crawshaw: Thank you for having me.

Matt Klein: Yeah.

David Crawshaw: It's true, I do like computers.

Matt Klein: So, so let's, let's actually start with that. So w-we'll, we'll fast-forward, but is it exe.dev or is it exe.dev?

David Crawshaw: Yeah. When we were prototyping, it was exe.dev because I was thinking very MS-DOS, you know, when we were doing it. It was all like, "Oh, executables. That's fundamentally what we're doing. We're executing things. Let's call it that." Uh, and then everyone called it exe, and, you know, there's only so much control you have over language, right?

And so if everyone kept saying exe, I'm like, "All right. Fine. It's exe now." That's, uh, what it is.

Matt Klein: It's actually funny, and people make fun of me for this, but I got my start programming for Windows. Yeah. Um, and you know, when-

David Crawshaw: Why would they make fun

of you? Windows

Matt Klein: is

David Crawshaw: good.

Matt Klein: Yeah, and it's like when you get your start doing a certain thing, you maintain habits, you know?

It's like the main goes in exe. That's the, that's the directory where it goes. [00:02:00] And, um, when I first laid out the source code for Envoy, you know, like main is in a directory called exe, and I think people have, people have comment on that over time. Anyway, um, we'll get back to your company, but where I really like to start is, um, you know, to learn about you.

Like tell us how did you get into computers? Uh, te-tell us a bit about your background.

David Crawshaw: Yeah, I mean, I've always been into computers, and I've always been into them because I had nothing else to do. You know, I grew up in a small city in northern Australia and, you know, it was the tropics. It was hot outside and, you know, we had air conditioning, and so I'll stay inside and I'll use a computer.

And, uh, now what do I do with it? Uh, figured out how to, you know, make it do things. Uh, and it's a lot of fun making computers do things. And so, you know, I grew up with it just like reading and writing, you know? It was... just from a young age. Uh, and it was all very much a sort of a self-taught, you know, play around thing.

Uh, and then I was, you know, very luckily [00:03:00] was, you know, in a very good environment for learning how to use computers because I lived on the floor above a, a, a medical center as they're called in Australia, but I think you'd call it a, a primary care practice here in the US. Uh, which, uh, my parents ran. My father was a, a general practitioner.

And as a hobby, he taught himself programming and wrote the medical record software the, the place used, and, uh... I hacked on that basically, and learned programming that way. And so it was a, a variant of sort of a dBase III-plus style programming language called Clipper, which was this DOS-based thing that, uh, it was an interpreted, uh, language based on P-code, that was, you know, it was actually really nice.

It was a very, um, it, it was a language that focused on brevity and focused on building Tuis. And so, you know, five lines of Clipper got you n- a nice little DOS Tui, uh, menu system, which is a perfect place for someone to learn to program. It was much easier than, say, learning to program using [00:04:00] JavaScript or something today, where you have complex DOM models and all of HTML and CSS to fight.

You know, I, I was just, like, laying out pixels by, like, counting X and counting Y- ... and saying, like, "The text goes here," and, you know, a kid could figure that stuff out. And so it was awesome. And then, you know, there was a stack of other stuff we did. We went, uh, we went on a trip to Hong Kong once and, uh, found a, a magnetic card reader Which plugs into the PS/2 port on a computer and acts like a keyboard, and like you swipe the magnetic card and it basically types stuff out.

And so then we built a, a thing for reading people's Medicare cards, they're called in Australia. So we could s- you know, the front desk could swipe people and save, you know, a little bit of, uh... check-in time, uh, which was just a thing no one was doing back then. It was a l- it was a lot of fun. You know, then I learned, like there's a, there's an algorithm for, uh, checking if a card number is valid, right?

Which applies to credit cards, and Medicare cards in Australia use the same algorithm because there's a lot of numbers that fit in that pattern. So, you know, it's, it's a fun way for a kid to learn data structures and algorithms. And, you know, I think the most exciting thing was, uh, [00:05:00] uh, buying a fancy laser printer to print s- prescriptions for people, uh, which talked PCL 5e, I think, which is sort of a PostScript, uh, alternative from HP back in the day.

And you know, that was... Learning that was something. That was, a lot of fun.

Matt Klein: Well, I mean, yeah, if, if you were able to get a printer to work-

David Crawshaw: Yeah

Matt Klein: ... that probably bodes well for your future career, um, because-

David Crawshaw: Yeah. Well-

Matt Klein: ... as we all, as we all know- So- ... getting printers to work is, uh, is fun ...

David Crawshaw: uh, they were easier to make work back then.

They were mechanically more reliable, and they had much simpler... Uh, you know, there were no drivers or anything. This was a network printer. You send PCL 5e to, uh, o- over IPX or to the right IP address if you turned on TCP/IP. Uh, it was, it was much-

Matt Klein: Yeah

David Crawshaw: ... much simpler. I, I couldn't make a printer work today.

Matt Klein: It, it is actually-

David Crawshaw: That's too hard.

Matt Klein: It's actually funny. I feel like printers have gotten worse over time, and I don't understand how that's the case. And people might be wondering, "Why do I have a printer?" Well, when you have kids, like you typically have a printer to print some stuff. And I finally got fed up recently and got rid of the inkjet whatever, which is always [00:06:00] breaking.

It didn't work. And I got an old school laser printer. And you know what?

David Crawshaw: Yeah.

Matt Klein: That thing always prints.

David Crawshaw: Nice.

Matt Klein: Every, every single time. That's, uh- It's pretty nice.

David Crawshaw: Yeah. Laser printers are much nicer. Uh, yeah, and it's really interesting what's wrong with printers. I think there's a lesson in software engineering hiding in there that, uh, I think is a bit of work to extract.

But it's, uh, it's all the kerfuffle around the printer that goes wrong, right? It's the layers and layers of driver machinery. Like a few years ago, I bought an HP laser printer because I was under the mistaken impression HP was a good brand, because it was when I was a kid. Uh, and, uh, uh, for printers anyway.

I don't know anything about what else they make. Uh, and, I was astonished because I have one Windows machine in my house, and there was a, uh, a strange executable running on this Windows machine in the task manager that I couldn't figure out, that just appeared one day. And like, you know, I, I have no business poking around the Windows task manager because I have no idea what's going on in there.

But I was like, "Where did this come from?" And I realized that when the printer was on [00:07:00] on the network Uh, it was sending out broadcast packets which triggered something inside the Windows that I was running to start up an HP driver that was built into the Windows machine. I'd never printed from this machine.

I never would.

Matt Klein: Spooky.

David Crawshaw: Yeah.

Matt Klein: That is, that's kind of insane actually.

David Crawshaw: It-- From a, from a network security perspective, it is. The idea that a, a random packet triggers an enormous amount of, uh, third-party code to execute on a, on a Windows machine, doesn't really bode well for it being like a really, you know, secure thing that, uh, can't be hacked remotely.

Matt Klein: But also like, is that code just part of Windows? Does HP pay them, or was it a computer that had all of the, you know, third-party garbage that had been pre-installed?

David Crawshaw: I, uh... No, no. It was, I installed the Windows on it, and it was a basic Windows. That's- And, uh, as far as I can tell, Microsoft quite reasonably ship mountains of third-party drivers-

Matt Klein: Yeah, yeah

David Crawshaw: with Windows.

Matt Klein: Right, right.

David Crawshaw: Which is fine.

Matt Klein: Yeah.

David Crawshaw: But the third-party drivers also come with like- extra machinery around them-

Matt Klein: Sure. Yeah

David Crawshaw: ... and apparently HP get to [00:08:00] ship lots of programs in there, and, uh, you know, it's... It didn't make me feel very good about Windows, I have to be honest.

Matt Klein: Yeah.

David Crawshaw: I, I came away from that being like, "What's, what's going on?"

Like-

Matt Klein: Yeah, yeah.

David Crawshaw: Yeah. And so yeah. It's, uh... You know, if I'd plugged it in via USB or something, maybe I could understand it, but this was literally Ethernet and, like, uh, a random, you know, broadcast packet.

Matt Klein: Was-

David Crawshaw: Like, this

is not cool ...

Matt Klein: was that your, uh, first major, major interest in networking? Or, I mean, h-had you, had you been previously interested before that event happened?

David Crawshaw: Oh, no. This was, this was just... This was very recently.

Matt Klein: Oh, oh,

recent.

David Crawshaw: This

was-

Matt Klein: Okay. Sorry. I-

David Crawshaw: Yeah. This was well off the tail scale. No.

Matt Klein: Oh, oh, oh, wow. Okay. No.

David Crawshaw: Net-networking-

Matt Klein: I, I, I- Yeah.

David Crawshaw: No, this was-

Matt Klein: Yeah

David Crawshaw: ... like three years ago or something. Uh, no, networking-

Matt Klein: That is scary.

David Crawshaw: Uh, yeah, right. This is the world we... Yeah.

This is post, you know, Microsoft's driver validation-

Matt Klein: I,

I thought... Right ... systems. Like, I thought this was a story from, like, the '90s or something.

David Crawshaw: No, no.

Matt Klein: No, no, no,

no.

David Crawshaw: This, this is-

Matt Klein: Wow

David Crawshaw: ... like Windows 10 or something.

Matt Klein: Yikes.

David Crawshaw: You know, something...

Matt Klein: All right.

David Crawshaw: Yeah. Yeah. No, really, it was very upsetting. Like, you know, if this was Windows XP before [00:09:00] SP3, I'd be like-

Matt Klein: Right

David Crawshaw: "Oh, yeah, of course it's terrible." But, like-

Matt Klein: Right

David Crawshaw: ... no, this is, uh-

Matt Klein: Yeah

David Crawshaw: ... uh, yeah, the world's a funny place. Uh, no, no, I got into networking, uh, back in that medical practice because the whole software my father wrote is we got a Novell NetWare file server, and it talked IPX. This was pre-TCP/IP network. We wired up the building with coax cable and put a PC, an MS-DOS PC, on every doctor's desk.

And like, you know, I was small, so I did some of the coax cable wiring through the roof and I, I, I, you know, and I complained every step of the way. You know, I was, I was not very fond of this.

Matt Klein: Yeah.

David Crawshaw: Uh, but you know, we, uh, uh... It was all built around, uh, uh, SMB basically, or some variation of SMB.

Matt Klein: Yep.

David Crawshaw: So the files, the, the, the, the database files lived on the Novell NetWare server, and each of the machines basically just were, uh, using some variation on a, on one of the file locking protocols that mostly worked, uh, to edit these databases directly.

[00:10:00] Uh, and then, you know, I remember trying to figure out TCP/IP because I heard that's what the internet ran on. This was all... You know, we didn't have the internet at this point.

Yeah, that's right.

Matt Klein: Yep.

David Crawshaw: And, uh, uh, you know, it existed, but, like, you would have to do a long distance phone call 2,000 miles from where I lived to, to get-

Matt Klein: Yes, sure

David Crawshaw: onto an ISP. So, you know, we... You know, at some point I migrated it all to TCP/IP and then, you know, Windows came out and I started playing with that, and eventually, again, on another trip to Hong Kong, we got, uh, some Slackware CDs and, uh, uh, got Linux running, uh, not easily and, you know, figured all that sort of stuff out.

Uh, and so, you know, networking, uh, was a thing I experienced pre-internet of. It was a more interesting way to use computers. It was to use several of them. And that feels like a world that has long passed. You know, modern software You know, sitting on a local LAN for the most part reaches out to the, you know, uh, servers on the internet to solve all of its problems.

Matt Klein: Yeah.

David Crawshaw: And you know, honestly, I suspect that experience, you know, was a useful piece of information in my head when designing Tailscale.

Matt Klein: Yeah, makes sense.

David Crawshaw: [00:11:00] Uh, I had two machines at home and, you know, this was- Tailscale existed at this point. We had a customer. But I had these two machines at home running Tailscale, and I was like, "Well, it's really annoying they don't talk directly on the local LAN."

I was like, "So let me, let me make that work." And so I went and figured that out. So, you know, all very... And you know, it was the same TCP/IP, you know, and everything looked really similar to how it looked 30 years ago, so.

Matt Klein: Yeah.

David Crawshaw: So.

Matt Klein: Yeah, for sure. Cool. So I, I guess, so you started out o- obviously, um, in your, uh, parents' business, which is quite cool.

Um, uh, I don't know. T- tell us how did you make it to the US? Um, like how did you... Because you- Oh ... obviously spent a bunch of time at Google. Like, what was your, what was your path to come over and, and start there?

David Crawshaw: Yeah, that was not a technical story at all. You know, I just met a girl, followed her home.

That's, uh... You know, she's from Berkeley, so I live in Berkeley now. That's it. Yeah. That's, uh- Uh, I'd never had a job in computing at all-

Matt Klein: Yeah

David Crawshaw: ... until I moved here, and a Google recruiter contacted me, and I took the [00:12:00] BART to the Caltrain down to Mountain View to, you know, do an interview there, uh, which was really fun.

And you know, I was like, "Oh yeah, I'll come work here for like six months. It'll be great." And you know, this was 2009 or something, and I had no industry experience, so they offered me like a base salary of like $90,000-

Matt Klein: Yeah

David Crawshaw: ... or something to go work at Google. I'm like, "Wow, that's so much money." And so, and so you know, I went down there, and then I ended up staying there for like seven years because they kept having things to teach me.

Matt Klein: Yeah, for sure.

David Crawshaw: So-

Matt Klein: Yeah ...

David Crawshaw: it's a, it's a great fun place. And you know, I'd done a lot of open source work before that because that's basically how I learned computing, was like first, you know, Usenet and then mailing lists and then, you know, using GCC to build stuff. That was the, the, the... E- everything, everything I really learned about computing, I learned through using open source, uh, software-

Matt Klein: Yeah

David Crawshaw: of some sort somewhere. And so, you know, it's, uh- I, uh,

Matt Klein: I love talking to people who are around our age or older. I, I don't know how old you are. We're probably about the same age.

David Crawshaw: Yeah,

probably.

Matt Klein: And, um- It is [00:13:00] just, I think people today cannot, especially with the modern tools of the last couple of years, like they cannot comprehend what it was like to learn some of these things before you had the internet, or especially now before you have all of these LLM tools.

You know, the fact that you basically had to go get a book or-

David Crawshaw: Yeah

Matt Klein: ... you had to get the source code and you had to read it. It's like there was no other way to actually learn anything. Um, so I guess like tell us a bit about your experience learning. I think that's pretty interesting.

David Crawshaw: Yeah. I mean, I-- again, there was no formal training where I was, uh, that was of any value to me.

Uh, and you know, the, before college there was nothing, of course. And also there's, there weren't really libraries with anything technical that was useful where I grew up. So all of the information I had to use had to come from the internet once we got the internet. And you know, it was quite a, it was quite of a journey getting the internet, right?

We went through several phases there, uh, including like, you know, scamming an account out of the local [00:14:00] university, which had its own 64 kilobit ISDN line, uh, connected to the internet, and so we could dial into it, uh, like a bulletin board thing and, uh, do a bunch of stuff on some SunOS machine.

Matt Klein: Yep.

David Crawshaw: Uh, but yeah, it was really spending way too long reading Usenet and then mailing lists just trying to piece things together.

And honestly, it's not a great way to learn is, is the truth. Uh, I spent years out in the weeds doing really dumb stuff is, uh, is approximately what I say. And like I, I got value out of doing that. Like it was years and years on topic trying to build things but, you know, trying to build complex Linux software by like downloading every single dependency manually and running the configure file and making it and trying to get the prefixes right and like digging through all of these libraries that like I know nothing about to try and like assemble a program out of like 50 pieces.

Yeah. Like what a waste of time. Uh- Uh, but also I learned a lot about build systems doing it. So-

Matt Klein: Yeah. I mean, but like you, you might say that you were doing dumb stuff, but o- [00:15:00] one thing I wonder when you say that is that, I mean, as an industry, like if you look at what people were doing 20 or 30 years ago compared to what they're doing now, I mean, wasn't everyone doing a lot of dumb stuff?

You know, it's like, do you think-- I guess what I'm asking is it, it, it seems like, um, the, um, just like the proliferation of information that we have today, I feel like is able to up-level people theoretically in a way that was not possible, y- you know, like 20 or 30 years ago.

David Crawshaw: No, I, I, uh, I totally agree with that, but I also I have seen people learn from, you know, people who are, you know, very knowledgeable.

Matt Klein: Sure. Yep.

David Crawshaw: And I had that experience at Google too, of like it was the first time in my life when I was surrounded by people who just knew so much more than me and had been doing it for longer than me-

Matt Klein: Yeah

David Crawshaw: ... and could teach me stuff. Uh, and the experience there is, uh, what, what they help you do is, [00:16:00] they help you choose, uh, more interesting paths to spend your time on.

If you assume that-- And, like, this is actually not generally true of people in all subjects. Uh, you have to find something you really care about to spend hours on it. But if you assume that you on your own will spend a certain number of hours and you with a teacher will spend the same number of hours, then it's better to have a teacher.

Matt Klein: Absolutely.

David Crawshaw: Yeah.

Matt Klein: Yep.

David Crawshaw: That's, uh, someone to, you know, help guide you. And, like, that's not always true. Like, sometimes you need to, you know, do it yourself to actually find the passion to spend the hours on it-

Matt Klein: Yep

David Crawshaw: ... in which case you're better without the teacher.

Matt Klein: Yeah. So. Yeah. I, I mean, I-- As I mentioned before, I started my career at Microsoft, and ob-obviously before that I had been in college, and, uh, you know, like, you don't know anything about programming.

I mean, I had done programming since high school, but as you said, you, you don't really know anyone to learn from. And even when I started at Microsoft, obviously the company in the early 2000s, uh, you know, around 2002, 2003, was fairly mature, but there's a big [00:17:00] variety of code quality. And then I had the opportunity to work on the Windows NT kernel, and for those people that don't know out there, this is a very high-quality piece of software.

Like, not necessarily the, the entirety of the Microsoft ecosystem, but the kernel, uh, was, is a pretty impressive piece of software. And I remember at the time just the exposure to that body of code and the people that worked on it was so formative for me. I mean, like, it, it, it is one of the big reasons that I am the engineer that I am today, is having that kind of even indirect mentorship of that code.

And, I mean, it sounds like you had a similar experience for when you showed up at Google and you got to learn from people that had been doing it for a long time. Um, but I, I think it's really amazing that, um, you know, this industry still is very apprentice-based.

David Crawshaw: Yeah.

Matt Klein: It's like you can learn a lot from other people, for sure.

David Crawshaw: I, I think that's to the industry's benefit, honestly. You know, it is [00:18:00] The vast majority of things that I learnt, I learnt through an apprenticeship style. Like, I, I think a well-executed PhD is you apprenticing yourself to someone.

Matt Klein: Sure. Yeah.

David Crawshaw: And like they don't call it that because, you know, why, why say apprentice when you can use a $5 word and sound fancy?

But like fundamentally, there's a person who knows what they're doing, and they tell someone who doesn't know what they're doing how to do what they're doing better.

Matt Klein: Yeah.

David Crawshaw: And like, you know, you work on a real thing as you do it, and that's, that's the best way to learn basically everything.

Matt Klein: Yeah.

David Crawshaw: Like, you know, why, why would you do anything else?

Matt Klein: Yeah.

David Crawshaw: It's, uh... If anything, I feel like most of the computer science courses people take, uh, teach really fun stuff. Like I really, I enjoy theoretical computer science. It's a great branch of mathematics and like, you know, I, I recommend people study as much of it as they enjoy studying-

Matt Klein: Yeah

David Crawshaw: ... because like, you know, even if it's not useful, it's fun and like really interesting and like great for the brain.

But like it's not great for like learning how to write computer programs.

Matt Klein: Yeah.

David Crawshaw: Like that's a different skill.

Matt Klein: For sure. Yep, yep.

David Crawshaw: And so

Matt Klein: [00:19:00] Yeah.

David Crawshaw: And

so like if that's what you wanna do, you know, find another way to learn it and, you know, find a programmer and like go build something with them.

Matt Klein: Yeah.

David Crawshaw: So.

Matt Klein: So, so tell us a bit about, obviously, as you mentioned, you spent seven years or so at, at Google and then, um, and to tell us a- at least briefly about that time and how that led you into founding Tailscale.

David Crawshaw: Yeah, I mean, the thing that really excited me about Google that I'd never been able to do before then was work on something HPC-like. Like I wanted to do something with lots of computers or very large computers because all of my work had been basically solo open source stuff, which meant my universe was constrained to the PC or the Mac or some machine on that scale.

And, uh, I had recently been doing a data analysis project with my now father-in-law, who's a professor at Haas, and you know, I'd, I'd built this thing because the dataset didn't fit in one machine, and so we found 10 machines somewhere at the university, and I, I tried to build something across them and, you know, hacked up some [00:20:00] very crude MapReduce-looking thing-

Matt Klein: Yeah

David Crawshaw: based on what I'd read about MapReduce. And I was like, "Oh, well, I could go learn this from the people who know what they're doing." And I did. I, I went and worked in logs and search quality and, uh- Some search infrastructure work, uh, when I started there. And it was great, and, you know, it was, uh... I learned, you know, that was my first real exposure to sort of large scale, low latency distributed systems, which is just a universe of its own, uh-

Matt Klein: Yep

David Crawshaw: and enormous fun. You know, uh, supercomputers are great. That's, uh, you know, well, whatever you actually call them. But you know, definitely those giant, you know, data centers full of machines are, at Google when you write programs that span thousands of machines, like it's fundamentally a supercomputer by some shape or form.

Uh, and, you know, that was a, it was a really wonderful learning experience. And then I went and did some product stuff. I worked on Google+ for a little bit, uh, again, helping them with that infrastructure side of things. That was really formative, working on a product that didn't work out. Highly recommend it.

Matt Klein: Yep.

David Crawshaw: That's, uh, everyone should do it. The, I met some great people there. You know, it's, it's, uh, still [00:21:00] really stands out in my mind. And then I ended up on the Go programming language team, because I, I, did one of the first major deployments of Go at inside of Google.

Matt Klein: Oh, wow.

David Crawshaw: Yeah.

Matt Klein: Yeah, that's cool.

David Crawshaw: Yeah. I, uh, I converted, uh, an existing project over to it, uh, quite successfully, and then converted more. Uh, and, that led me to the team itself. And then I spent some time in sort of compiler runtime linker stuff. It was mountains of fun. You know, it's, uh, it's, uh, it's a, it's an extremely mature side of software engineering where you can really feel, you know- You know, 60 something years of, you know-

Matt Klein: Yeah

David Crawshaw: solid theory behind, you know, how we build parsers and lexers and, uh, compilers and type systems and, uh, the type checkers, and all of the machinery, all of the beautiful algorithms around how we sort of then produce machine code in optimized ways, you know, SSA forms, all those sorts of things. It's great.

It's, uh-

Matt Klein: I

was gonna say too that, I mean, I, I have never worked on that stuff professionally, [00:22:00] but, you know, I do, a- as m- many engineers do, I like doing per- I like doing performance work. And at least for me, I would imagine that one of the things that's so satisfying about that kind of work is that the feedback cycles are very local.

It's like you can, like, tweak-

David Crawshaw: Yeah

Matt Klein: ... the compiler. You know, you can look at the assembly code. Then obviously I would imagine in a place like Google, you can, like, test the compile code at scale to see if it's actually working well. So I just would imagine that it's a very satisfying feedback cycle to, like, see really nice improvements.

I'm not sure if that was your experience, but it just strikes me that it would be very satisfying that way.

David Crawshaw: It absolutely was, especially coming from, like, the large C++ systems that took minutes to build-

Matt Klein: Yes

David Crawshaw: ... and like-

Matt Klein: Right. Yeah, for sure.

Yeah

David Crawshaw: ... you know, my first project at Google ran at 2:00 a.m. Mountain View time, uh, once a day, and, like, I would wake up in the morning and see how it did and, you know, spent months tuning it-

Matt Klein: Right

David Crawshaw: based on that. It's, uh, uh, that-- Comparing [00:23:00] that to, like, I put -s on it and then look at the assembly and, you know-

Matt Klein: Yes.

Absolutely. Yeah ...

David Crawshaw: my iteration time is two seconds.

Matt Klein: Right.

David Crawshaw: It's, uh, you know, it's, it's, uh, you know, they're radically different kinds of programming.

Matt Klein: Right.

David Crawshaw: They're both very fun in different ways.

Matt Klein: Yeah.

David Crawshaw: Uh, and yeah, it's, uh, highly recommend it. It's, it's a very strange sort of side of computing. Like, it's generally done inside big companies because It, it only makes sense to spend the resources there, uh, on those sorts of projects-

Matt Klein: Yeah

David Crawshaw: ... because they're not sort of self-contained products that you can build smaller businesses around.

But, you know, it's, it's well worth doing. So, and again, just, you know, enormous, you know, you know, some of the, some of my colleagues worked on the HotSpot compiler before that.

Matt Klein: Absolutely. Yeah.

David Crawshaw: Other programming languages that date from around the time I was born. You know, it, it's, uh, uh, enormous amounts of, knowledge that have been accumulated over many decades, which is, you know, really fascinating.

So the only downside of it is all of the strongly held opinions.

Matt Klein: Right.

David Crawshaw: Uh, everyone has really big opinions about programming languages. Uh-

Matt Klein: Yeah

David Crawshaw: ... one, one of the joys [00:24:00] actually is spending a few years working on them is throwing all of those opinions away by s- like, slowly holding each one, one after another.

Uh, and, you know, it's, it's great to come out the other side of that and just sort of be able to look across them all and be like- Yeah ... "I can build programs in any of these."

Matt Klein: Yeah.

David Crawshaw: It's, uh

Matt Klein: I- i- is there any, just, like, from your, from your time working on the Go team, is there any, I don't know, like, particular, I don't wanna say, I don't wanna say argument, but, like, was there any particular thing that stands out to you, you know, that the team just, like, really had a hard time reconciling that you can share?

David Crawshaw: Ooh, that the team had a hard time reconciling. I feel like the Go team was quite well aligned on the kind of language they wanted to build. And so the, the issues were always in the details around sort of the, the small of it. There were, there were sort of big questions, and the team would constantly have to deal with, uh, opinions from outside-

Matt Klein: Yes, of course

David Crawshaw: uh, around-

Matt Klein: Yeah

David Crawshaw: ... you know, the question of, like, should a [00:25:00] programming language be garbage collected?

Matt Klein: Of course. Yeah, for sure.

David Crawshaw: What fundamental limits does that put on-

Matt Klein: Yeah

David Crawshaw: ... the kinds of software you can write? And people have very strong, and I think, and, and you know, uh, I would say generally incorrect opinions about garbage collection.

Matt Klein: Sure.

Yeah.

David Crawshaw: Uh, especially when you, like- When you start digging through modern systems and you realize there's an extremely sophisticated garbage collection algorithm running inside the firmware on your NVMe, and like, you know, you're, you're not even noticing it as you program, and you don't think about it as all-- at all, uh, and, you know, somehow IO works.

Uh, and, you know, there's a, there's a lot of complex allocation going on-

Matt Klein: Yep

David Crawshaw: ... inside the Linux kernel, which, you know, it's not garbage collection by traditional sense, but there's a lot of complex arena machinery going on.

Matt Klein: Sure.

Yep.

David Crawshaw: And the layers and layers of software we use that uses algorithms at least as complicated everywhere, and then people say, "Oh, you couldn't possibly build that with a garbage collector."

I'm like, "I, I don't even know where to start arguing-

Matt Klein: Yeah

David Crawshaw: ... with you," because this is just-- [00:26:00] That's, you know, it's, it's not a true statement, but also like, yes, there are complicated trade-offs, and we should figure out how to make them. Anyway, that, that sort of stuff got old, honestly. Like-

Matt Klein: Yeah

David Crawshaw: ... it, it's not fun discussing those sorts of things.

Matt Klein: Yeah.

David Crawshaw: Uh.

Matt Klein: But I mean, but, but that, that would be like a discussion with the larger community, I, I guess, and like not-

David Crawshaw: That's right

Matt Klein: ... uh, not like a design decision that you had to make internally or something along those lines.

David Crawshaw: Yeah. There was no one on the Go team who said, "Oh, we shouldn't use garbage collection."

Matt Klein: Right. Yeah.

David Crawshaw: Like, you know, basically like if you're there, you're, you want to, you wanted a garbage collection language.

Matt Klein: Absolutely. Yeah.

David Crawshaw: And it's, uh-

Matt Klein: Yeah

David Crawshaw: ... um, you know, it's-

Matt Klein: Yeah ...

David Crawshaw: so, you know, I, I would say that in that sense, the team was very well aligned. But obviously, you know, a-as, as usual, there's a, there's a lore about this, right?

Uh, I forget his name. One of the Haskell people has this about, uh, uh, the time spent arguing in programming languages- ... uh, at different levels.

Matt Klein: Yes.

David Crawshaw: And like, uh, it's the level of syntax where-

Matt Klein: Right

David Crawshaw: ... all, all the energy is spent.

Matt Klein: Yeah. All right.

David Crawshaw: Uh, and so, you know, there's lots of syntax discussions, uh, which I mostly stayed out [00:27:00] of because I didn't have anything honestly useful to add.

Again, it was a very-

Matt Klein: Right.

David Crawshaw: Mm-hmm ... it was a, it was a team with a mountain of experience that, you know, didn't need my input on those sorts of things.

Matt Klein: Yeah.

David Crawshaw: So.

Matt Klein: Cool. All right. Well, then tell us a bit about how you went from there to Tailscale.

David Crawshaw: Yeah. I mean, my last project at Google, I spent most of a year working on the Fuchsia, uh, operating system, and I ported, uh, the gVisor network stack as Fuchsia's first network stack, because it was a microkernel design.

So again-

Matt Klein: Yeah

David Crawshaw: ... and, and again, garbage collector and network stack, how can you do it? It's like, well, don't allocate for every packet and you're okay. It, uh, it turns out. Uh, but uh... I left Google about the time I had my first kid, because, you know, it's, uh, I wanted to spend more time with the kid. Uh, and then, Avery, who I'd met at Google New York, left around the same time, you know, a few months later.

Uh, and then at one point, you know, we got on a call and we're like, "Oh, we should start a startup." We're like, "Yeah, let's do that." And so let's go start, you know... And so we started a startup. And, you know, we did it [00:28:00] with, uh, uh, uh, someone from undergrad Avery knew, David Carney. And so the three of us, you know, went looking for something to do.

And we, we came at it from like a, "Oh, we-- there's a lot of complicated things in computing that are too hard. We should make them simpler." And this, this comes back to the printer thing of, you know, uh, computers in general get better in every way. They're so much be-- you know, the, the, the watch I wear has so much more compute power than the first machine I compiled Apache 1.3 on.

Matt Klein: Yeah.

David Crawshaw: You know, we, we, we live in a glorious time. But also some things get worse too, and like they get worse just because things get complicated and we've made bad product decisions and, you know, we built the wrong abstractions. You know, we, we make... You know, it's a bumpy ride. And there were a whole set of things like that in computing.

Like we stared at Kubernetes and said like, "That's way too complicated a job scheduler. Like we should be able to make that simpler." We couldn't figure it out though. It was too hard. Uh we were like, like, "Well, I'm sure it can be simpler. I don't know how though." And, you know, it was, it was just a genuine failing on our part to, uh, to sort of wrap our heads around it.

Uh, I am continuing to [00:29:00] wrestle with that every day right now. Uh, but uh, we, we decided like let's, let's just go and consult for a bunch of companies, and see their problems, and try and solve them and find a product that way. Because we have like a general shape of like, let's build simpler stuff-

Matt Klein: Yeah

David Crawshaw: was very much the thing in our mind and like, you know, there's a whole lot of products we're willing to work on, so let's just, uh, find one by use. Uh-

Matt Klein: How did you... I, I mean, I think, um, you know, as, as any founder will tell you, starting a company is like monumentally hard. Um-

David Crawshaw: Yeah

Matt Klein: ... and I, I, I, I think doing the way that you did it is actually really interesting, is to go and just kind of consult and look for problems.

I, I'm actually curious about that. T- tell us more about like how did you find those people, like that you went and consulted for? Because I feel like putting yourself in the right position to see the right problems, to take that strategy is obviously really important. So I'm just curious like how you all thought about that part of the process.

David Crawshaw: Yeah, I mean, you basically just [00:30:00] call up people you know. Be the summary, you know. You just, uh, people are doing things. Uh, it's actually a problem if you've spent your whole career in a mega corp like Google, and then you go to call people up, and you'll find you don't know very many.

Matt Klein: Yeah.

David Crawshaw: Because like everyone in your Rolodex works at Google, and like they don't have problems you can solve.

Matt Klein: Right.

David Crawshaw: Like, they've got problems, but you can't solve them. That's, uh, you can't solve them because they're complex, interconnected political-

Matt Klein: Sure

David Crawshaw: ... issues within mega corps-

Matt Klein: Yeah, of course. Yeah

David Crawshaw: ... that you can't solve from the outside.

Matt Klein: Yeah.

David Crawshaw: Uh, what you want are people who are off doing their own thing. They're working for small companies, they've started small companies, those sorts of things.

And like one of the joys now of having been, you know, a, a startup founder for, you know, uh, I'm on, let's see, it's been seven years, I think, is now I know just mountains of these people. I, I could just go through LinkedIn and like most of the people I'm connected with these days are people in orgs of all sizes doing all sorts of things.

And so there, that was the biggest sort of, uh- Challenge for me was the lack of people.

Matt Klein: Yeah.

David Crawshaw: And you also, I'm from [00:31:00] a different continent, so like I, I don't have like college buddies I, I know or anything like that who, you know, it's, uh, uh, who, uh, who were useful for these problems. You know, I, I called a few people and like chatted with them about some things.

But the, uh, the one we eventually found was this small Canadian bank who, uh, Avery had done some work for many years ago. And so this was very much a case of like, you know, his pre-Google career. Again, he'd done a startup before, involved meeting lots of people and like, you know, knowing people. And so again, you just, you just start with where you are, and then you ask them, "Hey, do you know anyone who has troubles, uh, you want to talk?"

And th- this is that again, very non-technical side of starting a company and like it's just talking to people and like it's, uh, such annoying advice just to talk to people-

Matt Klein: I, I, it-

David Crawshaw: ... but it's also true.

Matt Klein: Well, I, I mean, like newsflash to those that are listening, um, much of starting a company is non-technical.

David Crawshaw: Yeah. That's, uh... Oh, absol- absolutely. It's, uh, I mean, and that was really, that was such a, such a visible thing for us at Tailscale because we were three technical founders.

Matt Klein: [00:32:00] Yes, for sure.

David Crawshaw: Um-

Matt Klein: Yeah ...

David Crawshaw: and you know, with speed running like, "So sales, how does that work?"

Matt Klein: Yes.

David Crawshaw: And you know, question, you know, big questions like that, "What is marketing exactly?"

Yeah. You know, it's, uh, you know, it's, uh, uh, you gotta learn a lot of stuff. Uh, but I would say the single biggest thing is just talking to people.

Matt Klein: Yeah.

David Crawshaw: And like that's just... It's, it's such shallow advice, and yet it's just so true simultaneously. Uh, y- And so many cold emails.

Matt Klein: Yeah,

yeah.

David Crawshaw: You know, it's, uh-

Matt Klein: So, so this, so this bank situation-

David Crawshaw: Yeah

Matt Klein: I mean, you, you like went in and talked to them and I'm guessing somehow it came out that they had some like really shitty VPN software. I'm just like, what was the, like what was the context, right?

David Crawshaw: Yeah. It's really, it's really fun.

Matt Klein: Like how, how did it come out of that?

David Crawshaw: Yeah. So like when you talk to someone, you're like, "So what problems do you have?"

And you, you have to reframe that question several times because if you just ask someone what problems they have, they don't know. Like no one, no one keeps like a running list of my problems.

Matt Klein: Yes. Right.

David Crawshaw: Like actually, maybe we should. Maybe every morning we should write a list down or something. But, uh, at least I don't.

And so, uh, I've got a lot, but [00:33:00] I can't tell you what they are off the top of my head. But you, you talk to them until you figure them out. And the problem, the biggest technical problem they were facing is they had recently done, uh, they'd had an auditor come in and do a phishing test, which they'd failed, which is, you know, extremely normal.

Everyone fails phishing tests.

Matt Klein: Yeah.

David Crawshaw: Uh, you know, you click on stuff in emails. Uh, we're, we're all guilty. And if you don't think you're guilty, then you're really in trouble, uh, because, uh, because, uh, trust me, you have. And, uh, the auditor gave very standard advice, which is good advice, which is put two-factor on everything.

That'll make, that'll make this a lot harder. And so they went to try and figure out how to do that, and their problem was they had a whole lot of software they'd written like 15 years before that in a sort of a .NET system, uh, built around Windows file sharing. And, uh, SMB on Windows at the time has no obvious two-factor story, and I still don't think it does.

And so as consultants we're like, "Okay, well, I'm sure this isn't the first bank that's had this, given that banks are still out there running COBOL. I'm sure there's banks out there still running this [00:34:00] sort of software. How do you solve this problem?" And so we asked some people, and the answer was very simple.

It's, uh, VDI, the virtual desktop machinery, uh, like Citrix Metaframe, all those sorts of things. Uh, and no one likes this for reasons that are actually a little bit mysterious to me, and there's really interesting product questions in there, but like virtual desktops are just not a fun experience of a computer in a computer and like-

Matt Klein: Yeah

David Crawshaw: you're operating something remotely. Uh, and so we're like, "Well, is there anything else we could do to achieve this?" We're like, "Well, you could... You know, it's too bad you can't just like two-factor onto the network periodically." Like, you know, once a day you get knocked off the network and you have to-

Matt Klein: Yeah

David Crawshaw: two-factor back on, and that, that's it. That's your two-factor into the software. And, uh, we're like, "Well, that's a good idea. Let's set them up a VPN and have them use that." And it was great because they actually wanted to try having some people work from home, uh, occasionally, which was a new thing for them.

And so we even had a small group within this small-

Matt Klein: Mm

David Crawshaw: ... company that want... It was a perfect test group for it of like, well, they need a VPN for the... And they didn't have one yet, uh, for their laptops, 'cause up until that point they'd all done their work in office. So we had a... Uh, and then that's your [00:35:00] ideal scenario for any kind of new software is like you don't just have a small group, but you have an even smaller group within it to try it on.

And so we did the obvious thing and we downloaded some, you know, VPN software to try out and, uh, it, it was all very unpleasant. Like the most unpleasant experience of all was one piece of software, uh, that I won't name 'cause I'm sure they've fixed it by now, uh, but this was true in 2019, was two-factor worked by it would pop up a box with your username and password that you'd type in, and then after you've typed in your password, you would take the code from your multi-factor device and append it to the password.

And, you know, uh, you know, you can, you can, you know, as an engineer, I'm sure you can see the long train of decisions and issues-

Matt Klein: Sure

David Crawshaw: ... that led them here, right? Like they couldn't-

Matt Klein: Right, yeah

David Crawshaw: ... fix the UI, and so, yeah. It's, uh- Yeah. Anyway.

Matt Klein: Nice.

Okay.

David Crawshaw: That was just one of those moments where I was like, "This is so dumb.

I could build this in a weekend. What are we doing?" And, you know, say the usual-

Matt Klein: Famous last

words ...

David Crawshaw: cool thing. Yeah.

Matt Klein: "Build it in a weekend," as we all say, but yeah.

David Crawshaw: Yeah, [00:36:00] exactly.

Matt Klein: Sure. Right.

David Crawshaw: And then Avery's like, "Well, I heard about this WireGuard thing that looks cool." And I went and looked at it, I'm like, "That is cool."

And then I looked at it and it's like, "Oh, it's written in Go. This is, like, super easy to use."

Matt Klein: Yeah.

David Crawshaw: I'm like, "Oh, I really can do this in a weekend." And, like, the great thing is, like, the problem of how do I configure my network in WireGuard turned out to fundamentally be like, I have to distribute a whole lot of config files-

Matt Klein: Mm-hmm

David Crawshaw: uh-

Matt Klein: Yeah

David Crawshaw: ... because I've got to get the keys right in each one and then hand them out to everyone. I'm like, "Oh, cool. This is a, this is a config file distr- distribution machine that I have to build," like a little control server that distributes these things. I'm like, "That's super easy." And it literally was a weekend to get the first prototype working.

Uh, and then, you know, it ran on a machine in my closet in Manhattan, and like that's, uh, uh, that's what they used for their first few deployments. And then, you know, uh, I started using it myself and, like, we started designing, the, the more formal version of it. And like-

Matt Klein: Yeah

David Crawshaw: ... if you read our early design docs, it's very much like, oh, well, each office will have a...

'cause this bank had multiple offices, will have a concentrator that everyone connects to, and then these will [00:37:00] have interconnects between them across the internet, and it looked like a very traditional sort of VPN architecture with like clients and servers and whatnot. And like I was also trying to use it at home, and I'm like You know, uh, at one point I was like: Well, this is, this is too hard.

Uh, you know, the building this is gonna take forever. Let's just hack something up that's wrong, and, like, the thing we'll hack up is I'll have one control server on the internet for everyone, them and me, and we'll, uh, we'll have the machines just connect directly, like in a, like a mesh network, but like a mesh network with no intermediate nodes.

Matt Klein: Yep. Yep.

David Crawshaw: So it's just a pure endpoint, uh-

Matt Klein: Yeah

David Crawshaw: ... so it, it, it met the endpoint principle, which was important to me. And you know, I, I remember writing an email to Avery and David about this, and, like, Avery wrote back like: "Well, mesh networks never work." And he's like: "But let's just try it anyway because we can fix it later."

'Cause we had fallback plans for how to do this. We're like: "Well, we can work around it for a bit with STUN and TURN, things like that, and, like, worst case, we can rip this out later and do the more traditional-

Matt Klein: Sure

David Crawshaw: ... design. This just lets us get there." And the weird thing was trying this thing that never works and that [00:38:00] we knew would never work because, like, you know, we, we-- but we'd all played with mesh networks of various sorts before, and they're all too complicated and really fiddly, and they go wrong in all these ways.

It turns out if they're just pure end-to-end networks like this, uh, they're not complicated and fiddly, and your fundamental problem just comes down to NAT traversal. And there was really good industry proof of concept out there that NAT traversal is a solvable problem in WebRTC.

Matt Klein: Right.

David Crawshaw: And there were even RFCs written about this, which are quite terrifying RFCs to read.

Matt Klein: Interesting, yeah.

David Crawshaw: But, like, it was, it was known technology how to solve this. And so we said to ourselves: "Well, yeah, STUN and TURN and all of those NAT busting techniques on top of this will solve this problem, and we can keep scaling it for a long time." And so we just kept pushing on this idea that we knew was wrong and would never work Because, you know, it solved my home personal problem and it solved the bank's problem at the same time.

Matt Klein: Yeah.

David Crawshaw: Uh, and that's how we sort of stumbled our way into the product. Yeah. So you know, it's a, it's a fun story, right? Like we started a company not knowing what to do. We found a customer who had a problem. The solution to their problem turned out to be really different-

Matt Klein: Yeah

David Crawshaw: ... [00:39:00] than what everyone else thought it was.

And then trying to like expand the scope of what that problem could solve to something that was fun for, uh, ourselves-

Matt Klein: Absolutely. Yeah ...

David Crawshaw: uh, created the company.

Matt Klein: I, I

mean, we could spend the entire, we could spend an entire hour talking about Tailscale and, obviously, we're a Tailscale user. It's a fantastically-

David Crawshaw: Right

Matt Klein: successful, awesome product. Um, I, I would love just in the interest of time, I, I'd love to fast-forward and talk a bit about your new company, um, because I, I'm, I'm interested both in what you're doing, at least in terms of what you can share, um, but I'm also interested in the journey of how you got there.

Like what, what led you to decide that, you know, you had done what you wanted to do on a day-to-day basis at, at Tailscale and it was time for something new? And take us through the same journey of like how you identified this new thing that you're going after now.

David Crawshaw: Yeah. I mean, the, I mean, the, the basic story is, you know, I, I, I hired myself out of a job at [00:40:00] Tailscale, right?

Like there was nothing useful for me to do. Like the team had grown to 40 something engineers.

Matt Klein: Yep.

David Crawshaw: Uh, and it just needed standard professional engineering management. It didn't need me coming up with wacky product or feature ideas every day because the company was in a phase of its life where it was so clear what we needed to do was just take what we had and make it work really well for-

Matt Klein: Yep

David Crawshaw: bigger and bigger customers. And like that's a very You know, the, the best thing to do when you're doing that is to not come up with a bunch of wacky feature ideas-

Matt Klein: Yeah

David Crawshaw: ... and try to solve new problems.

Matt Klein: For sure.

David Crawshaw: And so, like, I was in this place of like, well, I need to just go off into a corner and build my own thing for a bit.

And then I was like, well, I really wanna work on LLMs because they're really cool. This was sort of, uh, GPT-3-ish or so around this time. Mm-hmm. Uh, and, you know, there's no way to fit LLMs into the Tailscale product, like to actually use an LLM itself. Like, obviously Tailscale has Aperture now as an LLM gateway style product, which is very nice, and I think actually fits really well into sort [00:41:00] of, uh, uh, Tailscale as a product.

But, uh, uh, I was very interested in like, well, can I generate code with these things? Because like this was at a point where you could kinda get a 10-line JavaScript program out of an LLM that sometimes worked, uh, and that was very exciting. And so I really wanted to work on that, and it was just really obvious that the best place to do this is just, you know, not at Tailscale.

Matt Klein: Yep.

David Crawshaw: And so, you know, I stepped back and, you know, it was a, you know, it's a process, but, you know, I fi- I fully stepped back, uh, at the Series C, uh, and then started another company with a old friend of mine, Josh Bleicher Snyder, uh, and w- we-- 'cause we were both very excited about LLMs. Like, well, we can make them do something, surely.

And like we kept trying to do things and, you know, we, uh, we prototyped out a, a lovable like product that like would never work because the models were no good. And, you know, the models kept getting better. And then we built, uh, a coding agent before tool calling really worked Uh, which we launched about the time Claude, Claude Code came out.

And, you know, then we spent last year fighting our way through tool calling messes and trying to make that [00:42:00] work. We tried... The, the thing, our, our coding agent, uh, didn't work because we were very focused on the problem of we want to run the agent in a sandbox, which was not a thing anyone was talking about at that point.

So we... What would happen is you'd try using it, and it would boot up a container, and it would push your Git repo into the container. It would do everything there, and then it would pull Git commits out and put them on-

Matt Klein: Yeah

David Crawshaw: ... branches outside.

Matt Klein: I, I was gonna say, like, whoever solves this problem is going to make a lot of money, because, I mean, like very, very brief segue, but I, I literally spent a day last week making a new set of tooling to move my, like, AWS production credentials out into a secure shell that I know that the agent cannot get into.

David Crawshaw: Yes.

Matt Klein: Because it's simpler than, like, trying to run my other shit in a sandbox. Like-

David Crawshaw: Yeah

Matt Klein: ... it does not work. So now what I've done is I've refactored my computer where, like, the stuff that the agent must never, ever have access to is in a place where by [00:43:00] password, like, it cannot get access to it. And then-

David Crawshaw: Right

Matt Klein: I just, I just run in YOLO mode, which no one wants to hear, but I mean, it's like such a difficult problem right now.

David Crawshaw: Yeah.

That's, that's how I develop. And, you know-

Matt Klein: Yeah

David Crawshaw: ... it's... We built, we built the infrastructure for that, and that, that was, like, why we built Sketch originally, was, like, this problem of, like, well, these agents are so much more powerful if you give them root-

Matt Klein: Yeah

David Crawshaw: and you just let them do stuff.

Matt Klein: Yeah.

David Crawshaw: Because, like, you, you'll be working on a problem, and it will just start TCP dumping on the problem and, like, looking at the PCAP and, like, it'll figure out what went wrong.

Matt Klein: Yep.

David Crawshaw: And, like, it can't do that unless you give it a computer. And so it's so obvious it ha- you have to run in YOLO mode, right?

Like, all of the sandbox machinery built into Claude Code and code, it's all the wrong shape. Mm-hmm. It's so obviously wrong.

Matt Klein: Yeah.

David Crawshaw: And so we, we said, like, "Let's just give them computers." So we, we started with gVisor, uh, after we started with Docker locally. Then when in the cloud, we started with gVisor. We kept running into the problem of like, well, again, you need to give your agents a computer.

They've got to run CI, and they, they couldn't inside gVisor. Like, your, your CI would try to Docker compose or something, and that wouldn't work.

Matt Klein: Right.

David Crawshaw: And so that didn't work. In the end, like we need [00:44:00] VMs. Like it was so clear. Like we just give it a full Linux machine, like let it go nuts, uh, manage the VMs well.

Uh, and then we realized, you know, we built a lot of infrastructure for this, and we realized this was actually the most interesting thing we'd built because there's a, there's a problem that agents create that no product at that point solved. We now have many, many competitors who have copied this idea.

Matt Klein: Yes. Right.

David Crawshaw: Uh, but the, the core idea is like I have some amount of computer, and I want as many VMs as I can fit in it. And it's very much just like Docker containers of like I just wanna fit as many containers as possible into the compute-

Matt Klein: Yep

David Crawshaw: ... and they compete, and like if I run out, well, it's my job to manage them.

And so we built that product of like we start as many VMs as you want on a fixed slice of compute, and so that's, that's the core of what exe is. And then in that, we run, uh, we have our own agent called Shelley because it's a web agent, because we like web interfaces better than terminals, uh, which, you know, again, it has root inside that VM and it's fine.

Matt Klein: Yeah.

David Crawshaw: And, uh, we've been, we've been building all the machinery we need for it. So we develop [00:45:00] on Shelley. We treat it like a cloud agent. We develop in there, and we've built a bunch of machinery around it that is really important. So like you were talking about secrets, you're right. The trick is never give the agent the secret And so it turns out the best way to do that most of the time is a thing we call integrations, which is effectively, uh, a, a TLS reverse proxy that lives outside the, uh, the VM.

And so you go into, uh, you go into your exe.dev terminal, you set up an integration, and what it is, is, uh, uh, you give it the URL of the service and it, it generates an internal URL you can use.

Matt Klein: Yeah.

David Crawshaw: And then you say like, "Add this HTTP header, add this basic auth," and it injects the API keys, uh, as you, as your agent tries to reach the service.

Matt Klein: Yeah.

David Crawshaw: And this is great because you give it access to these things and, like, it, uh, uh, it can never exfil the key. It can never do anything really stupid with it.

Matt Klein: Yeah.

David Crawshaw: And then we've, we've augmented that with like OAuth-level support and, uh, uh, identity federation for AWS, so it can reach S3 [00:46:00] buckets and stuff like that, again, without ever getting hands on a credential.

Matt Klein: Yeah. I, I was gonna say, I, I don't know what you're willing to share in terms of how the underlying system works, but I started my career, as I mentioned, working on operating systems and virtualization and those kinds of things. And I, just from the outside, because I haven't worked on it professionally myself in a number of years, I think it's so interesting how much work has continued to happen in, like, the micro VM space.

Obviously, I think AWS started there with their Firecracker, and I'm sure Google has something and everyone else. And I just, I find it really interesting that people have put in so much effort y- you know, to bring up these miniature VMs that don't have all of the bio surface and all the other crap that you don't need.

And again, I don't know what you're willing to share about the underlying technology, but would love-

David Crawshaw: Yeah, um-

Matt Klein: But, but would love to learn a little more a- about, like, how you thought about building the actual infrastructure, because I think it's really interesting.

David Crawshaw: Yeah, I mean, we... Interestingly, the, the, the VMM itself [00:47:00] is not something we've done a lot of radical work on.

We started with, uh, Cloud-Hypervisor is its name, which is a, uh... It actually shares a lot of code with Firecracker, which is the AWS one. Uh, but it actually has a different, uh, historical lineage. I think it was supported by, uh, Intel for a long time. It's again, it's another one of these Rust-based-

Matt Klein: Yeah

David Crawshaw: VMMs.

Matt Klein: Right.

David Crawshaw: I think it's actually well worth pointing out that, the microVM thing was kind of an advertising thing, sort of. It was, it was, it was like two elements sort of wrapped together. But when people think about microVMs, they think about fast starting and lightweight, and that's not re-

That's not really the VMM that did the work. You can get fast-starting and lightweight VMM- VMs with, uh, with QEMU-

Matt Klein: Yeah ...

David Crawshaw: today, and you could at that point too, because the heavy lifting to make them fast and lightweight is actually inside the guest's Linux kernel, and all the heavy work was actually done by kernel developers.

Matt Klein: Interesting. Yeah.

David Crawshaw: And it's, uh... The work is starting Linux without turning on all of its hardware detection [00:48:00] machinery-

Matt Klein: Right ...

David Crawshaw: and giving these, these nice, simple virtual interface things onto the-

Matt Klein: Yeah

David Crawshaw: ... the VM. And that work was done for many years before Firecracker existed. Firecracker, I think, mostly came out of crosvm, which is actually a Google project, which is the open source VMM they built for, uh, uh, Chrome OS.

Matt Klein: Yep.

David Crawshaw: Uh, and so, you know, the, the lineage here is always very complicated.

Matt Klein: Absolutely.

David Crawshaw: Obviously Firecracker's its own thing these days.

Matt Klein: Yeah, yeah.

David Crawshaw: But the, the thing the microVM actually invented that was good was a, uh, a hype- a VMM with, uh, very little actual code in it. Like, there's a million lines of code in QEMU, and it's really, really complicated, and it's got all the BIOS emulation machinery-

Matt Klein: Yes

David Crawshaw: and all these other variations inside it. And so it's a scary code base, and you look at it and you say, like, "Well, is this safe? Can something escape QEMU?" And, uh, that's a really rough question to ask because it's just such a big code base. It's, it's kind of scary. Uh, the beauty of, uh, Firecracker, cloud hypervisor, all of these things is, you know, you can actually build a hypervisor in, like, 40,000 lines of code.

Matt Klein: Yeah.

David Crawshaw: There's nothing to them, [00:49:00] and that's actually... That's wonderful. And like, you know, I, I credit the microVM for that and for... You know, that turns it into a really beautiful sort of security boundary. And obviously we've had a couple of KVM escapes recently, which show that it's not a perfect security boundary.

Matt Klein: Yeah.

David Crawshaw: Uh, but it's, it's something, and it's, uh, it's definitely better, it's definitely better than all of our other options.

Matt Klein: Yeah.

David Crawshaw: And so, you know, I'm a, I'm a big fan of the hypervisor stuff. Uh, but, uh, our, for our product, because we have always-on VMs, you know, we treat them like real servers. They're not...

There's a, there's a lot of, there's mountains of hypervisor, uh, uh, innovation going on. Some of the innovation is, like, preserving network connections across live migrations, which is not something we think is important for our product, uh, at this point at least. Because almost all your connections are web and, uh, they, they handle connection resets really well.

Matt Klein: Yep.

David Crawshaw: Uh, and we handle SSH connections outside at our proxy level.

Matt Klein: Right.

David Crawshaw: And so, uh, you know, r- those aren't very important. Uh, other, other really great innovations in the hypervisor space, uh, focus on scale to zero.

Matt Klein: Yes.

David Crawshaw: It's all about [00:50:00] being able to turn them off. There are these beautiful products out there that, like Interpose the post-Postgres protocol and implement the heartbeats for the pro-Postgres protocol-

Matt Klein: Yeah.

Okay

David Crawshaw: ... and turn off the VM.

Matt Klein: Right,

right.

David Crawshaw: And, uh, bring the VM up to answer a query.

Matt Klein: Yeah.

David Crawshaw: It's beautiful. But again, uh, that's not particularly interesting to our product.

Matt Klein: Yeah.

David Crawshaw: All of our work is at the disk level, and so it's all file system machinery for us.

Matt Klein: Sorry, when you say at the disk level, it's, it's, um, both, like, in the replication of disks, the management of disks.

David Crawshaw: Yeah.

Matt Klein: Because, because b-based on hearing what you're saying, which I think is really interesting, is that at least right now at your product stage, if you take the hypervisor at least to be a somewhat solved entity, it seems like your problems are related to all the normal distributed system things, like-

David Crawshaw: Yes

Matt Klein: orchestration, networking-

David Crawshaw: Yep

Matt Klein: ... like all, all of those things. So would you say that that's where, like, a lot of the innovation is happening right now, is just how you manage this whole fleet of stuff?

David Crawshaw: That's where all our work is.

Matt Klein: Yeah.

David Crawshaw: So, like, you know, other... Again, other companies are building other things.

But for us, we built our own [00:51:00] global load balancer. We've built our own SSH proxy that's actually-

Matt Klein: Yeah

David Crawshaw: ... pretty complicated. There's a fun blog post on that, um, because it's, it's a surprisingly hard problem. Uh, and, uh, we've done a lot of work on sort of managing the, uh, the, the virtual block devices themselves.

Matt Klein: Right.

David Crawshaw: And I would say the most interesting thing, and, like, I really need to write this up because again, this isn't secret, but again, we, we really should sort of advertise all the hard work we've done. The most interesting thing to me in computing that's changed since, you know, when I started work at Google in, like, 2009 or so, is back then we used hard drives which have a 10-millisecond seek time-

Matt Klein: Mm-hmm

David Crawshaw: for random access. And the beautiful thing everyone realized is you can take your hard drives and put them on the network And this is really wonderful from a cloud perspective because your computers used to have like three fundamental dimensions you had to choose a size of, which is the amount of CPU, the amount of RAM, and the amount of disk.

Matt Klein: Yeah.

David Crawshaw: And the great thing about network disks is now you reduce them to two [00:52:00] dimensions. Mm-hmm. And so the number of SKUs is greatly reduced. Uh, the product space is in much better control. And even within that space, like the instance types are crazy.

Matt Klein: Yeah.

David Crawshaw: So, you know, it, it's good to cut that out.

Yeah. Well, the problem is that was true for hard drives.

Matt Klein: Yeah.

David Crawshaw: And the r-reason it was true for hard drives is your round trip time on Ethernet is a millisecond, and that's fine adding that to the 10 millisecond of seek time on a hard drive. Then we swapped them all out for SSDs, which have a 10 microsecond seek time equivalent.

They don't actually seek, right? But, you know, I mean-

Matt Klein: Yeah

David Crawshaw: ... it depends on what the garbage collector does actually. But, uh, we, we take this 10 microsecond, uh, random read time, uh, and then we put a millisecond of Ethernet round trip time on it, and like we've killed our IOPS.

Matt Klein: Yeah.

David Crawshaw: And this is the worst thing about clouds for me, is like we should be working on local NVMe, and we're not.

And so our infrastructure runs VMs directly on local NVMe, and then in the background is constantly syncing the changed blocks of the, the virtual block device off that machine to another machine.

Matt Klein: I see.

David Crawshaw: And so that's how we achieve durability. And like, again, I haven't even written [00:53:00] up that our machines are durable.

And like, if you-- This is why we don't call ourselves a sandbox provider, because the sandbox people just run a VM on a machine that they just hacked up. And like-

Matt Klein: Yeah

David Crawshaw: ... if the machine fails, sorry about your sandbox. But like we've got a level of durability beyond that.

Matt Klein: Interesting.

David Crawshaw: And the durability guarantees are very different from EBS because of that lag.

You know, there's, uh, there's many seconds, uh, before we've, uh, replicated the d- the blocks. But it works for almost everything you need. And it's such a different environment because if you need 100,000 IOPS, you've got them, and you can just use it and it's fine.

Matt Klein: Yeah.

David Crawshaw: Uh, unlike your EBS volume on AWS, which has like 3,000 IOPS by default, and like 100,000 IOPS costs you way too much money to use.

Matt Klein: Yeah.

David Crawshaw: So-

Matt Klein: Um, I, I, I could honestly, uh, spend all day talking to you about this topic. It's super interesting, so maybe we'll do a follow-up episode at, at some point. Um, we're, we're pretty much at time. Um-

David Crawshaw: Yeah. I'm happy

to talk tech

whenever you like.

Matt Klein: Yeah. I mean, it's, it's s- super interesting, and I, I guess I can feel your pain, especially when you're doing a startup.

It's like at, at bitdrift, [00:54:00] for what we do, I feel like I've actually solved a tremendous amount of really difficult technical problems, and it's like you don't have time. You're just too busy to like sit and write it all down, which is unfortunate. But-

David Crawshaw: Yeah ... so, so, so I guess- But again, so much of business is talking to people, right?

Matt Klein: Yeah. So- Yes. I, I agree. Yeah.

David Crawshaw: Yeah. Anyway, uh- It's, it's al- it's al- it's also really important, just to say one more thing on this, is, uh-

Matt Klein: Yeah

David Crawshaw: ... most startups are not doing really interesting technical work, and like it's actually correct for them not to because they're trying to go and solve people's product problems.

Matt Klein: Yeah.

David Crawshaw: And most product problems, you know, that people have don't need to be solved with complex, new, interesting technology.

Matt Klein: Yeah.

David Crawshaw: But if you're in the wonderful position of doing interesting technology as a startup, you gotta talk about it because-

Matt Klein: I agree

David Crawshaw: ... it's so unusual and fun.

Matt Klein: Yeah.

David Crawshaw: So I'd love to read more about bitdrift.

Matt Klein: Cool.

Yeah. Yeah. So- Um, any, any last parting words for our listeners that you would like to share?

David Crawshaw: Oh, I mean, you know, please play with exe I'd love to-

Matt Klein: Okay

David Crawshaw: ... I love feedback. You know, we have a lot of very happy users. Uh, we're, we're trying to build it out into a production cloud that you run your services on.

Uh, you know, we, we think [00:55:00] it's, uh, we think it's really fun and, uh, it's worth using. Uh, on computing generally, you know, if you, if you like computers, find a way to work with them in a way you actually like them. I know a lot of programmers who like computers and then spend their day doing things they don't like.

Matt Klein: Yeah.

David Crawshaw: So, you know, try enjoying it more, so.

Matt Klein: Those are fantastic parting words. Thank you so much, David. This was a fantastic episode.

David Crawshaw: Thanks,

Matt. Appreciate it.

Matt Klein: Um, that's a wrap for this episode of Beyond the Noise: Signals, Stories, and Spicy Takes. Huge thanks to David for joining and sharing his story. You can find this episode and all past ones on the bitdrift YouTube channel.

If you had fun, drop us a review, tell your friends, or yell your favorite hot take into the void and just make sure to tag us. I'm Matt Klein, and we'll see you next time. Thanks a lot.