Mike Shi: Your Agent Is Now Observability’s Most Important User

Episode #27 - Mike Shi

===

[00:00:00]

Matt Klein: All right, folks. Welcome to another episode of Beyond the Noise: Signals, Stories, and Spicy Takes, the show where we dig into the stories of the people shaping the future of app-based computing. I'm your host, Matt Klein, co-founder and CTO at bitdrift, as well as the founder of Envoy Proxy. Each episode, we'll talk with engineers, founders, and technical leaders who transform the way their companies build and understand what's happening inside their systems.

We'll dig into the challenges, the breakthroughs, the lessons learned, and we'll wrap it all up with their hottest takes. So let's dive in. Today, I am thrilled to have with us Mike Shey, who leads observability at ClickHouse, where he works on ClickStack. He joined through the acquisition of HyperDX, which he co-founded after years building developer tools and becoming increasingly obsessed with databases.

Welcome, Mike. Thank you for joining [00:01:00] us.

Mike Shi: Yeah. Thank you, Matt. Thanks for having me.

Matt Klein: Yeah. Awesome. I am excited for this conversation as I have, uh, written about a lot before. I'm a big fan of ClickHouse. We also use HyperDX internally at bitdrift, so, you know, definitely very familiar with what you all are doing.

So excited to talk about it, excited to learn more. Um, where we typically get started is, uh, to learn a little about you. So could you just, you know, give us a bit about your background, tell us a bit about how you got into computers, and, uh, tell us about your journey?

Mike Shi: Yeah, absolutely. I think I got into computers a few different ways.

Uh, I think starting off with robotics as a kid, I was lucky enough to have, you know, Lego robotics kids, uh, kits, uh, in my, uh, in my school. And then there were... I, I started with kind of the drag and drop programming, so that was actually kind of an interesting era. Uh, and then eventually you know, robotics became a bit of an expensive hobby for my parents to want me to keep going down.

Um, so software's a bit [00:02:00] easier for me to... You know, it was just easier to have a la- everyone has a laptop at that point. You know, we had a laptop, um, uh, that I shared with my parents growing up. So was able to kind of, um, you know, build software and then got into gaming and software and gaming became really natural.

You want to build like tools for games, you want to build games for yourself. So I got into .NET, C#, um, XNA, um, uh, Visual Basic .NET actually starting off with, so it was not quite C language, and then learned that C, you know, C-based language is the way to go. Went there. Um, eventually, uh, wanted to share my programs with my friends, so emailing executables was not really a thing you could do.

I don't know if you could still do that. No one does that anymore. But you know, back then it's like, "Oh, here's like a cool, you know, EXE I built." Uh, you want to email a friend, the, the, you know, the email doesn't really like that. So distributing was difficult. Uh,

Matt Klein: yeah, I mean, I was gonna say that, that, that sounds very sketchy to be, to be-

emailing around executables. But yeah. Anyway, go ahead.

Mike Shi: Yeah, I mean, yeah, you know, you're, you're a kid, right? So you didn't really think that much about it.

Matt Klein: Yeah.

Mike Shi: It's always like, "Oh, this is my program." But yeah, no, I understand why they were, they were [00:03:00] blocked. Um, so then, um, learned, you know, I, I went on the internet and Google like how, how do you, you know, how do you build like web apps, right?

'Cause that seemed a lot easier. Um, and then got into the, the LAMP stack, um, or, or I was also using Windows at the time, so also the WAMP stack locally. Um, built a few apps that my friends really liked and then got hooked into building products and I thought, "Oh, cool, I wanna be a software engineer." Um, went to college, uh, tried doing software engineering for a bit, and then I realized, well, there's actually a lot more I wanna do besides just software engineering.

Uh, and I think that's kinda where I found a... You know, I wanted to be a bit more of a f- like someone that does a lot more. So I tried to be a PM as well. Uh, and then I realized like actually being a founder felt a little bit more interesting, building products from zero to one, um, and also solving problems that I'm real passionate about from a variety of angles, not just the engineering part.

I mean, the engineering part's still really, really interesting to me and I still spend time, um, coding, but, um, uh, o- over time it's kind of expanded, uh, a bit beyond that. Uh, and you know, I can, [00:04:00] I can keep talking about that, uh-

Matt Klein: Yeah

Mike Shi: ... all day, I guess.

Matt Klein: I,

I, I mean, did you go directly into being a founder or had you worked for anyone before being a founder?

Mike Shi: That's a great question. So I did want to... I, I think I've tried to start companies all throughout my life to some extent. Um, I remember as soon as I turned 18, I could legally sign papers, I wanted to turn one of my side projects into a company- ... 'cause I thought, oh, this is gonna be, uh, this is like, you know, I was building things for my friends, I was in school.

I was like, "Oh, this is gonna be, like, the next, like, ed tech startup." Um, uh, and, and, you know, I wanna go down that path. Um, so I, I think to some extent I've, I've had that version with me for, uh, for a while. I... And, and I suspect it's 'cause, you know, my parents, if I had a bit of an influence and, you know, uh, kind of i- injecting me with some entrepreneurial spirit as, as I was a kid-

Matt Klein: Yeah

Mike Shi: so that kind of followed me throughout my childhood and adult life as well.

Matt Klein: Yeah, I, I mean, you know, there, there are so many stories out there of, like, [00:05:00] the mythical founder who starts their company, you know, in college or out of high school or something along those lines. And I, I, I'm just personally curious because I don't know what the breakdown is o- of people that have successful exits, you know, who that is their first experience, like, versus having had some experience working for someone else.

And I just wanted to get your thoughts on it because I think it's interesting, is that, you know, a- a- that was my question a- as to had you worked for anyone else-

Mike Shi: Mm-hmm

Matt Klein: ... prior to doing this? And then, I don't know, I mean, like, if, if you were giving advice to people, do you think that it's worth it to, to have experience working in a different environment?

Or do you think that that experience can actually m- m- make you, I don't know, like, not see things differently enough?

Mike Shi: Oh, man, this is such a tricky question. I've obviously spent my time reflecting 'cause, you know, before I started, you know, HyperDX, you know, I, I worked at a company as well beforehand

[00:06:00] so I, I would say that, um, I've definitely looked back at it every time and asked myself, "Well, was, you know, was the last thing I did really the right decision?" Not to, like, kind of regret on it, but more of like to figure out, "Hey, like, what should I be doing next? Should I be gaining more experience in, like, a more structured environment via employment?

Or is it hey, like, go, go and try entrepreneurship?" Um, I, I give a lot of credit to those who, you know, straight out of high school or college found a company really successful. I think that's extremely challenging. You know, when I've gone through that, I felt like I did not know enough things about what I should be doing and what's expected and, you know, what the norm is.

Matt Klein: Yeah.

Mike Shi: Uh, that made it really challenging to be successful. I mean, um, I, I'm super eager and action-oriented, so that's kinda where I like jumping head in, really excited to do it anyways, regardless of the kind of lack of experience of doing so. So naivety is definitely the bigger driving force to it. I think looking back at it, I definitely benefited from being able to, um, work at other, um, uh, companies and startups as well to be able to really absorb kind of the knowledge from [00:07:00] there and understanding a little bit more about kind of almost like how the world works and kind of what's expected-

Matt Klein: Yeah

Mike Shi: and the norms and connections that you build. So I I, I think it's probably better, you know, net-net, I think it's better to have a bit more experience going in before being a founder. But, you know, obviously some people pull it off without that and, you know, huge kudos to them 'cause that's, that's seriously impressive.

Matt Klein: Yeah, I, I mean, for me, it's, it's hard to, it's hard to imagine s- starting a company without f- you know, for me, without having that prior experience. Um, but I can see an argument where, you know, pe- people that wanna really buck against some trend, um, if they don't have any preconceived notions, like maybe, I don't know, maybe that naivete is actually useful, I guess.

Um, a- and that's what I find so interesting about this topic.

Mike Shi: Yeah, I, I think that part is interesting. I mean, I think there's kind of these common skills that are gonna be useful, such as like, you know, how, how do people expect to be managed in a [00:08:00] company, right? 'Cause eventually you're gonna hire people and bring them in and you're gonna create this culture-

Matt Klein: Sure.

Mike Shi: Yeah ... of work, um, that can be very difficult if you've never worked at a company for an extended period of time and knowing kinda, hey, how do you wanna model it after? Um, so I think there... Y- I think you'd have to have... I, I'd agree. I think it has to be some like extreme case where, you know, that fresh perspective and, you know, maybe that, that, um, what's it?

Uh, uh, kind of naiv- Like I, I see naivete. I, I... Hopefully this, this is, this comes off okay 'cause, you know, I feel it the same way. It's like, hey, like you're gonna be naive about some certain aspects, but that's actually a superpower, right? As a founder where you don't know- I agree ... all the terrible things you're gonna get into.

For sure. 'Cause that's the reason why most people don't try to do the thing you do, right? 'Cause, you know, they might have a bit more experience of like, "Hey, that's actually a really, really difficult path," but maybe once you get past that you'll get something much better. So I think there's definitely chance or, you know, places where that happen and, you know, that becomes your competitive advantage is that you just don't know how terrible it's gonna be, but you're willing to work through it one step at a time.

Matt Klein: Yeah, I mean, how, how, [00:09:00] uh, terrible it's gonna be. I mean, I think- ... I think that people definitely under- underestimate, you know, the mental anguish that goes into starting a company, but that's a whole, you know, that's a whole separate conversation for sure. Yeah. Um, but that's actually a good segue. I mean, tell us a bit about how you founded HyperDX, how you became interested in the observability space.

Mike Shi: Yeah. Um, I, I think, uh, I, I've always, you know, really cared about products that were reliable. You know, when you're using an application, I think everyone's frustrated. I think me intensely, maybe intensely a bit more frustrated than most. When something goes down, um, it, it's always not enjoyable, especially something that you really rely on, you know, um, uh, something that you're about to use, something that's really important to you, and it goes down.

That's always really, really frustrating. Availability is kind of a big problem. Um, so I've kind of, you know, ear- early on in my career I thought a, a bit about, "Hey, we should have, like, logging. We should have monitoring of services that we're building." Um, and then that led me to, uh, working at a log management startup.

It was called, uh, [00:10:00] LogDNA, now it's called Mezmo. They were a YC Winter of '15 company. Um, uh, I joined, uh, around, like, there were about, I think, like 80-ish people when I joined. So it was, it was kind of humming along. I've already kind of gone through kind of trying to build a, a company before that, whatnot.

Um, they're humming along, uh, and that's kind of where I got really into, like, the vendor, observability vendor business, uh, or I guess log management vendor business back then. Um, and that's kind of where, uh, I spent a lot more time. But throughout the time, I felt like the problem wasn't really being solved.

I mean, we were building products for really great brands. You know, we were being used by BMcloud, Reddit, Limebike. Uh, a ton of different startups or, uh, companies were definitely using, uh, what we were building, but it still felt like they had a lot of, unmet needs, right?

Uh, there was kind of a lot of challenge around correlating telemetry, figuring out root causes of issues. Like, log management clearly on its own wasn't enough. Um, spent a few years there, and at the end of it, I felt like, hey, like, there has to be something a bit better for reliability. And that's kind of where, uh, I, I set off to kind of explore, hey, what is reliability?

What should reliability really look [00:11:00] like? Um, and, and, you know, that's kind of how I started going to HyperDX, where actually the first version of it was, um, synthetic monitoring. Uh, I was taking it from the user perspective. Hey, since I l- hate downtime so much, one part of monitoring should be, hey, is there kind of a gap in the synthetic monitoring market?

Uh, that was a bit tricky to really execute very well. I think it was a bit difficult to get people to buy into, um, synthetic monitoring. Uh, maybe that's gotten a lot better, uh, since then. But I think at the time it was kind of difficult to convince people, "Hey, you should be spending money on synthetic monitoring, have people maintain these synthetic monitors."

It gets a bit messy in enterprises 'cause it's kind of hard to... You know, it's a blur-

Matt Klein: Yeah

Mike Shi: ... between developers and, and SREs.

Matt Klein: Sorry, for those that, that don't know what synthetic monitoring is, do, do you mean products like Thousand Eyes or, or, or, or like h- like how do you mean synthetic monitoring exactly?

Mike Shi: Yeah. Yeah, so there, I mean, there's, there's a few different ways to do it. I mean, I think, um, uh, Checkly would be really kind of the, the prime example to me of like a great synthetic monitoring product, um, these days, [00:12:00] um, where you're able to basically create some sort of effectively like a cron job that runs every once in a while-

Matt Klein: Yep

Mike Shi: and tests out your product-

Matt Klein: Yeah

Mike Shi: ... whether that's an API, you know, endpoint call, so you just curl something, see if it, it hits a 200 back to you, or if it's actually more involved as like a browser synthetic. So it actually spins up a virtual browser. You know-

Matt Klein: Yep

Mike Shi: ... you tell it, "Hey, click on, you know, checkout flow, go through it, see if you actually can get to the end of the checkout flow."

Um, so, so those are kind of the ways that I think about synthetic monitoring. Um, uh, and, and the product, I think, you know, they have a great product in that space as well.

Matt Klein: And then when you were thinking about entering that market, like what, what was the aspect of the existing products at, at the time that you thought were lacking or that you thought that, that you would be able to add to?

Mike Shi: Yeah, I mean, I thought the, the difficult part was kind of getting people to create the synthetic monitors. Uh, and a synthetic monitor is a bit more than just kind of the curl 200 and g- get an API, you know. It kind of validated an API endpoint, 'cause to me, that wasn't really, [00:13:00] um, the reliability bar you're going for.

Kind of, you know, we talked a lot about SLAs, SLOs these days, right? It's really about the end user experience. Um-

Matt Klein: Yeah

Mike Shi: ... and I think a lot of folks kind of focus on, um, monitoring and observability from the sort of service level perspective. So like, "Hey, like-

Matt Klein: Mm-hmm

Mike Shi: ... you know, my service is up, therefore I don't really have a problem."

But as a business, hey, if everyone's service is up but the user still can't check out, clearly something has gone wrong here, uh, and some definition has failed. So that's kinda where, you know, starting off from the synthetic endpoint seemed really interesting to me, or from that perspective seemed really interesting to me.

And then trying to solve it from trying to make it, you know, my, my hunch was that it was very difficult to get people to actually create a synthetic. So, hey, can we make it easier for people to just, you know, click around on the page, uh, be able to create a Playwright script from there, and then immediately launch that into their synthetic monitoring?

So, you know, from like you get your PM or, or whoever in your organization to do five clicks-

Matt Klein: Right

Mike Shi: ... set it up, and then you have great, you know, um, availability monitoring for your service end to end, and that was kind of the, the hope that we wanted to build. [00:14:00]

Matt Klein: Yeah, I, I mean, it's, it, it's almost the eternal observability question, which is that are you, are you going after how I think of, like, the n- known knowns or the unknown unknowns, if that makes sense, right?

You know-

Mike Shi: Yeah

Matt Klein: ... because I think that there's certainly pros in terms of making it easier for people, whether you're in web or whether you're on mobile, you know, to, to do these flows and make sure that these golden flows are working correctly. And I think we see organizations that, that wanna have a set of golden flows, and I think that makes sense.

It's more just, you know, I mean, as we know, so many things can go wrong, and then h- how do you debug and understand what's going on, you know-

Mike Shi: Mm-hmm

Matt Klein: ... when the thing that's gone wrong is not one of these synthetic flows? So I think my follow-up question for you, and maybe this is why you didn't end up doing a synthetic, uh, observability company, is, I don't know, like, how do you balance that- [00:15:00] synthetic testing versus the real-world reality of all the things that can break all the time.

You know, because listening to you talk, I mean, you're, you're obviously preaching to the choir is the reason that, that we started bitdrift is that so many times in my career, you know, you're sitting there looking at your dashboards and your backend service is like 99.99% success rate and like yet your app is totally down and no one knows, and it's just horrifying, right?

So I don't know. I, I find this topic just really, really interesting, um, in terms of how you, how you balance these different things, you know, in terms of, again, like helping people debug the reality of the situation or helping them set up some flows and know that those flows are working. So anyway, would love to get your thoughts on that.

Mike Shi: Yeah, I mean, I think the, the unknown unknown, you're right. That's kind of the thing which really led us down this production, um, debugging path and, and kind of, um, really thinking about... and, and we [00:16:00] have a lot of takes on, you know, I know you had Charity Majors on recently and, you know, highly, um, highly aligned with her viewpoints as well and I think going e-even a bit more of extreme of like, hey, like, you know, how can we live in a world where we don't have to sample a lot of this data either, right?

Matt Klein: Yeah.

Mike Shi: 'Cause a lot of these unknown unknowns, um, can pop up in, uh, really kind of support cases, right? A lot of times, you know, developers spend a good amount of time debugging like, hey, like this user reported this issue and, you know, this specific flow that they were working on, what actually went wrong, and actually getting down to the root cause can be extremely tricky, especially, um, if, if you're working in an environment where it happens very few times relative-

Matt Klein: Yep

Mike Shi: to, you know, other transactions.

Matt Klein: Sure.

Mike Shi: Um, uh, and, and you know, if you have a, a high sampling rate, that becomes a little bit difficult to get down. So I think that's kind of ... It, it definitely pushed us down that path of, hey, how do we build a platform that, um, you know, not only, you're right, solves kind of known, um, uh, known, known problems, uh, or at least, you know, detect known sort of workflows, but also kind of cover that, that tail, and that's kind of really where we felt most tools in, you know, entering [00:17:00] the observability market really fell short is that they all had some sort of limitation around the, the type of context or the correlation of the context that you could have that really hampered your ability to actually get in and find the root cause of what actually went wrong.

Matt Klein: Yeah. So, so when you were looking at it back then, I mean, it sounds like you technically wanted to find a solution that would be cost-effective to not sample. I mean, is that, is that something that you were thinking about? Like, where I'm going with this is obviously you, you, you know, thought that it would be a good idea to build on top of ClickHouse, and I would love to learn that part of the story.

Mm-hmm. Um, and I'm assuming that you picked ClickHouse because it's open source, it scales very well, you wanna do an open source product. But would, would love to understand how the selection of building on top of ClickHouse, how, how did that follow from what you were just saying?

Mike Shi: Yeah, I mean, um, uh, [00:18:00] y- you're right in terms of, yeah, like it definitely...

We, we had a lot of problems where we wanted to collect more and more telemetry around our existing systems that we're building, um, and it was getting, uh, really difficult to do so. And we also had like a million different tools to do it. Like we had a, a bunch of different, you know... We were buying a bunch of different products to try to instrument our application at the time.

Um, a- and, you know, we were doing the classic, like pick the timestamp from one tool, go to the next tab, put in the timestamp there.

Matt Klein: Yep, sure.

Mike Shi: Oh, no, it was UTC Pacific Time, et cetera, et cetera. Um, so it was kind of, kind of a mess going through that. Uh, so we knew that, you know, there was something that we wanted to do better.

Um, and we also knew it needed to be a platform that scaled. When it come to choosing ClickHouse, you know, at, at the last, uh, log management company, um, I was working at, we were built on top of, an Elastic-based stack. Um, and I think Elastic was kind of the defa- you know, this company started-

Matt Klein: Absolutely.

Yep

Mike Shi: ... uh, 2015. Super common de facto stack. In fact, a lot of people are still choosing it today.

Matt Klein: Yep.

Mike Shi: Um, but it was pretty clear from how the scale that we're running it at is [00:19:00] that, um, we were spending a lot of time, uh, just from a c- building perspective, 'cause, you know, I knew that if we were to build a zero platform, we wanna build it in a way that was quite comprehensive, quite complete and, and, you know, uh, the engineering velocity of our ability to actually build the product was quite, uh, an important part of deciding what technology we wanna build on.

As with anything else, right? You wanna build platform... You wanna choose platforms that allow you to build faster-

Matt Klein: Yeah

Mike Shi: ... on top of them. Um, and it was clear kinda building on top of Elastic, a lot of the work that we spent, um, uh, working there was, uh, trying to build, uh, around the database. Uh, 'cause like Elastic, you know, its properties are, you know, uh, fast reads, but, um, writes co- uh, often writes are, are quite expensive.

Um-

Matt Klein: Yeah

Mike Shi: ... and then aggregations are expensive. There's a few kinda different downsides to the technology that are kinda well-known at this point, right? Um, so we spent a lot of time kinda doing pre-aggregation and caching, and a lot of other things that we kinda built around the, the database itself that, uh, really slowed down our ability to build awesome products for the user.

Um, a- and then even scaling clusters to the scales that, you know, the data that we pump in was also a bit challenging, [00:20:00] um, a- as well at the time. So that's kinda where when, you know, uh, we were looking at our produce and say, "Hey, how do we actually build out this product? What platform we wanna build on?" We looked around.

It, it was kind of immediately obvious that, you know, ClickHouse was really the kind of up-and-coming data store that would help solve our challenges. As you mentioned, yeah, it's open source. Um, it's columnar. It, it's really great at aggregations. We understood that, you know, most of the time what people are gonna be looking at is, yeah, they wanna emit high cardinality wide events But-

Matt Klein: Yeah

Mike Shi: they're gonna want to do aggregations on top of them, right? It, it doesn't... It's like, great, I have a bunch of traces and they have all this rich attribution, but, hey, how do I get, like, the, the P95 grouped by this specific, uh, the domain attribute that might be super, super high cardinality? Um, so that's the kind of experience that we wanted to make sure we had.

And, you know, when you looked around at the different products and platforms that were mature for the use case, really, ClickHouse, um, was the only platform. Even today, like, I mean, I'm a bit more biased now I'm on the other side of the table, but, you know, uh- ... having been, been acquired in, but I still [00:21:00] think it's, uh, there's, there's really not many options, um, to, to, to be picking from, from the market still.

Matt Klein: Yeah. I, I, I, I mean, having, having not been acquired by ClickHouse, like we- ... we use ClickHouse heavily at, at bitdrift. And-

Mike Shi: Right

Matt Klein: ... I, I can say that I, I also don't think that there is, at least in the realm of that type of database that's also open source and has a cloud product-

Mike Shi: Right

Matt Klein: ... and all of those things, I can't think of any other solution that is like it currently, and I think it's an amazing piece of technology.

I think, um, so it's like s- sitting here right now, given how much ClickHouse has grown and all of those things, it seems today to be extremely obvious, right? I mean, well done, right? I mean, it seems extremely obvious to build an observability platform on top and, you know, given the natural growth of ClickHouse, that you're obviously rising, rising tide lifts all boats and all of those things.

Mike Shi: Mm-hmm.

Matt Klein: But, um, at the time, obviously, you don't know that. I mean, it's like you are- Right ... [00:22:00] starting a company, you're trying to figure it out. And what, what I'd love to learn about more is that you, you bet on ClickHouse, which seems like a great bet, but then-

Mike Shi: Mm-hmm

Matt Klein: ... you also decided to open source HyperDX itself, right?

Mike Shi: Yes.

Matt Klein: Um, and, and t-t-t-tell us a bit about... Because you could have gone down, obviously, the bitdrift path, which is that we're not open source on the backend, but we use ClickHouse-

Mike Shi: Yeah

Matt Klein: ... internally. That, that part is well known. Tell us a bit about how you thought at the time about whether your solution would be open source.

Because I find it so interesting around, you know, when you're developing this type of software, um, y-you know, I mean, and you make it open source, obviously there's, there, there's always this pro and con. I mean, it's like-

Mike Shi: Yeah

Matt Klein: ... open source is a, is a great lead gen, but then you have a lot of people that are not paying for it.

So I, I don't know. I just, like, would, would love to learn a little more [00:23:00] when you were having these discussions back in the day of how you thought about that, that part of it.

Mike Shi: Yeah. Um, you know, me and my co-founder, Warren, we spent a lot of time talking about this. Um, and I think the natural inclination, 'cause we're both, you know, engineers and technical people and, and, and wrote a bunch of code as well, you know, we, we both...

Our both inclination would be, hey, we wanna build something that's open source. Um, you know, personally use as much open source as I can. Uh, love, love the kind of community, and obviously building an open source company, I think, is just one of the cool things that, that, you know, you can do as an engineer, right?

And I think that's also... that's really exciting. On the other hand, I was kind of weighing the, well, you know, hey, we, we, we have investors, obviously we have a business. We can't just do a thing that we think is really neat. We have to, like, make something that's-

Matt Klein: Yep

Mike Shi: ... as you mentioned, kind of commercially viable as well.

Um, but when we thought more about it and looking at kind of, as you mentioned, sort of, kind of the other companies that went open source, what made us feel really comfortable is that there's definitely, you know, there's definitely risks in open source and, you know, kind of, hey, can you have your kind of lunch eaten, [00:24:00] uh, you know, through, through someone else kind of coming in and taking your product and then, and then, um, uh, hosting it or forking or doing whatnot with it.

Uh, but I think for us, the, the math, at least kind of our belief is that, hey, if we can build the best product and, you know, we can keep having a really high product velocity to the point where it doesn't make sense to really invest into any other, you know, provider of the product besides the people who are building it, and I think ClickHouse is very much kind of on that front as well, um, then it's, it's, it's totally fine to go open source 'cause, like, your, your competitive moat isn't, hey, all the things that you've built so far.

I mean, that's obviously very important, uh, what the users value, but also what's coming ahead, right? Uh, and we felt like, hey, we could build a really great, you know, product and foundation and have a really high velocity to the point where, you know, sure, someone else can come around and host a product, but do you really wanna use the version of it that's gonna be, you know, like, uh-

Matt Klein: Yeah

Mike Shi: a few versions behind? You know, they're struggling to merge their patches in 'cause of the conflicts that they have to maintain versus the product that's gonna be kind of moving along and building a lot faster. And, [00:25:00] and on top of that, I think there's just kind of, uh, as the thing I mentioned before, just kind of that personal belief, that personal want to build something open source.

You know, when we picked the license, we picked the most permissive one that's most, you know, MIT, kind of one of the most permissive, uh, licenses available, even, uh, technically more so than Apache 2 to some extent. Um, so we really wanted to go all the way and kind of embrace that sort of culture, and we felt kind of good that, hey, if we play our cards right, we can build an amazing product open source and still be commercially successful.

And of course, you know, we weren't the first ones to do it. You have people like, you know, CID and GitLab. There's, there's kind of a, a lot of great history of companies that have done it well as well.

Matt Klein: Yeah.

Mike Shi: Uh, so we felt assured that, hey, we weren't gonna be, you know, we're gonna be the only ones kind of believing this lunatic vision.

There's other things that we're gonna be lunatic about, but this one is like, hey, we're, we're kind of following footsteps and feel that we can be successful with it.

Matt Klein: Yeah. Now I'm gonna ask you a really tough question.

Mike Shi: Oh, yes.

Matt Klein: Which is that do you, do you think that Today the calculation-

Mike Shi: Mm-hmm

Matt Klein: ... has changed.

Like, would you, would you make the [00:26:00] same decision today? I mean, o- obviously things have changed. You wouldn't build the same company today.

Mike Shi: Right.

Matt Klein: But I find this topic so fascinating because obviously the agentic engineering tools have, have really changed the equation, in my opinion.

Mike Shi: Mm-hmm.

Matt Klein: And, and, and look, to be clear, I also love open source.

Absolutely love it. Um- Yeah ... working on, working on Envoy is pr- it, it probably will always be the highlight of my career, I mean, to, to, to contribute to that type of open source environment.

Mike Shi: Right.

Matt Klein: Um, but when I look at today at what can be created with these tools, es- especially for highly competent engineers, um, I, I think open source is changing, and I don't think we even know what that looks like anymore, you know?

So I'm just- ... curious. I mean, uh, obviously this is a segue into AI, which I'm sure we were gonna wind up-

Mike Shi: Right

Matt Klein: ... with anyway.

Mike Shi: Yes.

Matt Klein: But would, w- w- would l- [00:27:00] love to learn a little more about how you think about this topic now, right? Versus how-

Mike Shi: Yeah

Matt Klein: ... you would've thought about it a couple years ago.

Mike Shi: Yeah. I mean, I, I think...

I'm not sure if agents You, you know, obviously agents have changed the landscape a ton, right? And I think, you know, there, there's some, you know, some points of, hey, like, you know, um, you know, agents with your open source code are gonna be able to kind of replicate your product a lot easier than before, where users can point, you know, their agent at your open source code and then build kind of, uh, other versions of it as well.

I, I don't know if that really changes materially the kind of equation and trade-offs that we were thinking before, where it's kind of, hey, the users will wanna pick, um, one, the, the best product, right? Um, a-and two, there're probably just gonna be a huge, uh, uh, a cohort of people who wanna run them themselves, and we'd love to empower them to do that, right?

'Cause, hey, you're running it today, and at some point there might be... We know touched upon this earlier, but you know, the open source model, a lot of times like hey, you're running it today, [00:28:00] and at some point you might not wanna run it anymore, right? Like, petabyte scale databases are not really super-

Matt Klein: Absolutely.

Yeah

Mike Shi: ... not like a walk in the park to run anymore, uh, at, at a certain point. So there's kind of a, a point where, um, you know, some companies, hey, they might wanna invest into people and, and keep building it in-house and, and maintain in-house. That's, that's awesome. Obviously we wanna support those people and, you know, have them help, you know, uh, spread the word and, and make them super successful as well.

Um, but then there's gonna be a group of people who, you know, they're like, "Hey, our business isn't gonna be in the observability management business or the observability forwarding business or the more, you know, database scaling business." And those are, you know, they're gonna successfully move into kind of a commercial model where it could take more of the hands, uh, you know, hands-off approach, uh, from, from their perspective.

But with, with agents kinda coming in, I don't think it, it, it majorly changes sort of that dynamic even when you ha- you know, you have, you know, things are cheaper than ever, tokens are cheaper than ever. A, a lot of these, you know, at least from my perspective so far, a lot of these agents aren't fully autonomous, weren't fully out of a job.

They're able to, you know, replicate engineers. So there's still-

Matt Klein: Absolutely. Sure

Mike Shi: ... an effort, uh-

Matt Klein: Yeah

Mike Shi: ... [00:29:00] the organization has to put either, you know, depending on what angle you're looking at, whether it's kind of like, hey, taking your software, running it, there's still gonna be effort in maintaining it. Agents, you know, if you apply them to your infrastructure, uh, without any guards it's probably not the best idea to kind of maintain-

Matt Klein: Yep

Mike Shi: open source on your behalf. Um, yeah, all sorts of kind of, you know, challenges there, uh, kind of applying-

Matt Klein: Yeah

Mike Shi: ... agents to, to the space.

Matt Klein: I, I mean, to be completely clear, I am, I am not a like death of SaaS person. I mean, it's like-

Mike Shi: Mm-hmm

Matt Klein: ... I absolutely think that people are gonna pay other people to have-

Mike Shi: Right

Matt Klein: services that are absolutely awesome. I was mostly asking about it from the open source perspective, and I, I just in the past couple of weeks, I open sourced a SQL Kafka replacement thing that I built, right?

Mike Shi: Mm-hmm.

Matt Klein: Uh, and I mean, that, that never would... Like, I never would have invested the time in doing that before the current crop of tools.

Mike Shi: Right.

Matt Klein: And I, I was mostly just asking about it from the perspective of when you start an open source [00:30:00] company today, right? Like, I'm not talking about SaaS.

Mike Shi: Yeah.

Matt Klein: That part is super clear. I'm talking purely about, you know, what it, what it means to kinda use open source as lead gen effectively to-

Mike Shi: Mm-hmm

Matt Klein: eventually get you into the enterprise product and all of those things. And I, I'm mostly just asking the thought exercise of it seems like maybe the calculation has changed slightly, where someone that previously who might have said, "Okay, like, I'm gonna go pay for the enterprise, uh, open core solution that has-

Mike Shi: Yeah

Matt Klein: this authentication feature or something," now I'm just gonna tell the agent to go build me this enterprise feature, and I'm, like, not gonna talk to you. So I, I, you know, again, like, I completely buy that people are always gonna wanna go to the creators and are gonna wanna pay for it. I, I mostly just personally think that it's not clear yet how these calculations are changing right now.

That's all I'm saying.

Mike Shi: [00:31:00] Yeah. No, I, I, I agree, and I think, you know, um, when it comes to these kind of, I, I guess, like, the open core concept, right? And is that under kind of a threat by agents being able to-

Matt Klein: Yeah

Mike Shi: ... expand the core to the footprint that the user wants? Um I think there's still kind of the, uh, you know, with open source projects, really the project that's kind of voted as in like the consensus of community that wants to go behind a given product because we've seen forks happen before, right?

And forks where the fork, the, the, the fork off of the original project, uh-

Matt Klein: Yep

Mike Shi: ... has become, you know, more successful or equally successful to the original project. So really it's kind of a vote of a community of where it should go. And these are still pe- so far still people voting on which community that they want to believe in.

So I think there's one extent where, you know, companies can have their private forks and try to build around the core, but then they'll still be taking the maintenance burden of, hey, new shiny feature comes out. Oh, I have to then retrofit all my agent tech patches on top of that.

Matt Klein: Yep.

Mike Shi: I'm sure that's possible to some extent, but I think, you know, if there's kind of a good division between your core and [00:32:00] your, your, you know, your value-added features on top of that, and that's kind of a meaningful set to people in a way that's not like just super like thin layer on top of it, which, you know, always was gonna be a problem commercially anyways, um, then I think people are definitely safe.

I think there's gonna be a point where agents get powerful enough where they become a bit... that becomes a bit more tricky to think about. But if anything, I think we're not, we're not quite at that, you know, stage yet. I think companies are still doing the same trade-off they did before. Like, "Hey, do we really wanna fork it?

Do we really wanna maintain this internal version that patches the enterprise?"

Matt Klein: Yeah.

Mike Shi: Um, it is easier, and I think there's definitely companies are doing it, don't get me wrong. Uh, but I don't think it's, you know, definitely changed the equation to a large extent yet for companies who are debating whether they should be going open source or, or staying proprietary.

Matt Klein: No, for sure. I mean, a- and if anything, I think ClickHouse is a poster child of a company that has done this very, very well. I mean, they, they have a vibrant open source community, and o- obviously you are building a, a great cloud product, um, that people, including us, are paying for. So I mean, that, [00:33:00] that strategy is totally working, so that part makes absolute sense to me.

Um- I think, I think it's a good segue probably to start talking a bit about the, the current world in terms-

Mike Shi: Mm-hmm

Matt Klein: ... of, you know, the modern AI tooling, and I would love to pick your brain on how observability has changed, honestly, since-- I mean, things are moving so fast, right? So it's like-

Mike Shi: Yeah

Matt Klein: ... you, you, you, you started HyperDX not that many years ago-

in the grand scheme of things. Um, compared to when you started that company, when you chose that target in terms of the problems you were going after-

Mike Shi: Yeah

Matt Klein: ... and then you look at today, I mean, n-n-you know, unless you're living under a rock, like, no one's gonna claim that the world has not fundamentally changed in the last few years.

And would, would love to learn a bit about how you think about observability in this new world and the follow-up, and I'm sure this will be a long conversation, [00:34:00] but do you feel that your charter within ClickHouse has changed, like, in the last-

Mike Shi: Mm-hmm

Matt Klein: ... few years because of these new tools? So would love to learn about that.

Mike Shi: Yeah. Um, I, I mean, definitely things have changed. I think we've gone from, you know, uh, the era of, um, search being the dominant thing that people were looking for, right? Like, a lot of the, uh, uh, tools in, in the, in the 2000s until maybe 2010s, um, were kind of, "Hey, like, I wanna go find something, you know, I wanna search for something that's gone wrong," right?

And then we moved on to kind of analytics, and we wanna be able to do trends of things that are going wrong, 'cause we have more servers-

Matt Klein: Yeah

Mike Shi: ... and more systems and complexity, and therefore we wanna do, you know, data, you know, aggregations were a lot more important than single needle in a haystack sort of problems.

And I think now we're getting to the problem-- you-- We're getting to the statement where, hey, observability vendors are expected to know kind of why something has gone wrong. Um, that's kinda needed to be presented on a plate to our users a lot easier than ever before. Um, and I think that will continue going, [00:35:00] right?

Like, you know, why is only a part of it. We know a lot of it's gonna go into, like, hey, like, actually just fix the thing that has gone wrong, right? Um, in one way or another, within limited bounds, on infrastructure, or even, you know, passing it off to a coding agent or being the coding agent and submitting a PR into fixing what has gone wrong.

Matt Klein: Yeah.

Mike Shi: Um, so I think the, the, the kind of- Um, the, the ask that's been given of, you know, observability products have been continually shifting and we're doing, definitely moving towards the right, um, which is obviously, uh, great for, for our end users. Um, i-in terms of kind of, uh, how, how that's kinda shifted, um, my time at ClickHouse, yeah, I think coming in, uh, we were thinking, uh, we were still pretty thinking about like, you know, AI agents weren't really that big at the time.

Like it wasn't obvious that, you know, agents... how much agents were able to automate versus how much they were gonna be an assistant, um, and what pace that was gonna happen. Obviously, I think we, we kind of undershot, um, some of our expectations to it to some degree, uh, kind of looking back at it. So I, I wish we were a little bit more [00:36:00] thoughtful about, hey, how fast will agents continue to evolve and how much it will continue impacting.

How quickly will enterprises feel comfortable taking on agents into various workflows, or I guess developers taking on various workflows, which should've been a bit more obvious to us. But at the time I think we were thinking about it as like, hey, ClickHouse is a great, um, data store, and obviously people like Anthropic and OpenAI are using it for all their highest volume, most difficult workloads, but they're spending a lot of time engineering the bits around it.

Um, and we knew that, hey, we spent a lot of time building around ClickHouse who really essentially solved those problems. How do we make sure that out of the box ClickHouse isn't just a amazing data store a lot of these companies use, but also an amazing observability platform out of the box that people can use and they can kind of-

Matt Klein: Yeah

Mike Shi: you know, um, slot into their existing setup as it is And that was kind of our sort of, um, kind of north star at the time, say, "Hey, we just wanna make ClickHouse easier to adopt as an observability platform." I don't think that has really disappeared to any extent. Obviously, people are still expecting like, "Hey, when you come in, you know, um, we want a platform that just works out of the box," [00:37:00] not thinking about, you know, "Hey, how do I be a DBA?"

Or worrying about, like, kind of the intricacies of column or data store, but just like, "Hey, I got a bunch of logs, a bunch of traces, a bunch of metrics. I just wanna push them into-"

Matt Klein: Yeah

Mike Shi: ... "a place where I can, you know, give my users or my developers the ability to actually analyze and, and, and build better reliability 'cause of that."

Um, so that was kind of- I was gonna

Matt Klein: say

that-

Mike Shi: ...

the

initial thing.

Matt Klein: Yeah, yeah. Yeah. So sorry. I was j- I was gonna say really quick that we, we like to joke at bitdrift that every new feature that we do that involves ClickHouse at the, at the scale that we run, which is not small, is that it's like a day to get it working, and then it's-

like two months, you know, to get... Like, like there's a lot of intricacy-

Mike Shi: Yes

Matt Klein: ... in ClickHouse in terms of the table layout and the ingestion profiles and all of these things. Um, and don't get me wrong, it is an amazing piece of technology, but it is not actually easy to use. Um, so I can, I can completely buy, you know-

that, that, that there is room on top to, you [00:38:00] know, to make the technology easier to use. Sorry, uh,

keep going.

Mike Shi: Yeah. No, no, no. And I, I, I love that. I'm gonna go on a, a brief tangent as well, and I'll, I'll promise I'll circle back. Um, I, I totally agree with you, and that's actually one of the things I loved ClickHouse a lot actually while building on top of it, is that there were so many knobs you can tune.

A- as a builder, it's the perfect tool, and I think that's why it's so popular. It's not, "Hey, it's not one database that does, you know, um, uh, one thing really well. If you wanna do anything else, like, you know, that's wait for the developers to build-"

Matt Klein: Yeah

Mike Shi: ... "the feature you want."

Matt Klein: Right.

Mike Shi: It's like, "Hey, if you wanna build the thing that you want, um, it's gonna be possible."

There's gonna be a, a bunch of knobs which are available for you to tune. You might spend some time thinking about which are the knobs you need to tune to get your problem right, but the knobs are gonna be there. It is super buildable. You're never gonna feel limited by the data store.

Matt Klein: Yeah.

Mike Shi: It's really just about your ability to kind of figure out, "Hey, how do I construct the problem, and how do I kind of almost push the laws of physics or my NVMe drives or my IOPS," right?

Matt Klein: I, I,

I [00:39:00] agree. I, I actually call it- ClickHouse brain. Like, like you have to-

Mike Shi: Yeah.

Matt Klein: ... you, you have to, like, get your ClickHouse brain going. But anyway, sorry. Yeah.

Mike Shi: Right. Exactly. But, but once you do, you can solve really, really... Obviously, kind of once you format the problem the right way... And really, I, I think the great thing about ClickHouse is a lot of times when we were solving these hard problems, uh, as you mentioned, like, you know, hey, you got it working and then you have a customer comes in at scale and wants it to work on a lot bigger scale than before.

Matt Klein: Yeah.

Mike Shi: Um, a lot of the times, like the logic about it is understanding how the mechanics of ClickHouse works, and then just reasoning about kind of like, hey, like where's it limited? Usually it's, it's I/O limited, it's disk limited, it's, it's, you know, it's, it's limited by our latency to, to scanning S3 or non-contiguous range.

Matt Klein: Yeah.

Mike Shi: Like there's kind of a lot-

Matt Klein: Yeah

Mike Shi: ... like kind of like, almost like the laws of physics are literally what constrains your problem set and just kind of you're trying to like figure out how you can cheat those kind of constraints-

Matt Klein: Mm

Mike Shi: ... to get where you want.

Matt Klein: Sure. Yeah.

Mike Shi: But a lot of the times it isn't so much like, hey, ClickHouse is limited.

It's like, hey, ClickHouse is architecture, which then touches reality in some way, and those are the kind of constraints we need to [00:40:00] think about. And once we get those in the right way and kind of rotate the problem the right way, then the problem becomes so much better on top of ClickHouse. And I think that's one of the joys of, of, of the technology.

It isn't arbitrary limits. It's really just like, hey, you're touching disk, you're touching network, you're touching the CPU, you're evicting cache lines. Um, those sort of things are the things you have to think about, and then everything else, once you solve it, becomes kind of, you know, comes out of your way.

So that's... Anyways, just, just a bit of a tangent, but kind of one-

Matt Klein: No, no, no, no, no

Mike Shi: ... of the fun things about working on, you know, working on-

Matt Klein: No,

that's fine I, I, I also... I mean I, I could spend a l- I could spend a lot of time talking about ClickHouse. I think it's a very interesting database, um, and maybe we'll do a separate show just on ClickHouse.

But, but let's, let's come back to the agentic-

Mike Shi: Yeah

Matt Klein: ... revolution. Um, you were talking about how your charter has either changed or not changed, and would love to learn a little more about that.

Mike Shi: Yeah. And, and y- you're right. So, so we were talking a bit about sort of, hey, we wanna make ClickHouse really great to use out of the box for our users, which aren't gonna be your [00:41:00] typical, you know, developers who are really into databases, but they're gonna be, you know, folks who are, um, you know, they might be an SRE team, they might be a platform team, they might diff- all sort of different titles, of course, um, and smaller teams like, you know, the CTO, of course.

Um, and they might be used to running a database, right? 'Cause they're storing their logs somewhere, or maybe it's a SaaS product which has a database as well. But they're, you know, they're consuming somewhere, um, but they wanna make it easier to use. And that was kind of the problem set that we came in and spent a lot of time working on.

Um, but definitely as you can tell, with agents taking over for coding and then increasingly becoming better and better at root cause analysis and, and infrastructure, um, that's where, you know, our, our mandate's being shifted from... Or not mandate, but kind of like, you know, w- our mandate kind of like, "Hey, help figure out how we can make, you know-

Matt Klein: Yeah, of course

Mike Shi: make our customers and users happy."

Matt Klein: Right.

Mike Shi: But kind of how-

Matt Klein: Yeah

Mike Shi: ... we think about how we could fulfill that shifts from, hey, we wanna build a great interface, and it's great, easy to use on top of a data store, and to how do we make sure that the agents are a first-class user of our data store-

Matt Klein: Yeah

Mike Shi: ... as well?

'Cause increasingly, we talk to engineers [00:42:00] and, you know, uh, sure, there's a lot of people who are mixed, but then you... if you see kind of that leading path of early adopters, they're gonna be talking about, "Hey, I really rarely go in the UI anymore." Or mo- even a, a bigger chunk of the populations, "I go agent first, and then if there's something I need to dig in, I'll go in the UI."

So the agent has become kind of that most important user to us all of a sudden.

Matt Klein: Yeah.

Mike Shi: And that has definitely shifted how we think about how the product should be built, what should be prioritized, a- and what kind of, uh... and continually think about what sort of interfaces and, and value add can we still add on top of the data store, um, uh, for these agents to make them better and better for what we're trying to build.

Matt Klein: Yeah. I, I, I mean, I, a- again, like I find this topic s- so interesting and, um, the, the thing that came to mind while you were speaking that we also think about at bitdrift is that in the observability space, I feel like in the very initial stages of the AI revolution, there's been almost a fork in the road, and I think there's some set of vendors who are rushing to [00:43:00] build, I'll call it, like, integrated agentic experiences.

Mike Shi: Mm-hmm.

Matt Klein: You know, people that are building the flows into it, right?

Mike Shi: Yeah.

Matt Klein: It's like they want you to go into that tool, and they want you to ask the questions and do all those things. And, um, I promise that I'm actually getting to a question- ... is that at, at bitdrift, what we've seen... and maybe this is bias for the customers that we have, is that we know no customer that wants that.

All of our customers want to use our data source-

Mike Shi: Yeah

Matt Klein: ... to bring their own agentic experiences to, their own models, their own harnesses, their own... whatever. Um, and I, I wanted to ask you about that split, right?

Mike Shi: Yeah.

Matt Klein: Because I'm sure you've seen it in the market, you know, this market of the- Hundred percent

built-in tools versus the data provider and feeding into your own agents. Would love to learn about how you think about that, both from an industry perspective and also from how you're building [00:44:00] product.

Mike Shi: Oh, yeah. We spend a lot of time talking about this. Um, I was whiteboarding with my engineering counterpart, you know, just Friday, uh, afternoon, uh, for a couple of hours about kind of how that's continuing to evolve and what we're hearing from the market.

Um, I totally agree. I think a lot of the folks that I'm talking to, um, uh, see the same trend you do, Matt, um, including us, where it's people have their agents. It might be maybe it's a Claude code or a Cursor. It might be a, um, custom, uh, internal agent or-

Matt Klein: Yep

Mike Shi: ... you know, for Clickhouse there's a ton of them as well.

I'm sure a lot of these folks and you guys might as well. Um, and they wanna bring the data into their agent. And I think that's, you know, definitely the dominant pattern going forward 'cause I, I think there was like a, a tweet a while back. It's like, you know, uh, it, it's hard to imagine that for every SaaS we subscribe to, we're gonna be paying an additional, you know, say, twenty dollars a month, uh, per, per user for-

Matt Klein: Crazy

Mike Shi: their agent product, right? No. It's, it's kinda hard to imagine versus, "Hey, um, I already have, you know, uh, [00:45:00] my, uh, enterprise kinda way of running AI," whether it's a contract with the, you know, the, the, uh, lab, whether it's Anthropic, OpenAI, or Bedrock or others. Um, or it's a, you know, "Hey, we have our internal open source deployments that we vetted and feel comfortable running open source models internally for security reasons."

Um, there's kind of, there already, uh, an existing paradigm of how basically tokens are being vended within a company and then how, and then on top of that, kind of within the enterprise, and I think it's the right thing to do, how companies think about, "Hey, we wanna build our internal agent that's really good at doing our company things," right?

Matt Klein: Yeah.

Mike Shi: And it's gonna be customized all the way. It's gonna be plugged into all our data sources. Uh, versus when you go to a vendor, their agent's gonna not have any of that, doesn't know kind of, it knows maybe just-

Matt Klein: Absolutely. Yeah

Mike Shi: ... the segment company that it deals with, right? So it doesn't know, like, your Salesforce CRM, your operational OTL, you know, your Postgres databases.

It won't know

kind of your-

Matt Klein: Source code. I mean, I, I think, I think that's the most important one. It probably can't see your source code.

Mike Shi: Yeah.

Matt Klein: I mean, there's, there, there, there are so many, but yeah, sure. Yeah.

Mike Shi: 100%. Exactly. All these sources, you write that vendor, at least by default, will not have access to and will take a while to [00:46:00] get access.

And imagine that-

Matt Klein: Mm-hmm

Mike Shi: ... across every single vendor. It's very difficult to imagine that paradigm working out. A-and even just from a kind of an agent effectiveness perspective beyond that is do we really think there's gonna be one agent that can do, you know, uh, all your reliability problems, right, for every company under the sun?

And, and ClickHouse is a great example of that, is that, you know, we have a team that debugs database problems, and we have teams that debug frontend problems. And it's very hard-

Matt Klein: Absolutely. Yeah

Mike Shi: ... to imagine. And they don't have, they do not have the same workflows, believe it or not, right? Um, so it's really hard to imagine, hey, you're gonna have an agent that, you know, does one and does the other from a vendor.

That just works out of the box. It's, it's incredibly hard to imagine.

Matt Klein: Yeah.

Mike Shi: So therefore, you know, it, it has to be, you know, these teams with their expertise, with the workflows they built out, they're gonna be the experts building that agent. Um, and they're gonna have their way of bending the tokens to, to make that happen.

Um, and for us it's about how do we give them the right tools to accelerate them to make a really, really awesome experience out of the box, as [00:47:00] opposed to, you know, uh, having them to kind of figure out all the, all the hard problems of building, you know, or having agents integrate with, you know, ClickHouse and Observability data source.

Matt Klein: Yeah, I mean, and, and this might be a controversial statement, but it, it also seems to me from a business perspective that anyone that is building stuff directly on top of a model, whether that be OpenAI or Anthropic-

Mike Shi: Mm-hmm

Matt Klein: ... or any of, any of the open weight models There's not much moat there, right? I mean, it's like all, all you are is a pass-through for-

Mike Shi: Mm-hmm

Matt Klein: for something that's getting passed to a model, and that came to mind last week. I'm sure you saw it, but I think Anthropic announced their AI SRE product thing-

Mike Shi: Mm-hmm

Matt Klein: ... right? And that goes up against the 10 other AI SRE companies. And it, it, at least at, at bitdrift, and it sounds like you think about it the same way, it seems like in this new world, data is the moat, right?

I mean, it's like if you can provide the best [00:48:00] quality data and give it to people to help them consume it, again, in their custom harnesses or through third-party companies or something like that, that strikes me as the most valuable currency right now. Like, especially as intelligence is becoming commoditized and there's all these different companies that, as far as I can tell, mostly have the same product.

But anyway, again, like, that's potentially a controversial statement, but w- wanted to see if that resonates with you or not.

Mike Shi: Yeah, I mean, I think data is a big moat. Um, and that itself might be sufficient for a lot of companies as well. Um, I think we also think about how do we make sure that those teams are empowered to build beyond the data, right?

A-as you kinda mentioned, kind of it's almost like the, the similar thing you mentioned earlier, like getting it working day one as a prototype is very cool, but then how do you expand the envelope from there and, and move beyond, hey, I have my agent writing SQL against my database, which is very super powerful, by the way.

If, if folks [00:49:00] haven't tried that out, please connect your agent to a read-only, uh, instance of your database-

Matt Klein: Yeah.

Mike Shi: ... that you can trust with your agent, right? Uh-

Matt Klein: Read-only very, very,

very, very important, yes.

Mike Shi: Yeah. Right, right. Like, you know, like obvious- one, isolate it down to the permissions it should have, and then, uh, but two, once you have that, like, please try it out.

Have your agent connect to it-

Matt Klein: Yeah

Mike Shi: ... a-and ask it questions. It's, it's really amazing. But then moving from cool, we have this really cool analysis to be able to do some of these things, to be able to, hey, how do we make sure these agents are more reliable at more of these problems over time? Um, there's kind of all these, uh, you know, kind of, uh, maybe we could call it, like, harness-adjacent problems.

Uh, like, you know, how do we build the right tools for the agents? How do we help them-

Matt Klein: Yeah

Mike Shi: ... manage the context as best as possible? There's a lot of these kind of adjacent problems that we think will become important over time as well. We're kinda starting off with the kernel, which is put your, you know, as you mentioned, intelligence next to your data, and you get a lot of really great-

Matt Klein: Right

Mike Shi: you know, outcomes out of that. And then how do we make sure engineers can scale that beyond sort of that prototype or POC, um, or those basic workflows into things that are more and more advanced, [00:50:00] and how do we offer tools there? So I think there's gonna be continually value added by folks focusing on the specific area.

Data's the biggest core of all that, uh, but I think it does radiate out, uh, a bit as well, uh, in terms of, you know, other things that we can... You know, maybe there's, like, a really cool SRE harness specifically built, uh, that would be, you know, a huge value add on top of the data, and there's definitely opportunities there as well.

Matt Klein: Yeah, for sure. Cool. Um, I could spend all day talking about this, but unfortunately- ... we are al-almost up at time. Um, a-as we wrap up, um, I don't know, you know, where, where do you think things are going, uh, over the next couple of years? Um, you know, you said yourself that you had maybe under-underestimated the rate at, at which these tools are progressing in terms of their capabilities.

I'll be honest, I, I, I also personally underestimated it. Um, and I don't know. I, I mean, do you, do you think that things are going to plateau? Do, do you think that things are going to keep [00:51:00] increasing? Like how, how do you think it's gonna change what you all are doing on a day-to-day basis?

Mike Shi: I think the, one of the biggest challenges that we have at the moment is that the differing rate of adoptions for these tools-

Matt Klein: Hmm.

Interesting.

Yeah ...

Mike Shi: um, we can talk to, you know, uh, young companies, startups who, you know, they're kind of obviously being born in an AI native world, which must be incredible-

Matt Klein: Yeah, yeah

Mike Shi: ... by the way-

Matt Klein: Right

Mike Shi: ... uh, to, to, to do-

Matt Klein: Right

Mike Shi: ... to start a company like that. Um, and they're obviously gonna be way more centered on AI being maybe the only consumer of this telemetry, um, versus when we talk to a large enterprise and it might be, yeah, there's gonna be these, you know, fantastic early adopters, but there's a ton of people who are, um, either they're not AI native or they have limitations on their permissions to be AI native.

Like, there's a lot of things where they're gonna be a lot slower to adopt. And one of the-

Matt Klein: Yeah

Mike Shi: ... challenges we have now is not only, like, the rate of technology advancement, but it's also to some extent [00:52:00] kind of like, hey, how quickly will the market actually adopt these tools, and when can we build the right thing at the right time-

Matt Klein: Yeah, yeah

Mike Shi: for these users so that it's actually meaningful to them as opposed to being like a far out pipe dream that they can't adopt for a couple years and stuff like that. So I think that's gonna be a huge challenge. But I think, you know, it'll keep getting better for sure. Like, these agents are gonna keep, you know, keep getting better and better over time.

Matt Klein: Yeah. I mean, I think that's a, that's a fantastic point, and we also see the same thing. I mean, it's like we, we sell to some companies that they care about the dashboards. They, they, they want to have all the chart types and all these different things that you could possibly have. I'm sure you have customers that are-

Mike Shi: Yeah.

Matt Klein: asking for similar things. Um, and then, and then you have the polar opposite. You have people that just point it at the data and tell the agent to draw whatever chart they want, and it does it-

Mike Shi: Yep

Matt Klein: ... you know? I mean, I, I just think it's a, I think it's a fascinating world. And what we see also, which I'm sure that you see, though, is that there are a lot of hybrid organizations.

Like [00:53:00] even organizations-

Mike Shi: Yep

Matt Klein: ... that I would say are, are pretty AI forward, you know? They're, they're, they're, they're really into it, you know, they're heavily adopted, um, all the tools. We still see people that come into the dashboard and they wanna use the dashboard-

Mike Shi: Yeah

Matt Klein: ... to do certain things, and they're using the tools on the side.

So I, I think, I think we see that too, and I think that is fascinating in terms of how we develop products moving forward, because you were already talking about yourself, is that in this new world, are you API first? Is that all that matters? Like, do you do that-

Mike Shi: Yeah

Matt Klein: ... first and not worry about the dashboard, or, or do you do these things in lockstep?

I think it's so fascinating.

Mike Shi: Yeah. I think the, the, the last thing I would s- maybe two of the last things I would say is one is kind of, you know, and not even being API first, 'cause, you know, APIs have been really great for computers, but we've seen, you know, those APIs don't translate as well, at least today, and we've still seen, you know, y- as models get better, that hasn't proven out yet, where, you know, tools for agents, tools purpose-built for agents would be a lot better than just APIs for them.

Matt Klein: Yeah.

Mike Shi: [00:54:00] So I think that's one thing that's interesting. Um, the other bit too is that as people rely on their agents for more and more, um, I think there's gonna be a really interesting area that will hopefully expand over time as w- and we're obviously pushing the direction as well, in terms of how are the users gonna be interacting with their agents in a meaningful way to understand.

Like, the simplest case is this, right? "Hey, I wanna go, uh, draw a chart." Uh, and your agent might go out and spit out HTML file, and then where you're gonna share this HTML file in your Slack-

Matt Klein: Yes. Yeah

Mike Shi: ... and help you.

Matt Klein: Right.

Mike Shi: Like, it's, it's, it's kind of mess, like there's, there's ... And that's obviously the most simple case of it.

Not to mention, hey, you wanna have people collaborate with your agent, right? Hey, another-

Matt Klein: Yes. Yeah

Mike Shi: ... engineer's like, "Hey, I don't think that's right. Uh, can we go-"

Matt Klein: Yeah

Mike Shi: ... "in this direction instead?" So there's all this sort of really amazing sort of, um, user-to-agent interaction that's gonna happen that I think no one has really solved to any degree-

Matt Klein: I agree

Mike Shi: um, so far, and that's gonna be, I think, the next exciting frontier for the product builders out there who are still thinking about the user, is that there's still that huge unsolved problem, not to mention just the [00:55:00] agent-to kind of data-

Matt Klein: Yeah

Mike Shi: ... problem. There's gonna be the, the user-to-agent problem that's, uh, that's completely wide open right now.

Matt Klein: Cool. All right. That is a fantastic place to end. Uh, thank you, Mike. This was a great conversation. I think people are really gonna enjoy it. So, uh, that's a wrap for this episode of Beyond the Noise: Signals, Stories, and Spicy Takes. Huge thanks to Mike for joining and sharing his story. You can find this episode and all past ones on the bitdrift YouTube channel.

If you had fun, drop us a review, tell your friends, or yell your favorite hot take into the void, and just make sure to tag us. I'm Matt Klein, and we'll see you next time. Thanks a lot.