From Razr to Smart Carts: Building the Future of Payments & Retail Tech

[00:00:00]

Matt Klein: All right, everyone. Welcome to Beyond the Noise: Signals, Stories, and Spicy Takes the show where we dig into the stories of the people shaping the future of app-based computing with a special focus on mobile. I'm your host, Matt Klein, co-founder and CTO of bitdrift, and the creator of Envoy. Each episode we'll talk with engineers, founders, and technical leaders who've transformed the way their companies build and understand what's happening inside their systems.

We'll dig into the challenges, the breakthroughs, the lessons learned, and wrap it all up with their hottest takes. Let's dive in. Today I'm thrilled to have Michal Palczewski who is a Senior Staff Engineer at Instacart, where he works on the Android software for Caper smart [00:01:00] carts. He's been building mobile apps since the launch of the first Android phone with previous roles at Google on projects including Google Play, hands free and Android Auto.

Welcome Michal. Thanks for joining us.

Michal Palczewski: Hey, glad to be here.

Matt Klein: Great. Well you've had a really interesting career, so I think how I like to start these episodes is I'd love, just to learn a little bit about your background, and your path in terms of how you got into mobile development. So, feel free to take it away and we can go from there.

Michal Palczewski: Sure.

Yeah. So it was... so my first job programming was in the year 2000. So I've been doing this a while. So I remember the year 2000 bug, but, it was basically, it was for academic type software at a lab doing the genetics. And so, the guy wrote an email. And he said, 'Hey, is anybody interested in this job?'

I was like, oh, I guess I'm interested. This sounds interesting. I've been, I've been studying programming. And, when I got to the interview, he said, 'uh, so most [00:02:00] people sent their resume, but I just got this, Hey, I'm interested.'

But, but, I think, I was always very passionate about this kind of stuff and I think it...

Michal Palczewski: I interviewed well.

I got the job and I was doing academic programming for a long time. Eventually, I went to grad school at Florida State University. And, around that time I had a Motorola Razr. And I had just up- and then I upgraded it to one of those...

Matt Klein: sorry, I - I was gonna say, I think most people on this call probably don't recall what came before smartphones, you know?

Michal Palczewski: Fair enough. Yeah.

Matt Klein: They, they probably never saw Blackberry or never saw a flip phone or any of those things... so you and I are probably dating ourselves here, but anyway, go ahead.

Michal Palczewski: I'm definitely dating myself. Yeah. So...

yeah, I had, I had a... so the Razr phone was like this, it was the pinnacle of... dumb phones.

Matt Klein: I remember, yeah.

Michal Palczewski: Yeah. It was very sleek. And I got a... Sidekick. So [00:03:00] Sidekick had a hardware keyboard. The screen flipped up- upside down.

Matt Klein: I remember that one too. Yeah. Yeah.

Michal Palczewski: Yeah. So that was my first foray into smartphones and I would just be messaging people all the time. But what really frustrated me with it was they had a SDK, so you could program for it, but you couldn't just download the SDK, you had to apply to get the SDK.

And in order to apply to get the SDK you had have an idea. Yeah, this is, this is the kind of app I want to build. And if they deemed that it was good enough, they would send you the SDK.

Matt Klein: Sorry.

And, and this is the SDK for the Motorola Razr flip phone, basically.

Michal Palczewski: Oh, this is for the Sidekick.

Matt Klein: Oh, the Sidekick.

Okay, got it.

Michal Palczewski: The Sidekick. Yeah, the Danger. So funny enough, the people that made... Danger, Danger was the OS on Sidekick, and they're the same people that ended up making the Android. So it was the same... team.

Michal Palczewski: But yeah, you couldn't just get the SDK. I didn't bother applying for it. I just wanted to tinker. And, but when I saw that Android was coming out, it was by [00:04:00] Google.

There was a lot of open- it was open sourced, the SDK, the docs were just on there. You could start programming for it. I - I immediately jumped on it and, in some ways it wasn't as good. The battery life was worse, but it had a real web browser in it. And, what really frustrated me, and this is, this is what gets me into my first app, is, at my school we had a wifi network that was a captive network and you had to log in with your school credentials.

And those were like a - a long password with symbols and numbers,

Matt Klein: Yeah

Michal Palczewski: and capital letters and you had to type that in literally every time you'd pull your phone out of your pocket if you didn't want to use your limited mobile data, which is all you had at the time. And it wasn't very fast. It was 2G, so quite... yeah,

Matt Klein: quite slow,

yeah.

Michal Palczewski: Quite slow. Quite slow. Especially indoors. And, so I wrote an app that basically it would save your password. I saved it to shared preferences. [00:05:00] Very... very secure. I mean, or lack thereof.

Matt Klein: Probably shouldn't tell people that you were doing things like that, but yeah,

Michal Palczewski: probably shoudn't. If you doubt if you still have the app, go ahead and uninstall it.

I'm sure they've come up with better ways of logging in now.

Matt Klein: Right.

Michal Palczewski: And so, but it would just log you onto the network. And so you didn't have to, cons- so it was very convenient. You just open it up, you'd be on the network.

Matt Klein: And, and,

sorry, sorry. Just so I'm keeping track, this is when you were working a separate job in academia.

You just decided to pick this up on the side, basically?

Michal Palczewski: I was a grad student at the time.

Matt Klein: Yeah. Yeah.

Michal Palczewski: So I was basically a TA, yeah. So I just did this on the side and... I put it on the app store, sold it for 99 cents because that's how... right now it'd be a subscription for 99 cents a month.

Matt Klein: Right, right.

Michal Palczewski: But that, that, that's how you did it back then, made like $500, which you know, is not nothing.

A little, little help as a grad student.

Matt Klein: Yeah, of course. [00:06:00]

Michal Palczewski: And so, what happened next is I got an email from a software company. They were literally five minutes from my house and I was at home that day and they had a career fair and they said, come here and you'll get free lunch, which as a grad student, that sounds great.

Free lunch. And so I went. And they asked me if I knew anything about mobile app development and I said, well, I got this kind of little app that I put in the app store and it was immediate, 'Hey, you have to come talk to the Vice President of so- of Engineering. They, they got me to come in for an interview and I got an internship like...

pretty quickly, a paid internship. So, that also helped. Cause I was, again, I was a grad student. I had a stipend, but having a paid internship on the side was wonderful.

Matt Klein: Yeah. And, and, and, sorry, were you, were you a computer science grad student or were you studying something separate?

Michal Palczewski: I was studying scientific [00:07:00] computing.

Matt Klein: Oh, interesting. Okay. Yeah.

Michal Palczewski: Which is like, using computers to do science, a lot of numerical algorithm type stuff.

Matt Klein: Yeah,

Michal Palczewski: I- it was in genetics. So there's a pretty successful genetics paper that I have... a lot of citations for.

Matt Klein: Fantastic.

Michal Palczewski: Yeah.

Matt Klein: Great.

Michal Palczewski: And then immediately, first day on the job, they're like, okay, so we need you to go to iTunes U and learn iOS.

So I did, and I, and, and so I was doing iOS there, and then they wanted me to do Android again.

Matt Klein: But, but hold on. This is, this is what year? Because I - I don't remember. I- iPhone came out and Android came out in what year? Like 2006 ish or something like that?

Michal Palczewski: Yeah, 2006 ish, but I think I started there at 2000...

Well, I - I think Android was a year later, like a 2007 or...

Matt Klein: 2007? Okay. So this is fast forwarding. You had, you had been in grad school for a while at this point?

Michal Palczewski: I had been. So this is like 2010?

Matt Klein: Yeah.

Michal Palczewski: Ish. Yeah.

Matt Klein: Okay, got it, yeah.

Fun [00:08:00] fact, I - I actually started my career working on Windows Mobile, which was the precursor

Michal Palczewski: Okay.

Matt Klein: To... you know, Windows phone and then obviously it was pre iPhone, all of those things. So most of my later career has been in backend systems and stuff like that, but I actually started my career doing this stuff too. So I - I definitely remember this time.

Michal Palczewski: Awesome

Matt Klein: It was a very, very exciting time.

Anyway, keep going.

Michal Palczewski: Well, I was at Microsoft in 2003. I was working at their call center for developer support.

Matt Klein: We - we must have overlapped because I actually worked at Microsoft 2003 to 2008. Yeah, yeah,

Michal Palczewski: yeah. We definitely overlapped. And I remember they had Windows phone back in 2003, or whatever it was called exactly back then.

But they were like, this one is it. This is - this is the one.

Matt Klein: I mean, what, what's frustrating in hindsight, and we're getting a bit off topic, is that Windows phone, pre iPhone had all of the [00:09:00] pieces. They, they could have been iPhone

Michal Palczewski: yes.

Matt Klein: But Microsoft was so hyper-focused on the enterprise market and all of those things... that they just, they didn't see that the consumer market, you know, that

obviously, Apple and Google captured would be so, so big.

Michal Palczewski: Yeah I mean it's really...

Matt Klein: it's probably, probably one of, one of the biggest gaps that they've ever had anyway, yeah.

Michal Palczewski: Yeah. I mean, it took Apple to show everybody because

Matt Klein: Yeah.

Michal Palczewski: Even, even then, everybody was chasing Blackberry.

Matt Klein: Yeah, yeah.

Anyway, sorry, go ahead.

So we're in 2010 ish and you're now doing... iOS and Android development.

Michal Palczewski: Yeah. And, and so as an intern. We launched the... this was my favorite app that we launched as, as I was an intern. We launched a ticketing system for police officers to use in Puerto Rico.

Matt Klein: Okay.

Michal Palczewski: Yeah. So, it worked with like a Bluetooth printer.

There was some web views, so we, we didn't have to do too much, but we had to write this out. It - it deployed to Puerto Rico [00:10:00] and my big thing was I had to write the blue... I had to make this Bluetooth printer work. The SDK was not good enough to get the tickets to look like we wanted them to look like, and I had to, I decompiled their SDK just to figure out how to, how to work, work it...

Matt Klein: Fantastic.

Yeah.

Michal Palczewski: And...

Matt Klein: And, and, did you work on the backend software for that as well, or were you...

Michal Palczewski: I did not work

on the backend back then.

Matt Klein: Got it. Yeah.

Michal Palczewski: Yeah, we were like printing these tickets, sending it to QA, coming back, fixing it, sending it to QA. We were like making paper airplanes out of those tickets.

Throwing 'em at each other.

Matt Klein: Yeah.

Michal Palczewski: And, and then a few months later, it was... on the front page of a Puerto Rican newspaper. The governor was holding like one of my tickets. I was like, oh my God. That's, that's the ticket. That's... it was, it was, it was like my first like really exciting moment and like, 'oh, wow.

I, I did a thing. It made a difference.'

Matt Klein: It is - it is a pretty good [00:11:00] feeling when you see your software out there and you see it being used. I mean, I'm sure it's a different feeling when you know that you've been giving people parking tickets, but... you know, it's a, it's still, it's a, it's a fantastic feeling seeing things used.

Michal Palczewski: Yeah, absolutely. So eventually I graduate from grad school. I - I get it, get my gear together, defend my dissertation. And, I have like about a week where I'm not going to grad school and I'm working my... I had gotten my internship upgraded to a job. So I'm going to work. And then a week - a week later, Google calls me.

And they're like, 'Hey, do you want to come work for Google?' And this is in 2013.

Matt Klein: How did they find you? Like how did they, how did they know about it or, or what, you know?

Yeah

Michal Palczewski: You know, they found me on LinkedIn. And, at that time I decided, you know... facebook was really big back then. And instead of- I said [00:12:00] I was gonna spend more time on LinkedIn getting my profile

up to snuff than I will on Facebook. So I just made sure I stood out. I don't know if I, I don't know if I was even thinking about standing out. I was like, 'I'm gonna get it as good as I can get it.' And, I don't know, maybe they just keep track of like, who graduated recently or...

Matt Klein: Sure.

Yeah.

Michal Palczewski: how they did it,

but one of their recruiters just reached out to me and... I - I got the initial phone screen and I did not do very well. I'm surprised they called me back for a second phone screen to this day, i'm surprised. But I must have done well enough that they were like, let's try it again. But, taking that phone screen, I was like, okay, this is what these interviews are like.

Okay, I can get... I can prepare. And so then the second phone screen I did much better on and then they called me on site and I [00:13:00] was, I read a book called Programming Pearls. Very classic in the field. I recommend that to anybody who's taking like these Leet Code type interviews. I think it's... it gets your mind thinking in the right directions.

Matt Klein: Yep.

Michal Palczewski: And so, by the time I was on site, I - I killed it and, and got hired on at - at Google and that's what really... then my career really started. Also, it helps to have a coding job if you're coding all the time and then you have a coding interview, then...

Matt Klein: Yeah, for sure.

Michal Palczewski: It's definitely...

Matt Klein: Although, you know, I mean a lot of these interviews, especially those kinds of interviews, I - I feel like it's still somewhat of a crap shoot in terms of how it goes.

Fun fact, I have actually, over my career, I've interviewed at Google and I was rejected twice.

Michal Palczewski: Okay

Matt Klein: So, yeah. You know, it's just... so it goes. But anyway, so you, so you wound up at Google and were you working on Android at Google?

Michal Palczewski: When I started at Google, I was working on iOS. [00:14:00]

Matt Klein: Oh, okay.

Michal Palczewski: Yeah. We had this... I was working on a product that, when I joined it was... I was in the payments org. And, it was about getting... so you could buy things using your stored Google credentials. But only, non-digital goods because, digital goods, you have to use Apple's in app payments.

So... was not a very successful product. Not a lot of people are using Apple devices, want to go into an app, log in with Google, use their store payment.

Yeah. I was working on that for a few months and at that time, Google was constantly reorganizing. So I got re-orged and I got re-orged on this team that looked at payment declines, and we were trying to reduce payment declines.

I - I really still to this day, don't know what I did on that team other than some SQL queries. To give you a perspective: every year I was at Google, I wrote a hundred thousand lines of [00:15:00] code. They, they have a way to track that.

Matt Klein: Wow. Okay.

Michal Palczewski: And that... those two months, I wrote 200 lines of code, total.

Matt Klein: So, so were you, I - I mean, were you like just looking at analytics or like, what, what were you doing during that time?

Michal Palczewski: Yeah, yeah. I think I was... I was looking at analytics and trying to discern why payments were declining and what we could do to prevent payments from declining.

Matt Klein: And it was payments were declining... I - I guess I'm trying to understand, because this is obviously a topic that I'm, I'm very passionate about.

Was this stuff that was happening on, on the client that was hard to understand? Or was it the interaction between like the client and the payment processor? You know, it's just like a lot of these flows are very difficult to understand. So, what was, what was involved in trying to figure all this out?

Michal Palczewski: Yeah. I have, I'm, I'm not really sure. I - I... it was not on the client. It was definitely in the backend. I think Google has its own payment processor. Well, they...

Matt Klein: [00:16:00] I see

Michal Palczewski: they're... they still work with banks, right? But they're, basically have their own Stripe. And that was in that...

Matt Klein: Right, yeah

Michal Palczewski: And, and so it was like, okay, is their card expired?

Is that why it declined? Is it a fraudulent card? Can we ask them for another card? It was... trying to get data to help drive product decisions.

Matt Klein: Sure.

Michal Palczewski: And, and I - I don't know if I did anything, like, worthwhile... I - I was like looking at the time and I was like, is it time for me to go home yet?

It was probably like one of the least satisfying times in this, in this industry I've ever had. But, they needed me to come back to finish that original project, so that they could launch it on time and be done with it. And so I came back. And they had a new manager. I worked with him. He - he ended up being my manager for a very long time.

Because I asked, I was like, can I stay please? And he said, oh yeah, I'd love to have you on my team. I had to talk to the director and they... and so that's what got me started on the [00:17:00] hands-free project.

And, this was... a Larry Page idea. And it was basically, we want payments to work where you keep your phone in your pocket, you just, you just say, 'I wanna pay with Google,' and then you're done.

And, it took us a couple years to get this to work right. And we worked with McDonald's. And, you could eventually just walk into a McDonald's and say, 'I wanna pay with Google.'

Matt Klein: Is this - is this different from... because I use Google Pay all the time. You know, I go into the store and I take my phone out and I tap it on the console.

Is that that? Or is this something else that we're talking about?

Michal Palczewski: No, this is something else. This was a little bit more, more advanced. So we used Bluetooth, low energy facial recognition.

Matt Klein: Oh, I see. Interesting.

Michal Palczewski: Yeah.

Matt Klein: Okay.

Michal Palczewski: Yeah, so the Bluetooth low energy told... the BLE told you that the person was in the vicinity of, of the payment terminal. And [00:18:00] then the facial recognition allowed you to say who was actually in the front of the line.

Matt Klein: Well, that's fascinating. So was your software, is this a... is this terminal side only or would it also interact with your phone that - that has your identity? So is there like an SDK on both sides?

Michal Palczewski: Yes. So there was terminal side, there was a separate box in the store and an app installed on your phone. So you need to have the app installed.

Matt Klein: Okay.

Michal Palczewski: And this was both Android and iOS and I was leading both the mobile teams and eventually the whole team.

Matt Klein: Super interesting.

Yeah. And is that - is that technology, is it still widely deployed?

I actually don't know much about how this area works. Mostly because just as a consumer, as I said, I use the tap to pay all the time.

Michal Palczewski: Right.

Matt Klein: But I don't know that I've ever, at least to my knowledge, that I've done the facial recognition payments option.

Michal Palczewski: No. It, it... eventually we decided to [00:19:00] roll it into Android Pay, and then Google got sued for using, facial recognition in a public space and it kind of... so they didn't wanna launch this. Yeah.

Matt Klein: I mean, it sounds like a fascinating project though. So, you got as far as to get production prototypes, as you said, with McDonald's or some other store?

Michal Palczewski: Yeah, it launched at 40 McDonald's in the Bay area.

Matt Klein: Oh, wow, okay.

Michal Palczewski: Yeah. So yeah, so we were actually like using it to pay. And yeah, getting it to work reliably was the... freakiest part.

Matt Klein: That was actually gonna be my next question. I mean, I, I'd love just because I'm interested in these kinds of things. It's like, what, what was the hardest thing that you had to debug when, when you were trying to launch this thing?

I mean, it sounds like there's so many moving parts between the facial recognition, the terminal part, the phone part. I mean, it just sounds like there's so many pieces that can go wrong here.

Michal Palczewski: Yeah, so the, the phone [00:20:00] needed to be in a particular geolocation and then it would, listen for beacons and, and then it, it would check in and... but then we wanted it to work offline. The most challenging thing was that it could fail in so many

Matt Klein: Oh, absolutely.

Yeah.

Michal Palczewski: So many different ways and what, what I focused on was... I took a look at every single bug report that came in. There was a lot, and...

Matt Klein: Sure.

Michal Palczewski: So sometimes I'd be skimming over them, right? But I wanted something actionable to come out of every single bug report. And so sometimes the most actionable thing was to add more logging.

Matt Klein: Yep. Sure.

Michal Palczewski: So, so I built - I built the framework at the time. I think right now you get these kind of for free, but we wanted to make sure that every bug report had all of our logs attached to it.

Matt Klein: Yep.

Michal Palczewski: And so, we made sure that that was sent up, sent up with that, like literally our logcat logs were piped into into this thing that... so that it [00:21:00] was part of the bug report. And, and then it was like, yeah, okay, here's a bug.

Does it- do I need more logging? Do I need more... probably. Okay. What can I log? Where did it fail? So I was very methodical about this, but it was kind of like, how do you eat an elephant? One teaspoon at a time, and that's how it was. And so you didn't see any huge jumps in reliability, but then, you zoomed out and looked over like a six month period and it was just like a...

Matt Klein: Absolutely

Michal Palczewski: steady, yeah, steady.

Matt Klein: Well, it's also, I

mean... as you obviously know, mobile is so difficult compared to server because every time you gotta add a log, you have to update the firmware on the devices that are running in the store with the facial recognition software, you gotta get the app update, update out and all of those things, so it's so painful, you know? Versus what you can do on server.

Michal Palczewski: Yeah, it - it wasn't as bad on the firmware that was running on the- in the store, because that was our hardware. So it [00:22:00] was, okay, we'll just push, we'll just push a new, new firmware to that. But on the phones, you have to convince users to update your app.

Matt Klein: Absolutely. Update. Yep. For sure. Yeah.

Michal Palczewski: And that's, that's the trickiest part is...

Matt Klein: yeah.

Michal Palczewski: If they're not updating, then you're getting like, okay, well this is an old version. We fixed this.

Matt Klein: Absolutely. Yep.

Yeah. Cool. Well, so - so you, so you worked on the hands free stuff. Yeah. And then what, what happened after that?

Michal Palczewski: So after the, the project, our team just became the the Google Pay Growth team.

And so we worked on various, Google Pay, growth initiatives. And that was especially tricky because back then GMS Core or Google Play Services was launching maybe once every six months. And...

Matt Klein: Right.

Michal Palczewski: A big part of Android Pay was on Google Play Services. And so you had to get it right and if you wanted like a cherry pick late in the dev cycle, then it was going up to a vice president and he was gonna be grumpy about it.

Like it just... so [00:23:00] you had better get your ducks in a row and, and get things, get things, ready to go.

Matt Klein: Yeah. I mean, it's - it's hard to imagine these days of developing any software that's on a six month release cycle, but, I do remember back earlier in my career that that was not uncommon for sure.

Michal Palczewski: Yeah. Yeah. And,

they wanted it to make it more frequent, but they... stuff just kept slipping and...

Matt Klein: Yeah.

Michal Palczewski: Yeah.

Matt Klein: For sure. Yep.

Michal Palczewski: But Android Pay did launch every... I think it launched every two weeks. You had to get a bunch of approvals, every time you wanted to launch it. Which worked when, when I was on the hands free team, that was tough.

We didn't have a lot of people on that team. And then you had to go through the same approval process every time you wanted, launch a new version as, as, the Google Pay team, which was like much bigger and...

Matt Klein: Yep.

Michal Palczewski: They could have a TPM drive the whole thing.

Matt Klein: Yeah, of course.

Michal Palczewski: Yeah. And then eventually I was kind of got... I was like, I'm- I don't want to work in payments anymore.

Like, I've been [00:24:00] here four and a half years. And, I was like, okay. Oh, one of my colleagues that was on my team, he went, he went to work for Android Pay. Or I mean, sorry, Android Auto. So we recommended Android Auto, and that was, that was a fun team to be on. They... they had all these cars and so

you could, like one day you'd drive home with a Jeep Cherokee, another day you'd drive home with an Alpha Romeo and you'd be testing out the...

Matt Klein: all for testing purposes. Of course.

Michal Palczewski: All for test... Absolutely. Yeah.

Matt Klein: So which, so which part of Android Auto were you working on?

Michal Palczewski: The head unit. So this was Android auto embedded, which is different than what most people think of Android Auto is where it runs on your phone and projects to the...

Matt Klein: Yep, sure. Right

Michal Palczewski: unit, but a lot of Android head- but a lot of head units themselves are running Android. And so, that, that's the...[00:25:00]

that's the version of Android that I worked on.

Matt Klein: Yep.

Michal Palczewski: I - I did not write a lot of code during that time as much as I tried. I think they wanted me more because I was a Staff Engineer by that time and they wanted me more to like review, review, their -, their docs. Make sure that the design doc... the design is fine and, and drive projects and think of edge cases and, and it... I'm - I'm more of an execution kind of.

Matt Klein: Yeah.

Michal Palczewski: I like to be in the, weeds-

Matt Klein: seems like you like to get your hands dirty a bit more.

Michal Palczewski: Yeah. I love to get my hands dirty. And by that time I, a few people, I had known went to Instacart.

Matt Klein: Okay. Yeah.

Michal Palczewski: And so, I remember, I kind of interviewed there,

and started working there. And that was quite the change of pace. because that was back in 2019

Matt Klein: For sure. Oh wow. Yeah. I mean... and that was... well sorry, so that was right before the pandemic started then, right?

Michal Palczewski: A year before the pandemic.

Matt Klein: A - a year before, yeah. But obviously still a fairly early startup at that point.

Michal Palczewski: Right. Yeah.

Matt Klein: Yeah. So, so [00:26:00] yeah, I mean, I guess tell me more, just because you've, obviously in your career you went from being an academic obviously to working at Google and then going to Google and working at a - at a faster paced startup. That's obviously, those are very big transitions. So yeah, I would love to learn a little more what - what it was like to - to go between those.

Michal Palczewski: Yeah. So it, it was very much... I just loved how much less process there is in an early stage startup. So, we'd be constantly sending each other, like the ship, a Slack icon.

Matt Klein: Yep. Right.

Michal Palczewski: Ship it. Ship it. And so like at Google, when you wanted to launch a new version of the app, there was a whole review process.

Everybody had to okay it. And, like a whole stack of people, legal, vice presidents, whatnot. I, I don't even remember, like everybody that had to okay it. You go to Instacart and, and then a TPM actually hand you... you submit a PR, or I guess a CL [00:27:00] at Google and they... and it goes to a TPM and he'd make sure that everything is done and, and there's your binary...

Matt Klein: Right.

Michal Palczewski: And then he actually puts it up in the app store or puts it up at the Google Play Store. Whereas like you get to Instacart and it was like, okay, so you're now on call, so you have to upload it to Apple. And I was like, okay, cool. Yeah. And there was no approval. It gets released every week, it gets cut every week, it gets released every week,

Matt Klein: Right, yeah.

Michal Palczewski: The cadence was much quicker. The first thing I noticed was how easy it was to find a meeting room. If you ever needed a meeting room, they were all empty. And whereas at Google, like they were always full of people, so there was so much less meetings.

Matt Klein: Fantastic. So, so what were you working on there,

when you started?

Michal Palczewski: So I've been on so many different teams there. I, yeah, I joke that I haven't been on one team for more than a year except now I've been on Caper for two. So at first I was on the platform team but when I'm working on platform, I also like to write features. There's... I - I feel like if [00:28:00] you're just working on platform, it, it's kind of hard to understand like what the developers are going through.

Matt Klein: I could not agree more. You know, I, I think so many platform teams, whether they're client platform teams or server platform teams, if you're too divorced from the people that are using the tools, it's like, what are you actually doing? You know? I mean, it's like you're not, you're, you're not building tools that the application developers actually want to use or making their lives easier.

Michal Palczewski: Yeah, I think, I think you end up making your own life easier, but you're not...

Matt Klein: I agree. I agree. Yeah.

Anyway, go ahead.

Michal Palczewski: So shortly after that we decide our conversions were not great, so they took a bunch of people, put 'em on a conversion team. Let's get, let's get more people signing up and and so I became part of the growth team again. I was like, oh, okay, I'm on a growth... this is familiar. I was on Growth Free at Google Pay. Now I'm on Growth at Instacart.

Matt Klein: And, and, and, sorry, that was getting people to sign up or they had signed up and you're trying to [00:29:00] figure out why they're, why they're dropping off or why they're not using the app, or something along those lines?

Michal Palczewski: So it was after they've installed the app. Get them to make their first order.

Matt Klein: Right.

Michal Palczewski: So kind of... so that whole funnel, and it was like, add an item to the cart, go through checkout...

Matt Klein: Right.

Michal Palczewski: Yeah.

Matt Klein: So I mean, I - I guess from those kinds of things, tell me more about,

like what are the mechanics of, I don't know, 'cause I'm guessing you have to look at a bunch of analytics. You have to look at, I don't know, you know, how long it takes to launch the app and whether they drop off on different screens or something along those lines. So, you know, it's like what are the mechanics that you're doing to actually figure all of that stuff out?

Michal Palczewski: In a lot of my grow teams, I did not look at a lot of analytics.

Matt Klein: Interesting. Alright.

Michal Palczewski: And there's kind of two reasons for that. At Google we had project managers and they were looking at analytics and we had like a really great one that worked on Google Pay and he could look at analytics better than I [00:30:00] could.

So there was kind of like, lemme just see what he has to say.

Matt Klein: Right.

Michal Palczewski: And, and, on Instacart... I also didn't look at them directly. We had a funnel and we kind of knew that kind of from the app installed to getting the first order is what we wanted to do. And it was always more about experimentation rather than...

Matt Klein: Got it.

Michal Palczewski: And iteration.

Matt Klein: Yeah.

Michal Palczewski: Rather than like hard a- like, like sure you have your broad stroke analytics, but like your detailed ones we weren't about. And so, conversions are generally about... I found that they're generally about two things. Reducing friction. So basically making it as easy as possible for you to make your first order.

So like, don't require an account if - if you don't need an account to get, get started browsing the app.

Matt Klein: Yep.

Michal Palczewski: That, that reduces friction. So, you can, you can have a little bit more friction later in the process, but you definitely want, definitely want to reduce friction as much as [00:31:00] possible. I think Apple, Apple sign in came around that time.

That makes things a lot easier on iOS, for example, you definitely want to have Google sign in on Android because everybody's got a Google account. They just hit the button. I, I know we do like, we started doing free delivery. And I wrote a really annoying banner letting people know you get free delivery.

That was, that was like, that was the biggest thing.

Matt Klein: Yeah,

I would imagine.

Michal Palczewski: So your first delivery's free. Or, I think now we do three free deliveries, so

Matt Klein: Right.

Michal Palczewski: Use Instacart, get your free first three deliveries. But yeah, growth has always... so, that's conversions. There's kind of two parts to growth.

There's, getting the users to make their first order and getting the users to continue making orders. Conversion and retention is the way I've always split those two up. And there's different levers, right? So like conversions, reduce friction and then reduce any objections somebody might have.

Matt Klein: Yeah, of course.

Michal Palczewski: Which is like - like, free - free delivery. [00:32:00] And with retention, one of the best retention things was actually, I - I did it for Google Pay. We partnered with, Warner Brothers. They had just come out with a Justice League movie. And, what we started doing was, we started, showing a Justice League animation

whenever you used Google Pay. So it'd be like Wonder Woman flying in or, or, Batman. I actually have all the Justice League characters as little Android figurines for that project. And so it helps the user make like an emotional connection with the product.

Matt Klein: Right.

Makes sense.

Michal Palczewski: We also use stuff like punch cards. So, use Android Pay five times get, or Google Pay, I don't know what it's called now. I know we had a rebrand.

Matt Klein: Right.

Michal Palczewski: Use it five times. You get like $5 off your next purchase.

Matt Klein: Right.

Michal Palczewski: Or, or, we had another one was like a wheel... you spin the wheel and you get some kind of prize, maybe some Google Play credits or

Matt Klein: Yeah, sure.

Michal Palczewski: Or something.

Matt Klein: Yeah.

Michal Palczewski: Yeah. And so that Justice League, little promo, no [00:33:00] money changed hands between us and Warner Brothers. We kind of both benefited. And it... it made just as much of an impact as the paid retention.

Matt Klein: I bet. Yeah.

Michal Palczewski: Retention stuff. So, yeah. A lot of retention on Instacart involves, I don't wanna say too much, but we do have a membership program that we... similar to Amazon Prime.

Matt Klein: Right, makes sense.

Michal Palczewski: Called Instacart Plus. And...

Matt Klein: Yeah.

Michal Palczewski: And so if, if we can make you a member, then you'll probably keep using it.

That's the idea.

Matt Klein: Yeah. Cool. And then I guess from, from there, you, you went on to the shopping cart, which I would, I would love to learn more about selfishly.

Michal Palczewski: Yeah. Yeah. So, I work on the Caper carts. And, those are like, if you've ever gone to the store. And you've used self-checkout. Well now imagine that that whole self-checkout register is on the cart itself.

[00:34:00] So you scan an item, put it in the cart, scan an item, put it in the cart, and when you're ready to pay, you press a button. And you pay at the cart.

Matt Klein: I - I just like, I - I hear you even starting to talk about this and I mean, even, I think last weekend I went to Costco and I usually do self checkout. And it's like so infuriating that I am... I bring the cart up, I gotta unload every item and scan them and put them on the weigher thing, you know? And then I have to put them all back

into the cart. It's just like, it could be so much more efficient. So what you're, what you're talking about sounds amazing to me.

Michal Palczewski: Yeah. So if, if, someone at Costco's listening, you guys gotta get Caper carts.

Matt Klein: So, so is it, are you - are you scanning it? Like is there a QR scanner in the cart and you basically scan the item when you put it into the cart?

Is that how it works? Or are your cameras like, how does that, how does that process work?

Michal Palczewski: Yeah, so [00:35:00] I don't want to reveal too much of the

Matt Klein: No, that's fine. I mean, whatever you can share.

I'm, I'm just,

Michal Palczewski: yeah, it's absolutely like if you're, you'll...

Matt Klein: personally curious,

Michal Palczewski: you'll, you go in there, it'll beep when you, when you put... 'cause it'll see the barcode you put it in.

Matt Klein: Got it. Okay. Yeah.

Michal Palczewski: And yeah

Matt Klein: And, and, and then I think another thing, because again, just coming from my personal experience, when I do these self-checkout things, as I understand it, a lot of what they're trying to do is obviously to reduce theft. Me meaning you have to like take the item, you scan it, you put it on the scale, like it knows approximately what the item weighs.

So they're trying to reduce theft that way. And again, whatever you can share. You don't have to reveal any proprietary information. But, I am curious with all of these systems, because I'm sure there's a big anti-theft component as well. So it's like, how do you, I - I guess my question is how do you balance the usability, like versus all of the potential [00:36:00] anti-theft measures?

Michal Palczewski: Yeah, it's, it's actually... that's, that's a huge area because... obviously, and this is something where like obviously we want as many people to use these carts as possible.

Matt Klein: Of course, yeah.

Michal Palczewski: Yeah.

Matt Klein: Right.

Michal Palczewski: And the retailers are very much... they're very sensitive to, to theft in the industry it's called Shrink.

Matt Klein: Shrink. Got it. Okay.

Michal Palczewski: Shrink. Yeah.

Matt Klein: Yeah.

Michal Palczewski: But ultimately, we kinda have two customers, right? We gotta sell the retailers on the cart and then we gotta sell the customers to actually

Matt Klein: Absolutely

Michal Palczewski: use it.

Matt Klein: Yeah.

Michal Palczewski: Yeah. And so, yeah, it's definitely a balance. The way I think of it is... I try not to look for trade offs. Like I, I know as engineers we're always looking for trade offs... faster, but I try to look for synergies.

Matt Klein: Yeah.

Michal Palczewski: And it's like, can we make it better for the customer and for the shopper? Can we reduce, shrink and make it a smoother experience?

Matt Klein: I mean, one, one thing that I've wondered, and you've probably thought about this way more than me, is that [00:37:00] for these systems, again, coming back to my Costco example,

Michal Palczewski: yeah.

Matt Klein: Why don't I just roll the entire cart onto a scale, right? And then they weigh the whole thing, you know, and then at least I can, I can, I - I'm just trying to limit the amount of times, you know, that I have to move things back and forth.

So, like, I can imagine with a, with a system like yours is that if you're, you know, in some way, however you do it, if you're able to scan all the items when they come into the cart, and then I just weigh the whole cart. I mean, it seems like, you know, it just, it seems like there might be ways, you know, that you could

balance both the usability of this system, but also make it so that you kind of know if it doesn't weigh the right amount.

Michal Palczewski: Yeah. Yeah. 'Cause, the cart itself weighs the items.

Matt Klein: Oh, interesting. Okay, cool. Yeah.

Michal Palczewski: Yeah. That's public, so I think I can... yeah,

Matt Klein: Yeah.

Michal Palczewski: Yeah. And so you can imagine that you know how much each [00:38:00] item is supposed to weigh.

Matt Klein: Interesting.

Michal Palczewski: Yeah.

Matt Klein: Got it. So from, from the cart perspective. Again, whatever you're willing to share. I'm assuming you run some embedded software on that device. Is that embedded Android or is that your own custom thing? Like what, what type of software actually runs on the cart?

Michal Palczewski: Yeah, yeah. We're, we're running Android on, on the device.

Okay. So it's very, it's very familiar.

Matt Klein: Yeah.

Michal Palczewski: Yeah,

Matt Klein: And - and then, I mean, I would imagine too, again, just because again, there's a lot of moving pieces here in terms of how that cart interacts, I'm assuming with the retailer systems and all of that stuff. So kind of like your first example of the hands free, you know, there's a lot of moving parts here, so I would imagine that debugging these systems in the field can be difficult.

Michal Palczewski: Yeah, it's, it's my... biggest, biggest challenge, right now. 'cause, there is, there are some advantages because we own the carts. And so we have certain access to them that you wouldn't [00:39:00] have, if it was just, somebody's personal device or somebody else's device. So, that is, that is fortunate for us.

But...

Matt Klein: Meaning, you can push updates roughly whenever you want, or at night or something along those lines? You know, I...

Michal Palczewski: Some- something along those

lines.

Matt Klein: Yeah.

Yeah. Right. Got it.

Michal Palczewski: Like we, like the retailers wouldn't like it if we just started changing everything.

Matt Klein: Yeah, of course, right?

Michal Palczewski: But not just updates, but also what kind of data you can get on the cart.

It's is a little bit less limited if you, if you own the cart yourself.

Matt Klein: Sure, of course.

Yeah. Yeah. But then you, you must also... because those systems have to interface with the - with the retailer's payment systems. Of course. So is that, is that all like a backend connection or kind of like the hands free case, you know, do, do you offer like an SDK that somehow gets plugged into retailer systems in some way?

Michal Palczewski: Yeah. [00:40:00] So the cart itself, is not directly talking to the systems. Instead it's, it's a backend.

Matt Klein: Got it.

Michal Palczewski: It's a backend thing.

Matt Klein: Yep.

Michal Palczewski: Yeah. Which I'm not too involved in.

Matt Klein: No, that's fine. But I mean,

but the cart obviously still has to have connectivity to the backend, but I would also imagine, you know, I mean I - I could be wrong that you would still want people to do payments if there's backend connectivity issues.

So it's like, do you also have to deal with offline cases and all of those edge cases as well?

Michal Palczewski: Yeah. Yeah. So that, that comes up. I know... when we were, when we were at the McDonald's at Google, that came up a lot, because the whole store could be offline.

Matt Klein: Yeah, absolutely.

Michal Palczewski: The user's device could be offline.

Matt Klein: Yeah, absolutely.

Michal Palczewski: And so, we had to deal with that. And the first thing we did is we bought LTE backup for - for the [00:41:00] McDonald's themselves.

Matt Klein: Sure.

Michal Palczewski: So that's kind of like, okay, you're Google, you can throw money at the problem.

Matt Klein: Right, yeah.

Michal Palczewski: And, and make it work. So, so that was one way, another way, we just made- like if you go to McDonald's right now.

Maybe not anymore, but at the time they had AT&T DSL. And, frequently they'd lose internet connectivity.

Matt Klein: Absolutely, yeah.

Michal Palczewski: And they would just optimistically authorize your credit card. They, they'd have like a blacklist of credit cards that they knew they couldn't authorize. Generally they... like if, if your card declines later, you got a free meal.

And so-

Matt Klein: I - I mean, we, we work with some restaurant customers and even in 2025 we're, we're told that connectivity in these stores is not awesome. So, you know, I, I think there's still problems for sure.

Michal Palczewski: Yeah. And so, the way we solved this problem was basically, we added [00:42:00] a encrypted profile of every hands-free user, compressed and small, to, to the device that was at the store.

Our plan for scaling this, if we, if we did get a lot of users, was basically, well, we can predict who's in what area of the country.

Matt Klein: Right. And then send a - send a subsection of the population out to have it ready. Yeah.

Makes sense.

Michal Palczewski: Exactly. Yeah. So. And so then... but then if the box got compromised, you didn't watch all your user profiles leaking, so then we had to encrypt them.

Matt Klein: Right.

Michal Palczewski: And then decryption key was... half of it was on the box and half of it was on the user's phone itself.

Matt Klein: Okay.

Michal Palczewski: And so they could communicate offline, decrypt the key.

Matt Klein: Yeah.

Michal Palczewski: That way. And, and then, and then you got the full facial template and it, it just, the whole thing worked offline.

Matt Klein: It's - it's just so interesting to me because there, as you said before, we're talking about it throughout this entire conversation. These systems have so many edge cases, so many failure points. I mean, to me, that's what makes it so challenging. So I, I guess [00:43:00] I could, I could talk about this for an hour and unfortunately we're coming up on our time.

I - I guess I'd just love to learn. Last thing on the cart, you know, what, what's the biggest technical challenge that you're facing right now? Trying, trying to make it all work well?

Michal Palczewski: Well, it's, it... the thing does a lot. It's basically... it's, it's doing, you know, I - I said it's doing weighing, it's do-, it's figuring out what you're putting in there

Matt Klein: Right.

Michal Palczewski: So it's doing a lot of real time signal processing. And, for an Android app... the, the hardware can't always handle it.

Matt Klein: Right. Okay.

Michal Palczewski: And, and so the, that's the biggest challenge is, is making it all work very efficiently. And then managing, you know, you have a, we're very separated team...

and, and so getting everybody to work together and like, 'hey, you guys can't use all the resources' like...

Matt Klein: Yeah.

Are you,

[00:44:00] are you owning the entire stack, meaning like all the way down to the embedded hardware and the Android OS and all, all of the apps and just- the reason that I'm asking that is that I can imagine that for some of these systems, you know, you probably have a system on a chip and you have some things that are doing different things and you're trying to make them coordinate.

You know, as you said, that's, that's, that's a lot of hardware and some of it is real time components and I, I don't know, it's like the space fascinates me.

Michal Palczewski: Yeah. So, like me personally, I own the... my scope keeps increasing, but right now it's - it's the Android apps, that are, that are on the cart. But there's, there's other systems.

Matt Klein: Absolutely.

Michal Palczewski: That's... the, the low level kernel stuff. Like, I'm like, I haven't done that in a while. Like, let's...

Matt Klein: Yeah.

Michal Palczewski: Yeah.

Matt Klein: Yeah. Cool. Well, as I said, I, I could, I could talk about this forever. I think this is a really interesting topic, where I'd love to close [00:45:00] out,

is, you know, you've obviously been doing mobile development for a long time, you know. You've worked across iOS and Android. I'd love to just, you know, in our last few minutes, I'd love to pick your brain more generally around. What do you think are the biggest challenges that mobile developers are facing right now?

What, what are the biggest challenges that you face on your teams in terms of technical leadership, in terms of productivity? Would love to learn a little more about where you think the industry is going. So, whatever you're willing to share there would be fantastic.

Michal Palczewski: Well, I think the world has changed so much in the last year with

AI.

Like...

Matt Klein: Yeah.

Michal Palczewski: There's not anybody I know that's not talking about AI

Matt Klein: For sure.

Michal Palczewski: And how their, their productivity has changed. I was able to work on a personal project and there was no way I would've gotten it done so quickly. Like I had AI write me a Flutter app and a backend, push it to GCS, Google Cloud Services, and - and like, I'm kind of overseeing like, okay, yeah, [00:46:00] that, that looks right.

No, no, no. Like that's, that's a rookie mistake, AI, fix it. And, and getting the whole thing up, but... It just really, you know, I think like earlier I talked about writing a hundred thousand lines of code a year, and now like... anybody can do it. So the productivity seems to be increasing at a very rapid pace on like how much you can get done.

You're not spending a lot of time on unit tests, but then people are coming to like new paradigms about it. So just last week I was talking to an engineer and he's like, if - if I can't get AI to write the code that I wanna write, I start rethinking the problem space.

Matt Klein: Hmm, interesting.

Michal Palczewski: Yeah. It's, it's like he's using it as a code smell.

Like it must be too complicated, what I'm doing, I need to simplify. But it, it got me like thinking of like, what are the implications of that? In the long run I'm like, well, if that's your attitude, you might avoid a certain class of [00:47:00] problems that might be interesting, but you're probably gonna get so much more done than - than a lot of other people. So like...

Matt Klein: yeah. It's a - it's a super interesting space because I think for me, I use these tools a lot.

Michal Palczewski: Yeah.

Matt Klein: I've had similar experiences to you where, you know, especially for Greenfield projects, the amount of stuff that you can get done is absolutely mind blowing.

My personal success, I would say, has been less on bigger existing, like complicated code bases, you know, where you're trying to make targeted changes and have them make sense in the whole context of the system. But I have to believe that the tools are only going to continue to improve for sure.

Michal Palczewski: Yeah.

I find it's not too bad. Like I use Cursor and I use Max Mode and get it all the context window that it can get and... yeah, sometimes, sometimes it goes off on its own. It's like, don't - don't run Gradle. It's expensive. [00:48:00] I'm like, wait till I okay it. And I'll just be like, okay, run Gradle... like...

Matt Klein: Right.

Michal Palczewski: Yeah. But... so it sometimes has a hard time like doing what it's told, and if you give it too much info, it'll kind of go off in its own direction.

Matt Klein: Right.

Michal Palczewski: And it's prone to cargo cult programming.

Matt Klein: Yes,

I agree. Yeah.

Michal Palczewski: ' This is how it's always done,' but, but I, I do think that so many of these kind of rote tasks are automated. And, if you can give it a good plan, it can execute on it. And some days my biggest problem is I'll - I'll have been working all day and I'm- will be pretty confident that I did a bunch of cool shit, but then I did so much that I can't remember everything I did that day.

I'm like, wait, what? How many PRs did I have? What - what was that one about? Did I do that?

Matt Klein: And you think that's because of the tools?

Michal Palczewski: Yeah, absolutely. Well, 'cause you're not coding it yourself, you're not in the weeds as much anymore [00:49:00] and, and because you're doing... you're getting so much different things done that it's just hard to keep track of...

Matt Klein: Right.

Michal Palczewski: Of everything. Yeah. And then it, I feel like if you're- you still have to code... and be a little less efficient otherwise, I kind of find myself drifting off into a lazy land of my brain just doesn't wanna work anymore. It doesn't- it starts forgetting how to do things and you're like, no, no, no.

Gotta gotta keep that saw, fresh too.

Matt Klein: Yeah.

Do you - do you find that there's consistency in how your teams, are using these tools? Like, I guess one, one question that I have, is I feel like there's a learning curve in terms of how to use the tools well. And I guess one, you know, one general question is like, how, how do you find that people are... like, are they all teaching themselves? Or are - are you, like... are people converging on using the tools and the prompts in a certain way?

Like how, how has that been going for you?

Michal Palczewski: You know, there's kind of... [00:50:00] it's kind of two things is one, at one point in time I thought like, you know, the most productive use of my time is to write the rules for the AI that are set for the project, so everybody's kind of using the same rules and keep them updated.

And then I - I realized that it's a losing battle because the more rules that you have, the less of them get followed.

Matt Klein: Interesting.

Michal Palczewski: The more context you have.

Matt Klein: Right.

Michal Palczewski: And - and the less effective they are. And so it's like, okay, let's just have some base ones. But as far as like everybody... the, the way I'm thinking of it right now is I could help- I could show someone what I'm doing today, right?

And it might be useful for like a month. And then next month there's like a better way and stuff is moving so quickly. But I think that, where I see this heading is a year from now that proficiency with AI tools... that's gonna just be a job requirement. Like we don't... like when I got to Google, they didn't [00:51:00] teach me how to program an Android.

Matt Klein: Sure.

Michal Palczewski: When I gotta Instacart, they didn't teach me how to program on Android or iOS. That was just a core competency that I needed to have or figure it out on my own. And, and so I like - I think that's where it's headed. Being proficient with AI is gonna be a core job skill that you just can't work without.

Matt Klein: It's, it is amazing to me how fast some of these things are moving. And also, at least just from my own experience, I get different levels of value from them. Like it might be the code that I write. I feel like for some of the more intricate code that the AI tends to go off the rails more. But for example, as you said, when I'm working on things that are more on the beaten path, whether it be SQL queries or those kinds of things, I mean the - the productivity enhancements are - are very

incredible, for sure.

Michal Palczewski: Yeah. Like, I'm not... I don't know anybody that's written a regular expression in, in a

long time.

Matt Klein: Oh, absolutely. Yeah. Yeah.

Cool.

Michal Palczewski: But

I was gonna say, if you have

[00:52:00] like an ANR to debug, then... and it's not an obvious main thread blocking thing, but something else has caused ANR like, good luck.

Matt Klein: Yeah.

Michal Palczewski: Like you're, yeah...

Matt Klein: of course. Cool. Well, you know, again, I - I would love to keep talking, but we're we're almost at time. Any closing thoughts that you would like to share?

Michal Palczewski: No, not really.

Matt Klein: Alright, well, thank you so much for joining... learned a lot myself. Very interested to see where all of this stuff goes with the cart that you're working on,

and I'm hoping that you can sell it to Costco. So if anyone is listening from Costco, please, please talk to these folks because the self-checkout is very annoying.

So, thank you, Michal. All right folks.

Michal Palczewski: Thanks.

Matt Klein: That's a wrap for this episode of Beyond the Noise: Signals, stories, and Spicy Takes. Huge thanks to Michal for joining and sharing his story, and you can find this episode and all past ones on the bitdrift YouTube channel.

If you [00:53:00] had fun, drop us a review, tell your friends or yell your favorite hot take into the void, and just make sure to tag us. I'm Matt Klein and see you next time.