How Square debugs a global platform with bitdrift

The challenge: Balancing visibility with cost in a complex production environment

Square’s Principal Android Engineer Pierre-Yves Ricau (known as P-Y) specializes in deep technical investigations into complicated production issues that span Android devices, iOS devices, and Square’s own custom Android-based hardware. That work depends on being able to see what actually happened on a device before, during, and after potential issues.

Square historically relied on another major service provider for observability, which takes a send-everything approach to telemetry. But that model didn’t give P-Y the selective, on-demand log control he needed. It also meant the team was paying to collect far more data than any single investigation actually required.

“Mobile is a constant struggle,” he explained. “I need more logs, but I need to send less logs because it costs me money. It’s bad for the customer and bad for bandwidth. Those are two fighting constraints.”

Solution: Log everything on-device, send only what you need

P-Y was considering building a custom solution when he encountered bitdrift. Its selective, queryable approach to device logs piqued his interest. A common way teams keep observability costs under control is to either not log certain things at all, or to log a sampled fraction of everything. Both approaches come with tradeoffs:

1. If you skip the log entirely, it’s gone for good until the next release. If you want to add it back in, you need to ship a new build with the added logging and wait for the issue to happen again.
2. If you sample, you get a biased, incomplete picture. Your aggregate metrics skew toward whatever the sample happened to catch, and the one session you actually care about (when a customer hits a bug) might not be in the sample at all.

“I wasn’t satisfied with the traditional observability model of ‘send-everything and we’ll sort it out later,’” P-Y said.

He kicked off a trial with bitdrift, which demonstrated its unique ability to provide deep insight into the mobile experience beyond crashes without the cost overhead of other approaches.

“With bitdrift, you just keep logs on the device. You don’t send them until you need them.”

Square can retain rich, detailed sessions in a ring buffer on-device and pull them on demand, without paying to stream everything, everywhere, all the time. bitdrift also proved its stability to operate at scale during the evaluation, which was one of the key criteria for Square.

“The session timeline is honestly the killer feature of bitdrift,” added P-Y. “That was my a-ha moment, and it’s definitely the biggest value we get from bitdrift to date. I can get everything when I need it without having to pay the cost when I don’t need it. It’s a big win.”

Plugging into Square’s observability environment

Today, Square relies on bitdrift for full-resolution mobile observability and real user monitoring (RUM), which integrates well within Square’s larger observability ecosystem.

“bitdrift filled a gap no other tool was addressing for Square’s mobile stack,” said P-Y. “bitdrift sessions are a lot richer and more interesting to use.”

The benefits: precise root cause analysis and a deep understanding of the user experience

Since deploying bitdrift, Square finds the root cause of issues that were hard to identify before. “Being able to see how memory grows over time as you go through logs has been really helpful to understand what users are doing,” said P-Y.

In one instance, the Square team struggled to diagnose an out-of-memory (OOM) crash using breadcrumbs alone. Working from a Bugsnag crash report, P-Y used a direct link into the corresponding bitdrift session and traced the failure to a large file upload that triggered the creation of an oversized request object; a connection made visible through bitdrift’s heap-tracking logs. Without that session-level detail, the root cause would have stayed hidden behind a generic memory error.

“Bugsnag was reporting the heap size at crash time, and was showing a heap size at baseline size, not full at all, despite the OOM error message reporting much higher heap size,” explained P-Y. “The AI agent went down the wrong path looking at various OOM stacktraces. bitdrift helped us figure out exactly what was going on because it had this tracking of heap over time associated with logs; that helped invalidate the incorrect agent theory, correlate the network call with what was happening, and surface a seemingly unrelated log that was letting us know about an unexpectedly large file (which contained the contents we were trying to upload). What surfaced was a long-running problem.”

He added, “This error is tricky and really interesting because it’s an out of memory error. You’re trying to load a large list of transactions from disk into memory to then send it over the network. If at some point that fails, the work is interrupted… I don’t think we could have figured it out until we saw how memory was rising over time with what type of work, based on the logs and how it was going down right at the crash time. bitdrift was an unblock here, and it’s directly tied to the customer experience too.”

Looking ahead: Bringing Engineering and Customer Support closer together

P-Y’s next goal is to close the gap between customer support tickets and the technical data engineers need to act on them. Because customer complaints often arrive without enough diagnostic detail to be useful on their own, he wants to build a pipeline where customer-reported issues automatically trigger a bitdrift session capture, paired with bitdrift AI-driven analysis of the resulting logs.

This would allow support agents to get immediate, context-aware information and close issues faster, versus having to manually translate a vague complaint into a technical investigation.

Conclusion

With bitdrift, Square’s mobile team is better equipped to investigate issues that never would have surfaced through crash reports or breadcrumbs alone. It’s also opened a path toward faster, more automated triage. The team is ultimately able to deliver a better customer experience as a result.

Square is a bitdrift customer and participated in this case study as part of its customer agreement with bitdrift.