I'm Alex Chen, a staff software engineer in San Francisco. I build and scale distributed backends, obsess over latency tails, and write essays to make hard infrastructure ideas approachable.
Long-form notes on distributed systems, performance, and the occasional war story from production.
After three years of running gRPC in production across dozens of services, I've collected a set of patterns that consistently keep things healthy under load. In this post I walk through deadline propagation, retry budgets, and circuit breaking — and explain why naive retries are often the fastest path to a cascading outage. I also share the interceptor stack we standardized on for observability.
Read moreWebAssembly promises near-native speed, but the gap between the benchmark and your actual workload can be surprising. I profiled a real image-processing pipeline compiled from Rust and measured where the time actually goes: linear memory copies at the JS boundary, SIMD that wasn't being emitted, and a few GC interactions I didn't expect. Here's how I closed most of the gap, with flame graphs to back it up.
Read moreWe moved a fleet of forty services off a hand-rolled deployment system and onto Kubernetes over six months. It was not the clean win the conference talks promise. This is an honest retrospective: what genuinely improved (rollouts, autoscaling, self-healing), what got harder (debugging networking, cost visibility), and the organizational habits we had to build before the platform paid for itself.
Read moreNetworks fail in the middle of requests, clients retry, and suddenly a customer is charged twice. Idempotency keys are the standard fix, but the details matter enormously. I break down how to scope keys, where to store them, how long to keep them, and how to handle the awkward case where the original request is still in flight when the retry arrives.
Read moreOpen-source tools and side projects I maintain in my spare time.
I'm a staff software engineer based in San Francisco, where I've spent the last nine years building the kind of backend infrastructure that's supposed to be invisible when it works. Most recently I lead a platform team responsible for the service mesh and deployment tooling used by every engineer at a mid-sized fintech, which means I think a lot about reliability, developer experience, and the quiet trade-offs between them.
My background is in distributed systems and performance engineering. I got my start writing latency-sensitive trading systems in C++, moved into large-scale Go services, and these days I do most of my greenfield work in Rust. I care about systems that fail gracefully, codebases that newcomers can navigate, and observability that actually answers the question you're asking at 3 a.m.
Outside of work, I write essays to force myself to understand things deeply — if I can't explain a concept clearly, I usually don't understand it as well as I thought. When I'm not at a keyboard you'll find me bouldering in the East Bay, over-engineering my espresso setup, or mentoring early-career engineers through a local non-profit.
I'm open to consulting, advisory work, and conference talks. The fastest way to reach me is email — I read everything, even if it takes a few days to reply.
hello@alexchen.dev