Building software on rnewman

Removing downloaded macOS text to speech voices

Sat, 14 Mar 2026 00:00:00 +0000

I recently played around with using macos-speech-server for STT/TTS in Home Assistant. When experimenting with avspeech, the macOS-native speech engine, I discovered that macOS 26 no longer allows you to delete voice packs after installing them — the management UI is gone, and the advice online about swiping in the list doesn’t work.

At 100–500MB per voice, these add up.

Unfortunately, deleting the files from /System/Library/AssetsV2/ is blocked by system integrity protection: even sudo can’t do it.

Fortunately these are “mobile assets”, so there’s a macOS framework to manage them. Claude and I pulled together a Swift script that uses that framework to list the voice assets and let you delete the ones you no longer want. I put it on GitHub.

The incident spiral

Tue, 13 Jan 2026 00:00:00 +0000

I’ve watched this pattern unfold more than twice, so it’s time to write it down.

A team has a quality problem. Reliance on unreliable partners, technical debt after a couple of years of fast growth, accrued complexity — whatever the causes, we’ve become aware that things aren’t just humming along.
The team starts taking incidents more seriously. A new leader joins, or senior leadership puts the team under a microscope. The intent is good.
Incident retrospectives become more thorough; these take time. Each retro produces more action items, which take time to address. Some action items improve detection, so on-call engineers get paged more, and we open more incidents to track. The team is now spending 20–40% of its engineering bandwidth on incidents, retros, and action items.
Meanwhile, the team is expected to keep shipping features and absorbing company-wide changes. Those changes themselves increase the surface area for things to go wrong. Running at 60% capacity means corners are cut on feature work and people are tired. More incidents.

This is a bad situation: the more the team tries to meet expectations, by shipping features and taking incidents seriously, the worse things get.

Action items are a trap

In most companies, incident action items bypass normal prioritization. A P1 comes with a one-week deadline and supersedes other work for whichever engineer is assigned the ticket. This is by design, because, without that deadline and that accountability, the backlog of action items will keep getting punted until after the next sprint, when the cold light of day shines on the list of things you told your product manager you’d get done. You sat in a meeting and said these things were necessary to prevent recurrence, accelerate detection, help with mitigation, or reduce blast radius, right?

But consider the tradeoff that occurs in practice. Teams are always balancing feature and reliability work, and some of that planned reliability work might matter more than any incident action item.

A P1 action item for a minor incident might preempt a planned project to deprecate a problematic system or remove a single point of failure.

The incident action item addresses a failure mode we’ve just seen. The planned work might avoid a catastrophic failure that we haven’t yet seen. We prioritize the former because it’s vivid. We’ve been lucky on the latter — so far.

Lorin Hochstein captures this with a thought experiment he calls the Oracle of Delphi. Imagine an oracle tells you that if you do an incident’s follow-up work, you’ll avoid a recurrence… but you’ll suffer a novel eight-hour outage next month. If you do the reliability work that was already on your backlog, you’ll have another minor incident like the one you just had, but avoid the big one. Which do you choose?

Creating a P1 or P2 incident action item is a statement that this is the most important thing you can work on this week, based on an implicit assumption that the last incident is a strong predictor of future reliability issues. That assumption is often wrong. You were surprised before. You’ll be surprised again.

The complexity that was supposed to help

There’s a related problem with the kind of action items that retros produce. The natural response to an incident is to add something: a check, a cache, a retry, a fallback, a circuit breaker. Each is reasonable in isolation, and they’re easy for folks on the periphery of a system to imagine and propose as action items. Over time, they accumulate.

Hochstein has a conjecture about this: once a system reaches a certain level of reliability, most major incidents will involve either a manual intervention intended to mitigate a minor incident, or unexpected behavior of a subsystem whose primary purpose was to improve reliability.

This isn’t theoretical. The October 2025 AWS outage involved an unanticipated interaction between multiple reliability mechanisms: redundant enactor instances, a locking mechanism, a cleanup mechanism, a transactional mechanism, and a rollback mechanism. All sensible design decisions, but the incident emerged from their interaction. The December 2025 Cloudflare outage was triggered by a killswitch — a mechanism specifically designed to quickly disable misbehaving rules — that had worked well in the past but failed in a corner case.

Caches, retries, and bimodal fallback paths are all common contributors to incidents. I see them proposed as action items every week.

This doesn’t mean reliability mechanisms are bad. We need retries, timeouts, bulkheading, failovers, rate limiting, caches, circuit breakers, and all the rest. Complexity is inevitable, and indeed we must learn to ‘surf’ it, as Hochstein puts it. However, we must also continually work to simplify to make room for the new complexity we’re adding.

Breaking out

The urge to take incidents seriously is correct. The problem is that when every incident generates two weeks of follow-up work, the team can’t keep up. Prevention competes with response, and both compete with the roadmap.

Here are some approaches I’ve found helpful to reduce the burden and increase agency, starting with the two I think matter most.

Predictable external failures shouldn’t be incidents. One airline being unreachable isn’t an incident for a travel booking site. One restaurant being unexpectedly closed isn’t an incident for Uber Eats. Some failure modes exist by design — they will happen, and the system should handle them gracefully, informing the user as appropriate. (To do this right you might need to monitor external dependencies with synthetics, rather than just watching error rates.)

This doesn’t mean ignoring these failures. Track them. Build dashboards. Set up weekly reviews to spot trends. If a partner’s error rate doubles, that’s a conversation to have with the partner, or a reason to reconsider the integration. But don’t page someone at 2am.

You can achieve similar benefits by weakening guarantees: for example, retries, job queues, dead-letter queues, and so on are all mechanisms to turn failures into latency. If your users are OK with latency, you can use those mechanisms to make your system more resilient.

You might make a different decision here if you have a lot of influence or control over the partner or destination, if the partner is fungible, if there’s something you can do to unblock your own customer in extremis, or if you want to actively track these failures. However, I think this point will be immediately recognizable to teams trapped in a particular bad state.

Be more discriminating about action items. Action items should fill urgent gaps in detection or ability to mitigate, fix obvious bugs, and prevent cascading failures. Not every incident needs a code change. Good action items should address a class of incidents, not just the specific failure you observed.

Almost everything else you can think of, particularly migrations and rewrites, should be added to the backlog to be weighed against the other reliability work you were already planning.

I’m fond of asking in retros the questions I learned at Amazon. They’re simple, but they nudge you into taking different perspectives and generalizing to a class of failures: what would have halved your time to detection? What would have halved your time to recover? What would have halved the blast radius?

It’s also useful to ask what would halve the cost of this class of incident, because achieving that can help us break out of the spiral. Sometimes the answer is “nothing practical, and we accept this will happen occasionally” — that might be better than doing a bunch of work just to feel like you’re responding to the incident. If you’re never declining action items, you’re not being selective enough.

After an incident, look for complexity to remove, not just safeguards to add. We should be skeptical of action items that add new things that can interact and fail. After an incident, ask: what could we remove to make this simpler? Could we eliminate the dependency that failed, rather than wrap it in more error handling? Can we make two things share fate? Could we drop a feature that isn’t worth its operational cost?

Asking simplifying questions all the time is something I expect of staff engineers.

Be honest about staffing. A team spending 30% of its time on incidents might be understaffed for its scope. Sometimes the answer is “we can’t operate this much surface area with this many people.” That’s uncomfortable to say, but the alternative — denial, leading to degraded quality, burnout, and attrition — is worse.

Make the cost visible with an incident budget. Allocate a fixed percentage of engineering time — say, 15% — to incident work. When that’s exhausted, remaining action items compete with features through normal prioritization. This makes the cost visible, forces explicit tradeoffs, and creates pressure to make incidents cheaper.

None of these changes will fix everything overnight, but they might break the feedback loop.

Engineering needs to be sustainable. A team drowning in incidents can’t think strategically or make deliberate choices. The first step is to reclaim the ability to plan, then use that ability to chart a course towards holistic reliability.

An incident lens

Sun, 25 Aug 2024 00:00:00 +0000

I first started owning the operational posture of modern (SaaS, microservices, cloud, etc.) services in 2018, when I joined AWS.

With service ownership came incidents (and CoEs!), and the ensuing six years built my awareness of availability and formed the lenses I use in incident retrospectives. Those lenses now shape the advice I give.

Last week’s example: a senior engineer wanted to improve the deployment processes used by a nascent service. It is today manually deployed to a shared testing environment and a production environment, usually in order, once per week.

The first thought of the engineer and his manager was that we should build a ‘stable’ pre-production environment — let’s call it pilot for the sake of discussion — avoiding contention in the testing environment. Changes would bake there for a while before flowing to prod. Various additional suggestions, most of them good, built on that: make deployments include only a single change, so we know what broke; build a pipeline that a person would promote instead of running Terraform, and so on.

These are all worth doing! But from an incident perspective, we tend to ask some very simple questions:

How did you know something went wrong?
How long did it take you to find out?
How big was the impact?
How long did it take you to recover?

In a retrospective, we’ll rotate these questions a bit:

How would you have halved the time to detection?
How would you have halved the time to recovery?
How would you have halved the blast radius?

Thinking in this mindset led to my advice: there is little point in building a pilot environment or a release pipeline unless you can (a) reliably roll back deployments, (b) detect bustage in prod quickly enough to automatically decide to roll back, and (c) roll back quickly enough, and/or deploy to small enough populations, to minimize how many users are affected, depending on the maturity level and SLA of your service.

We have no automatic rollback other than ECS blue-green deploys, and while we have alarms on service metrics, many endpoints take little to no traffic. (I’ve seen this situation in a few places: uncommon endpoints take no traffic, so they can easily regress without being detected in any reasonable rollback window.)

Introducing a pilot environment would only give us a false sense of security, or even make things worse by improving developer productivity: we’d deploy broken changes, bake them without detecting any problems, then roll out the broken change to 100% of users in prod. Maybe we’d even watch the dashboards, and not see any problem for a week or two!

The need to detect bustage before it affects lots of users, and roll back quickly, dictates the roadmap I suggested:

Build synthetic tests (“canaries”) to ensure a volume of traffic that mimics real usage on every API (“which API endpoints are you willing to break without noticing?”).
Build detectors on those canaries and service metrics to fire alarms. Any bustage should be detectable. Now your time-to-detection is low.
Build a pipeline that rolls back if those alarms fire. Now your time-to-recovery for simple bugs is low.
Then build a pilot stage to serve as a smaller ‘canary’ population. This stage bakes a release before it goes out. Now you have reduced the blast radius of a bad change, introducing a place to run canaries and real internal users to generate novel traffic.

Other concerns, like smaller rapid deployments, are secondary to these. They might be important — perhaps developer velocity is hampered by the shared testing environment — but I’ve found that bad deployments are themselves a large source of friction, as well as being bad for the customer.

Tenets

Fri, 21 Oct 2022 00:00:00 +0000

I learned a lot during my time at AWS. One thing that I haven’t seen discussed much is how documented tenets can be used to structure a team’s path.

Tenets in this context are documented principles and beliefs that a team holds about itself, its users, and its products. They can be a useful guide for decision-making, and even the process of defining and phrasing the team’s tenets — and the ruthless prioritization needed to hit the expected limit of 5–7 — brings tremendous clarity. As with planning, the value is largely in the process, not the outcome!

Tenets can be set at the team level, the service level, or for smaller efforts and scopes that would benefit from directional clarity.

Writing good tenets is hard, and even tenured leaders often misunderstand the concept, losing some of the benefits.

A bad tenets list reads like a statement of values, a wishlist, simple priorities, or untethered aspirations. Our product will be fast, cheap, and available everywhere.

This blog post publishes some internal Amazon tenets (see the w.amazon.com links!).

A good example is from IoT:

Necessary offline operations: We help customers build systems in the cloud that work in predictable ways when connectivity is limited.

The point of writing up tenets is not to capture something that’s unambiguous or uncontested (products should be fast, the team should have a high bar for quality), nor to capture something that’s easily defined in some process doc or as a metric and thus does not require much human judgment. It’s to define a mental model for coping with conflicting priorities, to define a position in an exclusive value space, or to provide a mast to which a team under fire can tie themselves.

I’m sure it would be very convenient to tie some IoT capabilities to a network connection. Making a system work reliably with limited connectivity is hard and must be designed in from the start; it’s something that customers need help with; and it’s a requirement that engineering teams would love to drop! This tenet encodes an intention that steers decisions towards the inconvenient choice that’s right for customers.

Tenets like this are a statement in advance, while we still have the space and presence of mind to use good judgment, that we won’t do the expedient thing. It is a form of precommitment — the corporate equivalent of setting out your workout clothes the night before, because you know you won’t feel like exercising in the morning.

Accessibility is another good example. It’s important, it has top-down guidance in many product orgs, and yet teams will routinely sacrifice accessibility (and localization!) in order to hit a deadline, saying “yeah, well, accessibility is important, but of course we have to weigh it against shipping features…”.

A weak tenet like accessibility is important is strengthened by making it definite — our releases are always accessible to users of assistive technologies — or make the tradeoffs explicit: we prioritize accessibility over time to market, and will delay a release that doesn’t serve all of our customers.

A good tenet can capture discomfort and interaction patterns, too: we avoid excluding our remote teammates by making decisions asynchronously and with time for thoughtful comments, rather than in synchronous meetings. Again, there’s a tradeoff captured: that the team will go slower in order to be inclusive and make more measured decisions.

Other tenets can express an opinionated stance about product-market fit: our users are sophisticated and prefer detailed, accurate docs rather than vague marketing materials, or we accommodate our industry’s long sales cycles by treating trial users with the same priority as sold users.

At a leadership offsite my team tried an exercise: gather in small groups and write what you think the whole org’s tenets should be, then come together and discuss. The outcome was educational, if not surprising: many newer leaders wrote bad tenets; many wrote tenets that focused on the importance of their teams’ efforts, rather than the breadth of the business and our coordinated long-term success; and few wrote tenets that surfaced the essential tensions between their own priorities and those of other managers. This exercise is time well spent if you can spare an hour, but be prepared to work through the disagreements about what really matters for your teams!

Writing for an audience

Fri, 21 Oct 2022 00:00:00 +0000

Many Stripes are in the habit of writing “5:15s”: short updates that take fifteen minutes to write and five to read. My own staff notes are infrequent, on the order of every month or two.

In part that is because I find it hard to find the time to digest my week into a useful narrative. If you told 25-year-old me how many hours of meetings I would have each week, and how much time I spend on Slack, he wouldn’t have believed you, and would have gone back to writing code. (First he would have asked you what Slack was!) The raw dump would be overwhelming and unnavigable.

In part it’s because I spend a lot of time constructing custom-fit narratives for individuals or small groups, trying to serve their needs by shaping my experiences and perspectives into a parable and an account that succinctly gives them what they need for the situation in which they find themselves. A mentee recently apologized for presenting so many questions and problems for me to respond to, and I laughed — this is the most fun part of my job!

Mostly, though, it’s because I am acutely aware that a broadcast narrative — whether that’s an email, a project review update, a 5:15, or a conference talk — is a tool for changing an audience’s perspective, and getting that right requires both understanding the audience and providing (or assuming) enough context that the message can be practically brief. Often the context cannot feasibly be communicated, either because it is too early and unstable, or because it is so impractically large that the audience will extract no net value from what I have to say.

That leaves me where I am today: with occasional abstract updates.

Those of you who succeed at broader notes: I would love to hear your perspectives and ideas. Those who read these abstract notes: I am curious whether you find them valuable, or would prefer something more concrete (or different entirely).

First month at Stripe

Sun, 06 Feb 2022 00:00:00 +0000

I’m approaching the end of my first month at Stripe. I didn’t announce where I was landing in my earlier post out of a sense of counting my chickens, so I’m glad to change that!

It is a strange and amusing shift to go from working at a bookshop with a climate pledge to working at a publisher with a carbon sequestration arm.

After more than three years at AWS I was quite settled on my team, surrounded by people I knew well, and aware of the patterns swirling at various levels. I was also aware of my pace of learning having slowed, and I was tired of managing my cognitive dissonance. While I miss the people, the Principal Engineering community, and some unique aspects of my role, and I’m grateful for how much I had the opportunity to learn, I feel good about my decision to leave.

I also feel good so far about my decision to join Stripe: even as a relatively large startup, there is still a coherence of vision and some agility, and it really feels like we are all pulling in roughly the same direction. The people are kind, and that’s even expressed in the benefits decisions that management makes, like explicitly giving emergency medical and caring leave this year — too many of my friends at Amazon burned through PTO due to surprise COVID or caring responsibilities. No place is perfect, but so far I can live with the imperfections.

My first weeks at Stripe have been a firehose, as one might imagine: simultaneously learning a new domain (finance and payments), how Stripe’s businesses fit in to that, the technologies we use, the work that’s beginning, and the people. Stripe has a very thorough “101” onboarding program, but I quite quickly (too quickly?) ramped up on some real work, and I feel like I accidentally struck a good balance: after 19 working days I already have a web of relationships with my onboarding cohort, my new teams, and other staff engineers in my area, and I’m starting to develop the intuitions and questioning/routing behaviors that staff engineers use to do our jobs.

More importantly, I have managed to strike a good balance of time. Meetings happen 9–3 Pacific to avoid impacting east coasters, and Wednesdays are meeting-free for makers. I stop at 4 or 5, depending on when I started my day. Working at Amazon was always described to me as “intense”. So far Stripe is also intense, but bounded and mindfully so: everyone has given me advice to make sure that I don’t take on too much.

We have ambitious goals for the next twelve months, so it’s a relief to find that people view that as a long hike, not as a deathmarch. Onwards!

I'm back; things changed

Thu, 06 Jan 2022 00:00:00 +0000

As I wrote last year, I didn’t feel like I had the space to write for public consumption while performing my role at AWS, and I would be back when things changed.

They have: I decided to leave AWS at the start of 2022.

As is tradition I then spent a couple of days fiddling with blogging systems, migrating my posts out of Medium and recovering a couple of posts from an even older blog. This is the result.

I don’t expect a particular cadence of writing, and I’m trying to not let the perfect be the enemy of good, but I’m optimistic that I will publish things now that I have an easy place to do so.

More next week.

A long pause

Sat, 25 Sep 2021 00:00:00 +0000

Working at AWS is very different to working at smaller mission-driven companies like Mozilla.

Even in projects where I get to think or work in the open, there is policy around communication that has a chilling effect on writing, as well as cultural pressure to use my time to write more leveraged blog posts to serve the interests of the team. That means I don’t write here.

This culture is in stark contrast to my pieces on Medium from 2018, my last year at Mozilla, in which I wrote to explain — and wrote to think through! — two problems with which I have grappled for long stretches of my career: how different groups of people (individuals, teams, or even communities) can reuse information that overlaps in space and vocabulary; and how different devices can change shared data over time and space without the need for heavyweight central coordination.

The AWS blog certainly attracts more readers than this little Medium, but it doesn’t serve those purposes of explaining and thinking through problems.

In occasional bursts of free time I return to the topic of structured storage, whether by writing index chunk join code to understand an algorithm, or more recently by digging into an identity challenge that is intrinsically decentralized, only to find JSON-LD (RDF in disguise!) hiding under the covers. Writing prose for public consumption, however, has fallen by the wayside.

I’ll be back when things change.