On Safety Critical Software

When you read “safety critical software” your mind probably went to something like medical device software (perhaps the Therac-25), or maybe avionics software, or perhaps industrial control systems that run nuclear power plants. It’s pretty unlikely that you thought of the operating system kernel on a cell phone or a web browser. Congratulations, you have a solid command of the threat model for the overwhelming majority of people.

For nearly everyone, even a complete and total failure of their phone or web browser’s security would not be catastrophic. It would be potentially embarrassing. And probably financially damaging. And definitely extremely inconvenient. But not a matter of national security or life or death.

However, for a small group of people, it is. They’re nutritionists, they’re senior government officials, they’re human rights advocates, they’re Tibetans, and they’re Uyghurs. And for them their web browsers and OS kernels are bulwarks against oppressive forces. This may sound hyperbolic, but every one of those links contains a citation. And we know these tools aren’t living up to these users' hopes for their protection.

Any time you have two sets of users with radically different needs, you have a challenge for a product team; this is no different in security. On one side we have nearly everyone whose security concerns extend as far as needing a defense against high-volume bulk attacks. And on the other we have a small minority whose security needs require protection against motivated, and specifically targeted, attacks by nation states.

When you’re confronted by a product challenge like this, you always have three choices:

  1. You can decline to serve the minority use case
  2. You can find some way to serve both use cases with the same product
  3. You can build a second, more specialized, product for the second use case

The authors of software like web browsers, OS kernels, and mobile messengers need to answer this choice. Does their software endeavor to protect against these sorts of attacks, or do people at risk need to look elsewhere?

Let’s look at how Google handled this challenge for their server side products. For as long as I can remember, users have faced bulk phishing and other account takeover risks. However, in recent years there has been an uptake in more targeted account takeover efforts. None more famous than John Podesta in 2016. In 2017 Google launched their Advanced Protection Program. This is an opt-in program, designed to provide increased levels of security to people at the most risk. It does this by enforcing the use of 2FA with security keys, by blocking privileged OAuth grants (e.g. read/send email), and by disabling many password-reset/account-recovery routes.

One thing that is immediately obvious is that these have a tremendous impact on user experience. If Google enabled this for all of their users, they’d have an awful lot of annoyed people on their hands who don’t need this level of protection. But it’s available, and free, for those who do need it.

There is no equivalent program for iPhone or Android users. And this probably makes sense — increased protection against 0-day attacks against browsers and OS kernels shouldn’t require user experience changes (except, perhaps, performance), so any improvements here could be available to everyone. And that leaves us to ask: Do Google, Apple, Microsoft, and others intend to serve this use case?

It’s relatively common for software’s EULA to say the manufacturer isn’t liable if you use it to operate a nuclear power plant. There’s nothing inherently wrong with choosing a set of users to build for, and ignoring other concerns.

Making software secure enough to resist motivated attackers will not come for free, whether your bet is on memory safe languages, CPU architecture improvements, or something else. Achieving it will require an investment of resources, and possibly decelerating new feature development. This is a complex trade-off to consider, even if there are no other user experience impacts (in the vein of Advanced Protection). And not all software will reach the level of security required to be safety-critical for the most at-risk users — it’s simply impractical.

Collectively, we currently fall short of providing protection to the most at-risk users, so I hope the teams that make web browsers, OS kernels, and other software on the front-lines will decide to pursue even more secure systems. But either way, what you absolutely have to do is to say, out loud, what trade-offs you’re making. People shouldn’t be left guessing whether your threat model includes them or not.