Too often, I’ve read documents or listened to panel sessions that rehash misleading or just plain incorrect industry talking points regarding autonomous vehicle standards and regulations. The current industry strategy seems to boil down to “Trust us, we know what’s best,” “Don’t stifle innovation,” and “Humans are bad drivers, so computers will be better.”
As far as I can tell, what’s really going on is that automated-vehicle companies are saying what they say both to avoid being regulated and to avoid having to follow their own industry safety standards. That strategy has not yielded long-term safety in other industries that have tried it, however, and I predict that in the long term it will not serve the automotive industry well either. It certainly does not encourage trust.
In this essay, I address the usual industry talking points and provide summary rebuttals. I’m intentionally simplifying and generalizing each talking point for clarity. It is my hope that other stakeholders, policymakers, and regulators can use this information to encourage AV companies to talk about the things that matter, such as ensuring the safety of all road users. We need more transparency and honest discussion — not a continuation of the current, empty rhetoric.
It is important to be clear that, from everything I’ve seen, the rank-and-file engineers, and especially the safety professionals — if the company has any — are trying to do the right thing. It is the government relations and policy people, not the engineers, who are providing the facile talking points. And it is the high-level managers — the ones who set budgets, priorities, and milestones — who affect whether safety teams have sufficient resources and authority to build an AV that will in fact be acceptably safe. So this essay is directed at them, not at the engineers.
For more detailed guidance to state and municipal DOT and DMV regulators, see this blog post.
(I've numbered these points for easier reference. Making yourself a bingo card and bringing it to the next regulatory/policy panel session you attend to keep score is optional.)
- Myth #1: 94% of crashes are due to human driver error, so AVs will be safer.
- Myth #2: You can have either innovation or regulation, not both.
This is a false dichotomy. You can easily have both innovation and regulation, if the regulation is designed to permit it.
To consider a simple example, you could regulate road testing safety by requiring conformance to SAE J3018. That standard is all about making sure that the human safety driver is properly qualified and trained. It also helps ensure that testing operations are conducted in a responsible manner consistent with good engineering validation and road safety practices. It places no constraints on the autonomy technology being tested. (Adaptation would be needed for a safety driver in a chase car if it was deemed impractical to have a safety driver sit in a prototype vehicle for road testing; see this blog post.)
For more general approaches, you can switch from the current approach of track and road testing to more goal-based testing. For example, a regulation that tells you what symbol to put on a dashboard to tell the driver sitting in a driver seat that there is low tire pressure (Federal Motor Vehicle Safety Standard [FMVSS] 138) does indeed constrain design by requiring a light, a dashboard, a driver seat, and a driver. But the lighted symbol isn’t the point; getting the low tire pressure addressed is the point, and regulations can focus on that instead (see this blog post). To be sure, this requires a change in regulatory structure. But the choice isn’t between innovation and regulation; it is between old regulation and new regulation, and that is a far different matter — especially if the new elements of the regulatory approach are based on conforming to standards the industry itself wrote.
The primary industry standards for deployed AVs in the U.S. are ISO 26262 (functional safety), ISO 21448 (safety of the intended function, or SOTIF), and ANSI/UL 4600 (system-level safety). Indeed, the current US DOT proposal for regulation as of this writing is to get the AV industry to conform to precisely those standards. None of the standards stifle innovation. Rather, they promote a level playing field so companies can’t skimp on safety to gain a competitive timing advantage while putting other road users at undue risk.
If a company states that safety is their #1 priority, how can that possibly be incompatible with regulatory requirements to follow industry consensus safety standards written and approved by the industry itself?
- Myth #3: There are already sufficient regulations in place (for example in California).
Existing regulations (with one exception) do not require conformance to any industry computer-related safety standard, and do not set any level of required safety. At worst, it is the "Wild West." At best, there are requirements for driver licensing, insurance, and reporting. But requirements on assuring safety, if any, are little more than taking the manufacturer's word for it.
The one exception is the New York City DOT’s rule to require the SAE J3018 road testing safety standard, and to attest that road testing will not be more dangerous than a normal human driver. (See: https://rules.cityofnewyork.us/rule/autonomous-self-driving-vehicles/ . For bonus points, see if you can find any of these myths in the comments submitted in response to that standard, or in responses to the DOT proposal referenced under Myth #2.)
- Myth #4a: We don't need proactive AV regulation because of existing safety regulations.
The current Federal Motor Vehicle Safety Standard (FMVSS) regulations do not cover computer-based functionality safety. They are primarily about whether brakes work, whether headlights work, tire pressure, seat belts, airbags, and other topics that are basic safety building blocks at the vehicle behavior level. As the National Highway Traffic Safety Administration would tell you, merely passing FMVSS mandates is not enough to ensure safety on its own; it is simply a useful and important check to weed out the most egregious safety problems based on experience.
There are no FMVSS or other regulatory requirements for automotive software safety in general, let alone for AV-specific software safety.
- Myth #4b: We don’t need proactive regulation because of liability concerns and NHTSA recalls.
The National Highway Traffic Safety Administration generally operates reactively to bad events. Sometimes, car companies voluntarily disclose a problem. Other times, a number of people have to die or be seriously injured before NHTSA forces some action (for example, eleven crashes involving Tesla driver assistance "autopilot" with emergency vehicles occurred over a 3.5-year period before action was initiated, with an expectation of many months to resolution). For mature technology, maybe this is OK — if one makes the assumption that the industry is populated with only good faith actors. But even with that assumption, it isn’t enough for immature AV technology and manufacturers new to automotive safety.
Aircraft safety regulation used to wait for crashes, but air travel got a lot safer when FAA and airlines became proactive. Most importantly, a regulatory policy that waits for loss of life and limb before taking action can result in a process that takes years to solve problems, even as the loss events continue.
- Myth #5: Existing safety standards aren't appropriate because (pick one or more):
- they are not a perfect fit;
- no single standard applies to the whole vehicle;
- they would reduce safety because they prevent the developer from doing more;
- they would force the AV to be less safe;
- they were not written specifically for AVs.
These statements misrepresent how the real standards work. ISO 26262, ISO 21448, and ANSI/UL 4600 all permit significant flexibility to be used in a way that makes sense. All three work together to fit any safe AV.
ISO 26262 can apply to any light vehicle on the road for the parts that aren’t the machine-learning–based mechanisms. Cars still have motors, brakes, wheels, and other non-autonomous features that have to be safe. The hardware on which the autonomy software runs can still conform to ISO 26262. All of these are covered by ISO 26262, and the standard specifically permits extension to additional scope.
ISO 21448 is explicitly scoped for AVs in addition to ADAS. Its origin story includes being proposed as an addition to ISO 26262, and it is written to be compatible with that standard.
ANSI/UL 4600 is specifically written for AVs. It applies to the whole vehicle as well as support infrastructure. The voting committee includes experts on ISO 26262 and ISO 21448, so it is compatible with those standards, and in fact it leads naturally to using all three of these standards and not just a “single” standard. (Anyone who knows standards knows it is improbable that any safety-critical system would involve use of just a single standard.)
There is no reason not to conform to these standards, and U.S. DOT has already proposed this set for the United States. All of these standards allow developers to do more than required. All of them are flexible enough to accommodate any AV. None of them force a company to be less safe (really, that argument is laughable). None of the standards constrain the technical approach used.
- Myth #6: Local and state regulations need to be stopped to avoid a “patchwork” of regulations that inhibits innovation.
- Myth #7: We conform to the “spirit” of ISO 26262, etc.
AV developers typically justify their “in the spirit of” statements by advancing the theory that there might be a need for deviation from the standard (beyond any deviations that the standards already permit). The statements never specify what the possibly required deviations might be, and I’ve never heard a concrete example at any of the many standards meetings I have attended. (I’m on the U.S. voting committees for all the standards listed in this essay.)
I’ve never heard an AV company argue, when making its case, that it conforms to the intent of a standard, just to the spirit of the standard — whatever that means. Indeed, any “in the spirit” statement is meaningless, because the standards I’ve mentioned are all flexible enough that if you actually conform to the spirit and intent of the standard, you can conform to the standard. The standards explicitly permit not doing things that don’t apply and deviating from inapplicable clauses with appropriate decision processes. Doing that still permits conformance to the actual standard.
I worry that AV companies’ “spirit” claims are really code for cutting corners on safety when they think it is economically attractive to do so, or they’re in a hurry, or both.
A reasonable alternative explanation is simply that lawyers might want to avoid committing to something if nobody is forcing them to do so. That is understandable from their point of view, but it impairs transparency. The dark side of the strategy is that it provides cover for companies that are not the best actors to hide cutting corners. If companies are worried that they’ll be called out for not following a standard after a crash, they should spend the resources to actually follow the standard. Or they should not spend so much effort making public claims about safety being their top priority.
- Myth #8: Government regulators aren’t smart enough about the technology to regulate it, so there should be no regulations. Industry is smarter and should just do what it thinks is best.
Following the proposed U.S. DOT plan to invoke the industry standards mentioned earlier makes sense, because it addresses precisely this concern. Even two years ago, the standards weren’t really there, but now they are. Industry decided which standards made sense, and then they created them.
If we could trust industry — any industry — to self-police safety in the face of short-term profit incentive and organizational dysfunction, we wouldn’t need regulators. But that isn’t the real world. Trusting the automotive industry to regulate development with immature, novel technology is unlikely to work. It’s possible, and important, for the industry to achieve a healthy balance between taking responsibility for safety and accepting regulatory oversight. Near-zero regulatory influence until after the crashes start piling up is not the right balance.
Near-zero regulatory influence until after the crashes start piling up is not the right balance.
- Myth #9: Disclosing testing data gives away the secret sauce for autonomy.
Road testing safety is all about whether a human safety driver can effectively keep a test vehicle from creating elevated risk for other road users. That has nothing to do with the secret-sauce autonomy intellectual property. It is about the effectiveness of the safety driver.
Companies sometimes say it would be too difficult or expensive to get or provide data. If companies don’t have data to prove that they are testing safely, they shouldn’t be testing. If they think that providing testing safety data is too expensive, they can’t afford the price of admission for using public roads as a testbed.
Testing data need not include anything about the autonomy design or performance. An example would be revealing how often test drivers fall asleep while testing. A non-zero result might be embarrassing, but how does that divulge secret autonomy technology data?
By the same token, regulators should not be asking for autonomy performance data such as how often the system “disengages” because of an internal fault or software issue. They should be asking how often other road users are placed at risk, which is an entirely different story. So the miles and locations tested, along with the collisions and near-miss situations that occur, make sense as measures of public exposure to risk. But metrics related to the quality of the autonomy itself do not, unless and until that data is being used to justify testing without a safety driver.
- Myth #10: Delaying deployment of AVs is tantamount to killing people.
- Myth #11: We haven't killed anyone, so that must mean we are safe.
- Myth #12: Other states/cities let us test without any restrictions; you should too.
- Myth #13: Testing deaths are a regrettable, but necessary price to pay for improved safety.
- Myth #14: Self-certification has served the industry well, so it should not be changed.
Victims and their families involved with numerous wide-scale safety and environmental issues might think that self-certification has not served them well, even if the industry is happy with the situation. Exercise: Pick your favorite automotive industry safety or emissions scandal. Be sure to include class actions and death and injury suits, as well as criminal proceedings, verdicts, and settlements.
It is important to remember that industry “self-certification” is not required to address any functional or software safety standard, despite automotive-specific guidelines and standards describing how to do such safety in detail going back more than 25 years. So companies are not really required to certify anything except conformance to FMVSS, which is not about software and computer-based system safety.
Other industries actually follow their own safety standards (aviation, rail, chemical, power, mining, factory robotics, and HVAC are examples). As far as we can tell, most automotive original equipment manufacturers (OEMs) — the companies that actually sell cars, rather than those companies’ suppliers — do not. (The nuances are significant. Supply chains often follow safety standards if OEMs pay accordingly. And it is difficult to judge the validity of an OEM claim that it does something “similar to” or “better than” an industry standard. I haven’t found a single OEM statement that unambiguously reports conformance to ISO 26262 for their vehicle, which is the bedrock automotive safety standard, but go ahead and look for one. If you find one, tell me the source of that statement, and I’ll happily put a link right here: <none so far> ). (In fairness, it seems some organizations conform to ISO 26262 process chapters, but not (yet) the chapters regarding hardware and software design. The example I'm aware of is GWM, for parts 2, 3, 4, 7, 8, 9 but not parts 5 & 6.)
Automotive is the one life-critical equipment industry that does not even claim to follow its own industry safety standards.
Let that sink in.
A core concept of safety standards and safety is independence. Without independence, it is in practice impossible to get sustainable safety. Just ask Boeing. Yet car makers continually push back on any external oversight as well as conforming to standards that permit non-external but substantively independent checks and balances.
- Myth #15: Safety standards must be based on vehicle testing, via using "performance based safety standards."
Currently, FMVSS is based on vehicle testing, for a variety of reasons. The advantage is that test results can, at least in principle, be reproduced independently. However, the narrow testing parameters (for example, pavement temperature, air temperature, speed, tire pressure) mean FMVSS tests are a narrow check on minimum capability, not a robust characterization of safety across a full range of real-world conditions. That’s OK for what they do — which is to make sure certain features are present, not to ensure that features work across full environmental and usage conditions.
It has been known for decades that for computer-based systems such as AVs, you don’t get safety by testing. You get safety by following best practices and, where available, consensus industry safety standards. Testing is a way to spot check that you got safety right. Tests without required standards conformance won’t ensure safety.
Ironically, the FMVSS test-based regulations that the industry insists are the only ones we should have are probably the most unfriendly to innovation (see Myth #2).
When you hear someone say we should be using "performance based safety standards," that tends to be code for a testing-only approach (FMVSS-style) and avoiding process standards. It also implies rejecting conformance to industry safety standards and any requirement to perform accepted safety engineering practices.
- Myth #16: Following standards would not be cost effective, or would force inferior approaches compared to "superior" internal proprietary standards.
I can’t think of anything in these standards that forces companies to be less safe than any internal standard I’ve seen. Remember that the companies themselves participate in writing these standards and would specifically complain if the standards were to break existing industry practices. Industry standards are written to be compatible with industry practices.
Any responsible company is already following internal standards, which should be at least as rigorous as published consensus industry standards. (If not, how is that a good thing?) They say their standards are better, so those standards should be more rigorous and therefore more expensive to accomplish. If companies think that conforming to industry standards is too expensive, what does that say about the resources they spend conforming to their purportedly superior internal standards?
One could speculate that the way their own internal standards are “superior” is that they permit cutting corners on safety procedures to reduce cost and speed up time to market. This would be consistent with an argument that following industry standards is too expensive and “stifles innovation.” But if we can’t see their standards, we can’t know for sure. And doing less than is required by the industry’s consensus safety standards sounds like a bad idea.
- Myth #17: Regulations should be "standards neutral" to level the playing field.
That's ridiculous. The standards define the consensus level playing field.
All the standards mentioned go through an open industry-consensus process. Thousands of work-hours (at least) and multiple rounds of comments and balloting are spent making sure that all the stakeholders have their say. I can tell you from personal experience that the meetings are numerous and lengthy, and everyone who wants to have a say gets one. (At the end of the day, this is a good thing, even if those days are long.)
Anyone saying that regulations should be “standards neutral” more likely means they don’t want to have to follow standards at all.
All the standards mentioned are “technology neutral.” None of them require using a LiDAR, or a radar, or a camera, or whatever. What they do require is that whatever you decide to build into your vehicle ends up being acceptably safe.
- Myth #18: ANSI/UL 4600 <is broken or says something awful>.
