The Future
of Mobility

It was the 2007 DARPA Urban Challenge, an autonomous vehicle competition run by the US Defense Department’s research body, that has led to today’s self-driving technology. DARPA made things easier by eliminating pedestrians and cyclists from its simulation, but what the competing teams accomplished was still impressive. Most put their systems together largely from scratch in just 18 months.

At the time, the teams relied on rules-based programming techniques, which means the robotic systems of a decade ago tended to operate only in constrained environments assuming well-behaved road users who would not deviate much from established rules. In the past few years, the game has changed. Advances in processing power, storage and artificial intelligence (AI) combine to allow computers to think through problems without a programmed ‘script’. From massive volumes of data they learn to recognise patterns with astonishing accuracy, filtering out anomalous inputs from their sensors to focus on what matters.

The industry has come far over the last decade, but how close a truly autonomous vehicle (AV) is to realisation remains a big question. Industry morale spans a wide spectrum. At the optimistic end is Elon Musk, who has shared his vision for the roll-out of Tesla robo-taxis in 2021. At the other end are those who see the technology taking a decade to reach maturity.

© Bloomberg/Getty Images.

Most, including the likes of Waymo, Aurora and Cruise are somewhere in the middle, assuming deployment in constrained environments within a few years. Great challenges remain at each step of autonomous driving, from perception to prediction to planning.

Perception

Autonomous cars must detect and classify the objects around them. This isn’t easy. Even objects with the same functions come in various shapes and sizes, while weather, light and environment can interfere with sensors and reduce visibility. Also, things must be contextualised as well as identified. A stop sign on the road, on a bus, or held under the arm of a construction worker all mean different things.

Cars’ sensors are their eyes and ears, but they can’t ‘understand’ what they capture. A computer is needed to combine inputs from multiple sensors, then sort out errors and inconsistencies. Achieving a comprehensive, robust picture of the world for a computer to process is incredibly difficult.

Prediction

An AV must anticipate the next moves of objects around it and their interactions before they happen. The rules of the road set the pattern of the behaviours and speeds of different users, but people and things don’t always follow rules and the accidental movement of one can have knock-on effects. Also, road users deploy all sorts of non-verbal cues to communicate. Getting computers to understand facial expressions, postures or hand gestures is challenging.

Prediction is seen as the hardest problem in autonomous driving. As Chris Urmson, former technical lead of Waymo and founder CEO of Aurora, put it: ‘If I could wave a magic wand, what part of the system would I make work today to accelerate it [autonomous driving] as quickly as possible ... it's really ... perception forecasting capability. So if tomorrow you could give me a perfect model of what is happening and what will happen for the next five seconds around a vehicle on the roadway, that would accelerate things pretty dramatically.’

Planning

It’s fiendishly hard to specify rules for every action a car might need to take under any circumstance. Right now, self-driving vehicle companies use a hybrid model, so that when the software fails to act, a human ‘safety driver’ can take back control. The alternative to this stand-by is programming extreme risk-aversion into the car, for example defaulting to pulling over to stop or seeking alternatives to a confusing or potentially problematic route. One piece of passenger feedback from the Waymo One service in Phoenix, Arizona, is that the car drives too cautiously, for example taking too long to make an unprotected left turn. The next step is to build in algorithms that tell the car when it’s being too cautious, when it needs to ‘nudge’ forward in dense traffic or commit to an action consistently so that other road users can respond correctly.

One way around this difficulty is to get a computerised neural network to copy what humans do, a process known as ‘imitation learning’. By feeding the computers lots of human driving data, the neural network will learn what we humans would do in similar circumstances. That said, further complications arise where there are ethical choices, as in the famous who to save? moral quandary known as the ‘trolley problem’. The quandary is whether the onlooker should divert a runaway train from its current track, knowing that it will cause the death of one person working on the other line, to save five people working on the existing line. Professor Emilio Frazzoli, founder CTO at nuTonomy (acquired by Aptiv), makes the point that ‘​the real Achilles heel for AVs is we don't know how human-driven vehicles should behave’.

For all these challenges, there are grounds for optimism. First, sensors are getting cheaper and better. According to Aurora, LiDAR (light detection and sensing – a sensing system that uses pulsed laser beams to measure depth and distance to build up a 3-D map) can provide high resolution data from as far as 400 metres ahead. Its price has fallen dramatically, from $70,000–$100,000 to under $1,000. It will continue to fall thanks to continued technological progress and economies of scale.

Second, ‘deep learning’, AI that imitates our brains’ neurological patterns, has advanced very rapidly in recent years. Since it requires lots of data to train the network, it will improve as more data are collected. Waymo has reached over 15 billion miles in simulation and 20 million miles in the real world; Tesla has nearly a million cars on the road driving billions of miles; and Lyft offered the public autonomous driving data from its ‘level 5’ (the highest level of vehicle autonomy) self-driving fleet to help democratise access to self-driving technology.

Third, according to Dmitri Dolgov, chief technology officer at Waymo, the benefits of intra-fleet communications will accrue once more autonomous cars are on the road. If one car ‘learns’ something about road closures, construction sites, or accidents it can instantly share that information with other cars in the network to allow them to react in advance. Today, there are only a few hundred Waymo cars on the street. With a lot more, perception, prediction and planning could be significantly improved.

Chris Urmson believes that the industry has crossed the ‘0–1 threshold’ – the ‘ah ha!’ moment when self-driving cars can take to the road without safety drivers. At that point, the AV moves from being a mixed science-engineering product to a mixed engineering-commercialisation product. While further technological development is needed to improve safety, equally important questions include how to continue to scale, how to build the right business model and how the right customer experience could make autonomous cars more useful. Urmson is confident of large-scale autonomous vehicle deployment in 10 years’ time.

It’s common to see different approaches to a single problem at the earliest stages of a technology’s development. Three aspects are worth highlighting: one relates to the ‘philosophical’ approach to the technology, another to the technology itself and the third to the strategy to get the technology to market.

i. Philosophical approach

There are two schools of thought on achieving full automation. One holds that shooting straight to level 4 autonomous driving is flawed. The technology is not ready and there are already substantial incremental benefits to safety and drivers’ comfort from ‘level 2’ automation, which is basically ADAS (advanced driver assistance systems, such as automatic emergency braking, lane departure correction and adaptive cruise control). Most carmakers, including Tesla with its Autopilot function, follow this approach.

By contrast, those currently aiming for level 4, notably Chris Urmson, believe that level 2 is not a stepping stone for level 4, and that the technologies should diverge. He believes that level 2 autonomy is at odds with human behaviour: people quickly trust technology that works. If encouraged to sit back and relax, it’s hard for them to dip in and out of driving if a risk emerges.

There’s also the challenge of context. Once drivers take back control, they don’t always know enough about their surroundings to make the right decisions. So, while the active safety system is an important technology that should be integrated into vehicles, care should be taken on how level 2 is marketed and delivered.

For years, Waymo considered Tesla’s Autopilot irresponsible. A fatal collision in May 2016 caused by over-reliance on the technology is often blamed on the company’s gung-ho approach. The victim, 40-year-old Joshua Brown of Ohio, was a technology enthusiast so taken with Tesla’s Autopilot mode that he posted dozens of videos of himself using it on YouTube. Despite Tesla’s software warning him to resume control of the vehicle on seven occasions, he chose not to. His hands were on the steering wheel for only 25 seconds out of 37 minutes during which Autopilot was activated, resulting in him crashing into a lorry.

ii. Technological approach

Competition in autonomous driving has turned into ‘Tesla versus the rest’. Tesla’s approach uses cameras and computer vision while the others are based on LiDAR and high-definition (HD) maps. The differences are historical. When Waymo was founded a decade ago, ‘deep learning’ was not yet popular in the AI research community so cameras and computer vision weren’t available solutions. In fact, Waymo only started to apply deep neural networks to pedestrian detection in 2015. Being part of Google meant that Waymo could tap into its parent company’s expertise and resources, particularly Google Street View.

The Waymo team believed that, as well as providing directions, a detailed map of every street would hugely benefit autonomous driving. The argument is that if the system only has to process changes to a mapped area 1 per cent of the time, it can be up to two orders of magnitude safer than a system reliant on real-time perceptions of the world. Since other companies in the industry were founded by former Waymo engineers, they all follow a similar approach.

Tesla is the exception. Elon Musk has dismissed the LiDAR and maps combination as “crutches”. He argued that, as humans drove perfectly well without lasers on their foreheads, so too could computers. In his view, LiDAR sidesteps the fundamental problem of visual recognition needed for autonomy. Also, HD mapping is a laborious and expensive process, and systems that rely on maps are brittle and hard to scale up for multiple cities.

Tesla’s position makes sense for a company that can’t afford to install thousands of dollars’ worth of LiDAR equipment on cars it wants to mass produce. Tesla can also leverage the presence of nearly a million cars on the road collecting real-world data in diverse areas to train its computer vision. Waymo is constrained by gathering real-world data via a fleet of only 500–600 self-driving cars, currently only in Texas, California, Michigan, Arizona and Georgia. That’s why Waymo relies heavily on simulation. While this is a critical tool that helps quickly improve and iterate the system, it is questionable whether computers could ever simulate every real-world driving scenario. Musk may be right when he says: “If somebody can produce a driving simulation that matches the reality, that in itself [would be] a monumental achievement of human capability.”

Tesla’s more purist approach seems riskier, though it is more likely to win in the long term if the company can master computer vision before the cost of LiDAR falls to the tens of dollars. Urmson agrees that LiDAR is a “crutch”, but no more so than petrol-powered hybrids are crutches on the path to electric vehicles. Any technology can be replaced by superior technologies in the future. Urmson sees the existing transport model as being so broken that any technology that can come to market and save lives is welcome. Furthermore, although LiDAR is not cheap, he believes that with the right business model (ride sharing), the costs can be absorbed.

iii. Go-to-market strategies

Today’s autonomous vehicle landscape is a tangled web of partnerships, alliances and investment deals. There are three layers in the AV ecosystem: the cars, the self-driving software and the customer-facing service. Industry players differ over how many layers they are developing themselves.

Companies such as Aurora focus on software only. They want to build the ‘driver’ for driverless cars and rely on partner OEMs, such as Hyundai, for building the vehicles. Aurora sees OEMs as being more than mere “metal benders”. Designing and building millions of vehicles per year and having them operate in a vast range of circumstances for decades is a tough task best left to experienced carmakers, Aurora believes. It meanwhile can focus on building the software, a tremendously difficult task in itself. Besides its highly respected management team, Aurora’s pragmatism in collaborating with others in the ecosystem will increase its chances of success.

Waymo is less clear-cut as, besides software, it is attempting to create its own ride-sharing service with Waymo One, while also providing cars on Lyft’s network. Other companies such as Tesla, GM and Ford are trying to develop the ‘full stack’ or complete infrastructure. Building a commercial full-stack system requires massive resources and a huge range of technical talents. The economics are tricky: developers can’t gradually pay off the cost of development (billions of dollars) across other carmakers as no one wants to buy a competitor’s technology. Manufacturers, such as GM, are also wanting to expand into ride sharing, which is a highly competitive market and it’s not yet clear how it can be profitable. Such firms may be spreading themselves too thin. Nonetheless, despite the difficulties, the returns could be massive if they succeed. Each of the layers they are targeting is a market worth hundreds of billions of dollars.

Who might win?

It’s not yet clear which company’s technology and business strategy is better. Put simply: Aurora and Waymo could become the default operating system for autonomous cars, similar to Android for smartphones or Windows for PCs. Those going for the full stack will face more challenges, but there are historical examples in which the control of an entire architecture brings success (for example, Apple and early BlackBerry).

To my mind, Tesla’s model may stand the highest chance of success. Consider a comparison with Apple. Despite being a ‘closed’ system (only working on Apple products), iOS is still successful because it was developed for hardware devices that consumers desire. Apple is both a software and a hardware company, unlike other phone makers, and is capable of making devices that stand out from the crowd. iOS was developed to work exclusively and seamlessly with Apple hardware products, optimising itself to achieve the best user experience.

Isn’t this exactly what Tesla is doing? Musk’s company makes the electric cars that consumers love and one of its unique selling points is the integration between hardware and software (delivered via over-the-air upgrades). As we move away from complex cars with simple software to simple cars with complex software, driven by electrification, Tesla with its strong software position could gain massive advantage.

Tesla’s position in the electric car market remains strong, but we should be even more excited about Tesla’s position in autonomous driving. With the largest fleet, driving billions of miles on real roads, the data volumes must give it a significant edge.

I am sceptical about incumbent car manufacturers developing autonomous driving software. If software is not their expertise and their hardware is not differentiated, the least a car company should do is to develop an integrated model. If history is any guide, a joining of forces between Ford and Volkswagen to develop self-driving software may go the same way as Symbian, the phone operating system developed by Nokia, Motorola and Ericsson that lost out to Android. The fate of Nokia might have been very different if it had abandoned Symbian sooner and followed Samsung’s approach in adopting Android.

Similarly, it’s better for incumbent car companies to partner with autonomous software companies such as Waymo and Aurora, and focus on what they do best: the mechanical engineering of motorcars.