The Future Car – The verification and validation of autonomous vehicles – Transcript
In a recent episode of the Future Car podcast, I talked with Nand Kochhar, VP of Automotive and Transportation, and Gwen van Vugt, Sr. Director Autonomous Vehicles, both of Siemens Digital Industries Software, about the testing and validation of autonomous vehicle systems, and how companies are working to address the challenge of verifying and validating the most complex vehicle systems ever made.
We discussed the ways in which the automotive industry is continuing to improve advanced driver assistance systems, driving for higher quality and the introduction of new and powerful functionalities. As companies continue to pursue higher levels of autonomous driving, modern vehicles have increasingly come to resemble “computers on wheels”. These vehicles contain powerful computer chips and artificial intelligence algorithms to safely guide the vehicle through dynamic and often hectic surroundings.
Furthermore, we considered the responsibility of carmakers for ensuring their vehicles are safe for public use and what this entails for their testing, verification, and validation processes. AVs must be able to safely and reliably navigate the driving environment, over and over, day in and day out. So, how can the creators of these systems demonstrate such safety in all possible driving scenarios? And what exactly does that mean? When is an autonomous driving system safe enough to be released to market? How can manufacturers make sure that an autonomous driving system is robust enough that the passenger can let go of the steering wheel and rely entirely on the technology? To find out, you can listen to our podcast here, or read the transcript below:
[00:10] Conor Peick: So, today, we’re gonna talk to Nand Kochhar, VP of Automotive and Transportation, and Gwen van Vugt, Senior Director of Autonomous Vehicles, both of Siemens Digital Industries Software. We’re going to discuss the testing and validation of autonomous vehicle systems, which will be a crucial step in bringing these vehicles to market. So, first of all, I’d like to welcome you both on to the podcast. Nand, good morning, and great to talk to you again.
[02:48] Nand Kochhar: Good morning, Conor. Thanks for having me over.
[02:50] Conor Peick: And Gwen, welcome to you as well, I believe this is your first time on the show.
[02:54] Gwen van Vugt: Yeah, thank you.
[02:55] Conor Peick: So, to get started on our discussion, I think let’s address the autonomous vehicle as a trend. I know a lot has been made of it in the past. But in 2022, and going forward, there’s still a big trend in the automotive industry.
[03:08] Nand Kochhar: It is one of the big trends, both from a technology standpoint and a business transformation going on in the automotive industry. I say that it’s a trend, but at the same time, some of it has been a reality already if you look at the ADAS features. So, the majority of the OEMs out there have ADAS, SAE Level 2 – or some even Level 3 – type of vehicles on the road today. So, you can see, basically, since this trend has been there for few years, that has come to maturity or certain levels of autonomy. Then you continue to look that industry is driving higher and higher level of autonomy, going to SAE Level 4 autonomy, which is basically you could have vehicles in a closed environment do things like logistics. And it goes beyond just a car or a consumer product, it could go into also the commercial, for example, in the trucking business. Some also call it working in a geofenced area – so that’s the Level 4 of autonomy, which is, again, converting very fast from a trend into a reality in a lot of the scenarios.
[04:24] Conor Peick: So, we really are moving into a world where this is becoming a fact in the market rather than science fiction.
[04:31] Nand Kochhar: That is for sure. So, that’s why the levels of autonomy are very key in defining, converting research and development into a practical reality the consumers are experiencing and they’re enjoying the benefits of levels of autonomy.
[04:49] Conor Peick: So, then what are the current major challenges around autonomous vehicle development and how do you guys see that evolving in the next five years or so?
[04:57] Gwen van Vugt: What we see happening is what Nand just said. So, we have already an existence of SAE Level 2 systems on the road today. And there is also a big move and development of SAE Level 4 systems. But what we see happening, especially on those higher levels of autonomy, is that the industry is going more from research and development type of activities towards prioritization activities. And with that, of course, yeah, there are a few challenges. I actually see three major challenges happening there. So, the first thing is that when you move towards prioritization, efficiency comes into place because you need to build and develop autonomous vehicles in an efficient way. So, efficient engineering methodologies are getting more important. And of course, the whole validation topic is still a big challenge. Understanding how much testing you need to do in order to cover your entire system is still a big challenge. And of course, when have you tested enough? When is it good enough? That is also still a question mark that we need to solve as an industry.
[06:19] Conor Peick: That seems like a really interesting challenge to say, “Okay, we’ve done enough testing and we feel confident.” Is there anything that can help in that area, in ensuring that you’ve done enough testing?
[06:30] Nand Kochhar: I think you’ve said in your opening, a couple of big challenges: How do you make sure it’s safe enough and robust enough? And then you also touched on the RAND Corporation study, which is an extensive testing validation of 1 billion miles to get to that level where you build that confidence that its failure rate is better than a human driver. So, when you keep that in mind, then it becomes really important to look at the entire testing, validation, and the certification aspects of it. So, bigger challenges become like data collection, data management, data processing, and making sure you’re getting the right data and qualifying based off of those results. What can be done about it? One of the things is around, it’s not just the technical aspects, but also having standards to which you will do signups. So, in other words, you have to continue to work with the authorities, just like NITSA in the US and the other authorities all over the globe, that what is acceptable from a testing standpoint? So, if you use an example of passive safety, there are obviously standards for testing and acceptance and the star ratings on the vechicles. So, you need to get the standards organization, you need to get the testing authorities to develop those procedures that what’s going to be acceptable, and then technology, and those policies, and procedures can work; that’s when you’ll see that this is becoming more and more of a reality.
[08:11] Gwen van Vugt: Those are extremely important for the prioritization phase itself. So, as long as you’re in research and development, you don’t care too much about these standards and thresholds. But the more you move, also as an OEM, for example, into prioritization of these type of high levels of autonomy, you want to understand how much effort you need to put into the validation to prove that your systems are safe enough to go on to the streets. And methodology, but also the framework around that, it should be clear enough for you to make investment decisions.
[08:51] Conor Peick: So, a strong industry cooperation to develop some standards around what is acceptable and what you need to show in order to say, “Okay, this vehicle is safe and ready to go out to the market.” Am I hearing that correctly?
[09:05] Nand Kochhar: That’s right.
[09:06] Conor Peick: So, then the other thing I think people may have heard about is this idea of critical scenarios and needing to cover those in your testing. How can you make sure that you cover enough of these so-called critical scenarios, I think edge cases is the other term? How can you ensure that you’ve covered enough of those to prove the safety and reliability of your autonomous vehicle system?
[09:27] Nand Kochhar: This is where the simulation comes into play. As you mentioned, if you need a billion miles of physical driving testing, that’s not possible in the given product development time. So, simulation plays a key role in developing those edge cases and then validating through those edge cases. So, you can have a combination of physical testing for correlation, and then do majority of your work of collecting those miles and scenarios through simulations. This is where it’s not a simple simulation of, let’s say, vehicle dynamics or driving models. You also bring in artificial intelligence, machine learning, and a combination of, let’s say, the controls algorithms, and all those test scenarios into play. And the simulation in that sense is far advanced, bringing in not only machine learning, but all of the computer vision technologies and high intensive computing power behind it to validate and get through those critical scenarios, both in the generation, and then in the testing, and then the certification of that. That is the most important point, I think, in getting to the validation stage.
[10:44] Gwen van Vugt: What would you still see is that a lot of companies are driving around and collecting a lot of real-world’s data. The problem with collecting real-world data is that critical events rarely happen. That’s actually also where the billions of driving miles from RAND Corporation come from, because in order to capture a certain amount of critical elements, in order to statistically then prove that your system can handle those types of situations, yeah, that is basically unreliable and undoable to do in the real world. And the interesting thing here also is that for other systems, today, you see that the manufacturers are actually doing those types of, let’s say, million-mile driving programs to validate their systems. But that is still knowing the fact that in the end, it’s the human that is in the car, is, in the end, responsible for the safety and not our system in itself. If you go to the higher levels, Level 4 type of systems, you cannot rely anymore on the driver because the driver might not even be there in the car. For those types of systems, you need way more proof. And in order to collect sufficient critical scenarios, you need to come up with synthetic scenarios that you create in simulation, and to do that efficiently. So, coming up with the most critical scenarios because those are, basically, also dependent on the system that is the best. So, really finding the critical scenarios that are critical for your specific setup, that is very challenging, and that is also one of the things that, of course, we are working on.
[12:30] Conor Peick: Given this virtual approach, do you guys think that physical testing will ever become redundant? Well, we always need to do some level of physical testing.
[12:41] Nand Kochhar: We’ve always had this vision of zero prototype, or reducing or eliminating physical testing. However, my view is that we will always have some level of physical testing for validating as things continue to innovate, as you continue to try different algorithms, you still need to validate before you can rely on 100% virtual scenarios. So, from that sense, I’m sure – again, personal view – the innovation in the automotive industry is not stopping. We’re never going to get satisfied, “Hey, we reached good enough technology. Let’s stop our R&D right here.” I’ll use several examples: sensor development, camera, LIDAR/radar, those technologies continue to mature. So, when you bring in a new technology, and even in a basic automobile as a materials, when you bring in the new materials, you need to test them out in a physical environment to do that initial correlation. So, when you keep those kinds of things in mind, some level of testing. Of course, the goal is always to reduce or to eliminate testing, which does not give you any insights. You call it a redundant testing, a testing not really needed that you can rely on virtual verification and virtual validation for those scenarios. And of course, the path has continued to grow more and more of those. And so you have a reduced amount of physical prototypes, physical prototype testing. Only to do new things where you want to get some insights, the things you don’t know yet as an engineer or as a decision-maker.
[14:23] Gwen van Vugt: So, we definitely will continue to see physical testing, always a necessity. One other aspect of that is that the more you rely on simulation, especially, for example, for virtual homologate, for the virtual certification. You need to basically prove that to the simulation models that you’re using, the virtual models that you’re using for simulation, that those reflect real-world behavior. So, that correlation between your virtual models and the real world for that correlation to measure — I mean, for that, you also need physical testing. With those and correlation tests, you can then find out what the difference is between your virtual models and the physical worlds such that, later on, you can then also do analysis on how safe or how sure you can be that your virtual models are correct. That’s definitely a necessity that will continue. And also, one thing which we see there in the industry is, you can do those correlation testing on the entire vehicle, but also on the subsystems. So, like Nand says, central modeling and measuring the correlation on the component level, that is also possible. And then you can build up, basically, a virtual system, in the end, to prove a much more complex system with much less physical testing than what you do today.
[15:58] Conor Peick: Proving this idea of correlation is an interesting one, where it’s basically proving that your virtual world closely matches the real world. And as you prove that and show that over and over again, we could gain higher and higher confidence in these virtual simulations. But nevertheless, we’re always going to need to put a car on the road and make sure that it works correctly, more or less. I think this ties in with the next big topic, which is customer trust. And one of the big things that autonomous vehicles have faced is gaining the trust of the public; people are hesitant to release their control over the vehicle, at least until they’ve seen the technology work over and over again. And then, of course, I think people might come around to it. But did you guys think, are consumers ready for higher levels of autonomy today?
[16:46] Nand Kochhar: Yeah, I think that’s really interesting when you take a look at the history of how technologies, not only in automotive but in other industries, have evolved as well. There are several scenarios, it does take some time. Also, the consumer, it’s not that you have a single consumer out there; there’s a whole variety of consumers who are technology savvy and ready to try these technologies, and the others who need some time, who need to see that it works, and they need to see the proof, and then they adapt to those technologies. So, my take is, yes, we’ve seen examples of autonomy being implemented or the autonomous vehicles being implemented in certain cities within the US, let’s say, Arizona ran a program, and then Phoenix, where it wasn’t just for the general public, but they nominated and people volunteered who’re willing to try that, and they’ll use that as a shell system in a given environment, and people adapted to that. Others will want to, obviously, try it at a shorter span type of thing before they make it really, really comfortable that they can truly 100% adapt it. So, it’s a migration technology. And in my mind, it’s no different from how the first planes were designed, and people who tried flying a plane, it’s not that everybody adapted, “Let’s take a flight,” everybody felt safe and comfortable. So, I think it’s the trend. And it’s maturing at a really fast rate, especially when the new business models are coming in. And it’s getting adopted.
[18:31] Gwen van Vugt: I think the general public is, in principle, ready for automation. I mean, we are already seeing autonomous systems around us, I will say, basically, every day. There are a lot of us that step into an elevator almost every day and they don’t even think about it. But basically, an elevator is an autonomous system and it’s pretty safety-critical, I would say, as well. But the same applies to — I mean, there’s already autonomous metro systems deployed, where thousands of people, basically, take those autonomous trains. And actually, even if you go into a theme park, you go into the rollercoasters, which are also autonomous systems. So, of course, people trust those systems because they are engineered and validated with sound engineering principles. And actually, in a lot of those cases, simulation also has taken a big part of that in the engineering of those systems. So, I think that the general public is basically willing to trust autonomous systems as soon as they see the benefits of using that. Once we can prove that the systems are safe, people will start using it.
[19:46] Conor Peick: That’s interesting. I never would have thought of an elevator or a roller coaster or even a train, I guess, for that matter, as being an autonomous system. But I suppose it completely is. The key difference being, a lot of those have a very constrained path of motion. I think that just speaks to the additional challenge of trying to make an autonomous car, which in theory could drive anywhere. A lot more variables, I suppose.
[20:12] Gwen van Vugt: Yeah, that is definitely the big leap that we’re taking in terms of technology. And to be honest with you, I think that the deployment of autonomous vehicles because of that will also see a more gradual approach. So, you will, for example, see that first on highways because that’s also a much more controlled environment than the city centers. And you will basically see these autonomous systems not being deployed in certain areas. First, like Nand says, in Phoenix, Arizona, they already deployed, but also even they’re in very relatively small parts of the town because you need to have pretty good understanding of the environment in order to make sure that your system can cope with all the different scenarios that it can encounter.
[21:04] Conor Peick: We’re getting on to the idea of this transportation as a service in a way or deployment of vehicles. Before we get there, I wanted to ask you guys about the interaction of government with this technology. What role should government play in building policy or determining how autonomous vehicles are deployed in the future?
[21:24] Nand Kochhar: Yeah, I think, I very briefly touched on that. Is a very, very important role. It’s not an issue to be resolved by a single body, let’s say, an OEM or a supplier. It is a combination of standards organizations, certification organizations, the government authorities – in fact, you can even go beyond that – the policymakers, collectively things need to come together from all these angles for technologies like autonomous vehicles to make it a reality that and adaption at a very mass scale. All these things support each other if you have standards of testing “safe,” that needs to come from standards organizations or authorities and validated, then only the suppliers and the OEMs can develop and validate to those standards. And then the safety regulators need to be engaged either directly or indirectly through the standards organization to play that important role. And other things like the insurance companies will have a big role in adapting to these new ways of doing things. The entire community needs to come together, and that’s what makes it a little harder sometimes, and that’s why maybe the progress is not as fast as you would expect. Well, the technology continues to mature, you keep going on the next level from a technical perspective, but there’s a phase lag to make it a reality because all the other elements we touched on have to come together. But to answer your question very directly, very important role by all the other elements, not only just government but the standards organizations.
[23:10] Gwen van Vugt: I think this is directly tied to the trust topic that we just covered. And the examples that I just gave of all of us getting into an elevator and into a rollercoaster, you also do these types of things because you know that there is actually a regulated market out there. I mean, elevators are tested and validated towards certain standards that are accepted. But you also know that the manufacturers of these systems that they are being controlled, or being checked upon. And even in themeparks, you know that there is annual maintenance being done on these systems. So, the regulation parts adds also to the trustworthiness of these technologies. So, that’s also the reason why it’s really important for governments to step out here.
[24:01] Conor Peick: They certainly are going to play a big role in that: Building of trust, and also, as Nand mentioned, the creation of these safety standards and other regulations, or even just industry benchmarks.
[24:13] Gwen van Vugt: And in the end, it’s public acceptance. And that comes with industry, the general public, and governments working together to basically have a common understanding of when a system is safe enough and how to build that and how to maintain that.
[24:30] Conor Peick: So, then I think we can get back to the deployment aspect because that’s another very interesting topic within autonomous vehicles is, let’s say, we’ve developed very safe, reliable vehicles; how are they going to get deployed in the market? How do you guys see that evolving?
[24:47] Nand Kochhar: There are several scenarios. I think people typically think of as consumers or don’t have to buy cars, then they will be leveraging Transportation-as-a-Service. And we’ve seen many companies, especially as the fleet operators, trying to make that as a business case. So, in today’s case, Uber or Lyft or Ola and other parts of the company running that one of the big cost factors in there – the driver itself. So, if those fleets have the autonomous vehicles running, it makes a very strong business case because you are getting into a certain level of efficiency and business results delivery. So, Transportation-as-a-Service, as the levels of autonomy mature, becomes a very valid business case. And that’s why these two things need to come together: the maturity level, safety, trust, and then you can start to adapt several different versions of these business models.
[25:51] Conor Peick: Do you see there being any additional verification or validation challenges related to Transportation-as-a-Service type deployment?
[26:00] Nand Kochhar: When you take it all the way to Transportation-as-a-Service, it’s now going beyond just the vehicle itself; the vehicle needs to be connected, not only to the surrounding vehicles but also to where the service is being used. Let’s say if it’s a delivery service, so all of a sudden, you’re adding a degree of complexity, not only from manning the vehicle, and the traffic, and then the city environment around the traffic lights, and the pedestrians, and the cyclist. So, your system of systems becomes much broader than a vehicle itself. So, this is where, I think, from a technical perspective, we talk about it, the approaches like Model-Based Systems Engineering. We promote that that’s what is necessary to deliver, because in that, first, you have to define a system. So, when we’re doing a vehicle development, the vehicle is your system. And then when you take the vehicle on the road, then the road and the infrastructure becomes your system. But when you’re doing transportation as a system, just like I’ve seen some of the pilots happen in the US, as a pizza delivery happening in an autonomous way, obviously, in a geo-fenced area. Now, you can imagine other technologies, that it’s not even transportation of the person, that’s the transportation of the goods. So, when the pizza car, well, this was a pilot in an arbor, a few years ago, in one of the suburbs, outside of Metro Detroit, that when pizza car comes there, needs to communicate with the house owner that I’m outside. And then you have to have a system that, you know, the window pops up, you’re making sure the right person is picking up the pizza. So, now you can imagine this, just transporting of a good, as simple as a pizza delivery, now you talk about that having people leveraging Transportation-as-a-Service, and you can see how there are additional elements you need to take into account.
[28:00] Nand Kochhar: So, it becomes even more complex. And complexity is which is already increasing; from your introduction, you said, “Hey, anywhere from a chip to a system to a full vehicle.” There’s that level of complexity. And now you’re creating all these scenarios around driving, there’s another level of complexity. And now you take that into Transportation-as-a-Service, and you’re including other levels of complexity. So, the approaches like Model-Based Systems Engineering will help you navigate through those because, first, you have to define what your system definition is.
[28:36] Gwen van Vugt: I think Transportation-as-a-Service is definitely a very interesting concept. It will be enabled by autonomous vehicle technologies. But it will actually also go through deployment cycle because if you look at our transportation system, as we have today, is actually very inefficient. And we have a lot of vehicles standing out there, actually, a majority of the time are not driving. But also, for example, trucks that are the majority of the time driving are driving a lot of the time with half-empty loads. In order to have a Transportation-as-a-Service model, and even that’s more mature, like Nand says, that needs a new system thinking approach, which over time will come up with types of vehicles that we have probably not even thought about today. If you look to people movements, if you will do that all with a fleet of robo taxis, you need a lot of robo taxis to move all the people during rush hour. But then outside of rush hour, all these robo taxis are still standing somewhere doing nothing. So, maybe you should use those robo taxis outside of diversion. I want to transport goods or use another way of making use of them. And that is the invasion cycle that will go also through that Transportation-as-a-Service activity. But I think Transportation-as-a-Service becomes possible because of the electrification and because of the automation of vehicles.
[30:22] Conor Peick: The one other interesting technology topic, I think, that comes up related to this is this idea of smart infrastructure, where self-driving cars can communicate with actual pieces of infrastructure in the driving environment. And that, I think, starts to build out this idea of a mobility system. It’s not just you and your car anymore; it’s everyone’s car, or in this Transportation-as-a-Service model, fleets of cars that are all also communicating with the infrastructure in the area. How does that change the verification and validation requirements? Does it? And how does smart infrastructure play into all of this?
[31:03] Nand Kochhar: Even for autonomous driving, in my mind, the smarter infrastructure becomes part of it. So, if you’re truly driving in a city, then it needs to be talking to the traffic lighting system, let’s say, in a congested city, or it needs to be talking to the other devices. So, you touched on vehicle-to-vehicle communications, that’s key, especially when you’re driving on a highway, vehicle needs to be self-aware of his surroundings, in this case. Just like as a human being we are, when we are walking or driving, whatever, we have to be always aware of our surroundings. So, a vehicle needs to be be aware of its surroundings. We’ve touched on already; it’s not only the other cars and infrastructure, you might have to have smart cities where you need to be talking to the buildings or the people in the buildings. So, the pizza delivery example, that’s a good one, that you need to be talking to the person and their individual devices, which most likely will be their cell phones. So, the humans are connected in this environment. So, that’s what I mean by systems engineering approach, depending on how far we are taking this, and then you define your system, and then you mature it or you grow it from a total connected world. So, it’s not only autonomous vehicle, but now it is getting into the overall connectedness, which is also one of the transportation trends or the automotive trends.
[32:37] Gwen van Vugt: I think that connectivity – so, V to X type of communication – is actually required to deploy autonomous vehicles efficiently in the cities. So, if every vehicle just relies on its own information, you will not be able to increase the efficiency of the transportation throughput in the city. You need to be able to communicate between vehicles, but also between vehicles and, for example, traffic lights. And yeah, there are new possibilities coming up the moment you start communicating because with the sensor systems that you have on your vehicle, you cannot look around the corner. But if you have communication, and there is a vehicle driving around the corner, or there is an even an infrastructure sensing, sensor placed there because we know that there is difficult place to look around the corner. I mean, you can basically relay that information to the vehicle and the vehicle can start making use of that. That itself is a great use case. But another thing that starts to come up and that’s a big part of verification and validation is, I mean, how can I make sure that I can trust the information that comes from a different sensor system? If I’m an OEM and I’m building a car, I control each and every element of that car. But as soon as I start acting upon data that comes from a system that I have not designed myself, how can I be sure that the information that comes to me is actually trustworthy and I can act upon it? And that is another level of trustworthiness but also standardization between the different actors in the industry. So, that’s another dimension, basically, that you add to the already complex system of autonomous vehicles if you add communication to it. I mean, in the end, it’s necessary to get to an efficient transportation system.
[34:53] Conor Peick: Fascinating. Well, Nand and Gwen, unfortunately, I think we are just about out of time, but it’s been a great pleasure to have you guys on the show. Thank you again for your time and your expertise on this really interesting topic. So, thanks again.
[35:06] Nand Kochhar: Thank you, Conor. It’s always a pleasure talking to you.
[35:09] Gwen van Vugt: Yeah, thanks, Conor, for having me.
