Validation of lidar simulation – Partnering with XenomatiX
Recently during our lidar simulation webinar, we introduced a new physics-based lidar sensor in Simcenter Prescan. This new sensor follows our simulation philosophy, which is based on three core pillars: the physical device, the world and the simulation engine. We believe these three pillars are essential to providing sensor simulations that are accurate and representative of real-world behavior. However, in order to validate our claims of simulation accuracy, we must go through a validation process. This is where we can determine with certainty that our models are accurate. In this blog post, we will showcase the latest collaboration with the lidar company XenomatiX where we validated our physics based lidar simulation.
Lidar simulation methodology
To validate our physics-based sensor models, we always compare measurement data from real sensors against simulated data. We do this in a three-step approach. During the first step we focus on doing very specific tests under highly controlled conditions. This allows us to ensure we understand and control any variable that has an impact on sensor behavior.
During the second step we move into a semi-controlled environment. Here we go out onto a public road and setup our own testing ground. In this environment we perform a set of tests that allow us to generate data that that can be compared qualitatively to a similar environment simulated using Simcenter Prescan. Since we use known targets we are also able to perform a quantitative analysis of the data. The outcome of this step is what gives us confidence that our simulation is accurate enough to be used under normal driving conditions.
The last and final step is to move into an uncontrolled environment. In this case we want to have a final assessment of performance under normal driving conditions. This serves as a final sanity check and might also be used to identify special cases which might require their own independent validation activities in the future.
Partnership with XenomatiX
XenomatiX is a lidar company whose mission is to provide road safety and comfort by providing the market with solidly reliable, versatile and affordable LiDAR technology. They are one of the first lidar suppliers with a true solid-state unit that is currently on series implementation. The knowledge and expertise of their optical team together with its advanced true solid-state lidar technology made them an ideal partner for Siemens. Together, we felt confident in our decision to collaborate towards validation of our lidar simulation solutions.
1st validation step in lidar simulation
Working alongside XenomatiX, we used their XenoLidar to perform measurements for our validation activities. In the first step, XenomatiX used their lab environment to setup very specific scenarios. In these scenarios we selected four different targets. Two of those targets are very important for automotive use cases: a car paint and retro reflective sign. The two remaining targets have well defined reflective properties for a lidar.
Once the materials are selected, we create a specific setup for validation. We first put the lidar sensor on a tripod looking into the direction of a target. Finally, the targets are setup at different distances from the lidar and under different orientations. This allows us to measure various angles of reflections.
After determining the testing setups, XenomatiX used their XenoLidar to measure energy returned for each target. To make a comparison between simulation and measurement we need to reproduce the measurement environment in simulation. One aspect of the environment is how the targets interact with lidar beams. We captured the reflective properties of each target by using specialized equipment to measure reflections in the 840nm infra-red wavelength (the working wavelength of the XenoLidar). This allows us to get accurate Bidirectional scattering and distribution functions (BSDFs) for each target. Then the simulation engine can use the measured BSDFs to ensure that we simulate the energy reflections correctly, based on physical equations. This allows us to run the simulation and generate simulated energy returns.
The final step is to compare the measurement data against the simulated data. To do this comparison, Siemens and XenomatiX got together to interpret the results and determine the best comparison method. We will soon deliver a technical report that discusses the findings. We can say that the results of this comparison have been overwhelmingly positive.
2nd validation step in lidar simulation
In the second validation step, we setup a semi-controlled environment. Instead of performing tests in a lab, we now do the tests in a parking lot. Using the same four targets we take new measurements. In this case, instead of using a tripod, we use a test vehicle equipped with a XenoLidar.
At the same time, we now need to create a digital twin of the parking lot where the measurements are taken. For this, we use the world modelling capabilities of Simcenter Prescan to ensure that we have the right level of detail in simulation. Then we can run the simulation and generate simulated sensor data.
Finally, we follow the same approach as the 1st validation step. We take the measured data and compare it against simulated data. This is what allows us to say something about our simulation accuracy. The following statement from Nick Van den Broeck, a XenomatiX development engineer, gives us an insight about validation from an expert in the field: “Without validation, you cannot trust the end result of a simulation. That is why we pay special attention, to compare the Siemens physics based lidar, to the XenomatiX XenoLidar in order to make sure that the simulations represent physical circumstances in different situations. In that way we know that we can use and trust the Siemens physics based lidar for developing our XenoLidar prototypes”.
Final validation step in lidar simulation
The first two steps in the validation process are what really gives us confidence about the accuracy of the physics based lidar simulation. This final step focuses on getting data under normal operating conditions. The test vehicle from XenomatiX records data from a test drive and generates a set of points clouds. We can then use this to do a qualitative comparison against a similar test drive done with the simulation environment.
The final outcome
The results of our collaboration with XenomatiX tells us that we have a highly accurate lidar simulation. We see this as an important step to ensure trust on simulation results and it also validates the work done by our development team. Their goal has always been to develop the most accurate lidar simulation in the automotive market. Our experts take great pride on their work and good validation results are essential confirmation of their efforts. We are extremely satisfied with our validation results so far, but we know that there is still a lot of work ahead of us. If we really want to provide the best lidar simulation in the industry, we must continue to innovate. If we truly want to accelerate autonomous vehicle development, simulation must continue to improve. We are committed to doing this. Not only for lidar, but also for camera and radar.
If you would like to know more, feel free to contact us or visir the Simcenter Prescan page on our website.
