“Designed for the future of integrated Edge AI and general robotics” – this is how Nvidia announces the new NVIDIA  Jetson AGX Thor computing devices.

As you can imagine, real-time communication is essential for robotics applications. Fast and dynamic movements, short reaction times on sensor events and fast action time for actuators. These are only some of the various requirements which meet robotic applications.

And this is exactly what acontis EC-Master can achieve: our EtherCAT MainDevice Software Stack realizes highest performance with lowest CPU loads and hard real-time capabilities.

This blog will show you how to make your Nvidia Jetson AGX Thor’s EtherCAT controller on a Linux system real-time capable. To show the different performances depending on the optimization effort of the system, we divide this blog into three stages: Stage 1, 2 and 3 (see figure 1). Starting from minimal effort we go step by step more into the optimization of the system. We compare the three stages and, in the end, it’s up to the application, which level needs to be selected.

Figure 1: Different optimization stages of EtherCAT performance

Now let’s dig deeper in details …

Nvidia offers a Quick Start Guide which ensures a smooth Linux installation with the JetPack 7.0. The Linux system is already equipped with a SMP Preempt kernel.

Before we start, let’s have a look into the different stages of optimization. There are many different use cases of this system, and not all of them need highest performance. It is quite easy to get a ”plug&play” system where the EtherCAT communication can be set into operational mode without any big effort (Stage 1). But for sure, there are applications where the performance of the system plays an important role. And to reach this stage, a certain effort is needed (Stage 2). Last but not least, the system can also achieve highest real-time performance requirements, which needs some additional effort (Stage 3).   

For each stage, we want to check and measure the performance of the system. We realize this by adding our acontis performance measurement reference setup to the Nvidia Jetson AGX Thor, connected via an Ethernet Cable (see figure 2).

Figure 2: Test system with Nvidia Jetson AGX Thor and Beckhoff SubDevices

Acontis performance measurement reference setup consists of :

• EtherCAT coupler (Beckhoff EK1100)
• 2x EtherCAT 4-channel digital output Terminal (Beckhoff EL2004)
• 2x EtherCAT 4-channel digital input Terminal (Beckhoff EL1014)
• EtherCAT 2-channel analog output Terminal (Beckhoff EL4132)
• EtherCAT extension (Beckhoff EK1110)

As EC-Master application, we will use for this test setup the sample application EcMasterDemoDc. This application supports the functionality Distributed Clock and enables a detailed analyzing of the measurement results with the option -dcmlog.

Before we start, we need to define the meaning of real-time performance. Because the definition will vary depending on the requirements of the current system. For some, the system shall be real-time capable for a high amount of SubDevices while for others, the total amount of data (cyclic and/or acyclic process data) is important. Also optional features like Distributed Clock (DC), Hot Connect or Multiple Update Rates will have an effect on the real-time performance. And, of course, customers EtherCAT application itself has a huge impact on the performance. The way how the application handles the access to the EC-Master library will affect the real-time capability of the system.

Beyond the above-mentioned points, two big topics will for sure the most important benchmarks to define the real-time performance:

1. CPU Load:
The CPU load shows exactly the time which EC-Master needs to handle the EtherCAT communication. That means, the lower the CPU load, the more time is reserved for the CPU to handle all other processes in the system.

2. Jitter:
This is an important parameter which shows the systems “real-time” capability. Jitter is variation in time intervals between the frames transporting process data. A low jitter is mandatory for a precise synchronization and control of a system.

Let’s start with the “naked” system, installed according the Quick Start Guide from Nvidia. From acontis side, the complete folder including the “EcMasterDemoDc” application and the generic Linux sock raw driver are saved on the system.

Figure 3: Architecture of SockRAW

In this setup, the EC-Master stack communicates with the SubDevices via acontis network interface module SOCK_RAW (see figure 3). This module communicates with the Linux network stack and driver code and exchanges EtherCAT frames via the standard Ethernet Adapter (MAC). As there is a lot of internal communication in between, we expect poor performance of real-time performance, especially as the real-time kernel is not configured.

So let’s start our EC-Master with following command:

$ sudo ./EcMasterDemoDc -f eni.xml -sockraw enP2p1s0 -v 4 -b 1000 -perf -dcmlog

EcMasterDemoDc is using the ENI (EtherCAT Network Information) file with the configuration of the SubDevices for the performance reference (-f eni.xml) and the application is using the module SOCK_RAW (-sockraw) with the name of the network interface as following parameter. We set the verbosity level to a high level so we can see plenty of information as console output messages (-v). For our test, we use the cycle time of 1 ms (1000 us) which is specified with the parameter -b. Parameter -perf enables the performance measurement and with the function -dcmlog, we are able to print out the detailed performance data into a csv file.

Below in Figure 4, the results of the test with the EtherCAT communication show a quite high jitter of about +/- 330 µs (see deviation of min. and max. Cycle time). The whole job of the EC-Master last about 10,4 µs of the specified CPU cycle time of 1000µs. In other words, the EC-Master needs around 1,04% (10,4 µs of 1000 µs) of the CPU load – these are amazing performance values. So the average values are quite good, but we see high maximum values due to the lack of the Linux real-time kernel.

Figure 4 performance measurement results – Stage 1

The chart below (Figure 5) is recorded with the -dcmlog functionality and shows the value CtlErr (Controller Error). These values (in nanoseconds) represent the current deviation of the DCM controller. The chart shows some peaks which generate delays and which could cause EtherCAT frame losses and serious impacts on the system performance.

Figure 5 Cyclic frame jitter in ns – Stage 1

To sum up the results of this Stage 1 measurement, we see that the performance regarding the CPU load is great. But the jitter values of around +/- 330 µs which corresponds to 33% of the cycle time, which is in most cases to high. This jitter might be ok, for higher cycle times, e. g. 5 ms.

In this second stage, we realize some improvements on the Linux system.

As the real-time Kernel is already loaded within the JetPack 7.0, we now need to configure the kernel. If, in any case, there is a need to download the latest real-time Kernel, please check our blog Developing Compact and Powerful EtherCAT Controllers with acontis EC-Master on the NVIDIA Jetson Orin Nano for more information. Some more information about Linux optimization can also be found at the acontis webpage.

To improve the real-time behavior of the system, following  kernel-settings need to be adjusted:

$ sudo nano /boot/extlinux/etxlinux.conf

By adding following Kernel Boot parameter to the RT-Kernel config, we exclusively use CPU number 3 for the EtherCAT communication:

isolcpus=3

After adding this parameter, safe the RT-Kernel config file and restart the system. After the successful system reboot, we are able to check if the parameters we added to the Kernel Config are actually enabled:

$ cat /proc/cmdline

In our system, the Kernel config will look like below:

root=PARTUUID=b7a98bc4-af54-4ba7-b42b-01791f6ebacf rw rootwait rootfstype=ext4 mminit_loglevel=4 earlycon=tegra_utc,mmio32,0xc5a0000 console=tty0 console=ttyUTC0,115200 clk_ignore_unused firmware_class.path=/etc/firmware fbcon=map:0 nospectre_bhb efi=runtime bl_prof_dataptr=6225920@0x2008010000 bl_prof_ro_ptr=65536@0x20 isolcpus=3 bl_prof_dataptr=6225920@0x2008010000 bl_prof_ro_ptr=65536@0x2008000000

Now, we need to change the scaling governor for all 14 CPU cores of the Thor system. We assign to each CPU the highest performance with the maximum frequency to achieve the maximum computing power of the cores.

This can be realized by adding following commands:

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
…
echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor

So let’s start again our EC-Master with following command:

$ sudo ./ EcMasterDemoDc -f eni.xml -sockraw 1 1 -v 4 -a 3 -b 1000 -perf -dcmlog

With the command -a, the CPU affinity is specified. In our case, the EC-Master stack will use the CPU number 3 exclusively. This will optimize the performance of the EtherCAT communication and also ensure that the other applications can focus on their CPU loads. Let’s have a look to the result:

Figure 6: performance measurement results – Stage 2

We see immediately in Figure 6, that the jitter has more than halved to around +/- 125 µs. But nevertheless, especially for hard real-time applications with Distributed Clock functionality, these values are too high for a stable system.  This can also be seen in Figure 7, where still some peaks occur in the chart.

Figure 7: Cyclic frame jitter in ns – Stage 2

Let’s go to the third stage. To ensure that we reach a real-time performance of the Ethernet adapter, we need to replace the standard Linux Ethernet driver. At the Thor Development Kit, one of the onboard RJ45 Ethernet Adapter is a Realtek Ethernet Controller.

This specific real-time driver for this Realtek Ethernet Controller, which is developed and tested by acontis technologies, is transferred to a working directory in the user space.

To ensure, that the acontis driver is used for the Ethernet Controller, the standard driver needs to be removed from the controller. This can be realized by blacklisting the standard driver:

echo blacklist r8126 | sudo tee -a /etc/modprobe.d/blacklist.conf
update-initramfs -k all -u
sudo reboot

With the command above, the standard driver of the Ethernet Controller is added to the blacklist. After a reboot of the system, this driver is will not be loaded into the kernel any more.

To grant direct access from the ethernet driver to the Ethernet network adapter (MAC), the kernel module atemsys needs to be activated. First, download the atemsys module especially for the Thor system on the acontis download page and save the module to your working directory.

Following command will initiate the loading of the module:

$ sudo insmod atemsys.ko

The whole process allows now a direct hardware access of the acontis real-time Ethernet driver to enable high real-time and CPU-load performance of the system. An overview of the architecture is shown in Figure 8 below.

Figure 8: Architecture of acontis real-time driver

So let’s start our EC-Master with following command:  

$ sudo ./EcMasterDemoDc -f eni.xml -rtl8169 1 1 -v 4 -a 3 -b 1000 -perf -dcmlog

Our EcMasterDemoDc is using similar configuration as in our last test, but we replace the sockraw driver with the acontis real-time Ethernet driver for the Realtek Ethernet controller (-rtl8169). Now, we get following results:

Figure 9 performance measurement results – Stage 3

The results of this test shows that the system is able to handle the EtherCAT communication with a low jitter of about +/- 6 µs (see Figure 9, deviation of min. and max. Cycle time). The whole job of the EC-Master last about 13,8 µs of the specified CPU cycle time of 1000µs. In other words, the EC-Master needs around 1,38% (13,8 µs of 1000 µs) of the CPU load.

And finally we do not have any peaks in the CtlErr-chart (see Figure 10), so the system runs stable and there are no risks for the stability of the system.

Figure 10 Cyclic frame jitter in ns – Stage 3

With the knowledge, that a 1ms cycle time runs easily with the current system, we also can try out a cycle time of 250 µs:

Figure 11 performance measurement results with cycle time of 250 µs – Stage 3

Also here (Figure 11), we can find a stable system with the low jitter where the EC-Master needs around 5,4% (13,5 µs of 250 µs) of the CPU load. That is an amazing value and it shows that the EC-Master can handle cycle times of 250µs and below.