When you are running applications with a high precision of synchronization, you definitely need the Distributed Clock (DC) functionality. One typical example is a high-speed motion control application in the automation industry. Here, it is very important that all receiving and sending commands are synchronized to the different drives (SubDevices) so that all axis can move at the same time to ensure a dynamic, jerk-free and accurate movement.

EC-Master offers different possibilities to synchronize the internal timer of the EtherCAT MainDevice with the SubDevice acts as DS Reference Clock. This whole control mechanism that manages initialization, synchronization and stabilization is called Distributed Clock MainDevice Synchronization (DCM).

In this blog, we will have a more detailed look to the network timing of EtherCAT and we learn some interesting things about the different modes of the Distributed Clock MainDevice Synchronization (DCM).

After setting-up a test configuration on our desk, we will run some tests with different SubDevices within the network and we will find out the pros and cons of the different DC modes.

EtherCAT takes the standard Ethernet communication to send and receive data frames throughout the network to the different nodes. In each cycle, cyclic and acyclic data are processed “on the fly”, which means that each SubDevice adds own data to the Ethernet frame which is running through the network. This fast processing with very small delay, with fixed cycles and with high-precision synchronization allows a real-time performance within the network.

At the beginning of a cycle, the newly received data (inputs) are first updated. This is done by evaluating the previously received EtherCAT frames when calling the Process Inputs job function. The application takes this newly received data and calculates the output data  which will be sent to the network. These new output data is sent out when calling the Write Outputs job function. With the help of Direct Memory Access (DMA), the frame is transported from the memory to the Ethernet controller without loading the CPU and finally sent over to the physical network. The frame passes through all EtherCAT SubDevices on the network and is automatically received on returning to the MainDevice without the need for an interrupt. The MainDevice state machine and the SubDevice state machines on each individual SubDevice are executed when calling EC-Master Administration job function.

To ensure high precision and accuracy of the synchronization, the Distributed Clocks (DC) are mandatory. In this mode, it is important that the data frame with the cyclic data will be processed in the different SubDevices before the SYNC signal is raised by the EtherCAT SubDevice Controller (ESC). The data frame contains process data and also the command for distributing the system time (Register 0x910) through the network. The figure below (Figure 1) shows the network timing including the SYNC signals of the Distributed Clock (DC).

Figure 1: Network timing with cyclic frames and DC SYNC0 signals

Typically, the first SubDevice in an EtherCAT network provides the System Time and therefore acts as Reference Clock. Each EtherCAT SubDevice Controller (ESC) has a timer register which is driven by an oscillator and provides the so-called System Time. Based on this time, the DC-SYNC-Signal is raised by the ESC hardware.

On the other hand, the MainDevice is sending cyclic frames containing process data. On the MainDevice controller, the cycle is scheduled by the operating system based on a timer interrupt. And this timer is also driven by an oscillator.

As we have now two oscillators, a synchronization between the two systems is required. This synchronization, the so-called Distributed Clocks MainDevice Synchronization (DCM), ensures that the cyclic frame completely arrives on the SubDevice before the DC-SYNC-signal is raised.

Figure 2: Network timing without Distributed Clock MainDevice Synchronization (DCM)

The typical timing problem without DCM is shown in Figure 2. The MainDevice controller starts his cyclic task based on his own oscillator (see blue-dashed line) and sends out the EtherCAT frames to the network. During the first cycles, the DC-SYNC-signal, which is raised by the EtherCAT SubDevice Controller, is synchronous with the cyclic task of the MainDevice (see red-dashed line). The SubDevice is able to start the own firmware task with the available EtherCAT data. As the timer of the SubDevice is slightly different to the timer of the MainDevice, even if they are working with the same frequency, a certain drift will occur over the next cycles. After a period of time, the DC-SYNC-signal will raise even if the EtherCAT frame is not completely arrived (see red-striped area). As a result, the data of the frame are missed and during this specific cycle there will be not in/output data available.

To avoid the above-mentioned issues, EC-Master provides different possibilities to synchronize the two clocks. We will focus on the three most common modes: “MainDevice Shift”, “Bus Shift” and “MainDevice Is Ref Clock”.

If the “MainDevice Shift” mode is used, the system time is provided by the timer of a SubDevice (DC Reference Clock) and the MainDevice follows this time. Depending on the DC Reference Clock, the timer of the MainDevice is adjusted. This mode leads to reduced DCM controller error but adjusting the timer is not available and therefore this mode is not available on all operating systems.  The principle of this mode is shown in figure 3 below. The “MainDevice Shift” mode achieves an accuracy of the SYNC signals around +/- 20 ns jitter, which is an incredible value.

Figure 3: Time is controlled by Reference Clock in the “MainDevice Shift” mode

Using the “Bus Shift” mode, the timer of the MainDevice controls the Reference Clock. The system time is still provided by the much more precise SubDevice timer, but this timer will be adjusted with the cyclic frames from the MainDevice and set into the ESC register 0x0910. The SubDevice itself will now share(=distribute) this system time to the other SubDevices within the network. As displayed in Figure 4, the Reference Clock is still one of the SubDevices. The “Bus Shift” mode is the default mode of the EcMasterDemoDC and can be easily implemented into running systems.

Figure 4: Time is controlled by Reference Clock in the “Bus Shift” mode

The third mode is called “MainDevice Is Ref Clock” mode. In this mode, the MainDevice acts as DC Reference Clock and is providing the System Time in the cyclic frames to the SubDevices (see figure 5). The API function ecatDcmConfigure() gets the precise DC system time, so there is no DCM controller required, which lasts to a reduced CPU load. This mode is characterized by a high accuracy within a few nanoseconds but this mode also needs newest hardware timers with high resolution and accuracy.

Figure 5: MainDevice is acting as Reference Clock in the “MainDevice is Ref Clock” mode

A short overview of the advantages and requirements of the different modes are shown in figure 6 below.

Figure 6: Overview of the DCM modes

EC-Master stack supports all three different modes. The application EcMasterDemoDC is a perfect example how the Distributed Clock (DC) technology can be processed in the different synchronization modes. To start the application, following parameters  are used to run the application in DCM:

./EcMasterDemoDC -gem 1 1 rpi5 -f "eni.xml" -b 500 -dcmmode busshift -dcmlog -t 30000 -v 4 -perf

Snippet 1 Starting EcMasterDemoDC

After the parametrization of the Ethernet adapter (-gem) the eni.xml file is loaded from the application (-f “eni.xml”). The bus cycle time (-b) is set and now the specific DCM mode, which we described above, needs to be configured with the parameter -dcmmode. To get the relevant measurement data in the system, the DCM logging is activated (-dcmlog). For our tests, we define a 30 second period for logging all relevant data (-t 30000) and set the verbosity level to 4. Last but not least, we activate the performance measurement with the command -perf.

To setup and test the Distributed Clock (DC) functionalities, we defined the Compute Module 5 IO Board with the Raspberry Pi 5. EC-Master is running on a Linux system and we use our atemsys module to get direct access to our GEM-Ethernet driver.

We put three SubDevices into our network (see figure 6). Of course, we need to use SubDevices which support the DC functionality:

• AC Servo Drive Yaskawa Sigma
  • We use this servo drive as a Reference Clock. This device is the first SubDevice in the network.

• Beckhoff coupler EK1100 including following terminals:
  • 4x EtherCAT 4-channel digital output terminals (EL2002)
  • 3x EtherCAT 4-channel digital input terminals (EL1124)
  • 1x EtherCAT 2-channel analog input terminal with oversampling function (EL3702) and Distributed Clock functionality
  • EtherCAT extension (EK1110)

• Beckhoff EtherCAT Evaluation Kit (EL9820)
  • This development kit for EtherCAT SubDevices is equipped with several interfaces. It is perfect to develop on a platform using the ET9300 EtherCAT SubDevice Stack Code. This kit also provides the Distributed Clock functionality.

Figure 6: Test setup for Distributed Clock functionality

By scanning the network with EC-Engineer, we detect all SubDevices in the network. To activate the DC functionality, you first need to activate the functionality on the MainDevice tab. If you activate this option, the Reference Clock is automatically suggested as the first SubDevice (supporting DC) in the network. You can stay with that, but you can also select a different SubDevice as a Reference Clock. In addition, the required mode needs to be selected. As you see in figure 7, we selected the Yaskawa Servo Drive (SubDevice_1001) as our Reference Clock and selected the “Bus Shift” mode for our system.

Figure 7: Setting-up Distributed Clock functionality for MainDevice in the EC-Engineer

After setting the MainDevice configuration, we now need to configure the SubDevices. All SubDevices which are able to run DC also provide a separate tab to configure the DC settings. As an example, figure 8 shows the SubDevice configuration of the EtherCAT Evaluation Kit.

Figure 8: Setting-up Distributed Clock functionality for the SubDevice in the EC-Engineer

Now let’s start the EC-Master application EcMasterDemoDc. After creating the EtherCAT Network Information (ENI) file with the EC-Engineer, we safe this file on the Raspberry Pi 5 and we start the application with the commands as already shown in snippet 1 in the previous chapter. For our test, we use a network cycle time of 500 ms.

As shown in figure 9, we start the application in “Bus Shift” mode and we get the current status during the initialization phase of the EC-Master. After the initialization, the SubDevices are scanned within the network and then the system is synchronized. After the synchronization has been successfully finished, the EC-Master turns operational (OP).

Figure 9: Starting the EcMasterDemoDc

While starting up and running in “Bus Shift” mode, the DCM controller actively needs to control the Reference Clock. The deviation between MainDevice Clock and the Reference Clock is called control error (CtlErr). In a perfect world, this value would be zero, but in reality, this value is changing around the zero point, as shown in figure 10 below. The system is always busy to keep this value near zero. If the control error is too high, the speed of the Reference Clock needs to be speeded up/down depending on the current situation. A short sequence of approx. 30 sec is shown in figure 10 below. In this example of a Bus Shift mode, the clock needs to be adjusted mainly in the area of 20 µs.

Figure 10: DCM controlling of Controller Error in “Bus Shift” mode

To check the current status of synchronization, the EC-Engineer offers the following feature: After the connection via RAS-Server, we select the diagnosis mode. Now, we can have a look to each SubDevice which is running in Distributed Clock mode. As shown in figure 11, the EtherCAT Evaluation Kit (SubDevice 1012) is running with the bus cycle rate of 500ms. Currently, the SubDevice is synchronized within 719 ns to the Reference Clock (Yaskawa Servo Drive).

Figure 11: Online diagnosis with EC-Engineer in “Bus Shift” mode

If we want to configure the “MainDevice Shift” and the “MainDevice Is Ref Clock” mode in the EC-Engineer, we need to select the corresponding items in the MainDevice tab (see figure 12) and load the file onto the Raspberry Pi. To start the application, we use now the corresponding parameter -dcmmode mastershift or masterrefclock.

Figure 12: Setting-up Distributed Clock “MainDevice Shift” functionality for MainDevice in the EC-Engineer

Now we had some insights to the different DCM modes. But of course, we want also show some situations which appear in the real world in production environment or while setting up the system. We will have a look to three of the most common error scenarios which we face regularly in our support teams.

No synchronization

It is important, that the network cycle of the EC-Master is the same as the SubDevices defined in the ENI-file. To show this in an example, let’s run the application EcMasterDemoDc with 700µs while the system cycle time still remains at 500 µs. If you will have different cycle times, you will get synchronization error messages as displayed in figure 13. EC-Master detects that not all SubDevices are synchronized, so the Servo Drive will not turn operational and will stay in SAFEOP.

Figure 13: Synchronization error while starting-up EcMasterDemoDc

Drift too high/low

This error appears if the frequency between the timers does not fit to each other. This can be an electrical problem caused by an oscillator of a device or it can also be a software problem caused for example by inaccurate rounding of the timer.

The typical chart looks like in the figure 14 below. The drift between the MainDevice Clock and the Reference Clock is continuously increasing and the system is not able to stabilize the drift within the system. Within the 30 seconds of recording, the DCM is toggling between in sync / out of sync. But in the end, the drift of +/- 250 µs is too high  and the system is not be able to go operational.

Figure 14: DCM controlling error of a drift

Out of sync

This error is characterized by a typical curve which is shown in figure 15. During a running cycle, one or several events may occur which stresses the DCM in a way that the system is out of sync with a very high deviation between MainDevice Clock and the Reference Clock. But in the end, the DCM is able to synchronize the distributed clocks while the EC-Master is still operational.

Figure 15: DCM controlling of “out sync” error