Charting servo period clock jitter while running

More
18 May 2014 14:57 #47024 by DaBit
Occasionally I get a realtime error, and the text in the syslog is not very helpful:
In recent history there were
653944, 771452, 532944, 652396, and 651964
elapsed clocks between calls to the motion controller.
This time, there were 653188 which is so anomalously
large that it probably signifies a problem with your
realtime configuration.  For the rest of this run of
EMC, this message will be suppressed.

Compared to the other numbers 653188 clocks doesn't seem to be so anomalously large to me.

Another one:
In recent history there were
652056, 653344, 772080, 533296, and 652872
elapsed clocks between calls to the motion controller.
This time, there were 654140 which is so anomalously
large that it probably signifies a problem with your
realtime configuration.  For the rest of this run of
EMC, this message will be suppressed.

Same story. It doesn't seem to affect anything at all either, so I think these are false errors.
LinuxCNC 2.6.0~pre (but 2.7.0~pre0 behaves the same), 3.4.55-rtai-2 kernel, Asus P8Z77 mainboard, Celeron G550 processor.

With the latency test running a few days I cannot get latency above 5200, and that is already a spike. Some time ago I tried outputting a square wave on an LPT pin and monitoring that with a real oscilloscope. That resulted in a nice squarewave at half the servothread frequency, but modern digital oscilloscopes are blind 99% of the time.

- Can I chart the jitter in time elapsed between servo periods with LinuxCNC running? Using HALscope perhaps?
- Can I see what percentage of a servo period time slice was actually needed for processing?

Please Log in or Create an account to join the conversation.

More
18 May 2014 21:56 #47025 by PCW
motion.servo.last-period-ns

Also all the major functions in hal have time and tmax parameters
plotting these can be interesting...

Your numbers do look like theres a math error somewhere

Please Log in or Create an account to join the conversation.

More
19 May 2014 02:33 #47029 by DaBit
Is there a method to link a parameter (motion.servo.last-period-ns is a parameter, not a pin) to a net so I can plot it in halscope, feed it into a minmax component or dump it to file using sampler/halsampler?

When putting a watch on motion.servo.overruns and motion.servo.last-period-ns I see about a microsecond jitter on the servo period but no excessive values. The overrun counter increases in steps of 5 'sometimes'.

But the watch list in 'HAL configuration' is blind 99.9% of the time as well and with 'only' ~100 overruns per hour it won't catch it easily.

Please Log in or Create an account to join the conversation.

More
19 May 2014 03:13 #47030 by PCW
You can plot parameters just like pins

Please Log in or Create an account to join the conversation.

More
19 May 2014 21:27 #47049 by DaBit
If you select the 'parameters' tab, you can. If you select the 'pins'tab, you cannot.... :whistle: :blush:

I got results:



Once in a while there is a single +50usec spike followed by a -50usec spike: one servo period is delayed by 50usec. It never gets worse than this; had HALScope with a trigger on +302K for hours.

I'll see what causes this. Didn't play around with isolcpus/idle=poll/noirqbalance/SMI/etc. yet because I didn't have to. It is also strange that it seems to be exactly 50usec each and every time. The servo period does not matter; a 500usec servo period displays the same +50/-50usec peaks.
If I cannot catch and eliminate the root cause, I will leave it this way. Except for the realtime-error a few minutes after startup LinuxCNC operation seems to be completely unaffected.

The latency test does not catch these peaks BTW. LinuxCNC itself neither; I scanned /var/log/syslog and it's backups for similar numbers but didn't find them. It's all 'In recent history there were 526200, 653972, 649688, 654676, and 777580 elapsed clocks between calls to the motion controller. This time, there were 653804' and similar.

Please Log in or Create an account to join the conversation.

More
19 May 2014 22:48 #47057 by andypugh

- Can I chart the jitter in time elapsed between servo periods with LinuxCNC running? Using HALscope perhaps?


Possibly www.linuxcnc.org/docs/devel/html/man/man9/latencybins.9.html

Please Log in or Create an account to join the conversation.

Time to create page: 0.117 seconds
Powered by Kunena Forum