[Emc-users] Rough motion with servos

Date: Tue, 27 Jun 2017 15:01:34 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: [Emc-users] Rough motion with servos
I have been running my lathe using EMC for many years and decided to upgrade
to the latest LinuxCNC. The computer I am using is showing it's age and the
ancient version of Debian I have been running won't even boot properly on my
new computer.
The lathe has a 5i20 and a 7i29 to run the servos. Using the same PID
settings as I had in EMC I get odd random spikes in the Z axis motor command
signal, roughly about one to two a second. If I look at pid.z.output with
halscope I can see the spikes. This is on the same computer and hardware as I
was running EMC, which ran really smoothly. I changed the computer and the
results are the same. I even used Pncconf to create an absolute minimum spec
configuration and it does the same thing.
I tried both LinuxCNC 2.7.9 and Git head. Does anyone have any ideas why this
would be happening?
Les

Can you trace the PID output spike back to its source?

Maybe there's a encoder noise issue or something similar
(that may have changed signtly with a new PC due to grounding differences etc)

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Peter Wallace
Mesa Electronics

(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Dave Caroline

2017-06-27 15:00:58 UTC

New PC new possible latency problems?

Dave Caroline

Jon Elson

2017-06-27 15:15:38 UTC

Post by Les Newell
I have been running my lathe using EMC for many years and
decided to upgrade to the latest LinuxCNC. The computer I
am using is showing it's age and the ancient version of
Debian I have been running won't even boot properly on my
new computer.
The lathe has a 5i20 and a 7i29 to run the servos. Using
the same PID settings as I had in EMC I get odd random
spikes in the Z axis motor command signal, roughly about
one to two a second. If I look at pid.z.output with
halscope I can see the spikes. This is on the same
computer and hardware as I was running EMC, which ran
really smoothly. I changed the computer and the results
are the same. I even used Pncconf to create an absolute
minimum spec configuration and it does the same thing.
I tried both LinuxCNC 2.7.9 and Git head. Does anyone have
any ideas why this would be happening?

Have you run the latency tests? Are the numbers good? Any
time you change the hardware, OR kernel, you need to check
that the RT performance is acceptable.

Jon

Les Newell

2017-06-27 15:31:21 UTC

Same PC, same hardware, same settings. The only difference between the
two setups is the hard drive. Changing the PC (AMD processor VS Intel on
the original) had no effect on the symptoms. I checked latency on the
new PC and if I remember correctly worst case after a couple of hours
was ~150us. Not good for steppers but fine for servos.

I haven't had time to try tracing back from PID. That is probably my
next job.

Les

Post by Les Newell
I have been running my lathe using EMC for many years and decided to
upgrade to the latest LinuxCNC. The computer I am using is showing
it's age and the ancient version of Debian I have been running won't
even boot properly on my new computer.
The lathe has a 5i20 and a 7i29 to run the servos. Using the same PID
settings as I had in EMC I get odd random spikes in the Z axis motor
command signal, roughly about one to two a second. If I look at
pid.z.output with halscope I can see the spikes. This is on the same
computer and hardware as I was running EMC, which ran really
smoothly. I changed the computer and the results are the same. I even
used Pncconf to create an absolute minimum spec configuration and it
does the same thing.
I tried both LinuxCNC 2.7.9 and Git head. Does anyone have any ideas
why this would be happening?

Have you run the latency tests? Are the numbers good? Any time you
change the hardware, OR kernel, you need to check that the RT
performance is acceptable.
Jon
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Dave Caroline

2017-06-27 15:43:17 UTC

"The computer I am using is showing it's age and the ancient version
of Debian I have been running won't even boot properly on my new
computer." implies some change to us, what?, changing OS will change
the latency too

Dave

Les Newell

2017-06-27 16:07:10 UTC

Hi Dave,

Just ran latency test on the old computer with LinuxCNC 32us servo 54us
base. I moved some windows, opened a few programs etc. As I mentioned
before I had checked on the new computer and although it wasn't great it
was good enough. The fact that the fault is identical on two very
different computers tends to rule out the computer.

Les

Post by Dave Caroline
"The computer I am using is showing it's age and the ancient version
of Debian I have been running won't even boot properly on my new
computer." implies some change to us, what?, changing OS will change
the latency too
Dave
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Nicklas Karlsson

2017-06-27 16:28:53 UTC

Even though latency run well then I user computer for others at the same time I sometimes get a latency delay and are on the path to give up ordinary computer for servo threads. I think a small simpler computer will be more than enough and then I am free to move display.

On Tue, 27 Jun 2017 17:07:10 +0100

Post by Les Newell
Hi Dave,
Just ran latency test on the old computer with LinuxCNC 32us servo 54us
base. I moved some windows, opened a few programs etc. As I mentioned
before I had checked on the new computer and although it wasn't great it
was good enough. The fact that the fault is identical on two very
different computers tends to rule out the computer.
Les

Jon Elson

2017-06-27 16:19:42 UTC

Post by Les Newell
Same PC, same hardware, same settings. The only difference
between the two setups is the hard drive. Changing the PC
(AMD processor VS Intel on the original) had no effect on
the symptoms. I checked latency on the new PC and if I
remember correctly worst case after a couple of hours was
~150us. Not good for steppers but fine for servos.

150 uS?? YIKES! That is not generally good enough for even
a servo system, although Peter's PLL scheme probably makes
it usable, but not very good. With RTAI, I generally get
about 10 - 15 us on any machine I think is good enough to
use. On linux-preempt, it is more like 15-20 us. Note that
150 us is 15% of the 1 KHz servo cycle.

Post by Les Newell
I haven't had time to try tracing back from PID. That is
probably my next job.

First thing to do is bring up the encoder velocity, and look
for physically impossible velocity spikes. If the encoder
is being misread, or there are bad latency hiccups, you can
see spikes in the velocity that can't be real, that's an
immediate sign of some kind of difficulty between the
encoder shaft and the CPU. If you see them, then it can
take some detective work to determine where they are coming
from. If no spikes, then you may just need to retune the
PID. There could have been subtle changes in the way PID
works that make your old numbers less than optimum.

Jon

Les Newell

2017-06-27 18:50:18 UTC

150 uS?? YIKES! That is not generally good enough for even a servo
system

I have done a lot of work with PID loops in various non LinuxCNC systems
and in my general experience they generally seem to be very tolerant to
timing variation. After all we are dealing with motors that have a
bandwidth far lower than the PID loop speed. Anyway I have two boxes
here so I can run back-to-back tests at some point.

I think I may have found the problem. Pncconf doesn't connect
pid.n.feedback-deriv to the encoder velocity. The version of EMC I am
running doesn't have feedback-deriv so my config doesn't either. With no
connection, pid estimates the derivative and something funky is going on
with that estimation. Sometimes the machine twitches while stationary
which makes it a lot easier to see what is going on. Watching
feedback-deriv on halscope I can see it move at exactly the same moment
as the output jumps. A few milliseconds later the encoder velocity moves
in response.

After connecting the encoder velocity to feedback-deriv my tuning values
have to be very different but the twitching appears to have gone away.
The tuning at the moment is very rough with just P,D and FF used. I need
to experiment further and tune the axis properly before saying the
problem is definitely fixed. Now I need to try to figure out why my
jogwheels (one per axis) don't work.

Is there a reason why Pncconf doesn't connect the encoder velocity? By
the way Pncconf wouldn't let me set the maximum feed rate to more than
2000mm/min which is a bit annoying as the machine is capable of 4800.

Les

Les Newell

2017-06-27 20:06:45 UTC

Hmm, looks like that was a red herring. The problem is back. I can't do
any more testing because the pixies inside the old computer finally
abandoned ship and it now no longer boots. Doesn't even post. My
priority now is going to be to find a box that will run my old EMC.

Les

150 uS?? YIKES! That is not generally good enough for even a servo
system

I have done a lot of work with PID loops in various non LinuxCNC
systems and in my general experience they generally seem to be very
tolerant to timing variation. After all we are dealing with motors
that have a bandwidth far lower than the PID loop speed. Anyway I have
two boxes here so I can run back-to-back tests at some point.
I think I may have found the problem. Pncconf doesn't connect
pid.n.feedback-deriv to the encoder velocity. The version of EMC I am
running doesn't have feedback-deriv so my config doesn't either. With
no connection, pid estimates the derivative and something funky is
going on with that estimation. Sometimes the machine twitches while
stationary which makes it a lot easier to see what is going on.
Watching feedback-deriv on halscope I can see it move at exactly the
same moment as the output jumps. A few milliseconds later the encoder
velocity moves in response.
After connecting the encoder velocity to feedback-deriv my tuning
values have to be very different but the twitching appears to have
gone away. The tuning at the moment is very rough with just P,D and FF
used. I need to experiment further and tune the axis properly before
saying the problem is definitely fixed. Now I need to try to figure
out why my jogwheels (one per axis) don't work.
Is there a reason why Pncconf doesn't connect the encoder velocity? By
the way Pncconf wouldn't let me set the maximum feed rate to more than
2000mm/min which is a bit annoying as the machine is capable of 4800.
Les
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Jon Elson

2017-06-28 01:13:15 UTC

Post by Les Newell
Hmm, looks like that was a red herring. The problem is
back. I can't do any more testing because the pixies
inside the old computer finally abandoned ship and it now
no longer boots. Doesn't even post. My priority now is
going to be to find a box that will run my old EMC.

Whenever this happens, I reseat the memory sticks and maybe
even the CPU and that often brings it back.
Of course, it could have suffered the capacitor plague.

Jon

Les Newell

2017-06-28 08:50:38 UTC

OK, Tried the usual reseating components etc and that PC does not want
to play. I found a Dell SFF PC in the junk pile. After hacking it to
take a full size PCI card I fired it up. Running my old Debian +
LinuxCNC 2.6.0 it runs smooth. Latency jitter is 7us,10us. Running the
latest LinuxCNC image installed on an SSD it runs rough, latency jitter
is 37us,16us. 2.6.0 won't build on the latest image but I built 2.6.13
and it also runs rough, though maybe not quite as rough as 2.7.9.

Is there any way to install an older RTAI version?

Les

Post by Les Newell
Hmm, looks like that was a red herring. The problem is back. I can't
do any more testing because the pixies inside the old computer
finally abandoned ship and it now no longer boots. Doesn't even post.
My priority now is going to be to find a box that will run my old EMC.

Whenever this happens, I reseat the memory sticks and maybe even the
CPU and that often brings it back.
Of course, it could have suffered the capacitor plague.
Jon

Jon Elson

2017-06-28 15:49:42 UTC

Post by Les Newell
OK, Tried the usual reseating components etc and that PC
does not want to play. I found a Dell SFF PC in the junk
pile. After hacking it to take a full size PCI card I
fired it up. Running my old Debian + LinuxCNC 2.6.0 it
runs smooth. Latency jitter is 7us,10us. Running the
latest LinuxCNC image installed on an SSD it runs rough,
latency jitter is 37us,16us. 2.6.0 won't build on the
latest image but I built 2.6.13 and it also runs rough,
though maybe not quite as rough as 2.7.9.
Is there any way to install an older RTAI version?

I'm pretty sure some of the older ISO files are available,
but may not be listed directly on linuxcnc.org
Check with uname -a to see what kernel you are using, it
might be a preempt kernel, which will have worse latency
than rtai. But, 37 us shouldn't cause serious problems, so
I'm thinking it probably is not the kernel, but something to
do with the PID.

Jon

Jon Elson

2017-06-29 15:45:17 UTC

Just wanted to let everybody know about this deal:

Part # 33615 HD

http://www.mpja.com/06-29-17.asp?r=129492&s=16

The on line store shows $2.95, you might have to use
discount code "email SA".

Jon

Todd Zuercher

2017-06-27 16:39:08 UTC

What version of Linuxcnc (EMC2) were you running before the upgrade?
I think there were some changes in how PIDs were handled between v2.5 and 2.7 that usually require a little servo returning.

----- Original Message -----

Sent: Tuesday, June 27, 2017 10:01:34 AM
Subject: [Emc-users] Rough motion with servos
I have been running my lathe using EMC for many years and decided to
upgrade to the latest LinuxCNC. The computer I am using is showing
it's
age and the ancient version of Debian I have been running won't even
boot properly on my new computer.
The lathe has a 5i20 and a 7i29 to run the servos. Using the same PID
settings as I had in EMC I get odd random spikes in the Z axis motor
command signal, roughly about one to two a second. If I look at
pid.z.output with halscope I can see the spikes. This is on the same
computer and hardware as I was running EMC, which ran really
smoothly. I
changed the computer and the results are the same. I even used
Pncconf
to create an absolute minimum spec configuration and it does the same
thing.
I tried both LinuxCNC 2.7.9 and Git head. Does anyone have any ideas
why
this would be happening?
Les
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Todd Zuercher

2017-06-28 11:23:50 UTC

Yes, the latest v2.7.9 will run on the older Ubuntu 10.04 ISOs.
They can be down loaded here:
http://linuxcnc.org/iso/

----- Original Message -----
From: "Les Newell" <***@fastmail.co.uk>
To: "Enhanced Machine Controller (EMC)" <emc-***@lists.sourceforge.net>
Sent: Wednesday, June 28, 2017 4:50:38 AM
Subject: Re: [Emc-users] Rough motion with servos

OK, Tried the usual reseating components etc and that PC does not want
to play. I found a Dell SFF PC in the junk pile. After hacking it to
take a full size PCI card I fired it up. Running my old Debian +
LinuxCNC 2.6.0 it runs smooth. Latency jitter is 7us,10us. Running the
latest LinuxCNC image installed on an SSD it runs rough, latency jitter
is 37us,16us. 2.6.0 won't build on the latest image but I built 2.6.13
and it also runs rough, though maybe not quite as rough as 2.7.9.

Is there any way to install an older RTAI version?

Les

Post by Les Newell
Hmm, looks like that was a red herring. The problem is back. I can't
do any more testing because the pixies inside the old computer
finally abandoned ship and it now no longer boots. Doesn't even post.
My priority now is going to be to find a box that will run my old EMC.

Whenever this happens, I reseat the memory sticks and maybe even the
CPU and that often brings it back.
Of course, it could have suffered the capacitor plague.
Jon

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
Emc-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/emc-users

Todd Zuercher

2017-06-28 11:31:27 UTC

Yes, the latest v2.7.9 will run on the older Ubuntu 10.04 ISOs.
They can be down loaded here:
http://linuxcnc.org/iso/

Forgot to mention, You will need to change the old Ubuntu repo lines.

deb http://linuxcnc.org lucid base 2.7-rtai
deb http://old-releases.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-security main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse

----- Original Message -----
From: "Les Newell" <***@fastmail.co.uk>
To: "Enhanced Machine Controller (EMC)" <emc-***@lists.sourceforge.net>
Sent: Wednesday, June 28, 2017 4:50:38 AM
Subject: Re: [Emc-users] Rough motion with servos

Is there any way to install an older RTAI version?

Les

Les Newell

2017-06-28 16:51:51 UTC

Hi Todd,

I did that but I can't build because it needs Python >=2.7 and the
repositories only contain 2.6. I tried a very basic test config with
with the version that is shipped with 2.7.9 and it appeared to run
smoothly, at least once I figured out I needed to pet the watchdog. I'm
not going to admit how long it took me to figure that one out :-[
I can't run my lathe properly because it depends on a bunch of custom
modules which I can't sort out until I can build LinuxCNC.

Les

Post by Todd Zuercher
Yes, the latest v2.7.9 will run on the older Ubuntu 10.04 ISOs.
http://linuxcnc.org/iso/
Forgot to mention, You will need to change the old Ubuntu repo lines.
deb http://linuxcnc.org lucid base 2.7-rtai
deb http://old-releases.ubuntu.com/ubuntu/ lucid main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-security main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
----- Original Message -----
Sent: Wednesday, June 28, 2017 4:50:38 AM
Subject: Re: [Emc-users] Rough motion with servos
Is there any way to install an older RTAI version?
Les
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Todd Zuercher

2017-06-28 17:49:26 UTC

Are you trying to build Linuxcnc from source or installing/updating LInuxcnc from Buildbot?
I'm still not sure what distro you are working with. (You seem to have talked about several.)

I have not tried to build from source on Lucid for a very long time. (But I do recall having a great deal of difficulty meeting all the dependencies.)

Are you sure all of your custom modules are still "custom", or needed? There have been a lot of little goodies added to Linuxcnc since 2.5-2.6 days.
Maybe they can be added using halcompile (formerly known as comp)?

----- Original Message -----
From: "Les Newell" <***@fastmail.co.uk>
To: "Enhanced Machine Controller (EMC)" <emc-***@lists.sourceforge.net>
Sent: Wednesday, June 28, 2017 12:51:51 PM
Subject: Re: [Emc-users] Rough motion with servos

Hi Todd,

I did that but I can't build because it needs Python >=2.7 and the
repositories only contain 2.6. I tried a very basic test config with
with the version that is shipped with 2.7.9 and it appeared to run
smoothly, at least once I figured out I needed to pet the watchdog. I'm
not going to admit how long it took me to figure that one out :-[
I can't run my lathe properly because it depends on a bunch of custom
modules which I can't sort out until I can build LinuxCNC.

Les

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
Emc-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/emc-users

Les Newell

2017-06-28 18:26:52 UTC

Hi Todd,

I am building from source. I have tried several distros:
Debian - something old, I can't remember what. This has been running
fine for about 8 years but it lacks driver support for modern hardware.
Ubuntu 10.04 LinuxCNC ISO - appears to run smoothly but I can't test
properly until I get LCNC to build. It's also a bit long in the tooth.
Ubuntu 16.04 LinuxCNC ISO - runs rough but I can build fine. I've tried
various different branches of LinuxCNC, including one that is close to
what I was originally running, on this and they all run rough.

Post by Todd Zuercher
Are you sure all of your custom modules are still "custom", or needed? There have been a lot of little goodies added to Linuxcnc since 2.5-2.6 days.

Yes, some of the modules do have fairly direct replacements, such as my
original Modbus component and the new MB2HAl (which is based on my
original). However I have extra stuff to control my spindle and
electronically controlled gearbox which don't have direct replacements.

Post by Todd Zuercher
Maybe they can be added using halcompile (formerly known as comp)?

Todd Zuercher

2017-06-28 18:37:47 UTC

If you install from buildbot, both linuxcnc and linuxcnc-dev I am pretty sure that is all you need to be able to build hal components using halcompile.
You don't have to have all of the resources to build all of Linuxcnc from source.

----- Original Message -----
From: "Les Newell" <***@fastmail.co.uk>
To: "Enhanced Machine Controller (EMC)" <emc-***@lists.sourceforge.net>
Sent: Wednesday, June 28, 2017 2:26:52 PM
Subject: Re: [Emc-users] Rough motion with servos

Hi Todd,

I am building from source. I have tried several distros:
Debian - something old, I can't remember what. This has been running
fine for about 8 years but it lacks driver support for modern hardware.
Ubuntu 10.04 LinuxCNC ISO - appears to run smoothly but I can't test
properly until I get LCNC to build. It's also a bit long in the tooth.
Ubuntu 16.04 LinuxCNC ISO - runs rough but I can build fine. I've tried
various different branches of LinuxCNC, including one that is close to
what I was originally running, on this and they all run rough.

Post by Todd Zuercher
Are you sure all of your custom modules are still "custom", or needed? There have been a lot of little goodies added to Linuxcnc since 2.5-2.6 days.

Post by Todd Zuercher
Maybe they can be added using halcompile (formerly known as comp)?

Yes, most are comps but I still need a working build environment.

I am wondering if now would be a good time to move away from RTAI and
use preempt-rt instead. My reasoning behind that is that I have two
projects coming up that use HM2 ethernet boards so I'll have to work
with preempt-rt at some point. Keeping everything on the same platform
will simplify maintenance in the future.

Les

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
Emc-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/emc-users

Les Newell

2017-06-28 18:52:03 UTC

Hi Todd,

Does buildbot go back to 10.04 times? I thought it was only for recent
builds.

Post by Todd Zuercher
Thanks for your contributions that became MB2HAL. I'm using it.

You are welcome. If I manage to get a recent version of LCNC running
you'll probably see some changes to mb2hal. For a start I want to look
into why it uses so much CPU. Scanning 55 inputs and 55 outputs plus a
few registers at 20Hz ends up using 25% of the CPU which seems excessive.

I've just built a preempt-rt kernel for Ubuntu 16.04. I might have just
enough time tonight to download and build LinuxCNC Git master. It will
be interesting to see how it behaves under preempt-rt.

Les

Post by Todd Zuercher
If you install from buildbot, both linuxcnc and linuxcnc-dev I am pretty sure that is all you need to be able to build hal components using halcompile.
You don't have to have all of the resources to build all of Linuxcnc from source.
----- Original Message -----
Sent: Wednesday, June 28, 2017 2:26:52 PM
Subject: Re: [Emc-users] Rough motion with servos
Hi Todd,
Debian - something old, I can't remember what. This has been running
fine for about 8 years but it lacks driver support for modern hardware.
Ubuntu 10.04 LinuxCNC ISO - appears to run smoothly but I can't test
properly until I get LCNC to build. It's also a bit long in the tooth.
Ubuntu 16.04 LinuxCNC ISO - runs rough but I can build fine. I've tried
various different branches of LinuxCNC, including one that is close to
what I was originally running, on this and they all run rough.

Post by Todd Zuercher
Are you sure all of your custom modules are still "custom", or needed? There have been a lot of little goodies added to Linuxcnc since 2.5-2.6 days.

Post by Todd Zuercher
Maybe they can be added using halcompile (formerly known as comp)?

Yes, most are comps but I still need a working build environment.
I am wondering if now would be a good time to move away from RTAI and
use preempt-rt instead. My reasoning behind that is that I have two
projects coming up that use HM2 ethernet boards so I'll have to work
with preempt-rt at some point. Keeping everything on the same platform
will simplify maintenance in the future.
Les
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Les Newell

2017-06-29 12:42:35 UTC

I tried using Buildbot on a fresh install of Stretch amd64 and got a
bunch of missing dependencies:
python-gst0.10 (Stretch has python-gst1.0)
gstreamer0.10-plugins-base
hostmot2-firmware-all

Les

Les Newell

2017-06-29 15:20:16 UTC

Well, some progress figuring out what is going on.

I just ran some more tests using Ubuntu 16.04 with a freshly built
preempt-rt kernel and freshly built Git master. No surprise, it still
shows the fault. I took a Halscope log from the original setup. The
encoder velocity is rather noisy but no big spikes. On the new install
the encoder velocity is not as noisy but it has occasional spikes. The
encoders are wired to a 7i29.
This log is from the old setup <www.sheetcam.com/lathe/old.log>
This log is from the new setup <www.sheetcam.com/lathe/spike.log>

This lathe is fitted with high resolution handwheel encoders so I also
did something I should have done much sooner and looked at the handwheel
encoder velocity. Big difference - nice clean signals on both configs.

So it looks like something funky is going on when reading encoders
with the 7i29, resulting in noisy velocity estimation. I guess the
velocity estimation code has changed a bit over the years, giving the
different results. The fault has probably been there for a long time,
quite possibly since I built it. My immediate thought is noise on the
encoder signals but this machine never loses position. It will turn out
accurate parts all day. Peter, any idea how the velocity estimation can
be noisy while the overall counts seem to track accurately?

Some time back I was involved with developing flight controllers for RC
models. They used nested PID loops. The loops worked well even though
the accelerometer and gyro feedback signals were extremely noisy. I
think that explains why my old config seems to run smoothly. The signal
is noisy but consistent so the noise averages out. On the later config
the velocity estimation ends up with spikes which are noticeable in motion.

Les

Peter C. Wallace

2017-06-29 15:52:49 UTC

Date: Thu, 29 Jun 2017 16:20:16 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos
Well, some progress figuring out what is going on.
I just ran some more tests using Ubuntu 16.04 with a freshly built preempt-rt
kernel and freshly built Git master. No surprise, it still shows the fault. I
took a Halscope log from the original setup. The encoder velocity is rather
noisy but no big spikes. On the new install the encoder velocity is not as
noisy but it has occasional spikes. The encoders are wired to a 7i29.
This log is from the old setup <www.sheetcam.com/lathe/old.log>
This log is from the new setup <www.sheetcam.com/lathe/spike.log>
This lathe is fitted with high resolution handwheel encoders so I also did
something I should have done much sooner and looked at the handwheel encoder
velocity. Big difference - nice clean signals on both configs.
So it looks like something funky is going on when reading encoders with the
7i29, resulting in noisy velocity estimation. I guess the velocity estimation
code has changed a bit over the years, giving the different results. The
fault has probably been there for a long time, quite possibly since I built
it. My immediate thought is noise on the encoder signals but this machine
never loses position. It will turn out accurate parts all day. Peter, any
idea how the velocity estimation can be noisy while the overall counts seem
to track accurately?

Do you have and raw-write encoder filter tweaking stuff in the hal file:

The bitfile may have changed so the address is no longer correct (plus newwer
version of the driver let you set these as hal pins)

Note the noise in the encoder signals can generate nasty velocity noise
without losing counts

Some time back I was involved with developing flight controllers for RC
models. They used nested PID loops. The loops worked well even though the
accelerometer and gyro feedback signals were extremely noisy. I think that
explains why my old config seems to run smoothly. The signal is noisy but
consistent so the noise averages out. On the later config the velocity
estimation ends up with spikes which are noticeable in motion.
Les
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Peter Wallace
Mesa Electronics

(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Les Newell

2017-06-29 20:59:46 UTC

No. For testing purposes I am running the barest minimum needed to
achieve motion using a config generated by pncconf.

After spending a lot of time playing with halscope I have some
interesting findings. Most of what I thought was noise is actually phase
variation in the encoder counts. There is a cycle that repeats every 4
counts. When zoomed out of course it just looks like random hash. I
checked by monitoring the encoder counts and velocity at the same time.
I could clearly see the cycle repeating every 4 steps in the encoder count.

Now for the really weird bit. It didn't sink in before but the spikes
are always positive, no matter what direction I am moving! in other
words if I move in the positive direction the spikes show increasing
velocity. If I move in the negative direction the spikes show decreasing
velocity. I have a log that shows one spike reaching zero velocity for
one encoder count. I have another log when moving in the other direction
where the spike is nearly double velocity for one count. However in both
cases the width of the encoder count is pretty much the same as nearby
counts. I can't think of any way that electrical noise could give this
result. If anyone wants to see the logs I'll upload them tomorrow.

Not easy. The screw is difficult to get to and I can only turn it about
1/4 turn at a time.

Mesa does have some adjustments to the digital filter settings that
you could try,

I tried the turning the filter on and off and it made no discernible
difference.

or you may need to change the encoder cables, set up the shields
differently (you ARE using shielded cables, I hope!) of possibly have
to go to differential drive at the encoder end, with differential
receivers at the Mesa end.

I can't see it being pickup in the encoder cable. It's only about 18"
long and doesn't run near any noisy wiring, apart from a couple of
inches near the 7i29. It is the original screened cable fitted by the
manufacturer of the encoder. Apart from that I don't see how noise could
produce the results I am seeing.

Peter C. Wallace

2017-06-29 21:29:34 UTC

Date: Thu, 29 Jun 2017 21:59:46 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos
No. For testing purposes I am running the barest minimum needed to achieve
motion using a config generated by pncconf.

If you did this before I would make sure you duplicate this

The default encoder filter is quite short even with the filter bit on

On a 5I20 its 15x30ns (filter bit on) so only 450 ns which is in the range of
coupled impulse noise spikes from the 7I29

With current LinuxCNC dists (2.7 and 2.8) you can lower the encoder sampling
frequency by setting hm2_card.0.encoder.sample-frequency

If you set this to say 5000000, This will result in a 15*200ns =3 usec filter
on all encoder inputs that have their filter bit on, This will limit the
maximum count frequency to about 666 KHz and be much more resistant to
coupled impulse noise from the motor PWM
Peter Wallace
Mesa Electronics

(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Les Newell

2017-06-30 14:12:31 UTC

OK, this is embarrassing. It is noise and it has always been there.
Anyone got any spare crows for me to eat? :-[

Post by Peter C. Wallace
The bitfile may have changed so the address is no longer correct (plus
newwer version of the driver let you set these as hal pins)

You are correct. I had another trawl through the original hal files and
found something that looks suspiciously like it tweaks the filter by
directly writing to registers. I commented it out and the old setup
behaves in a very similar way.

As far as I can tell the problem is an earth loop between the drive and
the computer. The drive power supply and computer are both earthed to
the same star point but the ribbon cable also joins the drive and
computer earths, forming a loop. I have now installed the computer I
originally intended to use, which is a mini fanless pc powered by a
double insulated power brick and the noise has gone. It runs perfectly
with no filtering. The computer only has a mini PCI-e slot so I am using
a mini PCI-e to PCI riser, with the 5I20 mounted in a separate frame
beside the computer. If I had wanted to keep the original PC I probably
could have fixed it by connecting the PC earth directly to the drive
earth instead of to the star point. Earthing can be fun sometimes.

It's ironic really. After I found the original installation wouldn't
boot on the new computer I decided to work incrementally, doing a new
install on the old computer then working from there. If I'd just said
stuff it and installed the new version on the new computer it would have
worked fine...

Well, at least I have now proved you can use a 5i20 on a new computer
using PCIE or mini PCIE.
This is the PCI adapter
<http://www.ebay.co.uk/itm/PCI-E-Express-X1-to-Dual-PCI-Riser-Extend-Adapter-Card-With-2-6-FT-USB3-0-Cable/131861478412>.
They aren't available with a mPCIE plug so I had to buy a m-PCIE to PCIE
adapter like this one:
<http://www.ebay.co.uk/itm/PCI-E-USB-3-0-Express-1x-To16x-Mini-Extender-Riser-Card-SATA-Adapter-Power-Cable/282475427399>
just for the mPCIE plug and cable.

Les

Date: Thu, 29 Jun 2017 21:59:46 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos
No. For testing purposes I am running the barest minimum needed to
achieve motion using a config generated by pncconf.

If you did this before I would make sure you duplicate this
The default encoder filter is quite short even with the filter bit on
On a 5I20 its 15x30ns (filter bit on) so only 450 ns which is in the
range of coupled impulse noise spikes from the 7I29
With current LinuxCNC dists (2.7 and 2.8) you can lower the encoder
sampling frequency by setting hm2_card.0.encoder.sample-frequency
If you set this to say 5000000, This will result in a 15*200ns =3 usec
filter on all encoder inputs that have their filter bit on, This will
limit the maximum count frequency to about 666 KHz and be much more
resistant to
coupled impulse noise from the motor PWM
Peter Wallace
Mesa Electronics
(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Gene Heskett

2017-06-30 20:12:07 UTC

Post by Les Newell
OK, this is embarrassing. It is noise and it has always been there.
Anyone got any spare crows for me to eat? :-[

Post by Peter C. Wallace
Do you have and raw-write encoder filter tweaking stuff in the hal
file: The bitfile may have changed so the address is no longer
correct (plus newwer version of the driver let you set these as hal
pins)

You are correct. I had another trawl through the original hal files
and found something that looks suspiciously like it tweaks the filter
by directly writing to registers. I commented it out and the old setup
behaves in a very similar way.
As far as I can tell the problem is an earth loop between the drive
and the computer. The drive power supply and computer are both earthed
to the same star point but the ribbon cable also joins the drive and
computer earths, forming a loop. I have now installed the computer I
originally intended to use, which is a mini fanless pc powered by a
double insulated power brick and the noise has gone. It runs perfectly
with no filtering. The computer only has a mini PCI-e slot so I am
using a mini PCI-e to PCI riser, with the 5I20 mounted in a separate
frame beside the computer. If I had wanted to keep the original PC I
probably could have fixed it by connecting the PC earth directly to
the drive earth instead of to the star point. Earthing can be fun
sometimes.
It's ironic really. After I found the original installation wouldn't
boot on the new computer I decided to work incrementally, doing a new
install on the old computer then working from there. If I'd just said
stuff it and installed the new version on the new computer it would
have worked fine...
Well, at least I have now proved you can use a 5i20 on a new computer
using PCIE or mini PCIE.
This is the PCI adapter
<http://www.ebay.co.uk/itm/PCI-E-Express-X1-to-Dual-PCI-Riser-Extend-A
dapter-Card-With-2-6-FT-USB3-0-Cable/131861478412>. They aren't
available with a mPCIE plug so I had to buy a m-PCIE to PCIE adapter
<http://www.ebay.co.uk/itm/PCI-E-USB-3-0-Express-1x-To16x-Mini-Extende
r-Riser-Card-SATA-Adapter-Power-Cable/282475427399> just for the mPCIE
plug and cable.
Les

Les, when I have something like that, I often ask myself if it was worth
chewing thru the straps to get up and pee, last Monday morning. In
retrospect, the answer should have been no, several times now.

But I am glad you have it sorted. Is the 4 step pattern gone, or is it
the same even though it Just Works(TM) now?

Date: Thu, 29 Jun 2017 21:59:46 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos
No. For testing purposes I am running the barest minimum needed to
achieve motion using a config generated by pncconf.

If you did this before I would make sure you duplicate this
The default encoder filter is quite short even with the filter bit on
On a 5I20 its 15x30ns (filter bit on) so only 450 ns which is in the
range of coupled impulse noise spikes from the 7I29
With current LinuxCNC dists (2.7 and 2.8) you can lower the encoder
sampling frequency by setting hm2_card.0.encoder.sample-frequency
If you set this to say 5000000, This will result in a 15*200ns =3
usec filter on all encoder inputs that have their filter bit on,
This will limit the maximum count frequency to about 666 KHz and be
much more resistant to
coupled impulse noise from the motor PWM
Peter Wallace
Mesa Electronics
(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.
--------------------------------------------------------------------
----------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

----------------------------------------------------------------------
-------- Check out the vibrant tech community on one of the world's
most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Cheers, Gene Heskett

--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Les Newell

2017-06-30 21:36:25 UTC

Dammit, for some reason when I reply to messages on this list they often
end up going to the author of the message rather than to the list. Sorry
Gene, you'll probably end up receiving two copies of this.

Post by Gene Heskett
Les, when I have something like that, I often ask myself if it was worth
chewing thru the straps to get up and pee, last Monday morning. In
retrospect, the answer should have been no, several times now.

I have to admit I was beginning to lose the will to live.

Post by Gene Heskett
But I am glad you have it sorted. Is the 4 step pattern gone, or is it
the same even though it Just Works(TM) now?

The step pattern is still there. I can't say I can detect any issues in
operation so I'm not worrying about it.

One pleasant side effect of retuning is that I found I can crank Z up
from 4.8m/min to 6m/min (it starts faulting at 7m/min). I don't know if
I was just very conservative last time or if this is a result of the
changes. Either way it is good news. Z has always been a bit slow on
this machine.

Les

Jon Elson

2017-06-30 01:36:23 UTC

Post by Peter C. Wallace
Do you have and raw-write encoder filter tweaking stuff

I think it is time to put a scope on the encoder. it sounds
like there may be a big duty cycle or phase angle error in
that encoder. In other words, at constant speed, the 4
quadrature transitions are not evenly spaced, but at least
one of them is out of time.

Post by Les Newell
Now for the really weird bit. It didn't sink in before but
the spikes are always positive, no matter what direction I
am moving! in other words if I move in the positive
direction the spikes show increasing velocity. If I move
in the negative direction the spikes show decreasing
velocity. I have a log that shows one spike reaching zero
velocity for one encoder count. I have another log when
moving in the other direction where the spike is nearly
double velocity for one count. However in both cases the
width of the encoder count is pretty much the same as
nearby counts. I can't think of any way that electrical
noise could give this result. If anyone wants to see the
logs I'll upload them tomorrow.

Post by Peter C. Wallace
Assuming the spikes happen fairly often, you should try
setting up Halscope to display velocity and trigger on
where you think a spike might reach, and then move the
axis by hand for a while and see if the scope triggers.

Not easy. The screw is difficult to get to and I can only
turn it about 1/4 turn at a time.

With a little hacking of Hal commands, you should be able to
set up a roughly constant speed on the motor, running open
loop. Or, even run closed loop and just hook the scope to
the encoder signals.

Post by Les Newell
I can't see it being pickup in the encoder cable. It's
only about 18" long and doesn't run near any noisy wiring,
apart from a couple of inches near the 7i29. It is the
original screened cable fitted by the manufacturer of the
encoder. Apart from that I don't see how noise could
produce the results I am seeing.

OK, I did not know what your setup was there.

Jon

Les Newell

2017-06-30 11:11:09 UTC

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that encoder. In
other words, at constant speed, the 4 quadrature transitions are not
evenly spaced, but at least one of them is out of time.

Gene Heskett

2017-06-30 13:08:03 UTC

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that encoder.
In other words, at constant speed, the 4 quadrature transitions are
not evenly spaced, but at least one of them is out of time.

I agree Les, and I've made a filter out of some hal components that sends
as output, the sum of the last 4 velocity samples to the spindles PID as
the usual feedback. This reduces that noise to about 25% of its raw
value, and for the spindle of the G0704, allowed me to raise the P quite
a bit. When there is a 5% timing error in the quadrature, it slams the
PID output, sometimes from rail to rail, making Pgains above 1 unusable.
With the filtering, a Pgain of 1.75 is still slaming the gearing backlash
in the head unless its heavily loaded. But the noise isn't a bit
rythmic, so I am thinking a head teardown and replacement of the roller
skate quality bearings in it with decent bearings is in my future. From
the .ini:

[SPINDLE_9]
P = 1.75
I = 0.05
D = 0.002
FF0 = 1.00
FF1 = 0.085
FF2 = 0.00025

No clue if its correct, but its usable, and its quite stiff loading wise.

Makes lot of racket and is beating the cheap head bearings to
smithereens. This is an optical encoder with 268 edges. On an external
scope it looks pretty good, but the 5i25's encoder velocity output is
horrible. A similar encoder on TLM, I even went to the trouble of making
a per opto-interrupter led dimmer so I could fine tune the duty cycle to
minimize the amplitude velocity variation and the 4 cycle noise, but it
cannot be equalized to a smooth line. So thats the first of my machines
to get that averaging filter. That allows a Pgain of 3, and its quite
stiff in terms of load vs speed.

Cheers, Gene Heskett

--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Stephen Dubovsky

2017-06-30 13:14:39 UTC

May I opine that noise on one of the encoder lines may cause a zero speed
estimation. If you get two of the same channel edges in a row the encoder
logically must have reversed direction. It must go through zero speed to
change direction. Since a third edge (one on the other channel) is not yet
detected to know true speed in the other direction, zero is the best
estimate. Not sure if LinuxCNC does it this way but you can indeed detect
a 'zero' speed even without long pulses if purely looking at the
information the edges are telling you. You might have noise?

I think it is time to put a scope on the encoder. it sounds like

there may be a big duty cycle or phase angle error in that encoder. In
other words, at constant speed, the 4 quadrature transitions are not
evenly spaced, but at least one of them is out of time.

I can see the phase angle errors in halscope by looking at the measured
velocity at low speed. There is a regular 4 step cycle. I can see the same
effect on the other encoders, though it is not as pronounced. All encoders
have some phase angle error but I have to admit this one is worse than I
would like. The problem is compounded by it being a fairly low resolution
encoder. However this does not explain the occasional big spikes,
especially the example where I had a spike of zero velocity for one count.
The machine was moving at low speed so I could see individual counts in
Halscope. To read zero velocity that count should have been MUCH wider than
the others and it was not. There has to be something wrong with the the way
the velocity is calculated.
Les
------------------------------------------------------------
------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Peter C. Wallace

2017-06-30 13:27:29 UTC

Date: Fri, 30 Jun 2017 12:11:09 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that encoder. In
other words, at constant speed, the 4 quadrature transitions are not
evenly spaced, but at least one of them is out of time.

I dont think so, the velocity calculation in the driver has not changed for
many years, and electrical noise can simulate a reversal which will cause a 0
velocity reading...

Peter Wallace
Mesa Electronics

(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Gene Heskett

2017-06-30 16:22:24 UTC

Date: Fri, 30 Jun 2017 12:11:09 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that encoder.
In other words, at constant speed, the 4 quadrature transitions are
not evenly spaced, but at least one of them is out of time.

I dont think so, the velocity calculation in the driver has not
changed for many years, and electrical noise can simulate a reversal
which will cause a 0 velocity reading...

This has been a very consistent error for years Peter. And I've been
sitting here, with halscope running remotely on this machine for the
last hour watching it, and just had a thought. This consistency could be
explained if there was a difference in the gain as applied to the
velocity averager of about 2/1 between the a and b inputs. The timing,
and the resultant error I am seeing right now, with the spindle turning
around 95 rpms, based on hall effect devices on the 60 tooth bull gear,
is telling me the velocity is stepping over a 2 division (400mv p-p)
range centered on nominally 1.2 volts. The shape of this 4 step pattern
can be explained by a gain difference in the a vs b swings. Is this
something we've been ignoring for a decade or more? I can bend the
opto's a tad on TLM, getting the phasing as perfect as I can, and
adjusting the leds for a 50% duty cycle. But the velocity output on TLM,
triggered on the index, looks very similar regardless of which of the 3
machines I can look at. The velocity is consistently low while the b
input is high, and consistently high while the b input is low.

And of course if I crank it up to 1000 rpms, quantization noise is the
dominant noise in the a/b pattern, but the general shape of the velocity
error only changes because of the rise time of the halscope.

It may be Peter, that my encoders are junk, but why are they so
consistently junk in the same repeatable direction pattern? Optical or
hall effect?

--------------------------------------------------------------------
---------- Check out the vibrant tech community on one of the world's
most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Peter Wallace
Mesa Electronics
(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.
----------------------------------------------------------------------
-------- Check out the vibrant tech community on one of the world's
most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Cheers, Gene Heskett

--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Peter C. Wallace

2017-06-30 16:45:55 UTC

Date: Fri, 30 Jun 2017 12:22:24 -0400
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

Date: Fri, 30 Jun 2017 12:11:09 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that encoder.
In other words, at constant speed, the 4 quadrature transitions are
not evenly spaced, but at least one of them is out of time.

I dont think so, the velocity calculation in the driver has not
changed for many years, and electrical noise can simulate a reversal
which will cause a 0 velocity reading...

This has been a very consistent error for years Peter. And I've been
sitting here, with halscope running remotely on this machine for the
last hour watching it, and just had a thought. This consistency could be
explained if there was a difference in the gain as applied to the
velocity averager of about 2/1 between the a and b inputs. The timing,
and the resultant error I am seeing right now, with the spindle turning
around 95 rpms, based on hall effect devices on the 60 tooth bull gear,
is telling me the velocity is stepping over a 2 division (400mv p-p)
range centered on nominally 1.2 volts. The shape of this 4 step pattern
can be explained by a gain difference in the a vs b swings. Is this
something we've been ignoring for a decade or more? I can bend the
opto's a tad on TLM, getting the phasing as perfect as I can, and
adjusting the leds for a 50% duty cycle. But the velocity output on TLM,
triggered on the index, looks very similar regardless of which of the 3
machines I can look at. The velocity is consistently low while the b
input is high, and consistently high while the b input is low.
And of course if I crank it up to 1000 rpms, quantization noise is the
dominant noise in the a/b pattern, but the general shape of the velocity
error only changes because of the rise time of the halscope.
It may be Peter, that my encoders are junk, but why are they so
consistently junk in the same repeatable direction pattern? Optical or
hall effect?

Not sure, but the velocity average is nearly perfect with perfect quadrature
and the errors are proportional to the quadrature error as expected.
I have measured this by using a stepgen running in table mode fed back into
an encoder in table mode and making a 16 step long table with a 1 step
quadrature error (3/5 steps from edge to edge instead of 4/4), in this case
you get the expected 1/3 --> 1/5 velocity steps

This does show that a relatively small quadrature error causes a large
velocity modulation is the relative time between edge changes so much

--------------------------------------------------------------------
---------- Check out the vibrant tech community on one of the world's
most engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Cheers, Gene Heskett
--
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Peter Wallace
Mesa Electronics

(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Gene Heskett

2017-06-30 20:59:19 UTC

Date: Fri, 30 Jun 2017 12:22:24 -0400
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

Date: Fri, 30 Jun 2017 12:11:09 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that
encoder. In other words, at constant speed, the 4 quadrature
transitions are not evenly spaced, but at least one of them is
out of time.

I dont think so, the velocity calculation in the driver has not
changed for many years, and electrical noise can simulate a
reversal which will cause a 0 velocity reading...

This has been a very consistent error for years Peter. And I've been
sitting here, with halscope running remotely on this machine for the
last hour watching it, and just had a thought. This consistency
could be explained if there was a difference in the gain as applied
to the velocity averager of about 2/1 between the a and b inputs.
The timing, and the resultant error I am seeing right now, with the
spindle turning around 95 rpms, based on hall effect devices on the
60 tooth bull gear, is telling me the velocity is stepping over a 2
division (400mv p-p) range centered on nominally 1.2 volts. The
shape of this 4 step pattern can be explained by a gain difference
in the a vs b swings. Is this something we've been ignoring for a
decade or more? I can bend the opto's a tad on TLM, getting the
phasing as perfect as I can, and adjusting the leds for a 50% duty
cycle. But the velocity output on TLM, triggered on the index, looks
very similar regardless of which of the 3 machines I can look at.
The velocity is consistently low while the b input is high, and
consistently high while the b input is low.
And of course if I crank it up to 1000 rpms, quantization noise is
the dominant noise in the a/b pattern, but the general shape of the
velocity error only changes because of the rise time of the
halscope.
It may be Peter, that my encoders are junk, but why are they so
consistently junk in the same repeatable direction pattern? Optical
or hall effect?

Not sure, but the velocity average is nearly perfect with perfect
quadrature and the errors are proportional to the quadrature error as
expected. I have measured this by using a stepgen running in table
mode fed back into an encoder in table mode and making a 16 step long
table with a 1 step quadrature error (3/5 steps from edge to edge
instead of 4/4), in this case you get the expected 1/3 --> 1/5
velocity steps
This does show that a relatively small quadrature error causes a large
velocity modulation is the relative time between edge changes so much

That appears to also be true, but I don't have a perfect siggen. The
variations in timeing seen on the halscope are of course granular in
nature because the sample read from the card is non-sync to the encoders
input pulses, so moves the apparent edge times seen back and forth by
the thread period, which is nominally 1 millisecond, but this IS an r-pi
3b running locked at 1.2 Ghz, however putting it back in demand mode
makes little if any difference except in the heat. Latency overruns >10x
the 1ms servo-thread are 10 cents a dozen either way. I've the
recommended small heat sinks on it, and a video card fan running on 5
volts blowing directly on the sinks from about 3/8" away.

Now if we just had a kernel that could do decent latency on the armhf,
unfortunately we haven't had one of those critters since 2.6.20 on the
32 bit intel chips. And that predates the armhf's so we may as well
forget about it until we can build our own real time kernel that
actually works. What we have that are called real time by the builder,
are a far cry from it, good latency is 100+ microseconds, and get the
performance they claim by ignoreing 9 out of 10 keyboard or mouse
events. Miss a key up or mouse button release, and its game over
because you just smashed up both the workpiece, and the tool in the tool
post by running it into something, like a chuck jaw turning 1500 revs.

That can get expensive. Fast.

Post by Peter C. Wallace
Peter Wallace
Mesa Electronics
(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Cheers, Gene Heskett

--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Peter C. Wallace

2017-06-30 21:29:39 UTC

Date: Fri, 30 Jun 2017 16:59:19 -0400
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

Date: Fri, 30 Jun 2017 12:22:24 -0400
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

Date: Fri, 30 Jun 2017 12:11:09 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that
encoder. In other words, at constant speed, the 4 quadrature
transitions are not evenly spaced, but at least one of them is
out of time.

I dont think so, the velocity calculation in the driver has not
changed for many years, and electrical noise can simulate a
reversal which will cause a 0 velocity reading...

This has been a very consistent error for years Peter. And I've been
sitting here, with halscope running remotely on this machine for the
last hour watching it, and just had a thought. This consistency
could be explained if there was a difference in the gain as applied
to the velocity averager of about 2/1 between the a and b inputs.
The timing, and the resultant error I am seeing right now, with the
spindle turning around 95 rpms, based on hall effect devices on the
60 tooth bull gear, is telling me the velocity is stepping over a 2
division (400mv p-p) range centered on nominally 1.2 volts. The
shape of this 4 step pattern can be explained by a gain difference
in the a vs b swings. Is this something we've been ignoring for a
decade or more? I can bend the opto's a tad on TLM, getting the
phasing as perfect as I can, and adjusting the leds for a 50% duty
cycle. But the velocity output on TLM, triggered on the index, looks
very similar regardless of which of the 3 machines I can look at.
The velocity is consistently low while the b input is high, and
consistently high while the b input is low.
And of course if I crank it up to 1000 rpms, quantization noise is
the dominant noise in the a/b pattern, but the general shape of the
velocity error only changes because of the rise time of the
halscope.
It may be Peter, that my encoders are junk, but why are they so
consistently junk in the same repeatable direction pattern? Optical
or hall effect?

Not sure, but the velocity average is nearly perfect with perfect
quadrature and the errors are proportional to the quadrature error as
expected. I have measured this by using a stepgen running in table
mode fed back into an encoder in table mode and making a 16 step long
table with a 1 step quadrature error (3/5 steps from edge to edge
instead of 4/4), in this case you get the expected 1/3 --> 1/5
velocity steps
This does show that a relatively small quadrature error causes a large
velocity modulation is the relative time between edge changes so much

That appears to also be true, but I don't have a perfect siggen. The
variations in timeing seen on the halscope are of course granular in
nature because the sample read from the card is non-sync to the encoders
input pulses, so moves the apparent edge times seen back and forth by
the thread period, which is nominally 1 millisecond, but this IS an r-pi
3b running locked at 1.2 Ghz, however putting it back in demand mode
makes little if any difference except in the heat. Latency overruns >10x
the 1ms servo-thread are 10 cents a dozen either way. I've the
recommended small heat sinks on it, and a video card fan running on 5
volts blowing directly on the sinks from about 3/8" away.
Now if we just had a kernel that could do decent latency on the armhf,
unfortunately we haven't had one of those critters since 2.6.20 on the
32 bit intel chips. And that predates the armhf's so we may as well
forget about it until we can build our own real time kernel that
actually works. What we have that are called real time by the builder,
are a far cry from it, good latency is 100+ microseconds, and get the
performance they claim by ignoreing 9 out of 10 keyboard or mouse
events. Miss a key up or mouse button release, and its game over
because you just smashed up both the workpiece, and the tool in the tool
post by running it into something, like a chuck jaw turning 1500 revs.
That can get expensive. Fast.

You can have relatively bad latency without any serious encoder/stepgen side
effects

Latency has nearly 0 effect on the velocity estimation since its done on a
delta-counts/delta-time basis and delta-time is calculated from time stamp
differences (or from the free running timestamp counter if no counts have
occured since the last servo thread invocation)

The time stamp counter is driven by a crystal oscillator

The encoder (and stepgen) position will of course have latency related noise
(especially at high speeds) because of the position sampling error caused by
the sompling time jitter. If this is an issue, a config with the DPLL module
can be used to reduce the encoder (and stepgen) sampling time jitter to the
100 ns region

Peter Wallace
Mesa Electronics

(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Gene Heskett

2017-07-01 00:24:58 UTC

Date: Fri, 30 Jun 2017 16:59:19 -0400
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

Date: Fri, 30 Jun 2017 12:22:24 -0400
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

Date: Fri, 30 Jun 2017 12:11:09 +0100
Reply-To: "Enhanced Machine Controller (EMC)"
Subject: Re: [Emc-users] Rough motion with servos

Post by Jon Elson
I think it is time to put a scope on the encoder. it sounds
like there may be a big duty cycle or phase angle error in that
encoder. In other words, at constant speed, the 4 quadrature
transitions are not evenly spaced, but at least one of them is
out of time.

I dont think so, the velocity calculation in the driver has not
changed for many years, and electrical noise can simulate a
reversal which will cause a 0 velocity reading...

This has been a very consistent error for years Peter. And I've
been sitting here, with halscope running remotely on this machine
for the last hour watching it, and just had a thought. This
consistency could be explained if there was a difference in the
gain as applied to the velocity averager of about 2/1 between the
a and b inputs. The timing, and the resultant error I am seeing
right now, with the spindle turning around 95 rpms, based on hall
effect devices on the 60 tooth bull gear, is telling me the
velocity is stepping over a 2 division (400mv p-p) range centered
on nominally 1.2 volts. The shape of this 4 step pattern can be
explained by a gain difference in the a vs b swings. Is this
something we've been ignoring for a decade or more? I can bend the
opto's a tad on TLM, getting the phasing as perfect as I can, and
adjusting the leds for a 50% duty cycle. But the velocity output
on TLM, triggered on the index, looks very similar regardless of
which of the 3 machines I can look at. The velocity is
consistently low while the b input is high, and consistently high
while the b input is low.
And of course if I crank it up to 1000 rpms, quantization noise is
the dominant noise in the a/b pattern, but the general shape of
the velocity error only changes because of the rise time of the
halscope.
It may be Peter, that my encoders are junk, but why are they so
consistently junk in the same repeatable direction pattern?
Optical or hall effect?

Not sure, but the velocity average is nearly perfect with perfect
quadrature and the errors are proportional to the quadrature error
as expected. I have measured this by using a stepgen running in
table mode fed back into an encoder in table mode and making a 16
step long table with a 1 step quadrature error (3/5 steps from edge
to edge instead of 4/4), in this case you get the expected 1/3 -->
1/5 velocity steps
This does show that a relatively small quadrature error causes a
large velocity modulation is the relative time between edge changes
so much

That appears to also be true, but I don't have a perfect siggen.
The variations in timeing seen on the halscope are of course
granular in nature because the sample read from the card is non-sync
to the encoders input pulses, so moves the apparent edge times seen
back and forth by the thread period, which is nominally 1
millisecond, but this IS an r-pi 3b running locked at 1.2 Ghz,
however putting it back in demand mode makes little if any
difference except in the heat. Latency overruns >10x the 1ms
servo-thread are 10 cents a dozen either way. I've the recommended
small heat sinks on it, and a video card fan running on 5 volts
blowing directly on the sinks from about 3/8" away.
Now if we just had a kernel that could do decent latency on the
armhf, unfortunately we haven't had one of those critters since
2.6.20 on the 32 bit intel chips. And that predates the armhf's so
we may as well forget about it until we can build our own real time
kernel that actually works. What we have that are called real time
by the builder, are a far cry from it, good latency is 100+
microseconds, and get the performance they claim by ignoreing 9 out
of 10 keyboard or mouse events. Miss a key up or mouse button
release, and its game over because you just smashed up both the
workpiece, and the tool in the tool post by running it into
something, like a chuck jaw turning 1500 revs.
That can get expensive. Fast.

You can have relatively bad latency without any serious
encoder/stepgen side effects
Latency has nearly 0 effect on the velocity estimation since its done
on a delta-counts/delta-time basis and delta-time is calculated from
time stamp differences (or from the free running timestamp counter if
no counts have occured since the last servo thread invocation)
The time stamp counter is driven by a crystal oscillator
The encoder (and stepgen) position will of course have latency related
noise (especially at high speeds) because of the position sampling
error caused by the sompling time jitter. If this is an issue, a
config with the DPLL module can be used to reduce the encoder (and
stepgen) sampling time jitter to the 100 ns region

I believe I have that turned on. Let me scan the hal file. I think this
is it:

# This, ack the hostmot2 man page, should reduce following errors
# and it applies to all stepgens enabled
setp hm2_[HOSTMOT2](BOARD).0.stepgen.timer-number 1

Or is that something else? Doesn't sound like its encoder related
though.

Encoder sampling is: (from ini file)
ENCODER_SAMPLE = 500000
And is used in the hal file via:
A non-existent statement, so I am assuming 500 kilohertz is the default
because that is what the halmeter reports. Nevertheless I should add
that to the .hal file in case it ever needs a tweak.

Thanks Peter.

Post by Peter C. Wallace
Peter Wallace
Mesa Electronics
(\__/)
(='.'=) This is Bunny. Copy and paste bunny into your
(")_(") signature to help him gain world domination.

Cheers, Gene Heskett

--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

Jon Elson

2017-07-01 02:39:24 UTC

Post by Peter C. Wallace
This does show that a relatively small quadrature error
causes a large velocity modulation is the relative time
between edge changes so much

Good encoders have very close to 50% duty cycle on the
individual A and B outputs, at least when new. Depending on
the optical technology, the duty cycle can drift as the LED
ages. The phase angle between A and B is pretty well set by
mechanical features of the encoder, and is not likely to
drift. But, it could be off on poorly-built encoders.
Also, older encoders had mechanical adjustments for the
sensor gratings, and if dropped or handled roughly, that
factory adjustment could break loose.

Some very old encoders had 4 sensors and gratings such that
the sensors got differential light for each signal, and then
used a comparator. This pretty much eliminated the drift as
the LED (or bulb!) aged.

Anyway, I think many encoders have a spec of no more than 5%
duty cycle error, and maybe 5 or 10 electrical degrees in
the quadrature.

Jon

dave

2017-06-30 16:52:57 UTC

I think it is time to put a scope on the encoder. it sounds like
there may be a big duty cycle or phase angle error in that encoder. In
other words, at constant speed, the 4 quadrature transitions are not
evenly spaced, but at least one of them is out of time.

Years ago Jon Elson convinced me you almost could not too many counts/rev.
Going from 5000 c/in (glass scale) to 40K c/in made a huge difference in
how easy it was
to tune.
On my Z axis I mounted the encoder almost like an idler but on the cog
side of the belt.
Small timing pulley so I get upwards of 100K counts/in as opposed to
only 40K for X and Y.
Every bittle lit helps. :-)

Dave

Les Newell

2017-06-30 19:13:26 UTC

Post by dave
Years ago Jon Elson convinced me you almost could not too many
counts/rev.
Going from 5000 c/in (glass scale) to 40K c/in made a huge difference
in how easy it was
to tune.

I agree with Jon but changing the encoder is a pain and once I got past
the noise issues, tuning it was pretty straight forward. I have 200
counts/mm (~5k counts/inch) on Z and 819.2 counts/mm (~21k counts/inch)
on X. X is more important than Z on lathe as for the vast majority of
jobs I do diameter is more critical than length.

Les

Jon Elson

2017-06-29 16:02:26 UTC

Post by Les Newell
Well, some progress figuring out what is going on.
I just ran some more tests using Ubuntu 16.04 with a
freshly built preempt-rt kernel and freshly built Git
master. No surprise, it still shows the fault. I took a
Halscope log from the original setup. The encoder velocity
is rather noisy but no big spikes. On the new install the
encoder velocity is not as noisy but it has occasional
spikes. The encoders are wired to a 7i29.
This log is from the old setup
<www.sheetcam.com/lathe/old.log>
This log is from the new setup
<www.sheetcam.com/lathe/spike.log>
This lathe is fitted with high resolution handwheel
encoders so I also did something I should have done much
sooner and looked at the handwheel encoder velocity. Big
difference - nice clean signals on both configs.
So it looks like something funky is going on when reading
encoders with the 7i29, resulting in noisy velocity
estimation.

Assuming the spikes happen fairly often, you should try
setting up Halscope to display velocity and trigger on where
you think a spike might reach, and then move the axis by
hand for a while and see if the scope triggers. This would
be with the drives off and moving it by hand. If you can't
see these spikes with the drives off, then you have a strong
indication of electrical noise getting into the encoder
signals. Mesa does have some adjustments to the digital
filter settings that you could try, or you may need to
change the encoder cables, set up the shields differently
(you ARE using shielded cables, I hope!) of possibly have to
go to differential drive at the encoder end, with
differential receivers at the Mesa end.

Post by Les Newell
I guess the velocity estimation code has changed a bit
over the years, giving the different results. The fault
has probably been there for a long time, quite possibly
since I built it. My immediate thought is noise on the
encoder signals but this machine never loses position. It
will turn out accurate parts all day. Peter, any idea how
the velocity estimation can be noisy while the overall
counts seem to track accurately?

I think you are on the right track.

Jon

Sebastian Kuzminsky

2017-06-30 16:31:43 UTC

Post by Les Newell
I tried using Buildbot on a fresh install of Stretch amd64 and got a
python-gst0.10 (Stretch has python-gst1.0)
gstreamer0.10-plugins-base
hostmot2-firmware-all

Thanks for reporting this problem.

We've removed the dependency on python-gst and gstreamer for 2.7 and
master (2.8~pre) on Stretch for now, possibly breaking some parts of the
gmoccapy and gscreen guis (on Stretch only) until they get the attention
they need from their maintainers.

We've also added hostmot2 firmware debs for Stretch.

--
Sebastian Kuzminsky

Todd Zuercher

2017-06-28 18:39:46 UTC

PS.
Thanks for your contributions that became MB2HAL. I'm using it.

----- Original Message -----
From: "Les Newell" <***@fastmail.co.uk>
To: "Enhanced Machine Controller (EMC)" <emc-***@lists.sourceforge.net>
Sent: Wednesday, June 28, 2017 2:26:52 PM
Subject: Re: [Emc-users] Rough motion with servos

Hi Todd,

I am building from source. I have tried several distros:
Debian - something old, I can't remember what. This has been running
fine for about 8 years but it lacks driver support for modern hardware.
Ubuntu 10.04 LinuxCNC ISO - appears to run smoothly but I can't test
properly until I get LCNC to build. It's also a bit long in the tooth.
Ubuntu 16.04 LinuxCNC ISO - runs rough but I can build fine. I've tried
various different branches of LinuxCNC, including one that is close to
what I was originally running, on this and they all run rough.

Post by Todd Zuercher
Are you sure all of your custom modules are still "custom", or needed? There have been a lot of little goodies added to Linuxcnc since 2.5-2.6 days.

Post by Todd Zuercher
Maybe they can be added using halcompile (formerly known as comp)?

Peter C. Wallace

2017-06-28 18:48:02 UTC

Did you trace the source of the PID output spikes?
that is, are there glitches in feedback position of feedback velocity?

The whole thing sounds very strange

Are you using identical hal files for the system with the spikes/without
spikes?

Peter Wallace
Mesa Electronics

Les Newell

2017-06-28 19:14:40 UTC

Hi Peter,

As far as I can tell the spikes are in the feedback velocity. It can be
a little difficult to tell cause and effect.

Post by Peter C. Wallace
Are you using identical hal files for the system with the
spikes/without spikes?

When I built the 2.6.13 branch on the 2.7.9 ISO I only had to make some
very minor changes to the hal files to get it to run.

Les

Post by Peter C. Wallace
Did you trace the source of the PID output spikes?
that is, are there glitches in feedback position of feedback velocity?
The whole thing sounds very strange
Are you using identical hal files for the system with the
spikes/without spikes?
Peter Wallace
Mesa Electronics
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
https://lists.sourceforge.net/lists/listinfo/emc-users

Todd Zuercher

2017-06-28 20:33:33 UTC