[Emc-users] RPI saga continues

[Bertho Stultiens]

Hi all,

Too happy too soon. That is what happens.

There seems to be one additional problem when configuring the SPI port.
After a restart of the Pi everything went to bad again.

After some digging I noticed that there might be a data-barrier problem,
where peripheral register access can become out-of-order. The ARM has
the __sync_synchronize() (via gcc) to insert DMB (data-memory-barrier)
instructions when you need to guarantee ordering. Inserting DMB made
things worse, such that sometimes the setup is 32MHz and sometimes
50MHz. Well, actually, it exposes a deeper problem.

This is a Pi3, a 4-core SMP machine. DMBs and peripheral setup probably
requires that no other cores are crossing peripherals while doing setup
(or the process/thread is rescheduled onto a different core). For the
Pi3 you probably need to lock this specific setup operation to one core
and one core only.

I noticed at startup that several interesting things happen, and I
assume that a lot of processes and threads are spawned while doing
startup. That also means that all bets are off in this sensitive time
period and could explain why SPI keeps hopping between different setup
states.

Anybody care to comment or suggest a way to lock that part of the code
to a core?

--
Greetings Bertho

(disclaimers are disclaimed)

[Bertho Stultiens]

On 05/30/2017 02:32 AM, Bertho Stultiens wrote:
> After some digging I noticed that there might be a data-barrier problem,
> where peripheral register access can become out-of-order. The ARM has
> the __sync_synchronize() (via gcc) to insert DMB (data-memory-barrier)
> instructions when you need to guarantee ordering. Inserting DMB made
> things worse, such that sometimes the setup is 32MHz and sometimes
> 50MHz. Well, actually, it exposes a deeper problem.

I suddenly realized that the SPI peripheral is configured and accessed
in userspace. That will fail miserably if not extremely careful,
especially on SMP.

However, there already is a solution for this! The linux kernel has a
SPI driver, which is quite good (used it before). It solves all the
userspace problems and, to say the least, some very clever people have
had a crack at this problem before.

So, drop userspace access to the hardware and use the kernel interface.

1 - run raspi-config and enable the kernel SPI driver -> reboot
2 - you should now have /dev/spidev0.[01]
3 - replace the hm2_rpspi.c file in src/hal/drivers/mesa-hostmot2/ with
the one attached
4 - recompile
5 - run

Gene, with my (4GB) SD image do (you know the cmdline):
- run raspi-config to enable the kernel SPI driver and reboot
- save attached file on SD card as hm2_rpspi.c.new
$ cp hm2_rpspi.c.new linuxcnc-git/src/hal/drivers/mesa-hostmot2/hm2_rpspi.c
$ cd linuxcnc-git/src
$ make
$ sudo make setuid
$ ../scripts/linuxcnc -v ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini

I get the result as shown in the images. The advantage is that there are
no inter-word delays anymore. The time for chip select to be active
varies a bit (16.5 us in image 8 vs. 6.3 us data transfer in image 6).
This variability is inherent to the driver how it handles chip select
asynchronously. Still, it is an improvement, especially for large transfers.

I did add a rtapi module parameter - spiclk - which should set the clock
upon load. However, I've not been able to get that to work. Probably
something trivial I did wrong. Ah well, at least it runs at a good speed
now, also after reboots.

My hm2_rpspi driver hack is fixed to /dev/spidev0.0 (CE0 pin). This
should be a configurable parameter (features for the future).

--
Greetings Bertho

(disclaimers are disclaimed)

[Jeff Epler]

On Tue, May 30, 2017 at 02:20:38PM +0200, Bertho Stultiens wrote:
> I suddenly realized that the SPI peripheral is configured and accessed
> in userspace. That will fail miserably if not extremely careful,
> especially on SMP.
>
> However, there already is a solution for this! The linux kernel has a
> SPI driver, which is quite good (used it before). It solves all the
> userspace problems and, to say the least, some very clever people have
> had a crack at this problem before.
[snip]

We already have a driver for hostmot2 that uses /dev/spidev*,
hm2_spi. hm2_rpspi exists because its contributor stated that on
their system, hm2_spi did not perform adequately.

Jeff

[Bertho Stultiens]

On 05/30/2017 02:28 PM, Jeff Epler wrote:
>> However, there already is a solution for this! The linux kernel has a
>> SPI driver, which is quite good (used it before). It solves all the
>> userspace problems and, to say the least, some very clever people have
>> had a crack at this problem before.
> [snip]
>
> We already have a driver for hostmot2 that uses /dev/spidev*,
> hm2_spi. hm2_rpspi exists because its contributor stated that on
> their system, hm2_spi did not perform adequately.

My guess is that it was on a Pi2. The Pi3 should perform better. The
problem is that you cannot control the SMP behavior correctly from
userspace, unless all interrupts are disabled and caching is taken into
account.

The only performance issue I can see is the asynchronous chip select.
The original code in the hm2_rpspi is significantly slower in transfers
due to synchronous write/read sequences.

--
Greetings Bertho

(disclaimers are disclaimed)

[Bertho Stultiens]

On 05/30/2017 02:28 PM, Jeff Epler wrote:
> We already have a driver for hostmot2 that uses /dev/spidev*,
> hm2_spi. hm2_rpspi exists because its contributor stated that on
> their system, hm2_spi did not perform adequately.

That reminds me of setting scheduling/cpu affinity. Maybe the solution
can be simpler.

--
Greetings Bertho

(disclaimers are disclaimed)

[Gene Heskett]

On Tuesday 30 May 2017 08:20:38 Bertho Stultiens wrote:

> On 05/30/2017 02:32 AM, Bertho Stultiens wrote:
> > After some digging I noticed that there might be a data-barrier
> > problem, where peripheral register access can become out-of-order.
> > The ARM has the __sync_synchronize() (via gcc) to insert DMB
> > (data-memory-barrier) instructions when you need to guarantee
> > ordering. Inserting DMB made things worse, such that sometimes the
> > setup is 32MHz and sometimes 50MHz. Well, actually, it exposes a
> > deeper problem.
>
> I suddenly realized that the SPI peripheral is configured and accessed
> in userspace. That will fail miserably if not extremely careful,
> especially on SMP.
>
> However, there already is a solution for this! The linux kernel has a
> SPI driver, which is quite good (used it before). It solves all the
> userspace problems and, to say the least, some very clever people have
> had a crack at this problem before.
>
> So, drop userspace access to the hardware and use the kernel
> interface.
>
> 1 - run raspi-config and enable the kernel SPI driver -> reboot
> 2 - you should now have /dev/spidev0.[01]
> 3 - replace the hm2_rpspi.c file in src/hal/drivers/mesa-hostmot2/
> with the one attached
> 4 - recompile
> 5 - run
>
> Gene, with my (4GB) SD image do (you know the cmdline):
> - run raspi-config to enable the kernel SPI driver and reboot
> - save attached file on SD card as hm2_rpspi.c.new
> $ cp hm2_rpspi.c.new
> linuxcnc-git/src/hal/drivers/mesa-hostmot2/hm2_rpspi.c $ cd
> linuxcnc-git/src
> $ make
> $ sudo make setuid
> $ ../scripts/linuxcnc -v
> ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
>
Well, I am still trying to figure out how to run linuxcnc w/o a gfx
screen.

I followed the above instructions when it wouldn't run the first time,
and now I get a somewhat different message, so lemme see if I can log in
from here and duplicate the results, at least to the interesting part.

***@pionsheldon:~/linuxcnc-git/scripts
$ ./linuxcnc -v ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
Verbose mode on
RUN_IN_PLACE=yes
LINUXCNC_DIR=
LINUXCNC_BIN_DIR=/home/pi/linuxcnc-git/bin
LINUXCNC_TCL_DIR=/home/pi/linuxcnc-git/tcl
LINUXCNC_SCRIPT_DIR=
LINUXCNC_RTLIB_DIR=/home/pi/linuxcnc-git/rtlib
LINUXCNC_CONFIG_DIR=
LINUXCNC_LANG_DIR=/home/pi/linuxcnc-git/src/objects
INIVAR=inivar
HALCMD=halcmd
LINUXCNC_EMCSH=/usr/bin/wish8.6
LINUXCNC - 2.8.0~pre1
Machine configuration directory
is '/home/pi/linuxcnc-git/scripts/../../linuxcnc/configs/sheldon-lathe'
Machine configuration file is '7i90-axis.ini'
INIFILE=/home/pi/linuxcnc-git/scripts/../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
VERSION=1.0
PARAMETER_FILE=hm2-stepper.var
TASK=milltask
HALUI=
DISPLAY=axis
COORDINATES=X Z
KINEMATICS=trivkins coordinates=XZ
Starting LinuxCNC...
Starting LinuxCNC server program: linuxcncsvr
Loading Real Time OS, RTAPI, and HAL_LIB modules
Starting LinuxCNC IO program: io
Found file(REL): ./hm2-7i90-stepper.hal
Note: Using POSIX realtime
hm2: loading Mesa HostMot2 driver version 0.15
hm2_rpspi: spiclk=32000000 Hz
hm2_rpspi: Invalid cookie
hm2_rpspi: Read: 00000000 00000000 00000000 00000000
hm2_rpspi: rtapi_app_main: No such device (-19)
./hm2-7i90-stepper.hal:32: waitpid
failed /home/pi/linuxcnc-git/bin/rtapi_app hm2_rpspi
./hm2-7i90-stepper.hal:32: /home/pi/linuxcnc-git/bin/rtapi_app exited
without becoming ready
./hm2-7i90-stepper.hal:32: insmod for hm2_rpspi failed, returned -1
Shutting down and cleaning up LinuxCNC...
Killing task linuxcncsvr, PID=690
hm2_rpspi: not loaded
<commandline>:0: exit value: 255
<commandline>:0: rmmod failed, returned -1
hm2: unloading
<commandline>:0: unloadrt failed
Removing HAL_LIB, RTAPI, and Real Time OS modules
Note: Using POSIX realtime
Removing NML shared memory segments
LinuxCNC terminated with an error. You can find more information in the
log:
/home/pi/linuxcnc_debug.txt
and
/home/pi/linuxcnc_print.txt
as well as in the output of the shell command 'dmesg' and in the terminal
***@pionsheldon:~/linuxcnc-git/scripts $

Is that 14 pf cap to ground on the spi-clk0 now screwing it up?

On the first attempt, the error wording seemed to indicate it couldn't
find a display. I wish I'd have thought to capture it.

On the other card, I had 4 workspaces set up on the other card before I
had increased the framebuffer depth to 24, and I found they were still
there, just blank screens, and I could make my terminal session go away
by rolling the mouse wheel, and if I kept rolling the same direction it
wrapped and put me back on the 1st screen. Shade tree engineering at
its best. :) Doesn't work now of course.

Your turn, should you choose to accept it.

I'm going to put the other card back in and see if I have a good 7i90
under all those 7i42ta's. The one its running on now has some blown bus
buffers according to Peter. With 4 of them laying about, I may have
grabbed a bad one. So I'd better check before I bury it any deeper.

> I get the result as shown in the images. The advantage is that there
> are no inter-word delays anymore. The time for chip select to be
> active varies a bit (16.5 us in image 8 vs. 6.3 us data transfer in
> image 6). This variability is inherent to the driver how it handles
> chip select asynchronously. Still, it is an improvement, especially
> for large transfers.
>
> I did add a rtapi module parameter - spiclk - which should set the
> clock upon load. However, I've not been able to get that to work.
> Probably something trivial I did wrong. Ah well, at least it runs at a
> good speed now, also after reboots.
>
> My hm2_rpspi driver hack is fixed to /dev/spidev0.0 (CE0 pin). This
> should be a configurable parameter (features for the future).


Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Chris Albertson]

Gene,

So many words here I can't see the problem. If if you are trying to
reliably export the P's screen to your "your computer" You might just skip
trying to use X11 and go with VNC. I run a VNC server on my Pi3 and it
has been running continuously now for about a month. When I log in from
my iMac I get a graphic session.

If you are god with Google. Stop reading here and just Goole "VNC".



The down side of VNC is the lag. It's very noticeable . VNC is the same
technology Both Microsoft and Apple use for their remote desktop products.
There are a number of implementations of VNC both server and client some
open source, some closed but zero cost and some commercial. Some Windows
versions ship with it installed.

It does not depend on X11, so you can do tricks like have the Windows 10
desktop showing in the Pi or vice versa and it works even over the
Internet. Technical support people use it to debug user problems In fact
if you have VNC server running on the Pi, some expert 1,000 miles away
could look at your Pi, even while you were logged on.

But did I warn you about lag? It's so bad VNC uses a double cursor, one
moves in real time the other shows what the remote computer thinks, they
seem to be connected by a bungee cord.

One other good thing: I have VNC client on my iPad. So I can walk
around and access the Pi3 with no wires. Works for simple things within
limits of 7 inch screen and glass keyboard. Works on iPhone too but tiny
screen is useless.

Google for details but steps are to (1) install VNC server on one computer,
make sure it runs at boot timed (2) install client on computer #2 then from
menu select first computer or type in IP address if not on menu. I
suggest the first experiments be done on two computers you are comfortable
with, maybe you Linux desktop and your Windows box. THEN try on there Pi.

Al that said, for me, ssh works perfect with zero lag. Also I can export
ONE app from Pi to iMac (not the entire desktop session) and it work
lag-free also. My Pi is using a wired Ethernet connection.



On Tue, May 30, 2017 at 10:07 AM, Gene Heskett <***@shentel.net> wrote:

> On Tuesday 30 May 2017 08:20:38 Bertho Stultiens wrote:
>
> > On 05/30/2017 02:32 AM, Bertho Stultiens wrote:
> > > After some digging I noticed that there might be a data-barrier
> > > problem, where peripheral register access can become out-of-order.
> > > The ARM has the __sync_synchronize() (via gcc) to insert DMB
> > > (data-memory-barrier) instructions when you need to guarantee
> > > ordering. Inserting DMB made things worse, such that sometimes the
> > > setup is 32MHz and sometimes 50MHz. Well, actually, it exposes a
> > > deeper problem.
> >
> > I suddenly realized that the SPI peripheral is configured and accessed
> > in userspace. That will fail miserably if not extremely careful,
> > especially on SMP.
> >
> > However, there already is a solution for this! The linux kernel has a
> > SPI driver, which is quite good (used it before). It solves all the
> > userspace problems and, to say the least, some very clever people have
> > had a crack at this problem before.
> >
> > So, drop userspace access to the hardware and use the kernel
> > interface.
> >
> > 1 - run raspi-config and enable the kernel SPI driver -> reboot
> > 2 - you should now have /dev/spidev0.[01]
> > 3 - replace the hm2_rpspi.c file in src/hal/drivers/mesa-hostmot2/
> > with the one attached
> > 4 - recompile
> > 5 - run
> >
> > Gene, with my (4GB) SD image do (you know the cmdline):
> > - run raspi-config to enable the kernel SPI driver and reboot
> > - save attached file on SD card as hm2_rpspi.c.new
> > $ cp hm2_rpspi.c.new
> > linuxcnc-git/src/hal/drivers/mesa-hostmot2/hm2_rpspi.c $ cd
> > linuxcnc-git/src
> > $ make
> > $ sudo make setuid
> > $ ../scripts/linuxcnc -v
> > ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
> >
> Well, I am still trying to figure out how to run linuxcnc w/o a gfx
> screen.
>
> I followed the above instructions when it wouldn't run the first time,
> and now I get a somewhat different message, so lemme see if I can log in
> from here and duplicate the results, at least to the interesting part.
>
> ***@pionsheldon:~/linuxcnc-git/scripts
> $ ./linuxcnc -v ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
> Verbose mode on
> RUN_IN_PLACE=yes
> LINUXCNC_DIR=
> LINUXCNC_BIN_DIR=/home/pi/linuxcnc-git/bin
> LINUXCNC_TCL_DIR=/home/pi/linuxcnc-git/tcl
> LINUXCNC_SCRIPT_DIR=
> LINUXCNC_RTLIB_DIR=/home/pi/linuxcnc-git/rtlib
> LINUXCNC_CONFIG_DIR=
> LINUXCNC_LANG_DIR=/home/pi/linuxcnc-git/src/objects
> INIVAR=inivar
> HALCMD=halcmd
> LINUXCNC_EMCSH=/usr/bin/wish8.6
> LINUXCNC - 2.8.0~pre1
> Machine configuration directory
> is '/home/pi/linuxcnc-git/scripts/../../linuxcnc/configs/sheldon-lathe'
> Machine configuration file is '7i90-axis.ini'
> INIFILE=/home/pi/linuxcnc-git/scripts/../../linuxcnc/
> configs/sheldon-lathe/7i90-axis.ini
> VERSION=1.0
> PARAMETER_FILE=hm2-stepper.var
> TASK=milltask
> HALUI=
> DISPLAY=axis
> COORDINATES=X Z
> KINEMATICS=trivkins coordinates=XZ
> Starting LinuxCNC...
> Starting LinuxCNC server program: linuxcncsvr
> Loading Real Time OS, RTAPI, and HAL_LIB modules
> Starting LinuxCNC IO program: io
> Found file(REL): ./hm2-7i90-stepper.hal
> Note: Using POSIX realtime
> hm2: loading Mesa HostMot2 driver version 0.15
> hm2_rpspi: spiclk=32000000 Hz
> hm2_rpspi: Invalid cookie
> hm2_rpspi: Read: 00000000 00000000 00000000 00000000
> hm2_rpspi: rtapi_app_main: No such device (-19)
> ./hm2-7i90-stepper.hal:32: waitpid
> failed /home/pi/linuxcnc-git/bin/rtapi_app hm2_rpspi
> ./hm2-7i90-stepper.hal:32: /home/pi/linuxcnc-git/bin/rtapi_app exited
> without becoming ready
> ./hm2-7i90-stepper.hal:32: insmod for hm2_rpspi failed, returned -1
> Shutting down and cleaning up LinuxCNC...
> Killing task linuxcncsvr, PID=690
> hm2_rpspi: not loaded
> <commandline>:0: exit value: 255
> <commandline>:0: rmmod failed, returned -1
> hm2: unloading
> <commandline>:0: unloadrt failed
> Removing HAL_LIB, RTAPI, and Real Time OS modules
> Note: Using POSIX realtime
> Removing NML shared memory segments
> LinuxCNC terminated with an error. You can find more information in the
> log:
> /home/pi/linuxcnc_debug.txt
> and
> /home/pi/linuxcnc_print.txt
> as well as in the output of the shell command 'dmesg' and in the terminal
> ***@pionsheldon:~/linuxcnc-git/scripts $
>
> Is that 14 pf cap to ground on the spi-clk0 now screwing it up?
>
> On the first attempt, the error wording seemed to indicate it couldn't
> find a display. I wish I'd have thought to capture it.
>
> On the other card, I had 4 workspaces set up on the other card before I
> had increased the framebuffer depth to 24, and I found they were still
> there, just blank screens, and I could make my terminal session go away
> by rolling the mouse wheel, and if I kept rolling the same direction it
> wrapped and put me back on the 1st screen. Shade tree engineering at
> its best. :) Doesn't work now of course.
>
> Your turn, should you choose to accept it.
>
> I'm going to put the other card back in and see if I have a good 7i90
> under all those 7i42ta's. The one its running on now has some blown bus
> buffers according to Peter. With 4 of them laying about, I may have
> grabbed a bad one. So I'd better check before I bury it any deeper.
>
> > I get the result as shown in the images. The advantage is that there
> > are no inter-word delays anymore. The time for chip select to be
> > active varies a bit (16.5 us in image 8 vs. 6.3 us data transfer in
> > image 6). This variability is inherent to the driver how it handles
> > chip select asynchronously. Still, it is an improvement, especially
> > for large transfers.
> >
> > I did add a rtapi module parameter - spiclk - which should set the
> > clock upon load. However, I've not been able to get that to work.
> > Probably something trivial I did wrong. Ah well, at least it runs at a
> > good speed now, also after reboots.
> >
> > My hm2_rpspi driver hack is fixed to /dev/spidev0.0 (CE0 pin). This
> > should be a configurable parameter (features for the future).
>
>
> Cheers, Gene Heskett
> --
> "There are four boxes to be used in defense of liberty:
> soap, ballot, jury, and ammo. Please use in that order."
> -Ed Howdershelt (Author)
> Genes Web page <http://geneslinuxbox.net:6309/gene>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Emc-users mailing list
> Emc-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/emc-users
>



--

Chris Albertson
Redondo Beach, California

[Gene Heskett]

On Tuesday 30 May 2017 15:00:31 Chris Albertson wrote:

> Gene,
>
> So many words here I can't see the problem. If if you are trying to
> reliably export the P's screen to your "your computer" You might just
> skip trying to use X11 and go with VNC. I run a VNC server on my
> Pi3 and it has been running continuously now for about a month. When
> I log in from my iMac I get a graphic session.
>
> If you are god with Google. Stop reading here and just Goole "VNC".

I've thought of that, and the 2nd or 3rd message I read is squawking
about the lag.
>
> The down side of VNC is the lag. It's very noticeable . VNC is the
> same technology Both Microsoft and Apple use for their remote desktop
> products. There are a number of implementations of VNC both server and
> client some open source, some closed but zero cost and some
> commercial. Some Windows versions ship with it installed.
>
You are talking windows Chris. Only ones I have here are made of glass.
M$ products are verboten at the coyote.den.

> It does not depend on X11, so you can do tricks like have the Windows
> 10 desktop showing in the Pi or vice versa and it works even over the
> Internet. Technical support people use it to debug user problems In
> fact if you have VNC server running on the Pi, some expert 1,000 miles
> away could look at your Pi, even while you were logged on.
>
> But did I warn you about lag? It's so bad VNC uses a double cursor,
> one moves in real time the other shows what the remote computer
> thinks, they seem to be connected by a bungee cord.

I don't know as I could tolerate that.
>
> One other good thing: I have VNC client on my iPad. So I can walk
> around and access the Pi3 with no wires. Works for simple things
> within limits of 7 inch screen and glass keyboard. Works on iPhone
> too but tiny screen is useless.
>
> Google for details but steps are to (1) install VNC server on one
> computer, make sure it runs at boot timed (2) install client on
> computer #2 then from menu select first computer or type in IP address
> if not on menu. I suggest the first experiments be done on two
> computers you are comfortable with, maybe you Linux desktop and your
> Windows box. THEN try on there Pi.
>
> Al that said, for me, ssh works perfect with zero lag. Also I can
> export ONE app from Pi to iMac (not the entire desktop session) and it
> work lag-free also. My Pi is using a wired Ethernet connection.

So is mine. And I've not figured out yet how to stop whatever it is
thats bringing wlan0 up at boot. I'd remove its execute rights. This
one is not behind alu siding, so its attackable, successfully, by a
neighbor that I've not been able to ident.

> On Tue, May 30, 2017 at 10:07 AM, Gene Heskett <***@shentel.net>
wrote:
> > On Tuesday 30 May 2017 08:20:38 Bertho Stultiens wrote:
> > > On 05/30/2017 02:32 AM, Bertho Stultiens wrote:
> > > > After some digging I noticed that there might be a data-barrier
> > > > problem, where peripheral register access can become
> > > > out-of-order. The ARM has the __sync_synchronize() (via gcc) to
> > > > insert DMB (data-memory-barrier) instructions when you need to
> > > > guarantee ordering. Inserting DMB made things worse, such that
> > > > sometimes the setup is 32MHz and sometimes 50MHz. Well,
> > > > actually, it exposes a deeper problem.
> > >
> > > I suddenly realized that the SPI peripheral is configured and
> > > accessed in userspace. That will fail miserably if not extremely
> > > careful, especially on SMP.
> > >
> > > However, there already is a solution for this! The linux kernel
> > > has a SPI driver, which is quite good (used it before). It solves
> > > all the userspace problems and, to say the least, some very clever
> > > people have had a crack at this problem before.
> > >
> > > So, drop userspace access to the hardware and use the kernel
> > > interface.
> > >
> > > 1 - run raspi-config and enable the kernel SPI driver -> reboot
> > > 2 - you should now have /dev/spidev0.[01]
> > > 3 - replace the hm2_rpspi.c file in src/hal/drivers/mesa-hostmot2/
> > > with the one attached
> > > 4 - recompile
> > > 5 - run
> > >
> > > Gene, with my (4GB) SD image do (you know the cmdline):
> > > - run raspi-config to enable the kernel SPI driver and reboot
> > > - save attached file on SD card as hm2_rpspi.c.new
> > > $ cp hm2_rpspi.c.new
> > > linuxcnc-git/src/hal/drivers/mesa-hostmot2/hm2_rpspi.c $ cd
> > > linuxcnc-git/src
> > > $ make
> > > $ sudo make setuid
> > > $ ../scripts/linuxcnc -v
> > > ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
> >
> > Well, I am still trying to figure out how to run linuxcnc w/o a gfx
> > screen.
> >
> > I followed the above instructions when it wouldn't run the first
> > time, and now I get a somewhat different message, so lemme see if I
> > can log in from here and duplicate the results, at least to the
> > interesting part.
> >
> > ***@pionsheldon:~/linuxcnc-git/scripts
> > $ ./linuxcnc -v ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
> > Verbose mode on
> > RUN_IN_PLACE=yes
> > LINUXCNC_DIR=
> > LINUXCNC_BIN_DIR=/home/pi/linuxcnc-git/bin
> > LINUXCNC_TCL_DIR=/home/pi/linuxcnc-git/tcl
> > LINUXCNC_SCRIPT_DIR=
> > LINUXCNC_RTLIB_DIR=/home/pi/linuxcnc-git/rtlib
> > LINUXCNC_CONFIG_DIR=
> > LINUXCNC_LANG_DIR=/home/pi/linuxcnc-git/src/objects
> > INIVAR=inivar
> > HALCMD=halcmd
> > LINUXCNC_EMCSH=/usr/bin/wish8.6
> > LINUXCNC - 2.8.0~pre1
> > Machine configuration directory
> > is
> > '/home/pi/linuxcnc-git/scripts/../../linuxcnc/configs/sheldon-lathe'
> > Machine configuration file is '7i90-axis.ini'
> > INIFILE=/home/pi/linuxcnc-git/scripts/../../linuxcnc/
> > configs/sheldon-lathe/7i90-axis.ini
> > VERSION=1.0
> > PARAMETER_FILE=hm2-stepper.var
> > TASK=milltask
> > HALUI=
> > DISPLAY=axis
> > COORDINATES=X Z
> > KINEMATICS=trivkins coordinates=XZ
> > Starting LinuxCNC...
> > Starting LinuxCNC server program: linuxcncsvr
> > Loading Real Time OS, RTAPI, and HAL_LIB modules
> > Starting LinuxCNC IO program: io
> > Found file(REL): ./hm2-7i90-stepper.hal
> > Note: Using POSIX realtime
> > hm2: loading Mesa HostMot2 driver version 0.15
> > hm2_rpspi: spiclk=32000000 Hz
> > hm2_rpspi: Invalid cookie
> > hm2_rpspi: Read: 00000000 00000000 00000000 00000000
> > hm2_rpspi: rtapi_app_main: No such device (-19)
> > ./hm2-7i90-stepper.hal:32: waitpid
> > failed /home/pi/linuxcnc-git/bin/rtapi_app hm2_rpspi
> > ./hm2-7i90-stepper.hal:32: /home/pi/linuxcnc-git/bin/rtapi_app
> > exited without becoming ready
> > ./hm2-7i90-stepper.hal:32: insmod for hm2_rpspi failed, returned -1
> > Shutting down and cleaning up LinuxCNC...
> > Killing task linuxcncsvr, PID=690
> > hm2_rpspi: not loaded
> > <commandline>:0: exit value: 255
> > <commandline>:0: rmmod failed, returned -1
> > hm2: unloading
> > <commandline>:0: unloadrt failed
> > Removing HAL_LIB, RTAPI, and Real Time OS modules
> > Note: Using POSIX realtime
> > Removing NML shared memory segments
> > LinuxCNC terminated with an error. You can find more information in
> > the log:
> > /home/pi/linuxcnc_debug.txt
> > and
> > /home/pi/linuxcnc_print.txt
> > as well as in the output of the shell command 'dmesg' and in the
> > terminal ***@pionsheldon:~/linuxcnc-git/scripts $
> >
> > Is that 14 pf cap to ground on the spi-clk0 now screwing it up?
> >
> > On the first attempt, the error wording seemed to indicate it
> > couldn't find a display. I wish I'd have thought to capture it.
> >
> > On the other card, I had 4 workspaces set up on the other card
> > before I had increased the framebuffer depth to 24, and I found they
> > were still there, just blank screens, and I could make my terminal
> > session go away by rolling the mouse wheel, and if I kept rolling
> > the same direction it wrapped and put me back on the 1st screen.
> > Shade tree engineering at its best. :) Doesn't work now of course.
> >
> > Your turn, should you choose to accept it.
> >
> > I'm going to put the other card back in and see if I have a good
> > 7i90 under all those 7i42ta's. The one its running on now has some
> > blown bus buffers according to Peter. With 4 of them laying about,
> > I may have grabbed a bad one. So I'd better check before I bury it
> > any deeper.
> >
> > > I get the result as shown in the images. The advantage is that
> > > there are no inter-word delays anymore. The time for chip select
> > > to be active varies a bit (16.5 us in image 8 vs. 6.3 us data
> > > transfer in image 6). This variability is inherent to the driver
> > > how it handles chip select asynchronously. Still, it is an
> > > improvement, especially for large transfers.
> > >
> > > I did add a rtapi module parameter - spiclk - which should set the
> > > clock upon load. However, I've not been able to get that to work.
> > > Probably something trivial I did wrong. Ah well, at least it runs
> > > at a good speed now, also after reboots.
> > >
> > > My hm2_rpspi driver hack is fixed to /dev/spidev0.0 (CE0 pin).
> > > This should be a configurable parameter (features for the future).
> >
> > Cheers, Gene Heskett
> > --
> > "There are four boxes to be used in defense of liberty:
> > soap, ballot, jury, and ammo. Please use in that order."
> > -Ed Howdershelt (Author)
> > Genes Web page <http://geneslinuxbox.net:6309/gene>
> >
> > ------------------------------------------------------------
> > ------------------
> > Check out the vibrant tech community on one of the world's most
> > engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> > _______________________________________________
> > Emc-users mailing list
> > Emc-***@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/emc-users


Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Gene Heskett]

On Tuesday 30 May 2017 08:20:38 Bertho Stultiens wrote:

> On 05/30/2017 02:32 AM, Bertho Stultiens wrote:
> > After some digging I noticed that there might be a data-barrier
> > problem, where peripheral register access can become out-of-order.
> > The ARM has the __sync_synchronize() (via gcc) to insert DMB
> > (data-memory-barrier) instructions when you need to guarantee
> > ordering. Inserting DMB made things worse, such that sometimes the
> > setup is 32MHz and sometimes 50MHz. Well, actually, it exposes a
> > deeper problem.
>
> I suddenly realized that the SPI peripheral is configured and accessed
> in userspace. That will fail miserably if not extremely careful,
> especially on SMP.
>
> However, there already is a solution for this! The linux kernel has a
> SPI driver, which is quite good (used it before). It solves all the
> userspace problems and, to say the least, some very clever people have
> had a crack at this problem before.
>
> So, drop userspace access to the hardware and use the kernel
> interface.
>
> 1 - run raspi-config and enable the kernel SPI driver -> reboot
> 2 - you should now have /dev/spidev0.[01]
> 3 - replace the hm2_rpspi.c file in src/hal/drivers/mesa-hostmot2/
> with the one attached
> 4 - recompile
> 5 - run
>
> Gene, with my (4GB) SD image do (you know the cmdline):
> - run raspi-config to enable the kernel SPI driver and reboot
> - save attached file on SD card as hm2_rpspi.c.new
> $ cp hm2_rpspi.c.new
> linuxcnc-git/src/hal/drivers/mesa-hostmot2/hm2_rpspi.c $ cd
> linuxcnc-git/src
> $ make
> $ sudo make setuid
> $ ../scripts/linuxcnc -v
> ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
>
> I get the result as shown in the images. The advantage is that there
> are no inter-word delays anymore. The time for chip select to be
> active varies a bit (16.5 us in image 8 vs. 6.3 us data transfer in
> image 6). This variability is inherent to the driver how it handles
> chip select asynchronously. Still, it is an improvement, especially
> for large transfers.
>
> I did add a rtapi module parameter - spiclk - which should set the
> clock upon load. However, I've not been able to get that to work.
> Probably something trivial I did wrong. Ah well, at least it runs at a
> good speed now, also after reboots.
>
> My hm2_rpspi driver hack is fixed to /dev/spidev0.0 (CE0 pin). This
> should be a configurable parameter (features for the future).

Hi Bertho; I haven't heard any more, so I am wondering if you've found
any more "magic beans"?

I managed to get the box with the rebuilt controller mounted on the
outside of the motor drivers box door today, but nothing is wired up
yet. This is with the pi with the teeny capacitor (about 14 pf) loading
the clocking line, which seems to be about bulletproof in that so far it
has worked every time. This is with the new hm2_rpspi.c you had me
build.

So tomorrow, I'll start wiring this one back into the system. It will
take some time though as I either have to splice & extend the cables, or
start from scratch with longer cabling. I'll have to take inventory as
finding really suitable cabling is a bit of a chore out here in the
puckerbrush of West Virginia.

This box is ground isolated as I mounted it on insulation strips, except
for its power cord, so the first thing will be to extend the single
point ground to include this box, and cut the static ground in the power
cord off the back of the socket. Details, details. Boring.


Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Bertho Stultiens]

On 06/01/2017 02:13 AM, Gene Heskett wrote:
> Hi Bertho; I haven't heard any more, so I am wondering if you've found
> any more "magic beans"?

Still looking into options.

There is that persy dayjob taking away attention from the real work ;-)
Seriously, it is exams-period and that requires some minimum attention
from my side. The least I owe my students is to have read their work.


--
Greetings Bertho

(disclaimers are disclaimed)

[Bertho Stultiens]

On 06/01/2017 02:13 AM, Gene Heskett wrote:
> Hi Bertho; I haven't heard any more, so I am wondering if you've found
> any more "magic beans"?

Yes, I think I've tracked down (most of) the problem(s). There are
several factors that play a role. Not all are solved or maybe solvable,
but the timing is much more stable, and very fast most of the time.


Problem 1:
The RPI3 has dynamic frequency scaling activated by default (ondemand
governor). This makes the Pi hop between 600MHz and 1.2GHz core
frequency. Very annoying and makes realtime rather unpredictable.

Add a line to /etc/rc.local:
echo -n performance >
/sys/devices/system/cpu/cpufreq/policy0/scaling_governor

This will disable the frequency hopping on boot and lock the cores to
1.2GHz.


Problem 2:
The SPI peripheral input frequency appears to be hopping between 400MHz
and about 250MHz. This probably originates somewhere in the Linux kernel
as the kernel is in charge of clock-generation.

Changing the cpu's frequency scaling governor to "performance" makes the
clock more stable, hanging out at 400MHz most of the time, but I've seen
lower frequencies once in a while. The spidev driver actually reads the
(peripheral) clock frequency before a transfer starts and sets the
divider for each transfer. However, I have, at this moment, no clue how
to read the clock setting in userspace. This is a currently a minor
issue and should not give rise to major problems.


Problem 3:
The hm2_rpspi driver was miscalculating the clock-divider by statically
setting it to 8. This results in 50MHz SPI clock (400/8), which is what
we've been seeing. The 32MHz clock is the also explained by the
peripheral clock switching to ~250MHz.

The code is now changed to do the calculation correct and is
configurable with a module parameter "spiclk_rate" to set the clock (in
kHz) and defaults to 33000kHz.

It should be noted that the clock divider must be an even number,
resulting in frequencies of 200MHz, 100MHz, 66.6MHz, 50MHz, 40MHz,
33.3MHz, 28.5MHz, 25MHz, etc.. down to 6.1kHz.


Problem 4:
The hm2_rpspi driver was missing memory barriers in the peripheral
write/read operations. This resulted in inconsistencies. The code now
uses memory barrier instructions to ensure consistency.


I also unified the driver-code a bit and fixed some other problems I
could see. It should be functional. The SPI transfers are faster now
too. I've removed the inter-word delays by using the device's FIFO
properly. The chip-select latency is also minimal (factor 2..3 better
than spidev), but it can vary a bit if a (scheduling) interrupt delays
ending the transfer. The cookie-read is done in 5.7us versus 16.5us
using spidev.


Attached are the changed files. The .c and .h file are the modified
driver. These go in the source-tree at
"~/linuxcnc-git/src/hal/drivers/mesa-hostmot2/" (after copy do cd into
src and do make etc). The .hal and .ini files are the modified configs
for the sheldon lathe, adding the frequency configuration parameter, and
go in "~/linuxcnc/configs/sheldon-lathe/".

Try the changes if you will. I'm trying to build a new SD card with X
installed, but you should not stay awake for it to be finished (upload
alone of the compressed images takes already 3..4 hours).

--
Greetings Bertho

(disclaimers are disclaimed)

[Gene Heskett]

On Thursday 01 June 2017 19:46:29 Bertho Stultiens wrote:

> On 06/01/2017 02:13 AM, Gene Heskett wrote:
> > Hi Bertho; I haven't heard any more, so I am wondering if you've
> > found any more "magic beans"?
>
> Yes, I think I've tracked down (most of) the problem(s). There are
> several factors that play a role. Not all are solved or maybe
> solvable, but the timing is much more stable, and very fast most of
> the time.
>
>
> Problem 1:
> The RPI3 has dynamic frequency scaling activated by default (ondemand
> governor). This makes the Pi hop between 600MHz and 1.2GHz core
> frequency. Very annoying and makes realtime rather unpredictable.
>
> Add a line to /etc/rc.local:
> echo -n performance >
> /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
>
> This will disable the frequency hopping on boot and lock the cores to
> 1.2GHz.
>
Done but not rebooted for effect.
>
> Problem 2:
> The SPI peripheral input frequency appears to be hopping between
> 400MHz and about 250MHz. This probably originates somewhere in the
> Linux kernel as the kernel is in charge of clock-generation.
>
> Changing the cpu's frequency scaling governor to "performance" makes
> the clock more stable, hanging out at 400MHz most of the time, but
> I've seen lower frequencies once in a while. The spidev driver
> actually reads the (peripheral) clock frequency before a transfer
> starts and sets the divider for each transfer. However, I have, at
> this moment, no clue how to read the clock setting in userspace. This
> is a currently a minor issue and should not give rise to major
> problems.
>
>
> Problem 3:
> The hm2_rpspi driver was miscalculating the clock-divider by
> statically setting it to 8. This results in 50MHz SPI clock (400/8),
> which is what we've been seeing. The 32MHz clock is the also explained
> by the peripheral clock switching to ~250MHz.
>
> The code is now changed to do the calculation correct and is
> configurable with a module parameter "spiclk_rate" to set the clock
> (in kHz) and defaults to 33000kHz.
>
> It should be noted that the clock divider must be an even number,
> resulting in frequencies of 200MHz, 100MHz, 66.6MHz, 50MHz, 40MHz,
> 33.3MHz, 28.5MHz, 25MHz, etc.. down to 6.1kHz.
>
>
> Problem 4:
> The hm2_rpspi driver was missing memory barriers in the peripheral
> write/read operations. This resulted in inconsistencies. The code now
> uses memory barrier instructions to ensure consistency.
>
Sounds for sure like it ought to be more stable.
>
> I also unified the driver-code a bit and fixed some other problems I
> could see. It should be functional. The SPI transfers are faster now
> too. I've removed the inter-word delays by using the device's FIFO
> properly. The chip-select latency is also minimal (factor 2..3 better
> than spidev), but it can vary a bit if a (scheduling) interrupt delays
> ending the transfer. The cookie-read is done in 5.7us versus 16.5us
> using spidev.
>
>
> Attached are the changed files. The .c and .h file are the modified
> driver. These go in the source-tree at
> "~/linuxcnc-git/src/hal/drivers/mesa-hostmot2/" (after copy do cd into
> src and do make etc). The .hal and .ini files are the modified configs
> for the sheldon lathe, adding the frequency configuration parameter,
> and go in "~/linuxcnc/configs/sheldon-lathe/".
>
> Try the changes if you will.
I'll take a look. But when I had assembled the board stack and was
checking it out while sitting on the table saws table, I had to spend a
day sorting the .hal file out, it looked like I had last edited it with
gedit, and I am glad I had a printout dated the 10th so I could restore
it. gedit is a major screwup looking for a place to happen. I
blacklisted it over a year ago as it scrambles the contents of a file
around pretty reliably, so I've been using geany ever since. Yeah, it
has some warts, but nothing to compare to gedits open boils. Kate or
Kwrite might be ok, but I could never fall in love with either.

ATM, I have hit a bad path thru the 7i90 to the direction output from the
pwmgen.0 on gpio-21 to pin p1-43, thence thru the 7i42TA, its stuck on
ground, meaning if I had power on the vfd, it would be running the motor
backwards, so first thing tomorrow I have to remove at least 2 of the
7i42TA's to get under them, and after putting a bad card marker on it
try another 7i90 as I must have built it with a bad card I had
previously blown.

I need to ask Peter if he repairs them, and if so how much $. Seems I am
doing it in wholesale qty's, or I picked up the wrong card, I have 4
here, and I know for sure 3 are damaged. Power on the vfd & stepper
drivers is next.

But I'm hoping the 7i42TA's will solve the blown 7i90 problem.

> I'm trying to build a new SD card with X
> installed, but you should not stay awake for it to be finished (upload
> alone of the compressed images takes already 3..4 hours).

Ouch.

Thank you Bertho.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Bertho Stultiens]

On 06/02/2017 01:46 AM, Bertho Stultiens wrote:
> Problem 1:
> The RPI3 has dynamic frequency scaling activated by default
> (ondemand governor). This makes the Pi hop between 600MHz and 1.2GHz
> core frequency. Very annoying and makes realtime rather
> unpredictable.

There are actually two lines that must be added to rc.local:
echo -n 1200000 > /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq
echo -n performance >
/sys/devices/system/cpu/cpufreq/policy0/scaling_governor

The first one forces to hop to 1.2GHz. The current cpu frequency,
apparently, may otherwise hang onto the 600MHz value (in
.../cpu0/cpufreq/cpuinfo_cur_freq).


> Problem 2:
> The SPI peripheral input frequency appears to be hopping between
> 400MHz and about 250MHz. This probably originates somewhere in the
> Linux kernel as the kernel is in charge of clock-generation.

There may be an interaction with the power supply and the frequency-lock
of one of the BCM283[567] PLLs.

Several power-glitches caused the system to crash after I installed a
full graphical desktop version. It turned out I was using a cheapo usb
supply with bad regulation.

The problems seem to stop now that I have attached a bench-PSU set to 5V
directly on the 40-pin header's 5V input. I needed to increase the
voltage to 5.1V after the red LED still was blinking once in a while
(probably indicating too much noise on the power line with my long wires
without decoupling capacitor on the end).

It may be worth checking the actual 5V line for noise too.


--
Greetings Bertho

(disclaimers are disclaimed)

[Gene Heskett]

On Friday 02 June 2017 17:29:24 Bertho Stultiens wrote:

> On 06/02/2017 01:46 AM, Bertho Stultiens wrote:
> > Problem 1:
> > The RPI3 has dynamic frequency scaling activated by default
> > (ondemand governor). This makes the Pi hop between 600MHz and 1.2GHz
> > core frequency. Very annoying and makes realtime rather
> > unpredictable.
>
> There are actually two lines that must be added to rc.local:
> echo -n 1200000 >
> /sys/devices/system/cpu/cpufreq/policy0/scaling_min_freq

Added also.

> echo -n performance >
> /sys/devices/system/cpu/cpufreq/policy0/scaling_governor
>
> The first one forces to hop to 1.2GHz. The current cpu frequency,
> apparently, may otherwise hang onto the 600MHz value (in
> .../cpu0/cpufreq/cpuinfo_cur_freq).
>
> > Problem 2:
> > The SPI peripheral input frequency appears to be hopping between
> > 400MHz and about 250MHz. This probably originates somewhere in the
> > Linux kernel as the kernel is in charge of clock-generation.
>
> There may be an interaction with the power supply and the
> frequency-lock of one of the BCM283[567] PLLs.
>
> Several power-glitches caused the system to crash after I installed a
> full graphical desktop version. It turned out I was using a cheapo usb
> supply with bad regulation.
>
> The problems seem to stop now that I have attached a bench-PSU set to
> 5V directly on the 40-pin header's 5V input. I needed to increase the
> voltage to 5.1V after the red LED still was blinking once in a while
> (probably indicating too much noise on the power line with my long
> wires without decoupling capacitor on the end).
>
> It may be worth checking the actual 5V line for noise too.

5mv? 4 amp box, haven't had a problem.

Thanks Bertho.

One thing I've noted is that in one switch starts both the monitor and
the pi, the monitor isn't ready I assume to respond to an EDID query by
the time the pi issues it. So I've had to pull the pi's power plug for
about 2 or 3 seconds mid-boot. Then plug in the teeny usb plug again,
and I get video on that reboot.

Is there quick and likely dirty way to make the pi wait 2 or 3 seconds
before loading things up?

Thank you Bertho.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Bertho Stultiens]

On 06/03/2017 12:20 AM, Gene Heskett wrote:
[snip]
>> The problems seem to stop now that I have attached a bench-PSU set to
>> 5V directly on the 40-pin header's 5V input. I needed to increase the
>> voltage to 5.1V after the red LED still was blinking once in a while
>> (probably indicating too much noise on the power line with my long
>> wires without decoupling capacitor on the end).
>>
>> It may be worth checking the actual 5V line for noise too.
>
> 5mv? 4 amp box, haven't had a problem.

And you do not see any red led on the pi blink then?

If yes, then you man need to add decoupling.


[snip]
> One thing I've noted is that in one switch starts both the monitor and
> the pi, the monitor isn't ready I assume to respond to an EDID query by
> the time the pi issues it. So I've had to pull the pi's power plug for
> about 2 or 3 seconds mid-boot. Then plug in the teeny usb plug again,
> and I get video on that reboot.
>
> Is there quick and likely dirty way to make the pi wait 2 or 3 seconds
> before loading things up?

See:
https://www.raspberrypi.org/documentation/configuration/config-txt/boot.md

- bootcode_delay - seems the one you are looking for.
From the docs: "This is particularly useful to insert a delay before
reading the EDID of the monitor, for example if the Pi and monitor are
powered from the same source, but the monitor takes longer to start up
than the Pi."


--
Greetings Bertho

(disclaimers are disclaimed)

[Gene Heskett]

On Friday 02 June 2017 18:35:32 Bertho Stultiens wrote:

> On 06/03/2017 12:20 AM, Gene Heskett wrote:
> [snip]
>
> >> The problems seem to stop now that I have attached a bench-PSU set
> >> to 5V directly on the 40-pin header's 5V input. I needed to
> >> increase the voltage to 5.1V after the red LED still was blinking
> >> once in a while (probably indicating too much noise on the power
> >> line with my long wires without decoupling capacitor on the end).
> >>
> >> It may be worth checking the actual 5V line for noise too.
> >
> > 5mv? 4 amp box, haven't had a problem.
>
> And you do not see any red led on the pi blink then?
>
> If yes, then you man need to add decoupling.
>
Solid as a rock.
>
> [snip]
>
> > One thing I've noted is that in one switch starts both the monitor
> > and the pi, the monitor isn't ready I assume to respond to an EDID
> > query by the time the pi issues it. So I've had to pull the pi's
> > power plug for about 2 or 3 seconds mid-boot. Then plug in the teeny
> > usb plug again, and I get video on that reboot.
> >
> > Is there quick and likely dirty way to make the pi wait 2 or 3
> > seconds before loading things up?
>
> See:
> https://www.raspberrypi.org/documentation/configuration/config-txt/boo
>t.md
>
> - bootcode_delay - seems the one you are looking for.

Absolutely Bertho. I just set a 3 second delay & we'll see how the next
reboot works.

> From the docs: "This is particularly useful to insert a delay before
> reading the EDID of the monitor, for example if the Pi and monitor are
> powered from the same source, but the monitor takes longer to start up
> than the Pi."

Thank you Bertho.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Bertho Stultiens]

On 06/03/2017 01:08 AM, Gene Heskett wrote:
>>> 5mv? 4 amp box, haven't had a problem.
>> And you do not see any red led on the pi blink then?
>> If yes, then you man need to add decoupling.
> Solid as a rock.

That should be fine then.


Something different:
I see that the hm2_rpspi module is added to the servo-thread. I am
wondering if the control-loop will be more stable if one of the ARM
cores is isolated and assigned specifically to the servo-thread.

I do not know the rtapi internals too well, so I'm wondering if CPU
affinity has been implemented in the code. For the Pi, it would probably
be a real-time advantage to have the servo-thread isolated on one core
and have the core for itself the whole time.

Has anybody done an implementation of affinity in linuxcnc already? If
yes, how is it setup?

--
Greetings Bertho

(disclaimers are disclaimed)

[Gene Heskett]

On Friday 02 June 2017 19:59:52 Bertho Stultiens wrote:

> On 06/03/2017 01:08 AM, Gene Heskett wrote:
> >>> 5mv? 4 amp box, haven't had a problem.
> >>
> >> And you do not see any red led on the pi blink then?
> >> If yes, then you man need to add decoupling.
> >
> > Solid as a rock.
>
> That should be fine then.
>
>
> Something different:
> I see that the hm2_rpspi module is added to the servo-thread. I am
> wondering if the control-loop will be more stable if one of the ARM
> cores is isolated and assigned specifically to the servo-thread.
>
> I do not know the rtapi internals too well, so I'm wondering if CPU
> affinity has been implemented in the code. For the Pi, it would
> probably be a real-time advantage to have the servo-thread isolated on
> one core and have the core for itself the whole time.
>
> Has anybody done an implementation of affinity in linuxcnc already? If
> yes, how is it setup?

On the x86 stuff, in years past, we used "isolcpus"=3 (or whatever was
the last core) as a kernel argument at kernel load time.

On x86 stuff the cpu you have isolated is found and used automatically,
however the use cannot be detected by the various cpu monitoring
utilities such as htop. I had originally set that up to reserve core 3,
on the 4 core pi, and on core 3 of the athlons I don't know who
corrected me, but it was said that was no longer a useful item, so I
took it back out of /boot/cmdline.txt and out of some of the x86
machines too, but I see its still in as an =1 on the two atom boards out
in the shop. The one I removed it from hasn't indicated any unhappiness
with that turn of events.

Perhaps Sebastian or someone more familiar with it can shine a light on
us here? As in has it been deprecated on all arches. I still have 2
D525MW's with it in effect. And htop cannot see any usage of cpu-1 on
either of those 2 boxes.

Cheers Bertho, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Bertho Stultiens]

On 06/03/2017 03:36 AM, Gene Heskett wrote:
>> Has anybody done an implementation of affinity in linuxcnc already? If
>> yes, how is it setup?
>
> On the x86 stuff, in years past, we used "isolcpus"=3 (or whatever was
> the last core) as a kernel argument at kernel load time.
>
> On x86 stuff the cpu you have isolated is found and used automatically,
> however the use cannot be detected by the various cpu monitoring
> utilities such as htop. I had originally set that up to reserve core 3,
> on the 4 core pi, and on core 3 of the athlons I don't know who
> corrected me, but it was said that was no longer a useful item, so I
> took it back out of /boot/cmdline.txt and out of some of the x86
> machines too, but I see its still in as an =1 on the two atom boards out
> in the shop. The one I removed it from hasn't indicated any unhappiness
> with that turn of events.
>
> Perhaps Sebastian or someone more familiar with it can shine a light on
> us here? As in has it been deprecated on all arches. I still have 2
> D525MW's with it in effect. And htop cannot see any usage of cpu-1 on
> either of those 2 boxes.

Yes, isolcpus=X is the way to do that. However, doing so means that the
kernel never schedules anything on the core automatically. You must set
the thread/process affinity in the running code, using a syscall, to
move a specific thread/process onto that isolated core. Otherwise, it
will be idle all the time.


--
Greetings Bertho

(disclaimers are disclaimed)

[Gene Heskett]

On Friday 02 June 2017 21:50:13 Bertho Stultiens wrote:

> On 06/03/2017 03:36 AM, Gene Heskett wrote:
> >> Has anybody done an implementation of affinity in linuxcnc already?
> >> If yes, how is it setup?
> >
> > On the x86 stuff, in years past, we used "isolcpus"=3 (or whatever
> > was the last core) as a kernel argument at kernel load time.
> >
> > On x86 stuff the cpu you have isolated is found and used
> > automatically, however the use cannot be detected by the various cpu
> > monitoring utilities such as htop. I had originally set that up to
> > reserve core 3, on the 4 core pi, and on core 3 of the athlons I
> > don't know who corrected me, but it was said that was no longer a
> > useful item, so I took it back out of /boot/cmdline.txt and out of
> > some of the x86 machines too, but I see its still in as an =1 on the
> > two atom boards out in the shop. The one I removed it from hasn't
> > indicated any unhappiness with that turn of events.
> >
> > Perhaps Sebastian or someone more familiar with it can shine a light
> > on us here? As in has it been deprecated on all arches. I still have
> > 2 D525MW's with it in effect. And htop cannot see any usage of cpu-1
> > on either of those 2 boxes.
>
> Yes, isolcpus=X is the way to do that. However, doing so means that
> the kernel never schedules anything on the core automatically. You
> must set the thread/process affinity in the running code, using a
> syscall, to move a specific thread/process onto that isolated core.
> Otherwise, it will be idle all the time.

When I awoke about sunup, I had a message from amanda. I gad left lcnc
running when I came in for the and the pi apparently crashed very
quickly when amanda tried to access it.

So when I ran down at restoring it today, with 8 wires yet to hook up
verify I have them right, and I'll be at the same place I was,
everything running nicely, the morning of May 2, but I shut lcnc down
before hitting the light switch and closing the garage door for the
night. On my feet a good share of the day, I'll pay for it in the night
with leg cramps. I'll take an extra big b12 pill, which sometimes helps
because metformin flushes b12 out into a big white bowl. :)

I did write the new image to a card, but have not had the time to go into
it, setting hostnames, hosts etc, yadda, yadda yet. And killing
anything that can over-write /etc/resolv.conf thereby keeping my local
network running on the pi like it runs on everything else.

I did plug in that terabyte HD and had mc update the linuxcnc tree
already on that drive from a couple weeks ago, and its still plugged in.

And I'm bushed, but progress has been made. Curious, before I quit, I
fired up the scope and looked at everything I had hooked up the last 3
days, and was amazed! Whereas the old lashup was attacking the 7i90
with switching noises up in the 90+ mhz range, at peak voltages of 30+
volts before it hit any wire by wire filtering I'd been building, if
there is any noise there now, its well under 50 millivolts and I can't
see it on the analog scope at all. This was with the vfd up and running,
spinning the chuck at about 225 rpms. And yet every cable I am hooking
up is coming into the bottom of the motor drivers box, thru it just as
before. The cpu/controller box is insulated from the power box, and only
connected to the single point ground by one braided strap, which I had
to cover with shrink tubing because it was touching the spinx1 which I'd
remounted on the inside of the door. Didn't blow anything but the vfd
was going crazy when I moved the boxes door, which now has the pi &
controller box on the outside of the door.

Thanks Bertho, a lot.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[john mcintyre]

Good day Gene ,

Your saga continues, you must be related to a bulldog😊 ,once locked on it cannot let go.

All the best john


________________________________
From: Gene Heskett <***@shentel.net>
Sent: Sunday, 4 June 2017 9:47 AM
To: emc-***@lists.sourceforge.net
Subject: Re: [Emc-users] RPI saga continues - SPI probably solved

On Friday 02 June 2017 21:50:13 Bertho Stultiens wrote:

> On 06/03/2017 03:36 AM, Gene Heskett wrote:
> >> Has anybody done an implementation of affinity in linuxcnc already?
> >> If yes, how is it setup?
> >
> > On the x86 stuff, in years past, we used "isolcpus"=3 (or whatever
> > was the last core) as a kernel argument at kernel load time.
> >
> > On x86 stuff the cpu you have isolated is found and used
> > automatically, however the use cannot be detected by the various cpu
> > monitoring utilities such as htop. I had originally set that up to
> > reserve core 3, on the 4 core pi, and on core 3 of the athlons I
> > don't know who corrected me, but it was said that was no longer a
> > useful item, so I took it back out of /boot/cmdline.txt and out of
> > some of the x86 machines too, but I see its still in as an =1 on the
> > two atom boards out in the shop. The one I removed it from hasn't
> > indicated any unhappiness with that turn of events.
> >
> > Perhaps Sebastian or someone more familiar with it can shine a light
> > on us here? As in has it been deprecated on all arches. I still have
> > 2 D525MW's with it in effect. And htop cannot see any usage of cpu-1
> > on either of those 2 boxes.
>
> Yes, isolcpus=X is the way to do that. However, doing so means that
> the kernel never schedules anything on the core automatically. You
> must set the thread/process affinity in the running code, using a
> syscall, to move a specific thread/process onto that isolated core.
> Otherwise, it will be idle all the time.

When I awoke about sunup, I had a message from amanda. I gad left lcnc
running when I came in for the and the pi apparently crashed very
quickly when amanda tried to access it.

So when I ran down at restoring it today, with 8 wires yet to hook up
verify I have them right, and I'll be at the same place I was,
everything running nicely, the morning of May 2, but I shut lcnc down
before hitting the light switch and closing the garage door for the
night. On my feet a good share of the day, I'll pay for it in the night
with leg cramps. I'll take an extra big b12 pill, which sometimes helps
because metformin flushes b12 out into a big white bowl. :)

I did write the new image to a card, but have not had the time to go into
it, setting hostnames, hosts etc, yadda, yadda yet. And killing
anything that can over-write /etc/resolv.conf thereby keeping my local
network running on the pi like it runs on everything else.

I did plug in that terabyte HD and had mc update the linuxcnc tree
already on that drive from a couple weeks ago, and its still plugged in.

And I'm bushed, but progress has been made. Curious, before I quit, I
fired up the scope and looked at everything I had hooked up the last 3
days, and was amazed! Whereas the old lashup was attacking the 7i90
with switching noises up in the 90+ mhz range, at peak voltages of 30+
volts before it hit any wire by wire filtering I'd been building, if
there is any noise there now, its well under 50 millivolts and I can't
see it on the analog scope at all. This was with the vfd up and running,
spinning the chuck at about 225 rpms. And yet every cable I am hooking
up is coming into the bottom of the motor drivers box, thru it just as
before. The cpu/controller box is insulated from the power box, and only
connected to the single point ground by one braided strap, which I had
to cover with shrink tubing because it was touching the spinx1 which I'd
remounted on the inside of the door. Didn't blow anything but the vfd
was going crazy when I moved the boxes door, which now has the pi &
controller box on the outside of the door.

Thanks Bertho, a lot.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>
[http://geneslinuxbox.net:6309/gene/pix/EasterSundayCropped2004-1.jpg]<http://geneslinuxbox.net:6309/gene>

Gene's Web pages<http://geneslinuxbox.net:6309/gene>
geneslinuxbox.net
Welcome to Gene's web pages. Here you will find some of the things that make me tick, and that help keep me out of the bars. That is me & the missus, Dee (Elladene) I ...




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Emc-users mailing list
Emc-***@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/emc-users
Emc-users -- Enhanced Machine Controller (EMC)<https://lists.sourceforge.net/lists/listinfo/emc-users>
lists.sourceforge.net
This list is for users of the Enhanced Machine Controller (EMC). Topics include how to obtain, install, configure, and use EMC, as well as other general EMC related ...

[Bertho Stultiens]

On 06/04/2017 06:15 AM, john mcintyre wrote:
> Your saga continues, you must be related to a bulldog😊 ,once locked
> on it cannot let go.

Isn't this what you call a challenge?


Anyway, I've fixed a couple of things, like endianness, which I did
wrong in the previous versions. This implementation is hard-coded to use
the RPI hardware peripherals. Also, you can now attach to SPI0 or SPI1
ports on the 40-pin header and you may choose any CE line you wish
(updated man page also attached).

My tests so far have seen stable results, timing wise, although I cannot
test functionally on a real system (lacking a 7i90 in my inventory).
Though, the timing has improved with respect to the original version.

Read and write SPI clock frequencies may be different now. This will
help to overcome the SPI round-trip delay while reading (just lower the
read frequency setting), or when you want buffers on the SPI lines.

So, if anybody wants to have a go and test this, please do so and tell
me how it goes. In the meantime... need to get my hands on a 7i90 (and
upgrade the mill at the local hackerspace).

--
Greetings Bertho

(disclaimers are disclaimed)

[Peter C. Wallace]

On Mon, 5 Jun 2017, Bertho Stultiens wrote:

> Date: Mon, 5 Jun 2017 16:20:03 +0200
> From: Bertho Stultiens <***@vagrearg.org>
> Reply-To: "Enhanced Machine Controller (EMC)"
> <emc-***@lists.sourceforge.net>
> To: "Enhanced Machine Controller (EMC)" <emc-***@lists.sourceforge.net>
> Subject: Re: [Emc-users] RPI saga continues - SPI probably solved
>
> On 06/04/2017 06:15 AM, john mcintyre wrote:
>> Your saga continues, you must be related to a bulldog? ,once locked
>> on it cannot let go.
>
> Isn't this what you call a challenge?
>
>
> Anyway, I've fixed a couple of things, like endianness, which I did
> wrong in the previous versions. This implementation is hard-coded to use
> the RPI hardware peripherals. Also, you can now attach to SPI0 or SPI1
> ports on the 40-pin header and you may choose any CE line you wish
> (updated man page also attached).
>
> My tests so far have seen stable results, timing wise, although I cannot
> test functionally on a real system (lacking a 7i90 in my inventory).
> Though, the timing has improved with respect to the original version.
>
> Read and write SPI clock frequencies may be different now. This will
> help to overcome the SPI round-trip delay while reading (just lower the
> read frequency setting), or when you want buffers on the SPI lines.
>
> So, if anybody wants to have a go and test this, please do so and tell
> me how it goes. In the meantime... need to get my hands on a 7i90 (and
> upgrade the mill at the local hackerspace).
>
> --
> Greetings Bertho
>
> (disclaimers are disclaimed)
>

Thank you for working on this! The ability to set the clock speed should be
especially valuable. It means I can make a Async SPI interface version of the
firmware (as opposed to the current hardware clocked version), that
oversamples/digital filters the input data and clock for better noise
resistance. This will probably require lowering the SPI clock to say 16 MHz
to accomodate the increased turnaround delays from the digital filtering.

Peter Wallace
Mesa Electronics

[Gene Heskett]

On Monday 05 June 2017 10:20:03 Bertho Stultiens wrote:

> On 06/04/2017 06:15 AM, john mcintyre wrote:
> > Your saga continues, you must be related to a bulldog😊 ,once locked
> > on it cannot let go.
>
> Isn't this what you call a challenge?
>
>
> Anyway, I've fixed a couple of things, like endianness, which I did
> wrong in the previous versions. This implementation is hard-coded to
> use the RPI hardware peripherals. Also, you can now attach to SPI0 or
> SPI1 ports on the 40-pin header and you may choose any CE line you
> wish (updated man page also attached).
>
> My tests so far have seen stable results, timing wise, although I
> cannot test functionally on a real system (lacking a 7i90 in my
> inventory). Though, the timing has improved with respect to the
> original version.
>
> Read and write SPI clock frequencies may be different now. This will
> help to overcome the SPI round-trip delay while reading (just lower
> the read frequency setting), or when you want buffers on the SPI
> lines.
>
> So, if anybody wants to have a go and test this, please do so and tell
> me how it goes. In the meantime... need to get my hands on a 7i90 (and
> upgrade the mill at the local hackerspace).

Does the image I downloaded Saturday and wrote to an sd card yesterday
have this stuff in it? I got sidetracked today, doing a poor man's
set-true to the chuck, and getting it under a thou. Can't go any farther
w/o reaming the bolt holes in the backing plate a wee bit. Thats pretty
close, and I've no reamer kit. Yet. Just one of several items I need to
acquire, like a live center which I didn't get with it. And of course
more tooling, never have enough.

Back when I bought the ER-40 kit, someone said I couldn't use long
workpieces with it, but it clears several feet of nominally 7/8" OD
stock, so as long as the OD is no more than that, I don't see a problem,
but I'll have to make a spider for the left end of the spindle, easily
done if I can find some brass acorn nuts.

I'll see if I can pull the cover tomorrow and try the new sd card,
Bertho. Depends on the weather.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Bertho Stultiens]

On 06/06/2017 03:30 AM, Gene Heskett wrote:
> Does the image I downloaded Saturday and wrote to an sd card yesterday
> have this stuff in it?

No, you need to pull the most recent source-files (hm2_rpspi.c and
spi_common_rpspi.h) from the mailing list (or the forum).

Login as pi user and put them in
~/linuxcnc-git/src/hal/drivers/mesa-hostmot2/

Then do:
$ cd ~/linuxcnc-git/src
$ make
$ sudo make setuid
$ ../scripts/linuxcnc -v ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini

That should run your sheldon lathe config. You may need to update the
config as needed.

--
Greetings Bertho

(disclaimers are disclaimed)

[Gene Heskett]

On Tuesday 06 June 2017 11:46:27 Bertho Stultiens wrote:

> On 06/06/2017 03:30 AM, Gene Heskett wrote:
> > Does the image I downloaded Saturday and wrote to an sd card
> > yesterday have this stuff in it?
>
> No, you need to pull the most recent source-files (hm2_rpspi.c and
> spi_common_rpspi.h) from the mailing list (or the forum).
>
> Login as pi user and put them in
> ~/linuxcnc-git/src/hal/drivers/mesa-hostmot2/
>
Since I saved them on this machine and have not made sshfs live yet,
Thats best done by bringing the card in, plugging it into a reader to do
the transfers.

I have had it working a time or three with the buildbots old 2.8.0-pre,
but its quite fragile, never crashing at the same point twice.

> Then do:
> $ cd ~/linuxcnc-git/src
> $ make
> $ sudo make setuid
> $ ../scripts/linuxcnc -v
> ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
>
> That should run your sheldon lathe config. You may need to update the
> config as needed.

I've rsync'd that out to a backup drive from the card that was in it for
the last 4 or so days, then when this new finally booted to a usable
screen, rsynced it back in order to get the most recent edits in place.
ANAICT, everything works, but its very, very fragile, freezing up at the
drop of a 4-40 screw 10 feet away.

I was going to print the README.md, and build according to that, but
found that while the package manager says cups-client and friends is
installed, its not usable as there is no /etc/cups/cups-client.conf.

I haven't sorted that basket of rattlesnakes yet, been sitting on the
rider, trying to find my yard. That, by the time I loaded the 3x5x1
foot deep 2 wheeled wagon full of hedge trimmings stacked about 3 feet
high, to haul it to the burn pit wrote a ~30~ to my back for the day, so
the weedeating around the edges will wait till another dry day.

And its time (17:45) to round up something for dinner. So thats the
status for today. Thank you Bertho.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Gene Heskett]

On Tuesday 06 June 2017 11:46:27 Bertho Stultiens wrote:

> On 06/06/2017 03:30 AM, Gene Heskett wrote:
> > Does the image I downloaded Saturday and wrote to an sd card
> > yesterday have this stuff in it?
>
> No, you need to pull the most recent source-files (hm2_rpspi.c and
> spi_common_rpspi.h) from the mailing list (or the forum).
>
> Login as pi user and put them in
> ~/linuxcnc-git/src/hal/drivers/mesa-hostmot2/
>
> Then do:
> $ cd ~/linuxcnc-git/src
> $ make
I am ready to do the make. I've crashed it at least a dozen times so far
today, but we'll see if it can actually get thru a make. Did, building
only the hm2_rpspi.so

> $ sudo make setuid
did this too.

Now print this so I don't have any typo's when I goto the machine for the
test run.

> $ ../scripts/linuxcnc -v
> ../../linuxcnc/configs/sheldon-lathe/7i90-axis.ini
>
> That should run your sheldon lathe config. You may need to update the
> config as needed.


Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Gene Heskett]

On Sunday 04 June 2017 01:46:57 Erik Christiansen wrote:

> On 03.06.17 19:47, Gene Heskett wrote:
> > On my feet a good share of the day, I'll pay for it in the night
> > with leg cramps. I'll take an extra big b12 pill, which sometimes
> > helps because metformin flushes b12 out into a big white bowl. :)
>
> Gene, a magnesium supplement tablet fixes my nocturnal leg cramps,
> often with ten minutes, if chewed. YMMV, as it perhaps depends on a
> person's condition.

The form of magnesium (oxide) in the OTC stuff has very poor absorbsion,
like 4%. I ran out about 3 weeks ago and haven't picked up another
bottle of those, I probably should, and add 2 in the morning pockets,
and 2 in the evening pockets of my weekly pilltainer
>
> Dunno how you keep going if the B12 goes low, though. Is it the
> sub-lingual pill, as the others mostly go straight through with
> minimal absorption? Since switching to the under-the-tongue pills,
> I've not needed B12 jabs.

I've been washing those down with half a 8 oz glass of water. Big pale
red chalky looking pill I haven't bothered to taste. I just know I need
it. :)
>
Thanks for the reminder about the mag, Erik, its a reminder I needed.

Cheers, Gene Heskett
--
"There are four boxes to be used in defense of liberty:
soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author)
Genes Web page <http://geneslinuxbox.net:6309/gene>

[Erik Christiansen]

On 04.06.17 02:07, Gene Heskett wrote:
> On Sunday 04 June 2017 01:46:57 Erik Christiansen wrote:
> > Dunno how you keep going if the B12 goes low, though. Is it the
> > sub-lingual pill, as the others mostly go straight through with
> > minimal absorption? Since switching to the under-the-tongue pills,
> > I've not needed B12 jabs.
>
> I've been washing those down with half a 8 oz glass of water. Big pale
> red chalky looking pill I haven't bothered to taste. I just know I need
> it. :)

Medicos here insist that there's inadequate B12 absorption from tablets
in the gut, and so I had intramuscular injections for a while. But
absorption directly into the bloodstream, from an appropriate tablet
allowed to dissolve under the tongue, is a more convenient alternative
which I find to work as claimed.

Erik

> Thanks for the reminder about the mag, Erik, its a reminder I needed.

No worries. Now to remind myself to take the vit. D tablets, as it
apparently help somewhat to stave off dementia. ;-)

Erik
(Whose limited vegetarian diet can make a little targeted
supplementation worthwhile.)