Waveforms 2015 Segmentation Fault, Ubuntu 16.04.1, x86-64, Analog Discovery 1, Analog Discovery 2

[clf]

Waveforms 2015 fails with a segmentation fault whenever I connect an Analog Discovery 1 or an Analog Discovery 2.  This happens regardless of whether I connect the AD1/AD2 before starting Waveforms 2015 or while Waveforms 2015 is running.  Waveforms 2015 appears to work when no AD1 / AD2 is connected I'm running Ubuntu 16.04.1 on x86-64.

I installed:

digilent.adept.runtime_2.16.5-amd64.deb
digilent.waveforms_3.3.7_amd64.deb

"lsusb" output for the AD2:

Bus 001 Device 008: ID 0403:6014 Future Technology Devices International, Ltd FT232H Single HS USB-UART/FIFO IC

"uname" output:

$ uname -a
Linux 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Any suggestions?

Thanks,

Chris

[ballessay]

Hi guys,

I recently got my analog discovery 2 and experienced the same behavior an my notebook (but not on my desktop). I did a bit of research/debugging and got a functional but rather ugly workaround.

tl;dr: (re)compile your systems libc without --enable-lock-elision and extend the LD_LIBRARY_PATH of the /usr/bin/waveforms script to that lib-path (maybe better to not install it system wide)

Long version:
I got my AD2 and tried to get it running on my notebook. waveforms started fine without the hardware but always crashed with the AD2 attached. So I tested on my desktop system (practically the same arch linux installation) and it worked (yeah some messages about not enough power at the front usb ports but on a port on the back working fine). The difference my desktop has a i7 5930K, my notebook a i5 6300U - hmmm.

Skylake CPUs support TSX-NI (e. g. my notebook CPU [1]) and glibc has a feature called 'lock-elision' [2]. adept2 seems to have a bug/quirk with its locking which gets triggered in such environments. My stack traces ended, like nickswalkers (in the other thread), at _lll_unlock_elision(). After reading [3], [4] and [5] I decided to compile the arch glibc without the '--enable-lock-elision' configure flag. After copying the waveforms wrapper script to my local script directory (in my case $HOME/bin; which is in my $PATH), adding the the the output path, in my case '$HOME/data/digilent/glibc/usr/lib', to the LD_LIBRARY_PATH and starting waveforms with said script the application seems to work fine (haven't done long time testing).

If someone is interested I can post some more detailed building instructions.

The real solution would be that someone with access to the adept2 source code would look into this.

HTH

[1] http://ark.intel.com/de/products/88190/Intel-Core-i5-6300U-Processor-3M-Cache-up-to-3_00-GHz
[2] https://01.org/blogs/2014/lock-elision-glibc
[3] https://bbs.archlinux.org/viewtopic.php?id=203679
[4] https://bugs.archlinux.org/task/46064
[5] http://patrakov.blogspot.ch/2016/05/is-tsx-busted-on-skylake-too-no-its.html
 

[attila]

Hello,

The devices are working fine for me with the same Ubuntu and software.
I'm not sure what to suspect on your machine.

Do you have any other software installed that could cause this?
Could you try using USB hub?

[clf]

Thanks Attila.

I just did a fresh install of Ubuntu 16.04.1 on a different machine (also x86-64) and the Analog Discovery 2 & Waveforms 2015 is working just fine. There must be something botchy about my first Ubuntu machine.

Sorry for the noise & thanks for the feedback. I'm going to call this solved, as it appears the problem is on my end.

Thanks again!

Chris

[clf]

@attila

Hi Attila,

Can we call this "unsolved" again?  Sorry. 

I'm still experiencing the Waveforms 2015 crash every single time I connect the Analog Discovery 2 to my first machine.  In my previous post, I'd assumed that something must have been wrong with my Ubuntu 16.04.1 install, so I just tried running from a fresh Ubuntu 16.04.1 Live USB drive with persistence.  The crash still occurred.

I also tried running from the fresh Ubuntu 16.04.1 Live USB with a USB hub between the Analog Discovery 2 and the computer.  The crash still occurred.

However, I've had success so far with the problem machine when running Ubuntu 14.04 from a Live USB with persistence.

The problem machine is a Dell Precision 7510.  The only substantial hardware difference I can think of between this problem machine and my other Ubuntu 16.04.1 machine on which Waveforms 2015 seems to run fine is that the Dell Precision 7510 has USB-C.

Have you guys tested on a host with USB-C?  I have the Analog Discovery 2 connected to USB 3.0 ports, but I suspect they share a USB controller with the USB-C ports.

Any other ideas?

During the experiments in this particular post, I installed digilent.adept.runtime_2.16.5-amd64.deb and digilent.waveforms_3.2.5_amd64.deb.

Thanks,

Chris

[clf]

Possibly related:

 

 

[attila]

Hello,

Here is another post with similar glibc problem on Skylake and Broadwell CPUs:

 

[clf]

Another breadcrumb:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=800574

[clf]

I've captured a stack trace that indicates that I am experiencing a lock elision problem.

An Analog Discovery 2 was connected before & during the following commands.

$ export LD_LIBRARY_PATH=/usr/lib/digilent/waveforms/qtlibs

$ gdb /usr/lib/digilent/waveforms/waveforms

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

[...]

(gdb) r
Starting program: /usr/lib/digilent/waveforms/waveforms

Thread 4 "waveforms" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff6267f700 (LWP 3243)]
__lll_unlock_elision (lock=0x7fff5c0281f8, private=0) at ../sysdeps/unix/sysv/linux/x86/elision-unlock.c:29
29    ../sysdeps/unix/sysv/linux/x86/elision-unlock.c: No such file or directory.
(gdb) bt
#0  __lll_unlock_elision (lock=0x7fff5c0281f8, private=0) at ../sysdeps/unix/sysv/linux/x86/elision-unlock.c:29
#1  0x00007fff6146b12f in EventDestroy () from /usr/lib64/digilent/adept/libftd2xx.so
#2  0x00007fff6146557e in FT_Close () from /usr/lib64/digilent/adept/libftd2xx.so
#3  0x00007fffed590a11 in ?? () from /usr/lib64/digilent/adept/libdpcomm.so.2
#4  0x00007fffed58b29b in ?? () from /usr/lib64/digilent/adept/libdpcomm.so.2
#5  0x00007fffed5876f8 in ?? () from /usr/lib64/digilent/adept/libdpcomm.so.2
#6  0x00007fffed5853d6 in ?? () from /usr/lib64/digilent/adept/libdpcomm.so.2
#7  0x00007fffed5848dc in ?? () from /usr/lib64/digilent/adept/libdpcomm.so.2
#8  0x00007ffff573b70a in start_thread (arg=0x7fff6267f700) at pthread_create.c:333
#9  0x00007ffff4bd082d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) q

This was produced on a machine running Ubuntu 16.04.

$ cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 94
model name    : Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz
stepping    : 3
microcode    : 0x88
cpu MHz        : 799.980
cache size    : 8192 KB
physical id    : 0
siblings    : 8
core id        : 0
cpu cores    : 4
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 22
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
bugs        :
bogomips    : 5423.88
clflush size    : 64
cache_alignment    : 64
address sizes    : 39 bits physical, 48 bits virtual
power management:

[...repeats for each CPU core...]

This processor is a 6th-generation Core processor and is a member of the Skylake family (per https://ark.intel.com/products/codename/37572/Skylake#@Mobile).

[clf]

The Ubuntu community is discussing disabling either (a) providing a dynamic means of enabling/disabling lock elision or (b) disabling it entirely:

https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1560634

[clf]

The workaround posted above by @ballessay worked for me.  Using a glibc with lock elision disabled, I no longer experience a Waveforms 2015 on my Intel Core i7-6820HQ machine running Ubuntu 16.04.1 when I connect an Analog Discovery 1 / Analog Discovery 2 device.  I've successfully tried this with both an Analog Discovery 1 and an Analog Discovery 2.  I've successfully tried this when the device is connected before starting Waveforms 2015 and when the device is connected when Waveforms 2015 is already running.

I did update my BIOS and processor microcode to the latest.  Doing so did not resolve the crash.

I was unable to rebuild glibc on Ubuntu using "configure" and "make".

It was straightforward to rebuild glibc when I (loosely) followed the procedure posted by expipiplus1 / MyrtleSoftware at [1].  The build did fail the first time around, reporting that I did not have certain required packages installed.  It built without issue once I installed those packages with "sudo apt-get install".  Per MyrtleSoftware's recommendation, I did perform the build within a virtual machine.

The build process produced a number of output files.  I extracted the output file "libc6_2.23-0ubuntu4_amd64.deb" into a directory under my home directory:

/home/user/digilent-waveforms/glibc-2.23-disable-lock-elision$ dpkg-deb --extract libc6_2.23-0ubuntu4_amd64.deb .

I then modified the "waveforms" script to prepend the customized glibc libraries to the LD_LIBRARY_PATH.

$ sudo vim $(which waveforms)

#!/bin/sh
export LD_LIBRARY_PATH=/usr/lib/digilent/waveforms/qtlibs
export LD_LIBRARY_PATH="/home/user/digilent-waveforms/glibc-2.23-disable-lock-elision/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}"
/usr/lib/digilent/waveforms/waveforms "$@"

I still can't make a ruling on whether the root cause of this problem lies with my machine or with the Waveforms 2015 software.

Thanks guys!

Chris

[1] https://github.com/MyrtleSoftware/glibc-no-lock-elision

tags: "glibc" / "lock-elision" / "pthreads" / "mutex" / "Skylake" / "pthread_create" / "start_thread" / " __lll_unlock_elision" / "elision-unlock"

[clf]

There's discussion in the Ubuntu community of disabling lock elision in future releases, which would hopefully eliminate the need for this workaround.

If you'd like to see that happen, please visit https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1560634 and mark this bug as affecting you.

Chris

[alvarop]

Just want to stop by and say  I am also running into this issue on ubuntu 16.04 on an Intel NUC6i7KYK with a Core i7-6770.

I wonder if there's anything that can be done on the waveforms side so we don't have to re-compile glibc  

[alvarop]

Found a no-compile-needed solution here: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1642390

Looks like the patched libc will be included in ubuntu soon.

[clf]

Thanks for the update, @alvarop.

[lah]

I got this problem today, I've solved it on arch with following steps :
 

yaourt -S asp
asp checkout glibc
cd glibc/
asp update
git pull
cd trunk
# edit PKGBUILD and remove --enable-lock-elision compile flag
# build package
makepkg -cs
tar -xf glibc-2.26-11-x86_64.pkg.tar.xz -C ~/dev_tools/linux/glibc-2.26-11-disable-lock-elision/

Create /usr/local/bin/waveforms_glibc with this content or modify /usr/bin/waveforms :
 

#!/bin/sh
export LD_LIBRARY_PATH=/usr/lib/digilent/waveforms/qtlibs:/home/remi/dev_tools/linux/glibc-2.26-11-disable-lock-elision/usr/lib
/usr/lib/digilent/waveforms/waveforms

chmod a+x /usr/local/bin/waveforms_glibc

Done!