Why won't my cross-compiled CGO binary run on the Raspberry Pi (Raspbian)?

When I compile the following snippet of code (playground link):

<!-- language: go -->

package main
 
/*
#cgo LDFLAGS: -lbluetooth
 
#include <bluetooth/bluetooth.h>
#include <bluetooth/hci.h>
#include <bluetooth/hci_lib.h>
*/
import "C"
 
func main() {
        C.hci_get_route(nil)
}

…using these environment variables:

CC=arm-linux-gnueabihf-gcc
CGO_ENABLED=1
GOARCH=arm
GOARM=6

I get a 1.6Mb binary that:

• works on my ODROID-C1 (ARMv7, running Arch, from archlinuxarm.org), but
• does not work on my Raspberry Pi B+ (ARMv6, running Raspbian, from raspberrypi.org).

When I run it on the Pi, I get a "Segmentation fault".

When I run it with these environment variables:

LD_PRELOAD=/lib/arm-linux-gnueabihf/libSegFault.so
SEGFAULT_USE_ALTSTACK=1

I get this output:

*** Segmentation fault
Register dump:
 R0: 00000000   R1: 00000001   R2: bebf37f4   R3: bebf37fc
 R4: 00000000   R5: 00000000   R6: 00010c59   R7: 00000000
 R8: 00000000   R9: 00000000   SL: b6f33000   FP: 00000000
 IP: 0000004f   SP: bebf37ec   LR: 00010c6b   PC: 0005d7f4
 CPSR: 40000030
 Trap: 0000000e   Error: 00000817   OldMask: 00000000
 Addr: 00000001
Backtrace:
Memory map:
00010000-00125000 r-xp 00000000 b3:02 16193      /home/pi/test
00135000-0013b000 rw-p 00115000 b3:02 16193      /home/pi/test
0013b000-0014a000 rw-p 00000000 00:00 0 
0026b000-00290000 rw-p 00000000 00:00 0          [heap]
b6d53000-b6d73000 r-xp 00000000 b3:02 16080      /lib/arm-linux-gnueabihf/libgcc_s.so.1
b6d73000-b6d7a000 ---p 00020000 b3:02 16080      /lib/arm-linux-gnueabihf/libgcc_s.so.1
b6d7a000-b6d7b000 rw-p 0001f000 b3:02 16080      /lib/arm-linux-gnueabihf/libgcc_s.so.1
b6d7b000-b6e9f000 r-xp 00000000 b3:02 4254       /lib/arm-linux-gnueabihf/libc-2.13.so
b6e9f000-b6ea6000 ---p 00124000 b3:02 4254       /lib/arm-linux-gnueabihf/libc-2.13.so
b6ea6000-b6ea8000 r--p 00123000 b3:02 4254       /lib/arm-linux-gnueabihf/libc-2.13.so
b6ea8000-b6ea9000 rw-p 00125000 b3:02 4254       /lib/arm-linux-gnueabihf/libc-2.13.so
b6ea9000-b6eac000 rw-p 00000000 00:00 0 
b6eac000-b6ec0000 r-xp 00000000 b3:02 4247       /lib/arm-linux-gnueabihf/libpthread-2.13.so
b6ec0000-b6ec7000 ---p 00014000 b3:02 4247       /lib/arm-linux-gnueabihf/libpthread-2.13.so
b6ec7000-b6ec8000 r--p 00013000 b3:02 4247       /lib/arm-linux-gnueabihf/libpthread-2.13.so
b6ec8000-b6ec9000 rw-p 00014000 b3:02 4247       /lib/arm-linux-gnueabihf/libpthread-2.13.so
b6ec9000-b6ecb000 rw-p 00000000 00:00 0 
b6ecb000-b6ee0000 r-xp 00000000 b3:02 44311      /usr/lib/arm-linux-gnueabihf/libbluetooth.so.3.12.0
b6ee0000-b6ee7000 ---p 00015000 b3:02 44311      /usr/lib/arm-linux-gnueabihf/libbluetooth.so.3.12.0
b6ee7000-b6ee8000 r--p 00014000 b3:02 44311      /usr/lib/arm-linux-gnueabihf/libbluetooth.so.3.12.0
b6ee8000-b6eea000 rw-p 00015000 b3:02 44311      /usr/lib/arm-linux-gnueabihf/libbluetooth.so.3.12.0
b6ef8000-b6efa000 r-xp 00000000 b3:02 25761      /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so
b6efa000-b6f01000 ---p 00002000 b3:02 25761      /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so
b6f01000-b6f02000 rw-p 00001000 b3:02 25761      /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so
b6f02000-b6f04000 r-xp 00000000 b3:02 4246       /lib/arm-linux-gnueabihf/libSegFault.so
b6f04000-b6f0b000 ---p 00002000 b3:02 4246       /lib/arm-linux-gnueabihf/libSegFault.so
b6f0b000-b6f0c000 r--p 00001000 b3:02 4246       /lib/arm-linux-gnueabihf/libSegFault.so
b6f0c000-b6f0d000 rw-p 00002000 b3:02 4246       /lib/arm-linux-gnueabihf/libSegFault.so
b6f0d000-b6f2a000 r-xp 00000000 b3:02 4248       /lib/arm-linux-gnueabihf/ld-2.13.so
b6f2e000-b6f32000 rw-p 00000000 00:00 0 
b6f32000-b6f33000 r--p 0001d000 b3:02 4248       /lib/arm-linux-gnueabihf/ld-2.13.so
b6f33000-b6f34000 rw-p 0001e000 b3:02 4248       /lib/arm-linux-gnueabihf/ld-2.13.so
bebd3000-bebf4000 rw-p 00000000 00:00 0 
bee7f000-bee80000 r-xp 00000000 00:00 0          [sigpage]
ffff0000-ffff1000 r-xp 00000000 00:00 0          [vectors]
Segmentation fault

Removing /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so from /etc/ld.so.preload gives me the same output, except with no lines containing libcofi_rpi.so.

Trying to build a statically linked binary with -tags netgo -a -ldflags "-linkmode external -extldflags -static" gives me an "Illegal instruction" instead of the "Segmentation fault".

Trying to build the binary on the Pi itself works, however, I want to cross-compile this. And cross compilation works indeed, but only on the ODROID-C1/Arch, not on the Raspberry Pi/Raspbian.

The exact build steps I use can be reproduced using Docker:

$ docker run -it golang:1.4.1-cross bash

Inside docker, I run:

<!-- language: bash -->

echo "deb http://emdebian.org/tools/debian/ jessie main" >/etc/apt/sources.list.d/crosstools.list
curl -s http://emdebian.org/tools/debian/emdebian-toolchain-archive.key | apt-key add -
dpkg --add-architecture armhf
apt-get update
apt-get install -y crossbuild-essential-armhf libbluetooth-dev:armhf
export CC=arm-linux-gnueabihf-gcc
export CGO_ENABLED=1
export GOARCH=arm
export GOARM=6
go build -o test test.go

This gives me an executable named "test" that, as noted above, will not run on the Pi.

Here's some info about my test devices:

Raspberry Pi B+:

$ uname -a
Linux raspberrypi 3.18.5+ #744 PREEMPT Fri Jan 30 18:19:07 GMT 2015 armv6l GNU/Linux

$ cat /proc/cpuinfo
processor       : 0
model name      : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS        : 2.00
Features        : half thumb fastmult vfp edsp java tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xb76
CPU revision    : 7
Hardware        : BCM2708
Revision        : 0010
Serial          : 000000002a69d702

$ ldd test
        /usr/lib/arm-linux-gnueabihf/libcofi_rpi.so (0xb6f01000)
        libbluetooth.so.3 => /usr/lib/arm-linux-gnueabihf/libbluetooth.so.3 (0xb6ed4000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0xb6eb5000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0xb6d84000)
        /lib/ld-linux-armhf.so.3 (0xb6f0e000) 

Installed Bluetooth libraries:

$ dpkg -s libbluetooth3

Package: libbluetooth3
Status: install ok installed
Priority: optional
Section: libs
Installed-Size: 172
Maintainer: Debian Bluetooth Maintainers <pkg-bluetooth-maintainers@lists.alioth.debian.org>
Architecture: armhf
Multi-Arch: same
Source: bluez
Version: 4.99-2
Replaces: libsdp2 (<= 1.5-2)
Depends: libc6 (>= 2.13-28)
Pre-Depends: multiarch-support
Conflicts: libsdp2 (<= 1.5-2)
Description: Library to use the BlueZ Linux Bluetooth stack
 BlueZ is the official Linux Bluetooth protocol stack. It is an Open Source
 project distributed under GNU General Public License (GPL).
Homepage: http://www.bluez.org

ODROID-C1:

$ uname -a
Linux alarm 3.10.66-1-ARCH #1 SMP PREEMPT Wed Jan 28 18:12:35 MST 2015 armv7l GNU/Linux

$ cat /proc/cpuinfo
Processor       : ARMv7 Processor rev 1 (v7l)
processor       : 0
BogoMIPS        : 3.27
 
processor       : 1
BogoMIPS        : 3.27
 
processor       : 2
BogoMIPS        : 3.27
 
processor       : 3
BogoMIPS        : 3.27
 
Features        : swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc05
CPU revision    : 1

Hardware        : ODROIDC
Revision        : 000a
Serial          : 1b00000000000000

$ ldd test
       libbluetooth.so.3 => /usr/lib/libbluetooth.so.3 (0xb6eaf000)
       libpthread.so.0 => /usr/lib/libpthread.so.0 (0xb6e87000)
       libc.so.6 => /usr/lib/libc.so.6 (0xb6d48000)
       /lib/ld-linux-armhf.so.3 (0xb6ed1000)

Installed Bluetooth libraries & development headers files:

$ pacman -Qe bluez
bluez 5.27-1

Docker build environment

$ uname -a
Linux 094aa2ed34a5 3.18.5-gentoo-tp #1 SMP Fri Feb 6 20:14:55 CET 2015 x86_64 GNU/Linux

Installed Bluetooth development headers:

$ dpkg -s libbluetooth-dev:armhf
Package: libbluetooth-dev
Status: install ok installed
Priority: extra
Section: libdevel
Installed-Size: 429
Maintainer: Debian Bluetooth Maintainers <pkg-bluetooth-maintainers@lists.alioth.debian.org>
Architecture: armhf
Source: bluez (5.23-2)
Version: 5.23-2+b1
Replaces: libsdp2-dev (<= 1.5.2)
Provides: libbluetooth3-dev
Depends: libbluetooth3 (= 5.23-2+b1), libc6-dev | libc-dev
Suggests: pkg-config
Conflicts: bluez-pan, libbluetooth-dev, libsdp2-dev (<= 1.5.2)
Description: Development files for using the BlueZ Linux Bluetooth library
 BlueZ is the official Linux Bluetooth protocol stack. It is an Open Source
 project distributed under GNU General Public License (GPL).
Homepage: http://www.bluez.org

Installing the exact same version of libbluetooth-dev in the Docker container as the one that is available on the Pi makes no difference.

Copying over the newer libbluetooth.so from Jessie to Wheezie gives me this error:

./test: /lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.15' not found (required by /home/pi/libbluetooth.so.3.17.11)

UPDATE:

Tried with Arch on the Pi, the cross compiled binary also segfaults, giving the same message ("Illegal instruction (core dumped)") with GOARM=5, 6 or 7.

So far I haven't found a way to cross-compile CGO code for the Pi, linked against libbluetooth.

Will try with the Raspberry Pi 2 tomorrow.

So let me answer my own question by concluding, after weeks of trial and error, that…

…it just doesn't work.

As a last resort, I tried taking the very same Raspbian image that I used to compile the above code, and running it with QEMU. And guess what: I get a segfault. The very same instructions, using the very same image, gives me a working executable when I run it on the Pi, and a segfault when I emulate the Pi or try to cross-compile it in any way.

In the end, it is just not worth the hassle. We just set up dedicated Pi that builds our project and the "production" Pis just pull the updated binary.