Cross-compilation Go and Goc "symbol not found" error message on ARM

I have nowhere to ask, everything is so specific, search and googling doesn't work. Spent tens of hours on this.

I'm trying to build a Go program for my LinkSys MR8300 V1.1 router, which has armv7l processor (uname -a output), which is ARM v7, 32-bit CPU. I have OpenWrt operating system installed on the router.

The idea of the app is to connect USB coin acceptor, and enable internet for 30 minutes when you drop a coin. I have working app on the Ubuntu desktop which understands the device, and on desktop everything's fine.

However, the router is 32 bit, and ARM, so things are little bit different and more complicated.

cat /proc/cpuinfo on my router gives something like this:

processor       : 0
model name      : ARMv7 Processor rev 5 (v7l)
BogoMIPS        : 26.81
Features        : half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm
CPU implementer : 0x41
CPU architecture: 7
CPU variant     : 0x0
CPU part        : 0xc07
CPU revision    : 5

I'm using cross-compilation, since the router has only 19MB of disk space (and not all of the development packages are available).

For cross-compilation I'm using x86_64 Ubuntu desktop, and toolchain provided by OpenWrt. I'm already past the point when you need to configure and make the toolchain. I was able to compile and execute the basic C programs on my router, like:

#include <stdio.h>

int main()
{
  printf("Hello World");
  return 0;
}

And even Go programs like:

package main

func main() {
        println("Coin acceptor device control program")
}

When I upload binaries to the router they work, there is no any problem with that.

Problems start when I try to use USB library inside the router. Since it's cross-compilation, I should mention environment variables I have:

STAGING_DIR=/home/ro/work/openwrt/staging_dir
TOOLCHAIN_DIR=/home/ro/work/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.3.0_musl_eabi
LDCFLAGS=/home/ro/work/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.3.0_musl_eabi/usr/lib
LD_LIBRARY_PATH=/home/ro/work/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.3.0_musl_eabi/usr/lib
LDFLAGS=-L/home/ro/work/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.3.0_musl_eabi/usr/lib
CGO_LDFLAGS=-L/home/ro/work/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.3.0_musl_eabi/usr/lib
CFLAGS=-I/home/ro/work/openwrt/staging_dir/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.3.0_musl_eabi/include
CC=arm-openwrt-linux-gcc
GOARCH=arm
CROSS_COMPILE=arm-openwrt-linux-gcc
CGO_ENABLED=1
GOOS=linux
GOARM=7

With these environment variables go get github.com/karalabe/hid command works as expected (there is hid.a binary in my /home/ro/go/pkg/linux_arm/github.com/karalabe folder).

This url https://github.com/karalabe/hid is the library I use for USB, the title says "Gopher Interface Devices (USB HID)". However, it has dependencies on some C libraries... So here is where GOC comes into play I guess. The library says:

> Both of these dependencies are vendored directly into the repository and wrapped using CGO, making the hid package self-contained and go-gettable

As I mentioned before, I can compile simple apps on my Desktop and upload them to the router, and these apps work! However, when I try to use the library, things go little bit off the script.

Here is the app I have:

main.go

package main

import (
        "fmt"
)

func main() {
        println("Hello")

        devices := FindAll(0x0079)

        if len(devices) == 0 {
                fmt.Println("No coin acceptor found")
                return
        }

        println("Found coin acceptor")
}

device.go

package main

import (
	"errors"
	"github.com/karalabe/hid"
)

type Device struct {
	info hid.DeviceInfo
	dev  *hid.Device

	Name string
	PID  uint16
}

func (device *Device) Open() error {
	if device.dev != nil {
		return errors.New("device: Device handle is not null, but Open() called")
	}

	var err error
	device.dev, err = device.info.Open()

	return err
}

func (device *Device) Close() error {
	if device.dev == nil {
		return errors.New("device: Close() called for Device with null handle")
	}

	err := device.dev.Close()
	device.dev = nil

	return err
}

// Find all products with the given product ID.
func FindAll(product uint16) []Device {
	deviceInfo := hid.Enumerate(0x1e7d, product)

	if len(deviceInfo) == 0 {
		return []Device{}
	}

	devices := make([]Device, len(deviceInfo))

	for i, info := range deviceInfo {
		devices[i] = Device{
			info: info,
			Name: info.Product,
			PID:  product,
		}
	}

	return devices
}

Command I use to build a thing:

go build ./main.go ./device.go

The binary gets produces, and things look normal. Until the point when I upload and run the binary on the router. It pretty much says:

Error relocating ./main: __pthread_cond_timedwait_time64: symbol not found
Error relocating ./main: __nanosleep_time64: symbol not found
Error relocating ./main: __stat_time64: symbol not found
Error relocating ./main: __clock_gettime64: symbol not found

And that's it. It doesn't even print "Hello".

One thing to notice is that it has references to (and complains about) 64-bit functions for some reason. So, to sum this up:

• This is 32-bit ARM CPU.
• I can run simple C and Go programs, so I assume that my C and Go compiles provide correct binaries for the 32-bit CPU.
• When I try to attach the library none of the compilers (I assume the app gets compiled using Go and Goc) complains about using 64-bit functions.
• The binary itself complains about 64-bit functions.

I'm wondering what am I missing? What flags should I specify to the compiler(s) so there is no 64-bit functions usage? Maybe there is something inside these USB libraries written in C? I kinda ran out of any ideas and pretty much stuck at this point, since I cannot use my USB device with embedded OpenWrt device.

objdump -p main output:

objdump -p main

main:     file format elf32-little

Program Header:
    PHDR off    0x00000034 vaddr 0x00010034 paddr 0x00010034 align 2**2
         filesz 0x00000120 memsz 0x00000120 flags r--
  INTERP off    0x00000154 vaddr 0x00010154 paddr 0x00010154 align 2**0
         filesz 0x00000018 memsz 0x00000018 flags r--
    LOAD off    0x00000000 vaddr 0x00010000 paddr 0x00010000 align 2**16
         filesz 0x001164ec memsz 0x001164ec flags r-x
    LOAD off    0x00116d6c vaddr 0x00136d6c paddr 0x00136d6c align 2**16
         filesz 0x00013cc4 memsz 0x00029c94 flags rw-
 DYNAMIC off    0x00116f08 vaddr 0x00136f08 paddr 0x00136f08 align 2**2
         filesz 0x000000f8 memsz 0x000000f8 flags rw-
    NOTE off    0x0000016c vaddr 0x0001016c paddr 0x0001016c align 2**2
         filesz 0x00000064 memsz 0x00000064 flags r--
     TLS off    0x00116d6c vaddr 0x00136d6c paddr 0x00136d6c align 2**2
         filesz 0x00000000 memsz 0x00000008 flags r--
   STACK off    0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**4
         filesz 0x00000000 memsz 0x00000000 flags rw-
   RELRO off    0x00116d6c vaddr 0x00136d6c paddr 0x00136d6c align 2**0
         filesz 0x00000294 memsz 0x00000294 flags r--

Dynamic Section:
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so
  INIT                 0x0001437c
  FINI                 0x000a7abc
  INIT_ARRAY           0x00136d6c
  INIT_ARRAYSZ         0x00000004
  FINI_ARRAY           0x00136d70
  FINI_ARRAYSZ         0x00000004
  HASH                 0x000101d0
  GNU_HASH             0x00010acc
  STRTAB               0x00012820
  SYMTAB               0x000114c0
  STRSZ                0x00001568
  SYMENT               0x00000010
  DEBUG                0x00000000
  PLTGOT               0x00137000
  PLTRELSZ             0x00000360
  PLTREL               0x00000011
  JMPREL               0x0001401c
  REL                  0x00014014
  RELSZ                0x00000008
  RELENT               0x00000008
  VERNEED              0x00013ff4
  VERNEEDNUM           0x00000001
  VERSYM               0x00013d88

Version References:
  required from libgcc_s.so.1:
    0x0b792655 0x00 02 GCC_3.5

file main output:

main: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-armhf.so.1, Go BuildID=VEoRmm2_70EkDLdXjTm0/APdquCT6lGlorzOlV3Un/jnQVIeEEaRXWGrmNQ-mE/Bjx-ldvZODg_GFcJuIgW, not stripped

Finally!

I tried to re-build OpenWrt toolchain, but this time I switched to a branch similar to my router openwrt-21.02, did make menuconfig and make again. It didn't help by itself, but I slightly changed my env variables this way:

export STAGING_DIR=/home/ro/work/openwrt/staging_dir
export TOOLCHAIN_DIR=$STAGING_DIR/toolchain-arm_cortex-a7+neon-vfpv4_gcc-11.3.0_musl_eabi
export LDCFLAGS=$TOOLCHAIN_DIR/lib
export LD_LIBRARY_PATH=$TOOLCHAIN_DIR/lib
export LDFLAGS="-L${TOOLCHAIN_DIR}/lib"
export CGO_LDFLAGS="-Xlinker -rpath=${TOOLCHAIN_DIR}/lib/libc.so -static"
export CFLAGS="-I${TOOLCHAIN_DIR}/include"
export CC=arm-openwrt-linux-gcc-11.3.0
export PATH=$TOOLCHAIN_DIR/bin:$PATH
export GOARCH=arm
export CROSS_COMPILE=arm-openwrt-linux-gcc
export CGO_ENABLED=1
export GOOS=linux
export GOARM=7

The CGO_LDFLAGS specifies static linking to libc. Finally it all works, and I can access USB device from the embedded device!