pbuilder fails with "exec format error" when targeting arm64

I want to customize a debian binary package for ruby to set config flags enabling certain features. I want to build this binary package inside of the free github actions runners which are only amd64 and I want to also support the arm64 architecture, which means cross compiling.

I am using Ubuntu 22.04 (to get the latest tools) as my build environment, targeting focal, 20.04, mostly to keep access to openssl 1.1.

I am spinning up a reproducible packaging environment with Kubernetes and this is my pod definition (note I have enabled privileged mode if anything wanted to try and use qemu):

apiVersion: v1
kind: Pod
metadata:
  name: ruby-ubuntu-build
  namespace: default
spec:
  containers:
  - args:
    - sleep
    - infinity
    image: ubuntu:22.04
    imagePullPolicy: Always
    name: ruby-debian-build
    resources:
        requests:
            memory: 2048Mi
        limits:
            memory: 2048Mi
    securityContext:
      privileged: true
  preemptionPolicy: PreemptLowerPriority
  priority: 1000
  restartPolicy: Always

When I get in to that environment I install all my tools:

apt-get update
apt-get install -y git-buildpackage vim git build-essential ruby devscripts qemubuilder qemu-user-static

To recap my environment:

# uname -a
Linux ruby-ubuntu-build 5.4.149-73.259.amzn2.x86_64 #1 SMP Mon Sep 27 12:48:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"

And my failing pbuilder chroot environment creation:

# pbuilder create --distribution focal --architecture arm64 --mirror http://ports.ubuntu.com/ubuntu-ports/        
W: /root/.pbuilderrc does not exist
W: cgroups are not available on the host, not using them.
I: Distribution is focal.
I: Current time: Sat Feb 18 02:56:31 GMT 2023
I: pbuilder-time-stamp: 1676688991
I: Building the build environment
I: running debootstrap
/usr/sbin/debootstrap
I: Retrieving InRelease 
I: Checking Release signature
I: Valid Release signature (key id F6ECB3762474EDA9D21B7022871920D1991BC93C)
I: Retrieving Packages 
I: Validating Packages 
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://ports.ubuntu.com/ubuntu-ports...
I: Validating adduser 3.118ubuntu2
/// SNIP for brevity
W: Failure trying to run: chroot "/var/cache/pbuilder/build/29550" /bin/true
W: See /var/cache/pbuilder/build/29550/debootstrap/debootstrap.log for details
E: debootstrap failed
E: Tail of debootstrap.log:
2023-02-18 02:56:31 URL:http://ports.ubuntu.com/ubuntu-ports/dists/focal/InRelease [264892/264892] -> "/var/cache/pbuilder/build/29550/var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_focal_InRelease" [1]
gpgv: Signature made Thu Apr 23 17:34:17 2020 GMT
gpgv:                using RSA key 3B4FE6ACC0B21F32
gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>"
gpgv: Signature made Thu Apr 23 17:34:17 2020 GMT
gpgv:                using RSA key 871920D1991BC93C
gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>"
2023-02-18 02:56:32 URL:http://ports.ubuntu.com/ubuntu-ports/dists/focal/main/binary-arm64/by-hash/SHA256/e014b241021d8c0cf16cb94f44f4cafb7746507769baf36b77845d3ae22e2c1b [940188/940188] -> "/var/cache/pbuilder/build/29550/var/lib/apt/lists/partial/ports.ubuntu.com_ubuntu-ports_dists_focal_main_binary-arm64_Packages.xz" [1]
chroot: failed to run command '/bin/true': Exec format error
E: End of debootstrap.log
W: Aborting with an error

why is the chroot environment trying to run /bin/true during filesystem creation? I would understand more if this was an issue at build time.

Looks like I had issues with binfmt not automatically invoking qemu. First I learned about binfmts, checking what was setup and then enabling the right architecture.

# update-binfmts --display | grep "aarch64 (" -A 9
qemu-aarch64 (disabled):
     package = qemu-user-static
        type = magic
      offset = 0
       magic = \x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00
        mask = \xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff
 interpreter = /usr/libexec/qemu-binfmt/aarch64-binfmt-P
    detector = 

and then enabled it:

update-binfmts --enable qemu-aarch64

I create the debian filesystem:

# debootstrap --arch=arm64 --foreign --variant=buildd --verbose --include=fakeroot,build-essential --components=main --resolve-deps --no-merged-usr focal /tmp/focal-arm-sbuild http://ports.ubuntu.com/ubuntu-ports

I can now chroot:

# this is the regular shell, amd64 arch
root@ruby-ubuntu-build:/# uname -a
Linux ruby-ubuntu-build 5.4.149-73.259.amzn2.x86_64 #1 SMP Mon Sep 27 12:48:12 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@ruby-ubuntu-build:/# chroot /tmp/focal-arm-sbuild
# uname -a
Linux ruby-ubuntu-build 5.4.149-73.259.amzn2.x86_64 #1 SMP Mon Sep 27 12:48:12 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux