UDP packets in wrong order with simple Kubernetes setup

I am experiencing UDP packets coming out of my pods in a wrong order.

When using tshark inside a pod's container (through the kubernetes dashboard and using "exec") on the container eth0 interface :

• I see that some UDP packets are sent in bad order (example: 1-2-10-11-12-3-4-5-6-7...).
• This happens sporadically.
• This is problematic since these packets are carrying order dependent information.

For instance, when using an ffmpeg command inside one of my pod to send a video file via UDP to a service I am experiencing the problem. Using another program instead of ffmpeg I am still experiencing the same issue. It doesn't seem to come from the application itself.

The same applications (ffmpeg, custom application, ...) are working fine outside of the cluster with docker or when running the binaries. No UDP packets are sent in the wrong order here.

When I try to send the UDP packets from one container to another in the same pod, I am still experiencing the same issue.

I tried on 4 different kind of machines/setups with similar results:

• 2 x Intel® NUC Kit NUC8i7BEH (Intel Core i7-8559U @ 2.70GHz, 32GB ram):
  • deloyed with terraform
  • Rke cluster (container based)
  • container runtime: dockerd
• 2 x Intel® NUC Kit NUC8i7BEH (Intel Core i7-8559U @ 2.70GHz, 32GB ram):
  • deployed with kubadm
  • network plugin (cni): calico
  • container runtime: crio
• 3 x Old supermicro blade servers (Pentium 4 3Ghz, 2GB ram each):
  • deployed with kubadm
  • network plugin (cni): calico
  • container runtime: containerd
• 3 x VMs (12 cores each of a Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz):
  • deployed with kubadm
  • network plugin (cni): calico
  • container runtime: crio

All setups are using wired Ethernet connection.

What should I do to guarantee that my UDP packets will always be sent in the right order when coming out of my containers when in Kubernetes? Is there a networking configuration that needs to be made to the cluster ?

Edit:
Here is a simple kubernetes config that reproduces the issue:

We have two pods using ffmpeg, one sending the signal and the other one receiving.

Here is the sender's config:

apiVersion: v1
kind: Pod
metadata:
  name: ffmpeg-stream
  labels:
    app: ffmpeg-stream
spec:
  containers:
    - name: ffmpeg-record
      securityContext:
        privileged: true
      image: linuxserver/ffmpeg
      command: ["ffmpeg"]
      args: ["-re", "-f", "lavfi", "-i", "nullsrc=s=1280x720", "-filter_complex", "geq=random(1)*255:128:128;aevalsrc=-2+random(0)", "-b:v", "20M", "-c:v", "libx264", "-preset", "ultrafast", "-f", "mpegts", "udp://ffmpeg-record:26500?pkt_size=1316"]
      resources:
        limits:
          memory: "2Gi"
          cpu: 6
      volumeMounts:
        - mountPath: /data/discontinuity
          name: data
          subPath: discontinuity
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: nfs

Here is the receiver's config:

apiVersion: v1
kind: Pod
metadata:
  name: ffmpeg-record
  labels:
    app: ffmpeg-record
spec:
  containers:
    - name: ffmpeg-record
      securityContext:
        privileged: true
      image: linuxserver/ffmpeg
      command: ["ffmpeg"]
      args: ["-i", "udp://localhost:1234", "-c:v", "copy", "-c:a", "copy", "-sn", "test.ts"]
      resources:
        limits:
          memory: "1Gi"
          cpu: 3
      volumeMounts:
        - mountPath: /data/discontinuity
          name: data
          subPath: discontinuity
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: nfs

---

apiVersion: v1
kind: Service
metadata:
  name: ffmpeg-record
spec:
  selector:
    app: ffmpeg-record
  type: ClusterIP
  ports:
  - port: 26500
    targetPort: 1234
    protocol: UDP

> What should I do to guarantee that my UDP packets will always be sent in the right order

Strictly speaking, you cannot. What I think you are saying is you really want them to arrive in the same order. (Sending in the wrong order is probably a great way to have them arrive out of order...)

Packets get lost, arrive out of order. TCP has many layers of features to handle these problems and deliver a data stream to the recipient that is properly ordered.

UDP will simply hand the arriving data to the receiver. Based on what you are seeing on the sending site, the network stack is faithful the liberalities of UDP even when sending.

You didn't provide any code, so I don't think you can get a solution from here.

But historically, the solution would be to send over UDP only one item, then wait for a response that confirms, and then send the second... if memory serves me correctly, this was what TFTP (not FTP) did. The tradeoffs are pretty obvious, and many of the replacement protocols used a TCP-based solution.