Go Varint Returns Half of Expected Value

Why is the output of this code:

package main

import (
    "fmt"
    "encoding/binary"
)

func main() {
    var myByte byte = 18
    array := []byte{myByte}
    val, n := binary.Varint(array)
    fmt.Printf("value: %d, num bytes: %d\n", val, n)
}

value: 9, num bytes: 1 instead of value: 18, num bytes: 1

It probably has something to do with two's complement, but I'm not seeing how.

TLDR: Use the Uvarint method to properly decode an unsigned byte .. which is what a byte is by default.

The byte is stored unsigned (as a byte is unsigned by default - it is an alias for uint8 .. in most languages).

When you're decoding the number, you're calling binary.Varint .. which decodes a signed number. This results in the number being incorrect because of the sign bit.

Using binary.Uvarint .. that is, decode an unsigned number, you get the correct result:

val, n := binary.Uvarint(array) // val = 18, n = 1

Expanded example:

Lets take a look at your number - 18. In binary, it is this:

00010010

The binary.Varint function is below:

func Varint(buf []byte) (int64, int) {
	ux, n := Uvarint(buf) // ok to continue in presence of error
	x := int64(ux >> 1)
	if ux&1 != 0 {
		x = ^x
    }
    return x, n
}

Basically, it will first go and get the unsigned value of what you've provided: 18.

It will then shift all of the bytes across by 1. This results in:

00001001

That is the binary representation of 9. Notice the sign bit is still 0 - which means a positive number. It then checks whether to invert the result by bitwise AND'ing the original value (18) with 1. It does this because, it is operating with a "I know this number is signed" context - that's why the function exists:

00010010
00000001
--------
00000000
     = 0

At that point, zero does equal zero - so the method returns x - which is 9.

Lets try with 1

Using 1 as the input:

00000001

Shift it right:

00000000

AND the original number (1) with 1:

00000001
00000001
--------
     = 1

At this point, the result doesn't equal zero.. so the result is inverted:

11111111

This is the signed representation of -1 (notice the sign bit is now 1 .. indicating a negative number).

Yes. Check this modified version: http://play.golang.org/p/AyP2a4gue8

package main

import (
	"encoding/binary"
	"fmt"
)

func main() {
	for i := 0; i < 100; i++ {
		var myByte byte = byte(i)
		array := []byte{myByte}
		val, n := binary.Varint(array)
		fmt.Printf("int %d value: %d, num bytes: %d\n", i, val, n)

	}    
}

which produces the following output:

int 0 value: 0, num bytes: 1
int 1 value: -1, num bytes: 1
int 2 value: 1, num bytes: 1
int 3 value: -2, num bytes: 1
int 4 value: 2, num bytes: 1
int 5 value: -3, num bytes: 1
int 6 value: 3, num bytes: 1
int 7 value: -4, num bytes: 1
int 8 value: 4, num bytes: 1
int 9 value: -5, num bytes: 1
int 10 value: 5, num bytes: 1
int 11 value: -6, num bytes: 1
int 12 value: 6, num bytes: 1
int 13 value: -7, num bytes: 1
int 14 value: 7, num bytes: 1
int 15 value: -8, num bytes: 1
int 16 value: 8, num bytes: 1
int 17 value: -9, num bytes: 1
int 18 value: 9, num bytes: 1

You can see "zig-zagging" between the negatives and positives. This is because, according to the documented binary format, varints use "zig-zag" encoding so that values of small absolute value are encoded with small values.