How to append consecutively to a JSON file in Go?

I wonder how can I write consecutively to the same file in Go? Do I have to use os.WriteAt()?

The JSON is basically just an array filled with structs:

[
    {
        "Id": "2817293",
        "Data": "XXXXXXX"
    },
    {
        "Id": "2817438",
        "Data": "XXXXXXX"
    }
    ...
]

I want right data to it consecutively i.e. append to that JSON array more than once before closing the file.
The data I want to write to the file is a slice of said structs:

dataToWrite := []struct{
    Id   string
    Data string
}{}

What is the proper way to write consecutively to a JSON array in Go?

My current approach creates multiple slices in the JSON file and thus is not what I want. The write process (lying in a for loop) looks like this:

...
			// Read current state of file
			data := []byte{}
			f.Read(data)

			// Write current state to slice
			curr := []Result{}
			json.Unmarshal(data, &curr)

			// Append data to the created slice
			curr = append(curr, *initArr...)
			JSON, _ := JSONMarshal(curr)

			// Empty data container
			initArr = &[]Result{}

			// Write
			_, err := f.Write(JSON)
			if err != nil {
				log.Fatal(err)
			}
...

Write the opening [ to the file. Create an encoder on the file. Loop over slices and the elements of each slice. Write a comma if it's not the first slice element. Encode each slice element with the encoder. Write the closing ].

_, err := f.WriteString("[")
if err != nil {
	log.Fatal(err)
}

e := json.NewEncoder(f)
first := true
for i := 0; i < 10; i++ {

	// Create dummy slice data for this iteration.

	dataToWrite := []struct {
		Id   string
		Data string
	}{
		{fmt.Sprintf("id%d.1", i), fmt.Sprintf("data%d.1", i)},
		{fmt.Sprintf("id%d.2", i), fmt.Sprintf("data%d.2", i)},
	}

	// Encode each slice element to the file
	for _, v := range dataToWrite {

		// Write comma separator if not the first.
		if !first {
			_, err := f.WriteString(",\n")
			if err != nil {
				log.Fatal(err)
			}
		}
		first = false

		err := e.Encode(v)
		if err != nil {
			log.Fatal(err)
		}
	}
}
_, err = f.WriteString("]")
if err != nil {
	log.Fatal(err)
}

https://go.dev/play/p/Z-T1nxRIaqL

If it's reasonable to hold all of the slice elements in memory, then simplify the code by encoding all of the data in a single batch:

type Item struct {
	Id   string
	Data string
}

// Collect all items to write in this slice.
var result []Item

for i := 0; i < 10; i++ {
	// Generate slice for this iteration.
	dataToWrite := []Item{
		{fmt.Sprintf("id%d.1", i), fmt.Sprintf("data%d.1", i)},
		{fmt.Sprintf("id%d.2", i), fmt.Sprintf("data%d.2", i)},
	}

	// Append slice generated in this iteration to the result.
	result = append(result, dataToWrite...)
}

// Write the result to the file.
err := json.NewEncoder(f).Encode(result)
if err != nil {
	log.Fatal(err)
}

https://go.dev/play/p/01xmVZg7ePc

If you don't care about the existing file you can just use Encoder.Encode on the whole slice as @redblue mentioned.

If you have an existing file you want to append to, the simplest way is to do what you've shown in your edit: Unmarshal or Decoder.Decoder the whole file into a slice of structs, append the new struct to the slice, and re-decode the whole lot using Marshal or Encoder.Encode.

If you have a large amount of data, you may want to consider using JSON Lines to avoid the trailing , and ] issue, and write one JSON object per line. Or you could use regular JSON, seek back from the end of the file so you're writing over the final ], then write a ,, the new JSON-encoded struct, and finally a ] to make the file a valid JSON array again.

So it depends on a bit on your use case and the data size which approach you take.

NOTICE
This Answer is a solution or workaround if you care about the content of an existing file!
This means it allows you to append to an existing json file, created by your API.

Obviously this only works for arrays of same structs

actual working json format:

[
object,
...
object,
]

When writing to a file, do NOT write [ and ].
Just append to the file writing your serialized json object and append a ,

actual file content:

object,
...
object,

Finally when reading the file prepend [ and append ]

This way you can write to the file from multiple sources and still have valid JSON

Also load the file and have a valid input for your json-processor.

We write our logfiles like this and provide a vaild json via REST calls, which is then processed (for example by a JavaScript Grid)