Unable to process strings in files with different encoding [Not accepting answers]

I am trying to process the strings present in a particular file, The file is written in English. The problem arises when the encoding of the file differs from "UTF-8". But the file with encoding as "UTF-16-le" does not behave as expected. My main goal is manipulate the strings from the read file. For example the strings.TrimSpace() only works with the UTF-8 file,

I am aware that golang only supports UTF-8 by default, Any alternate approach would be helpful.

Personal Question

>Also I would like to point out, many new programming languages, do process the strings irrespective of the encoding, And why does Go only support UTF-8. If at least there would be an alternative way to pass the encoding format to the reader, that might still help.

What I tried

1. I tried using utf-8 and utf-16 std packages

Code

(main.go)

sample code to show the difference.

package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"strings"
)

func processFile(src string) {
	data, _ := ioutil.ReadFile(src)

	fmt.Println("--- original source ---")
	fmt.Println(string(data))
	fmt.Println(http.DetectContentType(data))

	fmt.Println("\n--- modified source ---")
	for _, val := range strings.Split(string(data), "\n") {
		fmt.Println(strings.TrimSpace(val))
	}
}

func main() {
	processFile("./utf-16-english.txt")
	processFile("./utf-8-english.txt")
}

File-1

(utf-8-english.txt)

Hello
	This is
		Sample
			Document

File-2

(utf-16-english.txt)

Hello
	This is
		Sample
			Document

EDIT

1. Seems that the only way to process strings in a better way is to convert them to UTF-8. Kindly refer the marked answer.

2. As per comments I have written the result from the program to respective files. And the special symbols are not present, but the process with strings, works fine with UTF-8

You have to decode the utf-16 encoded file. The decoding will convert the input to utf-8, after which you can use the string libraries to process the input.

You can use something like this:

import "unicode/utf16"

func processFile(src string, decode func(in[]byte) string) {
    data, _ := ioutil.ReadFile(src)

    fmt.Println("--- original source ---")
    fmt.Println(decode(data))

    fmt.Println("\n--- modified source ---")
    for _, val := range strings.Split(decode(data), "\n") {
        fmt.Println(strings.TrimSpace(val))
    }
}

func main() {
    processFile("./utf-16-english.txt",func(in []byte) string {
        return string(utf16.Decode(in)) })
    processFile("./utf-8-english.txt",func(in []byte) string {
         return string(in)})
}