从循环函数中将JSON导出为单个文件

huangapple go评论82阅读模式
英文:

Exporting JSON into single file from loop function

问题

我写了一些代码,通过调用一个公共API并将JSON输出保存到文件中。但是数据存储在文件中时是逐行存储的,而不是以单个JSON格式存储。

例如:

当前输出:

{"ip":"1.1.1.1", "Country":"US"}
{"ip":"8.8.8.8", "Country":"IN"}

期望的输出:

[
{"ip":"1.1.1.1", "Country":"US"},
{"ip":"8.8.8.8", "Country":"IN"}
]

我知道这应该很简单,我可能漏掉了什么。

我的当前代码是:

从文件中读取IP,并逐个IP调用API。

func readIPfromFile(filename string, outFile string, timeout int) {
	data := jsonIn{}
	//打开输入文件
	jsonFile, err := os.Open(filename) //打开输入文件
    ...

    ...
	jsonData := bufio.NewScanner(jsonFile)

	for jsonData.Scan() {

		//解组JSON数据并检查日志
		if err := json.Unmarshal(jsonData.Bytes(), &data); err != nil {
			log.Fatal(err)
		}
		//保存到文件
		url := fmt.Sprintf("http://ipinfo.io/%s", data.Host)
		GetGeoIP(url, outFile, timeout)

	}
}

使用自定义请求头进行HTTP请求,并调用写入文件的函数。

func GetGeoIP(url string, outFile string, timeout int) {
	geoClient := http.Client{
		Timeout: time.Second * time.Duration(timeout), // 5秒后超时
	}

	req, err := http.NewRequest(http.MethodGet, url, nil)
	if err != nil {
		log.Fatal(err)
	}

	req.Header.Set("accept", "application/json")

	res, getErr := geoClient.Do(req)
	if getErr != nil {
		log.Fatal(getErr)
	}

	if res.Body != nil {
		defer res.Body.Close()
	}

	body, readErr := ioutil.ReadAll(res.Body)
	if readErr != nil {
		log.Fatal(readErr)
	}

	jsonout := jsonOut{}
	jsonErr := json.Unmarshal(body, &jsonout)
	if jsonErr != nil {
		log.Fatal(jsonErr)
	}
	file, _ := json.Marshal(jsonout)
	write2file(outFile, file)
}

写入数据到文件:

func write2file(outFile string, file []byte) {
	f, err := os.OpenFile(outFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
	if err != nil {
		log.Fatal(err)
	}

	defer f.Close()
	if _, err = f.WriteString(string(file)); err != nil {
		log.Fatal(err)
	}
	if _, err = f.WriteString("\n"); err != nil {
		log.Fatal(err)
	}

我知道,我可以将f.WriteString("\n");修改为f.WriteString(",");以添加逗号,但是对我来说在文件中添加[]仍然是一个挑战。

英文:

I wrote some code which hits one public API and saves the JSON output in a file. But the data is storing line by line into the file instead of a single JSON format.

For eg.

Current Output:

{"ip":"1.1.1.1", "Country":"US"}
{"ip":"8.8.8.8", "Country":"IN"}

Desired Output:

[
{"ip":"1.1.1.1", "Country":"US"},
{"ip":"8.8.8.8", "Country":"IN"}
]

I know this should be pretty simple and i am missing out something.

My Current Code is:

To read IP from file and hit the API one by one on each IP.

func readIPfromFile(filename string, outFile string, timeout int) {
	data := jsonIn{}
	//open input file
	jsonFile, err := os.Open(filename) //open input file
    ...

    ...
	jsonData := bufio.NewScanner(jsonFile)

	for jsonData.Scan() {

		// marshal json data & check for logs
		if err := json.Unmarshal(jsonData.Bytes(), &data); err != nil {
			log.Fatal(err)
		}
		//save to file
		url := fmt.Sprintf("http://ipinfo.io/%s", data.Host)
		GetGeoIP(url, outFile, timeout)

	}
}

To make HTTP Request with custom request header and call write to file function.

func GetGeoIP(url string, outFile string, timeout int) {
	geoClient := http.Client{
		Timeout: time.Second * time.Duration(timeout), // Timeout after 5 seconds
	}

	req, err := http.NewRequest(http.MethodGet, url, nil)
	if err != nil {
		log.Fatal(err)
	}

	req.Header.Set("accept", "application/json")

	res, getErr := geoClient.Do(req)
	if getErr != nil {
		log.Fatal(getErr)
	}

	if res.Body != nil {
		defer res.Body.Close()
	}

	body, readErr := ioutil.ReadAll(res.Body)
	if readErr != nil {
		log.Fatal(readErr)
	}

	jsonout := jsonOut{}
	jsonErr := json.Unmarshal(body, &jsonout)
	if jsonErr != nil {
		log.Fatal(jsonErr)
	}
	file, _ := json.Marshal(jsonout)
	write2file(outFile, file)
}

To Write data to file:

func write2file(outFile string, file []byte) {
	f, err := os.OpenFile(outFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
	if err != nil {
		log.Fatal(err)
	}

	defer f.Close()
	if _, err = f.WriteString(string(file)); err != nil {
		log.Fatal(err)
	}
	if _, err = f.WriteString("\n"); err != nil {
		log.Fatal(err)
	}

I know, i can edit f.WriteString("\n"); to f.WriteString(","); to add comma but still adding [] in the file is challenging for me.

答案1

得分: 1

首先,请不要发明一种新的 JSON 编组方式,只需使用 Go 内置的 encoding/json 或其他在 GitHub 上的库。

其次,如果你想创建一个表示对象数组的 JSON 字符串,你需要在 Go 中创建对象数组,并将其编组为字符串(更准确地说,是字节数组的数组)。

我创建了一个简单的示例,但如果可能的话,请自行定制。

https://go.dev/play/p/RR_ok-fUTb_4

英文:

First, please do not invent a new way of json marshaling, just use golang built-in encoding/json or other library on github.

Second, if you want to create a json string that represents an array of object, you need to create the array of objects in golang and marshal it into string (or more precisely, into array of bytes)

I create a simple as below, but please DIY if possible.

https://go.dev/play/p/RR_ok-fUTb_4

huangapple
  • 本文由 发表于 2022年4月24日 15:15:19
  • 转载请务必保留本文链接:https://go.coder-hub.com/71986275.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定