How to parse an input in Bash

I am new to Bash and have an input file.log with the following contents:

ok: [test_performance=10.50.100.82] => {
    "msg": [
        [
            "DeviceName:78BAY08V2A/10",
            "DeviceName:GH7AK1A02097/10"
]]}
ok: [test_1=10.50.101.84] => {
    "msg": [
        [
            "DeviceName:8K251FDD4000D1/13",
            "DeviceName:99071JEGR10369/12"
]]}

I want to get output as below:

test_performance,10.50.100.82,78BAY08V2A/10
test_performance,10.50.100.82,GH7AK1A02097/10
test_1,10.50.101.84,8K251FDD4000D1/13
test_1,10.50.101.84,99071JEGR10369/12

I tried below command but it is not working. What can I try next?

grep -C 3 "DeviceName:" file.log | xargs | sed 's/ok/\n/g' | sed 's|[]{}[]||g' | sed 's/msg//g' | sed 's/ =>/,/g' | awk -F ":" '{print $4,$5}' | awk -F "--" '{print $1}' | sed 's/,    /,/g' | sed 's/, /-/g' | sed 's/ //g' | grep -E 'arda|kobra|pipeline|performance|integration|feature|kare|yjr' | sed 's/=/,/g' | sed 's/\(.*\)-/,/' | sed 's/-/,/g' | tee -a file2.csv

With GNU awk use ", :, =, ] and [ as field separators:

awk -F '[":=\\]\\[]' '/^ok/{ok=$3 "," $4} /DeviceName/{print ok "," $3}' file.log

Output:
<pre>
test_performance,10.50.100.82,78BAY08V2A/10
test_performance,10.50.100.82,GH7AK1A02097/10
test_1,10.50.101.84,8K251FDD4000D1/13
test_1,10.50.101.84,99071JEGR10369/12
</pre>

With your shown samples please try following awk code, written and tested in GNU awk. This will pick up lines from ok: to till } and process them as per requirement.

awk -v RS='(^|\n)ok: \\[[^}]*' '
RT{
  match(RT,/ok: \[([^=]*)=([^]]*)\]/,arr)
  val=arr[1]","arr[2]
  while(match(RT,/"DeviceName:([^"]*)"/,arr)){
    print val","arr[1]
    RT=substr(RT,RSTART+RLENGTH)
  }
}
' Input_file

The input looks like a list of JSON objects prefixed with labels like ok: [test_performance=10.50.100.82] =>. So the task of parsing can be divided into two parts:

1. Parsing the prefixes.
2. Parsing the JSON objects.

The first part doesn't look like a well-known format, so I'd parse it using awk, perl, or sed. For the JSON part, there is a popular jq tool.

Here is one of the possible solutions using sed and jq:

sed -e 's/\(^ok: .* => \){/"":{/;
    1{s/^/{/};
    ${s/$/}/};
    2~1{s/"ok:/,"ok:/}
' file.log | jq --raw-output -e '.|to_entries|.[]|. as $p|.[]
    |$p.value.msg|.[]|.[]
    |sub("^DeviceName:";"") as $d
    |$p.key|capture("ok: \\[(?<test>.*)=(?<ip>.*)\\]")
    |.test+","+.ip+","+$d'

Where the sed command makes a JSON object out of the whole input. It

• Adds curly braces before the first line(1{s/^/{/}) and after the last line(${s/$/}/}).
• Surrounds the "ok:..." prefixes with double quotes.

Then the resulting JSON object is piped to the jq command. For the rest, I suggest the reader refer to the documentation:

• man sed
• man jq.

Choosing the right tool

While it is technically possible to parse JSON with awk, sed, or even bash, we should avoid doing so because these are general-purpose tools. Implementing a correct parser requires a program that adheres to the entire JSON specification.

Similarly, generating CSV data actually requires software specifically written for this purpose. Therefore, personally, I wouldn't use the command above in a production environment. A proper solution involves using a programming language that is more powerful than bash, awk, or sed. Using a Perl, PHP, or Node.js script would be suitable as long as the appropriate libraries are utilized.

Here is a sed script process script somewhat similar to the accepted awk answer. Note env -S is a GNU coreutils extension of env.

If the line starts with ok: copy the line to the hold buffer (h).

If the line contains DeviceName: append the hold buffer to the pattern buffer (G). Extract and format and order the 3 fields of the now two lines in the pattern buffer and print (s/.../p) the result.

#!/usr/bin/env -S sed -E -n -f

/^ok:/ h
/DeviceName:/ {
	G
    s/.*:([^"]*).*\n.*\[([^=]*)=([^]]*).*/,,/p
}

and run it like this:

$ ./process file.log
test_performance,10.50.100.82,78BAY08V2A/10
test_performance,10.50.100.82,GH7AK1A02097/10
test_1,10.50.101.84,8K251FDD4000D1/13
test_1,10.50.101.84,99071JEGR10369/12

Here is a bash script process that uses the magic of case pattern matching:

#!/bin/bash

while read -r line
do
	case "$line" in
		ok:*)
			line=${line#*[}
			line=${line%]*}
			field12=${line/=/,}
			;;
		*DeviceName:*)
			line=${line#*:}
			field3=${line%\"*}
			echo "$field12,$field3"
			;;
	esac
done

and you would run it like this:

$ ./process < file.log
test_performance,10.50.100.82,78BAY08V2A/10
test_performance,10.50.100.82,GH7AK1A02097/10
test_1,10.50.101.84,8K251FDD4000D1/13
test_1,10.50.101.84,99071JEGR10369/12