jq loop over objects and use shell command output to add a field

I'm trying to iterate over a JSON and add a field to each object in the JSON.
I'm doing this in a Gitlab pipeline stage but I don't think is important.

The JSON is this:

{
 "service10": {
     "path_to_file" : "service10.txt"
     },
"service192": {
     "path_to_file" : "service192.txt"
     }
}
     

What I'm doing is using the path_to_file to check in a Gitlab repo if that file exists and I need to save the http response, the code now looks as follows:

check_files_existence:
  stage: update-external-repo
  script:
    - apt-get update --fix-missing
    - apt-get install jq -y
    - HTTP_RESPONSE_CODES=()

    # Check for file
    - |-
      for FILE_TO_CHECK in $(jq -r '.[].path_to_file ' $JSON_TO_READ); do
        
        last_status_code=($(curl [omitted]))
        
        HTTP_RESPONSE_CODES+=($last_status_code)      
        
       done

   # Write array of responses into artifact
    - echo "${HTTP_RESPONSE_CODES[@]}" >> ./http_response_code.txt

  artifacts:
    paths:
      - ./http_response_code.txt
    expire_in: 10 min

This generates a .txt file with the http responses separated by a whitespace.

What I want to do is add a field to each service while I loop over them with the http_response, so the JSON will be like:

{
 "service10": {
     "path_to_file" : "service10.txt",
     "http_response": 200
     },
"service192": {
     "path_to_file" : "service192.txt",
     "http_response": 404
     }
}

And then I'm going to export this an artifact to use it in another stage in my Gitlab Pipeline.

Using the above for loop I can get the file path but then it looks like I cannot update the JSON, if I use something like

for OBJECT in $(jq -r '.[] ' $JSON_TO_READ)

I cannot read the file path because using

jq '.path_to_file $OBJECT' 

returns an error.

Is there a way to do this?

Thanks.

This probably calls for two separate copies of jq. Assuming your input file were changed to be valid JSON:

jq -r 'to_entries[] | [.key, .value.path_to_file] | @tsv' <"$JSON_TO_READ" \
| while IFS=$'\t' read -r key value; do
    # comment this next line in your real system
    (( ++http_status )) # stand-in so this can be tested without calling curl
    # uncomment this next line and replace the url in your real system
    #http_status=$(curl -s -o /dev/null -w "%{http_code}" "http://example.com/$key") || continue
    printf '%s\t%s\n' "$key" "$http_status"
  done \
| jq -Rn --slurpfile input_file "$JSON_TO_READ" '
    reduce inputs as $inline
      ($input_file[0];
       (($inline | split("\t")) as $pieces |
        .[$pieces[0]].http_response = ($pieces[1])))'

Breaking down the shell pipeline:

• First, we have a copy of jq that emits a tab-separated stream with your key and value. An entry here would look like:

  service10<TAB>service10.txt
  

• Next, we have a shell pipeline (running a BashFAQ #1 loop) that reads these two variables, and runs curl (or in the example above, just adds an incrementing number as the status code) to determine the HTTP status code:

  service10<TAB>200
  

• Finally, we have another copy of jq that accepts your input file via --slurpfile, and reads the stream of tab-separated service keys and HTTP status codes from stdin, using a reducer to incrementally update the status file as new values come in. (Because --slurpfile contains a list of JSON documents found in the file, not just a single document, we're referring to $input_file[0] to refer to the first -- and only -- document in the file).

If your file isn't too big and you don't mind reading the same input file multiple times, then here is another solution that should work:

1. Count the number of key-value-pairs in the object
2. Extract single entry by index from object
3. Get path_to_file value from entry
4. Call curl
5. Add http_response to entry
6. Use jq -s (--slurp) to reconstruct the object from its entries

input='input.json'
length="$(jq 'length' "$input")"
for i in $(seq "$length"); do
  entry="$(jq --argjson idx "$i" 'to_entries[$idx-1]' "$input")"
  path="$(printf '%s' "$entry" | jq -r '.value.path_to_file')"
  last_status_code="$(curl ... "$path")"
  printf '%s' "$entry" | jq --argjson status "$last_status_code" '.value.http_response = $status'
done | jq -s 'from_entries'