I need to sum all the values in a column across multiple files

I have a directory with multiple csv text files, each with a single line in the format:

field1,field2,field3,560

I need to output the sum of the fourth field across all files in a directory (can be hundreds or thousands of files). So for an example of:

file1.txt
field1,field2,field3,560

file2.txt
field1,field2,field3,415

file3.txt
field1,field2,field3,672

The output would simply be:
1647

I've been trying a few different things, with the most promising being an awk command that I found here in response to another user's question. It doesn't quite do what I need it to do, and I am an awk newb so I'm unsure how to modify it to work for my purpose:

awk -F"," 'NR==FNR{a[NR]=$4;next}{print $4+a[FNR]:' file1.txt file2.txt

This correctly outputs 975.

However if I try pass it a 3rd file, rather than add field 4 from all 3 files, it adds file1 to file2, then file1 to file3:

awk -F"," 'NR==FNR{a[NR]=$4;next}{print $4+a[FNR]:' file1.txt file2.txt file3.txt
975
1232

Can anyone show me how I can modify this awk statement to accept more than two files or, ideally because there are thousands of files to sum up, an * to output the sum of the fourth field of all files in the directory?

Thank you for your time and assistance.

Here we go (no need to test NR==FNR in a concatenation):

$ cat file{1,2,3}.txt | awk -F, '{count+=$4}END{print count}' 
1647

Or same-same 🇹🇭 (without wasting some pipe(s)):

$ awk -F, '{count+=$4}END{print count}' file{1,2,3}.txt
1647

A couple issues with the current code:

• NR==FNR is used to indicate special processing for the 1st file; in this case there is no processing that is 'special' for just the 1st file (ie, all files are to be processed the same)
• an array (eg, a[NR]) is used to maintain a set of values; in this case you only have one global value to maintain so there is no need for an array

Since you're only looking for one global sum, a bit more simpler code should suffice:

$ awk -F',' '{sum+=$4} END {print sum+0}' file{1..3}.txt
1647

NOTES:

• in the (unlikely?) case all files are empty, sum will be undefined so print sum will display a blank link; sum+0 insures we print 0 if sum remains undefined (ie, all files are empty)
• for a variable number of files file{1..3}.txt can be replaced with whatever pattern will match on the desired set of files, eg, file*.txt, *.txt, etc

$ perl -MList::Util=sum0 -F, -lane'push @a,$F[3];END{print sum0 @a}' file{1..3}.txt
1647

$ perl -F, -lane'push @a,$F[3];END{foreach(@a){ $sum +=$_ };print "$sum"}' file{1..3}.txt
1647

$ cut -d, -f4 file{1..3}.txt | paste -sd+ - | bc
1647

The paste method is only viable for smaller inputs. When row count gets somewhat larger, like the example below of 34,999,999, compared to awk,

• gnu-paste is ~2.83x slower
• bsd-paste is ~3.72x slower

** adding perl for completeness ::: perl 5.36.0 is ~1.54x slower

perl -MList::Util=sum0 -F, -lane'push @a,$_;END{print sum0 @a}'

__='34999999'

( time ( jot "$__" | pvE0 | mawk2 'END { print +_ }{ _+=$__ }' ) )

sleep 1

( time ( jot "$__" | pvE0 | LC_ALL=C paste -sd+ - | bc ) )

sleep 1

( time ( jot "$__" | pvE0 | LC_ALL=C gpaste -sd+ - | bc ) )

|

      in0:  289MiB 0:00:06 [47.9MiB/s] [47.9MiB/s] [<=> ]
     1	612499982500000

( jot "$__" | pvE 0.1 in0 | mawk2 'END { print +_ }{ _+=$__ }'; )

  8.51s user 0.22s system 143% cpu 6.072 total

      in0:  289MiB 0:00:11 [25.5MiB/s] [25.5MiB/s] [ <=> ]
     1	612499982500000

( jot "$__" | pvE 0.1 in0 | LC_ALL=C paste -sd+ - | bc; ) 

27.87s user 1.15s system 128% cpu 22.594 total

      in0:  289MiB 0:00:06 [48.3MiB/s] [48.3MiB/s] [ <=> ]
     1	612499982500000

( jot "$__" | pvE 0.1 in0 | LC_ALL=C gpaste -sd+ - | bc; )

17.14s user 1.10s system 106% cpu 17.185 total