merging two files with columns of that have different lengths and possible comments into CSV like file

file1 looks like

# dsd
# dsd
1,2,5
2,3,5
1,2,5
2,3,5
3,4,5
3,4,5

file2 looks like

# s
1,2
1,2

I want to merge them to get

# dsd
# dsd
1,2,5,1,2
2,3,5,1,2
1,2,5,,
2,3,5,,
3,4,5,,
3,4,5,,

That is I want to keep the comment lines # from the first file after the comment lines, I want to paste columns from the second file, padding them to the column length of the first file. If there are any comment lines in the second file, ignore them.

I started with:

 paste $(grep -v '^#' file1) file2

but I get bash: /usr/bin/paste: Argument list too long

I guess this would be a job for awk but I am only familiar with single file processing and I have only found examples that deal with the same length files. Is there an easy way or one needs to go to longer bash script or python et al.?

You may use this awk solution:

awk -v OFS=, '
NR == FNR {
   if (!/^#/)
      a[++i] = $0
   next
}
{
   if (/^#/)
      print
   else {
      ++NR2
      if (NR2 in a)
         print $0, a[NR2]
      else
         print $0,"",""
   }
}' file2 file1

# dsd
# dsd
1,2,5,1,2
2,3,5,1,2
1,2,5,,
2,3,5,,
3,4,5,,
3,4,5,,

Using any awk:

$ cat tst.awk
BEGIN { FS=OFS="," }
FNR == 1 {
    lineNr = 0
    dflt = a[1]
    gsub("[^"FS"]+","",dflt)
}
/^#/ {
    if ( NR != FNR ) {
        print
    }
    next
}
{ ++lineNr }
NR == FNR {
    a[lineNr] = $0
    next
}
{ print $0, (lineNr in a ? a[lineNr] : dflt) }

<p>

$ awk -f tst.awk file2 file1
# dsd
# dsd
1,2,5,1,2
2,3,5,1,2
1,2,5,,
2,3,5,,
3,4,5,,
3,4,5,,

Using the great Miller, paste, cat and grep, you could run

paste -d ',' <(grep -v '^#' file1.txt) <(grep -v '^#' file2.txt) | mlr --csv -N --ragged cat >output
<file1.txt grep -P '^#' | cat - output > tmp.txt && mv tmp.txt output

to get

# dsd
# dsd
1,2,5,1,2
2,3,5,1,2
1,2,5,,
2,3,5,,
3,4,5,,
3,4,5,,

The steps:

• merge the two input files horizontally, removing the comments lines (via paste and grep);

• add missing commas (via mlr);

• add the comment lines of first file to the merged one (via grep and cat)

Here is a Ruby with the CSV module:

ruby -r csv -e '
f1=CSV.read(ARGV[0])
f2=CSV.read(ARGV[1]).select{|row| !row.join("")[/^\s*#/] }
f2=[""]*f1.slice_when{|a,b| b.to_s[/\d/]}.first.length+f2
f2c=f2.max_by{|row| row.length}.length
puts CSV.generate{|csv| 
    f1.zip(f2).each{|row| 
        if row.flatten.join("")[/^\s*#/] 
            csv<<row[0] 
        elsif row[-1].nil?
            csv<<row[0]+[nil]*f2c
        else 
            csv<<row.flatten
        end
    }
}
' file1 file2 

This is not limited to the assumption that file2 is only 2 columns.

It DOES assume that file1 is the longer of the two files. Easily changed if that is not true.

Prints:

# dsd
# dsd
1,2,5,1,2
2,3,5,1,2
1,2,5,,
2,3,5,,
3,4,5,,
3,4,5,,