Loop to merge csv files

Question

[Gonzalo]

Hello everyone,

I am trying to merge csv files. I have 87 pairs of files of 87 different samples, some are .maf and others .csv (the structure is the same, the maf file is also a csv) and I have to merge them. I tried to do it with the paste command and it is perfect, but it gives me a lot of work to paste them one by one. I tried a for loop to automate the process but it doesn't work for me. Can someone help me please?

My files are: Sample_1.maf, Sample_1.csv, Sample_2.maf, Sample_2.cvs, ..., Sample_79.maf, Sample_79.csv

I wanted to paste Sample_1.csv after Sample_1.maf, Sample_2.csv after Sample_2.maf, and so on for all files.

Any ideas to automate?

Sorry to bother with this

Thanks in advance

Answer 1

[Mensur Dlakic]

I think this is lazy on several levels. First, it is a trivial task, and the least you could have done is show us the command that didn't work for you. Second, it would be nice to know how do you want the files to be formatted (still comma-separated? or not? does .maf or .csv come first when pasting?) and the resulting file names after pasting. It also would be nice to know what shell you are using. As it is, we have to guess these things on our own.

for i in Sample_*.maf ; do paste -d"," ${i%.maf}.maf ${i%.maf}.csv > ${i%.maf}_merged.csv ; done

This will paste .maf first followed by .csv, retain comma separation and save in files that end in _merged.csv.

bash is not my shell of choice, but this should work. For (t)csh shell:

foreach i ( Sample_*.maf )
paste -d"," $i:r.maf $i:r.csv > $i:r_merged.csv
end

[Gonzalo]

Dear Mensur, thank you very much for your answer, you have been very kind and generous. It was a great help. I apologize for the omissions, all observations are correct, sorry.

This was my initial script

for i in $(ls *.maf)
do
  for j in $(ls *.csv)
  do
    paste $i $j > $i.M.csv
  done
done

But this always added the same csv file to all *.maf.

With your suggestion I changed it like this

for i in $(ls *.maf)
do
  for j in $(ls ${i%.maf}.csv)
  do 
    paste $i $j > $i.N.csv
  done
done

Your script also worked fine

Thank you very much and sorry for the inconvenience.

[cpad0112]

I think this is lazy on several levels. First, it is a trivial task, and the least you could have done is show us the command that didn't work for you. Second, it would be nice to know how do you want the files to be formatted (still comma-separated? or not? does .maf or .csv come first when pasting?) and the resulting file names after pasting. It also would be nice to know what shell you are using. As it is, we have to guess these things on our own.

I think this post, along with other posts from one EU members make me rethink forum's future path. I think it's the time to say bye to this forum.