Getting unusual characters while generating output files in a bash script

Question

[S AR]

Hi i im trying to run one command on mutliple files so i made a .sh script to do it:

 bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* > ERR040140.sam
bwa mem -M -R '@RG\tID:ERR046796\tLB:2799740\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR046796' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR046796_* > ERR046796.sam
bwa mem -M -R '@RG\tID:ERR046903\tLB:2806657\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR046903' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR046903_* > ERR046903.sam
bwa mem -M -R '@RG\tID:ERR067581\tLB:3712238\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067581' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067581_* > ERR067581.sam
bwa mem -M -R '@RG\tID:ERR067593\tLB:3712250\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067593' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067593_* > ERR067593.sam
bwa mem -M -R '@RG\tID:ERR067606\tLB:3712263\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067606' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067606_* > ERR067606.sam
bwa mem -M -R '@RG\tID:ERR067607\tLB:3712264\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067607' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067607_* > ERR067607.sam
bwa mem -M -R '@RG\tID:ERR067608\tLB:3712265\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067608' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067608_* > ERR067608.sam
bwa mem -M -R '@RG\tID:ERR067609\tLB:3712266\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067609' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067609_* > ERR067609.sam
bwa mem -M -R '@RG\tID:ERR067610\tLB:3712267\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067610' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067610_* > ERR067610.sam
bwa mem -M -R '@RG\tID:ERR067613\tLB:3712270\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067613' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067613_* > ERR067613.sam
bwa mem -M -R '@RG\tID:ERR067614\tLB:3712271\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR067614' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR067614_* > ERR067614.sam

But the output files are generating with unusual characters:

'ERR067610.sam'$'\r'
'ERR067614.sam'$'\r'

Whats the issue?

Can anyone help?

[Pierre Lindenbaum]

Whats the issue?

my guess: you're working with some windows files somewhere : https://en.wikipedia.org/wiki/Newline#Issues_with_different_newline_formats

see https://stackoverflow.com/questions/800030/remove-carriage-return-in-unix

[WouterDeCoster]

Try dos2unix on your script. It should remove windows line terminators.

Unrelated: there must be a more efficient way (e.g. using a loop) to execute these commands, rather than listing them all separately in a bash script.

[Kevin Blighe]

I agree - this looks like a job for dos2unix, a very useful tool too have. The solutions found through Pierre's links are also plausible. \r is an end-line character. Another commonly-found one is ^M.

[S AR]

Yes there are first i'm trying to do this if i get success that i make a loop or something. And i ran :

dos2unix ../pre_xdr_20_batch_run.sh

It outputted:

dos2unix: converting file ../pre_xdr_20_batch_run.sh to Unix format...

But i found no change..

[S AR]

And if anyone suggest a better way of doing it like a loop that can get file name from a list?

[Benn]

The unique read group header (string after -R) makes it a bit difficult for a for loop, unless you can extract that string easily?

[S AR]

The only thing that change in the read group header will be tID and tSM. that will be the file names.

[finswimmer]

Hello S AR ,

the similiar/same question you also asked on stackexchange. You should link to places where you asked the same question. Doing this everyone can see if your problem is maybe already solved.

Thanks.

fin swimmer

Answer 1

[S AR]

The error was resolved using :

#dos2unix
sed `echo "s/\r//"` ../pre_xdr_20_batch_run.sh > pre_xdr_20_batch_run.sh

Secondly I found an answer to run it in a loop :

https://bioinformatics.stackexchange.com/questions/5635/how-to-run-same-command-on-multiple-files/5640?noredirect=1#comment9431_5640