Script makes different file then the manual command, but the command is the same

Question

[stan.aanhane]

Hi,

I am working on a pipeline in a script, and want to have a specific section of my blast file in a txt file. But for some reason Linux can not work with the txt file. This is the code i use for this part:

blastn -db /exports/minor_data/BNGSA/refgenome/lclav.db -outfmt 6 -max_target_seqs 5 -query 50biggest.fasta > blastresults.txt
awk -F ' ' '{if ($3 >= "50.000" && $11 <="0.0") print $0}' blastresults.txt > Results_50_0.0.txt

And for some reason i get the error:

awk: cannot open blastresults.txt (No such file or directory).

The file is there, with the same size as the file when i do this manually. The code works manually, but when i use it in a script, it can not find the blastresults.txt file.

After some time looking I found that the text files are called 'blastresults.txt'$'\r' instead of blastresults.txt. And I have no clue why Linux makes this file, and how i fix this problem.

Can someone help me out?

Thanks!

[cpad0112]

Hope you have printed output location and awk execution path.

[stan.aanhane]

Well when i use the blastn command normal i will get the .txt file and the awk command works. But when i use it in the script, it says it can not find the file produced in the blastn command. But it is there, but it says TXT-file instead of textile behind the file. They are both the same size, so I'm really confused.

[GenoMax]

While a redirect > may work, you should use -out blastresult.txt option since that is the proper way to specify an output file.

[cpad0112]

can you try posting your script here?

[stan.aanhane]

    #! /bin/sh
echo "cp /exports/minor_home/reads/test/NINovo/contigs.fasta /exports/minor_home/reads/test"
cp /exports/minor_home/reads/test/NINovo/contigs.fasta /exports/minor_home/reads/test
wait
echo "awk '/^>/ {printf("\n%s\n",$0);next; } { printf("%s",$0);}  END {printf("\n");}' < contigs.fasta > contigs1line.fasta"
awk '/^>/ {printf("\n%s\n",$0);next; } { printf("%s",$0);}  END {printf("\n");}' < contigs.fasta > contigs1line.fasta
wait
echo "head -101 contigsoneline.fasta > 50biggest.fasta"
head -101 contigs1line.fasta > 50biggest.fasta
wait
echo "sed -i 's/NODE/lclav_config/g' 50biggest.fasta"
sed -i 's/NODE/lclav_config/g' 50biggest.fasta
wait
echo "sed -i 's/_/ /g' 50biggest.fasta"
sed -i 's/_/ /g' 50biggest.fasta
wait
echo "sed -i 's/length/lengte/g' 50biggest.fasta"
sed -i 's/length/lengte/g' 50biggest.fasta
wait

#Deelopdracht 8
echo "Deelopdracht 8"
echo "blastn -db /exports/minor_data/BNGSA/refgenome/lclav.db -outfmt 6 -max_target_seqs 5 -query 50biggest.fasta > BLASTResults.txt"
blastn -db /exports/minor_data/BNGSA/refgenome/lclav.db -outfmt 6 -max_target_seqs 5 -query 50biggest.fasta -out blastresults.txt
wait
echo "awk -F ' ' '{if ($3 >= "50.000" && $11 <="0.0") print $0}' blastresults.txt > Results_50_0.0.txt"
awk -F ' ' '{if ($3 >= "50.000" && $11 <="0.0") print $0}' exports/minor_home/reads/test/blastresults.txt > Results_50_0.0.txt
wait
cat Results_50_0.0.txt
echo "Pipeline Finished"

[GenoMax]

Looks like you are missing the first / in your path exports/minor_home/reads/test/blastresults.txt on awk command line.

[stan.aanhane]

unfortunately, this doesnt solve the problem... Same error. I think for some reason, when i use the blast command with the script the .txt file have different properties then the one is done manually. As you can see in the picture the one above(done manually) looks like a text file, and works. But the other one looks a bit odd, and doesn't work...

[cpad0112]

1. Can you try running ./script.sh or bash script.sh on small set and let us know if error still exists?
2. Try also running dos2unix on the script file before execution.
3. Can you edit this header line and see? from #! /bin/sh to #! /bin/bash (or wherever bash is installed.

[stan.aanhane]

1. I have tried this, with just 6 of the 50 configs, and the error still exists.
2. At this point dos2unix is not installed, I have to wait until i am able to install this (I'm working on a VM) 3 This doesn't make a different either

[lieven.sterck]

OK,

let's take this from the top:

• can you do (& post result) pwd in the directory where you see (or think) the blastresults.txt file is
• can you do (and post the result) of doing ls /exports/minor_home/reads/test/blastresults.txt
• can you do (& post result) pwd where you execute this script

[stan.aanhane]

• /exports/minor_home/reads/test
• When ls command is used, this error appears: ls: cannot access '/exports/minor_home/reads/test/blastresults.txt': No such file or directory
• /exports/minor_home/reads/test

The locations of the files look correct to me. The only thing is that i am not able to open the file in linux.

[cschu181]

Just to ensure: there is actual output in the blastresults.txt?

[stan.aanhane]

Yes there is. When i open the file, with Notepad, I can see the blast results. But my PC also does not recognise this file, I have to select open with Notepad to see the results. When I use the blast command manually, the pc recognises the file.

[GenoMax]

Sounds like you are accessing this file on unix partition from windows side using Notepad. That may be the cause of issues you are observing.

This used to be an issue once upon a time. This was supposed to have been fixed recently. If you don't try to open the file using Notepad do you still get the error?

[stan.aanhane]

When I open the file in Windows, it will open. I have to select notepad, but that's it. So no error, but the file does not change in Linux, the file cant be found. When I use the command on its own, the file looks different. If I open it in windows, it automatically opens, and I don't have to select notepad. And the file can be used in the next step.

[stan.aanhane]

I had a look at the files. Copied them both to windows, and for some reason, the script makes a .txt_ file instead of a .txt file. So that's the issue, but i don't see the reason why its making a .txt_ file

[GenoMax]

Not sure why that is happening. Closest issue that I could find where filenames got a trailing _ in the name is this thread but it is not directly applicable in your case.

You could try putting a mv blastresults.txt_ blastresults.txt before the awk command to see if it fixes the problem.

[stan.aanhane]

I tried, but the same result. Only at a different place. The error appears after the mv code now.

I was thinking, I can make the file with touch first, and add the data to this file. But even touch makes a .txt_ file. But this only shows when its copied to windows. When in Linux it still shows just .txt.

I am using Notepad++ for this script. Can the problem be within the program itself?

[5heikki]

1. Make sure you don't have the results file open in Windows (e.g. reboot)
2. Delete the results file
3. Confirm with ls that the results file no longer exists in that dir
4. Run your script

[stan.aanhane]

This dindt fix the problem. But something else drew my attention. The weird text files are called: 'blastresults.txt'$'\r' when I type in ls. But I don't know why Linux makes these type of files?

[lieven.sterck]

how are you running this script? From your windows machine/environment?

best would be to export that script to your linux environment and then do run dos2unix on it . (or on the output files generated by your script)

what you just descried is a typical thing when opening/running windows edited files in a linux enivronment (== differences in line endings between dos/windows and unix)

[stan.aanhane]

I am woking on my windows pc. Due to COVID-19 we are forced to work from home, so we made it possible, with the use of a VM(virtual machine) to work on the pc's at work, on our pc's at home. So I am working in Linux pc at a different location, from my own windows laptop.

I will have a look with dos2unix, but I think we need to install this, so I have to contact the right person.

If there is an other solution, that would be great!

[lieven.sterck]

you can custom implement what dos2unix does indeed, however, more efficient would be to just download the binary from the internet and install it in you home folder or such.

if you are working remotely you might need to look into an SSH terminal option, such as putty, mobaXterm, ... these will provide you access to your linux machines at the lab.

[stan.aanhane]

Thank you, I will have a look at that.

We indeed work from a SSH terminal called Bitvise.

[lieven.sterck]

that should be OK as well ...

in any case: top tip: when switching data/scripts/... from windows to linux or viceversa do never consider it will work as such ;-)

if in any way possible it would be best to choose one environment (linux ;) ) and stick to it :) , thus try to avoid opening scripts in notepad and then run them in linux

[GenoMax]

so we made it possible, with the use of a VM(virtual machine) to work on the pc's at work, on our pc's at home. So I am working in Linux pc at a different location, from my own windows laptop.

In this instance, your windows laptop becomes a means to work on the remote server. I am not sure why you are using a VM at all. You can use a simple SSH program like MobaXTerm to access your linux machine at work.

Then you said this in a following comment.

We indeed work from a SSH terminal called Bitvise.

So perhaps you don't use a real VM and are confusing a local terminal with a virtual machine. It appears that BitVise program has an secure FTP/scp graphical interface. You may be accessing files that are on your server via what looks like windows explorer file interface.

As @lieven said, if you only use the server for all of your work this odd behavior should not happen.

[stan.aanhane]

Well, I fixed the problem. It's the output of the Notepad++ file. It was still in windows format and added ^M to every line. I converted the line to Unix format, at it works fine now. Feel stupid to overlook this... But thanks for the help!

[GenoMax]

when i use it in a script, it can not find the blastresults.txt file.

Add full explicit path to the result file. Especially if you are running the script in a separate shell.

[stan.aanhane]

I tried this, but it gives the same error. I think the problem is in the file it makes. I ran it one time with the script(with -out) and one time manually. I added a picture so you can see how the files look. the top blastresults.txt file is the one made manually, the other is made with the script.

[5heikki]

How is it possible to have two files with the same name in the same directory?