Forum:When To Use Sed Awk Over Perl Or Python

Question

[Medhat]

Hi,

This is not debate or geeky question, It is simple as it is

sed program is a stream editor,AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool wikipedia, perl and python are programming languages and here in biostars a lot of talks cover perl vs python topic When To Choose Python Over Perl (And Vice Versa)?

My question

If you already knows Perl or python is it worth to learn Sed and Awk?

From your experience which tool of the four helped you best in solving bioinformatics problem and in what situation?

Conclusion: Which to use when?

Thanks

Answer 1

[Damian Kao]

For me it's about efficiency and not interrupting workflow.

I only use sed/awk for simple tasks when I know it will be faster to type out and execute than a perl/python script. I try to use sed/awk over scripts because they don't interrupt my workflow. When you are on the command line, it's nice to stay on the command line. Even having to open up vi, type out your script, and then saving can be annoying and interrupts your flow of thought.

However, if you spend 2-3 minutes perfecting a sed/awk one-liner when you could have written a script in 30 seconds, then that's probably not a very good use of your time and you have also interrupted your train of thought.

I think the important mentality to have when doing data analysis is that your goal is to work with the data, not to work with the tools.

[tospo]

you don't need to interrupt your CLI work - check out Perl one-liners (e.g. this great collection)

Answer 2

[Pavel Senin]

I walk to the office, use a bicycle to get to boulangerie, and a car to get to the ski station. While I can drive up to work and stop by a store on the way back, I really like walking and biking. Java can do all of that stuff perl an python do, but I still use awk/sed very often for no particular reason :).

[Medhat]

Nice and cheerful answer :)

[Pavel Senin]

Really sorry if you didn't like it. Personally, I think that it is awesome that there are so many tools around, so many ways to get things done, so many data sources... One can always find a tool or a way which makes more sense from particular personal (biased) perspective. I am trying to communicate that skills (such as in in sed and awk VS perl or pyton) are really driven by personal preferences and habits.

[Medhat]

I liked it by the way +1 :)

Answer 3

[Devon Ryan]

Sure, sometimes awk/sed prove simpler than perl/python (there are a HUGE number of awk examples on this site for that very reason). I use the former two for simple file processing all the time. If things are a bit more complicated, then you need perl/python. One caveat is that sed/awk lend themselves to one-off uses, so I generally just write something in python if I expect to use it numerous times.

[Medhat]

So sed and Awk are interchangeable but yet awk is more powerful?

[cts]

They're not really interchangeable. sed is mostly about changing the text of a line using regular expressions. Awk is about filtering, extracting lines and doing some maths on particular columns.

[Devon Ryan]

To reiterate what cts said, no, they're not interchangeable. I often use the two together since, they have different strengths.

Answer 4

[cts]

Perl can do everything that sed and awk can do on the command line, however I still use awk in many occasions because typing out an awk one-liner is usually shorter than the corresponding perl one-liner. I almost never use sed at all because I'm so used to perl's regular expression syntax. Awk has pretty much replaced my need for perl as it can do most of the basic data filtering that I used to do in perl scripts, but it does not support more complex data structures like having a hash or arrays etc. For that I usually use python, particularly if it is something that I'll use alot. I still use perl occasionally for quick scripts that I'm not likely to use many times as autovivification makes creating a complicated data structure very quick when I'm too lazy to write the same thing in python.

Answer 5

[alanh]

These days, almost anything that's more complex than a one-liner should probably be a Python script, but perl can still be useful.

For whatever reason, I learned Perl first back in the day and I've had little reason to learn awk or sed. Perl has command line options that are very nice for stream editing:

Extracted from the perl 5.32 command line help:

  -i[extension]     edit <> files in place (makes backup if extension supplied)
  -n                assume "while (<>) { ... }" loop around program
  -p                assume loop like -n but print line also, like sed

E.g. to replace all instance of foo with bar in file.txt and copy the original file to file.txt.bak:

perl -pi.bak -e 's/foo/bar/g' file.txt

or, say, if I want to extract only the lines that contain sample2 into a file, and correct a path:

cat file.txt | perl -n -e ' s{/wrong/path}[/path/right]g ; print if /sample2/' > file.only_sample2.corrected.txt