Forum:I want to re-open the old debate: python or perl ?

Question

[Gabriel R.]

Hello! I have a colleague who is interested in learning a scripting language to

• parse files in various formats relevant to computational biology (BAM,VCF,bed, genes lists in plain text etc,,,)
• automate processes
• do computations of course

I tried to encourage her to learn as much bash as she can but the shell has its limits. When a awk command becomes so convoluted and bloated, a little script can be helpful.

Now I am trying to think what is the best scripting language to learn for someone with no programming experience that will give her the best bang for her buck. First, a few reasons I dislike Perl

• Programming community not as active, there seems to be more libraries for the biological sciences in Python
• Can be hard to read/interpret

and reasons why I dislike python:

• The @$@#( indentation system drives me insane
• Seems slower than Perl, maybe it's just in my head
• Lack of "plug and play" regular expressions
• The weird type system. I fine with the rigid C++ type system and the anything goes of Perl. But python's system can be a pain.

What are your thoughts? Should I teach that person Perl or Python? What is easier and better in the long run?

[Devon Ryan]

That "obfuscated perl" is typically just a redundant phrase is argument enough for me to prefer python :)

My main problem with python is memory management, but with numpy and such there are usually good ways around that. Plus, ipython notebook and pandas are really useful, not sure if similar things exist for perl (they very well might, I honestly don't know).

[Ashutosh Pandey]

Here we go (again) :-)

[Devon Ryan]

If nothing else, this thread is proving to be a treasure trove of witticisms :)

• "Dealing with other people's Perl is often like dealing with other people's stool samples--it's not a job many programmers want." -Deedee
• "Using lambdas is tantamount to going to work while wearing a a deer skin poncho and bringing grilled rats for lunch ." -joe.cornish826

[Istvan Albert]

Here I am in the morning - trying to get some work done - instead I am laughing out loud -

I like lambdas - I try to replace my 'for loops' with function application via map and 'if conditionals' with filters - and I've come to believe that doing so makes code shorter and more robust. It seems those sections of code rarely have bugs. But then I do agree the code does look quite different - never considered it deer skin poncho though.

[pld]

You would be the first person I've spoken with that feels this way, almost every other case it is complaining. For the level of grief the python community will give someone over using these features, I've always wondered why they haven't been dropped from the language.

[Dan D]

Add me to the "lambdas are awesome" group. map, reduce, filter, and list comprehension syntax make life super-easy.

[Istvan Albert]

This a somewhat divisive issue and your post made me look into what the current zeitgeist might be on the subject.

Along the way I found this article: The Fall Of Perl, The Web's Most Promising Language - now it captures what I think - but then it might just be confirmation bias.

[SES]

A similar article was published on DDJ by Andrew Binstock, and there is a good response by brian d foy about why this type of analysis may be flawed. Regardless, I do agree with the main point that Perl is declining because fewer libraries are being published and more Python is being used. I think this is natural, Perl is over 25 years old so I would expect there to be stable libraries and less need to create new ones, and likewise, Python will decline at some point. I would not expect either to go away for a long time though, unless something revolutionary comes along (Perl 6? :-) ).

Answer 1

[Dan D]

I am fluent in both languages. I learned Python after Perl, and since then I've never had a reason to go back to Perl. Perl is a cool language but it has had its day. I can more quickly write working, testable code in Python than Perl, and getting things done quickly and correctly is the bottom line for me.

You pretty much answered your own questions based on what your colleague needs to do. Perl is messy and Python is better supported in the bioinformatics context. If your colleague has no programming experience, slightly easier implementation of regexs is a moot point. Same with the type system. Perl's contextual interpretation of variables can be very difficult to understand for the novice. Python's types aren't too hard to understand. The only thing I ever got hung up on was unexpected automatic conversion of arithmetic expressions to ints. Nothing too painful there.

EDIT: I totally forgot about OOP. I think its implementation in Perl is horrendous. It's like you're hacking the language to use it. In contrast, OOP is beautiful in Python. With my C# experience I picked it up at warp speed. When I look at OOP-ish Perl code my brain hurts. I use OOP quite a bit when solving novel bioinformatics problems, so this is a major consideration for me.

I'm going to argue that Perl vs Python is not simply a matter of taste. Yes there's a huge overlap in what can be done between Python and Perl. Yes, there is a large element of personal preference and comfort. But there's also a way to get readable, maintainable code written faster, and that way is Python.

[SES]

Your comments may have been valid 15 years ago, but this is an old argument that simply isn't relevant today. Moose is arguably the richest object system available, and it's been ported to other languages like JavaScript (see Joose). Moops is also an awesome library that allows syntax and features not available elsewhere, and these features will be in core next year (version 5.22) as part of Class::MOP (which was inspired by the object system in Perl 6).

[Ram]

The very fact that namespace.object.method() (from any sane OOP/OBP - Java/C#/Python/JS) becomes namespace::object->method() is syntactic hell enough. Unless Class::MOP brings in a whole new syntax, Perl OOP is not getting any more bearable, IMO. I'm all for Perl, but Perl OOP is a bad joke.

[SES]

Indirection is one way to call a method, another way is to give the fully-qualified name, and in this case the two are basically identical (different package separator). Let's not argue over this minor stuff though, that was certainly not my intention. All OOP systems are inspired by other communities, and they grow by bringing in new ideas. So do we, so let's try to keep an open mind. Use what you want and have fun.

[Dan D]

The zeitgeist (both in bioinformatics and the programming community in general) profoundly disagrees with your assertion about the validity of my comments. Furthermore, I don't see what would have been different 15 years ago about what I said, besides maybe OOP in light of your Moose reference. The problem with Moose, though, is that you still have to deal with Perl to use it. Looking through the Moose examples, the logic of the code doesn't pop out at me like Python (or Java/C#) code does. It does look pretty neat though. Wish I'd known about it earlier.

I have a soft spot for Perl. I just don't have any reason to use it anymore, despite its charms. Look at StackOverflow, look at coding challenge sites, look at Github. The move away from Perl is simply undeniable. I've always contended that it's like no other language in that you can see how someone's brain works by looking at their Perl code.

[SES]

Yes, I was referring specifically to the comment about OOP, sorry if I was unclear or if it came across a different way.

[Ram]

Hello there fellow C# dabbler! I must say, after LINQ, python's lambdas fall so SO short of expectations!

[Dan D]

Agreed. C# is wonderful in so many ways.

[Gabriel R.]

Thank you for your perspective !

Answer 2

[pld]

Honestly, who cares?

They're both high-level languages with different approaches to syntax. Python is designed on the idea of "least suprise", so the syntax is "low power" and rigid. Perl isn't designed like this so you have a "high-power" syntax with tons of flexibility. Python says there's one right way for each task, Perl says you can do one task any number of ways depending on how you feel.

Sometimes Python sucks because you have to write more code than you would have to in Perl, sometimes Perl sucks because you're trying to read dense code that you/someone wrote while thinking "oh man, I can be super cool and write this in 10 lines". One can write stupid Python code just as well as one can write readable Perl code.

Python is newer and in vogue, so it is growing but isn't as mature as Perl. Perl is older and not as popular, so it has a very high level of maturity but the community and industry usage may be shrinking. This is the only objective reason I can see picking one over the other, in the long run knowing Python may be a better place to start.

In the end, just like every programming language A vs B question, it doesn't matter too much. As long as A and B are reasonable choices, the only reasons one may be better are personal preferences/experience and case-specific requirements. You might pick A over B because you have more experience with A, but maybe performance really matters so you'll spend more resources on development to gain performance.

I wouldn't try and do absurd numerical method stuff with python, just like I wouldn't write a script to parse FASTA files with FORTRAN.

[Gabriel R.]

Thanks for your comment. Very interesting !

Answer 3

[lh3]

Newcomers should learn python. It is not just about taste. Python is a proper structured/OO language that helps students to develop skills and good habits in programming especially for large collaborative projects. Many of these skills can be transferred to programming in other industrial languages such as C++, C#, Java, etc. It is technically possible to learn perl the modern way, but it is really not designed for that.

Answer 4

[Alex Reynolds]

It's not in your head. Python is very slow at I/O and string manipulation, particularly, and so it should be avoided for I/O- and string-heavy work. Threading is a mess. Python is memory heavy. Perl is much more C-like and performant in that respect.

While OO design is easier in Python than Perl, Python is not optimal for projects that require designing more than one or two classes. It's better to use a faster, more featured and strongly-typed OO language like C++ at that point.

Python usage is split between two incompatible major versions. Some features do not work within a major version, between minor versions that are commonly found on older Linux workstations. Both aspects are problematic in that they make Python a nightmare to support for production scripts. Perl is mature and doesn't often break old code, however unreadable it might be.

Basically, where you want speed and maturity, use Perl. Where you might want to prototype something OO that you will ultimately rewrite in a better language, use Python.

[dariober]

About speed: In theory you are probably right (although it's not obvious from this comparison). But in practice? In my experience the difference in speed between the two is negligible in most if not all cases, either they are fast enough (say to stitch together a pipeline or to parse files of few 100s of MB) or they are both too slow (say to write an NGS aligner or parse TB of data). In my opinion code readability coupled with fast development is far more important than raw speed, within reasonable limits of course.

About maturity I would consider python fully mature (younger then perl, but still...). Apparently at Google they say "python where we can, C++ where we must".

I agree about python 2 vs 3. Version 3 seems a sort of false re-start. Personally I stick to python 2 and never had compatibility issues.

[lh3]

On performance, python and perl are similar in various benchmarks, except that on simple regex perl is way faster. This is actually a show stopper for me. On the other hand, python has pypy, but perl has nothing similar.

[Alex Reynolds]

Beyond performance problems, another serious problem with Python is that one cannot easily replace the values of parameters passed to functions, without using globals or classes. Globals are usually frowned upon in programming, when the goal is to write production code. Having to write a class to get around Python's restriction here defeats the purpose of using Python, i.e., to reduce verbosity. It seems odd, to me, that Perl gets so much grief when Python does certain things poorly, and apparently on purpose.

That said, it is useful to learn both languages, if only because they show up everywhere in this field. When in Rome, etc.

[Istvan Albert]

I do agree that there is one major language design flaw in Python with respect of the scope of variables. Now the majority of programmers and especially those that program with imperative style will never run into but it is painful when one realizes that this precludes us from building certain types of solutions.

Example:

x = 1

def test1():
    # This works
    print x

def test2():
    # This produces a UnboundLocalError traceback!!!
    x = x + 1
    print x

test1()
test2()

They've "fixed" this in Python 3 by introducing a new keyword nonlocal but that is not a pretty solution either.

The reason that this error exists is that in test2() listing the x on the left hand side turns it into a local variable on the lookup table - and the lookup does not cascade up the scope to find the global as it did in test1() now either way would be fine but what is not ok when the scope changes just by listing values on left vs right hand side of an assignment.

more details in blog posts like this: http://technotroph.wordpress.com/2012/10/01/python-closures-and-the-python-2-7-nonlocal-solution/

[Brian Bushnell]

What's wrong with a function returning a value?

[pld]

Why not curry? I might not be understanding you totally.

You can't really gripe too much about performance with Python, if you can, use a JIT compiler (e.g. PyPy) otherwise if performance is such an issue maybe Python isn't the best option. Or, be a terrible person and write a Python/Perl script for something that should be done in a different language and hog up your local cluster.

Nothing is worse than someone getting priority for their 4000 single threaded Python/Perl (and R) jobs because your single C/MPI 40 node job can't fit.

However gripes I do have is how expensive stack frame allocations and therefore function calls are in Python and that optimizing is sort of impossible. The stack frame cost can make purely functional (although cheating with file IO side effects) very slow, in some cases they're not worth pursuing even "for fun".

The optimization thing is a gripe because this is where the concept of "least surprise" starts to fall apart. I'm not talking about fine grain optimization after development, but in the general syntax. An great example is in joining strings. Another would be generating lists with map being slower than a list comprehension. These instances are the result of optimizations of specific cases or optimizations of specific approaches, but from the syntax of two approaches it isn't obvious that one would be faster than the other.

Answer 5

[Vivek]

Perl still appears to be about as actively used as Python even until very recently and I don't really agree that the community is underserved, I find CPAN to be pretty extensive and awesome.

http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html

I mostly use Perl for my scripting work because the existing code in our repositories is in Perl, I love the easy and quick regex and the indentation in Python is a major turn off.

[Gabriel R.]

Thanks for your comment !

Answer 6

[Ram]

I like (and hate) both equally. Transitioning from shell scripts to Perl is easier. Transitioning from a C/C++ bg to Python is more beneficial. Transitioning from no background to Python is ridiculously easy.

Both have good community support, though Python is outgrowing Perl by a lot. It is easier to write a moderately large application in Python.

Personally, because I come from a background where languages have blocks and scope defined by curly braces, Python drives me insane. It might be a minor reason, but I have to put in conscious effort to suppress my rage each time I write a bit of Python code and for this reason, I prefer Perl.

My MO is based on my convenience and the type of solution I want to design. It typically follows: Quick and dirty - shell. Slightly more complicated - Perl. Will structure and maintain a solution long term - Python.

[Gabriel R.]

Thanks for your input, I feel the same about the lack of curly brackets.

[girishwd]

I like your concerns about efficiency of software than 'quick-dirty'.

Answer 7

[Brian Bushnell]

The language of choice really depends on what you want to do. Write algorithms that do high-performance math? Choose a compiled imperative language. Pipelines that execute programs, move files, and track statuses? Interpreted languages tend to be good. Enjoy pure theory, and already have tenure? Stick with functional languages.

Perl teaches terrible programming practices that are hard to fix if you ever want to transition to a well-constructed language, so I would never recommend it to anyone, for any purpose. Python, on the other hand, while similarly low-performance, is somewhat properly-structured. The whitespace/indentation-based syntax and "under-under" conventions that are sort of required, but not really formalized, and lack of real threading or other features relevant to modern computers kill it for me as a serious choice for algorithm development. But still, it's quicker and easier to write a pipeline manager in Python than compiled languages.

For a first introduction to programming, I think Python is much better than Perl, though not great. Once a person understands the basics of programming, if he/she desires to develop algorithms, I recommend transitioning to a typed and compiled language; compilation really catches a lot of bugs that can be introduced by the "forgivingness" of interpreted languages like Perl, Python, and Matlab.

For pipeline-control programs that don't do extensive math, but mainly move files and execute other programs - I don't know of anything better than Python (but there are many languages out there I have not tried). Interpreted languages really shine here.

[Dan D]

Perl teaches terrible programming practices that are hard to fix if you ever want to transition to a well-constructed language, so I would never recommend it to anyone, for any purpose.

Ehhhhh...that's a pretty bold assertion. With that said, I get calls almost weekly from various desperate recruiters who saw "Perl" on my public resume asking if I'm interested in a particular Perl developer position that's been open for over eight months. The first item on the job description indicates that the role will involve updating and improving an existing Perl code base.

Dealing with other people's Perl is often like dealing with other people's stool samples--it's not a job many programmers want!

[SES]

I don't think it's fair to say "perl teaches terrible programming" because modern guides (3 books published in 2014) teach best practices that are common in all languages. I would agree that the flexibility can be abused (perl does has a rich type system, btw), but that is also an advantage. edit: To be clear, I should have said "system for type constraints" because it's not a full-blown type system like other languages (and I don't want to imply that).

[pld]

This is something that has always bothered me, excluding purposeful features designed to make code hard to read, there's really not any reason a given language should "cause" people to write bad code. I think there are cultural components in the user community, and flexibility in syntax can lead to hard to read code, but Larry Wall isn't holding you at gunpoint demanding you write ugly code.

I think culture, experience and training are major drivers. If all you've seen is college comp sci java and SO downvote tirades over non-pythonic Python, everything else will seem like a pile of garbage. Even languages considered to be fairly arcane (e.g. Haskell) aren't by definition hard to read, it just comes from the typically large inexperience with purely functional languages.

I have to admit, I write Perl scripts when I'm bored to tears with the banality of Python. Python is great but it is such a bore. There's very little room to play around unless you want to use some of the functional aspects (yay lambdas) and then you're just being a non-pythonic heathen from the wastelands. Using lambdas is tantamount to going to work while wearing a a deer skin poncho and bringing grilled rats for lunch.

[Istvan Albert]

There are certain types of constructs that been have proven to be more difficult to comprehend than others. For example how a language chooses to represent a data structure does matter, take the array and scalar context that data structures have in Perl. I still remember how much trouble I had with the sigil notations @ and $->{} etc. that in the end refer to the exact same data structure. And by now I forgot all that and would need to re-learn it.

Languages that rely on such constructs will be more difficult to learn - and hence the code written in them will be more difficult to deal with later on. I think that is really what happens with Perl - I for one believe that Perl's ease of use is an illusion - just as the ease of use of R is an illusion as well - they actually share many commonalities. It all comes down to too many ways to express a functionality by relying on tacit behaviors and non explicit effects. Only that R is still growing in popularity since it has no comparable free competitor as Python/Ruby are for Perl. If there were no Python and Ruby Perl would be the king of programming languages.

[pld]

I've always found R much more difficult, Perl can be ugly but at least there's some sort of consistency. R has always felt like it is a thousand different languages that all happen to work with the same interpreter. Perl might be hard to read, but at least one can grasp a foundation and start writing code. In R, predominantly in Bioconductor, it has always felt like I'm stuck trying to re-purpose vignettes or waiting for someone else who has somehow found the answer.

Maybe it is just me, but it seems that one can at least learn Perl. With R (again, Bioconductor) you're stuck kludging examples into working solutions and hoping someone answers your post (typically with something that zero previous experience would have indicated).

[SES]

About sigils in Perl, I think the actual issue is in dereferencing and that people use two different forms (circumfix and postfix). In version 5.14 there was a feature called autoderef that was added (and marked experimental in 5.18) which allows you to treat everything as a scalar. I guess there is ambiguity with this syntax and people want to know what they are working on as before. So, a new feature has been added to 5.20 called postfixderef (explained here). These are experimental but I think it is interesting to see how people are trying to improve the core.