File Without Redundancy Using Awk

Question

[thiagomafra]

Hi people,

i have two fasta file with redundant reads. I want one file with all reads without redundancy, using awk. Someone help me?

[Neilfws]

Can I encourage users not to answer questions which fail the "what have you tried" test.

[Neilfws]

Seems the answer is no, I cannot :)

Answer 1

[Matt LaFave]

If the IDs of the sequences (the bit after the >) are the same for identical sequences, you could do something like this:

cat file1.fa file2.fa | awk '{if($1 ~ /^>/){name=$1}else{print name"\t"$1}}' | sort | uniq | awk '{print $1"\n"$2}'

If the IDs are not the same, and you're only interested in the sequences themselves, you could get those with sed:

cat file1.fa file2.fa | sed -n '2~2p' | sort | uniq

[Frédéric Mahé]

Hi Matt, your first command line could be simpler:

awk '{printf (/^>/) ? $0"\t" : $0"\n"}' file1.fa file2.fa | sort -u | tr "\t" "\n"

You can use a conditional expression to shorten the awk command; sort has a -u option to remove duplicates. In your second example, you can use awk to select only the sequences:

awk '! /^>/' file1.fa file2.fa | sort -u

[Matt LaFave]

Good to know - thanks!

Answer 2

[Rm]

Try CD-Hit or Usearch See the similar question: Generating a non-redundant gene set

Answer 3

[ewre]

not sure if you are looking for tools to remove duplicate lines, if so, using vim this can be done by: in command mode enter :sort ,hit enter to run and then enter :g/^(.*)$\n\1$/d

Answer 4

[Prakki Rama]

This is not awk, but can be used for the same purpose. See FASTQ/A Collapser.