code for text file parsing

Question

[sharmatina189059]

Dear all can anyone tell me how to parse this type of file using either perl or awk

AUO97_RS0005
AUO97_RS0005    alpha  hydrolase wp_567465 GI:54365463
AUO97_RS0007
AUO97_RS0007    beta   hydrolase wp_567465 GI:65456475
AUO97_RS0020
AUO97_RS0020    gamma   hydrolase wp_567465 GI:4536473

I want to retrieve only only those values having data in next columns and remove duplicates. Output file:

AUO97_RS0005    alpha  hydrolase wp_567465 GI:54365463
AUO97_RS0007    beta   hydrolase wp_567465 GI:65456475
AUO97_RS0020    gamma   hydrolase wp_567465 GI:4536473

[kloetzl]

Your formatting makes any effort on our side impossible. Please put the example into code blocks to preserve new lines and other formatting.

[sharmatina189059]

AUO97_RS0005 
AUO97_RS0005 alpha hydrolase wp_567465 GI:54365463 
AUO97_RS0007 
AUO97_RS0007 beta hydrolase wp_567465 GI:65456475 
AUO97_RS0020
AUO97_RS0020 AUO97_RS0020 gamma hydrolase wp_567465 GI:4536473

[Alex Reynolds]

Your question is unanswerable as written.

[sharmatina189059]

file is like this:

AUO97_RS0005                
AUO97_RS0005    alpha   hydrolase   wp_567465   GI:54365463
AUO97_RS0007                
AUO97_RS0007    beta    hydrolase   wp_567465   GI:65456475
AUO97_RS0020                
AUO97_RS0020    gamma   hydrolase   wp_567465   GI:4536473

Answer 1

[EagleEye]

How is the second line from above example separated ( space or TAB ) ?

If TAB delimited, you can extract the lines containing TAB delimited entries.

grep -P "\t" YOUR_FILE.txt > DESIRED_OUTPUT.txt

If the single column entries in the above example also has TAB in the end,

grep -P "\t.*\t" YOUR_FILE.txt > DESIRED_OUTPUT.txt

[sharmatina189059]

It is tab separated but i have done it using awk. awk -F '\t' '$2!=""' file2. but thank you so much for your reply.

Answer 2

[Alex Reynolds]

I suspect this will generally be fastest on the larger inputs, by avoiding cat and sort and uniq operations:

$ awk 'NF > 1 && !a[$0]++' in.txt > answer.txt

[e.rempel]

Could you please explain what

&& !a[$0]++

does?

[Alex Reynolds]

It strips duplicate lines without needing to sort the input. It can be memory hungry, but memory is cheap and fast, these days. If you can do without sorting, go for it, I say.

Answer 3

[Joe]

Full bash solution, just for the sake of it!

#!/bin/bash
# Usage: bash scriptname.sh textfile.tsv

while read line ; do
    read -a fields <<< "$line"
    [ ! -z "${fields[1]}" ] && echo "$line"
done < $1

Or as a one-liner:

while read line ; do read -a fields <<< "$line" ; [ ! -z "${fields[1]}" ] && echo "$line" ; done < textfile.tsv

Answer 4

[kloetzl]

cat file.txt | awk 'NF > 1' | sort | uniq

Answer 5

[cpad0112]

$ cat test.txt 
AUO97_RS0005
AUO97_RS0005    alpha  hydrolase wp_567465 GI:54365463
AUO97_RS0007
AUO97_RS0007    beta   hydrolase wp_567465 GI:65456475
AUO97_RS0020
AUO97_RS0020    gamma   hydrolase wp_567465 GI:4536473

ouput: awk solution

$ awk '$2 == "" {next} {print}' test.txt
AUO97_RS0005    alpha  hydrolase wp_567465 GI:54365463
AUO97_RS0007    beta   hydrolase wp_567465 GI:65456475
AUO97_RS0020    gamma   hydrolase wp_567465 GI:4536473

non-awk solution:

$ grep -i GI test.txt 
AUO97_RS0005    alpha  hydrolase wp_567465 GI:54365463
AUO97_RS0007    beta   hydrolase wp_567465 GI:65456475
AUO97_RS0020    gamma   hydrolase wp_567465 GI:4536473

[GenoMax]

A similar solution has been used/posted by OP already.