Reading vcf file using python gives UnicodeDecodeError

Question

[Medhat]

python version Python 3.5.4

Trying to process vcf file as shown in the code below:

 with open(myfile, 'r') as datain:
    for myline in datain:
        if myline.startswith("#"):
            pass
        else:
            do something

In some line (after processing couple of hundreds of lines) it raises this error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 8077: invalid start byte

Any idea how to fix this error?

[Kevin Blighe]

Did you try this? - https://stackoverflow.com/questions/22216076/unicodedecodeerror-utf8-codec-cant-decode-byte-0xa5-in-position-0-invalid-s

[e.benn]

This se answer is related, but dealing with a python 2 encoding problem. python 3 behaviour is fairly different in this area.

[Medhat]

As you mentioned it is already decoded.

[Medhat]

@ Kevin Blighe If I used the code like: myline = myline.encode('utf-8').strip() It will give this error:

TypeError: startswith first arg must be bytes or a tuple of bytes, not str Then I could use str() but this will not help in fixing the main issue.

[GenoMax]

Are you using an European locale setting by any chance? What does locale produce? You may need to force the locale to be UTF-8. See this: https://unix.stackexchange.com/questions/87745/what-does-lc-all-c-do

[e.benn]

The error message shows that python is using utf8 to decode. The question is what codec of the input

Answer 1

[e.benn]

Python 3.0+ decodes files by default.

https://docs.python.org/3/library/functions.html#open

"the contents of the file are returned as str, the bytes having been first decoded using a platform-dependent encoding or using the specified encoding if given."

Apparently your file is not encoded in utf-8. My guesses would be that it is gzipped, or encoded in some other encoding. What happens if you file myfile.vcf ? Does it report gzip encoded data? Does it report the codec?

Try:

# read as binary (~ascii) data (like python 2)
with open(myfile, 'rb') as datain:

# specify the actual encoding
with open(myfile, 'r', encoding="ascii") as datain:

# the vcf is gzipped
with gzip.open(filename) as datain:

[Medhat]

Actually, it is not gzipped file. If I read as binary 'rb' I need to convert it again to string.

[e.benn]

The vcf spec just states 'text' encoding. https://samtools.github.io/hts-specs/VCFv4.2.pdf Most likely you have ascii. Specify ascii as the encoding parameter to open. There is some chance you don't have ascii if the file has been moved between platforms. What does the file program say?

[Medhat]

The file is data also the loop does not fail in the first line it reads couple of hundreds of line then raise that error.

[Medhat]

when using: with open(myfile, 'r', encoding="ascii") as datain:

It changes the error to : UnicodeDecodeError: 'ascii' codec can't decode byte 0x9d in position 8077: ordinal not in range(128)

Answer 2

[Medhat]

So I manage to work around , but did not fix the issue.

The original file was a result of using bcftools merge A.vcf B.vcf C.vcf > someresult.vcf and that gave me the error above.

But when using vcf-merge A.vcf B.vcf C.vcf > someresult.vcf The issue was fixed.

This does not clearly answer my question, But it fixed the issue.

[Ram]

Always specify output format with bcftools, it screws you over otherwise.

[Medhat]

I tryied your suggestion using:

-O z -o result.vcf.gz

But still have (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 8116: invalid start byte

[Ram]

Does reading the individual files work OK?

[Medhat]

Yes I will just stay with file from vcftools

[e.benn]

When you specify to bcftools that the output is to be gzipped, you get the same error. This again suggests to me that the original problem was a gzipped vcf file. What is the result of running file myfile.vcf

[Medhat]

The old file was data as I mentioned in a comment on your answer. C: Reading vcf file using python gives UnicodeDecodeError

[e.benn]

Apologies, I missed that. On my system a .vcf file is reported as : Variant Call Format (VCF) version 4.1, ASCII text A .vcf.gz files is reported as: gzip compressed data, extra field. It sounds like the bcftools command may be making a corrupt file, maybe a bug report would be useful. Did you check the bcftools command for an error code? What happens if you bcftools view the merged file? Which bcftools version?

[Medhat]

I do not have access now But what you are suggesting is really interesting, I will try and keep you updates (when using bcftools there were no errors)

[e.benn]

Hopefully you can get to the bottom of the issue. When I encounter a data problem like this I get scared - if such a well used program as bcftools has an issue with the input it could mean the data is corrupt - maybe the alternative tool vcf-merge is failing in a different way that seems to work, but is actually giving you garbage data.

[Ram]

There are always edge cases that make even well-tested tools like bcftools fail. An example is using bcftools norm on an indel-heavy VCF - I ran into a scenario where it made an entry with identical chr, pos, ref and alt entries, with just the DP/DQ/something-of-that-sort different.