Space Required to run 15 GB File in python

Question

[anasjamshed]

I want to know how much ram memory is required to run a 15 GB data file and to clean datasets through python pandas.

I have a file of 14.8 GB which contains genes information. It contains a total of 45804630 rows and 39 columns

When tried to open it by taking 1000000 rows through pandas it was working fine

Code:

data = pd.read_csv("CosmicGenomeScreensMutantExport.tsv", sep= '\t',nrows=1000000)

But when I am trying to read all datasets at once, it hangs my pc. I only have 4 GB ram. So should I increase my RAM?

[5heikki]

If you need to have a 15G file in memory then you obviously need more than 15G RAM, possibly a lot more depending on what precision you are reading your data (e.g. float16, float32, etc.). The pandas concat function could be of use to you, check e.g. this

[anasjamshed]

I am trying to use this code:

#Reading file in chunks as data is very large
chunks=pd.read_csv("CosmicGenomeScreensMutantExport.tsv",chunksize=1000000,sep='\t')

#Make dataframe by concatating list of chunks
df= pd.concat(list(chunks), ignore_index=True)

from my friend pc who has 8 gb so will it work?

[5heikki]

I don't understand why you would need to have the whole file in memory if you only need to "clean" it, whatever that means.. sounds like something that shouldn't require all the rows at the same time, so why not just process the file in chunks?

[anasjamshed]

but i want to clean all datasets that's why i need to upload all at once

[5heikki]

I'm just guessing here, but you probably need like minimum 64G RAM if you want to have your file in memory..

[anasjamshed]

Ohh no. How can I acquire 64 GB