Question on: MiSeq SOP (follow up)

[davidalcoi]

I'm trying
"mothur > cluster(column=final.dist, count=final.count_table)"
in another computers with more RAM but it results in an error (the same one):

[ERROR]: std::bad_allocRAM used: 61.5826Gigabytes . Total Ram: 62.6216Gigabytes.

has occurred in the OptiMatrix class function readColumn. This error indicates your computer is running out of memory. This is most commonly caused by trying to process a dataset too large, using multiple processors, or a file format issue. If you are using multiple processors, try running the command with processors=1, the more processors you use the more memory is required. Also, you may be able to reduce the size of your dataset by using the commands outlined in the Schloss SOP, http://www.mothur.org/wiki/Schloss_SOP. If you are unable to resolve the issue, please contact Pat Schloss at mothur.bugs@gmail.com, and be sure to include the mothur.logFile with your inquiry.

With processors=1 (or 10 or 20) doesn't run either.

My files final.dist, final.count_table, and final.fasta have 262 GB, 50 MB, and 1,2 GB, respectively.

How much RAM is required? I think I have enough disk space (>1 TB free).

Thank you very much for your kindness.

[mothur-westcott]

[ERROR]: std::bad_allocRAM used: 61.5826Gigabytes . Total Ram: 62.6216Gigabytes.

From the error message, I can see that your computer has ~62Gigs of RAM. The distance matrix is 262GB and is too large to cluster on your current machine. Do you have access to a computer with more RAM?

[davidalcoi]

I will expand the RAM to about 160 GB, is that enough?

[mothur-westcott]

That won't be enough. I am assuming the distance matrix was created with a cutoff, meaning the entire matrix must fit into memory. The count file must also fit into memory. Mothur also needs additional memory to calculate the clusters. ~300G would likely do it.

[pschloss]

You might also want to check out this blog post... https://mothur.org/blog/2014/Why-such-a-large-distance-matrix/

[davidalcoi]

Yes, the distance matrix was created with cutoff=0.03. I don't have access to a computer with ~300G, I will have to do the processing by sample groups.

Thank you very much.