I created a 127,545 x 1,234,620 array with np.memmap. If my math is right (127,545 1,234,620 4/8/1,000,000,000), my array should be 78.73 GB at 4 bits per cell. However, when look at it’s properties it is 586 GB. I’m definitely missing something here. However, I’m just glad it worked and now it has been created. This took about 2 hours to create.
In any event, I need to populate it now and I am deathly afraid of how long it will take to do this. Basically, I have a unique integer that represent each of the rows (so 127,545 integers) and set of integers that represent each column (so 1,234,620 sets of integers) . If the row integer is in the set of column integers, then that row/column cell gets a 1, otherwise 0.
Would love some advice on how to do this as efficiently as possible.