Memory usage of timeseries - lists of {Integer32, Real32}

4

2

A list of {Integer32, Real32} items in Mathematica seems to use 12-times as much memory than the C equivalent array of structs. And it looks like it can't be stored in a packed array, due to the integer/real mix. Besides splitting the time-series into two lists, one of timestamps and the other of values, so that each one can be packed individually, is there any other strategy?

Also, Mathematica 9 has a new function – TemporalData. Is that more efficient than a simple list?

Meh

Posted 2013-10-15T17:44:15.900

Reputation: 1 547

Mathematica's AbsoluteTime returns the Unix Time which is per default represented by a Real. Isn't it a possibility to use this?

– halirutan – 2013-10-15T18:04:58.533

@halirutan at least it's almost unix time: unixTime[date___]:=AbsoluteTime[date] - AbsoluteTime[{1970, 1, 1}] – ssch – 2013-10-15T18:16:15.383

Hmm, the source is UNIX time, but Mathematica seems to convert them to Integer when I use the following function: Epoch[timestamp_] = AbsoluteTime[{1970}] + timestamp – Meh – 2013-10-15T18:54:39.480

How do you assert that an integer is represented by Integer32 in memory? As far as I can tell, you have no control on that. – Hector – 2013-10-16T01:23:02.033

@Hector I don't, I just load them using BinaryReadList using {Integer32, Real32}. Even if Mathematica uses 64-bit ints/floats, there is still a 6x overhead over C. – Meh – 2013-10-16T01:46:38.603

Answers

1

I am not sure what seems to be the problem.

Module[{fn = "D:\\Temp\\exp.mx", lt = Transpose@{Range[10], N[Range[10]/3]}, at}, 
 BinaryWrite[fn, lt, {"Integer32", "Real32"}]; Close@fn; 
 Print[FileByteCount@fn]; 
 Print[at = BinaryReadList[fn, {"Integer32", "Real32"}, 10]]; 
 Print[lt];
 at - lt]

That file is as packaged as it can be; and, besides rounding errors, the information is recovered.

As for the contents in memory, you do not have control over those. I do not think Mathematica uses 64-bits integers but as intermediate objects. Otherwise, it would not be able to make arbitrary precision arithmetic. For example, you can fit $2^{30}$ in a 32-bit integer and store in disk as such. Once loaded to memory, Mathematica has no problem finding $(2^{30})^5$, which cannot fit in a machine-size integer. The 12x overhead space is the price we pay for the polymorphism that Mathematica gives.

Hector

Posted 2013-10-15T17:44:15.900

Reputation: 6 278

2I was asking about in memory usage, not on disk. ByteCount[ts] shows 12x overhead – Meh – 2013-10-15T18:56:24.533