Mathematica not releasing RAM

4

The following code is supposed to sequentially read data from a large CSV file and put it row by row into an SQL server. Although I thought RAM consumption should be constant the system keeps asking for more and more RAM.

$HistoryLength=3;
fileDestination="C:\\Users\\atfai\\Desktop\\destination.txt"
fileSource="C:\\Users\\atfai\\Desktop\\source.txt"

data=ReadString[fileSource,"\n"];
While[data=!=EndOfFile,
result=SQLInsert[conn,SQLTABLE["table_name"],availableColumns,StringSplit[data,","]];
If[result==$Failed,WriteString[fileDestination,data<>"\n"]];
data=ReadString[fileSource,"\n"];
];

Close[fileSource];
Close[fileDestination];

This code runs out of memory despite there being reading taking place row by row. What am I doing wrong here?

Also this code is slow since it it reading each row one by one. Is there a way to turn this into bulk insert but also be able to place rows which can't be inserted into the fileDestination?

I need help urgently if anyone can help.

user13892

Posted 2017-07-10T20:45:13.470

Reputation: 7 041

2A couple of observations: the $HistoryLength=3 is largely irrelevant here, as it only limits the length of completely evaluated expressions. The whole While loop is one complete expression, and retrhns Null in any event, so you're not saving memory. You could try printing the ByteCount of result, conn, and data, (and possibly also MemoryInUse[]) so you can track where the memoroy blow up is happening. I don't see anyting obviously wrong. – Itai Seggev – 2017-07-11T05:27:44.097

No answers