23

10

When we don't know the number of results that will be generated, the usual way to collect results is `Reap`

/`Sow`

. Another alternative is linked lists. Neither of these are available in compiled functions.

`AppendTo`

does work, but it has an $O(n^2)$ complexity, so it will be unacceptably slow for long result lists.

There was a very clever suggestion to use `Internal`Bag`

in these situations:

Unfortunately a compiled `Bag`

will only hold scalars. Let's try to use vectors:

```
cf = Compile[{},
Module[{bag = Internal`Bag[{{0, 0, 0}}]},
Do[Internal`StuffBag[bag, {i, i, i}], {i, {1, 2, 3}}];
Internal`BagPart[bag, All]
]]
```

`CompilePrint`

shows that this calls `MainEvaluate`

, so this does not work.

**What is the best way to collect a large number of results in a compiled function when the
number of results is not known before the computation and the result type is not a (fixed size) vector or matrix?**

### Benchmarking the answers

**Andy's answer**`cf = Compile[{len}, Module[{bag = Internal`Bag[Most[{0}]]}, Do[Internal`StuffBag[bag, {i, i, i}, 1], {i, len}]; Partition[Internal`BagPart[bag, All], 3] ] ]; Do[cf[500000], {100}]; // Timing (* ==> {2.87, Null} *)`

I needed to initialize the

`Bag`

using`bag = Internal`Bag[Most[{0}]]`

to let`Compile`

know that it is holding integers, not reals (see here).**Leonid's answer**`cf2 = Compile[{len}, Module[{arr, lim, ctr}, arr = ConstantArray[{0, 0, 0}, 10]; lim = Length[arr]; ctr = 1; Do[ If[ctr == lim, arr = Join[arr, Table[{0, 0, 0}, {lim}]]; lim = Length[arr]]; arr[[ctr++]] = {i, i, i}, {i, len} ]; Take[arr, ctr - 1] ] ]; Do[cf2[500000], {100}]; // Timing (* ==> {16.474, Null} *)`

Comparing the computational complexity of the two solutions by direct measurement:

```
data = Table[
{Round[2^k], First@AbsoluteTiming@Do[cf[Round[2^k]], {100}]},
{k, 13, 19, 1/2}];
data2 = Table[
{Round[2^k], First@AbsoluteTiming@Do[cf2[Round[2^k]], {100}]},
{k, 13, 19, 1/2}];
ListLogLogPlot[{data, data2}]
```

(They're the same.)

1I think the reason

`cf2`

performs poorly is that it is not completely compiled. There is still a call to`MainEvaluate`

. – RunnyKine – 2015-06-19T23:54:57.187Do you mean when the number of results is unknown and the type of each result is a fixed dimensional list? Or you're asking for the more general case of different lengths and dimensions of each result to be collected? – Rojo – 2012-02-17T20:17:34.673

@Rojo I am asking for the case when each result is the same type (a fixed dimensional tensor) but the number of results is not known beforehand. – Szabolcs – 2012-02-17T21:02:27.033