75

63

Since `Internal`Bag`

, `Internal`StuffBag`

and `Internal`BagPart`

can be compiled down, it is a precious source for various applications. There were already many questions why `AppendTo`

is so slow, and which ways exist to make a dynamically grow-able array which is faster. Since inside `Compile`

many tricks can simply not be used, which is for instance the case for `Sow`

and `Reap`

, this is a good alternative.

**A fast, compiled version of AppendTo:** For a comparison I will use

`AppendTo`

directly for an easy loop. Ignore the fact that this would not be necessary here, since we know the number of elements in the result list. In a real application, you maybe wouldn't know this.```
appendTo = Compile[{{n, _Integer, 0}},
Module[{i, list = Most[{0}]},
For[i = 1, i <= n, ++i,
AppendTo[list, i];
];
list
]
]
```

Using `Internal`Bag`

is not as expensive, since in the above code, the list is copied in each iteration. This is not the case for `Internal`Bag`

.

```
stuffBag = Compile[{{n, _Integer, 0}},
Module[{i, list = Internal`Bag[Most[{0}]]},
For[i = 1, i <= n, ++i,
Internal`StuffBag[list, i];
];
Internal`BagPart[list, All]
]
]
```

Comparing the run time of both functions uncovers the potential of `Internal`Bag`

:

```
First[AbsoluteTiming[#[10^5]]] & /@ {appendTo, stuffBag}
(*
{4.298237, 0.003207}
*)
```

## Usage and features

The following information was collected from different sources. Here is an article from Daniel Lichtblau who was kind enough to give some insider information. A question on MathGroup led to a conversation with Oleksandr Rasputinov who knew about the third argument of `Internal`BagPart`

. Various other posts on StackOverflow exist which I will not mention explicitly. I will restrict the following to the usage of `Internal`Bag`

and `Compile`

*together*. While we have 4 functions (`Internal`Bag`

, `Internal`StuffBag`

, `Internal`BagPart`

, `Internal`BagLength`

), only the first three can be compiled. Therefore, one has to explicitly count the elements which are inserted into the bag if needed (or use `Length`

on `All`

elements).

`Internal`Bag[]`

creates an empty bag of type real. When an`Integer`

is inserted it is converted to`Real`

.`True`

is converted to`1.0`

and`False`

to`0.0`

. Other types of bags are possible too. See below.`Internal`StuffBag[b, elm]`

adds an element`elm`

to the bag`b`

. It is possible to create a bag of bags inside compile. This way it is easy to create a tensor of arbitrary rank.`Internal`BagPart[b,i]`

gives the`i`

-th part of the bag`b`

.`Internal`BagPart[b,All]`

returns a list of all. The`Span`

operator`;;`

can be used too.`Internal`BagPart`

can have a third argument which is the used`Head`

for the returned expression.- Variables of
`Internal`Bag`

(or general inside`Compile`

) require a hint to the compile for deducing the type. A bag of integers can be declared as`list = Internal`Bag[Most[{0}]]`

- To my knowledge supported number-types contain
`Integer`

,`Real`

and`Complex`

.

## Examples

The important property of the following examples is that they are completely compiled. There is no call to the kernel, and using the `Internal`Bag`

in such a way should most likely speed things up.

The famous sum of Gauss; adding the numbers from 1 to 100. Note that the numbers are not explicitly added. I use the third argument to replace the `List`

head with `Plus`

. The only possible heads inside `Compile`

are `Plus`

and `Times`

and `List`

.

```
sumToN = Compile[{{n, _Integer, 0}},
Module[{i, list = Internal`Bag[Most[{0}]]},
For[i = 1, i <= n, ++i,
Internal`StuffBag[list, i];
];
Internal`BagPart[list, All, Plus]
]
];
sumToN[100]
```

Creating a rank-2 tensor by creating the inner bag directly inside the constructor of the outer one:

```
tensor2 = Compile[{{n, _Integer, 0}, {m, _Integer, 0}},
Module[{list = Internal`Bag[Most[{1}]], i, j},
Table[
Internal`StuffBag[
list,
Internal`Bag[Table[j, {j, m}]]
],
{i, n}];
Table[Internal`BagPart[Internal`BagPart[list, i], All], {i, n}]
]
]
```

An equivalent function which inserts every number separately

```
tensor2 = Compile[{{n, _Integer, 0}, {m, _Integer, 0}},
Module[{
list = Internal`Bag[Most[{1}]],
elm = Internal`Bag[Most[{1}]], i, j
},
Table[
elm = Internal`Bag[Most[{1}]];
Table[Internal`StuffBag[elm, j], {j, m}];
Internal`StuffBag[list, elm],
{i, n}];
Table[Internal`BagPart[Internal`BagPart[list, i], All], {i, n}]
]
]
```

A `Position`

for integer matrices:

```
position = Compile[{{mat, _Integer, 2}, {elm, _Integer, 0}},
Module[{result = Internal`Bag[Most[{0}]], i, j},
Table[
If[mat[[i, j]] === elm,
Internal`StuffBag[result, Internal`Bag[{i, j}]]
],
{i, Length[mat]}, {j, Length[First[mat]]}];
Table[
Internal`BagPart[pos, {1, 2}],
{pos, Internal`BagPart[result, All]}]
], CompilationTarget -> "C", RuntimeOptions -> "Speed"
]
```

This last example can easily be used to measure some timings against the kernel function:

```
times = Table[
Block[{data = RandomInteger[{0, 1}, {n, n}]},
Transpose[{
{n, n},
Sqrt[First[AbsoluteTiming[#[data, 1]]] & /@ {position, Position}]
}]
], {n, 100, 1000, 200}];
ListLinePlot[Transpose[times]]
```

## Open Questions

- Are there simpler/other ways to tell the compiler the type of a local variable? What bothers me here is that this is not really explained in the docs. It is only mentioned shortly how to
*define*(not*declare*) a tensor. When a user wants to have an empty tensor, it is completely unintuitive that he has to use a trick like`Most[{1}]`

. Declaring variables would be one of the first things I need, when I would be new to`Compile`

. In this tutorial, I didn't find any hint to this. - Are there further features of
`Bag`

which may be important to know in combination with`Compile`

? - The timing function of
`position`

above leaks memory. After the run`{n, 100, 3000, 200}`

there is 20GB of memory occupied. I haven't investigated this issue really deeply, but when I don't return the list of positions, the memory seems OK. Actually, the memory for the returned positions should be collected after the`Block`

finishes. My system here is Ubuntu 10.04 and*Mathematica*8.0.4.

Hi, halirutan. I am confused about the performance of Bag vs Append. I define

`appendc = Compile[{{x, _Integer, 1}, {y, _Integer}}, Append[x, y]]`

and`bagc = Compile[{{x, _Integer, 1}, {y, _Integer}}, Module[{bag}, bag = Internal`Bag[x]; Internal`StuffBag[bag, y]; Internal`BagPart[bag, All]]]`

. Timing`RepeatedTiming[Function[{x}, #[Range[100], x]] /@ Range[10000]][[ 1]] & /@ {appendc, bagc}`

shows {0.041, 0.079}, so my bagc is slower than appendc, what is wrong? I suppose bagc should be faster – matheorem – 2016-09-02T02:34:03.623Well, there is no use in using

`Bag`

or`Append`

to appendonenumber to a list! If you have a loop that calculates results one after another and you append every new result to the list of existing results,thisis that situation where`Append`

is bad and`Bag`

kicks in with all its performance advantages. – halirutan – 2016-09-04T22:27:55.553Oh, I understand. Create a bag is time consuming. If I define

`bagc = Compile[{{x, _Integer, 1}, {y, _Integer}}, Module[{bag}, bag = Internal`Bag[x]]]`

without any stuffbag manipulation, it is already slower then append version – matheorem – 2016-09-05T00:43:52.727