## How to avoid nested With[]?

80

52

With[
{v1 = #},
With[
{v2 = f[v1]},
g[v1, v2]
]
]


How to avoid nested With[] like the above? I'd like to use v1 and v2=f[v1] in the module's body. Is using Module[{v1, v2}, v2=f[v1]; g[v1, v2]] the best/only way to avoid nested module?

6

In Mathematica 10, you can do it with Inactivate, see example here.

– István Zachar – 2014-07-10T12:08:36.070

Strongly related: "A version of With that binds variables sequentially" and "Iterative constants and variables definitions."

– Alexey Popkov – 2014-07-15T07:45:45.570

65

I don't think one can avoid the need for nested With altogether - I find it a very common case to need declared variables use previously declared variables.

Since I once wrote the function (actually macro) that automates nesting With, and generates nested With at run-time, this is a good opportunity to (re)post it as an answer to an exact question that it actually addresses. I will partly borrow the discussion from this answer.

### Implementation

Edit Aug.3, 2015 - added RuleDelayed UpValue, per @Federico's suggestion

Here is the code for it (with added local-variable highlighting):

ClearAll[LetL];
SetAttributes[LetL, HoldAll];
SyntaxInformation[LetL] = {
"ArgumentsPattern" -> {_, _},
"LocalVariables" -> {"Solve", {1, Infinity}}
};
LetL /: (assign : SetDelayed | RuleDelayed)[
lhs_,rhs : HoldPattern[LetL[{__}, _]]
] :=
Block[{With},
Attributes[With] = {HoldAll};
assign[lhs, Evaluate[rhs]]
];
LetL[{}, expr_] := expr;
Block[{With}, Attributes[With] = {HoldAll};


What it does is to first expand into a nested With, and only then allow the expanded construct to evaluate. It also has a special behavior when used on the r.h.s. of function definitions performed with SetDelayed.

I find this macro interesting for many reasons, in particular because it uses a number of interesting techniques together to achieve its goals (UpValues, Block trick, recursion, Hold-attributes and other tools of evaluation control, some interesting pattern-matching constructs).

### Simple usage

First consider simple use cases such as this:

LetL[{a=1,b=a+1,c=a+b+2},{a,b,c}]

{1,2,5}


We can trace the execution to see how LetL expands into nested With:

Trace[LetL[{a=1,b=a+1},{a,b}],_With]

{{{{With[{b=a+1},{a,b}]},With[{a=1},With[{b=a+1},{a,b}]]},
With[{a=1},With[{b=a+1},{a,b}]]},
With[{a=1},With[{b=a+1},{a,b}]],With[{b$=1+1},{1,b$}]}


### Definition-time expansion in function's definitions

When LetL is used to define a function (global rule) via SetDelayed, it expands not at run-time, but at definition-time, having overloaded SetDelayed via UpValues. This is essential to be able to have conditional global rules with variables shared between the body and the condition semantics. For a more detailed discussion of this issue see the linked above answer, here I will just provide an example:

Clear[ff];
ff[x_,y_]:= LetL[{xl=x,yl=y+xl+1},xl^2+yl^2/;(xl+yl<15)];
ff[x_,y_]:=x+y;


We can now check the definitions of ff:

?ff

Globalff
ff[x_,y_]:=With[{xl=x},With[{yl=y+xl+1},xl^2+yl^2/;xl+yl<15]]

ff[x_,y_]:=x+y


Now, here is why it was important to expand at definition time: had LetL always expanded at run time, and the above two definitions would be considered the same by the system during definition time (variable-binding time), because the conditional form of With (also that of Module and Block) is hard-wired into the system; inside any other head, Condition has no special meaning to the system. The above-mentioned answer shows what happens with a version of Let that expands at run time: the second definition simply replaces the first.

### Remarks

I believe that LetL fully implements the semantics of nested With, including conditional rules using With. This is so simply because it always fully expands before execution, as if we wrote those nested With constructs by hand. In this sense, it is closer to true macros, as they are present in e.g. Lisp.

I have used LetL in a lot of my own applications and it never let me down. From my answers on SE, its most notable presence is in this answer, where it is used a lot and those uses illustrate its utility well.

3Out of curiosity, why did you call it "LetL" ? – QuantumDot – 2014-07-26T21:14:14.397

2@QuantumDot The Let part comes from an analogy with Lisp, where there is a similar function. The L stands for Leonid, because originally there was a mathgroup thread with different implementations, and I had to somehow differentiate mine from the others. – Leonid Shifrin – 2014-07-27T10:08:02.653

@QuantumDot I call it WithAll – shrx – 2015-05-19T13:02:31.917

3Why wrap Verbatim around SetDelayed? – bdforbes – 2015-06-23T02:05:34.503

@bdforbes That's a good question. I don't see now, why it is there. But I do recall that it was necessary at some stage. I did have several versions of this function, so presumably that was left from that older code. Unless I am overlooking some subtlety. – Leonid Shifrin – 2015-06-23T04:37:50.637

I can't find a situation where it changes the result. Based on my research, this type of usage of Verbatim is always to prevent evaluation of the head, e.g. as used here: http://mathematica.stackexchange.com/a/2867/8612 In the present context, we wouldn't expect SetDelayed to evaluate in any scenario, is that correct?

– bdforbes – 2015-06-23T04:43:15.087

Potentially, while you were developing the function, you were working on the pattern Verbatim[SetDelayed][lhs_, rhs : HoldPattern[LetL[{__}, _]]] on its own, and thus the Verbatim was actually necessary to keep the expression as a pattern? – bdforbes – 2015-06-23T04:47:31.517

@bdforbes More precisely, Verbatim is used to escape the pattern-matcher. A typical use case would be e.g. MatchQ[_Integer, Verbatim[_Integer]] (gives True), where not using Verbatim would lead to False. If you want to prevent the evaluation of the pattern, you use HoldPattern (which sometimes is also used for escaping, where Verbatim should be used). In any case, I don't see now, why in this particular case Verbatim could've been useful. – Leonid Shifrin – 2015-06-23T04:50:41.830

2I suggest the variant LetL /: (assign : SetDelayed | RuleDelayed)[lhs_, rhs : HoldPattern[LetL[{__}, _]]] := Block[{With}, Attributes[With] = {HoldAll}; assign[lhs, Evaluate[rhs]]];, so that one can also use it with RuleDelayed as in Range[10] /. x_Integer :> LetL[{y = x}, -y /; y <= 5]. – Federico – 2015-08-02T23:32:13.247

@Federico Thanks, good suggestion! Edited this in. – Leonid Shifrin – 2015-08-04T14:57:16.370

Is it possible to combine previously prepared lists with assignments with new variables, something like newfun[$a_,$b_]:=LetL[{a=$a,b=$b}~Join~longHeldListOfSetDelayeds, rv] and then compile newfun? – Åsmund Hj – 2016-03-10T13:32:44.633

@ÅsmundHj Your question is not clear. I won't have the time to answer today anyway. You could ask on the main site as a separate question, to have more eyes on it. – Leonid Shifrin – 2016-03-10T13:44:43.277

– Åsmund Hj – 2016-03-10T14:36:51.237

One tutorial calls it IterateWith, but their implementation with Inactivate is kinda broken, leaking some evaluations of heads: try g = {1};IterateWith[{}, g[[1]] = 1]

– masterxilo – 2016-07-19T14:54:40.823

@masterxilo Yes, I am aware of that. My code also supports shared local variables in definitions, which that one doesn't. – Leonid Shifrin – 2016-07-19T16:23:11.320

I was just bitten by this implementation: Because the rest of WL is not aware of this new scoping construct, it doesn't behave perfectly: With[{data = {a}}, With[{a = data}, data]] and With[{data = {a}}, LetL[{a = data}, data]] give different results. I think I'll start using the extended With suggested below instead. – masterxilo – 2016-08-23T15:44:20.470

1@masterxilo This is a problematic case indeed, and also something that no user-defined implementation can fight, because, as you noted, the outer scoping construct should be aware of the inner one to resolve the naming conflict right. That said, I would still think that this example is a pathology, rather than a typical use case. B.t.w., which extended With did you mean - the new extra built-in capabilities / syntax? – Leonid Shifrin – 2016-08-23T17:48:37.960

46

Introduced in V10.4 or earlier, but after V10.1

This functionality has snuck into With (ref: Daniel's comment). Note the use of the braces.

With[{v1 = #}, {v2 = f[v1]}, g[v1, v2]]
(*  g[#1, f[#1]]  *)


The syntax coloring has not caught up yet:

In V10 --

Needs["GeneralUtilities"];
?GeneralUtilitiesWhere


Where[ass1, ass2, ..., expr] is a version of With that supports multiple sequential assignments.

Needs["GeneralUtilities"];
Where[v1 = #, v2 = f[v1], g[v1, v2]]

(* g[#1, f[#1]] *)

Where[x = 2, t = x^2, Hold[x + t]]

(* Hold[2 + 4] *)


This reminds me of a Q&A I've been meaning to post for years but I never get around to. The basic code behind Where is one of the methods I was using. Maybe I'll finally get around to that soon. (+1 of course.) – Mr.Wizard – 2014-07-15T06:33:43.143

1I can't get Where to work in 10.4... am I alone? – user541686 – 2016-03-26T11:01:23.147

@Mehrdad Likewise; see my answer for the old definition. – Ronald Monson – 2016-04-10T01:33:41.753

1The undocumented With construct seems to be the way to go, so this should be the accepted answer IMO. It is the only variant that behaves correctly like nested Withs in the following case where LetL and Where fail: ClearAll@a; {With[{data = {a}}, With[{a = data}, With[{b = a}, {data, b}]]],With[{data = {a}},With[{a = data}, {b = a}, {data, b}]](*same result, more concise*), With[{data = {a}}, LetL[{a = data, b = a}, {data, b}]](*different result!*), Needs["GeneralUtilities"]; With[{data = {a}}, Where[a = data, b = a, {data, b}]](*crashes because of recursion*) }. – masterxilo – 2016-08-23T15:49:24.867

It looks like Where is more like Block since it can fail from recursions. – masterxilo – 2016-08-23T15:49:46.153

Nevermind about Where, it just isn't here anymore in 10.4, hence the recursion crash. – masterxilo – 2016-08-23T15:53:09.733

@masterxilo Yes, the Where functionality was incorporated into With and removed in V10.4, although it's still undocumented. – Michael E2 – 2016-08-23T16:28:25.427

@MichaelE2 Did you mean **In V10.4 or later**? This syntax does not work in 10.1 for example. – Mr.Wizard – 2018-10-30T02:27:25.963

1@Mr.Wizard I suppose I meant the With syntax worked in V10.4 and may have worked in V10.3, and maybe in V10.2, too. I probably couldn't check versions before V10.4. Daniel's comment was that it was introduced in "V10.something". No one has come along and verified which version was the version in which it was first implemented. – Michael E2 – 2018-10-30T03:23:25.897

Is there a way to fix the syntax highlights for With in v12? I tried SyntaxInformation[With] = ... after Unprotect, but it didn't work. – JEM_Mosig – 2019-12-30T15:06:42.397

@JEM_Mosig This is mentioned in Daniel's comment and it's still not fixed. I don't know of a fix nor why you can't fix it with SyntaxInformation.

– Michael E2 – 2019-12-30T15:18:15.683

28

With works by performing a substitution operation prior to executing its body, and likely it is only a single pass. So, inter-referencing the variables is not possible. Since With accepts the use of SetDelayed (:=), you might think that that could be used, instead. For example,

With[{v1 = #, v2 := f[v1]}, g[v1, v2]]& @ p
(* g[p, f[v1]] *)


which reveals the other use of With: localization. The v1 in f[v1] is not the same as the v1 used by With, so that method is out, also. The same problem exists for Module as it uses a similar form of localization.

However, Block works

Block[{v1 = #, v2 = f[v1]}, g[v1, v2]]& @ p
(* g[p, f[p]] *)


even though the syntax highlighting makes it appear that the v1 inside f is not localized. But, Block has the attribute HoldAll, so v2 = f[v1] is not executed until v1 has taken on its local definition, and, unlike in With and Module, it is not internally treated with Unique[v1].

With the potential effects of these different attributes on the program unknown, I think I'd better just use Module[{v1, v2}, v2=f[v1]; g[v1, v2]]. Thanks! – qazwsx – 2012-09-11T04:39:32.947

6+1 for some good points. However, my comment here is that Block solves an entirely different problem. With is unique in that it creates referentially-transparent code (no assignments to its "variables" in the body of With is possible), while Block allows the body to arbitrarily modify the variables it localizes. Another big difference: (nested) With can inject into held expressions, something Block or Module can not. So, I don't really view a replacement of (nested) With with Block or Module as a generally valid solution to the posed problem. – Leonid Shifrin – 2012-09-11T12:19:20.007

@LeonidShifrin absolutely. With because of its method of operation does things Block cannot do, and vice versa. But, the self-referential part, I thought, was essential to the problem. As long as the other parts of the operation of With are not needed, then Block should be a viable alternative. However, it is not a direct replacement. – rcollyer – 2012-09-11T12:43:30.010

@rcollyer Well, as long as the solution removes some defining properties of With, I would not consider it a general solution for With specifically. I would agree with you more (in that self-referential part is the main issue) if this was a question of nested Block or Module. – Leonid Shifrin – 2012-09-11T12:54:02.820

@LeonidShifrin except that it is well known that you can do Scope[{x,y}, x = f[y]] with either Module or Block, so I don't think that question would come up. Personally, I think the mutability of the variables in Block is the biggest issue with using as a replacement for With. The ability to use With to inject code into held expressions is less well known. But, for the OPs problem, it was the perfect fit, provided of course the representative example doesn't stray to far from the actual code. :) – rcollyer – 2012-09-11T13:09:21.303

@rcollyer I am a bit confused. If it's the case that With's ability to inject values into held expressions isn't that widely known, what do most people use With for? Or is it the case that they simply don't use it at all? (IMO, With should be used with some circumspection, because it can potentially copy each expression many times. So, it is a rather specialized tool, not meant to be used as an ordinary scoping construct.) – Oleksandr R. – 2012-09-11T14:40:38.007

@OleksandrR. I use With primarily to provide temporary named variables, usually in the case of nested pure functions, like With[{f = #}, {#, f[#]}& /@ Range[5]]& @ {x, y, z}, or similar. I avoid held expressions, if I can help it. – rcollyer – 2012-09-11T14:52:13.727

@OleksandrR. I actually use With the most in my code, much more often than Module or Block. Copying does not seem an issue, because I guess With does not perform a deep copy, since expression does not change. – Leonid Shifrin – 2012-09-11T15:45:14.860

@Leonid you are right--an experiment shows that With is actually much more conservative about copying than I had thought. Even though I use With a lot already, it looks like I've been avoiding it needlessly in some cases. Thanks! – Oleksandr R. – 2012-09-11T16:24:52.583

@OleksandrR. Well, good to have another confirmation - I never did specialized tests. I like With (and particularly nested With or my LetL) because they allow to avoid side-effects and they clearly state so from the outset. I am trying to write my code in such a way that one can see at a glance whether or not the body of a given function contains any side effects. I generally try to avoid them. – Leonid Shifrin – 2012-09-11T16:27:32.787

@Leonid I also prefer the side-effect-free style. What I did not expect, though, is that given e.g. data = RandomReal[{-1, 1}, 10^8]; f[tmp2_] := Total[tmp2 + tmp2 + tmp2]; g[data_] := With[{tmp2 = data}, Total[tmp2 + tmp2 + tmp2]]; h[data_] := With[{tmp1 = data}, With[{tmp2 = tmp1}, Total[tmp2 + tmp2 + tmp2]]];, the memory consumption of f[data], g[data], and h[data] is identical. No unnecessary copies occur no matter how deeply With is nested. LetL looks even better in this light: not only clear and robust, but also highly efficient. – Oleksandr R. – 2012-09-11T16:45:06.863

@OleksandrR. As to memory consumption, it seems that this is deeper than just With-related. Szabolcs gave a nice answer, and I added a few further observations in comments to it. Whereas for LetL, it should be almost as fast as nested With entered directly, and also it disappears in function's definitions, since there it expands at definition-time (as a true self-respecting macro should). Glad that you liked it. I use it in my code all the time.

– Leonid Shifrin – 2012-09-11T16:53:54.190

@Leonid I'm not sure that the internal structure is a directed acyclic graph as InternalBags can be self-referential--either bags are just one possible way of building an expression, or a DAG only gets built later on, either during parsing or as part of the evaluation process (so that the evaluator can know when to stop). In any case, I agree that this is a nice answer, and maybe I've been too pessimistic about With, given that it obviously behaves no less well in this respect than any other familiar function. – Oleksandr R. – 2012-09-11T17:44:51.790

@OleksandrR. An interesting point about the Bags, I wasn't aware of that. Still, I tend to think that they are an exception. They are probably as close to the lower-level references as one is allowed to get in Mathematica. My understanding is that the general absence of circular references is an essential feature of mma, which allows it to use a simpler reference - counting scheme in the garbage collector. – Leonid Shifrin – 2012-09-11T17:50:11.277

13

Perhaps this will work:

SetAttributes[BetterWith, HoldAll]
BetterWith[{x_}, expr_] := With[{x}, expr];
BetterWith[{x_, rest__}, expr_] := BetterWith[{x}, BetterWith[{rest}, expr]]
BetterWith[{s:Verbatim[Set][x_List, y_], rest___}, expr_] := Quiet @ With[
{x2 = Replace[Hold[x], z_Symbol :> Pattern[z, _], {2}]},
Replace[y, {Apply[HoldPattern,x2] :> BetterWith[{rest}, expr], _ -> $Failed}] ] BetterWith[{y=x+1,x=1},y] (* gives 2 *)  1Just hide the nested With[] under hood of a new symbol? – qazwsx – 2012-09-11T04:37:46.477 1@MaThEmAtika Exactly. – M.R. – 2012-09-11T04:42:18.853 7 As of 10.4 Where appears to no longer reside in "GeneralUtilities'" however I quite like its form so here is the <= 10.3 Definition (v. similar to M.R.'s answer) that can be placed in an init.m file. SetAttributes[Where, HoldAll]; Where[s : Verbatim[Set][x_List, y_], rest___, expr_] := With[{x2 = Quiet[Replace[Hold[x], z_Symbol :> z_, {2}]]}, Replace[y, {HoldPattern @@ x2 :> Where[rest, expr], _ ->$Failed}]]

Where[expr_] := expr

Where[x_List, expr_] := With[x, expr]

Where[x_, expr_] := With[{x}, expr]

Where[x_, rest__, expr_] := Where[x, Where[rest, expr]]


Note that this slightly improves the previous "GeneralUtilities'" definition by giving effect to a "multiple set" presumably originally intended by the first Where[s:Verbatim[Set] ... definition:

Where[
{x, y} = {1, 3 + x},
z = 2,
{x, y, z}]

(* {1, 3 + x, 2} *)


whereas

In 10.3.1

Needs["GeneralUtilities"];

Where[
{x, y} = {1, 3 + x},
z = 2,
{x, y, z}]

(*
With::lvset: Local variable specification {{x,y}={1,3+x}} contains {x,y}={1,3+x}, which is an assignment to {x,y}; only assignments to symbols are allowed. >>

With[{{x, y} = {1, 3 + x}}, Where[z = 2, {x, y, z}]]

*)


Update - turns out Where can (if rarely) let a few bullets through; To collate:

First, some compression helping:

SetAttributes[CompressCode,HoldAll];
CompressCode[code_] := Compress@Hold@code;
UncompressCode[compressed_] := ReleaseHold@Uncompress@compressed;
UncompressCode[compressed_List] := Scan[UncompressCode, compressed];


i.e. with the compressed strings coming from CompressCode[(* defs *)] with defs being substituted by the code that defines Where, LetL and Let (the latter two due to Leonid)

WhereDef = "1:eJytVNtOhDAQZUFdo9kH/QMTv8A/wFt8MEZ3E/e5SMk2Foq0TcCvt9NSunRhZY0vE2jPnDlnOu1VwpbZLAgCfqTCE6NpNoe/SxXuWF4yWaQPdVlhzgkrshD2FiqssIiFqEgihdo6VivrDVaoecsSU2rAZwZ8jylqcJpFsNbBDQRyXpEQuCq4lqKXjapTFd5xlSBBch4ZruGs2iQA9y1Fxac29Ez4CLzJgh58FwX5ypEwwAsLfJGUrvCXxMXHgH5Iwqpfg+ywuSZi4xru9LXeOPzUN87LmyTKcWTrLHFJkSrcO7LWv+Y5B5Ck2DZ80Py336sTqN7kCaNjCZ6fngOizyv0NLZtDne9DmoE7rgsaaO3wZYV0LakQ5lJs8fjWt5VAm5PsBZ2/YgItfW8wZwF+wdz5GC96h5p+AvpgXM7abgM6X/omnJBDpz1KeKmPBHTxLnbu7DAP95cN3ZbV23PLOoPeCl+APfNJQs=";

LetLDef = "1:eJzNk9tOAkEMhhfxFBGjvoGPgkiiCTGGNXpdoAsTygzOzBJ4ezs7e3APuMErbybtTvu3/ab7MFWTqBMEgTnl41nRPLp03j0fQ7XeqFjOR7uNRmOEkj7SBQwJQQ+IkrQx2nF04q76fIRoB9ZqMY0tZ2X35iLV5yQf2/WxXvPOOXtpYfciI6XXYLlcRTzxhLGFN4kJw1s2BnoRr1Fa8wbWopaVhMB5Z3w8EshV1S3L3bgkNQP6AC1gSmjKYqFLDBVtsfxd5Gx4AiGF3UfdDMk7LHjSJyTY47w8lRdxcNLWzTnbwLgX6RTXbj5yVwxly0ivPLhMrpd2nvp1QdcGLU3j2NU4zXHJIL30uepAXeedoIa3nwmH+BWjnGFjPe+q2aoskRifwi591IHtq6+NQ1FZtkSlLF5bvQJxDid/vdEWKAZWy3CcZIUK5hUaQX07f3J1N8hTVIAU39srdNorLBHmrU98oJWiWgO9TPyYfhv+1z/3a0HQoRU7bsD/sXwtmOub+Nsu5ITyyRPjNSb6BgoRTRw=";



and now some defining:

UncompressCode@{WhereDef, LetLDef, LetDef}


we get the following syntactic variants (ordered by rough chronology) that all do the same thing:

With[{x = 1}, With[{y = x}, {x, y}]]
LetL[{x = 1, y = x}, {x, y}]
Where[x = 1, y = x, {x, y}]
Let[x = 1, y = x, {x, y}]
With[{x = 1}, {y = x}, {x, y}]

(* {1,1}  *)

(* For all of the above*)


Where and Let also permit threaded List Assignments

Where[{x, y} = {1, 1}, {x, y}]
Let[{x, y} = {1, 1}, {x, y}]

(* {1,1}  *)

(* For both*)


while noting users' potential scoping oversights

Where[{x, y} = {1, x}, {x, y}]
Let[{x, y} = {1, x}, {x, y}]

(* {1,x}  *)

(* For both*)


(but which I suspect differ in more involved scenarios that suggest Let is more robust)

I think it is important to have a succinct, natural, robust and efficient version of this scoping construct given its ubiquity. My vote would be for Let based on how it satisfies the first two qualities while seeming to do likewise for the latter two.

4

Let's create nested With with knowledge about RawBoxes. I'd not use this method in this case, due to the performance but it is a good exercise:

SetAttributes[myWith, HoldAll];

myWith[{spec___}, body_] := ToExpression @@ Fold[
RawBoxes @ MakeBoxes[With[{#2}, #]] &,
RawBoxes @ MakeBoxes @ body,
RawBoxes /@ MakeBoxes /@ Unevaluated[{spec}] // Reverse
]


It is handy to nest with RawBoxes but we have to get rid of the very outer RawBoxes at the end. Notice that I'm doing this with ToExpression @@ which also make expression from what is left:

a = b = 4;
myWith[{a = 1, b = a + 1}, {a + b, Hold[a, b]}]
a
b

{3, Hold[1, 2]}
4
4
`