## Enforcing correct variable bindings and avoiding renamings for conflicting variables in nested scoping constructs

47

38

Using global variables the following turns an "expression" into a Function:

expr = 2 x;
Function[x, Evaluate[expr]]


Of course this doesn't work if for some reason a value is assigned to x. Thus, I want to turn expr into a Function inside a Module. But it seems like there is a scoping problem:

Module[{x, expr},
expr = 2 x;
Function[x, Evaluate[expr]]
]

(* Function[x$, 2 x$1713] *)


Clearly, the x used in the first argument of Function is not the same as the x in the second. I played around with Evaluate and even with Replace and Slot, also trying to create a function with an anonymous variable, but didn't succeed. The only way that I managed to get the right result is this one involving strings:

Module[{x, expr},
expr = 2 x;
"(" <> StringReplace[ToString@expr, ToString@x -> "#1"] <> ")&" // ToExpression
]


Is there a simpler, more straight-forward way to achieve this, without ever using any global variables?

1I modified the title because the problem you faced in an instance of a more general and quite important issue. But if you don't like the new one, please feel free to roll back. – Leonid Shifrin – 2013-03-07T00:00:00.147

That's totally fine with me. In fact I'm glad that this question turned out to be of this much interest and spawned such a nice and thorough answer. Thank you! – einbandi – 2013-03-07T00:23:26.233

Related: (10086), (13757)

– Mr.Wizard – 2013-08-22T20:20:40.087

54

### Scoping constructs, lexical scoping and variable renamings

It pays off to understand a bit deeper how the scoping constructs work and what happens behind the scenes when you execute one. In addition to the documentation, this was discussed in part here, but let us present some summary.

When the lexical scoping construct Sc[vars, body] executes (where Sc can stand for constructs such as With,Module, Function), evaluation roughly happens via the following steps (or at least this represents our current understanding of it):

• First, the list of local variables vars is analyzed.
• The body is analyzed, and the presence of inner scoping constructs is tested. This test is performed verbatim, so that, to be detected, some inner scoping construct ScInner has to be present in body in the form ScInner[innerVars, innerBody]. If the inner scoping construct is dynamically generated at run-time (via ScInner @@ ... or otherwise), it is not detected by Sc during this stage.
• If some inner scoping constructs are found where some variables conflict with vars, then Sc renames them. It is important to stress that it is Sc that does these renamings in the inner scoping constructs. Indeed, those are inert during that stage (since Sc has HoldAll attribute and so body is kept unevaluated), so Sc is the only function in a position to do those renamings.
• The actual variable binding happens. The body is searched for instances of vars, and those instances are lexically bound to the variables.
• Depending on the nature of the scoping construct, further actions may be performed. Function does nothing, With performs the replacements of symbols (variables) with their values in body (according to bindings), while Module creates var$xxx variables (according to the bindings, both in the initialization and body), and then performs variables initializations. • The code in body is actually allowed to evaluate. ### How to fool scoping constructs From the above description, it is clear that, if one wants to avoid renamings for some inner lexical scoping construct ScInner[innerVars, innerBody] for whatever reason, one has to dynamically generated this code, so that it is not present inside Sc verbatim. Again, depending on the situation, one may or may not want to evaluate innerVars and innerBody. More often than not one wants to prevent such evaluation, so it is typical to use something like With[{y = boundToZ}, With @@ Hold[{y = z}, boundToZ]]  or With[{y = boundToZ}, Hold[{y = z}, boundToZ] /. Hold -> With]  or anything else that would prevent the innerVars or innerBody from unwanted early evaluation. Here is a more meaningful example. The following is a new scoping construct, which executes some code code with a variable var bind to a current value extracted from a Java iterator object: ClearAll[javaIterate]; SetAttributes[javaIterate, HoldAll]; javaIterate[var_Symbol, code_][iteratorInstance_?JavaObjectQ] := While[iteratorInstance@hasNext[], With @@ Hold[{var = iteratorInstance@next[]}, code] ];  The iteratorInstance is expected to be a Mathematica reference for Java object implementing Iterator interface. The variable var is bound to the current value of the iterator (extracted via iteratorInstance@next[]), using With. This is non-trivial, since we construct this With from pieces, and therefore generate lexical binding of this var to the occurrences of var in code, at every iteration of the While loop. In this case, the outer protecting scoping construct is actually SetDelayed. And we need the construct With @@ Hold[...] to prevent variable var renaming, which is exactly what we don't want here. However, there are cases, where we do want some or all of innerVars or innerBody to evaluate before the binding stage for the inner scoping constructs. The case at hand falls into this category. In such a case, perhaps the most straight-forward way is to use Sc @@ {innerVars, innerBody}, which is what acl did in his answer. ### The case at hand It is now clear why this solution works: Module[{x, expr}, expr = 2 x; Function @@ {x, expr} ]  Function[x$5494, 2 x$5494]  Since there wasn't a line Function[x,...] present verbatim, Module did not detect it. And since we do want the variable and body to evaluate before Function performs the variable bindings, the second version (Function @@ {...}) has been used. You will note that Evaluate is not needed because List is does not have the HoldAll attribute. This specific syntax is not the only approach. For example h[x, expr] /. h -> Function or ReplacePart[{x, expr}, 0 -> Function] would also work because there is not an explicit Function[x, . . .] in the code. It is instructive to realize that this version works too: Module[{x, expr}, expr = 2 x; Function[Evaluate[x], Evaluate[expr]] ]  while Function[...] is present here, the presence of extra Evaluate around x in Function made it impossible for Module to detect the conflict. Of course, there are many other ways one can achieve the same effect. It is now also clear why the following will not work: Module[{x, expr}, expr = 2 x; Function[x, z] /. z -> expr ]  Function[x, 2 x$151]


The point is, the substitution z -> expr happens only at the stage when the body of the Module is evaluated, while the binding stage happens earlier (as described above). During the binding stage, Module detects the name conflict all right, and of course renames. Only then is x converted into a newly created x$151, and only after all that the code inside Module executes - by which time it is too late since the symbols inside Function and expr are different. ### The case of Block Block is a natural approach when guarding against global values, but as Szabolcs comments it must be used with care. Block is not seen a scoping construct for the purposes of the automatic renaming described in the tutorial. You can also see some additional relevant discussion here. Because of that you will not get the "protection" that you may be accustomed to. Using Szabolcs's example: f[z_] := Block[{x, expr}, expr = 2 x + z; Function[x, Evaluate[expr]]] f[x]  Function[x, 3 x]  Note that this function will triple its argument rather than doubling and adding Globalx which may or may not be what you were expecting. Such injection is often very useful but at the same time if you are accustomed to the automatic renaming behavior (even if you are unaware of the mechanism) this may come as a surprise. Re: "Function does nothing" (in your answer). This is only correct for Functions that use Slots. Please consider Function[x, Function[var, x]][2]. – Jacob Akkerboom – 2014-05-27T11:27:28.790 "In general, the Wolfram Language renames the formal parameters in an object like ScInner[vars,body] whenever body is modified in any way by the action of another Sc." Is this a good syntesis now or there are still specific cases to consider? – unlikely – 2016-03-08T01:18:16.880 @unlikely You seem to be quoting http://reference.wolfram.com/language/tutorial/VariablesInPureFunctionsAndRules.html except that you use "ScInner" instead of Function. If one realizes that Function is not the only scoping construct and that renaming will take place elsewhere I think that pretty well summarizes it. – Mr.Wizard – 2016-03-08T15:50:38.907 Yes, I "adapted" this sentence of the tutorial with the same "placeholder" symbols used in this answer because in this form appeared to me as a definitive synthesis but it also appeared (maybe) not completely in agreement with a sentence of this answer, ie. "If some inner scoping constructs are found where some variables conflict with vars, then Sc renames them." This last doesn't suggest that a conflict is searched between two declaration list of two nested scoping construct? – unlikely – 2016-03-09T08:11:27.127 @Leonid Do you have a reply to the comment above? I am not entirely sure I understand it and since you wrote that part (thanks again) I think you're in a better position to reply. – Mr.Wizard – 2016-03-09T16:51:14.000 @Mr.Wizard I think the quoted sentence actually does suggest exactly that "a conflict is searched between two declaration list of two nested scoping construct" - how else would one analyse the conflicts? So there is no contradiction. – Leonid Shifrin – 2016-03-09T16:57:40.550 @Leonid I thought maybe I do not understand unlikely's comment since I do not see the contradiction either. – Mr.Wizard – 2016-03-09T16:58:53.667 5+1. The real reason why the version with Function[x, z] /. z -> expr does not work is that the one we are trying to fool here is not Function, but Module. It is Module that performs the renaming inside Function, and it uses verbatim search for Function[something] in the code of Module. This is why things like Function @@ ... work. I think it is important to realize this. – Leonid Shifrin – 2013-03-06T20:50:10.520 @Leonid That is the point I was trying to make but I guess I didn't say it very well. I'll try again. – Mr.Wizard – 2013-03-06T20:56:37.677 I think it is still not clear enough. Do you mind if I contribute to your answer? – Leonid Shifrin – 2013-03-06T21:02:23.597 @LeonidShifrin Of course I don't mind. :-) – Mr.Wizard – 2013-03-06T21:08:21.613 @Leonid So what Module does is actually scan for Function, With, etc. not replace anything inside those? There isn't anything more complicated involved? – Szabolcs – 2013-03-06T21:23:18.530 @Szabolcs I did not really study the internal implementation, so all I can say about this has a status of an educated guess. I tried to describe this mechanism above, in my edit - see if you find this a satisfactory answer to your question. – Leonid Shifrin – 2013-03-06T21:57:06.460 That's quite the contribution @Leonid. :^) – Mr.Wizard – 2013-03-06T22:04:15.780 I hope it did not insult you that I changed your original post considerably. Please feel free to roll back if you don't like it. – Leonid Shifrin – 2013-03-06T22:05:16.093 @Leonid I didn't mean that at all. Thanks for the comprehensive addition! – Mr.Wizard – 2013-03-06T22:33:41.107 I am still not clear about what constitutes a conflict of variables. For example why Module[{body}, Function[x, body] /. body -> 2 x] and Module[{ }, Function[x, body] /. body -> 2 x] give different answers? It seems that in the first one, Module renames x, as it detects body inside Function, and this doesn't occur in the second one. – ecoxlinux – 2013-03-06T22:52:47.673 @ecoxlinux These renamings are excessive, it seems - and it is also mentioned in the docs. So, yes, it sees the conflict in the body but does renamings of function's variables. – Leonid Shifrin – 2013-03-06T23:18:16.330 @Mr.Wizard No problem. This is an important topic, so sooner or later we had to cover it somewhere. – Leonid Shifrin – 2013-03-06T23:19:14.737 @Leonid Thanks. – ecoxlinux – 2013-03-06T23:40:38.327 @Leonid, MrW nice answer; I don't think here's the best place to post it for visibility though. It is likely to be missed by people interested in this broad discussion. But I am not sure how to fix that (except by us linking to it in the future). – acl – 2013-03-06T23:51:28.847 @acl you could edit the question, or at least the title, to make some recognizable reference to the automatic renaming in nested scoping constructs. – Mr.Wizard – 2013-03-06T23:53:37.917 @Mr.Wizard I have been thinking about this for the past few minutes but don't see any lightweight modification that does the job. If you do, please change it (also @Leonid) – acl – 2013-03-06T23:57:10.367 1@acl See if you like the new title I just made for it. – Leonid Shifrin – 2013-03-06T23:59:04.017 @Leonid perfect – acl – 2013-03-06T23:59:19.303 3You've been Leonified! – Rojo – 2013-03-07T05:05:43.453 @LeonidShifrin Going back to the case of Function, I think the description of variable renaming doesn't always apply. Take for example Module[{z},Function[z,0]]. Nothing is renamed at all. But then add Module[{z$,z},Function[z,0]] and you get an error. – Jens – 2013-03-10T23:06:24.983

The problem with z$ is that it should count as a different variable since that's the replacement you get when doing this: Module[{z, expr}, expr = 1; Function[z, expr]]. Note here that the replacement is made from z to z$ in Function, whereas expr is actually not replaced at all again (it's the same name inside and outside Function). – Jens – 2013-03-10T23:29:38.323

@Jens Yes, the first case I actually noticed, but was too lazy to correct my wording. Basically, my main point was that the decisions about renamings are made by Module. In the case Module[{z},Function[z,0]], it was smart enough to realize that simply keeping x in Function would be enough to avoid cinflicts. As to the second case, this seems to be a general problem with Module (and more generally, the emulation of lexical scoping in Mathematica): take Module[{z\$, z}, 1], and you get the same. I wasn't aware of this particular issue, but I am not too surprised either. – Leonid Shifrin – 2013-03-10T23:36:34.543

@Jens So, thanks for the input, in particular it shows that some people read what I write, carefully enough :). I will make an edit and make my statements more precise, some time soon. – Leonid Shifrin – 2013-03-10T23:38:19.160

14

I think this should be OK

Module[{x, expr},
expr = 2 x;
Function @@ {x, expr}
]


(because {x,2*expr} gets evaluated before Function replaces the List head)

Thanks! This is clean and easy. I knew I was missing something here... – einbandi – 2013-03-06T18:42:16.873

4

You can also just replace Module with Block:

Block[{x, expr},
expr = 2 x;
Function[x, Evaluate[expr]]]


5Perhaps it's good to mention that the usual caveat applies: if this is used in a function in a way similar to f[z_] := Block[{x, expr}, expr = 2 x + z; Function[x, Evaluate[expr]]], then f[x] will give an incorrect result. It's necessary to make sure that the function doesn't take symbolic arguments that are used together with the Block-variables. @acl's solution doesn't suffer from this problem. – Szabolcs – 2013-03-06T19:33:13.697

Right. Although in some cases that may be also what you want to achieve when evaluating f[x] (i.e., treating x as the function variable). – ecoxlinux – 2013-03-06T19:45:28.297

2

To inject expressions into a function, I use one of the following:

f1 = With[{expr = 2 x},
Function @@ {x, expr}];

f1[5]

(* ==> 10 *)

f2 = Function[x, body] /. body -> (2 x);

f2[5]

(* ==> 10 *)


Modification in light of the comments

The first approach works fine inside Module, the second doesn't, as mentioned by Szabolcs. To fix this, one can do a more comprehensive replacement like this:

Module[{x},
f3 = Function[x1, body] /. {x1 -> x, body -> (2 x)};
]

f3[3]

(* ==> 6 *)


1Warning! Module[{x, expr, body}, expr = 2 x; Function[x, body] /. body -> expr ] won't make a correct Function. (it was also what I tried) – Szabolcs – 2013-03-06T20:42:16.007

@Szabolcs Yes; I tried to address this in my answer below. – Mr.Wizard – 2013-03-06T20:43:08.063

@Mr.Wizard I only got to reading it now :) – Szabolcs – 2013-03-06T20:47:30.683

@Szabolcs It is Module that we are trying to fool here, not Function, thus this behavior - see my comment below Mr.Wizard's answer. – Leonid Shifrin – 2013-03-06T20:51:45.553

@Szabolcs Thanks for pointing that out - I was in a hurry and overlooked how important the Module part of the question was... – Jens – 2013-03-06T23:26:58.417

0

You can also use a placeholder wrapper symbol, like so:

Module[{x, expr, f},
expr = 2 x;
Attributes[f] = {HoldAll};
f[x, Evaluate[expr]] /. f -> Function]
`

Mathematica provides 1001 ways to skin this particular cat, but this one has been useful to me on some occasions.