## Context of localised (dynamic) symbols

13

6

When you localise a symbol with Module, the created unique symbol is usually in the current context. There seems to be an exception to this which has further consequences.

Let us assume we define a symbol in a package. We can mimic this by defining a symbol in some context and append this context to the $ContextPath: AppendTo[$ContextPath, "MyContext"]
MyContextx = "Package Variable";


If you look at ??x you'll see that we really have only one x in MyContext. What is surprising at first glance is the following:

Module[{x},
Information[x]
]


While you might be tempted to say "who cares?", this has consequences! Without evaluating the following, try to guess the output:

Module[{x = "Local Variable"},
{x, MyContextx}
]


The output is

{"Local Variable", "Local Variable"}

There are further consequences: The standard context of DynamicModule variables is $CellContext. This means, even when you have defined some variable in Global context, if you use the same name inside DynamicModule, it comes into the $CellContext context. With our MyContextx this is different.

Evaluate this

Dynamic[MyContextx]


and use this minimal dynamic

DynamicModule[{x = .5},
{Slider[Dynamic[MyContextx]], Dynamic[MyContextx]}
]


Again, try to guess what you expect when you move the slider. After that, just change the local variable to

DynamicModule[{y = .5},
{Slider[Dynamic[MyContextx]], Dynamic[MyContextx]}
]


The important part is that in the first place, the DynamicModuleBox defines (look at the cell expression!)

DynamicModuleBox[{MyContextx$$= ....  while in the second example it is CellContext as expected. Question: Can anyone explain why I should regard this as good language design or as intuitive? Wasn't it possible to solve this more carefully? Can anyone offer an explanation why I shouldn't care about this (beside the fact that it obviously works for a large code-base)? Anyone who is further interested can read the chat log, because it was Kuba who brought the issue up. One reason might be that, given the Module approach to lexical scoping (symbol generation), and that these symbols may, under certain conditions, leak (persist after the execution leaves Module), it may be a good idea for them to at least leak into whatever context was there when they were constructed. If we think about it, once we state that Module implements lexical scoping by symbol generation, the currently existing scheme is the simplest possible. – Leonid Shifrin – 2015-02-18T16:52:37.783 My assumption (haven't added it to the question) was similar. Since Module[{x},..] creates an entry in the symbol table for x even if it uses then x123, it would lead to name clashes when it creates it in the global context and another x lives in another accessible context. – halirutan – 2015-02-18T16:55:24.170 1The fact that you use a variable with a long (full context) name does not prevent the variable capture in this case. Which is another good reason to use Private subcontexts for implementations (where the code with Module will sit). In that case, the generated variables will live in those private sub-contexts, and such annoying variable captures are much less likely. – Leonid Shifrin – 2015-02-18T16:56:06.210 In a sense, once symbol-generation + symbol renaming is chosen as a mechanism to implement lexical scoping, there aren't many natural choices left for the rest of it. I'd say, the current behavior is rather natural from this point of view. – Leonid Shifrin – 2015-02-18T16:57:11.943 @LeonidShifrin One question that persists is: Why isn't DynamicModule forcing every symbol into the CellContex? This context is not in the search path and name clashes are not possible. I mean, it does it for all Global symbols. Why not for the rest? – halirutan – 2015-02-18T17:01:25.950 I haven't tested this now (although I did before IIRC), but my guess would be that the distinction is not whether the symbol is in Global or not, but whether the symbol is referenced by its short name or not. And it probably has to do with conversion to boxes. When short names are used, they are converted to short string names when box conversion happend. My guess is that FE code assumes then that those are to be dynamic FE variables, and converts them accordingly. But when fully qualified symbols are converted, they result in long string names, and FE treats them as kernel variables. – Leonid Shifrin – 2015-02-18T17:07:49.153 Why the design here is as it is, I can't answer, but I've found this behavior useful on many occasions. – Leonid Shifrin – 2015-02-18T17:09:04.033 1 Somewhat related to my last comments. – Leonid Shifrin – 2015-02-18T17:13:09.263 1To correct my statement a bit: fully-qualified names are only converted to fully-qualified strings if the context of the symbol is not on the ContextPath - otherwise they are still converted as short names (strings). Conversely, if you have a symbol and pass it to the code which converts it to a string in an environment where the symbol's context is not on the ContextPath, the result would be a fully-qualified string name even if initially you referred to the symbol by its short name (the fact that I exploited in my answer in the linked question). – Leonid Shifrin – 2015-02-18T17:22:14.813 Related: (8295) -- the Module example at least is explained therein. – Mr.Wizard – 2015-02-18T18:55:04.620 @Mr.Wizard I have linked this very article in the beginning. Especially because of your first sentence "Symbols are created in the current context" which is not quite correct if I understand it as you intended it. That's why I almost quoted you. – halirutan – 2015-02-18T19:41:00.983 oops; right you are. Okay, I'll return to this later. Good question, by the way; thanks for it. – Mr.Wizard – 2015-02-18T19:46:30.657 ## Answers 16 I will make no attempt to defend the fact that Mathematica simulates scoping by means of variable renaming. However, the behaviour that we see is consistent with the principles under which Mathematica does operate. Whenever Mathematica tries to interpret a symbol name, it first checks to see whether a symbol with that name already exists in a package in the context path. If it does, that symbol is used. If not, a new symbol is created. So, when presented with the expression: Module[{x}, Information[x]]  The system will interpret it as if we wrote: Module[{MyContextx}, Information[MyContextx]]  When written in full this way, it is easy to see why we get the result we see -- especially when we take into account the fact that scoping is simulated by variable renaming. But what about the alternative? What if the input was interpreted as: Module[{Globalx}, Information[Globalx]]  This would mean that the symbol Globalx would be created. This would immediately shadow the definition from MyContextx, rendering it inaccessible in unqualified form. It would also place in doubt the meaning of any subsequent unqualified references to x. This behaviour would be much more surprising and irritating than the present behaviour. The present behaviour is confined, at least, to some uncommon corner cases. The consequences of the other examples are also easier to see when they are written out in full after symbol resolution (and taking the scope renaming rule into account): Module[{MyContextx = "Local Variable"}, {MyContextx, MyContextx}] DynamicModule[{MyContextx = .5}, {Slider[Dynamic[MyContextx]], Dynamic[MyContextx]}]  It is for these very reasons that the Mathematica naming recommendations are what they are. Exported names are expected to be long and descriptive to maximize the chance of global uniqueness. Global variables are expected to start with a dollar sign. It is considered bad practice to export short names that start with lower case letters (especially ones like x and y) because the chance of collision is near certainty. Short, common, names are expected to remain private to a package. Good language design? Not so sure. But we must remember that Mathematica is focused squarely upon symbolic manipulation. It tends to avoid prejudging the role that a symbol will play in an expression: is it a variable? a function? inert data? For expediency, the designers have chosen a middle ground between complete neutrality when interpreting expressions and completely conventional scoping and symbol interpretation. That middle ground attempts to give us the power of symbolic manipulation while still supporting some of the programming paradigms we use in other languages. I think that the present behaviour under discussion, as well as the various scope leaks etc, are inevitable consequences of the seam between these two extremes. Intuitive? I do not think that this behaviour can be considered intuitive when coming from another system. But I think that one can build a pretty good intuition over time provided one is willing to accept the heavily symbolic paradigm that Mathematica promotes. For the case in hand, this is where our intuition needs to expressly take on the concept that symbols are only created when they do not exist in a ContextPath package. We also need to take on the fact that scoping is simulated by variable renaming. Once we accept these ideas (perhaps grudgingly), our intuition can develop. DynamicModule Let's now examine the DynamicModule cases. It was observed that the expressions Dynamic[{x = .5}, ...] and Dynamic[{y = .5}, ...] behaved differently. Once we take into account the symbol resolution logic discussed above, we note that the first dynamic module is being asked to localize MyContextx. Thus, the internal references to that symbol will be replaced at execution time by localized symbols. That is what we see: when the slider is dragged, the outer Dynamic[MyContextx] does not change. In contrast, the second dynamic module is localizing Globaly. This is irrelevant. What is important is that MyContextx is not being localized. That is why the outer dynamic changes when we drag the slider. Even though there is no apparent scope leak in this case, there remains a worry stemming from the observation that the first dynamic module used the variable placeholder name MyContextx$$ whereas the second used a placeholder named $CellContexty$$. Can the absence of CellContext from the former placeholder lead to a scope leak at some point? The answer is "no". The key word in the preceding paragraph is placeholder. The symbols used in the cell expression are not the symbols that will be used at execution time. We can observe this by evaluating the following expression: DynamicModule[{v = 1}, { Dynamic[v], Dynamic[Unevaluated@v] }] (* {1, FEv$$21} *)  The cell expression of the result looks like this: Cell[BoxData[ DynamicModuleBox[{$CellContextv$$= 1}, ... DynamicBox[ToBoxes[CellContextv$$, StandardForm],
...
DynamicBox[ToBoxes[Unevaluated[$CellContextv$$], StandardForm], ...  Notice how the return value used the fully localized symbol FEv$$21 whereas the cell expression uses the placeholder $CellContextv$$. What exactly is a placeholder? It is best to think of a dynamic module as template from which user interface widgets can be generated. An important design goal of dynamic modules is that the user must be able to copy-and-paste a widget to obtain a completely detached and fully functional copy of that widget. Localization is a critically important factor that makes this possible. Placeholders are used for (at least) two purposes. First, they show where localized variables need to be inserted into the form. Second, they act as keys to hold the persistent values of the dynamic variables between front-end sessions. In these roles, the exhibited localization is adequate. It remains a mystery to me as to why the placeholders of the form Globalx get changed into CellContextx. It appears to serve some arcane internal purpose -- largely unrelated to the final execution scope. [Update: @Kuba points out that there is information about CellContext here]. The life-cycle of a dynamic module is actually a complicated business. There are multiple evaluations, performed in at least five distinct scopes. In interest of keeping this post merely unreasonably long, I shall only give a brief glimpse of the process. From a fresh session (both front-end and kernel), evaluate this expression: AppendTo[ContextPath, "MyContext"]; MyContextx = "Package Variable";  Then, as a separate action, evaluate this ugly piece of work: DynamicModule[{MyContextx = 0, y = 1}, Print @ Unevaluated @ {MyContextx, y} ; { Unevaluated @ {MyContextx, y} , Dynamic @ Unevaluated @ {MyContextx, y} , Button["Wormhole" , Print @ DynamicModule[{} , {Dynamic @ Unevaluated @ MyContextx, Dynamic @ MyContextx} , InheritScope -> True ] ] , Slider[Dynamic[MyContextx]] , Dynamic[MyContextx] } ] (* {x898, y898} {Unevaluated[{x$$, y$$}], {FEMyContextx$$25, FEy$$25}, <<button>>, <<slider>>} *)  Including our original variables, I count four versions of each of our x and y variables. If we examine the cell expression, we find the placeholders MyContextx$$ and \$CellContexty$$. That makes five versions. Each of these versions plays a role in the various evaluation stages of the dynamic module. When the dust clears at the end, the variables are pretty damn localized. Note how the final x and y symbols were successfully localized irrespective of the presence or absence of CellContext. There may be some bugs in this process, but they will not be from lack of trying. And if that were not enough, press the Wormhole button that is created, to produce this output: (* {FEMyContextx$$25, 0} *)


Aha! One might think that we can sneak into the dynamic module's scope like this:

Dynamic[FEMyContextx$$25]  If we actually try it, it does not work. Why? The answer lies in comparing the cell generated by the Wormhole button to our hand-coded Dynamic. Our hand-coded cell lacks the generated DynamicModuleBox option DynamicModuleParent->BoxObject[7345]. Without it, our evaluations do not receive notifications when the kernel variable changes. Another topic that this post will pass over. Reproducing These Results For those excited at the prospect of experimenting in this space, a few words of advice: • Remember that symbols are created after an expression is read but before it is evaluated. • Remember that active dynamics can generate symbols and perform evaluations merely by becoming visible (especially when a notebook is first opened). Failure to take these points into careful consideration will result in symbols appearing in unexpected contexts. How Many Scope Strategies Are There? If you have been keeping score, there are a lot of different scoping strategies touched upon in this post, and even more in Mathematica as a whole. Here is an incomplete list to give a taste: • module-style renaming: x123 • function-style renaming: x • placeholder-style renaming: CellContextx$$ or MyContextx$$ • wormhole-style renaming: FEx$$1234 or FEMyContextx$$1234 • block scope: first class dynamic scoping • compiled scope: first class traditional scoping (mostly) • pattern scope: first class traditional scoping?? Conclusion To conclude, I observe that most of the time we can blithely ignore these issues. This speaks to the fact that the designers seem to have done a pretty good job papering over the boundary between pure symbolic and "conventional" programming. But the boundary exists, and when we stumble over it the failure tends to be spectacular. We must defend ourselves by building our intuition about how Mathematica handles these cases. Only then do we stand a chance to foresee/avoid or diagnose/fix such problems (which unfortunately tend to arise at run-time, not design-time). (For the record, I am still of the opinion that none of this argumentation precludes the desirability of first-class module-like scoping facilities within in the language.) I realize that there is still more to be said about the double-evaluation and double-scoping of DynamicModule, but I've run out of time to extend this post at the moment :) I figured that I would post it anyway since it is too long for comments. – WReach – 2015-02-18T17:56:25.907 +1. This is very close to what I had in my mind when seeing this question and thinking about it, but I wouldn't be able to express this so eloquently. – Leonid Shifrin – 2015-02-18T18:05:01.813 Even without a discussion of DynamicModule this answer is very well written. Thanks. If you find the time to include your opinion about the statement in this comment, it would be very nice. – halirutan – 2015-02-18T19:55:49.603 1Thanks, really nice explanation. By "Mathematica naming recommendations" you mean any particular part of documentation/tutorials? I'm asking because it really grid my gears that I work with MMA, readTFM and still am surprised once a week. Maybe it's because I have no IT background but isn't MMA advertised as designed fur such folks? – Kuba – 2015-02-18T19:58:19.647 2@halirutan I took a crack at describing the DynamicModule behaviour. Now I need to lie down :) – WReach – 2015-02-19T03:13:45.200 @Kuba I don't know of any section in the documentation that pulls all of the recommendations together. There a bits and pieces scattered about in the sections about variables, packages, and the internals of Mathematica. I can also recall some rather lengthy essays and presentations by Stephen Wolfram on the subject. So I guess my so-called recommendations are actually just me trying to summarize my understanding of de facto established practice. – WReach – 2015-02-19T03:16:51.083 I thought it may be so :), thanks anyway. About DM/CellContext part. You can cut some parts, move them and link readers here: How does CellContext work?, it may fit there better, what do you think? – Kuba – 2015-02-19T07:32:41.840 @Kuba Thanks for that link -- I had not seen that question. I have added a reference to it at the part in which I said "I dunno what CellContext does". I'm not sure that I can add to the discussion to be found at that question. – WReach – 2015-02-19T12:31:23.943 I've lately forced myself to understand what I'm doing and I have two notes so far: "Globalx get changed into CellContextx" - I believe one of the reasons why it is done is to make users life easier. Especially users who work interactively/manually with notebooks. E.g. Button["foo", foo[] where foo is not scoped will remain a valid gui element even if you switch notebook's context to local and change foo - no need to reevaluate DynamicModule. – Kuba – 2016-09-01T22:02:24.697 start="2"> • "Dynamic[FEMyContextx$$25] -If we actually try it, it does not work." - not true. The only reason is this major bug: 100828. So, with fixes described there evaluate: DynamicModule[{MyContextx = 0}, {Dynamic@Unevaluated@{MyContextx},
• Slider[Dynamic[First@{MyContextx}, (MyContextx = #) &]], Dynamic[{MyContextx}]}], note the number and createSlider@Dynamic@FEMyContextx$$number. Works well, will stop as soon as kernel or notebook is closed and reopened but that is expected. – Kuba – 2016-09-01T22:05:30.797 otoh DynamicModuleParent is needed to parse Slider@Dynamic@MyContextx$$ correctly. – Kuba – 2016-09-02T05:32:28.003

+1 @WReach: you say 'the failure tends to be spectacular.' So would you say that this precludes MMA from being a serious language with which 'production systems' might be written? Has Wolfram built an increasingly tall and unstable Jenga tower where random leaks, shadowings and name conflicts can occur at any time? Have they built a system so spaghetti-ish and complex that even they don't understand it? – berniethejet – 2019-01-12T17:41:42.623

1@berniethejet I think WL production systems require a different mindset than more mainstream languages. Wolfram Alpha is reputed to be mostly pure WL. WL feels to me more optimized for improvisational programming by an individual. The forthcoming compiler developments look like they might broaden the scope for more conventional deployment. I'm reminded of issues associated with the symbolic and dynamic capabilities of early LISP. It took nothing less than a rework of language semantics before compiled LISP applications could fill the same roles as, say, C applications. – WReach – 2019-01-12T19:53:42.003

@WReach, thanks, that is very interesting, I didn't know any of that about LISP. I don't wonder whether Alpha isn't relaunching an instance of WL for every Alpha session, or every query even. Certainly I have never been able to keep MMA up and running for more than a few days without a crash in the stuff I do. – berniethejet – 2019-01-14T01:47:04.977