What are all the "magic" Symbols in the Mathematica language?

42

22

Leonid Shifrin once wrote (excerpted):

(Unevaluated) is one of a very few "magic symbols", along with Sequence and Evaluate - these are deeply wired into the system and can not be easily replicated or blocked, unlike Hold - in that sense, Unevaluated is more fundamental.

HoldPattern is a normal (usual) head with HoldAll attribute for the purposes of evaluation, but its magic shows in the pattern-matching: it is invisible to the pattern-matcher, and is very important ingredient of the language since it allows pattern-matcher to be consistent with the evaluation process.

David B. Wagner writes:

The symbols Evaluate, Unevaluated, and Sequence are magic cookies. This is a whimsical term used by computer scientists to refer to any type of value that has special significance to the software system of which it is a part. The behavior of these symbols is not the result of any values or attributes they possess; rather, it is "wired into" the kernel. This implies these behaviors simply cannot be altered, nor can they be duplicated.

These are fundamental to the language but they are not widely recognized as such. I have never attempted to make an exhaustive list of these "magic Symbols" but I think it would be useful to the community to have one.

What are all the "magic" Symbols in the Mathematica language?

What makes each of these Symbols special?

  • I believe these should be considered distinct from atomic objects, which while also being "deeply wired into the system" and "not easily replicated" are a form of data types with special handling.

A related question with a somewhat different scope:

Mr.Wizard

Posted 2016-08-02T09:44:37.597

Reputation: 259 163

ToExpression will definitely be one. – Wjx – 2016-08-02T10:27:48.550

@Wjx I would argue that it is not, at least by my understanding of the way others are using this therm. ToExpression does not seem to involve any special evaluation. That it performs I complex task does not make it "magic" since conceivably we could reimplement it ourselves. – Mr.Wizard – 2016-08-02T10:34:22.230

May I ask how? It seems that it's hard to convert an string to a complex expression? – Wjx – 2016-08-02T11:04:24.973

@Wjx A lot of hard work! I mean that with enough effort one could write out all the parsing rules and create an expression assembler. Leonid has actually done work in this area I believe. In no way do I mean to suggest that the function is trivial; far from it! However I am saying that it does not require anything that does not already exist in the language, as far as I know. By contrast we cannot write Unevaluated robustly ourselves. – Mr.Wizard – 2016-08-02T11:20:25.907

3Oh, gotcha! You mean it's writable, but in contrast, Unevaluate or so is simply CANNOT be written directly. – Wjx – 2016-08-02T11:25:58.177

Three are things that appear as magic cookies to the notebook user because the front-end transforms their input strings into something strange before it is seen by an evaluator (or maybe it is never seen by an evaluator). How do these fit into your question? – m_goldberg – 2016-08-02T17:07:16.457

3Would you consider things like Catch/Throw or Check/CheckAbort/CheckAll to fall into the "magic" category? They pretty much need special kernel hooks to function correctly, but it feels like a different level of magic than Unevaluated etc. – WReach – 2016-08-02T17:29:20.187

@WReach Thanks for the comment. I think those things should be mensioned here in their own category, like the pattern object "meta-expressions." I'll add that to my answer later, unless you would like to post separately about it. This question can be adjusted as necessary. I think all of this information is worth collecting. – Mr.Wizard – 2016-08-02T23:46:46.160

3

@WReach - I think Catch and Throw could be written - they're non trivial and would involve Catch rewriting the entire expression given as its first argument - but LISP macros are written do that kind of thing a lot and I don't think there's anything special about them here. I believe the Check family can be written as well. (For the curious, Let Over Lambda is probably the best book on LISP macros ... ever!)

– davidbak – 2016-08-03T00:14:46.020

1@davidbak It is exactly this kind of discussion/information that I hoped this question would inspire. Thank you. (I know this site isn't supposed to be for discussions per se, but discussion for the purpose of developing questions or answers is always welcome.) It sounds a little strange but it might be good to start an answer to list things that are not "magic" Symbols though they at first appear so, with explanations. – Mr.Wizard – 2016-08-03T05:21:25.903

How about operators like << and >>? – Brett Champion – 2016-08-03T19:20:14.437

Sooo... what's your definition of "magic" here, @Mr.Wizard? It is entirely not clear to me, and I couldn't grasp the essence - other than the ostensive definition you provide by picking certain functions that you find "magical". Could you perhaps provide a more solid organizing principle? Is it about being written entirely in C (irreproducible in Mathematica)? Or about functions controlling evaluation? – István Zachar – 2016-08-03T19:23:33.273

@István I am not quite sure myself! That's one of the things I am trying to figure out. Leonid Shifrin, David B. Wagner, and Robby Villegas all use the term; I did not create it. If the only "magic Symbols" are "Evaluate, Unevaluated, and Sequence" then I want to know why certain other symbols are not. (I think Szabolcs makes a convincing case that Function is also a "magic Symbol" for example.) I also want to identify related classes like pattern "meta-expressions" which have certain "magic" attributes yet are distinct. – Mr.Wizard – 2016-08-04T02:38:43.593

Answers

24

uh oh, another "work in progress" answer from me that may never be finished.


  • Unevaluated

    Robby Villegas, Working with Unevaluated Expressions:

    Unevaluated is not meant to be a function or stable data type. It is to be used as a wrapper on an argument in stage 1, before argument evaluation. It is a signal to the evaluator to suppress the usual evaluation of that argument.

    It is transparent to the function receiving the argument. You can think of it as a shuttle giving the argument safe transport to the function's code, keeping the evaluator away. Unevaluated vanishes before the argument is fed to the function, since its purpose is fulfilled.

  • Evaluate

  • Sequence

  • Function

    As Szabolcs wrote:

    I'll also note that Function is a very special symbol that has very special evaluation rules: the evaluator will hold y in an expression of the form f[x][y] only when f is Function (and in no other situation). In particular, it is not possible to bestow a custom symbol with this property.

  • Pattern elements

    Arguably a distinct class, these Symbols are "magic" only in pattern matching and not main evaluation itself. Robby Villegas describes these as having "meta-expression status."

    Via a method from jkuczm, May 8 '15

    {Alternatives, Blank, BlankNullSequence, BlankSequence, Condition, Except,
    HoldPattern, IgnoringInactive, Literal, Longest, Optional, OptionsPattern,
    OrderlessPatternSequence, Pattern, PatternSequence, PatternTest, Repeated,
    RepeatedNull, Shortest, Verbatim}
    
  • OptionsPattern / OptionValue

    Some nonstandard evaluation appears to take place here.

undetermined

These might have limited "magic" behavior in one form or another.

Mr.Wizard

Posted 2016-08-02T09:44:37.597

Reputation: 259 163

4There is also Nothing. It is like Sequence[], but it only gets removed from lists. – Szabolcs – 2016-08-02T09:51:11.290

@Szabolcs Unfortunately it is not present in the version I use therefore I cannot examine its behavior, or definition if such exists. Have you looked into its mechanism at all? – Mr.Wizard – 2016-08-02T09:52:52.540

2@Szabolcs What exactly is magical there? Common use case can be coded e.g.: not /: {a___, not, b___} := {a, b} – Kuba – 2016-08-02T09:53:43.880

@Kuba Okay, I just reviewed that section. I don't see anything that really fits the description of "magic" quoted in the question, e.g. While should be easy to emulate. Trace does things "we" cannot but I don't think it really fits here, though I forgot about it or I might have grouped it with Stack. – Mr.Wizard – 2016-08-02T10:07:37.743

Maybe it isn't special then. – Szabolcs – 2016-08-02T10:12:09.147

$ConditionHold probably falls into the undetermined category, right next to RuleCondition. – WReach – 2016-08-02T15:57:52.867

MatrixForm doesn't seem that special, but I couldn't work out how to replicate its behaviour. – mikado – 2016-08-02T19:43:36.173

@mikado MatrixForm basically is a formatting rule that creates a certain Box form (GridBox I believe, without checking). For the purpose of this question I am am not considering special Front End box forms and behaviors "magic" -- I should probably clarify that. – Mr.Wizard – 2016-08-02T23:48:37.160

Why is OrderlessPatternSequence in blue above? It's a built-in symbol. Someone needs to update the Mathematica keywords on this site. Who to contact? – QuantumDot – 2016-08-04T09:47:58.420

@Quantum, admittedly it hasn't been updated in a while.

– J. M.'s ennui – 2016-08-08T14:14:49.037

12

[This response is intended as a "big list" contribution rather than a comprehensive treatment.]

Program Control Symbols

If we define "magical symbol" to mean a symbol whose implementation cannot reasonably be achieved in user code, then the various program control symbols would almost certainly qualify.

Language Features:

While it is possible in principle to implement the language features in user code through clever code transformation, the effort amounts to replacing the built-in evaluator completely. The reason is that Mathematica supports evaluation of code generated at run-time. Static analysis of the code is not sufficient to identify all evaluatable code -- the program must actually be run. Thus, the code transformer would have to take over the full evaluation process in order to accomplish its task. The implementation effort would be larger than writing the built-in evaluator in the first place -- and the program control symbols would be "magical" in the rewritten evaluator.

Even Common Lisp, with far more well-defined evaluation semantics than Mathematica, was forced to provide throw, catch, go and unwind-protect as special forms for the same reasons.

Some function interact with the evaluator:

These are somewhat less magical than the language features, but they do require access to internal state of the evaluation implementation.

WReach

Posted 2016-08-02T09:44:37.597

Reputation: 62 787

I believe your description of Common Lisp's implementation of throw and catch is incorrect. They are "special forms" because catch has an implicit progn. "Special form", in Common Lisp, does mean "special evaluation" but in this case it is no more than not evaluating all arguments before the call. (In other words, it is the HoldAll attribute.) It could have been provided as a macro. (And perhaps is, in some implementations.) The reason these things are harder in Mathematica than Common Lisp (or Scheme) is that the Mathematica evaluator has a lot more kinds of "special form". – davidbak – 2016-08-04T02:56:13.807

@davidbak I think if it were possible to implement catch as a macro, it would have been. That was a driving goal in the design of Common Lisp: to reduce the set of special forms to the absolute minimum possible. – WReach – 2016-08-04T14:11:44.983