Making a symbol's new definitions be tried before all previously defined ones



Is there any way to "close" a package (or a symbol, or a context) in that if a user of the package adds definitions to the symbol they will be tried before the package defined ones, just like what happens with built-ins?


Posted 2012-03-10T22:16:51.110

Reputation: 40 993


This is a very important question. This will allow users to wrap package functions. Furthermore, certain "builtins", such as all Parallel* functions, are actually implemented as a package!!

– Szabolcs – 2012-03-14T19:37:53.557

1@Szabolcs, but in those cases their functions don't work "as regular built-ins". Try adding ParallelSubmit[___]:=8 after unprotecting... It gets added at the bottom of the DownValue list and ParallelSubmit[2+2] works "as usual" instead of catching the new definition – Rojo – 2012-03-14T22:27:25.883

@Rojo Works with my method for a double blank (although with a slight glitch, an error message is generated - due to a minor bug, but that does not affect the result). Try withUserDefs[ParallelSubmit, {SetAttributes[ParallelSubmit, HoldAllComplete]; ParallelSubmit[__] := 8}, ParallelSubmit[2 + 2]]. Although, tripple blank is the only pattern which my method does not currently cover. – Leonid Shifrin – 2012-03-15T00:09:28.713

@LeonidShifrin, your method is great, robust. But It is for cases when you're the user of a package and want to extend it. The other answers are attempts to deal with the case when you are creating the "extensible" package. What I just commented here was that the Parallel* built-ins don't behave as "closed" like other built-ins. You can extend them using your magic, but not right away. So, peeking at how they did it won't help improve those attempts – Rojo – 2012-03-15T00:15:01.807

@Rojo But this is a normal behavior for a function implemented on the top level. Yes, these are different from built-in rules which are not top-level and lower precedence than user-defined ones - which is probably what you meant. But I consider the separation between built-in and user-defined rules very artificial, having to do with performance and perhaps protecting the system, but IMO impairing the purity of language design, and the language consistency. You just observed the consequence of that: because the division itself is artificial, the "precedence" - rule works only one level. – Leonid Shifrin – 2012-03-15T00:21:54.277

1@Rojo I think, Szabolcs didn't mean to peek at how those were written - he just meant that it would be nice to be able to consistently override functions, no matter built-ins or top-level. My mechanism is not perfect, but possesses one important property - it can be nested. Nesting is important, because it gives some level of consistency. – Leonid Shifrin – 2012-03-15T00:24:39.780

@LeonidShifrin, I agree. But given that built-ins already work that way and MMA users know it, I find it better to have a package whose behaviour doesn't depend on how exactly I defined the patterns. I can just document: "works as built-ins, whatever you define will be tried first", and you can extend it without fear, wrapt it, and forget about those internals. Don't you think it would be nice if somehow packages worked that way? – Rojo – 2012-03-15T00:28:39.363

@LeonidShifrin, then your mechanism seems perfect for that – Rojo – 2012-03-15T00:29:50.437

@Rojo Absolutely, but the problem is that, for this to be robust, all packages must work this way. And the only way to do this is to enforce these rules, pretty much on a language level, or with a special framework. Should we say develop a special package format, we could perhaps impose these rules. May be this is a good idea - but this would require a framework which would automate it and enforce these rules. The hardest part would be to convince people to start using it... I'd gladly continue this in chat, but have to go now and get some sleep :). But this is a very interesting topic. – Leonid Shifrin – 2012-03-15T00:34:26.667

@Rojo My comment was in reply to your previous one. As to my mechanism, one thing I dislike about it is that it is a run-time thing, so, as in all cases where dynamic scoping is used, the code which would use this mechanism extensively would be hard to understand, particularly because at no given time all definitions exist simultaneously. I would not use it for systematic development unless absolutely necessary, rather for some introspection and debugging tools. It's literally like in OOP - prefer composition over inheritance (users can delegate to different symbols, avoiding clashes). – Leonid Shifrin – 2012-03-15T00:40:31.693

What do you think about a function that clones the symbol into some "private implementation" symbol (given by Module) and redefines the symbol so it calls the private implementation, like in my answer below? You think it would be robust? If it's locked well... Sad – Rojo – 2012-03-15T01:03:36.723

@Rojo I thought about it. I think this is a bit too intrusive and may lead to problems. The problem is, suppose you have two different modules and each of them does that...That may lead to a disaster. This is why I think that any global redefinitions are bad. In my method, redefinitions are local to a given dynamic environment. As to Locked symbols - well, there is not much one can do about them anyways, except write wrappers with different names. Actually, what we all try to do here is to implement something like OO inheritance, which is hard because OO is not natively supported in M. – Leonid Shifrin – 2012-03-15T10:25:06.180

@Rojo The problem would disappear had M supported OO. In a functional world, one does not need inheritance so much because one can use higher-order functions and closures, and parametrize general modules by more specific functionality, effectively subtyping. But this requires careful design, so that such subtyping is planned from the start. It seems harder to achieve ad hoc overloading in FP (which is not necessarily bad - I've seen too much code where blind overloading led to some terrible code). – Leonid Shifrin – 2012-03-15T10:31:15.643

let us continue this discussion in chat

– Rojo – 2012-03-15T13:47:46.293



I will suggest a solution for DownValues - based definitions, but it may be generalized to other types of definitions as well. I will only consider a case of a single symbol, but again, the generalization is rather straightforward. You will also have to execute your code in a special dynamic environment.

A first ingredient of my suggestion is a (slightly modified) symbol-cloning functionality described here:

GlobalProperties[] := 
 {OwnValues, DownValues, SubValues, UpValues, 
    NValues, FormatValues, Options, DefaultValues, Attributes};

Attributes[clone] = {HoldAll};
clone[s_Symbol, new_Symbol] :=
  With[{clone = new, sopts = Options[Unevaluated[s]]},
    With[{setProp = (#[clone] = (#[s] /. HoldPattern[s] :> clone)) &},
      Map[setProp, DeleteCases[GlobalProperties[], Options]];
      If[sopts =!= {}, Options[clone] = (sopts /. HoldPattern[s] :> clone)];
      HoldPattern[s] :> clone]]

Here comes my suggested dynamic environment then:

SetAttributes[withUserDefs, HoldAll];
withUserDefs[sym_Symbol, {defs__}, code_] :=
  Module[{s, inSym},
    clone[sym, s];
     sym[args___] /; ! TrueQ[inSym] :=
        Block[{sym, inSym = True},
          clone[s, sym];
          With[{result = sym[args]},
             result /; result =!= Unevaluated[sym[args]]

What is happening here: first, we clone the original symbol. Then, we Block it, and run the definitions which we want to override the previous ones. Then, we add a catch-all definition which uses the Villegas-Gayley trick, but also Block-s the symbol in question again, and inside the inner Block uses the clone to effectively "unblock" the symbol to its original defs by reverse-cloning it, and run those. The extra trick to use With and a shared local variable result is needed to avoid infinite iteration in cases when neither user-defined nor previous rules apply.

Here comes an example:

f[1] = 10;
f[x_] := x^2;
f[x_, y_] := x + y;

And now:

withUserDefs[f, {f[x_] := x^4}, {f[1], f[2], f[1, 2], f[1, 2, 3]}]

  ==> {1, 16, 3, f[1, 2, 3]}

you can see that in the first two cases, the modified definitions were used, in the third one the original definition was used, and the last one did not match any and evaluated to itself.

One can nest these constructs, and the inner one will override the outer ones then. For example:

withUserDefs[f, {f[x_] := x^4}, 
   withUserDefs[f, {f[1] := 100}, 
     {f[1], f[2], f[1, 2], f[1, 2, 3]}]]

  ==> {100, 16, 3, f[1, 2, 3]}

You can ask why I wasn't just using the Villegas-Gayley trick by itself, which is much simpler. The answer is that there is no guarantee that the ordering of definitions will be right with it, even if we manually reorder them, and moreover, cases like f[1] = 10 are immune to all reorderings of DownValues and will always be at the top, so can not be dealt with at all, within a pure Villegas-Gayley approach - but can be successfully dealt with in this more complex one.

The suggested approach is as good as the symbol's cloning procedure is. For SubValues, UpValues etc, it should be modified. The full solution involving all possible global rules and multiple symbols will likely look more complex, but I just wanted to illustrate the idea in the simplest possible setting. Also, we probably can not count on such generalization to be fully robust.

Leonid Shifrin

Posted 2012-03-10T22:16:51.110

Reputation: 108 027

2It's not a System`​ symbol, but Language`ExtendedDefinition is very useful for cloning. It is used extensively in the cloud deployment of Wolfram Language, so should be safe for users to use as well. It exists at least as far back as M9. – Carl Woll – 2017-08-17T19:26:44.737

@CarlWoll Thanks. I've been aware of it for a while now. Actually, learned about it from this discussion. But not at the time when I was writing this answer. Should've updated it.

– Leonid Shifrin – 2017-08-17T19:39:12.533


If we are the ones writing the package in question, then we could proceed as follows. First, we define a public version of the function that delegates all calls to a private version:

publicFn[args___] := privateFn[args]

Then we define the functionality we desire on the private function:

privateFn[x_] /; x < 0 := 100 x
privateFn[x_] := 10 x

Before users make any modifications, the public function behaves in the way we have defined:

publicFn /@ Range[-5, 5]


Now, if a user comes along and adds a definition to the public function...

publicFn[n_?EvenQ] := 0

... then the corresponding behaviour is overridden, but the other definitions remain in place.

publicFn /@ Range[-5, 5]


Since the public function is defined with the broadest possible definition, all more-specific user definitions will be tried before the pre-defined rules. Of course, if the user redefines the public function's one definition exactly, then all bets are off.

Handling Partial Functions

If the function is only partially defined, we may want the public function to return unevaluated when the arguments are outside its domain. To do this, we need a slightly more elaborate version of the public function:

publicFn[args___] := Module[{v = privateFn[args]}, v /; !MatchQ[v, _privateFn]]

Unfortunately, Mathematica no longer considers this definition to be "broad" due to the complex MatchQ condition. Consequently, this definition will no longer be applied after subsequent user-supplied definitions. What we need to do is take over the management of the down-values of publicFn so that our "broad" definition is always last in the list. This is accomplished by installing a new SetDelayed/Set definition as an up-value on publicFn:

publicFn /: (SetDelayed|Set)[h_publicFn, def_] :=
  (DownValues[publicFn] =
      DeleteCases[DownValues[publicFn], Verbatim@HoldPattern[h] :> _]
    , HoldPattern[h] :> def
    , -2

We'll change privateFn so that it is only partially defined:

privateFn[x_] /; x < 0 := 100 x

publicFn /@ Range[-5, 5]

{-500, -400, -300, -200, -100, publicFn[0], publicFn[1], publicFn[2], publicFn[3], publicFn[4], publicFn[5]}

Users can then add further partial definitions:

publicFn[n_?EvenQ] := 0
publicFn[5] = "will be replaced by a subsequent definition";
publicFn[5] = "five";

{-500, 0, -300, 0, -100, 0, publicFn[1], 0, publicFn[3], 0, "five"}


Posted 2012-03-10T22:16:51.110

Reputation: 62 787

Nice one +1 Any way to allow our package function to return unevaluated (publicFn[..]) when it doesn't match the definitions we gave? – Rojo – 2012-03-11T01:12:28.193

@Rojo Yes, see the new section Handling Partial Functions. – WReach – 2012-03-11T01:55:45.973

but that last one works because privateFn fails with even arguments. If I remove the negative condition the user defined version never gets a chance to overload – Rojo – 2012-03-11T01:59:06.530

@Rojo You are correct -- user definitions are ignored in the new version. The problem is that the condition in the new definition of publicFn causes Mathematica to regard the definition as specialized instead of broad. Consequently it is no longer applied after all other definitions. I'm withdrawing the section on partial functions until such time that I have an alternative solution. Assuming that time ever comes... :) – WReach – 2012-03-11T06:01:34.473

@Rojo I've added a new strategy to deal with partial functions. It is not as clean as the simple definition, but it seems work for user-supplied down-value redefinitions. – WReach – 2012-03-11T07:24:14.793

1Thanks WReach. With your and @MrWizard's ideas I think I came up with a way that doesn't require resorting DownValues or defining upvalues on set and posted it. – Rojo – 2012-03-13T03:23:17.977


Here is method based on WReach's answer. It works by explicitly sorting the list of DownValues after any new assignment is made, placing the ___ last. It allows the return the public function unevaluated. By nature this only works for DownValues definitions.

ClearAll[func, privateFn]

privateFn[x_] /; x < 0 := 100 x
privateFn[x_?NumericQ] := 10 x

func[args___] := With[{eval = privateFn[args]}, eval /; ! MatchQ[eval, _privateFn]]

func /: (set : Set | SetDelayed)[lhs_func, rhs_] /; ! TrueQ[$deffunc] :=
  Block[{$deffunc = True},
    lhs ~set~ rhs;
    DownValues[func] = 
       {MatchQ[ #[[1]], _[_[_[_, Verbatim[___]]]] ] &}


func[n_Integer?EvenQ] := 0
func[n_Integer] := 1

func /@ Range[-5, 5]
{1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1}




Posted 2012-03-10T22:16:51.110

Reputation: 259 163

2Just saw this. +2 for the pattern, -1 for making my head hurt first thing in the morning… ;P – J. M.'s ennui – 2015-05-30T01:12:48.317

6_[_[_[_, Verbatim[___]]]] that's the scariest pattern I've ever seen. (note I said seen, not understood) – Rojo – 2012-03-13T03:23:00.023

8@Rojo is _~_~Verbatim@___//_//_ more cheerful? :o) It's just a pattern to match HoldPattern[func[args___]] :> something where args is arbitrary. – Mr.Wizard – 2012-03-13T06:06:19.030

2Haha, ok, I'm glad you used the first version – Rojo – 2012-03-14T23:43:10.673


Ok, based on your answers...

Module[{guard = True},
 publicFn[args___] /; guard := 
  Block[{guard = False}, Module[{res = publicFn[args]},
    If[MatchQ[res, _publicFn],
     res = privateFn[args]]; res /; ! MatchQ[res, _privateFn]]

This doesn't suffer problems of the automatic DownValue resorting, and without resorting to upvalues on Set functions... Shouldn't have problems with UpValues either. The public definition of the function is found perhaps before or perhaps after some user defined definitions. But if it is found, it makes sure there aren't any other good public definitions before transferring the job to the private definition. If there are none, it returns unevaluated.

privateFn[x_] /; x < 0 := 100 x
privateFn[x_?NumericQ] := 10 x
privateFn[3] = 99

publicFn /@ Range[-5 , 5]

{-500, 0, -300, 0, -100, 0, 10, 0, 30, 0, 50}  

Now, making some definitions

publicFn[n_?EvenQ] := 0;
publicFn[3 | 33] = -100;
soh /: publicFn[soh] := "boo"

We see that

publicFn /@ Range[-5 , 5]


{-500, 0, -300, 0, -100, 0, 10, 0, -100, 0, 50}




Posted 2012-03-10T22:16:51.110

Reputation: 40 993