What is the most convenient way to read definitions of in-memory symbols when we don't have the source files? (Spelunking tools)

114

102

Note: I put Simon's implementation on GitHub. Contributions welcome!


When trying to read the definition of already defined (package or built-in) symbols using Information or FullDefinition, the biggest inconvenience is that lots of distracting private context names appear in front of all symbol names.

Currently I am using a little function contextFreeDefinition[] to avoid this problem. It will attempt to hide the most frequently appearing context name in the definition. contextFreeDefinition[] is based on this answer.

Compare for example ClearAttributes[RunThrough, ReadProtected]; Information[RunThrough] and contextFreeDefinition[RunThrough]. The latter is a lot less cluttered because the System`Dump` context is hidden in the definition. (I usually paste the output of this function into Workbench and re-indent it using the Source -> Format context menu item for better readability)


Unfortunately contextFreeDefinition[] does not always successfully hide contexts, for example try the following:

ImportString["1", "List"]; (* force Stub symbols to be loaded *)

System`Convert`TableDump`ImportList // contextFreeDefinition

and notice that several symbols (especially patterns) still have System`Convert`TableDump` prepended. For example, I see the following in the FullDefinition it prints:

protectRegEx[System`Convert`TableDump`s_String] := 
     StringReplace[System`Convert`TableDump`s, $ProtectedCharacterRules]

The symbol System`Convert`TableDump`s still has the context name prepended even though the function tried to hide exactly this context.

Question: How can contextFreeDefinition[] be fixed so it always hides the context, or what other alternative approaches are there to read the definitions of in-memory symbols?


The code of contextFreeDefinition[].

Clear[commonestContexts, contextFreeDefinition]

commonestContexts[sym_Symbol, n_: 1] := Quiet[
  Commonest[
   Cases[Level[DownValues[sym], {-1}, HoldComplete], 
    s_Symbol /; FreeQ[$ContextPath, Context[s]] :> Context[s]], n],
  Commonest::dstlms]

contextFreeDefinition::contexts = "Not showing the following contexts: `1`";

contextFreeDefinition[sym_Symbol, contexts_List] := 
 (If[contexts =!= {}, Message[contextFreeDefinition::contexts, contexts]];
  Internal`InheritedBlock[{sym}, ClearAttributes[sym, ReadProtected];
   Block[{$ContextPath = Join[$ContextPath, contexts]}, 
    Print@InputForm[FullDefinition[sym]]]])

contextFreeDefinition[sym_Symbol, context_String] := 
 contextFreeDefinition[sym, {context}]

contextFreeDefinition[sym_Symbol] := 
 contextFreeDefinition[sym, commonestContexts[sym]]

Understanding and using the function:

commonestContexts[sym, n] will find the n most frequently used contexts that are not in $ContextPath in the definition of symbol sym.

contextFreeDefinition[sym] will print the FullDefinition of sym, hiding the commonest context that would appear there. It will also issue a message with the name of the context being hidden.

contextFreeDefinition[sym, {"Context1`", "Context2`", ...}] will try to hide an explicitly given list of contexts.

Szabolcs

Posted 2012-02-14T14:55:29.663

Reputation: 213 047

@Szabolcs do you think it's worth highlighting the second most popular answer to this question? For 10.0+ users it's probably the simplest way, don't you think? – b3m2a1 – 2016-12-28T00:58:19.467

When you write "always removes the context" do you mean extraneous context or all context? – Mr.Wizard – 2012-02-14T16:02:59.640

Also, I get no output for System`Convert`TableDump`ImportList // contextFreeDefinition -- would you please try to provide an example that works in version 7? – Mr.Wizard – 2012-02-14T16:07:57.103

@Mr.Wizard This function tries to find the commonest non-public context, then prints the definition with that context hidden. It is also possible to explicitly pass a number of contexts to be hidden (as the second argument). – Szabolcs – 2012-02-14T16:27:27.850

I will accept either an answer that fixes contextFreeDefinition or one that will give a better suggestion on how to do this kind of system spelunking. – Szabolcs – 2012-02-14T16:35:09.460

Answers

91

Link to the code on GitHub


I have been using this. It's mostly Leonid's code from the stackoverflow question you linked to, but it uses Definition instead of DownValues. Symbol names are printed without any context, but the full symbol name is put into a Tooltip so you can always find out what context a symbol is in.

Update

FullDefinition[symbol] claims to "print the definitions given for symbol, and all symbols on which these depend", but sometimes one wants to explore deeper than the first level of dependency. Here is a version of Spelunk which uses plain Definition instead of FullDefinition, but allows you to click on symbols in the definition to get their definition. So you can dig right down into the dependency chain.

Update 2

The code now copes with definitions containing strings with backticks in, and cases where Definition throws an error.

Also, it now works for symbols which have OwnValues, e.g. Internal`$VideoEncodings.

BeginPackage["Spelunk`"];

Spelunk::usage = "Spelunk[symbol]";

Begin["`Private`"];

defboxes[symbol_Symbol] := Hold[symbol] /. _[sym_] :>
        If[MemberQ[Attributes[sym], Locked], "Locked",
          Internal`InheritedBlock[{sym},
            Unprotect[sym]; ClearAttributes[sym, ReadProtected];
            Quiet@Check[ToBoxes[Definition@sym], "DefError"] /. 
            InterpretationBox[a_, b___] :> a ]];

defboxes[s_String] := defboxes[#] &@ToExpression[s, InputForm, Unevaluated]

prettyboxes[boxes_] := 
  boxes /. {" "} -> {"\n-----------\n"} //. {RowBox[{left___, ";", 
       next : Except["\n"], right___}] :> 
     RowBox[{left, ";", "\n", "\t", next, right}], 
    RowBox[{sc : ("Block" | "Module" | "With"), "[", 
       RowBox[{vars_, ",", body_}], "]"}] :> 
     RowBox[{sc, "[", RowBox[{vars, ",", "\n\t", body}], "]"}]};

fancydefinition[symbol_Symbol] :=
  Cell[BoxData[
    prettyboxes[
     defboxes[symbol] /. 
      s_String?(StringMatchQ[#, __ ~~ "`" ~~ __] &) :> 
       First@StringCases[s, 
         a : (__ ~~ "`" ~~ b__) :> processsymbol[a, b]]]], "Output", 
   Background -> RGBColor[1, 0.95, 0.9],
   CellGroupingRules->"OutputGrouping",
   GeneratedCell->True,
   CellAutoOverwrite->True,
   ShowAutoStyles->True,
   LanguageCategory->"Mathematica",
   FontWeight->"Bold"
];

processsymbol[a_, b_] := Module[{db},
  Which[
   ! StringFreeQ[a, "\""], a,
   ! StringFreeQ[a, "_"] || (db = defboxes[a]) === "Null", 
   TooltipBox[b, a],
   db === "Locked", TooltipBox[b, a <> "\nLocked Symbol"],
   db === "DefError", TooltipBox[b, a <> "\nError getting Definition"],
   True, ButtonBox[TooltipBox[b, a], ButtonFunction :> Spelunk@a, 
    BaseStyle -> {}, Evaluator -> Automatic]]]

Spelunk[symbol_Symbol] := CellPrint[fancydefinition[symbol]];

Spelunk[s_String] := CellPrint[fancydefinition[#] &@ToExpression[s, InputForm, Unevaluated]];

SetAttributes[{defboxes, fancydefinition, Spelunk}, HoldFirst] 

End[];

EndPackage[];

Simon Woods

Posted 2012-02-14T14:55:29.663

Reputation: 81 905

I did not find a place to file bug reports, so I post it here: In Mathematica 10.0.2 Spelunk[WolframAlpha] produces an output with some error boxes. – Vladimir Reshetnikov – 2015-01-03T04:09:04.420

@VladimirReshetnikov, thanks for the notification. There are a few similar examples though I never got around to debugging it. If you come up with a workaround please let me know. – Simon Woods – 2015-01-03T13:18:32.600

Simon, I get a DefError when attempting Spelunk @ Image`ColorOperationsDump`iChromaticityPlot -- could you look into that when you have time? – Mr.Wizard – 2015-02-03T12:08:21.140

@Mr.Wizard, try Block[{Legended}, Spelunk @ ...] – Simon Woods – 2015-02-03T12:39:51.920

Thanks, that works. Should this be added to Spelunk itself? – Mr.Wizard – 2015-02-03T12:54:39.550

@Mr.Wizard, I guess so. There are a probably a few other typesetting functions that should be blocked too. I really need to give the code a bit more attention now that Mathematica has moved on a couple of versions. – Simon Woods – 2015-02-03T13:39:03.893

1

@Mr.Wizard, I've uploaded a new version of Spelunk to my fork of the GitHub. This uses the wonderful FrontEnd`UndocumentedTestFEParserPacket (which Kuba reminded me of) to extract boxes from the symbol definition. This seems to do a great job of stripping out the typesetting stuff which has been causing problems. It'll need some testing but so far it looks promising.

– Simon Woods – 2015-02-03T23:04:23.953

@Simon You should probably include a revision number in the package for future reference. – Mr.Wizard – 2015-02-05T17:22:18.777

2This is very nice! What a surprise after such a long time :) – Szabolcs – 2012-12-07T22:31:46.197

1

When you use BaseStyle->"Hyperlink" and maybe wrap it in a StyleBox to underline links, IMO the code is much better readable than with the buttons: http://i.stack.imgur.com/kNlOR.png Anyway, +1 for this nice work.

– halirutan – 2013-01-04T02:03:18.937

Would you mind if I put this on GitHub and start building on it? Or would you like to do that yourself? – Szabolcs – 2013-01-21T19:21:10.340

@Szabolcs, please do! You'll probably find it's more a case of "rebuild" than "build on" - the code became a bit of a tangle as I patched bugs. I look forward to seeing what you create :-) – Simon Woods – 2013-01-21T22:00:16.187

Just added your original code, no changes yet: https://github.com/szhorvat/Spelunking I use this a lot, and I wanted to find a more convenient home for the code than this SE post in a place where people can easily contribute. Even if you don't want to install git on your own computer, you can easily contribute using the online editor interface.

– Szabolcs – 2013-01-22T00:17:13.313

+1. And using this with Leonids depends is simply awesome. Almost automatic refactoring.

– Ajasja – 2013-01-22T09:55:17.373

@Szabolcs, thank you so much for the bounty! – Simon Woods – 2013-01-24T21:08:56.670

Simon, I added "GraphicsGrouping" to fancydefinition, so that the output will be grouped. "OutputGrouping" did not maintain grouping for subsequent invocations using the buttons, but "GraphicsGrouping" does. – rcollyer – 2013-03-05T15:11:03.940

@rcollyer, thanks! That's a really useful addition. – Simon Woods – 2013-03-05T15:44:04.583

@Simon Why exactly do you have this line, Quiet@Check[ToBoxes[Definition@sym], "DefError"]? Did you encounter a situation where this was needed? I am asking because if I turn on unpacking warnings (On["Packing"]), Definition may issue messages that don't actually indicate failure. So in this case it's better not to give up on messages. If you know about particular cases of failure where messages are shown, we could specialise Check for those messages only. – Szabolcs – 2013-03-05T15:46:22.610

@Szabolcs, I did encounter a situation where I got an error from Definition, but I can't remember what symbol had the problem! Perhaps the best thing would be to remove the check until the problem resurfaces or I remember what it was... – Simon Woods – 2013-03-05T16:27:38.510

@Szabolcs, a quick check shows that ColorDataFunction, EmpiricalDistribution and TagBox give an error. I only checked the System context, there are probably others too. It is actually ToBoxes that causes the problem - it seems to be related to symbols with special output forms. – Simon Woods – 2013-03-05T17:02:20.053

@Simon Thanks for the pointer! – Szabolcs – 2013-03-05T17:22:45.897

83

In version 10.1, I've built something like Spelunk into GeneralUtilities`.

To use it, run

Needs["GeneralUtilities`"]
PrintDefinitions[symbol];

This will pop up a window that shows all definitions of symbol. Here is a short summary of features:

  • The window shows code cells containing all DownValues, OwnValues, UpValues, SubValues, and Attributes of a symbol.
  • Most pathological kinds of StandardForm are avoided, so that Image, Graphics, Row, Column, SparseArray, etc. will show up as code, not UI elements. This is achieved via the PlainForm and CodeForm wrappers, which can also be found in GeneralUtilities`.
  • Hyperlinks are be injected as appropriate. Click on a hyperlinked symbol to print its definitions in a new window.
  • The CellContext of each code cell is set to reduce the amount of clutter from fully qualified private symbols. For symbols outside this context, the name is shown, the fully qualified name is tooltipped, and color coding is used: symbols in conventional private contexts like `Private` are brown/orange; system symbols are black; other symbols are dark gray if on the context path, otherwise light gray.

Here's a simple example of PrintDefinitions running on itself:

enter image description here

Taliesin Beynon

Posted 2012-02-14T14:55:29.663

Reputation: 10 244

Fantastic! This looks really useful. Is there really an internal function with more than 128 definitions? – Rolf Mertig – 2015-04-02T17:39:06.133

3@RolfMertig more that before we had associations, people would use symbols as a key-value store via their downvalues... and there might be thousands of definitions on such symbols. – Taliesin Beynon – 2015-04-02T18:46:28.323

1@RolfMertig Dataset is one of these functions, with 189 definitions. – Murta – 2015-05-07T22:55:40.900

1Is it by design that (Print)Definitions omits formatting definitions like MakeBoxes[...]^:=... (e.g. in case of PrintDefinitions@EvaluationObject)? Is it because some dynamic formatting box could lead to problems when evaluated for printing? @Rolf Try ChartElementData: it has 1853 definitions. – István Zachar – 2015-05-08T12:35:22.060

4I've only just seen this - awesome! It's really good to have something like this built in, many thanks. – Simon Woods – 2015-05-19T19:19:18.160

@SimonWoods cool, thanks for the inspiration! – Taliesin Beynon – 2015-08-07T22:27:27.857

@IstvánZachar that's just an oversight. You can use PrintDefinitions to patch Definitions, though that won't survive a kernel restart obviously. – Taliesin Beynon – 2015-08-07T22:28:07.960

Thanks for the reply. BTW, I've found PrintDefinitions@NonlinearStateSpaceModel to crash the kernel after a flood of error messages. I assume the problem is not in PD but in some loose NSSM definitions evaluating prematurely? – István Zachar – 2015-11-26T15:22:46.933

What's the matter with printing more than 128 definitions? I can't do PrintDefinitions[ComplexAnalysis`BranchCuts] because it has 166 definitions. Would you kindly take away this limitation or maybe add an option "MaxDefinitions" that sets the threshold? thanks! – QuantumDot – 2016-05-30T09:37:00.923

Just stumbled onto this. Very handy. Hopefully will one day become part of a FE IDE ... – Ronald Monson – 2016-06-15T10:29:54.773

@QuantumDot it's possible to escape that behavior by setting Length to max out at 127 using Villegas-Gayley. PrintDefinitions will work fine then, although it seems in 11.0 the max has been changed to 256.

– b3m2a1 – 2016-12-27T23:35:21.273

+1. Could it print something like 'there are 35000 DownValues and 1 Attributes for 'Temp'' instead of 'Temp has too many definitions (35001) to display.'? – berniethejet – 2019-01-12T23:34:21.963

34

I can now offer a solution which leverages the full power of the code formatter, in its new, more robust form.

Load the formatter:

Import["https://raw.github.com/lshifr/CodeFormatter/master/CodeFormatter.m"]

Some examples:

CodeFormatterSpelunk[RunThrough]

enter image description here

CodeFormatterSpelunk[PacletManager`CreatePaclet]

enter image description here

In the last example, using MakeBoxes would produce a slightly better result:

CodeFormatterSpelunk[PacletManager`CreatePaclet, MakeBoxes]

enter image description here

CodeFormatterSpelunk[PacletManager`PackPaclet]

enter image description here

Leonid Shifrin

Posted 2012-02-14T14:55:29.663

Reputation: 108 027

Great stuff Leonid! would you happen to know how to use this on a Manipulate source code? When I do CodeFormatterSpelunk[ Manipulate[Plot[Sin[a x], {x, 0, 1}], {{a, 1, "a"}, 1, 10, 1}]] I just get Mathematica graphics (I am trying to use your program to format my manipulate code, but not able to)....

– Nasser – 2013-11-03T02:14:57.027

@Nasser Thanks :). Re: manipulate - the CodeFormatterSpelunk function takes a symbol name, not code. You can do e.g. fn[] := Manipulate[Plot[Sin[a x], {x, 0, 1}], {{a, 1, "a"}, 1, 10, 1}], and then CodeFormatterPrint[fn], or you can do CellPrint[Cell[CodeFormatterMakeBoxes[Manipulate[Plot[Sin[a x], {x, 0, 1}], {{a, 1, "a"}, 1, 10, 1}]], "Input"]], but the latter for some reason does not format quite right...Perhaps a bug. Will look into this some time soon. – Leonid Shifrin – 2013-11-03T10:55:51.780

26

Since nobody has mentioned it yet...

V8 introduced the undocumented flag Debug`$ExamineCode. When it is set to true, the information functions will display the definitions of ReadProtected symbols:

Debug`$ExamineCode = True
??BinLists

It is sometimes useful to suppress some of the internal package names to make it easier to scan the definitions. Here is a quick-and-dirty way to do it:

Block[
  { $ContextPath = {"System`", "Internal`", "Statistics`BinningDump`"}
  , Debug`$ExamineCode = True
  }
, RawBoxes @ ToBoxes @ Information[BinLists]
]

Here, the $ContextPath is carefully chosen to include package names I do not want to see. This, of course, can be automated but as that is the subject of other answers I will pass over it here.

WReach

Posted 2012-02-14T14:55:29.663

Reputation: 62 787

17

I would just use strings, for all their fragility:

ClearAll[print];
print[sym_, {conts_String}] :=
 With[{altptrn =  Alternatives @@ Reverse[SortBy[{conts}, StringLength]]},
    Print@StringReplace[ToString[InputForm@FullDefinition@sym],
         (x : (_ | "") ~~ altptrn ~~ y : (_ | "")) /; ! (x === "\"" && y === "\"") :> 
            StringJoin[x, y]]]

contextFreeDefinition[sym_Symbol, contexts_List] :=
 (
    If[contexts =!= {}, Message[contextFreeDefinition::contexts, contexts]];
    print[sym, contexts]
 );

Note that my code to protect against modifications inside strings is not quite robust. But, if needed, it can be easily made more robust by preprocessing the string.

Leonid Shifrin

Posted 2012-02-14T14:55:29.663

Reputation: 108 027

10

I have not tested this yet but here is one possible approach:

contextFreeDefinition[sym_Symbol, contexts_List] :=
 Internal`InheritedBlock[{sym},
  ClearAttributes[sym, ReadProtected];
  If[contexts =!= {}, Message[contextFreeDefinition::contexts, contexts]];
  Block[{ipf = ToString @ InputForm @ FullDefinition @ sym},
   ipf = MakeExpression@StringSplit[ipf, "\n"] /. HoldComplete[Null] -> "";
   ipf /. x_Symbol /; MemberQ[contexts, Context[x]] :>
     With[{eval = StringDrop[ToString@Unevaluated@x, StringLength@Context@x]},
         eval /; True]
     /. HoldComplete -> HoldForm // Column
   ]
  ]

(sorry for the messy formatting)

Please point out all failings and I shall see if this is redeemable tomorrow.

Mr.Wizard

Posted 2012-02-14T14:55:29.663

Reputation: 259 163

Let me think hard and try to break this ;-) First attempt: contextFreeDefinition[someSymbol, "System`"]. Admittedly it's easy to put in a check if something from contexts is already in $ContextPath. – Szabolcs – 2012-02-15T11:07:47.963

9

This answer addresses hiding the context marks only, not spelunking. You could use the following functions to view output without contexts marks:

SetAttributes[{ContextFreeForm, ContextFreeInformation}, HoldAll]

ContextFreeForm /: MakeBoxes[ContextFreeForm[expr_], form_] := Block[
    {Internal`$ContextMarks=False},

    MakeBoxes[expr, form]
]

ContextFreeInformation[expr_] := With[
    {name = ToString[Unevaluated @ expr, InputForm]},

    Block[{Internal`$ContextMarks = False}, Information[name]]
]

First, here is the definition of System`Convert`TableDump`ImportList viewed using ContextFreeForm:

ContextFreeForm @ Definition @ System`Convert`TableDump`ImportList

ImportList[file_String,opts___]:=Block[{listsep},listsep=LineSeparators/. {opts}/. Options[ImportList];If[!ListQ[listsep],listsep={listsep}];Data->Flatten[Last[ImportTable[file,LineSeparators->listsep,FieldSeparators->listsep,RepeatedSeparators->False,opts]],1]]

Options[ImportList]={LineSeparators->{

, , , }}

For Information (i.e., ?symbol) there is no output, the definition is displayed as a side effect, so a different function is needed:

ContextFreeInformation @ System`Convert`TableDump`ImportList

System`Convert`TableDump`ImportList

ImportList[file_String,opts___]:=Block[{listsep},listsep=LineSeparators/. {opts}/. Options[ImportList];If[!ListQ[listsep],listsep={listsep}];Data->Flatten[Last[ImportTable[file,LineSeparators->listsep,FieldSeparators->listsep,RepeatedSeparators->False,opts]],1]]

Options[ImportList]={LineSeparators->{

, , , }}

It is also possible to just set Internal`$ContextMarks = False globally, but that can have issues (e.g., Information[System`Convert`TableDump`ImportList] will no longer work).

Carl Woll

Posted 2012-02-14T14:55:29.663

Reputation: 112 778