Why can't I inject expressions in Compile using (only) With

15

5

Generally speaking, With can be used to inject into arbitrarily held expressions, for example:

With[{args = 2},
  Hold @ Hold @ HoldComplete[args]
]
(* Out[1]= Hold[Hold[HoldComplete[2]]] *)
Attributes @ foo = HoldAll;
With[{args = {{x, _Real}}},
  foo[args, x + 2]
]
(* Out[2]= foo[{{x, _Real}}, x + 2] *)

However, trying to do the same exact thing to inject inside Compile does not work:

With[{args = {{x, _Real}}},
  Hold @ Compile[args, x + 2]
]
(* Hold[Compile[args, x + 2]] *)

Also, the only relevant attribute of Compile is HoldAll, which as shown in the first examples is not really a problem for this kind of injection. Does this mean that Compile follows some special evaluation rules (even though it's not listed among the magic symbols)?

Using a Trott-Strzebonski-like trick, like shown for example here, does work:

With[{args = {{x, _Real}}},
  Hold @ Compile[args, x + 2] /; True
][[1]]
(* Hold[Compile[{{x, _Real}}, x + 2]] *)

An extract from Leonid's explanation in the above linked answer explains why adding the Condition works:

The basic idea is to exploit the semantics of rules with local variables shared between the body of With and the condition, but within the context of local rules. Since the condition is True, it forced the eval variable to be evaluated inside the declaration part of With (...)

This explanation does not however explain why the use of Compile in particular makes a difference, why does it prevent With from injecting into it, and why can we avoid this constraint adding a Condition?


Some more examples of functions showing or not showing this kind of behaviour:

With[{args = x}, Hold@Compile[args, 2]]
With[{args = x}, Hold@Function[args, 2]]
With[{args = x}, Hold@Module[args, 2]]
With[{args = x}, Hold@With[args, 2]]
With[{args = x}, Hold@Block[args, 2]]
(*
  Out[1]= Hold[Compile[args, 2]]
  Out[2]= Hold[Function[args, 2]]
  Out[3]= Hold[Module[x, 2]]
  Out[4]= Hold[With[x, 2]]
  Out[5]= Hold[Block[x, 2]]
*)

Also, in at least some cases it matters whether there is a second argument:

With[{args = x}, Hold @ Compile[args]]
With[{args = x}, Hold @ Function[args]]
(* Out[1]= Hold[Compile[args]] *)
(* Out[2]= Hold[x &] *)

glS

Posted 2017-08-04T11:35:57.590

Reputation: 7 002

2For the last case With[{args = x}, Hold @ Function[args]], my guess would be that a function with a single arg is treated as a slot-based pure function, which is not considered a scoping construct in the context of protection of local variables, since it does not use named variables (it uses slots instead). Which is why such protection is not there in this case. – Leonid Shifrin – 2017-08-04T12:40:35.927

Answers

15

Compile is considered a scoping construct by the outer With, and its bindings are protected. This will work:

With[{args = {{x, _Real}}}, Hold[Compile @@ Hold[args, x + 2]]] 

or this (if you don't want to keep Apply in code):

Unevaluated[ 
  With[{args = {{x, _Real}}}, Hold[Compile[args, x + 2]]]
] /. Compile -> $compile /. $compile -> Compile 

One relevant link is this. If you don't want to care about inner scoping constructs, use replacement rules instead of With - the injector pattern at your service:

Hold @ Compile[args, x + 2] /. Unevaluated[args ->   {{x, _Real}}]

For the Trott-Strzebonski part, it works because there, With is a secondary device serving essentially replacement rules. There, you could directly use RuleCondition instead of With. So my point is, for Trott-Strzebonski trick, the essential part is that one uses replacement rules. And rules don't care about inner scoping constructs.

It has been noticed in comments and in the edit of the original question, that not all scoping constructs are treated in the same way, and in particular for Function (with named arguments) and Compile, the entire declaration is protected, while for With and Module the protection happens on the level of individual variables (in terms of name collisions).

Leonid Shifrin

Posted 2017-08-04T11:35:57.590

Reputation: 108 027

Another illustration: With[{args = x}, Hold@Function[args, x + 2]] – Mr.Wizard – 2017-08-04T12:12:01.593

1Leonid, wouldn't With[{args = {{x, _Real}}}, Compile @@@ Hold[Hold[args, x + 2]]] be a better substitution for this? – Mr.Wizard – 2017-08-04T12:12:51.083

Interesting. I have a few questions though: 1) why is Compile considered a scoping construct? And is there a list of functions that are considered scoping constructs by With? It doesn't show in Attributes or such. 2) running my second example with With instead of Compile works as expected (as in, it replaces args), doesn't this contradict your argument about inner scoping constructs? – glS – 2017-08-04T12:13:38.130

@Mr.Wizard thanks, I added that and some more examples in the question – glS – 2017-08-04T12:22:15.770

@Mr.Wizard It is a possibility, yes - there are surely many. The second one I gave in the answer is somewhat different though, which is why I thought it made sense to provide it. – Leonid Shifrin – 2017-08-04T12:22:18.007

2@glS 1) Compile binds variables (in this case, declares their types), so it has to protect those variables (they can be defined outside, and that should not affect Compile) - which is why it is considered a scoping construct. 2) I would guess that the variable - protecting mechanism treats Compile somewhat differently from other scoping constructs. For those, it looks at conflicts of individual scoped variables, while here, it protects the full declaration. Consider this, for an example or protection in action: With[{args = {{x, _Real}}}, Hold@With[{args = {1, 2, 3}}, args + 2]]. – Leonid Shifrin – 2017-08-04T12:28:16.097

1

not sure but cannot we use Evaluate in this case?

With[{args = {{x, _Real}}},
Hold@Compile[Evaluate@args, x + 2]
]
(* Hold[Compile[Evaluate[{{x, _Real}}], x + 2]] *)

Ali Hashmi

Posted 2017-08-04T11:35:57.590

Reputation: 8 448

2Evaluate is the wrong tool here, and it only "works" because the presence of Evaluate hides Compile from outer With (it does no longer recognize Compile as a scoping construct). But this would bite you later, for example if x has been globally defined. – Leonid Shifrin – 2017-08-04T11:55:52.793

@LeonidShifrin thanks i see. But as long as no global variable is defined the approach should work. I get your point, it is better to use Unevaluated on the whole expression and make the replacements inside – Ali Hashmi – 2017-08-04T12:04:36.277

Evaluate only works one level deep, and even there only for heads which are not HoldAllComplete. In this case, the expression is too deep for Evaluate to have any effect at the time when external With evaluates. But later, it will force Compile to evaluate its first argument, which is not desirable for a scoping construct. – Leonid Shifrin – 2017-08-04T12:06:28.063

@LeonidShifrin to better understand your last point do you mind giving an example that i can check, especially the case when the head has an attribute of HoldAllComplete – Ali Hashmi – 2017-08-04T12:19:21.610

Just try HoldComplete[Evaluate[1+1]] for a start. Also, read the chapter 7 of David Wagner's book (chapter on evaluation), he explains this very clearly. – Leonid Shifrin – 2017-08-04T12:29:34.773