## Is there something wrong with DistributeDefinitions?

7

I have the following code, which fails to distribute the DownValues for the symbol f. Could anybody please explain what is wrong.

f[1] = Range[10] + 1;
f[2] = Range[10] + 2;
f[3] = Range[10] + 3;

DownValues[f]

(* {HoldPattern[f[1]] :> {2, 3, 4, 5, 6, 7, 8, 9, 10, 11},
HoldPattern[f[2]] :> {3, 4, 5, 6, 7, 8, 9, 10, 11, 12},
HoldPattern[f[3]] :> {4, 5, 6, 7, 8, 9, 10, 11, 12, 13},
HoldPattern[f[4]] :> {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}} *)

ParallelEvaluate[Clear[f]]
DistributeDefinitions[f]

(* {Null, Null, Null, Null, Null, Null, Null, Null} *)
(* {} *)

ParallelEvaluate[DownValues[f]]

(* {{}, {}, {}, {}, {}, {}, {}, {}} *)


Any suggestions, pointers would be greatly appreciated

1@MarcoB I think there is something fishy. When I quit all kernels and start a fresh Matematica session I get the parallel downvalues correct. The problem is erratic, sometimes I get empty braces and sometimes the right definition. – Iconoclast – 2020-06-29T19:55:08.987

@MarcoB: I would request you to kindly run this short piece of code if you have time, please. I am really bogged down because of this as I want to do a huge parallel computation for my work. – Iconoclast – 2020-06-29T20:03:22.407

1@MarcoB Definition[f] is tricky. It does not evaluate, it just formats in a certain way. Try ParallelEvaluate[Echo@Definition[f]] to see that f has no definitions on the subkernels. – Szabolcs – 2020-06-29T20:21:54.010

I expect this happens because you were explicitly setting the definition of f on the subkernels, using Clear. If you omit that part, it works as expected. – Szabolcs – 2020-06-29T20:23:27.237

@MarcoB The result returned from the subkernels is {f[1], f[2], f[3]} because f has no definitions on the subkernels. But it does on the main kernel, so that result immediately evaluates further. – Szabolcs – 2020-06-29T20:39:01.397

@Szabolcs You are quite right. This is a nice self-reminder of why I don't "do" parallel :-) I'm going to delete my previous comments, since they muddy the issue. – MarcoB – 2020-06-29T20:51:45.437

@Szabolcs Yes, you seem to be right. Clearing f is causing the problem. But why, after distributing the definitions on f it is not being redefined? – Iconoclast – 2020-06-29T20:55:12.370

@Iconoclast Too tired to investigate tonight, but here's a guess: ParallelEvaluate[Clear[f]] does two things: first, it distributes the definitions of f (surprise!) because f appears in ParallelEvaluate. ParallelEvaluate used to not auto-distribute (unlike other parallel constructs) but this was changed in maybe 10.4 or 11.0 (??). Only after this initial distribution does it clear f in the subkernels. – Szabolcs – 2020-06-29T20:58:56.400

2However, by this point, the parallel tools framework has made a note for itself (on the main kernel) that "f is already distributed, no need to touch it until it changes on the main kernel again". You are not supposed to mess with it on the subkernel, as distribution is one way only (main kernel -> subkernel) and the system assumes that it is fully managing the definitions of f on the subkernel. It assumes that you are not interfering. – Szabolcs – 2020-06-29T21:00:05.333

2Your DistributeDefinitions[f] does nothing as the system believes that distribution has already happened, and there's no need to do it again. Notice that it returns {}. That means that nothing was distributed. – Szabolcs – 2020-06-29T21:00:49.843

This is my hypothesis. You can try to prove or disprove it :-) – Szabolcs – 2020-06-29T21:01:09.933

@MarcoB One of the most annoying failure modes for me: ParallelNeeds["SomePackage"]; LaunchKernels[]. Except I forgot that SomePackage is not on the path on the subkernels ... it's only on the path on the main kernel. Sadly, subkenels avoid evaluating any sort of init.m where one may have set the $Path ... Also sadly, while ParallelNeeds does affect kernels launched in the future, any errors will not be printed from these kernels. Only currently running kernels will complain that they can't find the package. Now I do ParallelTable[PackageFunction[i], {i, 1000}], which ... – Szabolcs – 2020-06-29T21:03:35.397 @MarcoB ... does absolutely nothing on the subkernels (since the package is not loaded and PackageFunction is not defined) but it returns a literal {PackageFunction[1], PackageFunction[2], ...} to the main kernel, which will then evaluate further. Everything appears to work. I do get the result I expect. It just doesn't run in parallel, there's no speedup at all ... – Szabolcs – 2020-06-29T21:04:54.277 @Szabolcs Yes, that behavior, which is perhaps a larger example of the pitfall I fell in, is positively wicked. I was trying a few more things before seeing your comment and was picking up on the odd lack of a speedup in embarrassingly parallel tasks, but it would have taken me a long time to twig to what was really going on. Really nasty. – MarcoB – 2020-06-29T21:21:07.730 Crossposted here. – Rohit Namjoshi – 2020-06-30T01:42:47.990 ## Answers 4 I believe the hypothesis of Szabolcs made in the comments is correct, and Mathematica only distributes definitions it deems to be "new". Digging a little into the definition of DistributeDefinitions, it calls ParallelProtectedDistDefs, which then calls ParallelParallelPrivateupdatedDefs. When the definitions of f have not yet been distributed, updatedDefs returns them untouched and creates the downvalue ParallelParallelPrivate$distributedDefs[HoldForm[f]], which in this case takes the form

ParallelParallelPrivate$distributedDefs[HoldForm[f]] = {OwnValues -> {}, SubValues -> {}, UpValues -> {}, DownValues -> {HoldPattern[f[1]] -> {2, 3, 4, 5, 6, 7, 8, 9, 10, 11}, HoldPattern[f[2]] -> {3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, HoldPattern[f[3]] -> {4, 5, 6, 7, 8, 9, 10, 11, 12, 13}}, NValues -> {}, FormatValues -> {}, DefaultValues -> {}, Messages -> {}, Attributes -> {}}  When DistributeDefinitions is called again without having modified f, updateDefs deletes the definitions of f from the the list of definitions to be distributed because ParallelParallelPrivateupdatedDefs[s : "HoldForm"[_Symbol] -> vals_List, True] /; ParallelParallelPrivate$distributedDefs[s] === vals :=
Sequence @@ {}


The solution to force a redistribution of definitions is to unset the appropriate $distributedDefs beforehand: f[1] = Range[10] + 1; f[2] = Range[10] + 2; f[3] = Range[10] + 3; ParallelEvaluate[Clear[f]]; ParallelParallelPrivate$distributedDefs[HoldForm[f]] =.

DistributeDefinitions[f]

(* {f} *)

ParallelEvaluate[DownValues[f]]

(* {{HoldPattern[f[1]]:>{2,3,4,5,6,7,8,9,10,11},
HoldPattern[f[2]]:>{3,4,5,6,7,8,9,10,11,12},
HoldPattern[f[3]]:>{4,5,6,7,8,9,10,11,12,13}}
,
{HoldPattern[f[1]]:>{2,3,4,5,6,7,8,9,10,11},
HoldPattern[f[2]]:>{3,4,5,6,7,8,9,10,11,12},
HoldPattern[f[3]]:>{4,5,6,7,8,9,10,11,12,13}}} *)


This sets the definitions in the subkernels correctly even when evaluated multiple times.

The simpler option would of course be not to clear f in the subkernels by hand, as DistributeDefinitions` does that in any case according to the documentation. But I guess this behavior is generally good to know about, if the subkernels run code that actively modifies the local definitions.