You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).
There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):
- The relevant clinical/medical knowledge must be in a state of
equipoise (it's not already clear that one approach is better than
another, or which is better is genuinely not known).
- Individuals should be aware that they are participating in a study, and that they are being routed to
group A or B, and have the option to decline their assignment (or,
conversely, they have been made aware of the experimental assignment
and have consented to participate in advance).
- An institutional review board should evaluate your proposed
experiment and signed off on it. This, of course, presupposes that
you have access to such a board composed of members able to make
Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:
- Prepare your recommender model
- Feed your data through the model, without looking at actual outcomes
- Match your model output to actual outcomes to see whether or not
people followed the recommendation (whether they ever saw that
recommendation or not)
- Compare the results of people that ended up going with each
recommended approach (A vs. B), as well as those who "followed" the
recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.