In the context of argumentation relevance falls under the subject area of informal logic. The canonical reference is Johnson and Blair's Logical Self-defense. According to the summary in SEP's Informal Logic:
"Following Johnson & Blair (1977, 1994) many informal logicians understand informal validity in terms of relevance and sufficiency, making the criteria for good argument acceptability, relevance and sufficiency (the “ARS” criteria). The premises of an argument count as relevant to its conclusion when they provide some support for the conclusion and sufficient when they provide enough support to establish it as plausible. Relevance can be contrasted with irrelevance, which occurs in various instances of non sequitor, as occurs in the case of “straw man” and “red herring” arguments, which are common in ordinary discourse."
Another aspect that should be mentioned is the relevance logic. The idea is that the usual material conditional, which validates inferences like "if the Moon is made of cheese then the Riemann hypothesis is false", is too permissive and counterintuitive. The idea is that conclusions should be inferred only if the premises are relevant to the them, this leads to introducing the relevant conditional. There is a bit of a tension in trying to formalize a notion which is based on conceptual content rather than form:
"But there is a formal principle that relevant logicians apply to force theorems and inferences to “stay on topic”. This is the variable sharing principle. The variable sharing principle says that no formula of the form A → B can be proven in a relevance logic if A and B do not have at least one propositional variable (sometimes called a proposition letter) in common and that no inference can be shown valid if the premises and conclusion do not share at least one propositional variable... As we shall see, however, relevant logic does provide us with a relevant notion of proof in terms of the real use of premises, but it does not by itself tell us what counts as a true (and relevant) implication. It is only when the formal theory is put together with a philosophical interpretation that it can do this."
Relevance logics are even used to block the destructive effects of contradictions in classical reasoning, with the relevant conditional the law of explosion is no longer valid, contradiction does not imply anything. In other words, relevance logics are paraconsistent.
Finally, Wikipedia's Relevance also has some relevant (pardon the pun) references, for example Sperber-Wilson's Precis of Relevance: Communication and Cognition, which looks at the cognitive side of it:
"Grice's idea that the very act of communicating creates expectations which it then exploits provides a starting point. Beyond that, the inferential model needs radical reworking in order to become truly explanatory. A psychologically realistic answer must be given to such basic questions as these: What shared information is exploited in communication? What forms of inference are used? What is relevance and how is it achieved? What role does the search for relevance play in communication?
[...] It should be clear that we are not trying to define the ordinary and rather fuzzy English word relevance. We believe, though, that there is an important psychological property - a property involved in mental processes - which the ordinary notion of relevance roughly approximates, and which it is therefore appropriate to call by that name, using it in a technical sense... We then define:
Extent condition 1: An assumption is relevant in a context to
the extent that its contextual effects in that context are large.
Extent condition 2: An assumption is relevant in a context to
the extent that the effort required to process it in that context is