What are bias and variance in machine learning?


I am studying machine learning, and I have encountered the concept of bias and variance. I am a university student and in the slides of my professor, the bias is defined as:

$bias = E[error_s(h)]-error_d(h)$

where $h$ is the hypotesis and $error_s(h)$ is the sample error and $error_d(h)$ is the true error. In particular, it says that we have bias when the training set and the test set are not independent.

After reading this, I was try to get a little deepr in the concept, so I searched on internet and found this video , where it defines the bias as the impossibility to capture the true relationship by a machine learning momdel.

I don't understand, are the two definition equal or the two type of bias are different?

together with this, I am also studying the concept of variance, and in the slides of my professor it is said that if I consider two different samples from the sample error may vary even if the model is unbiased, but in the video I posted it says that the variance is the difference in fits between training set and test set.

Also in this case the definitions are different, why?


Posted 2020-08-12T08:10:53.703

Reputation: 531



What are Bias and Variance?

Let's start with some basic definitions:

  • Bias: it's the difference between average predictions and true values.
  • Variance: it's the variability of our predictions, i.e. how spread out your model predictions are.

They can be understood from this image:

enter image description here


What to do about bias and variance?

If your model suffers from a bias problem you should increase its power. For example, if the prediction of your neural network is not good enough, add more parameters, add a new layer making it deeper, etc.

If your model suffers from a variance problem instead, the best possible solution is coming from ensembling. Ensembles of Machine Learning models can significantly reduce the variance in your predictions.

The Bias-Variance tradeoff

If your model is underfitting, you have a bias problem, and you should make it more powerful. Once you made it more powerful though, it will likely start overfitting, a phenomenon associated with high variance. For that reason, you must always find the right tradeoff between fighting the bias and the variance of your Machine Learning models.

enter image description here


Learning how to do that is more an art than a science!


Posted 2020-08-12T08:10:53.703

Reputation: 4 928

1What exactly do the bulleyes and the points on the bulleye diagram represent? I get that it's (probably) the error of individual predictions, but isn't that usually one-dimensional? Or is it just an abstract picture intended to convey an idea rather than being a concrete representation of some data? – NotThatGuy – 2020-08-12T22:18:44.950

The bullseys are hypothetical Loss functions, with the center being the global minimum to be reached. But you can think of them as actual bullseys, and the points being shots. – Leevo – 2020-08-13T14:02:47.720


Well this image explains it all : in ML, you have a bias/variance dilemma : you want to create a model that is precise-enough to learn things from your data, but not perfectly-precised so it learns a tendancy and not the exact values of your training set.

Variance and Bias are to be taken together : on a same model, when you tweak to lower Variance, you'll automatically increase Bias.

Your job is then to get the good compromise, as show in image : a variance high enough (ie a bias low enough) to make good predictions and learn something from your train, but not a too high variance (ie not a too low bias) to avoid overfitting.

enter image description here


Posted 2020-08-12T08:10:53.703

Reputation: 611

The image explains it fairly well to me (who already understands it), but an explanation could still help others who may not quite be seeing what you're seeing in the image (or if they're unable to view the image for whatever reason). The rest of your answer mostly addresses the tradeoff (kind of) between bias and variance rather than what they mean. – NotThatGuy – 2020-08-12T22:26:05.723

The answer was more about practical understanding of what bias and variance represent in machine learning, than what it is mathematically and theorically, which I'm not prefectly aware of – BeamsAdept – 2020-08-13T08:02:18.077

It doesn't have to involve mathematics or too much theory. If you assume someone reads this answer and they're unable to see or interpret the image, do you believe they would learn what bias and variance means based on the text alone? The text seems to assume the reader already knows what bias and variance is. The other answer could potentially serve as a good example of what I mean: there are images, but the answer doesn't rely on them.

– NotThatGuy – 2020-08-13T11:08:21.747