A Simplified and Intuitive Categorisation of Discriminative Models
A popular interview question.
In yesterday’s post, we discussed Generative and Discriminative Models.
Here’s the visual from that post for a quick recap:
In today’s post, we shall dive into a further categorization of discriminative models.
A Quick Recap
learn decision boundaries that separate different classes.
maximize the conditional probability:
P(Y|X)— Given an input X, maximize the probability of label Y.
are meant explicitly for classification tasks.
maximize the joint probability:
learn the class-conditional distribution
are typically not meant for classification tasks, but they can perform classification nonetheless.
Check yesterday’s post for more details: Generative and Discriminative Models.
Further distinction of discriminative models
In a gist, discriminative models directly learn the function f that maps an input vector (
x) to a label (
They can be further divided into two categories:
Direct labeling models
The following visual neatly summarizes their differences:
As the name suggests, probabilistic models provide a probabilistic estimate for each class.
They do this by learning the posterior class probabilities
P(Y|X) — given an input
X, what is the probability of a label
As a result, their predictions depict the model’s confidence in predicting a specific class label.
This makes them well-suited in situations when uncertainty is crucial to the problem at hand.
Conditional Random Fields (CRFs)
In contrast to probabilistic models, labeling models (also called distribution-free classifiers) directly predict the class label, without providing any probabilistic estimate.
As a result, their predictions DO NOT indicate a degree of confidence.
This makes them unsuitable when uncertainty in a model’s prediction is crucial.
That being said, it is important to note that these models, in some way, can be manipulated to output a probability.
For instance, Sklearn’s decision tree classifier does provide a
predict_proba() method, as shown below:
This may appear a bit counterintuitive at first.
However, in this case, the model outputs the class probabilities by looking at the fraction of training class labels in a leaf node.
In other words, say a test instance reaches a specific leaf node for final classification. The model will calculate the probabilities as the fraction of training class labels in that leaf node.
Yet, these manipulations do not account for the “true” uncertainty in a prediction.
This is because the uncertainty is the same for all predictions that land in the same leaf node.
Therefore, it is always wise to choose probabilistic classifiers when uncertainty is paramount.
👉 Over to you: Can you add one more model for probabilistic and labeling models?
Thanks for reading Daily Dose of Data Science! Subscribe for free to learn something new and insightful about Python and Data Science every day. Also, get a Free Data Science PDF (550+ pages) with 320+ tips.
👉 If you liked this post, don’t forget to leave a like ❤️. It helps more people discover this newsletter on Substack and tells me that you appreciate reading these daily insights.
The button is located towards the bottom of this email.
Thanks for reading!
Latest full articles
If you’re not a full subscriber, here’s what you missed last month:
To receive all full articles and support the Daily Dose of Data Science, consider subscribing:
👉 Tell the world what makes this newsletter special for you by leaving a review here :)
👉 If you love reading this newsletter, feel free to share it with friends!