F-score

In the world of artificial intelligence and data science, how do we truly know if our predictive systems are good? The F-score is a critical metric that helps us assess performance in tasks like identifying spam or diagnosing diseases. It offers a sophisticated way to balance two often-conflicting measures: how accurate our positive predictions are, and how many actual positives we managed to find. Let's delve into this elegant statistical tool and uncover its nuances. The F-score combines precision and recall into a single metric, providing a balanced view of a model's performance in binary classification tasks. Precision measures how many of a model's positive predictions were correct, while recall measures how many actual positive cases the model successfully identified. Different F-score variants, particularly the F-beta score, allow us to weigh precision or recall more heavily, adapting the evaluation to specific problem requirements.

Source: Wikipedia

AI Summary

The F-score combines precision and recall into a single metric, providing a balanced view of a model's performance in binary classification tasks.
Precision measures how many of a model's positive predictions were correct, while recall measures how many actual positive cases the model successfully identified.
Different F-score variants, particularly the F-beta score, allow us to weigh precision or recall more heavily, adapting the evaluation to specific problem requirements.

Measuring Predictive Performance

Imagine you're building a system to detect rare diseases or identify fraudulent transactions. How would you measure its success? It's not as simple as just counting correct answers. We need a nuanced approach, and that's where metrics like the F-score come into play.

Precision: Are We Right When We Say 'Yes'?

At the heart of the F-score are two fundamental concepts: precision and recall. Let's start with precision. This tells us, out of all the times our system predicted something was positive—like predicting a transaction was fraudulent—how many of those predictions were actually correct? It's a measure of exactness.

Term	Description
True Positives (TP)	Correctly predicted positive
False Positives (FP)	Incorrectly predicted positive (Type I error)

Think of a spam filter. If it flags an email as spam, how often is it right? A high precision means fewer legitimate emails end up in your spam folder. Mathematically, we calculate precision by dividing the number of 'True Positives' by the sum of 'True Positives' and 'False Positives'.

Term	Formula
Precision	TP / (TP + FP)

In diagnostic testing, precision is also known as the Positive Predictive Value—it tells you the probability that a positive test result is truly positive. It's crucial for minimizing false alarms.

Precision = \frac{TP}{TP + FP}

Recall: Did We Find All the 'Yes's?

Now, let's consider recall. This metric asks: out of all the things that were actually positive—all the true fraudulent transactions or all the patients with the disease—how many did our system successfully identify? It's a measure of completeness.

Term	Description
True Positives (TP)	Correctly predicted positive
False Negatives (FN)	Incorrectly predicted negative (Type II error)

Going back to our spam filter, recall measures how many actual spam emails it successfully caught. If it has high recall, very little spam makes it to your inbox. We calculate recall by dividing the number of 'True Positives' by the sum of 'True Positives' and 'False Negatives'.

Term	Formula
Recall	TP / (TP + FN)

In medicine, recall is often called 'sensitivity'—how sensitive is a test to detecting the actual presence of a condition? Often, there's a trade-off: increasing precision might decrease recall, and vice-versa. This is why we need a combined metric.

Recall = \frac{TP}{TP + FN}

The F1-Score: A Balanced Perspective

This is where the F-score comes in, specifically the F1-score. The F1-score is the harmonic mean of precision and recall. Unlike a simple arithmetic average, the harmonic mean gives more weight to lower values, ensuring that both precision and recall must be reasonably high for a good F1-score. If either one is very low, the F1-score will reflect that.

Metric	Range
F1-score	0.0 to 1.0

A perfect F1-score of 1.0 means your system has both perfect precision and perfect recall. Conversely, an F1-score of 0 means either precision or recall (or both) are zero. It provides a single, symmetrical number to judge a system's balance between these two crucial aspects.

F_1 = 2 \cdot \frac{Precision \cdot Recall}{Precision + Recall} = \frac{2TP}{2TP + FP + FN}

A Quick Detour: The Name's Origin

Where did the 'F' come from? The F-measure's name is believed to originate from an 'F' function mentioned in research by Cornelis J. Van Rijsbergen. It gained wider recognition when introduced at the Fourth Message Understanding Conference (MUC-4) in 1992, a key event in natural language processing.

F-beta Score: When Balance Isn't Enough

Sometimes, we don't need a perfectly balanced view. In some applications, recall might be far more important than precision, or vice-versa. For instance, in medical diagnosis, missing a true disease (low recall) might be more catastrophic than a false alarm (low precision). This is where the more generalized F-beta score steps in.

The F-beta score introduces a positive real factor, 'beta' (\beta). This beta allows us to weigh recall 'beta' times as important as precision. If beta is greater than 1, we prioritize recall. If beta is less than 1, we prioritize precision. When beta equals 1, we get back our original F1-score.

F_\beta = (1 + \beta^2) \cdot \frac{Precision \cdot Recall}{(\beta^2 \cdot Precision) + Recall}

Commonly, beta values of 2 (F2-score) and 0.5 (F0.5-score) are used. An F2-score would mean recall is considered twice as important as precision—great for those critical disease detection scenarios. An F0.5-score, meanwhile, gives precision twice the weight, useful when false positives are particularly costly, like in certain security applications.

Real-World Applications

The F-score is a workhorse in many technical fields. In information retrieval, it's vital for evaluating search engines, document classification, and understanding how well a system answers a query. It's especially useful when the 'positive' class you're looking for is rare.

Beyond search, F-scores are widely used in machine learning to evaluate classification models, particularly in natural language processing tasks like named entity recognition and word segmentation. The flexibility of F-beta scores allows researchers to fine-tune evaluations to specific performance goals, moving beyond the balanced F1 when necessary.

Why Not Always Use F-score? Criticisms

Despite its utility, the F-score isn't without its critics. One major point is that it gives equal importance to precision and recall in its default F1 form, but in reality, the costs of different misclassifications can vary wildly. A false positive in a medical diagnosis might be inconvenient, but a false negative could be fatal.

Another significant criticism is that the F-score, particularly F1, ignores 'True Negatives'—cases where the system correctly identified that something was not positive. This oversight can make the F-score misleading, especially when dealing with highly imbalanced datasets where true negatives might dominate. Metrics like the Matthews correlation coefficient (MCC) or Cohen's kappa account for all four outcomes: True Positives, True Negatives, False Positives, and False Negatives, offering a more complete picture.

Furthermore, the F-score can be sensitive to the arbitrary choice of what we define as 'positive' versus 'negative.' If you swap these labels, the F1-score might change, highlighting a lack of symmetry. For some, the 'expected cost' or 'expected utility' of classification decisions offers a more principled and robust evaluation approach, considering the real-world consequences of a model's errors.

Multi-Class Extension

While we've focused on binary classification, the F-score can also be adapted for scenarios with more than two classes. This is typically done by calculating the F-score for each class individually and then averaging these scores. There are a few ways to average, leading to 'Macro F1' and 'Micro F1'.

Macro F1

Macro F1 calculates the F1-score for each class independently and then takes the arithmetic mean of these per-class F1 scores. It gives equal weight to each class, regardless of how many samples belong to it, providing a good measure of overall performance across all classes.

Micro F1

Micro F1, on the other hand, aggregates the counts of true positives, false positives, and false negatives across all classes first, and then calculates the F1-score using these aggregated counts. In single-label multi-class classification, micro precision, micro recall, and thus micro F1 are all equal. However, it's crucial to remember that micro F1 does not generally equal accuracy, as accuracy considers true negatives while micro F1, like its binary counterpart, does not.

Article

F-score

Precision and recall

In statistical analysis of binary classification and information retrieval systems, the F-score or F-measure is a measure of predictive performance. It is calculated from the precision and recall of the test, where the precision is the number of true positive results divided by the number of all samples predicted to be positive, including those not identified correctly, and the recall is the number of true positive results divided by the number of all samples that should have been identified as positive. Precision is also known as positive predictive value, and recall is also known as sensitivity in diagnostic binary classification.

The F1 score is the harmonic mean of the precision and recall. It thus symmetrically represents both precision and recall in one metric. The more generic ${\displaystyle F{\beta }}$ score applies additional weights, valuing one of precision or recall more than the other.

The highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if the precision or the recall is zero.

Etymology

F-score

The name F-measure is believed to be named after a different F function in Van Rijsbergen's book, when introduced to the Fourth Message Understanding Conference (MUC-4, 1992).

Definition

F-score

The traditional F-measure or balanced F-score (F1 score) is the harmonic mean of precision and recall:

${\displaystyle F{1}={\frac {2}{\mathrm {recall} ^{-1}+\mathrm {precision} ^{-1}}}=2{\frac {\mathrm {precision} \cdot \mathrm {recall} }{\mathrm {precision} +\mathrm {recall} }}={\frac {2\mathrm {TP} }{2\mathrm {TP} +\mathrm {FP} +\mathrm {FN} }}}$

With precision = TP / (TP + FP) and recall = TP / (TP + FN), it follows that the numerator of F1 is the sum of their numerators and the denominator of F1 is the sum of their denominators.

${\displaystyle F{1}={\frac {2\mathrm {TP} }{2\mathrm {TP} +2\mathrm {FP} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FP} }}}$

${\displaystyle F{1}={\frac {2\mathrm {TP} }{2\mathrm {TP} +2\mathrm {FN} }}={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} }}}$

${\displaystyle F{1}={\frac {2\mathrm {TP} }{2\mathrm {TP} +2\mathrm {FP} }}={\frac {2\mathrm {TP} }{4\mathrm {TP} }}={\frac {1}{2}}=0.5}$

${\displaystyle F{1}={\frac {2\mathrm {TP} }{2\mathrm {TP} +2\mathrm {FN} }}={\frac {2\mathrm {TP} }{4\mathrm {TP} }}={\frac {1}{2}}=0.5}$

To see it as a harmonic mean, note that ${\displaystyle F{1}^{-1}={\frac {1}{2}}(\mathrm {recall} ^{-1}+\mathrm {precision} ^{-1})}$.

Fβ score

A more general F score, ${\displaystyle F{\beta }}$, that uses a positive real factor ${\displaystyle \beta }$, where ${\displaystyle \beta }$ is chosen such that recall is considered ${\displaystyle \beta }$ times as important as precision, is:

${\displaystyle F{\beta }={\frac {\beta ^{2}+1}{(\beta ^{2}\cdot \mathrm {recall} ^{-1})+\mathrm {precision} ^{-1}}}={\frac {(1+\beta ^{2})\cdot \mathrm {precision} \cdot \mathrm {recall} }{(\beta ^{2}\cdot \mathrm {precision} )+\mathrm {recall} }}}$

To see that as a weighted harmonic mean, note that ${\displaystyle F{\beta }^{-1}={\frac {1}{\beta +\beta ^{-1}}}(\beta \cdot \mathrm {recall} ^{-1}+\beta ^{-1}\cdot \mathrm {precision} ^{-1})}$.

In terms of Type I and type II errors this becomes:

${\displaystyle F{\beta }={\frac {(1+\beta ^{2})\cdot \mathrm {TP} }{(1+\beta ^{2})\cdot \mathrm {TP} +\beta ^{2}\cdot \mathrm {FN} +\mathrm {FP} }}\,={\frac {(1+\beta ^{2})\cdot \mathrm {TP} }{(\mathrm {TP} +\mathrm {FN} )\cdot \beta ^{2}+(\mathrm {TP} +\mathrm {FP} )}}\,}$

Two commonly used values for ${\displaystyle \beta }$ are 2, which weighs recall higher than precision, and 1/2, which weighs recall lower than precision.

The F-measure was derived so that ${\displaystyle F{\beta }}$ "measures the effectiveness of retrieval with respect to a user who attaches ${\displaystyle \beta }$ times as much importance to recall as precision". It is based on Van Rijsbergen's effectiveness measure

${\displaystyle E=1-\left({\frac {\alpha }{p}}+{\frac {1-\alpha }{r}}\right)^{-1}}$

Their relationship is: ${\displaystyle F{\beta }=1-E}$ where ${\displaystyle \alpha ={\frac {1}{1+\beta ^{2}}}}$

Diagnostic testing

F-score

This is related to the field of binary classification where recall is often termed "sensitivity".

<table><thead><tr><th></th><th></th><th>Predicted condition</th><th>Sources:</th></tr></thead><tbody><tr><td>Total population = P + N</td><td>Predicted positive</td><td>Predicted negative</td><td>Informedness, bookmaker informedness (BM) = TPR + TNR − 1</td><td>Prevalence threshold (PT) = ⁠√TPR × FPR − FPR/TPR − FPR⁠</td></tr><tr><td>Actual condition</td><td>Real Positive (P)</td><td>True positive (TP), hit</td><td>False negative (FN), miss, underestimation</td><td>True positive rate (TPR), recall, sensitivity (SEN), probability of detection, hit rate, power = ⁠TP/P⁠ = 1 − FNR</td><td>False negative rate (FNR), miss rate type II error = ⁠FN/P⁠ = 1 − TPR</td></tr><tr><td>Real Negative (N)</td><td>False positive (FP), false alarm, overestimation</td><td>True negative (TN), correct rejection</td><td>False positive rate (FPR), probability of false alarm, fall-out type I error = ⁠FP/N⁠ = 1 − TNR</td><td>True negative rate (TNR), specificity (SPC), selectivity = ⁠TN/N⁠ = 1 − FPR</td></tr><tr><td></td><td>Prevalence = ⁠P/P + N⁠</td><td>Positive predictive value (PPV), precision = ⁠TP/TP + FP⁠ = 1 − FDR</td><td>False omission rate (FOR) = ⁠FN/TN + FN⁠ = 1 − NPV</td><td>Positive likelihood ratio (LR+) = ⁠TPR/FPR⁠</td><td>Negative likelihood ratio (LR−) = ⁠FNR/TNR⁠</td></tr><tr><td>Accuracy (ACC) = ⁠TP + TN/P + N⁠</td><td>False discovery rate (FDR) = ⁠FP/TP + FP⁠ = 1 − PPV</td><td>Negative predictive value (NPV) = ⁠TN/TN + FN⁠ = 1 − FOR</td><td>Markedness (MK), deltaP (Δp) = PPV + NPV − 1</td><td>Diagnostic odds ratio (DOR) = ⁠LR+/LR−⁠ = ⁠TP × TN/FP × FN⁠</td></tr><tr><td>Balanced accuracy (BA) = ⁠TPR + TNR/2⁠</td><td>F1 score = ⁠2 PPV × TPR/PPV + TPR⁠ = ⁠2 TP/2 TP + FP + FN⁠</td><td>Fowlkes–Mallows index (FM) = √PPV × TPR</td><td>phi or Matthews correlation coefficient (MCC) = √TPR × TNR × PPV × NPV - √FNR × FPR × FOR × FDR</td><td>Threat score (TS), critical success index (CSI), Jaccard index = ⁠TP/TP + FN + FP⁠</td></tr></tbody></table>

Normalised harmonic mean plot where x is precision, y is recall and the vertical axis is F1 score, in percentage points

Precision-Recall Curve: points from different thresholds are color coded, the point with optimal F-score is highlighted in red

Dependence of the F-score on class imbalance

F-score

Precision-recall curve, and thus the ${\displaystyle F{\beta }}$ score, explicitly depends on the ratio ${\displaystyle r}$ of positive to negative test cases. This means that comparison of the F-score across different problems with differing class ratios is problematic. One way to address this issue (see e.g., Siblini et al., 2020) is to use a standard class ratio ${\displaystyle r{0}}$ when making such comparisons.

Applications

F-score

The F-score is often used in the field of information retrieval for measuring search, document classification, and query classification performance. It is particularly relevant in applications which are primarily concerned with the positive class and where the positive class is rare relative to the negative class.

Earlier works focused primarily on the F1 score, but with the proliferation of large scale search engines, performance goals changed to place more emphasis on either precision or recall and so ${\displaystyle F{\beta }}$ is seen in wide application.

The F-score is also used in machine learning. However, the F-measures do not take true negatives into account, hence measures such as the Matthews correlation coefficient, Informedness or Cohen's kappa may be preferred to assess the performance of a binary classifier.

The F-score has been widely used in the natural language processing literature, such as in the evaluation of named entity recognition and word segmentation.

Properties

F-score

The F1 score is the Dice coefficient of the set of retrieved items and the set of relevant items.

• The F1-score of a classifier which always predicts the positive class converges to 1 as the probability of the positive class increases. • The F1-score of a classifier which always predicts the positive class is equal to 2 proportionofpositiveclass / ( 1 + proportionofpositiveclass ), since the recall is 1, and the precision is equal to the proportion of the positive class. • If the scoring model is uninformative (cannot distinguish between the positive and negative class) then the optimal threshold is 0 so that the positive class is always predicted. • F1 score is concave in the true positive rate.

Criticism

F-score

David Hand and others criticize the widespread use of the F1 score since it gives equal importance to precision and recall. In practice, different types of mis-classifications incur different costs. In other words, the relative importance of precision and recall is an aspect of the problem.

According to Davide Chicco and Giuseppe Jurman, the F1 score is less truthful and informative than the Matthews correlation coefficient (MCC) in binary evaluation classification.

David M W Powers has pointed out that F1 ignores the True Negatives and thus is misleading for unbalanced classes, while kappa and correlation measures are symmetric and assess both directions of predictability - the classifier predicting the true class and the true class predicting the classifier prediction, proposing separate multiclass measures Informedness and Markedness for the two directions, noting that their geometric mean is correlation.

Another source of critique of F1 is its lack of symmetry. It means it may change its value when dataset labeling is changed - the "positive" samples are named "negative" and vice versa. This criticism is met by the P4 metric definition, which is sometimes indicated as a symmetrical extension of F1.

Finally, Ferrer and Dyrland et al. argue that the expected cost (or its counterpart, the expected utility) is the only principled metric for evaluation of classification decisions, having various advantages over the F-score and the MCC. Both works show that the F-score can result in wrong conclusions about the absolute and relative quality of systems.

Difference from Fowlkes–Mallows index

F-score

While the F-measure is the harmonic mean of recall and precision, the Fowlkes–Mallows index is their geometric mean.

Extension to multi-class classification

F-score

The F-score is also used for evaluating classification problems with more than two classes (Multiclass classification). A common method is to average the F-score over each class, aiming at a balanced measurement of performance.

Macro F1

Macro F1 is a macro-averaged F1 score aiming at a balanced performance measurement. To calculate macro F1, two different averaging-formulas have been used: the F1 score of (arithmetic) class-wise precision and recall means or the arithmetic mean of class-wise F1 scores, where the latter exhibits more desirable properties.

Micro F1

Micro F1 is the harmonic mean of micro precision and micro recall. In single-label multi-class classification, micro precision equals micro recall, thus micro F1 is equal to both. However, contrary to a common misconception, micro F1 does not generally equal accuracy, because accuracy takes true negatives into account while micro F1 does not.