K-nearest neighbors algorithm
In the vast landscape of machine learning, few algorithms are as intuitive and widely applicable as the K-nearest neighbors algorithm, or k-NN. It's a method that learns not by building a complex model, but by simply remembering all its data points and making predictions based on the 'company' a new data point keeps. Discover how this elegant simplicity allows k-NN to tackle everything from classifying images to predicting house prices. K-NN classifies new data by assigning it the most frequent class among its 'k' closest training examples. For regression tasks, k-NN predicts numerical values by averaging the values of its 'k' nearest neighbors. Despite its straightforward nature, k-NN's effectiveness hinges on careful selection of 'k', appropriate distance metrics, and managing high-dimensional data.
AI Summary
In the vast landscape of machine learning, few algorithms are as intuitive and widely applicable as the K-nearest neighbors algorithm, or k-NN. It's a method that learns not by building a complex model, but by simply remembering all its data points and making predictions based on the 'company' a new data point keeps. Discover how this elegant simplicity allows k-NN to tackle everything from classifying images to predicting house prices.
- K-NN classifies new data by assigning it the most frequent class among its 'k' closest training examples.
- For regression tasks, k-NN predicts numerical values by averaging the values of its 'k' nearest neighbors.
- Despite its straightforward nature, k-NN's effectiveness hinges on careful selection of 'k', appropriate distance metrics, and managing high-dimensional data.
What is K-Nearest Neighbors?
At its heart, the K-nearest neighbors algorithm, often simply called k-NN, is a non-parametric, supervised learning method. First conceived by Evelyn Fix and Joseph Hodges in 1951, it's a 'lazy learner' because it doesn't build a model during training. Instead, it just stores all available data, waiting until a prediction is needed.
How K-NN Classifies New Data
Imagine you have a new data point — say, a picture of an animal you need to identify. K-NN's classification process works like a 'plurality vote'. It looks at the 'k' closest known animal pictures (its neighbors) and assigns your new picture to the class that appears most frequently among them. If 'k' is 1, it simply takes on the identity of its single closest neighbor.
K-NN for Regression: Predicting Values
K-NN isn't just for classification; it can also predict numerical values. In what's known as k-NN regression, the algorithm calculates the average of the property values of its 'k' nearest neighbors. So, if you're predicting a house price, it might average the prices of the 'k' most similar houses nearby, a process sometimes called 'nearest neighbor smoothing'.
Measuring Closeness: The Distance Metric
The core idea of 'nearest neighbors' relies on defining what 'closeness' means. This is where distance metrics come in. For continuous variables, the Euclidean distance is a very common choice, measuring the straight-line distance between two points in a multi-dimensional space.
The Euclidean distance, shown here, calculates the distance between two points, 'p' and 'q', across 'n' different features. For discrete data, like text or categorical values, other metrics such as Hamming distance, which counts differing attributes, might be more suitable.
d(p,q) = \sqrt{\sum_{i=1}^{n} (p_i - q_i)^2}The Power of 'k'
The choice of 'k' — the number of neighbors — is crucial. A small 'k' (like 1) makes the model very sensitive to noise, potentially leading to overfitting. A large 'k', however, can smooth out the decision boundaries too much, potentially missing important local patterns in the data and leading to underfitting.
Weighted Contributions
Sometimes, not all neighbors should have an equal say. A useful technique is to assign weights to the contributions of neighbors, where closer neighbors have more influence than distant ones. A common method is to weight each neighbor inversely to its distance from the new data point.
This inverse distance weighting ensures that a neighbor right next door contributes more heavily to the prediction than one further away. It's a way to give more credence to the most relevant information.
Weight_i \propto \frac{1}{distance_i}Putting K-NN to Work
The 'Training' Phase: Just Storing Data
Unlike many other machine learning algorithms that construct a complex model during training, k-NN's training phase is remarkably simple. It merely involves storing all the feature vectors and their corresponding labels or values. All the 'computation' happens when you actually ask it to make a prediction.
Data Preparation is Key
Because k-NN relies on distance, the way your data is scaled is incredibly important. If one feature (like income) has a much larger range of values than another (like age), it can unfairly dominate the distance calculation. Normalizing or scaling your data—ensuring all features contribute equally—can dramatically improve accuracy.
Challenges and Solutions
The Curse of Dimensionality
One of the biggest hurdles for k-NN is dealing with high-dimensional data — situations where you have many features. In very high dimensions, the concept of 'distance' can become less meaningful, as all data points tend to appear almost equidistant from each other. This phenomenon is famously known as the 'curse of dimensionality'.
Feature Engineering & Dimension Reduction
To combat the curse of dimensionality, techniques like Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA) are often employed as pre-processing steps. These methods reduce the number of features while trying to preserve the most important information, making the data more manageable for k-NN.
Dealing with Skewed Data
A drawback of basic 'majority voting' in k-NN classification arises with skewed class distributions. If one class is far more common, its examples might dominate the 'k' nearest neighbors, leading to biased predictions. Weighting by distance, as we discussed, or using more abstract data representations can help mitigate this.
Choosing the Optimal 'k'
There's no magic number for 'k'; the best choice depends heavily on your specific dataset. Often, 'k' is chosen through trial and error, or using techniques like cross-validation. For binary classification problems, choosing an odd value for 'k' is a smart move — it cleverly avoids any ties in the voting process!
Metric Learning
Instead of using standard distance measures, researchers have developed 'metric learning' algorithms. These sophisticated methods actually learn the best distance function from the data itself, often significantly boosting k-NN's classification performance by finding a more meaningful way to define 'closeness'.
Under the Hood & Performance
Computational Considerations
The naive approach to k-NN involves calculating the distance from a new point to every single training example. For very large datasets, this can be computationally intensive. Fortunately, advanced data structures and approximate nearest neighbor search algorithms make k-NN feasible even for massive amounts of data by cleverly reducing the number of distances that need to be computed.
Reliability and Error Rates
Despite its simplicity, k-NN boasts strong consistency results. As the amount of data approaches infinity, the k-NN algorithm is guaranteed to yield an error rate no worse than twice the Bayes error rate — the theoretical minimum achievable error rate for a given data distribution. This mathematical robustness underscores its enduring utility in machine learning.
Data Reduction: Identifying Prototypes
Working with enormous datasets presents another challenge: often, not all data points are equally important for accurate classification. Data reduction techniques aim to identify 'prototypes'—key data points that efficiently represent the dataset. This can also involve detecting and removing 'class outliers'—training examples surrounded by examples of other classes—to improve the model's robustness.
Article
K-nearest neighbors algorithm
In statistics, the k-nearest neighbors algorithm (k-NN) is a non-parametric supervised learning method. It was first developed by Evelyn Fix and Joseph Hodges in 1951, and later expanded by Thomas Cover. Most often, it is used for classification, as a k-NN classifier, the output of which is a class membership. An object is classified by a plurality vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.
The k-NN algorithm can also be generalized for regression. In k-NN regression, also known as nearest neighbor smoothing, the output is the property value for the object. This value is the average of the values of k nearest neighbors. If k = 1, then the output is simply assigned to the value of that single nearest neighbor, also known as nearest neighbor interpolation.
For both classification and regression, a useful technique can be to assign weights to the contributions of the neighbors, so that nearer neighbors contribute more to the average than distant ones. For example, a common weighting scheme consists of giving each neighbor a weight of 1/d, where d is the distance to the neighbor.
The input consists of the k closest training examples in a data set. The neighbors are taken from a set of objects for which the class (for k-NN classification) or the object property value (for k-NN regression) is known. This can be thought of as the training set for the algorithm, though no explicit training step is required.
A peculiarity (sometimes even a disadvantage) of the k-NN algorithm is its sensitivity to the local structure of the data. In k-NN classification the function is only approximated locally and all computation is deferred until function evaluation. Since this algorithm relies on distance, if the features represent different physical units or come in vastly different scales, then feature-wise normalizing of the training data can greatly improve its accuracy.
Statistical setting
K-nearest neighbors algorithm
Suppose we have pairs ${\displaystyle (X{1},Y{1}),(X{2},Y{2}),\dots ,(X{n},Y{n})}$ taking values in ${\displaystyle \mathbb {R} ^{d}\times \{1,2\}}$, where Y is the class label of X, so that ${\displaystyle X|Y=r\sim P{r}}$ for ${\displaystyle r=1,2}$ (and probability distributions ${\displaystyle P{r}}$). Given some norm ${\displaystyle \|\cdot \|}$ on ${\displaystyle \mathbb {R} ^{d}}$ and a point ${\displaystyle x\in \mathbb {R} ^{d}}$, let ${\displaystyle (X{(1)},Y{(1)}),\dots ,(X{(n)},Y{(n)})}$ be a reordering of the training data such that ${\displaystyle \|X{(1)}-x\|\leq \dots \leq \|X{(n)}-x\|}$.
Algorithm
K-nearest neighbors algorithm
Example of k-NN classification. The test sample (green dot) should be classified either to blue squares or to red triangles. If k = 3 (solid line circle) it is assigned to the red triangles because there are 2 triangles and only 1 square inside the inner circle. If k = 5 (dashed line circle) it is assigned to the blue squares (3 squares vs. 2 triangles inside the outer circle).
The training examples are vectors in a multidimensional feature space, each with a class label. The training phase of the algorithm consists only of storing the feature vectors and class labels of the training samples.
In the classification phase, k is a user-defined constant, and an unlabeled vector (a query or test point) is classified by assigning the label which is most frequent among the k training samples nearest to that query point.
Application of a k-NN classifier considering k = 3 neighbors. Left - Given the test point "?", the algorithm seeks the 3 closest points in the training set, and adopts the majority vote to classify it as "class red". Right - By iteratively repeating the prediction over the whole feature space (X1, X2), one can depict the "decision surface".
A commonly used distance metric for continuous variables is Euclidean distance. For discrete variables, such as for text classification, another metric can be used, such as the overlap metric (or Hamming distance). In the context of gene expression microarray data, for example, k-NN has been employed with correlation coefficients, such as Pearson and Spearman, as a metric. Often, the classification accuracy of k-NN can be improved significantly if the distance metric is learned with specialized algorithms such as large margin nearest neighbor or neighborhood components analysis.
An animated visualization of k-means clustering with k = 3, grouping countries based on life expectancy, GDP, and happiness—demonstrating how k-NN operates in higher dimensions. Click to view the animation.
A drawback of the basic "majority voting" classification occurs when the class distribution is skewed. That is, examples of a more frequent class tend to dominate the prediction of the new example, because they tend to be common among the k nearest neighbors due to their large number. One way to overcome this problem is to weight the classification, taking into account the distance from the test point to each of its k nearest neighbors. The class (or value, in regression problems) of each of the k nearest points is multiplied by a weight proportional to the inverse of the distance from that point to the test point. Another way to overcome skew is by abstraction in data representation. For example, in a self-organizing map (SOM), each node is a representative (a center) of a cluster of similar points, regardless of their density in the original training data. k-NN can then be applied to the SOM.
Parameter selection
K-nearest neighbors algorithm
The best choice of k depends upon the data; generally, larger values of k reduces effect of the noise on the classification, but make boundaries between classes less distinct. A good k can be selected by various heuristic techniques (see hyperparameter optimization). The special case where the class is predicted to be the class of the closest training sample (i.e. when k = 1) is called the nearest neighbor algorithm.
The accuracy of the k-NN algorithm can be severely degraded by the presence of noisy or irrelevant features, or if the feature scales are not consistent with their importance. Much research effort has been put into selecting or scaling features to improve classification. A particularly popular approach is the use of evolutionary algorithms to optimize feature scaling. Another popular approach is to scale features by the mutual information of the training data with the training classes.
In binary (two class) classification problems, it is helpful to choose k to be an odd number as this avoids tied votes. One popular way of choosing the empirically optimal k in this setting is via bootstrap method.
The 1-nearest neighbor classifier
K-nearest neighbors algorithm
The most intuitive nearest neighbour type classifier is the one nearest neighbour classifier that assigns a point x to the class of its closest neighbour in the feature space, that is ${\displaystyle C{n}^{1nn}(x)=Y{(1)}}$.
As the size of training data set approaches infinity, the one nearest neighbour classifier guarantees an error rate of no worse than twice the Bayes error rate (the minimum achievable error rate given the distribution of the data).
The weighted nearest neighbour classifier
K-nearest neighbors algorithm
The k-nearest neighbour classifier can be viewed as assigning the k nearest neighbours a weight ${\displaystyle 1/k}$ and all others 0 weight. This can be generalised to weighted nearest neighbour classifiers. That is, where the ith nearest neighbour is assigned a weight ${\displaystyle w{ni}}$, with ${\textstyle \sum {i=1}^{n}w{ni}=1}$. An analogous result on the strong consistency of weighted nearest neighbour classifiers also holds.
Let ${\displaystyle C{n}^{wnn}}$ denote the weighted nearest classifier with weights ${\displaystyle \{w{ni}\}{i=1}^{n}}$. Subject to regularity conditions, which in asymptotic theory are conditional variables which require assumptions to differentiate among parameters with some criteria. On the class distributions the excess risk has the following asymptotic expansion ${\displaystyle {\mathcal {R}}{\mathcal {R}}(C{n}^{wnn})-{\mathcal {R}}{\mathcal {R}}(C^{\text{Bayes}})=\left(B{1}s{n}^{2}+B{2}t{n}^{2}\right)\{1+o(1)\},}$ for constants ${\displaystyle B{1}}$ and ${\displaystyle B{2}}$ where ${\displaystyle s{n}^{2}=\sum {i=1}^{n}w{ni}^{2}}$ and ${\displaystyle t{n}=n^{-2/d}\sum {i=1}^{n}w{ni}\left\{i^{1+2/d}-(i-1)^{1+2/d}\right\}}$.
The optimal weighting scheme ${\displaystyle \{w{ni}^{}\}{i=1}^{n}}$, that balances the two terms in the display above, is given as follows: set ${\displaystyle k^{}=\lfloor Bn^{\frac {4}{d+4}}\rfloor }$, ${\displaystyle w{ni}^{}={\frac {1}{k^{}}}\left[1+{\frac {d}{2}}-{\frac {d}{2{k^{}}^{2/d}}}\{i^{1+2/d}-(i-1)^{1+2/d}\}\right]}$ for ${\displaystyle i=1,2,\dots ,k^{}}$ and ${\displaystyle w{ni}^{}=0}$ for ${\displaystyle i=k^{}+1,\dots ,n}$.
With optimal weights the dominant term in the asymptotic expansion of the excess risk is ${\displaystyle {\mathcal {O}}(n^{-{\frac {4}{d+4}}})}$. Similar results are true when using a bagged nearest neighbour classifier.
Furthest-neighbor variants
K-nearest neighbors algorithm
A variation of the nearest neighbor approach uses the k furthest neighbors instead of the nearest ones. In this setting, neighbors are selected based on maximal dissimilarity rather than similarity. This approach has been proposed in recommender systems as an "inverted neighborhood" model, where the most dissimilar users are identified and their preferences are used inversely to generate recommendations. The goal is typically to increase diversity or novelty in recommended items, since traditional nearest-neighbor methods may favor popular or obvious items. Empirical evaluations have found that furthest-neighbor approaches generally achieve lower predictive accuracy than standard k-nearest neighbor methods, although user-perceived usefulness may be similar or higher in some cases.
Properties
K-nearest neighbors algorithm
k-NN is a special case of a variable-bandwidth, kernel density "balloon" estimator with a uniform kernel.
The naive version of the algorithm is easy to implement by computing the distances from the test example to all stored examples, but it is computationally intensive for large training sets. Using an approximate nearest neighbor search algorithm makes k-NN computationally tractable even for large data sets. Many nearest neighbor search algorithms have been proposed over the years; these generally seek to reduce the number of distance evaluations actually performed.
k-NN has some strong consistency results. As the amount of data approaches infinity, the two-class k-NN algorithm is guaranteed to yield an error rate no worse than twice the Bayes error rate (the minimum achievable error rate given the distribution of the data). Various improvements to the k-NN speed are possible by using proximity graphs.
For multi-class k-NN classification, Cover and Hart (1967) prove an upper bound error rate of ${\displaystyle R^{}\ \leq \ R{k\mathrm {NN} }\ \leq \ R^{}\left(2-{\frac {MR^{}}{M-1}}\right)}$ where ${\displaystyle R^{}}$ is the Bayes error rate (which is the minimal error rate possible), ${\displaystyle R{kNN}}$ is the asymptotic k-NN error rate, and M is the number of classes in the problem. This bound is tight in the sense that both the lower and upper bounds are achievable by some distribution. For ${\displaystyle M=2}$ and as the Bayesian error rate ${\displaystyle R^{}}$ approaches zero, this limit reduces to "not more than twice the Bayesian error rate".
Error rates
K-nearest neighbors algorithm
There are many results on the error rate of the k nearest neighbour classifiers. The k-nearest neighbour classifier is strongly (that is for any joint distribution on ${\displaystyle (X,Y)}$) consistent provided ${\displaystyle k:=k{n}}$ diverges and ${\displaystyle k{n}/n}$ converges to zero as ${\displaystyle n\to \infty }$.
Let ${\displaystyle C{n}^{knn}}$ denote the k nearest neighbour classifier based on a training set of size n. Under certain regularity conditions, the excess risk yields the following asymptotic expansion ${\displaystyle {\mathcal {R}}{\mathcal {R}}(C{n}^{knn})-{\mathcal {R}}{\mathcal {R}}(C^{\text{Bayes}})=\left\{B{1}{\frac {1}{k}}+B{2}\left({\frac {k}{n}}\right)^{4/d}\right\}\{1+o(1)\},}$ for some constants ${\displaystyle B{1}}$ and ${\displaystyle B{2}}$.
The choice ${\displaystyle k^{}=\left\lfloor Bn^{\frac {4}{d+4}}\right\rfloor }$ offers a trade off between the two terms in the above display, for which the ${\displaystyle k^{}}$-nearest neighbour error converges to the Bayes error at the optimal (minimax) rate ${\displaystyle {\mathcal {O}}\left(n^{-{\frac {4}{d+4}}}\right)}$.
Metric learning
K-nearest neighbors algorithm
The K-nearest neighbor classification performance can often be significantly improved through (supervised) metric learning. Popular algorithms are neighbourhood components analysis and large margin nearest neighbor. Supervised metric learning algorithms use the label information to learn a new metric or pseudo-metric.
Feature extraction
K-nearest neighbors algorithm
When the input data to an algorithm is too large to be processed and it is suspected to be redundant (e.g. the same measurement in both feet and meters) then the input data will be transformed into a reduced representation set of features (also named features vector). Transforming the input data into the set of features is called feature extraction. If the features extracted are carefully chosen it is expected that the features set will extract the relevant information from the input data in order to perform the desired task using this reduced representation instead of the full size input. Feature extraction is performed on raw data prior to applying k-NN algorithm on the transformed data in feature space.
An example of a typical computer vision computation pipeline for face recognition using k-NN including feature extraction and dimension reduction pre-processing steps (usually implemented with OpenCV):
• Haar face detection • Mean-shift tracking analysis • PCA or Fisher LDA projection into feature space, followed by k-NN classification
Dimension reduction
K-nearest neighbors algorithm
For high-dimensional data (e.g., with number of dimensions more than 10) dimension reduction is usually performed prior to applying the k-NN algorithm in order to avoid the effects of the curse of dimensionality.
The curse of dimensionality in the k-NN context basically means that Euclidean distance is unhelpful in high dimensions because all vectors are almost equidistant to the search query vector (imagine multiple points lying more or less on a circle with the query point at the center; the distance from the query to all data points in the search space is almost the same).
Feature extraction and dimension reduction can be combined in one step using principal component analysis (PCA), linear discriminant analysis (LDA), or canonical correlation analysis (CCA) techniques as a pre-processing step, followed by clustering by k-NN on feature vectors in reduced-dimension space. This process is also called low-dimensional embedding.
For very-high-dimensional datasets (e.g. when performing a similarity search on live video streams, DNA data or high-dimensional time series) running a fast approximate k-NN search using locality sensitive hashing, "random projections", "sketches" or other high-dimensional similarity search techniques from the VLDB toolbox might be the only feasible option.
Decision boundary
K-nearest neighbors algorithm
Nearest neighbor rules in effect implicitly compute the decision boundary. It is also possible to compute the decision boundary explicitly, and to do so efficiently, so that the computational complexity is a function of the boundary complexity.
Data reduction
K-nearest neighbors algorithm
Data reduction is one of the most important problems for work with huge data sets. Usually, only some of the data points are needed for accurate classification. Those data are called the prototypes and can be found as follows:
• Select the class-outliers, that is, training data that are classified incorrectly by k-NN (for a given k) • Separate the rest of the data into two sets: (i) the prototypes that are used for the classification decisions and (ii) the absorbed points that can be correctly classified by k-NN using prototypes. The absorbed points can then be removed from the training set.
Selection of class-outliers
A training example surrounded by examples of other classes is called a class outlier. Causes of class outliers include:
• random error • insufficient training examples of this class (an isolated example appears instead of a cluster) • missing important features (the classes are separated in other dimensions which we don't know) • too many training examples of other classes (unbalanced classes) that create a "hostile" background for the given small class
Class outliers with k-NN produce noise. They can be detected and separated for future analysis. Given two natural numbers, k>r>0, a training example is called a (k,r)NN class-outlier if its k nearest neighbors include more than r examples of other classes.
Condensed Nearest Neighbor for data reduction
Condensed nearest neighbor (CNN, the Hart algorithm) is an algorithm designed to reduce the data set for k-NN classification. It selects the set of prototypes U from the training data, such that 1NN with U can classify the examples almost as accurately as 1NN does with the whole data set.
Calculation of the border ratio
Three types of points: prototypes, class-outliers, and absorbed points.
• Scan all elements of X, looking for an element x whose nearest prototype from U has a different label than x. • Remove x from X and add it to U • Repeat the scan until no more prototypes are added to U.
Use U instead of X for classification. The examples that are not prototypes are called "absorbed" points.
It is efficient to scan the training examples in order of decreasing border ratio. The border ratio of a training example x is defined as
where ‖x-y‖ is the distance to the closest example y having a different color than x, and ‖x'-y‖ is the distance from y to its closest example x' with the same label as x.
The border ratio is in the interval [0,1] because ‖x'-y‖ never exceeds ‖x-y‖. This ordering gives preference to the borders of the classes for inclusion in the set of prototypes U. A point of a different label than x is called external to x. The calculation of the border ratio is illustrated by the figure on the right. The data points are labeled by colors: the initial point is x and its label is red. External points are blue and green. The closest to x external point is y. The closest to y red point is x' . The border ratio a(x) = ‖x'-y‖ / ‖x-y‖is the attribute of the initial point x.
Below is an illustration of CNN in a series of figures. There are three classes (red, green and blue). Fig. 1: initially there are 60 points in each class. Fig. 2 shows the 1NN classification map: each pixel is classified by 1NN using all the data. Fig. 3 shows the 5NN classification map. White areas correspond to the unclassified regions, where 5NN voting is tied (for example, if there are two green, two red and one blue points among 5 nearest neighbors). Fig. 4 shows the reduced data set. The crosses are the class-outliers selected by the (3,2)NN rule (all the three nearest neighbors of these instances belong to other classes); the squares are the prototypes, and the empty circles are the absorbed points. The left bottom corner shows the numbers of the class-outliers, prototypes and absorbed points for all three classes. The number of prototypes varies from 15% to 20% for different classes in this example. Fig. 5 shows that the 1NN classification map with the prototypes is very similar to that with the initial data set. The figures were produced using the Mirkes applet.
• CNN model reduction for k-NN classifiers • Fig. 1. The dataset. • Fig. 2. The 1NN classification map. • Fig. 3. The 5NN classification map. • Fig. 4. The CNN reduced dataset. • Fig. 5. The 1NN classification map based on the CNN extracted prototypes.
k-NN regression
K-nearest neighbors algorithm
In k-NN regression, also known as k-NN smoothing, the k-NN algorithm is used for estimating continuous variables. One such algorithm uses a weighted average of the k nearest neighbors, weighted by the inverse of their distance. This algorithm works as follows:
• Compute the Euclidean or Mahalanobis distance from the query example to the labeled examples. • Order the labeled examples by increasing distance. • Find a heuristically optimal number k of nearest neighbors, based on RMSE. This is done using cross validation. • Calculate an inverse distance weighted average with the k-nearest multivariate neighbors.
k-NN outlier
K-nearest neighbors algorithm
The distance to the kth nearest neighbor can also be seen as a local density estimate and thus is also a popular outlier score in anomaly detection. The larger the distance to the k-NN, the lower the local density, the more likely the query point is an outlier. Although quite simple, this outlier model, along with another classic data mining method, local outlier factor, works quite well also in comparison to more recent and more complex approaches, according to a large scale experimental analysis.
Validation of results
K-nearest neighbors algorithm
A confusion matrix or "matching matrix" is often used as a tool to validate the accuracy of k-NN classification. More robust statistical methods such as likelihood-ratio test can also be applied.