My goal here is two-fold: 1) Performance: How much scaling we can actually get from all these extra lanes with reasonable development effort. Ideally, it should be 16x for single-precision. 2) Programmability: Contrasting SIMD way of thinking about parallel programs with SIMT (Single Instruction Multiple Threads), specifically CUDA. (SPMD is probably a better term, but I’ll stick with SIMT here)

Benchmark Problem

Finding a good problem for this is actually not that trivial. The number of problems that 1) can be meaningfully accelerated by SIMD and 2) quickly be explained for a blogpost is not very large. The issue is memory, which is often the bottleneck for interesting problems, but ideal SIMD speedup can only come from problems that are compute bound.

Here’s arguably the most well-known example people use to introduce SIMD, including an interestingly titled talk from CppNow that I recently found- “How to Leverage SIMD Intrinsics for Massive Slowdowns”:

void axpy_scalar(const float *a, const float *x, const float *b, float *out std::size_t n) {
    for (std::size_t i = 0; i < n; ++i) {
        out[i] = a[i] * x[i] + b[i];
    }
}

The video talks about how explicit vectorization using intrinsics can lead to a slowdown compared to auto vectorization, due to things like loop unrolling, ILP etc. The problem is, it’s just a bad, bad example to talk about SIMD at all, regardless of whether it’s explicit or auto-vectorized. Take a guess: if we completely disable vectorization here (i.e. force the compiler to use purely scalar code), how much of a slowdown will we see vs a good vectorized code on a 16-lane wide SIMD? 16x? 8x? 8%?

The answer, if it’s not obvious from my tone already:), is 8%. Here’s a (very) simplistic explanation why: for the time it takes to feed 16 bytes of data to CPU in above code (to process 1 element), it can execute 32 compute ops. But we only have 2 ops here. So the ALUs are already sitting idle most of the time even in the scalar code, SIMD is trying optimizing something that doesn’t really matter.

My testcase of choice here is K-Means algorithm- an unsupervised clustering algorithm. It’s a pretty simple algorithm- a Numpy version shouldn’t take more than 15-20 lines.

centroids = sample K points from dataset (K=8 throughout this post)

while centroids are not converged:
    for each sample in dataset:   // compute_labels()
        assign it to a cluster with "closest" centroid

    for each cluster:             // compute_centroids()
        choose a better centroid by averaging each sample

The dataset samples in this case is going to be pixels of an image, so K-Means will basically do image segmentation based on pixel values. This K-Means variant should be a pretty good algorithm to try the SIMD paradigm out. We have lots of computations relative to data movement (aka high arithmetic intensity). Memory access is predictable and linear. The two functions above exhibit two different parallel programming patterns, which will come in handy in evaluating the programmability aspects.

Original Image

Segmentation with 8 clusters

Baseline(s)

We have two baselines here: the pure scalar version and auto-vectorized code produced by two compilers: GCC 14.2 and Intel’s ICPX 2024.2. The scalar code should give us an idea of how much AVX-512 actually scales (ideally 16x for single-precision data), and the auto-vectorized version will tell us if the considerable pain of writing intrinsics was really worth the trouble.

The results are pretty interesting. We see clear improvement with both AVX2 and AVX512 variants from both compilers, but nowhere close to the 8x or 16x ideal scaling.

Let’s look at these runtimes from another angle by doing a quick speed-of-light analysis. Having a single, obvious bottleneck (compute in this case) makes our life a lot easier. The theoretical max for the CPU here (AMD EPYC 9654) in single-precision is 3.7 GHZ: i.e. 3.7 GFlops/sec. With AVX-512, we have 16 32-bit ops/cycle, so 16*3.7=59.2 GFlops/sec. As for actual computations, our image have about 5 million pixels, we fix K-means iteration number to 20, and in each iteration with 8 centroids, a pixel requires roughly 200 flops. So we have total 5 million * 200 * 20 = 20 GFlops of computation. Ideally, the program should take about 20/59.2= 337ms. The best compilers can do here is 1.4 seconds, a 4.2x slowdown.

There are quite a few nuances in this calculation (e.g. we’re ignoring FMA, using boosted speed etc). But this doesn’t change the overall picture: we have a program that should be highly SIMD-iable, and auto-vectorization won’t take us there. On to the world of intrinsics we go.

AVX-512

Our K-means have two main functions. The first one, compute_labels, finds the closest centroid for each pixel. This is an example of “embarrassingly parallel” pattern, meaning what we do for each pixel is fully independent of what’s going on in other pixels. Below is a snippet from the function:

float dx_norm = static_cast<float>(dx) * inv_width;
float dy_norm = static_cast<float>(dy) * inv_height;
float spatial_norm = (dx_norm*dx_norm + dy_norm*dy_norm)
spatial_norm /= 2.0f;

const float weight = 0.85f;
float dist = weight * color_norm 
dist += (1.0f - weight) * spatial_norm;

if(dist < best_dist){      
    best_dist = dist;      
    best_k = k;            
}
out_labels[i] = best_k;

__m512 dx_normv = _mm512_mul_ps(_mm512_cvtepi32_ps(dxv), _mm512_set1_ps(inv_width));
__m512 dy_normv = _mm512_mul_ps(_mm512_cvtepi32_ps(dyv), _mm512_set1_ps(inv_height));

dx_normv = _mm512_mul_ps(dx_normv, dx_normv);
__m512 spatial_normv = _mm512_fmadd_ps(dy_normv, dy_normv, dx_normv);
spatial_normv = _mm512_mul_ps(spatial_normv, _mm512_set1_ps(0.5));
                
spatial_normv = _mm512_mul_ps(spatial_normv, _mm512_set1_ps(1-weight));
__m512 distv = _mm512_fmadd_ps(color_normv, color_norm_weight, spatial_normv);

__mmask16 mask = _mm512_cmplt_ps_mask(distv, best_dist);
best_dist = _mm512_mask_mov_ps(best_dist, mask, distv);  
best_k = _mm512_mask_mov_epi32(best_k, mask, _mm512_set1_epi32(k));

_mm512_storeu_si512(out_ptr, best_k);

SIMD intrinsics doesn’t really leave a good first impression, does it?

Interestingly, CUDA (an example of SIMT) would look pretty much the same as scalar code in this case. For those who don’t know, SIMT exposes a more “scalar”-like interface. In this case, your code would define the work for just a single pixel, and hardware/compiler together does the job of parallelizing the for loop.

I don’t want to anthropomorphize, but SIMD looks “objectively” ugly compared to CUDA/regular CPP code, and this is after ignoring certain real-world issues like complication from supporting different archs, extra loop for holdovers etc. But it’s not just looks. GCC fails to auto-vectorize the scalar version because of that simple if condition in the snippet, whereas CUDA gracefully handles this. However you look at it, CUDA wins for this function hands down.

But before we move on, it’s worth asking how exactly CUDA abstracts away these verbose SIMD code, because behind the scene, it’s using pretty similar SIMD-like hardware. The if here is implemented by the warp scheduler, which implements the masking necessary to turn off threads that don’t take a branch. And there’s a lot going on behind the simple store operation in the end: out_labels[i] = best_k; (or any load). These two abstractions lead to two of the most well-known performance bugs in CUDA code: Warp divergence and Uncoalesced memory access. For example, if the data here wasn’t laid out so neatly in the memory, maybe because we have Array-of-(large) structs layout or data is truly random access, both CUDA warp and SIMD store operations will have to resort to basically a for loop. SIMD makes that explicit, CUDA hides it from the developers. This can have a dramatic performance impact, more on GPU than CPU.

I should also note while the SIMD code is bit too verbose for my taste, once the hard part, architecture and data-structure is done, I didn’t feel like writing it was more difficult than writing scalar code. It requires a bit more typing (or lots more googling), sure, but not necessarily more thinking.

The second function, compute_centroids is more interesting. This gathers all the pixels that has been assigned a similar label, and compute a new centroid (and color) for this group. Here’s what the main loop looks like:

for(int h=0; h<height; h++){
    for(int w=0; w<width; w++){
        int i = h*width+w;
        int k = cluster[i];
        sum_r[k] += R[i]; 
        count[k]++;       
    }
}

for(int h=0; h<height; h++){
    for(int w=0; w<width; w+=L){     
        iv = [0..15] + h*width+w     
        __m512i kv = cluster[iv];    
        sum_r[kv] += R[iv];  // CONFLICT!        
        count[kv] += 1;      // CONFLICT!         
    }
}

Note that unlike the last code, there is potential conflict between SIMT threads (or SIMD lanes) here, since all pixels with a similar label are incrementing a single centroid data. This is an important detail.

For both CUDA and SIMD, we can take an easy route here: atomicAdd with CUDA and a serial for loop over SIMD length with AVX-512. The performant route with AVX-512 would probably include the instruction vpconflictd, but I couldn’t really find any elegant way to use it. Documentation and code samples were surprisingly sparse for such a common pattern. After spending some time optimizing this, I ended up with a version that has a for loop over the centroids, along with masks and reduction instruction. Not elegant, but seems to work pretty well.

In contrast, CUDA allows us to progressively optimize this code by gradually adding more complexity, for example by doing warp level and block level synchronizations first before writing to device memory. This is the positive. The downside is that each of this optimization forces us to move out of the warm comfort of SIMT abstraction and reckon with one more GPU hardware implementation detail. The final optimized CUDA code would look very, very different from this scalar version. Overall, getting close to hardware limit felt much easier with SIMD than CUDA. Official docs from Nvidia recommends developers to use their libraries instead of hand-rolling these primitives.

Anyway, how does our final performance look with intrinsics?

Not bad! This is 7-8.5x better compared to scalar code- around half of the ideal scaling, but closer to what people observe for SIMD-friendly code in practice. This is also 4x faster than best auto-vectorized codes. Interestingly, the final runtime (344ms) is surprisingly close to the rough speed-of-light estimate we had before. That doesn’t mean we’re at 98% of theoretical upper bound, that estimate was pretty rough. But I’ve been doing this sort of analysis before starting CUDA development, and I have never gotten this close to the upper limit, either for real or toy exercises.

Auto-vectorization did lot worse than I anticipated (4x), so I dug a little deeper. The main difference came from the first compute_labels function. Here’s what the key loops look like

for(int h=0; h<height; h++)}
    for(int w=0; w<width; w++){
        int best_k = 0;
        float best_dist = 999999999.0;
        for(int k=0; k<K; k++){
            ...
            if(dist < best_dist){
                best_dist = dist;
                best_k = k;
            }
    ...

GCC got tripped up by the presence of the if the conditional, the generated assembly was fully scalar. ICPX did vectorize, hence the ~40% better runtime. But the real issue is that both tried to vectorize the inner loop over centroids, not over the pixels. We know this is not a great idea since number of centroids will be lot lower than number of pixels, but there’s no easy way to encode that info in regular C++.

Final Thoughts

I’m concluding this exercise with pretty positive impressions for AVX-512, both in programmability and performance. I was expecting it to throw up some roadblocks and it didn’t. I have seen it argued that SIMD failed to get good adoption partly because it lacked a good programming model. At least for this admittedly simple exercise, I don’t really see how moving from this explicit SIMD-type code to an ISPC or CUDA like SIMT model would be beneficial. In fact, for more complicated code, I believe reasoning about performance would get more complicated with those abstractions compared to low-level SIMD.

I can’t comment on ISPC because I don’t know it. But I do know CUDA, it has many virtues, but simplicity is not of them. CUDA architects never had a dogmatic loyalty to the elegance of the SIMT model, they happily exposed every ugly details of underlying hardware to the programmer if that allows a bit more performance. Writing any good CUDA program requires looking at it from the thread, warp, thread block- every level of this deep hierarchy. I think it’s interesting that two of the most widely used DSLs that AI community uses to program GPUs, Triton and Cutlass, has a “Tile-based” model that is much closer in spirit to the explicit SIMD than SIMT.

In CPU world there is a desire to shield programmers from those low-level details, but I think there are two interesting forces at play now-a-days that’ll change it soon. On one hand, Dennard Scaling (aka free lunch) is long gone, hardware landscape is getting increasingly fragmented and specialized out of necessity, software abstractions are getting leakier, forcing developers to be aware of the lowest levels of abstraction, hardware, for good performance. On the flip side, thanks to LLMs, actual generation of code is becoming almost zero-cost, pushing developer responsibility to higher levels of abstractions like architecture and design. I think explicit SIMD is pretty perfect for this era- low-level enough to fully utilize the hardware but high level enough to exploit many compiler optimizations (and for human review). This creates a possible workflow where we developers architect our program in a hardware-friendly way (e.g. SoA memory layout instead of AoS), write the hot loops in scalar version, and hand off the task of porting to explicit SIMD to LLMs, optionally with some domain knowledge (“Parallelize over pixel loops since K is small”). With the latest batch of LLM models, I believe this is doable for a “real-world” program, but I’ll leave that for a future post (For now, see Appendix 2).

Appendix

[1] see Matt Pharr’s famous The story of ispc series.)

[2] https://parallelprogrammer.substack.com/p/why-we-need-simd-the-real-reason

[3] https://lemire.me/blog/2025/08/09/why-do-we-even-need-simd-instructions/

[4] https://lemire.me/blog/2025/02/14/avx-512-gotcha-avoid-compressing-words-to-memory-with-amd-zen-4-processors/

[5] https://www.youtube.com/watch?v=GleC3SZ8gjU

Appendix 2: LLMs

So far, I’ve used LLM for mainly 3 purposes:

1. Look-up right AVX-512 instructions. (e.g. “AVX512 instruction to reduce fp32 with mask?”) The official intel doc isn’t beginner friendly at all.
2. Create the performance plots.
3. Look at all dumped assembly and highlight regions of interest.

Here, I tried porting the scalar code to explicit AVX-512 using Codex 5.2 and Opus 4.5. A single prompt was used to port both functions at once: “Port compute_labels() and compute_centroids() functions here to AVX-512 using intrinsics. Parallelize the loop over pixel width at compute_labels(), assume width will be divisible by 16. No of centroids (K) will be low (<20). Order of floating point ops doesn’t matter”.

I truly one-shotted this, didn’t have to play around with prompt, didn’t need to do any further prompt. No context was supplied except the scalar code. Both models generated correct code at first try. And here’s the performance result:

Most of these observations grew out of my work on a research project a few months ago. As I worked, I realized I was forming some pretty strong takes that I can’t really put in an academic paper. So here I am.

One of the key advantages of rocRAND is it is open-source. So let’s start at their GitHub repo first.

Design

Going through the README, one of the first things you notice is AMD actually offers two random number libraries: rocRAND and hipRAND, the latter being a thin client that chooses cuRAND or rocRAND depending on the platform. So, for today’s discussion, we’ll set aside hipRAND.

Next comes a list of random number generators implemented in the library. You won’t find a discussion about them here (or anywhere else for that matter), Just a list of names. Moving on, in the Requirements section, ROCm is listed as a dependency for AMD platforms, as expected. However, clicking on the ROCm link leads to the first 404 error on this page. To run this library on CPU, you need something referred to as “HIP-CPU”. This link thankfully works, and the tagline of its Github repo reads- “An implementation of HIP that works on CPUs, across OSes.”

Let’s pause for a moment. We’re not even halfway through the README and we have already seen 3 different platforms from AMD- ROCm, HIP, HIP-CPU. I really wonder about the necessity or the wisdom behind this fragmentation- splitting HIP in particular. A single standard or library like SYCL or Kokkos seems to support multiple hardware platforms just fine under one codebase. To me this felt like a half-hearted attempt to tick one more box in a head-to-head battle with (intel-supported) SYCL. And I say half-hearted because HIP-CPU has been under development for more than 3 years, last commit pushed 3 months ago, and this is the first paragraph of its README: “Please note the library is being actively developed, and is known to be incomplet; it might also be incorrekt and there could be a few bad bugs lurking.” Let’s return to our focus on rocRAND.

One of the key challenges in developing a parallel, reproducible random number library is ensuring statistical robustness. This might not matter for most users, but for applications like Brownian simulations, a weak generator can silently wreak havoc. Rigorous testing with standard, widely accepted statistical frameworks is crucial - something cuRAND of course does. However, I couldn’t find any discussion on this for rocRAND, aside from two self-written simple tests. There’s mention of a statistical test suite in the README, but again, that link leads to a 404 error.

It’s not looking great, but at this point, I found a feature that cuRAND doesn’t have, a Python API! It’s an interesting choice: to attach such a high-level language interface for such a low-level library. So let’s go to the documentation and see what’s it for, shall we?

Documentation

That’s it! That’s the entirety of the Python API documentation – and no, those headers aren’t clickable. This is it! (Update: the doc page has since been updated.)

So, that was a bonus feature. What about the C++ API documentation? well, it exists, but it’s hardly any different. The API reference is almost entirely just a dump of function docstrings, with same comment copy/pasted for all the functions. And this mindless copy/pasting has predictable result- you’ll find, for example, the “documentation” mention 64 bit int return type for a function while it actually returns 32-bit.

The Programming Guide again starts (and ends) with the list of generators, with only one piece of extra information here, whether a generator is for pseudo-random or quasi-random number generation. The next (and final) section is titled “Ordering”, and the very first sentence starts talking about “how results are ordered in global memory.” If you just thought- wait, what results? that’s a very valid response. You might eventually figure out they are talking about the host-side API that generates a buffer of random numbers on device. Being GPU, it uses multiple threads behind the scene, and ordering here refers to how to order the numbers coming out of each thread in the output buffer. They list 5 ways of doing it, after commenting how this choice impacts performance and reproducibility. Go on, read about them a little bit, you’ll soon discover a pretty interesting relationship between them. For the lazy among you, here’s a clue:

They are all the same! Of course, they don’t say that directly, it’s another little thing for you to figure out. (well, technically I can’t say “all” are same, because they don’t mention the fifth one anywhere else in the page.)

Frankly, this isn’t just bad documentation; this is horrendous. There is no attempt anywhere to introduce or explain anything: just data dumps and lists. You get the sense, once again, that this “documentation work” was another box for someone to tick, without any consideration paid to a potential user of the software.

But the code follows the same API as cuRAND. So someone familar with cuRAND will be able to manage eventually. Let’s look at how that code fares against cuRAND next.

Performance

I’ll start with a real-world benchmark, using a classic example of GPGPU programming: Ray tracing in one weekend in cuda (Github). For meaningful performance comparison of random number libraries, we need a program that uses random numbers beyond just the initialization phase. Ray tracer is a good example of that. Both libraries offer a variety of generators; for this test, I chose Philox.

4.03 seconds vs 5.5s- the raytracer with the rocRAND version is 37% slower. Remember this isn’t a micro-benchmark of just random number generation part, the timings are for whole program. With that in mind, I think this is a pretty substantial slowdown.

The benchmark was performed on an Nvidia V100 GPU. Is that fair? I think yes, especially since rocRAND’s developers claimed to have performance parity with cuRAND on Nvidia GPUs. But maybe cuRAND has some hardware-specific optimizations? I really don’t think that’s the case. Philox algorithm isn’t that complicated, it doesn’t really need any advanced GPU primitives. But don’t take just my word for it: our lab made a pretty simple implementation of Philox, (you can find it here), it is orders of magnitude smaller than rocRAND’s implementation in terms of LOC, yet it performs on par with CuRAND (4.09 seconds).

Still, it’s just one benchmark. I’m sure there are other hardware-software combinations where this performance gap disappears. But, just to ensure that the ray tracer isn’t some outlier, I wrote a pretty basic 2D brownian dynamics simulation code. The story is even worse here for rocRAND, 6.30 seconds vs cuRAND’s 4.23- a 48% slowdown.

Final Thoughts

After the ChatGPT phenomenon, there has recently been lots of focus on Nvidia’s “CUDA moat”. As we all watched the vast AI riches going almost exclusively to Nvidia thanks mostly to that moat, many assumed this will be a big wake-up call for AMD, their Carthage must be destroyed moment that radically alters their well-known laid-back attitude to software. There are hints of this shift in their recent events and press releases, and I hope this trend continues.

But in my little corner of HPC world, I’m yet to see any meaningful movement in that regard. And AMD needs to hurry up- as I wrote this article, I took a cursory glance at Intel’s documentation for SYCL (a competitor of HIP) on this topic- a clean, well-organized, professional site- as you’d expect.

Like many, I’m looking forward to a real showdown in the GPGPU space someday- I’m just not sure that will necessarily be between Nvidia and AMD.

We’ll restrict ourselves to “matrix-form” datasets in this post. While many of the points raised here will no doubt apply to deep learning, there are enough practical differences for it to require a separate post. Python code for all the experiments and figures can be found in this link.

Why You Should Care

There are two important reasons:

1. Label noise can significantly harm performance: Noise in a dataset can mainly be of two types: feature noise and label noise; and several research papers have pointed out that label noise usually is a lot more harmful than feature noise. Figure 1 illustrates the impact of (artificially introduced) 30% label noise on the classification boundary of LinearSVC on a simple, linearly separable, binary classification dataset. We’ll talk about the impact more deeply later on, so let’s move on to the second point.

2. Label noise is very widespread: Label noise can creep into your dataset in many ways. One possible source is automatic labeling. This approach often uses meta-information (i.e. info not directly present in feature vectors) to generate labels- for example using hashtags to label images or using commit logs to detect defective modules in a software repository etc. This saves both time and money compared to labeling by domain experts, especially while dealing with large datasets, at the expense of quality. In software engineering domain, it was discovered that one of the leading auto-labeling algorithms to detect bug introducing commits (SZZ) has quite high noise rate [2], putting a big question mark on years of research that relied on SZZ to produce defect classification datasets.

In fact, this trade-off between quality and cost springs up quite often. For an example particularly relevant to the moment- say we want to create a COVID-19 prediction dataset using demographic features. For collecting the labels, i.e. whether someone actually have COVID-19, so far we basically have two options- we can either use slow, expensive and accurate RT-PCR test, or we can use fast, cheap but error-prone rapid testing kits.

But human labelers, or even experts are not infallible. In the medical domain, radiologists with 10 years of experience make mistakes 16.1% of the times while classifying MRIs [1]. Amazon Mechanical Turk has emerged as a quite popular data-labeling medium, but its widely known to contain illegal bots (and sometimes lazy humans) randomly labeling stuff. In fact, it’s hard to imagine a sufficiently large dataset that doesn’t contain at least some level of label noise. It perhaps wouldn’t be an overstatement to say that at least a basic awareness of label noise is pretty important for any data scientist working with real-world datasets.

How Classifiers Respond to Label Noise

Noise in dataset label will hurt the performance of any classifier, which is expected- the interesting question is by how much. Turns out the answer depends quite heavily on the classifier being used to mine the dataset. To show this, we’re going to carry out a little experiment. We’ll use seven datasets to mitigate any dataset-specific bias- Iris, Breast Cancer, Vowel, Segment, Digits, Wine and Spambase. 5-fold cross-validation is repeated 3 times to compute the accuracy of a single classifier-dataset pair. At each iteration of cross-validation, we corrupt (i.e randomly flip) 20% labels of training data. Note that only the training dataset is corrupted with noise, original i.e. clean labels are used for evaluation.

As Figure 2 shows, performance of all classifiers get worse, which is expected. But there is a pretty huge variation among classifiers. Decision Tree (DT) appears to be extremely vulnerable to noise. All 4 ensembles examined here: Random Forest (RF), Extra Tree (Extra), XGBoost (XGB) and LightGBM (LGB), have broadly similar performance on original data. But XGBoost’s performance takes a comparatively bigger hit due to noise, RF on the other hand seems comparatively robust.

How Exactly Label Noise Harms Performance

Well, an obvious answer is that low-quality data results in low-quality models. But defining quality isn’t straight as forward as it seems. We might be tempted to say dataset with higher noise level is of lower quality, and intuitively that makes sense. But remember figure 1? here is the same one, but now with 4000 samples instead of 400. In both cases the datasets contain 30% label noise, so should be of similar quality. Yet in this case, the decision boundary learned with noisy data is indistinguishable from the one learned with clean data. This isn’t to say that the intuition is wrong (it isn’t), but to emphasize that when it comes to explaining the performance loss, there’s more to the story (e.g. dataset size) than simple noise level.

Besides, the data-centric perspective alone doesn’t explain the huge disparity among different classifiers’ response to noise. So next we’re going to analyze it from the perspective of classifiers. In the figure below, we take the most brittle among all classifiers- Decision Tree, train it with both clean and noisy labels of Iris dataset, and plot the structure of resulting trees below.

We’re only applying 20% noise here. But even this small noise is enough to turn a relatively small decision tree (left) into a gigantic mess (right). This is admittedly a contrived and extreme example, but it does reveal an important insight that more or less applies to all classifiers: label noise increases the model complexity, making the classifiers overfit to noise.

Another good algorithm to demonstrate the impact of noise is Adaboost, the predecessor of current state-of-the-arts like XGBoost and LightGBM. It was among state-of-the-arts in early 2000s, but it is also very vulnerable to label noise. Adaboost begins by assigning each instance equal weight. At each iteration, it increases the weight of the instances it misclassified, and reduces the weight of others. In this way, it gradually concentrates more on the harder instances. But as you might have imagined, mislabeled instances are usually harder to classify than clean ones. So Adaboost ends up assigning higher weights to mislabeled samples i.e. it tries hard to correctly classify instances it should ideally ignore. The animation in figure 5 captures the distribution of weights assigned to noisy and clean samples by Adaboost on Breast Cancer dataset with 20% noise. After only 20 iterations, noisy instances collectively have twice more weight than clean ones, despite being outnumbered 4 to 1.

To make this discussion complete, let’s take a look at some classifiers of the opposite spectrum: ones that are more robust to noise than others. If you have looked at figure 2 attentively, you’ve probably discovered a quite remarkable fact: that two of the most robust classifiers there (Random Forest and Extra Tree) are nothing but a simple collection of the most brittle algorithm: Decision Tree. To explain this, let’s start with something that these forests doesn’t do- they don’t put extra emphasis on noisy instances like Adaboost (or SVM), all instances are treated equally during bootstrap aggregating (or Bagging) [3], a vital component of random forest.

The explanation of how a bunch of poor decision trees (DT) can band together to form such powerful random forest lies in a concept called bias–variance decomposition of classification error. Imagine lots of DTs each with exactly 60% accuracy on a binary classification dataset. Assuming there is no correlation between their predictions, given a new instance, we can expect (approximately) 60% of DTs to make the right prediction on it. So if we aggregate their result by majority voting, for any new instance majority (i.e. 60%) will make the right prediction, giving us a perfect 100% accuracy! Of course, that assumption of zero correlation is impossible to achieve in practice, so we can’t go quite as far as 100, but hopefully, you get the point.

Handling Label Noise

Before we start talking about mitigating the impact of noise, please remember the famous No Free Lunch theorem. None of the methods discussed below are panacea- they sometimes work and sometimes don’t. And sometimes when they do work, the improvement might be insignificant compared to added computational cost. So always remember to compare your ML pipeline against a simple baseline without any of these noise handling mechanisms.

That being said, broadly speaking, we can attack the label noise problem from two angles: by using noise robust algorithms, or by cleaning our data. In the first approach, we can simply pick algorithms that are inherently more robust, for example, bagging-based ensembles over boosting. There have also been many algorithms and loss functions designed specifically to be noise-robust, for example unhinged SVM [4][5]. Or, using the fact that label noise leads to overfitting, we can usually make brittle algorithms more robust just by introducing stronger regularization, as figure 6 shows.

For cleaning data, we can use the previously stated observation that mislabeled instances are harder to correctly classify. In fact, a good number of ML papers rely on this observation to design new data cleaning procedures [6]. The basic steps are: train a bunch of classifiers using a subset of training data, predict the labels of the rest of the data using them, and then the percentage of classifiers that failed to correctly predict a sample’s given label is the probability that the sample is mislabeled. Although a link to full code has already been shared, a sample implementation using an ensemble of decision trees is so simple that I can’t resist showing it here:

def detect_noisy_samples(X,y,thres=.5): #Returns noisy indexes
    rf = RandomForestClassifier(oob_score=True).fit(X,y)
    noise_prob = 1 - rf.oob_decision_function_[range(len(y)),y]
    return noise_prob>thres

On Spambase dataset with 25% label noise, this method detects 85% of mislabeled instances, while only 13% of clean instances get detected as noisy. For Breast Cancer, these numbers are 90% and 10% respectively.

But this method of training multiple classifiers only to preprocess dataset might be impractical for big datasets. Another closely related but far more efficient heuristic is: 1) find the K nearest neighbors of a sample, 2) compute the percentage of those neighbors with similar label, 3) Use that as a proxy for label reliability. As expected, it’s performance can be somewhat less impressive- it detects 2/3rd of noisy instances in Spambase while 10% of clean instances gets labeled noisy. But once again, a basic implementation is incredibly simple.

def detect_noisy_samples(X,y,thres=.5): #Returns noisy indexes
    knn = KNeighborsClassifier().fit(X,y)
    noise_prob = 1 - knn.predict_proba(X)[range(len(X)),y]
    return np.argwhere(noise_prob>thres).reshape(-1)

It is worth emphasizing that cleaning data doesn’t have to mean simply throwing away suspected mislabeled samples. Both of the heuristics described above returns continuous probability of (a sample) being mislabeled. We can use the inverse of that probability as a sort of reliability or confidence score, and use cost-sensitive learning to utilize this information. In this way, we get to retain all data points, which is especially important when the noise level is high or dataset size is small. Plus, this is more general than filtering- filtering is a special instance of cost-sensitive approach where the cost can be only 0 and 1.

Conclusion

Thanks for staying this far! I hope this article has been useful. But please remember that this is simply an introduction, and therefore leaves out a lot of interesting and important questions.

For example, we haven’t really talked about “Noise Model” here. We only focused on the overall percentage of mislabeled samples, assuming the wrong label for a mislabeled instance can come from any of the other labels with equal probability. This is not unrealistic, this so-called uniform noise model can arise e.g. when an amazon bot randomly assigns labels. But we know from common sense that a serious human annotator is much more likely to confuse between say 9 and 7 than 9 and 8, or between positive and neutral sentiment than positive and negative sentiment- and the uniform noise model doesn’t quite capture this uneven interaction between labels.

Another interesting question is regarding how we should act when we haven’t yet collected labels: Do we collect a big amount of low-quality data? or do we collect a small quantity of high-quality data?

Anyway, I hope this introduction serves as a good starting point. I’m planning to address these left-out issues and several others in a series of articles in near future, so stay tuned.

References

[1] https://escholarship.org/uc/item/9zt9g3wt

[2] https://ieeexplore.ieee.org/document/8765743

[3] G., Yves. “Bagging equalizes influence.” Machine Learning, (2004)

[4] http://papers.nips.cc/paper/5941-learning-with-symmetric-label

[5] http://papers.nips.cc/paper/5073-learning-with-noisy-labels

[6] https://ieeexplore.ieee.org/abstract/document/6685834