Mixup

Mixup is a augmentation technique which works as follows

  1. Take an item from X say X1 (example an image)

  2. Choose a Random Weight (value say λ)

  3. Choose another item from X say X2 (example a different image randomly)

  4. Take the weighted average of item from Point 1 and item from Point 3 i.e. λ(X1) + (1-λ)(X2)

  5. Similarly take the weighted average of labels of the X1 and X2.

  6. This will help in regularization since each input to the Learner is a combination of more than one input.

  7. It will take more time to learn, but will generalize well.

Label Smoothing

  1. This method is to avoid overfitting and is for the Labels

  2. Labels are generally one-hot encoded and hence will be a 0s or 1s.

  3. This method is change them as follows

    1. Take a small value ε and replace 0s with with ε/N where N is the number of classes. If it is a classifier which classifies into 10 categories, N is 10.

    2. Replace 1s with 1 – ε+ ε/N

  4. For label smoothing, above points are only to understand how it works

  5. In practical scenarios, to implement this, all that is needed is an appropriate Loss function. LabelSmoothingCrossEntropy() in FastAI is such loss function.