Notes on Loss Functions
A note on different loss functions.
Loss Function or Cost Function or Error Function
Background
Why Loss function
The purpose of loss function is to quantify the error between the output of an algorithm and the given target value.
Let’s say that I have 100 pieces of something and I want the algorithm to predict the number of pieces available. Here, 100 pieces are the ground truth or the target value. Now, if the algorithm predicts that there are only 90 pieces, there is a loss or error of 10 pieces.
Types of Loss Functions
- NCE
- InfoNCE
- NT-Xent
- Contrastive Loss
- Triplet Loss
InfoNCE (Information Noise Contrastive Estimation) loss uses positive examples and sampled negative examples to contrast a true data distribution with a noise distribution. InfoNCE uses categorical cross-entropy loss to identify the positive sample amongst a set of unrelated noise samples. InfoNCE is an extension of NCE loss that adds a concept of mutual information to the loss calculation. Mutual information measures the amount of information shared between two random variables. InfoNCE loss aims to maximize the mutual information between the representations of the true target and its context while minimizing the mutual info between the representations of the true target and the noise samples.
NT-XENT (Normalized Temperature scaled Cross entropy) loss is a variant of InfoNCE loss that introduces a temperature parameter to control the smoothness of the similarity distribution, which can help with training stability and convergence. The higher temperature (Tau) encourages a softer distribution, making it easier for the model to distinguish between positive and negative sample.
Thoughts:
Citation
@misc{kumar2024,
author = {{Chandan Kumar}},
title = {Notes on {Loss} {Functions}},
date = {2024-01-13},
langid = {en-GB}
}