Handling Class Imbalance and Long Tail Labels

Class imbalance is the norm in production text classification. A support ticket system might have 50 categories, but the top 5 account for 70 percent of volume while 30 categories each see less than 1 percent of traffic. Global accuracy or F1 metrics can look healthy, often 0.90 or higher, while rare label recall is near zero. A model that predicts the majority class for all rare labels still achieves high overall accuracy due to the skewed distribution. The solution is to report metrics per class and use macro averaging, which treats all classes equally regardless of frequency. Macro F1 averages the F1 score across all classes, so a class with 10 examples impacts the metric as much as a class with 10,000 examples. This exposes the true performance on rare labels. Additionally, compute confusion matrices per label to identify systematic errors, such as confusing adjacent categories in a hierarchy or conflating semantically similar labels. Per label thresholds are essential for automation decisions. A global threshold of 0.5 probability might work well for common labels but causes high false positive rates on rare labels because the model is poorly calibrated in low density regions of the embedding space. Calibrate thresholds per class on a validation set to hit business precision or recall targets. For example, a billing dispute category might require 0.95 precision to avoid incorrect automated refunds, so you set the threshold at 0.85 probability. A general inquiry category might tolerate 0.70 precision for higher recall and set the threshold at 0.30 probability. Cost sensitive training adjusts the loss function to penalize mistakes on rare classes more heavily. Assign class weights inversely proportional to frequency, so misclassifying a rare label with 100 examples costs 100 times more than misclassifying a common label with 10,000 examples. This encourages the model to allocate capacity to rare labels. The trade-off is potential overfitting on small classes and degradation on common classes if weights are too aggressive. Tune weights on a validation set while monitoring per class metrics. Active learning and human in the loop systems are critical for rare labels. When the model produces low confidence predictions, route items to human review and collect those labels for retraining. Prioritize annotation of items near decision boundaries or from underrepresented classes. A common pattern is to review 5 to 10 percent of predictions, focusing on items with probability between 0.3 and 0.7 or from labels with fewer than 500 training examples. This feedback loop improves rare label recall by 10 to 20 percentage points over quarterly retraining cycles. Hierarchical classification reduces confusion across distant labels and provides a natural way to handle rare labels. Predict coarse labels first, like technology versus healthcare, then predict fine grained labels within the coarse branch. This constrains the search space and reduces the chance of bizarre errors, like predicting a healthcare label for a clearly technical document. It also enables fallback: if the fine grained classifier is uncertain, return only the coarse label. Maintain confusion matrices per level and apply stricter thresholds at deeper levels to control error propagation.

Support ticket system with 50 categories: Top 5 account for 70% of volume, bottom 30 have <1% each. Global F1 is 0.91 but macro F1 is 0.68, exposing poor rare label performance. Apply per class thresholds and cost sensitive training to improve macro F1 to 0.79

E-commerce product taxonomy: Hierarchical classifier predicts department (electronics) at 0.95 confidence, then category (laptops) at 0.82 confidence. For confidence below 0.75 at fine level, return only coarse label to avoid bizarre misclassifications