馃殌 Feature
When training on datasets with class imbalance, it is critical to have every subgroup represented in every batch. PyTorch's built-in WeightedRandomSampler allows us to assign weights to each subgroup but still relies on random chance, meaning it does not strictly guarantee that rare classes will appear in a specific batch.
For algorithms like Group Distributionally Robust Optimization (DRO) or Contrastive Learning, this is critical since if a group is not represented in a batch, the loss component can potentially become zero and can destabilize training.
If this makes sense, we also have to take care of this question:
- Discarding or Oversampling:
StratifiedBatchSampler can either oversample the group with the least datapoints or it can create the total number of batches such that it exhausts the subgroup with the least amount of datapoints, which leads to wastage of datapoints of highly-represented classes. We should discuss whether we want to keep both options or not, and which option would act as the default then.
I already have a working prototype of this from a recent project and can modify it for this.
馃殌 Feature
When training on datasets with class imbalance, it is critical to have every subgroup represented in every batch. PyTorch's built-in
WeightedRandomSamplerallows us to assign weights to each subgroup but still relies on random chance, meaning it does not strictly guarantee that rare classes will appear in a specific batch.For algorithms like Group Distributionally Robust Optimization (DRO) or Contrastive Learning, this is critical since if a group is not represented in a batch, the loss component can potentially become zero and can destabilize training.
If this makes sense, we also have to take care of this question:
StratifiedBatchSamplercan either oversample the group with the least datapoints or it can create the total number of batches such that it exhausts the subgroup with the least amount of datapoints, which leads to wastage of datapoints of highly-represented classes. We should discuss whether we want to keep both options or not, and which option would act as the default then.I already have a working prototype of this from a recent project and can modify it for this.