The pipeline/end_to_end/evaluate.py defines model by model = GPT2.load_model(), but did not move model to GPU, thus the inference is slow. Consider adding these steps to load_model(): 1, check is GPU available in current machine; 2, if yes, move model to GPU before returning. See example: https://stackoverflow.com/questions/77237818/how-to-load-a-huggingface-pretrained-transformer-model-directly-to-gpu
The pipeline/end_to_end/evaluate.py defines model by model = GPT2.load_model(), but did not move model to GPU, thus the inference is slow. Consider adding these steps to load_model(): 1, check is GPU available in current machine; 2, if yes, move model to GPU before returning. See example: https://stackoverflow.com/questions/77237818/how-to-load-a-huggingface-pretrained-transformer-model-directly-to-gpu