u/Old-Shelter2517

Hi all,

I have been working on finetuning Llama3.2-1B on GSM8K for over a month. The best score I can get so far is 22.14 ( baseline is 6.07 evaluated with lm_eval on my server, few shot 8). I've tried adjusting hyperparameters like batchsize, learning rate, epochs, warm_up ratio, lr_scheduler.....

Since I am new in this field, I would like to know if there is anything I could do better. Or if this score is the ceiling of Llama3.2-1B.

I appreciate any comment or instruction, thanks!