Closed
Description
I was tinkering with the code and made the following change in line 977, main.cpp (as it seemed wrong to me):
from
if (embd.size() > params.n_batch) {
break;
}to
if (embd.size() >= params.n_batch) {
break;
}The model's (13B) outputs suddenly changed. Reverted changes and tried to play with the batch_size parameter, it really does affect the output.
Not sure if it's expected behaviour. As far as I understand it shouldn't be the case. A bug? Different batch sizes have different evaluation results (rounding error)?