HistGradientBoostingClassifier slow in prediction mode

While HistGradientBoostingClassifier is 100 faster than GradientBoostingClassifier when fitting the model, I found it to be very slow in case of predicting the class probabilities, in my case about 100 times slower :-(

For example:
GradientBoostingClassifier: 3.2 min for training for 1 million examples. 32 ms for 1000 predictions.
HistGradientBoostingClassifier: 7s for training. 1s for 1000 predictions