add fast image processor nougat #37661
Conversation
|
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the |
There was a problem hiding this comment.
Hi @NahieliV , thanks for your contribution! Noted a few things to change :)
| def to_channels_first(image): | ||
| """ | ||
| Converts a NumPy image from channels-last (H, W, C) to channels-first (C, H, W) | ||
| if needed. Leaves PyTorch tensors and non-NumPy types unchanged. | ||
| """ | ||
| if isinstance(image, np.ndarray): | ||
| if image.ndim == 3 and image.shape[-1] in [1, 3]: | ||
| return torch.tensor(np.transpose(image, (2, 0, 1))) | ||
| return torch.tensor(image) | ||
| return image | ||
|
|
||
|
|
There was a problem hiding this comment.
this shouldn't be needed if you make it so that all input images are channels_first
There was a problem hiding this comment.
I removed this for the tests of the method align_long_axis, which indeed returns the channels first if the input image is channels first.
There was a problem hiding this comment.
However, there is a small bug in the original implementation for the method crop_margin when the following condition is met.
max_val = data.max()
min_val = data.min()
Even when we set data_format = "channels_first", the image is returned chanels last. This is because the image input format is inferred at the beginning, and then when to_pil_image is called, it's transformed into channels last. Then, when calling to_channel_dimension_format, data_format and input_data_format are the same so the function just returns the image.
I can raise an issue and fix it if you agree.
There was a problem hiding this comment.
I see, nice catch! If you could fix the issue in this PR that would be great. Thanks!
86305b0 to
9aa40ea
Compare
41e8fab to
ea91a91
Compare
|
@yonigozlan , changes are done. There are small differences due to the difference in the implementation of interpolation.BICUBIC from PIL and PyTorch. For the test |
There was a problem hiding this comment.
Hi @NahieliV ! Sorry for the delay, but thanks for iterating! Looks ready to merge to me :). Very last thing to do is to add a comment explaining why we have a larger than usual difference with slow processor, and overwrite the equivalence tests with a higher threshold.
|
|
||
| new_size = (height, width) | ||
|
|
||
| return F.resize(image, new_size, interpolation=F.InterpolationMode.BICUBIC) |
There was a problem hiding this comment.
Indeed the difference with the slow image processor probably comes from here, as in the slow processor, reducing_gap=2.0 is used. Could you add a comment just above this line explaining this issue?
There was a problem hiding this comment.
Should be good now.
ea91a91 to
b4dd0fe
Compare
There was a problem hiding this comment.
Thanks @NahieliV for iterating on this! I had to make some small changes mainly because of recent updates in Transformers, but LGTM! Waiting for the PR to be green then I'll merge
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
What does this PR do?
Adds fast image processor for Nougat model.
#36978
Before submitting
Pull Request section?
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@yonigozlan