https://huggingface.co/collections/HuggingFaceTB/smolvlm2-smallest-video-lm-ever-67ab6b5e84bf8aaa60cb17c7
Tiny Image collection: https://huggingface.co/collections/HuggingFaceTB/smolvlm-256m-and-500m-6791fafc5bb0ab8acc960fb0
Codebase: https://github.com/huggingface/smollm\n","updatedAt":"2025-04-08T09:38:14.569Z","author":{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","fullname":"Andres Marafioti","name":"andito","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":483}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9228962659835815},"editors":["andito"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg"],"reactions":[{"reaction":"π","users":["orrzohar","AdinaY","onuralpszr","Aurelien-Morgan","iky1e","Inflammable1230"],"count":6},{"reaction":"π₯","users":["Jaward","Inflammable1230","pierrci"],"count":3}],"isReport":false}},{"id":"67f560f8612ec3067a466c36","author":{"_id":"65fc5dca9c557fcbc68a664e","avatarUrl":"/avatars/fbd2635ff79f4053f72b90db374f4b99.svg","fullname":"Nurmyrat","name":"Nurmyrat1998","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-04-08T17:46:32.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"ΠΡΠΈΠ²Π΅Ρ ","html":"` to the tokenizer, so these get unique tokens/embeddings in the tokenizer, and they are optimized during training. \nThis is different then using the naitive text tokenizer which: `` -> `<`,`row`,`_`,`{}`,`_`,`col`,`_`,`{}`,`>` so instead of treating these as on-learable string representations, we basically learn new tokens to represent the spatial position of each image patch.\n","html":"
Tiny Image collection: https://huggingface.co/collections/HuggingFaceTB/smolvlm-256m-and-500m-6791fafc5bb0ab8acc960fb0
Codebase: https://github.com/huggingface/smollm\n","updatedAt":"2025-04-08T09:38:14.569Z","author":{"_id":"65d66b494bbd0d92b641cdbb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg","fullname":"Andres Marafioti","name":"andito","type":"user","isPro":false,"isHf":true,"isHfAdmin":false,"isMod":false,"followerCount":483}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9228962659835815},"editors":["andito"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg"],"reactions":[{"reaction":"π","users":["orrzohar","AdinaY","onuralpszr","Aurelien-Morgan","iky1e","Inflammable1230"],"count":6},{"reaction":"π₯","users":["Jaward","Inflammable1230","pierrci"],"count":3}],"isReport":false}},{"id":"67f560f8612ec3067a466c36","author":{"_id":"65fc5dca9c557fcbc68a664e","avatarUrl":"/avatars/fbd2635ff79f4053f72b90db374f4b99.svg","fullname":"Nurmyrat","name":"Nurmyrat1998","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-04-08T17:46:32.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"ΠΡΠΈΠ²Π΅Ρ ","html":"
ΠΡΠΈΠ²Π΅Ρ
\n","updatedAt":"2025-04-08T17:46:32.520Z","author":{"_id":"65fc5dca9c557fcbc68a664e","avatarUrl":"/avatars/fbd2635ff79f4053f72b90db374f4b99.svg","fullname":"Nurmyrat","name":"Nurmyrat1998","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"ru","probability":0.9691303968429565},"editors":["Nurmyrat1998"],"editorAvatarUrls":["/avatars/fbd2635ff79f4053f72b90db374f4b99.svg"],"reactions":[{"reaction":"π","users":["parlorsky"],"count":1}],"isReport":false}},{"id":"67f56b4abd0e5f3893ae98e7","author":{"_id":"653a7d0732bd4db35d6f0067","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KSsNVRk4DwaTLJgoYwApj.jpeg","fullname":"Ihor VItenko","name":"Strongich","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-04-08T18:30:34.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Based on your findings in Section 3.4 that \"excessive CoT data harms compact model performance\", would you expect the same effect when CoT reasoning is learned through RL in compact VLMs? Or is this relationship not obviously transferable and would require separate testing?","html":"Based on your findings in Section 3.4 that \"excessive CoT data harms compact model performance\", would you expect the same effect when CoT reasoning is learned through RL in compact VLMs? Or is this relationship not obviously transferable and would require separate testing?
\n","updatedAt":"2025-04-08T18:30:34.571Z","author":{"_id":"653a7d0732bd4db35d6f0067","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KSsNVRk4DwaTLJgoYwApj.jpeg","fullname":"Ihor VItenko","name":"Strongich","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9726264476776123},"editors":["Strongich"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KSsNVRk4DwaTLJgoYwApj.jpeg"],"reactions":[],"isReport":false},"replies":[{"id":"67f57c6e42d840e0e712e69c","author":{"_id":"648c9605565e3a44f3c9bb7b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg","fullname":"Orr Zohar","name":"orrzohar","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":97},"createdAt":"2025-04-08T19:43:42.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"We did not try RL-style post-training, so it is hard to draw a definitive conclusion. However, larger LLMs (e.g., S1) can use SFT on CoT and succeed in 'distilling' such reasoning from CoT data. \n\nMy intuition is that these small models do not have that \"emergent\" property to learn to reason -- which is why CoT distillation was not helpful. ","html":"We did not try RL-style post-training, so it is hard to draw a definitive conclusion. However, larger LLMs (e.g., S1) can use SFT on CoT and succeed in 'distilling' such reasoning from CoT data.
\nMy intuition is that these small models do not have that \"emergent\" property to learn to reason -- which is why CoT distillation was not helpful.
\n","updatedAt":"2025-04-08T19:43:42.197Z","author":{"_id":"648c9605565e3a44f3c9bb7b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg","fullname":"Orr Zohar","name":"orrzohar","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":97}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9754781723022461},"editors":["orrzohar"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg"],"reactions":[{"reaction":"π","users":["Jaward","Strongich","Inflammable1230"],"count":3}],"isReport":false,"parentCommentId":"67f56b4abd0e5f3893ae98e7"}},{"id":"67f647d953800b079151c0c6","author":{"_id":"653a7d0732bd4db35d6f0067","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KSsNVRk4DwaTLJgoYwApj.jpeg","fullname":"Ihor VItenko","name":"Strongich","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false},"createdAt":"2025-04-09T10:11:37.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Great answer, much appreciated! I was reading through Section 3.1 about \"Learned Tokens\" and noticed you mentioned they work better than raw text tokens. Could you explain how these learned tokens actually work, or point me to a specific paper where I can learn more about this? I'd like to understand the mechanics behind them (: ","html":"Great answer, much appreciated! I was reading through Section 3.1 about \"Learned Tokens\" and noticed you mentioned they work better than raw text tokens. Could you explain how these learned tokens actually work, or point me to a specific paper where I can learn more about this? I'd like to understand the mechanics behind them (:
\n","updatedAt":"2025-04-09T10:11:37.722Z","author":{"_id":"653a7d0732bd4db35d6f0067","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KSsNVRk4DwaTLJgoYwApj.jpeg","fullname":"Ihor VItenko","name":"Strongich","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9805828332901001},"editors":["Strongich"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KSsNVRk4DwaTLJgoYwApj.jpeg"],"reactions":[],"isReport":false,"parentCommentId":"67f56b4abd0e5f3893ae98e7"}},{"id":"67f6da8881eb6175bfdca899","author":{"_id":"648c9605565e3a44f3c9bb7b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/648c9605565e3a44f3c9bb7b/W5chvk17Zol6-2QSWkFVR.jpeg","fullname":"Orr Zohar","name":"orrzohar","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":97},"createdAt":"2025-04-09T20:37:28.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"We add `We add <row_{}_col_{}> to the tokenizer, so these get unique tokens/embeddings in the tokenizer, and they are optimized during training.
This is different then using the naitive text tokenizer which: <row_{}_col_{}> -> <,row,_,{},_,col,_,{},> so instead of treating these as on-learable string representations, we basically learn new tokens to represent the spatial position of each image patch.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
\nThe following papers were recommended by the Semantic Scholar API
\n- \n
- Mobile-VideoGPT: Fast and Accurate Video Understanding Language Model (2025) \n
- Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI (2025) \n
- SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding (2025) \n
- Small Vision-Language Models: A Survey on Compact Architectures and Techniques (2025) \n
- FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression (2025) \n
- Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation (2025) \n
- Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection (2025) \n
Please give a thumbs up to this comment if you found it helpful!
\nIf you want recommendations for any Paper on Hugging Face checkout this Space
\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: \n\n@librarian-bot\n\t recommend
an audio overview (~17 minutes): https://youtu.be/FWkAxDz0-4g
by the way, the team's demo video is very impressive!
Unfortunately, smolvlm-256M-Instruct, 4 times of less params then internvl-1B, still slower then Internvl-v2-1B (event i enable flash attention, bf16) ?
\nIt seems less input token & dimension didn't contribute to the inference speed . Does anyone plz give me some suggestion about this issue ?
see github issue : https://github.com/huggingface/transformers/issues/41947
Andres Marafioti
Abstract
SmolVLM, a series of compact multimodal models, achieves high performance with minimal GPU memory usage, making efficient deployment on mobile and edge devices possible.
AI-generated summary
Large Vision-Language Models (VLMs) deliver exceptional performance but require significant computational resources, limiting their deployment on mobile and edge devices. Smaller VLMs typically mirror design choices of larger models, such as extensive image tokenization, leading to inefficient GPU memory usage and constrained practicality for on-device applications. We introduce SmolVLM, a series of compact multimodal models specifically engineered for resource-efficient inference. We systematically explore architectural configurations, tokenization strategies, and data curation optimized for low computational overhead. Through this, we identify key design choices that yield substantial performance gains on image and video tasks with minimal memory footprints. Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during inference and outperforms the 300-times larger Idefics-80B model, despite an 18-month development gap. Our largest model, at 2.2B parameters, rivals state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend beyond static images, demonstrating robust video comprehension capabilities. Our results emphasize that strategic architectural optimizations, aggressive yet efficient tokenization, and carefully curated training data significantly enhance multimodal performance, facilitating practical, energy-efficient deployments at significantly smaller scales.
Community
We are happy to introduce our tech report for SmolVLM :)
Video collection: https://huggingface.co/collections/HuggingFaceTB/smolvlm2-smallest-video-lm-ever-67ab6b5e84bf8aaa60cb17c7
Tiny Image collection: https://huggingface.co/collections/HuggingFaceTB/smolvlm-256m-and-500m-6791fafc5bb0ab8acc960fb0
Codebase: https://github.com/huggingface/smollm
ΠΡΠΈΠ²Π΅Ρ
Based on your findings in Section 3.4 that "excessive CoT data harms compact model performance", would you expect the same effect when CoT reasoning is learned through RL in compact VLMs? Or is this relationship not obviously transferable and would require separate testing?
We did not try RL-style post-training, so it is hard to draw a definitive conclusion. However, larger LLMs (e.g., S1) can use SFT on CoT and succeed in 'distilling' such reasoning from CoT data.
My intuition is that these small models do not have that "emergent" property to learn to reason -- which is why CoT distillation was not helpful.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Mobile-VideoGPT: Fast and Accurate Video Understanding Language Model (2025)
- Shakti-VLMs: Scalable Vision-Language Models for Enterprise AI (2025)
- SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding (2025)
- Small Vision-Language Models: A Survey on Compact Architectures and Techniques (2025)
- FCoT-VL:Advancing Text-oriented Large Vision-Language Models with Efficient Visual Token Compression (2025)
- Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation (2025)
- Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
an audio overview (~17 minutes): https://youtu.be/FWkAxDz0-4g
by the way, the team's demo video is very impressive!
Unfortunately, smolvlm-256M-Instruct, 4 times of less params then internvl-1B, still slower then Internvl-v2-1B (event i enable flash attention, bf16) ?
It seems less input token & dimension didn't contribute to the inference speed . Does anyone plz give me some suggestion about this issue ?
see github issue : https://github.com/huggingface/transformers/issues/41947
Models citing this paper 27
Browse 27 models citing this paperDatasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2504.05299 in a dataset README.md to link it from this page.