Image Preprocessing for Artistic Robotic Painting
<p>(<b>a</b>) The original image. It has a blue pale background due to aerial perspective. (<b>b</b>) The predominant directions of brushstrokes in some regions of the image. In these regions, the brushstrokes are highly coherent. (<b>c</b>) The hue histogram showing that it has two distinct peaks near complementary colors: orange <math display="inline"> <semantics> <mrow> <mi>H</mi> <mo>=</mo> <mn>0.1</mn> </mrow> </semantics> </math> and blue <math display="inline"> <semantics> <mrow> <mi>H</mi> <mo>=</mo> <mn>0.6</mn> </mrow> </semantics> </math>.</p> "> Figure 2
<p>(<b>a</b>) An example of highly enhanced aerial perspective in art: “Rain, Steam and Speed” by J.M.W. Turner, 1844. (<b>b</b>) A piece where aerial perspective is rendered through bluish gamut: “Paisaje con san Jerónimo” by Joachim Patinir, 1515–1519.</p> "> Figure 3
<p>Block diagram of the aerial perspective enhancement process.</p> "> Figure 4
<p>Generalization of the proposed hue modification function: regions with small slope provide histogram compression by the desired peaks <math display="inline"> <semantics> <msub> <mi>C</mi> <mn>1</mn> </msub> </semantics> </math> and <math display="inline"> <semantics> <msub> <mi>C</mi> <mn>2</mn> </msub> </semantics> </math>, while continuity of the function in regions with steep slope ensures continuity of the modified histogram. Here, Bezier spline was used for the illustration.</p> "> Figure 5
<p>Gamut correction methods: color harmonization by templates (<b>a</b>); and the proposed method of nonlinear compression (<b>b</b>).</p> "> Figure 6
<p>“Country road in Provence by night” by Vincent van Gogh, 1890. Painterly brushstrokes on the road follow its edges, while brushstrokes in the sky are parallel to edges of the star and the crescent.</p> "> Figure 7
<p>Edge field improvement by averaging: (<b>a</b>) a painterly rendered image using a smoothened edge field by the proposed approach; and (<b>b</b>) a rendered image with no averaging. Yellow frames highlight similar fragments of rendered images and show the underlying major eigenvectors used for brushstroke generation in each case.</p> "> Figure 8
<p>(<b>a</b>) The original image, which is apparently more “flat” than the processed one in (<b>b</b>).</p> "> Figure 9
<p>Aerial perspective enhancement of the source image (<b>a</b>) using the manually painted depth map (<b>b</b>) leads to a misty image (<b>c</b>), which is then rendered to obtain an etude-like composition (<b>d</b>) painted with a thick brush.</p> "> Figure 10
<p>The source image (<b>a</b>) was processed using all four stages: aerial perspective enhancement, gamut compression for achieving an effect of yellowed old paints, contrast-saturation correction and, finally, brushstroke rendering with improved coherence. The rendering result (<b>b</b>) represents a simulation of generated plotter file ready for robotic artistic painting.</p> "> Figure 11
<p>A slightly noisy source image (<b>a</b>) containing a number of monotonic areas can be rendered with well-coherent strokes generated with the technique of coherence enhancement (<b>b</b>). The rendered image (<b>c</b>) is then painted with the robot (<b>d</b>).</p> ">
Abstract
:1. Introduction
- Aerial perspective enhancement
- Gamut correction
- Averaging of the edge field extracted from the image for controlling brushstroke orientation
2. Aerial Perspective Enhancement
2.1. Related Work
2.2. Synthetic Aerial Perspective
3. Gamut Correction
3.1. Related Work
3.2. Continuous Gamut Correction
4. Brushstroke Coherence Control
4.1. Related Work
4.2. Coherence Enhancement by Averaging
Algorithm 1: Obtaining a smoothed edge field from an image |
input: a bitmap , parameters of matrices output: a vector field Fs // Load a bitmap I ← LoadImage ; // Find derivatives Fx ← ∗ I ; Fy ← ∗ I ; A ← Fx · Fx, B ← Fx · Fy, C ← Fy · Fy ; // Convolve with the Gaussian matrix and processing the tensor field A ← G ∗ A, B ← G ∗ B, C ← G ∗ C ; S ← GetTensorField (A, B, C) [U,V ] ← GetMajorEigenvectors (S); // Convolve with the averaging matrix U ← W ∗ U ; V ← W ∗ V ; // Obtain the final result Fs ← [U, V ]; |
5. Experimental Results
5.1. Experiments with Simulated Painting
5.2. Experiments with the Robotic Painting
6. Conclusions and Discussion
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Conflicts of Interest
References
- Cohen, H. How to Draw Three People in a Botanical Garden; AAAI: Menlo Park, CA, USA, 1988; Volume 89, pp. 846–855. [Google Scholar]
- Avila, L.; Bailey, M. Art in the digital age. IEEE Comput. Graph. Appl. 2016, 36, 6–7. [Google Scholar]
- Scalera, L.; Seriani, S.; Gasparetto, A.; Gallina, P. Non-photorealistic rendering techniques for artistic robotic painting. Robotics 2019, 8, 10. [Google Scholar] [CrossRef] [Green Version]
- Gülzow, J.; Grayver, L.; Deussen, O. Self-improving robotic brushstroke replication. Arts 2018, 7, 84. [Google Scholar] [CrossRef] [Green Version]
- Scalera, L.; Seriani, S.; Gasparetto, A.; Gallina, P. Watercolour robotic painting: A novel automatic system for artistic rendering. J. Intell. Robot. Syst. 2019, 95, 871–886. [Google Scholar] [CrossRef]
- Karimov, A.I.; Kopets, E.E.; Rybin, V.G.; Leonov, S.V.; Voroshilova, A.I.; Butusov, D.N. Advanced tone rendition technique for a painting robot. Robot. Auton. Syst. 2019, 115, 17–27. [Google Scholar] [CrossRef]
- Shiraishi, M.; Yamaguchi, Y. An algorithm for automatic painterly rendering based on local source image approximation. In Proceedings of the 1st International Symposium on Non-Photorealistic Animation and Rendering, Annecy, France, 5–7 June 2000; pp. 53–58. [Google Scholar]
- Litwinowicz, P. Processing images and video for an impressionist effect. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, USA, 4 August 1997; Citeseer: State College, PA, USA, 1997; pp. 407–414. [Google Scholar]
- Karimov, A.I.; Pesterev, D.O.; Ostrovskii, V.Y.; Butusov, D.N.; Kopets, E.E. Brushstroke rendering algorithm for a painting robot. In Proceedings of the 2017 International Conference “Quality Management, Transport and Information Security, Information Technologies” (IT&QM&IS), St. Petersburg, Rusia, 24–30 September 2017; pp. 331–334. [Google Scholar]
- Vanderhaeghe, D.; Collomosse, J. Stroke based painterly rendering. In Image and Video-Based Artistic Stylisation; Springer: London, UK, 2013; pp. 3–21. [Google Scholar]
- Zang, Y.; Huang, H.; Li, C.F. Artistic preprocessing for painterly rendering and image stylization. Vis. Comput. 2014, 30, 969–979. [Google Scholar] [CrossRef]
- Morovic, J.; Luo, M.R. The fundamentals of gamut mapping: A survey. J. Imaging Sci. Technol. 2001, 45, 283–290. [Google Scholar]
- Ames, A., Jr. Depth in pictorial art. Art Bull. 1925, 8, 5–24. [Google Scholar] [CrossRef]
- Kim, D.; Son, S.W.; Jeong, H. Large-scale quantitative analysis of painting arts. Sci. Rep. 2014, 4, 7370. [Google Scholar] [CrossRef] [Green Version]
- Zeng, K.; Zhao, M.; Xiong, C.; Zhu, S.C. From image parsing to painterly rendering. ACM Trans. Graph. 2009, 29, 2–11. [Google Scholar] [CrossRef]
- Cotte, P.; Dupraz, D. Spectral imaging of Leonardo Da Vinci’s Mona Lisa: A true color smile without the influence of aged varnish. In Proceedings of the Conference on Colour in Graphics, Imaging, and Vision. Society for Imaging Science and Technology, Leeds, UK, 19–22 June 2006; Volume 2006, pp. 311–317. [Google Scholar]
- Hertzmann, A. A survey of stroke-based rendering. IEEE Comput. Graph. Appl. 2003, 70–81. [Google Scholar] [CrossRef] [Green Version]
- Rosin, P.; Collomosse, J. Image and Video-Based Artistic Stylisation; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2012; Volume 42. [Google Scholar]
- Zhang, E.; Hays, J.; Turk, G. Interactive tensor field design and visualization on surfaces. IEEE Trans. Vis. Comput. Graph. 2006, 13, 94–107. [Google Scholar] [CrossRef] [Green Version]
- Igno-Rosario, O.; Hernandez-Aguilar, C.; Cruz-Orea, A.; Dominguez-Pacheco, A. Interactive system for painting artworks by regions using a robot. Robot. Auton. Syst. 2019, 121, 103263. [Google Scholar] [CrossRef]
- Messina, B.; Pascariello Ines, M. The architectural perspectives in the villa of Oplontis, a space over the real. In Proceedings of the Le vie dei Mercanti XIII International Forum, Aversa, Capri, Italy, 11–13 June 2015. [Google Scholar]
- He, K.; Sun, J.; Tang, X. Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 33, 2341–2353. [Google Scholar] [PubMed]
- Narasimhan, S.G.; Nayar, S.K. Contrast restoration of weather degraded images. IEEE Trans. Pattern Anal. Mach. Intell. 2003, 25, 713–724. [Google Scholar] [CrossRef] [Green Version]
- Morales, C.; Oishi, T.; Ikeuchi, K. Real-time rendering of aerial perspective effect based on turbidity estimation. IPSJ Trans. Comput. Vis. Appl. 2017, 9, 1. [Google Scholar] [CrossRef] [Green Version]
- Raskar, R.; Tan, K.H.; Feris, R.; Yu, J.; Turk, M. Non-photorealistic camera: Depth edge detection and stylized rendering using multi-flash imaging. ACM Trans. Graph. (TOG) 2004, 23, 679–688. [Google Scholar] [CrossRef]
- Hirschmuller, H.; Scharstein, D. Evaluation of cost functions for stereo matching. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; pp. 1–8. [Google Scholar]
- Eigen, D.; Puhrsch, C.; Fergus, R. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems; MIT Press: Cambridge, MA, USA, 2014; pp. 2366–2374. [Google Scholar]
- Peris, M.; Martull, S.; Maki, A.; Ohkawa, Y.; Fukui, K. Towards a simulation driven stereo vision system. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), Tsukuba, Japan, 11–15 November 2012; pp. 1038–1042. [Google Scholar]
- Cohen-Or, D.; Sorkine, O.; Gal, R.; Leyvand, T.; Xu, Y.Q. Color harmonization. ACM Trans. Graph. (TOG) 2006, 25, 624–630. [Google Scholar] [CrossRef]
- Hertzmann, A. Painterly rendering with curved brush strokes of multiple sizes. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, Orlando, FL, USA, 19–24 July 1998; ACM: New York, NY, USA, 1998; pp. 453–460. [Google Scholar]
- Hegde, S.; Gatzidis, C.; Tian, F. Painterly rendering techniques: A state-of-the-art review of current approaches. Comput. Animat. Virtual Worlds 2013, 24, 43–64. [Google Scholar] [CrossRef]
- Kyprianidis, J.E.; Kang, H. Image and video abstraction by coherence-enhancing filtering. In Computer Graphics Forum; Wiley Online Library: Hoboken, NJ, USA, 2011; Volume 30, pp. 593–602. [Google Scholar]
- Meier, B.J. Painterly rendering for animation. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New Orleans, LA, USA, 4–9 August 1996; Citeseer: State College, PA, USA, 1996; pp. 477–484. [Google Scholar]
- Umenhoffer, T.; Szirmay-Kalos, L.; Szécsi, L.; Lengyel, Z.; Marinov, G. An image-based method for animated stroke rendering. Vis. Comput. 2018, 34, 817–827. [Google Scholar] [CrossRef]
- Jähne, B.; Scharr, H.; Körkel, S. Principles of filter design. Handb. Comput. Vis. Appl. 1999, 2, 125–151. [Google Scholar]
- Li, J.; Yao, L.; Hendriks, E.; Wang, J.Z. Rhythmic brushstrokes distinguish van Gogh from his contemporaries: Findings via automated brushstroke extraction. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 34, 1159–1176. [Google Scholar]
- Karras, T.; Aittala, M.; Hellsten, J.; Laine, S.; Lehtinen, J.; Aila, T. Training generative adversarial networks with limited data. arXiv 2020, arXiv:2006.06676. [Google Scholar]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Karimov, A.; Kopets, E.; Kolev, G.; Leonov, S.; Scalera, L.; Butusov, D. Image Preprocessing for Artistic Robotic Painting. Inventions 2021, 6, 19. https://doi.org/10.3390/inventions6010019
Karimov A, Kopets E, Kolev G, Leonov S, Scalera L, Butusov D. Image Preprocessing for Artistic Robotic Painting. Inventions. 2021; 6(1):19. https://doi.org/10.3390/inventions6010019
Chicago/Turabian StyleKarimov, Artur, Ekaterina Kopets, Georgii Kolev, Sergey Leonov, Lorenzo Scalera, and Denis Butusov. 2021. "Image Preprocessing for Artistic Robotic Painting" Inventions 6, no. 1: 19. https://doi.org/10.3390/inventions6010019