Interactive Mesh Sculpting with Arbitrary Topologies in Head-Mounted VR Environments
<p>Two brush modes. The pink sphere in (<b>a</b>) is a spherical brush with its points selection strategy based on the Euclidean distance. The pink ray in (<b>b</b>) is a ray-shaped brush with its points selection strategy based on the geodesic distance. These brushes are oriented toward the negative direction of the Z-axis. Additionally, the green ray points toward the positive direction of the Y-axis, and the red ray points toward the positive direction of the X-axis.</p> "> Figure 2
<p>The process of dividing the octree structure and the schematic diagram of its structure.</p> "> Figure 3
<p>The geodesic distance and Euclidean distance between two points on the surface.</p> "> Figure 4
<p>Points selection scenarios for two brush modes. (<b>a</b>,<b>d</b>) is the origin dog model; the area in the red wireframe is the section we intend to sculpt. (<b>b</b>,<b>c</b>), respectively, correspond to the sculpting outcomes of the spherical brush and the ray-shaped brush on simple surfaces. (<b>e</b>,<b>f</b>), on the other hand, represent the sculpting results of the spherical brush and the ray-shaped brush in more complex area.</p> "> Figure 5
<p>Edge split. As the pink edge splits into two equal segments, the four created green edges are smaller than the largest initial edge.</p> "> Figure 6
<p>Edge collapse. The orange edge shorter than <span class="html-italic">d</span> will be collapsed, and the four pink edges will become two green edges.</p> "> Figure 7
<p>The illegal collapse operation. The collapse of the orange edge results in the original two adjacent green triangles becoming self-intersecting.</p> "> Figure 8
<p>Operation menu for switching between sculpting modes. (<b>a</b>) Pull. (<b>b</b>) Push. (<b>c</b>) Flatten. (<b>d</b>) Smooth.</p> "> Figure 9
<p>Pull and Push operations on spherical shape. (<b>a</b>) Original shape. (<b>b</b>) Pull (+) and Push (−).</p> "> Figure 10
<p>Position update in Laplacian smoothing. The gray points are the adjacent vertices of the yellow point, and the yellow point will be relocated to the position of the pink point.</p> "> Figure 11
<p>Different operations on the same part of the mesh. (<b>a</b>) shows the result obtained by the Pull operation, and (<b>b</b>) shows the result of the Push operation. While these two operations may appear similar to the union and difference operations in Boolean operations, the deformations are gradual processes that can be halted at any time to obtain intermediate results, unlike Boolean operations which directly yield final outcomes. (<b>c</b>) represents the result of the Flatten operation, and (<b>d</b>) shows the result of the Smooth operation.</p> "> Figure 12
<p>Topology fusion induced during the mesh deformation. (<b>a</b>) is the mesh before topology fusion, and (<b>c</b>) is the corresponding wireframe. (<b>b</b>) is the mesh before topology fusion, and (<b>d</b>) is the corresponding wireframe.</p> "> Figure 13
<p>Merge the neighborhoods of two non-adjacent vertices if the distance between them is less than the threshold <math display="inline"><semantics> <msub> <mi>d</mi> <mrow> <mi>t</mi> <mi>h</mi> <mi>i</mi> <mi>c</mi> <mi>k</mi> <mi>n</mi> <mi>e</mi> <mi>s</mi> <mi>s</mi> </mrow> </msub> </semantics></math>.</p> "> Figure 14
<p>The interface of two controllers. (<b>a</b>) is the left-hand controller operation interface, and (<b>b</b>) is the right-hand controller operation interface. (<b>c</b>,<b>d</b>) denote the left-hand and right-hand touchpad partitions, respectively.</p> "> Figure 15
<p>The case of mesh with and without the topological auto-fusion mechanism. (<b>a</b>) is the case where intersections appear as can be seen from the inside of the mesh without the topology auto-fusion mechanism. We can see that the outer mesh appears misaligned in (<b>b</b>), and the zoomed-in case can be seen in (<b>c</b>). Correspondingly, (<b>d</b>) corresponds to the case of the interior of the mesh with the topological auto-fusion mechanism. As can be seen in (<b>e</b>), the outer mesh maintains good properties, and the zoomed-in case can be seen in (<b>f</b>).</p> "> Figure 16
<p>Illegal and legal model adding operations. (<b>a</b>) presents the illegal operation, and the resulting broken mesh is shown in (<b>b</b>). (<b>c</b>) shows the legal operation, and it does not affect the subsequent sculpting as shown in (<b>d</b>).</p> "> Figure 17
<p>Scores of the two compared systems in six evaluation dimensions, including system ease of use, functional completeness, modeling robustness, detail implementation, topological freedom, and overall evaluation.</p> "> Figure 18
<p>Models created by novice users based on existing models. (<b>a</b>,<b>b</b>) are origin models. (<b>c</b>) is a rabbit with horns based on (<b>a</b>), and (<b>d</b>) is a dog with wings and collar based on (<b>b</b>).</p> "> Figure 19
<p>Models created by novice users from scratch. (<b>a</b>) is a monster head, and (<b>b</b>) is a fantasy arthropod.</p> "> Figure 20
<p>Sculpted colored models. (<b>a</b>) is a bunch of grapes, and (<b>b</b>) is a magic broom.</p> "> Figure 21
<p>Sculpting sequence of a vase model. (<b>a</b>) is the basic shape of the model. A ring is created with holes as shown in (<b>b</b>), and the connection between the ring and the model’s body is established in (<b>c</b>). In (<b>d</b>), the handles are added, and then we add the undulating motifs around the surface in (<b>e</b>). Finally, we sculpt some symmetrical textures for the visual impact as shown in (<b>f</b>).</p> "> Figure 22
<p>Three-dimensional (3D) printing model entities. (<b>a</b>) is the entity of vase, and (<b>b</b>) is the entity of fantasy arthropod.</p> ">
Abstract
:1. Introduction
- The concept of quasi-uniform mesh is extended to virtual reality sculpting systems. Through a series of meticulously designed algorithms, we achieve the capability to perform arbitrary topological deformations on mesh within a VR environment. Concurrently, the robustness of the mesh structure can be ensured throughout the sculpting process.
- Building on the foundation of achieving arbitrary topology and maintaining mesh robustness, we have integrated a suite of essential sculpting tools—pull, push, flatten, smooth, and paint—into our system. These tools are designed to cater to the diverse and universal sculpting needs of users, enabling a broad range of artistic and technical manipulations within the virtual environment.
- Considering the diversity of the user base, we have meticulously designed an intuitive operation interface for the sculpting system, ensuring it is accessible even to novices who have never engaged in sculpting before. This design significantly lowers the learning curve, allowing users to quickly become proficient with the system. The interface is especially user-friendly for beginners, facilitating an easy and efficient introduction to digital sculpting.
2. Related Work
2.1. Modeling Paradigm
2.2. Virtual Sculpting
3. System Implementation
3.1. Efficient Sculpting Region Selection
3.1.1. Octree-Accelerated Triangle-Vertices Query
3.1.2. Points Selection for Spherical Brush
3.1.3. Points Selection for Ray-Casting Brush
3.2. Mesh Optimization
- A tight mesh for a given closed manifold mesh M and a threshold , where M is considered tight if every edge in M is smaller than . This configuration has the advantage of possessing enough vertices to accurately reflect the geometry of the underlying surface with precision better than .
- A manifold mesh conforms to a minimum length d if it is derived by iterating over all the edges of the initial mesh M and collapsing those edges whose lengths are less than d.
3.2.1. Edge Split
3.2.2. Edge Collapse
3.3. Mesh Deformation
3.4. Topology Fusion
3.5. More Specific Advantages
3.5.1. Free Topology
- Ray Construction: Initialize from the center of the new spherical mesh, constructing a ray that extends in the positive z-axis direction.
- Region Search: Utilize the ray to search within the octree of the original mesh and identify the triangles in the impacted regions as candidate triangles.
- Triangle Traversal: Traverse through the candidate triangles and count how many are intersected by the ray.
- Legitimacy Check: If the ray intersects and odd number of triangles, this indicates that the new spherical mesh will be added inside the original mesh; if even, the new sphere will be added outside.
3.5.2. Symmetrical Sculpting
3.5.3. Vertex Coloring
3.5.4. Model Rotating Sculpting
4. Results and Discussion
4.1. User Interface
4.2. System Usability
4.3. Created Models
5. Conclusions and Future Work
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Abbreviations
XR | Extended Reality |
AR | Augmented Reality |
VR | Virtual Reality |
MR | Mixed Reality |
3D | three-dimensional |
2D | two-dimensional |
NURBS | Non-Uniform Rational B-Splines |
CAD | computer-aided design |
CPU | Central Processing Unit |
Appendix A. Usability Questionnaire
- Rate how easy it was for you to master the use of the system (1 = extremely difficult, 7 = extremely easy).
- Rate the completeness of the design functions in the system (1 = extremely incomplete, 7 = extremely complete).
- Rate the frequency of errors occurring during the sculpting process (1 = extremely high frequency, 7 = extremely low frequency).
- Rate how convenient it was for you to implement the model details (1 = extremely inconvenient, 7 = extremely convenient).
- Rate how freely you can alter the model’s topology (1 = extremely restricted, 7 = extremely free).
- Rate the overall system (1 = extremely poor, 7 = extremely good).
References
- Keefe, D.; Zeleznik, R.; Laidlaw, D. Drawing on air: Input techniques for controlled 3D line illustration. IEEE Trans. Vis. Comput. Graph. 2007, 13, 1067–1081. [Google Scholar] [CrossRef] [PubMed]
- Maxim, B.; Gorgan, D.; Moldoveanu, A.; Dix, A. ArtworkVR: Novel Interaction Techniques for Virtual Painter. In Proceedings of the RoCHI, Bucharest, Romania, 17–18 October 2019; pp. 30–37. [Google Scholar]
- Schkolne, S.; Pruett, M.; Schröder, P. Surface drawing: Creating organic 3D shapes with the hand and tangible tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Seattle, WA, USA, 31 March–5 April 2001; pp. 261–268. [Google Scholar]
- Medium. Available online: https://www.oculus.com/medium/ (accessed on 25 April 2024).
- Tilt Brush. Available online: https://www.tiltbrush.com/ (accessed on 25 April 2024).
- Vinod, T.; Kumaresan, N.; Gugan, I.; Dhanasekaran, S.; Ramprathap, K.; Chinnasamy, P. Online Automobile Rental and E-Marketplace with Augmented Reality (AR). In Proceedings of the 2022 International Conference on Advancements in Smart, Secure and Intelligent Computing (ASSIC), Bhubaneswar, India, 19–20 November 2022; IEEE: Piscataway, NJ, USA, 2022; pp. 1–5. [Google Scholar]
- Geiger, A.; Ziegler, J.; Stiller, C. Stereoscan: Dense 3d reconstruction in real-time. In Proceedings of the 2011 IEEE Intelligent Vehicles Symposium (IV), Baden-Baden, Germany, 5–9 June 2011; IEEE: Piscataway, NJ, USA, 2011; pp. 963–968. [Google Scholar]
- Izadi, S.; Kim, D.; Hilliges, O.; Molyneaux, D.; Newcombe, R.; Kohli, P.; Shotton, J.; Hodges, S.; Freeman, D.; Davison, A.; et al. Kinectfusion: Real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th Annual ACM Symposium on User interface Software and Technology, Santa Barbara, CA, USA, 16–19 October 2011; pp. 559–568. [Google Scholar]
- Mouragnon, E.; Lhuillier, M.; Dhome, M.; Dekeyser, F.; Sayd, P. Real time localization and 3d reconstruction. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), New York, NY, USA, 17–22 June 2006; IEEE: Piscataway, NJ, USA, 2006; Volume 1, pp. 363–370. [Google Scholar]
- Davies, R.H.; Cootes, T.F.; Twining, C.J.; Taylor, C.J. An Information Theoretic Approach to Statistical Shape Modelling. In Proceedings of the BMVC, Manchester, UK, 10–13 September 2001; Citeseer: Princeton, NJ, USA, 2001; pp. 1–10. [Google Scholar]
- Wang, H.; Markosian, L. Free-form sketch. In Proceedings of the 4th Eurographics Workshop on Sketch-Based Interfaces and Modeling, Riverside, CA, USA, 2–3 August 2007; pp. 53–58. [Google Scholar]
- Yu, E.; Arora, R.; Baerentzen, J.A.; Singh, K.; Bousseau, A. Piecewise-smooth surface fitting onto unstructured 3D sketches. ACM Trans. Graph. (TOG) 2022, 41, 1–16. [Google Scholar] [CrossRef]
- Mikaeili, A.; Perel, O.; Safaee, M.; Cohen-Or, D.; Mahdavi-Amiri, A. Sked: Sketch-guided text-based 3d editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Paris, France, 2–6 October 2023; pp. 14607–14619. [Google Scholar]
- Lamousin, H.J.; Waggenspack, N. NURBS-based free-form deformations. IEEE Comput. Graph. Appl. 1994, 14, 59–65. [Google Scholar] [CrossRef]
- Wong, J.P.; Lau, R.W.; Ma, L. Virtual 3d sculpting. J. Vis. Comput. Animat. 2000, 11, 155–166. [Google Scholar] [CrossRef]
- McDonnell, K.T.; Qin, H.; Wlodarczyk, R.A. Virtual clay: A real-time sculpting system with haptic toolkits. In Proceedings of the 2001 Symposium on Interactive 3D Graphics, Chapel Hill, NC, USA, 26–29 March 2001; pp. 179–190. [Google Scholar]
- Keefe, D.F.; Feliz, D.A.; Moscovich, T.; Laidlaw, D.H.; LaViola, J.J., Jr. CavePainting: A fully immersive 3D artistic medium and interactive experience. In Proceedings of the 2001 Symposium on Interactive 3D Graphics, Chapel Hill, NC, USA, 26–29 March 2001; pp. 85–93. [Google Scholar]
- Solidworks. Available online: https://www.solidworks.com/ (accessed on 25 April 2024).
- Inventor. Available online: https://www.autodesk.com/products/inventor/overview (accessed on 25 April 2024).
- Rhino. Available online: https://www.rhino3d.com (accessed on 25 April 2024).
- Maya. Available online: https://www.autodesk.com.tw/products/maya/overview (accessed on 25 April 2024).
- 3ds Max. Available online: https://www.autodesk.com.cn/products/3ds-max/overview (accessed on 11 June 2024).
- Blender. Available online: https://www.blender.org/ (accessed on 11 June 2024).
- Cinema 4D. Available online: https://www.maxon.net/cinema-4d (accessed on 25 April 2024).
- Stanculescu, L.; Chaine, R.; Cani, M.P. Freestyle: Sculpting meshes with self-adaptive topology. Comput. Graph. 2011, 35, 614–622. [Google Scholar] [CrossRef]
- Galyean, T.A.; Hughes, J.F. Sculpting: An interactive volumetric modeling technique. ACM SIGGRAPH Comput. Graph. 1991, 25, 267–274. [Google Scholar] [CrossRef]
- Lorensen, W.E.; Cline, H.E. Marching cubes: A high resolution 3D surface construction algorithm. In Seminal Graphics: Pioneering Efforts That Shaped the Field; Association for Computing Machinery: New York, NY, USA, 1998; pp. 347–353. [Google Scholar]
- Ferley, E.; Cani, M.P.; Gascuel, J.D. Practical volumetric sculpting. Vis. Comput. 2000, 16, 469–480. [Google Scholar] [CrossRef]
- Perng, K.L.; Wang, W.T.; Flanagan, M.; Ouhyoung, M. A real-time 3D virtual sculpting tool based on modified marching cubes. In Proceedings of the International Conference on Artificial Reality and Teleexistence, Quito, Ecuador, 27–29 October 2001; pp. 64–72. [Google Scholar]
- Ferley, E.; Cani, M.P.; Gascuel, J.D. Resolution adaptive volume sculpting. Graph. Models 2001, 63, 459–478. [Google Scholar] [CrossRef]
- Chen, C.W.; Hu, M.C.; Chu, W.T.; Chen, J.C. A real-time sculpting and terrain generation system for interactive content creation. IEEE Access 2021, 9, 114914–114928. [Google Scholar] [CrossRef]
- ZBrush. Available online: https://pixologic.com/ (accessed on 25 April 2024).
- Geomagic Freeform. Available online: https://www.artec3d.cn/3d-software/geomagic-freeform (accessed on 11 June 2024).
- De Goes, F.; James, D.L. Regularized kelvinlets: Sculpting brushes based on fundamental solutions of elasticity. ACM Trans. Graph. (TOG) 2017, 36, 1–11. [Google Scholar] [CrossRef]
- Jang, S.A.; Kim, H.i.; Woo, W.; Wakefield, G. Airsculpt: A wearable augmented reality 3d sculpting system. In Proceedings of the Distributed, Ambient, and Pervasive Interactions: Second International Conference, DAPI 2014, Held as Part of HCI Interational 2014, Heraklion, Crete, Greece, 22–27 June 2014. Proceedings 2; Springer: Berlin/Heidelberg, Germany, 2014; pp. 130–141. [Google Scholar]
- Lu, P.; Sheng, B.; Luo, S.; Jia, X.; Wu, W. Image-based non-photorealistic rendering for realtime virtual sculpting. Multimed. Tools Appl. 2015, 74, 9697–9714. [Google Scholar] [CrossRef]
- Callens, E.; Danieau, F.; Costes, A.; Guillotel, P. A tangible surface for digital sculpting in virtual environments. In Proceedings of the Haptics: Science, Technology, and Applications: 11th International Conference, EuroHaptics 2018, Pisa, Italy, 13–16 June 2018, Proceedings, Part II 11; Springer: Berlin/Heidelberg, Germany, 2018; pp. 157–168. [Google Scholar]
- Milliez, A.; Wand, M.; Cani, M.P.; Seidel, H.P. Mutable elastic models for sculpting structured shapes. Comput. Graph. Forum 2013, 32, 21–30. [Google Scholar] [CrossRef]
- Gao, Z.; Li, J.; Wang, H.; Feng, G. Digiclay: An interactive installation for virtual pottery using motion sensing technology. In Proceedings of the 4th International Conference on Virtual Reality, Qingdao, China, 22–24 October 2018; pp. 126–132. [Google Scholar]
- Gao, Z.; Wang, H.; Feng, G.; Guo, F.; Lv, H.; Li, B. RealPot: An immersive virtual pottery system with handheld haptic devices. Multimed. Tools Appl. 2019, 78, 26569–26596. [Google Scholar] [CrossRef]
- Dashti, S.; Prakash, E.; Navarro-Newball, A.A.; Hussain, F.; Carroll, F. PotteryVR: Virtual reality pottery. Vis. Comput. 2022, 38, 4035–4055. [Google Scholar] [CrossRef]
- Zeidler, D.; McGinity, M. Bodylab: In virtuo sculpting, painting and performing of full-body avatars. Proc. ACM Comput. Graph. Interact. Tech. 2023, 6, 1–12. [Google Scholar] [CrossRef]
- Xiaoqiang, Z.; Tao, Y. Interactive sculpture modeling based on mesh deformation in HMD VR environment. J. Zhejiang Univ. (Eng. Sci.) 2018, 52, 599–604. [Google Scholar]
- Bresenham, J.E. Algorithm for computer control of a digital plotter. In Seminal Graphics: Pioneering Efforts That Shaped the Field; Association for Computing Machinery: New York, NY, USA, 1998; pp. 1–6. [Google Scholar]
- Shapelab. Available online: https://shapelabvr.com (accessed on 2 July 2024).
- Poole, B.; Jain, A.; Barron, J.T.; Mildenhall, B. Dreamfusion: Text-to-3d using 2d diffusion. arXiv 2022, arXiv:2209.14988. [Google Scholar]
- Jambon, C.; Kerbl, B.; Kopanas, G.; Diolatzis, S.; Leimkühler, T.; Drettakis, G. Nerfshop: Interactive editing of neural radiance fields. Proc. ACM Comput. Graph. Interact. Tech. 2023, 6, 1–21. [Google Scholar] [CrossRef]
- Ding, L.; Dong, S.; Huang, Z.; Wang, Z.; Zhang, Y.; Gong, K.; Xu, D.; Xue, T. Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D priors. arXiv 2023, arXiv:2312.04963. [Google Scholar]
- Lin, C.H.; Gao, J.; Tang, L.; Takikawa, T.; Zeng, X.; Huang, X.; Kreis, K.; Fidler, S.; Liu, M.Y.; Lin, T.Y. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada, 17–24 June 2023; pp. 300–309. [Google Scholar]
- He, Y.; Bai, Y.; Lin, M.; Sheng, J.; Hu, Y.; Wang, Q.; Wen, Y.H.; Liu, Y.J. Text-Image Conditioned Diffusion for Consistent Text-to-3D Generation. Comput. Aided Geom. Des. 2024, 111, 102292. [Google Scholar] [CrossRef]
- Wang, Z.; Lu, C.; Wang, Y.; Bao, F.; Li, C.; Su, H.; Zhu, J. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Adv. Neural Inf. Process. Syst. 2023, 36, 8406–8441. [Google Scholar]
Brush Mode | Sculpting Mode | Points Selection | Mesh Optimization | Mesh Deformation | Total |
---|---|---|---|---|---|
Spherical Brush | Pull | 0.092 ms | 0.133 ms | 0.0019 ms | 3.944 ms |
Push | 0.071 ms | 0.101 ms | 0.0019 ms | 3.993 ms | |
Flatten | 0.068 ms | 0.022 ms | 0.0028 ms | 3.643 ms | |
Smooth | 0.075 ms | 0.028 ms | 0.0054 ms | 3.782 ms | |
Ray-Shaped Brush | Pull | 0.796 ms | 0.085 ms | 0.0017 ms | 4.737 ms |
Push | 0.869 ms | 0.086 ms | 0.0016 ms | 4.692 ms | |
Flatten | 1.009 ms | 0.029 ms | 0.0028 ms | 4.897 ms | |
Smooth | 0.897 ms | 0.041 ms | 0.0062 ms | 4.732 ms |
System | System Ease of Use | Functional Completeness | Modeling Robustness | Detail Implementation | Topological Freedom | Overall Evaluation |
---|---|---|---|---|---|---|
The Proposed | ||||||
Shapelab | ||||||
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhu, X.; Yang, Y. Interactive Mesh Sculpting with Arbitrary Topologies in Head-Mounted VR Environments. Mathematics 2024, 12, 2428. https://doi.org/10.3390/math12152428
Zhu X, Yang Y. Interactive Mesh Sculpting with Arbitrary Topologies in Head-Mounted VR Environments. Mathematics. 2024; 12(15):2428. https://doi.org/10.3390/math12152428
Chicago/Turabian StyleZhu, Xiaoqiang, and Yifei Yang. 2024. "Interactive Mesh Sculpting with Arbitrary Topologies in Head-Mounted VR Environments" Mathematics 12, no. 15: 2428. https://doi.org/10.3390/math12152428