CN101300602A - True sketch - Google Patents

Abstract

Systems and methods for sketching reality are described. In one aspect, a set of vector primitives is identified from a 2-D sketch. In one implementation, the 2-D sketch is hand-drawn by the user. The 2.5D geometric model is automatically generated from vector primitives. The 2.5D geometric model is automatically rendered and presented to the user. In one implementation, a user provides user input based on a 2-D sketch to modify one or more of lighting location, lighting direction, lighting intensity, texture, color, and geometry of a presentation.

Description

sketch real

Background technique

Achieving realism is a major goal of computer graphics, and many approaches ranging from physics-based modeling of light and geometry to image-based rendering (IBR) have been proposed. Photorealistic imagery has revolutionized the visualization process, allowing unbuilt scenes to be represented and explored. However, although professional software graphics applications such as AutoCAD can be used to create realistically drawn new content, these applications are still tedious and very time consuming. This problem is exacerbated when designing content. Two-and-a-half-dimensional (2.5D) modeling systems are generally inconvenient to use and thus not suitable for the early stages of design unless the design has been well planned. Also, the 2.5D modeling process requires expertise in object creation and applying textures to surfaces.

Contents of the invention

Systems and methods for sketching reality are described. In one aspect, a set of vector primitives is identified from the 2-D sketch. In one implementation, the 2-D sketch is drawn by hand by the user. 2.5D geometric models are automatically generated from vector primitives. Automatically draw and present 2.5D geometric models for users. In one implementation, the user provides 2-D sketch-based user input to modify one or more of: rendered lighting position, lighting direction, lighting intensity, texture, color, and geometry.

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

Description of drawings

Figure 1 illustrates an example system for sketching a reality, according to one embodiment.

Figure 2 shows an example overview of sketching real operations according to one implementation.

Figure 3 shows an exemplary set of sketches and image representations according to one embodiment.

Figure 4 exemplarily illustrates a building design according to one embodiment to show that lines can be sorted by line direction and vanishing point.

Fig. 5 shows an exemplary illustration of a distance function d(l, vp) between a line l and a vanishing point vp according to one embodiment.

Figure 6 illustrates a first set of exemplary line segment junctions, according to one embodiment.

FIG. 7 illustrates vertices identified by inputting a 2-D sketch and corresponding junction labels, according to one embodiment.

Figure 8 shows an exemplary set of vertex face assignments according to one embodiment.

Figure 9 shows four exemplary types of segmentation shapes/polygons according to one embodiment.

Figure 10 illustrates three exemplary types of shaded shapes, according to one embodiment.

Figure 11 shows another example of shaded shapes according to one embodiment.

FIG. 12 is an exemplary illustration showing regions having a common vertex that is not part of T-junctions belonging to the same object, according to one embodiment.

Figure 13 shows an exemplary set of generic building components for reconstructing ambiguous portions of a 2-D sketch, according to one embodiment.

FIG. 14 illustrates an exemplary sketch-based interactive user interface (UI) 1400 for photorealistic rendering from 2-D sketches, according to one embodiment.

FIG. 15 illustrates the exemplary sketch-based interactive user interface of FIG. 14 except that the exemplary rendering presented is non-photorealistic compared to the photorealistic rendering shown in FIG. 14 .

Figure 16 shows an exemplary illustration of specifying lighting direction and shadow intensity according to one embodiment.

Figure 17 illustrates exemplary designation of the boundaries of a shadow surface according to one embodiment.

Figure 18 illustrates an exemplary process for sketching a reality, according to one embodiment.

Detailed ways

Overview

Despite advances in modeling and drawing, designers still prefer freehand sketching for conceptual design. The appeal of sketching as an artistic medium lies in its low overhead in representing, probing, and communicating geometric ideas. Indeed, such conceptual representations are radically different in spirit and purpose from the medium in which the designer ultimately presents the design. A system and method for sketching realism will be described below with reference to FIGS. 1 to 18 , which provides a seamless way for users to move from freehand drawing of concept designs to drawing for photorealistic or non-photorealistic rendering. This is in contrast to conventional systems and methods that generate non-photorealistic renderings from input photorealistic images.

More specifically, the system and method for sketching reality interprets freehand drawing (2-D sketching) to produce realistic imagery. In this implementation and for illustrative purposes, the systems and methods for sketching reality are applied to architectural-based drawings. In various implementations, the systems and methods for sketching reality are applied to freehand drawings that are not architecturally based. In this implementation, the system and method for sketching reality is responsive to receiving or otherwise obtaining freehand sketch input (e.g., a 2-D sketch), generating a 2.5D geometric model from the sketch input, and allowing the user to provide sketch-based Input to refine the appearance of the 2.5D model for drawing and presentation to the user. A 2.5D model refers to a scene representation in which depth information is available from a single viewpoint, while a 3D model has depth for each viewpoint. 2.5D models can be regarded as incomplete 3D models. For example, a 2D shader with added depth information becomes a 2.5D model.

More specifically, for example, a user draws a building using a simple input device interface such as an electronic tablet. The system and method then analyzes the drawn lines to identify junctions and sets of 2.5D parallel lines, thereby generating a 2.5D geometric model/representation. In one implementation, the user refines the appearance of the resulting 2.5D geometric model through texture selection operations (e.g., by painting the textured rough surface on a 2-D sketch) and lighting direction specification (e.g., by hand-painting hatching on the surface) . In one implementation, the final product is a realistic rendering/version of the hand-drawn sketch, which can then be used as part of an animation or edited into a photograph.

These and other aspects of the systems and methods for sketching the reality are now described in more detail.

example system

Although not required, the systems and methods for sketching realities are described in the general context of computer-executable instructions being executed by a computing device, such as a personal computer. Program modules generally include routines, programs, objects, components, data structures, etc., which perform specific tasks or implement specific abstract data types. Although the systems and methods are described in the above context, acts and operations described hereinafter may also be implemented in hardware.

Figure 1 illustrates an example system for sketching a reality, according to one embodiment. System 100 includes a computing device 102, such as a general computing device, server, laptop, mobile computing device, and the like. Computing device 102 includes one or more processors 104 coupled to a tangible computer-readable storage medium (system memory 106 ). Processor 108 may be a microprocessor, microcomputer, microcontroller, digital signal processor, or the like. System memory 106 includes, for example, volatile random access memory (eg, RAM) and nonvolatile read-only memory (eg, ROM, flash memory, etc.). System memory 106 includes computer program instructions executable by processor 104 and program data generated and/or used by respective ones of these computer program instructions. These computer program instructions and program data are shown as program modules 108 and program data 110, respectively. Program modules 108 include, for example, a sketch-real module 112 and "other program modules" 114 such as an operating system (OS) for providing a runtime environment and including other applications such as device drivers.

The sketch reality module 112 automatically generates a 2½-dimensional (2.5D) geometric model 116 from a hand-drawn 2-D sketch 118 of one or more objects, thereby presenting to the user based on the 2.5D geometric model 116 and one or more user inputs based on Photorealistic or non-photorealistic drawing of sketches. In one implementation, 2-D sketches 118 represent architectural designs. A non-photorealistic image is an image (or animation) that appears to have been created by hand, watercolor, pen and ink, oil painting, etc. The sketch reality module 112 presents the drawing results 120 to the user in a user interface (UI). For purposes of illustration, such a user interface is shown as a corresponding portion of "Other Program Data" 122 (see also Figures 14 and 15 below). The user interacts with the UI to update the geometry of the 2.5D model, such as by redrawing or otherwise modifying the 2-D sketch 118 on an input device 124 (eg, graphics tablet, etc.). In addition, the user can add visual cues (user input) to the 2-D sketch 118 via the UI to interactively control the lighting (position, orientation, and intensity) and textures of the rendering 120, as well as specify material rendering associated with the rendering 120 wait. In response to creation of the 2.5D model 116 and sketch-based user interaction for specifying lighting, textures, colors, and/or other parameters, the sketch-realistic module 112 draws photorealistic or non-photorealistic results based on the 2.5D model 116 and user input 120.

FIG. 2 shows an exemplary overview of the operation of sketch reality module 112 according to one implementation. As shown and in this implementation, the operation of the Sketch Reality module 112 includes four distinct phases: (1) an input phase, (2) a geometric model reconstruction phase, (3) an interactive refinement of appearance phase, and (4) output/rendering stage. During the input phase, the sketch reality module 112 obtains a hand-drawn 2-D sketch 118 . In one implementation, the sketch reality module 112 includes logic that allows a user to draw a 2-D input sketch 118 directly using an input device 124 (e.g., an electronic tablet, etc.) for presentation in the UI 122 on the display device 126. In one implementation, the display device 126 includes a touch screen that allows a user to draw the 2-D input sketch 118 directly on the display device 126 . In another implementation, the sketch real module 112 loads the input sketch 118 from a data store (e.g., a corresponding portion of the program data 110), an input device 124 (e.g., a scanner, remote storage, etc.), etc., for display on a display device 126. presented to the user. In one implementation, a different application (shown as a corresponding portion of "other program modules" 114) is used to generate 2-D input sketch 118.

Referring to FIG. 2, in the second stage, or geometric model reconstruction stage, the sketch reality module 112 generates 2.5 D Geometry 116. In the third stage, or interactive refinement of the look, the user specifies one or more of texture, lighting direction, color, etc. In a fourth stage, or output stage, the sketch reality module 112 generates a rendering from the 2.5D model 116 according to user input to present a sketch-based rendering result 120 to the user via a display device 126 . Figure 3 shows an example collection of sketches and image representations according to one embodiment. Specifically, (a) shows an example 2-D input sketch 118; (b) shows an example 2.5D model representation 116 of the example input sketch; (c) shows an example of the example 2.5D model 116 after the user has specified texture and lighting assignments Appearance; and (d) represents the final sketch-based exemplary rendering result 120 (photocomposition). A 2-D sketch 118 is a sketch composed of a set of vector lines or curves.

Exemplary geometry reconstruction

In this implementation, the geometric model reconstruction operation of the sketch reality module 112 identifies (in 3-D) sets of mutually perpendicular lines, vanishing points, and camera intrinsic parameters by analyzing the input 2-D sketch 118 . Next, the sketch-real module 112 identifies the joint type associated with each vertex in the identified line segments, and associates an ordered set of vertices with each stage of the 2.5D model 116 . These operations are followed by estimating the 3-D positions of these vertices, thus identifying the geometry of the 2.5D model 116 .

Identify vector primitives

The sketch reality module 112 identifies vector primitives (eg, points and lines) associated with the 2-D input sketch 118 . These vector primitives are shown as corresponding parts of "Other Program Data" 122 . Since the identified lines may not be straight (eg, curved or non-straight hand-drawn lines), the sketch reality module 112 vectorizes the identified lines to produce straight lines and corresponding vector data. Table 1 shows an exemplary data set generated by vectorizing (ie, generating line vectors) line segments from a hand-drawn 2-D sketch 118 .

In the example of Table 1, sketch real module 112 determines that 2-D sketch 118 includes 20 lines. In one implementation, the least squares method is used to fit straight lines and cubic Bezier curves are used to fit curves. For freehand lines or curves, the line vector/curve is generated with two endpoints A, B. In this implementation, a line segment may represent a straight line or a curved line.

The sketch-real module 112 generates a graph structure G = (V, E) from the identified vector primitives, where V is the set of all endpoints of a line and E is the set of lines. The graph structure is shown as a corresponding part of "other program data" 122 . The graph structure indicates the spatial relationship between corresponding ones of the identified points and the straightened lines (ie, vector primitives). V{v _i }, i=0...n _v is the set of all points of sketch 118 . Furthermore, E{l _i }, i=0...n _l is the set of all lines in sketch 118, where both endpoints of l _i are in V. The sketch reality module 112 uses the graph structure to classify lines as left, right, or vertical and to identify line segment vanishing points. These classifications and vanishing points are then used to identify the set of line segment endpoint junctions/intersection types. Using the identified line segments, the identified line segment junction types, and the described line connectivity probability criteria, the sketch-real module 112 maps respective ones of the identified line segments to corresponding surfaces/surfaces of the 2.5D model 116 as follows polygon.

line analysis

Most architectural sketches consist of lines that are mutually orthogonal in 3-D space: left, right, or vertical (X, Y, and Z, respectively). The system 100 for sketching reality exploits this property to determine the set of lines from which the vanishing point (vps) is estimated. Fig. 4 schematically illustrates a building design according to one embodiment, which is used to illustrate that lines can be sorted by line direction and vanishing point. In the example of FIG. 4, the 2-D sketch includes two vanishing points: a left vanishing point and a right vanishing point. In another implementation, the 2-D sketch may include one vanishing point or three vanishing points. When the designer draws the 2-D sketch 118, the architect can choose one, two or three vanishing points. If there is only one intersection associated with most lines in the 2-D sketch, then there is only one vanishing point. In this case, the line passing through the vanishing point corresponds to the Y direction in the world coordinate system associated with the 2-D sketch. A line with K=0 corresponds to a vertical line in the X direction in the world coordinate system (k is the slope of the line, k=0 indicates a horizontal line, and k=∞ indicates a vertical line). The line of K=∞ corresponds to the horizontal line in the Y direction in the world coordinates.

To construct the geometric model 116, the sketch reality module determines a hypothetical vanishing point _vp for each pair of lines in the 2-D sketch 118 as follows, and aggregates the fits of all other lines. Given a line l and a hypothetical vanishing point _vp , the sketch real module 112 computes a measure of fit according to the following formula:

g(l, v _p )=1-d(l, v _p )/T _a (1)

where d(l, _vp ) is a function of the distance (d) between line l and the vanishing point _vp in the 2-D sketch 118, and T _a is a threshold (eg, a sensitivity parameter) for this distance. In this implementation, the threshold is set to 0.35. In different implementations, this threshold can be set to different values.

Fig. 5 illustrates a distance function d(l, _vp ) between a line l and a vanishing point according to one embodiment. Referring to FIG. 5 , (a) represents an infinite vanishing point, and shows that an example distance is an angle a between a line segment l and the direction of the infinite vanishing point. Referring to Fig. 5(b), a finite vanishing point is shown, where the distance is the angle a between the midpoint of the line segment l and the finite vanishing point. If the vanishing point is finite: d(l, v _p ) = angle (l, V(v _p )). If the vanishing point is infinite: d(l, _vp )=angle(l,l( _vp ,midpoint)).

Junction identification

FIG. 6 illustrates a first example set 600 of line segment joins, according to one embodiment. In this implementation, the junction types are Y, T, L, E, and X junctions. In different implementations, different junction types may be used. A junction is an intersection between two or more lines. Once the lines of the 2-D sketch 118 have been grouped by corresponding vanishing points determined, the sketch reality module 112 determines the type of junction at each vertex and lists each face and vertex of the 2.5D model 116 in an ordered list Associated. To determine the corresponding junction type for a shared vertex, the sketch reality module 112 determines a junction classification based on the angle at which line segments intersect and the number of intersecting lines. A Y-junction represents a common vertex where three visible surfaces meet. T-junctions occur when one surface is obscured by another surface, or when there is a split line itself above the surface. An L-joint is a corner of a face, while an E-joint occurs when two visible faces meet along a common line. More generally, an X-joint is the result of several faces meeting at a common vertex. FIG. 7 illustrates the vertices identified by the input 2-D sketch 118 and the corresponding junction labels, according to one embodiment. In this example, a set of shared vertices and corresponding junctions (Y, T, L, E, and X) are circled and labeled.

Exemplary assignment of ordered vertices to faces

Once the junction types and vanishing points associated with the line directions have been determined, the sketch reality module 112 determines the topology of the 2.5D model 116 by associating an ordered list of vertices for each face. Graph G=(V,E) represents this topology, which encodes connection information between lines E and vertices V. The sketch real module 112 assigns vertices from a single arbitrary vertex of the faces that contain the vertex to other vertices that are part of the faces, and so on. When examining a face F with a vertex V to determine its boundary lines, the candidate lines are those connected to the vertex V. In one implementation, a greedy algorithm is used to select the line with the greatest likelihood of being an edge of the face. Geometrically, it is impossible for lines of the same face to correspond to three vanishing points associated with three orthogonal directions.

For example, suppose there are currently k ordered lines l ₀ , ..., l _k-1 in front, and l _k is a candidate line. Suppose also that the surface is bounded by lines associated with two vanishing points v _pa and v _pb . The probability that F is defined as a valid face is:

The term p(v _p ) describes the probability that the vanishing point v _p coincides with F, ie p(v _p ) = max _i g( _li , v _p ), where g() is defined in (1). The term p(l _i , v _pa , v _pb ) is a measure of the likelihood that line l _i corresponds to the vanishing point of F, and is defined as p(l _i , v _pa , v _pb )=larger(g(l _i , v _pa ), g(l _i , v _pb )), where larger() returns the larger of the two input arguments.

Given three vanishing points v _p0 , v _p1 and v _p2 , the sketch real module 112 selects the pair of vanishing points that yields the maximum value of p(Feffective|l ₀ , . . . , l _k , v _pa , v _pb ) as the vanishing point of the face F. If the maximum p(Fvalid|l ₀ , . . . , l _k , v _pa , v _{pa )} is smaller, the face is considered not valid; in this case, the propagation is terminated at the current vertex.

Figure 8 illustrates an example set of vertices for vertex face assignment, according to one embodiment. Starting from an arbitrary vertex, in this case V0 of FIG. 8( a ), the sketch reality module 112 assigns a new face according to the local characteristics of the junction. Since the starting vertex in this case corresponds to the Y junction, there are three new faces (with originally assigned vertices): F0:V1-V0-V3, F1:V4-V0-V1, and F2:V4-V0 -V3, as shown in Figure 8(b). The Sketch Reality module 112 traverses along the edges of each face to obtain a closed shape. For example, referring to face F0 of Figure 8(b), and starting from V3, there are two candidate lines: V3-V2 and V3-V6. Sketch Reality module 112 selects V3-V2 because it closes the loop of F0. The loop is thus V1-V0-V3-V2-V1, as shown in Figure 8(c). Referring to plane F1 and starting from V1, the sketched real module 112 progresses towards V7 and V8. However, the propagation ends because V8-V9 does not coincide with this face, as shown in Fig. 8(d). Therefore, the sketched real module 112 propagates from the other end to V4 and then to V5 instead. Since V4-V5 is a shaded line (V5 is a T-junction), propagation is again terminated. In this example, V5-V4-V0-V1-V7-V8 ends with an incomplete ring, as shown in Figure 8(e). Referring to the face F2, the sketch real module 112 determines the vertices V4-V0-V3-V6-V4 by direct traversal. At this point and in this example, the sketch real module 112 proceeds to another vertex that is part of at least one unassigned face to repeat the vertex assignment operation described above. Finally, the result of each face assigned the corresponding vertex in Fig. 8 is obtained, as shown in Fig. 8(f).

shape split or complete

Incomplete areas caused by T-junctions can be caused by one or more dividing lines or obscured planes on the existing shape. Figure 9 illustrates four exemplary segmentation shapes/polygons according to one embodiment. When generating 2-D sketches 118, architects typically draw lines on an existing plan to divide it into regions. Separation lines are not classified as boundaries of any regions. The figure below shows some cases of dividing lines. In this example, the circled endpoints represent critical junctions. In these examples, the XY (left, right) lines are contained within a complete shape. The sketch reality module 112 thus divides the complete shape into several sub-shapes contained within the complete shape. These results fit one of the four types of segmentation shapes shown in Fig. 9. For left-to-right line segments, the following set of shape splitting criteria (or splitting shape completion criteria) are used:

• Referring to Figure 9, area (a) or ie Figure 9(a), there are four junctions: two intersecting lines with four T-junctions; each junction on each side of the quadrilateral. So the quadrilateral is split into four quadrilaterals.

• Referring to Figure 9(b), the two junctions are on opposite sides of the quadrilateral; the quadrilateral is split into two quadrilaterals.

· Referring to Figure 9(c), the two T-junctions are on the same line; the quadrilateral is divided into two large and small pieces.

• Referring to Figure 9(d), the two junctions are on the two connecting lines. The quadrilateral is divided into two large and small pieces.

Figure 10 illustrates three exemplary shaded shapes, according to one embodiment. In these examples, the circled endpoints represent T-junctions. In a shaded shape, the WXZ line (left, right, vertical direction) of the junction is the shaded line of the shape. The Sketch Reality module 112 adds new line segments to complete the shape with parallel and perpendicular constraints considering a set of shaded shape completion criteria. More specifically and in this implementation, to create the correct shape, the sketch real module 112 locates the best match in the T-junction database. Referring to FIG. 10( a ), XW of one junction is the same as ZX of the other junction. In this case, the Sketch Reality module 112 adds a line parallel to the X-point connecting line for each junction and selects an edge to form the larger shape. Referring to Figure 10(b), in this case the XW of one joint differs from the ZX of the others, the sketch reality module 112 extends the YX direction of each joint; the intersection is the added vertex of the shape. Referring to Figure 10(c), there is only one T-junction. In this case, the sketch real module 112 positions the last line parallel to XY to complete the shape between X and the end points of the line, as described above with reference to FIG. 10( a ). FIG. 10 illustrates an exemplary set of additional lines (dotted lines) that are used by the sketching reality system and method to complete one or more corresponding shaded shapes, according to one embodiment.

Figure 11 shows another example of shaded shapes according to one embodiment. Figure 11(a) shows the shaded shapes associated with T-junction and Y-junction vertices. Figure 11(b) shows the shaded shape associated to a vertex with two T-junctions. The last two examples of Figure 11 are special because the two bottom lines of the T-junction are parallel and no intersection can be found. In this case, the Sketch Reality module 112 checks whether the incomplete face is contained in another face, as described above in the split shape case, for simply connecting two T-junctions (Fig. 11(c)) . Otherwise, the sketch reality module 112 tries to find a different face to connect, as shown in FIG. 11(d).

As the sketch-real module 112 locates closed shapes representing the surfaces of the 2.5D model 116, the sketch-real module 112 evaluates the relationship between these shapes to determine whether one or more small shapes are contained within a larger shape. From the order in which the closed shapes are formed and the determined shape containment relationships, the sketch reality module 112 determines the general location of these shapes such that the first completed shape is on top of the 2.5D model 116 and the last completed shape is at the 2.5D model 116 The lowest floor of D-model 116. If one or more shapes are enclosed by another shape, the enclosed shapes are children of that shape (the parent shape). Sketch reality module 112 uses such parent-child relationships to determine the order of texture mapping operations. A region included in another region is on the same plane as the other region. Thus, the sketch reality module 112 groups regions with a containment relationship to the same surface. Each surface consists of several layers at different depths. The layer with the lowest depth is at the bottom of the surface and the layer with the highest depth is at the top. In this case there are several regions of similar type on each layer.

Geometry construction (2.5D coordinate calculation)

Figure 12 is an illustration showing regions having one common vertex as part of T-junctions that are not part of the same object, according to one embodiment. In the example of FIG. 12 , sketch reality module 112 identifies two objects (ie, 1202 and 1204 ) from 2-D sketch 118 . In this implementation, to recover the geometry of the object from the 2-D sketch 118, the sketch real module 112 calculates the relative positions between vertices. This is now possible because of knowledge of camera intrinsics, line grouping (for orthogonal constraints), and face and vertex labels. Given two adjacent vertices V ₁ and V ₂ (with 3D positions q ₁ and q ₂ , respectively) and a vanishing point corresponding to the line q ₁ q ₂ (v _p (=(x _p , y _p ) ^T ), then There are constraints:

${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="q"\; filenames="A20068004108600282.png"\; state="\{\{state\}\}">q</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; sgn</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span>\frac{{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; ,</span>$

where f is the precomputed focal length, q _z is the z component of q, H(q) is the projection of 3D point q onto the image plane, and the superscript "T" denotes vector transpose. The sign function sgn(x) is positive when x ≥ 0, and negative otherwise. If the vanishing point is at infinity, the constraints can be modified as follows:

${\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; <figure-callout\; id="2"\; label="q"\; filenames="A20068004108600282.png"\; state="\{\{state\}\}">q</figure-callout></span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; sgn</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span>\frac{{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; q</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; f</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; T</span>}}<span\; class="notranslate">\; .</span>$

The sketch real module 112 sets an arbitrary vertex as a reference 3D point (0, 0, f) ^T , and then uses the above formula to propagate its position to other vertices.

There is a possibility that a line does not correspond to any known vanishing point (and thus has an unknown direction). Therefore, the coordinates of the vertices of such lines cannot be calculated. To handle this, the Sketch Reality module 112 uses a model library of generic architectural components and reconstructs any ambiguous parts of the architectural drawing 118 with the best matching model. Each model is provided as a graph G = (V, E) labeled as described above. The label for each vertex is based on the relationship between the junction type and the edges to that vertex. FIG. 13 illustrates an exemplary set of general building components for reconstructing ambiguous portions of a 2-D sketch 118 according to one embodiment. As shown in Figure 13 and in this implementation, the vertex labels (1-11) are based on the junction type. Lines along one of the three known directions are brighter than the darker, unknown lines. The sketch-real module 112 adapts the graph matching algorithm to match labeled graphs using a known random walk process. The probability of following a given arc from a node is defined as $f_l(L_p,L_q)=\sqrt{L_p*L_q}.$ The probability of jumping from vertex q to vertex p is defined as f _j (L _p , L _q )=L _p .

Sketch-based photorealistic drawing

FIG. 14 illustrates an exemplary sketch-based interactive user interface (UI) 1400 for generating photorealistic renderings from 2-D sketches, according to one embodiment. In this implementation, UI 1400 represents UI 122 of FIG. 1 . In one implementation, the sketch reality module 112 presents on the display device 126 a user interface with a UI 1400 for the user to edit the 2-D sketch 118 in the sketch window. The edit specifies the lighting, textures, colors, etc. required for the final rendering result 120 (eg, as shown in the rendering result window), eg, based on the sketch. FIG. 15 illustrates the example sketch-based interactive user interface 1400 of FIG. 14 , except that the rendering 120 is non-photorealistic compared to the photorealistic rendering 120 of FIG. 14 . These and other aspects of UI 1400 are now described in more detail.

The UI 1400 provides users with sketch-based interactive drawing capabilities for textures, viewpoints, and lighting conditions, among others. In this implementation, the user edits the 2-D sketch 118, and in response to these user edits, the sketch reality module 112 draws and presents sketch-based drawing results 120 to the user. For example, in this implementation, a user can draw hatching on the 2-D sketch 118 to indicate lighting, textures, and the like. In this implementation, the user can also select textures directly from the texture library. In one implementation, the user can also adjust the color of the building using a brush.

Exemplary specifying lighting direction

In one implementation, the user uses the hatch editing tool 1402 to draw one or more hatches on the 2-D sketch 118 . The hatches drawn by the user on the sketch can represent lighting or textures. For lighting, one or more user-supplied hatches indicate the location and angle of shading or shadows to be drawn in the sketch-based drawing result 120 . These hatched strokes/inputs are shown as corresponding parts of "Other Program Data" of FIG. 1 for illustration purposes. Based on such hatching input, sketch reality module 112 determines light source parameters for 2.5D model 116 . These light source parameters include, for example, position, direction and intensity. Direction, position and strength are deduced from the position and strength of the hatch. In response to receiving such hatching input, the sketch reality module 112 renders the indicated shading/shading angles according to the light source parameters determined in the sketch-based rendering structure 120 .

Figure 16 shows an example illustration of specifying lighting direction and shadow intensity, according to one embodiment. Referring to Figure 16, (a) is a style rendering of a block showing the direction of illumination from the left. Figure 16(b) shows an exemplary 2.5D model 116 in which the user provides hatch strokes with respective different intensities on each face to specify the lighting direction ("L") and shadow intensity. In this example, "n1" and "n2" refer to the normals of the corresponding faces. In architecture, the light source is usually the sun. Therefore the rays are parallel. Assuming that the surface is mostly Lambertian, the intensity of the surface under directional light is:

I = k _d I _l cos(θ),

where _kd is the reflectance of the surface, _Il is the intensity of the light source, and θ is the angle between the surface normal and the direction of the light.

Given the sun as the light source, θ is constant anywhere on a flat surface. Therefore, the intensity is uniform across the flat surface (in one implementation, internal reflections are ignored). With this simplification and in one implementation, the sketch real module 112 interprets the light direction based on the relative intensities at different surfaces. For example, suppose the light source direction is L(l ₁ , l ₂ , l ₃ ), where ||L||=1. Then, for each surface, with the following constraints:

I _i = k _d I _l cos(θ _i ) = k _d I _l N _i L,

where N _i is the normal to surface i and I _i is the relative intensity of surface i calculated from the hatching drawn by the user. Here, I _i is defined using the average intensity of the hatched area on the surface.

The sketch-real module 112 assumes I _l and k _d to be constant for the entire object (one or more corresponding parts of the 2.5D model 116 ), such that k _d I _l =1. Given the potential for sketches 118 to be inaccurate, in one implementation, sketch reality module 112 sets constraints for each shaded surface as follows:

I _li ≤ k _d I _l N _i L ≤ I _hi ,

where I _li and I _hi are the average intensities of B _out and B _in , as shown in Figure 17. Figure 17 illustrates exemplary designation of the boundaries of a shaded surface according to one embodiment. In this example, B _out is the bounding box that covers all hatches, and B _in is the center box that is half the size of B _out . The sketch reality module 112 uses a number of constraints corresponding to the number of hatched surfaces. To find a unique solution for the lighting direction, the sketch reality module 112 optimizes the term ΣNl _j ·L, where Nl _j is the surface not shaded by the user. In other words, find a solution that maximizes the strength of the unshaded surface while satisfying the constraints of the shaded surface.

Exemplary Texture Selection

In one implementation, a user uses texture editing tool 1402 to paint or select a texture for an enclosed area of 2.5D model 116 . For example, in one implementation, the user draws hatched strokes directly on the 2-D sketch 118 or 2.5D model 116 to indicate what the texture on the object looks like (eg, directly draws a similar shape to the desired texture). Based on these user inputs, the sketch reality module 112 automatically selects the correct texture for the object. In one implementation, the sketch-real module 112 matches the user input to a stored painting of a texture or an index thereto to retrieve the best texture match. In one implementation, the sketch reality module 112 presents one or more possible textures corresponding to the user input on the UI 1400 for selection by the user. This texture provision is independent of the user's search for the texture of an object in a texture database, but is based on an objective measure of line similarity.

Line similarity is used in texture analysis to describe the shape of texels. Both stroke and indexed line painting in the texture library are black and white images composed of lines, so only shapes are considered during matching, which is well described by line similarity. The line similarity is calculated based on the directional co-occurrence matrix whose element P _d (i,j) is defined as the relative frequency of two gray values i and j at distance d along the edge direction on the image. For a direction d, the line similarity measure is defined as

${\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; line</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; [</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>\frac{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}}{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; ]</span>}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; ,</span>$

where n is the number of gray levels between 0 and 255. In one implementation, the sketch-real module 112 uses the four directions (1,0), (0,1), (1,1), (1,-1) to calculate the direction co-occurrence matrix _Pd and calculates the four The average _Fline of the directions serves as a measure of similarity between user-drawn strokes and texture-indexed line paintings. After computing a matching texture index, the system retrieves compatible textures. The user can then select any of these textures. Each texture has a notion of relative scale, so once a global scale has been selected, the selected texture is scaled appropriately and composited to cover the surface (not stretched to fit). In one implementation, the exceptions to this rule are windows and doors - where the texture is rescaled to map to the entire surface.

In one implementation, "Other Program Data" 122 includes a texture library based on a texture symbol index. The user draws textured symbols on the 2-D sketch 118 . The sketch real module 112 matches the drawn texture symbol to one or more symbols in the texture symbol index to identify the corresponding texture, which texture is retrieved from the library to texture the object or polygon. Standard architectural symbols representing different materials are known. In one implementation, sketch reality module 112 uses these symbols to index texture libraries. When illustrating the retrieved textures in rendering results 120 , sketch reality module 112 maps the retrieved textures to corresponding surfaces of 2.5D object 116 . In one implementation, and due to different texture sample sizes, the sketch real module 112 synthesizes textures on the surface. In another implementation, when providing textures to windows, doors, etc., the textures are mapped directly to the target quad.

Colorization

UI 1400 provides a color selection control 1404 for a user to change the color of a selected object associated with 2-D sketch 118. In one implementation, the sketch reality module 112 presents the user with a color selection control 1404 in the form of a set of color samples, a text input control with direct input of RGB values by the user, or the like. In one implementation, after the user selects a color, the UI 1400 allows the user to paint one or more blobs on the 2-D sketch 118 to modify the color of the associated surface of the 2.5D model 116.

Geometry update

The UI 1400 allows the user to modify the geometry of the 2-D sketch 118, such as by adding lines, deleting lines, adjusting dimensions, modifying angles, and the like. Techniques for performing these edits are known. In response to user modifications to the 2-D sketch 118 , the sketch reality module 112 updates the 2.5D geometric model 116 to correspond to the modifications to the 2-D sketch 118 .

exemplary process

FIG. 18 illustrates an exemplary process 1800 for sketching a reality, according to one embodiment. For purposes of illustration, the operations of process 1800 are described with reference to aspects described above with respect to FIGS. 1-17. In one implementation, the operations of process 1800 are implemented by sketch reality module 112 . The operation of block 1800 is to identify a set of line segments from the 2-D sketch. In one implementation, line segments represent vector primitives from 2-D sketch 118 . The operations of block 1804 vectorize the line segments to produce straight lines. For purposes of illustration, vectorized lines are hereinafter referred to as line segments or lines. For each line segment, the operations of block 1806 estimate a corresponding vanishing point and classify the direction of each line segment based on the line segment's corresponding vanishing point. The operations of block 1808 group the line segments into corresponding line segments of corresponding ones of the plurality of polygons based on the line direction and the vanishing point. The operations of block 1810 construct the geometry of the 2.5D model using closed polygons. In one implementation, the 2.5D model is represented by 2.5D model 116 . The operations of block 1812 receive user input corresponding to one or more of lighting, texture, color, and geometry modifications. The operations of block 1814 are responsive to receiving user input based on which photorealistic and non-photorealistic results are rendered from the 2.5D model. The operations of block 1816 present the drawing results to the user. In one implementation, the drawing result is a sketch-based drawing result 120 .

in conclusion

Although sketched reality has been described with reference to FIGS. 1 to 18 in language specific to structural features and/or methodological operations or acts, it is to be understood that the implementations defined in the appended claims are not necessarily limited to the described these features or actions. Rather, the specific features and acts described above are disclosed as example forms of implementing the appended claims.

Claims (20)

1. A method implemented at least in part by a computing device, the method comprising: Identify a set of vector primitives from a 2-D sketch; and automatically generating a 2.5D geometric model from said vector primitives; and The 2.5D geometric model is rendered for presentation to the user. 2 . The method according to claim 1 , wherein the vector primitives include one or more of a line segment set and a curve set. 3 . 3. The method of claim 1, wherein the 2-D sketch is hand drawn. 4. The method according to claim 1, wherein the automatic generation of the 2.5D geometric model also comprises, generating corresponding line vectors from corresponding some vector primitives in the vector primitives, so as to generate the corresponding line vectors from the vector primitives Primitives corresponding to some vector primitives remove non-linear aspects. 5. the method for claim 1, is characterized in that, real-time automatic generation 2.5D geometric model also comprises: estimating corresponding vanishing points for at least a subset of the vector primitives; and Wherein the corresponding faces of the 2.5D geometric model are based on corresponding ones of the vector primitives according to the associated vanishing points. 6. The method according to claim 1, wherein said automatically generating a 2.5D geometric model further comprises assigning corresponding ones of said line segments to polygons of said 2.5D model. 7. The method of claim 1, wherein rendering further comprises rendering the 2.5D geometric model as a photorealistic image. 8. The method of claim 1, wherein rendering further comprises rendering the 2.5D geometric model as a non-photorealistic image. 9. The method of claim 1, further comprising: receiving user input corresponding to one or more of lighting, texture, color and geometry modifications of the 2.5D geometric model; and In response to receiving the user input, the 2.5D geometric model is rendered with properties based on the user input. 10. The method of claim 9, wherein the user input is a hand-drawn hatching stroke by a user for specifying a lighting position, direction and intensity. 11. The method of claim 9, wherein the user input is a user-drawn hatched stroke for specifying texture. 12. The method of claim 9, wherein the user input is a texture symbol, and wherein the method further comprises: searching a symbol-indexed texture library for a texture corresponding to the texture symbol; and The texture is used to paint the regions of the 2.5D geometric model, the regions identified by the user placing the texture symbols. 13. A tangible computer readable storage medium embodying computer program instructions executable by a processor, said computer program instructions performing the following operations when executed by said processor: Objects that automatically construct 2.5D geometric models from vector primitives of hand-drawn 2-D sketches; and automatically generating a rendering of the 2.5D geometric model for presentation to a user, the rendering being rendered based on 2-D sketch-based user input specifying one or more of texture, lighting, and color for the rendering of. 14. The computer-readable storage medium of claim 13, wherein the 2-D sketch-based user input comprises hand-drawn hatching by a user on the 2-D sketch. 15. The computer-readable storage medium of claim 13, wherein the 2-D sketch-based user input corresponding to lighting specifies lighting position, direction, and intensity. 16. The computer-readable storage medium of claim 13, wherein the computer program instructions for automatically constructing the 2.5D geometric model comprise computer program instructions for performing the following operations: estimating vanishing points in the 2-D sketch to classify the vector primitives according to corresponding directions; assigning a respective joint type of a plurality of joint types to each common endpoint associated with the vector primitive based on the respective directions; and Selecting said respective ones of said vector primitives as connections defining respective surfaces of the object based on a respective likelihood that each of said respective ones of said vector primitives will form an enclosed region of said 2.5D geometric model Wire. 17. The computer-readable storage medium of claim 13 , wherein the computer program instructions for automatically constructing objects of the 2.5D geometric model comprise computer program instructions for performing the following operations: for all The one or more regions of the 2.5D geometric model that cannot be closed, the one or more regions are completed according to a set of shaded shape completion criteria. 18. The computer-readable storage medium of claim 13 , wherein the computer program instructions for automatically constructing objects of the 2.5D geometric model comprise computer program instructions for performing the following operations: for the 2.5D One or more regions of the D geometric model that cannot be closed, the one or more regions are completed according to a set of shape segmentation completion criteria. 19. A computing device comprising: processor; and A memory coupled to the processor, the memory containing computer program instructions executable by the processor, which when executed by the processor perform the following operations: receiving a set of vector primitives in response to a user drawing a 2-D sketch on the input device; In response to receiving the vector primitive: vectorizing the vector primitives to produce a set of vector data defining a plurality of line segments; estimating properties of the line segments to assign respective ones of the line segments to a surface of the 2.5D model; presenting a rendering of said 2.5D model to a user; receiving user input corresponding to one or more regions of the 2-D sketch; and In response to receiving the user input, the rendering is presented to illustrate one or more of lighting position, lighting direction, lighting intensity, texture, and color based on the user input. 20. The computing device of claim 19, wherein the rendering is photorealistic rendering.

Images