CN115202661B - A hybrid generation method with hierarchical layout and related equipment - Google Patents

Abstract

The invention discloses a hybrid generation method with hierarchical structure layout and related equipment, wherein the method comprises the following steps: receiving a plurality of layout templates input by a user, wherein the layout templates are represented by a multi-branch tree structure, leaf nodes in the multi-branch tree structure represent single elements in the layout templates, and the corresponding relation between the elements is calculated according to a node corresponding rule to obtain corresponding information; establishing a combined tree according to the corresponding information, and dynamically determining intermediate state elements and relations which should be met between the elements based on the combined tree according to node types and in combination with parameters generated during user interaction; and taking the positions and the sizes of the nodes as soft constraints, taking the relation which should be met between the nodes as hard constraints, optimizing an objective function, and generating a new diversity layout after the optimization solution is completed. The invention only needs to input a small amount of layout templates without a large data set for training, saves the computing resources and enables users to generate brand-new diversified layouts through simple interaction.

Description

A hybrid generation method with hierarchical layout and related equipment

technical field

The invention relates to the technical field of computer graphics, in particular to a hybrid generation method, system, terminal and computer-readable storage medium with hierarchical layout.

Background technique

Two-dimensional layouts are widely used in information carriers such as web pages, posters, and magazines. Different information modules can be divided by layout, so as to obtain a content presentation layout. Reasonable layout can meet the user's reading habits, so that users can quickly obtain key information. Therefore, it is very meaningful to explore how to generate two-dimensional layout and apply it to information display.

There are mainly the following implementation forms for different forms of layout generation: the first one is to generate the relationship between elements as a constraint solution optimization. For example, (1) It is required to input a series of substitutable relationships between elements, the size range of elements and the size of the display interface, connect multiple relationships between elements through "or" constraints, and solve the whole as a hard constraint A layout that adapts to the size of various display devices can be generated. (2) A method is proposed to automatically detect possible relationships between elements such as alignment and equal size, and allow users to manually edit the relationship between elements in an interactive way, and finally the position and size of these elements As a soft constraint, the relationship between elements is optimized as a hard constraint to generate a regular layout. (3) A framework that can interpolate 3D building layouts with similar structures is proposed. The input of this work is multiple building layouts. There is a clear correspondence between these layout elements, and the building layout boundaries are used as hard constraints. , the lighting, height and courtyard of the room are used as soft constraints. The combination of hard constraints and soft constraints defines the criteria of a "good layout", and then generates a new layout with the same structure as the input layout by means of linear interpolation. The other is a way to automatically generate layouts based on deep learning. (4) Using the deep neural network to train the generative model and then combining the user's input constraints can automatically generate a two-dimensional floor plan layout. (5) A new framework is proposed, which uses the self-attention mechanism to learn the contextual relationship between layout elements and then use it to generate new layouts.

The above methods have certain defects, such as (1) and (3) when doing layout generation, they require a clear correspondence between the input layout elements; (2) require the user to manually edit the relationship between elements; (4) and (5) These deep learning-based layout generation work often requires a large data set for training the model, and such methods require a large amount of computing resources.

Therefore, the prior art still needs to be improved and developed.

Contents of the invention

The main purpose of the present invention is to provide a hybrid generation method, system, terminal and computer-readable storage medium with a hierarchical layout, aiming at solving the problem in the prior art that generating a new layout is too complicated for information display.

In order to achieve the above object, the present invention provides a hybrid generation method with a hierarchical layout, and the hybrid generation method with a hierarchical layout includes the following steps:

Receive multiple layout templates input by the user, the layout template is represented by a multi-fork tree structure, the leaf nodes in the multi-fork tree structure represent a single element in the layout template, and the correspondence between elements is calculated according to the node correspondence rule relationship, get the corresponding information;

Establish a combination tree according to the corresponding information, and dynamically determine the intermediate state elements and the relationship between the elements according to the node type and the parameters generated during user interaction based on the combination tree;

The location and size of nodes are used as soft constraints, and the relationship between nodes is used as hard constraints, the objective function is optimized, and a new diversity layout is generated after the optimization solution is completed.

Optionally, in the hybrid generation method with hierarchical layout, the node correspondence rules include a first correspondence rule, a second correspondence rule, a third correspondence rule and a fourth correspondence rule;

The first corresponding rule includes:

Two leaf nodes with similar geometric information tend to correspond together;

Two branch nodes with similar substructures tend to correspond together;

Two leaf nodes with different semantic information cannot correspond together;

The second corresponding rule includes:

Leaf nodes and branch nodes are allowed to be empty;

The third corresponding rule includes:

和

对应在一起，则子树

中的节点只能对应子树

中的节点或者对应空；If two branch nodes

and

Corresponding together, the subtree

Nodes in can only correspond to subtrees

The node in or corresponds to empty;

The fourth corresponding rule includes:

Two nodes at different levels are allowed to correspond together;

Wherein, the parent node of the leaf node is a branch node.

Optionally, in the hybrid generation method with a hierarchical layout, wherein the calculation of the correspondence between elements according to the node correspondence rules to obtain the correspondence information also includes:

Use the recursive method to define the function of calculating the corresponding loss between two nodes and between nodes and space;

，第一惩罚项的引入使得两个语义信息不同的节点不能对应在一起，为以下形式：The corresponding cost of two leaf nodes defines the second normal form of the position difference of the two nodes plus the second normal form of the size difference plus a first penalty item related to semantic information

, the introduction of the first penalty item makes it impossible for two nodes with different semantic information to correspond together, which is in the following form:

；

;

；

;

和

分别表示两个叶子节点；

是计算两个叶子节点对应代价的函数；

和

分别表示两个叶子节点的位置；

和

分别表示两个叶子节点的大小；

和

分别表示两个叶子节点对应的语义信息；in,

and

represent two leaf nodes respectively;

is a function to calculate the corresponding cost of two leaf nodes;

and

represent the positions of the two leaf nodes respectively;

and

respectively represent the size of the two leaf nodes;

and

respectively represent the semantic information corresponding to the two leaf nodes;

为该叶子节点对应一个大小为0的节点的损失，具体为该节点大小与0差的二范式再加上一个第二惩罚项

，为：Leaf node corresponds to the empty cost

The loss of the leaf node corresponding to a node with a size of 0, specifically the second normal form of the difference between the size of the node and 0 plus a second penalty item

,for:

；

;

；Among them, set

;

The cost corresponding to the null of a branch node is defined as the cumulative sum of the costs corresponding to the null of all leaf nodes in the branch, which is:

；

;

是计算分支节点与空对应代价的函数，

是该分支节点下的叶子节点；in,

is a function to calculate the cost corresponding to the branch node and empty,

is the leaf node under the branch node;

The cost of the branch node corresponding to the leaf node: recursively call the calculation loss method to calculate the cost of the child node of the branch node and the leaf node, calculate the cost of the child node of the branch node and the cost of the leaf node to the null; use the calculated The cost value constructs a two-dimensional matrix and solves it through the Hungarian algorithm to obtain a minimum loss and an optimal matching result.

Optionally, in the hybrid generation method with hierarchical layout, the calculation costs of the branch nodes corresponding to the branch nodes include:

；Neither branch node sinks; recursively calls the compute loss method to compute the cost between the child node of the first branch node and the child node of the other branch node; computes the cost of the child node of the first branch node against null and The cost of the child node of the second branch node to the null; use the calculated cost value to construct a two-dimensional matrix and solve it through the Hungarian algorithm to obtain a loss value recorded as

;

；The first branch node sinks one level, and the second branch node does not sink; recursively call the calculation loss method to calculate the cost between the first branch node and the child node of another branch node; calculate the pair of the first branch node Null cost and the cost of the child node of the second branch node to the null; use the calculated cost value to construct a two-dimensional matrix and solve it through the Hungarian algorithm to obtain a loss value denoted as

;

；The first branch node does not sink, and the second branch node sinks one level; recursively call the calculation loss method to calculate the cost between the child node of the first branch node and another branch node; calculate the cost of the first branch node The cost of the child node to the null and the cost of the second branch node to the null; use the calculated cost value to construct a two-dimensional matrix and solve it through the Hungarian algorithm to obtain a loss value recorded as

;

，两个分支节点内的元素对应结果等于损失值最小的对应的匹配；将两个布局的根节点作为参数传入定义的递归函数中得到总体的对应损失值以及两个布局元素之间的对应信息。then the final loss value

, the corresponding result of the elements in the two branch nodes is equal to the corresponding matching with the smallest loss value; pass the root nodes of the two layouts as parameters into the defined recursive function to obtain the overall corresponding loss value and the correspondence between the two layout elements information.

Optionally, the hybrid generation method with hierarchical layout, wherein, based on the combination tree, according to the node type and in combination with the parameters generated during user interaction, the intermediate state elements and the relationship that should be satisfied between the elements are dynamically determined, Specifically include:

The combined tree includes three types of nodes. The first type of nodes has corresponding nodes in both trees; the second type of nodes only has corresponding nodes in the first tree; the third type of nodes only has corresponding nodes in the first tree. There are corresponding nodes in the two trees;

For the second type of nodes and the third type of nodes, a deletion threshold is evenly assigned according to the level where they are located, and the tree structure of the intermediate state and the relationship that should be satisfied between sibling nodes are dynamically determined according to the input parameters;

Among them, the relationship between sibling nodes includes equal size, equal spacing and alignment.

Optionally, the hybrid generation method with hierarchical layout, wherein the position and size of nodes are used as soft constraints, and the relationship that should be satisfied between nodes is used as hard constraints to optimize the objective function, specifically including:

小于0.5时，用第一棵树中的关系，否则，用第二树中的关系；When the input parameter value

When it is less than 0.5, use the relationship in the first tree, otherwise, use the relationship in the second tree;

When solving, the position and size are used as soft constraints, and the relationship that should be satisfied between nodes is used as hard constraints. The objective function to be optimized is as follows:

；

;

是度量位置差的能量方程，

是度量大小差的能量方程；in,

is the energy equation to measure the position difference,

is the energy equation to measure the magnitude difference;

；

;

；

;

是中间状态树中的一个节点；

代表该节点要求解的位置量；

代表该节点要求解的大小值；如果

为空，则指示函数

，否则

；

表示节点

对应在第一棵树中的节点

的位置；

表示节点

的大小；

表示节点

对应在第二棵树中的节点

的位置；

表示节点

的大小；在一系列硬约束下最小化目标函数。in,

is a node in the intermediate state tree;

Represents the position quantity that the node needs to solve;

Represents the size value of the node to be solved; if

is empty, the indicator function

,otherwise

;

represents a node

corresponds to the node in the first tree

s position;

represents a node

the size of;

represents a node

corresponds to the node in the second tree

s position;

represents a node

The size of ; minimizes the objective function under a set of hard constraints.

Optionally, the hybrid generation method with hierarchical layout, wherein, the hybrid generation method with hierarchical layout further includes:

If the two nodes are vertically aligned, then:

；

;

；

;

表示节点

对应元素的横坐标；

表示节点

对应元素的宽度；

表示节点

对应元素的横坐标；

表示节点

对应元素的宽度。in,

represents a node

The abscissa of the corresponding element;

represents a node

The width of the corresponding element;

represents a node

The abscissa of the corresponding element;

represents a node

Corresponds to the width of the element.

In addition, in order to achieve the above object, the present invention also provides a hybrid generation system with a hierarchical layout, wherein the hybrid generation system with a hierarchical layout includes:

The hybrid build system with hierarchical layout includes:

The information calculation module is used to receive multiple layout templates input by the user, the layout templates are represented by a multi-fork tree structure, and the leaf nodes in the multi-fork tree structure represent a single element in the layout template, according to the node corresponding rules Calculate the corresponding relationship between elements to obtain corresponding information;

The relationship determination module is used to establish a combination tree according to the corresponding information, based on the combination tree, according to the node type and in combination with the parameters generated during user interaction, dynamically determine the intermediate state elements and the relationship that should be satisfied between the elements;

The optimization generation module is used to use the position and size of nodes as soft constraints, and the relationship that should be satisfied between nodes as hard constraints, optimize the objective function, and generate a new diversity layout after the optimization solution is completed.

In addition, in order to achieve the above object, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and a hybrid generator with a hierarchical layout stored on the memory and operable on the processor A program, when the hybrid generation program with a hierarchical layout is executed by the processor, the steps of the above-mentioned hybrid generation method with a hierarchical layout are implemented.

In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a hybrid generation program with a hierarchical layout, and the hybrid generation program with a hierarchical layout is The processor executes the steps to implement the hybrid generation method with hierarchical layout as described above.

In the present invention, a plurality of layout templates input by the user are received, and the layout template is represented by a multi-fork tree structure, and the leaf nodes in the multi-fork tree structure represent a single element in the layout template, and the elements are calculated according to the node corresponding rules According to the corresponding relationship between them, the corresponding information is obtained; a combination tree is established according to the corresponding information, based on the combination tree, the intermediate state elements and the relationship that should be satisfied between the elements are dynamically determined according to the node type and combined with the parameters generated during user interaction; The position and size of nodes are used as soft constraints, and the relationship between nodes is used as hard constraints. The objective function is optimized, and a new diversity layout is generated after the optimization solution is completed. The present invention only needs to input a small number of layout templates and does not need a large data set for training, which saves computing resources and enables users to generate brand new and diverse layouts through simple interaction.

Description of drawings

Fig. 1 is the flow chart of the preferred embodiment of the hybrid generating method with hierarchical layout of the present invention;

Fig. 2 is a schematic diagram of the first corresponding rule in a preferred embodiment of the hybrid generation method with hierarchical layout in the present invention;

Fig. 3 is a schematic diagram of the second corresponding rule in a preferred embodiment of the hybrid generating method with hierarchical layout in the present invention;

Fig. 4 is a schematic diagram of the third corresponding rule in a preferred embodiment of the hybrid generation method with hierarchical layout in the present invention;

Fig. 5 is a schematic diagram of the fourth corresponding rule in a preferred embodiment of the hybrid generation method with hierarchical layout in the present invention;

Fig. 6 is a schematic diagram of a combination tree created according to corresponding information in a preferred embodiment of the hybrid generation method with hierarchical layout in the present invention;

Fig. 7 is a schematic diagram of the tree structure process of dynamically determining the intermediate state in the preferred embodiment of the hybrid generation method with hierarchical structure layout in the present invention;

Fig. 8 is a schematic diagram of user-generated layouts and their corresponding visualization results in a preferred embodiment of the hybrid generation method with hierarchical layouts in the present invention;

Fig. 9 is a schematic diagram of the principle of a preferred embodiment of the hybrid generation system with hierarchical layout in the present invention;

FIG. 10 is a schematic diagram of an operating environment of a preferred embodiment of the terminal of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention more clear and definite, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

The invention enables the user to generate a brand new layout through simple interaction. Before performing mixed interpolation, the correspondence between the layout elements must be calculated first, and the calculated corresponding information is consistent with the division structure of the layout; a combination is established according to the corresponding information. Formulate a suitable node deletion strategy after the tree, and determine the intermediate variables and the constraint relationships that should be satisfied between the variables according to the input parameters; the method of the present invention can also calculate the optimal correspondence when there is no clear correspondence between elements, and The element corresponding result is consistent with the division structure of the layout, and can also be used for hybrid interpolation; only need the user to input at least two layout templates and a new layout can be generated through simple interaction, and the generated results can be used by subsequent designers creation etc.

The hybrid generation method with hierarchical structure layout described in the preferred embodiment of the present invention, as shown in Figure 1, the described hybrid generation method with hierarchical structure layout includes the following steps:

Step S10: Receive a plurality of layout templates input by the user, the layout templates are represented by a multi-fork tree structure, and the leaf nodes in the multi-fork tree structure represent individual elements in the layout template, and calculate the relationship between the elements according to the node corresponding rules Correspondence between the corresponding information.

Specifically, firstly, the layout template input by the user is represented by a multi-fork tree structure (both the input layout and the output layout are represented by a multi-fork tree), the leaf nodes in the multi-fork tree structure represent a single element in the layout, and the branch nodes are The bounding box of its child nodes, the parent node of the leaf node is a branch node, for example: if there are two sibling nodes (child nodes) corresponding to two elements in the layout, then the parent node (branch node) of the two sibling nodes) It is a box that just "wraps" those two elements completely. Before doing hybrid interpolation, the correspondence between elements must be calculated first, so a series of node correspondence rules are pre-defined.

Wherein, the node correspondence rules include a first correspondence rule, a second correspondence rule, a third correspondence rule and a fourth correspondence rule.

As shown in Figure 2, the first corresponding rule includes:

(a) Two leaf nodes with similar geometric information tend to correspond together;

(b) Two branch nodes with similar substructures tend to correspond together;

(c) Two leaf nodes with different semantic information cannot correspond together.

As shown in Figure 3, the second corresponding rules include:

(a) Leaf nodes and branch nodes are allowed to be empty.

As shown in Figure 4, the third corresponding rule includes:

和

对应在一起，则子树

中的节点只能对应子树

中的节点或者对应空。(a) If two branch nodes

and

Corresponding together, the subtree

Nodes in can only correspond to subtrees

Nodes in or correspond to empty.

As shown in Figure 5, the fourth corresponding rule includes:

(a) Two nodes at different levels are allowed to correspond together.

Among them, the parent node of the leaf node is a branch node; the circle in Figure 2-Figure 5 is a node (branch node or leaf node). Leaf nodes; those rectangular boxes of different colors are the elements in the layout, represented by nodes.

Under the guidance of the corresponding rules, a recursive method is used to define the function of calculating the corresponding loss between two nodes and between nodes and space.

，第一惩罚项的引入使得两个语义信息不同的节点不能对应在一起，为以下形式：First, the corresponding cost of two leaf nodes defines the second normal form of the position difference of the two nodes plus the second normal form of the size difference plus a first penalty item related to semantic information

, the introduction of the first penalty item makes it impossible for two nodes with different semantic information to correspond together, which is in the following form:

；

;

；

;

和

分别表示两个叶子节点；

是计算两个叶子节点对应代价的函数；

和

分别表示两个叶子节点的位置；

和

分别表示两个叶子节点的大小；

和

分别表示两个叶子节点对应的语义信息。in,

and

represent two leaf nodes respectively;

is a function to calculate the corresponding cost of two leaf nodes;

and

represent the positions of the two leaf nodes respectively;

and

respectively represent the size of the two leaf nodes;

and

respectively represent the semantic information corresponding to the two leaf nodes.

为该叶子节点对应一个大小为0的节点的损失，具体为该节点大小与0差的二范式再加上一个第二惩罚项

，为：Leaf node corresponds to the empty cost

The loss of the leaf node corresponding to a node with a size of 0, specifically the second normal form of the difference between the size of the node and 0 plus a second penalty item

,for:

；

;

。Among them, set

.

The cost corresponding to the null of a branch node is defined as the cumulative sum of the costs corresponding to the null of all leaf nodes in the branch, which is:

；

;

是计算分支节点与空对应代价的函数，

是该分支节点下的叶子节点。in,

is a function to calculate the cost corresponding to the branch node and empty,

It is the leaf node under the branch node.

The cost of the branch node corresponding to the leaf node: recursively call the previously defined calculation loss method to calculate the cost of the child node of the branch node and the leaf node; similarly, calculate the cost of the child node of the branch node and the cost of the leaf node to the null cost; then use the calculated cost value to construct a two-dimensional matrix and solve it through the Hungarian algorithm to get a minimum loss and the best matching result.

The calculation cost of the branch node corresponding to the branch node is divided into three cases:

；(1) Both branch nodes are not sinking; recursively call the previously defined calculation loss method to calculate the cost between the child node of the first branch node and the child node of the other branch node; calculate the child of the first branch node The cost of the node to the null and the child node of the second branch node to the null; construct a two-dimensional matrix with the calculated cost value and solve it by the Hungarian algorithm to obtain a loss value recorded as

;

；(2) The first branch node sinks one level, and the second branch node does not sink; recursively call the previously defined calculation loss method to calculate the cost between the first branch node and the child node of another branch node; calculate The cost of the first branch node to the null and the child node of the second branch node to the null; use the calculated cost value to construct a two-dimensional matrix and solve it through the Hungarian algorithm to obtain a loss value recorded as

;

；(3) The first branch node does not sink, and the second branch node sinks one level; recursively call the previously defined calculation loss method to calculate the cost between the child node of the first branch node and another branch node; calculate The cost of the child node of the first branch node to the null and the cost of the second branch node to the null; use the calculated cost value to construct a two-dimensional matrix and solve it through the Hungarian algorithm to obtain a loss value recorded as

;

，这两个分支节点内的元素对应结果等于损失值最小的对应的匹配；将两个布局的根节点作为参数传入定义的递归函数中得到总体的对应损失值以及两个布局元素之间的对应信息。Then, the final loss value

, the corresponding result of the elements in these two branch nodes is equal to the corresponding match with the smallest loss value; pass the root nodes of the two layouts as parameters into the defined recursive function to get the overall corresponding loss value and the relationship between the two layout elements corresponding information.

Step S20, build a combination tree according to the corresponding information, based on the combination tree, dynamically determine the intermediate state elements and the relationship between the elements according to the node type and in combination with the parameters generated during user interaction.

]；第三类节点是只在第二棵树中有对应的节点，例如节点[

，7]。Specifically, according to the calculated correspondence between elements, a combined tree is established; as shown in Figure 6, the combined tree includes three types of nodes, and the first type of nodes has corresponding nodes in both trees , such as node [15, 16]; the second type of node has a corresponding node only in the first tree, such as node [10,

]; the third type of node is a node that only has a corresponding node in the second tree, such as node [

, 7].

As shown in Figure 7, a deletion threshold is evenly assigned to the second type of nodes and the third type of nodes according to the level they are in, and then the tree structure of the intermediate state and sibling nodes (with the same parent node) are dynamically determined according to the input parameters. Nodes are sibling nodes. For example, in the tree in the upper left corner of Figure 6, node 10 and node 11 are sibling nodes).

Among them, the relationship between sibling nodes includes equal size, equal spacing and alignment.

Step S30 , taking the positions and sizes of nodes as soft constraints, and the relationship that should be satisfied between nodes as hard constraints, optimizing the objective function, and generating a new diversity layout after the optimization solution is completed.

小于0.5时，用第一棵树（图6左上，即输入的第一个布局）中的关系，否则，用第二树中（图6右上，即输入的第二个布局）的关系。Specifically, when the input parameter value

When it is less than 0.5, use the relationship in the first tree (upper left in Figure 6, that is, the first input layout), otherwise, use the relationship in the second tree (upper right in Figure 6, that is, the second input layout).

When solving, the position and size are used as soft constraints, and the relationship that should be satisfied between nodes is used as hard constraints. The objective function to be optimized is as follows:

；

;

是度量位置差的能量方程，

是度量大小差的能量方程。in,

is the energy equation to measure the position difference,

is the energy equation that measures the magnitude difference.

；

;

；

;

是中间状态树中的一个节点；

代表该节点要求解的位置量；

代表该节点要求解的大小值；如果

为空，则指示函数

，否则

；

表示节点

对应在第一棵树中的节点

的位置；

表示节点

的大小；

表示节点

对应在第二棵树中的节点

的位置；

表示节点

的大小；在一系列硬约束下最小化目标函数。in,

is a node in the intermediate state tree;

Represents the position quantity that the node needs to solve;

Represents the size value that the node needs to solve; if

is empty, the indicator function

,otherwise

;

represents a node

corresponds to the node in the first tree

s position;

represents a node

the size of;

represents a node

corresponds to the node in the second tree

s position;

represents a node

The size of ; minimizes the objective function under a set of hard constraints.

For example, if two nodes are vertically aligned, then:

；

;

；

;

表示节点

对应元素的横坐标；

表示节点

对应元素的宽度；

表示节点

对应元素的横坐标；

表示节点

对应元素的宽度。in,

represents a node

The abscissa of the corresponding element;

represents a node

The width of the corresponding element;

represents a node

The abscissa of the corresponding element;

represents a node

Corresponds to the width of the element.

Similarly, other hard constraints can also be written in a similar linear equation form, and the position and size of each node can be obtained by solving the quadratic programming problem.

As shown in Figure 8, a number of experimenters are invited to participate in the experience of the interactive system, and the experimenters select three layout templates from the set of candidate layout templates as input, and the layout is automatically generated by the method of the present invention. The experimenter selected 5 different layouts from the generated results and saved them, and the above process was repeated 5 times. Finally, a user survey was conducted on the experimenters, and the experimenters indicated that various, reasonable and beautiful layouts can be generated by using the tool (method) of the present invention. The present invention further applies these results to the production of content such as posters, and obtains beautiful visualization effects.

The method proposed by the invention only needs to input a small number of layout templates and does not need a large data set for training, which saves computing resources; it is convenient for users to generate diverse layouts through simple interaction.

The main content of the present invention is a hybrid generation method with a hierarchical structure layout. After the user inputs multiple layout templates, the present invention can automatically calculate the correspondence between elements, and then build a combined tree according to the corresponding information, combined with the user interaction generated The parameters dynamically determine the intermediate state elements and the relationship between the elements, and finally optimize the solution to generate a new layout. Further, a local editing function is also provided, and the user can fix or change a certain part of the generated result separately to increase the diversity of the generated layout.

Further, as shown in FIG. 9 , based on the above hybrid generation method with hierarchical layout, the present invention also provides a hybrid generation system with hierarchical layout, wherein the hybrid generation system with hierarchical layout includes :

The information calculation module 51 is configured to receive multiple layout templates input by the user, the layout templates are represented by a multi-fork tree structure, and the leaf nodes in the multi-fork tree structure represent a single element in the layout template, according to the node correspondence The rule calculates the corresponding relationship between the elements and obtains the corresponding information;

The relationship determination module 52 is configured to establish a combination tree according to the corresponding information, based on the combination tree, according to the node type and in combination with the parameters generated during user interaction, dynamically determine the intermediate state elements and the relationship that should be satisfied between the elements;

The optimization generation module 53 is used to use the positions and sizes of nodes as soft constraints, and the relationships that should be satisfied between nodes as hard constraints, optimize the objective function, and generate a new diversity layout after the optimization solution is completed.

Further, as shown in FIG. 10 , based on the above hybrid generation method and system with hierarchical layout, the present invention also provides a terminal correspondingly, and the terminal includes a processor 10 , a memory 20 and a display 30 . Fig. 10 only shows some components of the terminal, but it should be understood that it is not required to implement all the components shown, and more or less components may be implemented instead.

The storage 20 may be an internal storage unit of the terminal in some embodiments, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk equipped on the terminal, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD ) card, flash memory card (Flash Card), etc. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the installed terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, a hybrid generation program 40 with a hierarchical layout is stored on the memory 20, and the hybrid generation program 40 with a hierarchical layout can be executed by the processor 10, thereby realizing the hybrid with a hierarchical layout in this application. generate method.

In some embodiments, the processor 10 may be a central processing unit (Central Processing Unit, CPU), a microprocessor or other data processing chips for running program codes stored in the memory 20 or processing data, for example Implementing the described hybrid generation method with hierarchical layout, etc.

In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, Organic Light-Emitting Diode) touch panel, and the like. The display 30 is used for displaying information on the terminal and for displaying a visualized user interface. The components 10-30 of the terminal communicate with each other via a system bus.

In one embodiment, when the processor 10 executes the hybrid generation program 40 in the memory 20, the steps of the method for generating a hybrid with a hierarchical layout are realized.

The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a hybrid generation program with a hierarchical layout, and when the hybrid generation program with a hierarchical layout is executed by a processor, the above Steps of the hybrid generative method with hierarchical layout described above.

To sum up, the present invention provides a hybrid generation method with a hierarchical layout and related equipment, the method includes: receiving multiple layout templates input by the user, the layout templates are represented by a multi-fork tree structure, and the multiple The leaf nodes in the fork tree structure represent a single element in the layout template, and calculate the corresponding relationship between elements according to the node corresponding rules to obtain corresponding information; establish a combination tree according to the corresponding information, and based on the combination tree according to the node type Combined with the parameters generated during user interaction, the intermediate state elements and the relationship that should be satisfied between the elements are dynamically determined; the position and size of the nodes are used as soft constraints, and the relationship that should be satisfied between nodes is used as hard constraints, the objective function is optimized, and the optimization solution is completed Then generate a new diversity layout. The present invention only needs to input a small number of layout templates and does not need a large data set for training, which saves computing resources and enables users to generate brand new and diverse layouts through simple interaction.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or terminal comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or terminal. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article or terminal comprising the element.

Of course, those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be realized by instructing related hardware (such as processors, controllers, etc.) through computer programs, and the programs can be stored in a In the computer-readable computer-readable storage medium, the program may include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium described herein may be a memory, a magnetic disk, an optical disk, and the like.

It should be understood that the application of the present invention is not limited to the above examples, and those skilled in the art can make improvements or transformations according to the above descriptions, and all these improvements and transformations should belong to the protection scope of the appended claims of the present invention.

Claims (7)

1. A hybrid generation method with a hierarchical layout, the hybrid generation method with a hierarchical layout comprising:

receiving a plurality of layout templates input by a user, wherein the layout templates are represented by a multi-branch tree structure, leaf nodes in the multi-branch tree structure represent single elements in the layout templates, and the corresponding relation between the elements is calculated according to a node corresponding rule to obtain corresponding information;

establishing a combined tree according to the corresponding information, and dynamically determining intermediate state elements and relations which should be met between the elements based on the combined tree according to node types and in combination with parameters generated during user interaction;

taking the positions and the sizes of the nodes as soft constraints, taking the relation which should be met among the nodes as hard constraints, optimizing an objective function, and generating a new diversity layout after the optimization solution is completed;

the node corresponding rules comprise a first corresponding rule, a second corresponding rule, a third corresponding rule and a fourth corresponding rule;

the first correspondence rule includes:

two leaf nodes with similar geometric information tend to correspond together;

two branch nodes with similar substructures tend to correspond together;

two leaf nodes with different semantic information cannot be corresponded together;

the second correspondence rule includes:

leaf nodes and branch nodes are allowed to be empty correspondingly;

the third rule of correspondence includes:

if two branch nodes

And

correspond together to form a subtree

A node in (2) can only correspond to a sub-tree

Node in or corresponding null;

the fourth rule of correspondence includes:

two nodes at different levels are allowed to correspond together;

wherein, the father node of the leaf node is a branch node;

the calculating the corresponding relation between the elements according to the node corresponding rule to obtain the corresponding information further comprises the following steps:

defining a function for calculating corresponding losses between two nodes and between the nodes and the spaces by using a recursive method;

the corresponding cost of two leaf nodes defines two normal forms of the position difference of two nodes, the two normal forms of the size difference and a first punishment item related to semantic information

The introduction of the first penalty term makes two nodes with different semantic information unable to be corresponded together, and is in the following form:

；

；

wherein,

and

respectively representing two leaf nodes;

is a function for calculating the corresponding cost of two leaf nodes;

and

respectively representing the positions of two leaf nodes;

and

respectively representing the sizes of two leaf nodes;

and

respectively representing semantic information corresponding to the two leaf nodes;

leaf node to null cost

Corresponding to a node loss with a size of 0 for the leaf node, specifically, adding a second penalty term to a two-normal form of the difference between the size of the node and 0

The method comprises the following steps:

；

wherein, setting

；

The empty cost corresponding to a branch node is defined as the cumulative sum of the empty costs corresponding to all leaf nodes in the branch, and is:

；

wherein,

is a function of the costs of the branch nodes corresponding to the nulls,

is a leaf node below the branch node;

cost of the branch node corresponding to the leaf node: recursively calling a calculation loss method to calculate the costs of the child nodes and the leaf nodes of the branch nodes, and calculating the costs of the child nodes of the branch nodes and the leaves of the leaf nodes; constructing a two-dimensional matrix by using the calculated cost values, and solving by using a Hungarian algorithm to obtain a minimum loss and an optimal matching result;

taking the position and the size of the nodes as soft constraints, taking the relation which should be satisfied between the nodes as hard constraints, and optimizing an objective function specifically comprises the following steps:

when the parameter value is inputted

If the value is less than 0.5, using the relation in the first tree, otherwise, using the relation in the second tree;

during solving, the position and the size are used as soft constraints, the relation which is required to be met between the nodes is used as hard constraints, and the objective function to be optimized is as follows:

；

wherein,

is an energy equation that measures the difference in position,

is an energy equation that measures the magnitude difference;

；

；

wherein,

is a node in the intermediate state tree;

representing the amount of position that the node is to solve for;

a magnitude value representing the node to be solved for; if it is used

Null, then indicating a function

Otherwise

；

Representing nodes

Corresponding to nodes in the first tree

The position of (a);

representing nodes

The size of (d);

representing nodes

Corresponding to nodes in the second tree

The position of (a);

representing nodes

The size of (d); the objective function is minimized under a series of hard constraints.

2. The hybrid generation method with hierarchical layout according to claim 1, wherein the calculation cost of the branch node corresponding to the branch node comprises:

neither branch node sinks; recursively invoking a compute penalty method to compute a cost between a child node of a first branch node and a child node of another branch node; calculating the cost of the child node of the first branch node to the null and the cost of the child node of the second branch node to the null; a two-dimensional matrix is constructed by using the calculated cost values and solved by Hungary algorithm to obtain a loss value which is recorded as

；

The first branch node sinks byStage, the second branch node does not sink; recursively invoking a compute loss method to compute a cost between a child node of a first branch node and another branch node; calculating the cost of the first branch node being empty and the cost of the child nodes of the second branch node being empty; constructing a two-dimensional matrix by using the calculated cost values and solving through a Hungarian algorithm to obtain a loss value which is recorded as

；

The first branch node does not sink, and the second branch node sinks one level; recursively invoking a compute penalty method to compute a cost between a child node of the first branch node and another branch node; calculating the cost of the child node of the first branch node to the null and the cost of the second branch node to the null; a two-dimensional matrix is constructed by using the calculated cost values, and a loss value is obtained and recorded by solving through Hungary algorithm

；

The final loss value

The element correspondence results in the two branch nodes are equal to the corresponding match with the smallest loss value; and transmitting the root nodes of the two layouts as parameters into a defined recursive function to obtain the total corresponding loss value and the corresponding information between the two layout elements.

3. The method of claim 2, wherein the dynamically determining intermediate state elements and relationships that should be satisfied between the elements based on the composition tree according to node types and in combination with parameters generated during user interaction specifically comprises:

the combined tree comprises three types of nodes, wherein the first type of node is a node corresponding to both the two trees; the second kind of nodes are corresponding nodes only in the first tree; the third kind of nodes are corresponding nodes only in the second tree;

respectively and uniformly distributing a deletion threshold value for the second class node and the third class node according to the level of the second class node and the third class node, and dynamically determining the relation which should be met between the tree structure of the intermediate state and the brother node according to the input parameters;

among them, the relationships between sibling nodes include equal size, equal spacing and alignment.

4. The hybrid generation method with hierarchical layout of claim 3, further comprising:

if two nodes are vertically aligned, then there are:

；

；

wherein,

representing nodes

The abscissa of the corresponding element;

representing nodes

The width of the corresponding element;

representing nodes

The abscissa of the corresponding element;

representing nodes

The width of the corresponding element.

5. A hybrid generation system having a hierarchical layout, the hybrid generation system having a hierarchical layout comprising:

the information calculation module is used for receiving a plurality of layout templates input by a user, wherein the layout templates are represented by a multi-branch tree structure, leaf nodes in the multi-branch tree structure represent single elements in the layout templates, and the corresponding relation between the elements is calculated according to a node corresponding rule to obtain corresponding information;

the relation determining module is used for establishing a combined tree according to the corresponding information and dynamically determining the intermediate state elements and the relation which should be met among the elements based on the combined tree according to the node type and in combination with the parameters generated during user interaction;

the optimization generation module is used for taking the positions and the sizes of the nodes as soft constraints, taking the relation which should be met among the nodes as hard constraints, optimizing an objective function, and generating a new diversity layout after the optimization solution is completed;

the node corresponding rules comprise a first corresponding rule, a second corresponding rule, a third corresponding rule and a fourth corresponding rule;

the first correspondence rule includes:

two leaf nodes with similar geometric information tend to correspond together;

two branch nodes with similar substructures tend to correspond together;

two leaf nodes with different semantic information cannot be corresponded together;

the second correspondence rule includes:

leaf nodes and branch nodes are allowed to be empty correspondingly;

the third rule of correspondence includes:

if two branch nodes

And

correspond together to form a subtree

A node in (2) can only correspond to a sub-tree

Node in or corresponding null;

the fourth rule of correspondence includes:

two nodes at different levels are allowed to correspond together;

wherein, the father node of the leaf node is a branch node;

the method for calculating the corresponding relation between the elements according to the node corresponding rule to obtain the corresponding information further comprises the following steps:

defining a function for calculating corresponding losses between two nodes and between the nodes and the spaces by using a recursive method;

the corresponding cost of two leaf nodes defines two normal forms of the position difference of two nodes, the two normal forms of the size difference and a first punishment item related to semantic information

The introduction of the first penalty term makes two nodes with different semantic information unable to be corresponded together, and is in the following form:

；

；

wherein,

and

respectively representing two leaf nodes;

calculating the corresponding cost of the two leaf nodes;

and

respectively representing the positions of two leaf nodes;

and

respectively representing the sizes of two leaf nodes;

and

respectively representing semantic information corresponding to the two leaf nodes;

leaf node to null cost

A loss corresponding to a node with a size of 0 for the leaf node, specifically a two-norm of the difference between the size of the node and 0 plus a second penalty term

The method comprises the following steps:

；

wherein, setting

；

The empty cost corresponding to a branch node is defined as the cumulative sum of the empty costs corresponding to all leaf nodes in the branch, and is:

；

wherein,

is a function of the costs of the branch nodes corresponding to the nulls,

is a leaf node below the branch node;

cost of branch node corresponding to leaf node: recursively calling a calculation loss method to calculate the costs of the child nodes and the leaf nodes of the branch nodes, and calculating the costs of the child nodes of the branch nodes and the leaves of the leaf nodes; constructing a two-dimensional matrix by using the calculated cost values, and solving by using a Hungarian algorithm to obtain a minimum loss and an optimal matching result;

the optimizing an objective function by using the positions and sizes of the nodes as soft constraints and using the relation which should be satisfied between the nodes as hard constraints specifically comprises the following steps:

when the parameter value is inputted

If the value is less than 0.5, using the relation in the first tree, otherwise, using the relation in the second tree;

during solving, the position and the size are used as soft constraints, the relation which is required to be met between the nodes is used as hard constraints, and the objective function to be optimized is as follows:

；

wherein,

is an energy equation that measures the difference in position,

is an energy equation that measures the magnitude difference;

；

；

wherein,

is a node in the intermediate state tree;

representing the amount of position to be solved for by the node;

representing a magnitude value to be solved for the node; if it is used

Null, then indicating a function

Otherwise

；

Representing nodes

Corresponding to nodes in the first tree

The position of (a);

representing nodes

The size of (d);

representing nodes

Corresponding to nodes in the second tree

The position of (a);

representing nodes

The size of (d); the objective function is minimized under a series of hard constraints.

6. A terminal, characterized in that the terminal comprises: memory, a processor and a hybrid generating program with a hierarchical layout stored on the memory and executable on the processor, the hybrid generating program with a hierarchical layout implementing the steps of the hybrid generating method with a hierarchical layout of any one of claims 1 to 4 when executed by the processor.

7. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a hybrid generation program with a hierarchical layout, which when executed by a processor implements the steps of the hybrid generation method with a hierarchical layout according to any one of claims 1 to 4.

Images