CN111222519B - Construction method, method and device of hierarchical colored drawing manuscript line extraction model - Google Patents
Construction method, method and device of hierarchical colored drawing manuscript line extraction model Download PDFInfo
- Publication number
- CN111222519B CN111222519B CN202010049526.4A CN202010049526A CN111222519B CN 111222519 B CN111222519 B CN 111222519B CN 202010049526 A CN202010049526 A CN 202010049526A CN 111222519 B CN111222519 B CN 111222519B
- Authority
- CN
- China
- Prior art keywords
- image
- edge
- extraction model
- line
- line draft
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/181—Segmentation; Edge detection involving edge growing; involving edge linking
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
本发明公开了一种层次化彩绘文物线稿提取模型构建、方法及装置，首先通过提出新的加权损失函数，在BDCN边缘检测神经网络中引入传统FDoG方法，从而得到更加完整的线稿粗提取结果；再通过提出新的多尺度U‑Net网络，在粗提取的基础上抑制病害，得到更为细致、干净的线稿图像。该方法有效结合了BDCN神经网络和传统FDoG方法的优点，在学习明显边缘的同时能够同时学习细节，使得提取的线稿更加完整，甚至补充部分受病害影响断裂的线稿，进一步的，通过多尺度U‑Net网络细化线稿，抑制病害。与现有方法相比，该方法在干净以及复杂背景下均能更有效提取文物线稿图像。
The invention discloses a model construction, method and device for extracting line drawings of layered painted cultural relics. First, by proposing a new weighted loss function, the traditional FDoG method is introduced into the BDCN edge detection neural network, thereby obtaining a more complete line drawing rough extraction Results; Then, by proposing a new multi-scale U-Net network, the disease is suppressed on the basis of rough extraction, and a more detailed and clean line drawing image is obtained. This method effectively combines the advantages of the BDCN neural network and the traditional FDoG method. It can learn the details while learning the obvious edges, making the extracted line drawings more complete, and even supplementing some broken lines affected by diseases. Further, through multiple The scale U‑Net network refines the line draft and suppresses diseases. Compared with existing methods, this method is more effective in extracting line art images of cultural relics in both clean and complex backgrounds.
Description
Technical Field
The invention relates to a method for extracting line drafts of colored drawing cultural relics, in particular to a method and a device for constructing a hierarchical colored drawing cultural relic line drafts extraction model.
Background
The colored drawing cultural relics are left as historical materials, contain cultural arts, religious beliefs and environmental changes of different generations and different regions, inherit the cultural essence of one country of history, enrich the diversity of world culture, and provide valuable materials for human research history. The line draft of the colored drawing cultural relic reflects the main structure and content of the colored drawing pattern and is an important link for copying and repairing the colored drawing cultural relic. However, since many painted cultural relics are affected by natural environment, human factors and the like in the long term, diseases of different degrees, such as nail lifting, cracking, fading and the like, are generated, so that the background of the painted cultural relics becomes complex, and the accurate extraction of the line manuscript is challenged.
The traditional line draft drawing mainly depends on manual work, and the drawing process is very complex and time-consuming. The method has the advantages of high efficiency, reusability and the like by utilizing the computer and the image processing technology to extract the line draft of the colored drawing cultural relic. The line draft extraction key technology relates to edge extraction. Currently, edge extraction techniques mainly involve two types: conventional image processing algorithms and algorithms based on deep learning. Compared with the classical traditional image processing algorithms including Canny operators, FDoG algorithms and the like, line drafts generated by the Canny operators are poor in smoothness and have a shaking phenomenon; the FDoG algorithm improves the continuity of lines, has a good extraction effect on the cultural relic image with a clean background, is greatly influenced by noise on a complex background, extracts a plurality of non-line draft edges, and greatly reduces the accuracy. In recent years, edge extraction techniques based on deep learning include: the neural networks such as the HED, the RCF and the BDCN can introduce priori knowledge through a large amount of sample learning, so that the accuracy of edge detection is greatly improved. However, if the neural network-based method is directly applied to cultural relic data, particularly cultural relic data with a complex background, the extracted edges have the serious problems of blurring, artifacts and detail loss.
Disclosure of Invention
The invention aims to provide a construction method and a method of a hierarchical colored drawing line draft extraction model of FDoG and neural network, which are used for solving the problem of low extraction quality of line drafts under complex disease backgrounds in the colored drawing line draft extraction method in the prior art.
In order to realize the task, the invention adopts the following technical scheme:
a method for constructing a hierarchical colored drawing manuscript extraction model is implemented according to the following steps:
performing edge extraction on the line manuscript image of each colored drawing cultural relic image in the first sample set by adopting an FDoG algorithm to obtain an FDoG label set;
acquiring a line draft image of each colored drawing cultural relic image in a first sample set, and acquiring a first line draft label set;
step 2, taking the first sample set as input, taking the FDoG label set and the first line draft label set as reference output, training an edge extraction network, and obtaining a coarse extraction model;
the edge extraction network is obtained by training a BDCN edge detection network by taking an edge data set as input and an edge label set as reference output;
the edge data set comprises a plurality of natural images, and the edge tag set comprises an edge tag image corresponding to each natural image in the edge data set and an FDoG edge tag image obtained after edge extraction is carried out on each natural image in the edge data set;
step 3, collecting a plurality of color-painted cultural relic images to obtain a second sample set; wherein each painted cultural relic image in the second sample set is different from the painted cultural relic image in the first sample set;
acquiring a line draft image of each colored drawing cultural relic image in the second sample set, and acquiring a second line draft label set;
step 4, inputting the second sample set obtained in the step 3 into the coarse extraction model obtained in the step 2 to obtain a coarse draft image set;
step 5, taking the thick line draft image set as input, taking the second line draft label set as reference output, training a convolutional neural network, and obtaining a thin line draft extraction model;
and 6, obtaining a line draft extraction model, wherein the line draft extraction model comprises a crude extraction model obtained in the step 2 and a fine line draft extraction model obtained in the step 5 which are sequentially connected.
Further, the convolutional neural network in the step 5 comprises a U-net network, a side output layer and a fusion output layer which are sequentially arranged;
the side output layer comprises a first convolution layer, an up-sampling layer and a first activation function layer which are sequentially arranged; the fusion output layer comprises a second convolution layer and a second activation function layer which are arranged in sequence.
Further, when the convolutional neural network is trained in the step 5, a random gradient is adoptedTraining with a descent algorithm, wherein the loss function L fine Comprises the following steps:
wherein ω is side Representing specific gravity, ω, in the loss function for different scale features fuse Representing the specific gravity of the fusion feature in the loss function; s denotes the scale, S =1,2, \ 8230, S, S is a positive integer,
in the form of a single-channel loss function,
for a fusion loss function, is>
Y represents an image output after inputting a thick line draft image into a thin line draft extraction model R A line draft label image corresponding to the thick line draft image is shown, and R shows the thick line draft image.
Further, when the BDCN edge detection network is trained by using the edge data set and the edge label set in step 2, the loss function L is obtained pre Comprises the following steps:
where S denotes the scale, S =1,2, \ 8230;, S, S are positive integers, α is the weight of the edge label in the objective function, β is the weight of the FDoG edge label, P is N Representing the output image after the natural image N is input to the BDCN edge detection network,
represents the edge label image corresponding to the natural image N, and>
representing the FDoG edge label image corresponding to the natural image N.
A method for extracting line draft of a layered colored drawing cultural relic comprises the steps of inputting a colored drawing cultural relic image to be extracted into a line draft extraction model obtained by a method for constructing the layered colored drawing cultural relic line draft extraction model, and obtaining a colored drawing cultural relic line draft image.
A hierarchical colored drawing manuscript extraction model building device comprises a first data set building module, a crude extraction model obtaining module, a second data set building module, a thick manuscript extraction module, a fine extraction model obtaining module and a line manuscript extraction model obtaining module;
the first data set construction module is used for acquiring a plurality of colored drawing cultural relic images to obtain a first sample set;
performing edge extraction on the line manuscript image of each colored drawing cultural relic image in the first sample set by adopting an FDoG algorithm to obtain an FDoG label set;
acquiring a line draft image of each colored drawing cultural relic image in a first sample set, and acquiring a first line draft label set;
the rough extraction model obtaining module is used for taking the first sample set as input, taking the FDoG tag set and the first line draft tag set as reference output, training an edge extraction network and obtaining a rough extraction model;
the edge extraction network is obtained by adopting an edge data set as input and adopting an edge label set as reference output to train the BDCN edge detection network;
the edge data set comprises a plurality of natural images, and the edge tag set comprises an edge tag image corresponding to each natural image in the edge data set and an FDoG edge tag image obtained after edge extraction of each natural image in the edge data set;
the second data set construction module is used for acquiring a plurality of colored drawing cultural relic images to obtain a second sample set; wherein each painted cultural relic image in the second sample set is different from the painted cultural relic image in the first sample set;
acquiring a line draft image of each colored drawing cultural relic image in a second sample set, and acquiring a second line draft label set;
the thick line manuscript extraction module is used for inputting the obtained second sample set into a thick extraction model to obtain a thick line manuscript image set;
the fine extraction model obtaining module is used for taking the thick line manuscript image set as input, taking the second line manuscript label set as reference output, training a convolution neural network and obtaining a fine line manuscript extraction model;
the line draft extraction model obtaining module is used for obtaining a line draft extraction model, and the line draft extraction model comprises a crude extraction model and a fine line draft extraction model which are sequentially connected.
Further, the convolutional neural network in the fine extraction model obtaining module comprises a U-net network, a side output layer and a fusion output layer which are sequentially arranged;
the side output layer comprises a first convolution layer, an up-sampling layer and a first activation function layer which are sequentially arranged; the fusion output layer comprises a second convolution layer and a second activation function layer which are arranged in sequence.
Further, when the convolutional neural network is trained in the fine extraction model obtaining module, a stochastic gradient descent algorithm is adopted for training, wherein a loss function L fine Comprises the following steps:
wherein ω is side Representing specific gravity, ω, in loss functions corresponding to different scale features fuse Representing the specific gravity of the fusion feature in the loss function; s denotes the scale, S =1,2, \ 8230, S, S is a positive integer,
in the form of a single-channel loss function,
for a fusion loss function, is>
Y represents an image output after inputting a thick line draft image into a thin line draft extraction model R A line draft label image corresponding to the thick line draft image is shown, and R shows the thick line draft image.
Further, when the coarse extraction model obtaining module adopts the edge data set and the edge label set to train the BDCN edge detection network, the loss function L is obtained pre Comprises the following steps:
where S denotes the scale, S =1,2, \ 8230;, S, S are positive integers, α is the weight of the edge label in the objective function, β is the weight of the FDoG edge label, P is N Representing the output image after the natural image N is input to the BDCN edge detection network,
represents the edge label image corresponding to the natural image N, and>
representing the FDoG edge label image corresponding to the natural image N.
A layered colored drawing cultural relic line draft extraction device is used for inputting the colored drawing cultural relic image to be extracted into a line draft extraction model obtained by a layered colored drawing cultural relic line draft extraction model construction device to obtain a colored drawing cultural relic line draft image.
Compared with the prior art, the invention has the following technical effects:
1. the invention provides a line draft crude extraction method for a hierarchical colored drawing manuscript of an FDoG and neural network, and provides a method and a device for constructing a line draft extraction model of the FDoG and neural network, wherein a new weighting loss function is designed, a traditional FDoG algorithm is introduced into a BDCN edge detection neural network, the FDoG algorithm and the BDCN edge detection neural network are jointly trained to perform transfer learning from an edge detection task to a colored drawing manuscript extraction task, and the advantages of extracting the coherence edge feature by a deep convolutional network and the advantage of extracting the detail feature by the traditional algorithm are fully combined, so that the line draft information of the crude extracted manuscript is more complete, and even a part of line drafts influenced by diseases can be supplemented;
2. the invention provides a line draft fine extraction method, which is used for constructing a hierarchical colored drawing manuscript extraction model of an FDoG (fully drawn wire) and a neural network, wherein a designed new multi-scale U-Net network samples the features of different scales of each layer in the decoding process of the traditional U-Net network to the size of an original image, and the features of each scale are calculated by utilizing a loss function and are fully fused to capture rich disease multi-scale features, so that the disease noise which cannot be removed by the traditional algorithm is effectively inhibited, and the line draft in the colored drawing manuscript can be further refined and reduced;
3. the method and the device for extracting the FDoG and neural network layered colored drawing manuscript divide line manuscript extraction into two steps, firstly solve the problem of insufficient data by using the mobility of a BDCN edge detection network, restore the complete information of line manuscript edge details, supplement broken lines affected by diseases and perform coarse extraction on the line manuscript. And then, the crude extraction is used as prior knowledge, and diseases which cannot be eliminated by introducing a traditional algorithm in the first step and line blurring caused by a BDCN model are inhibited on the basis of the prior knowledge. The two steps are fully matched, and the advantages of the BDCN edge detection network and the advantages of the U-Net network are effectively combined, so that the extracted cultural relic line manuscript is coherent, complete, clear and clean, and is more comprehensive and accurate.
Drawings
FIG. 1 is an image of a painted cultural relic provided in an embodiment of the invention; FIG. 1 (a) is an image of a painted cultural relic under test; fig. 1 (b) is a line draft image extracted by the FDoG algorithm; fig. 1 (c) is a real line draft image provided;
FIG. 2 is a natural image provided in one embodiment of the present invention; FIG. 2 (a) illustrates natural images used for training; fig. 2 (b) is a line draft image extracted by the FDoG algorithm; FIG. 2 (c) is the original edge label;
FIG. 3 is an internal structure diagram of a thin line script extraction model provided in an embodiment of the invention;
FIG. 4 is a graph of the results of two-step extraction of a painted manuscript in an ablation experiment analysis provided in an embodiment of the present invention; FIG. 4 (a) is a color-painted cultural relic graph used in the test; FIG. 4 (b) is a rough drawing of a painted cultural relic line manuscript; FIG. 4 (c) is the final fine-extracted line drawing of the painted cultural relic; fig. 4 (d) is a real line draft image of the color-painted manuscript drawn by the expert;
FIG. 5 is a graph illustrating the respective individual extraction of sketches of painted cultural relics using BDCN and U-Net networks in an ablation experimental analysis provided in an embodiment of the present invention; FIG. 5 (a) is a color-painted cultural relic graph used in the test; fig. 5 (b) is a line draft result diagram obtained by the colored drawing cultural relic data only trained on the BDCN network, fig. 5 (c) is a line draft result diagram obtained by the colored drawing cultural relic data only trained on the U-Net network, and fig. 5 (d) is a line draft result diagram extracted by the hierarchical framework of the present invention;
fig. 6 is a comparison result graph of results of an existing classical edge detection algorithm provided in an embodiment of the present invention, and fig. 6 (a) is a graph of extraction results of a line script by a flow-based gaussian difference algorithm FDoG; FIG. 6 (b) is a diagram of the extraction result of Edge-Boxes Edge detection algorithm on the line script; fig. 6 (c) is an extraction result of the RCF network on the line draft; fig. 6 (d) is a diagram showing the result of BDCN network line draft extraction; fig. 6 (e) is a diagram of an extraction result of the line script by the method provided by the present invention.
Detailed Description
The present invention will be described in detail below with reference to the accompanying drawings and examples. So that those skilled in the art can better understand the present invention. It is to be expressly noted that in the following description, a detailed description of known functions and designs will be omitted when it may obscure the main content of the present invention.
The following definitions or conceptual connotations relating to the present invention are provided for illustration:
BDCN network: the network is a convolutional neural network structure based on deep learning and is used for an edge detection task.
FDoG algorithm: the Gaussian difference filtering algorithm based on the flow is a classic traditional edge detection operator. The algorithm preserves salient edges in the image, and weak edges are guided to follow the consistent direction of the salient edges in the neighborhood thereof, so as to extract details in the line draft and maximize line continuity.
U-net network: the network is a convolutional neural network structure based on deep learning, and comprises a contraction path to learn deep advanced features and an expansion path to learn position features, and the downsampled original features are spliced into the upsampled features. The method is originally used for a medical image segmentation task, and is used for various tasks to realize feature extraction due to the simplicity and effectiveness of the structure.
Example one
The embodiment discloses a method for constructing a hierarchical colored drawing manuscript extraction model, which is implemented according to the following steps:
performing edge extraction on the line manuscript image of each colored drawing cultural relic image in the first sample set by adopting an FDoG algorithm to obtain an FDoG label set;
acquiring a line draft image of each colored drawing cultural relic image in a first sample set, and acquiring a first line draft label set;
in this step, the mode of collecting the plurality of color-painted cultural relic images can be shooting and collecting by a camera or other devices, and can also be calling and collecting from a database.
In this embodiment, the obtained colored drawing cultural relic image is as shown in fig. 1 (a), and an FDoG algorithm is adopted to perform edge extraction on the colored drawing cultural relic image as shown in fig. 1 (a) to obtain an edge label image as shown in fig. 1 (b); the line manuscript image obtained by extracting the hand-drawn line manuscript from the color-drawn document image shown in fig. 1 (a) is shown in fig. 1 (c).
Step 2, taking the first sample set as input, taking the FDoG label set and the first line draft label set as reference output, training an edge extraction network, and obtaining a coarse extraction model;
the edge extraction network is obtained by training a BDCN edge detection network by taking an edge data set as input and an edge label set as reference output;
the edge data set comprises a plurality of natural images, and the edge tag set comprises an edge tag image corresponding to each natural image in the edge data set and an FDoG edge tag image obtained after edge extraction of each natural image in the edge data set;
in this step, the edge label image corresponding to each natural image is a real edge image of each natural image, and this real edge image may be manually drawn; the FDoG edge label image corresponding to each natural image is obtained after edge extraction is carried out through an FDoG algorithm, and in the step, the two parts of data are combined to form a pre-training set of each group of three images serving as a crude extraction network. In this step, the natural image is not a painted cultural relic image.
In the step, the existing BDCN edge detection network is used as the basis of a rough extraction model, the BDCN edge detection network takes VGG16 as a basic structure, each layer is subjected to multiple expansion convolution and down sampling to obtain features of different scales, and finally, multi-scale information under different depths is fused.
In this embodiment, the BDCN Edge Detection Network is a Network structure proposed by "Bi-Directional Cascade Network for spatial Edge Detection" and used for Edge Detection task.
In this embodiment, to solve the problem of insufficient cultural relic data set, multiple sets of public data for edge detection are obtained to pre-train a crude extraction network, and a public data set of BSDS500 is obtained, where each set includes a natural image X N And a corresponding edge label
The edge label contains a large amount of edge information but lacks detail; making a corresponding label @ using a conventional edge detection algorithm for FDoG>
The label contains relatively rich detail information and containsA certain noise; the natural image data is illustrated in fig. 2 (a), the corresponding edge label is illustrated in fig. 2 (b), and the edge label obtained by performing the FDoG algorithm on fig. 2 (a) is illustrated in fig. 2 (c); natural image data X N Inputting the data into a BDCN edge detection network for training, and obtaining a multi-scale fusion characteristic P through learning N The feature has information of edges and details at the same time, an edge extraction network is obtained after training is completed, and the network model learns the information of edges and details of the natural image style at the same time;
in the step, the FDoG edge label image is used as a label and retrained on the edge detection model pre-trained by BDCN, so that the network function with strong edge detection capability is fully utilized, and the trained edge extraction network fully combines the advantage of extracting the marginal characteristic by the deep convolutional network and the advantage of extracting the detailed characteristic by the traditional algorithm, so that the extraction result of the edge extraction network is more accurate;
when training the BDCN edge detection network, continuously optimizing the loss function by adopting a random gradient descent algorithm until convergence.
Optionally, in step 2, when the BDCN edge detection network is trained by using the edge data set and the edge label set, the loss function L is used pre Comprises the following steps:
where S denotes the scale, S =1,2, \ 8230;, S, S are positive integers, α is the weight of the edge label in the objective function, β is the weight of the FDoG edge label, P is N Representing the output image after the natural image N is input to the BDCN edge detection network,
represents the edge label image corresponding to the natural image N, and>
FDoG edge tag image corresponding to natural image N。
In this embodiment, α is a weight for learning an edge detection tag in an objective function, β is a weight for learning an FDoG tag, and is used to balance the proportion of the edge tag and the detail tag, and directly affect the characteristics finally learned by the network, where α =0.9 and β =0.1, s represents a scale;
the L function is a cross entropy loss function defined as:
wherein,
defining a predicted map, based on the predicted location of the reference>
Referring to each normalized pixel in the prediction map, μ = λ · Y + |/(|Y + |+|Y - |)，ν＝|Y - |/(|Y + |+|Y - I) to balance edge and non-edge pixels, Y + Denotes a positive sample, Y - Indicating negative samples, and lambda controls the specific gravity of the positive and negative samples.
In this step, the edge extraction network is trained by using the first sample set as input and the FDoG tag set and the first line draft tag set as reference output, and finally the rough extraction model is obtained, wherein the purpose of using the first sample set as input and the FDoG tag set and the first line draft tag set as reference output is to enable the edge extraction network to learn to obtain the line draft feature P with the style characteristics of the colored drawing cultural relic R The feature contains information about the edges and details.
In this embodiment, a random gradient descent method is also used when the edge extraction network is trained to obtain a converged crude extraction model; similarly, the loss function at this time is:
the model is obtained by training colored drawing cultural relic data, can extract information with the style characteristics of the colored drawing cultural relic, and has strong edge detection network performance obtained by training a large amount of natural data so as to extract rich edges and details and supplement lines which are partially broken due to breakage. However, due to the complexity of the cultural relic background, the traditional algorithm FDoG cannot effectively inhibit diseases, so that the roughly extracted line draft contains noise caused by certain diseases.
Step 3, collecting a plurality of color-painted cultural relic images to obtain a second sample set; wherein each painted cultural relic image in the second sample set is different from the painted cultural relic image in the first sample set;
acquiring a line draft image of each colored drawing cultural relic image in a second sample set, and acquiring a second line draft label set;
in this step, an image different from the color-painted cultural relic image obtained in step 1 is used.
Acquiring other groups of colored drawing cultural relic data, wherein each group comprises a colored drawing cultural relic image X R And a label Y of line draft of the colored drawing cultural relic made by an expert R As a test set for the coarse extraction network.
Step 4, inputting the second sample set obtained in the step 3 into the coarse extraction model obtained in the step 2 to obtain a coarse line manuscript image set;
in the step, the model obtained in the step 2 is used for testing the colored drawing cultural relic data of the test set obtained in the step 3 to obtain a plurality of groups of crude extraction line manuscript image pairs, wherein each group comprises one crude extraction line manuscript image
And a label chart Y made by an expert R 。
Step 5, taking the thick line draft image set as input, taking the second line draft label set as reference output, training a convolutional neural network, and obtaining a thin line draft extraction model;
in the present step, the crude extract line draft outputted from step 4 is plotted
The composed thick line manuscript image set is used as input.
In the embodiment, the convolutional neural network is based on a U-Net network, but is different from the U-Net network in the prior art in that the features of each layer in the decoding process are up-sampled to the original image size to form feature maps with different scales, and then the feature maps with the scales are fused to form a network structure of a fused feature map.
Optionally, the convolutional neural network in step 5 includes a U-net network, a side output layer, and a fusion output layer that are sequentially arranged;
the side output layer comprises a first convolution layer, an up-sampling layer and a first activation function layer which are sequentially arranged; the fusion output layer comprises a second convolution layer and a second activation function layer which are sequentially arranged.
In this embodiment, a plurality of upsampling layers of the U-net network are respectively connected to side output layers, wherein the side output layers are connected to the upsampling layers by a first convolution layer, and the upsampling layers are connected to the first activation function layer. The upper sampling layer is spliced with the U-net output layer channel at the same time, and then is connected with the fusion output layer, and the fusion output layer is connected with the second activation function layer through the second convolution layer.
As shown in fig. 3, the U-net network is mainly composed of a maximum pooling (downsampling) module, a deconvolution (upsampling) module, and a ReLU nonlinear activation function. The whole network process is as follows:
the maximum pooling modules perform downsampling, as shown in fig. 3, each maximum pooling module includes two convolution blocks and one maximum pooling layer, and each convolution block includes 3 × 3 convolution layers, a normalization layer, and an activation function layer, which are sequentially connected;
in this embodiment, four maximal pooling modules are provided, and two 3 × 3 volume blocks are spliced after the 4 th maximal pooling.
The method comprises the following steps that up-sampling is carried out by deconvolution (up-sampling) modules, each up-sampling module comprises two convolution blocks and an up-sampling layer, and each convolution block comprises a 3 x 3 convolution layer, a normalization layer and an activation function layer which are sequentially connected;
in this embodiment, four upsampling modules are correspondingly arranged.
In this embodiment, an output 1 × 1 convolution layer and an activation function layer are connected in sequence after the last up-sampling module.
In this embodiment, as shown in fig. 3, the first convolution layer performs 1 × 1 convolution on the output of the second single 3 × 3 convolution block and the output of the first three up-sampling modules in the U-net network, and then outputs four convolved features, the first up-sampling layer is used to perform up-sampling on the four convolved features, and then outputs the four up-sampled features, and the four up-sampled features are input to the first activation function layer to obtain 4 single-channel loss values; inputting 1 x 1 convolution layer in the U-net output layer into the activation function layer, and obtaining 1 single-channel loss value; calculating the 5 single-channel losses to obtain 5-scale side outputs of the multi-scale U-Net;
in addition, the four up-sampled features are spliced with the output features of the output 1 × 1 convolution layer in the U-net network, and are simultaneously input into a second convolution layer to carry out convolution by 1 × 1, so that a fusion feature is obtained, the feature is input into a second activation function layer to obtain a fusion loss value, and finally, multi-scale fusion output is obtained, wherein the fusion output is a fine extraction line draft.
And 6, obtaining a line draft extraction model, wherein the line draft extraction model comprises a crude extraction model obtained in the step 2 and a fine line draft extraction model obtained in the step 5 which are sequentially connected.
Optionally, when the convolutional neural network is trained in step 5, a stochastic gradient descent algorithm is used for training, wherein the loss function L is fine Comprises the following steps:
wherein ω is side Representing specific gravity, ω, in the loss function for different scale features fuse Representing the specific gravity of the fusion features in the loss function(ii) a S denotes the scale, S =1,2, \ 8230, S, S is a positive integer,
is a function of the loss of a single channel,
for a fusion loss function, is>
Y represents an image output after inputting a thick line draft image into a thin line draft extraction model R A line draft label image corresponding to the thick line draft image is shown, and R shows the thick line draft image.
In this embodiment, the single-channel loss function and the fusion loss function are both cross-entropy loss functions.
In the present embodiment, S =5, ω fuse ＝1.1；
When s =1, ω side =0.8; when s =2, ω side =0.8; when s =3, ω side =0.4; when s =4, ω side =0.4; when s =5, ω side ＝0.4。
In the method for constructing the line draft extraction model of the color-drawing manuscript provided in this embodiment, the complete information of the edge details of the line draft is restored, the broken lines affected by the disease are supplemented, and the line draft is extracted roughly. And then, the diseases are inhibited on the basis of the rough extraction, and the noise introduced by the diseases is removed, so that a more detailed and clean line draft is obtained. The two steps are fully matched, and the advantages of the BDCN edge detection network and the advantages of the U-Net network are effectively combined, so that the extracted cultural relic line manuscript is more comprehensive and accurate. The embodiment provides a new weighting loss function, introduces a traditional FDoG method into a BDCN edge detection neural network, and performs transfer learning from an edge detection task to a painted manuscript extraction task by combining the two methods, so that the advantages of deep convolution network extraction of edge characteristics and the advantages of traditional algorithm extraction of detail characteristics are fully combined, the information of a roughly extracted manuscript line is more complete, and even a part of line manuscripts broken under the influence of diseases can be supplemented; the method provides a new multi-scale U-Net network for fine extraction of line draft, and samples the features of different scales of each layer in the decoding process of the U-Net network to the size of the original image, and performs reasonable feature fusion, thereby effectively inhibiting the disease noise which cannot be removed by the traditional algorithm, and further refining and restoring the line draft in the colored drawing cultural relic.
Example two
In this embodiment, a method for extracting line draft of a layered colored drawing cultural relic is provided, in which a to-be-extracted colored drawing cultural relic image is input into a line draft extraction model obtained by the method for constructing a layered colored drawing cultural relic line draft extraction model in the first embodiment, so as to obtain a colored drawing cultural relic line draft image.
In the embodiment, two types of line draft images of the colored drawing cultural relics are adopted to verify the effectiveness of the line draft extraction method provided by the invention; the method comprises the following steps of cleaning a colored drawing cultural relic image with a background, wherein the colored drawing cultural relic image has less disease distribution such as falling, cracks, stains and the like; colored drawing cultural relic images with complex backgrounds, which have a certain degree of cracks, shedding and noise under the influence of diseases. The clean background colored drawing cultural relics image adopted in the experiment is a copy Dunhuang wall painting image, the complex background colored drawing image is a Tang nationality prince tomb horse racing image and a Tang Wei prince tomb horse preparation image, and the sizes of the images are 480 multiplied by 1264, 180 multiplied by 336, 640 multiplied by 640 and 800 multiplied by 736 respectively.
In order to better evaluate the effectiveness of the line draft extraction method, the invention provides two experimental types, which are respectively as follows: ablation experiments for the algorithm of the present invention and comparison experiments with existing edge detection algorithms. The ablation experiment mainly verifies the effectiveness of the hierarchical extraction process, the comparison experiment is mainly compared with several existing Edge detection algorithms, including traditional Edge detection algorithms Canny, FDoG, edge-Box and the like, and meanwhile, the comparison experiment is also performed with Edge detection algorithms based on deep learning, such as HED, RCF and BDCN.
In the experiment, a pitorch software package is used for training a network, and in the first step of crude extraction network training, 9000 iterations are performed when natural image pair training is performed due to the fact that a pre-training model for edge detection is loaded; when training is carried out by using cultural relic image data, about 10500 iterations and batches are carried outThe size is set to 10; for the stochastic gradient descent algorithm, the initial learning rate is 1e -6 Setting the weight attenuation to 0.0002 and the momentum to 0.9; in the second step of fine extraction network training, approximately 20000 iterations are performed, and the batch size is set to 10; for the stochastic gradient descent algorithm, the initial learning rate is 1e -4 The weight attenuation is set to 0.0002 and the momentum to 0.9.
Ablation experiment analysis:
fig. 4 is a result of two-step line draft extraction of the colored drawing cultural relics in the ablation experimental analysis, and fig. 4 (a) is a colored drawing cultural relics graph used in the test; fig. 4 (b) is a rough-extracted colored drawing manuscript; fig. 4 (c) is the final fine-extracted colored drawing manuscript of the cultural relic line; fig. 4 (d) is a real label of the painted line manuscript drawn by the expert; by visually observing the crude extraction line draft image and the fine extraction line draft image, the line draft extracted by the crude extraction network contains more complete details and edge information, and simultaneously lines which are partially broken due to damage of diseases are connected, but noise introduced by the diseases cannot be effectively inhibited; the line draft extracted by the fine extraction network removes most of noise and further refines the line draft.
FIG. 5 is the result of extracting line draft of the colored drawing cultural relics by BDCN and U-Net network respectively in the analysis of ablation experiment, and FIG. 5 (a) is the graph of the colored drawing cultural relics used in the test; fig. 5 (b) is a line draft result obtained by only performing BDCN network training by using the color-painted historical relic data pair, fig. 5 (c) is a line draft result obtained by only performing U-Net network training by using the color-painted historical relic data pair, and fig. 5 (d) is a result of hierarchical line draft extraction by the method of the present invention; as can be seen from comparison, the BDCN network training result with the edge detection model migrated has a good image edge information extraction effect, but has artifacts and blurring phenomena; the training result of the U-Net network has no pixel blurring and artifacts, but the information loss is serious because there is no pre-trained edge detection model. The method firstly utilizes the advantages of the migratability of the BDCN network, combines the FDoG algorithm to extract more complete detail edge information, takes the result as the prior knowledge of the U-Net to train, solves the problem of fuzzy artifact of the BDCN, better utilizes the U-Net to extract fine line draft, and the final extraction result is superior to the former two in both the integrity of the line draft and the style reduction degree.
Comparing with a classical edge detection algorithm, analyzing by an experiment:
fig. 6 is a comparison result with a result of a conventional classical edge detection algorithm, and fig. 6 (a) is a result of extracting a line draft by a flow-based gaussian difference algorithm FDoG; FIG. 6 (b) is the extraction result of Edge-Box Edge detection algorithm on line draft; fig. 6 (c) is an extraction result of the RCF network on the line draft; fig. 6 (d) is the result of BDCN network line draft extraction; FIG. 6 (e) is the result of extracting line script by the algorithm of the present invention; visual observation shows that the line draft extraction based on the FDoG algorithm contains more details, but noise caused by diseases such as cracks and falling is obvious; the line draft extraction result of the Edge-Box algorithm is better than FDoG in noise removal, but is not clear, low in resolution and incomplete in details; the results extracted by the edge detection network based on RCF and BDCN are better in denoising and contour positioning, but have the phenomena of serious loss of details, thicker lines and artifacts; the result of extracting the line manuscript by the algorithm is superior to the algorithms in noise removal and detail expression, the problems of line thickness and artifacts extracted by an edge detection network are solved, and the extracted line manuscript image can clearly restore the original artistic characteristics of the colored drawing cultural relic.
Visual evaluation comparison can visually recognize the line draft extraction result, and meanwhile, objective evaluation indexes are used for evaluating the extraction result; the invention adopts three objective evaluation indexes of RMSE, SSIM and AP to carry out comprehensive evaluation on the image; wherein, RMSE is root mean square error, which is used for measuring the deviation between an observed value and a true value; SSIM is a structural similarity index, and the structural similarity of the image is evaluated through three aspects of contrast brightness, contrast and structure; the AP is average precision, measures the accuracy and integrity of a classification result and is a common objective index of a pixel-level classification task.
Table 1 shows the objective index comparison of line draft results extracted by the algorithm of the present invention and several classical edge detection algorithms; the first part is a comparison with classical conventional edge detection algorithms; the second part is compared with the edge detection algorithm based on deep learning in recent years, wherein the x represents the result of training by adopting the colored drawing cultural relic data pair; the third part is the comparison of the results of the traditional U-Net network for the line refinement extraction network and the multiscale U-Net network of the present invention. Most indexes of the method are superior to those of other methods, wherein the AP value of the method is far higher than that of other methods, namely the method is greatly improved in the integrity and the accuracy of online draft extraction compared with other methods, and meanwhile, the RMSE value and the SSIM value are also generally higher than those of other methods, which shows that the method is closer to the structural similarity of a label and has smaller error.
By integrating visual evaluation and objective index evaluation, the algorithm provided by the invention can well extract the line draft of the colored drawing cultural relic and more completely express the artistic characteristics of the colored drawing cultural relic.
TABLE 1 Objective index of extraction result of line draft of color-painted cultural relics
EXAMPLE III
The device comprises a first data set construction module, a crude extraction model obtaining module, a second data set construction module, a thick line draft extraction module, a fine extraction model obtaining module and a line draft extraction model obtaining module;
the first data set construction module is used for acquiring a plurality of colored drawing cultural relic images to obtain a first sample set;
performing edge extraction on the line manuscript image of each colored drawing cultural relic image in the first sample set by adopting an FDoG algorithm to obtain an FDoG label set;
acquiring a line draft image of each colored drawing cultural relic image in a first sample set, and acquiring a first line draft label set;
the rough extraction model obtaining module is used for taking the first sample set as input, taking the FDoG label set and the first line draft label set as reference output, training an edge extraction network and obtaining a rough extraction model;
the edge extraction network is obtained by training a BDCN edge detection network by using an edge data set as input and an edge label set as reference output;
the edge data set comprises a plurality of natural images, and the edge tag set comprises an edge tag image corresponding to each natural image in the edge data set and an FDoG edge tag image obtained after edge extraction of each natural image in the edge data set;
the second data set construction module is used for acquiring a plurality of color-drawing cultural relic images to obtain a second sample set; wherein each painted cultural relic image in the second sample set is different from the painted cultural relic image in the first sample set;
acquiring a line draft image of each colored drawing cultural relic image in a second sample set, and acquiring a second line draft label set;
the thick line manuscript extraction module is used for inputting the obtained second sample set into a thick extraction model to obtain a thick line manuscript image set;
the fine extraction model obtaining module is used for taking the thick line manuscript image set as input, taking the second line manuscript label set as reference output, training a convolutional neural network and obtaining a fine line manuscript extraction model;
the line draft extraction model obtaining module is used for obtaining a line draft extraction model, and the line draft extraction model comprises a crude extraction model and a fine line draft extraction model which are sequentially connected.
Optionally, the convolutional neural network in the fine extraction model obtaining module includes a U-net network, a side output layer, and a fusion output layer, which are sequentially arranged;
the side output layer comprises a first convolution layer, an up-sampling layer and a first activation function layer which are sequentially arranged; the fusion output layer comprises a second convolution layer and a second activation function layer which are sequentially arranged.
Optionally, when the convolutional neural network is trained in the fine extraction model obtaining module, the convolutional neural network is trained by using a stochastic gradient descent algorithm, wherein the loss function L is fine Comprises the following steps:
wherein ω is side Representing specific gravity, ω, in the loss function for different scale features fuse Representing the specific gravity of the fusion feature in the loss function; s represents scale, S =1,2, \8230, S is a positive integer,
in the form of a single-channel loss function,
for a fusion loss function, is>
Y represents an image output after inputting a thick line draft image into a thin line draft extraction model R A line draft label image corresponding to the thick line draft image is shown, and R shows the thick line draft image.
Optionally, when the BDCN edge detection network is trained by using the edge data set and the edge label set in the coarse extraction model obtaining module, the loss function L is used pre Comprises the following steps:
wherein S represents scale, S =1,2, \8230, S and S are positive integers, alpha is the weight of the edge label in the objective function, beta is the weight of the FDoG edge label, and P is N Representing the output image after the natural image N is input to the BDCN edge detection network,
represents the edge label image corresponding to the natural image N, and>
representing the FDoG edge label image corresponding to the natural image N.
Optionally, S =5, ω fuse ＝1.1；
When s =1, ω side =0.8; when s =2, ω side =0.8; when s =3, ω side =0.4; when s =4, ω side =0.4; when s =5, ω side ＝0.4。
Example four
In this embodiment, a device for extracting line draft of a layered colored drawing cultural relic is provided, which is used to input an image of a colored drawing cultural relic to be extracted into a line draft extraction model obtained by a device for constructing a line draft extraction model of a layered colored drawing cultural relic in the third embodiment, so as to obtain a line draft image of the colored drawing cultural relic.
Through the description of the above embodiments, those skilled in the art will clearly understand that the present invention may be implemented by software plus necessary general hardware, and certainly may also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present invention may be substantially implemented or a part of the technical solutions contributing to the prior art may be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a hard disk, or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
Claims (10)
1. A method for constructing a hierarchical colored drawing manuscript extraction model is characterized by comprising the following steps of:
step 1, collecting a plurality of colored drawing cultural relic images to obtain a first sample set;
performing edge extraction on the line manuscript image of each colored drawing cultural relic image in the first sample set by adopting an FDoG algorithm to obtain an FDoG label set;
acquiring a line draft image of each colored drawing cultural relic image in a first sample set, and acquiring a first line draft label set;
step 2, taking the first sample set as input, taking the FDoG label set and the first line draft label set as reference output, training an edge extraction network, and obtaining a coarse extraction model;
the edge extraction network is obtained by training a BDCN edge detection network by taking an edge data set as input and an edge label set as reference output;
the edge data set comprises a plurality of natural images, and the edge tag set comprises an edge tag image corresponding to each natural image in the edge data set and an FDoG edge tag image obtained after edge extraction of each natural image in the edge data set;
step 3, collecting a plurality of color-painted cultural relic images to obtain a second sample set; wherein each painted cultural relic image in the second sample set is different from the painted cultural relic image in the first sample set;
acquiring a line draft image of each colored drawing cultural relic image in a second sample set, and acquiring a second line draft label set;
step 4, inputting the second sample set obtained in the step 3 into the coarse extraction model obtained in the step 2 to obtain a coarse line manuscript image set;
step 5, taking the thick line draft image set as input, taking the second line draft label set as reference output, training a convolutional neural network, and obtaining a thin line draft extraction model;
and 6, obtaining a line draft extraction model, wherein the line draft extraction model comprises a crude extraction model obtained in the step 2 and a fine line draft extraction model obtained in the step 5 which are sequentially connected.
2. The method for constructing a layered colored drawing manuscript extraction model of claim 1, wherein the convolutional neural network in the step 5 comprises a U-net network, a side output layer and a fusion output layer which are sequentially arranged;
the side output layer comprises a first convolution layer, an up-sampling layer and a first activation function layer which are sequentially arranged; the fusion output layer comprises a second convolution layer and a second activation function layer which are arranged in sequence.
3. The method for constructing the layered colored drawing manuscript extraction model of claim 2, wherein in the step 5, when the convolutional neural network is trained, a stochastic gradient descent algorithm is adopted for training, wherein the loss function L is fine Comprises the following steps:
wherein omega side Representing specific gravity, ω, in loss functions corresponding to different scale features fuse Representing the specific gravity of the fusion feature in the loss function; s denotes the scale, S =1,2, \ 8230, S, S is a positive integer,
in the form of a single-channel loss function,
in order to fuse the loss functions of the two,
y represents an image output after inputting a thick line draft image into a thin line draft extraction model R A line draft label image corresponding to the thick line draft image is shown, and R shows the thick line draft image.
4. The method for constructing the hierarchical color-drawing manuscript extraction model of claim 3, wherein in the step 2, when the BDCN edge detection network is trained by adopting the edge data set and the edge label set, the loss function L is obtained pre Comprises the following steps:
where S denotes the scale, S =1,2, \ 8230;, S, S are positive integers, α is the weight of the edge label in the objective function, β is the weight of the FDoG edge label, P is N Represents to be fromThen the image N is input to the output image after the BDCN edge detection network,
representing the edge label image corresponding to the natural image N,
representing the FDoG edge label image corresponding to the natural image N.
5. A method for extracting line draft of a layered colored drawing cultural relic is characterized in that the image of the colored drawing cultural relic to be extracted is input into a line draft extraction model obtained by the method for constructing the line draft extraction model of the layered colored drawing cultural relic according to any one of claims 1 to 4, so as to obtain the line draft image of the colored drawing cultural relic.
6. A hierarchical colored drawing cultural relic line draft extraction model construction device is characterized by comprising a first data set construction module, a crude extraction model acquisition module, a second data set construction module, a thick line draft extraction module, a fine extraction model acquisition module and a line draft extraction model acquisition module;
the first data set construction module is used for acquiring a plurality of colored drawing cultural relic images to obtain a first sample set;
performing edge extraction on the line manuscript image of each colored drawing cultural relic image in the first sample set by adopting an FDoG algorithm to obtain an FDoG label set;
acquiring a line draft image of each colored drawing cultural relic image in a first sample set, and acquiring a first line draft label set;
the rough extraction model obtaining module is used for taking the first sample set as input, taking the FDoG tag set and the first line draft tag set as reference output, training an edge extraction network and obtaining a rough extraction model;
the edge extraction network is obtained by adopting an edge data set as input and adopting an edge label set as reference output to train the BDCN edge detection network;
the edge data set comprises a plurality of natural images, and the edge tag set comprises an edge tag image corresponding to each natural image in the edge data set and an FDoG edge tag image obtained after edge extraction is carried out on each natural image in the edge data set;
the second data set construction module is used for acquiring a plurality of colored drawing cultural relic images to obtain a second sample set; wherein each colored drawing cultural relic image in the second sample set is different from the colored drawing cultural relic image in the first sample set;
acquiring a line draft image of each colored drawing cultural relic image in a second sample set, and acquiring a second line draft label set;
the thick line manuscript extraction module is used for inputting the obtained second sample set into a thick extraction model to obtain a thick line manuscript image set;
the fine extraction model obtaining module is used for taking the thick line manuscript image set as input, taking the second line manuscript label set as reference output, training a convolution neural network and obtaining a fine line manuscript extraction model;
the line draft extraction model obtaining module is used for obtaining a line draft extraction model, and the line draft extraction model comprises a crude extraction model and a fine line draft extraction model which are sequentially connected.
7. The device for constructing the layered colored drawing manuscript extraction model of claim 6, wherein the convolutional neural network in the fine extraction model obtaining module comprises a U-net network, a side output layer and a fusion output layer which are sequentially arranged;
the side output layer comprises a first convolution layer, an up-sampling layer and a first activation function layer which are sequentially arranged; the fusion output layer comprises a second convolution layer and a second activation function layer which are arranged in sequence.
8. The device for constructing a hierarchical colored drawing manuscript extraction model of claim 7, wherein a stochastic ladder is adopted when the convolutional neural network is trained in the fine extraction model obtaining moduleTraining with a degree-dropping algorithm, wherein a loss function L is used fine Comprises the following steps:
wherein ω is side Representing specific gravity, ω, in the loss function for different scale features fuse Representing the specific gravity of the fusion feature in the loss function; s denotes the scale, S =1,2, \ 8230, S, S is a positive integer,
in the form of a single-channel loss function,
in order to fuse the loss functions of the image,
y represents an image output after inputting a thick line draft image into a thin line draft extraction model R A line draft label image corresponding to the thick line draft image is shown, and R shows the thick line draft image.
9. The device for constructing the hierarchical line drawing manuscript extraction model of painted relics of claim 8, wherein the crude extraction model obtaining module uses the edge data set and the edge tag set to train the BDCN edge detection network, and the loss function L is the loss function L pre Comprises the following steps:
where S denotes the scale, S =1,2, \ 8230;, S, S are positive integers, α is the weight of the edge label in the objective function, β is the weight of the FDoG edge label, P is N Representing the output image after the natural image N is input to the BDCN edge detection network,
representing the edge label image corresponding to the natural image N,
representing the FDoG edge label image corresponding to the natural image N.
10. A layered colored drawing manuscript extraction device, which is used for inputting the colored drawing manuscript image to be extracted into the line manuscript extraction model obtained by the layered colored drawing manuscript extraction model construction device according to any one of claims 6 to 9, so as to obtain the colored drawing manuscript image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010049526.4A CN111222519B (en) | 2020-01-16 | 2020-01-16 | Construction method, method and device of hierarchical colored drawing manuscript line extraction model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010049526.4A CN111222519B (en) | 2020-01-16 | 2020-01-16 | Construction method, method and device of hierarchical colored drawing manuscript line extraction model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111222519A CN111222519A (en) | 2020-06-02 |
| CN111222519B true CN111222519B (en) | 2023-03-24 |
Family
ID=70827122
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010049526.4A Active CN111222519B (en) | 2020-01-16 | 2020-01-16 | Construction method, method and device of hierarchical colored drawing manuscript line extraction model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111222519B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111680706B (en) * | 2020-06-17 | 2023-06-23 | 南开大学 | Dual-channel output contour detection method based on coding and decoding structure |
| CN112372855B (en) * | 2020-11-04 | 2024-10-18 | 陕西历史博物馆(陕西省文物交流中心) | Wall painting masonry stripping equipment |
| CN113554087B (en) * | 2021-07-20 | 2023-11-10 | 清华大学深圳国际研究生院 | An automatic generation method for archaeological line drawings based on generative adversarial networks |
| CN115393470B (en) * | 2022-10-28 | 2023-05-23 | 成都航空职业技术学院 | Drawing method, drawing system and drawing device for cultural relic digital line drawing |
| CN115409914B (en) * | 2022-11-02 | 2023-01-06 | 天津恒达文博科技股份有限公司 | Method, device, storage medium and electronic device for generating line drawing of cultural relics |
| CN115731450B (en) * | 2022-11-24 | 2025-11-28 | 西北大学 | Method for extracting colored drawing cultural relics line manuscript |
| CN115841625B (en) * | 2023-02-23 | 2023-06-06 | 杭州电子科技大学 | Remote sensing building image extraction method based on improved U-Net model |
| CN117649365B (en) * | 2023-11-16 | 2024-08-09 | 西南交通大学 | Paper book graph digital restoration method based on convolutional neural network and diffusion model |
| CN120761501A (en) * | 2025-07-23 | 2025-10-10 | 陕西历史博物馆 | A method and device for monitoring cultural relics damage using multi-source acoustic emission information |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018028255A1 (en) * | 2016-08-11 | 2018-02-15 | 深圳市未来媒体技术研究院 | Image saliency detection method based on adversarial network |
| US10008011B1 (en) * | 2014-11-26 | 2018-06-26 | John Balestrieri | Methods for creating a simulated watercolor-painted image from a source image |
| CN108510456A (en) * | 2018-03-27 | 2018-09-07 | 华南理工大学 | The sketch of depth convolutional neural networks based on perception loss simplifies method |
| CN108615252A (en) * | 2018-05-03 | 2018-10-02 | 苏州大学 | The training method and device of color model on line original text based on reference picture |
| CN108830913A (en) * | 2018-05-25 | 2018-11-16 | 大连理工大学 | Semantic level line original text painting methods based on User Colors guidance |
| CN109448093A (en) * | 2018-10-25 | 2019-03-08 | 广东智媒云图科技股份有限公司 | A kind of style image generation method and device |
| WO2019144575A1 (en) * | 2018-01-24 | 2019-08-01 | 中山大学 | Fast pedestrian detection method and device |
| CN110211192A (en) * | 2019-05-13 | 2019-09-06 | 南京邮电大学 | A kind of rendering method based on the threedimensional model of deep learning to two dimensional image |
| CN110223359A (en) * | 2019-05-27 | 2019-09-10 | 浙江大学 | It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation |
| CN110378985A (en) * | 2019-07-19 | 2019-10-25 | 中国传媒大学 | A kind of animation drawing auxiliary creative method based on GAN |
-
2020
- 2020-01-16 CN CN202010049526.4A patent/CN111222519B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10008011B1 (en) * | 2014-11-26 | 2018-06-26 | John Balestrieri | Methods for creating a simulated watercolor-painted image from a source image |
| WO2018028255A1 (en) * | 2016-08-11 | 2018-02-15 | 深圳市未来媒体技术研究院 | Image saliency detection method based on adversarial network |
| WO2019144575A1 (en) * | 2018-01-24 | 2019-08-01 | 中山大学 | Fast pedestrian detection method and device |
| CN108510456A (en) * | 2018-03-27 | 2018-09-07 | 华南理工大学 | The sketch of depth convolutional neural networks based on perception loss simplifies method |
| CN108615252A (en) * | 2018-05-03 | 2018-10-02 | 苏州大学 | The training method and device of color model on line original text based on reference picture |
| CN108830913A (en) * | 2018-05-25 | 2018-11-16 | 大连理工大学 | Semantic level line original text painting methods based on User Colors guidance |
| CN109448093A (en) * | 2018-10-25 | 2019-03-08 | 广东智媒云图科技股份有限公司 | A kind of style image generation method and device |
| CN110211192A (en) * | 2019-05-13 | 2019-09-06 | 南京邮电大学 | A kind of rendering method based on the threedimensional model of deep learning to two dimensional image |
| CN110223359A (en) * | 2019-05-27 | 2019-09-10 | 浙江大学 | It is a kind of that color model and its construction method and application on the dual-stage polygamy colo(u)r streak original text of network are fought based on generation |
| CN110378985A (en) * | 2019-07-19 | 2019-10-25 | 中国传媒大学 | A kind of animation drawing auxiliary creative method based on GAN |
Non-Patent Citations (1)
| Title |
|---|
| 基于生成对抗网络的漫画草稿图简化;卢倩雯等;《自动化学报》;20180418(第05期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111222519A (en) | 2020-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111222519B (en) | Construction method, method and device of hierarchical colored drawing manuscript line extraction model | |
| CN111340738B (en) | An image rain removal method based on multi-scale progressive fusion | |
| CN114820579A (en) | Semantic segmentation based image composite defect detection method and system | |
| CN109523470B (en) | Depth image super-resolution reconstruction method and system | |
| CN113807356B (en) | End-to-end low-visibility image semantic segmentation method | |
| CN110991257B (en) | Polarized SAR oil spill detection method based on feature fusion and SVM | |
| CN112102229A (en) | Intelligent industrial CT detection defect identification method based on deep learning | |
| CN109035267B (en) | A deep learning-based image target extraction method | |
| CN113128518B (en) | Sift mismatch detection method based on twin convolution network and feature mixing | |
| CN113256494B (en) | A Text-Image Super-Resolution Method | |
| CN109360179A (en) | Image fusion method, device and readable storage medium | |
| CN109635726B (en) | Landslide identification method based on combination of symmetric deep network and multi-scale pooling | |
| Zhou et al. | MSAR‐DefogNet: Lightweight cloud removal network for high resolution remote sensing images based on multi scale convolution | |
| CN113887472A (en) | Remote sensing image cloud detection method based on cascade color and texture feature attention | |
| Conrad et al. | Two-stage seamless text erasing on real-world scene images | |
| CN109165551B (en) | An expression recognition method based on adaptive weighted fusion of saliency structure tensor and LBP features | |
| CN113920516A (en) | Calligraphy character skeleton matching method and system based on twin neural network | |
| CN114549325B (en) | Bionic Eagle Eye multi-scale fusion super-resolution reconstruction model, method, device and storage medium for single image | |
| CN113610024B (en) | A multi-strategy deep learning remote sensing image small target detection method | |
| CN109977834B (en) | Method and device for segmenting human hand and interactive object from depth image | |
| CN118967467A (en) | Hyperspectral image fusion method based on high-frequency information extraction and multi-feature matrix | |
| CN109741351A (en) | A class-sensitive edge detection method based on deep learning | |
| CN114220013B (en) | Camouflage object detection method based on boundary alternate guidance | |
| CN115937205A (en) | Surface defect tile image generation method, device, equipment and storage medium | |
| CN120147868A (en) | A method for identifying active landslides |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |