[go: up one dir, main page]

CN116509326B - Tongue multispectral image generation method, device, equipment and storage medium - Google Patents

Tongue multispectral image generation method, device, equipment and storage medium Download PDF

Info

Publication number
CN116509326B
CN116509326B CN202310387046.2A CN202310387046A CN116509326B CN 116509326 B CN116509326 B CN 116509326B CN 202310387046 A CN202310387046 A CN 202310387046A CN 116509326 B CN116509326 B CN 116509326B
Authority
CN
China
Prior art keywords
image
tongue
multispectral
preset
picture
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310387046.2A
Other languages
Chinese (zh)
Other versions
CN116509326A (en
Inventor
党晓兵
麦海鹏
马翠
陈贤帅
杜如虚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Jianchi Biotechnology Co ltd
Original Assignee
Guangdong Jianchi Biotechnology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Jianchi Biotechnology Co ltd filed Critical Guangdong Jianchi Biotechnology Co ltd
Priority to CN202310387046.2A priority Critical patent/CN116509326B/en
Publication of CN116509326A publication Critical patent/CN116509326A/en
Priority to PCT/CN2023/133264 priority patent/WO2024212524A1/en
Application granted granted Critical
Publication of CN116509326B publication Critical patent/CN116509326B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0075Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence by spectroscopy, i.e. measuring spectra, e.g. Raman spectroscopy, infrared absorption spectroscopy
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/0059Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
    • A61B5/0077Devices for viewing the surface of the body, e.g. camera, magnifying lens
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/103Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/1032Determining colour of tissue for diagnostic purposes
    • A61B5/1034Determining colour of tissue for diagnostic purposes by means of colour cards
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/45For evaluating or diagnosing the musculoskeletal system or teeth
    • A61B5/4538Evaluating a particular part of the muscoloskeletal system or a particular medical condition
    • A61B5/4542Evaluating the mouth, e.g. the jaw
    • A61B5/4552Evaluating soft tissue within the mouth, e.g. gums or tongue
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/48Other medical applications
    • A61B5/4854Diagnosis based on concepts of alternative medicine, e.g. homeopathy or non-orthodox
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/56Extraction of image or video features relating to colour
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/58Extraction of image or video features relating to hyperspectral data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02ATECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Surgery (AREA)
  • Veterinary Medicine (AREA)
  • Public Health (AREA)
  • Animal Behavior & Ethology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Pathology (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Dentistry (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • Rheumatology (AREA)
  • Orthopedic Medicine & Surgery (AREA)
  • Physical Education & Sports Medicine (AREA)
  • Alternative & Traditional Medicine (AREA)
  • Measuring And Recording Apparatus For Diagnosis (AREA)

Abstract

本发明公开了一种舌象多光谱图像生成方法、装置、设备及存储介质,方法包括:通过拍照设备获取舌象检测图像;根据预设标准比色卡,计算出环境光及所述拍照设备的光谱响应;根据预设的神经网络模型,在所述舌象检测图像中提取舌象图像;根据所述环境光和拍照设备的光谱响应,对所述舌象图像进行光谱重建,从而生成舌象多光谱图像。本发明解决了现有技术中拍摄设备和周围环境光对舌象获取干扰大、舌象多光谱图像准确性低的技术问题,便于用户可随时随地的进行舌象多光谱图像的获取,来进行舌象的问诊,降低用户对周围环境条件和拍照设备的性能要求,从而提高用户的使用体验,并为中医舌诊提供确切的、专业的影像信息,并提供辅助临床诊断。

The present invention discloses a tongue image multi-spectral image generation method, device, equipment and storage medium, the method comprising: acquiring a tongue image detection image through a photographing device; calculating the ambient light and the spectral response of the photographing device according to a preset standard colorimetric card; extracting a tongue image from the tongue image detection image according to a preset neural network model; and spectrally reconstructing the tongue image according to the ambient light and the spectral response of the photographing device, thereby generating a tongue image multi-spectral image. The present invention solves the technical problems in the prior art that the photographing device and the ambient light have great interference in the acquisition of tongue images and the accuracy of the tongue image multi-spectral image is low, and facilitates users to acquire tongue image multi-spectral images anytime and anywhere to conduct tongue image consultation, reduces the user's requirements for the surrounding environmental conditions and the performance of the photographing device, thereby improving the user's experience, and provides accurate and professional image information for traditional Chinese medicine tongue diagnosis, and provides auxiliary clinical diagnosis.

Description

Tongue image multispectral image generation method, device, equipment and storage medium
Technical Field
The present invention relates to the field of medical detection technologies, and in particular, to a tongue image multispectral image generating method, apparatus, device, and storage medium.
Background
Tongue diagnosis is an important part of inspection in TCM. For example, the thickness of tongue coating and the color of tongue picture can reflect the physical condition of human body, and can provide important evaluation index for clinical medical diagnosis. In the clinical diagnosis of tongue manifestations, according to the tongue color and tongue coating color, tongue manifestations can be divided into 27 kinds, and different tongue manifestations correspond to different clinical diagnosis schemes. In addition, the nature of the disease, the severity of the disease, the trend of the disease, the severity of the disease, etc. can be inferred from the tongue color change law.
At present, the tongue diagnosis of traditional Chinese medicine is still qualitatively described by visual inspection. Because different doctors have differences on color sensation, the tongue diagnosis of traditional Chinese medicine is greatly influenced by subjective factors, objective and scientific evaluation indexes are lacking, and clinical observation materials of all medical institutions are difficult to incorporate into a system, so that the deep research and clinical application of the work are greatly influenced. The existing tongue image is obtained by shooting tongue images by a user through image equipment, and judging the tongue image by utilizing an artificial intelligence technology or an image analysis technology. The tongue diagnosis system does not eliminate the difference of shooting equipment and the influence of surrounding light, is greatly interfered by the outside, and is difficult to be used as an accurate index in clinical tongue diagnosis.
Thus, there is a need for a method of accurately and quickly generating tongue multispectral images that eliminates the effects of the photographing device and ambient light.
Disclosure of Invention
The invention provides a tongue picture multispectral image generation method, a device, equipment and a storage medium, which are used for solving the technical problems that shooting equipment and ambient light have large interference on tongue picture acquisition and low tongue picture multispectral image accuracy in the prior art.
In order to solve the above technical problems, an embodiment of the present invention provides a tongue image multispectral image generating method, including:
Acquiring a tongue picture detection image through photographing equipment;
calculating the spectrum response of the photographing equipment and the ambient light according to a preset standard color chart;
extracting a tongue image from the tongue image detection image according to a preset neural network model;
and carrying out spectrum reconstruction on the tongue image according to the ambient light and the spectral response of the photographing equipment, thereby generating a tongue image multispectral image.
As a preferable scheme, the tongue image detection image is obtained by a photographing device, specifically:
And photographing the tongue body and a preset standard color chart aligned with the tongue body through photographing equipment to obtain a tongue picture detection image, wherein the tongue picture detection image comprises the preset standard color chart area image and the tongue picture image of the tongue body.
As a preferred solution, the calculating the ambient light and the spectral response of the photographing device according to the preset standard color chart specifically includes:
extracting the preset standard colorimetric card region image in the tongue image detection image;
Converting the preset standard color card area image into an HSV image form, and carrying out color partition on the converted preset standard color card area image so as to extract corresponding standard colors in different color partitions;
And calculating RGB color gamut values of the standard colors, and calculating the spectral responses of the ambient light and the shooting equipment through an optimization algorithm according to the reflection spectrum curves of the standard colors and the RGB color gamut values of the standard colors.
As a preferred solution, the extracting a tongue image from the tongue image detection image according to a preset neural network model specifically includes:
according to a preset neural network model, performing image segmentation on the tongue picture detection image to generate a mask image of the tongue picture image;
and extracting tongue picture region information from the tongue picture detection image according to the mask image of the tongue picture image, so as to obtain a tongue picture image.
As a preferred solution, the method for constructing the preset neural network model includes:
Constructing an initial neural network model;
and setting a loss function and an optimizer in the model training process, and training the initial neural network model by using a preset tongue image data set, so as to obtain a trained neural network model.
Preferably, the spectrum reconstruction is performed on the tongue image according to the ambient light and the spectral response of the photographing device, so as to generate a tongue multispectral image, specifically:
Performing spectrum reconstruction on each pixel point of the tongue image, acquiring a color gamut value of the pixel point in the tongue image in the process of performing spectrum reconstruction on any pixel point, calculating an optimal characteristic coefficient by an optimization algorithm by combining the color gamut value and the spectrum response of the ambient light and photographing equipment, and calculating a pixel value of the pixel point in the multispectral image according to the characteristic coefficient and a preset characteristic vector;
And generating tongue image spectral images under different wave bands according to the spectral image pixel values corresponding to each pixel point under different wave band ranges after the corresponding multispectral image pixel values are calculated by each pixel point in the tongue image.
Preferably, after generating the tongue image multispectral image, the method further comprises:
Uploading the tongue image multispectral image to a data server for storage.
Correspondingly, the invention provides a tongue picture multispectral image generating device which comprises an image acquisition module, a standard colorimetric card module, a tongue picture module and a spectrum reconstruction module;
The image acquisition module is used for acquiring tongue image detection images through photographing equipment;
The standard colorimetric card module is used for calculating the spectral response of the photographing equipment and the ambient light according to a preset standard colorimetric card;
The tongue picture module is used for extracting tongue picture images from the tongue picture detection images according to a preset neural network model;
the spectrum reconstruction module is used for carrying out spectrum reconstruction on the tongue image according to the ambient light and the spectrum response of the photographing equipment so as to generate a tongue image multispectral image.
As a preferable scheme, the tongue image detection image is obtained by a photographing device, specifically:
And photographing the tongue body and a preset standard color chart aligned with the tongue body through photographing equipment to obtain a tongue picture detection image, wherein the tongue picture detection image comprises the preset standard color chart area image and the tongue picture image of the tongue body.
As a preferred solution, the calculating the ambient light and the spectral response of the photographing device according to the preset standard color chart specifically includes:
extracting the preset standard colorimetric card region image in the tongue image detection image;
Converting the preset standard color card area image into an HSV image form, and carrying out color partition on the converted preset standard color card area image so as to extract corresponding standard colors in different color partitions;
And calculating RGB color gamut values of the standard colors, and calculating spectral responses of the ambient light and the shooting equipment according to the reflection spectrum curves of the standard colors and the RGB color gamut values of the standard colors.
As a preferred solution, the extracting a tongue image from the tongue image detection image according to a preset neural network model specifically includes:
according to a preset neural network model, performing image segmentation on the tongue picture detection image to generate a mask image of the tongue picture image;
and extracting tongue picture region information from the tongue picture detection image according to the mask image of the tongue picture image, so as to obtain a tongue picture image.
As a preferred solution, the method for constructing the preset neural network model includes:
Constructing an initial neural network model;
and setting a loss function and an optimizer in the model training process, and training the initial neural network model by using a preset tongue image data set, so as to obtain a trained neural network model.
Preferably, the spectrum reconstruction is performed on the tongue image according to the ambient light and the spectral response of the photographing device, so as to generate a tongue multispectral image, specifically:
Performing spectrum reconstruction on each pixel point of the tongue image, acquiring a color gamut value of the pixel point in the tongue image in the process of performing spectrum reconstruction on any pixel point, calculating an optimal characteristic coefficient by an optimization algorithm by combining the color gamut value and the spectrum response of the ambient light and photographing equipment, and calculating a pixel value of the pixel point in the multispectral image according to the characteristic coefficient and a preset characteristic vector;
And generating tongue image spectral images under different wave bands according to the spectral image pixel values corresponding to each pixel point under different wave band ranges after the corresponding multispectral image pixel values are calculated by each pixel point in the tongue image.
Preferably, after generating the tongue image multispectral image, the method further comprises:
Uploading the tongue image multispectral image to a data server for storage.
Accordingly, the present invention provides a terminal device comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the tongue image multispectral image generation method according to any one of the above when executing the computer program.
Accordingly, the invention provides a computer readable storage medium comprising a stored computer program, wherein the computer program when run controls a device in which the computer readable storage medium is located to perform the tongue image multispectral image generation method according to any one of the above.
Compared with the prior art, the embodiment of the invention has the following beneficial effects:
According to the technical scheme, the tongue image detection image is obtained, the spectral response of the environment light and the photographing equipment is calculated through the preset standard colorimetric card, so that interference of the surrounding environment or the photographing equipment on the acquisition precision and accuracy of the tongue image is avoided by taking the preset standard colorimetric card as a reference, errors of the acquisition of the tongue image data of external factors such as the environment and the equipment are reduced, the tongue image is subjected to spectral reconstruction through the environment light and the spectral response of the photographing equipment, the influence of the environment and the equipment is eliminated by the collected tongue image, the tongue image multispectral image is accurately generated, meanwhile, the acquisition of the tongue image multispectral image can be conveniently carried out by a user at any time and any place, the inquiry of the tongue image is carried out, the requirements of the user on the surrounding environment condition and the performance of the photographing equipment are reduced, and the use experience of the user is improved.
Further, through the multispectral imaging technology in the technical scheme, areas with different tongue colors and tongue fur colors are distinguished, information which cannot be distinguished by naked eyes is obtained, and the influence of individuals on different color sensations is eliminated. Provides exact and professional image information for tongue diagnosis of traditional Chinese medicine and provides auxiliary clinical diagnosis.
Drawings
FIG. 1 is a flow chart of steps of a tongue image multispectral image generation method provided by an embodiment of the invention;
FIG. 2 is a schematic diagram of a standard color chart according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of tongue image multispectral image generation provided by an embodiment of the invention;
FIG. 4 is a block diagram of a tongue image multispectral image generating device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Example 1
Referring to fig. 1, a tongue image multispectral image generating method provided by an embodiment of the invention includes steps S101-S104:
step S101, acquiring tongue picture detection images through photographing equipment.
As a preferred solution of this embodiment, the capturing, by the photographing device, a tongue image detection image specifically includes:
And photographing the tongue body and a preset standard color chart aligned with the tongue body through photographing equipment to obtain a tongue picture detection image, wherein the tongue picture detection image comprises the preset standard color chart area image and the tongue picture image of the tongue body.
In this embodiment, before photographing the tongue body and the preset standard colorimetric card aligned with the tongue body, the standard colorimetric card needs to be placed in a proper area of the face, so that the tongue body area in the standard colorimetric card is aligned with the extended tongue body, and therefore, the generation of a tongue image spectrum image can be directly performed through a photographed single image, further, diagnosis can be directly performed, and meanwhile, the single image can ensure that the standard colorimetric card and the tongue body are in the same ambient light and photographing equipment conditions, thereby ensuring the accuracy of tongue image detection images, and reducing errors and interference of surrounding external factors on tongue image detection.
It should be noted that the photographing device includes, but is not limited to, a camera, a video camera, a terminal device with photographing function, and the like, preferably, the photographing device can photograph through a smart phone and a mobile phone camera, can collect and acquire images of the tongue body and the standard color chart efficiently and accurately, and can process the images directly after collecting, further can upload or diagnose the obtained tongue image multispectral image to a server, so that the efficiency of tongue image diagnosis is improved, the tongue image multispectral image can be obtained through the smart phone, accurate image information is provided for clinical tongue diagnosis, clinical diagnosis is assisted, and meanwhile, the use of a user is facilitated, and the user experience is improved.
Furthermore, the tongue image multispectral image can be obtained through a single device, the structure is simple, the use is convenient, the operation is simple, no extra professional instrument is needed, and the popularization of the intelligent tongue diagnosis system based on the tongue image multispectral image is facilitated.
And step S102, calculating the spectral response of the photographing equipment and the ambient light according to a preset standard color chart.
As a preferred solution of this embodiment, the calculating, according to a preset standard color chart, the spectral response of the photographing device and the ambient light specifically includes:
the method comprises the steps of detecting tongue images, extracting the preset standard colorimetric card region images in the tongue image detection images, converting the preset standard colorimetric card region images into HSV image forms, carrying out color partitioning on the converted preset standard colorimetric card region images, accordingly extracting corresponding standard colors in different color partitions, calculating RGB color gamut values of the standard colors, and calculating spectral responses of ambient light and the shooting equipment according to reflection spectrum curves of the standard colors and the RGB color gamut values of the standard colors.
Please refer to fig. 2, which illustrates a standard color chart according to an exemplary embodiment of the present invention, wherein a rectangular box in the standard color chart is a standard color block in the color chart. Preferably, the standard color blocks include orange standard color blocks, pink standard color blocks, yellow standard color blocks, green standard color blocks and blue standard color blocks, so that the standard color blocks can cover a larger range of color frequencies, namely visible light frequencies, and further, for part of tongue image images needing to be identified and acquired with high precision, the color frequency interval between the color blocks can be reduced, so that the coverage range of the visible light frequencies is reduced, and the adjustment step length of the color blocks corresponding to the visible light frequencies is further improved, so that the capability of identifying the difference in fine environmental light or fine performance of photographing equipment is improved. Preferably, the standard color chart has an area aligned with the tongue that is slightly wider than the surface width of the tongue of a normal adult.
In this embodiment, referring to fig. 3, for example, the image is template-matched by the cv2.Matchtemplate function, so as to extract the standard color chart area image in the tongue image detection image. The standard color comparison card area is converted into an HSV image form, the areas with different colors of the HSV image are partitioned by using a cv2.InRange function, and standard colors in the partitions are extracted respectively, so that the tone (H), the saturation (S) and the brightness (V) of each standard color are accurately obtained, and the standard color comparison card adopted by the exemplary embodiment is five standard colors. Furthermore, the RGB color gamut values of the standard colors are calculated, in the present exemplary embodiment, the occurrence frequencies of the RGB color gamut values of the standard color areas are counted one by one through a collection function, the RGB color gamut value with the highest occurrence frequency is searched by using a counter function, the RGB color gamut value is used as the RGB color gamut value of the standard colors, and the spectral responses of the ambient light and the photographing device are calculated through a quadratic optimization algorithm according to the RGB color gamut values of the standard colors and the known reflection spectrum curves of the standard colors. Preferably, the reflectance spectrum curve of the standard colorimetric card is measured by a spectrometer.
Step S103, extracting tongue image from the tongue image detection image according to a preset neural network model.
As a preferred solution of this embodiment, the extracting a tongue image from the tongue image detection image according to a preset neural network model specifically includes:
And extracting tongue picture region information from the tongue picture detection image according to the mask image of the tongue picture image, thereby obtaining the tongue picture image.
It should be noted that, preferably, the preset neural network model is Unet neural network model. Referring to fig. 3, image segmentation is performed on the tongue image detection image by loading Unet a neural network model, a mask image of the tongue image is generated, and tongue image region information of the detection image is extracted through the mask image, so that the tongue image is obtained.
As a preferred solution of this embodiment, the method for constructing the preset neural network model includes:
setting a loss function and an optimizer in the model training process, and training the initial neural network model by using a preset tongue image data set to obtain a trained neural network model.
It should be noted that, preferably, the Loss function in the training process is Dice Loss, and the Optimizer is Adam Optimizer. The tongue image data set comprises tongue images acquired under various different environmental lights or different photographing devices, and the neural network model of the embodiment is mainly used for extracting tongue image area information so as to obtain tongue image.
And step S104, carrying out spectrum reconstruction on the tongue image according to the ambient light and the spectral response of the photographing equipment, so as to generate a tongue image multispectral image.
As a preferable solution of this embodiment, the spectrum reconstruction is performed on the tongue image according to the spectral responses of the ambient light and the photographing device, so as to generate a tongue multispectral image, specifically:
And in the process of carrying out spectrum reconstruction on any pixel point, acquiring a color gamut value of the pixel point in the tongue image, combining the color gamut value with the spectrum response of the ambient light and photographing equipment, calculating an optimal characteristic coefficient through an optimization algorithm, calculating a pixel value of the pixel point in the multispectral image according to the characteristic coefficient through a preset characteristic vector, and generating a tongue image spectrum image under different wave bands according to the corresponding spectrum image pixel value of each pixel point after calculating the corresponding multispectral image pixel value of each pixel point in the tongue image.
In this embodiment, referring to fig. 3, a color gamut value of a pixel point in a tongue image is obtained, and an optimal feature coefficient is obtained by a quadratic optimization algorithm in combination with the color gamut value and the spectral response of the ambient light and the photographing device, so that a trained preset feature vector is used to obtain a multispectral image pixel value according to the feature coefficient. And traversing each pixel point in the tongue image, and obtaining the pixel values of the multispectral image of all the pixel points through the steps so as to generate the tongue image multispectral image.
As a preferred aspect of the present embodiment, after generating the tongue-image multispectral image, the method further includes:
Uploading the tongue image multispectral image to a data server for storage.
It can be understood that after the obtained tongue image multispectral image is uploaded to the data server, the tongue image multispectral image can be conveniently checked by a user, meanwhile, the tongue image multispectral image can be diagnosed through a server platform and the like, so that an on-line diagnosis mode is realized, and the convenience of the user is improved.
Furthermore, a multispectral image generating algorithm is used for replacing a traditional multispectral system to generate tongue image multispectral images, so that the use cost is reduced, the convenience is improved, meanwhile, each time of detection data can be automatically recorded, long-term change trend of tongue images can be checked through a client platform, a server platform, an APP and the like, dynamic monitoring is realized, and the digital medical level is improved.
It can be understood that according to the characteristic that tissues with different colors on the tongue picture have obvious differences in light absorption and reflection of different spectrums, the embodiment generates a tongue picture multispectral image, acquires specific information which is difficult to acquire by naked eyes or simple RGB on the tongue picture according to the multispectral image, and is applied to clinical diagnosis of the tongue picture, so that the multispectral image technology can be used as a standard scientific evaluation index in clinical diagnosis of the tongue picture.
The implementation of the above embodiment has the following effects:
According to the technical scheme, the tongue image detection image is obtained, the spectral response of the environment light and the photographing equipment is calculated through the preset standard colorimetric card, so that interference of the surrounding environment or the photographing equipment on the acquisition precision and accuracy of the tongue image is avoided by taking the preset standard colorimetric card as a reference, errors of the acquisition of the tongue image data of external factors such as the environment and the equipment are reduced, the tongue image is subjected to spectral reconstruction through the environment light and the spectral response of the photographing equipment, the influence of the environment and the equipment is eliminated by the collected tongue image, the tongue image multispectral image is accurately generated, meanwhile, the acquisition of the tongue image multispectral image can be conveniently carried out by a user at any time and any place, the inquiry of the tongue image is carried out, the requirements of the user on the surrounding environment condition and the performance of the photographing equipment are reduced, and the use experience of the user is improved.
Further, through the multispectral imaging technology in the technical scheme, areas with different tongue colors and tongue fur colors are distinguished, information which cannot be distinguished by naked eyes is obtained, and the influence of individuals on different color sensations is eliminated. Provides exact and professional image information for tongue diagnosis of traditional Chinese medicine and provides auxiliary clinical diagnosis.
Example two
The invention provides a tongue picture multispectral image generating device which comprises an image acquisition module 201, a standard colorimetric card module 202, a tongue picture module 203 and a spectrum reconstruction module 204.
The image acquisition module 201 is configured to acquire a tongue image detection image through a photographing device.
The standard color chart module 202 is configured to calculate the spectral response of the photographing device and the ambient light according to a preset standard color chart.
The tongue image module 203 is configured to extract a tongue image from the tongue image detection image according to a preset neural network model.
The spectrum reconstruction module 204 is configured to perform spectrum reconstruction on the tongue image according to the ambient light and the spectral response of the photographing device, so as to generate a tongue multispectral image.
As a preferred solution of this embodiment, the capturing, by the photographing device, a tongue image detection image specifically includes:
And photographing the tongue body and a preset standard color chart aligned with the tongue body through photographing equipment to obtain a tongue picture detection image, wherein the tongue picture detection image comprises the preset standard color chart area image and the tongue picture image of the tongue body.
As a preferred solution of this embodiment, the calculating, according to a preset standard color chart, the spectral response of the photographing device and the ambient light specifically includes:
the method comprises the steps of detecting tongue images, extracting the preset standard colorimetric card region images in the tongue image detection images, converting the preset standard colorimetric card region images into HSV image forms, carrying out color partitioning on the converted preset standard colorimetric card region images, accordingly extracting corresponding standard colors in different color partitions, calculating RGB color gamut values of the standard colors, and calculating spectral responses of ambient light and the shooting equipment according to reflection spectrum curves of the standard colors and the RGB color gamut values of the standard colors.
As a preferred solution of this embodiment, the extracting a tongue image from the tongue image detection image according to a preset neural network model specifically includes:
And extracting tongue picture region information from the tongue picture detection image according to the mask image of the tongue picture image, thereby obtaining the tongue picture image.
As a preferred solution of this embodiment, the method for constructing the preset neural network model includes:
setting a loss function and an optimizer in the model training process, and training the initial neural network model by using a preset tongue image data set to obtain a trained neural network model.
As a preferable solution of this embodiment, the spectrum reconstruction is performed on the tongue image according to the spectral responses of the ambient light and the photographing device, so as to generate a tongue multispectral image, specifically:
and (3) carrying out spectrum reconstruction on each pixel point of the tongue image, acquiring a color gamut value of the pixel point in the tongue image in the process of carrying out spectrum reconstruction on one pixel point, calculating an optimal characteristic coefficient by combining the color gamut value with the spectrum response of the ambient light and photographing equipment through an optimization algorithm, calculating a pixel value of the pixel point in the multispectral image according to the characteristic coefficient and a preset characteristic vector, and generating a tongue image spectrum image under different wave bands according to the multispectral image pixel value of each pixel point after calculating the corresponding multispectral image pixel value of each pixel point in the tongue image.
As a preferred aspect of the present embodiment, after generating the tongue-image multispectral image, the method further includes:
Uploading the tongue image multispectral image to a data server for storage.
It will be clear to those skilled in the art that, for convenience and brevity of description, reference may be made to the corresponding process in the foregoing method embodiment for the specific working process of the above-described apparatus, which is not described herein again.
The implementation of the embodiment of the invention has the following effects:
According to the technical scheme, the tongue image detection image is obtained, the spectral response of the environment light and the photographing equipment is calculated through the preset standard colorimetric card, so that interference of the surrounding environment or the photographing equipment on the acquisition precision and accuracy of the tongue image is avoided by taking the preset standard colorimetric card as a reference, errors of acquisition of the tongue image data of external factors such as the environment and the equipment are reduced, spectral reconstruction is carried out on the tongue image through the environment light and the spectral response of the photographing equipment, the influence of the environment and the equipment is eliminated by the collected tongue image, the tongue image multispectral image is accurately generated, meanwhile, the acquisition of the tongue image multispectral image can be conveniently carried out by a user at any time and any place, the inquiry of the tongue image is carried out, the requirements of the user on the surrounding environment condition and the photographing equipment are reduced, the use experience of the user is improved, meanwhile, the areas with different tongue colors and fur colors are distinguished by the multispectral imaging technology in the technical scheme, the information which cannot be distinguished by naked eyes is obtained, and the influence of individuals on different colors is eliminated. Provides exact and professional image information for tongue diagnosis of traditional Chinese medicine and provides auxiliary clinical diagnosis.
Example III
Correspondingly, the invention further provides a terminal device, which comprises a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, wherein the tongue image multispectral image generation method according to any one of the embodiments is realized when the processor executes the computer program.
The terminal device of this embodiment comprises a processor, a memory, a computer program stored in said memory and executable on said processor, computer instructions. The processor, when executing the computer program, implements the steps of the first embodiment described above, such as steps S101 to S104 shown in fig. 1. Or the processor, when executing the computer program, performs the functions of the modules/units in the above-described device embodiments, such as the tongue module 203.
The computer program may be divided into one or more modules/units, which are stored in the memory and executed by the processor to accomplish the present invention, for example. The one or more modules/units may be a series of computer program instruction segments capable of performing the specified functions, which instruction segments are used for describing the execution of the computer program in the terminal device. For example, the tongue image module 203 is configured to extract a tongue image from the tongue image detection image according to a preset neural network model.
The terminal equipment can be computing equipment such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The terminal device may include, but is not limited to, a processor, a memory. It will be appreciated by those skilled in the art that the schematic diagram is merely an example of a terminal device and does not constitute a limitation of the terminal device, and may include more or less components than illustrated, or may combine some components, or different components, e.g., the terminal device may further include an input-output device, a network access device, a bus, etc.
The Processor may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, which is a control center of the terminal device, and which connects various parts of the entire terminal device using various interfaces and lines.
The memory may be used to store the computer program and/or the module, and the processor may implement various functions of the terminal device by running or executing the computer program and/or the module stored in the memory and invoking data stored in the memory. The memory may mainly include a storage program area which may store an operating system, an application program required for at least one function, etc., and a storage data area which may store data created according to the use of the mobile terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart memory card (SMART MEDIA CARD, SMC), secure Digital (SD) card, flash memory card (FLASH CARD), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
Wherein the terminal device integrated modules/units may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as stand alone products. Based on such understanding, the present invention may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
Example IV
Correspondingly, the invention further provides a computer readable storage medium, which comprises a stored computer program, wherein the computer program controls equipment where the computer readable storage medium is located to execute the tongue image multispectral image generation method according to any embodiment.
The foregoing embodiments have been provided for the purpose of illustrating the general principles of the present invention, and are not to be construed as limiting the scope of the invention. It should be noted that any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art without departing from the spirit and principles of the present invention are intended to be included in the scope of the present invention.

Claims (8)

1. A tongue image multispectral image generation method, comprising:
Photographing the tongue body and a preset standard color chart aligned with the tongue body through photographing equipment to obtain a tongue picture detection image, wherein the tongue picture detection image comprises a preset standard color chart area image and a tongue picture image of the tongue body;
extracting the preset standard colorimetric card region image in the tongue image detection image;
Converting the preset standard color card area image into an HSV image form, and carrying out color partition on the converted preset standard color card area image so as to extract corresponding standard colors in different color partitions;
Calculating RGB color gamut values of the standard colors, and calculating spectral responses of ambient light and the photographing equipment according to reflection spectrum curves of the standard colors and the RGB color gamut values of the standard colors;
extracting a tongue image from the tongue image detection image according to a preset neural network model;
and carrying out spectrum reconstruction on the tongue image according to the ambient light and the spectral response of the photographing equipment, thereby generating a tongue image multispectral image.
2. The method for generating a tongue picture multispectral image according to claim 1, wherein the tongue picture image is extracted from the tongue picture detection image according to a preset neural network model, specifically:
according to a preset neural network model, performing image segmentation on the tongue picture detection image to generate a mask image of the tongue picture image;
and extracting tongue picture region information from the tongue picture detection image according to the mask image of the tongue picture image, so as to obtain a tongue picture image.
3. The tongue image multispectral image generating method according to claim 2, wherein the method for constructing the preset neural network model comprises the following steps:
Constructing an initial neural network model;
and setting a loss function and an optimizer in the model training process, and training the initial neural network model by using a preset tongue image data set, so as to obtain a trained neural network model.
4. A method for generating a tongue image multispectral image according to claim 3, wherein the tongue image is spectrally reconstructed according to the spectral responses of the ambient light and the photographing device, thereby generating a tongue image multispectral image, specifically:
Performing spectrum reconstruction on each pixel point of the tongue image, acquiring a color gamut value of the pixel point in the tongue image in the process of performing spectrum reconstruction on any pixel point, calculating an optimal characteristic coefficient by an optimization algorithm by combining the color gamut value and the spectrum response of the ambient light and photographing equipment, and calculating a pixel value of the pixel point in the multispectral image according to the characteristic coefficient and a preset characteristic vector;
And after each pixel point in the tongue image calculates the corresponding multispectral image pixel value, generating tongue image spectral images in different wave bands according to the corresponding spectral image pixel value of each pixel point in different wave band ranges.
5. A method of generating a tongue image multispectral image according to any one of claims 1-4, further comprising, after generating the tongue image multispectral image:
Uploading the tongue image multispectral image to a data server for storage.
6. The tongue image multispectral image generating device is characterized by comprising an image acquisition module, a standard colorimetric card module, a tongue image module and a spectrum reconstruction module;
The image acquisition module is used for photographing the tongue body and a preset standard color chart aligned with the tongue body through photographing equipment so as to obtain a tongue image detection image, wherein the tongue image detection image comprises the preset standard color chart area image and the tongue image of the tongue body;
The standard colorimetric card module is used for extracting the preset standard colorimetric card region image in the tongue image detection image, converting the preset standard colorimetric card region image into an HSV image form, and carrying out color partitioning on the converted preset standard colorimetric card region image, so that corresponding standard colors in different color partitions are extracted respectively, RGB color gamut values of the standard colors are calculated, and spectral responses of ambient light and the photographing equipment are calculated according to reflection spectrum curves of the standard colors and the RGB color gamut values of the standard colors;
The tongue picture module is used for extracting tongue picture images from the tongue picture detection images according to a preset neural network model;
the spectrum reconstruction module is used for carrying out spectrum reconstruction on the tongue image according to the ambient light and the spectrum response of the photographing equipment so as to generate a tongue image multispectral image.
7. A terminal device comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, the processor implementing the tongue image multispectral image generation method according to any one of claims 1 to 5 when the computer program is executed by the processor.
8. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored computer program, wherein the computer program, when run, controls a device in which the computer readable storage medium is located to perform the tongue image multispectral image generation method according to any one of claims 1 to 5.
CN202310387046.2A 2023-04-11 2023-04-11 Tongue multispectral image generation method, device, equipment and storage medium Active CN116509326B (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202310387046.2A CN116509326B (en) 2023-04-11 2023-04-11 Tongue multispectral image generation method, device, equipment and storage medium
PCT/CN2023/133264 WO2024212524A1 (en) 2023-04-11 2023-11-22 Tongue manifestation multispectral image generation method and apparatus, and device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310387046.2A CN116509326B (en) 2023-04-11 2023-04-11 Tongue multispectral image generation method, device, equipment and storage medium

Publications (2)

Publication Number Publication Date
CN116509326A CN116509326A (en) 2023-08-01
CN116509326B true CN116509326B (en) 2025-02-14

Family

ID=87402227

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310387046.2A Active CN116509326B (en) 2023-04-11 2023-04-11 Tongue multispectral image generation method, device, equipment and storage medium

Country Status (2)

Country Link
CN (1) CN116509326B (en)
WO (1) WO2024212524A1 (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116509326B (en) * 2023-04-11 2025-02-14 广东健齿生物科技有限公司 Tongue multispectral image generation method, device, equipment and storage medium
CN117611581B (en) * 2024-01-18 2024-05-14 之江实验室 Tongue image recognition method, device and electronic device based on multimodal information
CN118526159B (en) * 2024-05-17 2025-04-29 西湖智能视觉科技(杭州)有限公司 Tongue diagnosis and skin diagnosis method and device based on artificial intelligence non-scanning type spectral imaging

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2256120A1 (en) * 1997-12-23 1999-06-23 Melissa A. Schrader Method for removing oral cavity staining, method for processing hogs, apparatus for delivering stain removing solution, and method for reducing bacteria level in a hog carcass oral cavity
CN103222856A (en) * 2013-04-16 2013-07-31 天津中医药大学 Multispectral tongue picture collecting device

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006107947A2 (en) * 2005-04-04 2006-10-12 Hypermed, Inc. Hyperspectral imaging in diabetes and peripheral vascular disease
CN100412906C (en) * 2006-10-20 2008-08-20 清华大学 Digital Tongue Image Color Shift Correction Method
US8144320B2 (en) * 2006-11-22 2012-03-27 Optopo, Inc. Method and apparatus for reconstructing optical spectra in a static multimode multiplex spectrometer
KR101063343B1 (en) * 2009-11-24 2011-09-07 한국 한의학 연구원 Color Information Segmentation Method Using Tongue Image
DE102014101059A1 (en) * 2014-01-29 2015-07-30 Heraeus Kulzer Gmbh Gingiva indexing device and method of indexing the gingiva
JP6450085B2 (en) * 2014-04-14 2019-01-09 フェムトディプロイメンツ株式会社 Health condition inspection device
CN106893778B (en) * 2017-03-03 2020-03-31 青岛泱深生物医药有限公司 Molecular marker for diagnosing and treating tongue squamous carcinoma
GB201708520D0 (en) * 2017-05-27 2017-07-12 Dawood Andrew A method for reducing artefact in intra oral scans
CN110807775A (en) * 2018-08-06 2020-02-18 深圳市前海安测信息技术有限公司 Traditional Chinese medicine tongue image segmentation device and method based on artificial intelligence and storage medium
CN109700433A (en) * 2018-12-28 2019-05-03 深圳铁盒子文化科技发展有限公司 A kind of tongue picture diagnostic system and lingual diagnosis mobile terminal
CN113781488B (en) * 2021-08-02 2025-01-28 横琴鲸准智慧医疗科技有限公司 Tongue image segmentation method, device and medium
CN113724228B (en) * 2021-08-31 2024-05-10 平安科技(深圳)有限公司 Tongue color and tongue fur color identification method and device, computer equipment and storage medium
CN218738913U (en) * 2022-09-05 2023-03-28 厦门妙机智能有限公司 Traditional Chinese medicine tongue diagnosis instrument
CN115587986A (en) * 2022-10-17 2023-01-10 航天科工深圳(集团)有限公司 Tongue diagnosis method, electronic device and storage medium
CN115601358B (en) * 2022-12-01 2023-03-28 合肥云诊信息科技有限公司 Tongue picture image segmentation method under natural light environment
CN115691793A (en) * 2022-12-28 2023-02-03 智慧眼科技股份有限公司 Method and device for processing tongue picture data and related equipment
CN116509326B (en) * 2023-04-11 2025-02-14 广东健齿生物科技有限公司 Tongue multispectral image generation method, device, equipment and storage medium

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CA2256120A1 (en) * 1997-12-23 1999-06-23 Melissa A. Schrader Method for removing oral cavity staining, method for processing hogs, apparatus for delivering stain removing solution, and method for reducing bacteria level in a hog carcass oral cavity
CN103222856A (en) * 2013-04-16 2013-07-31 天津中医药大学 Multispectral tongue picture collecting device

Also Published As

Publication number Publication date
CN116509326A (en) 2023-08-01
WO2024212524A1 (en) 2024-10-17

Similar Documents

Publication Publication Date Title
CN116509326B (en) Tongue multispectral image generation method, device, equipment and storage medium
US8155413B2 (en) Method and system for analyzing skin conditions using digital images
US8027533B2 (en) Method of automated image color calibration
US20080080766A1 (en) Apparatus and Method for Analyzing Skin Using L*a*b* Colorspace
US20110286644A1 (en) Image calibration and analysis
US8666130B2 (en) Systems and methods for bio-image calibration
CN110691226A (en) Image processing method, device, terminal and computer readable storage medium
CN113609907A (en) Method, device and equipment for acquiring multispectral data
JP2011210156A (en) Image processing apparatus, image reading apparatus, image processing method and image processing program
Fernandes et al. Early skin cancer detection using computer aided diagnosis techniques
CN117252875B (en) Medical image processing methods, systems, media and equipment based on hyperspectral images
CN113749642B (en) Method, system, medium and terminal for quantifying degree of skin flushing response
CN112788200B (en) Method and device for determining frequency spectrum information, storage medium and electronic device
Zheng et al. Corneal‐Smart Phone: A novel method to intelligently estimate postmortem interval
Khan et al. Dental image enhancement network for early diagnosis of oral dental disease
CN115736791B (en) Endoscopic imaging device and method
CN110319933B (en) Light source spectrum optimization method based on CAM02-UCS color appearance model
CN118735830B (en) Endoscopic image brightness correction method, device, equipment and storage medium
EP3940370B1 (en) Method for extracting spectral information of object to be detected
JP2001311666A (en) Colorimetric conversion coefficient calculation method and colorimetric image conversion method, and colorimetric conversion coefficient calculation device and colorimetric image conversion device, and computer-readable information recorded with a colorimetric conversion coefficient calculation program or a colorimetric imaging program recoding media
JP4378810B2 (en) Colorimetric conversion coefficient calculation method, colorimetric imaging method, colorimetric conversion coefficient calculation device, colorimetric imaging device, and computer-readable information recording medium recording a colorimetric conversion program
US20210174147A1 (en) Operating method of image processing apparatus, image processing apparatus, and computer-readable recording medium
CN118434348A (en) Method and system for analyzing perfusion parameters of skin
JP2023143591A (en) Method for identifying artery and vein in ocular fundus image
CN114972065A (en) Training method, system, electronic device and mobile device for chromatic aberration correction model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant