CN113298089A

CN113298089A - Venous transfusion liquid level detection method based on image processing

Info

Publication number: CN113298089A
Application number: CN202110535142.8A
Authority: CN
Inventors: 邹章晨; 邹学玉; 巴桂; 喻旸
Original assignee: Yangtze University
Current assignee: Yangtze University
Priority date: 2021-05-17
Filing date: 2021-05-17
Publication date: 2021-08-24

Abstract

The invention provides a venous transfusion liquid level detection method based on image processing, which is characterized by comprising the following steps of: s1: collecting the picture of the infusion bottle during infusion by a camera; s2: processing the picture, and carrying out target detection on the infusion bottle through an SSD model; s3: detecting the liquid level of the detected target infusion bottle in the S2 by adopting a projection method, and detecting the position of the liquid level of the infusion bottle; s4: and judging whether the liquid level reaches a warning line. The venous transfusion liquid level monitoring device has the advantages that the change condition of the liquid level during venous transfusion can be known in time, and convenience can be provided for medical staff and patients.

Description

Venous transfusion liquid level detection method based on image processing

Technical Field

The invention relates to the field of image processing, in particular to a venous transfusion liquid level detection method based on image processing.

Background

The intelligent medical treatment combines technologies such as big data, cloud computing and Internet of things and is used for improving the relation among patients, medical care personnel and a medical environment. Meanwhile, the medical environment develops towards informatization and intelligent management, the service quality and efficiency in the medical process are greatly improved, and remote intelligent monitoring is realized. Intravenous infusion is a common treatment method in the medical treatment process, and has very wide application because of high drug delivery speed and good treatment effect. In medical infusion, the problem that the liquid level height of an infusion bottle cannot be checked in time is always that a patient and a nursing staff are quite headache. If the transfusion bottle is not processed in time, the transfusion bottle is empty, the blood return condition can occur, the injection part of the patient has swelling pain if the transfusion bottle is not processed in time, and the patient can be shocked if the transfusion bottle is not processed in time. When the number of patients is large and medical staff is limited, the condition of each patient cannot be considered. The existing liquid level detection method has the defects that the mechanical detection method has the problems of complex structure, low precision and the like; the non-contact measurement method is like ultrasonic wave and is expensive.

Patent document No. CN104819754B discloses a method for detecting the liquid level of a medicine bottle based on image processing. The key point of the invention is that a template matching method is used, a typical drug image is collected to create a reference template image, a matching template is established, and a matching template coordinate system is created; matching the collected images of the plurality of detected medicine bottles with the matching template one by one to obtain a displacement matrix of the images of the detected medicine bottles, and obtaining a corrected image of the images of the detected medicine bottles by affine transformation; then, dividing the corrected image to obtain a detection area and extracting the shape of the liquid level band; and then, calculating the lowest point of the liquid level based on the shape of the liquid level belt, and judging the qualification of the medicine bottle according to the relationship between the lowest point of the liquid level and the set threshold value of the highest point. The method has the disadvantage that when the difference between the template and the image to be detected is large, the detection cannot be completed well.

Patent document No. CN105181082B discloses a liquid level detection method and apparatus based on visible light and image processing. Firstly, establishing a measurement coordinate system with the background of the inner wall of a container for storing liquid in the vertical direction; the camera and the laser head for emitting visible light laser are arranged above the liquid level, and the camera is inclined downwards relative to the horizontal plane to face the background; the optical axis of the camera forms a certain included angle with the horizontal plane; then, detecting a moving target of the visible laser spot to obtain a moving track of the visible laser spot; and then acquiring the corner coordinates corresponding to the liquid level position, and solving the actual liquid level value based on the corner coordinates. The method has the disadvantage of being expensive.

Patent No. CN208877510U discloses a monitoring device for venous transfusion, in which a transfusion mechanism is provided with an amplifying structure, the state of liquid in a transfusion tube can be amplified, in addition, the detection mode adopts image acquisition, the liquid level can be clearly detected, in addition, a retaining ring is provided, a hose is buckled, and the shaking is reduced. The method is carefully designed, but also increases the cost of the equipment.

Disclosure of Invention

The invention aims to provide a venous transfusion liquid level detection method based on image processing aiming at the defects of the prior art, so that the change condition of the liquid level during venous transfusion can be known in time, and convenience can be provided for medical care personnel and patients.

The invention provides a venous transfusion liquid level detection method based on image processing, which is characterized by comprising the following steps of:

s1: collecting the picture of the infusion bottle during infusion by a camera;

s2: processing the picture, and carrying out target detection on the infusion bottle through an SSD model;

s3: detecting the liquid level of the detected target infusion bottle in the S2 by adopting a projection method, and detecting the position of the liquid level of the infusion bottle;

s4: and judging whether the liquid level reaches a warning line.

In the above technical solution, the step of establishing the SSD model in step S2 is as follows:

shooting pictures of the infusion bottle with different postures, directions, distances, brightness, definition and angles by using a camera to form a sample set;

making a data set through the sample set; marking an infusion bottle in a picture of a sample set by a marking tool, processing the picture to generate an xml file, and converting the xml file into a TF format file as a data set;

vgg-16.ckpt, setting hyper-parameters, and sending the data set into an SSD model file for training and learning;

after the SSD model is generated after training, the SSD model is sent to a test set for detection, and the result of model training is observed.

In the above technical solution, step S3 specifically includes the following steps: preprocessing the image of the target infusion bottle, including graying, image filtering and pooling; detecting the overall outline of the infusion bottle through Canny edge detection; acquiring a plurality of possible liquid level position information according to the overall outline of the infusion bottle; whether each liquid level position information is a temporary qualified line or not is judged by scanning the accumulated number of the edge points of each liquid level position information in the horizontal direction (the standard of the qualified line is that the accumulated number of the edge points reaches more than one tenth of the whole line); judging all the qualified lines one by one, and judging whether the qualified lines are larger than a preset value or not;

if only one qualified line is larger than the preset value, the liquid level position information represented by the qualified line is judged to be the position of the current liquid level;

if the qualified lines are larger than a preset value, sorting the qualified lines according to length; preferentially judging the longest qualified line; comparing the liquid level position represented by the qualified line with the liquid level of the previous frame of picture, judging that the liquid level position is not the position of the liquid level if no position change exists, recording the position which is not the liquid level and the liquid level length, and then continuously judging the next qualified line;

otherwise, the position of the liquid level is judged and the length of the liquid level is recorded.

In the above technical solution, the step S4 specifically includes the following steps:

calculating the length of a liquid level line of horizontal projection by an integral projection method, and setting the length of an alarm horizontal liquid level; when the length of the liquid level line of the horizontal projection is judged to be smaller than the length of the alarm horizontal liquid level, the fact that the transfusion is about to end is indicated, and a warning prompt is sent out; otherwise, the infusion is not completed.

In the above technical solution, the integral projection method projects an image from two directions, i.e. horizontal and vertical directions, and the horizontal projection h (n) and the vertical projection v (m) are respectively defined as follows:

wherein, (N, M) represents the position of the pixel, I (N, M) represents the gray value of the pixel, N represents the number of pixels in the whole row, and M represents the number of pixels in the whole column.

In the above technical solution, in step S3, the data amount of the internal picture obtained by canny edge detection is reduced by mean pooling, and convolution operation is performed on the picture by using 3 × 3 kernels, so that the content edge in the picture becomes a straight line.

In the above technical solution, in step S2, a target data set is input, feature extraction is performed through a basic convolutional neural network of a VGG16 pre-training model file, prior frames are generated in a feature extraction layer and a partial convolutional neural network layer, the prior frames and a top network are associated, an additional convolutional layer is added to the basic network to predict frame offset and a target category in the prior frames, and a non-maximum value is used to suppress a screening result.

In the above technical solution, in step S2, the hyper-parameters that need to be set for training the SSD model include: category number, tag number, maximum number of steps performed; in the training process of the SSD model, the model is evaluated by entering a TensorBoard visual interface and observing the change of the loss rate and the learning rate: the smaller the loss rate is, the more stable the loss rate is, the model is trained well, and the iteration number also reaches the appropriate requirement. With the increase of the training steps, the average loss rate value is in a descending trend, and the fluctuation amplitude of the average loss rate value is gradually reduced, so that the error between the model prediction sample and the real sample is evaluated; the learning rate is gradually reduced along with the increase of the training steps; the smaller the learning rate is, the more accurate the feature learning is, so as to evaluate the accuracy of model training.

The invention has the beneficial effects that the venous transfusion liquid level detection method based on image processing is provided for solving the problem of low precision of venous transfusion liquid level detection. According to the invention, a deep learning-based SSD (Single Shot Multi Box Detector) algorithm is adopted, the VGG16 is used as a basic network for feature extraction, a multi-scale frame mechanism is adopted for local feature learning, a Non-Maximum Suppression (NMS) is adopted to screen out a default frame, and finally, the position and the class label of a target frame are output, so that the target detection is carried out on the infusion bottle, and the position of the infusion bottle is conveniently and quickly positioned; the liquid level of the infusion bottle is detected by adopting an integral projection method based on image processing, the position of a liquid level line can be accurately and quickly detected, and therefore, the alarm line threshold value is judged according to the projection length of the liquid level line, and whether the alarm state is reached or not is judged. The invention successfully combines the SSD algorithm with the integral projection method, improves the precision and time of the intravenous infusion liquid level detection, and greatly improves the working efficiency of medical care personnel.

Drawings

FIG. 1 is a system block diagram of the present invention

FIG. 2 is a flow chart of the detection of the present invention

FIG. 3 is a flow chart of the SSD algorithm

FIG. 4a is an original image to be detected, and FIG. 4b is a result graph of object detection

FIG. 5 shows the results (parts) of the multiple tests of the embodiment

FIG. 6 is a flow chart of liquid level detection by projection method

FIGS. 7a-7c are graphs showing the effect of the liquid level detection of the present invention (group 1)

FIGS. 8a-8c are graphs of the effectiveness of the level detection of the present invention (group 2).

Detailed Description

The invention will be further described in detail with reference to the following drawings and specific examples, which are not intended to limit the invention, but are for clear understanding.

As shown in fig. 1, which is a block diagram of the intravenous infusion liquid level detection system of the present scheme, the frame is composed of a monitoring object (infusion bottle), a camera, and an image processing terminal:

(1) detecting an object: the real-time infusion liquid level of a single patient intravenous infusion bottle in the infusion chamber;

(2) a camera: the camera collects data of the infusion bottle and transmits the shot picture to the image processing terminal;

(3) an image processing terminal: and the PC end processes the pictures transmitted by the camera, and performs target detection and liquid level detection on the infusion bottle by adopting an SSD algorithm based on deep learning and an integral projection method based on image processing.

As shown in fig. 2, the process chart of the intravenous infusion liquid level detection of the present scheme is divided into the following steps:

(1) image acquisition: and a camera is used for taking a sample set from different angles such as shooting posture, direction, distance, brightness, definition and the like. The infusion bottle data set collection can only be carried out by self shooting because the public data set can not be obtained, and pictures of different liquid levels, angles and distances of the infusion bottle are shot as many as possible. A mobile phone rear camera with the resolution of 800 ten thousand and a back-illuminated/BSI CMOS is supposed to be adopted. Sampling and photographing the infusion state of the infusion bottle, and transmitting the sampled and photographed infusion state to the PC terminal through the network. The infusion bottle is shot and sampled at different illumination (bright or dark) and at different distances of 1 meter and 2 meters and at different angles.

(2) Target detection: and processing the picture, and carrying out target detection on the infusion bottle through an SSD model. In view of the complex environment, the infusion bottle needs to be quickly detected, a large amount of data of the infusion bottle is collected through an SSD target detection algorithm based on deep learning, model training is carried out on the data, and finally target detection is carried out. This step is critical to facilitate the deployment of subsequent jobs. Firstly, a data set is manually made, and a LabLeImage sample labeling tool is used for manually labeling a shot image. And generating an xml file after processing the picture, and then converting the xml file into a TF format file. And downloading a VGG16 model file and setting the hyper-parameters. Sending the training set into an SSD model for training and learning, extracting features of a VGG16 basic network, generating different default frames by a multi-scale frame mechanism, then carrying out intersection-comparison threshold screening, and outputting detection frames, categories and confidence scores through non-maximum value inhibition; after the SSD model is trained, the SSD model is sent to a test set for detection, and the quality of model training is observed.

The artifact labeling uses the label _ image tool to click on the Creat Rectbox, a small box of objects as possible, to learn more carefully during the training process. And (3) proportionally distributing a training verification set by the data set: test set 4: 1, the unified image size is 300 × 300.

The SSD model is established by the following steps:

making a data set through the sample set, marking an infusion bottle in a picture of the sample set through a marking tool, processing the picture to generate an xml file, and converting the xml file into a TF format file as the data set;

after the SSD model is generated after training, the SSD model is sent to a test set for detection, and the quality of model training is observed.

The training SSD model requires setting parameters: category number, tag number, maximum number of steps performed, etc. Limited to computer configuration, the maximum number of iteration steps set herein: max _ number _ of _ steps 10000, leaving _ rate 0.001, and batch _ size 4.

The hyper-parameters required to be set for training the SSD model comprise: category number, tag number, maximum number of steps performed; in the training process of the SSD model, the parameter curve of the evaluation model can be intuitively seen to continuously change along with the increase of the training steps by means of a TensorBoard visual interface. The parameters of the evaluation model which are important are loss rate and learning rate. The smaller the loss rate is, the more stable the loss rate is, the model is trained well, and the iteration number also reaches the appropriate requirement. With the increase of the number of steps, the average loss rate is in a descending trend, and the fluctuation amplitude is gradually reduced, so that the error between the model prediction sample and the real sample is evaluated. The learning rate gradually decreases as the number of steps increases; the smaller the learning rate is, the more accurate the feature learning is, so as to evaluate the accuracy of model training.

(3) Liquid level detection: and carrying out preprocessing on the image, including graying processing, image filtering and pooling.

Detecting the overall outline of the infusion bottle through Canny edge detection; since the liquid level is always horizontal and longest in the local area of the bottle, the pooled picture is detected by the canny edge, and although there is interference, the general information of the liquid level is still preserved. Whether the pixel points are temporary qualified lines is judged by scanning the accumulated number of the points (namely the detected edge points) with the gray value of 255 of the pixel points in the horizontal direction, and all the qualified lines are judged one by one to see whether the gray values are larger than a preset value, wherein the gray values are set to be 0.3 per srcmagebin.

Because of the horizontal edge L of the trademark of the infusion bottle_SIs generally positioned at the middle lower part of the infusion bottle and is far larger than the set alarm horizontal liquid level length L_TTherefore, useless horizontal edges are removed, and the interference of the horizontal edges of the trademarks on liquid level detection is avoided.

(4) The method adopts an integral projection method to detect the liquid level of the infusion bottle, and the integral projection method mainly projects an image from the horizontal direction and the vertical direction. The horizontal projection and the vertical projection are respectively defined as follows:

wherein, (N, M) represents the position of the pixel, I (N, M) represents the gray value of the pixel, N represents the number of pixels in the whole row, and M represents the number of pixels in the whole column. The horizontal projection is to accumulate the gray values of the pixels in the whole row, the vertical projection is to accumulate the gray values of the pixels in the whole column, and the integral projection function can truly display the total pixel value of the image from the vertical direction or the horizontal direction. The method of horizontal projection is used herein so that the level of the image can be located. The difficulty of liquid level detection is that: the infusion bottle is covered by a trademark, so that the horizontal edge of the trademark can generate an interference item on liquid level detection, and useless horizontal edges are eliminated by reasonably setting the length threshold of the horizontal projection line segment; in addition, the liquid surface is detected directly by canny edge, but the found liquid surface edge has an angle and becomes a 'oblique line', the picture data volume is reduced by pooling of the mean value, and the 'angle' is eliminated by performing convolution operation on the picture by using a 3 × 3 kernel, so that the found edge is 'a straight line'. The method can achieve a better effect to a certain extent and keep key information.

(5) Judging and measuring the alert position: an alarm threshold value is preset, and when the liquid level line approaches the set threshold value, judgment is carried out and an alarm is given out, so that a nurse draws a needle for a patient or changes liquid. And if the threshold value is not reached, continuing to execute the step (1).

As shown in fig. 3, a SSD algorithm flow chart is shown. The main step S2 is: inputting a target data set, carrying out feature extraction through a basic convolutional neural network of a VGG16 pre-training model file, generating a prior frame on a feature extraction layer and a partial convolutional neural network layer, associating the prior frame with a top-level network, adding an additional convolutional layer on the basic network to predict frame offset and a target class in the prior frame, and inhibiting a screening result by using a non-maximum value.

As shown in fig. 4a and 4b, the result is a single-infusion bottle detection result. In this case, box threshold value select _ threshold is 0.4, and non-maximum threshold value nms _ threshold is 0.4. As shown in the figure, fig. 4a is an original image to be detected, and fig. 4b is a result graph of target detection. The category label is output: 1, the accuracy rate is more than 94% and the testing time is within 1.02 s.

As shown in fig. 5, a diagram showing the results of target detection of a plurality of infusion bottles is shown. The average accuracy rate of the 60 test sets is more than 81%, and the average test time is within 1.22 s.

Fig. 6 shows a flow chart of liquid level detection. Preprocessing the picture, including graying, image filtering and pooling; detecting an overall contour through Canny edge detection; and (3) roughly positioning the liquid level by using an integral projection method, detecting the position of the liquid level, judging whether the position reaches a liquid level warning line, and sending an alarm to inform when the liquid level line is smaller than a set threshold value, otherwise, indicating that the transfusion is not finished.

As shown in fig. 7a-c and fig. 8a-c, the effect of the present invention is shown. Fig. 7a and 8a show the results of performing the mean pooling plus the edge detection, fig. 7b and 8b show the results of performing the projection calculation, and fig. 7c and 8c show the results of the final liquid level detection, and it can be seen from the figures that the method can detect the location of the liquid level line well, and in the process, the liquid level detection time is within 0.62s, and the liquid level line can be detected quickly and accurately. Meanwhile, by calculating the length of the liquid level line of the horizontal projection, when the length of the liquid level line of the horizontal projection is smaller than the length of the liquid level at the bottom of the bottle, the completion of the transfusion is indicated, and a warning prompt is sent out; when the length of the liquid level line is larger than the set threshold value, the transfusion is not finished. The difficulty of liquid level detection is as follows: the infusion bottle is covered by a trademark, so that the horizontal edge of the trademark can generate an interference item on liquid level detection, and useless horizontal edges are eliminated by reasonably setting the length threshold of the horizontal projection line segment; in addition, the mean pooling is used, so that the vertical edges can be well removed, the method can achieve a good effect to a certain extent, and key information is reserved.

Through the result display, the SSD algorithm is adopted to detect the target, the complexity is reduced, the real-time performance is improved, the position of the infusion bottle can be detected quickly and accurately, the average test time is within 1.22s, and the average accuracy rate is more than 81%. The method for detecting the liquid level of the infusion solution by adopting the projection method based on image processing is simple to operate, the liquid level line can be accurately and quickly detected, the liquid level detection time is within 0.62s, and the accuracy rate is more than 69%.

Practical researches show that the SSD target detection algorithm based on deep learning and the projection method based on image processing can better realize the liquid level detection function of venous transfusion, and have certain practical value. Can play a beneficial auxiliary role in the medical environment and improve the related working efficiency. In addition, the method can also be applied to other liquid level detection environments such as lake surface water level detection, milk bottle liquid level detection and the like.

Details not described in this specification are within the skill of the art that are well known to those skilled in the art.

Claims

1. A venous transfusion liquid level detection method based on image processing is characterized by comprising the following steps:

s1: collecting the picture of the infusion bottle during infusion by a camera;

s4: and judging whether the liquid level reaches a warning line.

2. The image processing-based intravenous infusion liquid level detection method according to claim 1, wherein the SSD model establishing step in step S2 is as follows:

3. The method for detecting the level of intravenous infusion solution based on image processing as claimed in claim 1, wherein the step S3 comprises the following steps: preprocessing the image of the target infusion bottle, including graying, image filtering and pooling; detecting the overall outline of the infusion bottle through Canny edge detection; acquiring a plurality of possible liquid level position information according to the overall outline of the infusion bottle; judging whether each liquid level position information is a temporary qualified line or not by scanning the accumulated number of edge points of each liquid level position information in the horizontal direction; judging all the qualified lines one by one, and judging whether the qualified lines are larger than a preset value or not;

4. The method for detecting the level of intravenous fluid infusion based on image processing as claimed in claim 1, wherein said step S4 specifically comprises the steps of:

5. The method of claim 3, wherein the integral projection method projects the image from both horizontal and vertical directions, and the horizontal projection H (n) and the vertical projection V (m) are respectively defined as follows:

6. The method according to claim 1, wherein in step S3, the average pooling is used to reduce the amount of internal picture data obtained by canny edge detection, and a 3 x 3 kernel is used to perform convolution operation with the picture, so that the content edge in the picture becomes a straight line.

7. The method of claim 2, wherein in step S2, the target data set is input, feature extraction is performed through a basic convolutional neural network of a VGG16 pre-training model file, prior frames are generated at a feature extraction layer and a partial convolutional neural network layer, the prior frames and a top-level network are associated, an additional convolutional layer is added to the basic network to predict frame offset and target classes in the prior frames, and a non-maximum value is used to suppress the screening result.

8. The image processing-based intravenous infusion liquid level detection method according to claim 2, wherein in step S2, the hyper-parameters required to be set for training the SSD model comprise: category number, tag number, maximum number of steps performed; in the training process of the SSD model, the model is evaluated by entering a TensorBoard visual interface and observing the change of the loss rate and the learning rate: the smaller the loss rate is, the more stable the loss rate is, the model is trained well, and the iteration times also reach the appropriate requirements; with the increase of the training steps, the average loss rate value is in a descending trend, and the fluctuation amplitude of the average loss rate value is gradually reduced, so that the error between the model prediction sample and the real sample is evaluated; the learning rate is gradually reduced along with the increase of the training steps; the smaller the learning rate is, the more accurate the feature learning is, so as to evaluate the accuracy of model training.