[go: up one dir, main page]

WO2015170321A1 - Système et procédé d'analyse de motilité de flux d'image in vivo - Google Patents

Système et procédé d'analyse de motilité de flux d'image in vivo Download PDF

Info

Publication number
WO2015170321A1
WO2015170321A1 PCT/IL2015/050465 IL2015050465W WO2015170321A1 WO 2015170321 A1 WO2015170321 A1 WO 2015170321A1 IL 2015050465 W IL2015050465 W IL 2015050465W WO 2015170321 A1 WO2015170321 A1 WO 2015170321A1
Authority
WO
WIPO (PCT)
Prior art keywords
motility
bar
image
image stream
contractile activity
Prior art date
Application number
PCT/IL2015/050465
Other languages
English (en)
Inventor
Michal Drozdzal
Santiago Segui-Mesquida
Jordi Vitria
Petia Radeva
Carolina Malagelada
Fernando Azpiroz
Original Assignee
Given Imaging Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Given Imaging Ltd. filed Critical Given Imaging Ltd.
Publication of WO2015170321A1 publication Critical patent/WO2015170321A1/fr

Links

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/00002—Operational features of endoscopes
    • A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
    • A61B1/00009—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope
    • A61B1/000094—Operational features of endoscopes characterised by electronic signal processing of image signals during a use of endoscope extracting biological structures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G06T7/0016—Biomedical image inspection using an image reference approach involving temporal comparison
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
    • A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
    • A61B1/041—Capsule endoscopes for imaging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30028—Colon; Small intestine

Definitions

  • the present invention relates to a method and system for image processing of an image stream captured in- vivo. More specifically, the present invention relates to systems and methods for analyzing motility properties of an image stream captured in a gastrointestinal (GI) tract, and for presenting a summarized display of the analysis.
  • GI gastrointestinal
  • In-vivo imaging methods such as performed by an in-vivo imaging system including a swallowable capsule, may be used to image body lumens within a patient.
  • the imaging system may capture and transmit, for example, images of the gastrointestinal (GI) tract to an external recording device, while the capsule passes through the GI lumen.
  • the capsule may capture images in variable frame rates of, for example, 1 - 40 frames per second. Large numbers of images, for example 100,000 to 300,000 images, may be collected for viewing during the imaging procedure, which may be performed in a duration of one to eight hours, and may be viewed and/or processed in real time.
  • the images may be combined in sequence, and an image stream or movie of, for example, 30 - 120 minutes in length, may be presented to a user.
  • the small bowel (also called small intestine) is a part of the GI tract, connecting the stomach with the large intestine.
  • the length of the small intestine in an adult is variable, and depending on the conditions can measure from 3 to 8 meters.
  • the main function of the small bowel is the digestion and absorption of nutrients and minerals found in the food. In order to do so, the small intestine pushes the food through by the means of a physiological mechanism called motility.
  • Intestinal motility can be divided into two categories: peristalsis, e.g. synchronized movement of the intestinal wall responsible for moving the food in one direction; and independent contractions, e.g. unsynchronized movement of the intestinal wall where the muscles squeeze substantially independently of each other, which may have the effect of mixing the contents but not moving them up or down.
  • peristalsis e.g. synchronized movement of the intestinal wall responsible for moving the food in one direction
  • independent contractions e.g. unsynchronized movement of the intestinal wall where the muscles squeeze substantially independently of each other, which may have the effect of mixing the contents but not moving them up or down.
  • Image analysis may be performed, for example in order to reduce a physician's viewing time of the captured image stream, or to improve the efficiency of review.
  • a summarized presentation of the image analysis may be displayed using a graphical user interface.
  • US Patent Number 7,215,338, incorporated herein by reference in its entirety discloses methods of generating and displaying a summarized color bar presentation, which includes a series of summaries of color data of one or more images from a data stream.
  • An image stream may be captured by an in vivo device, the image stream may include image frames, and each frame may include a plurality of pixels.
  • a processing unit e.g., a central processing unit, processing unit, computer processor or computer controller
  • the motility bar may be filtered to obtain a filtered representation thereof, and contractile activity patterns may be detected in the filtered representation of the motility, bar by detecting local maxima points.
  • the motility bar is filtered using a plurality of filters, each filter adjusted to detect contractions in the motility bar at a predetermined time scale, and results of each filter may be combined into a single signal representing contractions positions in the motility bar.
  • the processing unit may further be configured to generate a display representing the detected contractile activity patterns on a visual display unit.
  • the detected local maxima points may be converted into a signal representing contractions of the gastrointestinal tract, and the density of contractions in the image stream may be calculated. In some embodiments, the calculated density may be displayed on the visual display unit. Portions of the image stream with no contractile activity may be detected or identified, and/or segments of the image stream may be classified into contractile activity pattern categories selected from: contractile activity and no contractile activity.
  • the system may include a memory for storing frames of an image and a processing unit which is configured to obtain a motility bar representing motility properties of the image stream.
  • the processing unit may calculate motility bar intensity values to identify lumen regions in the motility bar, and use the identified lumen regions to identify lumen region values for each motility bar line.
  • a lumen region value may indicate a degree or level of lumen region presence in an image frame represented by the motility bar line.
  • the processing unit may generate a visual representation of lumen region presence in each line of the motility bar, and a display unit may be provided to display the visual representation of the of lumen region presence.
  • the processor may be configured to smooth identified lumen regions, and calculate or estimate lumen region values by calculating a percentage of lumen region occupying each line of the motility bar.
  • open tunnel segments of the image stream may be identified based on the lumen region values, by determining segments in the motility bar which attained a lumen region value within a predetermined value range.
  • Embodiments presented herein may be combined, for example implemented in a single system which detects contractile activity patterns and lumen region values, and displays a combined visual presentation of motility-related properties of the analyzed image stream.
  • Fig. 1 is a schematic diagram of an in-vivo imaging system according to an embodiment of the present invention
  • Fig. 2A is an example of a series of computer-generated images which represent intestinal contractions, according to embodiments of the invention
  • Fig. 2B is an example of a motility bar calculated for the images of Fig. 2A;
  • Fig. 2C presents examples of fixed cuts and adaptive cuts for calculating a motility bar according to embodiments of the present invention
  • Fig. 3 shows steps of a method for contractile activity detection in a motility bar according to embodiments of the present invention
  • Figs. 4A - 4B show an exemplary system architecture for applying filters to a motility bar image according to embodiments of the invention
  • Figs. 4C - 4D show exemplary filters which may be applied to a motility bar image according to embodiments of the invention
  • Fig. 5 shows steps of a method for contraction analysis according to embodiments of the present invention.
  • Fig. 6A is a flow chart of a method for generating a motility bar based on an in vivo image stream, according to embodiments of the present invention
  • Fig. 6B is a flow chart of a method for contractile activity analysis according to an embodiment of the invention.
  • Fig. 7 shows contractile activity pattern bars along with motility bars which were calculated for the same image stream according to embodiments of the present invention
  • Fig. 8A shows a density plot of colors for a motility bar before applying filtering
  • Fig. 8B shows the result after of applying filtering to the RGB feature space of a motility bar image, according to an embodiment of the invention
  • Fig. 9 shows images generated by applying a lumen diameter estimation method according to embodiments of the invention.
  • Fig. 10 is an exemplary display of a lumen diameter graph for an image stream according to embodiments of the invention.
  • Fig. 11 is a flow chart of a method for generating a display representing the lumen region presence in frames of an image stream according to an embodiment of the present invention.
  • Fig. 12 is a schematic illustration of a Graphic User Interface according to an embodiment of the present invention.
  • Imaging, receiving, processing, storage and/or display units suitable for use with embodiments of the present invention may be similar to embodiments described in US Patent Application Publication Number 2006/0074275, entitled “System and Method for Editing an Image Stream Captured In- Vivo", U.S. Pat. No. 7,009,634 to Iddan et al., entitled “Device for In- Vivo Imaging”, and/or US Patent Application Publication Number 2007/0118012, entitled “Method of Assembling an In- Vivo Imaging Device", each assigned to the common assignee of the present application and incorporated by reference herein in its entirety.
  • US Patent Number 7,215,338 to Horn et al. discloses in some embodiments a system and a method for creating a summarized graphical presentation of a data stream captured in-vivo.
  • the graphical presentation may be in the form of a color bar.
  • Devices and systems as described herein may have other configurations and other sets of components.
  • Devices, systems and methods according to embodiments of the present invention may be similar to the commercial PillCam® SB2 or PillCam® Colon capsules and the associated data recorders and RAPID® workstation provided by Given Imaging, Ltd.
  • An in vivo imaging capsule which may be swallowed by a patient, may progress passively along the GI tract, due to peristaltic contractions which move the intestinal tissue walls. During its journey, the capsule passes through different GI organs, such as the esophagus, the stomach, the small bowel and the colon. The capsule may capture images at different image capture rates. Due to the relatively narrow tunnel structure of the small bowel tissue walls, while the capsule is traveling in the small bowel, it may maintain a position which is parallel to the direction of the tunnel. The longitudinal axis of the imaging capsule may generally remain parallel to the direction that the capsule advances in the small bowel.
  • One or more imaging systems of the capsule may be positioned in at least one of the longitudinal ends of the capsule, such that the imaging is performed generally in a forward and/or backward looking position, to capture images of the opening and closing of the lumen quite regularly.
  • Image data capturing the intestinal tissue walls and/or the opening and closing of the lumen hole, in combination with the recordation of the time of capturing each image, may permit analysis, display and/or calculation of properties or diagnoses of the patient's GI tract. For example, intestinal motility events, or type and frequency of peristaltic activity, or the amount of intestinal content which is depicted in the images may be analyzed.
  • Embodiments of the invention may enable segmented display of analyzed information of the GI tract to a medical professional in an efficient way, which may assist diagnosis or determination of the patient's condition.
  • the small intestine pushes the ingesta through by means of a physiological mechanism called motility.
  • motility can be categorized as follows: • Peristalsis - synchronized movement of the intestinal wall responsible for moving the ingesta in one direction.
  • the peristalsis and the segmentation are regulated by three types of contractions:
  • Ultrapropulsive contractions that are used to move rapidly the intestinal content without regard for digestion or absorption. These contractions are two to four times larger in amplitude and four to six times longer in duration than phasic contractions. Moreover, since the goal of these contractions is to clean the intestine rapidly, they propagate uninterruptedly over long distances of the small intestine.
  • images of movement of intestinal tissue walls may be classified as depicting different types or categories of intestinal events.
  • intestinal events may be detected over a sequence of several consecutive image frames. The following categories are examples of intestinal events:
  • contracts may be further classified into sub-categories, such as rhythmic phasic, ultrapropulsive and tonic contractions.
  • Obtaining a significant amount of image data may allow a detailed analysis of physiological condition.
  • the large amounts of the data may require a long duration of video visualization, therefore diagnosis of a study by the physician may take a relatively long time.
  • Detection, characterization and display of specific segments or portions of the image stream which have similar frame properties may be performed according to embodiments of the present invention. Such display may be useful for determining diagnosis or assessing condition of a patient by the physician.
  • a physician when reviewing and analyzing a patient's condition, may be interested in viewing points of transition of the in vivo imaging device during the image capturing procedure, e.g. transition of the imaging capsule from the esophagus to the stomach, transition from the stomach into the small bowel (e.g. duodenum detection), and/or transition from the small bowel into the colon (e.g., cecum detection).
  • transition points of transition of the in vivo imaging device during the image capturing procedure e.g. transition of the imaging capsule from the esophagus to the stomach, transition from the stomach into the small bowel (e.g. duodenum detection), and/or transition from the small bowel into the colon (e.g., cecum detection).
  • Intestinal motility event analysis may be performed and displayed, e.g. as disclosed in embodiments in US Published Patent Application 2015-0016700 to Drozdzal et al.
  • a summarized graphical display may indicate segments of the GI tract in which a certain level of intestinal activity was detected. Areas of the GI tract in which a high level of activity was detected (e.g., a high frequency of contractions) may be displayed, as well as segments of the GI tract in which substantially no contractions or a very low number of contractions occurred.
  • a summarized graphical display which segments the GI tract according to the amount or level of intestinal turbid content which was detected in the images.
  • the summarized graphical display may indicate segments of the GI tract in which a high amount of turbid intestinal content was detected, and/or segments which were detected as clean of turbid content.
  • the level of intestinal content may be related to detected segments of lumen diameter opening in the images, and such information may be graphically presented alongside the summarized display of the intestinal content segments.
  • Embodiments of the present invention describe a system and method for displaying summarized in vivo data, based on analysis and processing of data extracted from image frames captured in the GI tract.
  • Each image frame may be represented as a two-dimensional array of pixels, for example a rectangular or square pixel array of a certain height and a certain width (e.g. , 320 X 320 pixels or 1000 X 1000 pixels).
  • Each pixel may consist of one or more bits of information, representing the brightness of the image at that point and possibly including color information which may be encoded, for example as RGB color values, based on the RGB color model.
  • the RGB color model is a mathematical model for representing and displaying colors in electronic systems, in which red, green, and blue (R, G, B) light are added together in various ways to reproduce a broad array of colors.
  • Data which relates to the image capture timestamp and/or to the position of the in vivo device while capturing the image, may be encoded and transmitted either separately or along with the frame pixel data.
  • Analysis and processing of the image data may be performed automatically by a processing device, without user intervention.
  • the display of summarized graphical data for example using a color bar, window or display may be performed, e.g., by one or more processors, a workstation, circuitry, a detector or any other computation device.
  • one or more summarized graphical display windows or bars may be displayed to a health professional for diagnosis.
  • summarized data regarding the contractions types, frequency, variance, or intensity may be calculated and displayed to a user, e.g. upon receiving a user request.
  • FIG. 1 illustrates a schematic diagram of an in- vivo imaging system according to an embodiment of the present invention.
  • the system includes a capsule 40 having one or more imagers 46, for capturing images, one or more illumination sources 42, for illuminating the body lumen, and a transmitter 41, for transmitting image and possibly other information to a receiving device.
  • the image capture device may correspond to embodiments described in in U.S. Patent No. 7,009,634 to Iddan et al., and/or in U.S Published Patent Application No. 2007-0118012 to Gilad, each incorporated by reference herein in its entirety, but in alternate embodiments may be other sorts of image capture devices.
  • the images captured by the imager system may be of any suitable shape including for example circular, square, rectangular, octagonal, hexagonal, etc.
  • an image receiver 12 typically including an antenna or antenna array
  • an image receiver storage unit 16 typically including an antenna or antenna array
  • a data processor 14 e.g., a central processing unit, processing unit, computer processor or computer controller
  • a data processor storage unit 19 e.g., a central processing unit, processing unit, computer processor or computer controller
  • an image monitor or other visual display unit 18 e.g., a computer monitor
  • Data processor storage unit 19 includes an image database 21.
  • data processor 14, data processor storage unit 19 (e.g., a memory) and monitor 18 are part of a personal computer or workstation 11, which includes standard components such as processor 14, a memory, a disk drive, and input-output devices such as a mouse and keyboard, although alternate configurations are possible.
  • Data processor 14 may include any standard data processor, such as a microprocessor, multiprocessor, accelerator board, or any other serial or parallel high performance data processor.
  • Data processor 14, as part of its functionality, may act as a controller controlling the display of the images (e.g., which images, the location of the images among various windows, the timing or duration of display of images, etc.).
  • Image monitor 18 is typically a conventional video display, but may, in addition, be any other device capable of providing image or other data.
  • the image monitor 18 presents image data, typically in the form of still and moving pictures, motility data and in addition may present other information.
  • the various categories of information are displayed in windows.
  • a window may be for example a section or area (possibly delineated or bordered) on a display or monitor; other windows may be used.
  • Multiple monitors may be used to display images, motility properties, motility events and other data, for example an image monitor may also be included in image receiver 12.
  • a window of a set or sequence (e.g., ordered by time of capture or receipt, or another ordering) of frames may be a sequential subset of image frames within a stream of image frames.
  • a subset of image frames is typically a sequence or set of frames chosen from a larger sequence or set of frames.
  • imager 46 captures images and may send data representing the images to transmitter 41, which transmits images to image receiver 12 (e.g., as frames) using, for example, electromagnetic radio waves.
  • Image receiver 12 transfers the image data to image receiver storage unit 16.
  • the image data stored in storage unit 16 may be sent to the data processor 14 or the data processor storage unit 19.
  • the image receiver 12 or image receiver storage unit 16 may be taken off the patient's body and connected to the personal computer or workstation which includes the data processor 14 and data processor storage unit 19 via a standard data link, e.g., a serial, parallel, USB, or wireless interface of known construction.
  • the image data is then transferred from the image receiver storage unit 16 to an image database 21 within data processor storage unit 19.
  • Data processor 14 may analyze the data and provide the analyzed data to the image monitor 18, where a user views the image data.
  • data processor 14, or another data processor e.g. in receiver 12
  • Data processor 14 operates software that, in conjunction with basic operating software such as an operating system and device drivers, controls the operation of data processor 14.
  • the software controlling data processor 14 includes code written in the C++ language, and may be implemented using various development platforms such as Microsoft's .NET platform, but may be implemented in a variety of known methods.
  • each frame of image data includes 320 rows of 320 pixels each (e.g., 320 rows and 320 columns), each pixel including bytes for color and brightness, according to known methods.
  • each imager pixel may include a color sensor which may correspond to a single primary color, such as red, green, or blue.
  • the brightness of the overall pixel may be recorded by a one byte (i.e., 0-255) brightness value.
  • Images may be stored, for example sequentially, in data processor storage unit 19.
  • the stored data is comprised of one or more pixel values, including color and brightness.
  • Other image formats may be used.
  • Data processor storage unit 19 may store a series of images recorded by a capsule
  • the images the capsule 40 records, for example, as it moves through a patient's GI tract may be combined consecutively to form a series of images displayable as an image stream.
  • the user is typically presented with one or more windows on monitor 18; in alternate embodiments multiple windows need not be used and only the image stream may be displayed.
  • an image window may provide the image stream, or still portions of that image.
  • Another window may include buttons or other controls that may alter the display of the image; for example, stop, play, pause, capture image, step, fast-forward, rewind, or other controls.
  • Such controls may be activated by, for example, a pointing device such as a mouse or a finger on a touch screen.
  • the image stream may be frozen to view one frame, speeded up, or reversed; sections may be skipped; or any other method for viewing an image may be applied to the image stream.
  • Data processor 14 may generate, calculate or otherwise obtain a motility bar presentation by calculating pixel-based properties for each frame of at least a selected subset of frames from the image stream.
  • the motility bar may be filtered, and a filtered representation of it may be obtained and stored.
  • Embodiments for generation of a motility bar are disclosed, for example, in US Published Patent Application 2015-0016700, to Drozdzal et al, assigned to the common assignee of the present application and incorporated by reference herein in its entirety.
  • a motility bar may be e.g., a display where, moving linearly along the display, change indicates change within the image stream. Such a bar may be e.g., rectangular, but may be a different shape.
  • a color bar, motility bar, or motility chart may be a display or graphic, typically (but not exclusively) a rectangle including a series of strips or lines of color, each strip or line summarizing, e.g., using a color, one or more reference frames.
  • a motility bar generator 24 may be included or operationally connected to data processor 14, e.g. as described with relation to Figure 1A of the above mentioned publication 2015-0016700, to Drozdzal et al.
  • Data processor 14 may be, include, or may be operationally connected to, a motility display generator 24.
  • processor 14 may execute software or instructions to carry out methods according to embodiments of the invention, including the functionality of motility display generator 24.
  • Motility display generator 24 may process images from the captured set of images, and may obtain visual properties or portions of the images for display in a motility section of the GUI. Motility display generator 24 may produce a motility bar, based on selected cuts from images in the image stream.
  • a set of images from the image stream may be provided to motility display generator 24.
  • the set of images may include, for example, all images captured by the imaging device.
  • a subset of images may be used for generation of a motility bar.
  • the subset of images may be selected, for example according to different selection criteria.
  • the motility display generator 24 may select, for example, a "strip" from each image of a set of images (e.g. each image in a complete image stream or in a shortened image stream).
  • a "strip" or a "line” of an image may include a cut, a pixel line or a slice of pixels from an image, e.g. one or more adjacent pixels arranged in a straight line, or one or more lines of pixels which may be obtained or copied from the image.
  • the strip may include, for example, one or more lines of pixels from an image frame.
  • the strip may be selected according to different criteria. For example, a strip may be a "fixed" strip, e.g. a portion of image pixels, the portion being of a fixed size and selected from a predetermined fixed location of the image. A predetermined column of pixels or a predetermined row of pixels may be a fixed strip. For example, line 530 shown in Fig.
  • a strip may be an "adaptive" strip, e.g. selected from a different position or portion in each image.
  • Each strip selected from an image frame may have different length and/or width or may have a fixed size per image.
  • a strip may be selected to run across an image, e.g. from one side of the image to another side.
  • two points may be selected on the circumference of the image and the line connecting the points may be the selected strip.
  • a line may pass through a predetermined point in the image, either at a predetermined (e.g., fixed) angle (which may be measured, for example, between the selected line and a fixed vertical axis in the images) or at a changing angle which may be determined based on image pixel properties, the ends of the strip may correspond to the borders of the image or of a predetermined region of the image which may be used for the motility bar generation
  • a region of interest may be defined for an image stream, as a fixed template or mask which is selected from each image, based on known optical properties and/or illumination properties of the imaging system.
  • lumen size maximization may be a criterion for selecting a certain strip of pixels from an image. For example, the lumen region may appear in different locations in the images, and may not be centered in each image. Different methods for selecting strips may result in different amounts of the lumen hole which is depicted in the selected strip of pixels for each image. According to one embodiment, the strips may be selected to maximize the lumen appearance, and thereby maximize the lumen hole appearance in the generated motility bar.
  • Fig. 6A is a flow chart of a method for generating a motility bar based on an in vivo image stream, according to embodiments of the present invention.
  • images of an image stream captured by an in vivo imaging device may be received, for example in a workstation such as workstation 11 or a receiving unit such as receiver 12.
  • the image stream may include in vivo images captured sequentially over a period of time by an in vivo imaging capsule (e.g. capsule 40), and may be stored in a storage unit such as storage 19 or storage 16.
  • the images may be processed, e.g. by a processing unit (for example processor 14).
  • Each image may be represented as a two- dimensional array of pixels, for example a rectangular or square pixel array of 320 X 320 pixels.
  • Each pixel may contain a value corresponding to a primary color, e.g. red, green or blue.
  • a plurality of images may be selected from the captured image stream, and the selected images may be used for generation of a motility bar.
  • all images of the image stream may be used (e.g., all images may be selected) while other embodiments allow selecting a subset of images from the complete set of images captured by the imaging device.
  • the method may include selecting a subset of images from the image stream according to certain criteria, for example selecting certain segments or portions of the image stream for producing a motility bar, while other segments may not be selected. For example, sequences of images captured in certain organs of the GI tract may be selected. In one example, only images captured in the small bowel may be used for generating the motility bar. Other portions or segments of the image stream may be selected. Images which are blurred, too bright or too dark may not be selected.
  • the selected images may include a plurality of sequential images (e.g. consecutive image frames captured sequentially by the imaging device), in order to maintain smoothness of the generated motility bar and to better emphasize contractile activity patterns or intestinal features in the image stream.
  • a strip or line of pixels may be selected from each image of a plurality of images selected from the image stream (e.g., a subset of images). Different criteria for selecting the strip may be determined (e.g. by a user, or predetermined).
  • the strip may be selected as a fixed cut or an adaptive cut. For example, a linear array of adjacent pixels may be selected.
  • the strip or line of pixels need not be oriented parallel to a row or column of pixels, but may be oriented at another angle to a row or column; e.g., the strip or line may be diagonal.
  • the selected strip of pixels may include, for example, a line of pixels selected from an image.
  • the strip may be a straight line of pixels.
  • the strip may be a fixed strip or cut, e.g. pixels located in the same coordinates in the image pixel array may be selected from each image. For example, if the image include an array of 256 horizontal rows of pixels and 256 vertical columns of pixels, the selected strip may include all pixels in the central column of the pixel array (e.g. column 128). More than one line of pixels may be selected in the strip, for example two or more adjacent colums (e.g. columns 127, 128 and 129 in the example of 256 X 256 pixels).
  • the selected strip may pass through a predetermined point in the image. In one embodiment, the predetermined point may be a central point in the image (e.g. the center of the pixel array). Other predetermined points may be selected.
  • the strip may be an adaptive line of pixels selected from an image.
  • the line may not necessarily pass at the same position in the images selected for producing the motility bar, and instead the line position, orientation or other characteristics may be selected for each image according to certain features or properties of the image.
  • a certain feature or property may be detected in the image (e.g. a dark lumen hole or a blob, a bright portion of the image, etc.).
  • the strip may be selected to pass through the detected feature or portion of the image, for example through a central point or the center of gravity of the detected feature.
  • the adaptive strip may pass through a predetermined point in the image (e.g., a central point or pixel in the pixel array), and different angles may be determined for selecting the strip in each image.
  • a predetermined point in the image e.g., a central point or pixel in the pixel array
  • Other embodiments allow selecting lines that do not pass through a predetermined (e.g. fixed) point in each image.
  • a term that controls the changes between the angles of strips selected from consecutive frames may bedetermined.
  • Other cuts may be selected from images, and in some embodiments a combination of different types of cuts may be used to generate a motility bar.
  • the selected strips or lines may be aligned or positioned, for example vertically, and may be adjoined or positioned adjacently to each other, in order to form a visual representation of motility events which occurred in an image stream, e.g. motility bar or display 2080.
  • the strips or lines, or copies of the strips or lines may be used to form a motility bar.
  • the length (and/or width) of the selected strips may be resized to different lengths, e.g. may be stretched or reduced.
  • the motility bar may (optionally) be created or generated for display, for example on a visual display unit such as monitor 18.
  • the motility bar may be displayed, for example, alongside a window displaying a video view of the image stream, e.g. alongside window 2001 shown in Fig. IB.
  • the motility bar may be analyzed, and sequences of images may be correlated to certain motility events. For example, tunnel sequences may be detected.
  • contractile activity pattern analysis e.g. analysis of different contractile frequency may be performed.
  • the motility bar may be useful for visual validation of different motility properties or events such as contractions, turbid sequences, tunnel, static frames, etc.
  • the motility bar may also be used for analysis of motility properties of the image stream.
  • Motility events may be detected in the motility evets bar, and may be indicated or labeled and displayed to a user. Different motility-related properties may be calculated and summarized, such as pattern, type, rhythm, frequency and/or duration of contractions, average duration of contraction, frequency of contractions in a certain region of the GI tract, etc. Other motility events may be detected and related properties may be calculated and presented or displayed. A range of normal and abnormal values may be presented to the user, for example along with the calculated property, to enable comparison between the normal range and the detected value of the property, and in some embodiments an indication may be provided regarding, for example, abnormal behavior which may have been detected. Other operations or series of operations may be used.
  • Data processor 14 may include, or may be operationally connected to, contractile activity analyzer 51 for automatically detecting contractile activity patterns in in vivo images captured by imaging device 40.
  • Data processor 14 and/or may store instructions for execution or rules for use by software for analyzing series of captured image(s) from storage unit 19 and determining contractile activity patterns.
  • Contractile activity analyzer 51 may use pattern recognition and/or feature extraction to analyze and classify captured image frame sequences and for example, detect contraction sequences, tunnel sequences, turbid sequences/occluded field of view or static closed lumen sequences in series of in vivo image frames.
  • Modules such as display generator 50, contractile activity analyzer 51, lumen region analyzer 52 and other modules or filters may be executed by a processor such as data processor 14.
  • Contractile activity analysis may include calculating properties and/or contractile activity patterns, relating to contractile activity of the GI tract.
  • Contractile activity patterns may include detection of gastrointestinal events and/or classification of image sequences into various types of gastrointestinal motion patterns, e.g. detection of phasic, propulsive, ultrapropulsive or tonic contractions in a series of in vivo images, and/or detection of lack of contractions or substantially no intestinal motion (e.g. sequences of tunnel images with open lumen or static closed lumen sequences).
  • Turbid lumen image sequences may also be a type or category of contractile activity pattern.
  • Contractile activity analyzer 51 may obtain a motility bar (e.g., by generating the motility bar, or by another module generating the motility bar), which is generated for example according to methods as mentioned herein, or received e.g. from a storage medium (e.g. storage 19) which is operationally connected to a processing unit (e.g. processor 14).
  • a motility bar may be generated (e.g. by processor 14 or another processing unit) for the image stream, by calculating pixel-based properties for each frame of at least a selected subset of frames from the image stream.
  • the motility bar includes a plurality of vertical lines os pixels, and may be generated for at least a selected subset of frames from an image stream, and each (e.g., vertical) line represents (or summarizes) pixel-based properties for a selected frame of the image stream being analyzed.
  • a motility bar may be generated, for example, by selecting a straight line or strip of pixels may be selected from the subset of image frames, such that the line passes across a dark region which is detected in the image frame.
  • the lumen is defined as the dark blob or area of an image frame depicting the tunnel ot tube opening of the GI tract captured in an image.
  • the lumen region includes pixels of the image frame which belong to the dark blob or dark region of an image frame which shows a lumen or part of a lumen.
  • the dark region is detected as the lumen area or lumen region in the image, and may be identified, for example, based on pixel intensity values.
  • the selected strips may be aligned or arranged in a spatial arrangement, e.g. adjacently, to form a motility bar.
  • the subset of images may include one or more sequences of consecutive image frames from the image stream.
  • the selected strip may be chosen, for example, to maximize the visibility of the lumen hole in the generated motility bar, or to pass through a fixed point or position in each of the subset of frames.
  • Contractile activity analyzer 51 may filter the motility bar, using one or more filters. For example, Gabor-like filters may be applied to each of the R, G, and B channels of the motility bar RGB image. The filters may be based on, a preset number of frames, or a preset duration of time selected from the motility bar. Various filters may be selected and used, for example based on parameters such as the image capturing rate of the imaging device, common contraction durations or typical contraction frequencies of the GI tract.
  • the filtered representation may be further processed by contractile activity analyzer 51, for example by sorting the values in each vertical line of pixels, normalizing the filters responses into a predetermined range, applying a norm on the filtered representation, summing, or applying other linear or non-linear functions to the filtered representation. Additionally or instead, shallow or weak contractions may be filtered out by by contractile activity analyzer 51, e.g. by applying a predetermined or calcualted threshold to the filtered representation.
  • Contractile activity analyzer 51 may detect local maxima points in the filtered representation, to identify contractile activity patterns.
  • the local maxima points may be detected according to certain conditions, e.g. a minimum height of a detected peak in the filtered representation, and/or a minimal distance between two neighbouring local maxima points.
  • the local maxima points may indicate the presence of a contraction sequence or contractile activity.
  • the local maxima points may be correlated to the motility bar and/or to other displayed information which is related to the image stream, e.g. a time bar or a tissue color bar, for example as disclosed in US Patent Number 7,215,338 to Horn et al.
  • the detected local maxima points may be used by display generator 50 to generate a graphical representation of the deteted contractile activity pattern to a user.
  • Display generator 50 may generate one or more graphs, bars or other representations which visualize parameters such as the quantity, intensity, frequency and/or density of detected contractile activity patterns of the in vivo image stream.
  • the display of two or more bars may be correlated, such that a contractile activity pattern bar, and a time bar for example, and/or a tissue color bar may be positioned adjacent to one another or synchronized with one another.
  • An exemplary display which may be generated (e.g. by display generator 50) based on contractile activity patterns detected by contractile activity analyzer 51 is shown in Fig. 7.
  • Lumen regions in the motility bar may be detected by lumen region analyzer 52 and used for further contractile activity pattern analysis.
  • Lumen region analyzer 52 may identify lumen regions in the motility bar, e.g. bycalculating motility bar intensity values. Noise in the color distribution of the motility bar image may be reduced, for example, the motility bar may be filtered or processed by Lumen region analyzer 52.
  • Lumen region analyzer 52 may transform the image to a gray-scale image or intensity image, and may apply a lumen segment threshold to the intensity values. As a result of the thresholding operation, the image is converted to black and white with black representing the lumen and white representing no lumen regions.
  • Lumen region analyzer 52 may use the identified lumen regions calculate or estimate a lumen region value for each vertical line of the motility bar.
  • the lumen region values may indicate a level or degree of lumen region presence in an image frame represented by the corresponding motility bar line.
  • the lumen region value may range from [0, 100], and may be calculated as the percent of pixels in a line of the motility bar which were identified as pixels belonging to a lumen region.
  • the lumen region value may indicate the number of pixels in the line that were identified as pixels belonging to a lumen region.
  • Lumen region analyzer 52 may calculate and use other values or scores as lumen region values indicating the amount of lumen region pixels identified in the motility bar line.
  • Display generator 50 may use the lumen region values calculated by lumen region analyzer 52 to generate a display comprising a visual representation of lumen region presence in each line of the motility bar.
  • the display may be generated, for example, on a display unit (e.g. monitor 18).
  • Display generator 50 may generate a display of a lumen region bar such as bars 1021 - 1031 of Fig. 10, either separately on a Graphical User Interface or combined with one or more other bars (e.g. a motility bar, a tissue color bar, a time bar, etc.).
  • the display may allow a healthcare professional (a doctor or nurse) to review the contractile activity pattern of the lumen closure and opening, and may assist in diagnosis of gastrointestinal motility abnormalities.
  • Contractile activity patterns detected e.g. by contractile activity analyzer 51 in series of GI images and the result of the contractile activity analysis may be summarized and presented to a user, e.g. using display generator 50.
  • Display generator 50 may generate a visual presentation of the analysis result to a user, e.g. a graph or table showing contraction density over time (calculated, for example, over a 1 -minute sliding window), which may be correlated to a motility bar, tissue bar and/or time bar generated for the captured image stream.
  • Alphanumeric results summarizing the detected contractile activity properties may be calculated and displayed, e.g.
  • Display generator 50 may be a physical device or may be instructions stored in a memory which when executed by a processor, e.g., data processor 14, may perform display functions on monitor 18.
  • display generator 50 may be executed by a separate processor or may be a dedicated processing module.
  • Pixel-based properties or descriptors of an image frame when referred to herein, may include features or functions calculated based on pixel values of the corresponding image frame or a plurality of image frames. For a single image frame, a set of various pixel-based properties may be calculated, and the set may be referred to as a vector of properties. Examples of pixel-based properties include: color value (e.g., the pixels' RGB (red, green and blue) color values, pixel intensity, brightness, saturation, and/or a combination of these properties or other properties which may be extracted from the pixel values in the captured image frame.
  • a series of images from the image stream may be provided to or accessed by contractile activity analyzer 51.
  • the series of images may include, for example, all images captured by the imaging device.
  • a subset of images may be used for contractile activity analysis.
  • the subset of images may be selected, according to different selection criteria, e.g. as disclosed in US Patent Number 7,986,337 to Davidson et al., incorporated by reference herein in its entirety, which discloses editing methods of an in vivo image stream, to create a shortened movie.
  • a subset of images used for contractile activity analysis may include images captured between certain anatomical landmarks which may be identified in the image stream, e.g. the duodenum, the cecal valve, the Z-line (indicating entrance to the stomach), etc.
  • Two anatomical landmarks may be selected (e.g. may be predetermined in the system, or selected by a user) and all images captured during the time the capsule traveled from a selected anatomical landmark which was captured first to the selected anatomical landmark which was captured later, may be included in the contractile activity analyzer.
  • images may be selected according to color parameters, image quality parameters, number of detected pathology candidates in the image, etc.
  • the contractile activity analysis may be performed for selected organs of the GI tract (e.g.
  • images may be merged or fused, e.g. based on similarity between adjacent images, and contractile activity analysis may be performed based on the subset of fused or merged images.
  • Other image selection methods may be used for determining or selecting the subset of images. Different image selection methods may be combined for producing the subset of images which may be used in contractile activity analysis.
  • the contractile activity analyzer 51 may extract or calculate pixel value properties from pixel data of a summarized graphical representation, e.g. a color bar displaying average colors of each image frame in an image stream, or a motility bar displaying motility events based on pixel lines selected from each image frame.
  • a summarized graphical representation e.g. a color bar displaying average colors of each image frame in an image stream, or a motility bar displaying motility events based on pixel lines selected from each image frame.
  • R, G, and/or B pixel color values, pixel brightness values, or other frame -related and/or pixel- related values may be extracted from image data and stored in a storage unit, e.g. storage unit 19 and/or storage unit 16 and/or image database 21.
  • Display generator 50, contractile activity analyzer 51, lumen region analyzer 52 and other modules or processes discussed herein may be, or may be executed by processor 14 or another processor executing software, and thus in some embodiments processor 14 may include display generator 50, contractile activity analyzer 51, lumen region analyzer 52 or other components or modules discussed herein.
  • display generator 50, contractile activity analyzer 51 and/or lumen region analyzer 52 may be software or code executed by processor 14.
  • Other methods may be used; for example display generator 50, contractile activity analyzer 51 and/or lumen region analyzer 52 may be dedicated hardware or circuitry.
  • motility bar representations e . g. motility bar rows
  • contractions are generally grouped, e.g. it is unusual to see one isolated contraction in the motility bar.
  • a group of contractions is referred to herein as a contractile sequence.
  • a synthetic data example of a contractile sequence 200 is presented in Figs. 2A and 2B.
  • Each black dot 205 of a varying size is a synthetic example of a lumen region which may be observed in a single frame of an in vivo image stream.
  • a contraction 210 can be described (in frame view) as open-close - open lumen sequence of frames as shown in Fig. 2A.
  • the separation zone 220 between two contractions may be defined as the sequence of open lumen between two wall closures.
  • the same synthetic data of a contractile sequence shown in Fig. 2A can be represented in a motility bar 240 which is shown in Fig. 2B.
  • a motility bar 240 which is shown in Fig. 2B.
  • either contractions or zones between contractions may be measured.
  • the number of separation zones 220 separating contractions is counted (e.g. instead of counting contractions) and one separation zone 220 is referred to as a valley.
  • FIG. 3 An embodiment of a method for analysis of contractile movements in motility bar is presented in Fig. 3.
  • the input to the method is a digital color image, or another data structure or reference to a stored data structure representing the motility bar, and the output may include a binary signal with value "1" (one) where contractile oscillation is detected, and value "0" (zero) when lack of contractile movement is detected.
  • the following set of steps may be performed, according to one embodiment.
  • Step 301 Detect valleys in the motility bar image to obtain a valley image
  • Step 302 Convert or translate the valley image into a one-dimensional signal representing valley positions.
  • Step 303 Detect local maxima or peaks representing contractions in valleys positions signal.
  • the RGB image 310 is converted to a valley image 320.
  • the first step of the method includes the detection of valleys in the motility bar.
  • a set of filters e.g. Gabor-like filters
  • Gabor-like filters are used in image processing for feature extraction, and texture analysis.
  • the impulse response of these filters is created by multiplying an Gaussian envelope function with a complex oscillation
  • FIG. 4A An exemplary system architechture for applying the filters of Step 301 is shown in Fig. 4A.
  • R (403), G (404) and B (405) channels of image 402 (which corresponds to image 310 in Fig. 3) are separately processed calculating the filter response (406, 407, 408) for each one.
  • the responses are combined using L3 norm (410).
  • L3 norm favors high values in comparison to low values (by cube factor). High values mean high filter responses, and using the L3 norm results in pushing up the high responses and pulling down the low responses.
  • RGB image 402 and a set of filters 401.
  • filters 401 a plurality of filters, e.g. 4 filters, may be used, that correspond to the detection of valleys at different time scales (different contractile velocities or frequencies).
  • filters correspond to second order derivative of a gaussian filter and are defined in the direction of time axis of the motility bar, so that they detect only vertical valleys.
  • All filters may have a fixed height of, for example, 10 pixels and may be normalized to have mean 0 and energy 1.
  • the length of the filter may be variable, for example as shown in Fig. 4C which shows the exemplary filters 471 - 474 which were used in this embodiment.
  • the filters can be characterized by a scale parameter (which indicates the duration of the crest of the contraction).
  • the following example filters may be used, to detect valleys in a motility bar of an image streams (other filters may be used):
  • Fig. 4B shows how the filters are applied to the red channel, as an example of applying the filters to each of the channels.
  • a convolution of the image 402 with the filter is performed (421 - 424).
  • hyperbolic tangent non-linearity (431 - 434), is applied in order to normalize the filters responses into a predetermined range, for example, between [-1 , 1]. Since the detection in this embodiment is performed for valleys (and not for ridges), all negative responses are set to 0 (this step may be referred to as rectified linear unit). In order to boost the high filter responses, the results are elevated to the power of 3.
  • summing (441 - 443) the result on neighbouring scales and joining the results using L3 norm (450) are performed.
  • Fig. 4D shows a plot of the profiles 481 - 484 corresponding to the filters 471 - 474 of Fig. 4C.
  • the valley image 320 is converted or translated into a one-dimensional graph 330 representing the positions of the detected valleys.
  • shallow or weak contractions may be filtered out, for example by applying, a predetermined percentile to the calculated valley image 320 so that a restriction on the oscillation size is imposed.
  • a 75 th percentile may be calculated for each vertical line of the motility bar.
  • Image 330 represents the 75 th percentile of each of the line of valley image 320.
  • the Matlab command prctile may be applied, e.g.
  • the predetermined percentile restriction is used to filter out small oscillations which may not be a result of contractions, but rather may caused by external oscillatory movements of the body e. g. respiratory oscillations or movements caused by heartbeat.
  • the one-dimensional graph 330 is converted or translated to a binary image or signal 340, which represents the detected contractile movements.
  • the local maxima points e.g. peaks in graph 330
  • Various restrictions or conditions may be imposed on the local maxima points, for example, a restriction on the minimum height of the peak (e.g., a value of 20, which corresponds to the height of the peaks in graph 330).
  • Graph 330 (y-axis) is scaled to a range of values between 0 and 100.
  • Another restriction in addition or instead, may be on the distance between two neighbouring local maxima points (used to avoid two detections that may be separated by a short amount of time). For example, a separation value of at least 7 may be applied, in order to avoid detection of more contractions than physiologically possible (the known maximum being 12 contractions per minute). One contraction may produce two local maxima in the graph 330.
  • Image 502 is a portion of a motility bar which was calculated for an in vivo image stream, e.g. according to methods disclosed in Fig. 12 of US Published Patent Application 2015-0016700.
  • Image 504 is the valley image obtained after converting the motility bar image 502 of Fig. 5A (e.g., according to Step 301 above). In this example, valleys of duration ranging between 10 and 28 corresponding frames were detected. While these values conform to the common range of duration of contractions, other durations may be selected instead.
  • the detection of valleys in the motility bar is shown by the white intensity regions in image 504.
  • a sorting step (e.g., according to Step 302 above) may be performed, e.g. by sorting and re-arranging pixels in each vertical line of the valley image 504 according to the pixel intesity values, to obtain a sorted signal, e.g. as shown in signal 506.
  • Graph 508 is the one-dimensional graph representation, obtained after applying percentile 75 to each vertical line of pixels in the sorted signal 506.
  • the binary signal 510 is obtained by detecting local maxima in the one-dimensional graph 508 (e.g. according to Step 303 above).
  • the binary signal 510 which summarizes detected contractile information may be used to estimate the density of contractions.
  • the contraction density may be estimated using, for example, a sliding window of a predetermined length, e.g. a 1 -minute sliding window as shown in density graph 512.
  • Density graph 512 indicates that a maximum value of 12 contractions per minute was detected in the analyzed image stream, and a minimal value of 0 (zero) contractions per minute.
  • a motility bar may be obtained, for example generated according to known methods as mentioned herein, or received e.g. from a storage medium (e.g. storage 19) which is operationally connected to a processing unit (e.g. processor 14).
  • the motility bar may be a line of pixels calculated, for example, based on pixel values or properties, for each frame of at least a selected subset of frames from an in vivo image stream captured by an imaging device (e.g. device 40).
  • the motility bar may be a digital image comprising pixels which are represented by R, G, and B values as known in the art.
  • the motility bar may be filtered, using one or more filters.
  • Gabor-like filters may be applied to each of the R, G, and B channels, e.g. as described in Figs. 4A and 4B.
  • the filters may be based on, e.g., a preset number of frames, or a preset duration of time selected from the motility bar.
  • Various filters may be selected and used, based on parameters such as the image capturing rate of the imaging device, common contraction durations or typical contraction frequencies of the GI tract.
  • the filtered representation may be further processed, for example by sorting the values in each vertical line of pixels, normalizing the filters responses into a predetermined range, applying a norm on the filtered representation, summing, or applying other linear or nonlinear functions to the filtered representaiton.
  • shallow or weak contractions may be filtered out, for example by applying a predetermined threshold to the filtered representation, so that a restriction on the oscillation size or duration (e.g. the minimal number of frames that may be detected as a single contraction sequence) is imposed.
  • the oscillation size refers to the number of frames in a contraction.
  • the threshold may be a percentile of 75. Other threholds may be selected, e.g. in the range between [50, 95].
  • local maxima points may be detected in the filtered representation, to identify contractile activity patterns.
  • One or more restrictions may be imposed on local maxima points, for example a minimum height of the peak, and/or a minimal distance between two neighbouring local maxima points. Other restrictions or conditions may be enforced in order to determine the local maxima points of the filtered representation of the motility bar.
  • the local maxima points may indicate the presence of a contraction sequence (e.g. a series of images which depict a contraction of the GI tract).
  • the local maxima points may be correlated to the motility bar and/or to other summarized and displayed information which is related to the image stream, e.g. a time bar or a tissue color bar, for example as disclosed in US Patent Number 7,215,338 to Horn et al.
  • the detected local maxima points may be used to generate a graphical representation of the contractile activity pattern to a user.
  • a display representing the detected contractile activity patterns may be generated, displayed or provided on a visual display unit, e.g. monitor 18 of Fig. 1.
  • the display may include one or more graphs, bars or other representations which visualize parameters such as the quantity, intensity, frequency and/or density of detected contractile activity patterns of the in vivo image stream.
  • the display of two or more bars may be correlated, such that a contractile activity pattern bar, and a time bar for example, and/or a tissue color bar may be positioned adjacent to one another or synchronized with one another.
  • An exemplary display is shown in Fig. 7.
  • Fig. 7 shows contractile activity pattern bars 721 - 731, generated for a complete image stream, along with the motility bars 701 - 711 which were calculated for the same stream according to embodiments of the present invention.
  • Bars 721-731 are made up of lines, e.g., motility bar line 749.
  • the two types of displayed bars are correlated to each other, meaning that each vertical line of data in the motility bars 701 - 711 corresponds to the vertical line of data in the contractile activity pattern bars 721 - 731 which is positioned exactly below the motility bars, and the displayed data was calculated based on the same image frames of the image stream.
  • the contractile activity pattern data shown in the same line 749 which also crosses contractile activity pattern bar 721 is calculated based on the same one or more image frames of the image stream.
  • regions or zones of the image stream in which there is substantially no contractile activity may be automatically or manually identified, e.g. zones 751 and 752.
  • regions or zones of the in vivo image stream which depict intense contractile activity may also be identified.
  • regions which depict a very low contractile activity or contractile density value (e.g. below a threshold) or very high contractile activity or contractile density value (e.g. above a threshold) may be marked, emphasized or highlighted on the displayed screen, to facilitate diagnosis of motility conditions of the patient being examined.
  • the activity thresholds, or contractile density thresholds may be predetermined, or may be adjusted based on data extracted from the current image stream being analyzed.
  • oscillations or contractile activity with the presence of intestinal content may be identified.
  • mixing contractions are identified in region 754 of the contractile activity pattern bars, where the intestinal content is mixed with variable contractile strength (the contraction density value which was detected in the corresponding portion of contractile activity pattern bar 723, has a high variabililty or divergence).
  • the lumen region is visually characterized as a dark region with low illumination intensity.
  • the detection of the lumen region may be based on the analysis of light intensity in the in vivo images or in a motility bar generated for an image stream.
  • a method for estimation of lumen diameter based on a motility bar may include for example:
  • Step 1 Reduce the noise in the color distribution of the motility bar image, for example by applying mean-shift clustering.
  • Step 2 Convert the noise-reduced image to an intensity image, and apply threshold to obtain the regions of lumen.
  • Step 3 Apply morphological operations on the received intensity image to obtain smooth regions of intestinal lumen.
  • Fig. 8A shows the density plots of colors for a motility bar, before applying mean- shift filtering, according to one embodiment.
  • the size of dots reflects the density of colors in the RGB space.
  • Fig. 8B shows the result after of applying a mean-shift filtering to the RGB feature space of a motility bar image, according to an embodiment of the invention.
  • the colors of in vivo images typically occupy only a small subspace of the RGB space cube.
  • the size of the mean-shift bandwidth which determines the degree of smoothing imposed on the motility bar image, may be set so that the visual perception of the lumen is not modified.
  • smoothing a data set may include generating an approximating function that attempts to capture noticeable patterns in the data, while reducing noise or other small-scale structures/rapid phenomena.
  • smoothing the data points of a signal are modified so individual points (caused by noise) are reduced, and points that are lower than the adjacent points are increased to generate a smoother signal.
  • Mean-shift filtering is one optional smoothing function; other functions may be applied in addition or instead.
  • Fig. 9 shows images which may be generated by applying a lumen diameter estimation method according to embodiments of the invention.
  • Image 900 is an example of a motility bar received for an image stream.
  • Image 902 shows the motility bar image 900 after applying the operation of mean-shift filtering as explained with relation to Figs. 8A and 8B. It can be seen that contours of the lumen depicted in the motility bar image 900 are preserved in mean-shifted image 902.
  • the image 902 is transformed to a gray scale image 904 and a threshold is applied.
  • the gray scale image 904 has values in the range of, for example, [0, 1] and the threshold may be predetermined - for example set to 0.3. Other thresholds may be selected.
  • the gray scale image 904 provides a lumen region segmentation (or detection) of the motility bar image 900.
  • the segmented lumen region in gray scale image 904 is shown in black color, and the detected non-lumen regions are white.
  • morphological operations of opening and closing are applied to gray scale image 904.
  • the lumen segments coincide with their original position in the motility bar image 900.
  • Image 906 is a plot representing an estimation of the percentage that each line of motility bar 900 is occupied by a lumen region.
  • the lumen region segmentation results in gray scale image 904 are converted to a numerical value.
  • the numerical value represents the percentage of each motility bar line (or cut) that is occupied by intestinal lumen. This value is calculated for each vertical line of the motility bar image 900.
  • Graph 906 shows the results for the lumen size estimation, where the maximum value means that 100% of the corresponding vertical line in the motility bar represents a lumen region.
  • Fig. 10 shows a display of a lumen diameter graph for the complete image stream according to one embodiment.
  • Bars 1001 - 1011 are the motility bars calculated for the image stream
  • lumen diameter graphs 1021 - 1031 are the corresponding lumen diameter percentage values calculated based on the motility bars.
  • lumen diameter percentage values may be calculated according to the operations described with relation to Fig. 9 herein.
  • Zones or segments of the image stream with no visible lumen region have a low lumen diameter value (e.g. segment 1050 of lumen diameter graph 1025, corresponding to motility bar 1005, and/or segment 1051 of lumen diameter graph 1028, corresponding to motility bar 1008).
  • Zones or segments of the lumen diameter graphs 1021 - 1031 where the lumen size occupies the whole motility bar show high lumen size value (e.g. segment 1060 of lumen diameter graph 1031, which corresponds to motility bar 1011).
  • the lumen diameter graph values may represent other calculated values based on the detected lumen, for example a relative or normalized size of the detected lumen region in the corresponding vertical line of the motility bar (e.g. the largest value represents the largest lumen region detected in the motility bar, and the lowest value represents the smallest lumen region detected in the motility bar).
  • Other values based on detected lumen diameter, area or perimeter may be calculated according to embodiments of the invention, and presented to a user in a graphic or visual representation or display.
  • Fig. 11 is a flow chart of a method for generating a display representing lumen region presence in frames of an image stream according to an embodiment of the present invention.
  • the method may include receiving an image stream captured by an in vivo device.
  • the image stream may include image frames, each frame containing a plurality of image pixels arranged in an array.
  • a motility bar may be obtained, for example received from a storage unit (e.g. storage 19) operationally connected to a processing unit (e.g. processor 14), or generated by a processing unit.
  • the motility bar comprises a plurality of vertical lines generated for at least a selected subset of frames from an image stream, and each vertical line represents (or summarizes) pixel-based properties for a selected frame of the image stream being analyzed.
  • a motility bar may be generated, for example, by selecting a straight line or strip of pixels may be selected from the subset of image frames, such that the line passes across a dark region which is detected in the image frame.
  • the dark region is detected as the lumen area or lumen region in the image, and may be identified, for example, based on pixel intensity values.
  • the selected strips may be aligned or arranged in a spatial arrangement, e.g. adjacently, to form a motility bar.
  • the subset of images may include one or more sequences of consecutive image frames from the image stream.
  • the selected strip may be chosen, for example, to maximize the visibility of the lumen hole in the generated motility bar, or to pass through a fixed point or position in each of the subset of frames.
  • motility bar intensity values may be calculated, to identify lumen regions in the motility bar.
  • Noise in the color distribution of the motility bar image may be reduced, for example, the motility bar may be filtered or processed, e.g. by applying a mean-shift filter, as explained with relation to Figs. 8A - 8B.
  • Other filters may be applied in addition or instead, for example bilateral filters.
  • mean-shift filter is that it is a non-parametrical feature space analysis method, and neither the size nor the number of clusters are predefined.
  • the image may be transformed to a gray-scale image or intensity image, and a lumen segment threshold may be applied to the intensity values.
  • the lumen segment threshold may be fixed and predetermined, or may be adjusted based on values of the motility bar image.
  • the image is converted to black and white, with black representing the lumen regions and white representing no lumen regions.
  • Small lumen regions which may be generated by the thresholding operations may be removed, for example by applying morphological operations of opening and closing.
  • the identified lumen regions may be used to calculate or estimate a lumen region value for each vertical line of the motility bar.
  • the lumen region values may indicate a level or degree of lumen region presence in an image frame represented by the corresponding motility bar line.
  • the lumen region value may range from [0, 100] , and may be calculated as the percent of pixels in a line which were identified in operation 1120 as pixels belonging to a lumen region.
  • the lumen region value may indicate the number of pixels in the line that were identified as pixels belonging to a lumen region.
  • Other values or scores indicating the amount of lumen region pixels identified in the motility bar line may be calculated and used as lumen region values.
  • a display comprising a visual representation of lumen region presence in each line of the motility bar may be generated on a display unit (e.g. monitor 18).
  • a display of a lumen region bar such as bars 1021 - 1031 of Fig. 10 may be displayed, either separately on a Graphical User Interface or combined with one or more other bars (e.g. a motility bar, a tissue color bar, a time bar, etc.).
  • the display may allow a healthcare professional (a doctor or nurse) to review the contractile activity pattern of the lumen closure and opening, and may assist in diagnosis of gastrointestinal motility abnormalities.
  • Fig. 12 is a schematic illustration of an exemplary Graphic User Interface (GUI) for contractile activity pattern analysis and display of an image stream captured by an in vivo device.
  • the GUI may include a set of editing tools which may be displayed on a monitor, such as the monitor 18 of Fig. 1, according to an embodiment of the present invention.
  • One or more image windows 2001 may display images of an image stream, for example a video view of an image stream, which may be a reduced image stream which contains a selected subset of images, or an original (e.g., as captured by the imaging device 40) image stream.
  • images may be displayed as one or more sets of reduced-size images, e.g. thumbnails or larger images, and not necessarily as a video stream.
  • Controls 2014 may alter the display of the image stream in one or more image windows 2001.
  • Controls 2014 may include for example stop, play, pause, capture image, step, fast-forward, rewind, or other controls, to freeze, speed up, or reverse the image stream in window 2001.
  • the user may operate controls 2014 using an input device (e.g., input device 22 of Fig. 1 such as a keyboard and/or mouse).
  • strips, portions, slices or cuts of images from an image stream may be summarized or represented in a visual representation of motility events which may be detected in an image stream, for example as shown in motility bars or displays 2080 and 2081, and may be displayed to the user.
  • the combined strips or cuts may provide an indication of motility events that occurred during the imaging procedure, and the generation of one or more motility bars or other presentations may simplify or assist analysis of motility events which occurred during an imaging procedure of the patient's GI tract.
  • a cursor 2090 may be positioned on at least one of the motility bars 2080 and 2081, and may indicate to the viewer the correlation between the motility bar and the image currently being displayed in image window 2001.
  • the strip of pixels obtained from the image frame displayed in window 2001 may be indicated by the cursor 2090.
  • the time of capture of the image frame may be indicated by the cursor 2090, and the motility bar may be positioned, for example, alongside one or more time and tissue color bars 2053.
  • the motility bars 2080 and 2081 may be displayed separately, e.g. in a different screen, not alongside image stream window 2001, or upon user request.
  • contractile activity pattern bars 2082 - 2084 may be displayed.
  • the contractile activity pattern bars may be calculated according to different sets of properties - for example, contractile activity pattern bars 2083 - 2084 may be calculated based on contractions density property, e.g. as described with relation to Fig. 3 - Fig. 7 herein, contractile activity pattern bars 2082 may be calculated based on a lumen size or lumen diameter property, described with relation to Fig. 8 - Fig. 11 herein. Contractile activity pattern bars which are calculated based on other properties may be displayed instead or in addition.
  • points of change or transition which were identified, for example, based on the contractile activity analysis described herein, may be visually marked or emphasized on the GUI display.
  • the contractile activity pattern bars 2082 - 2084 are an example of summarized representations of the image stream which may be analyzed according to a motility bar.
  • An edit/create control 2009 may be provided to allow user selection of criteria for generation of contractile activity pattern bars 2082 - 2084, from a list of a plurality of available criteria for selection(e.g., by clicking a tab, check-box, or marker indicating specific criteria). For example, a user may select certain portions or organs of the GI tract which will be used for generating the motility bars, and/or the contractile activity pattern bars.
  • the generation criteria may be modified by a user, e.g. maximizing the lumen hole in the generated bar, and/or maximizing other predefined features.
  • the selection of the pixel strip of a motility bar may also be influenced by user input, e.g. by enabling selection of parameters which determine the generated motility bar, e.g. selecting a fixed cut view and/or adaptive cut views.
  • more than one image stream may be displayed concurrently on the monitor, for example as disclosed in Figs. 9A, 9B and 10A, 10B of U.S. Patent No. 7,474,327 to Davidson et al., assigned to the common assignee of the present application and incorporated herein by reference in its entirety.
  • the imaging device includes more than one imaging system
  • one or more image streams obtained by an imaging system may be used to generate the motility display.
  • data used in a motility bar may be summarized from one or more image streams of a single imaging procedure.
  • Timeline or time bar 2010 may provide a timeline or time chart of the image stream, for example by displaying a line indicating the capture time of images, starting from the beginning of the imaging procedure.
  • a cursor 2052 may be positioned over time bar 2010 and may indicate capture time of an image being displayed, for example, currently in window 2001.
  • tissue color bar 2051/2053 may be generated, for example, according to embodiments described in US Published Application No. 2010-0053313 to Horn et al. incorporated by reference herein in its entirety. Tissue color bar 2051/2053 may overlap with a time bar, or may be presented separately. In some embodiments, tissue color bar 2051/2053 may be formed of a number of strips or elements aligned adjacent to each other, or assembled in a continuing bar. Each strip or element in tissue color bar 2051/2053 may represent summarized information, for example, a mean or average color value or intensity, of image frames captured during a predetermined time duration (e.g., a one-minute period). In some embodiments, each strip in the tissue color bar 2051/2053 may correspond to one image frame, e.g. may summarize an average color, intensity, pH level, etc. of a displayed image frame.
  • Thumbnail images 2054, 2056, 2058 and 2060 may be displayed with reference to an appropriate relative time on the time chart 2010.
  • Related annotations or summaries 2055, 2057, 2059 and 2061 may include the image capture time of the image corresponding to each thumbnail image, and summary information associated with the respective thumbnail image.
  • Time indicator 2050 may provide a representation of the absolute time elapsed for or associated with the current image being shown in image windows 2001, the total length of the edited image stream and/or the original unedited image stream.
  • Absolute time elapsed for the current image being shown may be, for example, the amount of time that elapsed between the moment the imaging device (e.g., capsule 40 of Fig.
  • an image receiver e.g., image receiver 12 of Fig. 1
  • an image receiver e.g., image receiver 12 of Fig. 1
  • One or more monitors or image windows 2001 may be used to display the image stream and other data.
  • motility bars 2080 - 2081 and/or contractile activity pattern bars 2082 - 2084 may be displayed as a bar at a predetermined location on the display screen, for example aligned with (or alongside) tissue color bar 2051, and/or aligned with (or alongside) tissue/time bar 2053.
  • a single cursor indicating the current image being displayed e.g., indicating the time of capturing the image, the corresponding pixel strip in the motility bar, and the contractile activity pattern values and/or lumen region values
  • contractile activity pattern information of the image stream may be displayed in a separate window or screen, e.g. not alongside the image stream window.
  • the contractile activity pattern bars 2082 - 2084 may be displayed upon user demand, for example in a pop-up window which may be presented.
  • Capsule position window 2070 may include a current position and/or orientation of the imaging device in the gastrointestinal tract of the patient, and may display different segments of the GI tract using different colors.
  • the capusle position may be an approximated or estimated position of the capsule inside the GI tract.
  • a highlighted segment may indicate the position of the imaging device when the currently displayed image (or plurality of images) was captured.
  • a bar or chart in window 2070 may indicate the total path length travelled by the imaging device, and may provide an estimation or calculation of the percentage of the path travelled at the time the presently displayed image was captured.
  • Buttons 2040 and 2042 may allow the viewer to select between a manual viewing mode, for example an unedited image stream, and an automatically edited viewing mode, in which the user may view only a subset of images from the stream edited according to predetermined criteria.
  • View buttons 2044 allow the viewer to select between viewing the image stream in a single window, or viewing multiple image streams in double, quadruple, or mosaic view mode.
  • the display buttons 2048 may display to the viewer images from the original stream, or only selected images with suspected bleeding indications.
  • Viewing speed bar 2012 may be adjusted by the user.
  • the slider may indicate the number of displayed frames per second.
  • Buttons 2016, 2018, 2020, 2022, 2024,and 2026 may allow a user to capture landmark images or thumbnail images, input a manual score or comment for an image, generate a report for the viewed image stream, and save the clinical findings and markings of the viewer.
  • Control 2028 may allow a user to access a dictionary or atlas of sample images, e.g. of pathologies or anatomical landmarks in the GI tract.
  • Control 2030 may allow resizing the image on display in window 2001 (e.g. zooming in or zooming out).
  • Embodiments of the invention may include an article such as a computer or processor readable non-transitory storage medium, such as for example a memory, a disk drive, or a USB flash memory encoding, including or storing instructions, e.g., computer-executable instructions, which when executed by a processor or controller, cause the processor or controller to carry out methods disclosed herein.
  • a computer processor or computer controller e.g., data processor 14, may be configured to carry out embodiments of the invention, for example by executing software or code stored in a memory connected to the processor, and/or by having dedicated circuitry.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Surgery (AREA)
  • Radiology & Medical Imaging (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Signal Processing (AREA)
  • Biomedical Technology (AREA)
  • Optics & Photonics (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Animal Behavior & Ethology (AREA)
  • Pathology (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Quality & Reliability (AREA)
  • Endoscopes (AREA)

Abstract

L'invention concerne un système et un procédé pour détecter et afficher des motifs d'activité contractile d'un tractus gastro-intestinal. Un flux d'image, capturé par un dispositif in vivo, peut être analysé automatiquement, le flux d'image comprenant des trames d'image, chaque trame comprenant une pluralité de pixels. Une unité de traitement peut être configurée pour obtenir une barre de motilité comprenant des lignes, la barre de motilité étant calculée sur la base d'au moins un sous-ensemble de trames sélectionnées du flux d'image, chaque ligne représentant des propriétés basées sur des pixels d'une trame d'image. La barre de motilité peut être filtrée par l'unité de traitement, qui est en outre configurée pour détecter des motifs d'activité contractile dans la représentation filtrée de la barre de motilité par détection de points maximaux locaux. Un affichage, représentant les motifs d'activité contractile détectés, peut être généré sur une unité d'affichage visuel.
PCT/IL2015/050465 2014-05-09 2015-05-04 Système et procédé d'analyse de motilité de flux d'image in vivo WO2015170321A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201461990990P 2014-05-09 2014-05-09
US61/990,990 2014-05-09
US201462031318P 2014-07-31 2014-07-31
US62/031,318 2014-07-31

Publications (1)

Publication Number Publication Date
WO2015170321A1 true WO2015170321A1 (fr) 2015-11-12

Family

ID=54392226

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IL2015/050465 WO2015170321A1 (fr) 2014-05-09 2015-05-04 Système et procédé d'analyse de motilité de flux d'image in vivo

Country Status (1)

Country Link
WO (1) WO2015170321A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7805178B1 (en) * 2005-07-25 2010-09-28 Given Imaging Ltd. Device, system and method of receiving and recording and displaying in-vivo data with user entered data
US20110044515A1 (en) * 2006-03-13 2011-02-24 Panagiota Spyridonos Device, system and method for automatic detection of contractile activity in an image frame
US20120057766A1 (en) * 2010-09-08 2012-03-08 Drozdzal Michal System and method for automatic detection of in vivo contraction video sequences
WO2013114361A2 (fr) * 2012-01-31 2013-08-08 Given Imaging Ltd. Système et procédé pour afficher des éléments de motilité dans un flux d'images in vivo

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7805178B1 (en) * 2005-07-25 2010-09-28 Given Imaging Ltd. Device, system and method of receiving and recording and displaying in-vivo data with user entered data
US20110044515A1 (en) * 2006-03-13 2011-02-24 Panagiota Spyridonos Device, system and method for automatic detection of contractile activity in an image frame
US20120057766A1 (en) * 2010-09-08 2012-03-08 Drozdzal Michal System and method for automatic detection of in vivo contraction video sequences
WO2013114361A2 (fr) * 2012-01-31 2013-08-08 Given Imaging Ltd. Système et procédé pour afficher des éléments de motilité dans un flux d'images in vivo

Similar Documents

Publication Publication Date Title
EP2810216B1 (fr) Système et procédé pour afficher des éléments de motilité dans un flux d'images in vivo
US12059125B2 (en) Systems and methods for generating and displaying a study of a stream of in-vivo images
EP2868100B1 (fr) Système et procédé pour afficher un flux d'image
US10204411B2 (en) System and method for sequential image analysis of an in vivo image stream
US9060673B2 (en) System and method for displaying portions of in-vivo images
EP1997076B1 (fr) Analyse en cascade pour detecter une contraction intestinale
US8005279B2 (en) Capsule endoscope image display controller
US20060164511A1 (en) System and method for displaying an image stream
JP2015509026A5 (fr)
US20120113239A1 (en) System and method for displaying an image stream
US9342881B1 (en) System and method for automatic detection of in vivo polyps in video sequences
Vilarino et al. Intestinal motility assessment with video capsule endoscopy: automatic annotation of phasic intestinal contractions
EP4093025A1 (fr) Système de traitement d'image médicale
CN104640496A (zh) 医疗装置
WO2015170321A1 (fr) Système et procédé d'analyse de motilité de flux d'image in vivo
Azpiroz et al. Adaptable image cuts for motility inspection using WCE

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15789245

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 15789245

Country of ref document: EP

Kind code of ref document: A1