[go: up one dir, main page]

WO2023083848A1 - Self learning grasp sequence for robot bin picking - Google Patents

Self learning grasp sequence for robot bin picking Download PDF

Info

Publication number
WO2023083848A1
WO2023083848A1 PCT/EP2022/081227 EP2022081227W WO2023083848A1 WO 2023083848 A1 WO2023083848 A1 WO 2023083848A1 EP 2022081227 W EP2022081227 W EP 2022081227W WO 2023083848 A1 WO2023083848 A1 WO 2023083848A1
Authority
WO
WIPO (PCT)
Prior art keywords
reflection
robot
pick point
beam profile
pick
Prior art date
Application number
PCT/EP2022/081227
Other languages
French (fr)
Inventor
Jakob Unger
Siddhant Chandrakant MEHTA
Sven Wanner
Stefan ZEISS
Maria Klodt
Original Assignee
Trinamix Gmbh
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Trinamix Gmbh filed Critical Trinamix Gmbh
Publication of WO2023083848A1 publication Critical patent/WO2023083848A1/en

Links

Classifications

    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • B25J9/1628Programme controls characterised by the control loop
    • B25J9/163Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • B25J9/1656Programme controls characterised by programming, planning systems for manipulators
    • B25J9/1669Programme controls characterised by programming, planning systems for manipulators characterised by special application, e.g. multi-arm co-operation, assembly, grasping
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/37Measurements
    • G05B2219/37571Camera detecting reflected light from laser
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/39Robotics, robotics to robotics hand
    • G05B2219/39536Planning of hand motion, grasping
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/40Robotics, robotics mapping to robotics vision
    • G05B2219/40317For collision avoidance and detection
    • GPHYSICS
    • G05CONTROLLING; REGULATING
    • G05BCONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
    • G05B2219/00Program-control systems
    • G05B2219/30Nc systems
    • G05B2219/40Robotics, robotics mapping to robotics vision
    • G05B2219/40565Detect features of object, not position or orientation

Definitions

  • the invention relates to a computer implemented method for picking of items comprised in a carrier by using at least one robot and a pick system.
  • the devices, methods and uses according to the present invention specifically may be employed for example in various areas of daily life, security technology, production technology, safety technology, agriculture, maintenance, cos- metics, medical technology or in the sciences.
  • Prior art Robots with grippers can lift and transport a broad variety of objects to automate commissioning in warehouses or manufacturing processes in industry. Therefore, robots are equipped with 3D vision sensors in order to detect objects and/or suitable pick points where the gripper can grasp objects.
  • the process may allow adapting networks to almost any scenario, even the orientation of the object can be derived from the network.
  • network training is performed manually.
  • the simula- tion needs to be carefully configured from the customer.
  • the network is adapted to a single setting (e.g. camera hardware, distance camera to scene, background). Any deviation from the training settings will lead to false detections or undetected objects.
  • Deep reinforcement learning is one of the most promising directions to achieve intelligent robotic behavior. Pioneers in the field of reinforcement learning for robotics are Google and UC Berkeley, see e.g. Kalashnikov D, Irpan A, Pastor P, et al., QT-Opt: Scalable Deep Reinforcement Learning for Vi-sion-Based Robotic Manipulation.
  • these terms may both refer to a situa- tion in which, besides the feature introduced by these terms, no further features are present in the entity described in this context and to a situation in which one or more further features are present.
  • the expressions “A has B”, “A comprises B” and “A includes B” may both refer to a situation in which, besides B, no other element is present in A (i.e. a situation in which A solely and exclusively consists of B) and to a situation in which, besides B, one or more further elements are present in entity A, such as element C, elements C and D or even further elements.
  • the terms “at least one”, “one or more” or similar expressions indi- cating that a feature or element may be present once or more than once typically will be used only once when introducing the respective feature or element.
  • the expressions “at least one” or “one or more” will not be repeated, non-withstanding the fact that the respective feature or element may be present once or more than once.
  • the terms “preferably”, “more preferably”, “particularly”, “more particularly”, “specifically”, “more specifically” or similar terms are used in conjunction with op- tional features, without restricting alternative possibilities.
  • the term "computer implemented method" as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to a method involving at least one computer and/or at least one computer network.
  • the computer and/or computer network may comprise at least one processor which is con-figured for performing at least one of the method steps of the method according to the present invention. Specifically, each of the method steps is performed by the computer and/or computer network. The method may be performed completely automatically, specifically without user interaction.
  • carrier as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to a device configured for contain- ing at least one item, in particular a plurality of items.
  • the carrier may have support surface on which the items can be placed.
  • the carrier may have sidewalls.
  • the carrier may be a box.
  • template as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the items may be made from at least one material selected from the group consisting of card- board, glass, metal or plastic.
  • the items may comprise screws, bottles, food and the like.
  • the carrier may comprise items of identical size, shape, material and reflection and/or absorption properties.
  • the carrier may comprise items having varying size, shape, material and reflection and/or absorption properties.
  • robot as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically specifically may refer, without limitation, to an arbitrary device configured for performing at least one task autonomously.
  • the robot may be configured for performing the task in the absence of external assistance or instructions, such as for example without assistance from a user or control by a user.
  • the robot may be configured to react to specific situations independent from external input, e.g. by using pre-programmed routines, self-learning mecha- nisms or the like.
  • the robot may be selected from the group consisting of a commercial robot; an industrial robot, specifically a manufacturing robot.
  • the robot may comprise at least one robot arm equipped with at least one end effector at one end of the robot arm.
  • the robot arm may be configured for moving the end effector along three axis of movement.
  • the robot may be configured for performing at least one gripping function.
  • gripping and grasping may be used as synonyms in the following.
  • the gripping function may comprise one or more of clamping, holding, tilting, lifting, positioning, moving, handling, or trans- porting the item.
  • the gripping may comprise picking the item.
  • picking as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to one or more of selecting the item, interacting with a surface of the item such as contacting, or lifting the item.
  • the gripping may comprise lifting and transporting items to automate commissioning in a warehouses and/or in a manufac- turing process.
  • pick point as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to point of interaction between the end effector and the item.
  • the pick point may be a point on the surface of the item.
  • the end effector may comprise at least one gripper.
  • the gripper may be a vacuum grabber.
  • the vacuum grabber may be configured for applying an attractive force to a surface of an item to be gripped.
  • the vacuum grabber may comprise at least one vacuum cup, also denoted as suction cup, such as a rubber suction cup or polyurethane suction cup to pick up items.
  • the vacuum cups may have a round shape.
  • the vacuum cup may be configured for interacting with the surface of the item.
  • the vacuum grabber may comprise at least one vacuum pump configured for generating vacuum.
  • the vacuum grabber may comprise at least one pneumatic valve config- ured for sucking air out of the vacuum cup for attaching the item to the vacuum cup, thereby picking up the item.
  • the robot arm may be configured for moving the vacuum grabber with at- tached item to a further position, in particular outside the carrier. Other embodiments of a robot are possible.
  • the method steps may be performed in the given order or may be performed in a different or- der. Further, one or more additional method steps may be present which are not listed. Further, one, more than one or even all of the method steps may be performed repeatedly.
  • the method may comprise using the display device ac- cording to the present invention, such as according to one or more of the embodiments given above or given in further detail below.
  • the method comprises the following steps: a) at least one carrier scanning step comprising imaging the carrier at a plurality of imag- ing positions by using at least one camera; b) at least one material detection step comprising b1) projecting at least one illumination pattern on a scene comprising the carrier by using at least one projector and imaging at least one reflection image using the cam- era, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) determining material characteristics for each reflection feature by evaluating the reflection image by using at least one processing unit, wherein the evaluation com- prises, for each reflection feature, an analysis of its respective beam profile; c) at least one pick point scoring step, wherein the pick point scoring step comprises the processing unit assigning a score according to their probability to lead to a successful grasp to pick point candidates using a trained scoring model, wherein the determined material characteristics in a close environment around the pick point candidate are used as input for the trained scoring model; d)
  • the term “scanning” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to a process of imaging at a plurality of imaging positions.
  • the imaging positions may refer to different positions of the camera in space.
  • the scanning may be performed to account for trajectory planning and collision avoidance for the subsequent picking.
  • the imaging positions may be set by moving the robot arm.
  • the robot arm may be equipped with the camera, the projector and an optional flood light source.
  • Step a) may comprise illuminating the carrier, e.g. by using at least one flood light source.
  • flood light source is a broad term and is to be given its ordinary and cus- tomary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to at least one arbitrary device adapted to provide the at least one illumination light beam for illumination of the carrier.
  • the flood light source may be configured for scene illumination.
  • scene illumination may refer to diffuse and/or uniform illumination of the scene.
  • the flood light source may be adapted to directly or indirectly illuminating the carrier, wherein the illumination is reflected or scattered by surfaces of the carrier and, thereby, is at least partially directed towards the cam- era.
  • the flood light source may be adapted to illuminate the carrier, for example, by directing a light beam towards the carrier, which reflects the light beam.
  • the flood light source may be con- figured for generating an illuminating light beam for illuminating the carrier.
  • the flood light source may comprise at least one light-emitting-diode (LED).
  • the flood light source may illuminate the scene with the LED and, in particular, without the illumination pattern, and the camera may be configured for capturing a two-dimensional image of the scene.
  • the flood light source may comprise a single light source or a plurality of light sources.
  • the light emitted by the flood light source may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm.
  • light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 ⁇ m. Specifically, the light in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm may be used.
  • the flood light source may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodiments, the flood light source may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths channels.
  • the projector and flood light source may constitute a coordinate system, wherein a longitudinal coordinate is a coordinate along the optical axis.
  • the coordinate system may be a polar coordi- nate system in which the optical axis forms a z-axis and in which a distance from the z-axis and a polar angle may be used as additional coordinates.
  • a direction parallel or antiparallel to the z- axis may be considered a longitudinal direction, and a coordinate along the z-axis may be con- sidered a longitudinal coordinate z.
  • Any direction perpendicular to the z-axis may be considered a transversal direction, and the polar coordinate and/or the polar angle may be considered a transversal coordinate.
  • the term “depth information” may relate to the longitudi- nal coordinate and/or information from which the longitudinal coordinate can be derived.
  • the term “projector” as used herein, is a broad term and is to be given its ordinary and custom- ary meaning to a person of ordinary skill in the art and is not to be limited to a special or cus- tomized meaning.
  • the term specifically may refer, without limitation, to at least one illumination device configured for providing the at least one illumination pattern.
  • the projector may be or may comprise at least one light source or at least one multiple beam light source.
  • the projector may comprise at least one laser source and one or more diffractive optical elements (DOEs).
  • DOEs diffractive optical elements
  • the projector may comprise at least one laser and/or laser source.
  • lasers such as semiconductor la- sers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed bragg reflector lasers, polariton lasers, hybrid sili- con lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, In- dium Arsenide lasers, transistor lasers, diode pumped lasers, distributed feedback lasers, quan- tum well lasers, interband cascade lasers, Gallium Arsenide lasers, semiconductor ring laser, extended cavity diode lasers, or vertical cavity surface-emitting lasers.
  • semiconductor la- sers such as semiconductor la- sers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed bragg reflector lasers, polariton lasers, hybrid sili- con lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, In- dium Ars
  • non-laser light sources may be used, such as LEDs, micro-light emitting diode (LED), and/or light bulbs.
  • the projector may comprise one or more diffractive optical elements (DOEs) configured for generating the illumination pattern.
  • DOEs diffractive optical elements
  • the projector may be adapted to generate and/or to project a cloud of points, for example the projector source may comprise one or more of at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light emitting diodes; at least one array of laser light sources.
  • the use of at least one laser source is partic- ularly preferred.
  • the projector and the flood light source may be integrated into a housing. Further, the projector may be configured for emitting modulated or non-modulated light. In case a plurality of light sources is used, the different light sources may have different modulation fre- quencies which later on may be used for distinguishing the light beams.
  • the light beam or light beams generated by the projector generally may propagate parallel to the optical axis or tilted with respect to the optical axis, e.g. including an angle with the optical axis.
  • the projector may be configured such that the light beam or light beams propagates from the projector towards the scene along an optical axis.
  • the projector may com- prise at least one reflective element, preferably at least one prism, for deflecting the illuminating light beam onto the optical axis.
  • the light beam or light beams such as the laser light beam
  • the optical axis may include an angle of less than 10°, preferably less than 5° or even less than 2°. Other embodiments, however, are feasible. Further, the light beam or light beams may be on the optical axis or off the optical axis.
  • the light beam or light beams may be parallel to the optical axis having a distance of less 10 than 10 mm to the optical axis, preferably less than 5 mm to the optical axis or even less than 1 mm to the optical axis or may even coincide with the optical axis.
  • the term “at least one illumination pattern” refers to at least one arbitrary pattern comprising at least one illumination feature adapted to illuminate at least one part of the scene.
  • the term “illumination feature” refers to at least one at least partially extended feature of the pattern.
  • the illumination pattern may comprise a single illumination feature.
  • the illumination pattern may comprise a plurality of illumination features.
  • the illumination pattern may be selected from the group consisting of: at least one point pattern; at least one line pat- tern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern compris- ing an arrangement of periodic or non periodic features.
  • the illumination pattern may comprise regular and/or constant and/or periodic pattern such as a triangular pattern, a rectangular pat- tern, a hexagonal pattern or a pattern comprising further convex tilings.
  • the illumination pattern may exhibit the at least one illumination feature selected from the group consisting of: at least one point; at least one line; at least two lines such as parallel or crossing lines; at least one point and one line; at least one arrangement of periodic or non-periodic feature; at least one ar- bitrary shaped featured.
  • the illumination pattern may comprise at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pat- tern; a random point pattern or a quasi random pattern; at least one Sobol pattern; at least one quasiperiodic pattern; at least one pattern comprising at least one pre-known feature at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern at least one pattern comprising convex uniform tilings; at least one line pat- tern comprising at least one line; at least one line pattern comprising at least two lines such as parallel or crossing lines.
  • a distance between two features, in particular spots, of the illumination pattern and/or an area of the at least one illumination feature may depend on the circle of confu- sion in the reflection image.
  • the projector may be adapted to generate and/or to project a cloud of points.
  • the projector may comprise the at least one light projector adapted to generate a cloud of points such that the illumination pattern may comprise a plurality of point pattern.
  • the projector may comprise at least one mask adapted to generate the illumination pattern from at least one light beam generated by the light source.
  • the projector comprises at least one laser light source, wherein the illumination pattern comprises a grid of laser spots.
  • the projector comprises at least one laser source which is designated for generating laser radiation.
  • the projector may comprise the at least one diffractive optical element (DOE).
  • DOE diffractive optical element
  • the projector may be at least one point projector, such as the at least one laser source and the DOE, adapted to project at least one periodic point pattern.
  • projecting at least one illumination pattern refers to providing the at least one illumination pattern for illuminating the at least one scene.
  • ray generally refers to a line that is perpendicular to wavefronts of light which points in a direction of energy flow.
  • beam generally refers to a collection of rays. In the following, the terms “ray” and “beam” will be used as synonyms.
  • the term “light beam” generally refers to an amount of light, specifically an amount of light traveling essentially in the same direction, including the possibility of the light beam having a spreading angle or widening angle.
  • the light beam may have a spatial exten- sion.
  • the light beam may have a non-Gaussian beam profile.
  • the beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile.
  • the trapezoid beam profile may have a plateau region and at least one edge region.
  • the light beam specifically may be a Gaussian light beam or a linear combination of Gaussian light beams, as will be outlined in further detail below. Other embodiments are fea- sible, however.
  • the light emitted by the projector may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 ⁇ m.
  • the laser spots may have wavelengths in a near infrared (NIR) regime. Specifically, the light in the part of the near infrared region where silicon photodi- odes are applicable specifically in the range of 700 nm to 1100 nm may be used.
  • NIR near infrared
  • the projector may be configured for emitting light beams at a wavelength range from 800 to 1000 nm, preferably at 940 nm, since terrestrial sun radiation has a local minimum in irradiance at this wavelength, e.g. as described in CIE 085- 1989 compromiseSolar spectral Irradiance”.
  • the projector may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region.
  • the projector may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths chan- nels.
  • the term “scene” as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to may refer to at least one arbitrary object or spatial region.
  • the scene may comprise the carrier and a surrounding environment.
  • the term "camera” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to a device having at least one im- aging element configured for recording or capturing spatially resolved one-dimensional, two-di- mensional or even three-dimensional optical data or information.
  • the camera may comprise at least one pixelated camera chip.
  • the camera may comprise at least one camera chip, such as at least one CCD chip and/or at least one CMOS chip configured for recording im- ages.
  • image specifically may relate to data rec- orded by using a camera, such as a plurality of electronic readings from the camera, such as the pixels of the camera chip.
  • the camera may comprise at least one optical sensor, in particular a plurality of optical sensors.
  • the optical sensor has at least one light sensitive area.
  • an “optical sensor” gen- erally refers to a light-sensitive device for detecting a light beam, such as for detecting an illumi- nation and/or a light spot generated by at least one light beam.
  • a “light- sensitive area” generally refers to an area of the optical sensor which may be illuminated exter- nally, by the at least one light beam, in response to which illumination at least one sensor signal is generated.
  • the light-sensitive area may specifically be located on a surface of the respective optical sensor. Other embodiments, however, are feasible.
  • the camera may comprise a plural- ity of optical sensors each having a light sensitive area.
  • the term “the optical sensors each having at least one light sensitive area” refers to configurations with a plurality of single optical sensors each having one light sensitive area and to configurations with one com- bined optical sensor having a plurality of light sensitive areas.
  • the term “optical sensor” further- more refers to a light-sensitive device configured to generate one output signal.
  • each optical sensor may be embodied such that precisely one light-sensitive area is present in the respective optical sensor, such as by providing precisely one light-sensitive area which may be illuminated, in response to which illu- mination precisely one uniform sensor signal is created for the whole optical sensor.
  • each optical sensor may be a single area optical sensor.
  • the use of the single area optical sensors renders the setup of the display device specifically simple and efficient.
  • commercially available photo-sensors such as commercially available silicon photodi- odes, each having precisely one sensitive area, may be used in the set-up.
  • the light sensitive area may be oriented essentially perpendicular to an optical axis.
  • the optical axis may be a straight optical axis or may be bent or even split, such as by using one or more deflection elements and/or by using one or more beam splitters, wherein the es- sentially perpendicular orientation, in the latter cases, may refer to the local optical axis in the respective branch or beam path of the optical setup.
  • the optical sensor specifically may be or may comprise at least one photodetector, preferably inorganic photodetectors, more preferably inorganic semiconductor photodetectors, most prefer- ably silicon photodetectors.
  • the optical sensor may be sensitive in the infrared spectral range. All pixels of the matrix or at least a group of the optical sensors of the matrix specifically may be identical. Groups of identical pixels of the matrix specifically may be pro- vided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Further, the pixels may be identical in size and/or with regard to their electronic or optoelec- tronic properties.
  • the optical sensor may be or may comprise at least one inorganic photodiode which are sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers.
  • the optical sensor may be sensitive in the part of the near infra- red region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm.
  • Infrared optical sensors which may be used for optical sensors may be commercially availa- ble infrared optical sensors, such as infrared optical sensors commercially available under the brand name HertzstueckTM from trinamiXTM GmbH, D-67056 Ludwigshafen am Rhein, Ger- many.
  • the optical sensor may comprise at least one optical sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodi- ode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode.
  • a Ge photodiode an InGaAs photodiode, an extended InGaAs photodi- ode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode.
  • the optical sensor may comprise at least one optical sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, a Si:As photodiode.
  • the optical sensor may com- prise at least one photoconductive sensor such as a PbS or PbSe sensor, a bolometer, prefera- bly a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bo- lometer.
  • the optical sensor may be sensitive in one or more of the ultraviolet, the visible or the infrared spectral range. Specifically, the optical sensor may be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably at 650 nm to 750 nm or at 690 nm to 700 nm. Specifi- cally, the optical sensor may be sensitive in the near infrared region.
  • the optical sensor may be sensitive in the part of the near infrared region where silicon photodiodes are ap- plicable specifically in the range of 700 nm to 1000 nm.
  • the optical sensor specifically, may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 micrometers.
  • the optical sensor each, independently, may be or may comprise at least one ele- ment selected from the group consisting of a photodiode, a photocell, a photoconductor, a pho- totransistor or any combination thereof.
  • the camera may be or may comprise at least one element selected from the group consisting of a CCD sensor element, a CMOS sen- sor element, a photodiode, a photocell, a photoconductor, a phototransistor or any combination thereof. Any other type of photosensitive element may be used.
  • the photosensitive element generally may fully or partially be made of inorganic materials and/or may fully or partially be made of organic materials. Most commonly, one or more photodiodes may be used, such as commercially available photodiodes, e.g. inorganic semiconductor photodiodes.
  • the camera may comprise at least one sensor element comprising a matrix of pixels.
  • the optical sensor may be part of or constitute a pixelated optical device.
  • the camera may be and/or may comprise at least one CCD and/or CMOS device.
  • the optical sensor may be part of or constitute at least one CCD and/or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area.
  • the term “sensor element” generally refers to a device or a combination of a plurality of devices configured for sensing at least one parameter.
  • the pa- rameter specifically may be an optical parameter
  • the sensor element specifically may be an optical sensor element.
  • the sensor element may be formed as a unitary, single device or as a combination of several devices.
  • the sensor element comprises a matrix of optical sensors.
  • the sensor element may comprise at least one CMOS sensor.
  • the matrix may be composed of independent pixels such as of independent optical sensors.
  • a matrix of inorganic photodi- odes may be composed.
  • a commercially available matrix may be used, such as one or more of a CCD detector, such as a CCD detector chip, and/or a CMOS detector, such as a CMOS detector chip.
  • the sensor element may be and/or may com- prise at least one CCD and/or CMOS device and/or the optical sensors may form a sensor array or may be part of a sensor array, such as the above-mentioned matrix.
  • the sensor element may comprise an array of pixels, such as a rectangular array, having m rows and n columns, with m, n, independently, being positive integers.
  • more than one column and more than one row is given, i.e. n>1, m>1.
  • n may be 2 to 16 or higher and m may be 2 to 16 or higher.
  • the ratio of the number of rows and the number of columns is close to 1.
  • the matrix may be composed of independent pixels such as of independent optical sensors.
  • a matrix of inorganic photodiodes may be composed.
  • a commer- cially available matrix may be used, such as one or more of a CCD detector, such as a CCD de- tector chip, and/or a CMOS detector, such as a CMOS detector chip.
  • the opti- cal sensor may be and/or may comprise at least one CCD and/or CMOS device and/or the opti- cal sensors of the display device may form a sensor array or may be part of a sensor array, such as the above-mentioned matrix.
  • the matrix specifically may be a rectangular matrix having at least one row, preferably a plural- ity of rows, and a plurality of columns.
  • the rows and columns may be oriented essentially perpendicular.
  • the term “essentially perpendicular” refers to the con- dition of a perpendicular orientation, with a tolerance of e.g.
  • the term “essentially paral- lel” refers to the condition of a parallel orientation, with a tolerance of e.g. ⁇ 20° or less, prefera- bly a tolerance of ⁇ 10° or less, more preferably a tolerance of ⁇ 5° or less.
  • tolerances of less than 20°, specifically less than 10° or even less than 5° may be acceptable.
  • the matrix specifically may have at least 10 rows, pref- erably at least 500 rows, more preferably at least 1000 rows.
  • the matrix may have at least 10 columns, preferably at least 500 columns, more preferably at least 1000 columns.
  • the matrix may comprise at least 50 optical sensors, preferably at least 100000 optical sensors, more preferably at least 5000000 optical sensors.
  • the matrix may comprise a number of pixels in a multi-mega pixel range. Other embodiments, however, are feasible. Thus, in setups in which an axial rotational symmetry is to be expected, circular arrangements or concentric ar- rangements of the optical sensors of the matrix, which may also be referred to as pixels, may be preferred.
  • the sensor element may be part of or constitute a pixelated camera.
  • the sensor element may be and/or may comprise at least one CCD and/or CMOS de- vice.
  • the sensor element may be part of or constitute at least one CCD and/or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area.
  • the sensor el- ement may employ a rolling shutter or global shutter method to read out the matrix of optical sensors.
  • the camera may be a fix-focus camera, having at least one lens which is fixedly adjusted with respect to the camera. Alternatively, however, the camera may also comprise one or more variable lenses which may be adjusted, automatically or manually.
  • the camera further may comprise at least one transfer device.
  • the camera may further com- prise one or more additional elements such as one or more additional optical elements.
  • the camera may comprise at least one optical element selected from the group consisting of: trans- fer device, such as at least one lens and/or at least one lens system, at least one diffractive op- tical element.
  • trans- fer device such as at least one lens and/or at least one lens system, at least one diffractive op- tical element.
  • the term “transfer device”, also denoted as “transfer system”, may generally refer to one or more optical elements which are adapted to modify the light beam, such as by modify- ing one or more of a beam parameter of the light beam, a width of the light beam or a direction of the light beam.
  • the transfer device may be adapted to guide the light beam onto the optical sensor.
  • the transfer device specifically may comprise one or more of: at least one lens, for ex- ample at least one lens selected from the group consisting of at least one focus-tunable lens, at least one aspheric lens, at least one spheric lens, at least one Fresnel lens; at least one diffrac- tive optical element; at least one concave mirror; at least one beam deflection element, prefera- bly at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; at least one multi-lens system.
  • the transfer device may have a focal length.
  • the term “focal length” of the transfer device refers to a dis- tance over which incident collimated rays which may impinge the transfer device are brought into a “focus” which may also be denoted as “focal point”.
  • the focal length constitutes a measure of an ability of the transfer device to converge an impinging light beam.
  • the transfer device may comprise one or more imaging elements which can have the effect of a converging lens.
  • the transfer device can have one or more lenses, in partic- ular one or more refractive lenses, and/or one or more convex mirrors.
  • the focal length may be defined as a distance from the center of the thin refractive lens to the principal focal points of the thin lens.
  • the focal length may be considered as being positive and may provide the distance at which a beam of collimated light impinging the thin lens as the transfer device may be focused into a single spot.
  • the transfer device can comprise at least one wavelength-selec- tive element, for example at least one optical filter.
  • the transfer device can be de- signed to impress a predefined beam profile on the electromagnetic radiation, for example, at the location of the sensor region and in particular the sensor area.
  • the abovementioned op- tional embodiments of the transfer device can, in principle, be realized individually or in any de- sired combination.
  • the transfer device may have an optical axis.
  • optical axis of the trans- fer device generally refers to an axis of mirror symmetry or rotational symmetry of the lens or lens system.
  • the transfer system may comprise at least one beam path, with the elements of the transfer system in the beam path being located in a rotationally symmetrical fashion with respect to the optical axis. Still, one or more optical elements located within the beam path may also be off-centered or tilted with respect to the optical axis.
  • the optical axis may be defined sequentially, such as by interconnecting the centers of the optical elements in the beam path, e.g. by interconnecting the centers of the lenses, wherein, in this context, the optical sensors are not counted as optical elements.
  • the optical axis generally may denote the beam path.
  • the camera may have a single beam path along which a light beam may travel from the object to the optical sensors, or may have a plurality of beam paths.
  • a single beam path may be given or the beam path may be split into two or more partial beam paths. In the latter case, each partial beam path may have its own optical axis.
  • the optical sensors may be located in one and the same beam path or partial beam path. Alternatively, however, the optical sensors may also be located in different partial beam paths.
  • the transfer device may constitute a coordinate system, wherein a longitudinal coordinate is a coordinate along the optical axis and wherein d is a spatial offset from the optical axis.
  • the co- ordinate system may be a polar coordinate system in which the optical axis of the transfer de- vice forms a z-axis and in which a distance from the z-axis and a polar angle may be used as additional coordinates.
  • a direction parallel or antiparallel to the z-axis may be considered a lon- gitudinal direction, and a coordinate along the z-axis may be considered a longitudinal coordi- nate. Any direction perpendicular to the z-axis may be considered a transversal direction, and the polar coordinate and/or the polar angle may be considered a transversal coordinate.
  • the camera is configured for imaging the carrier at a plurality imaging positions of the camera.
  • the images may be generated in response to the diffuse and/or uniform illumination of the car- rier by the flood light source.
  • the images generated in response to the diffuse and/or uniform illumination of the carrier by the flood light source may not comprise any reflection features gen- erated by the illumination pattern.
  • the image may be at least one two-dimensional image.
  • the term “two dimensional image” may generally refer to an image having infor- mation about transversal coordinates such as the dimensions of height and width.
  • the image may be an RGB (red green blue) image.
  • imaging at least one image may refer to capturing and/or recording the image.
  • the camera is configured for imaging the at least one reflection image.
  • the reflection image comprises a plurality of reflection features generated by the scene in response to the illumina- tion pattern.
  • the term “reflection feature” may refer to a feature in an image plane generated by the scene in response to illumination, specifically with at least one illumina- tion feature.
  • Each of the reflection features comprises at least one beam profile, also denoted reflection beam profile.
  • the term “beam profile” of the reflection feature may generally refer to at least one intensity distribution of the reflection feature, such as of a light spot on the optical sensor, as a function of the pixel.
  • the beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile and a linear combination of Gaussian beam profiles.
  • the evaluation of the reflection image may comprise identifying the reflection features of the re- flection image.
  • the processing unit may be configured for performing at least one image analy- sis and/or image processing in order to identify the reflection features.
  • the image analysis and/or image processing may use at least one feature detection algorithm.
  • the image analysis and/or image processing may comprise one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between an image created by the sensor signals and at least one offset; an inversion of sensor signals by inverting an image cre- ated by the sensor signals; a formation of a difference image between an image created by the sensor signals at different times; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a blob detector; applying a corner detector; applying a Determinant of Hessian filter; applying a principle curvature-based region detector; applying a maximally stable extremal regions detector; applying a generalized Hough-transformation; ap- plying a ridge detector; applying an affine invariant feature detector; applying an affine-adapted interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector
  • the region of interest may be determined manually by a user or may be determined automatically, such as by recognizing a feature within the image generated by the optical sensor.
  • the term “processing unit” generally refers to an arbitrary data pro- cessing device adapted to perform the named operations such as by using at least one proces- sor and/or at least one application-specific integrated circuit.
  • the at least one processing unit may comprise a software code stored thereon comprising a number of com- puter commands.
  • the processing unit may provide one or more hardware elements for perform- ing one or more of the named operations and/or may provide one or more processors with soft- ware running thereon for performing one or more of the named operations.
  • Operations, includ- ing evaluating the images may be performed by the at least one processing unit.
  • one or more instructions may be implemented in software and/or hardware.
  • the processing unit may comprise one or more programmable devices such as one or more computers, application-specific integrated circuits (ASICs), Digital Signal Processors (DSPs), or Field Programmable Gate Arrays (FPGAs) which are configured to perform the above-mentioned evaluation.
  • ASICs application-specific integrated circuits
  • DSPs Digital Signal Processors
  • FPGAs Field Programmable Gate Arrays
  • the processing unit may also fully or partially be embodied by hardware.
  • the processing unit and the camera may fully or partially be integrated into a single device.
  • the processing unit also may form part of the camera.
  • the processing unit and the camera may fully or partially be embodied as separate devices.
  • the processing unit may be or may comprise one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), and/or one or more data processing devic- es, such as one or more computers, preferably one or more microcomputers and/or microcon- trollers, Field Programmable Arrays, or Digital Signal Processors. Additional components may be comprised, such as one or more preprocessing devices and/or data acquisition devices, such as one or more devices for receiving and/or preprocessing of the sensor signals, such as one or more AD-converters and/or one or more filters. Further, the processing unit may com- prise one or more measurement devices, such as one or more measurement devices for meas- uring electrical currents and/or electrical voltages.
  • ASICs application-specific integrated circuits
  • data processing devic- es such as one or more computers, preferably one or more microcomputers and/or microcon- trollers, Field Programmable Arrays, or Digital Signal Processors. Additional components may be comprised,
  • the processing unit may comprise one or more data storage devices. Further, the processing unit may comprise one or more inter- faces, such as one or more wireless interfaces and/or one or more wire-bound interfaces.
  • the processing unit may be configured to one or more of displaying, visualizing, analyzing, dis- tributing, communicating or further processing of information, such as information obtained by the camera.
  • the processing unit may be connected or incorporate at least one of a display, a projector, a monitor, an LCD, a TFT, a loudspeaker, a multichannel sound sys- tem, an LED pattern, or a further visualization device.
  • It may further be connected or incorporate at least one of a communication device or communication interface, a connector or a port, capa- ble of sending encrypted or unencrypted information using one or more of email, text messages, telephone, Bluetooth, Wi-Fi, infrared or internet interfaces, ports or connections.
  • a processor may further be connected to or incorporate at least one of a processor, a graphics processor, a CPU, an Open Multimedia Applications Platform (OMAPTM), an integrated circuit, a system on a chip such as products from the Apple A series or the Samsung S3C2 series, a microcontroller or mi- croprocessor, one or more memory blocks such as ROM, RAM, EEPROM, or flash memory, timing sources such as oscillators or phase-locked loops, counter-timers, real-time timers, or power-on reset generators, voltage regulators, power management circuits, or DMA controllers.
  • Individual units may further be connected by buses such as AMBA buses or be integrated in an Internet of Things or Industry 4.0 type network.
  • the processing unit may be connected by or have further external interfaces or ports such as one or more of serial or parallel interfaces or ports, USB, Centronics Port, FireWire, HDMI, Ethernet, Bluetooth, RFID, Wi-Fi, USART, or SPI, or analogue interfaces or ports such as one or more of ADCs or DACs, or standardized interfaces or ports to further devices such as a 2D- camera device using an RGB-interface such as CameraLink.
  • the processing unit may further be connected by one or more of interprocessor interfaces or ports, FPGA-FPGA-interfaces, or serial or parallel interfaces ports.
  • the processing unit may further be connected to one or more of an optical disc drive, a CD-RW drive, a DVD+RW drive, a flash drive, a memory card, a disk drive, a hard disk drive, a solid state disk or a solid state hard disk.
  • an optical disc drive a CD-RW drive, a DVD+RW drive, a flash drive, a memory card, a disk drive, a hard disk drive, a solid state disk or a solid state hard disk.
  • the processing unit may be connected by or have one or more further external connectors such as one or more of phone connectors, RCA connectors, VGA connectors, hermaphrodite con- nectors, USB connectors, HDMI connectors, 8P8C connectors, BCN connectors, IEC 60320 C14 connectors, optical fiber connectors, D-subminiature connectors, RF connectors, coaxial connectors, SCART connectors, XLR connectors, and/or may incorporate at least one suitable socket for one or more of these connectors.
  • the processing unit may be configured for determining the beam profile of the respective reflec- tion feature.
  • the term “determining the beam profile” refers to identifying at least one reflection feature provided by the optical sensor and/or selecting at least one reflection fea- ture provided by the optical sensor and evaluating at least one intensity distribution of the reflec- tion feature.
  • a region of the matrix may be used and evaluated for determining the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the matrix.
  • a center of illumination by the light beam may be determined, such as by determining the at least one pixel having the highest illumination, and a cross-sectional axis may be chosen through the cen- ter of illumination.
  • the intensity distribution may an intensity distribution as a function of a coor- dinate along this cross-sectional axis through the center of illumination. Other evaluation algo- rithms are feasible.
  • the reflection feature may cover or may extend over at least one pixel of the refelction image. For example, the reflection feature may cover or may extend over plurality of pixels.
  • the pro- cessing unit may be configured for determining and/or for selecting all pixels connected to and/or belonging to the reflection feature, e.g. a light spot.
  • the processing unit may be config- ured for determining the center of intensity by wherein R coi is a position of center of intensity, r pixel is the pixel position and with j being the number of pixels j connected to and/or belonging to the reflection feature and I total be- ing the total intensity.
  • the processing unit is configured for determining material characteristics for each reflection fea- ture by analysis of its respective beam profile.
  • material characteris- tics may refer to arbitrary material property m derived from and/or relating to the beam profile of the reflection feature.
  • the material characteristics may be at least one information selected from the group consisting of information about softness, information about deformability, or infor- mation about permeability to air.
  • the material characteristics may be extracted for distinguishing between foreground and background, i.e. items to pick vs. carrier.
  • the processing unit may be configured for determining the material property m of the surface remitting the reflection feature by evaluating the beam profile of the reflection feature.
  • material property refers to at least one arbitrary property of the material con- figured for characterizing and/or identification and/or classification of the material.
  • the material property may be a property selected from the group consisting of: roughness, pene- tration depth of light into the material, a reflectivity, a specular reflectivity, a diffuse reflectivity, a surface property, a measure for translucence, a scattering, specifically a back-scattering behav- ior or the like.
  • the at least one material property may be a property selected from the group consisting of: a scattering coefficient, a translucency, a transparency, a deviation from a Lam- bertian surface reflection, a speckle, and the like.
  • the processing unit may be configured for identifying a reflection feature as to be generated by an item having a specific material property in case its reflection beam profile fulfills at least one predetermined or predefined criterion.
  • the term “at least one predetermined or predefined criterion” refers to at least one property and/or value suitable to distinguish material properties.
  • the predetermined or predefined criterion may be or may comprise at least one pre- determined or predefined value and/or threshold and/or threshold range referring to a material property.
  • the reflection feature may be indicated as to be generated by an item having a spe- cific material property in case the reflection beam profile fulfills the at least one predetermined or predefined criterion.
  • the term “indicate” refers to an arbitrary indication such as an electronic signal and/or at least one visual or acoustic indication.
  • the term “determining at least one material property” may refer to assigning the material property to respective reflection feature.
  • the processing unit may comprise at least one database comprising a list and/or table, such as a lookup list or a lookup table, of predefined and/or predetermined material properties.
  • the list and/or table of material properties may be de- termined and/or generated by performing at least one test measurement, for example by per- forming material tests using samples having known material properties.
  • the list and/or table of material properties may be determined and/or generated at the manufacturer site and/or by a user.
  • the material property may additionally be assigned to a material classifier such as one or more of a material name, a material group such as biological or non-biological material, translu- cent or non-translucent materials, metal or non-metal, fur or non-fur, carpet or non-carpet, re- flective or non-reflective, specular reflective or non-specular reflective, foam or non-foam, roughness groups or the like.
  • the processing unit may comprise at least one database compris- ing a list and/or table comprising the material properties and associated material name and/or material group.
  • beam profile analysis may be used for determining the material characteristic in step b). Spe- cifically, beam profile analysis makes use of reflection properties of coherent light projected onto object surfaces to classify materials.
  • the classification of materials may be performed as de- scribed in WO 2020/187719, in EP application 20159984.2 filed on February 28, 2020 and/or EP application 20154961.5 filed on January 31, 2020, the full content of which is included by reference.
  • a periodic grid of laser spots e.g. a hexagonal grid as described in EP application 20170905.2 filed on April 22, 2020, is projected and the reflection image is rec- orded with the camera.
  • Analyzing the beam profile of each reflection feature recorded by the camera may be performed by feature-based methods and/or using based on a convolutional neural network classifying the reflection features of the reflection image.
  • the feature based methods may be used in combination with machine learning methods which may allow para- metrization of a classification model.
  • Convolutional neuronal networks may be utilized to classify materials by using the reflection images as an input.
  • the feature-based methods may be explained in the following.
  • the processing unit may be con- figured for comparing the reflection beam profile with at least one predetermined and/or prere- corded and/or predefined beam profile.
  • the predetermined and/or prerecorded and/or prede- fined beam profile may be stored in a table or a lookup table and may be determined e.g. empir- ically, and may, as an example, be stored in at least one data storage device.
  • the predetermined and/or prerecorded and/or predefined beam profile may be determined during initial start-up of a device executing the method according to the present invention.
  • the predetermined and/or prerecorded and/or predefined beam profile may be stored in at least one data storage device of the processing unit, e.g. by software, specifically by the app downloaded from an app store or the like.
  • the reflection feature may be identified as to be gen- erated by an item having a material property m in case the reflection beam profile and the pre- determined and/or prerecorded and/or predefined beam profile are identical.
  • the comparison may comprise overlaying the reflection beam profile and the predetermined or predefined beam profile such that their centers of intensity match.
  • the comparison may comprise determining a deviation, e.g.
  • the processing unit may be adapted to compare the determined deviation with at least one threshold, wherein in case the determined deviation is below and/or equal the threshold the surface is indicated as biological tissue and/or the detection of biological tissue is confirmed.
  • the threshold value may be stored in a table or a lookup table and may be determined e.g. empirically and may, as an example, be stored in at least one data storage device of the processing unit. Additionally or alternatively, the material characteristics may be determined by applying at least one image filter to the refection image.
  • the term “image” refers to a two- dimensional function, f(x,y), wherein brightness and/or color values are given for any x,y-posi- tion in the image.
  • the position may be discretized corresponding to the recording pixels.
  • the brightness and/or color may be discretized corresponding to a bit-depth of the optical sensors.
  • the processing unit may be configured for determining at least one material feature ⁇ 2m by ap- plying at least one material dependent image filter ⁇ 2 to the image.
  • material dependent image filter refers to an image having a material dependent output.
  • the output of the material dependent image filter is denoted herein “material feature ⁇ 2m” or “mate- rial dependent feature ⁇ 2m”.
  • the material feature may be or may comprise at least one infor- mation about the at least one material property of the surface of the scene having generated the reflection feature.
  • the material dependent image filter may be at least one filter selected from the group consisting of: a luminance filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothness filter such as a Gaussian filter or median filter; a grey-level-occurrence-based con- trast filter; a grey-level-occurrence-based energy filter; a grey-level-occurrence-based homoge- neity filter; a grey-level-occurrence-based dissimilarity filter; a Law’s energy filter; a threshold area filter; or a linear combination thereof; or a further material dependent image filter ⁇ 2other which correlates to one or more of the luminance filter, the spot shape filter, the squared norm gradient, the standard deviation, the smoothness filter, the grey-level-occurrence-based energy filter, the grey-level-occurrence-based homogeneity filter, the grey-level-occurrence
  • the further material dependent image filter ⁇ 2other may correlate to one or more of the material dependent image filters ⁇ m by
  • the material dependent image filter may be at least one arbitrary filter ⁇ that passes a hypothe- sis testing.
  • the term “passes a hypothesis testing” refers to the fact that a Null- hypothesis H 0 is rejected and an alternative hypothesis H 1 is accepted.
  • the hypothesis testing may comprise testing the material dependency of the image filter by applying the image filter to a predefined data set.
  • the data set may comprise a plurality of beam profile images.
  • the exponential factor is identical for all Gaussian functions in all images.
  • the center-positions, are identical for all images f k :
  • Each of the beam profile images in the dataset may correspond to a material classifier and a distance.
  • the material classifier may be a label such as ‘Material A’, ‘Material B’, etc.
  • the beam profile images may be generated by using the above formula for f k (x, y) in combination with the following parameter table:
  • the values for x, y are integers corresponding to pixels with .
  • the images may have a pixel size of 32x32.
  • the dataset of beam profile images may be generated by using the above formula for f k in combination with a parameter set to obtain a continuous description of f k .
  • the values for each pixel in the 32x32-image may be obtained by inserting integer values from 0, ..., 31 for x, y, in f k (x, y). For example, for pixel (6,9), the value f k (6,9) may be com- puted.
  • the hypothesis testing may use a Null-hypothesis that the filter does not distinguish between material classifier.
  • the hypothesis testing may use as alternative hypothesis that the filter does distinguish between at least two material classifiers.
  • the alternative hypothesis may be given by
  • the term “not distinguish between material classifiers” refers to that the expectation values of the material classifiers are identical.
  • the term “distinguishes material classifiers” refers to that at least two expectation values of the material classifiers differ.
  • distinguishes at least two material classifiers is used synonymous to “suitable material classifier”.
  • the hypothesis testing may comprise at least one analysis of variance (ANOVA) on the generated feature values.
  • ANOVA analysis of variance
  • the hypothesis testing may comprise determining a mean-value of the feature values for each of the ⁇ materi- als, i.e.
  • the hypothesis testing may comprise determining a mean-value of all N feature values
  • the hypothesis testing may comprise determining a Mean Sum Squares within:
  • the hypothesis testing may comprise determining a Mean Sum of Squares between,
  • the hypothesis testing may comprise performing an F-Test:
  • I ⁇ is the regularized incomplete Beta-Function, , with the Euler Beta- Function being the incomplete Beta-Function.
  • the image filter may pass the hypothesis testing if a p-value, p, is smaller or equal than a pre-defined level of significance.
  • the filter may pass the hypothesis testing if p ⁇ 0.075, preferably p ⁇ 0.05, more preferably p ⁇ 0.025, and most preferably p ⁇ 0.01.
  • the Null-hypothesis H0 can be rejected and the alternative hypothesis H 1 can be accepted.
  • the image filter thus distin- guishes at least two material classifiers.
  • the image filter passes the hypothesis testing.
  • image filters are described assuming that the reflection image comprises at least one reflection feature, in particular a spot image.
  • a spot image ⁇ may be given by a func- tion wherein the background of the image f may be already subtracted.
  • the material dependent image filter may be a luminance filter.
  • the luminance filter may return a luminance measure of a spot as material feature.
  • the material feature may be de- termined by where f is the spot image.
  • the distance of the spot is denoted by z, where z may be obtained for example by using a depth-from-defocus or depth-from–photon ratio technique and/or by us- ing a triangulation technique.
  • the surface normal of the material is given by ⁇ ⁇ R ⁇ and can be obtained as the normal of the surface spanned by at least three measured points.
  • the vector is the direction vector of the light source. Since the position of the spot is known by using a depth-from-defocus or depth-from–photon ratio technique and/or by using a triangula- tion technique wherein the position of the light source is known as a parameter of the detector system, d ray , is the difference vector between spot and light source positions.
  • the material dependent image filter may be a filter having an output dependent on a spot shape. This material dependent image filter may return a value which correlates to the translucence of a material as material feature. The translucence of materials influences the shape of the spots.
  • the spot height h may be determined by where B r is an inner circle of a spot with radius r.
  • the material dependent image filter may be a squared norm gradient. This mate- rial dependent image filter may return a value which correlates to a measure of soft and hard transitions and/or roughness of a spot as material feature.
  • the material feature may be defined by
  • the material dependent image filter may be a standard deviation.
  • the standard deviation of the spot may be determined by Wherein ⁇ is the mean value given by
  • the material dependent image filter may be a smoothness filter such as a Gauss- ian filter or median filter.
  • this image filter may refer to the observation that volume scattering exhibits less speckle contrast compared to diffuse scattering materials.
  • This image filter may quantify the smoothness of the spot corresponding to speckle contrast as material feature.
  • the material feature may be determined by wherein F is a smoothness function, for example a median filter or Gaussian filter. This image filter may comprise dividing by the distance z, as described in the formula above.
  • the distance z may be determined for example using a depth-from-defocus or depth-from–photon ratio tech- nique and/or by using a triangulation technique. This may allow the filter to be insensitive to dis- tance.
  • the smoothness filter may be based on the standard deviation of an extracted speckle noise pattern.
  • a speckle noise pattern N can be de- scribed in an empirical way by where ⁇ ⁇ is an image of a despeckled spot.
  • N(X) is the noise term that models the speckle pat- tern.
  • the computation of a despeckled image may be difficult.
  • the despeckled image may be approximated with a smoothed version of f, i.e.
  • the material feature of this filter may be determined by Wherein Var denotes the variance function.
  • the material feature of the grey-level-occurrence-based contrast filter may be given by
  • the image filter may be a grey-level-occurrence-based energy filter. This material filter is based on the grey level occurrence matrix defined above.
  • the material feature of the grey-level-occurrence-based energy filter may be given by
  • the image filter may be a grey-level-occurrence-based homogeneity filter. This material filter is based on the grey level occurrence matrix defined above.
  • the material feature of the grey-level-occurrence-based homogeneity filter may be given by
  • the image filter may be a grey-level-occurrence-based dissimilarity filter. This ma- terial filter is based on the grey level occurrence matrix defined above.
  • the material feature of the grey-level-occurrence-based dissimilarity filter may be given by
  • the image filter may be a Law’s energy filter.
  • the image fk is convoluted with these matrices: and
  • the material dependent image filter may be a threshold area filter. This material feature may relate two areas in the image plane.
  • a first area ⁇ 1, may be an area wherein the function f is larger than ⁇ times the maximum of f.
  • a second area ⁇ 2 may be an area wherein the function f is smaller than ⁇ times the maximum of f, but larger than a threshold value ⁇ times the maximum of f.
  • may be 0.5 and ⁇ may be 0.05. Due to speckles or noise, the ar- eas may not simply correspond to an inner and an outer circle around the spot center.
  • ⁇ 1 may comprise speckles or unconnected areas in the outer circle.
  • f(x) > ⁇ max(f(x)) ⁇ and ⁇ 2 ⁇ x
  • the processing unit may be configured for using at least one predetermined relationship be- tween the material feature ⁇ 2m and the material property of the surface having generated the re- flection feature for determining the material property of the surface having generated the reflec- tion feature.
  • the predetermined relationship may be one or more of an empirical relationship, a semi-empiric relationship and an analytically derived relationship.
  • the processing unit may com- prise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.
  • Step b) may comprise using artificial intelligence, in particular convolutional neuronal networks. Using reflection images as input for convolutional neuronal networks may enable the generation of classification models with sufficient accuracy to differentiate between materials.
  • step b) at least one parametrized classification model may be used.
  • the para- metrized classification model may be configured for classifying materials by using the reflection image as an input.
  • the classification model may be parametrized by using one or more of ma- chine learning, deep learning, neural networks, or other form of artificial intelligence.
  • machine-learning as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to a method of using artificial intelli- gence (AI) for automatically model building, in particular for parametrizing models.
  • the classifi- cation model may be a classification model configured for discriminating materials. The material characteristics may be determined by applying an optimization algorithm in terms of at least one optimization target on the classification model.
  • the machine learning may be based on at least one neuronal network, in particular a convolutional neural network. Weights and/or topology of the neuronal network may be pre-determined and/or pre-defined. Specifically, the training of the classification model may be performed using machine-learning.
  • the classification model may comprise at least one machine-learning architecture and model parameters.
  • the machine-learning architecture may be or may comprise one or more of: linear regression, lo- gistic regression, random forest, naive Bayes classifications, nearest neighbors, neural net- works, convolutional neural networks, generative adversarial networks, support vector ma- chines, or gradient boosting algorithms or the like.
  • the term “training”, also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically specifically may refer, without limitation, to a process of building the model, in particular deter- mining and/or updating parameters of the model.
  • the classification model may be at least par- tially data-driven.
  • the classification model may be based on experimental data.
  • the training may comprise using at least one training dataset, wherein the training data set comprises images, in particular refection images, of a plurality of items with known ma- terial property.
  • multiple camera positions may be computed that cover the entire carrier to ensure that all items can be picked. These positions may be approached in a row until no item can be identified anymore and all items are covered.
  • Step b) may comprise the processing unit determining depth information for each reflection fea- ture by evaluating the reflection image. The evaluation may comprises, for each reflection fea- ture, an analysis of its respective beam profile using a depth-from-photon ratio technique.
  • the analysis of the beam profile comprise evaluating of the beam profile and may comprise at least one mathematical operation and/or at least one comparison and/or at least symmetrizing and/or at least one filtering and/or at least one normalizing.
  • the analysis of the beam pro- file may comprise at least one of a histogram analysis step, a calculation of a difference meas- ure, application of a neural network, application of a machine learning algorithm.
  • the pro- cessing unit may be configured for symmetrizing and/or for normalizing and/or for filtering the beam profile, in particular to remove noise or asymmetries from recording under larger angles, recording edges or the like.
  • the processing unit may filter the beam profile by removing high spatial frequencies such as by spatial frequency analysis and/or median filtering or the like. Summarization may be performed by center of intensity of the light spot and averaging all inten- sities at the same distance to the center.
  • the processing unit may be configured for normalizing the beam profile to a maximum intensity, in particular to account for intensity differences due to the recorded distance.
  • the processing unit may be configured for removing influences from background light from the beam profile, for example, by an imaging without illumination.
  • the processing unit may be configured for determining at least one longitudinal coordinate zDPR for each of the reflection features by analysis of the beam profile of the respective reflection fea- ture.
  • the processing unit may be configured for determining the longitudinal coordinate z DPR for the reflection features by using the so called depth-from-photon-ratio technique, also denoted as beam profile analysis.
  • depth-from-photon-ratio DPR
  • the processing unit may be configured for determining at least one first area and at least one second area of the reflection beam profile of each of the reflection features and/or of the reflec- tion features in at least one region of interest.
  • the processing unit is configured for integrating the first area and the second area.
  • the analysis of the beam profile of one of the reflection features may comprise determining at least one first area and at least one second area of the beam profile.
  • the first area of the beam profile may be an area A1 and the second area of the beam profile may be an area A2.
  • the pro- cessing unit may be configured for integrating the first area and the second area.
  • the pro- cessing unit may be configured to derive a combined signal, in particular a quotient Q, by one or more of dividing the integrated first area and the integrated second area, dividing multiples of the integrated first area and the integrated second area, dividing linear combinations of the inte- grated first area and the integrated second area.
  • the processing unit may be configured for deter- mining at least two areas of the beam profile and/or to segment the beam profile in at least two segments comprising different areas of the beam profile, wherein overlapping of the areas may be possible as long as the areas are not congruent.
  • the processing unit may be configured for determining a plurality of areas such as two, three, four, five, or up to ten areas.
  • the processing unit may be configured for segmenting the light spot into at least two areas of the beam profile and/or to segment the beam profile in at least two segments comprising differ- ent areas of the beam profile.
  • the processing unit may be configured for determining for at least two of the areas an integral of the beam profile over the respective area.
  • the processing unit may be configured for comparing at least two of the determined integrals.
  • the pro- cessing unit may be configured for determining at least one first area and at least one second area of the beam profile.
  • area of the beam profile generally refers to an arbitrary region of the beam profile at the position of the optical sensor used for determining the quotient Q.
  • the first area of the beam profile and the second area of the beam profile may be one or both of adjacent or overlapping regions.
  • the first area of the beam profile and the second area of the beam profile may be not congruent in area.
  • the processing unit may be configured for dividing a sensor region of the CMOS sensor into at least two sub-re- gions, wherein the processing unit may be configured for dividing the sensor region of the CMOS sensor into at least one left part and at least one right part and/or at least one upper part and at least one lower part and/or at least one inner and at least one outer part.
  • the camera may comprise at least two optical sensors, wherein the light-sensitive areas of a first optical sensor and of a second optical sensor may be arranged such that the first optical sensor is adapted to determine the first area of the beam profile of the reflection feature and that the second optical sensor is adapted to determine the second area of the beam profile of the reflection feature.
  • the processing unit may be adapted to integrate the first area and the second area.
  • the processing unit may be configured for using at least one predetermined rela- tionship between the quotient Q and the longitudinal coordinate for determining the longitudinal coordinate.
  • the predetermined relationship may be one or more of an empiric relationship, a semi-empiric relationship and an analytically derived relationship.
  • the processing unit may com- prise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.
  • the first area of the beam profile may comprise essentially edge information of the beam profile and the second area of the beam profile comprises essentially center information of the beam profile, and/or the first area of the beam profile may comprise essentially information about a left part of the beam profile and the second area of the beam profile comprises essentially infor- mation about a right part of the beam profile.
  • the beam profile may have a center, i.e. a maxi- mum value of the beam profile and/or a center point of a plateau of the beam profile and/or a geometrical center of the light spot, and falling edges extending from the center.
  • the second re- gion may comprise inner regions of the cross section and the first region may comprise outer regions of the cross section.
  • the term “essentially center information” generally refers to a low proportion of edge information, i.e. proportion of the intensity distribution corre- sponding to edges, compared to a proportion of the center information, i.e. proportion of the in- tensity distribution corresponding to the center.
  • the center information has a propor- tion of edge information of less than 10%, more preferably of less than 5%, most preferably the center information comprises no edge content.
  • the term “essentially edge infor- mation” generally refers to a low proportion of center information compared to a proportion of the edge information.
  • the edge information may comprise information of the whole beam pro- file, in particular from center and edge regions.
  • the edge information may have a proportion of center information of less than 10%, preferably of less than 5%, more preferably the edge infor- mation comprises no center content.
  • At least one area of the beam profile may be determined and/or selected as second area of the beam profile if it is close or around the center and com- prises essentially center information.
  • At least one area of the beam profile may be determined and/or selected as first area of the beam profile if it comprises at least parts of the falling edges of the cross section. For example, the whole area of the cross section may be determined as first region. Other selections of the first area A1 and second area A2 may be feasible.
  • the first area may comprise essentially outer regions of the beam profile and the second area may com- prise essentially inner regions of the beam profile.
  • the beam profile may be divided in a left part and a right part, wherein the first area may comprise essentially areas of the left part of the beam profile and the second area may comprise essentially areas of the right part of the beam profile.
  • the edge information may comprise information relating to a number of photons in the first area of the beam profile and the center information may comprise information relating to a number of photons in the second area of the beam profile.
  • the processing unit may be configured for de- termining an area integral of the beam profile.
  • the processing unit may be configured for deter- mining the edge information by integrating and/or summing of the first area.
  • the processing unit may be configured for determining the center information by integrating and/or summing of the second area.
  • the beam profile may be a trapezoid beam profile and the pro- cessing unit may be configured for determining an integral of the trapezoid.
  • the determination of edge and center signals may be re- placed by equivalent evaluations making use of properties of the trapezoid beam profile such as determination of the slope and position of the edges and of the height of the central plateau and deriving edge and center signals by geometric considerations.
  • A1 may correspond to a full or complete area of a feature point on the opti- cal sensor.
  • A2 may be a central area of the feature point on the optical sensor. The central area may be a constant value.
  • the central area may be smaller compared to the full area of the fea- ture point.
  • the central area may have a radius from 0.1 to 0.9 of a full radius of the feature point, preferably from 0.4 to 0.6 of the full radius.
  • the illumination pattern may comprise at least one line pattern.
  • A1 may cor- respond to an area with a full line width of the line pattern on the optical sensors, in particular on the light sensitive area of the optical sensors.
  • the line pattern on the optical sensor may be wid- ened and/or displaced compared to the line pattern of the illumination pattern such that the line width on the optical sensors is increased.
  • the line width of the line pattern on the optical sensors may change from one column to another col- umn.
  • A2 may be a central area of the line pattern on the optical sensor.
  • the line width of the central area may be a constant value, and may in particular correspond to the line width in the illumination pattern.
  • the central area may have a smaller line width compared to the full line width.
  • the central area may have a line width from 0.1 to 0.9 of the full line width, preferably from 0.4 to 0.6 of the full line width.
  • the line pattern may be segmented on the optical sensors.
  • Each column of the matrix of optical sensors may comprise center information of inten- sity in the central area of the line pattern and edge information of intensity from regions extend- ing further outwards from the central area to edge regions of the line pattern.
  • the illumination pattern may comprise at least point pattern.
  • A1 may corre- spond to an area with a full radius of a point of the point pattern on the optical sensors.
  • A2 may be a central area of the point in the point pattern on the optical sensors.
  • the central area may be a constant value.
  • the central area may have a radius compared to the full radius. For exam- ple, the central area may have a radius from 0.1 to 0.9 of the full radius, preferably from 0.4 to 0.6 of the full radius.
  • the illumination pattern may comprise both at least one point pattern and at least one line pat- tern. Other embodiments in addition or alternatively to line pattern and point pattern are feasi- ble.
  • the processing unit may be configured to derive a quotient Q by one or more of dividing the in- tegrated first area and the integrated second area, dividing multiples of the integrated first area and the integrated second area, dividing linear combinations of the integrated first area and the integrated second area.
  • the processing unit may be configured to derive the quotient Q by one or more of dividing the first area and the second area, dividing multiples of the first area and the second area, dividing linear combinations of the first area and the second area.
  • the processing unit may be config- ured for deriving the quotient Q by wherein x and y are transversal coordinates, A1 and A2 are the first and second area of the beam profile, respectively, and E(x,y) denotes the beam profile. Additionally or alternatively, the processing unit may be adapted to determine one or both of center information or edge information from at least one slice or cut of the light spot. This may be realized, for example, by replacing the area integrals in the quotient Q by a line integral along the slice or cut. For improved accuracy, several slices or cuts through the light spot may be used and averaged. In case of an elliptical spot profile, averaging over several slices or cuts may result in improved distance information.
  • the processing unit may be configured for evaluating the beam profile, by - determining the pixel having the highest sensor signal and forming at least one center sig- nal; - evaluating sensor signals of the matrix and forming at least one sum signal; - determining the quotient Q by combining the center signal and the sum signal; and - determining at least one longitudinal coordinate z of the object by evaluating the quotient Q.
  • a “sensor signal” generally refers to a signal generated by the optical sensor and/or at least one pixel of the optical sensor in response to illumination.
  • the sensor signal may be or may comprise at least one electrical signal, such as at least one analogue electrical signal and/or at least one digital electrical signal. More specifically, the sensor signal may be or may comprise at least one voltage signal and/or at least one current signal. More specifically, the sensor signal may comprise at least one photocurrent. Further, either raw sen- sor signals may be used, or the display device, the optical sensor or any other element may be adapted to process or preprocess the sensor signal, thereby generating secondary sensor sig- nals, which may also be used as sensor signals, such as preprocessing by filtering or the like.
  • the term “center signal” generally refers to the at least one sensor signal comprising essentially center information of the beam profile.
  • the term “highest sensor signal” refers to one or both of a local maximum or a maximum in a region of interest.
  • the center signal may be the signal of the pixel having the highest sensor signal out of the plurality of sen- sor signals generated by the pixels of the entire matrix or of a region of interest within the ma- trix, wherein the region of interest may be predetermined or determinable within an image gen- erated by the pixels of the matrix.
  • the center signal may arise from a single pixel or from a group of optical sensors, wherein, in the latter case, as an example, the sensor signals of the group of pixels may be added up, integrated or averaged, in order to determine the center sig- nal.
  • the group of pixels from which the center signal arises may be a group of neighboring pix- els, such as pixels having less than a predetermined distance from the actual pixel having the highest sensor signal, or may be a group of pixels generating sensor signals being within a pre- determined range from the highest sensor signal.
  • the group of pixels from which the center sig- nal arises may be chosen as large as possible in order to allow maximum dynamic range.
  • the processing unit may be adapted to determine the center signal by integration of the plurality of sensor signals, for example the plurality of pixels around the pixel having the highest sensor sig- nal.
  • the beam profile may be a trapezoid beam profile and the processing unit may be adapted to determine an integral of the trapezoid, in particular of a plateau of the trape- zoid.
  • the center signal generally may be a single sensor signal, such as a sensor signal from the pixel in the center of the light spot, or may be a combination of a plurality of sen- sor signals, such as a combination of sensor signals arising from pixels in the center of the light spot, or a secondary sensor signal derived by processing a sensor signal derived by one or more of the aforementioned possibilities.
  • the determination of the center signal may be per- formed electronically, since a comparison of sensor signals is fairly simply implemented by con- ventional electronics, or may be performed fully or partially by software.
  • the center signal may be selected from the group consisting of: the highest sensor signal; an average of a group of sensor signals being within a predetermined range of tolerance from the highest sen- sor signal; an average of sensor signals from a group of pixels containing the pixel having the highest sensor signal and a predetermined group of neighboring pixels; a sum of sensor signals from a group of pixels containing the pixel having the highest sensor signal and a predeter- mined group of neighboring pixels; a sum of a group of sensor signals being within a predeter- mined range of tolerance from the highest sensor signal; an average of a group of sensor sig- nals being above a predetermined threshold; a sum of a group of sensor signals being above a predetermined threshold; an integral of sensor signals from a group of optical sensors contain- ing the optical sensor having the highest sensor signal and
  • the term “sum signal” generally refers to a signal comprising essentially edge infor- mation of the beam profile.
  • the sum signal may be derived by adding up the sen- sor signals, integrating over the sensor signals or averaging over the sensor signals of the en- tire matrix or of a region of interest within the matrix, wherein the region of interest may be pre- determined or determinable within an image generated by the optical sensors of the matrix.
  • the actual optical sen- sors from which the sensor signal is generated may be left out of the adding, integration or aver- aging or, alternatively, may be included into the adding, integration or averaging.
  • the pro- cessing unit may be adapted to determine the sum signal by integrating signals of the entire matrix, or of the region of interest within the matrix.
  • the beam profile may be a trapezoid beam profile and the processing unit may be adapted to determine an integral of the entire trapezoid.
  • the determination of edge and center signals may be replaced by equivalent evaluations making use of properties of the trapezoid beam profile such as determination of the slope and position of the edges and of the height of the central plateau and deriving edge and center signals by geometric considera- tions.
  • the center signal and edge signal may also be determined by using segments of the beam profile such as circular segments of the beam profile.
  • the beam profile may be divided into two segments by a secant or a chord that does not pass the center of the beam profile.
  • one segment will essentially contain edge information, while the other segment will contain essentially center information.
  • the edge signal may further be subtracted from the center sig- nal.
  • the quotient Q may be a signal which is generated by combining the center signal and the sum signal.
  • the determining may include one or more of: forming a quotient of the center signal and the sum signal or vice versa; forming a quotient of a multiple of the center signal and a multiple of the sum signal or vice versa; forming a quotient of a linear combination of the cen- ter signal and a linear combination of the sum signal or vice versa.
  • the quotient Q may comprise an arbitrary signal or signal combination which contains at least one item of information on a comparison between the center signal and the sum signal.
  • the term “longitudinal coordinate for the reflection feature” refers to a distance between the optical sensor and the point of the scene remitting the corresponding illumination features.
  • the processing unit may be configured for using the at least one predetermined rela- tionship between the quotient Q and the longitudinal coordinate for determining the longitudinal coordinate.
  • the predetermined relationship may be one or more of an empiric relationship, a semi-empiric relationship and an analytically derived relationship.
  • the processing unit may com- prise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.
  • the processing unit may be configured for executing at least one depth-from-photon-ratio algo- rithm which computes distances for all reflection features with zero order and higher order.
  • the method may comprise simulating pick point approaches.
  • the processing unit may be con- figured for simulating for all pick point candidates whether approaching them leads to collisions with other items, e.g. carrier, robot or other items, wherein pick point candidates simulated to lead to collisions are filtered out for further consideration.
  • the simulation may be performed prior to pick point scoring.
  • the processing unit is configured for assigning a score to pick point candidates according to their probability to lead to a successful grasp using the trained scoring model.
  • pick point candidate as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to potential pick points.
  • a successful grasp is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to suitability of the pick point candidate to be picked up by the robot.
  • a successful grasp may be if the pick point candidate is suitable for allowing stable vacuum.
  • the grasp may be not successful if vac- uum cannot be maintained at the respective pick point candidate.
  • the probability to lead to a successful grasp may depend on one or more of softness, deformability, permeability to air and the like.
  • score is a broad term and is to be given its ordinary and cus- tomary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to a value assigned to the respective pick point candidate, wherein the value depends on the probability to lead to a successful grasp. For example, the score may range from 0 to 1. The score may be assigned from low score for low probability to high score for high probability. In step d) the pick point with the highest score is used as next pick point.
  • scoring model also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to a classification model and/or a regression model having as output a probability distribution over classes for the reflection features.
  • the output of the scoring model may be, for each of the re- flection features, one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high.
  • step c) in addition to the material characteristics, one or more of the following input parame- ters are provided to the scoring model: information about an image section, 3D information about the reflection features.
  • the term “input” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be lim- ited to a special or customized meaning.
  • the term specifically may refer, without limitation, to an input value or parameter for the trained scoring model and/or data which can be filled into the trained scoring model.
  • the trained scoring model may be configured for generating using the input at least one output, in particular a prediction.
  • the scoring model may be a classification and/or regression model.
  • the scoring model may be at least one model selected from the group consisting of a random forest (RF) or a convolution neural network (CNN).
  • Random Forest also denoted as random forest algorithm, as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to an ensemble learning method configured for at least classification and/or regression.
  • the random forest algorithm may be configured for constructing one or more decision trees and outputting at least one class selected from the mode of the classes and the mean prediction of the individual trees.
  • the random forest algorithm may generally be known, e.g. from journal paper by Leo Breiman, precedeRandom Forests“ in Machine Learning 45.1, Oct.2001.
  • the scoring model may be based on a CNN architecture configured for classification.
  • the con- volutional neural network may be a multilayer convolutional neural network.
  • the convolutional neural network may comprise a plurality of convolutional layers.
  • the convolutional layers may be followed by a plurality of fully-connected layers.
  • the convolutional neural network may com- prise a plurality of pooling layers.
  • the structure of convolutional neural networks is generally known to the skilled person such as from en.wikipedia.org/wiki/Convolutional_neural_net- work#Convolutional.
  • the CNN may be build by using the Keras library in Python. For Keras library in Python reference is made to https://keras.io/ or https://de.wikipedia.org/wiki/Keras.
  • material characteristics and 3D information may be fed into a dense layer with a Scaled Exponential Linear Unit (SELU) activation function.
  • the dense layer may be followed by a batch normalization layer.
  • the images determined in step a) may be fed, in a second branch of the CNN, into a 5x5 convolution kernel with 16 filters.
  • the convolu- tion kernel is followed by at least one batch normalization layer, a 2x2 Max Pooling layer and a dense layer with a SELU activation function.
  • the output of first and second branches may be fed into a dense layer with a SELU activation function followed by a dense layer having a sig- moid activation function.
  • the output may be a probability distribution over classes for the reflec- tion features.
  • the output of the scoring model may be, for each of the reflection features, one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high.
  • the term “trained scoring model”, also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically specifically may refer, without limita- tion, to the fact that the scoring model was trained using at least one training dataset.
  • the train- ing dataset may comprise a plurality of reflection images having reflection features of known probability for a successful grasp. The training may be performed for different items.
  • the trained scoring model can be re-trained and/or updated based on additional data.
  • the trained scoring model may be trained by using machine learning.
  • the method may comprise at least one train- ing step, wherein, in the training step, the scoring model is trained on the at least one training dataset. Specifically, the training step may be performed before performing step c).
  • the term “close environment around the pick point candidate”, also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of or- dinary skill in the art and is not to be limited to a special or customized meaning.
  • spe- cifically may refer, without limitation, to reflection features within a range or region around the pick point candidate.
  • the pick point candidate may coincide with a reflection feature.
  • the processing unit may select a range or image region around the pick point can- didate ensuring that at least one reflection feature is present, e.g. the reflection feature nearest to the pick point candidate.
  • the processing unit may be configured for considering the images imaged in step a) and the reflection image for identifying reflection features in a close environ- ment around the pick point candidate. Once the scoring is completed, the pick point with the highest score may be taken for the next pick.
  • the control unit is configured for selecting the next pick point of the robot considering the assigned scores.
  • control unit generally refers to an arbitrary device configured for performing the named operations, preferably by using at least one data processing device and, more preferably, by using at least one processor and/or at least one ap- plication-specific integrated circuit.
  • the at least one control unit may com- prise at least one data processing device having a software code stored thereon comprising a number of computer commands.
  • the control unit may provide one or more hardware elements for performing one or more of the named operations and/or may provide one or more proces- sors with software running thereon for performing one or more of the named operations.
  • the control unit may comprise one or more programmable devices such as one or more computers, application-specific integrated circuits (ASICs), Digital Signal Processors (DSPs), or Field Pro- grammable Gate Arrays (FPGAs) which are configured to perform steps b) and c). Additionally or alternatively, however, the control unit may also fully or partially be embodied by hardware.
  • the processing unit may be part of the control unit or an additional unit.
  • the method may comprises a self-learning grasp sequence for robot bin picking. The method may comprise in step d) approaching the selected next pick point with the robot and determin- ing, by at least one sensor of the robot, sensor data of the approached pick point relating to suit- ability for grasping.
  • the method may further comprise retraining the trained scoring model on the sensor data and repeating at least steps c) and d) of the method, preferably steps a) to d).
  • the method may comprise storing the determined sensor data in at least one database.
  • the model accumulates information with every pick and learns from it.
  • the robot comprises at least one robot arm, wherein the robot arm is equipped with a vacuum grabber and a vacuum sensor.
  • the control unit may be configured for moving the robot arm to a picking position for the next pick considering the assigned scores.
  • the method may comprise the control unit controlling the vacuum sensor for approaching the pick point, wherein each grasp is labeled as positive if the vacuum sensor returns stable vacuum val- ues after approaching the pick point or negative if vacuum cannot be maintained.
  • database as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be lim- ited to a special or customized meaning.
  • the term specifically may refer, without limitation, to an organized collection of data, generally stored and accessed electronically from a computer or computer system.
  • the database may comprise or may be comprised by a data storage device.
  • the database may comprise at least one data base management system, comprising a soft- ware running on a computer or computer system, the software allowing for interaction with one or more of a user, an application or the database itself, such as in order to capture and analyze the data contained in the data base.
  • the database management system may further encom- pass facilities to administer the database.
  • the database, containing the data may, thus, be comprised by a database system which, besides the data, comprises one or more associated applications.
  • Step d) may comprise the control unit to optimize one or more of the pick point selection, pick pose, depth offset and vacuum control of the gripper considering the determined material char- acteristics.
  • the computer program may be stored on a computer-readable data carrier and/or on a computer-readable storage medium.
  • the terms “computer-readable data carrier” and “computer-readable storage medium” specifically may refer to non-transitory data storage means, such as a hard-ware stor- age medium having stored thereon computer-executable instructions.
  • the computer-readable data carrier or storage medium specifically may be or may comprise a storage medium such as a random-access memory (RAM) and/or a read-only memory (ROM).
  • RAM random-access memory
  • ROM read-only memory
  • one, more than one or even all of method steps as indicated above may be performed by using a computer or a computer network, preferably by using a computer pro- gram.
  • a computer-readable storage medium comprising instructions which, when executed by a computer or computer network, cause to carry out at least steps a) to d), in par- ticular all steps, of the method according to the present invention.
  • a data carrier having a data structure stored thereon, which, after loading into a computer or computer network, such as into a working memory or main memory of the computer or computer network, may execute the method according to one or more of the embodiments disclosed herein.
  • a computer program product with program code means stored on a machine-readable carrier, in order to perform the method according to one or more of the embodiments disclosed herein, when the program is executed on a computer or computer network.
  • a computer program product refers to the program as a trad- able product.
  • the product may generally exist in an arbitrary format, such as in a paper format, or on a computer-readable data carrier and/or on a computer-readable storage medium.
  • the computer program product may be distributed over a data network.
  • a modulated data signal which contains instructions readable by a computer system or computer network, for performing the method according to one or more of the embodiments disclosed herein.
  • one or more of the method steps or even all of the method steps of the methods according to one or more of the embodi- ments disclosed herein may be performed by using a computer or computer network.
  • any of the method steps including provision and/or manipulation of data may be per- formed by using a computer or computer network.
  • these method steps may include any of the method steps, typically except for method steps requiring manual work.
  • a computer or computer network comprising at least one processor, wherein the processor is adapted to perform the method according to one of the embodiments described in this description, - a computer loadable data structure that is adapted to perform the method according to one of the embodiments described in this description while the data structure is being executed on a computer, - a computer program, wherein the computer program is adapted to perform the method according to one of the embodiments described in this description while the program is being executed on a computer, - a computer program comprising program means for performing the method according to one of the embodiments described in this description while the computer program is being executed on a computer or on a computer network, - a computer program comprising program means according to the preceding embodiment, wherein the program means are stored on a storage medium readable to a computer, - a storage medium, wherein a data structure is stored on the storage medium and wherein the data structure is adapted to perform the methods according to one of the embodiments described in this description after having been loaded into
  • a pick system is proposed.
  • the term "system” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning.
  • the term specifically may refer, without limitation, to an arbitrary set of interacting or interdependent components parts forming a whole.
  • the components may interact with each other in order to fulfill at least one common function.
  • the at least two components may be handled independently or may be coupled or connectable.
  • the components of the pick system may be configured for interacting for performing a picking function, in particular a gripping func- tion.
  • the pick system comprising - at least one robot configured for picking items comprised in a carrier; - at least one camera configured for imaging the carrier at a plurality of imaging positions; - at least one projector configured for projecting at least one illumination pattern on a scene comprising the carrier, wherein the camera is configured for imaging at least one reflection image, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; - at least one processing unit configured for determining material characteristics for each reflection feature by evaluating the reflection image, wherein the evaluation comprises, for each reflection feature, an analysis of its respective beam profile, wherein the processing unit is con- figured for assigning a score according to their probability to lead to a successful grasp to pick point candidates using a trained scoring model, wherein the determined material characteristics in a close environment around the pick point candidate are used as input for the trained scoring model; - at least one control unit configured for selecting the next pick point of the robot considering the assigned scores.
  • Embodiment 1 A computer implemented method for picking of items comprised in a carrier by using at least one robot, wherein the method comprises the following steps: a) at least one carrier scanning step comprising imaging the carrier at a plurality of imag- ing positions by using at least one camera; b) at least one material detection step comprising b1) projecting at least one illumination pattern on a scene comprising the carrier by using at least one projector and imaging at least one reflection image using the cam- era, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) determining material characteristics for each reflection feature by evaluating the reflection image by using at least one processing unit, wherein the evaluation com- prises, for each reflection feature, an analysis of
  • Embodiment 2 The method according to the preceding embodiment, wherein the method comprises a self learning grasp sequence for robot bin picking, wherein the method comprises approaching the selected next pick point with the robot and determining, by at least one sensor of the robot, sensor data of the approached pick point relating to suitability for grasping, retraining the trained scoring model on the sensor data and re- peating at least steps c) and d) of the method, preferably steps a) to d).
  • Embodiment 3 The method according to the preceding embodiment, wherein the method comprises storing the determined sensor data in at least one database.
  • Embodiment 4 The method according to any one of the preceding embodiments, wherein the score for the respective pin point candidate depends on the probability to lead to a successful grasp, wherein the score is assigned from low score for low probability to high score for high probability, wherein the pick point with the highest score is used as next pick point.
  • Embodiment 5 The method according to any one of the preceding embodiments, wherein the method comprises simulating pick point approaches, wherein the processing unit sim- ulates for all pick point candidates whether approaching them leads to collisions with other items, wherein pick point candidates simulated to lead to collisions are filtered out for further consideration.
  • Embodiment 6 The method according to any one of the preceding embodiments, wherein the determining of material characteristics is based on a convolutional neural network classifying the reflection features of the reflection image.
  • Embodiment 7 The method according to any one of the preceding embodiments, wherein the material characteristics is at least one information selected from the group consisting of information about softness, information about deformability, or information about permeability to air.
  • step d) comprises the control unit to optimize one or more of the pick point selection, pick pose, depth offset and vacuum control of the grabber considering the determined material characteristics.
  • Embodiment 9 The method according to any one of the preceding embodiments, wherein the scoring model is a classification and/or regression model, wherein the scoring model is at least one model selected from the group consisting of a random forest (RF) or a convolution neural network (CNN).
  • step b) comprises the processing unit determining depth information for each reflec- tion feature by evaluating the reflection image, wherein the evaluation comprises, for each reflection feature, an analysis of its respective beam profile using a depth-from- photon ratio technique.
  • Embodiment 11 The method according to any one of the preceding embodiments, wherein in step c) in addition to the material characteristics one or more of the following input pa- rameters are provided to the scoring model: information about an image section, 3D information about the reflection features.
  • Embodiment 12 The method according to any one of the preceding embodiments, wherein an output of the scoring model is a probability distribution over classes, wherein the out- put is one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high.
  • the robot comprises at least one robot arm, wherein the robot arm is equipped with a vac- uum grabber and a vacuum sensor.
  • Embodiment 14 The method according to the preceding embodiment, wherein the control unit is configured for moving the robot arm to a picking position for the next pick consider- ing the assigned scores.
  • Embodiment 15 The method according to any one of the two preceding embodiments, wherein the method comprises the control unit controlling the vacuum sensor for approaching the pick point, wherein each grasp is labeled as positive if the vacuum sensor returns stable vacuum values after approaching the pick point or negative if vacuum cannot be maintained.
  • the projector comprises at least one laser light source, wherein the illumination pattern comprises a grid of laser spots.
  • Embodiment 17 The method according to the preceding embodiment, wherein the laser spots have wavelengths in a near infrared (NIR) regime.
  • Embodiment 18 The method according to any one of the preceding embodiments, wherein the camera comprises at least one CCD chip and/or at least one CMOS chip.
  • Embodiment 19 The method according to any one of the preceding embodiments, wherein the camera is or comprises at least one near infrared camera.
  • Embodiment 20 The method according to any one of the preceding embodiments, wherein in step b), multiple camera positions are computed that cover the entire carrier to ensure that all items can be picked.
  • Embodiment 21 Computer program for picking of items comprised in a carrier by using at least one robot, configured for causing a computer or a computer network to fully or partially perform the method according to any one of the preceding embodiments, when exe- cuted on the computer or the computer network, wherein the computer program is configured for performing and/or executing at least steps a) to d) of the method ac- cording to any one of the preceding embodiments.
  • Embodiment 22 A computer-readable storage medium comprising instructions which, when ex- ecuted by a computer or computer network, cause to carry out at least steps a) to d) of the method according to any one of the preceding embodiments referring to a method.
  • Embodiment 23 A pick system comprising - at least one robot configured for picking items comprised in a carrier; - at least one camera configured for imaging the carrier at a plurality of imaging posi- tions; - at least one projector configured for projecting at least one illumination pattern on a scene comprising the carrier, wherein the camera is configured for imaging at least one reflection image, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; - at least one processing unit configured for determining material characteristics for each reflection feature by evaluating the reflection image, wherein the evaluation com- prises, for each reflection feature, an analysis of its respective beam profile, wherein the processing unit is configured for assigning a score according to their probability to lead to a successful grasp to pick point candidates using a trained scoring model, wherein the determined material characteristics in a close environment around the pick point candidate are used as input for the trained scoring model; - at least one control unit configured for selecting the next pick point of the robot con-
  • Embodiment 24 The pick system according to the preceding embodiment, wherein the pick system is configured for performing the method for picking of items comprised in a carrier by using at least one robot according to any one of the preceding embodiments referring to a method.
  • the pick system is configured for performing the method for picking of items comprised in a carrier by using at least one robot according to any one of the preceding embodiments referring to a method.
  • Figure 1 shows an embodiment of a method according to the present invention
  • Figure 2 shows an embodiment of a picking system according to the present invention
  • Figure 3 shows an embodiment of a scoring model comprising a CNN network structure.
  • Figure 1 shows an embodiment of a computer implemented method for picking of items 110 comprised in a carrier 112 by using at least one robot 114 according to the present invention.
  • the carrier 112 may be a device configured for containing at least one item 110, in particular a plurality of items 110.
  • the carrier 112 may have support surface on which the items 110 can be placed.
  • the carrier 112 may have sidewalls.
  • the carrier 112 may be a box.
  • the item 110 may be arbitrary objects to be picked.
  • the items 110 may be made from at least one material selected from the group consisting of cardboard, glass, metal or plastic.
  • the items may comprise screws, bottles, food and the like.
  • the carrier 112 may comprise items of identical size, shape, material and reflection and/or absorption properties.
  • the carrier 112 may comprise items having varying size, shape, material and reflection and/or absorption properties.
  • the robot 114 may be configured for performing a task in the absence of external assistance or instructions, such as for example without assistance from a user or control by a user. Specifically, the robot 114 may be configured to react to specific situations independent from external input, e.g. by using pre-programmed routines, self-learning mechanisms or the like.
  • the robot 114 may be selected from the group consisting of a commercial robot; an industrial robot, specifically a manufacturing robot.
  • the robot 114 may comprise at least one robot arm 116 equipped with at least one end effector 118 at one end of the robot arm 116.
  • the robot arm 116 may be configured for moving the end effector 118 along three axis of movement.
  • the robot 114 may be configured for performing at least one gripping function.
  • the gripping function may comprise one or more of clamping, holding, tilting, lifting, positioning, mov- ing, handling, or transporting the item.
  • the gripping may comprise picking the item.
  • the picking may comprise one or more of selecting the item 110, interacting with a surface of the item 110 such as contacting, or lifting the item 110.
  • the gripping may comprise lifting and transporting items 110 to automate commissioning in a warehouses and/or in a manufactur- ing process.
  • the pick point may be point of interaction between the end effector 118 and the item 110.
  • the pick point may be a point on the surface of the item 110.
  • the end effector 118 may comprise at least one gripper 120.
  • the gripper 120 may be a vacuum grabber.
  • the vacuum grabber may be configured for applying an attractive force to a surface of an item to be gripped.
  • the vacuum grabber may comprise at least one vacuum cup, also denoted as suction cup, such as a rubber suction cup or polyurethane suction cup to pick up items 110.
  • the vacuum cups may have a round shape.
  • the vacuum cup may be configured for interacting with the surface of the item 110.
  • the vacuum grabber may comprise at least one vacuum pump configured for generating vacuum.
  • the vacuum grabber may comprise at least one pneumatic valve configured for sucking air out of the vacuum cup for attaching the item 110 to the vacuum cup, thereby picking up the item.
  • the robot arm 116 may be configured for moving the vacuum grabber with attached item to a further position, in particular outside the carrier. Other embodiments of a robot are possible.
  • An embodiment of a pick system 113 comprising a robot 114 is shown in Figure 2.
  • the pick system 113 further may comprise the camera 124, the projector 130, the flood light source 148, the processing unit 134 and the control unit 144.
  • the method steps as shown in Figure 1 may be performed in the given order or may be per- formed in a different order. Further, one or more additional method steps may be present which are not listed. Further, one, more than one or even all of the method steps may be performed repeatedly.
  • the display device as discussed above.
  • the method may comprise using the dis- play device according to the present invention, such as according to one or more of the embodi- ments given above or given in further detail below.
  • the method comprises the following steps: a) at least one carrier scanning step 122 comprising imaging the carrier 112 at a plurality of imaging positions by using at least one camera 124; b) at least one material detection step 126 comprising b1) (128) projecting at least one illumination pattern on a scene comprising the car- rier 112 by using at least one projector 130 and imaging at least one reflection image using the camera 124, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) (132) determining material characteristics for each reflection feature by evalu- ating the reflection image by using at least one processing unit 134, wherein the eval- uation comprises, for each reflection feature, an analysis of its respective beam profile; c) at least one pick point scoring step 136, wherein the pick point scoring step 126 com- prises the processing unit 134 assigning a score according to their probability to lead to a successful grasp to pick point candidates 138 using a
  • the scanning 122 may be or may comprise a process of imaging at a plurality of imaging positions 146.
  • the imaging positions may refer to different positions of the camera in space.
  • the scanning may be performed to account for trajectory planning and collision avoidance for the subsequent picking.
  • the imaging positions may be set by moving the robot arm 116.
  • the robot arm 116 may be equipped with the camera 124, the projector 130 and an optional flood light source 148.
  • Step a) may comprise illuminating the carrier 112, e.g. by using the at least one flood light source 148. Additionally or alternatively, ambient light may be used.
  • the flood light source 148 may be configured for providing the at least one illumination light beam for illumination of the carrier 112.
  • the flood light source 148 may be configured for scene illumination.
  • the scene illumination may comprise diffuse and/or uniform illumination of the scene.
  • the flood light source 148 may be adapted to directly or indirectly illuminating the carrier 112, wherein the illumination is reflected or scattered by surfaces of the carrier 112 and, thereby, is at least partially directed towards the camera 124.
  • the flood light source 148 may be adapted to illuminate the carrier 112, for example, by directing a light beam towards the carrier 112, which reflects the light beam.
  • the flood light source 148 may be configured for generating an illuminating light beam for illuminating the carrier 112.
  • the flood light source 148 may comprise at least one light-emitting-diode (LED).
  • the flood light source 148 may illuminate the scene with the LED and, in particular, without the illumination pat- tern, and the camera 124 may be configured for capturing a two-dimensional image of the scene.
  • the flood light source 148 may comprise a single light source or a plurality of light sources.
  • the light emitted by the flood light source 148 may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm.
  • light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 ⁇ m.
  • the light in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm may be used.
  • the flood light source 148 may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodi- ments, the flood light source 148 may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths channels.
  • the projector 130 and flood light source 148 may constitute a coordinate system, wherein a lon- gitudinal coordinate is a coordinate along the optical axis.
  • the coordinate system may be a po- lar coordinate system in which the optical axis forms a z-axis and in which a distance from the z- axis and a polar angle may be used as additional coordinates.
  • a direction parallel or antiparallel to the z-axis may be considered a longitudinal direction, and a coordinate along the z-axis may be considered a longitudinal coordinate z. Any direction perpendicular to the z-axis may be con- sidered a transversal direction, and the polar coordinate and/or the polar angle may be consid- ered a transversal coordinate. Depth information may relate to the longitudinal coordinate and/or information from which the longitudinal coordinate can be derived.
  • the projector 130 may comprise at least one illumination device configured for providing the at least one illumination pattern.
  • the projector 130 may be or may comprise at least one light source or at least one multiple beam light source.
  • the projector 130 may comprise at least one laser source and one or more diffractive optical elements (DOEs).
  • DOEs diffractive optical elements
  • the projector 130 may comprise at least one laser and/or laser source.
  • lasers may be employed, such as semi- conductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed bragg reflector lasers, polariton la- sers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grat- ing lasers, Indium Arsenide lasers, transistor lasers, diode pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, Gallium Arsenide lasers, semiconductor ring laser, extended cavity diode lasers, or vertical cavity surface-emitting lasers.
  • non-laser light sources may be used, such as LEDs, micro-light emitting diode (LED), and/or light bulbs.
  • the projector 130 may comprise one or more diffractive optical ele- ments (DOEs) configured for generating the illumination pattern.
  • DOEs diffractive optical ele- ments
  • the projector 130 may be adapted to generate and/or to project a cloud of points, for example the projector source may comprise one or more of at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light emitting diodes; at least one array of laser light sources.
  • the use of at least one la- ser source is particularly preferred.
  • the projector 130 and the flood light source 148 may be in- tegrated into a housing. Further, the projector 130 may be configured for emitting modulated or non-modulated light. In case a plurality of light sources is used, the different light sources may have different modula- tion frequencies which later on may be used for distinguishing the light beams.
  • the illumination pattern may be an arbitrary pattern comprising at least one illumination feature adapted to illuminate at least one part of the scene.
  • the illumination feature may be at least one at least partially extended feature of the pattern.
  • the illumination pattern may comprise a single illumination feature.
  • the illumination pattern may comprise a plurality of illumination features.
  • the illumination pattern may be selected from the group consisting of: at least one point pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern comprising an arrangement of periodic or non periodic features.
  • the illumination pattern may comprise regular and/or constant and/or periodic pattern such as a triangular pat- tern, a rectangular pattern, a hexagonal pattern or a pattern comprising further convex tilings.
  • the illumination pattern may exhibit the at least one illumination feature selected from the group consisting of: at least one point; at least one line; at least two lines such as parallel or crossing lines; at least one point and one line; at least one arrangement of periodic or non-periodic fea- ture; at least one arbitrary shaped featured.
  • the illumination pattern may comprise at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo- random point pattern; a random point pattern or a quasi random pattern; at least one Sobol pat- tern; at least one quasiperiodic pattern; at least one pattern comprising at least one pre-known feature at least one regular pattern; at least one triangular pattern; at least one hexagonal pat- tern; at least one rectangular pattern at least one pattern comprising convex uniform tilings; at least one line pattern comprising at least one line; at least one line pattern comprising at least two lines such as parallel or crossing lines.
  • a distance between two features, in particular spots, of the illumination pattern and/or an area of the at least one illumination feature may depend on the circle of confusion in the reflection image.
  • the projector 130 may be adapted to generate and/or to project a cloud of points.
  • the projector 130 may comprise the at least one light projector adapted to generate a cloud of points such that the illumination pattern may comprise a plurality of point pattern.
  • the projector 130 may comprise at least one mask adapted to generate the illumination pattern from at least one light beam generated by the light source.
  • the projector 130 comprises at least one laser light source, wherein the illumina- tion pattern comprises a grid of laser spots.
  • the projector 130 comprises at least one laser source which is designated for generating laser radiation.
  • the projector may comprise the at least one diffractive optical element (DOE).
  • DOE diffractive optical element
  • the projector 130 may be at least one point projector, such as the at least one laser source and the DOE, adapted to project at least one periodic point pattern.
  • the light emitted by the projector 130 may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 ⁇ m.
  • the laser spots may have wavelengths in a near infrared (NIR) regime. Specifically, the light in the part of the near infrared region where silicon photodi- odes are applicable specifically in the range of 700 nm to 1100 nm may be used.
  • NIR near infrared
  • the projector 130 may be configured for emitting light beams at a wavelength range from 800 to 1000 nm, preferably at 940 nm, since terrestrial sun radiation has a local minimum in irradiance at this wavelength, e.g. as described in CIE 085- 1989 compromiseSolar spectral Irradiance”.
  • the projector 130 may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodiments, the projector may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths channels.
  • the camera 124 may have at least one imaging element configured for recording or capturing spatially resolved one-dimensional, two-dimensional or even three-dimensional optical data or information.
  • the camera 124 may comprise at least one pixelated camera chip.
  • the camera 124 may comprise at least one camera chip, such as at least one CCD chip and/or at least one CMOS chip configured for recording images.
  • the image may be data recorded by using the camera 124, such as a plurality of electronic readings from the camera 124, such as the pixels of the camera chip.
  • the camera 124 may be or may comprise at least one photodetector, preferably inorganic pho- todetectors, more preferably inorganic semiconductor photodetectors, most preferably silicon photodetectors.
  • the camera 124 may be sensitive in the infrared spectral range.
  • the camera 124 may be or may comprise at least one inorganic photodiode which are sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 microme- ters. Specifically, the camera 124 may be sensitive in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm. Infrared opti- cal sensors which may be used for the camera 124 may be commercially available infrared opti- cal sensors, such as infrared optical sensors commercially available under the brand name HertzstueckTM from trinamiXTM GmbH, D-67056 Ludwigshafen am Rhein, Germany.
  • the camera 124 may comprise at least one optical sensor of an intrinsic photovol- taic type, more preferably at least one semiconductor photodiode selected from the group con- sisting of: a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode.
  • the cam- era 124 may comprise at least one optical sensor of an extrinsic photovoltaic type, more prefer- ably at least one semiconductor photodiode selected from the group consisting of: a Ge:Au pho- todiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, a Si:As photodiode.
  • the camera 124 may comprise at least one pho- toconductive sensor such as a PbS or PbSe sensor, a bolometer, preferably a bolometer se- lected from the group consisting of a VO bolometer and an amorphous Si bolometer.
  • the camera 124 may be sensitive in one or more of the ultraviolet, the visible or the infrared spectral range. Specifically, the camera 124 may be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably at 650 nm to 750 nm or at 690 nm to 700 nm. Specifically, the camera 124 may be sensitive in the near infrared region.
  • the camera 124 may be sensitive in the part of the near infrared region where silicon photodiodes are applicable spe- cifically in the range of 700 nm to 1000 nm.
  • the camera 124 specifically, may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 micrometers.
  • the camera 124 may be or may comprise at least one element selected from the group consist- ing of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photocon- ductor, a phototransistor or any combination thereof. Any other type of photosensitive element may be used.
  • the photosensitive element generally may fully or partially be made of inorganic materials and/or may fully or partially be made of organic materials.
  • one or more photodiodes may be used, such as commercially available photodiodes, e.g. inorganic semiconductor photodiodes.
  • the camera 124 may be a fix-focus camera, having at least one lens which is fixedly adjusted with respect to the camera. Alternatively, however, the camera 124 may also comprise one or more variable lenses which may be adjusted, automatically or manually.
  • the camera 124 is configured for imaging the carrier 112 at a plurality imaging positions 146 of the camera 124. The images may be generated in response to the diffuse and/or uniform illumi- nation of the carrier 112 by the flood light source 148.
  • the images generated in response to the diffuse and/or uniform illumination of the carrier 112 by the flood light source 148 may not com- prise any reflection features generated by the illumination pattern.
  • the image may be at least one two-dimensional image.
  • the image may be an RGB (red green blue) image.
  • the camera 124 is configured for imaging the at least one reflection image.
  • the reflection im- age comprises a plurality of reflection features generated by the scene in response to the illumi- nation pattern.
  • the reflection feature may be a feature in an image plane generated by the scene in response to illumination, specifically with at least one illumination feature.
  • Each of the reflection features comprises at least one beam profile, also denoted reflection beam profile.
  • the beam profile of the reflection feature may generally refer to at least one intensity distribution of the reflection feature, such as of a light spot on the camera 124, as a function of the pixel.
  • the beam profile may be selected from the group consisting of a trapezoid beam profile; a trian- gle beam profile; a conical beam profile and a linear combination of Gaussian beam profiles.
  • the evaluation of the reflection image may comprise identifying the reflection features of the re- flection image.
  • the processing unit 134 may be configured for performing at least one image analysis and/or image processing in order to identify the reflection features.
  • the image analysis and/or image processing may use at least one feature detection algorithm.
  • the image analysis and/or image processing may comprise one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between an image created by the sensor signals and at least one offset; an inversion of sensor signals by inverting an image cre- ated by the sensor signals; a formation of a difference image between an image created by the sensor signals at different times; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a blob detector; applying a corner detector; applying a Determinant of Hessian filter; applying a principle curvature-based region detector; applying a maximally stable extremal regions detector; applying a generalized Hough-transformation; ap- plying a ridge detector; applying an affine invariant feature detector; applying an affine-adapted interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector
  • the region of interest may be determined manually by a user or may be determined automatically, such as by recognizing a feature within the image generated by the camera 124.
  • the processing unit 134 may be configured for determining the beam profile of the respective reflection feature.
  • the determining the beam profile may comprise identifying at least one reflec- tion feature provided and/or selecting at least one reflection feature and evaluating at least one intensity distribution of the reflection feature.
  • a region of a matrix constituted by pixels of the camera 124 may be used and evaluated for determining the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the matrix.
  • a center of illumination by the light beam may be determined, such as by determining the at least one pixel having the highest illumination, and a cross-sectional axis may be chosen through the center of illumination.
  • the intensity distribution may an intensity distribution as a function of a coordinate along this cross- sectional axis through the center of illumination.
  • Other evaluation algorithms are feasible.
  • the reflection feature may cover or may extend over at least one pixel of the refelction image.
  • the reflection feature may cover or may extend over plurality of pixels.
  • the pro- cessing unit 134 may be configured for determining and/or for selecting all pixels connected to and/or belonging to the reflection feature, e.g. a light spot.
  • the processing unit 134 may be con- figured for determining the center of intensity by wherein R coi is a position of center of intensity, r pixel is the pixel position and with j being the number of pixels j connected to and/or belonging to the reflection feature and I total be- ing the total intensity.
  • the processing unit 134 is configured for determining material characteristics for each reflection feature by analysis of its respective beam profile.
  • the material characteristics may be an arbi- trary material property m derived from and/or relating to the beam profile of the reflection fea- ture.
  • the material characteristics may be at least one information selected from the group con- sisting of information about softness, information about deformability, or information about per- meability to air.
  • the material characteristics may be extracted for distinguishing between fore- ground and background, i.e. items 110 to pick vs. carrier 112.
  • the processing unit 134 may be configured for determining the material property m of the sur- face remitting the reflection feature by evaluating the beam profile of the reflection feature.
  • the material property may be at least one arbitrary property of the material configured for character- izing and/or identification and/or classification of the material.
  • the material prop- erty may be a property selected from the group consisting of: roughness, penetration depth of light into the material, a reflectivity, a specular reflectivity, a diffuse reflectivity, a surface prop- erty, a measure for translucence, a scattering, specifically a back-scattering behavior or the like.
  • the at least one material property may be a property selected from the group consisting of: a scattering coefficient, a translucency, a transparency, a deviation from a Lambertian surface re- flection, a speckle, and the like.
  • the processing unit 134 may be configured for identifying a reflection feature as to be gener- ated by an item 110 having a specific material property in case its reflection beam profile fulfills at least one predetermined or predefined criterion.
  • the at least one predetermined or prede- fined criterion may be at least one property and/or value suitable to distinguish material proper- ties.
  • the predetermined or predefined criterion may be or may comprise at least one predeter- mined or predefined value and/or threshold and/or threshold range referring to a material prop- erty.
  • the reflection feature may be indicated as to be generated by an item 110 having a spe- cific material property in case the reflection beam profile fulfills the at least one predetermined or predefined criterion.
  • the determining at least one material property may comprise assigning the material property to respective reflection feature.
  • the processing unit 134 may comprise at least one database com- prising a list and/or table, such as a lookup list or a lookup table, of predefined and/or predeter- mined material properties.
  • the list and/or table of material properties may be determined and/or generated by performing at least one test measurement, for example by performing material tests using samples having known material properties.
  • the list and/or table of material proper- ties may be determined and/or generated at the manufacturer site and/or by a user.
  • the mate- rial property may additionally be assigned to a material classifier such as one or more of a ma- terial name, a material group such as biological or non-biological material, translucent or non- translucent materials, metal or non-metal, fur or non-fur, carpet or non-carpet, reflective or non- reflective, specular reflective or non-specular reflective, foam or non-foam, roughness groups or the like.
  • the processing unit 134 may comprise at least one database comprising a list and/or table comprising the material properties and associated material name and/or material group.
  • beam profile analysis may be used for determining the material characteristic in step b). Spe- cifically, beam profile analysis makes use of reflection properties of coherent light projected onto object surfaces to classify materials.
  • the classification of materials may be performed as de- scribed in WO 2020/187719, in EP application 20159984.2 filed on February 28, 2020 and/or EP application 20154961.5 filed on January 31, 2020, the full content of which is included by reference.
  • a periodic grid of laser spots e.g. a hexagonal grid as described in EP application 20170905.2 filed on April 22, 2020, is projected and the reflection image is rec- orded with the camera.
  • Analyzing the beam profile of each reflection feature recorded by the camera may be performed by feature-based methods and/or using based on a convolutional neural network classifying the reflection features of the reflection image.
  • the feature based methods may be used in combination with machine learning methods which may allow para- metrization of a classification model.
  • Convolutional neuronal networks may be utilized to classify materials by using the reflection images as an input.
  • Step b) may comprise using artificial intelligence, in particular convolutional neuronal networks.
  • Using reflection images as input for convolutional neuronal networks may enable the generation of classification models with sufficient accuracy to differentiate between materials.
  • at least one parametrized classification model may be used.
  • the parametrized classi- fication model may be configured for classifying materials by using the reflection image as an input.
  • the classification model may be parametrized by using one or more of machine learning, deep learning, neural networks, or other form of artificial intelligence.
  • the classification model may be configured for discriminating materials.
  • the material characteristics may be determined by applying an optimization algorithm in terms of at least one optimization target on the classifi- cation model.
  • the machine learning may be based on at least one neuronal network, in particu- lar a convolutional neural network. Weights and/or topology of the neuronal network may be pre-determined and/or pre-defined.
  • the training of the classification model may be performed using machine-learning.
  • the classification model may comprise at least one ma- chine-learning architecture and model parameters.
  • the machine-learning architec- ture may be or may comprise one or more of: linear regression, logistic regression, random for- est, naive Bayes classifications, nearest neighbors, neural networks, convolutional neural net- works, generative adversarial networks, support vector machines, or gradient boosting algo- rithms or the like.
  • the classification model may be at least partially data-driven.
  • the classification model may be based on experimental data.
  • training of the classi- fication model may comprise using at least one training dataset, wherein the training data set comprises images, in particular refection images, of a plurality of items with known material property.
  • multiple camera positions 146 may be computed that cover the entire carrier 112 to ensure that all items 110 can be picked. These positions may be approached in a row until no item can be identified anymore and all items 110 are covered.
  • Step b) may comprise the processing unit 134 determining depth information for each reflection feature by evaluating the reflection image. The evaluation may comprises, for each reflection feature, an analysis of its respective beam profile using a depth-from-photon ratio technique.
  • the analysis of the beam profile comprise evaluating of the beam profile and may comprise at least one mathematical operation and/or at least one comparison and/or at least symmetrizing and/or at least one filtering and/or at least one normalizing.
  • the analysis of the beam profile may comprise at least one of a histogram analysis step, a calculation of a differ- ence measure, application of a neural network, application of a machine learning algorithm.
  • the processing unit 134 may be configured for symmetrizing and/or for normalizing and/or for filter- ing the beam profile, in particular to remove noise or asymmetries from recording under larger angles, recording edges or the like.
  • the processing unit 134 may filter the beam profile by re- moving high spatial frequencies such as by spatial frequency analysis and/or median filtering or the like. Summarization may be performed by center of intensity of the light spot and averaging all intensities at the same distance to the center.
  • the processing unit 134 may be configured for normalizing the beam profile to a maximum intensity, in particular to account for intensity differ- ences due to the recorded distance.
  • the processing unit 134 may be configured for removing influences from background light from the beam profile, for example, by an imaging without illu- mination.
  • the processing unit 134 may be configured for determining at least one longitudinal coordinate zDPR for each of the reflection features by analysis of the beam profile of the respective reflection feature.
  • the processing unit 134 may be configured for determining the longitudinal coordinate z DPR for the reflection features by using the so called depth-from-photon-ratio technique, also denoted as beam profile analysis.
  • depth-from-photon-ratio (DPR) technique ref- erence is made to WO 2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, the full content of which is included by reference.
  • the method may comprise simulating pick point approaches 150.
  • the processing unit 134 may be configured for simulating for all pick point candidates 138 whether approaching them leads to collisions with other items, e.g. carrier 112, robot 114 or other items 110, wherein pick point candidates 138 simulated to lead to collisions are filtered out for further consideration.
  • the sim- ulation may be performed prior to pick point scoring 136.
  • the processing unit 134 is configured for assigning a score to pick point candidates 138 according to their probability to lead to a successful grasp using the trained scoring model 140.
  • a successful grasp may be if the pick point candidate 138 is suitable for al- lowing stable vacuum.
  • the grasp may be not successful if vacuum cannot be maintained at the respective pick point candidate.
  • the probability to lead to a successful grasp may depend on one or more of softness, deformability, permeability to air and the like.
  • the score may be a value assigned to the respective pick point candidate 138, wherein the value depends on the probability to lead to a successful grasp. For example, the score may range from 0 to 1.
  • the score may be assigned from low score for low probability to high score for high probability.
  • the pick point with the highest score is used as next pick point.
  • the scoring model 140 may be or may comprise classification model and/or a regression model having as output a probability distribution over classes for the reflection features.
  • the output of the scoring model 140 may be, for each of the reflection features, one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high.
  • step c) in addition to the material characteristics, one or more of the following input parame- ters are provided to the scoring model 140: information about an image section, 3D information about the reflection features.
  • the trained scoring model 140 is configured for generating using the input at least one output, in particular a prediction.
  • the scoring model 140 may be at least one model selected from the group consisting of a ran- dom forest (RF) or a convolution neural network (CNN).
  • RF ran- dom forest
  • CNN convolution neural network
  • the convolutional neural network may be a multilayer convolutional neural network.
  • the convolutional neural network may com- prise a plurality of convolutional layers.
  • the convolutional layers may be followed by a plurality of fully-connected layers.
  • the convolutional neural network may comprise a plurality of pooling layers.
  • the structure of convolutional neural networks is generally known to the skilled person such as from en.wikipedia.org/wiki/Convolutional_neural_network#Convolutional.
  • the CNN may be build by using the Keras library in Python.
  • Keras library in Python reference is made to https://keras.io/ or https://de.wikipedia.org/wiki/Keras.
  • material characteristics and 3D information may be fed 154 into a dense layer with a Scaled Exponential Linear Unit (SELU) activation function 156.
  • SELU Scaled Exponential Linear Unit
  • the dense layer 156 may be followed by a batch normalization layer 158.
  • the images de- termined in step a) may be fed 160, in a second branch of the CNN 162, into a 5x5 convolution kernel 164 with 16 filters.
  • the convolution kernel 164 is followed by at least one batch normali- zation layer 166, a 2x2 Max Pooling layer 168 and a dense layer with a SELU activation func- tion 170.
  • the output of first and second branches may be fed into a dense layer with a SELU activation function 172 followed by a dense layer having a sigmoid activation function 174.
  • the output 176 may be a probability distribution over classes for the reflection features.
  • the output 176 of the scoring model 140 may be, for each of the reflection features, one score number be- tween 0, in case probability for successful grasp is low, and 1, in case the probability for a suc- cessful grasp is high.
  • the trained scoring model 140 was trained using at least one training dataset.
  • the training da- taset may comprise a plurality of reflection images having reflection features of known probabil- ity for a successful grasp.
  • the training may be performed for different items.
  • the trained scoring model 140 can be re-trained and/or updated based on additional data.
  • the trained scoring model 140 may be trained by using machine learning.
  • the method may comprise at least one training step, wherein, in the training step, the scoring model 140 is trained on the at least one training dataset. Specifically, the training step may be performed before performing step c).
  • step c) the determined material characteristics in a close environment around the pick point candidate 138 are used as input for the trained scoring model 140.
  • the close environment around the pick point candidate 138 may be or may comprise reflection features within a range or region around the pick point candidate 138.
  • the pick point candidate 138 may coincide with a reflection feature. However, other embodiments are possible wherein the pick point candidate 138 and the reflection feature do not coincide.
  • the processing unit 134 may select a range or image region around the pick point candidate 138 ensuring that at least one reflection feature is present, e.g. the reflection feature nearest to the pick point candidate 138.
  • the processing unit 134 may be configured for considering the images imaged in step a) and the reflection image for identifying reflection features in a close environment around the pick point candidate 138. Once the scoring is completed, the pick point with the highest score may be taken for the next pick.
  • the control unit 144 is configured for selecting the next pick point of the robot 114 consid- ering the assigned scores.
  • the method may comprises a self-learning grasp sequence 178 for robot bin picking.
  • the method may comprise in step d) approaching 180 the selected next pick point with the robot 114 and determining, by at least one sensor 182 of the robot 114, sensor data of the approached pick point relating to suitability for grasping.
  • the processing unit 134 may be configured for distinguishing between suitable pick points and non-suitable pick points.
  • the method may further comprise retraining 184 the trained scoring model on the sensor data and repeating at least steps c) and d) of the method, preferably steps a) to d).
  • the method may comprise storing the determined sensor data in at least one database.
  • the model accu- mulates information with every pick and learns from it.
  • the robot 114 comprises the at least one robot arm 116, wherein the robot arm 116 is equipped with a vacuum grabber and a vacuum sensor.
  • the control unit 144 may be con- figured for moving the robot arm 116 to a picking position for the next pick considering the as- signed scores.
  • the method may comprise the control unit 144 controlling the vacuum sensor for approaching the pick point, wherein each grasp is labeled as positive if the vacuum sensor re- turns stable vacuum values after approaching the pick point 186 or negative if vacuum cannot be maintained 188. In case of negative success, data relating to 2D, 3D, material information may be dumped and the CNN may be retrained. The next pick point with the next highest score may be approach and tested next 190.
  • Step d) may comprise the control unit to optimize one or more of the pick point selection, pick pose, depth offset and vacuum control of the gripper 120 considering the determined material characteristics.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

A computer implemented method for picking of items (110) comprised in a carrier (112) by using at least one robot (114) is disclosed. The method comprises the following steps: a) at least one carrier scanning step (122) comprising imaging the carrier (112) at a plurality of imaging positions (146) by using at least one camera (124); b) at least one material detection step (126) comprising b1) projecting (128) at least one illumination pattern on a scene comprising the carrier (112) by using at least one projector (130) and imaging at least one reflection image using the cam- era (124), wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) determining material characteristics (132) for each reflection feature by evaluating the reflection image by using at least one processing unit (134), wherein the evaluation comprises, for each reflection feature, an analysis of its respective beam profile; c) at least one pick point scoring step (136), wherein the pick point scoring step (136) comprises the processing unit (134) assigning a score according to their probability to lead to a successful grasp to pick point candidates (138) using a trained scoring model (140), wherein the determined material characteristics in a close environment around the pick point candi- date (138) are used as input for the trained scoring model (140); d) at least one picking step (142) comprising at least one control unit (144) selecting the next pick point of the robot (114) considering the assigned scores.

Description

Self learning grasp sequence for robot bin picking Description Field of the invention The invention relates to a computer implemented method for picking of items comprised in a carrier by using at least one robot and a pick system. The devices, methods and uses according to the present invention specifically may be employed for example in various areas of daily life, security technology, production technology, safety technology, agriculture, maintenance, cos- metics, medical technology or in the sciences. However, other applications are also possible. Prior art Robots with grippers can lift and transport a broad variety of objects to automate commissioning in warehouses or manufacturing processes in industry. Therefore, robots are equipped with 3D vision sensors in order to detect objects and/or suitable pick points where the gripper can grasp objects. However, objects are often in disorganized, and due to sensor noise, obstructions, and occlusions, automated bin picking can be extremely challenging. Moreover, object size, shape, material and reflection and/or absorption properties can vary greatly. In this respect, picking al- gorithms are often tailored to specific objects or object families and cannot be generalized easily. Advances in hardware and algorithms in the field of machine learning have shifted the focus from geometrically driven methods (such as CAD matching) to data-driven methods. Prominent exam- ples are Single-, Multi-Grasp Detection, see e.g. J. Redmon and A. Angelova, Real-time grasp detection using convolutional neural networks. CoRR, abs/1412.3128, 2014. [27] S. Ren, I. Lenz, H. Lee, and A. Saxena, Deep learning for detecting robotic grasps, in RSS, 2013 and J. Mahler, M. Matl, X. Liu, A. Li, D. V. Gealy, and K. Goldberg, Dex-Net 3.0: Computing Robust Robot Suc- tion Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning. CoRR, abs/1709.06670, 2017. Even hand-eye coordination can be learned from data if the size of the database is sufficient in an end-to-end learning manner S. James, A. J. Davison, and E. Johns, Transferring end-to-end visuomotor control from simula-tion to real world for a multi-stage task. In CoRL, 2017. However, one major drawback is the demand for huge amounts of data to ensure that the network training converges properly. Another strategy uses simulated data such as J. Mahler, M. Matl, X. Liu, A. Li, D. V. Gealy, and K. Goldberg, Dex-Net 3.0: Computing Robust Robot Suction Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning. CoRR, abs/1709.06670, 2017. Data is generated from thousands of 3D models in randomized poses on a table and with varying light conditions. Simulated data avoid time-consuming collection of real-world grasps and corresponding manual labeling, see J. Mahler, M. Matl, X. Liu, A. Li, D. V. Gealy, and K. Goldberg, Dex-Net 3.0: Com- puting Robust Robot Suction Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning. CoRR, abs/1709.06670, 2017, but the approach is still limited to a set of trained objects and cannot generalize to unseen objects. Therefore, start-ups such as Cambrian Intelli- gence focus on tailored solutions. The customer needs to upload relevant 3D models of the de- sired target objects and define environment settings for the simulation. After performing the sim- ulated training, the customer receives the trained network that can be tested with real objects to evaluate whether the performance is satisfactory or if further optimization needs to be done. The process may allow adapting networks to almost any scenario, even the orientation of the object can be derived from the network. However, network training is performed manually. The simula- tion needs to be carefully configured from the customer. Moreover, the network is adapted to a single setting (e.g. camera hardware, distance camera to scene, background). Any deviation from the training settings will lead to false detections or undetected objects. Deep reinforcement learning is one of the most promising directions to achieve intelligent robotic behavior. Pioneers in the field of reinforcement learning for robotics are Google and UC Berkeley, see e.g. Kalashnikov D, Irpan A, Pastor P, et al., QT-Opt: Scalable Deep Reinforcement Learning for Vi-sion-Based Robotic Manipulation. ArXiv e-prints, 2018. The robot continuously updates its grasp strategy based on the most recent observations. The authors reported 96% grasp success on unseen objects built on a training set comprising about 1,000 visually and physically diverse objects. While the results are promising, reinforcement learning for robotics is still in its infancy and learned tasks are still rather simple and specific. Other approaches built self-learning pick systems that are already available on the market. They claim that the system can independently learn from failed picks or drops without any manual intervention or optimization. The major drawback is still the lack of sensor information. Additional material characteristics fundamentally impacts approaching and pick strategies. Problem addressed by the invention It is therefore an object of the present invention to provide devices and methods facing the above-mentioned technical challenges of known devices and methods. Specifically, it is an ob- ject of the present invention to provide devices and methods which allow picking of arbitrary items with a low technical effort and with low requirements in terms of technical resources and cost. Summary of the invention This problem is solved by the invention with the features of the independent patent claims. Ad- vantageous developments of the invention, which can be realized individually or in combination, are presented in the dependent claims and/or in the following specification and detailed embodi- ments. As used in the following, the terms “have”, “comprise” or “include” or any arbitrary grammatical variations thereof are used in a non-exclusive way. Thus, these terms may both refer to a situa- tion in which, besides the feature introduced by these terms, no further features are present in the entity described in this context and to a situation in which one or more further features are present. As an example, the expressions “A has B”, “A comprises B” and “A includes B” may both refer to a situation in which, besides B, no other element is present in A (i.e. a situation in which A solely and exclusively consists of B) and to a situation in which, besides B, one or more further elements are present in entity A, such as element C, elements C and D or even further elements. Further, it shall be noted that the terms “at least one”, “one or more” or similar expressions indi- cating that a feature or element may be present once or more than once typically will be used only once when introducing the respective feature or element. In the following, in most cases, when referring to the respective feature or element, the expressions “at least one” or “one or more” will not be repeated, non-withstanding the fact that the respective feature or element may be present once or more than once. Further, as used in the following, the terms "preferably", "more preferably", "particularly", "more particularly", "specifically", "more specifically" or similar terms are used in conjunction with op- tional features, without restricting alternative possibilities. Thus, features introduced by these terms are optional features and are not intended to restrict the scope of the claims in any way. The invention may, as the skilled person will recognize, be performed by using alternative fea- tures. Similarly, features introduced by "in an embodiment of the invention" or similar expres- sions are intended to be optional features, without any restriction regarding alternative embodi- ments of the invention, without any restrictions regarding the scope of the invention and without any restriction regarding the possibility of combining the features introduced in such a way with other optional or non-optional features of the invention. In a first aspect of the present invention, a computer implemented method for picking of items comprised in a carrier by using at least one robot is disclosed. The term "computer implemented method" as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a method involving at least one computer and/or at least one computer network. The computer and/or computer network may comprise at least one processor which is con-figured for performing at least one of the method steps of the method according to the present invention. Specifically, each of the method steps is performed by the computer and/or computer network. The method may be performed completely automatically, specifically without user interaction. The term “carrier” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a device configured for contain- ing at least one item, in particular a plurality of items. The carrier may have support surface on which the items can be placed. The carrier may have sidewalls. For example, the carrier may be a box. The term “item” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to arbitrary objects to be picked. The items may be made from at least one material selected from the group consisting of card- board, glass, metal or plastic. For example, the items may comprise screws, bottles, food and the like. The carrier may comprise items of identical size, shape, material and reflection and/or absorption properties. The carrier may comprise items having varying size, shape, material and reflection and/or absorption properties. The term “robot” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an arbitrary device configured for performing at least one task autonomously. The robot may be configured for performing the task in the absence of external assistance or instructions, such as for example without assistance from a user or control by a user. Specifically, the robot may be configured to react to specific situations independent from external input, e.g. by using pre-programmed routines, self-learning mecha- nisms or the like. The robot may be selected from the group consisting of a commercial robot; an industrial robot, specifically a manufacturing robot. The robot may comprise at least one robot arm equipped with at least one end effector at one end of the robot arm. The robot arm may be configured for moving the end effector along three axis of movement. The robot may be configured for performing at least one gripping function. The terms gripping and grasping may be used as synonyms in the following. For example, the gripping function may comprise one or more of clamping, holding, tilting, lifting, positioning, moving, handling, or trans- porting the item. Specifically, the gripping may comprise picking the item. The term “picking” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to one or more of selecting the item, interacting with a surface of the item such as contacting, or lifting the item. For example, the gripping may comprise lifting and transporting items to automate commissioning in a warehouses and/or in a manufac- turing process. The term “pick point” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to point of interaction between the end effector and the item. The pick point may be a point on the surface of the item. The end effector may comprise at least one gripper. For example, the gripper may be a vacuum grabber. The vacuum grabber may be configured for applying an attractive force to a surface of an item to be gripped. The vacuum grabber may comprise at least one vacuum cup, also denoted as suction cup, such as a rubber suction cup or polyurethane suction cup to pick up items. The vacuum cups may have a round shape. The vacuum cup may be configured for interacting with the surface of the item. The vacuum grabber may comprise at least one vacuum pump configured for generating vacuum. The vacuum grabber may comprise at least one pneumatic valve config- ured for sucking air out of the vacuum cup for attaching the item to the vacuum cup, thereby picking up the item. The robot arm may be configured for moving the vacuum grabber with at- tached item to a further position, in particular outside the carrier. Other embodiments of a robot are possible. The method steps may be performed in the given order or may be performed in a different or- der. Further, one or more additional method steps may be present which are not listed. Further, one, more than one or even all of the method steps may be performed repeatedly. For details, options and definitions, reference may be made to the display device as discussed above. Thus, specifically, as outlined above, the method may comprise using the display device ac- cording to the present invention, such as according to one or more of the embodiments given above or given in further detail below. The method comprises the following steps: a) at least one carrier scanning step comprising imaging the carrier at a plurality of imag- ing positions by using at least one camera; b) at least one material detection step comprising b1) projecting at least one illumination pattern on a scene comprising the carrier by using at least one projector and imaging at least one reflection image using the cam- era, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) determining material characteristics for each reflection feature by evaluating the reflection image by using at least one processing unit, wherein the evaluation com- prises, for each reflection feature, an analysis of its respective beam profile; c) at least one pick point scoring step, wherein the pick point scoring step comprises the processing unit assigning a score according to their probability to lead to a successful grasp to pick point candidates using a trained scoring model, wherein the determined material characteristics in a close environment around the pick point candidate are used as input for the trained scoring model; d) at least one picking step comprising at least one control unit selecting the next pick point of the robot considering the assigned scores. The term “scanning” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a process of imaging at a plurality of imaging positions. The imaging positions may refer to different positions of the camera in space. The scanning may be performed to account for trajectory planning and collision avoidance for the subsequent picking. The imaging positions may be set by moving the robot arm. The robot arm may be equipped with the camera, the projector and an optional flood light source. Step a) may comprise illuminating the carrier, e.g. by using at least one flood light source. The term “flood light source” as used herein, is a broad term and is to be given its ordinary and cus- tomary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to at least one arbitrary device adapted to provide the at least one illumination light beam for illumination of the carrier. The flood light source may be configured for scene illumination. As used herein, the term “scene illumination” may refer to diffuse and/or uniform illumination of the scene. The flood light source may be adapted to directly or indirectly illuminating the carrier, wherein the illumination is reflected or scattered by surfaces of the carrier and, thereby, is at least partially directed towards the cam- era. The flood light source may be adapted to illuminate the carrier, for example, by directing a light beam towards the carrier, which reflects the light beam. The flood light source may be con- figured for generating an illuminating light beam for illuminating the carrier. The flood light source may comprise at least one light-emitting-diode (LED). The flood light source may illuminate the scene with the LED and, in particular, without the illumination pattern, and the camera may be configured for capturing a two-dimensional image of the scene. The flood light source may comprise a single light source or a plurality of light sources. As an example, the light emitted by the flood light source may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 µm. Specifically, the light in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm may be used. The flood light source may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodiments, the flood light source may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths channels. The projector and flood light source may constitute a coordinate system, wherein a longitudinal coordinate is a coordinate along the optical axis. The coordinate system may be a polar coordi- nate system in which the optical axis forms a z-axis and in which a distance from the z-axis and a polar angle may be used as additional coordinates. A direction parallel or antiparallel to the z- axis may be considered a longitudinal direction, and a coordinate along the z-axis may be con- sidered a longitudinal coordinate z. Any direction perpendicular to the z-axis may be considered a transversal direction, and the polar coordinate and/or the polar angle may be considered a transversal coordinate. As used herein, the term “depth information” may relate to the longitudi- nal coordinate and/or information from which the longitudinal coordinate can be derived. The term “projector” as used herein, is a broad term and is to be given its ordinary and custom- ary meaning to a person of ordinary skill in the art and is not to be limited to a special or cus- tomized meaning. The term specifically may refer, without limitation, to at least one illumination device configured for providing the at least one illumination pattern. The projector may be or may comprise at least one light source or at least one multiple beam light source. For example, the projector may comprise at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the projector may comprise at least one laser and/or laser source. Various types of lasers may be employed, such as semiconductor la- sers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed bragg reflector lasers, polariton lasers, hybrid sili- con lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, In- dium Arsenide lasers, transistor lasers, diode pumped lasers, distributed feedback lasers, quan- tum well lasers, interband cascade lasers, Gallium Arsenide lasers, semiconductor ring laser, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alterna- tively, non-laser light sources may be used, such as LEDs, micro-light emitting diode (LED), and/or light bulbs. The projector may comprise one or more diffractive optical elements (DOEs) configured for generating the illumination pattern. For example, the projector may be adapted to generate and/or to project a cloud of points, for example the projector source may comprise one or more of at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light emitting diodes; at least one array of laser light sources. On account of their generally defined beam profiles and other properties of handleability, the use of at least one laser source is partic- ularly preferred. The projector and the flood light source may be integrated into a housing. Further, the projector may be configured for emitting modulated or non-modulated light. In case a plurality of light sources is used, the different light sources may have different modulation fre- quencies which later on may be used for distinguishing the light beams. The light beam or light beams generated by the projector generally may propagate parallel to the optical axis or tilted with respect to the optical axis, e.g. including an angle with the optical axis. The projector may be configured such that the light beam or light beams propagates from the projector towards the scene along an optical axis. For this purpose, the projector may com- prise at least one reflective element, preferably at least one prism, for deflecting the illuminating light beam onto the optical axis. As an example, the light beam or light beams, such as the laser light beam, and the optical axis may include an angle of less than 10°, preferably less than 5° or even less than 2°. Other embodiments, however, are feasible. Further, the light beam or light beams may be on the optical axis or off the optical axis. As an example, the light beam or light beams may be parallel to the optical axis having a distance of less 10 than 10 mm to the optical axis, preferably less than 5 mm to the optical axis or even less than 1 mm to the optical axis or may even coincide with the optical axis. As used herein, the term “at least one illumination pattern” refers to at least one arbitrary pattern comprising at least one illumination feature adapted to illuminate at least one part of the scene. As used herein, the term “illumination feature” refers to at least one at least partially extended feature of the pattern. The illumination pattern may comprise a single illumination feature. The illumination pattern may comprise a plurality of illumination features. The illumination pattern may be selected from the group consisting of: at least one point pattern; at least one line pat- tern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern compris- ing an arrangement of periodic or non periodic features. The illumination pattern may comprise regular and/or constant and/or periodic pattern such as a triangular pattern, a rectangular pat- tern, a hexagonal pattern or a pattern comprising further convex tilings. The illumination pattern may exhibit the at least one illumination feature selected from the group consisting of: at least one point; at least one line; at least two lines such as parallel or crossing lines; at least one point and one line; at least one arrangement of periodic or non-periodic feature; at least one ar- bitrary shaped featured. The illumination pattern may comprise at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pat- tern; a random point pattern or a quasi random pattern; at least one Sobol pattern; at least one quasiperiodic pattern; at least one pattern comprising at least one pre-known feature at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern at least one pattern comprising convex uniform tilings; at least one line pat- tern comprising at least one line; at least one line pattern comprising at least two lines such as parallel or crossing lines. A distance between two features, in particular spots, of the illumination pattern and/or an area of the at least one illumination feature may depend on the circle of confu- sion in the reflection image. For example, the projector may be adapted to generate and/or to project a cloud of points. The projector may comprise the at least one light projector adapted to generate a cloud of points such that the illumination pattern may comprise a plurality of point pattern. The projector may comprise at least one mask adapted to generate the illumination pattern from at least one light beam generated by the light source. For example, the projector comprises at least one laser light source, wherein the illumination pattern comprises a grid of laser spots. For example, the projector comprises at least one laser source which is designated for generating laser radiation. The projector may comprise the at least one diffractive optical element (DOE). The projector may be at least one point projector, such as the at least one laser source and the DOE, adapted to project at least one periodic point pattern. As further used herein, the term “projecting at least one illumination pattern” refers to providing the at least one illumination pattern for illuminating the at least one scene. As used herein, the term “ray” generally refers to a line that is perpendicular to wavefronts of light which points in a direction of energy flow. As used herein, the term “beam” generally refers to a collection of rays. In the following, the terms “ray” and “beam” will be used as synonyms. As further used herein, the term “light beam” generally refers to an amount of light, specifically an amount of light traveling essentially in the same direction, including the possibility of the light beam having a spreading angle or widening angle. The light beam may have a spatial exten- sion. Specifically, the light beam may have a non-Gaussian beam profile. The beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile. The trapezoid beam profile may have a plateau region and at least one edge region. The light beam specifically may be a Gaussian light beam or a linear combination of Gaussian light beams, as will be outlined in further detail below. Other embodiments are fea- sible, however. The light emitted by the projector may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 µm. The laser spots may have wavelengths in a near infrared (NIR) regime. Specifically, the light in the part of the near infrared region where silicon photodi- odes are applicable specifically in the range of 700 nm to 1100 nm may be used. For example, the projector may be configured for emitting light beams at a wavelength range from 800 to 1000 nm, preferably at 940 nm, since terrestrial sun radiation has a local minimum in irradiance at this wavelength, e.g. as described in CIE 085- 1989 „Solar spectral Irradiance”. The projector may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodiments, the projector may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths chan- nels. The term “scene” as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to may refer to at least one arbitrary object or spatial region. The scene may comprise the carrier and a surrounding environment. The term "camera" as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a device having at least one im- aging element configured for recording or capturing spatially resolved one-dimensional, two-di- mensional or even three-dimensional optical data or information. The camera may comprise at least one pixelated camera chip. As an example, the camera may comprise at least one camera chip, such as at least one CCD chip and/or at least one CMOS chip configured for recording im- ages. As used herein, without limitation, the term “image” specifically may relate to data rec- orded by using a camera, such as a plurality of electronic readings from the camera, such as the pixels of the camera chip. The camera may comprise at least one optical sensor, in particular a plurality of optical sensors. The optical sensor has at least one light sensitive area. As used herein, an “optical sensor” gen- erally refers to a light-sensitive device for detecting a light beam, such as for detecting an illumi- nation and/or a light spot generated by at least one light beam. As further used herein, a “light- sensitive area” generally refers to an area of the optical sensor which may be illuminated exter- nally, by the at least one light beam, in response to which illumination at least one sensor signal is generated. The light-sensitive area may specifically be located on a surface of the respective optical sensor. Other embodiments, however, are feasible. The camera may comprise a plural- ity of optical sensors each having a light sensitive area. As used herein, the term “the optical sensors each having at least one light sensitive area” refers to configurations with a plurality of single optical sensors each having one light sensitive area and to configurations with one com- bined optical sensor having a plurality of light sensitive areas. The term “optical sensor” further- more refers to a light-sensitive device configured to generate one output signal. In case the camera comprises a plurality of optical sensors, each optical sensor may be embodied such that precisely one light-sensitive area is present in the respective optical sensor, such as by providing precisely one light-sensitive area which may be illuminated, in response to which illu- mination precisely one uniform sensor signal is created for the whole optical sensor. Thus, each optical sensor may be a single area optical sensor. The use of the single area optical sensors, however, renders the setup of the display device specifically simple and efficient. Thus, as an example, commercially available photo-sensors, such as commercially available silicon photodi- odes, each having precisely one sensitive area, may be used in the set-up. Other embodiments, however, are feasible. Preferably, the light sensitive area may be oriented essentially perpendicular to an optical axis. The optical axis may be a straight optical axis or may be bent or even split, such as by using one or more deflection elements and/or by using one or more beam splitters, wherein the es- sentially perpendicular orientation, in the latter cases, may refer to the local optical axis in the respective branch or beam path of the optical setup. The optical sensor specifically may be or may comprise at least one photodetector, preferably inorganic photodetectors, more preferably inorganic semiconductor photodetectors, most prefer- ably silicon photodetectors. Specifically, the optical sensor may be sensitive in the infrared spectral range. All pixels of the matrix or at least a group of the optical sensors of the matrix specifically may be identical. Groups of identical pixels of the matrix specifically may be pro- vided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Further, the pixels may be identical in size and/or with regard to their electronic or optoelec- tronic properties. Specifically, the optical sensor may be or may comprise at least one inorganic photodiode which are sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the optical sensor may be sensitive in the part of the near infra- red region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm. Infrared optical sensors which may be used for optical sensors may be commercially availa- ble infrared optical sensors, such as infrared optical sensors commercially available under the brand name HertzstueckTM from trinamiXTM GmbH, D-67056 Ludwigshafen am Rhein, Ger- many. Thus, as an example, the optical sensor may comprise at least one optical sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodi- ode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode. Additionally or alterna- tively, the optical sensor may comprise at least one optical sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, a Si:As photodiode. Additionally or alternatively, the optical sensor may com- prise at least one photoconductive sensor such as a PbS or PbSe sensor, a bolometer, prefera- bly a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bo- lometer. The optical sensor may be sensitive in one or more of the ultraviolet, the visible or the infrared spectral range. Specifically, the optical sensor may be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably at 650 nm to 750 nm or at 690 nm to 700 nm. Specifi- cally, the optical sensor may be sensitive in the near infrared region. Specifically, the optical sensor may be sensitive in the part of the near infrared region where silicon photodiodes are ap- plicable specifically in the range of 700 nm to 1000 nm. The optical sensor, specifically, may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 micrometers. For example, the optical sensor each, independently, may be or may comprise at least one ele- ment selected from the group consisting of a photodiode, a photocell, a photoconductor, a pho- totransistor or any combination thereof. For example, the camera may be or may comprise at least one element selected from the group consisting of a CCD sensor element, a CMOS sen- sor element, a photodiode, a photocell, a photoconductor, a phototransistor or any combination thereof. Any other type of photosensitive element may be used. The photosensitive element generally may fully or partially be made of inorganic materials and/or may fully or partially be made of organic materials. Most commonly, one or more photodiodes may be used, such as commercially available photodiodes, e.g. inorganic semiconductor photodiodes. The camera may comprise at least one sensor element comprising a matrix of pixels. Thus, as an example, the optical sensor may be part of or constitute a pixelated optical device. For ex- ample, the camera may be and/or may comprise at least one CCD and/or CMOS device. As an example, the optical sensor may be part of or constitute at least one CCD and/or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. As used herein, the term “sensor element” generally refers to a device or a combination of a plurality of devices configured for sensing at least one parameter. In the present case, the pa- rameter specifically may be an optical parameter, and the sensor element specifically may be an optical sensor element. The sensor element may be formed as a unitary, single device or as a combination of several devices. The sensor element comprises a matrix of optical sensors. The sensor element may comprise at least one CMOS sensor. The matrix may be composed of independent pixels such as of independent optical sensors. Thus, a matrix of inorganic photodi- odes may be composed. Alternatively, however, a commercially available matrix may be used, such as one or more of a CCD detector, such as a CCD detector chip, and/or a CMOS detector, such as a CMOS detector chip. Thus, generally, the sensor element may be and/or may com- prise at least one CCD and/or CMOS device and/or the optical sensors may form a sensor array or may be part of a sensor array, such as the above-mentioned matrix. Thus, as an example, the sensor element may comprise an array of pixels, such as a rectangular array, having m rows and n columns, with m, n, independently, being positive integers. Preferably, more than one column and more than one row is given, i.e. n>1, m>1. Thus, as an example, n may be 2 to 16 or higher and m may be 2 to 16 or higher. Preferably, the ratio of the number of rows and the number of columns is close to 1. As an example, n and m may be selected such that 0.3 ≤ m/n ≤ 3, such as by choosing m/n = 1:1, 4:3, 16:9 or similar. As an example, the array may be a square array, having an equal number of rows and columns, such as by choosing m=2, n=2 or m=3, n=3 or the like. The matrix may be composed of independent pixels such as of independent optical sensors. Thus, a matrix of inorganic photodiodes may be composed. Alternatively, however, a commer- cially available matrix may be used, such as one or more of a CCD detector, such as a CCD de- tector chip, and/or a CMOS detector, such as a CMOS detector chip. Thus, generally, the opti- cal sensor may be and/or may comprise at least one CCD and/or CMOS device and/or the opti- cal sensors of the display device may form a sensor array or may be part of a sensor array, such as the above-mentioned matrix. The matrix specifically may be a rectangular matrix having at least one row, preferably a plural- ity of rows, and a plurality of columns. As an example, the rows and columns may be oriented essentially perpendicular. As used herein, the term “essentially perpendicular” refers to the con- dition of a perpendicular orientation, with a tolerance of e.g. ±20° or less, preferably a tolerance of ±10° or less, more preferably a tolerance of ±5° or less. Similarly, the term “essentially paral- lel” refers to the condition of a parallel orientation, with a tolerance of e.g. ±20° or less, prefera- bly a tolerance of ±10° or less, more preferably a tolerance of ±5° or less. Thus, as an example, tolerances of less than 20°, specifically less than 10° or even less than 5°, may be acceptable. In order to provide a wide range of view, the matrix specifically may have at least 10 rows, pref- erably at least 500 rows, more preferably at least 1000 rows. Similarly, the matrix may have at least 10 columns, preferably at least 500 columns, more preferably at least 1000 columns. The matrix may comprise at least 50 optical sensors, preferably at least 100000 optical sensors, more preferably at least 5000000 optical sensors. The matrix may comprise a number of pixels in a multi-mega pixel range. Other embodiments, however, are feasible. Thus, in setups in which an axial rotational symmetry is to be expected, circular arrangements or concentric ar- rangements of the optical sensors of the matrix, which may also be referred to as pixels, may be preferred. Thus, as an example, the sensor element may be part of or constitute a pixelated camera. For example, the sensor element may be and/or may comprise at least one CCD and/or CMOS de- vice. As an example, the sensor element may be part of or constitute at least one CCD and/or CMOS device having a matrix of pixels, each pixel forming a light-sensitive area. The sensor el- ement may employ a rolling shutter or global shutter method to read out the matrix of optical sensors. As an example, the camera may be a fix-focus camera, having at least one lens which is fixedly adjusted with respect to the camera. Alternatively, however, the camera may also comprise one or more variable lenses which may be adjusted, automatically or manually. The camera further may comprise at least one transfer device. The camera may further com- prise one or more additional elements such as one or more additional optical elements. The camera may comprise at least one optical element selected from the group consisting of: trans- fer device, such as at least one lens and/or at least one lens system, at least one diffractive op- tical element. The term “transfer device”, also denoted as “transfer system”, may generally refer to one or more optical elements which are adapted to modify the light beam, such as by modify- ing one or more of a beam parameter of the light beam, a width of the light beam or a direction of the light beam. The transfer device may be adapted to guide the light beam onto the optical sensor. The transfer device specifically may comprise one or more of: at least one lens, for ex- ample at least one lens selected from the group consisting of at least one focus-tunable lens, at least one aspheric lens, at least one spheric lens, at least one Fresnel lens; at least one diffrac- tive optical element; at least one concave mirror; at least one beam deflection element, prefera- bly at least one mirror; at least one beam splitting element, preferably at least one of a beam splitting cube or a beam splitting mirror; at least one multi-lens system. The transfer device may have a focal length. As used herein, the term “focal length” of the transfer device refers to a dis- tance over which incident collimated rays which may impinge the transfer device are brought into a “focus” which may also be denoted as “focal point”. Thus, the focal length constitutes a measure of an ability of the transfer device to converge an impinging light beam. Thus, the transfer device may comprise one or more imaging elements which can have the effect of a converging lens. By way of example, the transfer device can have one or more lenses, in partic- ular one or more refractive lenses, and/or one or more convex mirrors. In this example, the focal length may be defined as a distance from the center of the thin refractive lens to the principal focal points of the thin lens. For a converging thin refractive lens, such as a convex or biconvex thin lens, the focal length may be considered as being positive and may provide the distance at which a beam of collimated light impinging the thin lens as the transfer device may be focused into a single spot. Additionally, the transfer device can comprise at least one wavelength-selec- tive element, for example at least one optical filter. Additionally, the transfer device can be de- signed to impress a predefined beam profile on the electromagnetic radiation, for example, at the location of the sensor region and in particular the sensor area. The abovementioned op- tional embodiments of the transfer device can, in principle, be realized individually or in any de- sired combination. The transfer device may have an optical axis. As used herein, the term “optical axis of the trans- fer device” generally refers to an axis of mirror symmetry or rotational symmetry of the lens or lens system. The transfer system, as an example, may comprise at least one beam path, with the elements of the transfer system in the beam path being located in a rotationally symmetrical fashion with respect to the optical axis. Still, one or more optical elements located within the beam path may also be off-centered or tilted with respect to the optical axis. In this case, how- ever, the optical axis may be defined sequentially, such as by interconnecting the centers of the optical elements in the beam path, e.g. by interconnecting the centers of the lenses, wherein, in this context, the optical sensors are not counted as optical elements. The optical axis generally may denote the beam path. Therein, the camera may have a single beam path along which a light beam may travel from the object to the optical sensors, or may have a plurality of beam paths. As an example, a single beam path may be given or the beam path may be split into two or more partial beam paths. In the latter case, each partial beam path may have its own optical axis. In case of a plurality of optical sensors, the optical sensors may be located in one and the same beam path or partial beam path. Alternatively, however, the optical sensors may also be located in different partial beam paths. The transfer device may constitute a coordinate system, wherein a longitudinal coordinate is a coordinate along the optical axis and wherein d is a spatial offset from the optical axis. The co- ordinate system may be a polar coordinate system in which the optical axis of the transfer de- vice forms a z-axis and in which a distance from the z-axis and a polar angle may be used as additional coordinates. A direction parallel or antiparallel to the z-axis may be considered a lon- gitudinal direction, and a coordinate along the z-axis may be considered a longitudinal coordi- nate. Any direction perpendicular to the z-axis may be considered a transversal direction, and the polar coordinate and/or the polar angle may be considered a transversal coordinate. The camera is configured for imaging the carrier at a plurality imaging positions of the camera. The images may be generated in response to the diffuse and/or uniform illumination of the car- rier by the flood light source. The images generated in response to the diffuse and/or uniform illumination of the carrier by the flood light source may not comprise any reflection features gen- erated by the illumination pattern. The image may be at least one two-dimensional image. As used herein, the term “two dimensional image” may generally refer to an image having infor- mation about transversal coordinates such as the dimensions of height and width. The image may be an RGB (red green blue) image. The term “imaging at least one image” may refer to capturing and/or recording the image. The camera is configured for imaging the at least one reflection image. The reflection image comprises a plurality of reflection features generated by the scene in response to the illumina- tion pattern. As used herein, the term “reflection feature” may refer to a feature in an image plane generated by the scene in response to illumination, specifically with at least one illumina- tion feature. Each of the reflection features comprises at least one beam profile, also denoted reflection beam profile. As used herein, the term “beam profile” of the reflection feature may generally refer to at least one intensity distribution of the reflection feature, such as of a light spot on the optical sensor, as a function of the pixel. The beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile and a linear combination of Gaussian beam profiles. The evaluation of the reflection image may comprise identifying the reflection features of the re- flection image. The processing unit may be configured for performing at least one image analy- sis and/or image processing in order to identify the reflection features. The image analysis and/or image processing may use at least one feature detection algorithm. The image analysis and/or image processing may comprise one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between an image created by the sensor signals and at least one offset; an inversion of sensor signals by inverting an image cre- ated by the sensor signals; a formation of a difference image between an image created by the sensor signals at different times; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a blob detector; applying a corner detector; applying a Determinant of Hessian filter; applying a principle curvature-based region detector; applying a maximally stable extremal regions detector; applying a generalized Hough-transformation; ap- plying a ridge detector; applying an affine invariant feature detector; applying an affine-adapted interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform; applying a scale-space extrema detector; applying a local feature detector; applying speeded up robust features algorithm; applying a gra- dient location and orientation histogram algorithm; applying a histogram of oriented gradients descriptor; applying a Deriche edge detector; applying a differential edge detector; applying a spatio-temporal interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transformation; applying a Radon-transfor- mation; applying a Hough-transformation; applying a wavelet-transformation; a thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, such as by recognizing a feature within the image generated by the optical sensor. As further used herein, the term “processing unit” generally refers to an arbitrary data pro- cessing device adapted to perform the named operations such as by using at least one proces- sor and/or at least one application-specific integrated circuit. Thus, as an example, the at least one processing unit may comprise a software code stored thereon comprising a number of com- puter commands. The processing unit may provide one or more hardware elements for perform- ing one or more of the named operations and/or may provide one or more processors with soft- ware running thereon for performing one or more of the named operations. Operations, includ- ing evaluating the images may be performed by the at least one processing unit. Thus, as an example, one or more instructions may be implemented in software and/or hardware. Thus, as an example, the processing unit may comprise one or more programmable devices such as one or more computers, application-specific integrated circuits (ASICs), Digital Signal Processors (DSPs), or Field Programmable Gate Arrays (FPGAs) which are configured to perform the above-mentioned evaluation. Additionally or alternatively, however, the processing unit may also fully or partially be embodied by hardware. The processing unit and the camera may fully or partially be integrated into a single device. Thus, generally, the processing unit also may form part of the camera. Alternatively, the processing unit and the camera may fully or partially be embodied as separate devices. The processing unit may be or may comprise one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs), and/or one or more data processing devic- es, such as one or more computers, preferably one or more microcomputers and/or microcon- trollers, Field Programmable Arrays, or Digital Signal Processors. Additional components may be comprised, such as one or more preprocessing devices and/or data acquisition devices, such as one or more devices for receiving and/or preprocessing of the sensor signals, such as one or more AD-converters and/or one or more filters. Further, the processing unit may com- prise one or more measurement devices, such as one or more measurement devices for meas- uring electrical currents and/or electrical voltages. Further, the processing unit may comprise one or more data storage devices. Further, the processing unit may comprise one or more inter- faces, such as one or more wireless interfaces and/or one or more wire-bound interfaces. The processing unit may be configured to one or more of displaying, visualizing, analyzing, dis- tributing, communicating or further processing of information, such as information obtained by the camera. The processing unit, as an example, may be connected or incorporate at least one of a display, a projector, a monitor, an LCD, a TFT, a loudspeaker, a multichannel sound sys- tem, an LED pattern, or a further visualization device. It may further be connected or incorporate at least one of a communication device or communication interface, a connector or a port, capa- ble of sending encrypted or unencrypted information using one or more of email, text messages, telephone, Bluetooth, Wi-Fi, infrared or internet interfaces, ports or connections. It may further be connected to or incorporate at least one of a processor, a graphics processor, a CPU, an Open Multimedia Applications Platform (OMAPTM), an integrated circuit, a system on a chip such as products from the Apple A series or the Samsung S3C2 series, a microcontroller or mi- croprocessor, one or more memory blocks such as ROM, RAM, EEPROM, or flash memory, timing sources such as oscillators or phase-locked loops, counter-timers, real-time timers, or power-on reset generators, voltage regulators, power management circuits, or DMA controllers. Individual units may further be connected by buses such as AMBA buses or be integrated in an Internet of Things or Industry 4.0 type network. The processing unit may be connected by or have further external interfaces or ports such as one or more of serial or parallel interfaces or ports, USB, Centronics Port, FireWire, HDMI, Ethernet, Bluetooth, RFID, Wi-Fi, USART, or SPI, or analogue interfaces or ports such as one or more of ADCs or DACs, or standardized interfaces or ports to further devices such as a 2D- camera device using an RGB-interface such as CameraLink. The processing unit may further be connected by one or more of interprocessor interfaces or ports, FPGA-FPGA-interfaces, or serial or parallel interfaces ports. The processing unit may further be connected to one or more of an optical disc drive, a CD-RW drive, a DVD+RW drive, a flash drive, a memory card, a disk drive, a hard disk drive, a solid state disk or a solid state hard disk. The processing unit may be connected by or have one or more further external connectors such as one or more of phone connectors, RCA connectors, VGA connectors, hermaphrodite con- nectors, USB connectors, HDMI connectors, 8P8C connectors, BCN connectors, IEC 60320 C14 connectors, optical fiber connectors, D-subminiature connectors, RF connectors, coaxial connectors, SCART connectors, XLR connectors, and/or may incorporate at least one suitable socket for one or more of these connectors. The processing unit may be configured for determining the beam profile of the respective reflec- tion feature. As used herein, the term “determining the beam profile” refers to identifying at least one reflection feature provided by the optical sensor and/or selecting at least one reflection fea- ture provided by the optical sensor and evaluating at least one intensity distribution of the reflec- tion feature. As an example, a region of the matrix may be used and evaluated for determining the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the matrix. As an example, a center of illumination by the light beam may be determined, such as by determining the at least one pixel having the highest illumination, and a cross-sectional axis may be chosen through the cen- ter of illumination. The intensity distribution may an intensity distribution as a function of a coor- dinate along this cross-sectional axis through the center of illumination. Other evaluation algo- rithms are feasible. The reflection feature may cover or may extend over at least one pixel of the refelction image. For example, the reflection feature may cover or may extend over plurality of pixels. The pro- cessing unit may be configured for determining and/or for selecting all pixels connected to and/or belonging to the reflection feature, e.g. a light spot. The processing unit may be config- ured for determining the center of intensity by
Figure imgf000019_0001
wherein Rcoi is a position of center of intensity, rpixel is the pixel position and
Figure imgf000019_0002
with j being the number of pixels j connected to and/or belonging to the reflection feature and Itotal be- ing the total intensity. The processing unit is configured for determining material characteristics for each reflection fea- ture by analysis of its respective beam profile. As used herein, the term “material characteris- tics” may refer to arbitrary material property m derived from and/or relating to the beam profile of the reflection feature. The material characteristics may be at least one information selected from the group consisting of information about softness, information about deformability, or infor- mation about permeability to air. The material characteristics may be extracted for distinguishing between foreground and background, i.e. items to pick vs. carrier. The processing unit may be configured for determining the material property m of the surface remitting the reflection feature by evaluating the beam profile of the reflection feature. As used herein, the term “material property” refers to at least one arbitrary property of the material con- figured for characterizing and/or identification and/or classification of the material. For example, the material property may be a property selected from the group consisting of: roughness, pene- tration depth of light into the material, a reflectivity, a specular reflectivity, a diffuse reflectivity, a surface property, a measure for translucence, a scattering, specifically a back-scattering behav- ior or the like. The at least one material property may be a property selected from the group consisting of: a scattering coefficient, a translucency, a transparency, a deviation from a Lam- bertian surface reflection, a speckle, and the like. The processing unit may be configured for identifying a reflection feature as to be generated by an item having a specific material property in case its reflection beam profile fulfills at least one predetermined or predefined criterion. As used herein, the term “at least one predetermined or predefined criterion” refers to at least one property and/or value suitable to distinguish material properties. The predetermined or predefined criterion may be or may comprise at least one pre- determined or predefined value and/or threshold and/or threshold range referring to a material property. The reflection feature may be indicated as to be generated by an item having a spe- cific material property in case the reflection beam profile fulfills the at least one predetermined or predefined criterion. As used herein, the term “indicate” refers to an arbitrary indication such as an electronic signal and/or at least one visual or acoustic indication. As used herein, the term “determining at least one material property” may refer to assigning the material property to respective reflection feature. The processing unit may comprise at least one database comprising a list and/or table, such as a lookup list or a lookup table, of predefined and/or predetermined material properties. The list and/or table of material properties may be de- termined and/or generated by performing at least one test measurement, for example by per- forming material tests using samples having known material properties. The list and/or table of material properties may be determined and/or generated at the manufacturer site and/or by a user. The material property may additionally be assigned to a material classifier such as one or more of a material name, a material group such as biological or non-biological material, translu- cent or non-translucent materials, metal or non-metal, fur or non-fur, carpet or non-carpet, re- flective or non-reflective, specular reflective or non-specular reflective, foam or non-foam, roughness groups or the like. The processing unit may comprise at least one database compris- ing a list and/or table comprising the material properties and associated material name and/or material group. For determining the material characteristic in step b), beam profile analysis may be used. Spe- cifically, beam profile analysis makes use of reflection properties of coherent light projected onto object surfaces to classify materials. The classification of materials may be performed as de- scribed in WO 2020/187719, in EP application 20159984.2 filed on February 28, 2020 and/or EP application 20154961.5 filed on January 31, 2020, the full content of which is included by reference. Specifically, a periodic grid of laser spots, e.g. a hexagonal grid as described in EP application 20170905.2 filed on April 22, 2020, is projected and the reflection image is rec- orded with the camera. Analyzing the beam profile of each reflection feature recorded by the camera may be performed by feature-based methods and/or using based on a convolutional neural network classifying the reflection features of the reflection image. The feature based methods may be used in combination with machine learning methods which may allow para- metrization of a classification model. Convolutional neuronal networks may be utilized to classify materials by using the reflection images as an input. The feature-based methods may be explained in the following. The processing unit may be con- figured for comparing the reflection beam profile with at least one predetermined and/or prere- corded and/or predefined beam profile. The predetermined and/or prerecorded and/or prede- fined beam profile may be stored in a table or a lookup table and may be determined e.g. empir- ically, and may, as an example, be stored in at least one data storage device. For example, the predetermined and/or prerecorded and/or predefined beam profile may be determined during initial start-up of a device executing the method according to the present invention. For exam- ple, the predetermined and/or prerecorded and/or predefined beam profile may be stored in at least one data storage device of the processing unit, e.g. by software, specifically by the app downloaded from an app store or the like. The reflection feature may be identified as to be gen- erated by an item having a material property m in case the reflection beam profile and the pre- determined and/or prerecorded and/or predefined beam profile are identical. The comparison may comprise overlaying the reflection beam profile and the predetermined or predefined beam profile such that their centers of intensity match. The comparison may comprise determining a deviation, e.g. a sum of squared point to point distances, between the reflection beam profile and the predetermined and/or prerecorded and/or predefined beam profile. The processing unit may be adapted to compare the determined deviation with at least one threshold, wherein in case the determined deviation is below and/or equal the threshold the surface is indicated as biological tissue and/or the detection of biological tissue is confirmed. The threshold value may be stored in a table or a lookup table and may be determined e.g. empirically and may, as an example, be stored in at least one data storage device of the processing unit. Additionally or alternatively, the material characteristics may be determined by applying at least one image filter to the refection image. As further used herein, the term “image” refers to a two- dimensional function, f(x,y), wherein brightness and/or color values are given for any x,y-posi- tion in the image. The position may be discretized corresponding to the recording pixels. The brightness and/or color may be discretized corresponding to a bit-depth of the optical sensors. As used herein, the term “image filter” refers to at least one mathematical operation applied to the beam profile and/or to the at least one specific region of the beam profile. Specifically, the image filter Ф maps an image f, or a region of interest in the image, onto a real number, Ф(f(x,y)) = φ, wherein φ denotes a feature, in particular a material feature. Images may be subject to noise and the same holds true for features. Therefore, features may be random variables. The features may be normally distributed. If features are not normally distributed, they may be trans- formed to be normally distributed such as by a Box-Cox-Transformation. The processing unit may be configured for determining at least one material feature φ2m by ap- plying at least one material dependent image filter Ф2 to the image. As used herein, the term “ material dependent” image filter refers to an image having a material dependent output. The output of the material dependent image filter is denoted herein “material feature φ2m” or “mate- rial dependent feature φ2m”. The material feature may be or may comprise at least one infor- mation about the at least one material property of the surface of the scene having generated the reflection feature. The material dependent image filter may be at least one filter selected from the group consisting of: a luminance filter; a spot shape filter; a squared norm gradient; a standard deviation; a smoothness filter such as a Gaussian filter or median filter; a grey-level-occurrence-based con- trast filter; a grey-level-occurrence-based energy filter; a grey-level-occurrence-based homoge- neity filter; a grey-level-occurrence-based dissimilarity filter; a Law’s energy filter; a threshold area filter; or a linear combination thereof; or a further material dependent image filter Ф2other which correlates to one or more of the luminance filter, the spot shape filter, the squared norm gradient, the standard deviation, the smoothness filter, the grey-level-occurrence-based energy filter, the grey-level-occurrence-based homogeneity filter, the grey-level-occurrence-based dis- similarity filter, the Law’s energy filter, or the threshold area filter, or a linear combination thereof by |ρФ2other,Фm|≥0.40 with Фm being one of the luminance filter, the spot shape filter, the squared norm gradient, the standard deviation, the smoothness filter, the grey-level-occurrence-based energy filter, the grey-level-occurrence-based homogeneity filter, the grey-level-occurrence- based dissimilarity filter, the Law’s energy filter, or the threshold area filter, or a linear combina- tion thereof. The further material dependent image filter Ф2other may correlate to one or more of the material dependent image filters Фm by |ρФ2other,Фm|≥0.60, preferably by |ρФ2other,Фm|≥0.80. The material dependent image filter may be at least one arbitrary filter Φ that passes a hypothe- sis testing. As used herein, the term “passes a hypothesis testing” refers to the fact that a Null- hypothesis H0 is rejected and an alternative hypothesis H1 is accepted. The hypothesis testing may comprise testing the material dependency of the image filter by applying the image filter to a predefined data set. The data set may comprise a plurality of beam profile images. As used herein, the term “beam profile image” refers to a sum of NB Gaussian radial basis functions,
Figure imgf000022_0001
wherein each of the NB Gaussian radial basis functions is defined by a center
Figure imgf000022_0003
a prefac- tor, alk, and an exponential factor α = 1/∈. The exponential factor is identical for all Gaussian functions in all images. The center-positions,
Figure imgf000022_0004
are identical for all images fk:
Figure imgf000022_0002
Each of the beam profile images in the dataset may correspond to a material classifier and a distance. The material classifier may be a label such as ‘Material A’, ‘Material B’, etc. The beam profile images may be generated by using the above formula for fk(x, y) in combination with the following parameter table:
Figure imgf000022_0006
The values for x, y, are integers corresponding to pixels with . The images
Figure imgf000022_0005
may have a pixel size of 32x32. The dataset of beam profile images may be generated by using the above formula for fk in combination with a parameter set to obtain a continuous description of fk. The values for each pixel in the 32x32-image may be obtained by inserting integer values from 0, …, 31 for x, y, in fk(x, y). For example, for pixel (6,9), the value fk(6,9) may be com- puted. Subsequently, for each image fk, the feature value φk corresponding to the filter Φ may be cal- culated, Φ(fk(x, y), zk) = φk, wherein zk is a distance value corresponding to the image fk from the predefined data set. This yields a dataset with corresponding generated feature values φk The hypothesis testing may use a Null-hypothesis that the filter does not distinguish between material classifier. The Null-Hypothesis may be given by H0: μ1 = μ2 = ⋯ = μJ, wherein μm is the expectation value of each material-group corresponding to the feature values φk. Index m denotes the material group. The hypothesis testing may use as alternative hypothesis that the filter does distinguish between at least two material classifiers. The alternative hypothesis may be given by As used herein, the term “not distinguish between material
Figure imgf000023_0001
classifiers” refers to that the expectation values of the material classifiers are identical. As used herein, the term “distinguishes material classifiers” refers to that at least two expectation values of the material classifiers differ. As used herein “distinguishes at least two material classifiers” is used synonymous to “suitable material classifier”. The hypothesis testing may comprise at least one analysis of variance (ANOVA) on the generated feature values. In particular, the hypothesis testing may comprise determining a mean-value of the feature values for each of the ^ materi- als, i.e. in total J mean values, for m ∈ [0,1, ⋯ , J − 1], wherein Nm gives the number
Figure imgf000023_0002
of feature values for each of the J materials in the predefined data set The hypothesis testing may comprise determining a mean-value of all N feature values The hypothesis
Figure imgf000023_0003
testing may comprise determining a Mean Sum Squares within:
Figure imgf000023_0006
The hypothesis testing may comprise determining a Mean Sum of Squares between,
Figure imgf000023_0005
The hypothesis testing may comprise performing an F-Test:
Figure imgf000023_0004
Herein, I^ is the regularized incomplete Beta-Function,
Figure imgf000023_0008
, with the Euler Beta- Function being the incomplete
Figure imgf000023_0007
Beta-Function. The image filter may pass the hypothesis testing if a p-value, p, is smaller or equal than a pre-defined level of significance. The filter may pass the hypothesis testing if p ≤ 0.075, preferably p ≤ 0.05, more preferably p ≤ 0.025, and most preferably p ≤ 0.01. For exam- ple, in case the pre-defined level of significance is α =0.075, the image filter may pass the hy- pothesis testing if the p-value is smaller than α =0.075. In this case the Null-hypothesis H0 can be rejected and the alternative hypothesis H1 can be accepted. The image filter thus distin- guishes at least two material classifiers. Thus, the image filter passes the hypothesis testing. In the following, image filters are described assuming that the reflection image comprises at least one reflection feature, in particular a spot image. A spot image ^ may be given by a func- tion wherein the background of the image f may be already subtracted. However,
Figure imgf000024_0004
other reflection features may be possible. For example, the material dependent image filter may be a luminance filter. The luminance filter may return a luminance measure of a spot as material feature. The material feature may be de- termined by
Figure imgf000024_0001
where f is the spot image. The distance of the spot is denoted by z, where z may be obtained for example by using a depth-from-defocus or depth-from–photon ratio technique and/or by us- ing a triangulation technique. The surface normal of the material is given by ^ ∈ ℝ^ and can be obtained as the normal of the surface spanned by at least three measured points. The vector is the direction vector of the light source. Since the position of the spot is known by
Figure imgf000024_0005
using a depth-from-defocus or depth-from–photon ratio technique and/or by using a triangula- tion technique wherein the position of the light source is known as a parameter of the detector system, dray, is the difference vector between spot and light source positions. For example, the material dependent image filter may be a filter having an output dependent on a spot shape. This material dependent image filter may return a value which correlates to the translucence of a material as material feature. The translucence of materials influences the shape of the spots. The material feature may be given by
Figure imgf000024_0002
wherein 0 < α, β < 1 are weights for the spot height h, and H denotes the Heavyside function, i.e. H(x) = 1 ∶ x ≥ 0, H(x) = 0 ∶ x < 0. The spot height h may be determined by
Figure imgf000024_0003
where Br is an inner circle of a spot with radius r. For example, the material dependent image filter may be a squared norm gradient. This mate- rial dependent image filter may return a value which correlates to a measure of soft and hard transitions and/or roughness of a spot as material feature. The material feature may be defined by
Figure imgf000025_0001
For example, the material dependent image filter may be a standard deviation. The standard deviation of the spot may be determined by
Figure imgf000025_0002
Wherein µ is the mean value given by
Figure imgf000025_0003
For example, the material dependent image filter may be a smoothness filter such as a Gauss- ian filter or median filter. In one embodiment of the smoothness filter, this image filter may refer to the observation that volume scattering exhibits less speckle contrast compared to diffuse scattering materials. This image filter may quantify the smoothness of the spot corresponding to speckle contrast as material feature. The material feature may be determined by
Figure imgf000025_0004
wherein F is a smoothness function, for example a median filter or Gaussian filter. This image filter may comprise dividing by the distance z, as described in the formula above. The distance z may be determined for example using a depth-from-defocus or depth-from–photon ratio tech- nique and/or by using a triangulation technique. This may allow the filter to be insensitive to dis- tance. In one embodiment of the smoothness filter, the smoothness filter may be based on the standard deviation of an extracted speckle noise pattern. A speckle noise pattern N can be de- scribed in an empirical way by
Figure imgf000025_0005
where ^^ is an image of a despeckled spot. N(X) is the noise term that models the speckle pat- tern. The computation of a despeckled image may be difficult. Thus, the despeckled image may be approximated with a smoothed version of f, i.e. , wherein F is a smoothness opera-
Figure imgf000025_0006
tor like a Gaussian filter or median filter. Thus, an approximation of the speckle pattern may be given by
Figure imgf000025_0007
The material feature of this filter may be determined by
Figure imgf000025_0008
Wherein Var denotes the variance function. For example, the image filter may be a grey-level-occurrence-based contrast filter. This material filter may be based on the grey level occurrence matrix whereas is
Figure imgf000026_0006
Figure imgf000026_0007
the occurrence rate of the grey combination (g1,g2)=[f(x1,y1),f(x2,y2)], and the relation ρ defines the distance between (x1,y1) and (x2,y2), which is ρ(x,y)=(x+a,y+b) with a and b selected from 0,1. The material feature of the grey-level-occurrence-based contrast filter may be given by
Figure imgf000026_0001
For example, the image filter may be a grey-level-occurrence-based energy filter. This material filter is based on the grey level occurrence matrix defined above. The material feature of the grey-level-occurrence-based energy filter may be given by
Figure imgf000026_0002
For example, the image filter may be a grey-level-occurrence-based homogeneity filter. This material filter is based on the grey level occurrence matrix defined above. The material feature of the grey-level-occurrence-based homogeneity filter may be given by
Figure imgf000026_0003
For example, the image filter may be a grey-level-occurrence-based dissimilarity filter. This ma- terial filter is based on the grey level occurrence matrix defined above. The material feature of the grey-level-occurrence-based dissimilarity filter may be given by
Figure imgf000026_0004
For example, the image filter may be a Law’s energy filter. This material filter may be based on the laws vector L5=[1,4,6,4,1] and E5=[-1,-2,0,-2,-1] and the matrices L5(E5)T and E5(L5)T. The image fk is convoluted with these matrices: and
Figure imgf000026_0005
Figure imgf000027_0001
Whereas the material feature of Law’s energy filter may be determined by
Figure imgf000027_0002
For example, the material dependent image filter may be a threshold area filter. This material feature may relate two areas in the image plane. A first area Ω1, may be an area wherein the function f is larger than α times the maximum of f. A second area Ω2, may be an area wherein the function f is smaller than α times the maximum of f, but larger than a threshold value ε times the maximum of f. Preferably α may be 0.5 and ε may be 0.05. Due to speckles or noise, the ar- eas may not simply correspond to an inner and an outer circle around the spot center. As an ex- ample, Ω1 may comprise speckles or unconnected areas in the outer circle. The material fea- ture may be determined by
Figure imgf000027_0003
wherein Ω1 = {x| f(x) > α⋅max(f(x))} and Ω2 = {x| ε⋅max(f(x)) < f(x) < α⋅max(f(x))}. The processing unit may be configured for using at least one predetermined relationship be- tween the material feature φ2m and the material property of the surface having generated the re- flection feature for determining the material property of the surface having generated the reflec- tion feature. The predetermined relationship may be one or more of an empirical relationship, a semi-empiric relationship and an analytically derived relationship. The processing unit may com- prise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table. Step b) may comprise using artificial intelligence, in particular convolutional neuronal networks. Using reflection images as input for convolutional neuronal networks may enable the generation of classification models with sufficient accuracy to differentiate between materials. Since only physically valid information is passed to the network by selecting important regions in the reflec- tion image, only compact training data sets may be needed. Additionally, very compact network architectures can be generated. Specifically, in step b) at least one parametrized classification model may be used. The para- metrized classification model may be configured for classifying materials by using the reflection image as an input. The classification model may be parametrized by using one or more of ma- chine learning, deep learning, neural networks, or other form of artificial intelligence. The term “machine-learning” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a method of using artificial intelli- gence (AI) for automatically model building, in particular for parametrizing models. The classifi- cation model may be a classification model configured for discriminating materials. The material characteristics may be determined by applying an optimization algorithm in terms of at least one optimization target on the classification model. The machine learning may be based on at least one neuronal network, in particular a convolutional neural network. Weights and/or topology of the neuronal network may be pre-determined and/or pre-defined. Specifically, the training of the classification model may be performed using machine-learning. The classification model may comprise at least one machine-learning architecture and model parameters. For example, the machine-learning architecture may be or may comprise one or more of: linear regression, lo- gistic regression, random forest, naive Bayes classifications, nearest neighbors, neural net- works, convolutional neural networks, generative adversarial networks, support vector ma- chines, or gradient boosting algorithms or the like. The term “training”, also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a process of building the model, in particular deter- mining and/or updating parameters of the model. The classification model may be at least par- tially data-driven. For example, the classification model may be based on experimental data. For example, the training may comprise using at least one training dataset, wherein the training data set comprises images, in particular refection images, of a plurality of items with known ma- terial property. In step b), multiple camera positions may be computed that cover the entire carrier to ensure that all items can be picked. These positions may be approached in a row until no item can be identified anymore and all items are covered. Step b) may comprise the processing unit determining depth information for each reflection fea- ture by evaluating the reflection image. The evaluation may comprises, for each reflection fea- ture, an analysis of its respective beam profile using a depth-from-photon ratio technique. The analysis of the beam profile comprise evaluating of the beam profile and may comprise at least one mathematical operation and/or at least one comparison and/or at least symmetrizing and/or at least one filtering and/or at least one normalizing. For example, the analysis of the beam pro- file may comprise at least one of a histogram analysis step, a calculation of a difference meas- ure, application of a neural network, application of a machine learning algorithm. The pro- cessing unit may be configured for symmetrizing and/or for normalizing and/or for filtering the beam profile, in particular to remove noise or asymmetries from recording under larger angles, recording edges or the like. The processing unit may filter the beam profile by removing high spatial frequencies such as by spatial frequency analysis and/or median filtering or the like. Summarization may be performed by center of intensity of the light spot and averaging all inten- sities at the same distance to the center. The processing unit may be configured for normalizing the beam profile to a maximum intensity, in particular to account for intensity differences due to the recorded distance. The processing unit may be configured for removing influences from background light from the beam profile, for example, by an imaging without illumination. The processing unit may be configured for determining at least one longitudinal coordinate zDPR for each of the reflection features by analysis of the beam profile of the respective reflection fea- ture. The processing unit may be configured for determining the longitudinal coordinate zDPR for the reflection features by using the so called depth-from-photon-ratio technique, also denoted as beam profile analysis. With respect to depth-from-photon-ratio (DPR) technique reference is made to WO 2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, the full content of which is included by reference. The processing unit may be configured for determining at least one first area and at least one second area of the reflection beam profile of each of the reflection features and/or of the reflec- tion features in at least one region of interest. The processing unit is configured for integrating the first area and the second area. The analysis of the beam profile of one of the reflection features may comprise determining at least one first area and at least one second area of the beam profile. The first area of the beam profile may be an area A1 and the second area of the beam profile may be an area A2. The pro- cessing unit may be configured for integrating the first area and the second area. The pro- cessing unit may be configured to derive a combined signal, in particular a quotient Q, by one or more of dividing the integrated first area and the integrated second area, dividing multiples of the integrated first area and the integrated second area, dividing linear combinations of the inte- grated first area and the integrated second area. The processing unit may configured for deter- mining at least two areas of the beam profile and/or to segment the beam profile in at least two segments comprising different areas of the beam profile, wherein overlapping of the areas may be possible as long as the areas are not congruent. For example, the processing unit may be configured for determining a plurality of areas such as two, three, four, five, or up to ten areas. The processing unit may be configured for segmenting the light spot into at least two areas of the beam profile and/or to segment the beam profile in at least two segments comprising differ- ent areas of the beam profile. The processing unit may be configured for determining for at least two of the areas an integral of the beam profile over the respective area. The processing unit may be configured for comparing at least two of the determined integrals. Specifically, the pro- cessing unit may be configured for determining at least one first area and at least one second area of the beam profile. As used herein, the term “area of the beam profile” generally refers to an arbitrary region of the beam profile at the position of the optical sensor used for determining the quotient Q. The first area of the beam profile and the second area of the beam profile may be one or both of adjacent or overlapping regions. The first area of the beam profile and the second area of the beam profile may be not congruent in area. For example, the processing unit may be configured for dividing a sensor region of the CMOS sensor into at least two sub-re- gions, wherein the processing unit may be configured for dividing the sensor region of the CMOS sensor into at least one left part and at least one right part and/or at least one upper part and at least one lower part and/or at least one inner and at least one outer part. Additionally or alternatively, the camera may comprise at least two optical sensors, wherein the light-sensitive areas of a first optical sensor and of a second optical sensor may be arranged such that the first optical sensor is adapted to determine the first area of the beam profile of the reflection feature and that the second optical sensor is adapted to determine the second area of the beam profile of the reflection feature. The processing unit may be adapted to integrate the first area and the second area. The processing unit may be configured for using at least one predetermined rela- tionship between the quotient Q and the longitudinal coordinate for determining the longitudinal coordinate. The predetermined relationship may be one or more of an empiric relationship, a semi-empiric relationship and an analytically derived relationship. The processing unit may com- prise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table. The first area of the beam profile may comprise essentially edge information of the beam profile and the second area of the beam profile comprises essentially center information of the beam profile, and/or the first area of the beam profile may comprise essentially information about a left part of the beam profile and the second area of the beam profile comprises essentially infor- mation about a right part of the beam profile. The beam profile may have a center, i.e. a maxi- mum value of the beam profile and/or a center point of a plateau of the beam profile and/or a geometrical center of the light spot, and falling edges extending from the center. The second re- gion may comprise inner regions of the cross section and the first region may comprise outer regions of the cross section. As used herein, the term “essentially center information” generally refers to a low proportion of edge information, i.e. proportion of the intensity distribution corre- sponding to edges, compared to a proportion of the center information, i.e. proportion of the in- tensity distribution corresponding to the center. Preferably, the center information has a propor- tion of edge information of less than 10%, more preferably of less than 5%, most preferably the center information comprises no edge content. As used herein, the term “essentially edge infor- mation” generally refers to a low proportion of center information compared to a proportion of the edge information. The edge information may comprise information of the whole beam pro- file, in particular from center and edge regions. The edge information may have a proportion of center information of less than 10%, preferably of less than 5%, more preferably the edge infor- mation comprises no center content. At least one area of the beam profile may be determined and/or selected as second area of the beam profile if it is close or around the center and com- prises essentially center information. At least one area of the beam profile may be determined and/or selected as first area of the beam profile if it comprises at least parts of the falling edges of the cross section. For example, the whole area of the cross section may be determined as first region. Other selections of the first area A1 and second area A2 may be feasible. For example, the first area may comprise essentially outer regions of the beam profile and the second area may com- prise essentially inner regions of the beam profile. For example, in case of a two-dimensional beam profile, the beam profile may be divided in a left part and a right part, wherein the first area may comprise essentially areas of the left part of the beam profile and the second area may comprise essentially areas of the right part of the beam profile. The edge information may comprise information relating to a number of photons in the first area of the beam profile and the center information may comprise information relating to a number of photons in the second area of the beam profile. The processing unit may be configured for de- termining an area integral of the beam profile. The processing unit may be configured for deter- mining the edge information by integrating and/or summing of the first area. The processing unit may be configured for determining the center information by integrating and/or summing of the second area. For example, the beam profile may be a trapezoid beam profile and the pro- cessing unit may be configured for determining an integral of the trapezoid. Further, when trape- zoid beam profiles may be assumed, the determination of edge and center signals may be re- placed by equivalent evaluations making use of properties of the trapezoid beam profile such as determination of the slope and position of the edges and of the height of the central plateau and deriving edge and center signals by geometric considerations. In one embodiment, A1 may correspond to a full or complete area of a feature point on the opti- cal sensor. A2 may be a central area of the feature point on the optical sensor. The central area may be a constant value. The central area may be smaller compared to the full area of the fea- ture point. For example, in case of a circular feature point, the central area may have a radius from 0.1 to 0.9 of a full radius of the feature point, preferably from 0.4 to 0.6 of the full radius. In one embodiment, the illumination pattern may comprise at least one line pattern. A1 may cor- respond to an area with a full line width of the line pattern on the optical sensors, in particular on the light sensitive area of the optical sensors. The line pattern on the optical sensor may be wid- ened and/or displaced compared to the line pattern of the illumination pattern such that the line width on the optical sensors is increased. In particular, in case of a matrix of optical sensors, the line width of the line pattern on the optical sensors may change from one column to another col- umn. A2 may be a central area of the line pattern on the optical sensor. The line width of the central area may be a constant value, and may in particular correspond to the line width in the illumination pattern. The central area may have a smaller line width compared to the full line width. For example, the central area may have a line width from 0.1 to 0.9 of the full line width, preferably from 0.4 to 0.6 of the full line width. The line pattern may be segmented on the optical sensors. Each column of the matrix of optical sensors may comprise center information of inten- sity in the central area of the line pattern and edge information of intensity from regions extend- ing further outwards from the central area to edge regions of the line pattern. In one embodiment, the illumination pattern may comprise at least point pattern. A1 may corre- spond to an area with a full radius of a point of the point pattern on the optical sensors. A2 may be a central area of the point in the point pattern on the optical sensors. The central area may be a constant value. The central area may have a radius compared to the full radius. For exam- ple, the central area may have a radius from 0.1 to 0.9 of the full radius, preferably from 0.4 to 0.6 of the full radius. The illumination pattern may comprise both at least one point pattern and at least one line pat- tern. Other embodiments in addition or alternatively to line pattern and point pattern are feasi- ble. The processing unit may be configured to derive a quotient Q by one or more of dividing the in- tegrated first area and the integrated second area, dividing multiples of the integrated first area and the integrated second area, dividing linear combinations of the integrated first area and the integrated second area. The processing unit may be configured to derive the quotient Q by one or more of dividing the first area and the second area, dividing multiples of the first area and the second area, dividing linear combinations of the first area and the second area. The processing unit may be config- ured for deriving the quotient Q by
Figure imgf000032_0001
wherein x and y are transversal coordinates, A1 and A2 are the first and second area of the beam profile, respectively, and E(x,y) denotes the beam profile. Additionally or alternatively, the processing unit may be adapted to determine one or both of center information or edge information from at least one slice or cut of the light spot. This may be realized, for example, by replacing the area integrals in the quotient Q by a line integral along the slice or cut. For improved accuracy, several slices or cuts through the light spot may be used and averaged. In case of an elliptical spot profile, averaging over several slices or cuts may result in improved distance information. For example, in case of the optical sensor having a matrix of pixels, the processing unit may be configured for evaluating the beam profile, by - determining the pixel having the highest sensor signal and forming at least one center sig- nal; - evaluating sensor signals of the matrix and forming at least one sum signal; - determining the quotient Q by combining the center signal and the sum signal; and - determining at least one longitudinal coordinate z of the object by evaluating the quotient Q. As used herein, a “sensor signal” generally refers to a signal generated by the optical sensor and/or at least one pixel of the optical sensor in response to illumination. Specifically, the sensor signal may be or may comprise at least one electrical signal, such as at least one analogue electrical signal and/or at least one digital electrical signal. More specifically, the sensor signal may be or may comprise at least one voltage signal and/or at least one current signal. More specifically, the sensor signal may comprise at least one photocurrent. Further, either raw sen- sor signals may be used, or the display device, the optical sensor or any other element may be adapted to process or preprocess the sensor signal, thereby generating secondary sensor sig- nals, which may also be used as sensor signals, such as preprocessing by filtering or the like. The term “center signal” generally refers to the at least one sensor signal comprising essentially center information of the beam profile. As used herein, the term “highest sensor signal” refers to one or both of a local maximum or a maximum in a region of interest. For example, the center signal may be the signal of the pixel having the highest sensor signal out of the plurality of sen- sor signals generated by the pixels of the entire matrix or of a region of interest within the ma- trix, wherein the region of interest may be predetermined or determinable within an image gen- erated by the pixels of the matrix. The center signal may arise from a single pixel or from a group of optical sensors, wherein, in the latter case, as an example, the sensor signals of the group of pixels may be added up, integrated or averaged, in order to determine the center sig- nal. The group of pixels from which the center signal arises may be a group of neighboring pix- els, such as pixels having less than a predetermined distance from the actual pixel having the highest sensor signal, or may be a group of pixels generating sensor signals being within a pre- determined range from the highest sensor signal. The group of pixels from which the center sig- nal arises may be chosen as large as possible in order to allow maximum dynamic range. The processing unit may be adapted to determine the center signal by integration of the plurality of sensor signals, for example the plurality of pixels around the pixel having the highest sensor sig- nal. For example, the beam profile may be a trapezoid beam profile and the processing unit may be adapted to determine an integral of the trapezoid, in particular of a plateau of the trape- zoid. As outlined above, the center signal generally may be a single sensor signal, such as a sensor signal from the pixel in the center of the light spot, or may be a combination of a plurality of sen- sor signals, such as a combination of sensor signals arising from pixels in the center of the light spot, or a secondary sensor signal derived by processing a sensor signal derived by one or more of the aforementioned possibilities. The determination of the center signal may be per- formed electronically, since a comparison of sensor signals is fairly simply implemented by con- ventional electronics, or may be performed fully or partially by software. Specifically, the center signal may be selected from the group consisting of: the highest sensor signal; an average of a group of sensor signals being within a predetermined range of tolerance from the highest sen- sor signal; an average of sensor signals from a group of pixels containing the pixel having the highest sensor signal and a predetermined group of neighboring pixels; a sum of sensor signals from a group of pixels containing the pixel having the highest sensor signal and a predeter- mined group of neighboring pixels; a sum of a group of sensor signals being within a predeter- mined range of tolerance from the highest sensor signal; an average of a group of sensor sig- nals being above a predetermined threshold; a sum of a group of sensor signals being above a predetermined threshold; an integral of sensor signals from a group of optical sensors contain- ing the optical sensor having the highest sensor signal and a predetermined group of neighbor- ing pixels; an integral of a group of sensor signals being within a predetermined range of toler- ance from the highest sensor signal; an integral of a group of sensor signals being above a pre- determined threshold. Similarly, the term “sum signal” generally refers to a signal comprising essentially edge infor- mation of the beam profile. For example, the sum signal may be derived by adding up the sen- sor signals, integrating over the sensor signals or averaging over the sensor signals of the en- tire matrix or of a region of interest within the matrix, wherein the region of interest may be pre- determined or determinable within an image generated by the optical sensors of the matrix. When adding up, integrating over or averaging over the sensor signals, the actual optical sen- sors from which the sensor signal is generated may be left out of the adding, integration or aver- aging or, alternatively, may be included into the adding, integration or averaging. The pro- cessing unit may be adapted to determine the sum signal by integrating signals of the entire matrix, or of the region of interest within the matrix. For example, the beam profile may be a trapezoid beam profile and the processing unit may be adapted to determine an integral of the entire trapezoid. Further, when trapezoid beam profiles may be assumed, the determination of edge and center signals may be replaced by equivalent evaluations making use of properties of the trapezoid beam profile such as determination of the slope and position of the edges and of the height of the central plateau and deriving edge and center signals by geometric considera- tions. Similarly, the center signal and edge signal may also be determined by using segments of the beam profile such as circular segments of the beam profile. For example, the beam profile may be divided into two segments by a secant or a chord that does not pass the center of the beam profile. Thus, one segment will essentially contain edge information, while the other segment will contain essentially center information. For example, to further reduce the amount of edge information in the center signal, the edge signal may further be subtracted from the center sig- nal. The quotient Q may be a signal which is generated by combining the center signal and the sum signal. Specifically, the determining may include one or more of: forming a quotient of the center signal and the sum signal or vice versa; forming a quotient of a multiple of the center signal and a multiple of the sum signal or vice versa; forming a quotient of a linear combination of the cen- ter signal and a linear combination of the sum signal or vice versa. Additionally or alternatively, the quotient Q may comprise an arbitrary signal or signal combination which contains at least one item of information on a comparison between the center signal and the sum signal. As used herein, the term “longitudinal coordinate for the reflection feature” refers to a distance between the optical sensor and the point of the scene remitting the corresponding illumination features. The processing unit may be configured for using the at least one predetermined rela- tionship between the quotient Q and the longitudinal coordinate for determining the longitudinal coordinate. The predetermined relationship may be one or more of an empiric relationship, a semi-empiric relationship and an analytically derived relationship. The processing unit may com- prise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table. The processing unit may be configured for executing at least one depth-from-photon-ratio algo- rithm which computes distances for all reflection features with zero order and higher order. The method may comprise simulating pick point approaches. The processing unit may be con- figured for simulating for all pick point candidates whether approaching them leads to collisions with other items, e.g. carrier, robot or other items, wherein pick point candidates simulated to lead to collisions are filtered out for further consideration. The simulation may be performed prior to pick point scoring. The processing unit is configured for assigning a score to pick point candidates according to their probability to lead to a successful grasp using the trained scoring model. The term “pick point candidate” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to potential pick points. The term “to lead to a successful grasp” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to suitability of the pick point candidate to be picked up by the robot. For example, a successful grasp may be if the pick point candidate is suitable for allowing stable vacuum. The grasp may be not successful if vac- uum cannot be maintained at the respective pick point candidate. The probability to lead to a successful grasp may depend on one or more of softness, deformability, permeability to air and the like. The term “score” as used herein is a broad term and is to be given its ordinary and cus- tomary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a value assigned to the respective pick point candidate, wherein the value depends on the probability to lead to a successful grasp. For example, the score may range from 0 to 1. The score may be assigned from low score for low probability to high score for high probability. In step d) the pick point with the highest score is used as next pick point. The term “scoring model”, also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to a classification model and/or a regression model having as output a probability distribution over classes for the reflection features. The output of the scoring model may be, for each of the re- flection features, one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high. In step c), in addition to the material characteristics, one or more of the following input parame- ters are provided to the scoring model: information about an image section, 3D information about the reflection features. The term “input” as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be lim- ited to a special or customized meaning. The term specifically may refer, without limitation, to an input value or parameter for the trained scoring model and/or data which can be filled into the trained scoring model. The trained scoring model may be configured for generating using the input at least one output, in particular a prediction. The scoring model may be a classification and/or regression model. The scoring model may be at least one model selected from the group consisting of a random forest (RF) or a convolution neural network (CNN). The term “Random Forest”, also denoted as random forest algorithm, as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an ensemble learning method configured for at least classification and/or regression. Specifically, the random forest algorithm may be configured for constructing one or more decision trees and outputting at least one class selected from the mode of the classes and the mean prediction of the individual trees. The random forest algorithm may generally be known, e.g. from journal paper by Leo Breiman, „Random Forests“ in Machine Learning 45.1, Oct.2001. The scoring model may be based on a CNN architecture configured for classification. The con- volutional neural network may be a multilayer convolutional neural network. The convolutional neural network may comprise a plurality of convolutional layers. The convolutional layers may be followed by a plurality of fully-connected layers. The convolutional neural network may com- prise a plurality of pooling layers. The structure of convolutional neural networks is generally known to the skilled person such as from en.wikipedia.org/wiki/Convolutional_neural_net- work#Convolutional. The CNN may be build by using the Keras library in Python. For Keras library in Python reference is made to https://keras.io/ or https://de.wikipedia.org/wiki/Keras. For example, in a first branch of the CNN, material characteristics and 3D information may be fed into a dense layer with a Scaled Exponential Linear Unit (SELU) activation function. The dense layer may be followed by a batch normalization layer. The images determined in step a) may be fed, in a second branch of the CNN, into a 5x5 convolution kernel with 16 filters. The convolu- tion kernel is followed by at least one batch normalization layer, a 2x2 Max Pooling layer and a dense layer with a SELU activation function. The output of first and second branches may be fed into a dense layer with a SELU activation function followed by a dense layer having a sig- moid activation function. The output may be a probability distribution over classes for the reflec- tion features. The output of the scoring model may be, for each of the reflection features, one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high. The term “trained scoring model”, also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limita- tion, to the fact that the scoring model was trained using at least one training dataset. The train- ing dataset may comprise a plurality of reflection images having reflection features of known probability for a successful grasp. The training may be performed for different items. The trained scoring model can be re-trained and/or updated based on additional data. The trained scoring model may be trained by using machine learning. The method may comprise at least one train- ing step, wherein, in the training step, the scoring model is trained on the at least one training dataset. Specifically, the training step may be performed before performing step c). The term “close environment around the pick point candidate”, also denoted learning, as used herein, is a broad term and is to be given its ordinary and customary meaning to a person of or- dinary skill in the art and is not to be limited to a special or customized meaning. The term spe- cifically may refer, without limitation, to reflection features within a range or region around the pick point candidate. The pick point candidate may coincide with a reflection feature. However, other embodiments are possible wherein the pick point candidate and the reflection feature do not coincide. The processing unit may select a range or image region around the pick point can- didate ensuring that at least one reflection feature is present, e.g. the reflection feature nearest to the pick point candidate. The processing unit may be configured for considering the images imaged in step a) and the reflection image for identifying reflection features in a close environ- ment around the pick point candidate. Once the scoring is completed, the pick point with the highest score may be taken for the next pick. The control unit is configured for selecting the next pick point of the robot considering the assigned scores. As further used herein, the term “control unit” generally refers to an arbitrary device configured for performing the named operations, preferably by using at least one data processing device and, more preferably, by using at least one processor and/or at least one ap- plication-specific integrated circuit. Thus, as an example, the at least one control unit may com- prise at least one data processing device having a software code stored thereon comprising a number of computer commands. The control unit may provide one or more hardware elements for performing one or more of the named operations and/or may provide one or more proces- sors with software running thereon for performing one or more of the named operations. The control unit may comprise one or more programmable devices such as one or more computers, application-specific integrated circuits (ASICs), Digital Signal Processors (DSPs), or Field Pro- grammable Gate Arrays (FPGAs) which are configured to perform steps b) and c). Additionally or alternatively, however, the control unit may also fully or partially be embodied by hardware. The processing unit may be part of the control unit or an additional unit. The method may comprises a self-learning grasp sequence for robot bin picking. The method may comprise in step d) approaching the selected next pick point with the robot and determin- ing, by at least one sensor of the robot, sensor data of the approached pick point relating to suit- ability for grasping. The method may further comprise retraining the trained scoring model on the sensor data and repeating at least steps c) and d) of the method, preferably steps a) to d). The method may comprise storing the determined sensor data in at least one database. Thus, the model accumulates information with every pick and learns from it. For example, the robot comprises at least one robot arm, wherein the robot arm is equipped with a vacuum grabber and a vacuum sensor. The control unit may be configured for moving the robot arm to a picking position for the next pick considering the assigned scores. The method may comprise the control unit controlling the vacuum sensor for approaching the pick point, wherein each grasp is labeled as positive if the vacuum sensor returns stable vacuum val- ues after approaching the pick point or negative if vacuum cannot be maintained. This fully au- tomated labeling can be used to build up a comprehensive database during operation without any manual processing. The term "database" as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be lim- ited to a special or customized meaning. The term specifically may refer, without limitation, to an organized collection of data, generally stored and accessed electronically from a computer or computer system. The database may comprise or may be comprised by a data storage device. The database may comprise at least one data base management system, comprising a soft- ware running on a computer or computer system, the software allowing for interaction with one or more of a user, an application or the database itself, such as in order to capture and analyze the data contained in the data base. The database management system may further encom- pass facilities to administer the database. The database, containing the data, may, thus, be comprised by a database system which, besides the data, comprises one or more associated applications. Step d) may comprise the control unit to optimize one or more of the pick point selection, pick pose, depth offset and vacuum control of the gripper considering the determined material char- acteristics. In a further aspect of the present invention a computer program for picking of items comprised in a carrier by using at least one robot configured for causing a computer or a computer network to fully or partially perform the method according to the present invention, when executed on the computer or the computer network, wherein the computer program is configured for performing and/or executing at least steps a) to d) of the method according to the present invention. Specif- ically, the computer program may be stored on a computer-readable data carrier and/or on a computer-readable storage medium. As used herein, the terms “computer-readable data carrier” and “computer-readable storage medium” specifically may refer to non-transitory data storage means, such as a hard-ware stor- age medium having stored thereon computer-executable instructions. The computer-readable data carrier or storage medium specifically may be or may comprise a storage medium such as a random-access memory (RAM) and/or a read-only memory (ROM). Thus, specifically, one, more than one or even all of method steps as indicated above may be performed by using a computer or a computer network, preferably by using a computer pro- gram. In a further aspect a computer-readable storage medium comprising instructions which, when executed by a computer or computer network, cause to carry out at least steps a) to d), in par- ticular all steps, of the method according to the present invention. Further disclosed and proposed herein is a data carrier having a data structure stored thereon, which, after loading into a computer or computer network, such as into a working memory or main memory of the computer or computer network, may execute the method according to one or more of the embodiments disclosed herein. Further disclosed and proposed herein is a computer program product with program code means stored on a machine-readable carrier, in order to perform the method according to one or more of the embodiments disclosed herein, when the program is executed on a computer or computer network. As used herein, a computer program product refers to the program as a trad- able product. The product may generally exist in an arbitrary format, such as in a paper format, or on a computer-readable data carrier and/or on a computer-readable storage medium. Specifi- cally, the computer program product may be distributed over a data network. Finally, disclosed and proposed herein is a modulated data signal which contains instructions readable by a computer system or computer network, for performing the method according to one or more of the embodiments disclosed herein. Referring to the computer-implemented aspects of the invention, one or more of the method steps or even all of the method steps of the methods according to one or more of the embodi- ments disclosed herein may be performed by using a computer or computer network. Thus, generally, any of the method steps including provision and/or manipulation of data may be per- formed by using a computer or computer network. Generally, these method steps may include any of the method steps, typically except for method steps requiring manual work. Specifically, further disclosed herein are: - a computer or computer network comprising at least one processor, wherein the processor is adapted to perform the method according to one of the embodiments described in this description, - a computer loadable data structure that is adapted to perform the method according to one of the embodiments described in this description while the data structure is being executed on a computer, - a computer program, wherein the computer program is adapted to perform the method according to one of the embodiments described in this description while the program is being executed on a computer, - a computer program comprising program means for performing the method according to one of the embodiments described in this description while the computer program is being executed on a computer or on a computer network, - a computer program comprising program means according to the preceding embodiment, wherein the program means are stored on a storage medium readable to a computer, - a storage medium, wherein a data structure is stored on the storage medium and wherein the data structure is adapted to perform the methods according to one of the embodiments described in this description after having been loaded into a main and/or working storage of a computer or of a computer network, and - a computer program product having program code means, wherein the program code means can be stored or are stored on a storage medium, for performing the method ac- cording to one of the embodiments described in this description, if the program code means are executed on a computer or on a computer network. In a further aspect, a pick system is proposed. With respect to definitions and embodiments of the pick system reference is made to definitions and embodiments described with respect to the method. The term "system" as used herein is a broad term and is to be given its ordinary and customary meaning to a person of ordinary skill in the art and is not to be limited to a special or customized meaning. The term specifically may refer, without limitation, to an arbitrary set of interacting or interdependent components parts forming a whole. Specifically, the components may interact with each other in order to fulfill at least one common function. The at least two components may be handled independently or may be coupled or connectable. The components of the pick system may be configured for interacting for performing a picking function, in particular a gripping func- tion. The pick system comprising - at least one robot configured for picking items comprised in a carrier; - at least one camera configured for imaging the carrier at a plurality of imaging positions; - at least one projector configured for projecting at least one illumination pattern on a scene comprising the carrier, wherein the camera is configured for imaging at least one reflection image, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; - at least one processing unit configured for determining material characteristics for each reflection feature by evaluating the reflection image, wherein the evaluation comprises, for each reflection feature, an analysis of its respective beam profile, wherein the processing unit is con- figured for assigning a score according to their probability to lead to a successful grasp to pick point candidates using a trained scoring model, wherein the determined material characteristics in a close environment around the pick point candidate are used as input for the trained scoring model; - at least one control unit configured for selecting the next pick point of the robot considering the assigned scores. The pick system may be configured for performing the method for picking of items comprised in a carrier by using at least one robot according to the present invention. Overall, in the context of the present invention, the following embodiments are regarded as pre- ferred: Embodiment 1 A computer implemented method for picking of items comprised in a carrier by using at least one robot, wherein the method comprises the following steps: a) at least one carrier scanning step comprising imaging the carrier at a plurality of imag- ing positions by using at least one camera; b) at least one material detection step comprising b1) projecting at least one illumination pattern on a scene comprising the carrier by using at least one projector and imaging at least one reflection image using the cam- era, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) determining material characteristics for each reflection feature by evaluating the reflection image by using at least one processing unit, wherein the evaluation com- prises, for each reflection feature, an analysis of its respective beam profile; c) at least one pick point scoring step, wherein the pick point scoring step comprises the processing unit assigning a score according to their probability to lead to a successful grasp to pick point candidates using a trained scoring model, wherein the determined material characteristics in a close environment around the pick point candidate are used as input for the trained scoring model; d) at least one picking step comprising at least one control unit selecting the next pick point of the robot considering the assigned scores. Embodiment 2 The method according to the preceding embodiment, wherein the method comprises a self learning grasp sequence for robot bin picking, wherein the method comprises approaching the selected next pick point with the robot and determining, by at least one sensor of the robot, sensor data of the approached pick point relating to suitability for grasping, retraining the trained scoring model on the sensor data and re- peating at least steps c) and d) of the method, preferably steps a) to d). Embodiment 3 The method according to the preceding embodiment, wherein the method comprises storing the determined sensor data in at least one database. Embodiment 4 The method according to any one of the preceding embodiments, wherein the score for the respective pin point candidate depends on the probability to lead to a successful grasp, wherein the score is assigned from low score for low probability to high score for high probability, wherein the pick point with the highest score is used as next pick point. Embodiment 5 The method according to any one of the preceding embodiments, wherein the method comprises simulating pick point approaches, wherein the processing unit sim- ulates for all pick point candidates whether approaching them leads to collisions with other items, wherein pick point candidates simulated to lead to collisions are filtered out for further consideration. Embodiment 6 The method according to any one of the preceding embodiments, wherein the determining of material characteristics is based on a convolutional neural network classifying the reflection features of the reflection image. Embodiment 7 The method according to any one of the preceding embodiments, wherein the material characteristics is at least one information selected from the group consisting of information about softness, information about deformability, or information about permeability to air. Embodiment 8 The method according to any one of the preceding embodiments, wherein step d) comprises the control unit to optimize one or more of the pick point selection, pick pose, depth offset and vacuum control of the grabber considering the determined material characteristics. Embodiment 9 The method according to any one of the preceding embodiments, wherein the scoring model is a classification and/or regression model, wherein the scoring model is at least one model selected from the group consisting of a random forest (RF) or a convolution neural network (CNN). Embodiment 10 The method according to any one of the preceding embodiments, wherein step b) comprises the processing unit determining depth information for each reflec- tion feature by evaluating the reflection image, wherein the evaluation comprises, for each reflection feature, an analysis of its respective beam profile using a depth-from- photon ratio technique. Embodiment 11 The method according to any one of the preceding embodiments, wherein in step c) in addition to the material characteristics one or more of the following input pa- rameters are provided to the scoring model: information about an image section, 3D information about the reflection features. Embodiment 12 The method according to any one of the preceding embodiments, wherein an output of the scoring model is a probability distribution over classes, wherein the out- put is one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high. Embodiment 13 The method according to any one of the preceding embodiments, wherein the robot comprises at least one robot arm, wherein the robot arm is equipped with a vac- uum grabber and a vacuum sensor. Embodiment 14 The method according to the preceding embodiment, wherein the control unit is configured for moving the robot arm to a picking position for the next pick consider- ing the assigned scores. Embodiment 15 The method according to any one of the two preceding embodiments, wherein the method comprises the control unit controlling the vacuum sensor for approaching the pick point, wherein each grasp is labeled as positive if the vacuum sensor returns stable vacuum values after approaching the pick point or negative if vacuum cannot be maintained. Embodiment 16 The method according to any one of the preceding embodiments, wherein the projector comprises at least one laser light source, wherein the illumination pattern comprises a grid of laser spots. Embodiment 17 The method according to the preceding embodiment, wherein the laser spots have wavelengths in a near infrared (NIR) regime. Embodiment 18 The method according to any one of the preceding embodiments, wherein the camera comprises at least one CCD chip and/or at least one CMOS chip. Embodiment 19 The method according to any one of the preceding embodiments, wherein the camera is or comprises at least one near infrared camera. Embodiment 20 The method according to any one of the preceding embodiments, wherein in step b), multiple camera positions are computed that cover the entire carrier to ensure that all items can be picked. Embodiment 21 Computer program for picking of items comprised in a carrier by using at least one robot, configured for causing a computer or a computer network to fully or partially perform the method according to any one of the preceding embodiments, when exe- cuted on the computer or the computer network, wherein the computer program is configured for performing and/or executing at least steps a) to d) of the method ac- cording to any one of the preceding embodiments. Embodiment 22 A computer-readable storage medium comprising instructions which, when ex- ecuted by a computer or computer network, cause to carry out at least steps a) to d) of the method according to any one of the preceding embodiments referring to a method. Embodiment 23 A pick system comprising - at least one robot configured for picking items comprised in a carrier; - at least one camera configured for imaging the carrier at a plurality of imaging posi- tions; - at least one projector configured for projecting at least one illumination pattern on a scene comprising the carrier, wherein the camera is configured for imaging at least one reflection image, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; - at least one processing unit configured for determining material characteristics for each reflection feature by evaluating the reflection image, wherein the evaluation com- prises, for each reflection feature, an analysis of its respective beam profile, wherein the processing unit is configured for assigning a score according to their probability to lead to a successful grasp to pick point candidates using a trained scoring model, wherein the determined material characteristics in a close environment around the pick point candidate are used as input for the trained scoring model; - at least one control unit configured for selecting the next pick point of the robot con- sidering the assigned scores. Embodiment 24 The pick system according to the preceding embodiment, wherein the pick system is configured for performing the method for picking of items comprised in a carrier by using at least one robot according to any one of the preceding embodiments referring to a method. Brief description of the figures Further optional details and features of the invention are evident from the description of pre- ferred exemplary embodiments which follows in conjunction with the dependent claims. In this context, the particular features may be implemented in an isolated fashion or in combination with other features. The invention is not restricted to the exemplary embodiments. The exem- plary embodiments are shown schematically in the figures. Identical reference numerals in the individual figures refer to identical elements or elements with identical function, or elements which correspond to one another with regard to their functions. Specifically, in the figures: Figure 1 shows an embodiment of a method according to the present invention; Figure 2 shows an embodiment of a picking system according to the present invention; and Figure 3 shows an embodiment of a scoring model comprising a CNN network structure. Detailed description of the embodiments: Figure 1 shows an embodiment of a computer implemented method for picking of items 110 comprised in a carrier 112 by using at least one robot 114 according to the present invention. The carrier 112 may be a device configured for containing at least one item 110, in particular a plurality of items 110. The carrier 112 may have support surface on which the items 110 can be placed. The carrier 112 may have sidewalls. For example, the carrier 112 may be a box. The item 110 may be arbitrary objects to be picked. The items 110 may be made from at least one material selected from the group consisting of cardboard, glass, metal or plastic. For exam- ple, the items may comprise screws, bottles, food and the like. The carrier 112 may comprise items of identical size, shape, material and reflection and/or absorption properties. The carrier 112 may comprise items having varying size, shape, material and reflection and/or absorption properties. The robot 114 may be configured for performing a task in the absence of external assistance or instructions, such as for example without assistance from a user or control by a user. Specifically, the robot 114 may be configured to react to specific situations independent from external input, e.g. by using pre-programmed routines, self-learning mechanisms or the like. The robot 114 may be selected from the group consisting of a commercial robot; an industrial robot, specifically a manufacturing robot. The robot 114 may comprise at least one robot arm 116 equipped with at least one end effector 118 at one end of the robot arm 116. The robot arm 116 may be configured for moving the end effector 118 along three axis of movement. The robot 114 may be configured for performing at least one gripping function. For example, the gripping function may comprise one or more of clamping, holding, tilting, lifting, positioning, mov- ing, handling, or transporting the item. Specifically, the gripping may comprise picking the item. The picking may comprise one or more of selecting the item 110, interacting with a surface of the item 110 such as contacting, or lifting the item 110. For example, the gripping may comprise lifting and transporting items 110 to automate commissioning in a warehouses and/or in a manufactur- ing process. The pick point may be point of interaction between the end effector 118 and the item 110. The pick point may be a point on the surface of the item 110. The end effector 118 may comprise at least one gripper 120. For example, the gripper 120 may be a vacuum grabber. The vacuum grabber may be configured for applying an attractive force to a surface of an item to be gripped. The vacuum grabber may comprise at least one vacuum cup, also denoted as suction cup, such as a rubber suction cup or polyurethane suction cup to pick up items 110. The vacuum cups may have a round shape. The vacuum cup may be configured for interacting with the surface of the item 110. The vacuum grabber may comprise at least one vacuum pump configured for generating vacuum. The vacuum grabber may comprise at least one pneumatic valve configured for sucking air out of the vacuum cup for attaching the item 110 to the vacuum cup, thereby picking up the item. The robot arm 116 may be configured for moving the vacuum grabber with attached item to a further position, in particular outside the carrier. Other embodiments of a robot are possible. An embodiment of a pick system 113 comprising a robot 114 is shown in Figure 2. The pick system 113 further may comprise the camera 124, the projector 130, the flood light source 148, the processing unit 134 and the control unit 144. The method steps as shown in Figure 1 may be performed in the given order or may be per- formed in a different order. Further, one or more additional method steps may be present which are not listed. Further, one, more than one or even all of the method steps may be performed repeatedly. For details, options and definitions, reference may be made to the display device as discussed above. Thus, specifically, as outlined above, the method may comprise using the dis- play device according to the present invention, such as according to one or more of the embodi- ments given above or given in further detail below. The method comprises the following steps: a) at least one carrier scanning step 122 comprising imaging the carrier 112 at a plurality of imaging positions by using at least one camera 124; b) at least one material detection step 126 comprising b1) (128) projecting at least one illumination pattern on a scene comprising the car- rier 112 by using at least one projector 130 and imaging at least one reflection image using the camera 124, wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) (132) determining material characteristics for each reflection feature by evalu- ating the reflection image by using at least one processing unit 134, wherein the eval- uation comprises, for each reflection feature, an analysis of its respective beam profile; c) at least one pick point scoring step 136, wherein the pick point scoring step 126 com- prises the processing unit 134 assigning a score according to their probability to lead to a successful grasp to pick point candidates 138 using a trained scoring model 140, wherein the determined material characteristics in a close environment around the pick point candidate 138 are used as input for the trained scoring model 140; d) at least one picking step 142 comprising at least one control unit 144 selecting the next pick point of the robot 114 considering the assigned scores. The scanning 122 may be or may comprise a process of imaging at a plurality of imaging positions 146. The imaging positions may refer to different positions of the camera in space. The scanning may be performed to account for trajectory planning and collision avoidance for the subsequent picking. The imaging positions may be set by moving the robot arm 116. The robot arm 116 may be equipped with the camera 124, the projector 130 and an optional flood light source 148. Step a) may comprise illuminating the carrier 112, e.g. by using the at least one flood light source 148. Additionally or alternatively, ambient light may be used. The flood light source 148 may be configured for providing the at least one illumination light beam for illumination of the carrier 112. The flood light source 148 may be configured for scene illumination. The scene illumination may comprise diffuse and/or uniform illumination of the scene. The flood light source 148 may be adapted to directly or indirectly illuminating the carrier 112, wherein the illumination is reflected or scattered by surfaces of the carrier 112 and, thereby, is at least partially directed towards the camera 124. The flood light source 148 may be adapted to illuminate the carrier 112, for example, by directing a light beam towards the carrier 112, which reflects the light beam. The flood light source 148 may be configured for generating an illuminating light beam for illuminating the carrier 112. The flood light source 148 may comprise at least one light-emitting-diode (LED). The flood light source 148 may illuminate the scene with the LED and, in particular, without the illumination pat- tern, and the camera 124 may be configured for capturing a two-dimensional image of the scene. The flood light source 148 may comprise a single light source or a plurality of light sources. As an example, the light emitted by the flood light source 148 may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 µm. Specifically, the light in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm may be used. The flood light source 148 may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodi- ments, the flood light source 148 may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths channels. The projector 130 and flood light source 148 may constitute a coordinate system, wherein a lon- gitudinal coordinate is a coordinate along the optical axis. The coordinate system may be a po- lar coordinate system in which the optical axis forms a z-axis and in which a distance from the z- axis and a polar angle may be used as additional coordinates. A direction parallel or antiparallel to the z-axis may be considered a longitudinal direction, and a coordinate along the z-axis may be considered a longitudinal coordinate z. Any direction perpendicular to the z-axis may be con- sidered a transversal direction, and the polar coordinate and/or the polar angle may be consid- ered a transversal coordinate. Depth information may relate to the longitudinal coordinate and/or information from which the longitudinal coordinate can be derived. The projector 130 may comprise at least one illumination device configured for providing the at least one illumination pattern. The projector 130 may be or may comprise at least one light source or at least one multiple beam light source. For example, the projector 130 may comprise at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the projector 130 may comprise at least one laser and/or laser source. Various types of lasers may be employed, such as semi- conductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed bragg reflector lasers, polariton la- sers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grat- ing lasers, Indium Arsenide lasers, transistor lasers, diode pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, Gallium Arsenide lasers, semiconductor ring laser, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources may be used, such as LEDs, micro-light emitting diode (LED), and/or light bulbs. The projector 130 may comprise one or more diffractive optical ele- ments (DOEs) configured for generating the illumination pattern. For example, the projector 130 may be adapted to generate and/or to project a cloud of points, for example the projector source may comprise one or more of at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light emitting diodes; at least one array of laser light sources. On account of their generally defined beam profiles and other properties of handleability, the use of at least one la- ser source is particularly preferred. The projector 130 and the flood light source 148 may be in- tegrated into a housing. Further, the projector 130 may be configured for emitting modulated or non-modulated light. In case a plurality of light sources is used, the different light sources may have different modula- tion frequencies which later on may be used for distinguishing the light beams. The illumination pattern may be an arbitrary pattern comprising at least one illumination feature adapted to illuminate at least one part of the scene. The illumination feature may be at least one at least partially extended feature of the pattern. The illumination pattern may comprise a single illumination feature. The illumination pattern may comprise a plurality of illumination features. The illumination pattern may be selected from the group consisting of: at least one point pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern comprising an arrangement of periodic or non periodic features. The illumination pattern may comprise regular and/or constant and/or periodic pattern such as a triangular pat- tern, a rectangular pattern, a hexagonal pattern or a pattern comprising further convex tilings. The illumination pattern may exhibit the at least one illumination feature selected from the group consisting of: at least one point; at least one line; at least two lines such as parallel or crossing lines; at least one point and one line; at least one arrangement of periodic or non-periodic fea- ture; at least one arbitrary shaped featured. The illumination pattern may comprise at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo- random point pattern; a random point pattern or a quasi random pattern; at least one Sobol pat- tern; at least one quasiperiodic pattern; at least one pattern comprising at least one pre-known feature at least one regular pattern; at least one triangular pattern; at least one hexagonal pat- tern; at least one rectangular pattern at least one pattern comprising convex uniform tilings; at least one line pattern comprising at least one line; at least one line pattern comprising at least two lines such as parallel or crossing lines. A distance between two features, in particular spots, of the illumination pattern and/or an area of the at least one illumination feature may depend on the circle of confusion in the reflection image. For example, the projector 130 may be adapted to generate and/or to project a cloud of points. The projector 130 may comprise the at least one light projector adapted to generate a cloud of points such that the illumination pattern may comprise a plurality of point pattern. The projector 130 may comprise at least one mask adapted to generate the illumination pattern from at least one light beam generated by the light source. For example, the projector 130 comprises at least one laser light source, wherein the illumina- tion pattern comprises a grid of laser spots. For example, the projector 130 comprises at least one laser source which is designated for generating laser radiation. The projector may comprise the at least one diffractive optical element (DOE). The projector 130 may be at least one point projector, such as the at least one laser source and the DOE, adapted to project at least one periodic point pattern. The light emitted by the projector 130 may have a wavelength of 300 to 1100nm, especially 500 to 1100 nm. Additionally or alternatively, light in the infrared spectral range may be used, such as in the range of 780 nm to 3.0 µm. The laser spots may have wavelengths in a near infrared (NIR) regime. Specifically, the light in the part of the near infrared region where silicon photodi- odes are applicable specifically in the range of 700 nm to 1100 nm may be used. For example, the projector 130 may be configured for emitting light beams at a wavelength range from 800 to 1000 nm, preferably at 940 nm, since terrestrial sun radiation has a local minimum in irradiance at this wavelength, e.g. as described in CIE 085- 1989 „Solar spectral Irradiance”. The projector 130 may be configured for emitting light at a single wavelength. Specifically, the wavelength may be in the near infrared region. In other embodiments, the projector may be adapted to emit light with a plurality of wavelengths allowing additional measurements in other wavelengths channels. The camera 124 may have at least one imaging element configured for recording or capturing spatially resolved one-dimensional, two-dimensional or even three-dimensional optical data or information. The camera 124 may comprise at least one pixelated camera chip. As an example, the camera 124 may comprise at least one camera chip, such as at least one CCD chip and/or at least one CMOS chip configured for recording images. The image may be data recorded by using the camera 124, such as a plurality of electronic readings from the camera 124, such as the pixels of the camera chip. The camera 124 may be or may comprise at least one photodetector, preferably inorganic pho- todetectors, more preferably inorganic semiconductor photodetectors, most preferably silicon photodetectors. Specifically, the camera 124 may be sensitive in the infrared spectral range. Specifically, the camera 124 may be or may comprise at least one inorganic photodiode which are sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 microme- ters. Specifically, the camera 124 may be sensitive in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm. Infrared opti- cal sensors which may be used for the camera 124 may be commercially available infrared opti- cal sensors, such as infrared optical sensors commercially available under the brand name HertzstueckTM from trinamiXTM GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, as an example, the camera 124may comprise at least one optical sensor of an intrinsic photovol- taic type, more preferably at least one semiconductor photodiode selected from the group con- sisting of: a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode. Additionally or alternatively, the cam- era 124 may comprise at least one optical sensor of an extrinsic photovoltaic type, more prefer- ably at least one semiconductor photodiode selected from the group consisting of: a Ge:Au pho- todiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, a Si:As photodiode. Additionally or alternatively, the camera 124 may comprise at least one pho- toconductive sensor such as a PbS or PbSe sensor, a bolometer, preferably a bolometer se- lected from the group consisting of a VO bolometer and an amorphous Si bolometer. The camera 124 may be sensitive in one or more of the ultraviolet, the visible or the infrared spectral range. Specifically, the camera 124 may be sensitive in the visible spectral range from 500 nm to 780 nm, most preferably at 650 nm to 750 nm or at 690 nm to 700 nm. Specifically, the camera 124 may be sensitive in the near infrared region. Specifically, the camera 124 may be sensitive in the part of the near infrared region where silicon photodiodes are applicable spe- cifically in the range of 700 nm to 1000 nm. The camera 124, specifically, may be sensitive in the infrared spectral range, specifically in the range of 780 nm to 3.0 micrometers. For example, the camera 124 may be or may comprise at least one element selected from the group consist- ing of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photocon- ductor, a phototransistor or any combination thereof. Any other type of photosensitive element may be used. The photosensitive element generally may fully or partially be made of inorganic materials and/or may fully or partially be made of organic materials. Most commonly, one or more photodiodes may be used, such as commercially available photodiodes, e.g. inorganic semiconductor photodiodes. As an example, the camera 124 may be a fix-focus camera, having at least one lens which is fixedly adjusted with respect to the camera. Alternatively, however, the camera 124 may also comprise one or more variable lenses which may be adjusted, automatically or manually. The camera 124 is configured for imaging the carrier 112 at a plurality imaging positions 146 of the camera 124. The images may be generated in response to the diffuse and/or uniform illumi- nation of the carrier 112 by the flood light source 148. The images generated in response to the diffuse and/or uniform illumination of the carrier 112 by the flood light source 148 may not com- prise any reflection features generated by the illumination pattern. The image may be at least one two-dimensional image. The image may be an RGB (red green blue) image. The camera 124 is configured for imaging the at least one reflection image. The reflection im- age comprises a plurality of reflection features generated by the scene in response to the illumi- nation pattern. The reflection feature may be a feature in an image plane generated by the scene in response to illumination, specifically with at least one illumination feature. Each of the reflection features comprises at least one beam profile, also denoted reflection beam profile. The beam profile of the reflection feature may generally refer to at least one intensity distribution of the reflection feature, such as of a light spot on the camera 124, as a function of the pixel. The beam profile may be selected from the group consisting of a trapezoid beam profile; a trian- gle beam profile; a conical beam profile and a linear combination of Gaussian beam profiles. The evaluation of the reflection image may comprise identifying the reflection features of the re- flection image. The processing unit 134 may be configured for performing at least one image analysis and/or image processing in order to identify the reflection features. The image analysis and/or image processing may use at least one feature detection algorithm. The image analysis and/or image processing may comprise one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between an image created by the sensor signals and at least one offset; an inversion of sensor signals by inverting an image cre- ated by the sensor signals; a formation of a difference image between an image created by the sensor signals at different times; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a blob detector; applying a corner detector; applying a Determinant of Hessian filter; applying a principle curvature-based region detector; applying a maximally stable extremal regions detector; applying a generalized Hough-transformation; ap- plying a ridge detector; applying an affine invariant feature detector; applying an affine-adapted interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform; applying a scale-space extrema detector; applying a local feature detector; applying speeded up robust features algorithm; applying a gra- dient location and orientation histogram algorithm; applying a histogram of oriented gradients descriptor; applying a Deriche edge detector; applying a differential edge detector; applying a spatio-temporal interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transformation; applying a Radon-transfor- mation; applying a Hough-transformation; applying a wavelet-transformation; a thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, such as by recognizing a feature within the image generated by the camera 124. The processing unit 134 may be configured for determining the beam profile of the respective reflection feature. The determining the beam profile may comprise identifying at least one reflec- tion feature provided and/or selecting at least one reflection feature and evaluating at least one intensity distribution of the reflection feature. As an example, a region of a matrix constituted by pixels of the camera 124 may be used and evaluated for determining the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the matrix. As an example, a center of illumination by the light beam may be determined, such as by determining the at least one pixel having the highest illumination, and a cross-sectional axis may be chosen through the center of illumination. The intensity distribution may an intensity distribution as a function of a coordinate along this cross- sectional axis through the center of illumination. Other evaluation algorithms are feasible. The reflection feature may cover or may extend over at least one pixel of the refelction image. For example, the reflection feature may cover or may extend over plurality of pixels. The pro- cessing unit 134 may be configured for determining and/or for selecting all pixels connected to and/or belonging to the reflection feature, e.g. a light spot. The processing unit 134 may be con- figured for determining the center of intensity by
Figure imgf000051_0001
wherein Rcoi is a position of center of intensity, rpixel is the pixel position and with j
Figure imgf000051_0002
being the number of pixels j connected to and/or belonging to the reflection feature and Itotal be- ing the total intensity. The processing unit 134 is configured for determining material characteristics for each reflection feature by analysis of its respective beam profile. The material characteristics may be an arbi- trary material property m derived from and/or relating to the beam profile of the reflection fea- ture. The material characteristics may be at least one information selected from the group con- sisting of information about softness, information about deformability, or information about per- meability to air. The material characteristics may be extracted for distinguishing between fore- ground and background, i.e. items 110 to pick vs. carrier 112. The processing unit 134 may be configured for determining the material property m of the sur- face remitting the reflection feature by evaluating the beam profile of the reflection feature. The material property may be at least one arbitrary property of the material configured for character- izing and/or identification and/or classification of the material. For example, the material prop- erty may be a property selected from the group consisting of: roughness, penetration depth of light into the material, a reflectivity, a specular reflectivity, a diffuse reflectivity, a surface prop- erty, a measure for translucence, a scattering, specifically a back-scattering behavior or the like. The at least one material property may be a property selected from the group consisting of: a scattering coefficient, a translucency, a transparency, a deviation from a Lambertian surface re- flection, a speckle, and the like. The processing unit 134 may be configured for identifying a reflection feature as to be gener- ated by an item 110 having a specific material property in case its reflection beam profile fulfills at least one predetermined or predefined criterion. The at least one predetermined or prede- fined criterion may be at least one property and/or value suitable to distinguish material proper- ties. The predetermined or predefined criterion may be or may comprise at least one predeter- mined or predefined value and/or threshold and/or threshold range referring to a material prop- erty. The reflection feature may be indicated as to be generated by an item 110 having a spe- cific material property in case the reflection beam profile fulfills the at least one predetermined or predefined criterion. The determining at least one material property may comprise assigning the material property to respective reflection feature. The processing unit 134 may comprise at least one database com- prising a list and/or table, such as a lookup list or a lookup table, of predefined and/or predeter- mined material properties. The list and/or table of material properties may be determined and/or generated by performing at least one test measurement, for example by performing material tests using samples having known material properties. The list and/or table of material proper- ties may be determined and/or generated at the manufacturer site and/or by a user. The mate- rial property may additionally be assigned to a material classifier such as one or more of a ma- terial name, a material group such as biological or non-biological material, translucent or non- translucent materials, metal or non-metal, fur or non-fur, carpet or non-carpet, reflective or non- reflective, specular reflective or non-specular reflective, foam or non-foam, roughness groups or the like. The processing unit 134 may comprise at least one database comprising a list and/or table comprising the material properties and associated material name and/or material group. For determining the material characteristic in step b), beam profile analysis may be used. Spe- cifically, beam profile analysis makes use of reflection properties of coherent light projected onto object surfaces to classify materials. The classification of materials may be performed as de- scribed in WO 2020/187719, in EP application 20159984.2 filed on February 28, 2020 and/or EP application 20154961.5 filed on January 31, 2020, the full content of which is included by reference. Specifically, a periodic grid of laser spots, e.g. a hexagonal grid as described in EP application 20170905.2 filed on April 22, 2020, is projected and the reflection image is rec- orded with the camera. Analyzing the beam profile of each reflection feature recorded by the camera may be performed by feature-based methods and/or using based on a convolutional neural network classifying the reflection features of the reflection image. The feature based methods may be used in combination with machine learning methods which may allow para- metrization of a classification model. Convolutional neuronal networks may be utilized to classify materials by using the reflection images as an input. Step b) may comprise using artificial intelligence, in particular convolutional neuronal networks. Using reflection images as input for convolutional neuronal networks may enable the generation of classification models with sufficient accuracy to differentiate between materials. Specifically, in step b) at least one parametrized classification model may be used. The parametrized classi- fication model may be configured for classifying materials by using the reflection image as an input. The classification model may be parametrized by using one or more of machine learning, deep learning, neural networks, or other form of artificial intelligence. The classification model may be configured for discriminating materials. The material characteristics may be determined by applying an optimization algorithm in terms of at least one optimization target on the classifi- cation model. The machine learning may be based on at least one neuronal network, in particu- lar a convolutional neural network. Weights and/or topology of the neuronal network may be pre-determined and/or pre-defined. Specifically, the training of the classification model may be performed using machine-learning. The classification model may comprise at least one ma- chine-learning architecture and model parameters. For example, the machine-learning architec- ture may be or may comprise one or more of: linear regression, logistic regression, random for- est, naive Bayes classifications, nearest neighbors, neural networks, convolutional neural net- works, generative adversarial networks, support vector machines, or gradient boosting algo- rithms or the like. The classification model may be at least partially data-driven. For example, the classification model may be based on experimental data. For example, training of the classi- fication model may comprise using at least one training dataset, wherein the training data set comprises images, in particular refection images, of a plurality of items with known material property. In step b), multiple camera positions 146 may be computed that cover the entire carrier 112 to ensure that all items 110 can be picked. These positions may be approached in a row until no item can be identified anymore and all items 110 are covered. Step b) may comprise the processing unit 134 determining depth information for each reflection feature by evaluating the reflection image. The evaluation may comprises, for each reflection feature, an analysis of its respective beam profile using a depth-from-photon ratio technique. The analysis of the beam profile comprise evaluating of the beam profile and may comprise at least one mathematical operation and/or at least one comparison and/or at least symmetrizing and/or at least one filtering and/or at least one normalizing. For example, the analysis of the beam profile may comprise at least one of a histogram analysis step, a calculation of a differ- ence measure, application of a neural network, application of a machine learning algorithm. The processing unit 134 may be configured for symmetrizing and/or for normalizing and/or for filter- ing the beam profile, in particular to remove noise or asymmetries from recording under larger angles, recording edges or the like. The processing unit 134 may filter the beam profile by re- moving high spatial frequencies such as by spatial frequency analysis and/or median filtering or the like. Summarization may be performed by center of intensity of the light spot and averaging all intensities at the same distance to the center. The processing unit 134 may be configured for normalizing the beam profile to a maximum intensity, in particular to account for intensity differ- ences due to the recorded distance. The processing unit 134 may be configured for removing influences from background light from the beam profile, for example, by an imaging without illu- mination. The processing unit 134 may be configured for determining at least one longitudinal coordinate zDPR for each of the reflection features by analysis of the beam profile of the respective reflection feature. The processing unit 134 may be configured for determining the longitudinal coordinate zDPR for the reflection features by using the so called depth-from-photon-ratio technique, also denoted as beam profile analysis. With respect to depth-from-photon-ratio (DPR) technique ref- erence is made to WO 2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, the full content of which is included by reference. The method may comprise simulating pick point approaches 150. The processing unit 134 may be configured for simulating for all pick point candidates 138 whether approaching them leads to collisions with other items, e.g. carrier 112, robot 114 or other items 110, wherein pick point candidates 138 simulated to lead to collisions are filtered out for further consideration. The sim- ulation may be performed prior to pick point scoring 136. In step c), the processing unit 134 is configured for assigning a score to pick point candidates 138 according to their probability to lead to a successful grasp using the trained scoring model 140. For example, a successful grasp may be if the pick point candidate 138 is suitable for al- lowing stable vacuum. The grasp may be not successful if vacuum cannot be maintained at the respective pick point candidate. The probability to lead to a successful grasp may depend on one or more of softness, deformability, permeability to air and the like. The score may be a value assigned to the respective pick point candidate 138, wherein the value depends on the probability to lead to a successful grasp. For example, the score may range from 0 to 1. The score may be assigned from low score for low probability to high score for high probability. In step d) the pick point with the highest score is used as next pick point. The scoring model 140 may be or may comprise classification model and/or a regression model having as output a probability distribution over classes for the reflection features. The output of the scoring model 140 may be, for each of the reflection features, one score number between 0, in case probability for successful grasp is low, and 1, in case the probability for a successful grasp is high. In step c), in addition to the material characteristics, one or more of the following input parame- ters are provided to the scoring model 140: information about an image section, 3D information about the reflection features. The trained scoring model 140 is configured for generating using the input at least one output, in particular a prediction. The scoring model 140 may be at least one model selected from the group consisting of a ran- dom forest (RF) or a convolution neural network (CNN). For example, in Figure 3 an embodiment is shown in which the scoring model 140 may be based on a CNN architecture configured for classification. The convolutional neural network may be a multilayer convolutional neural network. The convolutional neural network may com- prise a plurality of convolutional layers. The convolutional layers may be followed by a plurality of fully-connected layers. The convolutional neural network may comprise a plurality of pooling layers. The structure of convolutional neural networks is generally known to the skilled person such as from en.wikipedia.org/wiki/Convolutional_neural_network#Convolutional. The CNN may be build by using the Keras library in Python. For Keras library in Python reference is made to https://keras.io/ or https://de.wikipedia.org/wiki/Keras. For example, in a first branch 152 of the CNN, material characteristics and 3D information may be fed 154 into a dense layer with a Scaled Exponential Linear Unit (SELU) activation function 156. The dense layer 156 may be followed by a batch normalization layer 158. The images de- termined in step a) may be fed 160, in a second branch of the CNN 162, into a 5x5 convolution kernel 164 with 16 filters. The convolution kernel 164 is followed by at least one batch normali- zation layer 166, a 2x2 Max Pooling layer 168 and a dense layer with a SELU activation func- tion 170. The output of first and second branches may be fed into a dense layer with a SELU activation function 172 followed by a dense layer having a sigmoid activation function 174. The output 176 may be a probability distribution over classes for the reflection features. The output 176 of the scoring model 140 may be, for each of the reflection features, one score number be- tween 0, in case probability for successful grasp is low, and 1, in case the probability for a suc- cessful grasp is high. The trained scoring model 140 was trained using at least one training dataset. The training da- taset may comprise a plurality of reflection images having reflection features of known probabil- ity for a successful grasp. The training may be performed for different items. The trained scoring model 140 can be re-trained and/or updated based on additional data. The trained scoring model 140 may be trained by using machine learning. The method may comprise at least one training step, wherein, in the training step, the scoring model 140 is trained on the at least one training dataset. Specifically, the training step may be performed before performing step c). In step c), the determined material characteristics in a close environment around the pick point candidate 138 are used as input for the trained scoring model 140. The close environment around the pick point candidate 138 may be or may comprise reflection features within a range or region around the pick point candidate 138. The pick point candidate 138 may coincide with a reflection feature. However, other embodiments are possible wherein the pick point candidate 138 and the reflection feature do not coincide. The processing unit 134 may select a range or image region around the pick point candidate 138 ensuring that at least one reflection feature is present, e.g. the reflection feature nearest to the pick point candidate 138. The processing unit 134 may be configured for considering the images imaged in step a) and the reflection image for identifying reflection features in a close environment around the pick point candidate 138. Once the scoring is completed, the pick point with the highest score may be taken for the next pick. The control unit 144 is configured for selecting the next pick point of the robot 114 consid- ering the assigned scores. The method, as shown in Figure 1, may comprises a self-learning grasp sequence 178 for robot bin picking. The method may comprise in step d) approaching 180 the selected next pick point with the robot 114 and determining, by at least one sensor 182 of the robot 114, sensor data of the approached pick point relating to suitability for grasping. The processing unit 134 may be configured for distinguishing between suitable pick points and non-suitable pick points. The method may further comprise retraining 184 the trained scoring model on the sensor data and repeating at least steps c) and d) of the method, preferably steps a) to d). The method may comprise storing the determined sensor data in at least one database. Thus, the model accu- mulates information with every pick and learns from it. For example, the robot 114 comprises the at least one robot arm 116, wherein the robot arm 116 is equipped with a vacuum grabber and a vacuum sensor. The control unit 144 may be con- figured for moving the robot arm 116 to a picking position for the next pick considering the as- signed scores. The method may comprise the control unit 144 controlling the vacuum sensor for approaching the pick point, wherein each grasp is labeled as positive if the vacuum sensor re- turns stable vacuum values after approaching the pick point 186 or negative if vacuum cannot be maintained 188. In case of negative success, data relating to 2D, 3D, material information may be dumped and the CNN may be retrained. The next pick point with the next highest score may be approach and tested next 190. This fully automated labeling can be used to build up a comprehensive database during operation without any manual processing. Step d) may comprise the control unit to optimize one or more of the pick point selection, pick pose, depth offset and vacuum control of the gripper 120 considering the determined material characteristics. List of reference numbers 110 item 112 carrier 113 pick system 114 robot 116 robot arm 118 end effector 120 gripper 122 carrier scanning step 124 camera 126 material detection step 128 projecting 130 projector 132 determining material characteristics 134 processing unit 136 pick point scoring step 138 pick point candidates 140 scoring model 142 picking step 144 control unit 146 imaging position 148 flood light source 150 simulating approaches 152 first branch 154 fed 156 dense layer with SELU 158 batch normalization layer 160 fed 162 second branch 164 5x5 convolution kernel 166 batch normalization layer 168 2x2 Max Pooling layer 170 dense layer with SELU 172 dense layer with SELU 174 dense layer having a sigmoid activation function 176 output 178 self-learning grasp sequence 180 approaching 182 sensor 184 retraining 186 positive 188 negative 190 approach pick point with next highest score

Claims

Claims 1. A computer implemented method for picking of items (110) comprised in a carrier (112) by using at least one robot (114), wherein the method comprises the following steps: a) at least one carrier scanning step (122) comprising imaging the carrier (112) at a plu- rality of imaging positions (146) by using at least one camera (124); b) at least one material detection step (126) comprising b1) projecting (128) at least one illumination pattern on a scene comprising the car- rier (112) by using at least one projector (130) and imaging at least one reflection image using the camera (124), wherein the reflection image comprises a plurality of reflection features generated by the scene in response to the illumination pattern, wherein each of the reflection features comprises a beam profile; b2) determining material characteristics (132) for each reflection feature by evalu- ating the reflection image by using at least one processing unit (134), wherein the evaluation comprises, for each reflection feature, an analysis of its respective beam profile; c) at least one pick point scoring step (136), wherein the pick point scoring step (136) comprises the processing unit (134) assigning a score according to their probability to lead to a successful grasp to pick point candidates (138) using a trained scoring model (140), wherein the determined material characteristics in a close environment around the pick point candidate (138) are used as input for the trained scoring model (140); d) at least one picking step (142) comprising at least one control unit (144) selecting the next pick point of the robot (114) considering the assigned scores.
2. The method according to the preceding claim, wherein the method comprises a self learn- ing grasp sequence (178) for robot bin picking, wherein the method comprises approach- ing the selected next pick point with the robot (114) and determining, by at least one sen- sor (182) of the robot (114), sensor data of the approached pick point relating to suitability for grasping, retraining (184) the trained scoring model on the sensor data and repeating at least steps c) and d) of the method, preferably steps a) to d).
3. The method according to the preceding claim, wherein the method comprises storing the determined sensor data in at least one database.
4. The method according to any one of the preceding claims, wherein the score for the re- spective pin point candidate (138) depends on the probability to lead to a successful grasp, wherein the score is assigned from low score for low probability to high score for high probability, wherein the pick point with the highest score is used as next pick point.
5. The method according to any one of the preceding claims, wherein the method comprises simulating pick point approaches (150), wherein the processing unit (134) simulates for all pick point candidates (138) whether approaching them leads to collisions with other items (110), wherein pick point candidates (138) simulated to lead to collisions are filtered out for further consideration.
6. The method according to any one of the preceding claims, wherein the material character- istics is at least one information selected from the group consisting of information about softness, information about deformability, or information about permeability to air.
7. The method according to any one of the preceding claims, wherein the scoring model (140) is a classification and/or regression model, wherein the scoring model (140) is at least one model selected from the group consisting of a random forest (RF) or a convolu- tion neural network (CNN).
8. The method according to any one of the preceding claims, wherein the robot (114) com- prises at least one robot arm (116), wherein the robot arm (116) is equipped with a vac- uum grabber and a vacuum sensor, wherein the control unit (144) is configured for moving the robot arm (116) to a picking position for the next pick considering the assigned scores.
9. The method according to any one of the two preceding claims, wherein the method com- prises the control unit (144) controlling the vacuum sensor for approaching the pick point, wherein each grasp is labeled as positive if the vacuum sensor returns stable vacuum val- ues after approaching the pick point or negative if vacuum cannot be maintained.
10. The method according to any one of the preceding claims, wherein the projector (130) comprises at least one laser light source, wherein the illumination pattern comprises a grid of laser spots.
11. The method according to any one of the preceding claims, wherein the camera (124) comprises at least one CCD chip and/or at least one CMOS chip.
12. Computer program for picking of items (110) comprised in a carrier (112) by using at least one robot (114), configured for causing a computer or a computer network to fully or partially perform the method according to any one of the preceding claims, when executed on the computer or the computer network, wherein the computer program is configured for performing and/or executing at least steps a) to d) of the method according to any one of the preceding claims.
13. A computer-readable storage medium comprising instructions which, when executed by a computer or computer network, cause to carry out at least steps a) to d) of the method according to any one of the preceding claims referring to a method.
14. A pick system (113) comprising - at least one robot (114) configured for picking items (110) comprised in a carrier (112); - at least one camera (124) configured for imaging the carrier (112) at a plurality of imaging positions (146); - at least one projector (130) configured for projecting at least one illumination pattern on a scene comprising the carrier (112), wherein the camera (124) is configured for imaging at least one reflection image, wherein the reflection image comprises a plu- rality of reflection features generated by the scene in response to the illumination pat- tern, wherein each of the reflection features comprises a beam profile; - at least one processing unit (134) configured for determining material characteristics for each reflection feature by evaluating the reflection image, wherein the evaluation comprises, for each reflection feature, an analysis of its respective beam profile, wherein the processing unit is configured for assigning a score according to their prob- ability to lead to a successful grasp to pick point candidates (138) using a trained scor- ing model (140), wherein the determined material characteristics in a close environ- ment around the pick point candidate (138) are used as input for the trained scoring model (140); - at least one control unit (144) configured for selecting the next pick point of the robot (114) considering the assigned scores.
15. The pick system (113) according to the preceding claim, wherein the pick system (113) is configured for performing the method according to any one of the preceding claims referring to a method.
PCT/EP2022/081227 2021-11-09 2022-11-09 Self learning grasp sequence for robot bin picking WO2023083848A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP21207082.5 2021-11-09
EP21207082 2021-11-09

Publications (1)

Publication Number Publication Date
WO2023083848A1 true WO2023083848A1 (en) 2023-05-19

Family

ID=78592586

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2022/081227 WO2023083848A1 (en) 2021-11-09 2022-11-09 Self learning grasp sequence for robot bin picking

Country Status (1)

Country Link
WO (1) WO2023083848A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117523157A (en) * 2023-11-21 2024-02-06 深圳华天高科电子有限公司 AI machine vision recognition method and system under dynamic sunlight irradiance
WO2024137630A3 (en) * 2022-12-20 2024-07-25 Liberty Reach Inc. Method and system for manipulating target items supported on a susbstantially horizontal support surface

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016010968A1 (en) * 2014-07-16 2016-01-21 Google Inc. Multiple suction cup control
US20160167228A1 (en) * 2014-12-16 2016-06-16 Amazon Technologies, Inc. Generating robotic grasping instructions for inventory items
WO2018091649A1 (en) 2016-11-17 2018-05-24 Trinamix Gmbh Detector for optically detecting at least one object
WO2020187719A1 (en) 2019-03-15 2020-09-24 Trinamix Gmbh Detector for identifying at least one material property
US10919151B1 (en) * 2018-03-23 2021-02-16 Amazon Technologies, Inc. Robotic device control optimization using spring lattice deformation model

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016010968A1 (en) * 2014-07-16 2016-01-21 Google Inc. Multiple suction cup control
US20160167228A1 (en) * 2014-12-16 2016-06-16 Amazon Technologies, Inc. Generating robotic grasping instructions for inventory items
WO2018091649A1 (en) 2016-11-17 2018-05-24 Trinamix Gmbh Detector for optically detecting at least one object
WO2018091640A2 (en) 2016-11-17 2018-05-24 Trinamix Gmbh Detector for optically detecting at least one object
WO2018091638A1 (en) 2016-11-17 2018-05-24 Trinamix Gmbh Detector for optically detecting at least one object
US10919151B1 (en) * 2018-03-23 2021-02-16 Amazon Technologies, Inc. Robotic device control optimization using spring lattice deformation model
WO2020187719A1 (en) 2019-03-15 2020-09-24 Trinamix Gmbh Detector for identifying at least one material property

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
BERENSON D ET AL: "Grasp planning in complex scenes", HUMANOID ROBOTS, 2007 7TH IEEE-RAS INTERNATIONAL CONFERENCE ON, IEEE, PISCATAWAY, NJ, USA, 29 November 2007 (2007-11-29), pages 42 - 48, XP031448821, ISBN: 978-1-4244-1861-9 *
J. MAHLERM. MATLX. LIUA. LID. V. GEALYK. GOLDBERG: "Dex-Net 3.0: Computing Robust Robot Suction Grasp Targets in Point Clouds using a New Analytic Model and Deep Learning", CORR, 2017
J. REDMONA. ANGELOVA: "Real-time grasp detection using convolutional neural networks", CORR, 2014
KALASHNIKOV DIRPAN APASTOR P ET AL.: "QT-Opt: Scalable Deep Reinforcement Learning for Vi-sion-Based Robotic Manipulation", ARXIV E-PRINTS, 2018
LEO BREIMAN: "Random Forests", MACHINE LEARNING, vol. 45, October 2001 (2001-10-01), pages 1
S. JAMES, A. J. DAVISON, E. JOHNS: "Transferring end-to-end visuomotor control from simula-tion to real world for a multi-stage task", CORL, 2017
S. REN, I. LENZ, H. LEE, A. SAXENA: "Deep learning for detecting robotic grasps", RSS, 2013

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024137630A3 (en) * 2022-12-20 2024-07-25 Liberty Reach Inc. Method and system for manipulating target items supported on a susbstantially horizontal support surface
CN117523157A (en) * 2023-11-21 2024-02-06 深圳华天高科电子有限公司 AI machine vision recognition method and system under dynamic sunlight irradiance

Similar Documents

Publication Publication Date Title
US11868863B2 (en) Systems and methods for joint learning of complex visual inspection tasks using computer vision
US11989896B2 (en) Depth measurement through display
US11436753B2 (en) Machine vision-based method and system to facilitate the unloading of a pile of cartons in a carton handling system
US9694498B2 (en) Imager for detecting visual light and projected patterns
WO2023083848A1 (en) Self learning grasp sequence for robot bin picking
WO2019075276A1 (en) Systems and methods for object identification
US20230078604A1 (en) Detector for object recognition
EP4111358A1 (en) Gesture recognition
WO2023072905A1 (en) Extended material detection involving a multi wavelength projector
CA3204014A1 (en) Machine vision-based method and system to facilitate the unloading of a pile of cartons in a carton handling system
US11906421B2 (en) Enhanced material detection by stereo beam profile analysis
US20230130353A1 (en) Method and System for Decanting a Plurality of Items Supported on a Transport Structure at One Time with a Picking Tool for Placement into a Transport Container
US20230121334A1 (en) Method and System for Efficiently Packing a Transport Container with Items Picked from a Transport Structure
US20230124076A1 (en) Method and System for Manipulating a Target Item Supported on a Substantially Horizontal Support Surface
US20230118445A1 (en) Method and System for Optimizing Pose of a Picking Tool with Respect to an Item to be Picked from a Transport Structure
US20230120703A1 (en) Method and System for Quickly Emptying a Plurality of Items from a Transport Structure
Sushkov Detection and pose determination of a part for bin picking
EP4490542A1 (en) 8bit conversion
WO2024137630A2 (en) Method and system for manipulating a multitude of target items supported on a susbstantially horizontal support surface one at a time
WO2024137732A2 (en) Method and system for manipulating a target item supported on a substantially horizontal support surface
Sushkov Detekce součástky a určení její polohy pro úlohu vybírání
Muttenthaler Object recognition for robotic grasping applications

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22813608

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22813608

Country of ref document: EP

Kind code of ref document: A1