CN116551672A - Systems and methods for robotic systems with object handling capabilities - Google Patents

Abstract

Systems and methods for a robotic system with object handling capabilities are disclosed. A computing system configured for object transfer is provided. The computing system includes at least one processing circuit configured to identify a pickable area of an object based on image information of the object. The pickable area may be determined from a surface cost map indicating the smoothness of each area of the image information, the surface cost map being determined from the height difference and the normal difference. The identification of pickable regions can be used in motion planning operations for conveying objects.

Description

Systems and methods for robotic systems with object handling capabilities

This application is a divisional application of the invention patent application 202310238393.9 with a filing date of March 8, 2023, entitled "System and Method for a Robotic System with Object Handling Capability".

Cross References to Related Applications

This application claims the benefit of US Provisional Application No. 63/317,877, filed March 8, 2022, entitled "ROBOTIC SYSTEM WITH OBJECTDETECTION," which is incorporated herein by reference in its entirety.

technical field

The present technology is directed generally to robotic systems, and more specifically to systems, processes and techniques for detecting and handling objects. More particularly, the present technique may be used to identify pickable regions of objects in containers.

Background technique

With the ever-improving performance and decreasing cost of robots, many robots (eg, machines configured to automatically/autonomously perform physical actions) are now widely used in various fields. For example, robots may be used to perform various tasks (eg, manipulating or transporting objects through a space) in manufacturing and/or assembly, packaging and/or packaging, transportation and/or shipping, and the like. When performing tasks, robots can replicate human movements, thereby replacing or reducing human involvement that would otherwise be required to perform dangerous or repetitive tasks.

However, despite technological advances, robots often lack the sophistication required to replicate the human interactions required to perform larger and/or more complex tasks. Accordingly, there remains a need for improved techniques and systems for managing operations and/or interactions between robots.

Contents of the invention

In an embodiment, a computing system is provided. The computing system includes: a control system configured to communicate with a robot having a robotic arm that includes or is attached to an end effector device; and at least one processing circuit that is configured to transmit Upon arrival in an object handling environment of an object source of a destination within the object handling environment, the processing circuitry is configured to: obtain image information of the object; identify one or more selected ones of the selected ones of the objects by Pickable regions of objects: generating a surface cost map from image information, segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects and generating a pickable area detection result including at least the at least one or more pickable areas; and generating a motion plan for the robotic system to transport the one or more selected objects, the motion plan being based on the pickable area detection result .

In an embodiment, there is provided a method of object transfer performed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm including or in addition to a camera, and to communicate with a camera. Connect to the end effector unit. The method includes: obtaining image information of one or more objects contained in an object source; identifying pickable regions of one or more selected ones of the objects by: generating a surface based on the image information a cost map, segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; and generating a pickable region including at least the one or more pickable regions region detection results; and generating, for the robotic system, a motion plan for conveying the one or more selected objects, the motion plan based on the pickable region detection results.

In an embodiment, a non-transitory computer readable medium configured with executable instructions for object transfer through a robot having at least one processing circuit and configured to communicate with a robot having a robotic arm and with The camera communicates with the control system implemented by the robotic arm that includes or is attached to an end effector device. The instructions may be configured to: obtain image information of one or more objects contained in the object source; identify pickable regions of one or more selected ones of the objects by: according to the image generating a surface cost map, segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; and generating at least the one or more pickable regions and generating, for the robotic system, a motion plan for transporting the one or more selected objects, the motion plan being based on the pickable area detection results.

Description of drawings

Figure 1A illustrates a system for performing or facilitating the detection, identification and retrieval of objects according to embodiments herein.

Figure IB illustrates an embodiment of a system for performing or facilitating the detection, identification and retrieval of objects according to embodiments herein.

Figure 1C illustrates another embodiment of a system for performing or facilitating the detection, identification and retrieval of objects according to embodiments herein.

Figure ID illustrates yet another embodiment of a system for performing or facilitating detection, identification and retrieval of objects according to embodiments herein.

2A is a block diagram illustrating a computing system configured to perform or facilitate detection, identification, and retrieval of objects, consistent with embodiments herein.

2B is a block diagram illustrating an embodiment of a computing system configured to perform or facilitate detection, identification, and retrieval of objects, consistent with embodiments herein.

2C is a block diagram illustrating another embodiment of a computing system configured to perform or facilitate detection, identification, and retrieval of objects, consistent with embodiments herein.

2D is a block diagram illustrating yet another embodiment of a computing system configured to perform or facilitate detection, identification, and retrieval of objects, consistent with embodiments herein.

Figure 2E is an example of image information processed by the system and is consistent with embodiments herein.

Figure 2F is another example of image information processed by the system and is consistent with embodiments herein.

FIG. 3A illustrates an exemplary object handling environment for operating a robotic system according to embodiments herein.

Figure 3B illustrates an exemplary object handling environment for operating a robotic system according to embodiments herein.

FIG. 3C illustrates an exemplary object handling environment for operating a robotic system according to embodiments herein.

4 is a flowchart illustrating an example process for handling detected objects.

FIG. 5A illustrates an example of 2D image information of a scene consistent with embodiments herein.

FIG. 5B illustrates an example of 3D image information of a scene consistent with embodiments herein.

FIG. 6A provides an example flowchart for a method of generating a surface cost map consistent with embodiments herein.

6B-6E provide examples of aspects of a surface costmap generation method consistent with embodiments herein.

Figure 6F provides an example of a height gradient costmap consistent with embodiments herein.

Figure 6G provides an example of a normal differences costmap consistent with embodiments herein.

Figure 6H provides an example of a surface cost map consistent with embodiments herein.

Figure 6I provides an example of an object consistent with embodiments herein.

Figure 7A provides an example of a segmentation method consistent with embodiments herein.

7B-7E provide examples of aspects of segmentation methods consistent with embodiments herein.

8A and 8B provide examples of aspects of detection maskinformation generation consistent with embodiments herein.

9A and 9B provide examples of aspects of safety volume generation consistent with embodiments herein.

Detailed ways

Systems and methods related to object detection, identification, and retrieval are described herein. In particular, the disclosed systems and methods may facilitate object detection, identification of pickable areas, and object retrieval if the object is located in a container. As discussed herein, objects may include boxes, bags, bags, and the like. Object handling in such situations can be challenging due to the irregular arrangement of the objects and the difficulty in identifying object regions or parts suitable for picking, for example with a suction gripping device. Accordingly, the systems and methods described herein are designed to identify pickable regions of objects from a set of objects, where the individual objects may be arranged at different locations, at different angles, etc. The systems and methods discussed herein may include robotic systems. Robotic systems configured in accordance with embodiments herein can autonomously perform integrated tasks by coordinating the operation of multiple robots. As described herein, robotic systems may include any suitable combination of robotic devices, actuators, sensors, cameras, and computing systems configured to control, issue commands, receive information from robotic devices and sensors, access, analyze, and process Data generated by robotic devices, sensors, and cameras to generate data or information that can be used to control robotic systems and plan actions for robotic devices, sensors, and cameras. As used herein, a robotic system does not require direct access to or control of robotic actuators, sensors, or other devices. As described herein, a robotic system may be a computing system configured to enhance the performance of such robotic actuators, sensors, and other devices by receiving, analyzing, and processing information.

The techniques described herein provide technical improvements to robotic systems configured for object identification, pickable area identification, and object delivery. The technological improvements described in this paper can increase the speed, precision, and accuracy of these tasks and further facilitate the detection, pickable area identification, and transfer of objects from source containers or storage to destinations. The robotic system and computing system described herein address the technical problems of identifying, detecting pickable areas, and retrieving objects from containers in which objects may be irregularly arranged. By solving this technical problem, techniques for object identification, pickable area detection, and object retrieval are improved.

This application relates to systems and robotic systems. As discussed herein, a robotic system may include robotic actuator assemblies (eg, robotic arms, robotic grippers, etc.), various sensors (eg, cameras, etc.), and various computing or control systems. As discussed herein, the computing system or control system may be referred to as "controlling" various robotic components, such as robotic arms, grippers, cameras, and the like. Such "control" may refer to direct control of and interaction with various actuators, sensors, and other functional aspects of robotic components. For example, a computing system can control a robotic arm by issuing or providing all the needed signals to cause the various motors, actuators, and sensors to cause the robot to move. Such "control" may also issue abstract or indirect commands to another robot control system, which then translates such commands into the necessary signals for causing the robot to move. For example, a computing system may control a robotic arm by issuing commands describing a trajectory or destination position to which the robotic arm should move, and another robotic control system associated with the robotic arm may receive and interpret such commands and then provide Various actuators and sensors provide the necessary direct signals to cause the desired movement.

In particular, techniques described herein facilitate robotic systems to interact with target objects among multiple objects in a container. The methods and systems described herein can identify pickable regions for selected objects from among a set of objects. As described herein, a robotic transport mechanism (eg, a robotic arm) may include a suction cup or suction gripper as part of an end effector device for grasping, picking up, or grasping an object. When applied to a smooth surface of an object (e.g., a portion of an object that has a sufficiently smooth surface profile such that a suction cup can engage and form a seal between the surface of the object and the suction cup to lift and transport the object), this suction-based grasping The performance of the grip device could be better. A surface that is smooth enough to properly engage a suction gripping device and large enough to accommodate one or more suction gripping devices in a robotic delivery system may be referred to as a "pickable area." When objects are loosely organized in source repositories or containers, the systems and methods described herein may be employed to identify pickable regions of objects.

In the following, specific details are set forth to provide an understanding of the presently disclosed technology. In embodiments, the techniques described herein may be practiced without including every specific detail disclosed herein. In other instances, well-known features such as specific functions or routines have not been described in detail to avoid unnecessarily obscuring the present disclosure. References in this specification to "an embodiment," "one embodiment," etc. mean that the particular feature, structure, material, or characteristic being described is included in at least one embodiment of the present disclosure. Thus, appearances of such phrases in this specification are not necessarily all referring to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, a particular feature, structure, material or characteristic described with respect to any one embodiment may be combined in any suitable manner with those of any other embodiment, unless such items are mutually exclusive. It should be understood that the various embodiments shown in the figures are illustrative representations only and are not necessarily drawn to scale.

In the interest of clarity, several details are not set forth in the following description, describing structures or processes that are well known and commonly associated with robotic systems and subsystems, but that might unnecessarily obscure some important aspects of the disclosed technology. Furthermore, while the following disclosure sets forth several embodiments of different aspects of the technology, several other embodiments may have different configurations or different components than those described in this section. Accordingly, the disclosed technology is capable of other embodiments with additional elements or without several of the elements described below.

Many of the embodiments or aspects of the disclosure described below may take the form of computer or controller-executable instructions, including routines, executed by a programmable computer or controller. Those skilled in the relevant art will understand that the disclosed techniques may be practiced on or with computer or controller systems other than those shown and described below. The techniques described herein may be embodied in a special purpose computer or data processor that is specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Accordingly, the terms "computer" and "controller" as generally used herein refer to any data processor, and may include Internet appliances and handheld devices (including palmtop computers, wearable computers, cellular or mobile phones, multiprocessor systems , processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information processed by these computers and controllers can be presented on any suitable display medium including liquid crystal displays (LCDs). Instructions for performing computer- or controller-executable tasks may be stored in or on any suitable computer-readable medium including hardware, firmware, or a combination of hardware and firmware. The instructions may be contained in any suitable memory device including, for example, a flash drive, USB device, and/or other suitable media.

The terms "coupled" and "connected," along with their derivatives, may be used herein to describe structural relationships between components. It should be understood that these terms are not intended as synonyms for each other. In contrast, in particular embodiments, "connected" may be used to indicate that two or more elements are in direct contact with each other. Unless the context clearly dictates otherwise, the term "coupled" may be used to indicate that two or more elements contact each other, directly or indirectly (through other intervening elements between them), or that two or more elements co-operate or interact with each other (eg, as in causality, such as for signal transmission/reception or function calls), or both.

Any reference herein to image analysis by a computing system may be performed in accordance with or using spatial structure information, which may include depth information describing corresponding depth values at various locations relative to selected points. Depth information can be used to identify objects or estimate how objects are spatially arranged. In some cases, spatial structure information may include or be used to generate a point cloud describing the location of one or more surfaces of an object. Spatial structure information is only one form of possible image analysis, and others known to those skilled in the art may be used in accordance with the methods described herein.

FIG. 1A shows a system 1000 for performing object detection, or more specifically object recognition. More particularly, system 1000 may include computing system 1100 and camera 1200 . In this example, the camera 1200 may be configured to generate image information that describes or otherwise represents the environment in which the camera 1200 is located, or more specifically represents the field of view of the camera 1200 (also referred to as the camera field of view). in the environment. The environment may be, for example, a warehouse, manufacturing plant, retail space, or other location. In particular embodiments described herein, an environment may be an object handling environment including one or more source repositories and one or more destination repositories. In such a case, the image information may represent an image of an object (such as a box, bag, bag, cabinet, case, etc.) located at such a location. Such objects can be located in both source and destination repositories. System 1000 may be configured to generate, receive, and/or process image information to perform object identification or object registration based on the image information, such as by using the image information to distinguish between individual objects in the field of view of the camera, and/or to perform object identification or object registration based on the image information. Robot interaction planning is performed, as discussed in more detail below (the terms "and/or" and "or" are used interchangeably in this disclosure). Robot interaction planning can be used, for example, to control a robot at a location to facilitate robot interaction between the robot and a container or other object. Computing system 1100 and camera 1200 may be co-located, or may be remote from each other. For example, computing system 1100 may be part of a cloud computing platform hosted in a data center remote from a warehouse or retail space, and may communicate with camera 1200 via a network connection.

In an embodiment, the camera 1200 (which may also be referred to as an image sensing device) may be a 2D camera and/or a 3D camera. For example, FIG. 1B shows system 1500A (which may be an embodiment of system 1000 ) that includes computing system 1100 and cameras 1200A and 1200B, both of which may be embodiments of camera 1200 . In this example, camera 1200A may be a 2D camera configured to generate 2D image information comprising or forming a 2D image describing the visual appearance of an environment in the camera's field of view. Camera 1200B may be a 3D camera (also referred to as a spatial structure sensing camera or spatial structure sensing device) configured to generate 3D image information comprising or forming a spatial structure about the environment in the camera's field of view information. The spatial structure information may include depth information (eg, a depth map) that describes respective depth values relative to various locations of the camera 1200B, such as locations on the surface of various objects in the field of view of the camera 1200B. These locations in the camera's field of view or on the surface of an object may also be referred to as physical locations. In this example, depth information can be used to estimate how objects are spatially arranged in three-dimensional (3D) space. In some cases, the spatial structure information may include or be used to generate a point cloud (also referred to as a 3D point cloud) that describes the location on one or more surfaces of objects in the field of view of the camera 1200B. More specifically, spatial structure information may describe various locations on the structure (also referred to as object structure) of one or more objects.

In an embodiment, system 1000 may be a robot operating system for facilitating robotic interactions between the robot and various objects in the environment of camera 1200 . For example, FIG. 1C shows a robot operating system 1500B, which may be an embodiment of the system 1000/1500A of FIGS. 1A and 1B . The robot operating system 1500B may include a computing system 1100 , a camera 1200 and a robot 1300 . As noted above, robot 1300 may be used to interact with one or more objects in the environment of camera 1200 , such as with a box, bag, bag, crate, cabinet, tray, or other container. For example, robot 1300 may be configured to pick objects from one location and move them to another location. In some cases, robot 1300 may be used to perform depalletizing operations in which a set of containers or other objects is unloaded and moved to, for example, a conveyor belt. In some implementations, camera 1200 may be attached to robot 1300 or robot 3300, as discussed below. This is also known as a handheld camera or handheld camera solution. The camera 1200 may be attached to the robot arm 3320 of the robot 1300 . The robotic arm 3320 may then move to various pick areas to generate image information about those areas. In some implementations, camera 1200 may be separate from robot 1300 . For example, camera 1200 may be mounted to the ceiling or other structure of a warehouse and may remain fixed relative to the structure. In some implementations, multiple cameras 1200 may be used, including multiple cameras 1200 separate from the robot 1300 and/or cameras 1200 separate from the robot 1300 used with the palm camera 1200 . In some implementations, the camera 1200 or cameras 1200 may be mounted or fixed to a dedicated robotic system separate from the robot 1300 used for object manipulation, such as a robotic arm, gantry, or configured for camera movement other automated systems. Throughout the description, "controlling" the camera 1200 may be discussed. For a handheld camera solution, control of the camera 1200 also includes control of the robot 1300 to which the camera 1200 is mounted or attached.

In an embodiment, the computing system 1100 of FIGS. 1A-1C may form or be integrated into a robot 1300, also referred to as a robot controller. A robot control system may be included in system 1500B and may be configured, for example, to generate commands for robot 1300, such as robot interaction move commands for controlling robot interactions between robot 1300 and containers or other objects. In such embodiments, computing system 1100 may be configured to generate such commands based on, for example, image information generated by camera 1200 . For example, computing system 1100 may be configured to determine a motion plan based on image information, where the motion plan may be aimed at, for example, grasping or otherwise picking up an object. Computing system 1100 may generate one or more robot interaction movement commands to perform motion planning.

In an embodiment, computing system 1100 may form or be part of a vision system. A vision system may be a system that generates, for example, visual information describing the environment in which the robot 1300 is located, or alternatively or additionally, describing the environment in which the camera 1200 is located. Visual information may include the 3D image information and/or 2D image information discussed above, or some other image information. In some cases, if computing system 1100 forms a vision system, the vision system may be part of the robotic control system discussed above, or may be separate from the robotic control system. If the vision system is separate from the robot control system, the vision system may be configured to output information describing the environment in which the robot 1300 is located. The information may be output to a robotic control system, which may receive such information from the vision system and perform motion planning and/or generate robot interactive movement commands based on this information. Further information on the vision system is detailed below.

In an embodiment, computing system 1100 may be connected via a direct connection, such as via a dedicated wired communication interface, such as an RS-232 interface, a Universal Serial Bus (USB) interface, and/or via a local computer bus, such as a Peripheral Component Interconnect (PCI) bus) to communicate with the camera 1200 and/or the robot 1300. In an embodiment, computing system 1100 may communicate with camera 1200 and/or with robot 1300 via a network. The network may be any type and/or form of network, such as a Personal Area Network (PAN), a Local Area Network (LAN) (eg, an Intranet), a Metropolitan Area Network (MAN), a Wide Area Network (WAN), or the Internet. Networks can utilize different technologies and protocol layers or stacks including, for example, Ethernet protocol, Internet Protocol Suite (TCP/IP), ATM (Asynchronous Transfer Mode) technology, SONET (Synchronous Optical Network) protocol or SDH (Synchronous Digital Hierarchy) protocol.

In embodiments, computing system 1100 may communicate information directly with camera 1200 and/or with robot 1300, or may communicate via an intermediate storage device or, more generally, via an intermediate non-transitory computer-readable medium. For example, FIG. 1D illustrates a system 1500C, which may be an embodiment of the system 1000/1500A/1500B, that includes a non-transitory computer-readable medium 1400 that may be stored in the computing system 1100 external, and may act as an external buffer or repository for storing image information generated by the camera 1200, for example. In such examples, computing system 1100 may retrieve or otherwise receive image information from non-transitory computer-readable medium 1400 . Examples of non-transitory computer readable medium 1400 include electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. The non-transitory computer readable medium can form, for example, a computer floppy disk, hard disk drive (HDD), solid state drive (SDD), random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read only memory (CD-ROM), digital versatile disk (DVD) and/or memory stick.

As mentioned above, the camera 1200 may be a 3D camera and/or a 2D camera. A 2D camera may be configured to generate 2D images, such as color images or grayscale images. The 3D camera may be, for example, a depth sensing camera, such as a time-of-flight (TOF) camera or a structured light camera, or any other type of 3D camera. In some cases, the 2D camera and/or the 3D camera may include an image sensor, such as a Charge Coupled Device (CCD) sensor and/or a Complementary Metal Oxide Semiconductor (CMOS) sensor. In embodiments, a 3D camera may include a laser, a LIDAR device, an infrared device, a light/dark sensor, a motion sensor, a microwave detector, an ultrasonic detector, a RADAR detector, or any other device configured to capture depth information or other spatial structure information. other devices.

Image information may be processed by computing system 1100, as described above. In an embodiment, the computing system 1100 may include or be configured as a server (e.g., with one or more server blades, processors, etc.), a personal computer (e.g., a desktop computer, a laptop computer, etc.), a smartphone, a tablet computing device and/or any other computing system. In an embodiment, any or all of the functions of computing system 1100 may be performed as part of a cloud computing platform. Computing system 1100 may be a single computing device (eg, a desktop computer), or may include multiple computing devices.

FIG. 2A provides a block diagram illustrating an embodiment of a computing system 1100 . Computing system 1100 in this embodiment includes at least one processing circuit 1110 and a non-transitory computer-readable medium (or media) 1120 . In some cases, processing circuitry 1110 may include a processor (e.g., a central processing unit (CPU) configured to execute instructions (e.g., software instructions) stored on non-transitory computer-readable medium 1120 (e.g., computer memory) CPU), dedicated computer and/or onboard server). In some embodiments, the processor may be included in a separate/independent controller operatively coupled to other electronic/electrical devices. The processor may implement program instructions to control/interface with other devices to cause the computing system 1100 to perform actions, tasks and/or operations. In an embodiment, the processing circuitry 1110 includes one or more processors, one or more processing cores, a programmable logic controller ("PLC"), an application specific integrated circuit ("ASIC"), a programmable gate array ("PGA") ”), Field Programmable Gate Array (“FPGA”), any combination thereof, or any other processing circuitry.

In embodiments, the non-transitory computer-readable medium 1120 that is part of the computing system 1100 may be an alternative or in addition to the intermediate non-transitory computer-readable medium 1400 discussed above. The non-transitory computer readable medium 1120 may be a storage device such as an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof, such as, for example, a computer floppy disk, a hard disk drive (HDD ), Solid State Drive (SDD), Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM or Flash), Static Random Access Memory (SRAM), Portable CD ROM (CD-ROM), Digital Versatile Disk (DVD), memory stick, any combination thereof, or any other storage device. In some cases, non-transitory computer readable medium 1120 may include multiple storage devices. In some implementations, the non-transitory computer-readable medium 1120 is configured to store image information generated by the camera 1200 and received by the computing system 1100 . In some cases, non-transitory computer readable medium 1120 may store one or more object recognition templates for performing the methods and operations discussed herein. The non-transitory computer readable medium 1120 may alternatively or additionally store computer readable program instructions which, when executed by the processing circuit 1110, cause the processing circuit 1110 to perform one or more of the methods described herein .

FIG. 2B depicts computing system 1100A, which is an embodiment of computing system 1100 and includes communication interface 1130 . Communication interface 1130 may be configured, for example, to receive image information generated by camera 1200 of FIGS. 1A-1D . Image information may be received via the intermediary non-transitory computer readable medium 1400 or network discussed above, or via a more direct connection between the camera 1200 and the computing system 1100/1100A. In an embodiment, the communication interface may be configured to communicate with the robot 1300 of FIG. 1C. If the computing system 1100 is external to the robotic control system, the communication interface of the computing system 1100 may be configured to communicate with the robotic control system. A communication interface may also be referred to as a communication component or a communication circuit, and may include, for example, a communication circuit configured to perform communication through a wired or wireless protocol. As examples, communication circuitry may include RS-232 port controllers, USB controllers, Ethernet controllers, controller, PCI bus controller, any other communication circuitry, or a combination thereof.

In an embodiment, as shown in FIG. 2C , non-transitory computer readable medium 1120 may include storage space 1125 configured to store one or more data objects discussed herein. For example, the memory space may store object recognition templates, detection hypotheses, image information, object image information, robotic arm movement commands, and any additional data objects that the computing systems discussed herein may need to access.

In an embodiment, the processing circuit 1110 may be programmed by one or more computer-readable program instructions stored on the non-transitory computer-readable medium 1120 . For example, FIG. 2D illustrates computing system 1100C, which is an embodiment of computing system 1100/1100A/1100B, in which processing circuitry 1110 is composed of one or more components including object recognition module 1121, motion planning module 1129, and object manipulation planning module 1126. Module programming. The processing circuit 1110 may also be programmed with an object registration module 1130 and a pickable area detection module 1132 . Each of the above modules may represent computer-readable program instructions configured to, when instantiated on one or more of the processors, processing circuits, computing systems, etc. described herein, perform certain Task. Each of the above modules can cooperate with each other to realize the functions described herein. Various aspects of the functionality described herein may be performed by one or more of the software modules described above, and the software modules and their descriptions should not be construed as limiting the computing architecture of the systems disclosed herein. For example, although a specific task or function may be described with respect to a particular module, that task or function may also be performed by a different module as desired. Furthermore, the system functions described herein may be performed by different sets of software modules configured with different functional decomposition or distribution.

In an embodiment, the object recognition module 1121 may be configured to acquire and analyze image information, as discussed throughout this disclosure. The methods, systems, and techniques discussed herein with respect to image information may use the object recognition module 1121 . The object recognition module may also be configured for object recognition tasks related to object identification, as discussed herein.

The motion planning module 1129 may be configured to plan and execute movements of the robot. For example, the motion planning module 1129 may interact with other modules described herein to plan the motion of the robot 3300 for object fetching operations and for camera placement operations. The methods, systems, and techniques discussed herein with respect to robotic arm movements and trajectories may be performed by the motion planning module 1129 .

The object manipulation planning module 1126 may be configured to plan and execute object manipulation activities of the robotic arm, eg, grasp and release objects and execute robotic arm commands to facilitate and facilitate such grasping and releasing.

Object registration module 1130 may be configured to acquire, store, generate, and otherwise process object registration and detection information that may be required for the various tasks discussed herein. The object registration module 1130 may be configured to interact or communicate with any other necessary modules.

The pickable area detection module 1132 may be configured to identify a pickable area on a surface of one or more objects, eg, as described with respect to FIG. 4 . The pickable area detection module 1132 may be configured to interact or communicate with any other necessary modules.

Referring to Figures 2E, 2F, 3A, 3B and 3C, methods related to the object recognition module 1121 and object registration module 1130 that may be performed for image analysis are explained. 2E and 2F illustrate example image information associated with image analysis methods, while FIGS. 3A-3C illustrate example robotic environments associated with image analysis methods. References herein related to image analysis by a computing system may be performed from or using spatial structure information, which may include depth information describing respective depth values for various locations relative to selected points. Depth information can be used to identify objects or estimate how objects are spatially arranged. In some cases, spatial structure information may include or be used to generate a point cloud describing the location of one or more surfaces of an object. Spatial structure information is only one form of possible image analysis, and others known to those skilled in the art may be used in accordance with the methods described herein.

In an embodiment, computing system 1100 may acquire image information representing objects in a camera field of view (eg, 3210 ) of camera 1200 . The steps and techniques described below for obtaining image information may be image information capture operations. In some cases, the object may be an object 3520 from a plurality of objects 3520 in a source container 3510 in the field of view 3210 of the camera 1200 . Image information 2600 , 2700 may be generated by camera (eg, 1200 ) while objects 3520 are (or have been) within camera field of view 3210 and may describe one or more of individual objects 3520 . The object appearance describes the appearance of the object 3520 from the viewpoint of the camera 1200 . If there are multiple objects 3520 in the camera field of view, the camera may generate image information representing multiple objects or a single object as required (such image information related to a single object may be referred to as object image information). The image information may be generated by the camera (eg, 1200 ) while the set of objects is (or has been) in the camera's field of view, and may include, for example, 2D image information and/or 3D image information.

As an example, Figure 2E depicts a first set of image information, or more specifically, 2D image information 2600, as described above, generated by camera 1200 and representing objects 3520, such as those shown in Figures 3A-3C. More specifically, 2D image information 2600 may be a grayscale or color image and may describe the appearance of object 3520 from the viewpoint of camera 1200 . In an embodiment, 2D image information 2600 may correspond to a single color channel (eg, red, green, or blue channel) of a color image. If the camera 1200 is disposed above the object 3520 , the 2D image information 2600 may represent an appearance of a corresponding top surface of the object 3520 . In the example of FIG. 2E , 2D image information 2600 may include respective portions 2000A/2000B/2000C/2000D/2550 representing respective surfaces of object 3520 , also referred to as image portions or object image information. In FIG. 2E, each image portion 2000A/2000B/2000C/2000D/2550 of 2D image information 2600 may be an image range, or more specifically, a pixel range (if the image is formed of pixels). Each pixel in the pixel range of 2D image information 2600 may be characterized as having a position described by a set of coordinates [U,V] and may have a value relative to the camera coordinate system or some other coordinate system, as shown in Figure 2E and 2F is shown. Each pixel may also have an intensity value, such as a value between 0 and 255 or 0 and 1023. In further embodiments, each pixel may include any additional information associated with the pixel in various formats (eg, hue, saturation, intensity, CMYK, RGB, etc.).

As mentioned above, in some embodiments the image information may be all or part of an image, such as 2D image information 2600 . For example, computing system 3100 may be configured to extract image portion 2000A from 2D image information 2600 to obtain only image information associated with corresponding object 3520 . In the case where an image portion (such as image portion 2000A) points to a single object, it may be referred to as object image information. Object image information need not contain only information about the object it points to. For example, the object it is pointing at may be near, below, above or otherwise located near one or more other objects. In such cases, object image information may include information about the object it is pointing at as well as one or more neighboring objects. Computing system 1100 may extract image portion 2000A by performing image segmentation or other analysis or processing operations based on 2D image information 2600 and/or 3D image information 2700 shown in FIG. 2F . In some implementations, segmentation or other processing operations may include detecting image locations where physical edges of objects (e.g., edges of objects) occur in 2D image information 2600, and using such image locations to identify images limited to representations of the camera's field of view (e.g., , 3210) and basically exclude object image information of other objects. "Substantial exclusion" means that image segmentation or other processing techniques may be designed and configured to exclude non-target objects from object image information, but it is understood that errors may arise, noise may be present, and various other factors may lead to the inclusion of other part of the object.

FIG. 2F depicts an example in which the image information is 3D image information 2700 . More particularly, 3D image information 2700 may include, for example, a depth map or point cloud indicating respective depth values for various locations on one or more surfaces (eg, top or other exterior surfaces) of object 3520 . In some implementations, image segmentation operations for extracting image information may involve detecting image locations where physical edges of objects (e.g., edges of boxes) occur in 3D image information 2700, and using such image locations to identify objects limited to representations Image portions (eg, 2730 ) of individual objects (eg, 3520 ) in the camera's field of view.

The respective depth values may be relative to the camera 1200 that generated the 3D image information 2700, or may be relative to another reference point. In some implementations, 3D image information 2700 may include a point cloud (3D point cloud) that includes corresponding coordinates at various locations on structures of objects in the camera's field of view (eg, 3210 ). In the example of FIG. 2F , the point cloud may include respective sets of coordinates describing positions on respective surfaces of object 3520 . The coordinates may be 3D coordinates, such as [XY Z] coordinates, and may have values relative to the camera coordinate system or some other coordinate system. For example, 3D image information 2700 may include a first image portion 2710 (also referred to as image portion) indicating respective depth values for a set of locations 2710 ₁ - 2710 _n on the surface of object 3520 , also referred to as physical locations. In addition, the 3D image information 2700 may further include second, third, fourth and fifth parts 2720 , 2730 , 2740 and 2750 . These parts may then also indicate corresponding depth values for a set of locations which may be represented by 2720 ₁ -2720 _n , 2730 ₁ -2730 _n , 2740 ₁ -2740 _n and 2750 ₁ -2750 _n respectively. These figures are examples only, and any number of objects with corresponding image portions may be used. Similar to the above, the acquired 3D image information 2700 may in some cases be part of the first set of 3D image information 2700 generated by the camera. In the example of FIG. 2E , if the acquired 3D image information 2700 represents the single object 3520 of FIG. 3A , the 3D image information 2700 may be reduced to only the reference image portion 2710 . Similar to the discussion of 2D image information 2600, identified image portions 2710 may pertain to individual objects and may be referred to as object image information. Accordingly, as used herein, object image information may include 2D and/or 3D image information.

In an embodiment, an image normalization operation may be performed by computing system 1100 as part of acquiring image information. An image normalization operation may involve transforming an image or portion of an image generated by the camera 1200 to generate a transformed image or portion of an image. For example, if acquired image information, which may include 2D image information 2600, 3D image information 2700, or a combination of both, may undergo an image normalization operation in an attempt to bring the image information to the point of view, object pose, The lighting conditions are changed. Such normalization can be performed to facilitate more accurate comparisons between image information and model (eg, template) information. The point of view may refer to the pose of the object relative to the camera 1200, and/or the angle at which the camera 1200 is viewing the object when the camera 1200 generates an image representing the object.

For example, image information may be generated during an object recognition operation where a target object is in the camera field of view 3210 . When the target object has a specific posture with respect to the camera, the camera 1200 may generate image information representing the target object. For example, the target object may have a pose with its top surface perpendicular to the optical axis of the camera 1200 . In such examples, the image information generated by camera 1200 may represent a specific viewpoint, such as a top view of a target object. In some cases, when the camera 3210 generates image information during an object recognition operation, the image information may be generated under certain lighting conditions such as lighting intensity. In such cases, the image information may represent a particular lighting intensity, lighting color, or other lighting conditions.

In an embodiment, the image normalization operation may involve adjusting the image or image portion of the scene generated by the camera so that the image or image portion better matches the viewpoint and/or lighting conditions associated with the information of the object recognition template. Adjustment may involve transforming the image or image portion to generate a transformed image that matches at least one of object pose or lighting conditions associated with the visual description information of the object recognition template.

Viewpoint adjustment may involve processing, warping, and/or shifting the image of the scene such that the image represents the same viewpoint as the visual description information that may be included within the object recognition template. For example, manipulation includes changing the color, contrast, or lighting of an image, warping of a scene may include changing the size, dimensions, or proportions of an image, and translating an image may include changing the position, orientation, or rotation of an image. In an example embodiment, processing, warping, and/or shifting may be used to alter objects in an image of a scene to have an orientation and/or size that matches or better corresponds to the visual descriptor of the object recognition template. If the object recognition template describes a head-on view (eg, a top view) of an object, the image of the scene may be warped to also represent the head-on view of the object in the scene.

Other aspects of the object recognition methods implemented herein are described in more detail in U.S. Application No. 16/991,510, filed August 12, 2020, and U.S. Application No. 16/991,466, filed August 12, 2020, which Each of the US applications is incorporated herein by reference.

In various embodiments, the terms "computer readable instructions" and "computer readable program instructions" are used to describe software instructions or computer code configured to perform various tasks and operations. In various embodiments, the term "module" broadly refers to a collection of software instructions or code configured to cause the processing circuit 1110 to perform one or more functional tasks. The modules and computer readable instructions may be described as performing various operations or tasks when the modules or computer readable instructions are being executed by a processing circuit or other hardware component.

3A-3C illustrate example environments in which pickable area (or grasping area) detection operations and/or motion planning operations may be performed. More specifically, FIG. 3A depicts system 3000 (which may be an embodiment of system 1000/1000A/1000B/1000C of FIGS. 1A-1D ) that includes computing system 1100 , robot 3300 and camera 1200 . Camera 1200 may be an embodiment of camera 1200 and may be configured to generate image information representative of a scene in camera field of view 3210 of camera 1200, or more specifically an object (such as object 3520 ₁ to 3520 _n , may include, for example, objects 3520 ₁ , 3520 ₂ , 3520 ₃ , 3520 ₄ , 3520 ₅ , . . . 3520 _n ) or structures thereof. In the embodiment of FIGS. 3A-3C , robot 3300 may be configured to manipulate or otherwise interact with each of one or more of objects 3520 ₁ - 3520 _n , such as by picking or otherwise grasping Live one of 3520 ₁ -3520 _n to lift the object from its current position and move the object to its destination position.

In some cases, some or all of objects 3520 ₁ through 3520 _n may be flexible objects. For example, each of objects 3520 ₁ through 3520 _n may be a package having a piece of clothing (e.g., a shirt or pants) or other textile or fabric, where the clothing or other textile may be wrapped in a sheet of packaging material, such as a sheet of plastic )middle. In some cases, sheets of plastic or other packaging material are generally immune to air or other liquids. In the example of FIG. 3A , objects 3520 ₁ through 3520 _n may be deployed in containers 3510, such as boxes or boxes for keeping 3520 ₁ through 3520 _n in a facility, such as a warehouse associated with a clothing manufacturer or retailer. Box. In some cases, some or all of objects 3520 ₁ through 3520 _n may include items such as boxes, bags, bags, and other items.

In some cases, the flexible objects (e.g., 3520 ₁ ) of the embodiments herein may be sufficiently flexible to allow the flexible object to change its shape when the robot 3300 moves or otherwise manipulates or when placed in the container 3510. out of shape. Sufficiently high flexibility may correspond to sufficiently low stiffness or rigidity to prevent the object from maintaining its shape when moved or otherwise manipulated by robot 3300 . In some cases, the flexible object may have a high enough flexibility to allow the weight of the flexible object to cause deformation of the object's own shape when the robot 3300 lifts the flexible object. Deformation may involve, for example, bending of the flexible object, or more specifically, sagging under its own weight when lifted by the robot 3300 . The flexibility of a flexible object may result from, for example, the dimensions of the flexible object and/or the material of the flexible object. In one example, a flexible object can have a thin profile, which can introduce flexibility (also called flexibility) into the flexible object. More specifically, the flexible object may have a thickness dimension that is very small relative to a transverse dimension (eg, length dimension or width dimension). In one example, the flexible object can be made of a material that is soft enough to introduce flexibility into the flexible object. In some cases, the material of the flexible object may be soft enough to sag under the material's own weight when the robot 3300 lifts the object. For example, if the flexible object is a package with a piece of clothing, it may be formed from a material such as cotton or wool fabric that lacks sufficient stiffness to prevent the material from sagging under its own weight when lifted by the robot 3300 .

In an embodiment, robot 3300 (which may be an embodiment of robot 1300 ) may include a robotic arm 3320 with one end attached to robot base 3310 and the other end attached to or controlled by end effector device 3330 . form. Robotic base 3310 may be used to mount one end of robotic arm 3320, while the other end of robotic arm 3320, or more specifically end effector assembly 3330, may be used to interact with one or more objects ( For example, 3520 ₁ , 3520 ₂ , etc.) interactions. Interactions may include, for example, grabbing and lifting one or more objects, and/or moving one or more objects from a current location to a destination location.

In an embodiment, end effector assembly 3330 may include one or more suction cups 3332 ₁ - 3332 _n (also referred to herein as suction grippers and suction gripping devices) for picking up or otherwise lifting objects , such as one of objects 3520 ₁ -3520 _n . In some implementations, each of suction cups 3332 ₁ - 3332 _n (also referred to as end effector suction cups) can be a mechanical device configured to, when pressed into contact against a surface of an object (e.g., 3520 ₁ ), Fluid pressure (eg, air pressure) in the space between the suction cup and the surface of the object (also referred to as the object surface) is reduced. In an example, the object surface may be formed from a generally fluid-impermeable (or more generally non-porous) material, such as the plastic wrap used to wrap a piece of clothing. A reduced fluid pressure, such as a partial or full vacuum, results in a pressure differential between the fluid pressure outside the space and the fluid pressure inside the space. More specifically, the fluid pressure inside the space can be lower than the fluid pressure outside the space, which creates a negative fluid pressure that causes the higher fluid pressure to exert a net force that causes the suction cup to adhere to the surface of the object. The net force can act as an adhesive force, allowing the suction cup to adhere to the surface and thus grip the surface. In embodiments, each of the suction cups (eg, 3332 ₁ or 3332 _n ) can have various shapes (eg, circular shapes) and sizes, and can be of various materials, such as plastic, silicone, nitrogen, fluorine Viton, vinyl, urethane, rubber, or some other flexible material. Suction cups are discussed in more detail in US Patent No. 10,576,630, entitled "Robotic system with a robot arm suction control mechanism and method of operation thereof," the entire contents of which are incorporated herein by reference. In an embodiment, the strength of the adhesive force between the suction cup and the surface of the object may depend on how tightly the suction cup is able to seal the space between itself and the surface of the object. For example, a tight seal can maintain a pressure differential and thus maintain adhesion, while a loose seal prevents the pressure differential from being maintained and therefore interferes with the suction cup's ability to grip the surface of the object. In an embodiment, the ability of the suction cup to form a tight seal may depend on the smoothness of the area of the object's surface (also referred to as the surface area) where the suction cup attempts to grip the object's surface. Accordingly, as discussed in more detail below, computing system 1100 can be configured to identify or search for surface regions that are smooth enough to be used as gripping regions to which suction cups can reliably adhere to grip objects surface.

In an embodiment, camera 1200 may be configured to generate image information representative of objects 3520 ₁ - 3520 _n and container 3510 or any other object(s) in camera field of view 3210 . The camera 1200 may be a 3D camera configured to generate 3D image information and/or a 2D camera configured to generate 2D image information. In an embodiment, the 3D image information may represent the collective object surface of object 3520, or more specifically describe the physical structure of the object surface. For example, 3D image information may include a depth map, or more generally, depth information that may describe depth values for various locations in the camera's field of view 3210 relative to the camera 1200 or relative to other reference points. The locations corresponding to the respective depth values may be locations on various surfaces in the camera's field of view 3210, such as locations on respective object surfaces of objects ₃₅₂₀₁ through _3520n . In some cases, the 3D image information may include a point cloud, which may include a plurality of 3D coordinates describing positions on respective object surfaces of objects 3520 ₁ through 3520 _n in camera field of view 3210 .

In an embodiment, an object surface of an object (eg, 3520 ₁ ) may refer to an outer surface (eg, a top surface) of the object. In such embodiments, the 3D image information may include information representative of the exterior surface, or more specifically may describe the physical structure of the exterior surface. For example, if the camera 1200 generates 3D image information by sensing light (eg, laser or structured light) or other signals reflected from the exterior surface, the 3D information may represent, for example, the surface profile of the exterior surface. The 3D information can still represent the outer surface of the object if the outer surface is formed from a transparent material, such as a flexible plastic sheet used as packaging material. More particularly, in this case, camera 1200 may sense light or other signals reflected from a non-transparent material, such as a piece of clothing fabric that is underlying or otherwise covered by a transparent material. Reflected light or signals can pass through the transparent material and can be detected by the camera 1200 to generate 3D information. In this case, the transparent material (eg, a plastic sheet) may be thin enough that the distance between the outer surface and the surface of the non-transparent material can be considered negligible. Therefore, in an embodiment, it can be considered that the 3D information describes the depth information of various positions on the outer surface of the object. Furthermore, if the transparent material forms the outer surface, the transparent material may be flexible enough so that all or many portions of the transparent material follow the surface contour of the underlying non-transparent material. Therefore, the 3D image information in this case can be regarded as describing the outer surface of the object, or more specifically the physical structure or surface contour of the outer surface.

In an embodiment, the 2D image information may include, for example, a color image or a grayscale image representing the appearance of one or more objects in the camera's field of view 3210 . For example, if a surface of an object has a visual marking (eg, a logo) or other visual detail printed on it, the 2D image information may describe or otherwise represent the visual detail. As noted above, an object surface may be the outer surface of an object, which may in some cases be formed of a transparent material. In such cases, the 2D image information may represent light (eg, visible light) or other signals reflecting from the surface of the underlying non-transparent material (eg, shirt) and passing through the transparent material forming the outer surface. Since in this case the 2D image information is based on light or other signals passing through the outer surface, the 2D image information can still be considered to represent the outer surface. Also, in some cases the transparent material forming the outer surface may be thin enough and transparent enough to have little or negligible effect on the appearance of the object such that it may be considered that the appearance of the object or the outer surface of the object refers to the underlying The appearance of non-transparent materials (for example, clothing materials).

In an embodiment, system 3000 may include multiple cameras. For example, FIG. 3B illustrates system 3000A (which may be an embodiment of system 3000 ) including camera 1200A having camera field of view 3210A, and including camera 1200B having camera field of view 3210B. Camera 1200A (which may be an embodiment of camera 1200A) may be, for example, a 2D camera configured to generate 2D images or other 2D image information, while camera 1200B (which may be an embodiment of camera 1200B) may be, for example, configured to generate 3D images 3D camera for information.

In an embodiment, the camera 1200 / 1200A / 1200B may be fixed relative to a reference point, such as the floor on which the container 3510 is placed, or relative to the robot base 3310 . For example, the camera 1200 in FIG. 3A may be mounted to a ceiling, such as the ceiling of a warehouse, or to a mounting frame that remains fixed relative to the floor, relative to the robot base 3310, or some other reference point. In an embodiment, the camera 1200 may be mounted on the robot arm 3320 . For example, FIG. 3C depicts system 3000B (which may be an embodiment of system 1000 ) in which camera 1200 is attached or otherwise mounted on end effector device 3330 forming the distal end of robotic arm 3320 . Such an embodiment may provide the robot 3300 with the ability to move the camera 1200 to different poses via movement of the robotic arm 3320 .

Computing system 1100 may be configured to generate pickable area detection results for one or more objects 3520 at source container 3510 . For example, source container 3510 may include a container of objects 3520 with random orientations, poses, and positions. In addition to the pickable area, the pickable area detection result may also include additional information, such as detection mask information, safety volume or a combination thereof, each of which is described in detail below.

Robot 3300 may also include other sensors configured to obtain information for accomplishing tasks, such as for manipulating structural members and/or for transporting robotic units. These sensors may include devices configured to detect or measure one or more physical characteristics of robot 3300 and/or the surrounding environment (eg, the state, condition and/or position of one or more structural members/joints thereof). Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, and the like.

4 provides a flowchart illustrating the general flow of methods and operations for identifying a pickable area for one or more selected one or more of the objects in a container. Pickable area identification method 4000 may include any combination of features of the sub-methods and operations described herein. The method 4000 can be performed or performed by any suitable systems and devices described herein.

At operation 4002, method 4000 includes obtaining image information. Image information of a set of objects or objects contained in a source container may be obtained by a computing system. The image information may be obtained, for example, by controlling a camera and/or may be obtained from a data storage device on which the image information has been stored. As discussed herein, image information of objects in a scene may include 3D image information 2700 . 5A and 5B provide a representative example of a scene including a plurality of objects represented by 2D image information 5600 (FIG. 5A) and 3D image information 5700 (FIG. 5B) representing the scene.

Figure 5A depicts 2D image information, or more specifically 2D image information 5600, generated by cameras 1200/1200A and representing objects 3520i _{- 3520n} and container 3510 of Figures 3A-3C. More specifically, 2D image information 5600 may describe the appearance of objects 3520 ₁ - 3520 _n and containers 3510 in which objects 3520 ₁ - 3520 _n are deployed. More specifically, 2D _image information 5600 may include image portions 5610 _{1 , 5620 2 , 5620 3 , 5620 4} , 5620 _{5 ,} ..., 5620n _-3 , 5620n _-2 , 5620n _-1 , _5620n (e.g. pixel area). In an embodiment, the 2D image information may represent an object surface of an object (eg, 3520 ₁ ). As noted above, an object surface may be an exterior surface (eg, a top surface) of an object, and may be formed from a transparent material, a non-transparent material (eg, a translucent or opaque material), or a combination thereof. As further noted above, if the outer surface is formed of a transparent material overlying an underlying non-transparent material, the transparent material may be sufficiently thin and transparent that it may be considered to have a negligible effect on the appearance of the object. In such cases, the appearance of the underlying non-transparent material may also be considered to be that of the outer surface of the object, such that the 2D image information is considered to represent the appearance of the outer surface of the object.

FIG. 5B illustrates an example of 3D image information 5700 . More particularly, 3D image information 5700 may include, for example, a depth map or other depth information indicating various locations (such as locations 5700 ₁ , 5700 ₂ , . _. . Can be the corresponding depth value of a grid of positions organized into rows and columns. In some implementations, the depth map may include pixels indicating depth values for locations 5700 ₁ -5700 _n . In an embodiment, at least some of locations 5700 ₁ - 5700 _n are locations on one or more object surfaces, such as object surfaces of objects 3520 ₁ - 3520 _n . For example, 3D image information 5700 may include image portions 5720 ₁ , 5720 ₂ , 5720 ₃ , 5720 ₄ , 5720 ₅ , ... 5720 _n-3 , 5720 _n-2 , 5720 _n-1 , 5720 _n , where each image A portion may include depth values for a corresponding set of locations on an object surface of a corresponding object (eg, 3520 ₁ , 3520 ₂ , 3520 ₃ , . . . or 3520 _n ). In some cases, the 3D image information may include a point cloud, which may include a collection of coordinates describing locations 5700i _{- 5700n} , respectively. The coordinates may be 3D coordinates, such as [XYZ] Cartesian coordinates, and may have values relative to the camera coordinate system or some other coordinate system. In this example, the [XYZ] coordinates of a particular location (eg, 5700 ₁ ) may have a Z component that is equal to or based on a depth value for that location. The depth value may be relative to the camera that generated the 3D image information (eg, 1200/1200A), or may be relative to some other reference point.

In an embodiment, the 3D image information may describe the surface contour of the surface of the object. For example, the 3D image information 5700 in FIG. 5A may have at least an image portion 5720 ₁ describing the surface contour of the object surface of the object 3520 ₁ . The surface profile of an object surface can describe the physical structure of the object surface. In some cases, the physical structure of the surface of the object may be completely or substantially smooth. In some cases, the physical structure of the object's surface may include physical features, such as wrinkles, bumps, ridges, creases, or depressions, which may form one or more non-smooth portions of the object's surface.

As noted above, an object surface may be an exterior surface (eg, a top surface) of an object, and may be formed from a transparent material, an opaque material (eg, a translucent or opaque material), or a combination thereof. As further noted above, if the outer surface is formed of a transparent material overlying an underlying opaque material, the transparent material may be thin enough and flexible enough to be considered to have a negligible effect on the physical structure or surface profile of the object. In such cases, 3D image information representing the physical structure or surface contour of the underlying opaque material may be considered to also represent the physical structure or surface contour of the outer surface of the object. Additionally, if the transparent material is thin enough, its thickness can be considered to have a negligible effect on the depth measurement of the camera (eg, 1200). In this case, various positions (such as the position of the image portion 5720 ₁ ) having depth values represented in the 3D image information may be regarded as positions on the outer surface of the corresponding object (eg, 3520 ₁ ).

In embodiments, obtaining image information (which may include object detection and object registration) may be performed by any suitable means. In an embodiment, identifying or detecting a plurality of objects 3520 may include a process including object registration, template generation, feature extraction, hypothesis generation, hypothesis refinement, and hypothesis validation, as performed, for example, by object registration module 1130 . Such a process is described in detail in US Patent Application Serial No. 17/884,081, filed August 9, 2022, which is hereby incorporated by reference in its entirety.

Object registration is the process that includes obtaining and using object registration data (eg, known, previously stored information about objects 3520) to generate object recognition templates for identifying and recognizing similar objects in a physical scene. Template generation is the process that includes generating a set of object recognition templates for use by the computing system in identifying objects 3520 for further operations related to object picking. Feature extraction (also referred to as feature generation) is the process that involves extracting or generating features from object image information for use in object recognition template generation. Hypothesis generation is a process that includes generating one or more object detection hypotheses, for example based on a comparison between object image information and one or more object recognition templates. Hypothesis refinement is the process performed on the matching of the object recognition template to the object image information (even in the case that the object recognition template does not exactly match the object image information). Hypothesis validation is the process by which a single hypothesis is selected from among multiple hypotheses as the best fit or best choice for the object 3520 .

At operation 4004, the method 4000 includes generating a surface cost map for the plurality of objects 3520 in the scene. The surface cost map may be an image map indicative of the smoothness of the surface of the collected objects 3520 or a portion of the objects 3520 . A surface cost map may be an image map that identifies irregularities or discontinuities in the surface of the collected plurality of objects 3520 or a portion of objects 3520 . The surface costmap may include a surface costmap value for each point or pixel representing the surface or top layer of the collected plurality of objects 3520 or a portion thereof. Accordingly, the surface costmap may assign a surface costmap value to each point of a point cloud representing number of objects 3520 or a portion thereof. As discussed above, each point/pixel of a point cloud can be represented by three coordinates (x, y, z). A surface costmap value represents the difference between a collection of points (referred to herein as a kernel or cell) and adjacent subdivisions. Thus, a surface costmap value assigned to any point or subdivision may represent the difference between that point or subdivision and an adjacent point or subdivision.

The surface cost map generated according to the image information 5700 may represent the difference in height and angle between a subdivision area or cell and an adjacent subdivision area or cell. The surface cost map may include a height gradient map and a normal difference map, or may be computed from a combination of the height gradient map and the normal difference map. Surface cost maps may be calculated or determined according to various means to represent height and angular differences between adjacent portions of 3D image information 5700 representing multiple objects in a scene. In an embodiment, referring to FIGS. 6A-6I , the calculation of the surface cost map may be performed as follows.

FIG. 6A provides an example flowchart of a method 6000 of generating a surface cost map. Method 6000 can be performed by any suitable processor or computing device described herein. The steps of Figure 6A are provided by way of example only. The steps of FIG. 6A may be performed in any suitable order or combination, and additional steps may be incorporated as desired. Additionally, alternative methods of generating surface cost maps may be employed without departing from the scope of the present disclosure.

A surface cost map may be generated from 3D image information 5700 to include or provide a combination of a height gradient map and a normal disparity map based on several cost map parameters. Such costmap parameters, explained in more detail below, may include subdivision regions, strides, distance thresholds, normal thresholds, and normal weighting factors. As described further below, costmap parameters may be determined manually or automatically.

In operation 6002 of the surface cost map generation method 6000 , 3D image information 5700 may be overlaid with a grid 6100 of cells 6101 . 6B and 6C illustrate grid operations of the surface costmap generation method. The cell 6101 may be rectangular or square, and may be sized according to subdivision areas. The subdivision area can represent the size of each cell 6101 (as shown by the dimension 6105) in units of points or pixels of the point cloud represented by the 3D image information 5700, such as 2×2, 4×4, 6×6, 8 ×8, 10×10, 15×15, 20×20 or any other suitable size. Cells 6101 form a grid on which surface costmap calculations can be performed. In an embodiment, 3D image information 5700 may be gridded with a single non-overlapping set of cells 6101, as shown in Figure 6B. The cell centers 6102 are each spaced apart from each other by a stride whose length (dimension 6106) is equal to the subdivision area size, resulting in a non-overlapping grid.

In a further embodiment, grid 6100 overlaying 3D image information 5700 may include a collection of overlapping cells 6101 . Each cell 6101 may overlap multiple other cells 6101, with cell centers 6102 spaced in steps smaller than the subdivision area size. Thus, for example, as shown in FIG. 6C, cells 6101 may have cell centers 6102 separated by a stride size that is half the subdivision area size. In FIG. 6C , grid 6100 includes cell centers 6102 each spaced apart by a stride size and cells 6101 with width and length dimensions equal to twice the stride size. In FIG. 6C, the size of a single cell 6101 is illustrated by the shaded area. Each cell 6101 overlaps the other four cells 6101 .

In the following discussion of surface costmap calculations, surface costmap values are assigned to cell centers 6102, and when calculations are performed, each cell 6101 is compared to its non-overlapping neighbors. Thus, for purposes of clarity, reference will be made to the non-overlapping arrangement of Figure 6B.

In operation 6004 , the surface costmap generation method 6000 may include the step of fitting a plane to each cell 6101 . FIG. 6D illustrates a set of planes 6220 corresponding to grid 6100 . For each cell 6101 , the plane 6201 may be determined from the x, y and z coordinates of the points covered by the cell 6101 in the 3D image information 5700 . Therefore, for a subdivision area size of 20×20, 400 points of 3D image information 5700 can be used to determine the plane 6201 . Plane 6201 may be determined according to any suitable method, including, for example, least squares. In another example, the plane 6201 may be determined according to the average value of normal vectors at each point in each cell 6101 in the 3D image information 5700 . Each plane 6201 includes a centroid 6202 and a normal 6203 . Centroid 6202 is located at the geometric center of plane 6201 , and normal 6203 extends from centroid 6202 normal to plane 6201 . The height of each plane 6201 can be defined as the height of its centroid 6202.

In operation 6006 , the surface costmap generation method 6000 may include calculating or determining the height gradient of each plane 6201 relative to its neighboring planes 6201 . FIG. 6F illustrates a height gradient 6200 overlaid on a representation of a source container 3510 containing an object 3520 . The height gradient of each plane 6201 may be a mathematical combination of individual height gradients between the plane 6201 and its eight adjacent planes 6201 . The height gradient of each plane 6201 can be determined in several different ways. As shown in FIG. 6F , open circles illustrate portions of low height gradients, filled circles illustrate portions of higher elevation gradients, and crosses illustrate portions that cannot be detected, for example due to unreliable detection or detection of source container 3510. Identify the part of the object. For purposes of illustration, these values are shown as high and low, although in practice these values may span a range of possible values. It can be seen that the height gradient at the border of object 3520 is greater than the height gradient across the central portion of object 3520 .

In an embodiment, the costmap height gradient between a plane 6201 and an adjacent plane 6201 may be determined with reference to FIG. 6E as follows. First, the height difference between the two planes (6201A and 6201B) can be determined. In an embodiment, the height difference of adjacent planes may be based on the extension of one plane 6201B over another plane 6201A (eg, extended plane 6201BA). For example, the height difference can be determined as the height difference between the first extended plane 6201BA and the centroid 6202A of the second plane 6201A according to the length of the normal vector of either plane or according to the z-direction of the 3D point cloud. Vector length calculation. For example, the height difference may be determined as an average height difference between corresponding points on the first extended plane 6201BA and the second plane 6201A, where the corresponding points correspond to grid points in the point cloud of the 3D image information 5700 . In an embodiment, the height difference between two planes 6201A and 6201B may be determined as the maximum or average height difference between: the height determined by extending plane 6201B above (or below) plane 6201A difference, and the height difference determined by extending plane 6201A above (or below) plane 6201B. This method of height difference determination may result in the same height difference regardless of which of the two planes is selected as the "first" plane and which of the two planes is selected as the "second" plane.

A height difference between two planes 6201 may be directly assigned to a position between cells 6101 corresponding to the two planes 6201 . For example, a height difference between plane 6201 corresponding to cells 6101D and 6101E (see FIG. 6B ) may be assigned to the location at point DE. Therefore, since the height difference between the plane 6201 corresponding to cells 6101D and 6101E does not correspond to the centroid of plane 6201 corresponding to cell 6101E, a correction can be applied to determine the height difference to assign to and cell 6101E The height difference between the plane 6201 corresponding to 6101D corresponds to the cell 6101E. In an embodiment, this correction may be applied by averaging the height difference assigned to point DE and the height difference assigned to point EF (e.g., the height difference assigned to point EF is based on plane 6201 corresponding to cells 6101E and 6101F height difference). The total height gradient of each cell 6101 can be determined as the average of the eight height differences from neighboring cells. The total height gradient of each cell 6101 may be assigned as the value associated with the point in the height gradient cost map 6200 at the center of that cell.

In further embodiments, the height difference can be determined according to different methods. The height difference may be based, for example, on the height difference between the centroids 6202A/6202B of the planes 6201A/6201B or on the height difference (or average height difference) between planar views along the boundary of the cell 6101 corresponding to the plane 6201 . Other altitude difference calculations and definitions may be used without departing from the scope of this disclosure.

With respect to the grid 6100 of FIG. 6B , the above discussion represents the calculation of the height gradient at the center point of each cell 6101 in the grid 6100 . Because the stride size can be smaller than the subdivision region size, the number of points for which height gradients are calculated can be larger (even significantly larger) than the number of subdivision regions that can be fitted to the mesh 6100 . For example, for a stride size of 1, any particular point in a 3D point cloud can have associated height gradients, each gradient determined based on a grid of cells 6101 of the subdivision region size where that particular point is the subdivision The center of one of the cells 6101 of the area size. For a stride size of 2, every other point will have an associated height gradient.

Accordingly, height gradient cost map 6200 may include a series of values representing the height gradient of a point (in some embodiments, all points) in the 3D point cloud relative to neighboring points in the 3D point cloud. As discussed above, the points in height gradient cost map 6200 may be those points in 3D point cloud image information 5700 separated by a step. For each point in the height gradient cost map 6200 that is assigned a value, the value is calculated based on the plane 6201 (whose 2D projection has subdivision region dimensions) and the plane's relationship to neighboring planes 6201 .

In an embodiment, the computation of the height gradient cost map 6200 may be simplified or optimized by reusing the height difference between two planes 6201 . For example, in some embodiments, as discussed above, the calculation of the height difference from the first plane 6201 to the second plane 6201 results in exactly the same calculation as the calculation of the height difference between the second plane 6201 and the first plane 6201 value. Therefore, only a single calculation of the height difference between two planes 6201 may be required, allowing the total number of height gradient calculations to be reduced by approximately 50%.

In an embodiment, a distance threshold parameter may be used in determining the altitude difference. The distance threshold parameter may be a threshold beyond which any height difference is assigned a maximum value. If the height difference between two planes exceeds a distance threshold, the height difference may be set to a predetermined value (eg, in some embodiments, the distance threshold). Using a distance threshold reduces the weight of large height differences between two planes when computing the total height gradient. In an embodiment, a distance threshold parameter may also be used to threshold the height gradient assigned to the cell 6101. After averaging the height differences from neighboring cells, the distance threshold may be applied to change the determined height gradient to a predetermined value if the determined height gradient exceeds the distance threshold.

In operation 6008 of surface costmap generation method 6000, a normal difference may be calculated. FIG. 6G illustrates a normal difference cost map 6300 overlaid on a representation of a source container 3510 containing an object 3520 . Referring now to FIG. 6D, the difference between the normal 6203 of each plane 6201 and its adjacent plane 6201 can be determined. The normal difference may be determined as the dot product of the normal 6203 of one plane 6201 and the normal 6203 of an adjacent plane 6201 . Thus, each plane 6201 can have eight different calculated normal differences. These normal differences can be averaged and assigned to the cell 6101 associated with the plane 6201 (eg, the point at the center of the cell 6101). In this way, a normal difference cost map 6300 can be generated in which each point within the surface cost map is assigned a normal difference indicating the difference between the plane 6201 centered at that point and the adjacent plane 6201. angle difference. As shown in FIG. 6G , open circles illustrate portions of low normal variance, filled circles illustrate portions of large normal variance, and crosses illustrate areas of failure, for example due to unreliable detection or detection of the source container 3510. The part identified as an object. For purposes of illustration, these values are shown as high and low, although in practice these values may span a range of possible values. It can be seen that the difference in normals at the boundaries of object 3520 is greater than the difference in normals across the central portion of object 3520 .

In an embodiment, a normal threshold parameter may be used in determining the normal difference. The normal threshold parameter may be a threshold beyond which any height difference is assigned a maximum value. If the normal difference between two planes exceeds a normal threshold, then the normal difference may be set to a predetermined value (eg, in some embodiments, the normal threshold). Using a normal threshold reduces the weight of large normal differences between two planes when computing the average normal difference.

In operation 6010 of surface cost map generation method 6000, a surface cost map may be generated. FIG. 6H illustrates a surface cost map 6400 overlaid on a representation of a source container 3510 containing an object 3520 . Surface cost map 6400 may be generated as a mathematical combination of height gradient cost map 6200 and normal difference cost map 6300 . In an embodiment, the computer system may combine the height difference and normal difference values according to a filtering operation, such as an averaging filter or a SOBEL filter. In an embodiment, the values in the height gradient cost map 6200 and the normal difference cost map 6300 may be normalized and combined. In an embodiment, a weighting factor may be applied to the height difference value or the normal difference value to control how strongly the surface costmap depends on the corresponding difference value. The weighting factor may be a normal weighting factor, for example, multiplied by the normalized normal difference to determine how strongly the final surface cost map 6400 should be determined from the normal difference or how strongly the final surface cost map 6400 should be determined from the height difference factor. As discussed below, the selection of normal weighting factors can be performed based on the intended object type. As shown in Figure 6H, open circles illustrate portions of low surface cost map values, filled circles illustrate portions of larger surface cost map values, and crosses illustrate 3510 and cannot be identified as part of an object. For purposes of illustration, values are shown as high and low, although in practice these values may span a range of possible values. It can be seen that the surface costmap values at the boundaries of the object 3520 are greater than the surface costmap values across the central portion of the object 3520 .

As discussed above, surface costmap generation may be performed based on one or more parameters including subdivision region size, stride size, distance threshold, normal threshold, and normal weight.

The subdivision area size and stride size may be selected or determined based on various factors to achieve various results. In an embodiment, smaller subdivision region sizes may be chosen to provide results that are more sensitive to small changes in the 3D point cloud, but smaller subdivision region sizes are also more sensitive to noise. In an embodiment, a larger subdivision region size may be chosen to smooth out small changes in the 3D point cloud, whether these changes are due to noise or changes in the actual object being imaged. In an embodiment, a small step size may be selected to provide a high resolution, detailed surface cost map, although such a small step size may require increased computing power and/or increased processing time. In an embodiment, a larger stride size results in downsampling of the 3D point cloud, which may provide faster results and/or lower computational resource usage at the expense of some detail. In an embodiment, selecting a stride size that is less than 0.5, less than 0.4, and/or less than 0.3 of the subdivision area size may provide an appropriate amount of detail while still providing faster results. In an embodiment, a stride size that is at or about half the size of the subdivision region may provide a balance between reduced resolution, speed, and level of detail. It will be appreciated that the choice of subdivision area and stride size will be influenced by the availability of processing or computing power. Increased computing resources may allow more detailed surface cost maps to be generated without adversely increasing processing time.

In an embodiment, the composition of the objects in the object source affects the optimal values of the subdivision region size and the stride size. For example, collections of objects with small, sharp discontinuities can benefit from smaller stride sizes in order to capture finer details. In another example, collections with rough but deformable objects can benefit from larger subdivision region sizes to provide greater smoothing. In another example, if the subdivision region size is large compared to the object size (e.g., the object size is only 2, 3 or 4 times the subdivision region size), then the surface costmap may include few smooth regions , since many of the subdivisions covering the object will also overlap the edges of the object where the discontinuity exists. In another example, objects with smooth curved surfaces with small radii can result in incorrectly high costs if the subdivision region size and stride size are too large.

In an embodiment, the composition of the objects in the object source also affects the optimal values for the distance threshold, normal threshold and normal weighting factor. For example, referring now to FIG. 61 , consider a box-like object 6500 and a bag-like object 6501 (which are examples of object 3520 ). The central portion of the objects 6500/6501 has a smoothness property that describes the overall or body smoothness of the objects 6500/6501, while the edges of the objects 6500/6501 describe the transitions between the objects 6500/6501. Therefore, it is advantageous to choose parameters that can take advantage of this.

For example, the distance threshold may be chosen according to object size. Any detected altitude difference equal to or greater than the distance threshold will be set to the maximum altitude difference. Thus, height differences at the edges of objects 6500/6501 may have the same impact on surface cost map 6400 regardless of whether the object is on top of a stack of several objects or just one object. The greater drop in height at the edges of objects 6500/6501 (eg, because objects 6500/6501 are stacked on top of other objects 6500/6501) does not provide any additional information for identifying object transitions.

In another example, the normal threshold can be chosen based on object shape. For example, for a box-like object 6500, it is expected that the normals will have low variance. In this case, the normal threshold can be chosen to be a value larger than the expected variation due to noise. Therefore, in the normal difference costmap, any normal difference that is identified as larger than any normal difference caused by the noise difference is set to the maximum value. In box-like objects 6500, since all object surfaces are likely to be planar, any change in the normal that can be identified as real (because it exceeds the noise value) can indicate a discontinuity between objects 6500. For such objects 6500, the normal weighting factor may also be chosen so as to give roughly equal weight to normal differences and height differences. In another example, a bag-like object, such as object 6501, may have portions where the angle changes significantly, rather than representing an object discontinuity. In this case, the normal weighting factor can be chosen to give greater weight to height differences, since differences in normals provide less information about object discontinuities. In yet another example, deformable bags can be expected to have large variations in normals, so the normal weighting factor can be chosen to give greater weight to height differences, since differences in normals provide very little information about the discontinuity of the object.

As discussed above, different parameters can give better or worse results in surface costmap generation depending on the object type and object size in the source container. In an embodiment, surface costmap generation parameters may be manually selected, for example based on the expected type and size of objects in the source container. In a further embodiment, parameter selection may be automated and may be performed based on, for example, the obtained 2D image information 2600 and/or the obtained 3D image information 5700 . As discussed above, object detection (including, for example, object registration) may be performed on the obtained 2D image information 2600 and/or the obtained 3D image information 5700 to identify the size, shape and/or type of objects in the source container . Depending on object detection (eg, object registration), surface costmap generation parameters can be automatically selected, including subdivision region size, stride size, distance threshold, normal threshold, and normal weighting factor.

In embodiments including source containers with multiple different types of objects, distance thresholds, normal thresholds, and normal weighting factors may be adjusted within the surface cost map for regions associated with different types of registered objects.

Returning now to FIG. 4 , at operation 4006 , method 4000 includes segmentation of image information (eg, 2D image information 2600 and/or 3D image information 5700 ). Segmentation may be performed from the surface cost map 6400 generated by the methods described above or by any suitable method. The segmented image information may provide a plurality of image segments that use the values of the surface costmap 6400 to identify various objects in the scene. An image segmentation process according to an embodiment is described with respect to FIGS. 7A-7E .

In operation 7002 of the image segmentation method 7000, an initial segmentation of the surface cost map may be performed by applying a cost threshold. Applying a cost threshold generates a thresholded boundary 7102 between object parts 7101 in a thresholded mask 7100, as shown in FIG. 7B. Threshold boundaries 7102 represent regions with surface costmap values exceeding the threshold, while object parts 7101 represent regions with surface costmap values not exceeding the threshold. Thus, the threshold boundary 7102 may be represented by a "false" value in the thresholded mask 7100, while the object portion 7101 is represented by a "true" value. The assignment of "false" and "true" values is by convention only, and any suitable distinction may be applied. Object portion 7101 represents a first estimate of an object surface, while threshold boundary 7102 represents a first estimate of an object boundary or discontinuity. Object boundary 7103 represents the actual object boundary and is provided for comparison purposes.

In an operation 7004 of the image segmentation method 7000, the thresholded mask 7100 may be further defined in a mask definition operation. Mask definition operations may include one or more of connected component analysis and mask erosion, as explained with respect to FIG. 7C . Thresholded mask 7100 may be further defined to generate defined mask 7200 .

Generating defined mask 7200 may include performing mask erosion on thresholded mask 7100 . Mask etching is an operation that reduces or erodes the boundaries of a mask according to the structuring elements. A structuring element may represent, for example, an NxN set of pixels or points with an output pixel/point which may be located at the center of the structuring element. When placed on a mask, the output point of the structuring element in the eroded mask is set to true if every point in the mask that coincides with a point in the structuring element is true. Therefore, for a point in the eroded mask to be true, every surrounding point in the original mask up to the size of the structured element must also be true. Etching thus has the effect of removing one or more dots at the edge of the mask and smoothing any irregularities in the mask. In an example, structuring elements that are half the size of the minimum pickable area size (e.g. the smallest area size that can be grasped by a robotic arm, which can be, for example, the size required for a suction gripper to achieve a firm grip) can be used A mask etch is performed on the thresholded mask 7100 . This etch operation can thus be used to disconnect any portion of the mask that is smaller than the minimum pickable area size.

In operation 7006 of image segmentation method 7000 , object regions may be identified within defined cost mask 7200 . Still referring to FIG. 7C , connected component analysis may be performed on the defined cost mask 7200 to identify object regions 7201 that lie within the defined cost mask 7200 . Object region 7201 may represent a finer estimate of object location and boundaries than previously discussed object portion 7101 .

In operation 7008 of image segmentation method 7000 , an image segment 7301 from within object region 7201 may be selected and further defined. Referring now to FIGS. 7C and 7D , an image segment 7301 may be selected as an object region 7201 having a seed 7204 located therein. The seed 7204 may be the point in the surface cost map with the lowest cost (eg, the smoothest point that is least likely to represent a boundary or discontinuity). A segment map 7300 (FIG. 7D) containing image segments 7301 may be generated by removing all object regions 7201 that do not include a seed. The image segment 7301 may then be dilated with structuring elements corresponding to half the size of the smallest pickable area. Dilation is the opposite operation to erosion. During dilation, the output pixels/points of the structured elements become the input points. When overlaid on the fragment graph 7300, if the point on the fragment graph 7300 corresponding to the input point of the structured element is true, then all points in the fragment graph 7300 corresponding to the structured element are set to true. The dilation has the effect of extending the border of the image segment 7301 by an amount corresponding to the size of the structuring element.

In an operation 7010 of the image segmentation method 7000, an image segment 7301 may be verified. Validation of the image segment 7301 may be performed to determine whether the identified image segment 7301 represents a feasible object of the plurality of objects. A bounding box 7305 (eg, a square or rectangular box) may be fitted around the identified image segment 7301 . The bounding box 7305 can then be compared to the maximum object candidate size and the minimum object candidate size. The maximum candidate object size and the minimum candidate object size represent the maximum and minimum possible object sizes determined during the object detection process. If the bounding box is larger than the maximum candidate object size or smaller than the minimum candidate object size, the image segment 7301 may be determined to be invalid, requiring iterations of operations 7002 , 7004 , 7006 , and 7008 . Iterations can be performed with a reduced cost threshold if the bounding box is larger than the maximum candidate object size. Iterations can be performed with an increased cost threshold if the bounding box is smaller than the minimum candidate object size.

In an embodiment, the bounding box may also be compared to the expected minimum pickable area size. The smallest pickable area size may correspond to the smallest possible area size that can be picked, for example to the size of a single suction gripper of a robotic arm. In embodiments, the robotic arm may employ more than one suction gripper, eg 2 or 4. The desired minimum pickable area size may be a parameter corresponding to the size of the area necessary to achieve a selected or desired grip (eg, the area necessary to enable 2 or 4 suction grippers to achieve a grip). If the bounding box is smaller than the desired minimum pickable area size, operations 7002, 7004, 7006, and 7008 may be iterated with increasing thresholds.

After the image segment 7301 has been verified, it can be stored as a pickable region for further analysis. Image segment 7301 may then be removed from surface cost map 6400, and operations 7002-7010 may be repeated to identify additional image segments 7301. In an embodiment, the cost threshold may be increased before repeating operations 7002-7010. Method 7000 can be repeated and the cost threshold can be increased until no additional fragments are detected or identified. FIG. 7E illustrates a set of image segments 7301 identified from the surface cost map 6400 . In an embodiment, the identified image segment 7301 may be designated as a pickable area. In an embodiment, the identified image segment 7301 may be further analyzed to determine pickable regions therein.

At operation 4008, method 4000 includes generating a detection mask. A detection mask may be generated to refine or further define potentially pickable regions of objects corresponding to the image segment 7301 determined from the image segmentation operation 4006 .

For example, as shown in Figure 8A, because the bounding box of operation 7010 is a two-dimensional construct, it may not correspond exactly to the actual height of the point on the object. In FIG. 8A , a bounding box 8021 has been fitted to an object 8022 . However, due to the deformable nature of the object 8022, the actual points 8023 on the surface of the object 8022 do not all fall within the bounding box 8021. Accordingly, in operation 4008, detection mask information may be generated to identify portions of the object within the bounding box that are more or less suitable for object pickup.

FIG. 8B illustrates detection mask information 8300 . The detection mask information 8300 may include information about objects within the bounding box 8021 (eg, the bounding box for the image segment 7301 generated during operation 7010). Inspection mask information 8300 includes identified regions 8024 and 8027 and unidentified region 8026 . Identified areas 8024 and 8027 may include detected area 8024 (which includes detected and unoccluded area) and occluded area 8027 . An occluded area 8027 may be unsafe or useless for object picking, while a detected area 8024 may be safe for picking. Unidentified regions 8026 may include regions that are not identified for either occlusion or pick-up and are generally not used for or relied upon for detection. A minimum pickable area 8025 is also illustrated in Figure 8B. As shown, it can be seen that the detected area 8024 labeled "B" is not large enough to accommodate the minimum pickable area 8025 . Accordingly, detection mask information 8300 may be used in conjunction with the image segmentation techniques described above to identify pickable regions of objects.

At operation 4010, method 4000 may include determining a safety volume for use in a motion planning operation. A safe volume may represent a volume that an object selected for picking may occupy. The safe volume is chosen to reduce the probability that the selected object, once picked up, will collide with other objects in the object disposal environment.

Referring now to FIG. 9A , a safe volume 9100 is provided around a pickable area 9201 designated as the pickable area of an object 3520 . The safe volume may be determined to have a size twice the difference between the designated pickable area 9201 for picking and the expected object size. This safe volume size thus creates a volume around the pickable area 9201 that may provide a margin of error in the potential dimensions of the object, for example, if the pickable area 9201 is not centered on the object 3520 to be picked. The size of the safety volume 9100 can then be modified as follows.

First, the Safe Volume 9100 is compared to the 3D point cloud. If the 3D point cloud does not support the safe volume 9100 size (e.g., the safe volume 9100 is too large and would extend beyond the boundaries of the 3D point cloud, which correspond to the boundaries of the source container 3510), then the size of the safe volume 9100 can Reduced to a size supported by the 3D point cloud. The safety volume 9100 can then be aligned to the edges of the 3D point cloud. FIG. 9B illustrates a situation where safe volume 9100 is reduced to safe volume 9101 because the boundaries of safe volume 9100 would extend beyond the 3D point cloud associated with source container 3511 .

If the safety volume 9100/9101 is larger than the specified maximum allowable size of the destination container, the safety volume 9100/9101 may be further reduced. For example, if the destination container is smaller than the source container, then the safe volume 9100/9101 may be too large for the destination container. The safety volume 9100/9101 can thus be reduced or adjusted accordingly. In an embodiment, where the safety volume 9100/9101 is larger than the destination container and cannot be adjusted to be smaller than the size of the destination container, if the object 3520 is known to fit into the destination container, then a Deterministic motion planning.

If the detection bounding box of operation 7010 extends beyond the safe volume 9100/9101, then the safe volume 9100/9101 may be further adjusted. This may happen, for example, due to shrinking or realigning the safe volume as described above, or if the bounding box is arranged in an inconvenient manner relative to the pickable area 9201 forming the basis of the safe volume 9100/9101. In an embodiment, to address this issue, the safety volume 9100/9101 may be moved to include the bounding box, or the bounding box may be moved and aligned to the safety volume 9100/9101.

At operation 4012, method 4000 includes outputting a pickable area detection result. Pickable area detection results may include any or all of the information generated in operations 4002-4010, including, for example, identified image segments 7301, their associated bounding boxes 7305, identified pickable areas 9201, and safe volumes 9100/9101 . The pickable area detection results may include pickable area detection result information regarding any or all detected objects 3520 within the source container 3510 .

At operation 4014, method 4000 may include generating and/or outputting a motion plan based on the pickable region detection results. Motion planning may include robotic instructions for following a trajectory, grasping or picking up an object 3520 through an identified pickable area 9201, and delivering the object 3520 to a destination container, while based on the determined safe volume of the object 3520 9100/9101 to consider a potential collision.

It will be apparent to those of ordinary skill in the relevant art that other suitable modifications and adaptations may be made to the methods and applications described herein without departing from the scope of any embodiment. The embodiments described above are illustrative examples and the present disclosure should not be construed as being limited to these specific embodiments. It should be understood that the various embodiments disclosed herein may be combined in combinations other than those specifically presented in the specification and drawings. It should also be understood that, depending on the example, certain acts or events of any process or method described herein may be performed in a different order, added to, combined, or omitted entirely (e.g., all described acts or events may not be performed in the manner of performing the method or necessary for the process). In addition, although certain features of the embodiments herein are described for clarity as being performed by a single component, module or unit, it is to be understood that the features and functions described herein may be implemented by any combination of components, units or modules implement. Accordingly, various changes and modifications can be made by those skilled in the art without departing from the spirit or scope of the invention as defined in the appended claims.

Further examples may include:

Embodiment 1 is a computing system comprising: a control system configured to communicate with a robot having a robotic arm that includes or is attached to an end effector device and to communicate with a camera; at least one processing circuit, configured to, when the robot is located in the object handling environment including a source of objects for delivery to a destination within the object handling environment: obtain image information of the object; identify the object by pickable regions of one or more selected ones of the objects: generating a surface cost map from the image information, segmenting the surface cost map to obtain one or more image segments, the one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; and generating a pickable region detection result including at least the one or more pickable regions; and generating a pickable region detection result for the robotic system A motion plan for the one or more selected objects is transmitted, the motion plan is based on the pickable area detection result.

Embodiment 2 is the system of embodiment 1, wherein the surface cost map represents a smoothness of the one or more selected objects.

Embodiment 3 is the system of embodiment 1 or 2, wherein the image information includes three-dimensional information, and the processing circuit is further configured to calculate height gradients between cells defined in the image information and normal difference to generate the surface costmap.

Embodiment 4 is the system of any one of embodiments 1 to 3, wherein the at least one processing circuit is further configured to generate the surface cost map based on surface cost map parameters.

Embodiment 5 is the system of any one of embodiments 1 to 4, wherein the at least one processing circuit is further configured to: register the one or more objects based on the image information to create object registration information; and determining the surface cost map parameters according to the object registration information.

Embodiment 6 is the system of any one of embodiments 1 to 5, wherein the at least one processing circuit is further configured to generate detection mask information indicative of the one or more pickable regions of the image segment, The detection mask information includes detected regions and occluded regions within the one or more image segments.

Embodiment 7 is the system of any one of embodiments 1 to 6, wherein segmenting the surface cost map comprises: applying a cost threshold to the surface cost map to generate a thresholded mask; eroding the surface cost map; thresholding the mask to generate an eroded mask; and applying connected component analysis to the eroded mask to identify a first image segment.

Embodiment 8 is the system of any one of embodiments 1 to 7, wherein segmenting the surface cost map further comprises: removing the first image segment from the surface cost map; applying a second cost threshold to the the remainder of the surface costmap to generate a second thresholded mask; erode the second thresholded mask to generate a second eroded mask; and apply connected component analysis to the second eroded mask mask to identify the second image segment.

Embodiment 9 is the system of any one of embodiments 1 to 8, wherein generating the pickable area detection result further comprises: generating a safety volume surrounding the one or more pickable areas, the safety volume indicating The estimated remainder of the one or more selected objects.

Embodiment 10 is a method of object transfer performed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm comprising or to which a camera is attached to an end effector apparatus, the method comprising: obtaining image information of one or more objects contained in an object source; identifying possible ones of one or more selected ones of the objects by Picking up regions: generating a surface cost map based on the image information, segmenting the surface cost map to obtain one or more image segments, the one or more image segments identifying one corresponding to the one or more selected objects or a plurality of pickable areas; and generating a pickable area detection result including at least the one or more pickable areas; and generating a motion plan for the robotic system to transport the one or more selected objects, the Motion planning is based on the pickable region detection results.

Embodiment 11 is the method of embodiment 10, wherein the surface cost map represents smoothness of the one or more selected objects.

Embodiment 12 is the method of embodiment 10 or 11, wherein the image information includes three-dimensional information, and the method further includes generating The surface cost map.

Embodiment 13 is the method of any one of embodiments 10 to 12, further comprising generating the surface cost map based on surface cost map parameters.

Embodiment 14 is the method of any one of embodiments 10 to 13, further comprising: registering the one or more objects based on the image information to create object registration information; and determining the surface based on the object registration information Costmap parameters.

Embodiment 15 is the method of any one of embodiments 10 to 14, further comprising generating detection mask information indicative of the one or more pickable regions of the image segment, the detection mask information comprising the Detected and occluded regions within one or more image segments.

Embodiment 16 is the method of any one of embodiments 10 to 15, wherein segmenting the surface cost map comprises: applying a cost threshold to the surface cost map to generate a thresholded mask; eroding the surface cost map thresholding the mask to generate an eroded mask; and applying connected component analysis to the eroded mask to identify a first image segment.

Embodiment 17 is the method of any one of embodiments 10 to 16, wherein segmenting the surface cost map further comprises: removing the first image segment from the surface cost map; applying a second cost threshold on the remainder of the surface costmap to generate a second thresholded mask; erode the second thresholded mask to generate a second eroded mask; and apply connected component analysis to the A second etched mask to identify a second image segment.

Embodiment 18 is the method of any one of embodiments 10 to 17, wherein generating the pickable area detection result further comprises: generating a safety volume surrounding the one or more pickable areas, the safety volume indicating The estimated remainder of the one or more selected objects.

Embodiment 19 is a non-transitory computer readable medium configured with executable instructions for object transfer performed by a control system having at least one processing circuit and configured to communicate with a robotic arm having and communicating with a camera, the robotic arm comprising or attached to an end effector device, the instructions being configured to: obtain image information of one or more objects contained in the object source; by To identify the pickable area of one or more selected objects among the selected objects: generate a surface cost map according to the image information, segment the surface cost map to obtain one or more image segments, and The one or more image segments identify one or more pickable areas corresponding to the one or more selected objects; and generating a pickable area detection result including at least the one or more pickable areas; and for The robotic system generates a motion plan for communicating the one or more selected objects, the motion plan based on the pickable area detection results.

Embodiment 20 is the non-transitory computer readable medium of embodiment 19, wherein the image information includes three-dimensional information, and the instructions are further configured to, based on the defined height between cells in the image information gradient and normal differences to generate the surface costmap.

Claims (20)

1. A computing system comprising: a control system configured to communicate with the robot having a robotic arm that includes or is attached to the end effector device and to communicate with the camera; At least one processing circuit configured to perform the following operations while the robot is in the object handling environment including an object source for delivery to a destination within the object handling environment: obtaining image information of the object; Identifies the pickable area of one or more of the selected ones of said objects by: generating a surface cost map from said image information, segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; and generating a pickable area detection result including at least the one or more pickable areas; and A motion plan for conveying the one or more selected objects is generated for a robotic system, the motion plan based on the pickable area detection results. 2. The system of claim 1, wherein the surface cost map represents a smoothness of the one or more selected objects. 3. The system of claim 1 , wherein the image information includes three-dimensional information, and the processing circuit is further configured to calculate the height gradient between cells defined in the image information and Line differences to generate the surface cost map. 4. The system of claim 3, wherein the at least one processing circuit is further configured to generate the surface cost map based on surface cost map parameters. 5. The system of claim 4, wherein the at least one processing circuit is further configured to: registering the one or more objects based on the image information to create object registration information; and The surface cost map parameters are determined according to the object registration information. 6. The system of claim 1 , wherein the at least one processing circuit is further configured to generate detection mask information indicative of the one or more pickable regions of an image segment, the detection mask information Including detected areas and occluded areas in the one or more image segments. 7. The system of claim 1, wherein segmenting the surface costmap comprises: applying a cost threshold to the surface cost map to generate a thresholded mask; etching the thresholded mask to produce an etched mask; and Connected component analysis is applied to the eroded mask to identify a first image segment. 8. The system of claim 7, wherein segmenting the surface costmap further comprises: removing said first image segment from a surface costmap; applying a second cost threshold to the remainder of the surface cost map to generate a second thresholded mask; etching the second thresholded mask to generate a second etched mask; and Connected component analysis is applied to the second eroded mask to identify a second image segment. 9. The system of claim 1, wherein generating the pickable region detection result further comprises: A safe volume is generated around the one or more pickable regions, the safe volume indicating an estimated remainder of the one or more selected objects. 10. A method of object transfer performed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm comprising or attached to a terminal An actuator device, the method comprising: obtaining image information of one or more objects contained in the object source; Identifies the pickable area of one or more of the selected ones of said objects by: generating a surface cost map from said image information, segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; and generating a pickable area detection result including at least the one or more pickable areas; and A motion plan for conveying the one or more selected objects is generated for a robotic system, the motion plan based on the pickable area detection results. 11. The method of claim 10, wherein the surface cost map represents a smoothness of the one or more selected objects. 12. The method according to claim 10, wherein the image information includes three-dimensional information, and the method further comprises generating the The surface cost map described above. 13. The method of claim 12, further comprising generating the surface cost map from surface cost map parameters. 14. The method of claim 13, further comprising: registering the one or more objects based on the image information to create object registration information; and The surface cost map parameters are determined according to the object registration information. 15. The method of claim 10, further comprising generating detection mask information indicative of the one or more pickable regions of the image segments, the detection mask information comprising the one or more image segments The detected area and the occluded area within. 16. The method of claim 10, wherein segmenting the surface costmap comprises: applying a cost threshold to the surface cost map to generate a thresholded mask; etching the thresholded mask to produce an etched mask; and Connected component analysis is applied to the eroded mask to identify a first image segment. 17. The method of claim 16, wherein segmenting the surface costmap further comprises: removing the first image segment from the surface costmap; applying a second cost threshold to the remainder of the surface cost map to generate a second thresholded mask; etching the second thresholded mask to generate a second etched mask; and Connected component analysis is applied to the second eroded mask to identify a second image segment. 18. The method of claim 10, wherein generating the pickable region detection result further comprises: A safe volume is generated around the one or more pickable regions, the safe volume indicating an estimated remainder of the one or more selected objects. 19. A non-transitory computer readable medium configured with executable instructions for object transfer performed by a control system having at least one processing circuit and configured to communicate with a robot having a robotic arm In communication with and with a camera, the robotic arm includes or is attached to an end effector device, the instructions being configured to: obtaining image information of one or more objects contained in the object source; Identifies the pickable area of one or more of the selected ones of said objects by: generating a surface cost map from said image information, segmenting the surface cost map to obtain one or more image segments identifying one or more pickable regions corresponding to the one or more selected objects; and generating a pickable area detection result including at least the one or more pickable areas; and A motion plan for conveying the one or more selected objects is generated for a robotic system, the motion plan based on the pickable area detection results. 20. The non-transitory computer-readable medium of claim 19, wherein the image information includes three-dimensional information, and the instructions are further configured to height gradient and normal difference to generate the surface costmap.