CN114683299B - Robot tool and method of operating the same - Google Patents

Abstract

A system and method for operating a robotic system to manipulate objects using two or more tools. The robotic system may coordinate planning processes and/or planning implementations based on grouping the objects and available manipulation tools. The robotic system may use two or more tools to coordinate planning processes and/or implementations to reduce task completion time and/or task error rates.

Description

Robot tools and how to operate them

This application is a divisional application of Chinese application CN202180004739.8. The application date is November 3, 2021, and the invention is titled “Robot Tool and Its Operation Method”.

Cross-references to related applications

This application claims the benefit of U.S. Provisional Patent Application Serial No. 63/109,870, filed on November 5, 2020, which is incorporated by reference in its entirety.

Technical field

The present technology relates generally to robotic systems, and more specifically, to robotic tools configured to grasp and hold objects.

Background technique

Robots (eg, machines configured to perform physical actions automatically/autonomously) are now widely used in many fields. Robots may be used, for example, to perform various tasks (eg, manipulate or handle objects) in manufacturing, packaging, transportation and/or shipping, etc. Robots are capable of replicating human movements when performing tasks, thereby replacing or reducing the human involvement that would otherwise be required to perform dangerous or repetitive tasks. Robots often lack the level of sophistication necessary to replicate the human sensitivity and/or adaptability required to perform more complex tasks. For example, robots often have difficulty grasping objects in certain suboptimal positions or postures. Therefore, there remains a need for improved robotic systems and techniques for handling objects using sets of grasping tools.

Description of the drawings

Figure 1 is an illustration of an exemplary environment in which a robotic system transports objects in accordance with one or more embodiments of the present technology.

Figure 2 is a block diagram illustrating a robotic system in accordance with one or more embodiments of the present technology.

Figure 3 is a top view of a robotic system in accordance with one or more embodiments of the present technology.

Figure 4A is an illustration of an exemplary handling unit in accordance with one or more embodiments of the present technology.

Figure 4B is an illustration of an exemplary tool set in accordance with one or more embodiments of the present technology.

Figure 5A is an illustration of a standard grasping scenario in accordance with one or more embodiments of the present technology.

Figure 5B is an illustration of an angled grip scenario in accordance with one or more embodiments of the present technology.

Figure 6A is an illustration of a standard release scenario in accordance with one or more embodiments of the present technology.

Figure 6B is an illustration of a first angled release scenario in accordance with one or more embodiments of the present technology.

Figure 6C is an illustration of a second angled release scenario in accordance with one or more embodiments of the present technology.

Figure 7 is an example timing diagram in accordance with one or more implementations of the present technology.

Figure 8 is a flow diagram for operating a robotic system in accordance with some embodiments of the present technology.

Detailed ways

This article describes systems and methods for selecting, changing, and using end effector tool sets. For example, a robotic system may include a transport robot configured to selectively connect to an end effector tool. Robotic systems can be selected and connected to different end effector tools to grasp and handle objects. Robotic systems can access and select from tool sets. In some embodiments, the tool set may include standard fixed tools, fixed angle tools, and/or flexible head tools. Standard stationary tools may have end effectors rigidly attached to the tool arm. Standard stationary tools can be configured to grip objects with a general orientation or top surface that is orthogonal to the orientation of the tool arm. Fixed angle tools can be configured to have non-orthogonal angles between the tool arm and the gripper head/interface. The flexible or adjustable tool head can be configured such that the relative angle/orientation between the tool arm and the gripper head/interface can be adjusted based on the posture of the target object.

In some embodiments, the robotic system may select an end effector tool based on: simulating and/or planning the handling of the object based on the selected tool; deriving a measure of the handling (e.g., costs and/or benefits); targeting a set of goals The object writes a set of plans; and/or selects a combination/sequence of plans that optimizes a corresponding metric (eg, total handling time and/or estimated loss/error rate). Additionally or alternatively, the robotic system may implement the selected plan in parallel with the derivation of the plan for subsequent objects.

Tool groups (e.g., multiple selectable tools) and/or coordinated planning and tool usage of a group of objects (e.g., rather than based on individual objects as they are detected) enable reduced resource consumption and/or or reduced error. In some embodiments, the planning and verification process may last less than the duration necessary to execute/implement the plan (e.g., one to five seconds) and/or the duration necessary to change tools (e.g., five to ten seconds). time (e.g., one second or less). Thus, by planning handling for a set of multiple objects (eg, rather than planning for one object at a time), the robotic system can derive a set of actions that minimize the highest cost (eg, time/duration) such as tool change operations. plan.

In the following, numerous specific details are set forth to provide a thorough understanding of the presently disclosed technology. In other embodiments, the techniques introduced herein may be practiced without these specific details. In other instances, well-known features such as specific functions or routines have not been described in detail in order to avoid unnecessarily obscuring the disclosure. Reference in this description to "an embodiment," "one embodiment," or similar language means that a particular feature, structure, material or characteristic described is included in at least one embodiment of the present disclosure. Accordingly, the appearances of such phrases in the specification are not necessarily all referring to the same embodiment. On the other hand, such references are not necessarily mutually exclusive. Furthermore, the particular features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments. It should be understood that the various embodiments shown in the figures are illustrative representations only and are not necessarily drawn to scale.

For purposes of brevity, several details are not set forth in the following description that describe structures or processes that are well known and often associated with robotic systems and subsystems and that may unnecessarily obscure some important aspects of the disclosed technology. Furthermore, while the following disclosure sets forth several embodiments of different aspects of the technology, several other embodiments may have different configurations or components than those described in this section. Accordingly, the disclosed technology may have other embodiments with additional elements or without several of the elements described below.

Many embodiments or aspects of the disclosure described below may be in the form of computer or controller executable instructions, including routines executed by a programmable computer or controller. Those skilled in the relevant art will understand that the disclosed technology may be practiced on computer or controller systems other than those shown and described below. The techniques described herein may be embodied in a special purpose computer or data processor specifically programmed, configured, or constructed to execute one or more of the computer-executable instructions described below. Thus, the terms "computer" and "controller" as used generally herein refer to any data processor and may include Internet appliances and handheld devices (including handheld computers, wearable computers, cellular or mobile phones, multi-processor systems, Processor-based or programmable consumer electronics, network computers, minicomputers, etc.). Information processed by these computers and controllers may be presented at any suitable display medium, including liquid crystal displays (LCDs). Instructions for performing computer or controller-executable tasks may be stored in or on any suitable computer-readable medium, including hardware, firmware, or a combination of hardware and firmware. Instructions may be contained in any suitable memory device (including, for example, a flash drive, USB device, and/or other suitable media, including tangible, non-transitory computer-readable media).

The terms "coupled" and "connected" and their derivatives may be used herein to describe structural relationships between components. It should be understood that these terms are not intended as synonyms for each other. Rather, in particular embodiments, "connected" may be used to mean that two or more elements are in direct contact with each other. Unless the context clearly dictates otherwise, the term "coupled" may be used to mean that two or more elements are in direct or indirect contact with each other (with other intervening elements in between), or that two or more elements cooperate or interact with each other ( For example, if in a causal relationship, such as for signal transmission/reception or for function calling), or both.

suitable environment

Figure 1 is an illustration of an exemplary environment in which a robotic system 100 transports objects in accordance with one or more embodiments of the present technology. Robotic system 100 may include one or more units (eg, robots) configured to perform one or more tasks and/or communicate with the one or more units. Aspects of tool selection and usage can be practiced or implemented by various units.

For the example shown in FIG. 1 , the robotic system 100 may include an unloading unit 102 , a handling unit 104 (eg, a palletizing robot and/or a picking robot), a transport unit 106 , a loading unit 108 located in a warehouse or distribution/shipping hub. , or their combination. Each of the units in robotic system 100 may be configured to perform one or more tasks. Tasks may be combined in sequence to perform operations that achieve a goal, such as unloading objects from a truck or van and storing them in a warehouse, or unloading objects from a storage location and preparing them for shipping. As another example, the task may include placing an object at a target location (eg, on top of a pallet and/or inside a bin/cage/box/box). As described below, the robotic system may derive a plan for placing and/or stacking objects (eg, placement locations/orientations, sequences for handling objects, and/or corresponding motion plans). Each of the units may be configured to perform a sequence of actions (eg, by operating one or more components therein) to perform the task according to one or more of the derived plans for performing the task.

In some embodiments, the task may include manipulating (eg, moving and/or reorienting) target object 112 (eg, one of a package, box, bin, cage, pallet, etc. corresponding to the task being performed) , such as moving target object 112 from starting position 114 to task position 116 . For example, the unloading unit 102 (eg, an unpacking robot) may be configured to transport the target object 112 from a location in a carrier (eg, a truck) to a location on a conveyor belt. Additionally, the handling unit 104 may be configured to handle the target object 112 from one location (eg, a conveyor belt, a pallet, or a bin) to another location (eg, a pallet, a bin, etc.). As another example, the handling unit 104 (eg, a palletizing robot) may be configured to handle the target object 112 from a source location (eg, a pallet, a pick area, and/or a conveyor) to a destination pallet. Upon completion of the operation, the transport unit 106 may transport the target object 112 from an area associated with the handling unit 104 to an area associated with the loading unit 108 , and the loading unit 108 may transport the target object 112 from the handling unit 104 (e.g., via The pallet carrying the target object 112 is moved to a storage location (eg, a location on a shelf). Details regarding tasks and associated actions are described below.

For illustrative purposes, robotic system 100 is described in the context of a shipping center; however, it is understood that robotic system 100 may be configured to operate in other environments/for other purposes (such as for manufacturing, assembly, packaging, health care, etc.) care and/or other types of automation) to perform tasks. It should also be understood that the robotic system 100 may include other units not shown in Figure 1, such as manipulators, service robots, modular robots, and the like. For example, in some embodiments, the robotic system 100 may include a depalletizing unit for moving objects from a cage or pallet onto a conveyor or other pallet, a depalletizing unit for moving objects from one container to another. Container exchange units, packaging units for wrapping objects, classification units for grouping objects according to one or more properties of the objects, for manipulating objects in different ways according to one or more properties of the objects (e.g. , sorting, grouping and/or handling) pickup units, or their combination.

Robotic system 100 may include and/or be coupled to physical or structural members (eg, robotic manipulator arms) connected at joints for movement (eg, rotational and/or translational displacement). Structural members and joints may form a kinetic chain configured to manipulate end effectors (e.g., grippers) configured to perform one or more tasks depending on the purpose/operation of robotic system 100 ( For example, grasping, spinning, welding, etc.). Robotic system 100 may include actuation devices (e.g., motors, actuators, wires) configured to drive or manipulate (e.g., displace and/or reorient) structural members about or at corresponding joints. , artificial muscles, electroactive polymers, etc.). In some embodiments, the robotic system 100 may include transport motors configured to transport corresponding units/chassis around.

Robotic system 100 may include sensors configured to obtain information for performing tasks, such as for manipulating structural members and/or for transporting robotic cells. Sensors may include one or more physical characteristics of the robotic system 100 (e.g., the state, condition, and/or position of one or more of its structural members/joints) and/or the surrounding environment. Physical properties of the device. Some examples of sensors may include accelerometers, gyroscopes, force sensors, strain gauges, tactile sensors, torque sensors, position encoders, etc.

In some embodiments, for example, a sensor may include one or more imaging devices configured to detect the surrounding environment (e.g., visual and/or infrared cameras, 2D and/or 3D imaging cameras, distance measurements such as lidar or radar devices, etc.). The imaging device may generate a representation of the detected environment, such as a digital image and/or point cloud, that may be processed by machine/computer vision (eg, for automated inspection, robot guidance, or other robotic applications). Robotic system 100 may process digital images and/or point clouds to identify target object 112, starting location 114, task location 116, pose of target object 112, or a combination thereof.

For manipulating target object 112 , robotic system 100 may capture and analyze images of a designated area (eg, a pickup location, such as the interior of a truck or on a conveyor belt) to identify target object 112 and its starting location 114 . Similarly, the robotic system 100 may capture and analyze another designated area (e.g., a drop location for placing an object on a conveyor, a location for placing an object inside a container, or a location on a pallet for stacking purposes). ) to identify the task location 116. For example, the imaging device may include one or more cameras configured to generate images of the pickup area and/or one or more cameras configured to generate images of the task area (eg, drop area). As described below, based on the captured images, the robotic system 100 may determine the starting position 114, the task position 116, associated poses, packing/placement plans, handling/packing sequences, and/or other processing results.

In some embodiments, for example, the sensors may include position sensors configured to detect the position of structural members of the robotic system 100 (eg, robotic arms and/or end effectors) and/or corresponding joints. Robotic system 100 may use position sensors to track the position and/or orientation of structural members and/or joints during performance of a task.

robotic system

Figure 2 is a block diagram illustrating a robotic system 100 in accordance with one or more embodiments of the present technology. In some embodiments, for example, robotic system 100 (eg, at one or more of the units and/or robots described above) may include electronic/electrical devices, such as one or more processors 202, One or more storage devices 204, one or more communication devices 206, one or more input-output devices 208, one or more actuating devices 212, one or more transport motors 214, one or more sensors 216, or a combination of them. Various devices may be coupled to each other through wired connections and/or wireless connections. For example, the robotic system 100 may include a bus such as a system bus, a Peripheral Component Interconnect (PCI) bus or a PCI Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), IIC (I2C) bus, or Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also known as "FireWire"). Additionally, for example, the robotic system 100 may include a bridge, adapter, processor, or other signal-related device for providing wired connections between devices. Wireless connections may be based on, for example, cellular communication protocols (e.g., 3G, 4G, LTE, 5G, etc.), wireless local area network (LAN) protocols (e.g., Wireless Fidelity (WIFI)), peer-to-peer or inter-device communication protocols (e.g., Bluetooth, Near field communication (NFC), etc.), Internet of Things (IoT) protocols (e.g., NB-IoT, LTE-M, etc.) and/or other wireless communication protocols.

Processor 202 may include a data processor (eg, central processing unit (CPU), special purpose computer, and/or onboard computer) configured to execute instructions (eg, software instructions) stored on storage device 204 (eg, computer memory). server). In some embodiments, processor 202 may be included in a separate/stand-alone controller operatively coupled to other electronic/electrical devices shown in FIG. 2 and/or the robotic unit shown in FIG. 1 . The processor 202 may implement program instructions that control/interact with other devices, thereby causing the robotic system 100 to perform actions, tasks, and/or operations.

Storage device 204 may include non-transitory computer-readable media having program instructions (eg, software) stored thereon. Some examples of storage 204 may include volatile memory (eg, cache and/or random access memory (RAM)) and/or non-volatile memory (eg, flash memory and/or disk drives). Other examples of storage 204 may include portable memory drives and/or cloud storage.

In some embodiments, storage 204 may be used to further store and provide access to processing results and/or predetermined data/thresholds. For example, storage device 204 may store master data including descriptions of objects (eg, boxes, bins, and/or products) that may be manipulated by robotic system 100 . In one or more embodiments, master data may include registration data for each such object. Registration data may include dimensions, shapes (e.g., templates of potential poses and/or computer-generated models for identifying objects in different poses), color schemes, images, identification information (e.g., , barcodes, Quick Response (QR) codes, logos, etc. and/or their intended location), expected weight, other physical/visual characteristics, or a combination thereof. In some embodiments, the master data may include information related to the manipulation of the objects, such as center of mass (CoM) positions or estimates thereof on each of the objects, expected sensor measurements corresponding to one or more actions/manipulations ( For example, for force, torque, pressure and/or contact measurements), or combinations thereof.

Communication device 206 may include circuitry configured to communicate with external or remote devices over a network. For example, communication device 206 may include a receiver, transmitter, modulator/demodulator (modem), signal detector, signal encoder/decoder, connector port, network card, or the like. Communication device 206 may be configured to send, receive, and/or process electrical signals in accordance with one or more communication protocols (eg, Internet Protocol (IP), wireless communication protocols, etc.). In some embodiments, robotic system 100 may use communications device 206 to exchange information between elements of robotic system 100 and/or with systems or devices external to robotic system 100 (e.g., for reporting, data collection, analysis and/or troubleshooting purposes).

Input-output devices 208 may include user interface devices configured to communicate information to and/or receive information from a human operator. For example, input-output device 208 may include display 210 and/or other output devices for communicating information to a human operator (eg, speakers, haptic circuitry, or tactile feedback devices, etc.). Additionally, input-output devices 208 may include control or receiving devices such as keyboards, mice, touch screens, microphones, user interface (UI) sensors (eg, cameras for receiving motion commands), wearable input devices, and the like. In some implementations, robotic system 100 may use input-output devices 208 to interact with a human operator performing actions, tasks, operations, or combinations thereof.

In some implementations, a controller (eg, a separate electronic device) may include a processor 202 , a storage device 204 , a communication device 206 , and/or an input-output device 208 . The controller may be a separate component or part of the unit/assembly. For example, each unloading unit, handling assembly, transport unit, and loading unit of system 100 may include one or more controllers. In some embodiments, a single controller may control multiple units or independent components.

Robotic system 100 may include physical or structural components (eg, robotic manipulator arms) connected at joints for movement (eg, rotational and/or translational displacement). Structural members and joints may form a kinetic chain configured to manipulate end effectors (e.g., grippers) configured to perform one or more tasks depending on the purpose/operation of robotic system 100 ( For example, grasping, spinning, welding, etc.). Robotic system 100 may include an actuation device 212 (e.g., motor, actuator, Wires, artificial muscles, electroactive polymers, etc.). In some embodiments, the robotic system 100 may include a transport motor 214 configured to transport a corresponding unit/chassis around. For example, the actuation device 212 and transport motor are connected to or are part of a robotic arm, linear slide, or other robotic component.

Sensors 216 may be configured to obtain information for performing tasks, such as for manipulating structural members and/or for transporting robotic units. Sensors 216 may include one or more physical characteristics of the robotic system 100 (e.g., the state, condition, and/or position of one or more of its structural members/joints) and/or the surrounding environment. A device with physical characteristics. Some examples of sensors 216 may include contact sensors, proximity sensors, accelerometers, gyroscopes, force sensors, strain gauges, torque sensors, position encoders, pressure sensors, vacuum sensors, and the like.

In some embodiments, for example, sensors 216 may include one or more imaging devices 222 configured to detect the surrounding environment (e.g., visual and/or infrared cameras, 2- and/or 3-dimensional imaging cameras, such as lidar or radar) distance measuring device, etc.). Imaging device 222 may include cameras (including visual and/or infrared cameras), lidar devices, radar devices, and/or other distance measurement or detection devices. Imaging device 222 may generate a representation of the detected environment, such as a digital image and/or point cloud, that may be processed by machine/computer vision (eg, for automated inspection, robot guidance, or other robotic applications).

For manipulating target object 112 , robotic system 100 (eg, via the various circuits/devices described above) may capture and analyze images of a designated area (eg, a pickup location, such as inside a truck or on a conveyor) to identify target object 112 and its starting position 114. Similarly, the robotic system 100 may capture and analyze another designated area (e.g., a drop location for placing an object on a conveyor, a location for placing an object inside a container, or a location on a pallet for stacking purposes). ) to identify the task location 116. For example, imaging device 222 may include one or more cameras configured to generate images of the pickup area and/or one or more cameras configured to generate images of the task area (eg, drop area). As described below, based on the captured images, the robotic system 100 may determine the starting position 114, the task position 116, associated poses, packing/placement plans, handling/packing sequences, and/or other processing results.

In some embodiments, for example, sensors 216 may include position sensors 224 configured to detect the position of structural members of robotic system 100 (eg, robotic arms and/or end effectors) and/or corresponding joints. Robotic system 100 may use position sensors 224 to track the position and/or orientation of structural members and/or joints during performance of a task. Robotic system 100 may use detected positions, tracked positions, tracked orientations, etc. from sensors 216 to derive tracking data representing a current position and/or a set of past positions of target object 112 and/or structural members of FIG. 1 .

Figure 3 is a top view of a robotic system 100 in accordance with one or more embodiments of the present technology. In some implementations, robotic system 100 may include a system manager 302 , a planner 304 , and/or a robotic arm 306 . System manager 302 and/or planner 304 may be implemented or include one or more of the circuits shown in Figure 2 (eg, processor 202, storage device 204, communications device 206, etc.).

System manager 302 may include mechanisms (eg, devices and/or software applications) configured to manage the overall operation of one or more task stations and/or corresponding robotic units. For example, system manager 302 may include a facility management system such as for a warehouse or shipping hub. In managing overall operations, system manager 302 may receive order input 312 (eg, customer requests for a set of objects accessible by robotic system 100). System manager 302 may derive various tasks, interactions/controls between task stations and/or corresponding robots, associated sequences or timing, etc., to collect the objects listed in order input 312 . System manager 302 may further interact with robotic arm 306 to implement/perform tasks.

Planner 304 may include mechanisms (eg, devices, software applications/features, or combinations thereof) configured to derive detailed controls for operating one or more robots or components thereof. Planner 304 may derive detailed steps, such as motion plans for the robot and/or communication protocols or sequences with other subsystems, to operate one or more robotic units and complete tasks determined by system manager 302 .

To operate robotic arm 306 , robotic system 100 may use planner 304 to derive a handling plan 314 corresponding to a path for moving one or more items from starting location 114 of FIG. 1 to task location 116 of FIG. 1 . For example, robotic system 100 may obtain one or more images of starting location 114 and task location 116 via imaging device 222 of FIG. 2 . The robotic system 100 may process the images to identify or identify the ordered objects at the starting location 114 and/or their posture within a bin (eg, a container with at least one vertical wall). Similarly, robotic system 100 may use images of task location 116 to derive or identify placement locations for each of the target objects at starting location 114 . The planner 304 may derive a handling plan 314 that includes operating the robotic arm 306 (eg, the actuating device 212 of FIG. 2 and/or the transport motor 214 of the actuating device 212 of FIG. 2 ) to operate the robot arm 306 in order to The path for transporting the target object between corresponding positions and/or the corresponding commands, settings, timing, etc. In some implementations, the planner 304 may iteratively derive the move plan 314 by starting at a placement location and iteratively deriving the next incremental location as it moves toward the destination. The next incremental position may be a test position that satisfies one or more predetermined rules (such as for avoiding collisions, minimizing distance/time, etc.).

The robotic system 100 may implement the handling plan 314 and operate the robotic arm 306 accordingly to handle one or more target objects identified in the order input 312 . For example, the system manager 302 may interact with the robotic arm 306 such as by communicating commands/settings for the path and/or handling plan 314 to the robotic arm 306 . The robotic arm 306 can execute the received information to perform the task.

As an illustrative example, system manager 302 may interact with one or more robots (e.g., transportation units such as automated guided vehicles (AGVs), conveyors, etc., and/or subsystems) to access storage areas therein Containers of ordered objects (including, for example, starting bin 322). Accordingly, the robotic system 100 may operate the transport unit to transport the starting bin 322 to the starting position 114 of the robotic arm 306 . Similarly, a destination container 324 (eg, a packaging box or a destination bin) may be placed at the task location 116 of the robotic arm 306 . Alternatively, task location 116 may correspond to a drop or placement location on a conveyor or robot (eg, AGV). Once the starting position 114 and the task position 116 are prepared, the system manager 302 can interact with the robotic arm 306 according to the handling plan 314 . Accordingly, the robotic arm 306 may grasp the target object 112 (eg, one or more of the objects specified in the order input 312 ) and carry it/them from the starting location 114 to the task location 116 . For example, the robotic arm 306 may pick up ordered objects from one or more starting bins 322 at the starting location 114 and place them in one or more target containers 324 and/or designated locations at the task location 116 to complete the order. .

In some embodiments, the robotic arm 100 may utilize a set of tools (eg, specialized end effectors) to perform different tasks using the same robot and/or to improve performance of a given task. For example, the robotic system 100 may selectively connect the robotic arm 306 to a gripper, welder, or cutter to perform the corresponding function according to the assigned task. Additionally, the robotic system 100 may selectively connect the robotic arm 306 to a clamp gripper or vacuum based on the physical characteristics of the target objects and/or their surrounding environment (e.g., relative position of other objects, availability of access paths, etc.) Gripper.

In some embodiments, as described in detail below, the robotic system 100 can selectively connect the robotic arm 306 to grippers with different angled interfaces based on the target object pose. For example, the robotic system 100 may select the first tool when the target object is positioned flat on the bottom of the bin, exhibiting a relatively horizontal top surface and/or a relatively vertical circumferential surface. Alternatively, the robotic system 100 may select the second tool when the target object has an angled posture relative to the lateral plane, such as when the target object has non-parallel surfaces or is placed on top of an uneven or non-planar contact point. The second tool may include a contact or gripping interface configured to grip such angled objects.

When utilizing a tool set, the system manager 302 may provide target selections 313 to the planner 304 to identify the tools and/or target objects 112 selected for one or more of the tasks/objects. The planner 304 may derive a handling plan 314 based on the target selection 313 and derive feedback 318 accordingly. For example, when the derivation is successful (eg, one or more threshold conditions are met, such as for collision avoidance or minimum error estimate met), feedback 318 may include a handling plan 314 for handling the target object using the designated tool. Feedback 318 may include an error message when the export is unsuccessful, such as when the specified tool is not suitable for gripping and/or handling the target object (due to, for example, causing a collision event). In alternative embodiments, planner 304 may select a tool and derive a corresponding move plan 314 without interacting with system manager 302 regarding tool selection.

When multiple tools are available, the robotic system 100 can export and evaluate multiple tasks or actions as a single group rather than processing each task or action individually. Alternatively or additionally, the robotic system 100 may coordinate the sequence or timing between derivation/evaluation, tool changes, and plan implementation to improve or maximize the efficiency (eg, overall completion time) of a set of tasks. Because export processes, tool change processes, and the implementation of actions have different costs and benefits, exporting and evaluating individual tasks and actions as a group can lead to overall performance improvements. For example, the duration necessary to derive and evaluate the handling plan 314 (eg, one second or less) may be less than the duration necessary to change tooling (eg, five seconds or more) and/or less than is implemented at the robotic arm 306 The required duration of the schedule (for example, one to five seconds). In some embodiments, changing tools may take longer than the average or maximum duration necessary to complete the handling of an item.

To derive and evaluate a set of tasks, the robotic system 100 (eg, system manager 302 and/or planner 304) may determine tool-based object groupings 316 that identify or group the items according to tools appropriate for their manipulation. . For example, the robotic system 100 may derive a tool-based object grouping 316 based on the pose of the target object within the starting bin 322 (using, for example, the angle or orientation of the exposed surface relative to one or more predetermined horizontal/vertical lines or planes). The robotic system 100 may evaluate different sequences or combinations of handling plans 314 based on a set of predetermined rules that take into account the cost of changing and utilizing tools. In one or more embodiments, the robotic system 100 may calculate costs associated with each handling plan 314, such as estimated handling times and/or estimated error probabilities. The robotic system 100 may evaluate any handling plan that results in a collision and/or has a probability of error (eg, probability of loss) that exceeds a predetermined threshold as described above, and therefore determines a plan to use appropriate tools to grasp the object. The robotic system 100 may derive different sequences of the determined set of plans and include or plan tool change operations accordingly. For each sequence, the robotic system 100 may calculate an overall cost metric (eg, overall completion time or estimated completion time taking into account possible error rates) for evaluation. For implementation, the robotic system 100 may select a sequence that minimizes the overall cost metric.

In some embodiments, the robotic system 100 can improve the overall performance of a task through sequential and parallel implementation of control plans, tool changes, and plan implementation. For example, the robotic system 100 may identify a tool that has been attached to the robotic arm 306 when the object is identified or when the starting bin 322 is placed at the starting position 114 . Robotic system 100 may identify a first set of target objects that can be grasped with existing tools. Robotic system 100 may select one of the objects in the group and derive a corresponding plan. Once the first plan is completed, the robotic system 100 may implement the first plan in parallel with the derivation of the second plan in the group. When the plan for the first set of objects is completed, the robotic system 100 may derive a plan for the second set of objects in parallel with (eg, concurrently/simultaneously with) the implementation of the first set of plans and/or tool change operations.

In other exemplary embodiments, system manager 302 may identify an initial tool that has been attached to robotic arm 306 . System manager 302 can interact with planner 304 to identify and plan a first object that can be grasped and carried using the attached tool. During implementation of the first plan, system manager 302 may interact with planner 304 to plan the second object. When the export fails, the system manager 302 may select and interact to iteratively plan different objects in parallel with previously planned implementations. When the export is successful for existing tools, the corresponding objects and plans can then be implemented. When there are no remaining target objects also suitable for the current connection, the robotic system 100 may plan to implement the tool change operation while the plan is in progress or at the end of a previous implementation. In alternative embodiments, the robotic system 100 may derive and evaluate the feasibility and cost of all available tools for each target object. The robotic system 100 may analyze the entire group derivation when determining a preferred order.

Example tools

Figure 4A is an illustration of an exemplary handling unit (eg, robotic arm 306) in accordance with one or more embodiments of the present technology. The robotic arm 306 may be the handling unit 104 of FIG. 1 (eg, a picking or bin-picking robot) or a portion thereof. Robotic arm 306 may include an arm portion 402 configured to manipulate an end effector (eg, a gripper) across an operating space. Arm portion 402 may include a set of structural members (e.g., beams, columns, etc.), a set of joints between the structural members, and/or a corresponding set of actuators configured to move the set of structural members about the joints. motor.

The end effector tool 404 may be attached to the arm portion 402 such as at its distal end. The end effector tool 404 may include a tool connector 412 (eg, a selective locking/attachment mechanism) configured to interface with and attach the tool to the arm portion 402 . The tool connector 412 may be structurally connected to or integral with the tool arm 414 with end effectors attached at opposite ends. For the example shown in Figure 4A, the end effector may include a grasper configured to grasp an object for manipulation (eg, transport or displacement across space). The gripper end effector may include a gripper head 416 that houses or facilitates a gripping interface 418 (eg, a set of suction cups for a vacuum-based gripper). Gripping interface 418 may be used to generate an attachment force or mechanism (eg, vacuum) that attaches the target object to gripper head 416 and/or robotic arm 306 .

Figure 4B is an illustration of an exemplary tool set 430 in accordance with one or more embodiments of the present technology. Tool set 430 may represent tools that may be used by robotic system 100 of FIG. 1 to perform one or more tasks. In some embodiments 440 , the tool set 430 may include a standard fixed gripping tool 440 , a fixed angle gripping tool 450 , and/or an adjustable gripping tool 460 . Although not shown in Figure 4B, it is understood that tool set 430 may include other tools, such as finger-based pinch grippers, grippers with different types of suction/contact interfaces, different types of ends Actuators (e.g., such as non-gripping end effectors for welding or cutting), etc.

The standard fixed gripping tool 440 may be configured to grip objects that are positioned relatively flat compared to the floor of the starting bin 322 . For example, a standard fixed gripping tool 440 may include a laterally oriented grip interface 418 for gripping a laterally oriented top surface of an object. The joint connecting the tool arm 414 to the gripper head 416 may be secured with the tool arm 414 extending either orthogonally to the suction interface or parallel to the clamping interface.

Fixed angle gripping tool 450 may be configured to grip objects positioned at an angle compared to the floor of starting bin 322 . For example, the fixed angle gripping tool 450 may include a suction gripping interface 418 and a tool arm 414 configured at non-orthogonal angles for gripping a non-lateral top surface of an object. The angled joint 452 may securely connect the tool arm 414 and the gripper head 416 together with structures forming corresponding non-orthogonal angles or orientations.

Adjustable gripping tool 460 may be configured to adjust the orientation of gripper head 416 relative to tool arm 414 . For example, the adjustable gripping tool 460 may include an orientation control mechanism 464 configured to adjust the orientation of the gripper head relative to the tool arm 414 and about the rotatable joint 462, such as a manipulator arm, a set of cables, an actuator, Motor etc. Accordingly, the robotic system 100 may operate the orientation control mechanism 464 to adjust the posture or orientation of the grasping interface 418 based on the orientation of the target object or one or more portions thereof (eg, the exposed top surface).

Example tool usage

Figure 5A is an illustration of a standard grasping scenario 500 in accordance with one or more embodiments of the present technology. A standard grasping scenario 500 may be used for grasping an object 502 in a flat pose. Flat pose object 502 may correspond to a target object having an orientation parallel to a bottom surface of starting bin 322 (eg, the bottom surface of the object). Therefore, standard gripping scenario 500 may correspond to standard fixed gripping tool 440 . For example, the robotic system 100 of FIG. 1 may operate the robotic arm 306 of FIG. 3 to position the standard fixed grasping tool 440 directly over the object 502 in a flat posture and lower the end effector to grasp the object.

Figure 5B is an illustration of an angled grip scenario 510 in accordance with one or more embodiments of the present technology. Angled grasp scenario 510 may be used to grasp angled objects 512 . Angled objects 512 may include objects that lean or rest on uneven contact points (eg, rest along non-horizontal surfaces) and/or objects that have non-parallel opposing surfaces. In some implementations, angled object 512 may correspond to a pose having one or more surfaces oriented along an angled direction/plane relative to a horizontal/vertical reference direction. Thus, angled grip scenario 510 may correspond to fixed angle grip tool 450 . For example, the robotic system 100 of FIG. 1 may operate the robotic arm 306 of FIG. 3 to position the fixed-angle grasping tool 450 directly over the angled object 512 and lower the end effector to grasp the object.

For comparison, a standard fixed gripping tool 440 may not be suitable for the angled gripping scenario 510. For example, to grip the angled object 512 , the standard fixed gripping tool 440 may be tilted or angled such that the gripping interface 418 is oriented parallel to the interface connection portion of the angled object 512 . Lowering the standard fixed gripping tool 440 into a tilt may cause a collision event between the tool and/or the robotic arm 306 and the starting bin 322 (indicated by an 'X' in Figure 5B).

In some implementations, robotic system 100 may select fixed angle gripping tool 450 based on surface pose 514 of angled object 512 . For example, robotic system 100 may process one or more images (eg, top view images) of starting bin 322 and/or angled objects 512 therein as captured by imaging device 222 of FIG. 2 . Robotic system 100 may identify edges depicted in images based on edge detection mechanisms (eg, Sobel filters). The robotic system 100 may identify each continuous surface depicted in the image based on determining connections and/or relative orientations between a set of edges and/or identifying shapes, colors, and/or designs located between the edges. Robotic system 100 may map the surfaces to a three-dimensional image (eg, a depth map) and use the depth measure to calculate one or more slopes for each surface. Using the one or more calculated slopes, the robotic system 100 may derive a surface pose 514 for each surface. The robotic system 100 may determine the angled object 512 based on comparing the surface pose 514 to a threshold representing a horizontal or flat surface.

Figure 6A is an illustration of a standard release scenario 600 in accordance with one or more embodiments of the present technology. Standard release scenario 600 may be used to release a grasped object (eg, target object 112 of FIG. 1 ) from standard fixed grasping tool 440. Standard release scenario 600 may be used to place a grasped object (eg, target object 112 of FIG. 1 ) at a corresponding placement location at task location 116 and/or in target container 324 of FIG. 3 . The standard release scenario 600 may correspond to a flat placement of the grasped object and/or an upright orientation of the standard fixed grasping tool 440 .

Figure 6B is an illustration of a first angled release scenario 610 in accordance with one or more embodiments of the present technology. The first angled release scenario 610 may be used to release a grasped object (eg, target object 112 of FIG. 1 ) from the fixed angle grasping tool 450. The first angled release scenario 610 may be used to place a grasped object (eg, target object 112 of FIG. 1 ) at a corresponding placement location at task location 116 and/or in target container 324 of FIG. 3 . The first angled release scene 610 may correspond to a flat lay posture 612 of the grasped object. Accordingly, the robotic system 100 of FIG. 1 may orient the fixed-angle gripping tool 450 into an angled tool release posture 614 (eg, in which the tool arm is in a non-vertical or tilted orientation). Tool release posture 614 may correspond to angled joint 452 of FIG. 4B of fixed angle gripping tool 450.

Figure 6C is an illustration of a second angled release scenario 620 in accordance with one or more embodiments of the present technology. The second angled release scenario 620 may be used to release a grasped object (eg, target object 112 of FIG. 1 ) from the fixed angle grasping tool 450. The second angled release scenario 620 may be used to place a grasped object (eg, target object 112 of FIG. 1 ) at a corresponding placement location at task location 116 and/or in target container 324 of FIG. 3 . The second angled release scene 620 may correspond to the angled placement pose 622 of the grasped object. Accordingly, the robotic system 100 of FIG. 1 can orient the fixed-angle gripping tool 450 to an upright tool release posture 624 (eg, in which the tool arm is in a vertical orientation). The angled placement position 622 may correspond to the angled joint 452 of FIG. 4B of the fixed angle gripping tool 450. The angled placement posture 622 and the upright tool release posture 624 may be used to release a grasped object without achieving the orientation manipulation associated with the tool release posture 614 and the flat placement posture 612 shown in Figure 6B. Accordingly, the angled placement posture 622 and the upright tool release posture 624 may correspond to lower implementation costs (eg, shorter implementation time) than the tool release posture 614 and the flat placement posture 612 . The angled position 622 and the upright tool release position 624 may be implemented for objects with predetermined characteristics, such as for softer and/or more flexible objects that are less susceptible to impact damage.

Example task sequence

Figure 7 is an example timing diagram 700 in accordance with one or more implementations of the present technology. Timing diagram 700 may represent a sequence or temporal relationship of operations and/or processes of robotic system 100 of FIG. 1 (eg, system manager 302 of FIG. 3, planner 304 of FIG. 3, and/or robotic arm 306 of FIG. 3). . For example, timing diagram 700 may show the temporal relationship between planning schedule 702 and implementation schedule 704 . The plan schedule 702 may represent a sequence of processes for each scenario in which the handling plan 314 of FIG. 3 is derived, such as for handling a target object with assigned tooling as specified by the target selection 313 of FIG. 3 . Implementation schedule 704 may represent a planned implementation/execution sequence at/by robotic arm 306 .

The robotic system 100 can perform the plan derivation process and the plan implementation process in parallel with improving overall efficiency and reducing overall task execution time. For the example shown in Figure 7, system manager 302 may send a first object selection (eg, target selection 313) identifying Object 1 to be grasped and handled using Part 1 (eg, standard fixed gripping tool 440 of Figure 4B) Case). In response to the first object selection, planner 304 may derive a first object plan 712 for gripping and handling object 1 with part 1 . Upon successful export, planner 304 may communicate first object plan 712 to system manager 302 via corresponding feedback 318 of FIG. 3 , and system manager 302 may interact with robotic arm 306 to implement first object plan 712 . Implementation of first object handling 732 of schedule 704 may represent implementation of first object plan 712 at robotic arm 306 .

Additionally, upon successful export, the system manager 302 may send a second object selection identifying the object 2 to be grasped and carried using the connected part 1 (a new case of target selection 313). In response to the second object selection, planner 304 may derive a second object plan 714, such as using the exemplary iterative process described above. When the second object is the angled object 512 of Figure 5B, the planner 304 may return a failure status indicating that a successful plan cannot be derived, such as due to an estimated collision event that occurred while attempting to grasp the object. Planner 304 may communicate the failure status to system manager 302, such as through corresponding feedback 318. In some implementations, system manager 302 may identify other target objects to be considered for planning with the currently connected tool. Such as when any remaining objects cannot be grasped with the currently attached tool and/or gripper head orientation, the system manager 302 may utilize updated tool selections (e.g., fixed angle gripping tool 450 of FIG. 4B and/or the The alternative head angle of the adjustable gripping tool 460) identifies the second object for planning. Using the updated tools, planner 304 may derive an updated second object plan 716 . Implementation of second object handling 736 of schedule 704 may represent implementation of updated second object plan 716 at robotic arm 306 . Various planning processes may occur in parallel with (eg, independently of) implementation schedule 704 .

When a follow-up plan is successfully exported, system manager 302 may queue its implementation in implementation schedule 704 . When a successful export is based on providing an updated tool, such as for the updated second object plan 716 , in target selection 313 , system manager 302 may include tool changes 734 in implementation schedule 704 .

In some embodiments, robotic system 100 (located, for example, at system manager 302 and/or planner 304) may estimate implementation metrics for various implementations. For example, the robotic system 100 may derive a first plan metric 722 for the first object plan 712 (e.g., including the total execution time to pick and/or place the object), a second plan metric 726 for the updated second object plan 716, and/or or a tool change measure 724 of the tool change 734 (eg, maximum and/or average time required). Robotic system 100 may use the achieved metrics to derive a plan sequence that minimizes the combined metrics. In some embodiments, the tool change measure 724 may represent the cost associated with switching the posture of the gripping interface 418 on the adjustable gripping tool 460 between functioning as a standard fixed gripping tool 440 and a fixed angle gripping tool 450 . Tool change metrics 724 may also include or take into account costs (eg, time to failure and/or change) associated with achieving tool release gesture 614 (eg, additional maneuvers compared to standard release from a standard fixed grip tool). ). Additionally, tool change measure 724 may also include or take into account costs associated with drag changes, such as due to angled gripping and/or additional manipulation that reduces drag or associated with changes in surface area relative to the direction of movement. air resistance.

As an illustrative example, robotic system 100 may derive a planned schedule 702 for moving multiple objects from a starting location (eg, a bin) to a destination location (eg, a different bin or conveyor). Robotic system 100 may derive estimated implementation metrics for handling one or more of the objects using the corresponding tool. The estimated measurements may take into account any speed changes of different tools, additional manipulations, and/or adjusted pick/drop manipulations or gestures. In some implementations, the estimated metrics may further take into account expected error rates associated with corresponding object tool combinations. For example, the estimated measure may include handling time with the calculated amount increased to account for remedial measures to be deployed in the event of an error (eg, lost piece). Additional adjustments may be calculated based on weighting the average duration of the remediation action by the error rate associated with the object tool combination and/or corresponding movement settings (eg, speed). Thus, the robotic system 100 may balance (1) any cost of using a suboptimal tool to handle an object against (2) the cost of changing tooling to use the optimal tool to handle the object and the impact of the tool change in handling other objects. Negative conditioning. Accordingly, the robotic system 100 may evaluate the overall cost of different combinations of motion plans and select one motion plan that minimizes the cost of handling one or more or all of the target objects. In some embodiments, the estimated measure may be expressed as picks per minute or hour using the same robotic unit.

Operating procedures

8 is a flowchart of a method 800 for operating a robotic system (eg, robotic system 100 of FIG. 1 ) in accordance with one or more embodiments of the present disclosure. Method 800 may be implemented using one or more of the devices described above, such as system manager 302 of Figure 3, planner 304 of Figure 3, and/or robotic arm 306 of Figure 3. Method 800 may be implemented using one or more of the components described above (such as processor 202 of Figure 2, storage device 204 of Figure 2, etc.). Method 800 may be used to plan and implement a task (eg, move an object) using a tool set (eg, tool set 430 of Figure 4B). As described in detail below, method 800 may correspond to planning and implementing tasks in parallel and/or processing objects in groups according to appropriate tools for minimizing tool changes 734 of FIG. 7 .

At block 802, the robotic system 100 may receive an order for a set of items (eg, order input 312 of Figure 3). For example, robotic system 100 (via communication device 206, such as FIG. 2) may receive order input 312 from a customer or requesting warehouse. Additionally, order input 312 may correspond to internally generated orders to repackage or regroup items for storage, such as to rearrange/combine objects and reduce storage containers. Thus, the robotic system 100 may identify a set of objects that need to be moved from a storage location to a different location (eg, an outgoing container) to fulfill an order.

At block 804, the robotic system 100 may coordinate the retrieval of the ordered object. The robotic system 100 (via, for example, the system manager 302 of Figure 3) may identify one or more storage locations for the ordered objects. For example, the robotic system 100 may compare the order input 312 to a record of stored/available objects. Accordingly, the robotic system 100 identifies the bin and/or object corresponding to the order input 312 shown at block 805 . The robotic system 100 may determine the storage location and/or identifier of the container in which the ordered object is stored.

The robotic system 100 may use the determined location and/or container identifier to coordinate retrieval of the ordered object. For example, the robotic system 100 may directly operate one or more transport units (eg, AGVs and/or conveyors) to transport target containers from the storage location to the starting location 114 of FIG. 1 for the robotic arm 306 of FIG. 3 . The robotic system 100 may similarly operate the transport unit to transport the target container 324 of FIG. 3 from its storage location to the task location 116 of FIG. 1 . Additionally or alternatively, robotic system 100 may interact or communicate with one or more subsystems (eg, a storage access system) to place one or more target containers and/or target containers 324 at their respective locations.

At block 806, the robotic system 100 may obtain image data. For example, robotic system 100 may use imaging device 222 of FIG. 2 to obtain two- and/or three-dimensional images depicting the storage container at the starting location. Thus, the image data obtained may depict one or more of the ordered objects stored within the storage container in real time.

At block 808, the robotic system 100 may identify groupings of objects (eg, target or order objects) in the storage container. The robotic system 100 may process the obtained image data to detect or identify objects within the starting bin 322 . For example, the robotic system 100 may identify surfaces based on detection lines (via, for example, a Sobel detection mechanism) and connections/arrangements between the detected lines. Robotic system 100 may compare the depicted image or portions thereof (eg, portions within the surface) to surface image data in the master data. Additionally or alternatively, the robotic system 100 may compare the dimensions of the identified surface to the object dimensions stored in the master data. Robotic system 100 may identify objects based on a match of the depicted image and/or measured dimensions with corresponding images and/or predetermined dimensions stored in the master data.

The robotic system 100 may compare the identified set of objects in the starting bin 322 to the target or expected portion of the order input 312 . Therefore, the robotic system 100 can locate the target object in the starting bin 322 . The robotic system 100 may further process the target objects to classify/group the objects according to the tool group 430 , thereby determining the tool-based object grouping 316 of FIG. 3 . For example, the robotic system 100 may determine whether each of the target objects is the flat pose object 502 of FIG. 5A or the angled object 512 of FIG. 5B. Robotic system 100 may calculate one or more slopes using depth measurements associated with the depicted surface. Robotic system 100 may use the calculated slopes to derive surface pose 514 of FIG. 5B for each of the target objects. Surface pose 514 may be used to classify or group objects, such as flat pose objects 502, angled objects 512, etc., according to tools that are appropriate or assigned to manipulate the corresponding objects.

The robotic system 100 may further classify/group objects based on other characteristics corresponding to different end effectors and/or target tasks. For example, the robotic system 100 may respond based on structural stiffness associated with different types of grippers (e.g., different types of contact interfaces such as suction grippers, suction cup size/location, finger-based grippers, etc.) and/or outer surface material (e.g., boxes, plastic wraps, bags, etc.), overall shape, etc. to group objects. Additionally, the robotic system 100 may group objects that are bound together and thus require a cutting tool separately from other unbound objects. Accordingly, the grouped objects may correspond to tool-based object grouping 316 .

As an illustrative example, robotic system 100 may determine at least a first set of objects (e.g., a set of flat-posed objects 502 ) and a second set of objects (eg, a set of angled objects) based on image data depicting objects within starting bin 322 512). A first set of objects may have one or more aspects (e.g., posture) that correspond to characteristics of the first tool (e.g., angle/orientation of gripping interface 418), and a second set of objects may have features that correspond to the second tool. One or more aspects of a characteristic (eg, angle/orientation of gripping interface 418). The flat posture objects 502 may each have a top portion parallel to the bottom surface of the starting bin 322 , thus corresponding to a standard fixed gripping tool 440 . The angled objects 512 may each have a top portion surface that is angled relative to the bottom surface of the starting bin 322 , thereby corresponding to the fixed angle gripping tool 450 .

At block 810, the robotic system 100 may track and/or update the connected tool. Robotic system 100 may use internal mechanisms (eg, radio frequency identification (RFID) circuitry, hard-coded identifiers, etc.) to dynamically detect tools connected to robotic arm 306 . Additionally or alternatively, the robotic system 100 may track tool change operations and tool selections to track connected tools.

In tracking and/or updating connected tools, the robotic system 100 may access or determine the set of tools available to the robotic system 100 . For example, the robotic system 100 may access predetermined data regarding the tools and/or corresponding characteristics of the tool set 430 available to the robotic arm 306 (eg, the posture of the grasping interface 418). Additionally or alternatively, the robotic system 100 may dynamically track (eg, in real time) the tool being used or connected to the robotic arm. Thus, the robotic system 100 may determine tools that are remaining (eg, not connected to any other robotic arms) and available for acquisition/connection. The robotic system 100 may further access predetermined or previously measured data to determine tool change measures 724 of FIG. 7 associated with switching between tools (e.g., disconnecting from one tool and connecting to a different tool) while manipulating the target object. For example, costs such as time). In some embodiments, tool change measure 724 may represent the cost associated with changing the posture of gripping interface 418 for adjustable gripping tool 460, such as by manipulating orientation control mechanism 464 of FIG. 4 .

Additionally, at block 810, the robotic system 100 may update the connected tool, such as based on a status determination described below (eg, decision block 820). For example, when it is determined based on the status that a new tool is needed to manipulate/handle an object, the robotic system 100 may operate the robotic arm 306 to disconnect the attached tool and connect to a new tool suitable for subsequent handling. Robotic system 100 may update tracked tool information. In some implementations, robotic system 100 may schedule tool updates (eg, tool changes 734 ) and update tracked information when tool changes 734 are implemented.

At block 812, the robotic system 100 may select one or more objects for planning. Robotic system 100 may select from groupings corresponding to tools connected to robotic arm 306 . Within the selected group, the robotic arm 100 may iteratively select an object for handling. The robotic system 100 may make selections based on a set of predetermined selection rules (such as for selecting higher positioned objects first and/or more closely positioned objects).

At block 814, the robotic system 100 (via, eg, the planner 304) may derive one or more plans (eg, the handling plan 314) for selection. For example, system manager 302 may provide target selections 313 to planner 304 based on the selected object and/or connected tool. In some implementations, robotic system 100 may provide a list of objects to a group. In response, planner 304 may derive one or more handling plans for each object. When the plan export is successful, planner 304 may provide the resulting move plan 314 to system manager 302 via feedback 318 . Otherwise, when plan derivation is unsuccessful (due to estimated collisions, e.g., due to improper tool assignment), the planner 304 may provide an indication of reporting an unsuccessful status (e.g., an inappropriate match between the assigned tool and the target object). ) feedback318.

Accordingly, the robotic system 100 may derive a set of handling plans 314 for handling target objects based on the tool. In some embodiments, for example, the robotic system 100 may follow a process for planning handling based on tool change metrics 724 (according to, for example, a set of rules and/or a framework for minimizing overall handling cost/time). For systems where tool changes have a larger average/minimum magnitude than the planned process and/or planned implementation, the robotic system 100 may plan the handling to minimize tool changes. As an example, the robotic system 100 may use connected tools to plan and schedule its implementation of applicable objects/groups, and then plan for a set of different objects. In some embodiments, the robotic system 100 may iterate through the selection and derivation process and implement the derived plan accordingly. In other embodiments, the robotic system 100 may derive plans for one or more groupings of objects and arrange/queue the plans according to a set of rules for minimizing overall cost.

In deriving the plan, the robotic system 100 may calculate the metrics as shown at block 816 (eg, plan metrics 720 of Figure 7). For example, the robotic system 100 may calculate an estimated duration for each handling plan based on the calculated distance traveled, number and/or type of maneuvers, corresponding settings, and/or the like.

For the example shown in Figure 8, the robotic system 100 may derive a set of plans based on iteratively selecting and planning objects in one grouping before processing another grouping. At decision block 818, planner 304 may determine whether the plan derivation was successful and provide corresponding feedback 318 to system manager 302. When the plan derivation is unsuccessful, the robotic system 100 may determine whether any objects remain in the currently processed object grouping, such as shown at decision block 820 . When an object remains in the currently selected/processed group, flow may proceed to block 812 and the robotic system 100 may select the next object in the group. When there are no more objects remaining in the group (eg, tool-based object group 316 is empty), flow may pass to block 810 and the robotic system 100 may update the connected tool (eg, by performing tool change 734). The robotic system 100 may select and connect a new tool based on the remaining objects in the bin and one or more corresponding tools. Along with tool changes, the robotic system 100 may derive plans for different groups associated with the new tools.

In some embodiments, when planner 304 derives a plan, robotic system 100 may derive and/or update a sequence for implementing the plan. For example, system manager 302 may schedule implementation of plans, such as by queuing the plans at system manager 302 or robotic arm 306 .

In one or more alternative embodiments, robotic system 100 may derive multiple plans for each object (eg, one plan for each available tool). At block 822 , the robotic system 100 may derive the planned sequence/combination based on the corresponding plan metrics and tool change metrics 724 . The robotic system 100 may select and schedule the sequence with the lowest overall cost (eg, overall implementation/handling time).

As an illustrative example of blocks 808-822, robotic system 100 may identify a first tool (eg, standard fixed gripping tool 440) connected to robotic arm 306 prior to handling a set of target objects in starting bin 322. Robotic system 100 may determine and select a first group of objects corresponding to a first tool (eg, in the case of tool-based object grouping 316 , such as flat pose objects 502 ). The robotic system may iteratively derive a first set of plans for operating the robotic arm 306 to handle the first set of objects. In some implementations, robotic system 100 may derive a first set of plans based on selecting objects in the set, deriving test plans for the selected objects, and determining whether the test plans are feasible for implementation. After deriving the first set of plans, the robotic system 100 may schedule tool changes 734 to follow the final plan in the first set. Accordingly, after handling the first set of objects, the robotic system 100 may schedule a second tool (eg, fixed angle gripping tool 450 ) to be connected to the robotic arm 306 . Robotic system 100 may continue to derive a plan for the second set of objects.

At block 824, the robotic system 100 (via, for example, the system manager 302) may implement the scheduled plan. In other words, the robotic system 100 can operate the robotic arm 306 and handle objects according to the derived handling plans and their scheduling sequence. The plan can be implemented in parallel (eg, independently) with the subsequent export process. For example, the system manager 302 may initiate implementation of the first plan at the robotic arm 306 immediately before or after providing the target selection 313 to the planner 304 to initiate plan derivation for the second object. Therefore, the derivation process for the second object or any subsequent objects may occur while the robot arm 306 is handling the first object. Accordingly, the robotic system 100 may continuously derive and implement motion plans until the starting position or the set of target objects is empty, as shown at block 826 .

In some embodiments, the robotic system 100 may include operations for reducing tension in the orientation control mechanism 464 of the adjustable gripping tool 460 in the handling plan. The robotic system 100 may include corresponding commands and/or settings (1) after bringing the corresponding object into contact with at least a portion of the grasping interface and/or (2) before grasping the corresponding object. Accordingly, during implementation, the gripping interface 418 may shift and/or rotate and increase the contact area with the exposed surface of the target object. In other words, the gripping interface 418 can rotate or adjust about the rotatable joint 162 and match the surface pose 514 of the target object. The reduced tension may occur before or during activation of the gripping interface 418 (eg, suction cup) in order to improve the grip established between the gripping interface 418 and the target object.

in conclusion

The above detailed description of examples of the disclosed technology is not intended to be exhaustive or to limit the disclosed technology to the precise forms disclosed. Although specific examples of the disclosed technology are described for illustrative purposes, those skilled in the relevant art will recognize that various equivalent modifications are possible within the scope of the disclosed technology. For example, although processes or blocks are presented in a given order, alternative implementations may perform routines with steps or employ systems with blocks in a different order, and may be deleted, moved, added, subdivided, combined, and/or modified Some processes or boxes to provide alternatives or subcombinations. Each of these processes or boxes can be implemented in a variety of different ways. Additionally, although processes or blocks are sometimes shown being executed in series, these processes or blocks may instead be executed or implemented in parallel, or may be executed at different times. Furthermore, any specific numbers noted herein are examples only; alternative implementations may employ different values or ranges.

These and other changes may be made to the disclosed technology in light of the above detailed description. While the detailed description describes certain examples of the disclosed technology and the best mode contemplated, the disclosed technology may be practiced in many ways, no matter how detailed the above description may appear. The details of the systems may vary widely in their specific implementations but are still covered by the technology disclosed herein. As noted above, the use of specific terms in describing certain features or aspects of the disclosed technology should not be taken to imply that the terms are redefined herein to be limited to any specific features associated with the disclosed technology. Characteristics or aspects. Accordingly, the invention is not limited except by the appended claims. In general, the terms used in the following claims should not be construed as limiting the disclosed technology to the specific examples disclosed in the specification, unless such terms are expressly defined in the Detailed Description section above.

Although certain aspects of the invention are presented below in the form of certain claims, the applicant may contemplate various aspects of the invention in the form of any number of claims. Accordingly, the applicant reserves the right to add additional claims after filing this application, either in this application or in a successor application in the form of such additional claims.

Claims (20)

1. A method for operating a robotic system, the method comprising:

identifying a tool set comprising two or more tools each uniquely configured to manipulate an object, wherein each tool is configured to be selectively attached to a robotic arm;

obtaining image data representing a set of target objects at a starting location, wherein the image data depicts representations of two or more objects in the set of target objects;

grouping the two or more objects according to a set of physical features and/or a current pose of the two or more objects, wherein the set of physical features and/or the current pose corresponds to a unique configuration of the two or more tools in the tool set;

deriving a set of plans comprising at least a first plan and a second plan based on the image data, wherein the set of plans is (1) for operating a robotic arm to handle at least a portion of the set of target objects from the starting position to a target position and (2) in connection with different tool change usages;

Determining one or more tool change metrics for the derived plan, wherein each tool change metric represents a cost associated with removing a previous tool from the robotic arm and attaching a subsequent tool to the robotic arm;

calculating a cost metric for each plan in the set of plans, wherein the calculated cost metric accounts for (1) the tool change metric and/or (2) a handling time associated with picking up, handling, and placing at least the portion of the set of target objects;

a plan is selected from the set of plans based on the cost metric, wherein the selected plan corresponds to a lowest cost metric associated with handling at least the portion of the set of target objects.

2. The method of claim 1, wherein:

the first plan representing effecting a tool change from an initial tool to a subsequent tool, and then transporting an object with the subsequent tool, wherein the first plan corresponds to a first cost measure accounting for the tool change and a first transport time of the object;

the second plan representing the handling of the object with the initial tool, wherein the second plan corresponds to a second cost measure that accounts for a second handling time of the object that is longer than the first handling time; and is also provided with

Selecting the plan includes comparing at least the second cost measure to a combination of the first cost measure and the tool change measure.

3. The method of claim 1, wherein:

grouping the two or more objects includes determining (1) a set of flat pose objects and/or (2) a set of angle objects based on the image data, wherein

The set of objects in a flat pose represents a first subset of objects each having a top portion parallel to a bottom surface of the starting position,

the angled objects represent a second subset of objects each having a top portion surface pose forming an angle with respect to a bottom surface of the starting position;

the tool set represents an adjustable tool and/or two or more fixed tools configured to change an angle between the robotic arm and a gripping interface;

selecting the plan includes (1) minimizing tool changes and/or (2) evaluating cost differences associated with (a) changing tools to accommodate different gestures and (b) maintaining additional maneuvers and/or speed adjustments necessary for the tools across the different gestures.

4. A method as claimed in claim 3, wherein the set of plans is derived based on:

Identifying a first tool connected to the robotic arm prior to handling the set of target objects, wherein the first tool represents one of: (1) A standard fixed gripping tool configured to grip the set of flat pose objects and (2) a fixed angle gripping tool configured to grip the set of angle objects;

determining a first object set corresponding to the first tool, wherein the first object set represents a corresponding one of (1) the set of objects in a flat pose and (2) the set of angled objects;

deriving a first set of plans for operating the robotic arm to use the first tool to handle the first set of objects; and

scheduling a tool change after deriving the first set of plans, wherein the tool change is for connecting a second tool to the robotic arm instead of the first tool after handling the first set of objects, the second tool representing the remainder of: (1) A standard fixed gripping tool configured to grip the set of flat pose objects and (2) a fixed angle gripping tool configured to grip the set of angled objects.

5. The method of claim 4, wherein the first set of plans is derived based on iteratively:

Selecting an object in the first object group;

deriving a test plan for handling the selected object using the first tool; and

determining whether the test plan is viable for implementation, wherein

Repeating the selecting, deriving, and determining in the first object group.

6. The method of claim 5, wherein iteratively deriving the first set of plans comprises:

deriving a first plan for handling a first object, wherein the first plan is verified as viable for implementation; and

deriving a second plan for handling a second object, wherein the second plan is derived after the first plan and verified as viable for implementation;

the method further comprises the steps of:

the first plan is implemented in parallel with deriving the second plan, wherein the first plan is for operating the robotic arm to transport the first object from a storage container to the target location.

7. The method of claim 1, wherein deriving the set of plans comprises:

deriving a test plan for each tool in the tool set for each object depicted in the image data and/or the set of target objects;

Determining a set of validated plans based on determining a feasibility of implementation of each plan according to one or more predetermined rules and/or thresholds; and is also provided with

The method further comprises the steps of:

an implementation sequence is derived based on minimizing a sum of costs for handling the set of target objects, wherein the implementation sequence is a handling plan for implementing each object of the set of target objects.

8. The method of claim 7, wherein:

the robotic system includes (1) a system manager configured to coordinate handling of the set of target objects and (2) a planner configured to derive one or more plans for operating the robotic arm to handle corresponding objects;

deriving the set of validated plans includes deriving the set of validated plans at the planner; and is also provided with

Deriving the implementation sequence includes deriving the implementation sequence at the system manager.

9. The method of claim 1, wherein the calculated cost measure represents additional robotic operations for achieving a tool release pose and/or an adjusted conveyance speed.

10. The method of claim 1, wherein:

the robotic system includes (1) a system manager configured to coordinate handling of the set of target objects and (2) a planner configured to derive one or more plans for operating the robotic arm;

Deriving the set of plans includes

Transmitting a first target selection from the system manager to the planner specifying a first object of the set of target objects;

receiving, at the system manager, a first transfer plan corresponding to the first target selection;

initiating implementation of the first transfer plan using the system manager to operate the robotic arm to transfer the first object; and

during implementation of the first handling plan, a second target selection is sent from the system manager to the planner specifying a second object of the set of target objects.

11. The method of claim 10, wherein deriving the set of plans comprises:

receiving feedback from the planner at the system manager, wherein the feedback represents a failure to derive a transfer plan based on the second target selection; and

in response to the feedback, a third target selection specifying remaining objects in the group associated with the connected tool is sent from the system manager to the planner.

12. The method of claim 4, wherein:

the robotic system includes (1) a system manager configured to coordinate handling of the set of target objects and (2) a planner configured to derive one or more plans for operating the robotic arm;

Deriving the set of plans includes

Sending a first target selection from the system manager to the planner specifying a first object of the set of target objects to be handled using the first tool;

receiving feedback from the planner at the system manager, wherein the feedback represents a failure to derive a transfer plan based on the first target selection;

in response to the feedback, sending a second target selection from the system manager to the planner specifying the first object to be handled using the second tool; and

scheduling a tool change in response to receiving a transfer plan corresponding to the second target selection, wherein the scheduled tool change precedes the transfer plan and represents an operation of the robotic arm to disconnect from the first tool and connect to a second tool between transfers of objects.

13. The method of claim 12, wherein the derived plan and/or the tool changes are queued at the system manager to control timing for implementing the derived plan and/or the tool changes.

14. The method of claim 12, further comprising:

the derived plan and/or the tool change are communicated to the robotic arm, which is configured to implement the derived plan and/or the tool change according to a receiving order.

15. The method of claim 1, wherein the tool change metric represents a cost associated with toggling a gesture of a grip interface on an adjustable grip tool.

16. The method of claim 15, wherein deriving the set of plans comprises deriving a command and/or setting for reducing tension of an orientation control mechanism of the adjustable gripping tool to allow the gripping interface to rotate about a rotatable joint and match a surface pose, wherein the tension reduction occurs (1) after contacting a corresponding object with at least a portion of the gripping interface and (2) before gripping the corresponding object.

17. The method of claim 1, further comprising:

determining a container identifier representing a bin comprising the set of target objects, wherein the bin comprises two or more vertical walls;

wherein:

the set of derived plans includes commands and/or settings for operating a vacuum gripper to contact and grip one or more objects while avoiding contact between the vertical wall and the robotic arm and attached tool.

18. A tangible, non-transitory, computer-readable medium having stored thereon processor instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

Determining at least a first set of objects and a second set of objects based on image data depicting the objects at a starting location, wherein the first set of objects and the second set of objects correspond to a first tool and a second tool, respectively, in a tool set;

determining a tool change metric associated with switching between the first tool and the second tool for handling the object;

deriving at least a first plan and a second plan based on the image data, wherein

The first plan is for operating a robotic arm to carry the first set of objects with the first tool, carry the second set of objects with the second tool, and switch between the first tool and the second tool, and

the second plan is for operating the robotic arm to handle at least a portion of the first set of objects and the second set of objects with the first tool prior to switching tools or without switching tools;

calculating a first cost metric for the first plan, wherein the first cost metric represents an overall handling time for handling the first and second sets of objects and the handoff tool according to the first plan;

Calculating a second cost measure for the second plan, wherein the second cost measure represents an overall handling time for handling the first set of objects and the second set of objects according to the second plan; and

one of the first plan or the second plan is selected to be implemented based on the lower of the respective first cost measure or the second cost measure.

19. A robotic system, comprising:

a communication circuit configured to communicate data, commands, and/or settings with (1) a planner configured to derive a plan for operating the robotic arm and/or (2) a robotic arm configured to selectively connect to a set of tools according to a corresponding plan and grip and handle objects using the connected tools;

at least one processor coupled to the communication circuit and configured to:

determining at least a first set of objects and a second set of objects based on image data depicting a set of objects within a container, wherein the first set of objects and the second set of objects correspond to a first tool and a second tool, respectively, of the set of tools;

determining a tool change metric associated with switching between the first tool and the second tool for handling the object, wherein the tool change metric represents a cost associated with removing a previous tool from the robotic arm and attaching a subsequent tool to the robotic arm; and is also provided with

Deriving a set of plans based on the tool change metrics, wherein the set of plans is for operating the robotic arm to handle the first set of objects with the first tool, handle the second set of objects with the second tool, and switch between the first tool and the second tool;

calculating a cost metric for each plan in the set of plans, wherein the calculated cost metric accounts for (1) the tool change metric and/or (2) a handling time associated with picking up, handling, and placing at least a portion of the object;

a plan is selected from the set of plans based on the cost measure, wherein the selected plan corresponds to a lowest cost measure associated with handling at least a portion of the object.

20. The system of claim 19, further comprising:

the scheduler, the scheduler communicatively coupled to the communication circuit;

the robotic arm communicatively coupled to the communication circuit; and

the set of tools includes the first tool and the second tool configured to (1) selectively connect to the robotic arm and (2) grasp one or more objects for manipulation.