KR102067878B1 - Robot hand for performing task using regrasping and control method thereof - Google Patents

Abstract

The present invention relates to a robot hand for performing a task using a re-grip, and a control method thereof. The present invention relates to a robot hand performing a specific task in a currently held form in order to perform a task for a given purpose. The present invention relates to a robot hand and a control method thereof capable of accomplishing a specific task by using regrasping when it is not possible. A robot hand that performs a task by moving a plurality of objects to achieve the above technical problem, the robot hand comprising: a hand for holding and moving a first object of the plurality of objects; And a controller configured to determine whether the task cannot be performed. If it is determined that the task cannot be performed, the controller controls the hand to release the first object held by the hand and performs the task. The hand may be controlled to re-hold the first object to perform the operation.

Description

ROBOT HAND FOR PERFORMING TASK USING REGRASPING AND CONTROL METHOD THEREOF}

The present invention relates to a robot hand for performing a task using a re-grip, and a control method thereof. The present invention relates to a robot hand performing a specific task in a currently held form in order to perform a task for a given purpose. The present invention relates to a robot hand capable of accomplishing a specific task by using a regrasping operation and a control method thereof.

Robot hand refers to the hand of a machine that constrains or moves an object by a plurality of fingers. The robot hand is a limited area, while the robot arm makes an alternative positioning within a wide working area. It is used to make fine manipulations and grasp objects in the interior.

In addition, the artificial intelligence and robot technology is spreading in various fields, and can be used as a core technology for breaking through handling technology of gripping and assembling parts in a manufacturing environment.

However, most of the operations performed in the actual manufacturing environment based on the above-described robot hands are handling operations for holding and assembling parts, and most handling operations are performed by human processes except for simple repetitive operations in a standardized environment. There is a problem that it is performed.

Therefore, in order to solve the above problems and to increase production efficiency in response to changes in the manufacturing environment, it is urgent to develop a grip / assembly technology equipped with artificial intelligence.

(1) Korean Patent Office Registration No. 10-1323217 (2) Registration number 10-1048762 of the Korean Intellectual Property Office

The present invention relates to a robot hand for performing a task using re-gripping and a control method thereof, and the present invention cannot perform manipulation for a given purpose in a form grasping in order to perform a specific task by the robot hand. When you use regrasping to provide a robot hand that can accomplish a specific task and its control method to the user.

In addition, an object of the present invention is to propose a technique for performing a specific task by using a regrasping when the robot hand can perform a specific task, when the currently grasping form does not allow manipulation for a given purpose. .

In addition, the present invention proposes an optimal algorithm required for reassembling one or both objects when the two-armed robot can assemble the object, if the assembly operation can not be carried out while holding the current object. The purpose.

In addition, an object of the present invention is to provide a handling technology capable of classifying and learning an object by additionally using a tactile sensor and measuring shear force, conductivity of an object, and the like as well as normal force measurement, which is widely used.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are clearly to those skilled in the art from the following description. It can be understood.

A robot hand that performs a task by moving a plurality of objects to achieve the above technical problem, the robot hand comprising: a hand for holding and moving a first object of the plurality of objects; And a controller configured to determine whether the task cannot be performed. If it is determined that the task cannot be performed, the controller controls the hand to release the first object held by the hand and performs the task. The hand may be controlled to re-hold the first object to perform the operation.

In addition, when it is determined that at least one of the direction and location of the first object is changed and the task cannot be performed, the controller may control the release and re-holding operation of the hand.

In addition, the task may be performed by moving the plurality of objects in a preset order, and when it is determined that the first object is moved and cannot perform the task differently from the preset order, the controller controls the hand. Can control the release and re-holding of

The task may be performed by moving the plurality of objects in a preset order, and the controller may control the first object held by the hand so that the first object is disposed in a direction and a position suitable for re-holding. Can be controlled to release.

The controller may control the hand to rotate the released first object at an angle, and control the hand to re-hold the rotated first object.

In addition, the controller may repeat the release and re-holding of the hand until the task is completed.

The apparatus may further include a camera configured to acquire images of the plurality of objects, and the controller may determine at least one of a changed direction and a position of the first object based on the image acquired through the camera. The hand may be controlled to re-hold the first object using at least one of the changed direction and position.

In addition, when the hand is in contact with the first object for the gripping, a tactile sensor that detects at least one tactile factor of normal force, shear force, and conductive of the first object It may further include; and may control to re-hold the first object by using the hand together with at least one tactile factor detected by the tactile sensor.

The hand may include a plurality of fingers, and each of the plurality of fingers may include a distance sensor configured to sense a distance spaced from the first object, and when the first object is gripped or re-gripping, Based on the information obtained through the distance sensor, each of the plurality of fingers may move in close proximity to the first object at the same speed.

Meanwhile, in a method of performing a task by moving a plurality of objects, which is another aspect of the present invention for achieving the above technical problem, a hand that grips and moves a first object of the plurality of objects; Stage 1; A second step of determining, by a controller, whether the task cannot be performed; A third step of releasing, by the hand, the first object held by the hand when it is determined that the task cannot be performed; And a fourth step of the hand re-holding the first object to perform the task under the control of the controller.

In addition, in the second step, when it is determined that at least one of the direction and location of the first object is changed to perform the task, in the third and fourth steps, the controller releases the hand. And re-holding operation.

The task may be performed by moving the plurality of objects in a preset order, and in the second step, when it is determined that the first object is moved and cannot perform the task differently from the preset order. In the third and fourth steps, the controller may control the release and re-holding of the hand.

The task may be performed by moving the plurality of objects in a predetermined order, and in the third step, the controller may be configured to hold the first object in a direction and a position suitable for re-holding. It is possible to control to release one first object.

The method may further include rotating the first object released by the hand at a predetermined angle between the third and fourth steps. In the fourth step, the hand may include the rotated first object. Can be re-held.

In addition, the second to fourth steps may be repeatedly performed until the task is completed.

The method may further include: obtaining, by the camera, images of the plurality of objects between the first and second steps. Between the third and fourth steps, the controller may control the camera through the camera. And determining at least one of a changed direction and a position of the first object based on the acquired image. In the fourth step, the hand uses the at least one of the changed direction and the position together to perform the determination. The first object can be retried.

In addition, between the first and second steps, when the hand is in contact with the first object for the gripping, the tactile sensor causes the normal force, shear force and conductivity of the first object. and detecting at least one tactile factor of the conductive, in the fourth step, the hand re-holds the first object using at least one tactile factor detected by the tactile sensor. can do.

The present invention can provide a technique for performing a specific task by using a regrasping when the robot hand can perform a specific task, the currently grasping form can not be manipulated for a given purpose.

 In addition, according to the present invention, it is possible to detect not only normal force, but also shear force, which is widely used, and to develop a tactile sensor that can classify objects by measuring conductivity of an object and utilizing such tactile information as a human being. It is possible.

In addition, the present invention is a real-time position / posture / state recognition technology of parts using visual and tactile information, the optimal grip type reasoning intelligence technology for stable gripping of parts, intelligent gripping technology based on recognition information and experience (using a single gripper / hand) , More than 30 kinds of objects), position / direction manipulation of parts using visual and tactile information, and experience-based assembly strategy learning of various parts. It is possible to solve the current problem which is being performed by the manual process.

On the other hand, the effect obtained in the present invention is not limited to the above-mentioned effects, other effects that are not mentioned will be clearly understood by those skilled in the art from the following description. Could be.

1 is a block diagram illustrating the configuration of a robot hand system proposed by the present invention.
2 is a view for explaining the configuration of the robot hand proposed by the present invention.
Figure 3 illustrates a specific example of gripping an object in accordance with the vision, control and machine learning of the present invention.
4 is a block diagram illustrating a configuration of a robot hand system according to the present invention for gripping an object based on a vision system.
FIG. 5 is a flowchart illustrating a method of grasping an approximate position and shape of an object through a vision and attempting grasping, and learning a failed event according to an algorithm when a task fails.
6 is a block diagram illustrating a configuration of a robot hand system according to the present invention for gripping an object by additionally using a tactile sensor in addition to a vision system.
7A and 7B illustrate an example of a specific shape of a robot gripping using vision information and tactile information in connection with the present invention.
8 is a flowchart illustrating the operation of the robot gripping using vision information and tactile information in conjunction with the present invention.
Fig. 9 shows an example of the miniature model and the three-dimensional position fitting used for the gripping of the present invention.
Figure 10 illustrates in detail an example of applying to the gripping by fitting to the three-dimensional position according to the present invention.
FIG. 11 is a flowchart illustrating the overall operation of the robot gripping using vision information and tactile information in conjunction with the present invention.
12 illustrates an example of a flowchart illustrating object gripping using learning in connection with the present invention.
FIG. 13 illustrates an example in which a gripping method changes according to a change in a direction and a position of an object in relation to the present invention.
14 is a view for explaining the need for re-hold according to the present invention.
15 illustrates an example of a rephasing learning model according to the present invention.
16 is a view for explaining a method for performing re-gripping according to the gripping success probability of the present invention.
FIG. 17 is a view for explaining execution of a task to which re-pagment is applied in accordance with the present invention.
FIG. 18 is a flowchart illustrating a method of accomplishing a specific task by using regrasping when the present grasping form does not allow manipulation for a given purpose.

Prior to the detailed description of the present invention, the configuration of the robot hand system applied to the present invention will be described with reference to the drawings.

1 is a block diagram illustrating the configuration of a robot hand system proposed by the present invention.

Referring to FIG. 1, the robot hand system 100 may include a wireless communication unit 110, an audio / video input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, The memory 160, the interface unit 170, the controller 180, the power supply unit 190, and the robot hand 200 may be included.

However, since the components shown in FIG. 1 are not essential, a robot hand system having more or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the robot hand system and the wireless communication system or between the device and the network in which the device is located.

For example, the wireless communication unit 110 may include a mobile communication module 112, a wireless internet module 113, a short range communication module 114, a location information module 115, and the like.

The mobile communication module 112 transmits and receives a wireless signal with at least one of a base station, an external device, and a server on a mobile communication network.

It may include various types of data according to text and multimedia message transmission and reception.

The wireless internet module 113 refers to a module for wireless internet access and may be embedded or external to the robot hand system. Wireless Internet technologies may include Wireless LAN (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), and the like.

The short range communication module 114 refers to a module for short range communication. Short range communication technologies include Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Wireless Fidelity (Wi-Fi). Can be used.

The position information module 115 is a module for obtaining the position of the robot hand system, and a representative example thereof is a GPS (Global Position System) module.

Referring to FIG. 1, the A / V input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the photographing mode. The processed image frame may be displayed on the display unit 151.

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110.

Two or more cameras 121 may be provided according to the use environment.

The microphone 122 receives an external sound signal by a microphone in a recording mode or a voice recognition mode, and processes the external sound signal into electrical voice data. The processed voice data may be converted into a form transmittable to the mobile communication base station through the mobile communication module 112 and output. The microphone 122 may implement various noise removing algorithms for removing noise generated in the process of receiving an external sound signal.

The user input unit 130 generates input data for the user to control the operation of the robot hand system. The user input unit 130 may include a key pad dome switch, a touch pad (static pressure / capacitance), a jog wheel, a jog switch, and the like.

The sensing unit 140 detects the current state of the robot hand system such as the open / closed state of the robot hand system, the position of the robot hand system, the presence or absence of user contact, the orientation of the robot hand system, the acceleration / deceleration of the robot hand system, and the like. Generates a sensing signal for controlling the operation of.

The sensing unit 140 may sense whether the power supply unit 190 supplies power or whether the interface unit 170 is coupled to an external device.

On the other hand, the sensing unit 140 may include a proximity sensor (not shown).

The output unit 150 is used to generate an output related to visual, auditory or tactile senses, which includes a display unit 151, an audio output module 152, an alarm unit 153, a haptic module 154, and a projector module ( 155) may be included.

The display unit 151 displays (outputs) information processed in the robot hand system.

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display (flexible). and at least one of a 3D display.

Some of these displays can be configured to be transparent or light transmissive so that they can be seen from the outside. This may be referred to as a transparent display. A representative example of the transparent display is TOLED (Transparant OLED). The rear structure of the display unit 151 may also be configured as a light transmissive structure.

There may be two or more display units 151 according to the implementation form of the robot hand system. For example, in the robot hand system, a plurality of display units may be spaced apart or integrally disposed on one surface, or may be disposed on different surfaces, respectively.

When the display unit 151 and a sensor for detecting a touch operation (hereinafter, referred to as a touch sensor) form a mutual layer structure (hereinafter referred to as a touch screen), the display unit 151 may be configured in addition to an output device. Can also be used as an input device. The touch sensor may have, for example, a form of a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 151 or capacitance generated in a specific portion of the display unit 151 into an electrical input signal. The touch sensor may be configured to detect not only the position and area of the touch but also the pressure at the touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and then transmits the corresponding data to the controller 180. As a result, the controller 180 can know which area of the display unit 151 is touched.

The proximity sensor (not shown) may be disposed in the inner region of the robot hand system surrounded by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object present in the vicinity without using a mechanical contact by using an electromagnetic force or infrared rays. Proximity sensors have a longer life and higher utilization than touch sensors.

Examples of the proximity sensor include a transmission photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. When the touch screen is capacitive, the touch screen is configured to detect the proximity of the pointer by the change of the electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, the act of allowing the pointer to be recognized without being in contact with the touch screen so that the pointer is located on the touch screen is referred to as a "proximity touch", and the touch The act of actually touching the pointer on the screen is called "contact touch." The position where the proximity touch is performed by the pointer on the touch screen refers to a position where the pointer is perpendicular to the touch screen when the pointer is in proximity proximity.

The proximity sensor detects a proximity touch and a proximity touch pattern (for example, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, and a proximity touch movement state). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

In addition, the robot hand system 100 according to the present invention may include a gyro sensor 141.

The gyro sensor 141 is a sensor for measuring a change in azimuth of an object by using a property that always maintains a constant direction initially set with high accuracy regardless of the rotation of the earth. The gyroscope has an optical method using a mechanical method and light. have.

In addition, the robot hand system 100 according to the present invention may include an acceleration sensor 142.

The acceleration sensor 142 processes an output signal to measure dynamic force such as acceleration, vibration, and impact of an object.

The acceleration sensor 142 is classified into a detection method, and includes an inertial type, a gyro type, and a silicon semiconductor type. An accelerometer or an inclinometer may also be regarded as a kind of acceleration sensor.

In addition, the robot hand system 100 according to the present invention may include a pressure sensor 143.

The pressure sensor 143 is a device and an element which detect the pressure of a liquid or gas, and convert it into an electrical signal which is easy to use for measurement or control, and transmits it.

There are many kinds of measurement principles such as displacement, deformation and thermal conductivity of molecular density. Recently, a strain gauge type pressure sensor made of silicon has been developed and used for precise pressure measurement. Integrated pressure sensors have also been developed that build integrated circuits on the same substrate and even process signals.

In addition, the robot hand system 100 according to the present invention may include a tactile sensor 144.

Tactile sensor (144, Tactile sensor) is a pressure sensor that is intended to artificially realize human tactile sense in a robot, and is classified into a contact sensor, a pressure sensor, a slip sensor, and a temperature sensor, which are necessary to realize an advanced human tactile system. Technology.

The tactile sensor array is a sensor for obtaining two-dimensional information by arranging several to several dozen contact angle sensors 144 or pressure sensors in a planar shape, and can also be used for detection of shape or motion.

Many of these sensors form an array type sensor by dividing either one of a conductive rubber, a piezoelectric polymer, and electrodes on both sides of a reduced pressure polymer, and a two-dimensional pressure distribution is detected from a resistance change or a voltage output between the electrodes.

In particular, the tactile sensor 144 according to the present invention may detect at least one of normal force, shear force, and conductive of an object that the robot hand 200 contacts.

The sound output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a recording mode, a voice recognition mode, a broadcast receiving mode, and the like. The sound output module 152 may also output sound signals related to functions performed in the robot hand system. The sound output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying occurrence of an event of the robot hand system.

The alarm unit 153 may output a signal for notifying occurrence of an event in a form other than a video signal or an audio signal, for example, vibration.

The video signal or the audio signal may be output through the display unit 151 or the audio output module 152, so that they 151 and 152 may be classified as part of the alarm unit 153.

The haptic module 154 generates various haptic effects that a user can feel. Vibration is a representative example of the haptic effect generated by the haptic module 154. The intensity and pattern of vibration generated by the haptic module 154 can be controlled.

For example, different vibrations may be synthesized and output or may be sequentially output.

In addition to vibration, the haptic module 154 may be configured to provide a pin array that vertically moves with respect to the contact skin surface, a jetting force or suction force of air through an injection or inlet, grazing to the skin surface, contact with an electrode, electrostatic force, and the like. Various tactile effects can be generated, such as effects by the endothermic and the reproduction of a sense of cold using the elements capable of endotherm or heat generation.

The haptic module 154 may not only deliver the haptic effect through direct contact, but also may implement the user to feel the haptic effect through a muscle sense such as a finger or an arm. The haptic module 154 may be provided with two or more according to the configuration of the robot hand system.

The projector module 155 is a component for performing an image project function using a robot hand system, which is the same as or at least the image displayed on the display unit 151 according to a control signal of the controller 180. Some may display other images on an external screen or wall.

In detail, the projector module 155 may generate a light source (not shown) for generating light (for example, laser light) for outputting the image to the outside, and an image to be output to the outside using the light generated by the light source. And an image generating means (not shown), and a lens (not shown) for expanding and outputting the image to the outside at a predetermined focal length. In addition, the projector module 155 may include an apparatus (not shown) which may mechanically move the lens or the entire module to adjust the image projection direction.

The projector module 155 may be divided into a cathode ray tube (CRT) module, a liquid crystal display (LCD) module, a digital light processing (DLP) module, and the like, according to the device type of the display means. In particular, the DLP module may be advantageous in miniaturization of the projector module 151 by expanding and projecting an image generated by reflecting light generated from a light source to a digital micromirror device (DMD) chip.

Preferably, the projector module 155 may be provided in the longitudinal direction on the side, front or back of the robot hand system. Of course, the projector module 155 may be provided at any position of the robot hand system as necessary.

Meanwhile, the memory unit 160 may store a program for processing and controlling the controller 180 and may temporarily store input / output data (for example, a message, audio, still image, video, etc.). It can also perform a function. The memory unit 160 may also store the frequency of use of each of the data. In addition, the memory unit 160 may store data regarding vibration and sound of various patterns output when a touch input on the touch screen is performed.

The memory 160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM (Random Access Memory, RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic It may include a storage medium of at least one type of disk, optical disk. The robot hand system may operate in connection with a web storage that performs a storage function of the memory 160 on the Internet.

The interface unit 170 serves as a passage with all external devices connected to the robot hand system. The interface unit 170 receives data from an external device, receives power, transfers the power to each component inside the robot hand system, or transmits data inside the robot hand system to an external device. For example, wired / wireless headset ports, external charger ports, wired / wireless data ports, memory card ports, ports for connecting devices with identification modules, audio input / output (I / O) ports, The video input / output (I / O) port, the earphone port, and the like may be included in the interface unit 170.

The identification module is a chip that stores various types of information for authenticating the use rights of the robot hand system. The identification module (User Identify Module, UIM), Subscriber Identify Module (SIM), Universal Subscriber (Universal Subscriber) Identity Module, USIM) and the like. A device equipped with an identification module (hereinafter referred to as an 'identification device') may be manufactured in the form of a smart card. Therefore, the identification device can be connected to the robot hand system through the port.

The interface unit may be a passage through which power from the cradle is supplied to the robot hand system when the robot hand system is connected to an external cradle, or various command signals input from the cradle by a user are transmitted to the mobile device. It can be a passage. Various command signals or power input from the cradle may be operated as signals for recognizing that the mobile device is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the robot hand system.

The power supply unit 190 receives an external power source and an internal power source under the control of the controller 180 to supply power for operation of each component.

Various embodiments described herein may be implemented in a recording medium readable by a computer or similar device using, for example, software, hardware or a combination thereof.

According to a hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and the like. It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. The described embodiments may be implemented by the controller 180 itself.

According to the software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code may be implemented in software applications written in a suitable programming language. The software code may be stored in the memory 160 and executed by the controller 180.

In addition, the system 100 according to the present invention may include a robot hand 200.

The robot hand 200 largely includes a palm 210 and a finger 200.

The finger part 200 may be configured to have at least one node, and may move at the same speed with respect to an object through a distance sensor (not shown).

The object according to the present invention may be a target of an object in a state where no information exists at all, an object in a state in which information received from the outside exists, an object in a state in which information has accumulated information through a failure event, and the like. .

The state in which the information about the object according to the present invention includes not only a state in which complete information on the object is constructed but also a state in which information is accumulated to some extent through failure events and capabilities.

On the other hand, the robot hand 200 may be provided in the system 100 in plurality, and performing a task having a specific order and coupling direction as well as a simple gripping operation through the plurality of robot hands 200. It is possible.

Based on the configuration of the system 100 of the present invention described above will be described with reference to the drawings the specific technology to be proposed in the present specification.

Embodiment 1-Technique for Handling Unknown Objects Using a Visual Sensor

Most of the work performed in the actual manufacturing environment based on the conventional robot hand is a handling work for holding and assembling parts, and most of the handling work is performed by manual processes except for simple repetitive work in a standardized environment. There was a problem.

Accordingly, the present invention aims to solve the above problems and to propose a gripping / assembly technology equipped with artificial intelligence in order to increase production efficiency in response to changes in the manufacturing environment.

That is, the present invention attempts grasping by grasping an approximate position, shape, and the like of an object through a vision based on the configuration of the system 100 described above. In addition, it provides a handling technique that attempts regrasping, and additionally, provides a technique that increases the efficiency of the learning algorithm by using a tactile sensor.

In addition, the present invention, by developing a conventional model-free method and model-based method for the development of gripping and manipulation method through the conventional machine learning to overcome the weak points such as long learning time, there is a model uncertainty Its purpose is to provide handling techniques that can be used in situations.

2 is a view for explaining the configuration of the robot hand proposed by the present invention.

Referring to FIG. 2, a robot hand 200 for gripping an object 10 is shown.

In addition, a plurality of cameras 121 are arranged in the system 1 according to the invention.

Herein, the controller 180 may control the robot hand 200 to grasp the object 10 in a situation where there is no information about the object 10.

That is, in the present invention, since there is no acquired information or learned information about the object, based on the vision system through the plurality of cameras 121, the approximate position and shape of the object are detected, and the robot is based on the detected information. The object 10 may be gripped by controlling the hand 200.

Specifically, in this case, the controller 180 determines at least one of the coordinates, the size, and the center of the object 10 by using the acquired image, and the robot hand 200 is the object 10 immediately based on the identified information. Can be controlled to hold.

In this case, the controller 180 may allow the robot hand 200 to grip the object 10 based on power grasping rather than precise grasping.

Since the robot hand 200 does not grip the object 10 based on the correct information, an event for releasing the object 10 may occur after failing or holding the object 10. .

In this case, the present invention intends to apply a machine learning algorithm for a failed event.

That is, when the robot hand 200 fails to grip the object 10, the controller 180 may determine at least one of the coordinates, the size, and the center of the object 200 again based on the failure event.

The robot hand 200 then attempts to grip the object 10 again using the determined factor.

Such machine learning algorithms may be repeatedly performed when the object 10 cannot be grasped, and ultimately, using much less attempts, calculation amounts, and time than grasping the object 10 after obtaining specific information. To hold the object 10.

Figure 3 illustrates a specific example of gripping an object in accordance with the vision, control and machine learning of the present invention.

Referring to FIG. 3, the robot hand 200 according to the present invention includes one palm 210 and three fingers 220, and each finger 220 includes a plurality of nodes.

Referring to FIG. 3A, the robot hand 200 according to the present invention may grip the object 10 by spreading and pinching a plurality of finger parts 220 based on the palm part 210. have.

That is, based on the image acquired through the camera 121, the controller 180 determines at least one of the coordinates, the size, and the center of the object 10, and the robot hand 200 immediately based on the identified information. Control to grip the object (10).

In this case, when an event for releasing the object 10 occurs after the robot hand 200 does not hold or grip the object 10, the controller 180 coordinates the object 200 based on the failure event. Determine at least one of the size and the center, and the robot hand 200 attempts to grasp the object 10 again using the determined factor.

According to the machine learning algorithm, the appearance of the robot hand 200 holding the object 10 may be changed.

That is, as shown in (a) of FIG. 3, an attempt may be made to grab from the right side of the object 10, and an attempt may be made to grab from the left side as shown in (b), and (c) As shown, it is possible to grip the other finger portion 220 on the left side while supporting the lower end of the object 10.

4 is a block diagram illustrating a configuration of a robot hand system according to the present invention for gripping an object based on a vision system.

Referring to FIG. 4, the tactile sensor 144 may be additionally used in addition to the configuration using the vision system of the system 100, which will be described below in detail with reference to the accompanying drawings.

In addition, in gripping the object 10, the controller 180 may determine grasping safety of the unknown object 10 currently held by using vision information (which may be used together with tactile sensor information).

Specifically, the controller 180 can check the safety by measuring the force / torque required as a whole while moving the holding object 10.

In addition, the controller 180 may estimate the force / torque required in the current grasping mode in consideration of the expected physical properties of the object 10.

In this case, when it is determined that grasping safety of the robot hand 200 is lower than a preset value, the controller 180 may control the robot hand 200 to release the object 10 and re-grip based on the learned information. It may be.

It is possible to identify the optimal condition with vision information and tactile information for regrasping and to grab objects for the purpose of grasping and manipulating objects during regrasing process.

FIG. 5 is a flowchart illustrating a method of grasping an approximate position and shape of an object through a vision and attempting grasping, and learning a failed event according to an algorithm when a task fails.

Referring to FIG. 5, first, an operation (S10) of acquiring an image of at least one first object 10 by the camera 121 is performed.

In the present invention, there is no prior information on the first object 10 and attempts to grasp the first object 10 based only on information obtained through the camera 121.

Thereafter, the controller 180 determines at least one of the coordinates, the size, and the center of the first object 10 by using the acquired image (S11).

In operation S11, when the control unit 180 does not acquire all the information about the coordinates, the size, and the center through the image, but determines at least one factor of the coordinates, the size, and the center, the hand 180 uses the determined factor. 200 immediately controls to hold the first object 10 (S12).

In operation S12, since the robot hand 200 does not hold accurate information, the robot hand 200 may fail to grasp the first object 10 (S13).

For example, the first object 10 does not exist in the place where the robot hand 200 is moved or the degree that the finger 220 of the robot hand 200 is opened is smaller than the width of the first object 10 or An event that cannot hold the first object 10 may occur due to an element such as the robot hand 200 moving toward the center of the first object 10 and another center.

At this time, the controller 180 determines again at least one of the coordinates, the size, and the center of the first object based on the failure event (S14), and the robot hand 200 uses the determined factor again to determine the first object 10. It is to hold again (S15).

Until the object 10 is successfully grabbed, steps S13 to S15 according to the present invention may be repeatedly performed.

In addition, even when the gripping of the object 10 is successful, as described above, when the gripping is determined to be unstable through the stability test, the gripping of the object 10 may be released and re-gripping.

Therefore, according to the present invention, by developing both the proposed model-free method and the model-based method for developing the gripping and operating method through machine learning, it is possible to overcome the weak points such as the long learning time and to use it even in the situation of model uncertainty. Can be complementary developments.

Second Embodiment Techniques for Handling Objects Using a Camera and a Tactile Sensor Together

In the present invention, in order to increase the efficiency of the machine learning described in the first embodiment, the technique using the tactile sensor 144 in addition to the camera 121, the robot hand 200 is to propose a technique to try to grip.

That is, the present invention overcomes weaknesses such as long learning time by developing the existing model-free method and the model-based method for developing the gripping and operating method through the conventional machine learning, and has a model uncertainty. Its purpose is to provide handling techniques that can be used in situations.

In addition, in the second embodiment, by using the tactile sensor 144 additionally, as well as the conventional normal force measurement, as well as measuring the shear force, the conductivity of the target object and the like to provide a handling technology that can classify and learn the object. For the purpose of

6 is a block diagram illustrating a configuration of a robot hand system according to the present invention for gripping an object by additionally using a tactile sensor in addition to a vision system.

The configuration shown in FIG. 6 is the same as that shown in FIG. 2, but additionally uses a tactile sensor 144.

The tactile sensor 144 according to the present invention may sense at least one of normal force, shear force, and conductive of the object 10 held by the robot hand 200.

Referring to FIG. 6, a plurality of cameras 121 are disposed in the system 1 according to the present invention, and a plurality of tactile sensors 144 may be provided in at least a part of the finger part 220 of the robot hand 200. Can be.

In the present invention, since there is no acquired information or learned information about the object, based on the vision system through the plurality of cameras 121, the approximate position and shape of the object are detected, and the robot hand is based on the detected information. The object 10 may be gripped by controlling the 200. In detail, the controller 180 determines at least one of the coordinates, the size, and the center of the object 10 by using the acquired image, and determines the identified information. Based on this, the robot hand 200 may directly control the gripping of the object 10.

Since the robot hand 200 does not grip the object 10 based on accurate information, an event for releasing the object 10 may occur after failing or holding the object 10. In the present invention, the machine learning algorithm for the failed event is applied.

That is, when the robot hand 200 fails to grip the object 10, the controller 180 may determine at least one of the coordinates, the size, and the center of the object 200 again based on the failure event.

In this case, unlike in FIG. 2, in FIG. 6, when the finger part 220 of the robot hand 200 contacts the object 10 for gripping, the object 10 obtained through the tactile sensor 144 is obtained. At least one of the normal force, the shear force and the conductive (conductive) of is additionally utilized for re-holding.

That is, since the robot hand 200 grips the object 10 again using the determined factor and at least one tactile factor detected by the tactile sensor 144, the robot hand 200 uses much less trial and calculation amount than when using only the vision system. Using the time, the object 10 can be gripped.

7A and 7B illustrate an example of a specific shape of a robot gripping using vision information and tactile information in connection with the present invention.

Referring to FIG. 7A (a), the robot hand 200 according to the present invention includes one palm 210 and two fingers 220, and each finger 220 has a plurality of nodes. It is composed.

Robot hand 200 according to the present invention, as shown in (a) to (d) of Figure 7a, the object through the operation of unfolding and pinching the plurality of fingers 220 relative to the palm portion 210 10) can be gripped.

That is, based on the image acquired through the camera 121, the controller 180 determines at least one of the coordinates, the size, and the center of the object 10, and the robot hand 200 immediately based on the identified information. Control to grip the object (10).

In this case, when an event for releasing the object 10 occurs after the robot hand 200 does not hold or grip the object 10 in FIG. 7A, the controller 180 based on the failure event. Re-determining at least one of the coordinates, the size, and the center of the furnace object 200, the robot hand 200 attempts to grasp the object 10 again using the determined factor. When the finger 220 is in contact with the object 10 for holding, the normal force, shear force, or conductive force of the object 10 obtained through the tactile sensor 144 is At least one is further utilized for re- grasping.

According to such a machine learning algorithm, the gripping of the object 10 by the robot hand 200 may be used to perform a specific task in addition to the simple gripping.

That is, as shown in FIG. 7B, two robot hands 200 are provided, the first robot hand 200a includes a first palm portion 210a and a first finger portion 220a, and a second The robot hand 200b may include a first palm portion 210b and a first finger portion 220b.

In addition, in FIG. 7B, the first robot hand 200a and the second robot hand 200b are used to perform the operation of inserting the second object 10b into the first object 10a.

Since the operation of inserting the second object 10b into the first object 10a is not a simple gripping operation, the two robot hands 200 may continuously fail to grasp the above-described vision system. The learning algorithm accordingly is used repeatedly.

Furthermore, at least one of normal force, shear force, and conductive of the object 10 obtained through the tactile sensor 144 is further utilized for re-holding.

For example, as in the case of manipulating the water bottle 10 containing only half of water, the tactile information may be used to grasping and manipulate the object while adjusting the force according to the disturbance of the object (the effect of changing the center of gravity).

In addition, in the present invention, the shape of the hand 200 holding the object 10 may be calculated and applied in advance.

In addition, as described above, in order to perform a specific task, an algorithm for changing a method, a direction, or the like of the object 10 may be applied to the purpose of grasping and manipulating the object 10.

The tactile information used in the second embodiment includes information such as normal force, shear force, electrical conductivity, and the like, and information that can be caught in contact with an external environment (bottom surface, etc.) when the small object 10 is held. Can also be obtained additionally.

Therefore, by using vision information and tactile information together, precise assembly (including shear force) can be measured, enabling precise assembly work.

8 is a flowchart illustrating the operation of the robot gripping using vision information and tactile information in conjunction with the present invention.

Referring to FIG. 8, first, the camera 121 acquires an image of at least one first object 10 (S20).

Thereafter, the controller 180 determines at least one factor of the coordinates, the size, and the center of the first object by using the acquired image (S21), and the hand 200 uses the determined factor to determine the first factor. The object 10 is gripped (S22).

At this time, unlike the method in FIG. 5, in the second embodiment, when the hand 200 is in contact with the first object 10 for the gripping, the normal force and the shear force of the first object 10. (S22) in which the tactile sensor 144 detects at least one of shear force and conductive.

After step S22, when the robot hand 200 fails to grasp the first object 10 due to insufficient information (S23), the controller 180 controls the coordinates, the size, and the center of the first object based on the failure event. At least one of the factors is determined again (S24).

Based on the step S24, when the hand 200 intends to grip the first object again, unlike the method of FIG. 5, in the second embodiment, the hand 200 uses the determined factor and at least one tactile factor detected by the tactile sensor. The first object is gripped again (S25).

Therefore, according to the present invention, it is possible to detect not only normal force, but also shear force, which is widely used, and develop a tactile sensor that can classify objects by measuring conductivity of an object and use such tactile information as a human being. Do.

In addition, real-time position / posture / state recognition technology of parts using visual and tactile information, optimal grip type inference intelligence technology for stable gripping of parts, intelligent gripping technology based on recognition information and experience (using a single gripper / hand, 30 types) Most of the handling work except the simple repetitive work through the process of position / direction manipulation (In-Hand) of parts using abnormal objects), visual and tactile information, and learning of assembly strategy of various parts based on experience. It is possible to solve the current problem being performed by.

Third Embodiment-Method of Grasping an Object More Accurately Based on Learned Information

In the third embodiment according to the present invention, if the information on the object 10 is learned based on the above-described first and second embodiments, an additional method for faster and more accurate grip of the object 10 can be obtained. Suggest.

Further methods applied in the third embodiment are as follows.

1) How to solve the difference between the pre-stored reference picture and the input picture through an error function (error function)

2) A method of determining a small-size model by determining a two-dimensional point in determining the grabbing area, and fitting it to a three-dimensional position and applying it to a phage.

First, a method of resolving a difference between a previously stored reference picture and an input picture through an error function will be described.

In the first method, the system 100 may store information related to the object 10 in advance through the memory 160 or receive the information from the outside through the wireless communication unit 110.

In addition, while attempting to grip the object 10 through the camera 121 or the tactile sensor 144, information about the object 10 may be obtained.

In this case, the controller 180 compares the information about the object 10 previously stored or received from the outside with information obtained through the camera 121 to determine the degree of error, and uses the previously stored information or obtained How to modify the information can be performed.

In this case, the controller 180 compares the information about the object 10 previously stored or received from the outside with the information obtained through the camera 121 and the like is as follows.

The cover error is ε cover (u, v, x, y) and the range error is range (u, v, x, y, z). These error terms are evaluated for each pixel in coordinates (u, v) of the input range image. The pixel translation value (x, y, z) of the reference range image determines its position with respect to the input range image.

The function is summed across all image pixels u, v using the weight λ (e.g. λ = 10). Normalization coefficients Ncover and Nrange make the error independent of object and image size. The error may be minimal if the picture R is probably aligned with a partially shielded object in the input image I.

Next, a method of determining a small-size model by determining a two-dimensional point in determining a grabbing area, fitting it to a three-dimensional position, and applying it to the gripping will be described.

FIG. 9 illustrates an example of a miniature model and three-dimensional position fitting used in the gripping of the present invention, and FIG. 10 specifically illustrates an example of fitting to a three-dimensional position in accordance with the present invention.

Referring to FIG. 9, the controller 180 receives a designation of a "recognition area" and a range to be performed in the captured image through the input device. Here, the "recognition area" is an area set by the control unit 180 in the photographed image in order to designate a portion to be gripped by the hand 200. The recognition area is DT that can be used as the area for measurement of distance.

Referring to FIG. 9A, recognition regions 11, 12, and 13 representing regions to be gripped by the controller 180 in the object 10 are illustrated.

The recognition area 11 designates the body part of the cup, the recognition area 12 designates the edge portion of the cup opening, and the recognition area 13 designates the handle part.

In addition, as illustrated in FIG. 9B, an icon list 20 representing a two-dimensional figure associated with the corresponding recognition area 13 may be further displayed on the display unit 151.

Icons 21 to 24 shown in FIG. 9B each show an example of a small model.

That is, icon 41 is a square cylinder model, icon 42 is a flat model, icon 43 is a cylinder model, and icon 43 is a truncated cone model.

Each miniature model has shape parameters to define shape and size, and placement parameters to define position and attitude.

In addition, the type of the miniature model is not limited to that shown in FIG. 9B, and of course, various shapes such as a secondary ellipsoid, an L-shaped footnote, a C-shaped cylinder, and the like can be further used.

In addition, referring to FIG. 9C, it is a conceptual diagram showing acquisition of three-dimensional position data 30 in the work space corresponding to the recognition areas 11, 12, 13.

The three-dimensional position data 30 in FIG. 9C shows the depth seen from the robot 200 in the working space together with the recognition area.

In addition, referring to FIG. 9 (d), a small-size model (specifically, a cylinder model, 40) is overlaid and shows a fitting result when the user selects the cylinder model.

10, the data structure specific example and the data content regarding the holding pattern applicable to the cylindrical model 40 are shown.

If the types of the robot hand 200 are different, the possible gripping patterns may be different and the gripping patterns applicable to the same small model may also be different.

That is, in FIG. 10, the hand 200 is a grip pattern applicable when the hand type is "flat three joint two finger hand" and the small model is a cylindrical model, and four grip patterns are recorded together with the application conditions.

Specifically, the four gripping patterns may be used as a side scissors gripping, a cross-section scissors gripping, a gripping containing grip and an edge scissor gripping.

Therefore, in the present invention, the controller 180 determines the 2D model corresponding to the point to be gripped by the robot hand 200 based on the acquired image, and determines the 2D model as a 3D model through depth information. The robot hand 200 may grip the object 10 by determining a posture optimized for grip of the determined 3D model.

Through this, the robot hand 200 can quickly grip the object 10 without obtaining additional information based on the obtained information.

4th Example  - vision  How to perform a task by holding on the basis of information and tactile information

In addition, in the present invention, a method for the robot hand 200 gripping the object 10 and performing a specific task will be described based on the first to third embodiments described above.

FIG. 11 is a flowchart illustrating the overall operation of the robot gripping using vision information and tactile information in connection with the present invention, and FIG. 12 is a flowchart illustrating an object gripping using learning in connection with the present invention. It is.

Referring to FIG. 11, based on the information obtained from the tactile sensor 144 and the non-metallic sensor 145 and the information obtained through the camera 122, the controller 180 may display properties such as physical properties of the object 10. It predicts and attempts to grasp the object 10 based on this.

At this time, the gripping on the object may fail, and the controller 180 adjusts a factor for re-gripping based on the information obtained through the camera 122 whenever a failure event occurs, and the tactile sensor 144 The central tactile sensation process (S30) is performed based on the information obtained through the information.

That is, in step S30, the control signal is fed back, pressure and roughness information are provided, or the physical property information about the object 10 is transmitted to the controller 180.

At this time, the controller 180 learns the information about the object 10 based on the information obtained through the camera and the information obtained through the step S30 (S31), and based on the learned information to the object 10 Resetting the holding condition for (S32), it is possible to repeatedly hold the object 10 based on the reflected reset condition to perform a specific task.

Referring to FIG. 12, a process 1100 of learning based on information obtained through the tactile sensor 144 and a process 1000 of learning based on information obtained through the camera 122 are illustrated. A process 1200 of performing a gripping operation by the robot hand based on the input information and the tactile input information is shown.

Fifth Embodiment-Method of Performing Task Through Re-Gripping Operation

On the other hand, in order for the robot hand 200 to perform a specific task, it may be necessary to use regrasping when the grasping form does not allow manipulation for a given purpose.

Therefore, in the present invention, in order to perform a specific task (robot hand), the present invention is to propose a technique that can accomplish a specific task using regrasping when the grasping form can not perform the operation for a given purpose. It is done.

That is, the present invention, when the two-arm robot hand 200 assembling the object, when the assembly operation can not be made in the state of holding the current object, the optimum algorithm required to re-assembly so that the assembly operation is possible to place one or both objects The purpose is to propose.

In addition, the present invention provides a handling technique for classifying and learning an object by measuring shear force, conductivity of an object, as well as normal force measurement, which is conventionally utilized by additionally using the tactile sensor 144 in re-holding operation. It aims to provide.

FIG. 13 illustrates an example in which the gripping method changes according to a change in the direction and position of an object in relation to the present invention.

Referring to FIG. 13A, the object 10 may be changed in position or direction by an external force or an operation to be gripped.

In addition, although a plurality of objects 10 exist, and in order to perform a specific task, the order in which the plurality of objects 10 are combined is determined, an event for holding the object 10 that does not correspond to the order May occur.

In this case, as illustrated in (b) of FIG. 13, the gripping of the object 10 should be performed in consideration of the change in the direction and position of the changed object 10.

In addition, as shown in FIG. 13B, a plurality of objects 10 exist, and in order to perform a specific task, the order in which the plurality of objects 10 are combined is determined. When an event for grasping the object 10 that does not correspond to the event occurs, a method of releasing the grasped object 10 and grasping the object 10 should be applied.

Also, in re-holding, a method of gripping the point of the object 10 optimized for the next process or the number object 10 whose position, direction, etc. of the object 10 is changed to be suitable for the next process may be used. Can be.

14 is a view for explaining the need for re-hold according to the present invention.

In FIG. 14, a plurality of objects 10 exist, and an object thereof is to perform a specific task of assembling the plurality of objects 10.

In this case, the controller 180 may store information about the order of assembling the plurality of objects 10 in advance or may be instructed from the outside through the communication unit 110 to perform a specific task.

As shown in (a) of FIG. 14, the object 10 may be completed when the first object 10a, the second object 10b, and the third object 10c are combined in turn. After the object 10a and the second object 10b are combined, the corresponding task may be performed only when the third object 10c is combined with the second object 10b.

However, at this time, as shown in FIG. 14B, an event in which the plurality of robot hands 200 grips and combines the second object 10b and the third object 10c first may be generated. Can be.

As shown in FIG. 14B, the second object 10b and the third object 10c are combined and positioned at the left side as shown in FIG. 14C, and the first object 10a is illustrated in FIG. 14B. ) May be located on the right side.

At this time, as shown in (d) of FIG. 14, an event in which the first object 10a is combined at the bottom of the form in which the second object 10b and the third object 10c are combined as shown in FIG. A problem arises in which (10) cannot be assembled.

Therefore, in this case, the task may be performed only after the first object 10a and the second object 10b are combined with the third object 10c and the second object 10b in the predetermined order. This can be done by rephasing.

That is, in FIG. 14B, two robotic hands 200 must grasp the first object 10a and the second object 10b, and the second object 10b and the third object 10c are gripped. When the event occurs, the robot hand 200 may release the gripped objects and allow the robot to grasp the correct object again.

In addition, in the present invention, when an invalid event occurs, the retargeting may be performed by changing the object target and the order to be gripped in response to the corresponding event.

That is, when an incorrect event in which the second object 10b and the third object 10c are held and combined in FIG. 14C occurs, the second object 10b and the third object 10c in response to the event. ) Releases the combined structure, rotates the combined structure to the left or right, or moves and rotates the first object 10a to the left side, and then performs a re-holding operation of combining the combined structure and the first object 10a. This allows you to perform specific tasks.

15 illustrates an example of a rephasing learning model according to the present invention.

Referring to FIG. 15, in ①, the robot hand 200 succeeds in holding the object 10 vertically.

However, the controller 180 may determine that the shape of ① is not preferable in performing a specific task, and may release the object 10 in the gripping operation such as ② and ③.

In addition, since the controller 180 knows the following process for completing a specific task, the controller 180 may change a position and a direction that are disposed when the object 10 is released in the release operation of ② and ③.

Subsequently, under the control of the controller 180, the robot hand 200 performs a ④ operation of gripping and lifting the object 10 at a predetermined angle from the upper left corner.

In the performing of the operation, the operation of gripping using the information acquired through the tactile sensor 144 and the information obtained through the camera 122 is performed based on the operation.

Thereafter, the operation ⑥ of moving the gripped object 10 through the operation ⑤ according to a predetermined order and process may be performed.

Also, FIG. 16 is a diagram for describing a re-gripping method according to a gripping success probability according to the present invention.

Referring to FIG. 16A, the Kernel density estimation method may be used in the present invention.

That is, based on the success rate of grasp P (grasped | s, a) after the re-gripping operation, it is determined whether the grasp operation has a high success rate, and the object state probability P (s_new | s, a, grasped ) To determine the center of the object can be used to determine what kind of motion is good.

As shown in (b) of FIG. 16, in re-holding the object 10, a Kernel density estimation method is used, and based on P (grasped | s, a), which is a probability of success after re-holding, Whether the gripping operation has a high success rate, and as shown in (c) of FIG. It can be used to determine whether the operation is good.

FIG. 17 is a view for explaining execution of a task to which re-pagment is applied in accordance with the present invention. FIG.

FIG. 17A illustrates a process of releasing and gripping immediately when an event deviating from a predetermined process occurs while moving a plurality of objects 10 according to a predetermined process to perform a specific task. The process is illustrated.

In contrast, FIG. 17B illustrates a specific task performed in a state where an event deviating from the predetermined process occurs in a process of moving a plurality of objects 10 according to a predetermined process to perform a specific task. The control unit 180 newly determines a process most suitable for the following, and accordingly, a process of performing the re-holding of the hibernation and gripping is shown.

In the present invention, all of the contents shown in FIGS. 17A and 17B may be applied.

In addition, FIG. 18 is a flowchart illustrating a method of accomplishing a specific task using regrasping when the present grasping form does not allow manipulation for a given purpose.

Referring to FIG. 18, the camera 122 acquires images of the plurality of objects 10 (S40).

Subsequently, in operation S41, the robot hand 200 grips and moves a first object of the plurality of objects based on a preset order based on the control of the controller 180.

Thereafter, the controller 180 determines whether the task cannot be performed (S41).

In this case, an event in which a task cannot be performed cannot be carried out according to a predetermined process because at least one of the position and the orientation of the first object has been changed and the first object cannot be grasped as desired, or an incorrect grip and combination has already been performed. And the like.

If it is determined in step S41 that the task cannot be performed, the controller 180 controls the robot hand 200 to release the gripped first object (S42).

After the step S42, the controller 180 grips the first object according to the original plan or newly establishes a plan suitable for an incorrectly progressed process, and based on the image acquired through the camera to grasp the first object, At least one of the changed direction and position is determined (S43).

In addition, the robot hand 200 proceeds to step S44 of re-holding the first object based on the changed direction and position.

The controller 180 may control the steps S41 to S44 to be repeatedly learned and performed until a specific task is performed.

The present invention is a regrasping when assembling, knowing in advance information about the shape of the movement and the order of assembly, so that the following process is known, and the object is released from the gripping so as to have a good arrangement for the next work, and the re-greasing operation Can be.

In addition, in the present invention, when an event occurs in which the object 10 cannot be gripped due to the expected shape, the corresponding part of the object 10 is rotated through the robot hand 200, and the visual information is obtained again, and then re-gripping. You can also perform an operation.

In addition, the present invention generates a geometric feature reflection file (assembly direction, size, etc.) for the assembly of each component based on the obtained information, permutation, and secure the primary assembly sequence through the combination of all components After that, it can be confirmed whether all the holes of each part to be assembled are used. In addition, the assembly sequence can be searched and the problem solved by the constraint satisfaction problem (confirmation of inter-component interference in the assembly direction during assembly).

Therefore, the present invention can provide a technique for performing a specific task by using a regrasping when the robot hand can perform a specific task, the currently grasping form can not be manipulated for a given purpose.

In addition, according to the present invention, it is possible to detect not only normal force, but also shear force, which is widely used, and develop a tactile sensor that can classify objects by measuring conductivity of an object and utilize such tactile information as a human being. Can be.

In addition, real-time position / posture / state recognition technology of parts using visual and tactile information, optimal grip type inference intelligence technology for stable gripping of parts, intelligent gripping technology based on recognition information and experience (using a single gripper / hand, 30 types) Most of the handling work except the simple repetitive work through the process of position / direction manipulation (In-Hand) of parts using abnormal objects), visual and tactile information, and learning of assembly strategy of various parts based on experience. It is possible to solve the current problem being performed by.

Meanwhile, the above-described embodiments of the present invention may be implemented through various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of a hardware implementation, a method according to embodiments of the present invention may include one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), and Programmable Logic Devices (PLDs). It may be implemented by field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, the method according to the embodiments of the present invention may be implemented in the form of a module, a procedure, or a function that performs the functions or operations described above. The software code may be stored in a memory unit and driven by a processor. The memory unit may be located inside or outside the processor, and may exchange data with the processor by various known means.

The detailed description of the preferred embodiments of the invention disclosed as described above is provided to enable those skilled in the art to implement and practice the invention. Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the present invention. For example, those skilled in the art can use each of the configurations described in the above-described embodiments in combination with each other. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The invention can be embodied in other specific forms without departing from the spirit and essential features of the invention. Accordingly, the above detailed description should not be interpreted as limiting in all aspects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention. The present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In addition, the claims may be combined to form an embodiment by combining claims that do not have an explicit citation relationship or may be incorporated as new claims by post-application correction.

Claims (17)

In the robot hand that performs a task by moving a plurality of objects,
A hand for gripping and moving a first object of the plurality of objects; And
And a controller configured to determine whether the task cannot be performed.
If it is determined that the task can not be performed,

The control unit,
Control the hand to release the gripped first object,
Compare the information about the first object previously stored or received from the outside with the image acquired by the camera to determine an error degree, and determine at least one of a changed direction and a position of the first object based on the acquired image; ,
Control the hand to re-grasp the first object so as to perform the task;

After it is determined that the hand cannot perform the task after the first re-holding attempt,
The control unit,
Determine the error between the stored image and the input image by using the learned information, correct the obtained information, and determine a two-dimensional model corresponding to the recognition region to be gripped in the region of the first object,
Converting the two-dimensional model into a three-dimensional model to determine a holding posture for the hand to hold the three-dimensional model,
And the hand controls to re-grip the first object based on the determined factor and the determined gripping pose to perform the task.
The method of claim 1,
When it is determined that at least one of the direction and the position of the first object is changed, the task cannot be performed.
And the controller controls the release and re-holding operation of the hand. The method of claim 1,
The task may be performed by moving the plurality of objects in a preset order.
If it is determined that the first object is moved out of the preset order and cannot perform the task,
And the controller controls the release and re-holding operation of the hand. The method of claim 1,
The task may be performed by moving the plurality of objects in a preset order.
The control unit,
And the hand controls to release the first object held by the hand so that the first object is disposed in a direction and a position suitable for re-holding. The method of claim 4, wherein
The control unit,
Control the hand to rotate the released first object at an angle;
And the hand controls to re-hold the rotated first object. delete delete The method of claim 1,
A tactile sensor that senses at least one tactile factor of normal force, shear force, and conductive when the hand is in contact with the first object for holding; Including more,
And the hand controls to re-grip the first object using at least one tactile factor detected by the tactile sensor. The method of claim 1,
The hand comprises a plurality of fingers,
Each of the plurality of fingers includes a distance sensor for sensing a distance spaced from the first object,
When holding or re-holding the first object,
Based on the information obtained through the plurality of distance sensors, each of the plurality of fingers move closer to the first object at the same speed. In the method of performing a task by moving a plurality of objects,
A first step of the hand grasping and moving a first object of the plurality of objects;
A second step of determining, by a controller, whether the task cannot be performed;
A third step of releasing, by the hand, the first object held by the hand when it is determined that the task cannot be performed;
The controller may compare the information about the first object previously stored or received from the outside with an image acquired by the camera to determine an error degree, and at least one of a changed direction and a position of the first object based on the acquired image. A fourth step of determining one;
A fifth step of the hand re-holding the first object to perform the task under the control of the controller;
A sixth step after the first re-grasping attempt, determining that the hand cannot perform the task;
The information obtained by determining the error between the stored image and the input image is corrected using the learned information, and a two-dimensional model corresponding to a recognition region to be gripped in the region of the first object is determined, Converting to a 3D model to determine a holding position for the hand to hold the 3D model; And
An eighth step of the hand re-holding the first object based on the determined factor and the determined gripping pose to perform the task.
The method of claim 10,
In the second step, when it is determined that at least one of the direction and location of the first object is changed to perform the task,
And in the third to fifth steps, the control unit controls the release and re-holding operation of the hand. The method of claim 10,
The task may be performed by moving the plurality of objects in a preset order.
In the second step, if it is determined that the first object is moved out of the predetermined order and cannot perform the task,
And in the third to fifth steps, the control unit controls the release and re-holding operation of the hand. The method of claim 10,
The task may be performed by moving the plurality of objects in a preset order.
And wherein in the third step, the controller controls the hand to release the first object held by the hand so that the first object is disposed in a direction and a position suitable for re-holding. The method of claim 13,
Between the third to fifth steps,
And rotating the released first object by the hand at an angle.
And in the fifth step, the hand re-holds the rotated first object. delete delete The method of claim 10,
Between the first and second steps,
When the hand is in contact with the first object for the gripping, the tactile sensor detecting at least one tactile factor of normal force, shear force, and conductive of the first object More;
In the fifth step,
And the hand re-holds the first object using at least one tactile factor detected by the tactile sensor.

Images