CN113942009B - Robot bionic hand grabbing method - Google Patents

Abstract

The present disclosure provides a method for grabbing a robot bionic hand, including: acquiring an image of a target object; analyzing the image of the target object to obtain visual information of the target object; based on the visual information of the target object, determining the visual grasping parameters of the robot bionic hand; The visual grasping parameters of the robot bionic hand are used to grasp the target object; the tactile information of the target object and the grasping result are obtained; the soft and hard attribute data of the target object are obtained based on the tactile information of the target object and the visual information of the target object; and based on The soft and hard attribute data of the target object, adjust the tactile grasping parameters of the robot bionic hand and continue to grasp.

Description

Robot bionic hand grasping method

technical field

The disclosure relates to a grabbing method of a robot bionic hand, which belongs to the technical field of robots.

Background technique

With the continuous development of robot technology, robots have been used in various fields to replace people to complete complex tasks.

When the robot is used in different fields, its work content is different, and different grippers need to be customized for the robot, and the traditional robot grippers are based on manual teaching or visual guidance, which cannot cope with objects of different materials, and are easy to Causes damage to soft material objects.

How to plan the correct operation action according to the softness and hardness of the object, so as to ensure that the operated object will not fall or be damaged, is the current difficulty in the field of robotics.

Contents of the invention

In order to solve one of the above technical problems, the present disclosure provides a grasping method of a robot bionic hand.

According to one aspect of the present disclosure, the present disclosure provides a method for grasping by a robot bionic hand, which includes:

Obtain the image of the target object;

Input the image of the target object into the visual detection neural network model for identification, and obtain the visual information of the target object, the visual information including the abscissa of the target object, the ordinate of the target object, the width of the target object, the height of the target object and the category of the target object;

Based on the visual information of the target object, determine the visual grasping parameters of the robot bionic hand;

Grasping the target object based on the visual grasping parameters of the bionic hand of the robot;

Obtaining the tactile information and the grasping result of the target object;

Acquiring soft and hard attribute data of the target object based on the tactile information of the target object and the visual information of the target object; and

Based on the soft and hard attribute data of the target object, adjust the tactile grasping parameters of the robot bionic hand and continue to grasp;

Wherein, the parameter setting of the visual detection neural network model includes: 6 convolutional layers, 6 pooling layers, 10000 training times, 20 training samples per batch, and 0.001 learning rate;

The training process of the visual detection neural network model includes:

Collecting visual samples includes: using a depth camera to collect an image containing a target object, performing target preprocessing on the image, and obtaining image label information, the preprocessing includes scaling the image containing the target object, and the obtaining image label information Including obtaining the abscissa of the target object in the image, the ordinate of the target object, the width of the target object, the height of the target object and the category of the target object;

Inputting the training set of the collected visual samples into the visual detection neural network model for training to obtain the trained visual detection neural network model; and

Input the test set of collected visual samples into the visual detection neural network model for test verification, and obtain the available visual detection neural network model after passing the verification;

The acquisition of the tactile information and grasping results of the target object includes: the tactile sensor of the robot bionic hand acquires contact force data in time series for different contact points, and the grasping results in the corresponding time series, the The contact force data forms a two-dimensional array corresponding to time and point;

The acquisition of soft and hard attribute data of the target object based on the tactile information of the target object and the visual information of the target object includes: forming the contact force data of the target object into a two-dimensional array corresponding to time and point and the target Input the object category into the tactile detection neural network model for identification, and obtain the soft and hard attribute data of the target object;

The tactile detection neural network model is a cyclic neural network model, and the parameter settings of the cyclic neural network model include: 64 hidden layers, 10000 training times, 20 training samples per batch, and a learning rate of 0.001; The network model building process includes:

Collecting tactile samples, within the range of time series, for different contact points, said collecting tactile samples, said collecting tactile samples is performed every time a visual sample is collected;

Inputting the training set of collected tactile samples into the tactile detection neural network model for training to obtain a trained tactile detection neural network model; and

The test set of collected tactile sense samples is input into the trained tactile sense detection neural network model for test verification, and after the verification is passed, an available tactile sense detection neural network model is obtained.

According to the robot bionic hand grasping method according to at least one embodiment of the present disclosure, the acquisition of the image of the target object includes:

The image of the target object is acquired through a depth camera, which is installed above the robot and has no contact with the robot.

According to the grasping method of the robotic bionic hand in at least one embodiment of the present disclosure, the determination of visual grasping parameters of the robotic bionic hand based on the visual information of the target object includes:

Based on the visual information of the target object, the trajectory and grasping posture of the robot bionic hand are determined.

According to the grasping method of the robotic bionic hand in at least one embodiment of the present disclosure, the adjustment of the tactile grasping parameters of the robotic bionic hand based on the soft and hard attribute data of the target object includes:

Based on the soft and hard attribute data of the target object, the contact force between the robot bionic hand and the target object is adjusted.

Description of drawings

The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain the principles of the disclosure, are included to provide a further understanding of the disclosure, and are incorporated in and constitute this specification. part of the manual.

Fig. 1 is a schematic flowchart of a method for grabbing a robotic bionic hand according to an embodiment of the present disclosure.

Fig. 2 is a schematic diagram of a visual detection neural network structure according to an embodiment of the present disclosure.

Fig. 3 is a schematic diagram of a structure of a cyclic neural network for touch detection according to an embodiment of the present disclosure.

Fig. 4 is a schematic flowchart of a visual-tactile sample collection method according to an embodiment of the present disclosure.

Figure 5 is a schematic illustration of a metal filled box according to one embodiment of the present disclosure.

Figure 6 is a schematic diagram of an empty box according to one embodiment of the present disclosure.

FIG. 7 is a schematic diagram of a haptic data sample according to an embodiment of the present disclosure.

FIG. 8 is a diagram illustrating details of a haptic data sample according to an embodiment of the present disclosure.

Fig. 9 is a schematic structural diagram of a robot bionic hand grasping system according to an embodiment of the present disclosure.

Explanation of reference signs

1000 robot bionic hand grasping system

1002 Robot Bionic Hand

1004 visual information acquisition module

1006 Tactile information acquisition module

1008 Visual Analysis Module

1010 Tactile Analysis Module

1012 grab control module

1100 bus

1200 processor

1300 memory

1400 Other circuits.

Detailed ways

The present disclosure will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific implementation manners described here are only used to explain relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only parts related to the present disclosure are shown in the drawings.

It should be noted that, in the case of no conflict, the implementation modes and the features in the implementation modes in the present disclosure can be combined with each other. The technical solutions of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with implementation manners.

Unless otherwise specified, the illustrated exemplary embodiments/embodiments are to be understood as exemplary features providing various details of some manner in which the technical idea of the present disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various embodiments/embodiments may be additionally combined, separated, interchanged, and/or rearranged without departing from the technical concept of the present disclosure.

The use of cross-hatching and/or shading in the figures is generally used to clarify the boundaries between adjacent features. As such, unless stated otherwise, the presence or absence of cross-hatching or shading conveys or indicates no specific material, material properties, dimensions, proportions, commonality between the illustrated components, and/or any other characteristic of the components, Any preferences or requirements for attributes, properties, etc. Also, in the drawings, the size and relative sizes of components may be exaggerated for clarity and/or descriptive purposes. While exemplary embodiments may be implemented differently, a specific process sequence may be performed in an order different from that described. For example, two consecutively described processes may be performed substantially simultaneously or in an order reverse to that described. In addition, the same reference numerals denote the same components.

When an element is referred to as being "on" or "over", "connected to" or "coupled to" another element, the element may be directly on, directly connected to, or Or directly bonded to the other component, or intermediate components may be present. However, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element, there are no intervening elements present. To this end, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

For descriptive purposes, this disclosure may use terms such as "under", "beneath", "below", "under", "above", "on", "on" ... above", "higher" and "side (e.g., as in "side walls")", etc., to describe one component compared to another (other) component as shown in the drawings Relationship. Spatially relative terms are intended to encompass different orientations of the device in use, operation and/or manufacture in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the exemplary term "below" can encompass both an orientation of "above" and "beneath". In addition, the device may be otherwise positioned (eg, rotated 90 degrees or at other orientations), and as such, the spatially relative descriptors used herein are interpreted accordingly.

The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, the singular forms "a" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. In addition, when the terms "comprising" and/or "comprising" and their variants are used in this specification, it means that the stated features, integers, steps, operations, parts, components and/or their groups exist, but do not exclude One or more other features, integers, steps, operations, parts, components and/or groups thereof are present or in addition. Note also that, as used herein, the terms "substantially," "about," and other similar terms are used as terms of approximation and not as terms of degree, and as such, they are used to explain what one of ordinary skill in the art would recognize. Inherent deviations from measured, calculated and/or supplied values.

Fig. 1 is a schematic flowchart of a method for grabbing a robotic bionic hand according to an embodiment of the present disclosure.

As shown in FIG. 1, a method S100 for grabbing a robot bionic hand includes:

S102: Acquiring an image of the target object;

S104: Analyze the image of the target object to obtain visual information of the target object;

S106: Based on the visual information of the target object, determine the visual grasping parameters of the robot bionic hand;

S108: Grasping the target object based on the visual grasping parameters of the robot bionic hand;

S110: Obtain tactile information of the target object and a grasping result;

S112: Obtain soft and hard attribute data of the target object based on the tactile information of the target object and the visual information of the target object; and,

S114: Based on the soft and hard attribute data of the target object, adjust the tactile grasping parameters of the robotic bionic hand and continue grasping.

Among them, the image of the target object is obtained, including:

The image of the target object is obtained through the depth camera, which is installed above the robot and has no contact with the robot.

Among them, the image of the target object is analyzed to obtain the visual information of the target object, including:

The image of the target object is input into the visual detection neural network model for recognition, and the visual information of the target object is obtained. The visual information includes the abscissa of the target object, the ordinate of the target object, the width of the target object, the height of the target object, and the category of the target object.

Among them, the parameter settings of the visual detection neural network model include: 6 convolutional layers, 6 pooling layers, 10,000 training times, 20 training samples per batch, and 0.001 learning rate;

The training process of the visual detection neural network model, including:

Collecting visual samples, including: using a depth camera to collect images containing target objects, performing target preprocessing on images, and obtaining image label information, preprocessing includes scaling images containing target objects, and obtaining image label information includes obtaining target objects in images The abscissa, the ordinate of the target object, the width of the target object, the height of the target object and the category of the target object;

Inputting the training set of the collected visual samples into the visual detection neural network model for training to obtain the trained visual detection neural network model; and,

Input the test set of collected visual samples into the visual detection neural network model for test verification, and obtain the usable visual detection neural network model after passing the verification.

Among them, based on the visual information of the target object, the visual grasping parameters of the robot bionic hand are determined, including:

Based on the visual information of the target object, the trajectory and grasping posture of the robot bionic hand are determined.

Among them, the tactile information of the target object and the grasping results are obtained, including:

The tactile sensor of the robot bionic hand acquires contact force data for different contact points in time series, and, for the grasping results in the corresponding time series, the contact force data forms a two-dimensional array corresponding to time and point.

Among them, based on the tactile information of the target object and the visual information of the target object, the soft and hard attribute data of the target object are obtained, including:

The contact force data of the target object is formed into a two-dimensional array corresponding to time and point, and the category of the target object is input into the tactile detection neural network model for identification to obtain the soft and hard attribute data of the target object.

Among them, the tactile detection neural network model is a recurrent neural network model,

The parameter settings of the cyclic neural network model include: the number of hidden layers is 64, the number of training times is 10000, the number of training samples per batch is 20, and the learning rate is 0.001;

The establishment process of the tactile detection neural network model includes:

The tactile samples are collected, and the tactile samples are collected for different contact points within the range of time series, and the tactile samples are collected every time a visual sample is collected.

Input the training set of the collected tactile sense samples into the tactile detection neural network model for training to obtain the trained tactile detection neural network model; and,

The test set of collected tactile samples is input into the trained tactile detection neural network model for test verification. After the verification is passed, a usable tactile detection neural network model is obtained.

Among them, based on the soft and hard attribute data of the target object, the tactile grasping parameters of the robot bionic hand are adjusted, including:

Based on the soft and hard attribute data of the target object, the contact force between the robot bionic hand and the target object is adjusted.

In summary, through the acquisition and analysis of the visual information of the target object, the detection, classification and positioning of the target can be realized. When the robot is close to the object, the soft and hard properties of the object can be obtained through the sense of touch, and then the robot can be controlled according to the material properties of the object. The bionic hand performs different grasping strategies, adjusts the magnitude of the grasping force, the position and posture of the grasping, so as to complete the stable and safe grasping of the object.

Detect, classify and locate the target through visual information. When the robot is close to the object, it can obtain the soft and hard properties of the object through the sense of touch, and then control the system according to the material properties of the object. Different grasping strategies, adjusting the size of the grasping force and The position and posture of the grasping, so as to complete the stable and safe grasping of the object. By simulating the cognitive thinking of human experts, constructing a visual and tactile state experience knowledge base, through interactive learning between the robot and the environment, with the help of vision and force sense to perceive the state of the robot in the working process, accumulate and update the robot experience knowledge base, and monitor the robot's operation process The execution status of the robot, and based on the deep learning method, evaluate the completion status of the robot's work and judge whether it is successful. It can be seen from this that it can help robots replace humans to better complete various complex tasks and improve production efficiency.

Fig. 2 is a schematic diagram of a visual detection neural network structure according to an embodiment of the present disclosure.

As shown in Figure 2, image input: [512,512,3], image size 512×512, three channels, the input and output data dimensions of each convolution layer and pooling layer are as follows:

The first convolutional layer, parameters: [5,5,3,32], convolution kernel 5×5, input channel 3, output channel 32, activation function uses relu, input: 512×512×3, output: 512 ×512×32;

The first layer of pooling layer, maximum pooling processing, input: 512×512×32, output is 256×256×32;

The second convolutional layer, parameter: [5,5,32,64], convolution kernel 5×5, input channel 32, output channel 64, activation function uses relu, input: 256×256×32, output: 256 ×256×64;

The second pooling layer, maximum pooling processing, input: 256×256×64, output 128×128×64;

The third convolutional layer, parameter: [3,3,64,128], convolution kernel 5×5, input channel 64, output channel 128, activation function uses relu, input: 128×128×64, output: 128×128 ×128;

The third pooling layer, maximum pooling processing, input: 128×128×128, output 64×64×128;

The fourth convolutional layer, parameter: [3,3,64,128], convolution kernel 5×5, input channel 128, output channel 192, activation function uses relu, input: 64×64×128, output: 64×64 ×192;

The fourth pooling layer, maximum pooling processing, input: 64×64×192, output 32×32×192;

The fifth convolutional layer, parameters: [3,3,192,256], convolution kernel 3×3, input channel 192, output channel 256, activation function uses relu, input: 32×32×192, output: 32×32×256 ;

The fifth pooling layer, maximum pooling processing, input: 32×32×256, output 16×16×256;

The sixth convolutional layer, parameter: [3,3,256,512], convolution kernel 3×3, input channel 256, output channel 512, activation function uses relu, input: 16×16×256, output: 16×16×512 ;

The sixth pooling layer, maximum pooling processing, input: 16×16×512, output 8×8×512;

Fully connected layer 1, flatten the previous output result, input: 8×8×512, output: 1024;

Fully connected layer 2, the output object category, input: 1024, output: N (custom, equal to the number of categories collected samples).

Fig. 3 is a schematic structural diagram of a tactile detection recurrent neural network provided according to an embodiment of the present disclosure.

As shown in Figure 3, the tactile detection recurrent neural network structure includes:

Network input layer: the feature dimension of x, the 25 continuous time series selected here;

Hidden layer: The feature dimension of the hidden layer determines the dimension of the hidden state hidden_state, which can be simply regarded as constructing a weight, where the hidden layer is selected as 64; and,

LSTM layer: The number of LSTM layers, the default is 1.

To sum up, the present invention constructs a visual and tactile state experience knowledge base by simulating the cognitive thought of human experts, through interactive learning between the robot and the environment, with the help of vision and force sense to perceive the state of the robot during the working process, and accumulate and update the robot experience knowledge base. Monitor the execution status of the robot during the operation process, and evaluate the completion status of the robot's work based on the deep learning method to judge whether it is successful or not. The invention can help the robot to replace the human to complete various complicated work and improve the production work efficiency.

Fig. 4 is a schematic structural diagram of a robot bionic hand grasping system according to an embodiment of the present disclosure.

The device may include corresponding modules for executing each or several steps in the above flow chart. Therefore, each step or several steps in the above flowcharts may be performed by corresponding modules, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform the corresponding steps, or be implemented by a processor configured to perform the corresponding steps, or be stored in a computer-readable medium for implementation by the processor, or be implemented by a some combination to achieve.

The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application of the hardware and the overall design constraints. The bus connects together various circuits including one or more processors, memory and/or hardware modules. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, external antennas, and the like.

The bus can be an Industry Standard Architecture (ISA, Industry Standard Architecture) bus, a Peripheral Component Interconnect (PCI, Peripheral Component) bus or an Extended Industry Standard Architecture (EISA, Extended Industry Standard Component) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, only one connection line is used in this figure, but it does not mean that there is only one bus or one type of bus.

Any process or method descriptions in flowcharts or otherwise described herein may be understood to represent modules, segments or portions of code comprising one or more executable instructions for implementing specific logical functions or steps of the process , and the scope of preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including substantially concurrently or in reverse order depending on the functions involved, which shall It is understood by those skilled in the art to which the embodiments of the present disclosure belong. The processor executes the various methods and processes described above. For example, method embodiments in the present disclosure may be implemented as a software program tangibly embodied on a machine-readable medium, such as memory. In some implementations, part or all of the software program may be loaded and/or installed via memory and/or a communication interface. One or more steps in the methods described above may be performed when a software program is loaded into memory and executed by a processor. Alternatively, in other implementation manners, the processor may be configured to perform one of the above-mentioned methods in any other suitable manner (for example, by means of firmware).

The logic and/or steps shown in the flowcharts or otherwise described herein can be embodied in any readable storage medium for instruction execution systems, devices or devices (such as computer-based systems, processor-included system or other systems that may fetch and execute instructions from an instruction execution system, device, or device), or be used in conjunction with such an instruction execution system, device, or device.

As far as this specification is concerned, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit programs for instruction execution systems, devices or devices or use in conjunction with these instruction execution systems, devices or devices. More specific examples (non-exhaustive list) of readable storage media include the following: electrical connection with one or more wires (electronic device), portable computer disk case (magnetic device), random access memory (RAM), Read Only Memory (ROM), Erasable and Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Read Only Memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be scanned, for example, by optical scanning of the paper or other medium, followed by editing, interpretation or other suitable means if necessary. processing to obtain programs electronically and store them in memory.

It should be understood that various parts of the present disclosure may be realized by hardware, software or a combination thereof. In the embodiments described above, various steps or methods may be implemented by software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one or combination of the following techniques known in the art: Discrete logic circuits, ASICs with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc.

Those of ordinary skill in the art can understand that all or part of the steps to realize the above-mentioned implementation method can be completed by instructing related hardware through a program, and the program can be stored in a readable storage medium. When the program is executed, it includes One or a combination of steps of a method embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing module, each unit may exist separately physically, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. If the integrated modules are realized in the form of software function modules and sold or used as independent products, they can also be stored in a readable storage medium. The storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like.

Fig. 4 is a schematic flowchart of a visual-tactile sample collection method according to an embodiment of the present disclosure.

As shown in FIG. 4 , the visual and tactile sample collection method S200 includes:

S201: After placing the target object, start and execute the collection task;

S202: Manually control the movement of the robot to approach the target object;

S203: The camera takes pictures, and collects the visual information of the target object;

S204: Pre-fetch the target object, and at the same time, obtain tactile data through tactile sensing;

S205: Determine whether the capture is successful, if the capture is successful, go to S206, otherwise go to S204;

S206: move the target object, and go to S207; and,

S207: Determine whether the target object is sliding, if there is sliding, go to S205, otherwise, go to S201.

Figure 5 is a schematic illustration of a metal filled box according to one embodiment of the present disclosure.

Figure 6 is a schematic diagram of an empty box according to one embodiment of the present disclosure.

FIG. 7 is a schematic diagram of a haptic data sample according to an embodiment of the present disclosure.

FIG. 8 is a diagram illustrating details of a haptic data sample according to an embodiment of the present disclosure.

Fig. 9 is a schematic structural diagram of a robot bionic hand grasping system according to an embodiment of the present disclosure.

As shown in Figure 9, a robot bionic hand grasping system 1000 includes:

Robotic bionic hand 1002 for grasping objects;

The visual information collection module 1004 is used to collect the image of the target object. Preferably, the visual information collection module is a depth camera, and the depth camera is installed above the robot and has no contact with the robot;

The tactile information collection module 1006 is used to collect the tactile information of the target object. Preferably, the tactile information collection module is an array type tactile sensor, and the array type tactile sensor is placed on the claw body of the bionic hand;

A visual analysis module 1008, configured to analyze the image of the target object to obtain the category of the target object, the position of the target object, and the size of the target object;

The tactile analysis module 1010 is connected to the robot bionic hand for receiving and analyzing the data collected by the tactile information collection module to obtain the soft and hard attribute data of the object material; and,

The grasping control module 1012 is communicated with the robot bionic hand, and controls the robot bionic hand to grasp the object based on the type of the target object, the position of the target object, the size of the target object, and the soft and hard attribute data of the target object.

In the description of this specification, descriptions referring to the terms "one embodiment/mode", "some embodiments/modes", "examples", "specific examples", or "some examples" mean that the embodiments/modes are combined The specific features, structures, materials or characteristics described in or examples are included in at least one embodiment/mode or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment/mode or example. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or more embodiments/modes or examples in an appropriate manner. In addition, those skilled in the art may combine and combine different embodiments/modes or examples and features of different embodiments/modes or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

It should be understood by those skilled in the art that the above-mentioned embodiments are only for clearly illustrating the present disclosure, rather than limiting the scope of the present disclosure. For those skilled in the art, other changes or modifications can be made on the basis of the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims (4)

1. A robot bionic hand grabbing method is characterized by comprising the following steps:

acquiring a target object image;

inputting a target object image into a visual detection neural network model for identification, and obtaining visual information of a target object, wherein the visual information comprises a target object abscissa, a target object ordinate, a target object width, a target object height and a target object category;

determining vision grabbing parameters of the bionic hand of the robot based on the vision information of the target object;

grabbing a target object based on the vision grabbing parameters of the bionic hand of the robot;

obtaining the tactile information and the grabbing result of the target object;

acquiring soft and hard attribute data of the target object based on the touch information of the target object and the visual information of the target object; and

based on the soft and hard attribute data of the target object, adjusting the touch grabbing parameters of the bionic hand of the robot and continuously grabbing;

wherein, the parameter setting of the visual detection neural network model comprises the following steps: the convolution layer number is 6, the pooling layer number is 6, the training times are 10000, the training sample number of each batch is 20, and the learning rate is 0.001;

the training process of the visual inspection neural network model comprises the following steps:

collecting a visual sample comprising: acquiring an image containing a target object by using a depth camera, performing target preprocessing on the image, and acquiring image tag information, wherein the preprocessing comprises zooming the image containing the target object, and the acquiring of the image tag information comprises acquiring a target object abscissa, a target object ordinate, a target object width, a target object height and a target object category in the image;

inputting a training set of the collected visual samples into a visual detection neural network model for training to obtain a trained visual detection neural network model; and

inputting the collected test set of the visual sample into a visual detection neural network model for test verification, and obtaining an available visual detection neural network model after the verification is passed;

the obtaining of the haptic information and the grasping result of the target object includes: the method comprises the steps that a touch sensor of a bionic hand of a robot acquires contact force data and a grabbing result in a corresponding time sequence from different contact point positions according to the time sequence, and the contact force data form a two-dimensional array corresponding to time and the point positions;

the acquiring soft and hard attribute data of the target object based on the tactile information of the target object and the visual information of the target object comprises: forming a two-dimensional array corresponding to time and point positions and the type of the target object into the touch detection neural network model for recognition, and obtaining soft and hard attribute data of the target object;

the tactile detection neural network model is a recurrent neural network model, and the parameter setting of the recurrent neural network model comprises the following steps: the hidden layer number is 64, the training times are 10000, the training sample number of each batch is 20, and the learning rate is 0.001; the haptic sensation detection neural network model establishing process comprises the following steps:

acquiring a touch sample, wherein the touch sample is acquired for different contact points in a time sequence range, and the acquisition of the touch sample is performed when one visual sample is acquired;

inputting a training set of the collected tactile samples into a tactile detection neural network model for training to obtain a trained tactile detection neural network model; and

and inputting the collected test set of the touch samples into the trained touch detection neural network model for test verification, and obtaining an available touch detection neural network model after the verification is passed.

2. The robotic biomimetic hand-grabbing method as recited in claim 1, wherein the acquiring an image of a target object comprises:

the target object image is acquired by a depth camera, which is mounted above the robot and is contactless with the robot.

3. The method of claim 1, wherein determining the visual grasping parameters of the biomimetic robotic hand based on the visual information of the target object comprises:

and determining the motion trail and the grabbing posture of the bionic hand of the robot based on the visual information of the target object.

4. The method according to claim 1, wherein adjusting the parameters of the robotic biomimetic hand for grabbing the tactile sensation based on the soft and hard attribute data of the target object comprises:

and adjusting the contact force of the bionic hand of the robot and the target object based on the soft and hard attribute data of the target object.

Images