CN102501252A - Method and system for controlling movement of tail end of executing arm - Google Patents
Method and system for controlling movement of tail end of executing arm Download PDFInfo
- Publication number
- CN102501252A CN102501252A CN2011103016120A CN201110301612A CN102501252A CN 102501252 A CN102501252 A CN 102501252A CN 2011103016120 A CN2011103016120 A CN 2011103016120A CN 201110301612 A CN201110301612 A CN 201110301612A CN 102501252 A CN102501252 A CN 102501252A
- Authority
- CN
- China
- Prior art keywords
- sign
- arm
- camera
- terminal
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention discloses a method for controlling the movement of a tail end of an executing arm. The method comprises the following steps of: 11) arranging a camera at the tail end of the executing arm and arranging an identifier in a visible area of the camera; 12) obtaining an image of the identifier by the camera in real time; 13) pre-storing an initial image of the identifier at an initial position, comparing the initial image with the real-time obtained image of the identifier, and calculating the real-time displacement of the tail end of the executing arm relative to the initial position; and 14) controlling the movement of the tail end of the executing arm according to the real-time displacement of the tail end of the executing arm. According to the control method, the real-time displacement of the tail end of the executing arm is obtained through direct image analysis, so that an instruction to control the movement of the executing arm is output. Compared with a manner of overlapping and calculating of a plurality of sensor data signals in the prior art, the manner of directly comparing and calculating the images is higher in accuracy. The invention further discloses a system for controlling the movement of the tail end of the executing arm.
Description
Technical field
The present invention relates to technical field of engineering machinery, particularly a kind of method of carrying out the terminal motion of arm of controlling.The invention still further relates to a kind of control system that carries out the terminal motion of arm.
Background technology
Various engineering machinery has jib structure, and such as concrete mixer, crane, fire fighting truck etc., jib is in order to carry manpower or material to specifying the job location.In work progress; Often need jib to according to same in stable condition motion, at this moment, need accurately to gather the terminal location parameter of jib; Calculate the terminal distance that moves relative to initial position of jib according to positional parameters vary, and then control the track of the terminal motion of jib in real time.
In the prior art, take the mode of indirect calculation to obtain the terminal position of jib usually, control jib end moves in a certain fixed position, and main account form is following:
(1) measures the angle that respectively saves arm respectively at the equal setting angle sensor of arm that respectively saves of jib, combine respectively to save the terminal position of length computation jib of arm again.Yet each saves arm can constantly produce deformation in the course of the work, and it is inaccurate calculating terminal vertical height through angle and the jib length that do not produce deformation.Even, at each joint arm installation position displacement sensor, calculate the deformation that respectively saves arm, calculate again in conjunction with deformation quantity, also can't guarantee the accuracy of result of calculation.Comprise more piece joint arm mainly due to, jib, the accuracy that each stack computing that saves arm will certainly reduce result of calculation can't satisfy intellectuality, automated construction operational requirements.
(2) at the jib end distance measuring sensor is set, in order to measure the terminal location parameter of jib.This kind mode can directly be measured the terminal vertical height of jib; But measuring distance is limited, precision is not high; Particularly when there was the object (the concrete height that pile up on ground, jib end below when making pump such as concrete mixer can constantly change) that highly constantly changes in jib end below, measurement result was inaccurate.
Therefore, how improving the control method of the terminal motion of jib, improve control accuracy, is those skilled in the art's technical issues that need to address.
Summary of the invention
In view of above-mentioned technical problem, the object of the invention is that a kind of method of carrying out the terminal motion of arm of controlling is provided, and this control method is utilized image processing method formula real time direct to obtain and carried out the terminal displacement of arm, carries out the terminal movement locus of arm thereby can accurately control.Another object of the present invention provides a kind of control system that carries out the terminal motion of arm.
For reaching the object of the invention, the present invention provides a kind of method of carrying out the terminal motion of arm of controlling, and may further comprise the steps:
11) in said execution arm end camera is set, and sign is set in the viewing area of said camera;
12) said camera obtains the image of said sign in real time;
13) the prestore initial pictures of the said sign of initial position, the image of said initial pictures of comparative analysis and the said sign obtained in real time calculates the real-time displacement that obtains the said relatively sign of said execution arm end;
14) control the terminal motion of said execution arm based on the terminal real-time displacement of said execution arm.
Preferably,
In the step 11), also the window near-end at said camera is provided with scale, and the position relation of said scale and said camera keeps constant;
In the step 12), said camera obtains the image that comprises said sign and said scale;
In the step 13); The initial position that prestores comprises the initial pictures of said sign and said scale; The image that comprises said sign and said scale that the said initial pictures analysis of comparative analysis is obtained in real time calculates the real-time displacement that obtains the said relatively sign of said execution arm end.
Preferably,
In the step 11), the horizontal ruler and vertical scale of the said scale of setting for intersecting; In the step 13); With the sign in the initial pictures and obtain in real time in the image sign all with the horizontal ruler of corresponding said scale and vertically the imaging of scale compare analysis, calculate the left and right displacement and the upper and lower displacement that obtain the said relatively sign of said execution arm end.
Preferably,
In the step 11), on said camera, angular transducer is set;
In the step 12), said camera is followed the tracks of the center of said sign, and obtains the image of said sign;
In the step 13); Write down said camera level angle and vertical angle that said angular transducer detects; Said initial pictures of comparative analysis and the identification image that obtains in real time; Obtain said the camera extremely initial distance and real-time distance of said sign center, obtain the real-time displacement of the said relatively initial position of said execution arm end according to corner, initial distance, distance, initial level angle, initial vertical angle, real-time level angle and vertical angle calculating in real time in real time.
Preferably, between step 12) and step 13), also have following steps:
123) the adjustment camera makes the center of aiming at said sign in the window center of the said camera of initial position.
Preferably, control the focal length of said camera according to the image of the said sign of obtaining.
Preferably, in the step 11), the working region around said execution arm end is provided with at least two signs.
The method of the terminal motion of arm is carried out in control provided by the present invention; Can be directly through initial pictures and the identification image that obtains in real time analysis relatively; Obtain the real-time displacement of sign, and then know the real-time displacement of carrying out the arm end, the terminal instruction of moving of arm is carried out in output control.Directly compare calculation mode,, have higher accuracy with respect to utilizing a plurality of sensor data signals stack calculation mode in the prior art through image.
In further technical scheme, at the near-end of camera scale is set, scale can be eliminated the error that the camera shake causes when analyzing the normative reference of sign; And, because the imaging of scale is comparatively steady and audible, the shape of the comparison that is easy to be designed to be convenient for reference, handling procedure that can the simplified image processing module improves accuracy; In addition, under the situation that focal length changes, can calculate the focal length variable through the change of scale imaging size, and then accurately calculate the real-time displacement of sign, and need not to obtain the focal length variable through other approach.
In order to reach second purpose of the present invention, the present invention also provides a kind of control system that carries out the terminal motion of arm, comprising:
Camera, it is terminal to be located at said execution arm;
Be arranged at the sign of said camera viewing area, said camera obtains the image of said sign;
Image processing module; Said image processing module receives the image that said camera obtains; And the initial pictures of the said sign of initial position that prestores; And the image of said initial pictures of compare of analysis and the said sign obtained in real time, calculate the real-time displacement that obtains the said relatively sign of said execution arm end;
Carry out arm controller, receive the real-time displacement of the said relatively sign of said execution arm end of said image processing module acquisition, carry out the terminal motion of arm based on the real-time displacement control of the said relatively sign of said execution arm end.
Preferably, the scale that also comprises the window near-end that is arranged at said camera; Said camera obtains the image that comprises said sign and said scale; The said image processing module initial position that prestores comprises the initial pictures of said sign and said scale; And said initial pictures of compare of analysis and the image that comprises said sign and said scale that obtains in real time, calculate the real-time displacement that obtains the said relatively sign of said execution arm end.
Preferably, the horizontal ruler and vertical scale of said scale for intersecting; Said image handle with the sign in the initial pictures with obtain in real time in the image sign all with the horizontal ruler of corresponding said scale and vertically the imaging of scale compare analysis, calculating acquisition relative initial position left and right displacement of said execution arm end and upper and lower displacement.
Preferably, also comprise the control module of camera, said control module is adjusted said camera according to the image of the said sign of obtaining, and makes the center of aiming at said sign in the window center of the said camera of initial position.
Preferably, said camera is provided with angular transducer; Said camera is followed the tracks of the center of said sign, and obtains the image of said sign; Said image processing module writes down the real-time corner of the said camera of said angular transducer detection with respect to initial position; And said initial pictures of comparative analysis and the identification image that obtains in real time; Obtain the initial distance and real-time distance of said camera to said sign center, obtain the terminal real-time displacement of initial position relatively of said execution arm according to corner, initial distance and real-time distance calculation.
Preferably, said control module is controlled the focal length of said camera according to the image of the said sign of obtaining.
Preferably, the working region around said execution arm end is provided with at least two signs.
Preferably, said sign is processed by reflectorized material; And be set to identify the illuminating lamp that illumination is provided.
The control system of the terminal motion of execution arm provided by the present invention; Directly analyze initial pictures that relatively identifies and the image that obtains in real time by image processing module; Obtain the real-time displacement of carrying out the terminal relative sign of arm based on variable quantity, and by the instruction of carrying out the motion of arm motion controller output control execution arm.With respect to utilizing a plurality of sensor data signal stack calculation mode in the prior art, this control system possesses high-precision analysis and Control ability.And in carrying out the process that arm moves, the near-end working condition chart that can also utilize above-mentioned control system will carry out the terminal working region of arm looks like to be transferred to the display unit in the control system, operates so that carry out Remote.Further can analyze carrying out the terminal peripheral operating mode of arm, avoid carrying out the terminal and large-scale barrier collision of arm, realize carrying out the terminal intelligent barrier avoiding of arm through above-mentioned control system.
Description of drawings
Fig. 1 is the flow chart that the terminal movement technique of arm is carried out in control provided by the present invention;
Fig. 2 is the flow chart that first kind of specific embodiment of the terminal movement technique of arm carried out in control provided by the present invention;
Fig. 3 is the flow chart that second kind of specific embodiment of the terminal movement technique of arm carried out in control provided by the present invention;
Fig. 4 is the sketch map that first kind of concrete account form of the terminal movement technique of arm carried out in control provided by the present invention;
Fig. 5 is the sketch map that second kind of concrete account form of the terminal movement technique of arm carried out in control provided by the present invention;
Fig. 6 is the sketch map that terminal the third the concrete account form of movement technique of arm is carried out in control provided by the present invention;
Fig. 7 is the structural representation of a kind of specific embodiment of the terminal kinetic control system of execution arm provided by the present invention;
Fig. 8 is the control principle sketch map of control system among Fig. 7.
The specific embodiment
Core of the present invention is that a kind of method of carrying out the terminal motion of arm of controlling is provided, and this control method is utilized image processing method formula real time direct to obtain and carried out the terminal displacement of arm, carries out the terminal movement locus of arm thereby can accurately control.Another core of the present invention provides a kind of control system that carries out the terminal motion of arm.
In order to make those skilled in the art understand technical scheme of the present invention better, the present invention is done further detailed description below in conjunction with accompanying drawing and specific embodiment.
Please refer to Fig. 1, Fig. 1 is the flow chart that the terminal movement technique of arm is carried out in control provided by the present invention.
The method of the terminal motion of arm is carried out in control provided by the present invention, mainly may further comprise the steps:
Step S11) in carrying out the arm end camera is set, and sign is set in the viewing area of camera;
Carry out the terminal viewing area of arm, near the position that refers to carry out the terminal manipulating object of arm, or this position so that after state in the step, be positioned at and carry out the image that the terminal camera of arm obtains sign.
Camera is arranged to carry out arm terminal, then camera is with carrying out terminal the moving of arm, and is promptly consistent with the movement locus of execution arm end.
Step S12) camera obtains the image of sign in real time;
Step S13) the prestore initial pictures of initial position sign, and the image of comparative analysis initial pictures and the sign of obtaining in real time calculate the real-time displacement that obtains to carry out the terminal relative initial position of arm;
In carrying out the terminal motion process of arm; Camera moves with it; And sign is arranged in the viewing area of carrying out the arm end, thus be identified at the process that is imaged as dynamic change in the camera, therefore; Analyze the variable quantity of the image of sign, can obtain to carry out the terminal real-time displacement of initial position relatively of arm with respect to initial pictures.
Step S14) carries out the terminal motion of arm based on the real-time displacement control of sign.
Carry out when arm is terminal to be moved, there are relative motion in sign and camera, such as; When sign moves to left; Be equivalent to camera and move to right, and the displacement of camera is terminal consistent with the execution arm, so according to the terminal displacement of the execution arm that obtains; Need in conjunction with actual motion, can control and carry out the terminal movement locus of arm.
Can know through above-mentioned argumentation; The control method of the terminal motion of execution arm provided by the present invention; Can be directly through initial pictures and the identification image that obtains in real time analysis relatively; Obtain and carry out the terminal real-time displacement of initial position relatively of arm, and then know the real-time displacement of carrying out the arm end, the instruction of arm motion is carried out in output control.Directly compare calculation mode,, have higher accuracy with respect to utilizing a plurality of sensor data signals stack calculation mode in the prior art through image.
Please refer to Fig. 2, Fig. 2 is the flow chart that first kind of specific embodiment of the terminal movement technique of arm carried out in control provided by the present invention.Comprise the steps:
Step S21), in carrying out the arm end camera is set, and in the viewing area of camera sign is set, also the window near-end at camera is provided with scale;
Be arranged at the near-end of window, guarantee that promptly scale all the time can be through the camera imaging, the position relation of scale and camera keeps constant; Promptly under the constant situation of focal length; The imaging size of scale equates all the time, when focal length changes, also can calculate the variation of focal length based on the variation of scale imaging size; So that accurately calculate the variation of the sign imaging size that causes owing to focal length variations, make the graphical analysis of sign have stable referential.The set-up mode of scale has multiple, such as, when camera was set, scale can be fixed in the place ahead of cam lens, also can be located at the inside of camera, and scale can also be fixed with miscellaneous part, guarantees it lens location with respect to camera is constant to get final product.
Step S22), camera obtains the image that comprises sign and scale;
Because scale can form images all the time, so the image that camera obtains comprises sign and scale simultaneously, certainly, under the situation that sign moves, in the different images constantly of acquisition, the relative position relation and the relative size of sign and scale all possibly produce dynamic change.
Step S23), the initial position that prestores comprises the sign and the initial pictures of scale, and comparative analysis initial pictures and the image that comprises sign and scale that obtains in real time, calculates the real-time displacement of acquisition execution arm end.
Analytical calculation principle in this step and step S13) identical.Because the imaging of scale is comparatively steady and audible, the shape of the comparison that is easy to be designed to be convenient for reference at this moment, can be calculated the reference of scale as sign imaging change calculations to carry out the terminal real-time displacement of arm.Therefore, the handling procedure that all right simplified image processing module is set of scale improves accuracy; In addition, under the situation that focal length changes, can calculate the focal length variable through the change of scale imaging size, and then accurately calculate and carry out the terminal real-time displacement of arm, and need not to obtain the focal length variable through other approach.
Step S24) carries out the terminal motion of arm based on the real-time displacement control of sign.With step S14) identical.
Please refer to Fig. 3, the flow chart of second kind of specific embodiment of the terminal movement technique of arm is carried out in Fig. 3 control provided by the present invention.Comprise the steps:
Step S31), in carrying out the arm end camera is set, and in the viewing area of camera sign is set, also the window near-end at camera is provided with scale;
Step S32), camera obtains the image that comprises sign and scale;
Step S31), step S32) with step S21), step S22) identical.
Step S33), the adjustment camera makes the center of aiming at sign in the window center of initial position camera.
Be with the difference of first kind of specific embodiment, increased this step.After adding this step; In initialization procedure, can adjust the height or the angle of camera, make the camera window aim at the center of sign as far as possible; The initial pictures of the sign that then obtains is positioned at the middle part of entire image; So still in the cam lens coverage, the error of calculation is also relatively low simultaneously in its imaging behind the mobile certain distance of sign.
Step S34), the initial position that prestores comprises the sign and the initial pictures of scale, and comparative analysis initial pictures and the image that comprises sign and scale that obtains in real time, calculates the real-time displacement of acquisition execution arm end.
Step S35) carries out the terminal motion of arm based on the real-time displacement control of sign.Step S34) and step S23), step S35) with step S24) and step S14) identical.
Please refer to Fig. 4, Fig. 4 is the sketch map that first kind of concrete account form of the terminal movement technique of arm carried out in control provided by the present invention.
This concrete account form is with sign 10 the initial imaging and the imaging of other imaging and scales 20 constantly Calculation Method of comparing; For ease of calculating sign 10 left and right displacement and front and back displacement (the vertically displacement of short transverse); Can sign 10 horizontal rulers that are set to intersect 201 and vertical scale 202 (cross scale) be equivalent to set up the perpendicular coordinate system.During processing; Can measure the height A 1 of the upper end of sign 10 initial imagings to horizontal ruler 201, the lower end of sign 10 initial imagings is to the height B 1 of horizontal ruler 201, and sign 10 initial imaging one ends are to vertical scale 202 length D1; The angle Q1 of sign 10 initial imagings and horizontal ruler 201; And the sign 10 imaging two ends in a certain moment are respectively A2, B2 to the height of horizontal ruler 201, and one of which end to the vertical length of scale 202 is D2, with the angle of horizontal ruler 201 be Q2; The physical length of sign 10 is L, then can calculate the distance Y that moves up and down of carrying out the terminal initial position relatively of arm through following formula:
Y＝Y1-Y2；
Y1＝L×SINQ1×(A1-B1)÷(A1+B1)；
Y2＝L×SINQ2×(A2-B2)÷(A2+B2)；
This method can also calculate through following formula and carry out the terminal move left and right distance X of initial position relatively of arm:
X＝X1-X2；
X1＝L×SINQ1×D1÷(A1+B1)；
X2＝L×SINQ2×D2÷(A2+B2)。
In fact, above-mentioned angle Q1 and Q2 also can draw through height and the length computation that identifies 10 imagings, correspondingly, through angle and the length of measuring, also can draw the height of imaging, and the aforementioned calculation method can be provided with flexibly.To the required precision of upper and lower displacement and left and right displacement when all higher; Can identify 10 structure and be set to the cross sign; Promptly comprise vertical sign and level sign; These computational methods are identical with the principle of first kind of account form, and the vertical sign imaging that the displacement basis was roughly vertically extended before and after difference only was is calculated, and left and right displacement is calculated according to the level sign imaging of roughly along continuous straight runs extension.
When before and after carrying out the arm end, producing displacement; The size of sign 10 imagings can change; Corresponding width and length all can change; Can do image processing (like processing modes such as the size that forms images, binocular range findings) the calculating front and back displacement of routine separately through imaging law, can also comprehensively take methods such as laser ranging, infrared distance measurement to realize the longitudinal separation measurement.Certainly; When front and back produced displacement, the aforementioned calculation formula also need be done suitable distortion, can measure the length of sign 10 initial imagings; The imaging length in a certain moment; Through the variation relation of two length values, revise the input variable in the above-mentioned formula, and then elimination moves forward and backward the error of calculation of being brought; Certainly, when length changes and influences above certain limit, also can reinitialize program, with new position point as a reference.
Please refer to Fig. 5, Fig. 5 is the sketch map that second kind of concrete account form of the terminal movement technique of arm carried out in control provided by the present invention.
In this account form; Sign 10 is set to grating 101 forms, and the width of every group of grating 101 can require design (such as control accuracy is 0.1 meter, and grating 101 width can be designed to 0.05 meter) according to control accuracy; Certainly; Grating 101 width should be tried one's best widelyer satisfying under the prerequisite of control accuracy, be convenient to imaging analysis, and the horizontal ruler imaging of scale 20 can be held a grating 101.If grating 101 width are A, it is M that grating 101 moves quantity, through the graphical analysis in the initial pictures and a certain moment, can draw the mobile quantity of grating 101, then identifies front and back displacement L=A * M of 10.Sign imaging shown in the figure is roughly vertically extended; When needing the measurement left and right displacement, can sign 10 be set to have level sign and the structure that vertically identifies, and all be made as raster mode; When then calculating left and right displacement, identical with the calculating principle of upper and lower displacement.When before and after carrying out the arm end, producing displacement, can carry out error compensation according to the imaging change width deflection scale position of grating.
Certainly, according to the difference of sign 10 forms, can also obtain by other account forms and carry out the terminal displacement of arm.
In above-mentioned two kinds of account forms, all with the imaging of scale 20 as a reference, in fact, also can in the image processing process in later stage, virtual scale be set.Need to prove; When scale 20 is fixed on other structures; Even camera has slight jitter, it is stable that the angle of the imaging of scale 20 still keeps, so the entity scale of setting 20 is with respect to virtual scale; Can be all the time exactly as the normative reference that identifies 10 image change, so the scale that scale 20 forms references preferably is set.
Please refer to Fig. 6, Fig. 6 is the sketch map that terminal the third the concrete account form of movement technique of arm is carried out in control provided by the present invention.
In this account form, camera is in tracking mode, makes corresponding angular turn with the change of sign 10 positions, the centre position of real-time tracking sign 10.According to the image of taking; In the time of can obtaining initial position; Camera to identify 10 center apart from S1; A certain moment camera to identify 10 centers apart from S2; And can then can calculate the distance Y that moves up and down of carrying out the terminal 500 relative initial positions of arm by the level angle P2, the vertical angle Q2 (, setting up the XYZ three-dimensional system of coordinate among the figure) that are located under level angle P1, vertical angle Q1 and the current state that angular transducer on the camera obtains the camera initial position for ease of understanding through following formula:
Y＝Y1-Y2；
Y1＝S1×SINQ1；
Y2＝S2×SINQ2；
Carry out the move left and right distance X of the terminal 500 relative initial positions of arm:
X＝X1-X2；
X1＝S1×COSQ1×COSP1；
X2＝S2×COSQ2×COSP2；
Carry out moving forward and backward of the terminal 500 relative initial positions of arm apart from Z
Z＝Z1-Z2；
Z1＝S1×COSQ1×SINP1；
Z2＝S2×COSQ2×SINP2；
In the third account form, the distance that identifies between 10 centers and the camera can calculate through the imaging analysis of scale 20 with sign 10, also can obtain through image analysis methods such as dual camera range findings.
In addition, to above-mentioned each embodiment, step S14) in, can also be according to the focal length of the image control camera 100 of the sign of obtaining in real time 10.When predefined focal length possibly can't obtain sign 10 images that satisfy the definition requirement, can control camera 100 zooms, during analytical calculation, the image size variation amount that zoom causes is introduced in the computing formula.
The foregoing description one and example two all build on, and sign 10 is positioned at the viewing area of carrying out arm terminal 500, and the actuating range of execution arm terminal 500 is less.When the construction location of carrying out arm terminal 500 changes, and luffing is when big, promptly carries out terminal 500 viewing areas of arm when changing, and can at least two signs be set in whole working region, selects wherein 1 as calculating with sign 10.The viewing area of carrying out arm terminal 500 refers to carry out the observable any zone of terminal 500 cameras of arm; A plurality of signs are set; Can enlarge the monitoring range of carrying out terminal 500 cameras of arm, guarantee to carry out arm action in any direction, all have the corresponding sign can be for reference; Thereby obtain continuous position signalling, regulation and control wider.
Above account form can calculate the top to bottom, left and right, front and rear displacement of carrying out arm terminal 500 easily, and at initial position, selected certain position is the origin of coordinates, then can set up the three dimensional space coordinate system that carries out arm terminal 500.Monitor the location parameter of carrying out arm terminal 500 in real time according to the displacement of calculating; When needs are carried out arm end 500 conversion work positions; Can be reference with this parameter, accurately control has the optional position that the execution arm terminal 500 of camera moves in the working region.
In fact; In the working region of carrying out arm; A plurality of execution arms terminal 500 possibly work simultaneously (working region of a plurality of execution arms terminal 500 overlaps); A plurality of signs 10 that then can be provided with avoid singular id 10 to be blocked by other execution arm terminal 500, reach the movement locus of controlling a plurality of execution arms terminal 500 simultaneously, realize the purpose that accurate intellectuality is controlled.
Except above-mentioned control method, the present invention also provides a kind of control system that carries out the terminal motion of arm.Please refer to Fig. 7 and Fig. 8, Fig. 7 is the structural representation of a kind of specific embodiment of the terminal kinetic control system of execution arm provided by the present invention; Fig. 8 is the control principle sketch map of control system among Fig. 7.
This control system comprises sign 10, camera, the image processing module that is arranged at terminal 500 work tops of execution arm, and carries out arm controller.
Sign 10 shape is preferably regular shape, is convenient to form images and the analytical calculation of successive image.Such as; Sign 10 shown in Fig. 6 is shaft-like, can know according to foregoing, and sign 10 can be raster mode; Also the color that be different from the upper and lower can be set at the middle part of rod-like members, or with the two ends of rod-like members be provided with other colors with mid portion as sign 10.The preferred mid portion that adopts is a white; Other parts are black, or mid portion is black, and other parts are white; Also can be other color combination with obvious difference; Possess eye-catching effect, state imaging and graphical analysis that step is mentioned after being convenient to, prevent that promptly background colour from causing imaging unintelligible with sign 10 color similarities.
In order to enlarge to carrying out the control range of terminal 500 motions of arm; Arm terminal 500 at least two signs 10 of working region setting all around can also carried out; Make and carry out arbitrary position that arm terminal 500 moves in the working region; All having at least one corresponding sign 10 includes in the viewing area of carrying out terminal 500 cameras of arm, for your guidance.When choosing the sign 10 of diverse location, can the position residual quantity between the different identification 10 be integrated with in the calculation procedure of image processing module 200, make the referential of graphical analysis keep unified; Or when carrying out arm and getting in the scope that is different from previously selected sign 10, image processing module 200 is reinitialized, with the image of the sign in the job area 10 as the initial pictures of selecting again; Or can know by said method; A plurality of signs 10 are set; Can set up the three dimensional space coordinate system that carries out arm terminal 500; Accurately the optional position that arm terminal 500 moves in the working region is carried out in control, in the working region, has a plurality of execution arms at terminal 500 o'clock, can set up the execution arm control system of scale.
Sign is set more than one at 10 o'clock at the place, viewing area that carries out arm terminal 500; The camera 100 of carrying out arm terminal 500 can obtain the image near the sign 10 of camera 100 window center; So that camera 100 selects suitable sign 10 as tracking target according to the imaging definition and the size of sign 10.
At initial position; Can adjust the relative position of camera 100 and sign 10; Its advantage is existing the argumentation in control method (be primarily aimed in first and second kinds of computational methods, sign 10 be adjusted to the center of camera 100), correspondingly; The control module 400 of control camera 100 can be set, the variation of control camera 100 height or angle.The control module 400 of camera 100 can be connected with execution arm controller 300 or image processing module 200; The latter two all can be according to the sign of obtaining 10 images; Judgement is at initial position; Whether the image of sign 10 is suitable as initial pictures, and sends the control module 400 of control signal corresponding to camera 100, by the height and the angle of control module 400 control cameras 100.
Carry out arm controller 300, in order to receiving the realtime graphic of the sign 10 that image processing module 200 obtains, calculate the real-time displacement of carrying out arm terminal 500 according to the initial pictures and the realtime graphic of sign 10, and then the motion of arm end 500 is carried out in control.Carry out arm controller 300 and also can be connected through wired or wireless mode with image processing module 200, the two also can be integrated on the same module.
The control system of terminal 500 motions of execution arm then provided by the present invention; The image that relatively identifies 10 initial pictures and obtain in real time by image processing module 200 direct analyses; Obtain the real-time displacement of carrying out the terminal 500 relative initial positions of arm based on variable quantity, and carry out the instruction of arm motion by carrying out the 300 output controls of arm motion controller.With respect to utilizing a plurality of sensor data signal stack calculation mode in the prior art, this control system possesses higher control accuracy.
Further, the control system can also comprise the scale 20 of the window near-end that is arranged at camera 100, cross scale as shown in Figure 6, and scale 20 is constant with the position relation maintenance of camera 100.Comprise sign 10 and scale 20 in the image that then camera 100 obtains, and under the constant situation of focal length, the imaging of scale 20 size keeps constant.During initialization; Image processing module 200 initial position that prestores comprises the sign 10 and the initial pictures of scale 20; And the image that comprises sign 10 and scale 20 that analysis is obtained in real time according to initial pictures, calculate the real-time displacement that obtains to carry out the terminal 500 relative initial positions of arm.The operation principle of scale 20 and the technique effect of generation are also discussed in the embodiment of control method, do not give unnecessary details at this.
The control module 400 of above-mentioned camera 100 can also be controlled the focal length of camera 100 according to the image of the sign of obtaining 10.In the process of carrying out terminal 500 motions of arm, all can control the zoom of control module 400, the variation of sign 10 that focal length variations causes or scale 20 imagings will count in the middle of the computing formula.
In control method, the concrete account form with regard to image processing module is described, can be with reference to understanding.
In addition, operation during for ease of night or insufficient light, sign 10 can be processed by reflectorized material, so that image recognition.And, can also be set to identify 10 illumination is provided illuminating lamp, then under the effect of illuminating lamp, be convenient to obtain and identify 10 images clearly.Preferably, with illuminating lamp place the sign 10 near.
In addition, gather the near-end working condition chart picture of carrying out terminal 500 working regions of arm, this image is shown on the display unit of engineering machinery, can realize that engineering machinery is carried out the arm end carries out long-range operating mode observation through camera 100.Further analyze through the image of 200 pairs of execution of image processing module arm terminal 500; When carrying out arm terminal 500 near certain large-scale barrier when wall (for example vertical); Image processing module 200 produces warning information and sends to carries out arm controller 300; Carry out arm controller 300 controls and stop to carry out the motion of arm or revising motion path, thereby can realize carrying out the intelligent barrier avoiding of arm terminal 500.The associated picture analysis and processing method is well known to those skilled in the art, and repeats no more at this.
More than a kind of method and control system of carrying out the terminal motion of arm of controlling provided by the present invention is described in detail.Used specific case herein principle of the present invention and embodiment are set forth, the explanation of above embodiment just is used for helping to understand method of the present invention and core concept thereof.Should be pointed out that for those skilled in the art, under the prerequisite that does not break away from the principle of the invention, can also carry out some improvement and modification to the present invention, these improvement and modification also fall in the protection domain of claim of the present invention.
Claims (15)
1. control the method for carrying out the terminal motion of arm for one kind, it is characterized in that, may further comprise the steps:
11) in said execution arm end camera is set, and sign is set in the viewing area of said camera;
12) said camera obtains the image of said sign in real time;
13) the prestore initial pictures of the said sign of initial position, the image of said initial pictures of comparative analysis and the said sign obtained in real time calculates and obtains the terminal real-time displacement of initial position relatively of said execution arm;
14) control the terminal motion of said execution arm based on the terminal real-time displacement of said execution arm.
2. the method for the terminal motion of arm is carried out in control according to claim 1, it is characterized in that,
In the step 11), also the window near-end at said camera is provided with scale, and it is constant that the position of the said relatively camera of said scale keeps;
In the step 12), said camera obtains the image that comprises said sign and said scale;
In the step 13); Prestore and comprise the initial pictures of said sign and said scale under the initial position; The image that comprises said sign and said scale that the said initial pictures analysis of comparative analysis is obtained in real time calculates and obtains the terminal real-time displacement of initial position relatively of said execution arm.
3. the method for the terminal motion of arm is carried out in control according to claim 2, it is characterized in that,
In the step 11), the horizontal ruler and vertical scale of the said scale of setting for intersecting;
In the step 13); With the sign in the initial pictures and obtain in real time in the image sign all with the horizontal ruler of corresponding said scale and vertically the imaging of scale compare analysis, calculate and obtain said execution the arm terminal left and right displacement and the upper and lower displacement of initial position relatively.
4. the method for the terminal motion of arm is carried out in control according to claim 2, it is characterized in that,
In the step 11), on said camera, angular transducer is set;
In the step 12), said camera is followed the tracks of the center of said sign, and obtains the image of said sign;
In the step 13); Write down the level angle and the vertical angle of the said camera of said angular transducer detection; Said initial pictures of comparative analysis and the identification image that obtains in real time; Obtain said the camera extremely initial distance and real-time distance of said sign center, obtain the real-time displacement of the terminal relative initial position of said execution arm according to initial distance, distance, initial level angle, initial vertical angle, real-time level angle and vertical angle calculating in real time in real time.
5. the terminal method of moving of arm is carried out in each described control according to claim 1 to 3, it is characterized in that between step 12) and step 13), also having following steps:
123) the adjustment camera makes the center of aiming at said sign in the window center of the said camera of initial position.
6. the terminal method of moving of arm is carried out in each described control according to claim 1 to 4, it is characterized in that, controls the focal length of said camera according to the image of the said sign of obtaining.
7. the terminal method of moving of arm is carried out in each described control based on claim 1 to 4, it is characterized in that, in the step 11), the working region around said execution arm end is provided with at least two signs.
8. a control system that carries out the terminal motion of arm is characterized in that, comprising:
Camera, it is terminal to be located at said execution arm;
Be arranged at the sign of said camera viewing area, said camera obtains the image of said sign;
Image processing module; Said image processing module receives the image that said camera obtains; And the initial pictures of the said sign of initial position that prestores, and the image of said initial pictures of compare of analysis and the said sign obtained in real time calculate and obtain the terminal real-time displacement of said execution arm;
Carry out arm controller, receive the terminal real-time displacement of said execution arm that said image processing module obtains, carry out the terminal motion of arm based on the real-time displacement control that said execution arm is terminal.
9. the control system of the terminal motion of execution arm according to claim 8 is characterized in that,
The scale that also comprises the window near-end that is arranged at said camera; Said camera obtains the image that comprises said sign and said scale; The said image processing module initial position that prestores comprises the initial pictures of said sign and said scale, and said initial pictures of compare of analysis and the image that comprises said sign and said scale that obtains in real time, calculates and obtains the terminal real-time displacement of said execution arm.
10. based on the control system of the described execution arm end of claim 9 motion, it is characterized in that the horizontal ruler and vertical scale of said scale for intersecting; Said image processing module with the sign in the initial pictures and obtain in real time in the image sign all with the horizontal ruler of corresponding said scale and vertically the imaging of scale compare analysis, calculate and obtain said execution the arm terminal left and right displacement and the upper and lower displacement of initial position relatively.
11. the control system of the terminal motion of execution arm according to claim 8; It is characterized in that; The control module that also comprises camera; When being positioned at initial position, said control module is adjusted said camera according to the image of the said sign of obtaining, and makes the window center of said camera aim at the center of said sign.
12. the control system of the terminal motion of execution arm according to claim 11 is characterized in that said camera is provided with angular transducer; Said camera is followed the tracks of the center of said sign, and obtains the image of said sign; Said image processing module writes down the level angle and the vertical angle of the said camera of said angular transducer detection; And said initial pictures of comparative analysis and the identification image that obtains in real time; Obtain said the camera extremely initial distance and real-time distance of said sign center, obtain the terminal real-time displacement of said execution arm according to initial distance, distance, initial level angle, initial vertical angle, real-time level angle and vertical angle calculating in real time in real time.
13. the control system of the terminal motion of execution arm according to claim 12 is characterized in that, said control module is controlled the focal length of said camera according to the image of the said sign of obtaining.
14. to the control system of the terminal motion of 13 each described execution arms, it is characterized in that the working region around the said execution arm end is provided with at least two signs according to Claim 8.
15. to the control system of the terminal motion of 13 each described execution arms, it is characterized in that said sign is processed by reflectorized material according to Claim 8; And be set to identify the illuminating lamp that illumination is provided.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103016120A CN102501252A (en) | 2011-09-28 | 2011-09-28 | Method and system for controlling movement of tail end of executing arm |
| PCT/CN2012/074304 WO2013044626A1 (en) | 2011-09-28 | 2012-04-18 | System and method for controlling movement of tail end of execution arm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2011103016120A CN102501252A (en) | 2011-09-28 | 2011-09-28 | Method and system for controlling movement of tail end of executing arm |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102501252A true CN102501252A (en) | 2012-06-20 |
Family
ID=46213415
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2011103016120A Pending CN102501252A (en) | 2011-09-28 | 2011-09-28 | Method and system for controlling movement of tail end of executing arm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102501252A (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102897696A (en) * | 2012-10-18 | 2013-01-30 | 三一重工股份有限公司 | Elevating fire truck boom control system, elevating fire truck boom control method and elevating fire truck |
| CN103195251A (en) * | 2013-03-28 | 2013-07-10 | 中联重科股份有限公司 | Method, device and system for controlling motion of tail end of boom and multi-section boom vehicle |
| CN103663149A (en) * | 2013-12-12 | 2014-03-26 | 中联重科股份有限公司 | Crane jib tail end position detection device, method and system and crane |
| CN104802174A (en) * | 2013-10-10 | 2015-07-29 | 精工爱普生株式会社 | Robot control system, robot, program and robot control method |
| CN105599241A (en) * | 2016-01-13 | 2016-05-25 | 重庆世纪精信实业（集团）有限公司 | Automatic alignment method and device based on image recognition technology for manipulator |
| WO2017045468A1 (en) * | 2015-09-18 | 2017-03-23 | 中兴通讯股份有限公司 | Method and device for correcting position error |
| CN108175504A (en) * | 2017-11-27 | 2018-06-19 | 北京理工大学 | Blood vessel interventional operation conduit/seal wire axial direction operative position moves detection device and method |
| TWI652153B (en) | 2018-01-05 | 2019-03-01 | 均豪精密工業股份有限公司 | Mechanical arm device and mechanical arm device control method |
| CN109855568A (en) * | 2019-01-16 | 2019-06-07 | 北京百度网讯科技有限公司 | Detection method, device, electronic equipment and the storage medium of automatic Pilot sensor |
| CN110834320A (en) * | 2018-08-16 | 2020-02-25 | 株式会社三丰 | Auxiliary measurement position coordinate determination system for use with a robot |
| CN110834322A (en) * | 2018-08-16 | 2020-02-25 | 株式会社三丰 | Robot system with auxiliary measuring position coordinate determination system |
| CN110936373A (en) * | 2018-09-24 | 2020-03-31 | 株式会社三丰 | Robot system with end tool metrology position coordinate determination system |
| CN111498474A (en) * | 2020-03-13 | 2020-08-07 | 广东九联科技股份有限公司 | Control system and method for taking and placing module |
| CN111671615A (en) * | 2020-06-01 | 2020-09-18 | 上海电气集团股份有限公司 | Training method and system of rehabilitation robot and storage medium |
| CN112191354A (en) * | 2020-10-09 | 2021-01-08 | 矿冶科技集团有限公司 | Control method, device, system, control equipment and computer readable storage medium |
| CN113120801A (en) * | 2019-12-31 | 2021-07-16 | 锥能机器人（上海）有限公司 | Forklift control method, forklift control device, and computer-readable storage medium |
| CN114147769A (en) * | 2021-12-21 | 2022-03-08 | 乐聚(深圳)机器人技术有限公司 | Factory detection method, device, equipment and storage medium for robot |
| US11745354B2 (en) | 2018-08-16 | 2023-09-05 | Mitutoyo Corporation | Supplementary metrology position coordinates determination system including an alignment sensor for use with a robot |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2669257A1 (en) * | 1990-11-21 | 1992-05-22 | Renault Automatique | Method for calibrating a robot |
| JPH07248209A (en) * | 1994-03-11 | 1995-09-26 | Natl Space Dev Agency Japan<Nasda> | Object position and attitude measuring device and part assembling apparatus loading the device |
| JP3155455B2 (en) * | 1995-12-11 | 2001-04-09 | 三菱電機株式会社 | Work positioning device |
| US7532949B2 (en) * | 2004-01-16 | 2009-05-12 | Fanuc Ltd | Measuring system |
| CN101633168A (en) * | 2009-07-28 | 2010-01-27 | 三一重工股份有限公司 | Control method and control system of large engineering manipulator |
-
2011
- 2011-09-28 CN CN2011103016120A patent/CN102501252A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2669257A1 (en) * | 1990-11-21 | 1992-05-22 | Renault Automatique | Method for calibrating a robot |
| JPH07248209A (en) * | 1994-03-11 | 1995-09-26 | Natl Space Dev Agency Japan<Nasda> | Object position and attitude measuring device and part assembling apparatus loading the device |
| JP3155455B2 (en) * | 1995-12-11 | 2001-04-09 | 三菱電機株式会社 | Work positioning device |
| US7532949B2 (en) * | 2004-01-16 | 2009-05-12 | Fanuc Ltd | Measuring system |
| CN101633168A (en) * | 2009-07-28 | 2010-01-27 | 三一重工股份有限公司 | Control method and control system of large engineering manipulator |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102897696A (en) * | 2012-10-18 | 2013-01-30 | 三一重工股份有限公司 | Elevating fire truck boom control system, elevating fire truck boom control method and elevating fire truck |
| CN102897696B (en) * | 2012-10-18 | 2015-03-18 | 三一重工股份有限公司 | Elevating fire truck boom control system, elevating fire truck boom control method and elevating fire truck |
| CN103195251A (en) * | 2013-03-28 | 2013-07-10 | 中联重科股份有限公司 | Method, device and system for controlling motion of tail end of boom and multi-section boom vehicle |
| CN103195251B (en) * | 2013-03-28 | 2015-06-10 | 中联重科股份有限公司 | Method, device and system for controlling motion of tail end of boom and multi-section boom vehicle |
| CN104802174B (en) * | 2013-10-10 | 2016-09-07 | 精工爱普生株式会社 | Robot control system, robot, program and robot control method |
| CN104802174A (en) * | 2013-10-10 | 2015-07-29 | 精工爱普生株式会社 | Robot control system, robot, program and robot control method |
| CN103663149B (en) * | 2013-12-12 | 2015-08-05 | 中联重科股份有限公司 | Crane jib tail end position detection device, method and system and crane |
| CN103663149A (en) * | 2013-12-12 | 2014-03-26 | 中联重科股份有限公司 | Crane jib tail end position detection device, method and system and crane |
| WO2017045468A1 (en) * | 2015-09-18 | 2017-03-23 | 中兴通讯股份有限公司 | Method and device for correcting position error |
| CN106558077A (en) * | 2015-09-18 | 2017-04-05 | 中兴通讯股份有限公司 | The antidote and device of site error |
| CN105599241A (en) * | 2016-01-13 | 2016-05-25 | 重庆世纪精信实业（集团）有限公司 | Automatic alignment method and device based on image recognition technology for manipulator |
| CN108175504A (en) * | 2017-11-27 | 2018-06-19 | 北京理工大学 | Blood vessel interventional operation conduit/seal wire axial direction operative position moves detection device and method |
| CN108175504B (en) * | 2017-11-27 | 2023-09-05 | 深圳市爱博医疗机器人有限公司 | Vascular intervention operation catheter/guide wire axial operation displacement detection device and method |
| TWI652153B (en) | 2018-01-05 | 2019-03-01 | 均豪精密工業股份有限公司 | Mechanical arm device and mechanical arm device control method |
| CN110834320A (en) * | 2018-08-16 | 2020-02-25 | 株式会社三丰 | Auxiliary measurement position coordinate determination system for use with a robot |
| CN110834322B (en) * | 2018-08-16 | 2023-02-28 | 株式会社三丰 | Robot system with auxiliary measuring position coordinate determination system |
| US11745354B2 (en) | 2018-08-16 | 2023-09-05 | Mitutoyo Corporation | Supplementary metrology position coordinates determination system including an alignment sensor for use with a robot |
| CN110834322A (en) * | 2018-08-16 | 2020-02-25 | 株式会社三丰 | Robot system with auxiliary measuring position coordinate determination system |
| CN110834320B (en) * | 2018-08-16 | 2023-02-24 | 株式会社三丰 | Auxiliary measurement position coordinate determination system for use with a robot |
| CN110936373A (en) * | 2018-09-24 | 2020-03-31 | 株式会社三丰 | Robot system with end tool metrology position coordinate determination system |
| CN110936373B (en) * | 2018-09-24 | 2023-02-28 | 株式会社三丰 | Robot system with end tool metrology position coordinate determination system |
| WO2020147498A1 (en) * | 2019-01-16 | 2020-07-23 | 北京百度网讯科技有限公司 | Detection method and apparatus for automatic driving sensor, and electronic device |
| US11933604B2 (en) | 2019-01-16 | 2024-03-19 | Apollo Intelligent Driving Technology (Beijing) Co., Ltd. | Detection method and apparatus for automatic driving sensor, and electronic device |
| CN109855568A (en) * | 2019-01-16 | 2019-06-07 | 北京百度网讯科技有限公司 | Detection method, device, electronic equipment and the storage medium of automatic Pilot sensor |
| CN113120801A (en) * | 2019-12-31 | 2021-07-16 | 锥能机器人（上海）有限公司 | Forklift control method, forklift control device, and computer-readable storage medium |
| CN111498474A (en) * | 2020-03-13 | 2020-08-07 | 广东九联科技股份有限公司 | Control system and method for taking and placing module |
| CN111671615A (en) * | 2020-06-01 | 2020-09-18 | 上海电气集团股份有限公司 | Training method and system of rehabilitation robot and storage medium |
| CN112191354A (en) * | 2020-10-09 | 2021-01-08 | 矿冶科技集团有限公司 | Control method, device, system, control equipment and computer readable storage medium |
| CN114147769A (en) * | 2021-12-21 | 2022-03-08 | 乐聚(深圳)机器人技术有限公司 | Factory detection method, device, equipment and storage medium for robot |
| CN114147769B (en) * | 2021-12-21 | 2024-06-11 | 乐聚(深圳)机器人技术有限公司 | Method, device, equipment and storage medium for factory detection of robot |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102501252A (en) | Method and system for controlling movement of tail end of executing arm | |
| US10171802B2 (en) | Calibration method and calibration device | |
| US9300954B2 (en) | Surrounding information-obtaining device for working vehicle | |
| KR102032070B1 (en) | System and Method for Depth Map Sampling | |
| CN101832760B (en) | Remote three-dimensional micro-deformation visual on-line monitoring method and system | |
| CN102511162B (en) | Wide-angle camera apparatus and Analytical system | |
| US11906305B2 (en) | Movable marking system, controlling method for movable marking apparatus, and computer readable recording medium | |
| CN105136058B (en) | The on-line proving device and its scaling method of laser sensing three-dimension measuring system | |
| EP3113146A1 (en) | Local location computation device and local location computation method | |
| US20150377606A1 (en) | Projection system | |
| JP6557896B2 (en) | Radar axis deviation amount calculation device and radar axis deviation amount calculation method | |
| CN107843202B (en) | Gap and face difference vision measurement device and method matched with robot | |
| US20170026636A1 (en) | Method for the positionally accurate projection of a mark onto an object, and projection apparatus | |
| CN102508495B (en) | Method and system for controlling motion of tail end of execution arm | |
| CN104268882A (en) | High-speed moving object detecting and speed measuring method and system based on double-linear-array cameras | |
| CN110944927A (en) | Guide information display device, crane having the same, and guide information display method | |
| CN112967344A (en) | Method, apparatus, storage medium, and program product for camera external reference calibration | |
| WO2018169467A1 (en) | A vehicle with a crane with object detecting device | |
| WO2018160119A1 (en) | A vehicle provided with an arrangement for determining a three dimensional representation of a movable member | |
| JP6080450B2 (en) | Surveillance camera device | |
| KR102498028B1 (en) | Surveillance Camera Systems and Mothod of Using the Same | |
| JP2020118575A (en) | Inter-vehicle distance measurement device, error model generation device, learning model generation device, and method and program thereof | |
| Liu et al. | Parallel camera network: Motion-compensation vision measurement method and system for structural displacement | |
| JPH06347220A (en) | Image monitoring device and its usage | |
| KR20160149843A (en) | Apparatus for registration of scan data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20120620 |