JP7453126B2 - Work observation system, work analysis method, and educational support system - Google Patents

Description

The present invention relates to a work observation system.

Qualitative guidance is the main focus in training workers who require skilled techniques, such as fine processing of precision equipment, and the training involves repeating trial and error while checking the results of the technique to help them acquire the technique. The quality of work has been improved.

As background art in this technical field, there are JP2012-198644A (Patent Document 1), JP2011-134224A (Patent Document 2), and JP2017-71033A (Patent Document 3).

Japanese Unexamined Patent Publication No. 2012-198644 discloses a storage unit that stores a first series of work image information regarding one work section, and an acquisition section that acquires a second series of work image information regarding the one work section from a camera. , the first series of work image information is obtained by comparing the first series of work image information stored in the storage unit and the second series of work image information acquired by the acquisition unit. a determination unit that determines the quality of the work at the time of photographing the second series of work image information for the work at the time of photography; A work support device is described that includes an updating section that updates to a second series of work image information, and an output section that outputs the update result by the updating section.

Japanese Unexamined Patent Publication No. 2011-134224 discloses an assembly work support system that supports workers in assembly work, which includes a work scenario recording means for recording a work scenario, and a work scenario recorder for recording a work scenario. A working time measuring means for measuring working time, a video capturing the work from the shortest working time measured by the working time measuring means to a predetermined order, and the working time measuring means measuring the working time. An assembly work support system is described, which is characterized by comprising a video recording means for recording a video of the work taken from the longest work time to a predetermined order.

JP 2017-71033A discloses a method of positioning an index in a working reference object that is used in a robot device having a robot arm and has an index that can be optically detected and adjust the coordinate system used to control the robot arm. It is characterized by comprising a measuring step of measuring displacement information, and a recording step of converting the positional displacement information measured in the measuring step into an optically readable information image and recording it on the work reference object. A method for manufacturing a reference object for work is described.

Japanese Patent Application Publication No. 2012-198644 Japanese Patent Application Publication No. 2011-134224 JP 2017-71033 Publication

Qualitative guidance is the main focus in training workers who require skilled techniques, such as fine processing of precision equipment, and the training involves repeating trial and error while checking the results of the technique to help them acquire the technique. ing. Therefore, in the case of skilled techniques such as micro-machining, the worker receiving the instruction may not be able to directly observe the movements of the skilled worker who is instructing the instructor during micro-machining or the state of the workpiece. This can take several months or more, so it has been desired to quickly improve the skills of workers and improve productivity.

Although the above-mentioned prior art can support the training of skilled workers, the work support device described in Japanese Patent Application Laid-open No. 2012-198644 determines a better video to serve as a model video for the worker and updates it. Furthermore, in the assembly work support system described in Japanese Patent Application Laid-open No. 2011-134224, a better model video for the worker is determined and updated. Furthermore, in the method for manufacturing a work reference object described in Japanese Patent Application Laid-open No. 2017-71033, coordinates of a marker at the tip of a robot arm are acquired using a fixed camera to determine positional deviation, but cameras are placed on the fixed camera and the robot arm. As described above, none of the prior art techniques takes into consideration the early development of skilled techniques for fine processing.

The present invention aims to provide a work observation system that trains workers more quickly than before and levels out work quality.

A typical example of the invention disclosed in this application is as follows. That is, it is a work observation system that images a state in which a worker works in a work area using a tool, and includes a first imaging device that images the work area from a first direction to obtain a first image. a second imaging device that images the work area from a second direction to obtain a second image; and an analysis device that analyzes the work using the first image and the second image. The analysis device is characterized in that the analysis device calculates at least feature quantities of the position and orientation of the tool reflected in the second image, and outputs the feature quantities of the position and orientation of the tool in the work. .

According to one aspect of the present invention, work quality can be improved. Problems, configurations, and effects other than those described above will be made clear by the description of the following examples.

FIG. 1 is a diagram showing the configuration of a work observation system. It is a diagram showing the arrangement of each device of the work observation system. It is a figure showing an example of composition of a work. It is a figure showing an example of composition of a tool. FIG. 2 is a diagram showing the physical configuration of an analysis device. FIG. 2 is a diagram showing a logical configuration of an analysis device. It is a flowchart of calibration processing. FIG. 3 is a diagram showing an example of a calibration screen. FIG. 3 is a diagram showing an example of a calibration screen. It is a flowchart of measurement processing. It is a figure showing an example of a measurement screen. It is a figure showing an example of a measurement screen. It is a figure which shows the example of an analysis result display screen. It is a figure which shows the example of an analysis result display screen. It is a figure which shows the example of an analysis result display screen.

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing the configuration of a work observation system according to an embodiment of the present invention, and FIG. 2 is a diagram showing the arrangement of each device in the work observation system.

First, an overview of the work observation system of this embodiment will be explained. Side cameras 22 and 23 that take stereo images are installed at the hands of the worker, and two minute second markers 52A and 52B attached to the tool 5 held by the worker are placed on the two side cameras 22 and 23. Shoot so that the images appear at the same time. The microscope camera 20 is attached to the lens barrel of the microscope 2 used by the worker, and captures an image that the worker visually recognizes during work, that is, the workpiece 3 to which a plurality of markers 33A to 33D are attached.

In pre-work calibration, the geometric relationship between the images taken by the microscope camera 20 and the side cameras 22 and 23 is obtained. The three-dimensional positions of the markers 52A, 52B of the tool 5 are measured by the side cameras 22, 23, and the position of the tip 51 of the tool 5 is calculated. Then, the calculated tip position is converted into the coordinates of the microscope camera 20 (S115). Further, the markers 33A to 33D on the workpiece 3 are measured with the microscope camera 20 to obtain minute movements of the tool 5 with respect to the workpiece 3.

Next, details of the work observation system of this embodiment will be explained. The work observation system of this embodiment includes a microscope camera 20 that photographs the work area 1, side cameras 22 and 23 that photographs the work area 1, and a signal generator that supplies synchronization signals to the plurality of cameras 20, 22, and 23. 24, an analysis device 10 that performs work analysis, a display device 27 and a speaker 28 connected to the analysis device 10.

The microscope camera 20 magnifies and photographs the work area 1 and transmits the photographed image to the analysis device 10. A microscope 2 equipped with a microscope camera 20 has an eyepiece 21 through which an operator can view the work area 1. Thereby, the microscope camera 20 can capture the same image as the image viewed by the operator.

In the work area 1, a jig 4 on which a workpiece 3 to be processed is placed is installed. The operator processes the workpiece 3 using the tool 5. The microscope 2 is connected via a microscope support 6 to a jig 4 placed on a workbench.

In an example of the workpiece 3, as shown in FIG. 3, a circular portion 32 is attached to a main body portion 31 having a rectangular outer shape. The circular portion 32, which is a circular electronic component, is a component that needs to be electrically connected to the main body portion 31. The circular portion 32 is provided with an electrode portion 300a, and the main body portion 31 is provided with an electrode portion 300b. A wiring section 300c is connected to the electrode section 300b, and the connection destination of the wiring section 300c is not shown in FIG. 3.

The electrode part 300a and the electrode part 300b have corresponding shapes, and the electrode part 300b is generally larger than the electrode part 300a so that the solder after reflow can easily connect the electrode part 300a and the electrode part 300b. The electrode parts 300a and 300b may have shapes other than those shown. Further, if the electrode portion 300a is arranged so as to extend not only to the top surface of the circular portion 32 but also to the side surface thereof, the connection between the electrode portion 300a and the electrode portion 300b becomes stronger. Note that from FIG. 8 onward, illustrations of the electrode portions 300a, 300b, and 300c provided on the circular portion 32 and the main body portion 31 are omitted.

First, the operator applies solder paste to the electrode portion 300b. After that, the operator temporarily determines the positions of the electrode part 300a and the electrode part 300b on the main body part 31, and arranges the circular part 32. Next, the operator shapes the solder paste using the tool 5 so that it has an appropriate thickness and application amount. When reflowing the solder paste that has been applied and shaped using the tool 5, the circular portion 32, which is an electronic component, may move or tilt depending on the amount and thickness of the solder paste, so it is necessary to use an appropriate amount of solder paste. .

In addition, since there are variations in the shape and thickness of the circular part 32 and the electrode part 300a, which are electronic components, the operator should 5 Use to stretch and shape the solder paste. The tip portion 51 of the tool 5 may have a thin shape, or may be a member made of rubber or resin such as a squeegee that has a flat tip and is deformable. Although the circular portion 32 has been described as a representative electronic component, the circular portion 32 may have another shape or may include a plurality of electrode portions 300a. Further, the circular portion 32 may be a substrate having a plurality of electronic components.

As another example, the electrode part 300a and the electrode part 300b may be directly soldered without providing a solder paste and a reflow process. In this case, the positions of one set of electrode parts 300a and 300b can be temporarily determined by soldering, and the other set of electrode parts 300a and 300b can be aligned by soldering.

Next, a method for recognizing markers will be explained. The circular portion 32 is provided with a plurality of markers 33A to 33D. It is preferable that the markers 33A to 33D have a color or shape (for example, a square of a predetermined color) that is easy to recognize with a camera in the work area 1. For example, by using a color different from the background of the work area 1, the visibility of the markers 33A to 33D in the work area 1 with a camera can be improved. The illustrated workpiece 3 is provided with four markers 33A to 33D, but if at least two markers are provided, positional deviations and rotations of the main body portion 31 and the circular portion 32 can be measured. Moreover, by making each marker 33A to 33D distinguishable (for example, having a different color and/or shape), it is possible to measure the positional shift and rotation of the main body portion 31 and the circular portion 32. Further, the electrode portion 300a or the electrode portion 300b may be used as a marker.

As another example, the circular part 32 and the main body part 31 are formed of resin, and the main body part 31 is provided with a convex part (not shown) in the circular part 32 and a concave part corresponding to the convex part of the circular part 32. A case will be explained in which the structure is such that the concavo-convex portions are fitted together.

When the resin members of the circular portion 32 and the main body portion 31 are manufactured by injection molding, layered manufacturing, etc., the fine uneven portions may not fit together due to burrs or heat shrinkage of the uneven portions. In such a case, a scribing needle, an end mill, a cutter, or the like is used as the tool 5 to cut or deform the uneven portions of the circular portion 32 and the main body portion 31 to perform fine processing so that they can be fitted together.

As shown in FIG. 4, the tool 5 includes a handle 53 to which a tip 51 is attached. A plurality of markers 52A and 52B are attached to the tip portion 51. The markers 52A and 52B have colors and shapes (for example, white spherical shapes) that are easy to recognize with a camera in the work area 1. The markers 52A, 52B and the tip of the tip portion 51 are recognized in a calibration process described later, and their positional relationship is measured. Even if the mounting positions of the markers 52A and 52B differ depending on the tool 5, by measuring their positional relationship, the distance between the markers 52A and 52B and the distance between the marker 52A and the tip of the tip portion 51 can be known. . Then, using the measurement results in the calibration process, the tip position of the tip portion 51 and the inclination of the tip portion 51 can be calculated from the positions of the markers 52A and 52B measured during the work.

The side cameras 22 and 23 photograph the markers 52A and 52B in the work area 1, that is, at the worker's hand, from a direction different from that of the microscope camera 20. The side camera 22 and the side camera 23 constitute a stereo camera that photographs the work area 1 from different directions to photograph a stereo image. That is, by photographing the markers 52A, 52B with the side cameras 22, 23, a three-dimensional image including distance information of the markers 52A, 52B is obtained. The side cameras 22 and 23 may be configured integrally and configured to output a distance image including image information and distance information.

The side cameras 22 and 23 are preferably installed at the same distance from the work area 1. For example, the side cameras 22 and 23 are arranged on a frame so that they can move freely in the horizontal direction, or arranged so that they can move on an arcuate rail centered on the work area 1. Further, the side camera 22 may be installed so that its position in the vertical direction can be changed (for example, on a platform with an elevator). Further, the side camera 22 may be installed on a pan head that can swing its head up and down and left and right, so that the shooting direction can be changed. The pan head may simply be a pedestal, a tripod, or the like.

The signal generator 24 supplies synchronization signals to the plurality of cameras 20, 22, 23. The multiple cameras 20, 22, and 23 start capturing moving images using a synchronization signal outputted from the signal generator 24 as a trigger. Further, the signal generator 24 may provide a common time to the plurality of cameras 20, 22, and 23, and each camera 20, 22, and 23 may capture a moving image at a synchronized time. Furthermore, one camera may supply synchronization signals to other cameras without providing the signal generator 24. The synchronization signal in this case may be a moving image shooting start trigger or a common time. Although it is not necessarily necessary to synchronize each camera, by synchronizing each camera, the movements of the operator and the tip portion 51 can be detected with higher accuracy.

Each camera 20, 22, 23 and the analysis device 10 are preferably connected through a network, but may also be connected through a dedicated cable. Also, it is preferable to connect the signal generator 24 and each camera 20, 22, 23 with a dedicated cable to avoid delays in signal transmission, but if a protocol that is not affected by delays is used, the network May be connected with

As shown in FIG. 5, the analysis device 10 is constituted by a computer having a processor (CPU) 11, a memory 12, an auxiliary storage device 13, a communication interface 14, an input interface 15, and an output interface 18.

Processor 11 executes a program stored in memory 12. Specifically, when the processor 11 executes the program, an image acquisition section 41, a calibration section 42, a tool position estimation section 43, a deviation determination section 44, and an output section 45, which will be described later, are realized. . Note that a part of the processing performed by the processor 11 by executing the program may be performed by using other types of (for example, hardware) arithmetic devices (for example, FPGA (Field Programable Gate Array), ASIC (Application Specific Integrated Circuit), CPLD ( Complex Programmable Logic Device), etc.).

The memory 12 includes a ROM that is a nonvolatile storage element and a RAM that is a volatile storage element. The ROM stores programs (eg, BIOS) and the like. The RAM is a high-speed and volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores programs executed by the processor 11 and data used during execution of the programs.

The auxiliary storage device 13 is a large-capacity, non-volatile storage device such as a magnetic storage device (HDD: Hard Disk Drive), flash memory (SSD: Solid State Drive), etc., and stores programs and programs executed by the processor 11. Stores data used at runtime. That is, the program is read from the auxiliary storage device 13, loaded into the memory 12, and executed by the processor 11. The auxiliary storage device 13 may be provided inside the analysis device 10 or may be a storage device provided separately from the analysis device 10.

The communication interface 14 is a network interface device that controls communication with other devices (microscope camera 20, side cameras 22, 23, etc.) according to a predetermined protocol.

The input interface 15 is an interface to which a keyboard 16, a mouse 17, etc. are connected, and receives input from an operator. The output interface 18 is an interface to which a display device 27, a speaker 28, etc. are connected, and outputs the results of program execution in a format that can be viewed by the operator. The input interface 15 and the output interface 18 may be tablet terminals in which input means using a touch panel and output means capable of displaying a screen using a liquid crystal or the like are integrated.

The program executed by the processor 11 is provided to the analysis device 10 via a removable medium (CD-ROM, flash memory, etc.) or a network, and is stored in a non-volatile auxiliary storage device 13, which is a non-temporary storage medium. For this reason, the analysis device 10 preferably has an interface for reading data from removable media.

The analysis device 10 is a computer system that is physically configured on one computer or on multiple logically or physically configured computers, and operates on the same computer in separate threads. It may also operate on a virtual computer built on multiple physical computer resources. Furthermore, each functional unit of the analysis device 10 may be realized on a different computer.

FIG. 6 is a diagram showing the logical configuration of the analysis device 10.

The analysis device 10 includes an image acquisition section 41, a calibration section 42, a tool position estimation section 43, a deviation determination section 44, and an output section 45 as functional blocks. The analysis device 10 also stores image data 46, camera parameters 47, standard work model (threshold) 48, and tool position data 49 in the auxiliary storage device 13.

The image acquisition unit 41 stores moving images taken by the cameras 20, 22, and 23 as image data 46. The image acquisition unit 41 also sends calibration still images taken by the cameras 20 , 22 , and 23 to the calibration unit 42 . Further, the image acquisition section 41 sends moving images taken by the cameras 20 , 22 , and 23 during the work to the tool position estimation section 43 .

The calibration unit 42 executes a calibration process using the calibration still images taken by the cameras 20, 22, and 23, and records the calibration results in the camera parameters 47 (FIGS. 7, 8, and 9). reference).

The tool position estimation section 43 estimates the feature amounts of the position and posture of the tool 5 from the moving images taken by the cameras 20 , 22 , and 23 during work, sends them to the deviation determination section 44 , and records them in the tool position data 49 . For example, the feature amounts of the position and orientation estimated by the tool position estimating unit 43 are as follows.
・Distance r from the origin position of the workpiece 3 to the tip position of the tip part 51
・Declination angle θ of the tip position of the tip portion 51 from the reference direction of the workpiece 3
・Declination angle φ of the tip position of the tip part 51 from the reference direction in the image
- Inclination of the tool 5 (angle between the tip 51 and the surface (horizontal plane) of the workpiece 3)
・Working time from the start of the work to the end of the work ・The angular velocity at which the tip 51 moves around the outer circumference of the circular part 32 ・The number of times the tip 51 contacts the workpiece 3 in one step of work ・In the work of one step, the tip・Position where the tip 51 first comes into contact with the work 3 ・Position where the tip 51 is separated from the work 3 in the 1st process operation ・Time from when the tip 51 comes into contact with the work 3 until it leaves the work 3 in the 1st process

The deviation determination unit 44 monitors the position of the tool 5 during work estimated by the tool position estimation unit 43, compares it with a threshold determined as a standard work model 48, and notifies the output unit 45 if it is determined to be abnormal. do.

The output unit 45 outputs an alarm to the operator when the deviation determination unit 44 determines that the position of the tool 5 is abnormal. In the work observation system of this embodiment, since the worker does not take his eyes off the eyepiece 21 during work, it is preferable to output a sound or an image visible through the eyepiece 21 to warn the worker. Further, the output unit 45 outputs the moving image data during the work stored in the image data 46 and the estimation result of the position of the tool 5 recorded in the tool position data 49 (see FIG. 13).

The image data 46 is a moving image taken by the cameras 20, 22, and 23 and acquired by the image acquisition unit 41 during work. The camera parameters 47 are the results of calibration processing by the calibration section 42. The standard work model 48 is data on the position and orientation of the tool 5 defined as standard work, and records threshold values for determining deviation. Further, the position data of the tool 5 recorded in the work of an expert may be recorded in the standard work model 48. The tool position data 49 is time series data of the result of estimating the position of the tool 5 by the tool position estimation unit 43.

Next, the calibration process will be explained. FIG. 7 is a flowchart of the calibration process executed by the calibration section 42, and FIGS. 8 and 9 are diagrams showing examples of calibration screens.

When the calibration process is started, first, the tool 5 placed horizontally is photographed, the markers 52A, 52B attached to the tool 5 and the tips of the tip portions 51 are recognized, and the positions and tips of the markers 52A, 52B are determined. The position of the tip of the tip portion 51 is measured, and the distance between the markers 52A and 52B and the distance between the marker 52A and the tip of the tip portion 51 are calculated (S101). Using these calculated distances, the position of the tip of the tool 5 can be calculated from the positions of the markers 52A, 52B measured during operation. Note that if the positions of the markers 52A and 52B are known, the calibration process in step S101 can be omitted. Further, in a camera such as the microscope camera 20 whose optical axis coincides with the vertical direction, it is preferable to set the tool 5 horizontally and take a picture. By setting it up in this way, the ratio of distances measured on the image matches the ratio of distances in the actual three-dimensional world, excluding factors caused by lens distortion, making calibration easier. By using a camera with less lens distortion and high magnification, such as the microscope camera 20, more accurate calibration can be achieved.

Next, internal parameters (for example, focal lengths of the side cameras 22, 23) and external parameters (positional relationship between the side cameras 22, 23) of the side cameras 22, 23 are estimated (S102). For example, the two side cameras 22 and 23 photograph a figure (for example, a checkered test pattern) in which the relative positions of three points are determined, and from the positional relationship of the three points in the image, the side cameras 22 and 23 Obtain the positional relationship of points in the image taken by . At least three points are required for calibration, but accuracy can be improved by using four or more points. Note that if the side cameras 22 and 23 are stereo cameras in which the positional relationship and photographing parameters (zoom magnification) remain unchanged, the calibration in step S102 is not necessary.

Next, internal parameters (for example, focal length) and external parameters (relative positional relationship between the microscope camera 20 and the side cameras 22 and 23) of the microscope camera 20 are estimated (S103). For example, the calibration unit 42 recognizes an object (calibration sphere) of known size from images taken by the cameras 20, 22, and 23 of a predetermined calibration jig. On the calibration screen shown in FIG. 8, the image of the microscope camera 20 is displayed on the right side, and the images of the side cameras 22 and 23 are displayed on the left side. When the "Start Calibration" button is operated, the calibration process in step S103 is started, and a shooting instruction is displayed on the screen. When the operator operates the "shoot" button, a frame image is captured, and this process is repeated to capture a plurality of still images (for example, 15).

Since the distance between the microscope camera 20 and the jig 4 is known, the zoom magnification of the microscope camera 20 can be calculated based on the recognition result (size of the calibration sphere) in the image taken by the microscope camera 20. In addition, since the zoom magnification of the side cameras 22 and 23 has been calculated, the side cameras 22 and 23 and the correction The distance of tool 4 can be calculated. This allows the positional relationship between the microscope camera 20 and the side cameras 22 and 23 to be determined. The calibration sphere is recognized in the three images taken at the same time, and the positional relationship within each image is calculated. Further, by averaging the sizes of the calibration spheres recognized from a plurality of still images, the accuracy of the calculated positional relationship can be improved.

Note that in step S103, even if the size of the calibration sphere is unknown, another object of known size and distance between the recognition targets (for example, a test pattern on which a figure of a predetermined size is drawn) is used. The zoom magnification of each camera 20, 22, and 23 may be estimated by doing this. In this case, the zoom magnification of each camera 20, 22, 23 can be determined by the test pattern, and the positional relationship of each camera 20, 22, 23 can be determined by the calibration sphere.

Next, the position and inclination of the upper surface of the workpiece 3 are measured (S104). For example, the tool 5 is placed with the tip of the tip portion 51 in contact with the workpiece 3, and the work area 1 is photographed using the microscope camera 20 and side cameras 22 and 23. The image of the microscope camera 20 is displayed on the calibration screen shown in FIG. When the "Start Calibration" button is operated, the calibration process in step S104 is started, and a shooting instruction is displayed on the screen. When the operator operates the "shoot" button, frame images shot by the side cameras 22 and 23 are captured, and this process is repeated to shoot a plurality of still images (for example, three). Markers 52A and 52B are recognized in the photographed image. Since the distance between the markers 52A and 52B and the distance between the marker 52A and the tip of the tip portion 51 have been calculated in the calibration in step S101, the vertical direction of the top surface of the workpiece 3 is determined from the images taken by the side cameras 22 and 23. The position of can be calculated. Further, by averaging the sizes of the calibration spheres recognized from a plurality of still images, the accuracy of the calculated positional relationship can be improved.

After all calibrations are completed, the device enters the measurement standby state.

FIG. 10 is a flowchart of the measurement process executed by the analysis device 10.

First, when the measurement process is started from the measurement standby state, the image acquisition unit 41 displays the screen shown in FIG. 11, and starts a countdown to start imaging by operating the "measurement start/stop" button. Then, at the timing when the count reaches 0, the image acquisition unit 41 sends a shooting start instruction to the signal generator 24 so that each camera 20, 22, 23 starts shooting moving images in synchronization (S101). . The signal generator 24 sends a shooting start trigger to the microscope camera 20 and the side cameras 22 and 23 at the timing when the shooting start counter of the analysis device 10 becomes 0.The microscope camera 20 and the side cameras 22 and 23 start shooting. When a start trigger is received, video shooting starts in synchronization.

The screen shown in FIG. 11 displays the time until the start of measurement as a status display in the upper left, an area where the image taken by the microscope camera 20 is displayed on the left side of the screen, and an area where the image taken by the microscope camera 20 is displayed on the right side of the screen, and an area where the tool 5 (the tip of the tip part 51 ) has an area where changes over time in position data obtained by analyzing the movement of the camera are displayed.

In addition, at the top of the screen, there is a "worker name" button to select the worker name to be displayed on the screen, a "work selection" button to select the work performed by the worker, and a button to start or stop measurement. A "measurement start/stop" button is provided to do so. In addition, at the bottom of the screen, there is a "Back" button to return to the previous screen, a "Play" button to play back previously recorded video images, and an "Analysis" button to analyze the currently displayed work record. ” button, the “Deviation Alert” switch (checkbox) to switch whether to issue a warning indicating that the measured value during work deviates from the predetermined threshold, and the “Deviation Alert” switch (checkbox) to switch whether to issue a warning indicating that the measured value during work deviates from the predetermined threshold. A “threshold” switch (check box) is provided for switching whether to display the threshold value or not.

Returning to FIG. 10, the explanation of the flowchart will be continued. Next, the image acquisition unit 41 acquires moving images from each of the cameras 20, 22, and 23, and stores them in the image data 46 (S102).

Next, the tool position estimation unit 43 recognizes the positions of the markers 52A and 52B from the images taken by the side cameras 22 and 23. Then, using the distance between the markers 52A and 52B calculated in S101 of the calibration process and the distance between the marker 52A and the tip of the tip 51, the tip of the tip 51 is calculated from the recognized positions of the markers 52A and 52B. The position is calculated and recorded in the tool position data 49 (S113).

Next, the tool position estimating unit 43 calculates the position of the workpiece based on the positions of the markers 52A and 52B recognized from the images taken by the side cameras 22 and 23 and the inclination of the top surface of the workpiece 3 calculated in S104 of the calibration process. 3 is calculated and recorded in the tool position data 49 (S114).

Next, the tool position estimating unit 43 estimates the tip position of the tip portion 51 in the coordinates of the image taken by the microscope camera 20, and records it in the tool position data 49 (S115). For example, the microscope coordinate system for an image taken by the microscope camera 20 may be a polar coordinate system in which one point in the image is the origin and a predetermined direction in the image is 0 degrees, or a specific side of the image with one point in the image as the origin. These are orthogonal coordinates with the In the workpiece 3 shown in FIG. 3, polar coordinates are used in which the center of the circular part 32 is the pole (origin) and the direction of one side of the image taken by the microscope camera 20 is 0 degrees in the plane including the upper surface of the workpiece 3. It is preferable to calculate the distance r from the origin and the declination angle φ from the reference axis.

Next, the tool position estimating unit 43 calculates the origin and reference direction of the workpiece coordinates from the positions of the markers 33A to 33D of the workpiece 3 photographed by the microscope camera 20, calculates conversion parameters with the coordinates of the microscope camera image, Using the calculated transformation parameters and mathematical methods such as linear transformation and projective transformation, the tip position of the tip portion 51 calculated in step S115 in the workpiece coordinate system is calculated and recorded in the tool position data 49 (S116 ). For example, the microscope coordinates of an image taken by the microscope camera 20 may be polar coordinates in which a point on the workpiece 3 is the origin and a predetermined direction in the image is 0 degrees, or polar coordinates in which a point on the workpiece 3 is the origin and a specific side of the workpiece 3. These are orthogonal coordinates with the In the workpiece 3 shown in FIG. 3, polar coordinates are defined with the center of the circular part 32 as the pole (origin) and the position of any one of the markers 33A to 33D of the circular part 32 as 0 degree in a plane including the upper surface of the workpiece 3. It is preferable to estimate the declination angle θ by using Note that the polar coordinates used in step S115 and the polar coordinates used in step S116 have the same radius r, but the reference direction of θ and the reference direction of φ differ due to the rotation of the circular part 32, so the reference direction θ and φ differ by the difference.

The analysis by the tool position estimating unit 43 may be performed for each frame of the moving image, or may be performed for every several frames as long as there is no delay in issuing the deviation alarm.

Next, the deviation determination unit 33 compares the position of the tool 5 estimated in steps S111 to S114 with a threshold value determined as the standard work model 48, and issues a deviation alarm if the position deviates from the normal value range. This is notified to the output unit 45 (S117).

Next, the output unit 45 reads out the accumulated image data 46 and the recorded tool position data 49, and outputs display data for displaying a moving image and a time series chart (S118). For example, as shown in FIG. 12, the output unit 45 displays "Measuring" and the elapsed time since the start of photography as a status display on the upper left side of the screen, an image taken by the microscope camera 20 on the left side of the screen, and a tool on the right side of the screen. 5 (the tip of the distal end portion 51), the position data obtained by analyzing the movement is displayed. Other elements of the screen shown in FIG. 12 are the same as the screen shown in FIG. 11.

13 to 15 are diagrams showing examples of analysis result display screens output by the output unit 45. The analysis result display screen shown in FIG. 13 displays the measured worker's movements and the previously measured teacher data of the expert, and the analysis result display screen shown in FIG. Display a time series chart that analyzes the behavior of the system for comparison.

At the top of the screen, there is a "work selection" button for selecting the work done by the worker, a "worker selection" button for selecting the worker name to be displayed on the screen, and a button for selecting the teacher data to be played on the screen. A ``select teacher data'' button for selecting a moving image file or a time-series chart of analysis results, and a ``select moving image'' button for selecting a moving image file or a time-series chart of an operator to be reproduced are provided.

In the center of the screen, there is a "Video/Chart" button to switch between displaying a video image (Figure 13) or a time series chart (Figure 14), and a teacher button to the left of the "Video/Chart" button. A teacher data display area for displaying data and a worker data display area for displaying data of the worker are provided on the right side of the "video/chart" button.

Further, at the bottom of the screen, there are provided a playback button for controlling playback of the moving image, a time slider for specifying the generation position of the moving image, and a playback method selection radio button. Further, at the bottom of the control area, there are provided a "return" button for returning to the previous screen and an "evaluation" button for displaying the evaluation results of the operator's actions (see FIG. 15).

The moving image of the expert may not only be displayed on the display device 27, but may also be made visible to the worker during the work. For example, a moving image of an expert may be projected onto the work 3 by projection mapping and shown to the worker. Furthermore, a prism may be arranged in the optical path from the eyepiece lens 21 of the microscope 2 to the objective lens to guide the image of the small display device to the eyepiece lens 21, so that a moving image of an expert can be visually recognized from the eyepiece lens 21. It's okay. Furthermore, when using a digital microscope, a moving image of an expert may be superimposed on the image viewed through the eyepiece 21 so that the operator can view it.

On the analysis result display screens shown in Figures 13 and 14, by displaying video images and time-series charts that analyze the movements of skilled workers and workers side by side, it is possible to clearly display the differences between the tasks of workers and experts. , can improve the learning effect.

The analysis result display screen shown in FIG. 15 displays the evaluation results obtained by scoring the analysis results of the worker's movements in the form of a radar chart.

At the top of the screen, there are buttons "Select work 1 video" and "Select work 2 video" for selecting the video file of the work performed by the worker, and "Select work 2 video" buttons for selecting the worker name to be displayed on the screen. "Select member" button is provided.

At the center of the screen, there are provided a data display area that displays a time-series chart that analyzes the selected work, and a radar chart display area that displays the worker's scored actions.

Additionally, a "back" button is provided at the bottom of the screen to return to the previous screen.

The analysis result display screen shown in FIG. 15 allows the worker's proficiency level compared to that of an expert to be displayed in an easy-to-understand manner, and the learning effect can be improved.

Next, an educational support system to which the work observation system of the above-described embodiment is applied will be described. The educational support system of this embodiment compares and analyzes the work data stored in the work observation system described above and the movements of the expert, and can clearly display the differences between the work of the worker and the expert, resulting in a high educational effect. You can get In addition, the evaluation results obtained by converting the analysis results of the worker's movements into scores are displayed, so that the superiority or inferiority of the worker's movements can be clearly displayed, and a high educational effect can be obtained.

As described above, the work observation system of the present embodiment includes a first imaging device (microscope camera 20) that images the work area 1 from a first direction to obtain a first image, and a second imaging device (microscope camera 20). Using a second imaging device (side cameras 22, 23) that images the work area 1 from the direction and obtains a second image including distance information, and the first image and the second image, The analysis device 10 calculates the feature amount of the position and orientation of the tool 5 shown in at least the second image, and outputs the feature amount of the position and orientation of the tool 5 in the work. Therefore, work quality can be improved. In other words, in the past, techniques were learned through qualitative instruction that involved repeating trial and error while checking the result of the technique, and it took several months to train a skilled person. Therefore, by digitizing the movements of workers and comparing the techniques of ordinary workers and those of skilled workers, it is possible to develop workers at an early stage.

Furthermore, the microscope camera 20 and the side cameras 22 and 23 may photograph the work area 1 at different magnifications, and the microscope camera 20 may photograph the work area 1 at a higher magnification than the side cameras 22 and 23. . For example, fine manual work (fine machining) can be three-dimensionally localized with an accuracy of 1 millimeter or less using an enlarged image from the microscope camera 20. Fine manual work or fine processing is work in which it is difficult for a third party to observe with the naked eye the movements of the operator during processing, the workpiece 3 to be processed, and the tool 5 used for processing. In other words, when observing with the naked eye, when the tool 5 is hidden by the workpiece 3 or the workpiece 3 is hidden by the tool 5, the operator's movements during machining, the workpiece 3, and the tool 5 are observed with the naked eye in multiple fields of view. That's difficult. Since the machining area of the workpiece 3 and the tool 5 are small, it is possible to observe it with the naked eye by increasing the magnification using a magnifying glass, but it is difficult to observe it with the naked eye from other fields of view. Therefore, it becomes possible to observe three-dimensionally using the above-mentioned cameras with different magnifications.

Further, since at least two first markers 33A to 33D are attached to the work object (work 3) placed in the work area, a circular part such as the circular part 32 is included in the work 3. Even if there is, the rotation of the circular portion 32 can be detected.

In addition, since the first image was taken from the same direction as the image viewed by the worker, by using the image from the worker's perspective, the work content can be explained to the worker in an easy-to-understand manner, increasing the educational effect. You can improve.

Further, since the microscope camera 20 and the side cameras 22 and 23 photograph the work area based on synchronized times, images without time lag can be photographed, and feature quantities can be accurately analyzed.

Further, each of the first markers 33A to 33D is different from the background of the work area 1 and colored in a different color from each other, so the visibility of the markers 33A to 33D in the work area 1 can be improved, and the markers 33A to 33D are colored in different colors. 33D visibility erroneous recognition can be reduced.

Furthermore, since at least two second markers 52A and 52B are attached to the tool 5, the feature quantities of the position and orientation of the tool 5 can be calculated by measuring the positions of the markers 52A and 52B.

Further, the second markers 52A, 52B are spherical, and each of the second markers 52A, 52B is provided at a different position from the tip of the tool 5, so that the feature quantities of the position and posture of the tool 5 can be easily determined. Can calculate.

Furthermore, since the analysis device 10 calculates the position and orientation of the tool 5 using the positional relationship between the second markers 52A and 52B, it is possible to easily calculate the feature amounts of the position and orientation of the tool 5.

In addition, the analysis device 10 compares the position and posture features of the tool 5 with pre-stored standard position and posture features and outputs the comparison results, so that the operator and the expert Differences in tasks can be clearly displayed, improving learning effectiveness.

Moreover, the feature amount of the position of the tool 5 reflected in the second image includes distance information from the microscope camera 20 to the tool 5, distance information from the side camera 22 to the tool 5, and distance information from the side camera 23 to the tool 5. Since the information is included, the position of the tool 5 can be accurately measured through calibration even if each camera is not installed accurately.

In addition, the output unit 45 outputs data for displaying a graph created by the work observation system based on the feature amount measured for the first worker and the feature amount measured for the second worker. Therefore, differences between workers can be displayed in an easy-to-understand manner, and effective training can be carried out.

Note that the present invention is not limited to the embodiments described above, and includes various modifications and equivalent configurations within the scope of the appended claims. For example, the embodiments described above have been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Further, a part of the configuration of one embodiment may be replaced with the configuration of another embodiment. Further, the configuration of one embodiment may be added to the configuration of another embodiment. Further, other configurations may be added, deleted, or replaced with a part of the configuration of each embodiment.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example by designing an integrated circuit, and a processor realizes each function. It may also be realized by software by interpreting and executing a program.

Information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or a recording medium such as an IC card, a flash memory, or a DVD.

Furthermore, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily show all the control lines and information lines necessary for implementation. In reality, almost all configurations can be considered interconnected.

1 Working area 2 Microscope 20 Microscope camera 21 Eyepiece 3 Workpiece 31 Main body 32 Circular part 33 Tool position estimation parts 33A, 33B, 33C, 33D Marker 4 Workbench 5 Tool 51 Tip parts 52A, 52B Marker 53 Handle 6 Microscope support 10 Analysis device 11 Processor 12 Memory 13 Auxiliary storage device 14 Communication interface 15 Input interface 16 Keyboard 17 Mouse 18 Output interfaces 22, 23 Side camera 24 Signal generator 27 Display device 28 Speaker 41 Image acquisition unit 42 Calibration unit 43 Deviation determination unit 44 Deviation determination unit 45 Output unit 46 Image data 47 Camera parameters 48 Standard work model 49 Tool position data

Claims (16)

A work observation system that images a state in which a worker works in a work area using a tool,
a first imaging device that images the work area from a first direction to obtain a first image;
a second imaging device that images the work area from a second direction to obtain a second image;
an analysis device that analyzes the work using the first image and the second image,
The analysis device includes:
Calculating feature amounts of the position and orientation of the tool that appears in at least the second image,
A work observation system characterized by outputting feature amounts of the position and orientation of the tool in the work. The work observation system according to claim 1,
A work observation system characterized in that at least two first markers are attached to the work object placed in the work area. The work observation system according to claim 1,
The work observation system is characterized in that the first image is taken from the same direction as the image visually recognized by the worker. The work observation system according to claim 1,
A work observation system characterized in that the first imaging device and the second imaging device photograph the work area at different magnifications. The work observation system according to claim 4,
The work observation system is characterized in that the first imaging device photographs the work area at a higher magnification than the second imaging device. The work observation system according to claim 1,
The work observation system is characterized in that the first imaging device and the second imaging device photograph the work area based on synchronized times. The work observation system according to claim 2,
A work observation system characterized in that each of the first markers is colored in a color different from a background of the work area and mutually different. The work observation system according to claim 1,
A work observation system characterized in that at least two second markers are attached to the tool. The work observation system according to claim 8,
the second marker is spherical;
The work observation system is characterized in that each of the second markers is provided at a different position from the tip of the tool. The work observation system according to claim 9,
The work observation system is characterized in that the analysis device calculates the position and orientation of the tool using the positional relationship of the second marker. The work observation system according to claim 1,
The work observation system is characterized in that the analysis device compares the feature values of the position and orientation of the tool with pre-stored standard feature values of the position and orientation, and outputs the results of the comparison. The work observation system according to claim 1,
The work observation system is characterized in that the analysis device calculates any one of the tip position of the tool, the angle of the tool, and the number of times the tool touches the workpiece as the feature quantity of the position and posture. The work observation system according to claim 1,
The work observation system is characterized in that the feature quantity of the position of the tool reflected in the second image includes distance information from each of the imaging devices to the tool. The work observation system according to claim 1,
The analysis device includes an output unit that outputs data for displaying a graph created based on the feature amount measured for the first worker and the feature amount measured for the second worker. A work observation system characterized by: A work analysis method in which an analysis device analyzes work performed by a tool in a work area, the method comprising:
The analysis device includes:
It has an arithmetic device that executes predetermined arithmetic processing to realize each of the following functional units, and a storage device that stores data used in the arithmetic processing,
The work analysis method is
The arithmetic unit includes a first imaging device that captures a first image of the work area from a first direction, and a second imaging device that captures a second image of the work area from a second direction. Obtain an image from the second imaging device,
the analysis device calculates at least feature amounts of the position and orientation of the tool shown in the second image;
A work analysis method, characterized in that the arithmetic device outputs feature quantities of the position and orientation of the tool in the work. An educational support system that images a state in which a worker works in a work area using a tool, the system comprising:
a first imaging device that images the work area from a first direction to obtain a first image;
a second imaging device that images the work area from a second direction to obtain a second image;
an analysis device that analyzes the work using the first image and the second image,
The first imaging device and the second imaging device photograph the work area at different magnifications,
The analysis device includes:
Calculating feature amounts of the position and orientation of the tool that appears in at least the second image,
An educational support system characterized by outputting feature amounts of the position and posture of the tool in the work.

Images