WO2025014468A1

WO2025014468A1 - Calibrating free moving equipment with camera-based augmented reality

Info

Publication number: WO2025014468A1
Application number: PCT/US2023/027198
Authority: WO
Inventors: Richard Gary Mcdaniel
Original assignee: Siemens Corporation
Priority date: 2023-07-10
Filing date: 2023-07-10
Publication date: 2025-01-16

Abstract

An augmented reality (AR) computing device can identify a robotic device and a pose of the robotic device. Based on identifying the robotic device and the pose of the robotic device, the AR computing device can obtain a world model of the robotic device. While performing an operation of the robotic device, the robotic device can be displayed on the user interface of the AR computing device. In an example aspect, a geometry of the robotic device can be overlayed in a simulation position on the user interface that also displays the actual position of the robotic device. Offsets can be determined between the world model and the physical environment, for instance using the AR computing device. In various examples, the world model is adjusted based on the offsets, so as to calibrate the world model for the operation.

Description

CALIBRATING FREE MOVING EQUIPMENT WITH CAMERA-BASED AUGMENTED

REALITY

BACKGROUND

[0001] Robotic operations in automation, such as robotic grasping and assembly, in various environments present various technical challenges. In order to perform automated operations, such as grasping and assembly, robots are calibrated. Calibrating equipment for pick and place and other similar automation jobs can be a tedious and error prone task. For example, in some cases, an integration team is tasked with performing calibration in which machines are aligned so that they correctly line up with the materials that they manipulate. Without calibration, it is often not possible to set up devices with perfect alignment, even with careful measurement. Thus, it is recognized herein that currently, approaches to calibrating free moving equipment in various automation scenarios can be inefficient and imprecise, among other technical shortcomings.

BRIEF SUMMARY

[0002] Embodiments of the invention address and overcome one or more of the described- herein shortcomings or technical problems by providing methods, systems, and apparatuses for calibrating robots and other devices in various automation systems, such as an assembly system. For example, the world model or digital twin of various objects can be calibrated, such that a given device can operate in accordance with its model, so as to calibrate various operations of the device.

[0003] In an example aspect, a robotic device is disposed in a physical environment with an augmented reality (AR) computing device that defines a camera and a user interface. The AR computing device can identify the robotic device and a pose of the robotic device. Based on identifying the robotic device and the pose of the robotic device, the AR computing device can obtain a world model of the robotic device. The world model can define a complete three- dimensional (3D) geometry, including outlines, of the robotic device. In an example, an operation of the robotic device is simulated using the world model, so as to define a simulated operation that defines a simulated position of the robotic device. Furthermore, the operation of the robotic device can be performed in the physical environment, so as to define a real-world operation that defines an actual position of the robotic device. While performing the operation of the robotic device, the robotic device can be displayed on the user interface of the AR computing device. In an example aspect, the geometric geometry of the robotic device can be overlayed in the simulation position on the user interface that displays the actual position of the robotic device. A first offset can be determined between the world model and the physical environment, for instance using the AR computing device. In various examples, the world model is adjusted based on the first offset, so as to calibrate the world model for the operation.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0004] The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:

[0005] FIG. 1 shows an example automation system that includes a robot and an augmented reality (AR) computing device configured to calibrate the robot, in accordance with an example embodiment.

[0006] FIG 2 illustrates an example architecture of an AR-based calibration system, in accordance with an example embodiment.

[0007] FIG. 3 is a flow diagram that illustrates example operations that can be performed by the AR-based calibration system including the AR computing device, in accordance with an example embodiment.

[0008] FIG. 4 illustrates a computing environment within which embodiments of the disclosure may be implemented.

DETAILED DESCRIPTION

[0009] By way of background, it is recognized herein that robots are typically calibrated by changing the numeric parameters of the commands for the robot, rather than by making physical adjustments. For example, the program that runs a given robot might be augmented to slow down and stop at key moments of its operations. At those key moments, the end effectors of the robot can be measured for how well they hit their marks. In some cases, adjustments are made to the program, and it is run repeatedly to make more adjustments to try to get the final positions to align. It is recognized herein that the adjustments sometimes are not transferred back to the original engineering design, and therefore the version of the application on the shop floor might be different from the designed version.

[0010] In accordance with various embodiments described herein, free moving devices, such as robots for example, are calibrated using augmented reality (AR) technology. Referring generally to FIG. 1, an AR computing device 101, for instance a tablet, mobile device, personal device assistant (PDA), laptop, or the like, that has a built-in camera or otherwise is coupled to a camera can capture images of a device being calibrated, for instance a robot 104. The images can also include one or more objects that are being manipulated by the robot. The camera can define an RGBD (color and depth) camera, such that the images can be scanned so as to measure how accurately the robot is moving in order to grasp or otherwise interact with the object. In various examples, by measuring positions of the device and detecting offsets, the device 101 may transfer the offsets back to a controller of the robot 104 so as to change how the robot 104 moves, thereby reducing the offsets and increasing precision of the robot 104.

[0011] Referring initially to FIG. 1 , an example industrial or physical environment or workspace or scene 100 is shown. As used herein, a physical environment, workspace, or scene can refer to any industrial environment. Unless otherwise specified, physical environment, workspace, and scene can be used interchangeably herein, without limitation. For purposes of example, one or more objects 106 can be disposed within the environment 100. For example, objects 106 can be positioned for grasping from a bin or container 107, or positioned for placement into the bin 107. Although a container is illustrated for purposes of example, it will be understood that the environment 100 can include alternative fixtures involved in assemblies or other automation processes, and all such fixtures and assembly components are contemplated as being within the scope of this disclosure. It will be further understood that the objects 106 are merely presented as examples, such that objects referred to herein can be alternatively shaped or define alternative structures, and all such objects are contemplated as being within the scope of this disclosure.

[0012] The physical environment 100 can include a computerized automation system 102 configured to perform one or more manufacturing operations, such as assembly, transport, or the like. The automation system 102 can include one or more robot devices or assembly machines, for instance a robot device 104, configured to perform one or more industrial tasks, such as bin picking, grasping, placing, assembly, or the like. The system 102 can include one or more computing processors configured to process information and control operations of the system 102, in particular the robot 104. The robot 104 can include one or more processors, for instance a processor 108, configured to process information and/or control various operations associated with the machine 104. A system for operating a machine within a physical environment can further include a memory for storing modules. The processors can further be configured to execute the modules so as to process information and generate models based on the information. It will be understood that the illustrated environment 100 and the system 102 are simplified for purposes of example. The environment 100 and the system 102 may vary as desired, and all such systems and environments are contemplated as being within the scope of this disclosure.

[0013] Still referring to FIG. 1, the robot 104 can further include a robotic arm or manipulator 110 and a base 112 configured to support the robotic manipulator 110. The robot 104 can further include an end effector 116 attached to the robotic manipulator 110. The end effector 116 can include one or more tools configured to grasp, move, join, or otherwise transform objects 106. Example end effectors 116 include finger grippers or vacuum-based grippers. The robotic manipulator 110 can be configured to move so as to change the position of the end effector 116, for example, so as to place or move objects 106 within the physical environment 100. The system 102 can further include one or more cameras or sensors 118, for instance a depth camera or three-dimensional (3D) point cloud camera, configured to detect or record objects 106 within the physical environment 100. In other examples, a camera 105 that is defined by the AR computing device 101, for instance mounted to, or part of, the AR computing device 101 or otherwise communicatively coupled to the AR computing device 101, is the only camera in the system 102.

[0014] Still referring to FIG. 1, the automated system 102 can further include the AR computing device 101 that can include the camera 105 configured as an RGB-D camera defining a color and depth channel. The camera 105 of the computing device 101 can be positioned, for instance held by a user or otherwise positioned (e.g., mounted or placed), so as to capture images of the environment 100. In particular, by way of example, the camera 105 can be positioned so as to capture images of the robot 104, for instance the end effector 116, and objects 106 that the robot 104 can manipulate in some way. [0015] The AR computing device 101 can define a user interface or interactive screen (display) 103 configured to receive input from a user. For example, the user interface 103 can define a touch screen configured to receive drawings, visual markings, numerical or textural input, and the like, from users. In some cases, the user interface 103 can be configured to receive commands from the user, such that the robot device 104 can be controlled by the commands received via the user interface 103. The user interface can also be configured to display video input from the camera 105, and various AR visual indications such as, for example and without limitations, markers, outlines of objects, geometric shapes or indications, color overlays, indications of obstacles or humans, and the like. The screen 103 can define user options and graphical overlays for calibrating a given device with respect to its digital twin in the world model, for instance the robot 104.

[0016] In some examples, the screen 103 can display real- world or live images of the physical environment 100 and AR graphics overlayed on the real-world or live images. In an example, via the interactive screen 103, a user may select a device, for instance the robot 104, that will be calibrated or corrected relative to its digital twin. In some cases, the selected device can be controlled directly via the user interface 103, so that the selected device (e.g., robot 104) can be guided based on positions that are detected by the AR computing device 101. Additionally, the user may use the interface 103 to correct mislabeled objects in the scene or to enable a recognition system to properly to align its internal world model to the camera images, as further described herein. The screen 103 may also be used for messages and other user communication. By way of example, and without limitation, the screen 103 may indicate that the user is not pointing the camera 105 of the AR computing device 101 at a given device (e.g., robot 104), whether the system 102 can find the device in the captured image or frame, whether the viewing angle defined by the camera 105 of the AR computing device 101 relative to the given device is poor, whether the device is moving according to expected control signals, among other indications.

[0017] The device that is calibrated using the AR computing device 101 is presented herein as the robot device 104 for purposes of example, though it will be understood that the device that is the subject of the AR computing device 102 can vary as desired, for instance any free-moving device that is open to viewing and not enclosed in a hidden area, and all such devices are contemplated as being within the scope of this disclosure. In various examples, the device defines a size that a camera can view in a frame, for instance a size that is not microscopic or exceptionally large. Similarly, in some cases, the objects 106 that the device manipulates are visible to the camera 105 and might be disposed in such a manner that minimizes reflections, transparencies, glare, and other factors that degrade camera-based measurements.

[0018] Referring now to FIG. 2, in example operations, the system 102 can obtain a world model 202 of the device that is being calibrated, for instance the robot 104. The world model 202 can also be referred to as a digital twin. The world model 202 generally refers to an abstract representation of spatial or temporal dimensions of objects and devices that might be encountered in the automated system 102. In some cases, the world model 202 can define a model that is used to program the robot device 104. By way of example, the robot device 104 might have free range of motion, and thus the robot device 104 might be programmed using motion planning where the kinematics, 3D shape, and relative positions of the robot 104 and objects in the environment are used to predict where the robot 104 may contact other entities and avoid collisions. Such programming is more general than a robot that is programmed directly using teach points or specific motion paths that are simply provided and cannot be changed. The 3D geometry for the world model 202 can be provided via computer-aided design (CAD) tools or CAD input 204. In an example, the world model 202 contains the knowledge of what the robot 104 is intended to do and how it is meant to interact with the material objects in a given application. For example, the knowledge may be encoded as a list of commands that the robot 104 follows, or it may be more general such as a set of goals that the system 102 must accomplish.

[0019] In various examples, based on the specific application for which the robot is being implemented, the system 102 can determine actions associated with the robot 102, which can be referred to as key actions. An example key action is when the robot 104 touches, releases, or otherwise manipulates an object. In some examples, the key actions are implicit in the commands themselves, for instance in a “pick command” for grasping an object. Alternatively, or additionally, the system 102 might identify key actions, for example, based on changes in state of the end effector 116. In particular, for example, the system can identify an action associated with the gripper opening or closing. Alternatively still, key actions of the robot 104 can be simulated before the action are performed in real-time, so as to determine when contact occurs when simulated objects or devices touch each other.

[0020] In various examples, when the system 102 obtains commands for the robot 104, the system can convert the commands to motion paths, for instance a motion plan 206. For example, the system 102 can include a controller 208 configured to control various operations (e.g., movement) of the robot device 104. The system 102 can convert commands it obtains or receives, for instance via the world model 202, into the motion plan 206. The system 102 can include a motion planner that generates the motion plan that can include various paths and states for the robot 104. The motion plan can be fed into the controller 208 of the robot 104, so that the robot 104 can perform particular operations for a given application.

[0021] In various examples, before the robot 104 is calibrated, the position of the effector 116 can be uncertain, so the system 102 may slow down so that a user may interact with the AR computing device 101 while the device 104 is running. For example, if the camera 105 is not pointing at the device 104 being calibrated, or if the system 102 is not able to recognize the device 104 well, then the system 102 can be stopped until the scene is better recognized. When the device 104 approaches a contact, for instance when the end effector 116 moves toward one of the objects 106, the system 102 may also stop the device 104 and ask that the user (e.g., via the user interface 103) point the camera 105 to find different views of the area of contact to achieve better measurements.

[0022] In some cases, there is an error between an expected position of the robot device 104 (or a portion thereof) based on the world model 202 and the actual position of the robot device

104. If the amount of error is so great such that the camera 105 cannot see a given object or portion of the robot device 104 that is supposed to be in a given scene captured by the camera

105, the system 102 can stop the robot device 104. Thus, the system 102 can stop operations when the system 102 cannot find or identify objects or devices that are supposed to be measured. In such a scenario, for instance when there is an error or offset between an expected position of an object or device and the actual position of the object or device, a user may add (at via the 210 via the user interface 103) error amounts or an offset estimation 212 to the world model 202. The offset estimation 212 may define distances between the actual location of objects or devices and the expected location of the objects or devices. In some cases, the system 102 can use the offset estimation 212 in the world model 202 to determine when to slow down the robot device 104 so as to provide a search space for objects that it needs to find.

[0023] In various embodiments, the camera 105 captures images of the robot device 104, and based on the images, the system 102 identifies the device 104 and determines the position and orientation of the device 104. For purposes of example, the camera 105 is described as an RBGD camera, though it will be understood that alternative or additional cameras (e.g., stereoscopic cameras or markers) can detect objects and determine poses of objects, and all such cameras are contemplated as being within the scope of this disclosure. For example, it is recognized herein that depth measurement can become garbled or inaccurate when the camera is close to the subject being captured, and therefore the system 102 can measure the offset from the 2D images.

[0024] It is recognized herein that devices in the virtual world model 202 can vary as compared to their corresponding physical devices. In particular, for example, color and reflectivity can vary in the real-world environment 100 as compared to the world model 202. By way of further example, portions of the device, such as wiring or other objects attached to the devices, might not be present in the world model 202. Thus, the system 102 can identify objects or devices and match them to their corresponding world model even when parts of the image are excluded or occluded or where there are attached masses that follow the device around. In some cases, the camera 105 can be used to add new modeled objects to the world model 202, such as previously unknown obstructions or background objects. For example, the camera 105 can be used to map out the shape of an unknown object, and then a 3D model of the unknown object can be added to the world model 202 in approximately the same shape. By way of further example, if that object is attached to a moving device such as the robotic arm 110, the world model 202 can be updated so that the new shape is attached in the same area of the device 104. In particular, for example, attached objects such as flexible wires may be modeled as a blob area for the purposes of motion planning and might be ignored for recognition of the device pose.

[0025] In an example calibration of the robot device 104, the system 102 can first find the device 104 in the view of the camera 105. To do so, at 214, the system 102 can retrieve a geometry or geometric model (e.g., 3D geometric model or outline) of the device 104 from the world model 202, and overlay the geometry of the device 104 on video from the camera 105, for display on the screen 103. Graphical overlays that are produced by the system 102 generally define drawings on the screen 103 that are intended to be interpreted by the user. A graphical overlay (e.g., geometric model or outline) of the device 104 can be shown on the screen 103 using the intrinsic properties of the camera 105 for projection. For example, via the user interface or interactive screen 103, the user may be asked to align the geometry of the device 104 defined by the graphical overlay with the shape or outline of the actual device defined by image from the camera 105 until the system identifies the device 104 in the image and tracks the device 105 automatically. In various examples, once the device 104 or portion of the device 104 is tracked, the camera 105 can be placed in a location suitable for the user to view the screen 103 while the device 104 remains in the view of the camera 105. In some cases, the world model 202 can be used to estimate whether the position of the camera 105 is achievable. In particular, for example, the system 102 can determine whether the AR computing device 101 (e.g., phone or tablet) can be physically placed in a given location. By way of examples, a given location might be determined to not be achievable if the location requires the AR computing device 101 to placed inside a solid surface, there are objects blocking the view of the camera 105 in the given location, or there is not room for a human being to hold the camera 105 at the given location. Such determinations can be determined from the world model 202 by searching the space (volume) that surrounds the location at which the camera 105 is expected to be placed. In particular, for example, the surrounding space can be searched in the world model 202 for the presence of collision surfaces, obstructions, and sufficient space for users of the AR computing device 101 to stand or sit.

[0026] If a given camera position is not achievable, the user may request that the system 102 produce a new location for the camera 105 or may manipulate the position of the outline model to be in a place that is more convenient. In some cases, a user can determine that a given viewing location is infeasible or otherwise undesired. For example, the user can determine a given viewing angle is undesirable by looking at the location at which the camera 105 is disposed to create the viewing angle. In particular, for example, the user might determine that a given viewing angle is impossible because it might require them to be inside a machine cabinet, under a table, otherwise in a physically prohibitive position.

[0027] In various examples, after objects and device positions are accurately measured, the system 102 can determine how to manipulate the world model 202 so that motions of the device 104 result in their intended positions. The positions of objects can be measured by using the RGBD camera 105. For example, the camera 105 can record color information (RGB information) and depth information (D) for each pixel. This arrangement of colored points in space can define a point cloud 216. It is recognized herein that the number of points in the point cloud 216 can be noisy or voluminous, so at 218, sampling and smoothing functions can be performed so as to reduce the number of points and smooth out any noise in the data, thereby filtering the point cloud 216. At 220, the points in the filtered point cloud 216 can be converted into a formula that represents the surface characteristics of the visible shape defined the filtered point cloud 216. By doing so, at 222, a set of points from the filtered point cloud 316 can be matched to shapes in the 3D world model 202, which defines geometric data that can be fed into a 3D patch matching module.

[0028] It is recognized herein that various robotic or automation devices can be bolted down or otherwise not easily moved, so corrections might be made to the positions of the objects that the device manipulates. For example, when the robot device 104 moves, it does so with a purpose with respect to some target. In a given operation that defines a target for the robot device 104, the device 104 can be controlled to move to a place where that target is modeled in the world model 202, but when that target in the world model 202 is not aligned or coinciding with the actual position of the real target with respect to the position of the device 104, the user can determine that the device 104 is moving to the wrong place in the real-world. To align an actual position or target with the position or target defined in the world model 202, an offset can be applied that represents the difference between the position of the target (robot device 104) in the world model 202 and the position of the robot device 104 in the real-world environment 100 after the robot device 104 is moved to the target defined in the world model. In some cases, when two or more devices are interacting with one another and offsets need to be applied, one or more of devices can be selected to move in the world model 202, and the distances for moving each of the one or more devices can be determined accordingly.

[0029] In an example in which multiple devices are moving such that multiple devices are calibrated, a constraint system can be set up between the coordinates of the devices. The constraint system can define a mathematical set of equations that can describe the relationship between various entities. By way of example, a constraint can be defined in the world model 202 that stipulate that a fixture in the physical environment 100 must be flush to a table in the physical environment. Another example constraint might define the dimensions of various parts of the robot 104, such as the end effector 116 or the robotic arm 110. When new information is added to the world model 202, such as when the offset estimation 212 is added, the system 102 can determine other unknowns by determining values that satisfy the constraints. By way of example, the system 102 might solve the exact location in the coordinate system of the robot’s base 112 or the exact location of a fixture for assembly. The constraint system can include constraints that can be relaxed over multiple measurements, so as to evenly distribute a correction or calibration between the devices. A constraint system can also be set up and implemented when multiple devices interact with a common target object. Constraints between the target object and the multiple devices can be created, and resolution occurs to shift the devices and the target to resolve the differences.

[0030] Referring again to FIG. 1, a device’s motion, for instance the motion of the robot device 104, can consist of multiple stages wherein some motions are free and open to change, and other motions are fixed and performed relative to some target object. By way of example, during a picking or placing operation, the robot 104 can move along any direction to reach the object being picked or container for placement. In an example placing operation, when the robot 104 reaches the container for placement, the robot 104, in particular the arm 110, might only move in a fixed downward motion or direction so as to properly place objects in a given container. Different kinds of containers or fixtures or work materials can require different kinds of fixed or final motions. For example, screwing in a fastener can require a rotary motion that travels in a fixed line at the angle in which the surface material is set. Thus, different final motions might require that a given object or device is disposed at a specific position before the final fixed motion. Calibrating the robot device 104 using the AR computing device 101 can ensure that the device 104 is disposed at the precise specific positions required for various final fixed motions.

[0031] In some cases, for a goal-based robotic application, the functional description of the robotic application might not be changed by the offsets incurred during calibration. For example, the indicators and directives can be set up to be target-specific, such that the calibration can be performed by moving target objects into a more correct position in the world model 202. A command-based robotic application, in some cases, does not tie back to the modeled objects in the same way that a goal-based robotic application might. Therefore, to calibrate a command-based robotic application, the parameters of the commands can be modified. In various examples, a user might determine which portions of the motion of the robotic device 104 are free and which are fixed, for instance at 210. Thus, the user can direct identify offsets to the motion paths via the user interface 103.

[0032] After the world model 202 is updated with calibrated position information, the updated world model 202 can be saved and new motion paths can be generated, so as to generate the motion plan 206, which can define a new or calibrated motion plan. The new paths in the new motion plan can reflect the calibrated positions. Furthermore, in various examples, the calibration motion plan 206 can function as manually input teach points. Further still, after the robot device 104 is calibrated for a first application (e.g., a pick and place application), a user can use the same AR computing device 101 having the same calibration application, to calibrate other devices or calibrate the same device 104 for other operations, or to re-calibrate various devices as devices age or lose accuracy in their movement.

[0033] Referring now to FIG. 3, example operations 300 can be performed by the system 100 with the AR computing device 101. At 302, the AR computing device 101 can identify an object (for instance the robotic device 104) and a pose of the object. To identify objects, in particular the robotic device 104, the camera 105 can be positioned so that the robotic device is in view of the camera 105. Furthermore, the camera 105 can capture video of the robotic device 104, so as to define video input. The AR computing device 101 can match the video input to a first object of the world model, wherein the first object corresponds to the robotic device 104. At 304, based on identifying, for example, the robotic device 104 and the pose of the robotic device 104, the AR computing device 101 can obtain a world model of the robotic device 104. The world model can define the complete geometric and logical states of the robotic device 104. In an example, at 306, an operation of the robotic device 104 is simulated using the world model, so as to define a simulated operation that defines a simulated position of the robotic device. Furthermore, at 308, the operation of the robotic device 104 can be performed in the physical environment 100, so as to define a real-world operation that defines an actual position of the robotic device 104. In various examples, the AR computing device 101 can control the operation of the robotic device 104. For example, user commands can be received via the user interface 103. Based on the user commands, the operation of the robotic device 104 can be performed in the physical environment 100. While performing the operation of the robotic device, the robotic device 104 can be displayed on the user interface 103 of the AR computing device 101, at 312. At 314, the outline of the robotic device can be overlayed in the simulation position on the user interface 103 that displays the actual position of the robotic device.

[0034] A first offset can be determined between the world model and the physical environment 100, for instance using the AR computing device 101. In some cases, the AR computing device 101 can receive an input via the user interface 103. The input might indicate a difference, for instance a distance, between the simulated position of the robotic device as shown on the user interface 103 and the actual position of the robotic device 104 as shown on the user interface 103. For example, the input might define a marker, for instance a location marker, on the user interface 103 or a numeric input. Thus, the first offset can define a distance between the outline of the robotic device 104 in the simulation position and the robotic device 104 in the actual position. Alternatively, or additionally, the first offset can define a difference between a first position of an object 106 relative to the robotic device 104 in the simulation position, and a second position of the object 106 relative to the robotic device 104 in the actual position. Thus, in some cases, the simulation of the robotic device is equivalent to the actual position of the robotic device, but the object 106 might be in a different position than expected based on the world model. For example, based on identifying the robotic device 104 and the pose of the robotic device 104, the AR computing device 101 can obtain a model of the object 106 from the world model. The model can define a first orientation and a first location of the object 106 that is different than a second orientation and a second location of the object 106 in the second position relative to the robotic device 104.

[0035] Thus, in various examples, at 316, the world model is adjusted based on the first offset, so as to calibrate the world model for the operation of the robotic device 104. In various examples, after calibration, the process can return to 308, such that the operation of the robotic device 104 in the physical environment is re-performed. After re-performing the operation, a second offset between the world model and the physical environment can be determined, wherein the second offset less than the first offset.

[0036] It is recognized herein that robotic devices can be calibrated by changing the numeric parameters of its commands, instead of by physical adjustment. The program that runs the robot can be augmented to slow down and stop at key moments of its operations at which time, the end effectors of the robot can be measured for how well they hit their marks. Adjustments can be made to the program, and it is often run again and again while making more adjustments to try to get the final positions to align. Furthermore, the adjustments might not be transferred back to the original engineering design, and the version of the application on the shop floor can be different from the designed version. In contrast, by performing calibration with the Al computing device 101 as described herein, an adjustment to achieve the desired final position can be determined in a single iteration, and the world model can be updated automatically such that the changes on the shop floor are reflected in the digital twin. Thus, in accordance with various embodiments described herein, calibration adjustments are recorded when the adjustments are made.

[0037] FIG. 4 illustrates an example of a computing environment that can include the AR computing device 101 within which embodiments of the present disclosure may be implemented. A computing environment 400 includes a computer system 410 that may include a communication mechanism such as a system bus 421 or other communication mechanism for communicating information within the computer system 410. The computer system 410 further includes one or more processors 420 coupled with the system bus 421 for processing the information.

[0038] The processors 420 may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as described herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general purpose computer. A processor may include any type of suitable processing unit including, but not limited to, a central processing unit, a microprocessor, a Reduced Instruction Set Computer (RISC) microprocessor, a Complex Instruction Set Computer (CISC) microprocessor, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), a System-on-a-Chip (SoC), a digital signal processor (DSP), and so forth. Further, the processor(s) 420 may have any suitable microarchitecture design that includes any number of constituent components such as, for example, registers, multiplexers, arithmetic logic units, cache controllers for controlling read/write operations to cache memory, branch predictors, or the like. The microarchitecture design of the processor may be capable of supporting any of a variety of instruction sets. A processor may be coupled (electrically and/or as comprising executable components) with any other processor enabling interaction and/or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device. [0039] The system bus 421 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may permit exchange of information (e.g., data (including computer-executable code), signaling, etc.) between various components of the computer system 410. The system bus 421 may include, without limitation, a memory bus or a memory controller, a peripheral bus, an accelerated graphics port, and so forth. The system bus 421 may be associated with any suitable bus architecture including, without limitation, an Industry Standard Architecture (ISA), a Micro Channel Architecture (MCA), an Enhanced ISA (EISA), a Video Electronics Standards Association (VESA) architecture, an Accelerated Graphics Port (AGP) architecture, a Peripheral Component Interconnects (PCI) architecture, a PCI -Express architecture, a Personal Computer Memory Card International Association (PCMCIA) architecture, a Universal Serial Bus (USB) architecture, and so forth.

[0040] Continuing with reference to FIG. 4, the computer system 410 may also include a system memory 430 coupled to the system bus 421 for storing information and instructions to be executed by processors 420. The system memory 430 may include computer readable storage media in the form of volatile and/or nonvolatile memory, such as read only memory (ROM) 431 and/or random access memory (RAM) 432. The RAM 432 may include other dynamic storage device(s) (e.g., dynamic RAM, static RAM, and synchronous DRAM). The ROM 431 may include other static storage device(s) (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, the system memory 430 may be used for storing temporary variables or other intermediate information during the execution of instructions by the processors 420. A basic input/output system 433 (BIOS) containing the basic routines that help to transfer information between elements within computer system 410, such as during start-up, may be stored in the ROM 431. RAM 432 may contain data and/or program modules that are immediately accessible to and/or presently being operated on by the processors 420. System memory 430 may additionally include, for example, operating system 434, application programs 435, and other program modules 436. Application programs 435 may also include a user portal for development of the application program, allowing input parameters to be entered and modified as necessary.

[0041] The operating system 434 may be loaded into the memory 430 and may provide an interface between other application software executing on the computer system 410 and hardware resources of the computer system 410. More specifically, the operating system 434 may include a set of computer-executable instructions for managing hardware resources of the computer system 410 and for providing common services to other application programs (e.g., managing memory allocation among various application programs). In certain example embodiments, the operating system 434 may control execution of one or more of the program modules depicted as being stored in the data storage 440. The operating system 434 may include any operating system now known or which may be developed in the future including, but not limited to, any server operating system, any mainframe operating system, or any other proprietary or non-proprietary operating system.

[0042] The computer system 410 may also include a disk/media controller 443 coupled to the system bus 421 to control one or more storage devices for storing information and instructions, such as a magnetic hard disk 441 and/or a removable media drive 442 (e.g., floppy disk drive, compact disc drive, tape drive, flash drive, and/or solid state drive). Storage devices 440 may be added to the computer system 410 using an appropriate device interface (e.g., a small computer system interface (SCSI), integrated device electronics (IDE), Universal Serial Bus (USB), or FireWire). Storage devices 441 , 442 may be external to the computer system 410.

[0043] The computer system 410 may perform a portion or all of the processing steps of embodiments of the invention in response to the processors 420 executing one or more sequences of one or more instructions contained in a memory, such as the system memory 430. Such instructions may be read into the system memory 430 from another computer readable medium of storage 440, such as the magnetic hard disk 441 or the removable media drive 442. The magnetic hard disk 441 (or solid state drive) and/or removable media drive 442 may contain one or more data stores and data files used by embodiments of the present disclosure. The data store 440 may include, but are not limited to, databases (e.g., relational, object-oriented, etc.), file systems, flat files, distributed data stores in which data is stored on more than one node of a computer network, peer-to-peer network data stores, or the like. The data stores may store various types of data such as, for example, skill data, sensor data, or any other data generated in accordance with the embodiments of the disclosure. Data store contents and data files may be encrypted to improve security. The processors 420 may also be employed in a multi-processing arrangement to execute the one or more sequences of instructions contained in system memory 430. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software. [0044] As stated above, the computer system 410 may include at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to the processors 420 for execution. A computer readable medium may take many forms including, but not limited to, non-transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks, such as magnetic hard disk 441 or removable media drive 442. Non-limiting examples of volatile media include dynamic memory, such as system memory 430. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up the system bus 421. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0045] Computer readable medium instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, statesetting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure. [0046] Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer readable medium instructions.

[0047] The computing environment 400 may further include the computer system 410 operating in a networked environment using logical connections to one or more remote computers, such as remote computing device 480. The network interface 470 may enable communication, for example, with other remote devices 480 or systems and/or the storage devices 441, 442 via the network 471. Remote computing device 480 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 410. When used in a networking environment, computer system 410 may include modem 472 for establishing communications over a network 471, such as the Internet. Modem 472 may be connected to system bus 421 via user network interface 470, or via another appropriate mechanism.

[0048] Network 471 may be any network or system generally known in the art, including the Internet, an intranet, a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a direct connection or series of connections, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 410 and other computers (e.g., remote computing device 480). The network 471 may be wired, wireless or a combination thereof. Wired connections may be implemented using Ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection generally known in the art. Wireless connections may be implemented using Wi-Fi, WiMAX, and Bluetooth, infrared, cellular networks, satellite or any other wireless connection methodology generally known in the art. Additionally, several networks may work alone or in communication with each other to facilitate communication in the network 471.

[0049] It should be appreciated that the program modules, applications, computer-executable instructions, code, or the like depicted in FIG. 4 as being stored in the system memory 430 are merely illustrative and not exhaustive and that processing described as being supported by any particular module may alternatively be distributed across multiple modules or performed by a different module. In addition, various program module(s), script(s), plug-in(s), Application Programming Interface(s) (API(s)), or any other suitable computer-executable code hosted locally on the computer system 410, the remote device 480, and/or hosted on other computing device(s) accessible via one or more of the network(s) 471, may be provided to support functionality provided by the program modules, applications, or computer-executable code depicted in FIG. 4 and/or additional or alternate functionality. Further, functionality may be modularized differently such that processing described as being supported collectively by the collection of program modules depicted in FIG. 4 may be performed by a fewer or greater number of modules, or functionality described as being supported by any particular module may be supported, at least in part, by another module. In addition, program modules that support the functionality described herein may form part of one or more applications executable across any number of systems or devices in accordance with any suitable computing model such as, for example, a client-server model, a peer-to-peer model, and so forth. In addition, any of the functionality described as being supported by any of the program modules depicted in FIG. 4 may be implemented, at least partially, in hardware and/or firmware across any number of devices.

[0050] It should further be appreciated that the computer system 410 may include alternate and/or additional hardware, software, or firmware components beyond those described or depicted without departing from the scope of the disclosure. More particularly, it should be appreciated that software, firmware, or hardware components depicted as forming part of the computer system 410 are merely illustrative and that some components may not be present or additional components may be provided in various embodiments. While various illustrative program modules have been depicted and described as software modules stored in system memory 430, it should be appreciated that functionality described as being supported by the program modules may be enabled by any combination of hardware, software, and/or firmware. It should further be appreciated that each of the above-mentioned modules may, in various embodiments, represent a logical partitioning of supported functionality. This logical partitioning is depicted for ease of explanation of the functionality and may not be representative of the structure of software, hardware, and/or firmware for implementing the functionality. Accordingly, it should be appreciated that functionality described as being provided by a particular module may, in various embodiments, be provided at least in part by one or more other modules. Further, one or more depicted modules may not be present in certain embodiments, while in other embodiments, additional modules not depicted may be present and may support at least a portion of the described functionality and/or additional functionality. Moreover, while certain modules may be depicted and described as sub-modules of another module, in certain embodiments, such modules may be provided as independent modules or as sub-modules of other modules.

[0051] Although specific embodiments of the disclosure have been described, one of ordinary skill in the art will recognize that numerous other modifications and alternative embodiments are within the scope of the disclosure. For example, any of the functionality and/or processing capabilities described with respect to a particular device or component may be performed by any other device or component. Further, while various illustrative implementations and architectures have been described in accordance with embodiments of the disclosure, one of ordinary skill in the art will appreciate that numerous other modifications to the illustrative implementations and architectures described herein are also within the scope of this disclosure. In addition, it should be appreciated that any operation, element, component, data, or the like described herein as being based on another operation, element, component, data, or the like can be additionally based on one or more other operations, elements, components, data, or the like. Accordingly, the phrase “based on,” or variants thereof, should be interpreted as “based at least in part on.”

[0052] Although embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments. Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments could include, while other embodiments do not include, certain features, elements, and/or steps. Thus, such conditional language is not generally intended to imply that features, elements, and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements, and/or steps are included or are to be performed in any particular embodiment.

Claims

CLAIMS What is claimed is:

1. A computer-implemented method for calibrating a robotic device disposed in a physical environment with an augmented reality (AR) computing device that defines a camera and a user interface, the method comprising: the AR computing device identifying the robotic device and a pose of the robotic device; based on identifying the robotic device and the pose of the robotic device, obtaining a world model of the robotic device, the world model defining a geometry of the robotic device; simulating an operation of the robotic device using the world model, so as to define a simulated operation that defines a simulated position of the robotic device; performing the operation of the robotic device in the physical environment, so as to define a real-world operation that defines an actual position of the robotic device; while performing the operation of the robotic device, displaying the robotic device on the user interface of the AR computing device; and overlaying the geometry of the robotic device in the simulation position on the user interface that displays the actual position of the robotic device.

2. The computer- implemented method as recited in claim 1, the method further comprising: determining a first offset between the world model and the physical environment.

3. The computer-implemented method as recited in claim 2, wherein determining the first offset further comprises: receiving an input via the user interface, the input indicating a difference between the simulated position of the robotic device as shown on the user interface and the actual position of the robotic device as shown on the user interface.

4. The computer-implemented method as recited in claim 3, wherein the input defines a marker on the user interface or a number input.

5. The computer-implemented method as recited in claim 2, wherein the first offset defines a distance between the geometry of the robotic device in the simulation position and the robotic device in the actual position.

6. The computer-implemented method as recited in claim 2, wherein the first offset defines a difference between a first position of an object relative to the robotic device in the simulation position, and a second position of the object relative to the robotic device in the actual position.

7. The computer-implemented method as recited in claim 6, wherein the simulation position of the robotic device is equivalent to the actual position of the robotic device.

8. The computer- implemented method as recited in claim 7, the method further comprising: based on identifying the robotic device and the pose of the robotic device, obtaining a model of the object from the world model, the model defining a first orientation and a first location of the object that is different than a second orientation and a second location of the object in the second position relative to the robotic device.

9. The computer- implemented method as recited in claim 2, the method further comprising: adjusting the world model based on the first offset, so as to calibrate the world model for the operation.

10. The computer- implemented method as recited in claim 9, the method further comprising: re-peforming the operation of the robotic device in the physical environment; and after re-performing the operation, determining a second offset between the world model and the physical environment, the second offset less than the first offset.

11. The computer- implemented method as recited in claim 1, wherein identifying the robotic device further comprises: positioning the camera so that the robotic device is in view of the camera; capturing, by the camera, video of the robotic device, so as to define video input; and matching, by the AR computing device, the video input to a first object of the world model, the first object corresponding to the robotic device.

12. The computer- implemented method as recited in claim 1, the method further comprising: the AR computing device controlling the operation of the robotic device.

13. The computer- implemented method as recited in claim 1, the method further comprising: receiving a user command via the user interface; and based on the user command, performing the operation of the robotic device in the physical environment.

14. The computer- implemented method as recited in claim 1, the camera is positioned at the robotic device so that that robotic device is visible on the user interface of the AR computing device while the operation of the robotic device is performed.

15. An augmented reality (AR) computing system, the AR computing system comprising: a user interface configured to receive input from users and display video input and augmented reality visual indications; a camera configured to capture the video input; a processor; and a memory storing instructions that, when executed by the processor, cause the AR computing system to: identify a robotic device and a pose of the robotic device within a physical environment; obtain a world model of the robotic device, the world model defining a geometry of the robotic device; simulate an operation of the robotic device using the world model, so as to define a simulated operation that defines a simulated position of the robotic device; perform the operation of the robotic device in the physical environment, so as to define a real-world operation that defines an actual position of the robotic device; and overlay the geometry of the robotic device in the simulation position on the user interface that displays the actual position of the robotic device.