CN104782122B - Controlled three-dimensional communication endpoint system and method - Google Patents

Abstract

A controlled three-dimensional (3D) communication endpoint system and method for emulating in-person communication between participants in an online meeting or conference and providing easy scalability of the virtual environment when additional participants join. This gives the participant the illusion that the other participants are in the same room and sitting around the same table as the viewer. The controlled communication endpoint includes a plurality of camera clusters that capture video of the participants 360 degrees around the participants. The controlled communication endpoint also includes a display device configuration including a display device positioned at least 180 degrees around the participant and displaying a virtual environment including geometric proxies of other participants. Scalability is easily achieved by placing participants at the virtual round table and increasing the diameter of the virtual table when additional participants are added.

Description

Controlled three-dimensional communication endpoint system and method

background

Current videoconferencing technologies typically use a single camera to capture RGB data (from the red, blue, and green (RGB) color model) of the local scene. This local scene typically includes people participating in a video conference, known as conference participants. The data is then transmitted in real time to the remote location and then displayed to another meeting participant at a different location than the other meeting participants.

Although advances have been made in videoconferencing technology to help provide higher resolution capture, compression, and transmission, the experience often falls short of recreating the face-to-face experience of an in-person meeting. One reason for this is the lack of eye gaze and other corrective conversation geometry that is typical of the video conferencing experience. For example, often, the person being captured remotely does not meet your eyes the way it is experienced in a face-to-face conversation. Furthermore, three-dimensional (3D) elements like motion parallax and image depth and freedom to change the perspective in the scene are lacking since there is only a single fixed camera capturing the scene and the conference participants.

Contents of the invention

This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of the controlled three-dimensional (3D) communication endpoint systems and methods simulate in-person communication between participants in an online meeting or conference. Additionally, embodiments of the controlled 3D communication endpoint system and method allow for easy scaling of a virtual environment containing participants such that additional participants can be added by merely increasing the size of a virtual table contained in the virtual environment. Furthermore, controlled endpoints allow the viewer to feel as if other participants are in the same room as him.

Specifically, embodiments of the controlled 3D communication endpoint systems and methods use multiple camera clusters at the endpoint to capture 3D video images of participants. The camera clusters in the controlled endpoints are arranged such that they allow the participant to be captured 360 degrees around the participant. Based on the data captured by the video, geometric proxies are created for the participants. Using RGB data and depth information from captured video, geometric proxies are created for each participant.

Scene geometry is created by embodiments of the systems and methods from eye gaze and dialogue geometry that would exist in presence communications. The general concept of scene geometry is to create relative geometry between actors. The scenes are virtually aligned to simulate real life scenarios as if the participants were in the same physical location and participating in the in-person communication.

Scene geometry uses virtual boxes to maintain relative, consistent geometry between participants. A conference with two participants (or one-to-one (1:1) scene geometry) includes two boxes occupying the space in front of the respective monitors (not shown) of the two participants. When there are three participants, the scene geometry includes three virtual boxes placed equidistantly around the virtual round table.

Scene geometry also includes virtual cameras. A virtual camera is a composite of images from two or more of multiple camera groups to obtain a camera view that was not captured by any one camera group alone. This allows embodiments of the systems and methods to achieve natural eye gaze and connection between people. Face tracking technology can be used to improve performance by helping the virtual camera stay in alignment with the viewer's eye gaze. This means that the virtual camera remains horizontal and aligned with the viewer's eyes both vertically and horizontally. The virtual camera interacts with face tracking to create a virtual point of view with the user looking where the user's eyes are looking. Thus, if the user is looking into the distance, the virtual viewpoint starts from the angle from which the user is looking into the distance. If the user is looking at another participant, the virtual point of view is from the angle from which the user is looking at the other participant. This is done not by artificially making it look like the user is looking at another participant, but by creating virtual geometry that correctly represents where the user is looking.

Geometry proxies are rendered relative to each other and placed into the virtual environment along with the scene geometry. The rendered geometry proxy and scene geometry are communicated to each of the participants. The virtual environment is displayed to the viewer (who is also one of the participants) in the controlled environment of the endpoint. Specifically, each endpoint contains a display device configuration that displays a virtual environment to a viewer using a virtual viewpoint. The virtual point of view depends on the position and orientation of the viewer's eyes. Depending on the position and orientation of the eyes, the viewer sees different angles of other participants in the meeting and other aspects of the virtual environment.

The registration of the real space to the virtual space ensures that the displayed image is what the viewer would see if she were looking at other participants in the virtual environment. Additionally, face tracking technology can be used to track the viewer's eyes to know what the virtual viewpoint should display. Controlling the size and placement of endpoints makes it easier to build solutions in order to create real geometry for participants at scale in an efficient manner and to help maintain the illusion that participants are all in one physical location.

A display device configuration contains multiple display devices (such as monitors or screens). The display device configuration controls the endpoint environment such that the display devices are arranged at least 180 degrees around the viewer. This ensures that the viewer has an immersive experience and feels as if he is actually in the same physical space as the other participants.

Embodiments of the system and method also allow for easy scalability. Specifically, in some embodiments, the virtual table is a circular (or annular) virtual table having a first diameter. A geometric proxy for each of the participants is placed in the virtual environment around the virtual table. This ensures that viewers can see every one of the participants surrounding the virtual table. If more participants are added to the online meeting, the size of the virtual round table is expanded to a second diameter that is larger than the first diameter. The second diameter may be any diameter greater than the first diameter. This extension keeps each of the participants still in view for viewing and gives the illusion of being in the same room with the other participants around the table.

Embodiments of the systems and methods also include facilitating multiple participants at a single endpoint. In some embodiments, face tracking technology tracks two different faces and then provides different views to different viewers. In other embodiments, each of the plurality of participants at the endpoint wears glasses, and in some embodiments, has a shutter on the glasses that shows each wearer the information displayed by the monitor that is tuned to each Alternate frames of a pair of glasses. Other embodiments use monitors with multiple viewing angles such that a viewer who is looking at the monitor from the right sees one scene and another viewer who is looking at the monitor from the left sees a different scene.

It should be noted that alternative embodiments are possible and that steps and elements discussed herein may be changed, added or eliminated depending on the particular embodiment. These alternative embodiments include alternative steps and elements that may be used, and structural changes that may be made without departing from the scope of the present invention.

Brief description of the drawings

Referring now to the drawings, like reference numerals designate corresponding parts throughout:

1 is a block diagram illustrating a general overview of embodiments of a controlled three-dimensional (3D) communication endpoint system and method implemented in a computing environment.

FIG. 2 is a block diagram showing system details of the 3D communication processing system shown in FIG. 1 .

FIG. 3 is a block diagram showing details of an exemplary embodiment of a camera cluster of embodiments of the controlled 3D communication endpoint and method shown in FIG. 1 .

FIG. 4 illustrates an exemplary embodiment of a camera cluster layout such as that shown in FIG. 2 using four camera clusters.

FIG. 5 shows an exemplary embodiment of a display device configuration such as that shown in FIG. 1 using three display devices.

Figure 6 illustrates a simplified example of a general-purpose computer system on which various embodiments and elements of the 3D communication window systems and methods described herein and shown in Figures 1-5 and 7-15 may be implemented.

FIG. 7 is a flowchart showing the overall operation of the controlled 3D communication endpoint system shown in FIG. 1 .

FIG. 8 is a flowchart showing the overall operation of the 3D communication processing system shown in FIG. 1 .

Figure 9 illustrates an exemplary embodiment of extending the embodiments of the systems and methods to accommodate additional endpoints.

Figure 10 shows an exemplary overview of creating a geometric proxy for a single conference participant.

Figure 11 shows an exemplary embodiment of scene geometry between participants when there are two participants (at two different endpoints) in an online meeting.

Figure 12 shows an exemplary embodiment of scene geometry between participants when there are three participants at three different endpoints in an online meeting.

Figure 13 illustrates an exemplary embodiment of a virtual camera based on where a participant is looking.

FIG. 14 illustrates an exemplary embodiment in which depth is provided by motion parallax based on where the viewer is facing.

15 illustrates an exemplary embodiment of a technique for handling multiple participants at a single endpoint using monitors with multiple viewing angles.

A detailed description

In the following description of the controlled three-dimensional (3D) communication endpoint system and method, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration an aspect of the practicable 3D communication endpoint system and method A specific example of each embodiment. It is to be understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the claimed subject matter.

I. System overview

Embodiments of the controlled 3D communication endpoint system and method create a controlled capture and viewing space for immersive online meetings and meetings. Embodiments of the system and method ensure consistency at endpoints when participants join an online meeting or meeting. Each endpoint is fully controlled during the online meeting, including lighting, room design, and geometry. In addition, endpoints include gear for capturing and viewing immersive meetings in 3D, such that to the viewer it appears that other participants are actually in the same room (or same physical space) as the participant.

An endpoint is a physical location, such as a room or other type of environment, that contains at least one of the participants of an online meeting or meeting. Each online meeting has at least two endpoints, where each endpoint has at least one participant. Each endpoint can have two or more participants. The manner in which endpoints with two or more participants are handled is discussed in detail below.

1 is a block diagram illustrating a general overview of embodiments of a controlled three-dimensional (3D) communication endpoint system 100 and method implemented in a computing environment. Embodiments of the system 100 and method include various components and systems that work together to create an immersive experience for participants in an online meeting or conference.

As shown in FIG. 1 , the system 100 and method includes a 3D communication processing system 105 that facilitates a participant's immersive experience. The 3D communication processing system 105 is implemented on a computing device 110 . This computing device can be a single computing device or can be distributed across multiple devices. Moreover, computing device 110 may be virtually any device having a processor, including desktop computers, tablet computing devices, and embedded computing devices.

Various embodiments of the system 100 and method include at least two endpoints. For pedagogical and ease of explanation purposes, Figure 1 shows only two endpoints. It should be noted, however, that various embodiments of the system 100 and method may include several more endpoints. Furthermore, while only a single participant is shown for each endpoint in Figure 1, it should be noted that any number of participants may be included at any endpoint.

Various embodiments of the system 100 and method include a first endpoint 115 and a second endpoint 120 . In FIG. 1 , the first end point 115 and the second end point 120 are shown in plan view. In other words, if the first and second endpoints 115, 120 are rooms, then Figure 1 is a plan view of the room.

The first endpoint 115 includes a first participant 125 contained therein. The first endpoint 115 also includes a number of capture and viewing devices. The viewing devices at the first endpoint 115 include a first monitor 130 , a second monitor 135 , and a third monitor 140 . The viewing device provides the first participant 125 with an immersive experience in the online meeting such that the first participant 125 feels as if he is in the room with the other participants.

Embodiments of the system 100 and method include monitor configurations having monitors or screens arranged such that they circle at least 180 degrees around a participant. The configuration of monitors can be virtually any arrangement as long as they are placed at least 180 degrees around the participant. As explained in detail below, this ensures that the participants' experience is fully immersive and enables scaling depending on the number of online meeting participants.

The monitor configuration in FIG. 1 shows second and third monitors 135 , 140 in the first end point 115 at right angles to the first monitor 130 . Additionally, the monitors 130 , 135 , 140 in the first endpoint 115 circle the first participant 125 at least 180 degrees. In alternative embodiments, the monitor configurations may be curved, such as semicircles, or may be at less than right angles to each other.

Embodiments of the system 100 and method also include a capture device for capturing at least a portion of the first participant 125 within the first endpoint 115 . Embodiments of the system 100 and method use multiple camera clusters as capture devices. It should be noted that while six camera clusters are shown in Figure 1, fewer or more camera clusters may be used.

As shown in FIG. 1 , the first endpoint 115 includes a first plurality of camera clusters 145 located in front of the first participant 125 and a second plurality of camera clusters 150 located behind the first participant 125 . The details of each camera group are explained in detail below. FIG. 1 shows a first plurality of camera clusters 145 attached to a first monitor 130 and a second plurality of cameras attached to a support structure at a first end point 115, such as a wall in a room or the floor of a room. Camera group 150. It should be noted, however, that in alternative embodiments, the first and second plurality of camera groups 145, 150 may be mounted on some other structure, or some camera groups may be mounted on the first monitor 130 and other cameras Swarms are mounted on other structures.

The second endpoint 120 includes a second participant 155 contained therein. Similar to the first endpoint 115, the second endpoint 120 also includes multiple capture and viewing devices. The viewing devices at the second endpoint 120 include a fourth monitor 160 , a fifth monitor 165 , and a sixth monitor 170 . These monitors 160, 165, 170 provide the second participant 155 with an immersive experience in the online meeting, making the first participant 125 feel as if he is in the room with the other participants.

The monitor configuration in FIG. 1 shows fifth and sixth monitors 165 , 170 in the second end point 120 having an angle of less than 90 degrees from the fourth monitor 160 . Additionally, the monitors 160 , 165 , 170 in the second endpoint 120 circle the second participant 155 at least 180 degrees. In alternative embodiments, the monitor configuration may also be curved, such as a semicircle.

Embodiments of the system 100 and method also include a capture device for capturing at least a portion of the second participant 155 within the second endpoint 120 . Embodiments of the system 100 and method use multiple camera clusters as capture devices. It should be noted that although ten camera clusters are shown in the second endpoint 120 in FIG. 1 , fewer or more camera clusters may be used.

As shown in FIG. 1 , the second endpoint 120 includes a third plurality of camera clusters 175 located in front of the second participant 155 and a fourth plurality of camera clusters 180 located behind the second participant 155 . The details of each camera group are explained in detail below. Additionally, the fifth plurality of camera groups 185 is located to the left of the second participant 155 and the sixth plurality of camera groups 190 is located to the right of the second participant 155 .

1 shows that the third plurality of camera groups 175 is attached to the fourth monitor 160, the fifth plurality of camera groups 185 is attached to the fifth monitor 165, and the sixth plurality of camera groups 190 is attached to the sixth Monitor 170. A fourth plurality of camera groups 180 is attached to a supporting structure of the second end point 120 (eg, a wall in a room or the floor of a room). However, it should be noted that in alternative embodiments, the third, fourth, fifth, and sixth plurality of camera groups 175, 180, 185, 190 may be mounted on some other structure, or some of the camera groups may be mounted on On other structures within the second endpoint 120.

The first participant 125 is captured by the camera cluster in the first endpoint 115 and the second participant is captured by the camera cluster in the second endpoint 120 . This captured information is then communicated to various embodiments of the 3D communication processing system 105, as explained in detail below. The capture device of the first endpoint 115 communicates with the 3D communication processing system 105 through the network 195 . Communication between network 195 and first endpoint 115 is facilitated using a first communication link. Similarly, communication between network 195 and second endpoint 120 is facilitated by second communication link 198 . In FIG. 1 , various embodiments of 3D communication processing system 105 are shown as residing on network 195 . It should be noted, however, that this is only one way in which the 3D communication processing system 105 may be implemented within various embodiments of the system 100 and method.

The captured information is processed and sent to endpoints for viewing on monitors. Embodiments of the system 100 and method provide virtual viewpoints to each participant at each endpoint. As explained in detail below, virtual viewpoints allow viewers to view an online meeting from variable perspectives that depend on the position and orientation of the viewer's face. In some embodiments, face tracking is used to track the viewer's eye gaze and determine how the processed information should be presented to the viewer.

II. System Details

Embodiments of the system 100 and method include various components and devices that are used together to provide participants with an immersive experience in an online meeting. These components and devices will now be discussed. It should be noted that other embodiments are possible and other devices may be used or substituted to achieve the purpose and function of the components and devices discussed.

Embodiments of the system 100 and method include three main components that work together to create an "in-person" communication experience. The first component is to capture and create a 3D video image of everyone participating in the meeting. The second component is to create relevant scene geometry based on the number of participants in the conference. And, the third component is to render and provide a virtual view as if the camera were placed where the viewer is looking, thus recreating the same scene geometry that the participants would have if they were talking in person.

II.A. 3D communication processing system

FIG. 2 is a block diagram showing system details of the 3D communication processing system 105 shown in FIG. 1 . As shown in FIG. 2 , the 3D communication processing system 105 includes a capture and creation component 200 , a scene geometry component 210 , and a virtual viewpoint component 220 . Capture and creation component 200 is used to capture and create 3D video images of participants at endpoints.

Specifically, capture and creation component 200 includes a camera cluster layout 230 that includes multiple camera clusters. Camera group layout 230 is used to capture participants from multiple perspectives. Computer vision methods were used to create high-fidelity geometric proxies of each meeting participant. As explained in detail below, this is achieved by taking the RGB data obtained from the RGB data collection module 235 and the depth information obtained and calculated by the depth information calculation module 240 . From this information, geometric proxy creation module 245 creates a geometric proxy 250 for each participant. Image-based rendering methods are used to create realistic textures for geometry proxy 250, such as with view-dependent texture mapping.

The scene geometry component 210 is used to create the correct scene geometry to simulate the participants having a real conversation together. This scenario geometry depends on the number of participants in the conference. The 3D registration module 260 is used to obtain precise registration of the display device or monitor with the camera cluster. Additionally, the spatial alignment module 265 aligns the orientation of the camera population to the real world. For a 1:1 meeting (with two endpoints), this is simply lining up the two physical spaces facing each other in a virtual environment. The capture area recreated for each participant is the area in front of the monitor.

Once the textured geometric proxies 250 are created for each conference participant and the participants are represented in a 3D virtual space in relation to the other participants in the conference, the geometric proxies are rendered to each other in a manner consistent with the conversation geometry . Also, this rendering is done based on the number of participants in the meeting.

Geometric proxies (and in some cases registration and alignment information) are communicated by transmission module 270 to remote participants. The virtual viewpoint component 220 is used to enhance the virtual viewpoint rendered to the remote participant. The 'on-set' experience is enhanced by using a motion parallax module 280 that adds motion parallax and depth to the scene behind the participant. Horizontal and lateral movement of either participant changes the point of view shown on their local display, and the participant sees the scene they are looking at and the people in it from a different perspective. This greatly enhances the experience of meeting participants.

II.B. Camera group

As noted above, the capture and creation component 200 of the system 100 and method includes multiple camera clusters that are used to capture participants and scenes in endpoints. Each camera group has multiple sensors. FIG. 3 is a block diagram showing details of an exemplary embodiment of a camera cluster 300 of embodiments of the controlled 3D communication endpoint system 100 and method shown in FIG. 1 . As shown in FIG. 1 , various embodiments of the system 100 and method generally include more than one camera cluster 300 . However, for educational purposes only, a single camera cluster will be described. Furthermore, it should be noted that multiple camera clusters do not necessarily have to include the same sensor. Some embodiments of the system 100 and method may include multiple camera clusters that include different sensors from each other.

As shown in FIG. 3 , camera group 300 includes a plurality of camera sensors. These sensors include a stereo sensor infrared (IR) camera 310 , an RGB camera 320 , and an IR emitter 330 . To capture 3D images of participants and endpoints, camera cluster 300 captures RGB data and depth coordinates to compute a depth map. Figure 3 shows an IR stereo IR camera 310 and IR emitter 330 are used to capture depth calculations. RGB camera 320 is used for texture acquisition and depth cues are enhanced using depth segmentation. Depth segmentation, well known in the field of computer vision, seeks to separate objects in an image from the background using background removal.

In an alternative embodiment, instead of the IR structured light approach, the camera cluster 300 uses time-of-flight sensors or ultrasound to achieve stereo sensing. A time-of-flight camera is a range-imaging camera system that calculates the distance of each point in an image based on the speed of light and by measuring the time-of-flight of the light signal between the camera and the object. Ultrasound techniques can be used to calculate distance by generating ultrasound pulses in a certain direction. If there is an object in the path of the pulse, some or all of that pulse will be reflected back to the transmitter as an echo. Distance can be derived by measuring the difference between the transmitted pulse and the received echo. In other embodiments, the distance may be derived by performing RGB depth calculations using a stereo pair of RGB cameras.

II.C. Camera Group Layout

The one or more camera clusters are configured in a particular layout to capture 3D images of the endpoint including one or more of the participants. The number of camera clusters directly affects the quality of captured images as well as the amount of occlusions. As the number of camera clusters increases, more RGB data is available and this improves image quality. Furthermore, the number of occlusions will decrease as the number of camera clusters increases.

As shown in FIG. 1 , the first endpoint 115 includes 6 camera clusters and the second endpoint 120 includes 10 camera clusters. In alternative embodiments, any number of cameras may be used. In fact, there may be lower end versions that use a single camera cluster. For example, a single camera cluster could be mounted on top of a monitor and use image distortion correction techniques to correct any imaging errors. The criterion is that the camera cluster layout should have enough camera clusters to provide a 3D view of the endpoint containing the participants.

FIG. 4 illustrates an exemplary embodiment of a camera cluster layout such as that shown in FIG. 2 using four camera clusters. As shown in FIG. 4 , four camera groups 300 are embedded in a frame of a monitor 400 . Monitor 400 can be virtually any size, but larger monitors provide more life-size re-projections. This generally provides a more realistic experience to the user. Displayed on monitor 400 are remote participants 410 participating in an online meeting or meeting.

As shown in FIG. 4, four camera clusters 300 are arranged in a diamond configuration. This allows embodiments of the system 100 and method to capture users from top to bottom and side to side. Furthermore, two middle top and bottom camera clusters can be used to seamlessly obtain the real texture of the user's face. Note that cameras in corners will often cause seam problems. In other embodiments, virtually any configuration and arrangement of the four camera clusters 300 may be used and mounted anywhere on the monitor 400 . In still other embodiments, one or more of the four camera clusters 300 are mounted at locations other than the monitor 400 .

In an alternate embodiment, three camera clusters are used and are located on the top or bottom of the monitor 400 . Some embodiments use two camera clusters located at the top and bottom corners of monitor 400 . In still other embodiments, N camera groups are used, where N is greater than four (N>4). In this embodiment, N camera clusters are placed around the outer edge of the monitor 400. In yet other embodiments, multiple camera clusters are located behind the monitor 400 to capture the 3D scene including the participants' endpoints.

II.D. Display device configuration

Several display devices, such as monitors and screens, are configured in a particular layout to display and present to each participant the captured images of at least some of the other participants. Embodiments of the system 100 and method configure the display devices such that the arrangement is at least 180 degrees around the participants in the endpoints. This ensures that embodiments of the system 100 and method can scale and provide participants with an immersive experience. In other words, providing at least a 180 degree display device to the participants in the endpoint enables them to see everyone at the virtual table at the same time. Using at least a 180 degree display device, as the viewer looks right and left around the virtual round table, she will be able to see everyone at the table.

FIG. 5 shows an exemplary embodiment of a display device configuration such as that shown in FIG. 1 using three display devices. As shown in FIG. 5 , display device configuration 500 is deployed in endpoint environment 510 . The display device configuration 500 includes a monitor #1 520 positioned such that it is in front of a participant (not shown) in the endpoint environment 510 . The display device configuration also includes monitor #2530 and monitor #3540 located on both sides of monitor #1520. As shown in Figure 5, monitor #2 530 and monitor #3 540 are each attached to or in contact with monitor #1 520 at a 45 degree angle.

Embodiments of the system 100 and method use the endpoint environment 510 for capture and display. In some embodiments, display device configuration 500 may be a 360-degree configuration. In other words, there may be a display device that completely surrounds a participant in endpoint environment 510 . In other embodiments, the display device may comprise a display device arranged to surround the endpoint environment 510 in any degree ranging from 180 degrees to 360 degrees inclusive. In yet other embodiments, the display device configuration 500, wherein all walls and ceilings of the endpoint environment 510 are display devices. This type of display configuration allows participants to fully immerse themselves in a purely virtual environment.

III. Exemplary Operating Environment

Before continuing further with the operational overview and details of embodiments of the controlled 3D communication endpoint system 100 and method, an exemplary operating environment in which embodiments of the controlled 3D communication endpoint system 100 and method may operate will now be presented discussion. Embodiments of the controlled 3D communication endpoint system 100 and method are operable within numerous types of general purpose or special purpose computing system environments or configurations.

FIG. 6 shows a simplified example of a general-purpose computer system on which various embodiments and elements of the 3D communication endpoint system 100 and method described herein and shown in FIGS. 1-5 and 7-15 may be implemented. It should be noted that any blocks represented by broken or dashed lines in FIG. 6 represent alternative implementations of a simplified computing device, and that any or all of these alternative implementations described below may be combined with other alternative implementations described throughout this document. use.

For example, FIG. 6 shows a general system diagram showing a simplified computing device 10 . The simplified computing device 10 may be a simplified version of the computing device 110 shown in FIG. 1 . Such computing devices are typically found in devices with at least some minimal computing capabilities, including but not limited to personal computers, server computers, handheld computing devices, laptop or mobile computers, communication devices such as cell phones and PDAs , multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, audio or video media players, etc.

To allow a device to implement embodiments of the controlled 3D communication endpoint system 100 and method described herein, the device should have sufficient computing power and system memory to enable basic computing operations. Specifically, as shown in FIG. 6 , computing power is typically represented by one or more processing units 12, and may also include one or more GPUs 14, either or both of which are associated with the system The memory 16 communicates. Note that the processing unit 12 of a general-purpose computing device may be a special-purpose microprocessor, such as a DSP, VLIW, or other microcontroller, or may be a conventional CPU with one or more processing cores, including GPU-based special-purpose CPUs in multi-core CPUs. nuclear.

Additionally, the simplified computing device 10 of FIG. 6 may also include other components, such as a communication interface 18 and the like. The simplified computing device 10 of FIG. 6 may also include one or more conventional computer input devices 20, such as a stylus, pointing device, keyboard, audio input device, video input device, tactile input device, for receiving wired or wireless data transmissions. equipment, etc.). The simplified computing device 10 of FIG. 6 may also include other optional components, such as, for example, one or more conventional computer output devices 22 (e.g., a display device 24, an audio output device, a video output device, for wired or wireless data transmission) equipment, etc.). Note that typical communication interfaces 18, input devices 20, output devices 22, and storage devices 26 of a general purpose computer are well known to those skilled in the art and will not be described in detail here.

The simplified computing device 10 of FIG. 6 may also include various computer-readable media. Computer-readable media can be any available media that can be accessed by simplified computing device 10 via storage device 26, and includes both volatile and non-volatile media that are removable 28 and/or non-removable 30 for storing information such as computer-readable Information such as readable or computer-executable instructions, data structures, program modules, or other data. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media. Computer storage media includes, but is not limited to: computer or machine readable media or storage devices such as DVDs, CDs, floppy disks, tape drives, hard drives, optical drives, solid state memory devices, RAM, ROM, EEPROM, flash memory or other memory technologies, Magnetic tape cartridges, tapes, disk storage or other magnetic storage devices, or any other device that can be used to store desired information and can be accessed by one or more computing devices.

Information such as computer-readable or computer-executable instructions, data structures, program modules, etc. may also be retained by encoding one or more modulated data signals or carrier waves or other transport mechanisms using any of the various aforementioned communication media or communication protocol, and includes any wired or wireless information transfer mechanism. Note that the terms "modulated data signal" or "carrier carrier" generally refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. For example, communication media includes wired media such as a wired network or direct-wire connection, which carry one or more modulated data signals, and wireless media such as acoustic, RF, infrared, laser and other wireless media for transmitting and/or receiving one or more A wireless medium with a modulated data signal or carrier wave. Any combination of the above communication media should also be included within the scope of communication media.

In addition, the controlled 3D software embodying the controls described herein may be stored, received, transmitted, or read from computer- or machine-readable media or any desired combination of storage devices and communication media in the form of computer-executable instructions or other data structures. Some or all of the software, programs and/or computer program products or portions thereof in various embodiments of the communication endpoint system 100 and method.

Finally, embodiments of the controlled 3D communication endpoint system 100 and methods described herein may also be described in the general context of computer-executable instructions, such as program modules, being executed by a computing device. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Embodiments described herein can also be practiced in distributed computing environments where tasks are performed by one or more remote processing devices that are linked through one or more communications networks, or in a cloud of one or more devices. In a distributed computing environment, program modules may be located in both local and remote computer storage media including media storage devices. Furthermore, the above-described instructions may be implemented in part or in whole as hardware logic circuits that may or may not include a processor.

IV. OPERATIONAL OVERVIEW

FIG. 7 is a flowchart showing the overall operation of the controlled 3D communication endpoint system 100 shown in FIG. 1 . As shown in FIG. 7, operation of the system 100 begins by capturing 3D video of a local participant at a local endpoint (block 700). As an example, a local endpoint may be a room in an office building. Captured video is obtained using multiple camera clusters that capture both RGB data and depth information (block 705). The plurality of camera clusters are positioned 360 degrees around the local participant. In other words, the captured video contains views all the way around the local participant.

Embodiments of the method then use the captured 3D video to create a local geometric proxy of the local participant (block 710). Next, the method generates scene geometry consistent with in-person communications (block 715). The general concept is to create a virtual environment that mimics the dynamics of in-person communication. The method then places native geometry proxies into the scene geometry to create the virtual environment (block 720). The local geometry proxy and scene geometry are communicated to the remote participant at the remote endpoint (block 725).

Similarly, multiple camera fleets are used to capture remote participants and any other participants participating in an online conference or meeting and create geometric proxies for each of them. Each of these geometric proxies is rendered and placed within the scene geometry of the virtual environment. These rendered geometric proxies and scene geometry are then communicated to other participants.

The received virtual environment is displayed to a viewer (eg, the remote participant) on a display device in the endpoint occupying a space of at least 180 degrees around the remote participant (block 730). This provides remote participants with a virtual viewpoint into the virtual environment. As explained in detail below, what the viewer sees when looking at the virtual viewpoint depends in part on the position and orientation of the viewer's head.

Embodiments of the method define virtual tables within a virtual environment. Each of the rendered participants is then placed around a virtual table in the virtual environment. In some embodiments, the virtual table has a circular shape with a first diameter (block 735). This allows scaling to occur easily. Specifically, the virtual environment may be expanded by increasing the number of participants beyond the current two participants (local participant and remote participant) (block 740). To accommodate this increase in participants, the method then increases the size of the virtual table from the first diameter to a second diameter, wherein the second diameter is larger than the first diameter (block 745). The participants' geometric proxies are placed at the virtual table with an increased size so that remote participants can see each participant at the virtual table in the virtual environment (block 750).

Various embodiments of the system 100 and method include a 3D communication processing system 105 . FIG. 8 is a flowchart showing the overall operation of the 3D communication processing system 105 shown in FIG. 1 . As shown in FIG. 8 , operation of the 3D communication processing system 105 begins by capturing images of each of the participants in an online meeting or meeting (block 800 ). At least one of the participants is a remote participant, meaning that the remote participant is not at the same physical location or endpoint as the other participants. The capture of each participant is achieved using a camera cluster.

Embodiments of the method then use data from the captured images to create geometric proxies for each participant (block 810). The number of participants is then determined (block 820). This determination can be performed out of sequence such that the number of participants is predetermined or known. Embodiments of the method then generate scene geometry based on the number of participants in the online meeting (block 830). This scene geometry generation helps simulate the experience of having an in-person conversation or meeting with remote participants.

Each geometric proxy for a particular participant is then rendered to other geometric proxies of other participants within the scene geometry (block 840). This rendering is performed such that the geometry agents are arranged in a manner consistent with the presence dialog. These rendered geometry proxies and scene geometry are then communicated to the participants (block 850). A varying virtual viewpoint is displayed to each of the participants such that the virtual viewpoint is dependent on the position and orientation of the viewer's face (block 860). For additional realism, motion parallax and depth are added to enhance the participant's viewing experience (block 870). As explained in detail below, motion parallax and depth are dependent on the eye gaze of the viewer relative to the display device or monitor on which the viewer is viewing the meeting or meeting.

V. Operating Details

Operational details of various embodiments of the controlled 3D communication endpoint system 100 and method will now be discussed. This includes details of scalability of the system 100, geometry proxy creation, and creation of scene geometry. Additionally, the concept of virtual cameras, adding motion parallax and depth to geometric proxies and scene geometry, and handling more than one actor in the same environment and viewing the same display device or monitor are discussed.

V.A. Scalability

Embodiments of the controlled 3D communication endpoint system 100 and method are scalable. This means that embodiments of the system 100 and method can be easily extended to accommodate additional endpoints whenever they are added to the online meeting. FIG. 9 illustrates an exemplary embodiment of extending various embodiments of the system 100 and method to accommodate additional endpoints.

Scalability is enhanced because of a display device configuration of at least 180 degrees. For example, if a single flat screen is on a wall and there are two endpoints, each with a participant, the two participants can be placed at a round table in the virtual environment. Each participant will be able to see the other participant. If this is extended and if 10 participants at 10 endpoints try to join an online meeting, the viewer can see people from him across the table, but every other person will get lost in the crowd. However, with at least a 180-degree display configuration, as long as the on-screen participants form a circle in the virtual environment, the circle can be made as large as needed and the viewer will still be able to see each participant .

Of course, this means that the more participants are added, the larger the virtual table must be. At some point, the number of participants becomes so large that the farthest participants at the table are too small for the viewer to identify them. Also, while the virtual table doesn't need to be round, with other shapes there is occlusion and people start blocking each other.

As shown in FIG. 9, a virtual environment 900 illustrates how various embodiments of the system 100 and method arrange participant geometric agents relative to each other. On the left side of FIG. 9 , three participants 905 , 906 , 907 are arranged around a first virtual round table 910 . In this virtual environment, each of the participants 905, 906, 907 views the online meeting through a virtual window. Specifically, virtual windows 920, 925, 930 are positioned in front of each of the three participants 905, 906, 907, respectively. These virtual windows 920 , 925 , 930 give the three participants 905 , 906 , 907 virtual viewpoints around the first virtual round table 910 . This allows each participant to feel as if he is actually in the room with the other participants.

Arrow 935 indicates that additional endpoints have been added to virtual environment 900 . With the addition of additional participants, the first virtual round table 910 has been expanded into a second virtual round table 940 . Eight participants 950 , 951 , 952 , 953 , 954 , 955 , 956 , 957 are arranged around the second virtual round table 940 . Additionally, a plurality of virtual windows 960 are positioned in front of each of the eight participants 950 , 951 , 952 , 953 , 954 , 955 , 956 , 957 . Each of the plurality of virtual windows 960 gives participants 950 , 951 , 952 , 953 , 954 , 955 , 956 , 957 a virtual viewpoint around the second virtual round table 940 . This gives each participant the illusion that each of the participants is together in one large virtual room.

V.B. Geometry Proxy Creation

Another part of the capture and creation component 200 is the geometry proxy creation module 245 . Module 245 creates geometric proxies for each of the participants in the conference or meeting. Depth information is calculated from distance data captured by the camera group 300 . Once the depth information is obtained, a sparse point cloud is created from the depth points contained in the captured depth information. A dense depth point cloud is then generated using known methods and captured depth information. In some embodiments, a mesh is constructed from the dense point cloud and a geometric proxy is generated from the mesh. In an alternative embodiment, dense point clouds are textured to generate geometric proxies.

Figure 10 shows an exemplary overview of creating a geometric proxy for a single conference participant. As shown in FIG. 10 , RGB data 1000 is captured from RGB cameras of camera group 300 . In addition, depth information 1010 is calculated from depth data obtained by the camera group 300 . The RGB data 1000 and depth information 1010 are added together to create a geometric proxy 250 of an individual conference participant. This geometric proxy creation is performed for each of the participants such that each participant has a corresponding geometric proxy.

V.C. Registration of 3D volumes and alignment in 3D space

A second component of embodiments of the controlled 3D communication endpoint system 100 and method is the scene geometry component 210 . This includes both the registration of the 3D volume captured by the camera cluster 300 and the alignment of the 3D space. The general concept of the scene geometry component 210 is to create relative geometry between conference participants. The scenes need to be aligned exactly as if the participants were in the same physical location and participating in the in-person communication.

Embodiments of the system 100 and method create scene geometry that is a 3D scene anchored at endpoints (or capture environments). In order to achieve this, it is desirable to have an accurate estimate of the environment containing each of the participants. Once this is obtained, various embodiments of the system 100 and method calculate an exact registration of the display device (or monitor) and the camera. This produces an orientation in virtual space that aligns with the real world. In other words, the virtual space is aligned with the real space. This registration and alignment is achieved using known methods. In a preferred embodiment of the system 100 and method, calibration is performed at the time of manufacture. In other embodiments, calibration is performed using reference objects in the environment.

Scene geometry seeks to create the relative geometry between local and remote actors. This includes creating eye gaze and conversation geometry as if participants were in an in-person meeting. One way to get eye gaze and dialogue geometry correct is to have relative, consistent geometry between the participants. In some embodiments, this is accomplished through the use of virtual boxes. Specifically, if boxes are drawn around participants in real space while the participants are in a room together, these virtual boxes are recreated in a virtual layout to create the scene geometry. The shape of the geometry matters less than its consistency among participants.

Certain input form factors like single monitor or multiple monitors will affect the optimal layout and scalability of the solution. Scene geometry also depends on the number of participants. A conference with two participants (a local participant and a remote participant) is a different one-to-one (1:1) scene geometry than where there are three or more participants. Furthermore, as will be seen from the examples below, the scene geometry includes eye gaze between participants.

Figure 11 shows an exemplary embodiment of scene geometry between participants when there are two participants (at two different endpoints) in an online meeting. As shown in FIG. 11 , this scene geometry 1100 for a 1:1 conference includes a third participant 1110 and a fourth participant 1120 . These participants are not in the same physical location. In other words, they are at different endpoints.

In this scene geometry 1100 for a 1:1 conference, the geometry includes two boxes occupying the space in front of the participants' 1100, 1120 respective display devices or monitors (not shown). A first virtual frame 1130 is drawn around the third participant 1110 and a second virtual frame 1140 is drawn around the fourth participant 1120 . Assuming equally sized monitors and consistent settings allows embodiments of the system 100 and method to know that the scene geometry is correct without requiring any manipulation of the captured data.

In an alternative embodiment of the system 100 and method, there are multiple remote participants and the geometry is different from the scene geometry 1100 of a 1:1 conference. Figure 12 shows an exemplary embodiment of scene geometry between participants when there are three participants at three different endpoints in an online meeting. This is the scene geometry 1200 for the 3 endpoint session. As mentioned above, an endpoint is the environment containing the participants of a meeting or meeting. In a 3-endpoint conference, there are participants at three different physical locations.

In Figure 12, the scene geometry 1200 for a 3-endpoint conference includes Participant #1 1210, Participant #2 1220, and Participant #3 1230 surrounding a virtual round table 1235. Virtual frame #1 1240 is drawn around participant #1 1210, virtual frame #2 1250 is drawn around participant #2 1220, and virtual frame #3 1260 is drawn around participant #3 1230. Each of the virtual boxes 1240 , 1250 , 1260 is placed to surround the virtual round table 1235 in an equidistant manner. This creates a scene geometry of 1200 for the 3-endpoint meeting. Note that this scene geometry can be extended for additional endpoints, as discussed above with respect to scalability.

V.D. Virtual Camera

The scene geometry component 210 also includes a virtual camera. The virtual camera defines a perspective projection from which a novel view of the 3D geometric proxy will be rendered. This allows embodiments of the system 100 and method to achieve natural eye gaze and connection between people. A glitch in current video conferencing occurs when people are not looking where the camera is located, making the remote participant in the conference feel as if the other person is not looking at them. This is unnatural and doesn't usually happen in an in-person conversation.

The virtual cameras in embodiments of the system 100 and method are created using the virtual space from the scene geometry and each participant's 3D geometry proxy (with detailed texture information). This virtual camera is not bound to the location of the real camera cluster that was used to capture the image. Additionally, some embodiments of the system 100 and method use face tracking (including eye gaze tracking) to determine where various participants are located and where they are looking in their virtual space. This allows virtual cameras to be created based on where in the scene the participant is looking. This is used to accurately communicate the participant's correct gaze to other participants and provide them with the correct view. Thus, the virtual camera facilitates natural eye gaze and conversational geometry of interactions between meeting participants.

These virtual cameras are created by creating scene geometry and placing additional people or things in that geometry. The virtual camera is able to move around the scene geometry based on the multiple viewpoints obtained by the camera swarm. For example, if the head is considered to be a balloon, the front of the balloon will be captured by the cluster of cameras in front of the balloon, and one side of the balloon will be captured by the cluster of cameras on that side of the balloon. By compositing images from these two camera clusters, a virtual camera can be created anywhere between directly in front and to the side. In other words, a virtual camera view is created as a composite of images from different cameras covering a particular space.

Figure 13 illustrates an exemplary embodiment of a virtual camera based on where a participant is looking. This can also be thought of as using virtual gaze to obtain natural eye gaze. As shown in FIG. 13 , monitor 400 displays remote participant 410 to local participant 1300 . The monitor 400 includes four camera groups 300 . A virtual eye gaze box 1310 is drawn around the remote participant's 1320 eyes and the local participant's 1330 eyes. The virtual eye gaze frame 1310 is horizontal so that in the virtual space, the eyes of the remote participant 1320 and the eyes of the local participant 1330 are looking at each other.

Some embodiments of the virtual camera use face tracking to improve performance. Face tracking helps embodiments of the system 100 and method change the perspective so that the participants face each other. Face tracking helps the virtual camera stay level with the viewer's eye gaze. This mimics how the human eye works during an in-person conversation. The virtual camera interacts with face tracking to create a virtual point of view where the user is looking directly at another participant. In other words, face tracking is used to change the virtual viewpoint of the virtual camera.

V.E. Depth via Motion Parallax

A third component of the system 100 and method is the virtual viewpoint component 220 . Once the rendered geometry proxy and scene geometry is delivered to each participant, it is rendered on the participant's monitor. To increase the realism of the scene displayed on the monitor, depth is added using motion parallax to provide the nuanced changes in view that occur when the position of someone viewing something changes.

Motion parallax is added using high speed head tracking that causes the camera view to change as the viewer's head moves. This creates the illusion of depth. FIG. 14 illustrates an exemplary embodiment in which depth is provided by motion parallax based on where the viewer is facing. As shown in FIG. 14 , a monitor 400 with four camera clusters 300 displays an image of a remote participant 410 . Note that in FIG. 14 , remote participants 410 are shown as dashed character 1400 and implementing character 1410 . Dashed figure 1410 shows remote participant 410 looking to his left and thus has a first field of view 1420 that includes dashed participant 1430 . Solid line figure 1410 shows that remote participant 410 is looking to its right and thus has a second field of view 1440 that includes solid line participant 1450 .

As the viewpoint of the remote participant 410 moves from side to side, his perspective of other spaces changes. This gives the remote participant 410 a different view of the other participants and the room (or environment) in which the other participants are located. Thus, if the remote participant moves left, right, up, or down, he will see a slightly different view of the participant the remote participant 410 is interacting with, and the background behind that person also changes. This gives the scene a sense of depth, and the people in the scene the sense of volume they get when they talk to someone in person. The viewpoint of the remote participant is tracked using head tracking or low latency face tracking techniques. The sense of volume is dynamically enhanced through the depth of motion parallax, while providing complete freedom of movement, as the viewer is not locked to one camera perspective.

V.F. Multiple Participants at a Single Endpoint

Embodiments of the system 100 and method also include situations where there is more than one participant at an endpoint. The above technique for depth through motion parallax works well for a single viewer due to the ability to track that viewer and provide the appropriate view on the monitor based on their viewing angle and position. However, where there is a second person at the same endpoint and looking at the same monitor, this will not work because the monitor can only provide one scene at a time and it will lock to one person. This takes the view away from another viewer who is not being tracked.

Embodiments of the system 100 and method have several ways to address this issue. In some embodiments, monitors are used that provide different images to different viewers. In these embodiments, the face tracking technology tracks two different faces and then provides different views to different viewers. In other embodiments, motion parallax is removed and a fixed virtual camera is locked to the center of the monitor. This creates a substandard experience when more than one participant is at an endpoint. In still other embodiments, each of the plurality of participants at the endpoint wears glasses. Each pair of glasses is used to provide a different view. In still other embodiments, the glasses have shutters on them that show each wearer a different frame than the monitor. Alternate frames displayed by the monitor are tuned to each pair of glasses and provide the correct image to each viewer based on that viewer's position.

Another embodiment uses a monitor with multiple viewing angles. 15 illustrates an exemplary embodiment of a technique for handling multiple participants at a single endpoint using monitors with multiple viewing angles. This provides each viewer in front of the monitor with a different view of the remote participant 410 and the room behind the remote participant 410 .

As shown in FIG. 15 , a monitor 1500 with a lenticular display (which allows for multiple viewing angles) and with four camera clusters 300 is displaying a remote participant 410 . A first viewer 1510 is looking at the monitor 1500 from the left side of the monitor 1500 . The eyes of the first viewer 1520 are looking at the monitor 1500 from the left side and have a left field of view 1530 of the monitor 1500 . A second viewer 1540 is looking at the monitor 1500 from the right side of the monitor 1500 . The eyes of the second viewer 1550 are looking at the monitor 1500 from the right side and have a right field of view 1560 . Because of the lens display on the monitor 1500, the left field of view 1530 and the right field of view 1560 are different. In other words, the first viewer 1510 and the second viewer 1540 are provided with different views of the remote participant 410 and the room behind the remote participant 410 . Thus, even though the first viewer 1510 and the second viewer 1541 are side by side, they will see different things on the monitor 1500 based on their point of view.

Furthermore, although the subject matter has been described in language specifically describing structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims (19)

1. A method for simulating in-person communication, comprising: capturing 3D video of the local participant at the local endpoint; creating a local geometric proxy for the local participant using the captured 3D video, further comprising adding the captured RGB data and captured depth information from the 3D video to create the local geometric proxy; Generate scene geometry and dialogue geometry with correct eye gaze consistent with in-person communications; placing the native geometry proxy in the scene geometry to create a virtual environment; and communicating the local geometry proxy and the scene geometry to a remote participant at a remote endpoint, wherein the local endpoint and the remote endpoint are at different physical locations to simulate a relationship between the local participant and the remote participant in-person communication. 2. The method of claim 1, further comprising capturing the 3D video at the local endpoint in a controlled manner using a plurality of camera clusters surrounding the local participant The latter captures both RGB data and depth information about the local participant in 360 degrees. 3. The method of claim 1, further comprising: capturing three-dimensional video of the remote participant at the remote endpoint; creating a remote geometry proxy for said remote participant; placing the remote geometry proxy in the scene geometry and virtual environment; and Both the local geometry proxy and the remote geometry proxy are rendered towards each other in the scene geometry and virtual environment. 4. The method of claim 3, further comprising communicating a rendered local geometry proxy, a rendered remote geometry proxy, and the scene geometry to the local endpoint and the remote endpoint. 5. The method of claim 1 , further comprising displaying the virtual environment to the remote participant at the remote endpoint on a display device occupying a space of at least 180 degrees around the remote participant. A participant to provide the remote participant with a virtual viewpoint of the virtual environment. 6. The method of claim 5, wherein the display devices comprise a first display device disposed in front of the remote participant, a second display device to one side of the first display device, and a third display device on the other side of the first display device. 7. The method of claim 6, further comprising: positioning the second display device at right angles to the first display device; and The third display device is positioned at right angles to the first display device. 8. The method of claim 6, further comprising: positioning the second display device at a first angle less than 90 degrees to the first display device; and The third display device is positioned at a second angle less than 90 degrees to the first display device. 9. The method of claim 8, further comprising setting the first angle and the second angle equal to each other. 10. The method of claim 3, further comprising: defining virtual tables in said virtual environment; and The local geometry agent and the remote geometry agent are placed around the virtual table to simulate the presence communication in the virtual environment. 11. The method of claim 10, further comprising defining the virtual table to have a circle with a first diameter. 12. The method of claim 11, further comprising: Scale the virtual environment by increasing the number of participants from two to more than two participants; increasing the size of the virtual table from a first diameter to a second diameter, wherein the second diameter is larger than the first diameter; and A geometric proxy for each of the participants is placed at the virtual table. 13. The method of claim 5, further comprising adding depth to the virtual viewpoint using motion parallax. 14. The method of claim 13, further comprising: tracking the remote participant's head; and What is displayed to the remote participant through the virtual viewpoint is changed based on the position and orientation of the remote participant's head. 15. A controlled three-dimensional 3D endpoint system comprising: a plurality of camera clusters arranged around a first endpoint to capture 3D video of a participant at said first endpoint such that 360 degrees around said participant is captured by said plurality of camera clusters ; a geometric proxy of the participant obtained by adding captured RGB data and captured depth information from the 3D video; Scene geometry and dialog geometry with correct eye gaze for creating virtual environments consistent with in-person communications; and a display device configuration having a plurality of display devices located at a second end point such that the display devices are positioned at least 180 degrees around a viewer at the second end point to enable the viewer to The participant is viewed through a virtual viewpoint, wherein the viewer's perspective of the participant in the virtual environment varies based on the position and orientation of the viewer's head. 16. The controlled three-dimensional 3D endpoint system of claim 15, further comprising: a virtual round table located in said virtual environment; and A rendered geometric proxy of the participant placed around the virtual round table, along with other participants at other endpoints of the online meeting. 17. A method for scaling the number of participants in an online meeting comprising: organizing a controlled capture environment at endpoints, the endpoints having a plurality of camera clusters disposed around each of the endpoints; capturing three-dimensional video of each participant at each endpoint using the plurality of camera clusters; creating a geometric proxy for each of the participants, further comprising adding captured RGB data and captured depth information from the 3D video to create the geometric proxy; generating scene geometry including virtual tables based on the number of participants; rendering each of the geometric agents toward each other in the scene geometry consistent with presence communication; placing a rendered geometry proxy in the scene geometry around the virtual table to create a virtual environment; organizing a controlled viewing environment at the endpoint having a display device that wraps at least 180 degrees around a participant at the endpoint; displaying the virtual environment to the participants in the controlled viewing environment using the display device; changing the virtual point of view of participants viewing the display device based on the position and orientation of each participant's head; increase the number of participants such that additional participants are added; and The size of the virtual table is increased to accommodate the additional participants. 18. The method of claim 17, further comprising defining the virtual table as a virtual round table having a diameter. 19. The method of claim 18, wherein increasing the size of the virtual table further comprises increasing a diameter of the virtual round table to accommodate the additional participants.