CN117321987A - Immersive viewing experience - Google Patents

Abstract

The patent discloses a method for recording images in a manner larger than the visual range of a user. Thus, allowing the user to view naturally through head tracking and eye tracking enables the person to see and view the scene as if it were naturally in real-time on-site. Also taught herein is an intelligent system that analyzes viewing parameters of a user and streams custom images to be displayed.

Description

Immersive viewing experience

Technical field

Aspects of this disclosure generally relate to uses for distribution of works.

Related application cross-references

This application is a PCT of US17/237,152 filed on April 22, 2021. US17/237,152 is a continuation-in-part of US patent application 17/225,610 filed on April 7, 2021. US patent application 17/225,610 was filed on April 7, 2021. A continuation-in-part of U.S. patent application 17/187,828 filed on February 28, 2021.

introduction

Movies are a form of entertainment.

Contents of the invention

All examples, aspects and features mentioned in this article may be combined in any technically conceivable way. This patent teaches a method, software, and apparatus for an immersive viewing experience.

In summary, this patent improves upon the technology taught in U.S. Patent Application 17/225,610, filed on April 7, 2021, the entire contents of which are incorporated herein by reference. Some devices described in US Patent Application No. 17/225,610 have the ability to generate very large data sets. This patent improves the display of such very large data sets.

This patent discloses a system, method, device and software to achieve an improved immersive viewing experience. First, the user's viewing parameters are uploaded to the cloud, where the cloud stores the imagery (which in the preferred embodiment is a very large data set). Viewing parameters may include any movement, posture, body position, eye gaze angle, eye convergence/vergence, or input (eg, via a graphical user interface). Thus, the user's viewing parameters are characterized in near real-time (e.g., via various devices such as eye-facing cameras, gesture-recording cameras) and sent to the cloud. Second, a set of user-specific images is optimized from the images, wherein the user-specific images are based at least on the above viewing parameters. In a preferred embodiment, the field of view of a user-specific image is smaller than that of the image. In a preferred embodiment, locations where the user is looking have high resolution and locations where the user is not looking have low resolution. For example, if the user is looking at an object on the left, then the user-specific image on the left will be high resolution. In some embodiments, user-specific imagery will be streamed in near real-time.

In some embodiments, the user-specific image includes a first portion having a first spatial resolution and a second portion having a second spatial resolution, wherein the first spatial resolution is higher than the second spatial resolution. Some embodiments include, wherein the viewing parameters include a viewing position, wherein the viewing position corresponds to the first portion.

Some embodiments include wherein the user-specific image includes a first portion having a first zoom setting and a second portion having a second zoom setting, wherein the first zoom setting is higher than the second zoom setting. Some embodiments include, wherein the first portion is determined by the above viewing parameters, wherein the above viewing parameters include at least one of the group consisting of: the position of the user's body; the orientation of the user's body; the user's hands the posture of the user; the facial expression of the user; the position of the user's head; and the direction of the user's head. Some embodiments include wherein the first portion is determined by a graphical user interface (such as a mouse or controller).

Some embodiments include where the image includes a first field of view (FOV) and the user-specific image includes a second field of view, where the first FOV is larger than the second FOV.

Some embodiments include, wherein the image includes a stereoscopic image, wherein the stereoscopic image is obtained by a stereoscopic camera or a stereoscopic camera cluster.

Some embodiments include, wherein the image includes a stitched image, wherein the stitched image is generated by at least two cameras.

Some embodiments include, wherein the image includes a composite image, wherein the composite image is generated by capturing a first image of the scene with a first set of camera settings, wherein the first set of camera settings causes the first object to be in focus and the second object to be in focus. out of focus; and capturing a second image of the scene with a second set of camera settings, wherein the second set of camera settings causes the second object to be in focus and the first object to be out of focus. Some embodiments include wherein the first image is presented to the user when the user looks at the first object, and the second image is presented to the user when the user looks at the second object. Some embodiments include combining at least the first object from the first image and the second object from the second image into the composite image.

Some embodiments include performing image stabilization. Some embodiments include, wherein the viewing parameter includes convergence. Some embodiments include where the user-specific image is a 3D image (three-dimensional image), where the 3D image is presented on an HDU, a pair of stereoscopic glasses, or a pair of polarized glasses.

Some embodiments include wherein the user-specific image is presented to the user on a display, wherein the user has a field of view of at least 0.5π steradians.

Some embodiments include wherein user-specific images are presented on the display. In some embodiments, the display is a screen (eg, a TV, a reflective screen coupled to a projector system, an extended reality head display unit including an augmented reality display, a virtual reality display, or a mixed reality display).

Description of drawings

Figure 1 shows

Figure 1 shows a retrospective display of stereoscopic images.

Figure 2 illustrates a method of determining which stereo pair to display to the user at a given point in time.

Figure 3 shows the display of video recordings on the HDU.

Figure 4 shows a pre-recorded stereoscopic viewing performed by user 1.

Figure 5 shows long-distance stereo imaging of distant objects using a stereo camera cluster.

Figure 6 illustrates the post-acquisition ability to adjust the image based on user eye tracking to get the best possible picture by generating a stereo composite image.

Figure 7A shows a moving image and the application of image stabilization processing.

FIG. 7B shows a moving image displayed in the HDU.

Figure 7C shows applying image stabilization to an image using stereoscopic imaging.

Figure 8A shows left and right images with a first camera setup.

Figure 8B shows left and right images with a second camera setup.

Figure 9A shows a top view of all data for a scene collected at one point in time.

Figure 9B shows a video-recorded wide-angle display 2D image frame.

Figure 9C shows a top view of user A with a viewing angle of -70° and 55° FOV.

Figure 9D shows what User A would see given his viewing angles of -70° and 55° FOV.

Figure 9E shows a top view of user B's viewing angles of +50° and 85° FOV.

Figure 9F shows what User B would see given User B's viewing angles of +50° and 85° FOV.

Figure 10A shows the field of view captured by the left camera at the first point in time.

Figure 10B shows the field of view captured by the right camera at the first point in time.

Figure 1OC shows the first user's personalized field of view (FOV) at a given point in time.

Figure 10D shows the second user's personalized field of view (FOV) at a given point in time.

Figure 10E shows a third user's personalized field of view (FOV) at a given point in time.

Figure 10F shows the fourth user's personalized field of view (FOV) at a given point in time.

Figure 11A shows a top view of the first user's left eye view.

Figure 11B shows a top view of the first user's left eye view with the convergence point proximal to the left and right eyes.

Figure 11C shows the view of the left eye at time point 1 without convergence.

Figure 1 ID shows the left eye view with convergence at time point 2.

Figure 12 shows the reconstruction of various stereoscopic images from previously acquired wide-angle stereoscopic images.

Figure 13A shows a top view of a home theater.

Figure 13B shows a side view of the home theater shown in Figure 13A.

Figure 14A shows a top view of a home theater.

Figure 14B shows a side view of the home theater shown in Figure 14A.

FIG. 15A shows an approximately spherical TV in which the user looks straight ahead at time point #1.

Figure 15B shows the portion of the television and field of view observed by the user at time point #1.

Figure 15C shows an approximately spherical TV in which the user is looking straight ahead at time point #2.

Figure 15D shows the portion of the television and field of view observed by the user at time point #2.

Figure 15E shows an approximately spherical TV in which the user is looking straight ahead at time point #3.

Figure 15F shows the portion of the television and field of view observed by the user at time point #3.

Figure 16A shows an unzoomed image.

Figure 16B shows a digital enlargement of a portion of the image.

Figure 17A shows an unzoomed image.

Figure 17B shows optical magnification on a portion of the image.

Figure 18A shows a single resolution image.

Figure 18B shows a multi-resolution image.

Figure 19A shows a large field of view in which a first user is looking at a first part of the image and a second user is looking at a second part of the image.

Figure 19B shows that only the first portion of the image in Figure 19A and the second portion of the image in Figure 19A are high resolution, while the remainder of the image is lower resolution.

Figure 20A shows a low resolution image.

Figure 20B shows a high resolution image.

Figure 20C shows the composite image.

Figure 21 illustrates a method and process for performing near real-time streaming of custom images.

Figure 22A illustrates the use of resection with stereo cameras, where the first camera position is unknown.

Figure 22B illustrates the use of resection in conjunction with a stereo camera where the object position is unknown.

Figure 23A shows a top view of a person looking forward towards the center of the home theater screen.

Figure 23B shows a top view of a person looking forward toward the right side of the home theater screen.

Figure 24 illustrates methods, systems and devices for optimizing stereo camera setup during image acquisition on the move.

Detailed ways

Flowcharts do not describe the syntax of any particular programming language. Rather, the flowchart illustrations illustrate the functional information that would be required by one of ordinary skill in the art to fabricate circuits or generate computer software to perform the processes required in accordance with the present invention. Note that many routine elements are not shown, such as loops and initialization of variables and the use of temporary variables. It will be understood by those of ordinary skill in the art that, unless otherwise stated herein, the specific sequence of steps described is only illustrative and may be changed without departing from the spirit of the invention. Therefore, unless otherwise stated, the steps described below are unordered, which means that where possible, the steps can be performed in any convenient or desired order.

Figure 1 shows a retrospective display of stereoscopic images. 100 shows step A, determining the location where the viewer is looking at time point n (eg, coordinates (α _n , β _n , _rn )). Note #1: This location can be a near convergence point, an intermediate convergence point, or a distant convergence point. Note #2: A series of stereoscopic images were collected and recorded. Step A follows the acquisition process and is performed at some subsequent time period while the user is watching. 101 shows step B, determining the FOV n corresponding to the position (eg, (α _n , β _n , _rn ) coordinates at time point n, Note: the user can select the FOV ₎ . 102 shows step C, selecting the camera(s) corresponding to the left eye FOV, with the option to perform additional image processing (e.g., using composite images, using convergence zones) to generate a personality at time point n ize the left eye image (PLEI _n ). 103 shows step D, selecting the camera(s) corresponding to the right eye FOV, with the option to perform additional image processing (e.g., use composite images, use convergence zones) to generate the personality at time point n Transform the right eye image (PREI _n ). 104 shows step E, displaying _PLEIn on the left eye display of the HDU. 105 shows step F, displaying PREI _n on the right eye display of the HDU. 106 shows step G, incrementing the time step to n+1 and going to step A above.

Figure 2 illustrates a method of determining which stereo pair to display to the user at a given point in time. 200 shows a text box that analyzes user parameters to determine which stereoscopic image to display to the user. First, use the viewing direction of the user's head. For example, if the user's head is in a forward direction, the first stereo pair may be used, and if the user's head is in a left direction, the second stereo pair may be used. Second, use the perspective of the user’s gaze. For example, if the user is looking in the direction toward a distant object (eg, a distant mountain), a distant (eg, area 3) stereoscopic image pair will be selected for that point in time. Third, use the convergence of users. For example, if the viewing direction of a near object (e.g., leaves on a tree) is very similar to the viewing direction of a distant object (e.g., a distant mountain), then choose to use a combination of convergence and perspective. Fourth, use the adjustment of the user's eyes. For example, monitor the user's pupil size and use changes in size to indicate where the user is looking (near/far).

Figure 3 shows the display of video recordings on the HDU. 300 shows establishing the coordinate system. For example, use the camera mark as the origin and the direction of the camera as the axis. This is discussed in more detail in US Patent Application No. 17/225,610, the entire contents of which are incorporated herein by reference. 301 shows a wide angle recording of the execution scene. For example, recording data with a larger FOV than the FOV displayed to the user). 302 shows performing analysis of the user as discussed in Figure 2 to determine where the user is looking in the scene. 303 shows optimizing the display based on the analysis in 302. In some embodiments, characteristics of the physical object (e.g., location, size, shape, orientation, color, brightness, texture, classification by AI algorithms) determine characteristics of the virtual object (e.g., location, size, shape, orientation, color , brightness, texture). For example, a user is using a mixed reality display in a room of a house where some areas of the room are bright (e.g., windows during the day) and some areas of the room are dark (e.g., dark blue walls). In some embodiments, virtual objects are placed based on the location of objects within the room. For example, if the background is a dark blue wall, virtual objects can be tinted white to stand out. For example, if the background is a white wall, virtual objects can be tinted blue to stand out. For example, the virtual object can be placed (or repositioned) so that its background enables the virtual object to be displayed, thereby optimizing the user's viewing experience.

Figure 4 shows a pre-recorded stereoscopic viewing performed by user 1. 400 shows user 1 performing stereoscopic recording using a stereoscopic camera system (eg, smartphone, etc.). This is discussed in more detail in US Patent Application No. 17/225,610, the entire contents of which are incorporated herein by reference. 401 shows storing a stereoscopic recording on a storage device. 402 shows a user (eg, User 1 or other user(s)) retrieving a stored stereoscopic recording. Note that the stereoscopic recording can be transmitted to the above-mentioned other user(s), and the above-mentioned other user(s) will receive the above-mentioned stored stereoscopic recording. 403 shows a user (eg, user 1 or other user(s)) viewing the above-mentioned stored stereoscopic recording on a stereoscopic display unit (eg, augmented reality, mixed reality, virtual reality display).

Figure 5 shows long-distance stereo imaging of distant objects using a stereo camera cluster. 500 shows two camera clusters placed at least 50 feet apart. 501 shows selecting a target at least 1 mile away. 502 shows the precise alignment of each camera cluster so that the centerline of focus intersects the target. 503 shows acquiring a stereoscopic image of the target. 504 illustrates viewing and/or analyzing the acquired stereoscopic image. Some embodiments use a camera with a telephoto lens instead of a camera cluster. Additionally, some embodiments have stereo separation less than or equal to 50 feet to optimize viewing less than 1 mile away.

Figure 6 illustrates the post-acquisition ability to adjust the image based on user eye tracking to get the best possible picture by generating a stereo composite image. There are several objects in the stereoscopic image displayed at this point in time that may be of interest to the person viewing the scene. Therefore, at each time point, a stereoscopic composite image is generated to match at least one user's input. For example, if the user is viewing (eye tracking determines the viewing position) mountain 600 or cloud 601 at the first point in time, a stereoscopic composite image pair transmitted to the HDU will be generated such that the distant object mountain 600 or cloud 601 is in focus, while including The close objects of Deer 603 and Flower 602 are out of focus. If the user is viewing (eye tracking determined viewing position) deer 603, the stereo composite image presented in this frame will be optimized for medium distances. Finally, if the user is viewing (eye tracking determines the viewing position) a flower 603 nearby, the stereo composite image will be optimized for close range (e.g., to achieve convergence, and blur distant items such as deer 603, mountains 600 and cloud 601). Various user inputs can be used to instruct the software suite how to optimize the stereo composite image. Gestures such as squinting can be used to optimize stereo composite images of more distant objects. Gestures such as leaning forward can be used to push distant objects closer. A GUI can also be used to improve the immersive viewing experience.

Figure 7A shows a moving image and the application of image stabilization processing. 700A shows a left eye image of an object with motion blur around the edges of the object. 701A shows a left eye image of an object to which image stabilization processing is applied.

FIG. 7B shows a moving image displayed in the HDU. 702 shows the HDU. 700A shows a left eye image of an object with motion blur around the edges of the object. 700B shows a right eye image of an object with motion blur around the edges of the object. 701A shows a left eye display aligned with the user's left eye. 701B shows a right eye display aligned with the user's right eye.

Figure 7C shows applying image stabilization to an image using stereoscopic imaging. A key task in image processing is image stabilization using stereoscopic imagery. 700A shows a left eye image of an object to which image stabilization processing has been applied. 700B shows a left eye image of an object to which image stabilization processing is applied. 701A shows a left eye display aligned with the user's left eye. 701B shows a right eye display aligned with the user's right eye. 702 shows the HDU.

Figure 8A shows left and right images with a first camera setup. Note that the text on the monitor is in focus, while the distant object of the knob on the cabinet is out of focus.

Figure 8B shows left and right images with a second camera setup. Note that the text on the monitor is out of focus, while the distant object of the knob on the cabinet is in focus. An innovative point is the use of at least two cameras. The first image is obtained from the first camera. The second image is obtained from the second camera. The first camera and the second camera are at the same viewing angle. Furthermore, they are images of a scene (e.g., a still scene, or a scene with movement/change at the same point in time). A composite image is generated, wherein a first part of the composite image is obtained from the first image and a second part of the composite image is obtained from the second image. Note that in some embodiments, objects within the first image may be segmented, and the same objects within the second image may also be segmented. The first image of the object and the second image of the object can be compared to see which one has better quality. Images with better image quality can be added to the composite image. However, in some embodiments, intentional selection of some unclear parts may be performed.

Figure 9A shows a top view of all data for a scene collected at one point in time.

Figure 9B shows a video-recorded wide-angle display 2D image frame. Note that assuming the user's internal FOV (human eye FOV) does not match the camera system's FOV, the entire field of view displayed to the user will be distorted.

Figure 9C shows a top view of user A with a viewing angle of -70° and 55° FOV. A key innovation is that users can select parts of the stereoscopic image based on their viewing angle. Note that the selected section may actually be up to ~180°, but no larger.

Figure 9D shows what User A would see given his viewing angles of -70° and 55° FOV. This is an improvement over existing techniques as it allows different viewers to see different parts of the field of view. Although humans have a horizontal field of view slightly larger than 180 degrees, humans can only read text over a field of view of about 10 degrees, evaluate shapes only over a field of view of about 30 degrees, and only over a field of view of about 60 degrees Evaluate color. In some embodiments, filtering (subtraction) is performed. A person has a vertical field of view of approximately 120 degrees, with an upward (above the horizontal) field of view of 50 degrees and a downward (below the horizontal) field of view of approximately 70 degrees. However, the maximum movement of the eyeball is limited to about 25 degrees above the horizontal and about 30 degrees below the horizontal. Typically, normal line of sight from a seated position is approximately 15 degrees below horizontal.

Figure 9E shows a top view of user B's viewing angles of +50° and 85° FOV. A key innovation is that users can select parts of the stereoscopic image based on their viewing angle. Also, note that User B's FOV is larger than User A's FOV. Note that the selected part may actually reach ~180°, but will not be larger due to limitations of the human eye.

Figure 9F shows what User B would see given User B's viewing angles of +50° and 85° FOV. This is an improvement over existing technology as it allows different viewers to see different parts of the field of view. In some embodiments, multiple cameras record 240° movies. In one embodiment, 4 cameras (each camera captures a 60° sector) are used for simultaneous recording. In another embodiment, sectors are captured sequentially - recorded one at a time. Some scenes in the film could be shot sequentially, while other scenes could be shot simultaneously. In some embodiments, overlapping camera setups may be used for image stitching. Some embodiments include use of the camera ball system described in U.S. Patent Application No. 17/225,610, the entire contents of which are incorporated herein by reference. After the images are recorded, the images from the camera are edited so that the scenes are synchronized and stitched together. LiDAR (LIDAR) devices can be integrated into camera systems for precise camera direction pointing.

Figure 10A shows the field of view captured by the left camera at the first point in time. Left camera 1000 and right camera 1001 are shown. The left FOV 1002 is shown by the white area, is approximately 215°, and would have an alpha range from +90° to -135° (sweeping counterclockwise from +90° to -135°). The area not imaged within left FOV 1003 is approximately 135° and would have an alpha range from +90° to -135° (sweeping clockwise from +90° to -135°).

Figure 10B shows the field of view captured by the right camera at the first point in time. Left camera 1000 and right camera 1001 are shown. The right FOV 1004 is shown by the white area, is approximately 215°, and would have an alpha range of +135° to -90° (sweeping counterclockwise from +135° to -90°). The area not imaged within the right FOV 1005 is approximately 135° and would have an alpha range from +135° to -90° (sweeping counterclockwise from +135° to -90°).

Figure 1OC shows the first user's personalized field of view (FOV) at a given point in time. 1000 shows the left camera. 1001 shows the right camera. 1006a shows the left boundary of the first user's left eye FOV, shown in light gray. 1007a shows the right boundary of the first user's left eye FOV, shown in light gray. 1008a shows the left boundary of the first user's right eye FOV, shown in light gray. 1009a shows the right boundary of the first user's right eye FOV, shown in light gray. 1010a shows the centerline of the first user's left eye FOV. 1011a shows the centerline of the first user's right eye FOV. Note that the center line 1010a of the first user's left eye FOV and the center line 1011a of the first user's right eye FOV are parallel, which is equivalent to a convergence point at infinity. Notice that the first user is looking forward. It is recommended that when photographing movement, most of the action in the scene occurs in this front-view direction.

Figure 10D shows the second user's personalized field of view (FOV) at a given point in time. 1000 shows the left camera. 1001 shows the right camera. 1006b shows the left boundary of the second user's left eye FOV, shown in light gray. 1007b shows the right boundary of the second user's left eye FOV, shown in light gray. 1008b shows the left boundary of the second user's right eye FOV, shown in light gray. 1009b shows the right boundary of the second user's right eye FOV, shown in light gray. 1010b shows the centerline of the second user's left eye FOV. 1011b shows the centerline of the second user's right eye FOV. Note that the center line 1010b of the second user's left eye FOV and the center line 1011b of the second user's right eye FOV meet at convergence point 1012. This allows the second user to view small objects in greater detail. Notice that the second user is looking forward. It is recommended that when photographing movement, most of the action in the scene occurs in this front-view direction.

Figure 10E shows a third user's personalized field of view (FOV) at a given point in time. 1000 shows the left camera. 1001 shows the right camera. 1006c shows the left boundary of the third user's left eye FOV, shown in light gray. 1007c shows the right boundary of the third user's left eye FOV, shown in light gray. 1008c shows the left boundary of the third user's right eye FOV, shown in light gray. 1009c shows the right boundary of the third user's right eye FOV, shown in light gray. 1010c shows the centerline of the third user's left eye FOV. 1011c shows the centerline of the third user's right eye FOV. Note that the center line 1010c of the third user's left eye FOV and the center line 1011c of the third user's right eye FOV are approximately parallel, which is equivalent to looking at a long distance. Note that the third user is looking moderately to the left. Note that the overlap of the left eye FOV and right eye FOV provides stereoscopic viewing for the third viewer.

Figure 10F shows the fourth user's personalized field of view (FOV) at a given point in time. 1000 shows the left camera. 1001 shows the right camera. 1006d shows the left boundary of the fourth user's left eye FOV, shown in light gray. 1107d shows the right boundary of the fourth user's left eye FOV, shown in light gray. 1008d shows the left boundary of the fourth user's right eye FOV, shown in light gray. 1009d shows the right boundary of the fourth user's right eye FOV, shown in light gray. 1010d shows the centerline of the fourth user's left eye FOV. 1011d shows the centerline of the fourth user's right eye FOV. Note that the center line 1010d of the fourth user's left eye FOV and the center line 1011d of the fourth user's right eye FOV are approximately parallel, which is equivalent to looking at a long distance. Note that the fourth user is looking to the extreme left. Note that the first user, the second user, the third user and the fourth user all watch different views of the movie at the same point in time. It should be noted that some of these designs, such as the camera cluster or spherical system described

FIG. 11A shows a top view of the first user's left eye view at time point 1 . 1100 shows the left eye viewpoint. 1101 shows the right eye viewpoint. 1102 shows the portion of the field of view (FOV) not covered by either camera. 1103 shows the portion of the FOV covered by at least one camera. 1104A shows the medialportion of the high-resolution FOV used by the user, which corresponds to α = +25°. This is discussed in more detail in US Patent Application No. 17/225,610, the entire contents of which are incorporated herein by reference.

1105A shows the lateral portion of the high-resolution FOV used by the user, which corresponds to α = -25°.

Figure 11B shows a top view of the first user's left eye view with the convergence point proximal to the left and right eyes. 1100 shows the left eye viewpoint.

1101 shows the right eye viewpoint. 1102 shows the portion of the field of view (FOV) not covered by either camera. 1103 shows the portion of the FOV covered by at least one camera. 1104B shows the inner portion of the high-resolution FOV used by the user, which corresponds to α = -5°. 1105B shows the outer portion of the high-resolution FOV used by the user, which corresponds to α = +45°.

Figure 11C shows the view of the left eye at time point 1 without convergence. Note that flower 1106 is shown in the image, which is positioned along the viewing angle α = 0°.

Figure 1 ID shows the left eye view with convergence at time point 2. Note that the image shows flower 1106, which is still positioned along the viewing angle α = 0°. However, users have converged during this point in time. This convergence behavior changes the left eye's field of view from a horizontal field of view with α between -25° and 25° (as shown in Figures 11A and 11C) to an α between -5° and +45° (as shown in Figures 11A and 11C) shown in Figures 11B and 11D). This system improves upon existing technology as it provides stereo convergence on a stereo camera by moving the image according to the left (right) field of view. In some embodiments, a portion of the display is non-optimized, as described in U.S. Patent 10,712,837, which is incorporated herein by reference in its entirety.

Figure 12 shows the reconstruction of various stereoscopic images from previously acquired wide-angle stereoscopic images. 1200 shows acquiring images from a stereo camera system. This camera system is discussed in more detail in U.S. Patent Application No. 17/225,610, the entire contents of which are incorporated herein by reference. 1201 shows the use of a first camera for a left eye viewing angle and a second camera for a right eye viewing angle. 1202 illustrates selecting the field of view of the first camera based on the left eye perspective and selecting the field of view of the second camera based on the right eye perspective. In a preferred embodiment, the selection will be performed by a computer (eg integrated into the head display unit) based on an eye tracking system that tracks the user's eye movements. It should also be noted that in the preferred embodiment, there is also an inward image shift on the display closer to the nose during the convergence process, as taught in U.S. Patent 10,712,837, particularly Figures 15A, 15B, 16A, and 16B, the entire contents of which are incorporated herein by reference. 1203 shows the left eye field of view being presented to the user's left eye and the right eye field of view being presented to the user's right eye. There are several options in this situation. First, a synthetic stereoscopic image pair is used, in which the left-eye image is generated by at least two lenses (e.g., the first is optimized for close-up images and the second is optimized for distant imaging), and in which the right-eye image is generated by at least Two lenses are generated (e.g. the first is optimized for close-up images and the second is optimized for distant imaging). When the user is looking at a near object, a stereoscopic image pair is presented in which the near object is in focus and the far object is out of focus. When the user is looking at a distant object, a stereoscopic image pair is presented in which the near object is out of focus and the far object is in focus. Second, use various display devices (e.g., augmented reality, virtual reality, mixed reality displays).

Figure 13A shows a top view of a home theater. 1300 shows the user. 1301 shows a projector. 1302 shows the screen. Note that the field of view of this immersive home theater display is larger than that of the user 1300 . For example, if the user 1300 is looking straight ahead, the home theater will display a horizontal FOV greater than 180 degrees. Therefore, the home theater's FOV will completely cover the user's horizontal FOV. Similarly, if the user is looking straight ahead, the home theater will display a vertical FOV greater than 120 degrees. Therefore, the home theater's FOV will completely cover the user's vertical FOV. AR/VR/MR headsets can be used with this system, but are not required. Inexpensive anaglyph or disposable colored glasses can also be used. Traditional IMAX polarized projectors can be used with IMAX-style disposable polarized glasses. Home theaters can vary in size. Home theater walls can be constructed using white reflective panels and frames. Projectors will have multiple heads to cover a larger field of view.

Figure 13B shows a side view of the home theater shown in Figure 13A. 1300 shows the user. 1301 shows a projector. 1302 shows the screen. Note that the field of view of this immersive home theater display is larger than that of the user 1300 . For example, if the user 100 is looking forward in a recliner, the home theater will display a vertical FOV greater than 120 degrees. Therefore, the home theater's FOV will completely cover the user's FOV. Similarly, if the user is looking straight ahead, the home theater will display a horizontal FOV greater than 120 degrees. Therefore, the home theater's FOV will completely cover the user's FOV.

Figure 14A shows a top view of a home theater. 1400A shows a first user. 1400B shows the first user. 1401 shows a projector. 1402 shows the screen. Note that the field of view displayed by the immersive home theater is larger than the FOV of the first user 1400A or the second user 1400B. For example, if first user 1400A is looking straight ahead, first user 1400A will see a horizontal FOV greater than 180 degrees. Therefore, the home theater's FOV will completely cover the user's horizontal FOV. Similarly, if first user 1400A is looking straight ahead, the home theater will display a vertical FOV greater than 120 degrees, as shown in Figure 14B. Therefore, the home theater's FOV will completely cover the user's vertical FOV. AR/VR/MR headsets can be used with this system, but are not required. Inexpensive stereoscopic films or polarized glasses may also be used. Traditional IMAX polarized projectors can be used with IMAX-style disposable polarized glasses. Home theaters can vary in size. Home theater walls can be constructed using white reflective panels and frames. Projectors will have multiple heads to cover a larger field of view.

Figure 14B shows a side view of the home theater shown in Figure 14A. 1400A shows a first user. 1401 shows a projector. 1402 shows the screen. Note that the field of view of this immersive home theater display is larger than the field of view of the first user 1400A. For example, if user 1400A is looking forward in a recliner, the user will see a vertical FOV greater than 120 degrees. Therefore, the FOV of the home theater will completely cover the FOV of the first user 1400A. Similarly, if first user 1400A is looking straight ahead, the home theater will display a horizontal FOV greater than 120 degrees. Therefore, the FOV of the home theater will completely cover the FOV of the first user 1400A.

A typical high-resolution display has 4000 pixels at a distance of 1.37 meters. This is equivalent to 10×10 ⁶ pixels per 1.87 square meters. Consider data for a hemispherical theater. Assume that the radius of the hemispherical theater is 2 meters. The surface area of the hemisphere is 2×π×r ² , which is equal to (4)(3.14)(2 ² ) or 50.24m ² . Assuming that the spatial resolution is desired to be equal to that of a typical high-resolution display, this would equal (50.24m ² ) (10×10 ⁶ pixels per 1.87m ² ) or 429 million pixels. Assume the frame rate is 60 frames per second. This is equivalent to 26 times the data volume of a standard 4K monitor.

Some embodiments include building a home theater that matches the geometry of the projector. Preferred embodiments are sub-spherical (eg hemispherical). One low-cost construction is to use reflective surfaces spliced together with multiple projectors. In some embodiments, the field of view includes a spherical coverage of 4π steradians. This can be achieved through HDU. In some embodiments, the field of view includes subspherical coverage of at least 3π steradians. In some embodiments, the field of view includes subspherical coverage of at least 2π steradians. In some embodiments, the field of view includes subspherical coverage of at least 1π steradian. In some embodiments, the field of view includes subspherical coverage of at least 0.5π steradians. In some embodiments, the field of view includes subspherical coverage of at least 0.25π steradians. In some embodiments, the field of view includes subspherical coverage of at least 0.05π steradians. In some embodiments, a sub-spherical IMAX system is created to improve the movie theater experience for many viewers. Chairs will be placed in similar positions to a standard movie theater, but the screen will be sub-spherical. In some embodiments, non-spherical shapes may also be used.

Figure 15A shows time point #1 where the user is looking straight ahead, seeing a horizontal field of view of approximately 60 degrees horizontally and 40 degrees vertically with a fairly precise field of view (e.g., the user can see the shape and color of the peripheral FOV) .

Figure 15B shows the center portion of the TV and the field of view observed by the user at time point #1. Note that in some embodiments, the data will be streamed (eg, over the Internet). Note that an innovative feature of this patent is called "viewing parameter-directed streaming." In this embodiment, viewing parameters are used to guide the streamed data. For example, if user 1500 is looking straight ahead, the first set of data will be streamed to correspond to user 1500's straight-looking perspective. However, if the user is looking to the side of the screen, a second set of data will be streamed to correspond to the user's 1500 side-viewing perspective. Other viewing parameters that can control the viewing angle include, but are not limited to, the following parameters: user's vergence; user's head position; user's head direction. Broadly speaking, any characteristics (age, gender, preferences) or actions (perspective, location, etc.) of the user can be used to guide streaming. Note that another innovative feature is the streaming of at least two image qualities. For example, a first image quality (eg, high quality) will be streamed according to a first parameter (eg, within the user's 30° horizontal FOV and 30° vertical FOV). And, a second image quality (eg, lower quality) that does not meet this criterion (eg, is not within the user's 30° horizontal FOV and 30° vertical FOV) will also be streamed. Surround sound will be implemented in this system.

Figure 15C shows time point #2 where the user is looking to the left of the user's screen, seeing approximately 60 degrees horizontally and 40 degrees vertically with a fairly precise field of view (e.g., the user can see the shape and color of the peripheral FOV) degree of horizontal field of view.

Figure 15D shows time point #2, where the field of view observed by the user is different from Figure 15B. The area of interest is half of time point #1. In some embodiments, the user is provided with more detail and higher resolution of objects within a small FOV within the scene. Outside this high-resolution field of view area, lower resolution image quality can be rendered on the screen.

Figure 15E shows time point #3 where the user is looking to the right side of the user's screen.

Figure 15F shows time point #3, where a circular high-resolution FOV is seen.

Figure 16A shows an unzoomed image. 1600 shows the image. 1601A shows a box representing an area within image 1600 that is set to magnification.

Figure 16B shows a digital enlargement of a portion of the image. This can be achieved by the method described in US Patent 8,384,771 (eg, 1 pixel becomes 4 pixels), which is incorporated herein by reference in its entirety. Note that the area to be enlarged can be achieved through various user inputs, including: gesture tracking systems; eye tracking systems; and graphical user interfaces (GUI). Note that the area within image 1601A represented in Figure 16A is now enlarged, as shown at 1601B. Note that the resolution of area 1601B is equal to the resolution of image 1600, just larger. Note that 1600B shows an unamplified portion of 1600A. Note that 1601A is now magnified, and note that portions of 1600A are not visualized.

Figure 17A shows an unzoomed image. 1700 shows the image. 1701A shows a box representing the area within image 1700 that is set to be magnified.

Figure 17B shows an optical magnification of a portion of the image. Note that the area to be enlarged can be achieved through various user inputs, including: gesture tracking systems; eye tracking systems; and graphical user interfaces (GUIs). Note that the area within image 1701A represented in Figure 17A is now enlarged as shown at 1701B, and also note that the image within 1701B exhibits higher image quality. This can be accomplished by selectively displaying the highest quality image in area 1701B and zooming in on area 1701B. Not only are the clouds bigger, the resolution of the clouds is also better. Note that 1700B shows an unamplified portion of 1700A (note that some of the unamplified portions of 1700A are now covered by enlarged areas).

Figure 18A shows a single resolution image. 1800A shows the image. 1801A shows a box representing an area within image 1800A that is set to have increased resolution.

Figure 18B shows a multi-resolution image. Note that areas of increased resolution can be achieved through a variety of user inputs, including: gesture tracking systems; eye tracking systems; and graphical user interfaces (GUIs) including joysticks or controllers. Note that the area within image 1801A represented in Figure 18A is now displayed at a higher resolution, as shown at 1801B. In some embodiments, the image within 1801B may also be changed in other options (e.g., different color schemes, different brightness settings, etc.). This may be accomplished by selectively displaying higher (eg, highest) quality images in area 1801B without enlarging area 1701B.

Figure 19A shows a large field of view in which a first user is looking at a first part of the image and a second user is looking at a second part of the image. The 1900A is the first resolution with a large field of view. 1900B is the position where the first user is looking, as shown in Figure 19B, which is set to become high resolution. 1900C is where the second user is looking, as shown in Figure 19B, which is set to become high resolution.

Figure 19B shows that only the first part of the image in Figure 19A and the second part of the image in Figure 19A are high resolution, while the remainder of the image is low resolution. 1900A is a large field of view with first resolution (low resolution). 1900B is the location of the first user's high resolution area, which has a second resolution (high resolution in this example). 1900C is the location of the second user's high resolution area, which has a second resolution (high resolution in this case). Therefore, the first high resolution area is used for the first user. Also, a second high resolution area may be used for a second user. As shown in Figures 14A and 14B, this system may be useful for home theater displays.

Figure 20A shows a low resolution image.

Figure 20B shows a high resolution image.

Figure 20C shows the composite image. Note that this composite image has a low resolution first part 2000 and a high resolution second part 2001. This is described in US Patent 16/893,291, which is incorporated herein by reference in its entirety. The first part is determined by the user's viewing parameters (eg, viewing angle). One novelty lies in the near real-time streaming of a first part 2000 with a first image quality and a second part with a second image quality. Note that the first part can be displayed differently than the second part. For example, the first part and the second part may differ in visual presentation parameters, including: brightness; color scheme; or others. Thus, in some embodiments, a first portion of the image may be compressed without compressing a second portion of the image. In other embodiments, a composite image is generated for display to the user by an arrangement that stitches together some high-resolution images and some low-resolution images. In some embodiments, some portions of a large image (eg, 429 million pixels) are high resolution and some portions of a large image are low resolution. A high-resolution portion of a large image will be streamed based on the user's viewing parameters (e.g., convergence point, viewing angle, head angle, etc.).

Figure 21 illustrates methods and processes for performing near real-time streaming of custom images.

Regarding the display 2100, the display includes, but is not limited to, the following: a large TV; extended reality (eg, augmented reality, virtual reality, or mixed reality display); on-screen projector system; computer monitor, etc. A key component of the display is the ability to track where the user is looking in the image and what the viewing parameters are.

Regarding viewing parameters 2101, viewing parameters include, but are not limited to, the following parameters: viewing angle; vergence/convergence; user preferences (e.g., objects of special interest, filtering - certain objects rated "R" may be filtered for a specific user, etc. wait).

Regarding cloud 1202, each frame in a movie or video will be extremely large data (especially if the home theater shown in Figures 14A and 14B is used in conjunction with the camera cluster described in U.S. Patent Application No. 17/225,610, of which The entire contents are incorporated herein by reference). Note that cloud refers to storage, databases, etc. Note that the cloud is capable of cloud computing. An innovative aspect of this patent is that the viewing parameters of the user(s) are sent to the cloud, where the viewing parameters are processed (e.g., selecting a field of view or synthesizing a stereoscopic image pair, as discussed in Figure 12), and determine which portions of very large data to stream to optimize the experience for individual users. For example, multiple users can watch their videos simultaneously. Each user has personalized optimized data for that specific point in time streamed from the cloud to their mobile device. Each user then views their own optimization data on their own device. This will result in an improved immersive viewing experience. For example, suppose there is a dinner scene at a single point in time, with a chandelier, a dog, an old man, a bookcase, a long table, a rug, and wall accessories. A user named Dave could be looking at a dog, and Dave's image would be optimized (e.g., an image of the dog with maximum resolution and optimized colors would be streamed to Dave's mobile device and displayed on Dave's HDU). A user named Kathy could be looking at a chandelier, and Kathy's image would be optimized (e.g., an image of the chandelier with maximum resolution and optimized colors would be streamed to Kathy's mobile device and displayed on Kathy's HDU). Finally, a user named Bob could be looking at an old man, and Bob's image would be optimized (e.g., an image of the old man with maximum resolution and optimized colors would be streamed to Bob's mobile device and displayed on Bob's HDU) . It is important to note that the cloud will store huge data sets at each point in time, but only parts of the data sets will be streamed, and these parts are determined by the user's viewing parameters and/or preferences. Therefore, the bookcase, long table, rug, and wall accessories may all be within the field of view of Dave, Kathy, and Bob, but these objects will not be optimized for display (e.g., the highest possible resolution of these images stored in the cloud not being streamed).

Finally, the concept of preemption is introduced. This additional image frame may be pre-emptively streamed if it is predicted that an upcoming scene may cause a particular user's viewing parameters to change (eg, the user turns their head). For example, if the movie's time is at 1:43:05 and the dinosaur makes a sound and jumps out from the left side of the screen at 1:43:30, you can download the entire scene in a low-resolution format and use it as needed ( For example, download additional data sets for selective portions of the FOV based on the user's viewing parameters, based on predictions of upcoming dinosaur scenes that the user wants to see). Therefore, the dinosaur that pops out will always be at its maximum resolution. This technology creates a more immersive and improved viewing experience.

Figure 22A illustrates the use of resection in conjunction with a stereo camera. Camera #1 has a known location (e.g. latitude and longitude from GPS). The distance (2 miles) and direction (330° north-northwest) from camera #1 to object 2200 are known. The position of object 2200 can be calculated. Camera #2 has an unknown location, but a known distance (1 mile) and direction (30° north-northeast) to object 2200. Since the position of object 2200 can be calculated, the geometry can be solved and the position of camera #2 determined.

Figure 22A illustrates the use of resection in conjunction with a stereo camera. Camera #1 has a known location (e.g. latitude and longitude from GPS). Camera #1 and Camera #2 have known locations. The direction from camera #1 to object 2200B is known (330° north-northwest). The direction from camera #2 to object 2200B is known (30° north-northeast). The position of object 2200B can be calculated.

Figure 23A shows a top view of a person looking forward towards the center of the home theater screen. Person 2300 is looking forward toward center portion 2302B of screen 2301 of the home theater. During this point in time, the stream is customized to have an optimized center portion 2302B (eg, highest possible resolution), a non-optimized left portion 2302A (eg, low resolution or black), and a non-optimized right portion. Side portion 2302C (e.g., low resolution or black). Note that for proper streaming, input to the monitoring system (which detects the user's viewing direction and other viewing parameters such as posture or facial expressions) or the controller (which receives the user's commands) must also be in place.

Figure 23B shows a top view of a person looking forward toward the right side of the home theater screen. Person 2300 is looking forward toward right portion 2302C of screen 2301 of the home theater. During this point in time, the stream is customized to have an optimized right portion 2302C (eg, highest possible resolution), a non-optimized left portion 2302A (eg, low resolution or black), and a non-optimized Middle portion 2302B (eg, low resolution or black). Note that for proper streaming, input to the monitoring system (which detects the user's viewing direction and other viewing parameters such as posture or facial expressions) or the controller (which receives the user's commands) must also be in place.

Figure 24 illustrates methods, systems and devices for optimizing stereo camera setup during image acquisition on the move. 2400 illustrates determining the distance of an object at a point in time (eg, using a laser rangefinder). Object tracking/target tracking systems can be implemented. 2401 shows adjusting the zoom setting of the stereo camera system so that it is optimized for the distance determined in step 2400. In a preferred embodiment, this will be performed while using a zoom lens as opposed to performing digital zoom. 2402 shows adjusting the separation distance between stereo cameras (stereo distance) so that it is optimized for the distance determined in step 2400. Note that there is an option to also adjust the direction of the camera so that it is optimized for the distance determined in step 2400. 2403 shows acquiring the stereoscopic image of the target at the time point of step 2400. 2404 illustrates recording, viewing and/or analyzing the acquired stereoscopic images.

Claims (20)

1. A method including: Upload the user's viewing parameters to the cloud via the Internet; wherein said cloud storage image, wherein said cloud is capable of cloud computing, and wherein the viewing parameter of the user includes a viewing angle; In the cloud, optimize user-specific display images based on the images; wherein said user-specific display image is based at least on said viewing parameters; wherein the user-specific display image includes a first part and a second part; wherein said first portion of said user-specific display image is different from said second portion of said user-specific display image; wherein said first portion of said user-specific display image includes a first image quality; wherein said first portion of said user-specific display image corresponds to said viewing angle; wherein the second portion of the user-specific display image includes a second image quality; and wherein the second image quality is lower than the first image quality; Downloading the user-specific display image via the Internet; and The user-specific display image is displayed to the user. 2. The method of claim 1, further comprising: wherein the user-specific display image includes a first portion with a first spatial resolution and a second portion with a second spatial resolution; and Wherein the first spatial resolution is higher than the second spatial resolution. 3. The method of claim 1, further comprising wherein the images comprise video images. 4. The method of claim 1, further comprising: wherein the user-specific display image includes, wherein the first portion includes a first zoom setting and the second portion includes a second zoom setting; and wherein said first zoom setting is higher than said second zoom setting. 5. The method of claim 4, further comprising, wherein the first portion is determined by at least one of the group consisting of: The position of the user's body; the direction of the user's body; the posture of the user's hands; the user’s facial expression; the position of the user's head; and The direction of the user's head. 6. The method of claim 4, further comprising wherein the first portion is determined by a graphical user interface. 7. The method of claim 1, further comprising: wherein the image includes a first field of view FOV; wherein the user-specific display image includes a second FOV; and Wherein the first FOV is larger than the second FOV. 8. The method of claim 1, further comprising: wherein the images include stereoscopic images; and The stereoscopic image is obtained through a stereoscopic camera or a stereoscopic camera cluster. 9. The method of claim 1, further comprising, wherein the image includes a stitched image, wherein the stitched image is generated by at least two cameras. 10. The method of claim 1, further comprising: wherein said images include composite images; The composite image is generated in the following manner: Capture a first image of the scene using a first set of camera settings such that a first object is in focus and a second object is out of focus; and A second image of the scene is captured using a second set of camera settings such that the second object is in focus and the first object is out of focus. 11. The method of claim 10, further comprising: When the user looks at the first object, the first image is presented to the user; and When the user looks at the second object, the second image is presented to the user. 12. The method of claim 10, further comprising combining at least the first object from the first image and the second object from the second image into the composite image. 13. The method of claim 1, further comprising wherein the viewing angle is moveable by the user. 14. The method of claim 1, further comprising wherein the viewing parameter includes convergence. 15. The method of claim 1, further comprising wherein the user-specific image is a 3D image, wherein the 3D image is presented on a head display unit (HDU). 16. The method of claim 15, further comprising wherein the viewing angle is determined by the direction of the HDU. 17. A method comprising: determining viewing parameters of the user, wherein the viewing parameters of the user include a viewing angle; sending the viewing parameters of the user to a cloud over the Internet, wherein the cloud is capable of cloud computing; wherein the cloud computing generates user-specific display images from images stored on the cloud; wherein said user-specific display image is based at least on said viewing parameters of said user; wherein the user-specific display image includes a first part and a second part, wherein said first portion of said user-specific display image is different from said second portion of said user-specific display image; wherein said first portion of said user-specific display image includes a first image quality; wherein said first portion of said user-specific display image corresponds to said viewing angle; wherein the second portion of the user-specific display image includes a second image quality; and wherein the second image quality is lower than the first image quality; Receive the user-specific display image via the Internet; and The user-specific display image is displayed on a head display unit HDU, wherein the HDU includes a left eye display and a right eye display. 18. The method of claim 1, further comprising wherein the user-specific display image is presented to the user on a display, wherein the user has a field of view of at least 0.5π steradians. 19. The method of claim 18, further comprising, wherein the display includes at least one of the group consisting of: screens and projectors; TV; and monitor. 20. A method comprising: Receive the user's viewing parameters via the Internet at the cloud; wherein the viewing parameter of the user includes a viewing angle; wherein said cloud is capable of cloud computing; using cloud computing to generate user-specific display images from images stored on the cloud; wherein said user-specific display image is based at least on said viewing parameters of said user; wherein the user-specific display image includes a first part and a second part; wherein said first portion of said user-specific display image is different from said second portion of said user-specific display image; wherein said first portion of said user-specific display image includes a first image quality; wherein said first portion of said user-specific display image corresponds to said viewing angle; wherein the second portion of the user-specific display image includes a second image quality; and wherein the second image quality is lower than the first image quality; sending the user-specific display image to the head display unit HDU through the Internet, Wherein the HDU includes a left eye display and a right eye display; The HDU displays the user-specific display image.