CN102215217B - Create a video conference during a call - Google Patents

Abstract

The invention relates to establishing a video conference during a call. Some embodiments provide a method of initiating a video conference with a first mobile device. The method presents, on a first mobile device, a selectable User Interface (UI) item for switching from an audio call to a video conference during the audio call with a second device over a wireless communication network. The method receives a selection of the selectable UI item. The method initiates the video conference without terminating the audio call. The method terminates the audio call before allowing the first and second devices to present the audio and video data exchanged over the video conference.

Description

Create a video conference during a call

When a prototype of Apple's iPhone 4 was stolen from an Apple engineer on March 25, 2010, the inventions disclosed and claimed in this application were released to the public in advance and without Apple's authorization. Prior to this apparent theft, the US priority application on which this application is based had not been filed.

Background technique

Many current portable devices, such as smart phones, have video capture capabilities. Through the camera on the phone, the user of the portable device is able to capture still images and video. However, after video capture is complete, in order to transfer the captured video to another party, the user typically must send the video directly to the other party, or upload the video to another location (e.g., an Internet video hosting site). ). Unfortunately, this does not allow the other party to view the live video stream while the portable device is capturing the video.

Additionally, standard portable devices are equipped with only one camera, and processing information from that camera is quite difficult. An ideal device would have multiple cameras and be able to send live video which is a composite of video from at least two cameras. This is a particularly difficult problem in view of the limited resources available to portable devices in terms of devices handling multiple captured video streams, and the network connected to said devices responsible for handling the transmission of the real-time video streams.

Contents of the invention

Some embodiments of the invention provide a mobile device with two cameras capable of taking pictures and video. The mobile device of some embodiments has a display screen that displays captured photo images and video images. It also includes memory to save captured images for later transfer to another device. The device also has a network interface that allows the device to transmit captured images to one or more devices during a real-time communication session between users of the plurality of devices. The device also includes an encoder that can be used to encode captured images for local storage or transmission to another device. The mobile device also includes a decoder that allows the device to decode diagrams taken by another device during a real-time communication session, or to decode locally saved images.

An example of a real-time communication session involving the transmission of captured video images is video conferencing. In some embodiments, the mobile device is only capable of transmitting video images captured by one camera at any given time during the videoconference. However, in other embodiments, the mobile device can simultaneously transmit video images captured from its two cameras during a video conference or other real-time communication session.

During a video conference with another device, the mobile device of some embodiments communicates other types of content by, along with video captured by one or both of its cameras. An example of such other content includes low-resolution or high-resolution photo images captured by one camera of the device while another camera of the device is capturing video for a video conference. Other examples of such other content include (1) files and other content stored on the device, (2) the device's screen display (i.e., content displayed on the device's screen), (3) Content received from another device, etc. during a communication session.

The mobile device of some embodiments employs novel in-conference adjustment techniques to make adjustments during a video conference. For example, during a video conference when only video captured by one camera is being transmitted, the mobile device of some embodiments can dynamically switch to transmitting video captured by its other camera. In this case, the mobile device of some embodiments notifies any other device participating in the video conference of this switch so that the other device can provide on its end a smooth transition between the video captured by the two cameras.

In some embodiments, the request to switch cameras can originate not only from a "local" device that is switching between its cameras during a video conference, but also from another "remote" device that is receiving video captured by the local device. equipment. Furthermore, allowing one device to instruct another device to switch cameras is just one example of the remote control capabilities of the devices of some embodiments. In some embodiments, examples of other operations that can be commanded remotely to the device include exposure adjustment operations (eg, auto exposure), focus adjustment operations (eg, auto focus), and the like. Another example of a novel in-session adjustment that can be specified locally or remotely is identifying a region of interest (ROI) in a captured video, and using said ROI identification to modify the behavior of the capturing camera, modifying the image processing of a device with the capturing camera operation, or modify the coded operation of a device with a capture camera.

Yet another example of the novel mid-meeting adjustments of some embodiments involves real-time modification of device-generated composite video displays. Specifically, in some embodiments, a mobile device generates a composite display that simultaneously displays multiple videos captured by multiple cameras of one or more devices. In some cases, composite display places video in adjacent display areas (eg, in adjacent windows). In other cases, the composite display is a picture-in-picture (PIP) display that includes at least two display areas displaying two different videos, one of which is a background main display area and the other display area is an overlay The foreground inset display area on top of the background main display area.

Real-time modification of the composite video display in some embodiments involves moving one or more display regions within the composite display in response to user selection and movement of the display regions. Some embodiments also rotate the composite display during a video conference when the screen of the device providing the composite display rotates. Additionally, the mobile device of some embodiments allows a user of the device to swap video in a PIP display (i.e., have video in the foreground inset display appear in the background main display, and have video in the background main display appear in the foreground inset displaying).

The foregoing summary of the invention is intended to briefly introduce some embodiments of the invention. It is not intended to be an introduction or overview of all inventive subject matter disclosed in this document. The following detailed description and the drawings referred to in the detailed description will further illustrate the embodiments described in this summary, as well as other embodiments. Therefore, an understanding of all of the embodiments described in this document requires a complete review of the Summary, Detailed Description and Drawings.

Description of drawings

The novel features of the invention are set forth in the appended claims. However, for purposes of illustration, several embodiments of the invention are shown in the drawings described below.

Figure 1 illustrates a synthetic display of some embodiments.

Figure 2 illustrates another synthetic display of some embodiments.

Figure 3 conceptually illustrates the software architecture of the video processing and encoding modules of the dual camera mobile device of some embodiments.

Figure 4 conceptually illustrates a captured image processing unit of some embodiments.

Figure 5 conceptually illustrates an example of different frame rates based on different vertical blanking intervals (VBI).

Figure 6 conceptually illustrates an example of different interlaced frame rates based on different VBIs.

Figure 7 conceptually illustrates another captured image processing unit of some embodiments.

Figure 8 conceptually illustrates another captured image processing unit of some embodiments.

Figure 9 conceptually illustrates the software architecture of the video conferencing and processing modules of the dual camera mobile device of some embodiments.

Figure 10 conceptually illustrates an example video conference request messaging sequence of some embodiments.

Figure 11 illustrates a user interface for video conference setup operations of some embodiments.

Figure 12 illustrates a user interface for accepting a video conference invitation of some embodiments.

Figure 13 illustrates another user interface of some embodiments for accepting a video conference invitation.

Figure 14 illustrates another user interface for video conference setup operations of some embodiments.

Figure 15 conceptually illustrates the process of setting the bit rate for a video conference of some embodiments.

Figure 16 conceptually illustrates another software architecture for the video conferencing and processing module of a dual camera mobile device of some embodiments.

Figure 17 conceptually illustrates another software architecture for a dual camera mobile device of some embodiments.

FIG. 18 conceptually illustrates the processing performed by a video conference manager of some embodiments as illustrated in FIG. 16 .

Figure 19 conceptually illustrates the software architecture of the temporal noise reduction module of some embodiments.

Figure 20 conceptually illustrates the process of reducing temporal noise of video images of some embodiments.

FIG. 21 conceptually illustrates the processing performed by the image processing manager of some embodiments as illustrated in FIG. 9 .

Figure 22 illustrates a user interface for exposure adjustment operations of some embodiments.

Figure 23 illustrates the user interface for the focus adjustment operation of some embodiments.

FIG. 24 conceptually illustrates the perspective correction process performed by the image processing manager of some embodiments as illustrated in FIG. 16 .

Figure 25 conceptually illustrates an example perspective correction operation of some embodiments.

FIG. 26 conceptually illustrates the software architecture of an encoder driver of some embodiments as illustrated in FIG. 16 .

FIG. 27 conceptually illustrates image scaling processing performed by an encoder driver of some embodiments as illustrated in FIG. 26 .

FIG. 28 conceptually illustrates the software architecture of the decoder driver of some embodiments as illustrated in FIG. 16 .

FIG. 29 conceptually illustrates the image extraction process performed by a decoder driver of some embodiments as illustrated in FIG. 28 .

Figure 30 illustrates an encoder driver including two rate controllers of some embodiments.

FIG. 31 conceptually illustrates the software architecture of a networking manager of some embodiments as illustrated in FIG. 16 .

Figure 32 illustrates the user interface for the snap to corner operation of some embodiments.

Figure 33 illustrates another user interface for the lock to corner operation of some embodiments.

Figure 34 illustrates a user interface for a PIP display rotation operation of some embodiments.

Figure 35 illustrates another user interface for a PIP display rotation operation of some embodiments.

Figure 36 illustrates another user interface for a PIP display rotation operation of some embodiments.

Figure 37 illustrates another user interface for a PIP display rotation operation of some embodiments.

Figure 38 illustrates a user interface for scaling a foreground inset display area in a PIP display of some embodiments.

Figure 39 illustrates another user interface for zooming inset display areas in a PIP display of some embodiments.

Figure 40 illustrates another user interface for zooming inset display areas in a PIP display of some embodiments.

Figure 41 illustrates another user interface for zooming inset display areas in a PIP display of some embodiments.

Figure 42 illustrates a user interface of some embodiments for identifying a region of interest in a display.

Figure 43 illustrates another user interface of some embodiments for identifying a region of interest in a display.

Figure 44 illustrates another user interface of some embodiments for identifying a region of interest in a display.

Figure 45 illustrates the process of locally switching camera operations on a dual camera mobile device of some embodiments.

Figure 46 illustrates a user interface for switching camera operations of some embodiments.

Figure 47 illustrates another user interface for switching camera operations of some embodiments.

Figure 48 illustrates another user interface for switching camera operations of some embodiments.

Figure 49 illustrates another user interface for switching camera operations of some embodiments.

Figure 50 illustrates the process of remotely switching camera operations on a dual camera mobile device of some embodiments.

Figure 51 illustrates a user interface for remote control switching camera operations of some embodiments.

Figure 52 illustrates another user interface for remote control switching camera operation of some embodiments.

Figure 53 illustrates another user interface for remote control switching camera operation of some embodiments.

Figure 54 illustrates another user interface for remote control switching camera operation of some embodiments.

Figure 55 conceptually illustrates the process of some embodiments for performing exposure adjustment operations.

Figure 56 illustrates a user interface for performing exposure adjustment operations of some embodiments.

Figure 57 illustrates another user interface of some embodiments for exposure adjustment operations.

Figure 58 illustrates another user interface of some embodiments for exposure adjustment operations.

FIG. 59 conceptually illustrates exposure adjustment processing performed by an image processing manager of some embodiments as illustrated in FIG. 16 .

Figure 60 conceptually illustrates the exposure adjustment operation of some embodiments.

Figure 61 conceptually illustrates the process of performing a focus adjustment operation of some embodiments.

Figure 62 illustrates the user interface for the focus adjustment operation of some embodiments.

Figure 63 illustrates another user interface for the focus adjustment operation of some embodiments.

Figure 64 illustrates another user interface for the focus adjustment operation of some embodiments.

Figure 65 illustrates different display arrangements of some embodiments for video captured from one or more dual camera mobile devices.

Figure 66 illustrates a user interface of some embodiments that overlays the foreground of the inserted video over the background video in the PIP display.

Figure 67 illustrates a technique of some embodiments for determining the foreground of a video image.

Figure 68 illustrates a user interface for swapping an inset display and a background display in a PIP display during a video conference, of some embodiments.

Figure 69 illustrates the user interface for the lock to corner operation of some embodiments.

Figure 70 illustrates the user interface of some embodiments for pin to corner and push operations.

Figure 71 illustrates a user interface for a PIP display rotation operation of some embodiments.

Figure 72 illustrates another user interface for a PIP display rotation operation of some embodiments.

Figure 73 illustrates a user interface for selecting one of two remote videos during a video conference, in some embodiments.

Figure 74 illustrates a user interface for selecting one of two local videos during a video conference of some embodiments.

Figure 75 illustrates a user interface for pre-meeting selection of video for a video conference of some embodiments.

Figure 76 illustrates an example of bandwidth allocation between two videos taken by a dual camera mobile device of some embodiments.

Figure 77 conceptually illustrates the arbiter module of some embodiments managing the rate controller of a dual camera mobile device.

Figure 78 conceptually illustrates a method of some embodiments for encoding an image captured by a camera of a dual camera mobile device.

Figure 79 conceptually illustrates another method of some embodiments for encoding an image captured by a camera of a dual camera mobile device.

FIG. 80 illustrates example image composition for the method illustrated in FIG. 79 .

Figure 81 conceptually illustrates another method of some embodiments for encoding an image captured by a camera of a dual camera mobile device.

Figure 82 conceptually illustrates a method of some embodiments for decoding an image captured by a camera of a dual camera mobile device.

Figure 83 conceptually illustrates another method of some embodiments for decoding an image captured by a camera of a dual camera mobile device.

Figure 84 conceptually illustrates another software architecture for the video conferencing and processing module of a dual camera mobile device of some embodiments.

Figure 85 illustrates a user interface for a multi-party video conference of some embodiments.

Figure 86 illustrates another user interface for a multi-party video conference of some embodiments.

Figure 87 illustrates another user interface for a multi-party video conference of some embodiments.

Figure 88 conceptually illustrates the application programming interface (API) architecture of some embodiments.

Figure 89 illustrates the architecture of a dual camera mobile computing device of some embodiments.

Figure 90 conceptually illustrates a touch input/output (I/O) device of some embodiments.

Figure 91 conceptually illustrates an example communication system of some embodiments.

Figure 92 conceptually illustrates another example communication system of some embodiments.

Detailed ways

In the following description, numerous details are set forth for purposes of explanation. However, one of ordinary skill in the art will recognize that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to obscure the description of the invention with unnecessary detail.

Some embodiments of the invention provide a mobile device with two cameras capable of taking pictures and video. Examples of mobile devices include mobile phones, smart phones, personal digital assistants (PDAs), laptop computers, tablet personal computers, or any other type of mobile computing device. Photographs as used herein refer to still photo images captured by a camera one at a time in single-shot mode, or several at a time in quick-shot mode. Video, on the other hand, refers to a series of video images captured by a camera at a specific rate (often called a frame rate). Typical frame rates for shooting video are 25 frames per second (fps), 30fps, and 60fps. The camera of the mobile device of some embodiments is capable of capturing video images (ie, video frames) at these and other frame rates.

Mobile devices of some embodiments are (1) capable of displaying captured photo images and video images, (2) capable of saving captured images for later transfer to another device, (3) capable of sharing images between multiple users of multiple devices transmit captured images to one or more devices during a real-time communication session between them, and (4) be able to encode captured images for local storage or transmission to another device.

An example of a real-time communication session involving the transmission of captured video images is video conferencing. In some embodiments, the mobile device can only transmit video images captured by one camera at any given time during the videoconference. However, in other embodiments, during a video conference or other real-time communication session, the mobile device can simultaneously transmit video images captured by its two cameras.

The mobile device of some embodiments produces a composite display that includes a simultaneous display of multiple videos captured by multiple cameras of one or more devices. In some cases, composite display places video in adjacent display areas (eg, in adjacent windows). Figure 1 illustrates one such example of a composite display 100 that includes two adjacent display areas 105 and 110 that simultaneously display images taken by two cameras of a device, or by Two videos captured by two cameras of two different devices participating in a video conference.

In other cases, the composite display is a PIP display comprising at least two display areas displaying two different videos, one of which is a background main display area and the other display area is a foreground inset superimposed on top of the background main display area display area. One such example of a composite PIP display 200 is illustrated in FIG. 2 . Composite PIP display 200 includes a background main display area 205 and a foreground inset display area 210 superimposed on the background main display area. The two display areas 205 and 210 simultaneously display two videos captured by two cameras of one device, or two videos captured by two cameras of two different devices participating in the video conference. Although the example composite PIP display illustrated and discussed herein is similar to composite PIP display 200 that displays the entire foreground inset display area 210 within background main display area 205, but with an overlay on background main display area 205, but not Other composite PIP displays with a foreground inset display area 210 entirely within the background main display area 205 are also possible.

In addition to communicating video content during a video conference with another device, the mobile device of some embodiments is capable of communicating other types of content along with the video content of the meeting. An example of such other content includes low or high resolution photo images captured by one of the device's cameras while another camera of the device is capturing video for a video conference. Other examples of such other content include (1) files and other content stored on the device, (2) the device's screen display (i.e., content displayed on the device's screen), (3) Content received from another device, etc. during a communication session.

The mobile device of some embodiments employs novel mid-meeting adjustment techniques to make adjustments during a video conference. For example, when only video captured by one camera is being transmitted during a video conference, the mobile device of some embodiments can dynamically switch to transmitting video captured by its other camera. In this case, the mobile device of some embodiments notifies any other device participating in the videoconference of this switch so that the other device can provide a smooth transition between the video captured by the two cameras on its end .

In some embodiments, the request to switch cameras can originate not only from a "local" device that is switching between its cameras during a video conference, but also from another "remote" device that is receiving video captured by the local device. equipment. Furthermore, allowing one device to instruct another device to switch cameras is just one example of the remote control capabilities of the devices of some embodiments. In some embodiments, examples of other operations that can be commanded remotely to the device include exposure adjustment operations (eg, auto exposure), focus adjustment operations (eg, auto focus), and the like. Another example of a novel in-session adjustment that can be specified locally or remotely is identifying a region of interest (ROI) in a captured video, and using said ROI identification to modify the behavior of the capturing camera, modifying the image processing of a device with the capturing camera operation, or modify the coded operation of a device with a capture camera.

Yet another example of the novel mid-meeting adjustments of some embodiments involves real-time modification of device-generated composite video displays. Specifically, in some embodiments, real-time modification of the composite video display involves moving one or more display regions within the composite display in response to user selection and movement of the display regions. Some embodiments also rotate the composite display during a video conference when the screen of the device providing the composite display rotates. Additionally, the mobile device of some embodiments allows the user of the device to flip the order of the videos in the PIP display (i.e., have the video in the foreground inset display appear in the background main display, and the video in the background main display appear in the background main display). foreground inset display).

Several more detailed embodiments are described below. Section I provides a description of the video processing architecture of some embodiments. Subsequently, Section II describes the captured image processing unit of some embodiments. In some embodiments, the captured image processing unit is a component of the device responsible for processing raw images captured by a camera of the device.

Next, Section III describes the videoconferencing architecture of some embodiments. Section III also describes the video conferencing module of some embodiments, and several ways to set up a single camera video conference. Section IV then describes in-session moderation and control operations of some embodiments. Section V then describes video conferencing features of some embodiments for transmitting and displaying multiple videos from various devices during a video conference. Next, Section VI describes the transmission of real-time video along with non-real-time content during videoconferencing. Finally, Section VII describes the hardware architecture of the dual camera device of some embodiments.

I. Video shooting and processing

Figure 3 conceptually illustrates a video processing and encoding module 300 of a dual camera mobile device of some embodiments. In some embodiments, module 300 processes images captured by cameras of a dual camera mobile device and encodes video. As shown in FIG. 3 , the module 300 includes a captured image processing unit (CIPU) driver 305 , a media exchange module 310 , an encoder driver 320 and a video processing module 325 .

In some embodiments, the media exchange module 310 allows programs on the device (consumers and producers of media content) to exchange media content and instructions regarding the processing of the media content. Within video processing and encoding module 300 , media exchange module 310 of some embodiments routes these instructions and media content between video processing module 325 and CIPU driver 305 , and between video processing module 325 and encoder driver 320 . To facilitate the routing of the instructions and media content, the media exchange module 310 of some embodiments provides a set of application programming interfaces (APIs) for use by consumers and producers of media content. In some of these embodiments, media exchange module 310 is a set of one or more frameworks that are part of an operating system running on a dual camera mobile device. An example of such a media switching module 310 is the Core Media architecture provided by Apple Inc.

The video processing module 325 performs image processing on images and/or videos captured by the camera of the device. Examples of such operations include exposure adjustment operations, focus adjustment operations, perspective correction, dynamic range adjustment, image scaling, image compositing, and the like. In some embodiments, some image processing operations may also be performed by the media exchange module 310 . For example, as shown in FIG. 3 , the media exchange module 310 of some embodiments performs temporal noise reduction (TNR) operations (e.g., with TNR 315) that reduce noise in video images captured by the device's camera. Additional examples of such image processing operations by video processing module 325 and media exchange module 310 are provided below.

Through the media exchange module 310, the video processing module 325 interfaces with the CIPU driver 305 and the encoder driver 320, as described above. The CIPU driver 305 serves as a communication interface between the captured image processing unit (CIPU) 330 and the media exchange module 310 . As described further below, the CIPU 330 is a component of a dual camera device responsible for processing images captured during image capture or video capture operations of the device's cameras. The CIPU driver 305 receives requests from the video processing module 325 via the media exchange module 310 for images and/or video from one or both cameras of the device. The CIPU driver 305 transmits the request to the CIPU 330, as a response, receives the requested image and/or video from the CIPU 330, and the CIPU driver 305 then sends the received image and/or video to the video processing module through the media exchange module 310 325. Through CIPU driver 305 and media exchange module 310, video processing module 325 of some embodiments also sends instructions to CIPU 330 in order to modify some of its operations (e.g., modify camera frame rate, exposure adjustment operations, focus adjustment operations, etc.).

The encoder driver 320 acts as a communication interface between the media exchange module 310 and the encoder hardware 335 (eg, an encoder chip, an encoding component on a chip system, etc.). In some embodiments, encoder driver 320 receives images and requests to encode images from video processing module 325 via media exchange module 310 . The encoder driver 320 sends the image to be encoded to the encoder 335, which then photo-encodes or video-encodes the image. When the encoder driver 320 receives the encoded image from the encoder 335 , the encoder driver 320 sends the encoded image back to the video processing module 325 through the media exchange module 310 .

In some embodiments, the video processing module 325 can perform different operations on the encoded images it receives from the encoder. Examples of such operations include saving the encoded image in the device's memory, transmitting the encoded image in a video conference through the device's network interface, and the like.

In some embodiments, some or all modules of video processing and encoding module 300 are implemented as part of an operating system. For example, some embodiments implement all four components 305, 310, 320, and 325 of video processing and encoding module 300 as part of the device's operating system. Other embodiments implement the media exchange module 310, CIPU driver 305, and encoder driver 320 as part of the device's operating system, while the video processing module 325 is an application running on the operating system. Additionally, other implementations of module 300 are possible.

The operation of the video processing and encoding module 300 during a video capture session is now described. To start a video capture session, the video processing module 325 initializes several components required for the video capture session. In some embodiments, these components include (1) CIPU 330, (2) scaling and compositing module (not shown) of video processing module 325, (3) image processing module (not shown) of video processing module 325 , and (4) encoder 335 . Additionally, the video processing module 325 of some embodiments initializes a network manager (not shown) when participating in a video conference.

Through the media exchange module 310 and the CIPU driver 305, the video processing module sends its initialization request to the CIPU 330, so that one or two cameras of the device start video shooting. In some embodiments, the request specifies a specific frame rate, exposure, and scale size for each camera that needs to capture the video. In response to the request, the CIPU 330 begins returning video images from the requested camera at the specified frame rate, exposure, and scale size. These video images are returned to the video processing module 325 through the CIPU driver 305 and the media exchange module 310. As mentioned above, the media exchange module 310 performs a TNR operation on the video images before providing them to the video processing module 325. At video processing module 325, video images are stored in a buffer (not shown) for additional image processing.

The image processing module of the video processing module 325 retrieves the video images stored in the buffer for additional video processing. The scaling and compositing module then retrieves the processed video image for scaling the video image if necessary for real-time display on the device's display screen. In some embodiments, the module creates a composite image from images taken by both cameras of the device, or from images taken by the camera of the device and the camera of another device during a video conference, to provide a captured video on the device Real-time display of images, or creation of composite video images for encoding.

The processed and/or composited video images are provided to the encoder 335 through the encoder driver 320 and the media exchange module 310 . Encoder 335 then encodes the video images. The encoded image is then returned to video processing module 325 (again via encoder driver 320 and media exchange module 310) for storage on the device, or for transmission during a video conference. When a device participates in a videoconference, the network manager (initialized by the video processing module 325) then retrieves these encoded images, packetizes the encoded images, and transmits the encoded images to one or more other devices.

II. Shooting image processing

A. Single pipeline

Images captured by the cameras of the dual camera mobile device of some embodiments are raw, unprocessed images. Before these images can be used for other operations, such as transmitting the image to another device (eg, during a video conference), saving the image, or displaying the image, these images need to be transformed into a specific color space. Additionally, images captured by the camera need to be processed to correct errors and/or distortions, and to adjust the color, size, etc. of the image. Accordingly, some embodiments perform several processing operations on images prior to saving, transmitting, and displaying such images. Part of the processing of the image is performed by CIPU 330.

An example of the CIPU is illustrated in FIG. 4 . In particular, Figure 4 schematically illustrates captured image processing unit (CIPU) 400 of some embodiments. The CIPU 400 includes a single processing pipeline that either processes images from one of the device's cameras at a time, or simultaneously processes images from both of the device's cameras in a time-division multiplexed manner (i.e., in a time-interleaved manner). pipeline)485. Processing pipeline 485 of CIPU 400 may be configured differently to handle different characteristics and/or operating settings of different cameras. Examples of different camera characteristics in some embodiments include different resolutions, noise sensors, lens types (fixed or zoom), etc. Additionally, in some embodiments, examples of different operational settings at which the device can operate the camera include image resolution size, frame rate, zoom level, exposure, and the like.

As shown in FIG. 4, CIPU 400 includes sensor module 415, line/frame buffer 417, bad pixel correction (BPC) module 420, lens shading (LS) module 425, demosaic module 430, white balance (WB) module 435 , gamma module 440, color space conversion (CSC) module 445, hue, saturation and contrast (HSC) module 450, scaler module 455, filter module 460, statistics engine 465, two sets of registers 470, and controller module 475. In some embodiments, all modules of CIPU 400 are implemented in hardware (e.g., ASIC, FPGA, SOC with microcontroller, etc.), while in other embodiments, some or all modules of CIPU 400 It is realized by software.

As shown in FIG. 4, sensor module 415 is communicatively coupled to two pixel arrays 410a and 410b of the device's two cameras, and two sets of sensors 405a and 405b. In some embodiments, this communication coupling is facilitated through a Mobile Industry Processor Interface (MIPI) for each camera sensor.

Through this communicative coupling, the sensor module 415 is able to forward instructions to the cameras to control various aspects of each camera's operation, such as its power level, zoom level, focal length, exposure, and the like. In some embodiments, each camera has four operating power modes. In the first operating power mode, the camera is powered off. For the second power mode of operation, the camera is powered on, but the camera is not yet configured. In a third operating power mode, the camera is powered on, the camera's sensor is configured, and the pixels of the camera sensor collect photons and convert the collected photons into digital values. However, the camera sensor has not yet sent an image to the sensor module 415 . Finally, in the fourth power mode of operation, the camera is in the same power mode of operation as the third power mode, except that the camera is now sending images to the sensor module 415 .

During operation of the device, the camera may switch from one operating power mode to another several times. When switching the working power mode, some embodiments require the camera to switch the working power mode in the above order. Therefore, in these embodiments, the camera in the first working power mode can only be switched to the second working power mode. When the camera is in the second working power mode, it can switch to the first working power mode or the third working power mode. Similarly, the camera can switch from the third power mode of operation to the second power mode of operation or the fourth power mode of operation. When the camera is in the fourth working power mode, it can only switch back to the third working power mode.

Furthermore, switching from one operating power mode to the next or previous operating power mode requires a certain amount of time. Thus, switching between two or three operating power modes is slower than switching between one operating power mode. Different operating power modes also consume different amounts of power. For example, the fourth operating power mode consumes the largest amount of power, the third operating power mode consumes more power than the first and second operating power modes, and the second operating power mode consumes more power than the first A working power mode. In some embodiments, the first power mode of operation does not consume any power.

When the camera is not in the fourth operating power mode for capturing images, the camera can be kept in one of the other operating power modes. The determination of which operating mode to keep an unused camera in depends on how much power the camera is allowed to consume and how quickly the camera needs to respond to a request to start taking images. For example, a camera configured to operate in a third operating power mode (eg, standby mode) consumes more power than a camera configured in a first operating power mode (ie, powered off). However, when the camera is instructed to take an image, the camera operating in the third operating power mode can switch to the fourth operating power mode faster than the camera operating in the first operating power mode. Thus, depending on different requirements (eg, response time to a request to capture an image, power consumption), the camera can be configured to operate in different operating power modes when not capturing an image.

As further described below, when the video processing module 325 requests one or both cameras to begin capturing images, and the sensor module 415 receives this request through the controller module 475, through its communicative coupling with each camera, the sensor module 415 415 can instruct one or two sets of camera sensors to start capturing images. A Bayer filter is superimposed on each camera sensor so that each camera sensor outputs a Bayer formatted image, which is stored in an array of pixels associated with each camera sensor. A Bayer format image is one in which each pixel holds only one color value: red, blue, or green.

Through its coupling to pixel arrays 410a and 410b, sensor module 415 retrieves the raw Bayer format images stored in camera pixel arrays 410a and 410b. By controlling the rate at which the sensor module 415 retrieves images from the camera's pixel array, the sensor module 415 is able to control the frame rate of video images captured by a particular camera. By controlling the rate at which its images are retrieved, the sensor module 415 is also able to interleave the reading of images captured by different cameras in order to interleave the image processing by the CIPU processing pipeline 485 of captured images from different cameras. The sensor module 415's control of its image retrieval is further described below in subsections II.A.1 and II.A.2.

Sensor module 415 stores image lines (ie, rows of pixels of an image) that it retrieves from pixel arrays 410 a and 410 b in line/frame buffer 417 . Each image line in the line/frame buffer 417 is processed through the CIPU processing pipeline 485 . As shown in FIG. 4, CIPU processing pipeline 485 consists of BPC module 420, LS module 425, demosaic module 430, WB module 43, gamma module 440, CSC module 445, HSC module 450, scaler module 455 and filter module 460 composition. In some embodiments, the CIPU processing pipeline 485 processes images from the line/frame buffer 417 row-by-row (i.e., row-by-row), while in other embodiments, the CIPU processing pipeline 485 processes images from the row/frame buffer frame-by-frame. The entire image of the device 417.

In the example pipeline illustrated in FIG. 4 , the BPC module 420 is the module that retrieves images from the line/frame buffer 417 . The BPC module 420 performs a bad pixel removal operation that seeks to correct bad pixels in the retrieved images that may be caused by one or more defective camera sensors (e.g., a defective photon sensor that does not sense light at all, incorrectly senses light, etc.). In some embodiments, BPC module 420 detects bad pixels by comparing a particular pixel in the image to one or more neighboring pixels in the image. If the difference between the value of the particular pixel and the values of the neighboring pixels is greater than a threshold amount, then the value of several neighboring pixels of the same color (i.e., red, green, and blue) as the particular pixel The average value replaces the value of that particular pixel.

The operation of the BPC module 420 is controlled in part by the values held in two sets of registers 470 of the CIPU 400 for that module. Specifically, to process images captured by two different cameras of a device, some embodiments configure the CIPU processing pipeline 485 differently with respect to each camera, as described above. The CIPU processing pipeline 485 is configured for two different cameras by storing two different sets of values in two different sets of registers 470a (Ra) and 470b (Rb) of the CIPU 400. Each set of registers 470 includes one register (Ra or Rb) for each module 420 - 460 within the CIPU processing pipeline 485 . Each register in each register bank holds a set of values that define the operation of a processing pipeline block. Thus, as shown in FIG. 4, register set 470a is used to indicate the operating mode of each processing pipeline module of one camera (camera A) of a dual-camera mobile device, while register set 470b is used to indicate the other of the dual-camera mobile device. The working mode of each processing pipeline module of a camera (Camera B).

One example of configuring the CIPU processing pipeline 485 differently for each camera is to configure the modules of the CIPU processing pipeline 485 to process images of different sizes. For example, if camera sensor 405a is 640 x 480 pixels and camera sensor 405b is 2048 x 1536 pixels, then register set 470a is configured to hold values that instruct the various modules of CIPU processing pipeline 485 to process a 640 x 480 pixel image, and register set 470b It is configured to hold values that instruct various modules of the CIPU processing pipeline 485 to process a 2048x1536 pixel image.

In some embodiments, different processing pipeline configurations (ie, register values) are stored in different profile settings. In some of these embodiments, a user of the mobile device is allowed to select one of the profile settings (eg, via a user interface displayed on the mobile device) to set the operation of one or more cameras. For example, a user can choose to configure a camera with a profile setting that takes high-resolution video, configure the same camera with a profile setting that takes low-resolution video, or configure both cameras for a high-resolution still image. Profile settings. A variety of different configurations are possible and can be saved in many different profile settings. In other such embodiments, instead of allowing the user to select a profile setting, the profile setting is automatically selected based on the application or activity selected by the user. For example, if the user selects a videoconferencing application, a profile that configures both cameras to capture video is automatically selected, if the user selects a camera application, a profile that configures one of the cameras to capture still images is automatically selected, and so on.

After the BPS module 420, the LS module 425 receives the bad pixel corrected image. The LS module 425 performs lens shading correction operations to correct image defects caused by camera lenses that produce a light falloff effect (ie, light gradually decreases towards the edges of the camera sensor). This effect results in uneven image illumination (eg, darker in corners and/or edges). To correct for these image defects, the LS module 425 of some embodiments estimates a mathematical model of the illumination falloff of the lens. The estimated model is then used to compensate for lens falloff of the image to uniformly illuminate parts of the image that are not uniformly illuminated. For example, if the corners of the image are half as bright as the center of the image, the LS module 425 of some embodiments multiplies the corner pixel values by 2 in order to produce a uniform image.

The demosaicing module 430 performs a demosaicing operation to produce a full-color image from the sampled-color image. As mentioned above, the camera sensor outputs a Bayer format image, which is incomplete because each pixel of the Bayer format image stores only one color value. The demosaicing module 430 reconstructs a red, green, blue (RGB) image from the Bayer format image by interpolating the color values of each set of colors in the Bayer format image.

The WB module 435 performs a white balance operation on the RGB image received from the demosaic module 430 so that the color of the image content is similar to the color of the content perceived by human eyes in real life. The WB module 435 adjusts the white balance by adjusting the color of the image to correctly render neutral colors (eg, gray, white, etc.). For example, an image of a piece of white paper under incandescent light may appear yellow, while the human eye perceives the piece of paper to be white. In order to solve the difference between the color of the image captured by the sensor and the color of the image perceived by the human eye, the WB module 435 adjusts the color value of the image so that the captured image correctly reflects the color perceived by the human eye.

The statistical engine 465 collects image data at various stages of the CIPU processing pipeline 485 . For example, FIG. 4 shows statistical engine 465 collecting image data after LS module 425 , demosaicing module 430 , and WB module 435 . Different embodiments collect data from any number of different stages of the CIPU processing pipeline 485 . Statistics engine 465 processes the collected data and adjusts the operation of camera sensors 405a and 405b via controller module 475 and sensor module 415 based on the processed data. Examples of such operations include exposure and focusing. Although FIG. 4 shows statistical engine 465 controlling camera sensors 405 a and 405 b via controller module 475 , other embodiments of statistical engine 465 control camera sensors via sensor module 415 .

The processed data can also be used to adjust the operation of the various modules of the CIPU 400. For example, statistics engine 465 of some embodiments adjusts the operation of WB module 435 based on data collected after WB module 435 . In some of these embodiments, statistical engine 465 provides an automatic white balance (AWB) function by utilizing the processed data to adjust the white balance operation of WB module 435 . Other embodiments may use processed data collected from any number of stages of the CIPU processing pipeline 485 to adjust the operation of any number of modules within the CIPU processing pipeline 485 . Additionally, statistics engine 465 can also receive instructions from controller module 475 to adjust the operation of one or more modules of CIPU processing pipeline 485 .

After receiving an image from WB module 435, gamma module 440 performs a gamma correction operation on the image to encode and decode luminance or tristimulus values for the camera system. The gamma module 440 in some embodiments corrects the gamma value of the image by converting the 10-12-bit linear signal into an 8-bit non-linear code, and performs gamma value correction. Some embodiments correct gamma values using a lookup table.

The CSC module 445 transforms the image received from the gamma module 440 from one color space to another. Specifically, the CSC module 445 transforms the image from the RGB color space to the luma and chrominance (YUV) color space. However, other embodiments of the CSC module 445 may transform images to and from any number of color spaces.

The HSC module 450 may adjust the hue, saturation, contrast, or any combination thereof of the image received from the CSC module 445 . For example, HSC module 450 may adjust these properties to reduce noise or enhance images. For example, you can increase the saturation of an image captured by a low-noise camera sensor to make the image appear more vibrant. Conversely, images captured by noisy camera sensors can be desaturated to reduce the color noise of such images.

After the HSC module 450, the scaler module 455 may scale the image to adjust the pixel resolution of the image, or to adjust the data size of the image. For example, the scaler module 455 may also reduce the size of the image to fit smaller displays. For example, the scaler module 455 can scale images in many different ways. For example, the scaler module 455 can scale up (ie, enlarge) and scale down (ie, reduce) the image. The scaler module 455 can also scale the image proportionally, or anamorphically scale the image.

Filter module 460 applies one or more filtering operations to the image received from scaler module 455 to alter one or more properties of some or all of the pixels of the image. Examples of filters include low-pass filters, high-pass filters, band-pass filters, bilateral filters, Gaussian filters, and the like. Thus, filter module 460 is capable of applying any number of different filters to the image.

The controller module 475 of some embodiments is a microcontroller that controls the operation of the CIPU 400. In some embodiments, the controller module 475 (1) controls the operation of the camera sensor (e.g., exposure) through the sensor module 41, (2) controls the operation of the CIPU processing pipeline 485, (3) controls the operation of the CIPU processing pipeline 485 Timing (e.g., when to switch camera sensors, when to switch registers, etc.), and (4) control the flash/strobe (not shown), which is the dual camera movement of some embodiments part of the device.

Some embodiments of controller module 475 process instructions received from statistics engine 465 and CIPU driver 480 . In some embodiments, the instructions received from CIPU driver 480 are instructions from a dual camera mobile device (i.e., received from a native device), while in other embodiments, the instructions received from CIPU driver 480 are from another device Instructions (for example, remote control during a video conference). According to the processed instruction, the controller module 475 can adjust the operation of the CIPU 400 by programming the value of the register 470. Additionally, the controller module 475 is capable of dynamically reprogramming the values of the registers 470 during operation of the CIPU 400.

As shown in FIG. 4, CIPU 400 includes a number of modules in CIPU processing pipeline 485. However, those of ordinary skill in the art will recognize that the CIPU 400 may be implemented with only some of the illustrated modules, or with additional different modules. Additionally, the processing performed by the different modules may be applied to the image in an order different from that illustrated in FIG. 4 .

Referring now to FIG. 4, an example operation of the CIPU 400 is illustrated. For purposes of illustration, a set of registers Ra is used to process images captured by the camera sensor 405a of a dual camera mobile device, and a set of registers Rb is used to process images captured by the camera sensor 405b of a dual camera mobile device. The controller module 475 receives instructions from the CIPU driver 480 to generate an image captured by one of the cameras of the dual camera mobile device.

The controller module 475 then initializes the various modules of the CIPU processing pipeline 485 to process images captured by one of the cameras of the dual camera mobile device. In some embodiments, this includes the controller module 475 checking whether the correct set of registers 470 is used. For example, if the CIPU driver 480 instructs the controller module 475 to generate an image captured by the camera sensor 405a, the controller module 475 checks whether the set of registers Ra is the set of registers read by the modules of the CIPU 400. If not, then the controller module 475 switches between the two sets of registers such that the set of registers Ra is the set of registers read by the modules of the CIPU 400.

For each module in the CIPU processing pipeline 485, the mode of operation is indicated by the value held in the set of registers Ra. As previously mentioned, the values in a set of registers 470 can be dynamically reordered during operation of the CIPU 400. Thus, the processing of one image differs from the processing of the next image. Although the discussion of this example operation of the CIPU 400 describes each module in the CIPU 400 reading a value held in a register indicating the mode of operation of the respective module, in some software-implemented embodiments, the parameters are passed to the CIPU 400 instead of each module.

In some embodiments, the controller module 475 initializes the sensor module 415 by instructing the sensor module 415 to delay for a certain amount of time after retrieving an image from the pixel array 410a. In other words, the controller module 475 instructs the sensor module 415 to retrieve images from the pixel array 410a at a certain rate.

Subsequently, the controller module 475 instructs the camera sensor 405 a to capture an image through the sensor module 415 . In some embodiments, the controller module 475 also provides exposure parameters and other camera operating parameters to the camera sensor 405a. In other embodiments, the camera sensor 405a uses default values for the camera sensor operating parameters. According to the parameters, the camera sensor 405a captures a raw image, which is stored in the pixel array 410a. Sensor module 415 retrieves the raw image from pixel array 410a and sends the image to line/frame buffer 417 for storage before CIPU processing pipeline 485 processes the image.

In some cases, images may be discarded by the line/frame buffer 417 . When the camera sensors 405a and/or 405b are capturing images at a high rate, the sensor module 415 can receive images faster than the BPC module 420 can retrieve the images from the line/frame buffer 417 and store the images in the line/frame buffer 417 During normal operation (eg, when shooting high frame rate video), the line/frame buffer 417 can become completely full. When this occurs, the line/frame buffer 417 of some embodiments discards images (ie, frames) on a first-in-first-out basis. That is, when line/frame buffer 417 discards a frame of image, line/frame buffer 417 discards that frame of image received before all other images in line/frame buffer 417 .

The image processing of the CIPU processing pipeline 485 fetches the image from the BPC module 420 back to the line/frame buffer 417, starting with correcting any bad pixels in the image. The BPC module 420 then sends the image to the LS module 425 to correct for any non-uniform illumination in the image. After correcting the illumination of the image, the LS module 425 sends the image to the demosaicing module 430, which processes the original image to generate an RGB image from the original image. Subsequently, the WB module 435 receives the RGB image from the demosaic module 430, and adjusts the white balance of the RGB image.

Statistics engine 465 may have collected some data at various points in CIPU processing pipeline 485, as described above. For example, as illustrated diagrammatically in FIG. 4 , statistical engine 465 collects data after LS module 425 , demosaicing module 430 , and WB module 435 . Based on the collected data, statistics engine 465 may adjust the operation of camera sensor 405a and/or the operation of one or more modules in CIPU processing pipeline 485 in order to adjust the capture of subsequent images from camera sensor 405a. For example, according to the collected data, the statistical engine 465 may determine that the exposure of the current image is too low, and thus instruct the camera sensor 405a through the sensor module 415 to increase the exposure of subsequent images. Thus, the statistics engine 465 of some embodiments acts as a feedback loop for some processing operations.

After the WB module 435 adjusts the white balance of the image, it sends the image to the gamma module 440 for gamma correction (eg, adjusting the gamma curve of the image). The CSC module 445 receives the gamma-corrected image from the gamma module 440 and performs color space transformation. In this example, the CSC module 445 converts the RGB image to a YUV image. In other words, the CSC module 445 converts an image expressed in RGB color space into an image expressed in YUV color space. The HSC module 450 receives the YUV image from the CSC module 445 and adjusts the hue, saturation and contrast attributes of each pixel in the image. Following the HSC module 450, a scaler module 455 scales the image (eg, enlarges or reduces the image). After receiving an image from scaler module 455, filter module 460 applies one or more filters to the image. Finally, the filter module 460 sends the processed image to the CIPU driver 480 .

In this example of the operation of the CIPU 400 described above, each module in the CIPU processing pipeline 485 processes images in some fashion. However, other images processed by CIPU 400 may not require processing by all modules of CIPU processing pipeline 485. For example, images may not require white balance adjustments, gamma corrections, scaling, or filtering. Thus, the CIPU 400 is capable of processing images in any of a variety of ways based on various inputs received, such as instructions from the CIPU driver 480, or data collected by the statistics engine 465.

Different embodiments control the rate at which images are processed (ie, frame rate) differently. One way to control the frame rate is through manipulation of the vertical blanking interval (VBI). For some embodiments that retrieve image rows to process the image row by row, the VBI retrieves the last row of an image from the pixel array for video captured by the cameras of a dual camera mobile device, and the next row of the video from the pixel array. The time difference between the first lines of an image. In other embodiments, the VBI is the time difference between retrieving from the pixel array one image of the video captured by the camera of the dual camera mobile device, and retrieving the next image of the video from the pixel array.

One example where VBI can be used is between sensor module 415 and pixel arrays 410a and 410b. For example, some embodiments of sensor module 415 retrieve images from pixel arrays 410a and 410b row by row, and other embodiments of sensor module 415 retrieve images from pixel arrays 410a and 410b image by image. Thus, by adjusting the VBI of the sensor module 415, the frame rate can be controlled: increasing the VBI decreases the frame rate, and decreasing the VBI increases the frame rate.

1. Application of VBI to single camera: frame rate control

Figure 5 conceptually illustrates examples of different frame rates 505, 510, and 515 based on different VBIs. Each sequence shows, at various times 525-555 along the timeline 520, an image of a character holding a guitar captured by one of the cameras of the dual camera mobile device. In addition, the time between each moment 525-555 is the same, which is called a time unit. For purposes of illustration, FIG. 5 will now be described with reference to sensor module 415 and pixel array 410a of FIG. 4 . Thus, each image represents a moment along the timeline 520 at which the sensor module 415 retrieves the image from the pixel array 410a.

In the example frame rate 505, the VBI of the sensor module 415 with respect to the pixel array 410a is set to 3 time units (eg, by the controller module 475). That is, the sensor module 415 retrieves a frame of image from the pixel array 410 a at every two moments along the time line 520 . As shown in example frame rate 505 , sensor module 415 retrieves images at times 525 , 540 , and 555 . Thus, the example frame rate 505 has a frame rate of one image frame every three time units.

The example frame rate 510 is similar to the example frame rate 505 except that the VBI is set to 2 time units. Therefore, the sensor module 415 retrieves a frame of image from the pixel array 410 a at every other moment along the time line 520 . The example frame rate 510 represents that the sensor module 415 retrieves images from the pixel array 410 a at times 525 , 535 , 545 and 555 . Because the VBI of example frame rate 510 is less than the VBI of example frame rate 505 , the frame rate of example frame rate 510 is higher than the frame rate of example frame rate 505 .

The example frame rate 515 is also similar to the example frame rate 505 except that the VBI of the sensor module 415 with respect to the pixel array 410a is set to 1 time unit. Thus, the sensor module 415 is instructed to retrieve a frame of image from the pixel array 410a at each moment along the timeline 520 . As shown, sensor module 415 retrieves an image from pixel array 410a at times 525-555. The VBI for example frame rate 515 is less than the VBI for example frame rates 505 and 510 . Thus, the frame rate of the example frame rate 515 is greater than the example frame rates 505 and 510 .

2. Application of VBI to dual cameras

Some embodiments may wish to operate both cameras of a dual-camera mobile device simultaneously (eg, to transmit video from both cameras during a video conference). Different embodiments of dual-camera mobile devices that include a single processing pipeline provide different mechanisms for simultaneously operating the two cameras of the dual-camera mobile device.

One such mechanism is to interleave the processing of the images captured by the two cameras by controlling the VBI of each camera. That is, during a VBI of one camera, one or more images captured by another camera are captured and processed, and vice versa. Since the CIPU 400 described above has a single processing pipeline 485, this mechanism can be implemented in the CIPU 400 of some embodiments. In such an embodiment, the sensor module 415 retrieves images from one of the pixel arrays 410a and 410b, and the retrieved images are processed by the CIPU 400 during a VBI of the sensor module 415 with respect to the other pixel array.

The VBI of the sensor module 415 with respect to each pixel array may be set to a specific value. However, in some embodiments, VBI is not set to a value that is less than the time it takes for CIPU 400 to retrieve and process a frame of image. Some embodiments set the VBI of the sensor module 415 to the same value for each pixel array. For example, when the VBI of sensor module 415 is set to the same value for each pixel array, sensor module 415 alternately retrieves images from pixel arrays 410a and 410b. Other embodiments set the VBI of the sensor module 415 to different values for each pixel array. In some such embodiments, the VBI of the sensor module 415 with respect to one pixel array is set to be a multiple of the VBI of the sensor module 415 with respect to the other pixel array. For example, the VBI of the sensor module 415 with respect to one pixel array is set to 2 time units, and the VBI of the sensor module 415 with respect to the other pixel array is set to 4 time units. In this example, sensor module 415 retrieves two frames of image from the one pixel array for every frame of image retrieved by sensor module 415 from the other pixel array.

Figure 6 conceptually illustrates an example of different interleaved frame rates 605, 610, and 615 based on different VBIs. FIG. 6 is similar to FIG. 5 except that FIG. 6 includes 13 time instants 625 - 685 along timeline 620 . Additionally, the image of the character holding the guitar represents the moment along timeline 620 when the image was retrieved from one pixel array, while the image of the character wearing the mortarboard represents the moment along timeline 620 when the image was retrieved from the other pixel array. time.

For purposes of illustration, an image of a character holding a guitar is assumed to have been captured by camera sensor 405a of a dual camera mobile device, and an image of a character wearing a mortarboard is assumed to have been captured by camera sensor 405b of a dual camera mobile device. Furthermore, FIG. 6 will now be described with reference to sensor module 415 and pixel arrays 410 a and 401 b of FIG. 4 .

In the example interleaved frame rate 605, the VBI of the sensor module 415 with respect to the pixel array 410a and the pixel array 401b is set to 2 time units. As shown in example interleaved frame rate 605, sensor module 415 retrieves images from pixel array 410a at times 625, 635, 645, 655, 665, 675, and 685 along timeline 620, and sensor module 415 retrieves images from pixel array 410a at times along timeline 620. At times 630, 640, 650, 660, 670, and 680 on timeline 620, images are retrieved from pixel array 410b. That is, at each time unit, the sensor module 415 alternately retrieves images from the pixel array.

Example interleaved frame rate 610 is similar to example interleaved frame rate 605 except that the VBI of sensor module 415 with respect to pixel array 410a and pixel array 401b is set to 4 time units. Example interleaved frame rate 610 shows sensor module 415 retrieving images from pixel array 410 a at times 625 , 645 , 665 , and 685 along timeline 620 , and sensor module 415 at times 635 , 655 and 685 along timeline 620 . 675. Retrieve an image from the pixel array 410b. Because the VBI of example interleaved frame rate 610 is greater than the VBI of example interleaved frame rate 605 , the frame rate of example interleaved frame rate 610 is less than the frame rate of example interleaved frame rate 605 .

The example interleaved frame rate 615 is also similar to the example interleaved frame rate 605 except that the VBI of the sensor module 415 with respect to the pixel array 410a and the pixel array 401b is set to 6 time units. As shown in FIG. 6, sensor module 415 retrieves images from pixel array 410a at times 625, 655, and 685 along timeline 620, and sensor module 415 retrieves images from pixel array 410a at times 640 and 670 along timeline 620. Array 410b retrieves the image. The VBI of the example interleaved frame rate 615 is greater than the VBIs of the example interleaved frame rates 605 and 610 . Thus, the frame rate of the example interleaved frame rate 615 is less than the frame rates of the example interleaved frame rates 605 and 610 .

B. Multiple pipelines

Figure 7 conceptually illustrates another captured image processing unit (CIPU) 700 of some embodiments. The CIPU 700 implements the same functions as the CIPU 400 described above, except that the CIPU 700 is implemented by two front-end processing pipelines, a memory, and a back-end processing pipeline instead of a single processing pipeline. Accordingly, the description of the functions of the CIPU 700 will be explained with reference to the modules of the CIPU 400.

As shown, CIPU 700 includes front-end processing pipeline 715 for camera sensor 405a and pixel array 410a, front-end processing pipeline 720 for camera sensor 405b and pixel array 410b, memory 725, controller module 730, and back-end processing Line 735. Camera sensors 405a and 405b of some embodiments are sensors of a camera of a dual camera mobile device.

Front-end processing pipelines 715 and 720 of some embodiments perform a portion of CIPU 400's image processing. Thus, different embodiments may include different numbers of modules of CIPU 400. For example, each front-end processing pipeline 715 and 720 of some embodiments includes the sensor module 415, the BPC module 420, the LS module 425, the demosaic module 430, the WB module 435, and the statistics engine 465 of the CIPU 400.

Although front-end processing pipelines 715 and 720 perform the same type of image processing by having the same modules, each module in each front-end processing pipeline 715 and 720 can be configured differently through different register values as explained above with respect to CIPU 400 . Furthermore, since each camera sensor 405a and 405b has its own front-end processing pipeline, the front-end processing pipelines 715 and 720 are able to process images independently of each other. For example, front-end processing pipelines 715 and 720 can process images in parallel (ie, at the same time), at different times, and at different rates.

In some embodiments, each front-end processing pipeline 715 and 720 is capable of retrieving images from its corresponding camera sensor and pixel array. For example, front-end processing pipeline 715 retrieves images captured by camera sensor 405a from pixel array 410a, and front-end processing pipeline 720 retrieves images captured by camera sensor 405b from pixel array 410b. When one of front-end processing pipelines 715 and 720 retrieves an image from its corresponding camera sensor and pixel array, the front-end processing pipeline processes the image and sends the processed image to memory 725 . In addition, each front-end processing pipeline 715 and 720 communicates with controller module 730 (eg, through each front-end processing pipeline's statistics engine) as described above.

The memory 725 of some embodiments holds partially processed images for the back-end processing pipeline 735 to complete processing. In these embodiments, memory 725 receives partially processed images from front-end processing pipelines 715 and 720 and sends the partially processed images to back-end processing pipeline 735 . Some embodiments implement memory 725 as volatile memory (e.g., random access memory (RAM)), while other embodiments implement memory 725 as non-volatile memory (e.g., flash memory, hard disk, optical disk, etc.) . Furthermore, memory 725 of some embodiments is internal memory (eg, RAM), while memory 725 of other embodiments is external memory (eg, Compact Flash (CF) card, Secure Digital (SD) card, etc.).

Some embodiments of the backend processing pipeline 735 perform a portion of the CIPU 700's image processing. In some embodiments, the back-end processing pipeline 735 includes modules of the CIPU 400 that the front-end processing pipelines 715 and 720 do not include. For example, referring to the example above, the backend processing pipeline 735 should include the CSC module 445, the gamma module 440, the HSC module 450, the scaler module 455 and the filter module 460 of the CIPU 400. Thus, the back-end processing pipeline 735 of such an embodiment performs the remaining image processing of the CIPU 400 that the front-end processing pipelines 715 and 720 do not perform. Accordingly, the backend processing pipeline 735 retrieves the partially processed image from the memory 725 and performs the remaining image processing on the partially processed image. After processing the image, the backend processing pipeline 735 sends the processed image to the CIPU driver 480 .

The controller module 730 performs the same functions as explained above with reference to FIG. 4 . As shown in FIG. 7 , controller module 730 interacts with front-end processing pipelines 715 and 720 and back-end processing pipeline 735 . In some embodiments, the controller module 730 is included in the back-end processing pipeline 735 , while in other embodiments, the controller module 730 is included in one of the front-end processing pipelines 715 and 720 .

Operation of the CIPU 700 is now described with reference to camera sensors 405a and 405b, pixel arrays 401a and 410b, front-end processing pipelines 715 and 720, memory 725, and back-end processing pipeline 735 illustrated in FIG. When one of the front-end processing pipelines 715 and 720 retrieves an image from its corresponding camera sensor and pixel array, the front-end processing pipeline processes the image and sends the partially processed image to memory 725 . For example, front-end processing pipeline 715 may retrieve images captured by camera sensor 405a from pixel array 410a, or front-end processing pipeline 720 may retrieve images captured by camera sensor 405b from pixel array 410b. As noted above, both front-end processing pipelines 715 and 720 are capable of processing images in parallel.

The backend processing pipeline 735 retrieves the partially processed image from the memory 725 and processes the partially processed image to complete the image processing of the image. In some embodiments, backend processing pipeline 735 retrieves and processes images stored in memory 725 on a first-in, first-out basis. In other words, a particular image in memory 725 will be processed after all images received and stored in memory 725 prior to that particular image, however, that particular image will be processed after it is received and stored in memory 725 before processing the image. After the backend processing pipeline 735 processes the image, it sends the processed image to the CIPU driver 480 .

Figure 8 conceptually illustrates another captured image processing unit (CIPU) 800 of some embodiments. The CIPU 800 performs the same functions as the CIPU 400 described above, except that the CIPU 800 is implemented by two separate processing pipelines, each camera sensor having its own separate processing pipeline. Thus, a description of the functions of the CIPU 800 will be explained with reference to the modules of the CIPU 400.

As shown, CIPU 800 includes a processing pipeline 815 for camera sensor 405a and pixel array 410a, and a processing pipeline 820 for camera sensor 405b and pixel array 410b. Each of the processing pipelines 815 and 820 of some embodiments includes all of the modules contained in the CIPU 400. Thus, the operation of each of the processing pipelines 815 and 820 of these embodiments is identical to the operation of the CIPU 400.

Since each camera sensor 405a and 405b has its own processing pipeline, processing pipelines 815 and 820 are able to process images independently of each other. For example, processing pipelines 815 and 820 can process images in parallel (ie, at the same time), at different times, and at different rates. Additionally, each of the processing pipelines 815 and 820 of some embodiments can be configured differently through the different register values described above with reference to the CIPU 400.

In some embodiments, many of the modules of CIPU 400 include one or more line/frame buffers for performing some or all of the operations of that module. For example, the filtering module 460 of some embodiments is implemented to perform 3x3 low-pass filtering. In such an embodiment, the 3x3 low pass filter processes three consecutive lines in the image to apply the 3x3 low pass filter to the middle row of the three consecutive lines. Thus, the filtering module 460 of this embodiment requires at least three line/frame buffers in order to implement 3x3 low-pass filtering. Other modules in CIPU 400 also include one or more line/frame buffers, such as BPC module 420 and LS module 425.

The CIPU 800's processing pipeline can all have different line/frame buffer sizes to tailor image processing to the characteristics of its corresponding camera. For example, if one camera of a dual camera mobile device has a 2048x1500 pixel sensor, the processing pipeline for the 2048x1500 pixel sensor can include a 2048 pixel wide line/frame buffer. Similarly, if the other camera of a dual camera mobile device has a 640x480 pixel sensor, the processing pipeline for the 640x480 pixel sensor may include a line/frame buffer that is 640 pixels wide. That is, the size of a line/frame buffer included in each module of one processing pipeline may be different from the size of a line/frame buffer included in each module of another processing pipeline.

III. Video conferencing

A. Video conferencing architecture

Figure 9 conceptually illustrates the software architecture of a video conferencing and processing module 900 of a dual camera mobile device of some embodiments. The video conferencing and processing module 900 includes a CIPU driver 905 , a media switching module 910 and an encoder driver 920 similar to the corresponding modules and drivers 305 , 301 and 320 described above with reference to FIG. 3 . The video conferencing and processing module 900 also includes a video conferencing module 925 , a video conferencing client 945 and a network interface 950 for realizing various video conferencing functions. Similar to the video processing and encoding module 300, the video conferencing and processing module 900 processes and encodes images captured from the cameras of the dual camera mobile device.

As described above with reference to FIG. 3 , the media exchange module 910 allows users and producers of media content in the device to exchange media content and instructions related to the processing of the media content. CIPU driver 905 serves as a communication interface with captured image processing unit (CIPU) 955 and encoder driver 920 serves as a communication interface with encoder hardware 960 (eg, an encoder chip, an encoding component on a system-on-chip, etc.).

The videoconferencing module 925 of some embodiments is responsible for various videoconferencing functions, such as image processing, videoconferencing management, and networking. As shown, video conferencing module 925 interacts with media exchange module 910 , video conferencing client 945 and network interface 950 . In some embodiments, videoconferencing module 925 receives instructions from, and sends instructions to, videoconferencing client 945 . The video conferencing module 925 of some embodiments also sends data to and receives data from the network through the network interface 950, for example, the network is a local area network (LAN), a wireless local area network (WLAN), a wide area network (WAN), a network of networks ( a network of networks), code division multiple access (CDMA) network, GSM network, etc.

The video conferencing module 925 includes an image processing layer 930 , a management layer 935 and a network layer 940 . In some embodiments, image processing layer 930 performs image processing operations on images for use in video conferencing. For example, the image processing layer 930 of some embodiments performs exposure adjustment, image scaling, perspective correction, and dynamic range adjustment, as described in further detail below. The image processing layer 930 of some embodiments sends requests for images from the CIPU 955 through the media exchange module 910 .

The management layer 935 of some embodiments controls the operation of the video conferencing module 925 . For example, in some embodiments, the management layer 935 initializes one/both cameras of a dual camera mobile device, processes images and audio for transmission to a remote device, and processes images and audio received from a remote device. In some embodiments, management layer 935 generates a composite (eg, PIP) display for the device. Additionally, the management layer 935 may alter the operation of the video conferencing module 925 based on the networking reports received from the network layer 940 .

In some embodiments, the network layer 940 implements some or all of the network functions for video conferencing. For example, as described below, the network layer 940 of some embodiments establishes a network connection (not shown) between a videoconferencing dual-camera mobile device and a remote device, transmits images to the remote device, and Receive an image from a remote device. Additionally, the network layer 940 receives networking data such as packet loss, one-way latency, and round-trip delay time, among other miscellaneous data, processes such data, and reports the data to the management layer 935 .

The video conferencing client 945 of some embodiments is an application that utilizes the video conferencing functionality of the video conferencing module 925, such as a video conferencing application, a voice over IP (VOIP) application (eg, Skype), or an instant messaging application. In some embodiments, video conferencing client 945 is a stand-alone application, while in other embodiments video conferencing client 945 is integrated into another application.

In some embodiments, network interface 950 is a communication interface that allows video conferencing module 925 and video conferencing client 945 to send and receive data over a network (e.g., a cellular network, a local area network, a wireless network, a network of networks, the Internet, etc.). For example, if videoconferencing module 925 wants to send data (e.g., an image captured by a camera of a dual-camera mobile device) to another device on the Internet, then videoconferencing module 925 sends the image to the other device via network interface 950. equipment.

B. Video conferencing setup

Figure 10 conceptually illustrates an example video conference request messaging sequence 1000 of some embodiments. FIG. 10 shows a video conference request messaging sequence 1000 between video conference client 1010 running on device 1005 , video conference server 1015 , and video conference client 1025 running on device 1020 . In some embodiments, video conferencing clients 1010 and 1025 are the same as video conferencing client 945 shown in FIG. 9 . As shown in FIG. 10, one device (ie, device 1005) requests a video conference, and another device (ie, device 1020) responds to the request. The dual-camera mobile device described in this application is capable of performing both operations (ie, making a request and responding to a request).

The video conferencing server 1015 of some embodiments routes messages between video conferencing clients. While some embodiments implement video conferencing server 1015 on one computing device, other embodiments implement video conferencing server 1015 on multiple computing devices. In some embodiments, the video conferencing server is a publicly accessible server capable of processing and routing messages for numerous conferences simultaneously. Each of videoconferencing clients 1010 and 1025 of some embodiments communicates with videoconferencing server 1015 via a network interface, such as network interface 950 described above, over a network (e.g., a cellular network, a local area network, a wireless network, a network of networks, the Internet, etc.) communication.

Video conference request messaging sequence 1000 of some embodiments begins when video conference client 1010 receives (at operation 1 ) a request from a user of device 1005 to start a video conference with device 1020 . When a user of device 1005 selects a certain user interface (UI) item of a user interface displayed on device 1005, video conferencing client 1010 of some embodiments receives a request to start a video conference. Examples of such user interfaces are illustrated in Figures 11 and 14 described below.

After videoconferencing client 1010 receives the request, videoconferencing client 1010 sends (at operation 2) a videoconferencing request to videoconferencing server 1015, the request indicating device 1020 as the recipient based on user input. The video conference server 1015 forwards (at operation 3 ) the video conference client 1025 of the device 1020 to the video conference request. In some embodiments, the video conference server 1015 forwards the video conference request to the video conference client 1025 using push technology. That is, when a request is received from videoconferencing client 1010, videoconferencing server 1015 begins transmitting the videoconferencing request to videoconferencing client 1025, rather than waiting for client 1025 to send a request for any messages.

When the video conference client 1025 of some embodiments receives a video conference request, a user interface is displayed on the device 1020 to indicate to the user of the device 1020 that the user of the device 1005 has sent a request to start the video conference, and to prompt the user of the device 1020 The user accepts or rejects the video conference request. An example of such a user interface is illustrated in Figure 12 described below. In some embodiments, when the video conferencing client 1025 receives (at operation 4) a request to accept a video conference request from the user of the device 1005, the video conferencing client 1025 sends (at operation 5) a video conference request to the video conferencing server 1015. Meeting accepted. When a user of device 1020 selects a certain user interface item of the user interface as illustrated in FIG. 12 , video conferencing client 1025 of some embodiments receives a request to accept a video request.

After video conference server 1015 receives the video conference acceptance from video conference client 1025 , video conference server 1015 forwards (at operation 6 ) the video conference acceptance to video conference client 1010 . Some embodiments of the video conference server 1015 forward the video conference acceptance to the video conference client 1010 using the push technology described above.

When a video conference acceptance is received, some embodiments establish (at operation 7 ) a video conference between device 1005 and device 1020 . Different embodiments set up the video conference differently. For example, video conference setup of some embodiments includes negotiating a connection between device 1005 and device 1020 , determining a bit rate to encode the video, and exchanging video at device 1005 and device 1020 .

In the above example, the user of device 1020 accepts a video conference request. In some embodiments, device 1020 may be configured (eg, via device preferences) to automatically accept incoming video conference requests without displaying a UI. Additionally, the user of device 1020 can also decline (at operation 4) the video conference request (eg, by selecting a certain user interface item of the user interface displayed on device 1020). Instead of sending a video conference acceptance, video conference client 1025 sends a video conference rejection to video conference server 1015 , which forwards the video conference rejection to video conference client 1010 . Thereby no video conference is established at all.

1. Video conference setting user interface

In some embodiments, a video conference is initiated based on an ongoing call. That is, when a user of a mobile device is on a call with a second user, the user can turn the call into a video conference with the permission of the other party. Figure 11 illustrates the initiation of such a video conference by a dual camera handheld mobile device 1100 for some embodiments of the invention. FIG. 11 illustrates the initiation of a video conference using five operational stages 1110 , 1115 , 1120 , 1125 , and 1130 of a user interface (“UI”) 1105 of device 1100 .

As shown in FIG. 11 , UI 1105 includes a name field 1135, a selection menu 1140, and selectable UI items 1145. The name column 1135 displays the name of the person with whom the user at the other end of the call wishes to request a video conference. In this example, selectable UI item 1145 (which can be implemented as a selectable button) provides the user with a selectable End Call option to end the call. Selection menu 1140 displays a menu of selectable UI items, such as speakerphone item 1142, mute item 1144, numeric keypad item 1146, phonebook item 1148, hold item 1152, video conferencing item 1154, and the like. Different embodiments display the selection menu differently. For the embodiment illustrated in FIG. 11, the selection menu 1140 includes several icons of the same size, each icon representing a different operation. Other embodiments provide scrollable menus, or give priority to certain items (eg, by making them larger).

Operation of the UI 1105 will now be described with reference to the states of the UI 1105 in five stages 1110, 1115, 1120, 1125, and 1130 illustrated in FIG. 11 . In a first stage 1110, a call has been established between the handheld mobile device user and Nancy Jones. The second stage 1115 displays the UI 1105 after the user selects a selectable video conferencing option 1154 (e.g., with a single-finger tap of finger 1150) to activate the video conferencing tool. In this example, video conference option 1154 (which can be implemented as a selectable icon) allows the user to initiate a video conference during a call. In the second stage, the video conferencing option 1150 is highlighted to indicate that the video conferencing tool has been activated. Different embodiments may indicate such selection in different ways (eg, by highlighting a border or text of an item).

The third stage 1120 displays the UI 1105 after the device 1100 has initiated the video conferencing process upon selection of the select video conferencing option 1154. The third phase is the transition hold phase while the device is waiting to establish a video conference (eg, while the device is waiting for the device on the other end of the call to accept or decline the video conference). In a third stage 1120, while the video conference connection is being established, the user of device 1100 is still able to talk to the user of another device (ie, Nancy Jones). Additionally, some embodiments allow the user of device 1100 to cancel the video conference request in the third stage 1120 by selecting an optional UI item (not shown) displayed on UI 1105 for canceling the video conference request. During this hold phase, different embodiments indicate this wait state using a different display in UI 1105.

As shown in FIG. 11 , in some embodiments, the third stage wait state is illustrated with a full screen display of the video captured by device 1100 , and a "Preview" symbol at the bottom of the video. Specifically, in FIG. 11 , the third stage 1120 illustrates the start of the video conferencing process by displaying in the display area 1160 of the UI 1105 a full-screen representation of the video captured by the device's camera. In some embodiments, the front camera is the default camera selected by the device when starting a video conference. Typically, the front camera is pointed at the user of the device when the video conference is started. Thus, in the example illustrated in FIG. 11 , the third stage 1120 graphically represents the device 1100 as presenting a full screen video of the user of the device 1100 . A "Preview" indication located below the video that appears in the display area 1160 during the third stage 1120 further highlights the waiting state of the device.

In some embodiments, the transitional third hold phase 1120 can be represented differently. For example, some embodiments allow a user of device 1100 to select the camera on the back as the camera to initiate a video conference. To allow for this choice, some embodiments allow the user (e.g., via a menu preference setting) to designate the back camera as the default camera to Select Rear Camera from the menu and Front Camera. In any of these cases, the UI 1105 (e.g., the display area 1160) displays the video captured by the camera on the back during the third hold phase 1120.

Additionally, other embodiments may display a smaller version of the video captured by device 1100, by displaying a still image stored on device 1100, by providing a message that highlights the device's waiting state (e.g., by displaying a "Conference Being Established" ( The meeting is being established), by not displaying the "Preview" indication, etc., can indicate the activation of the video conferencing tool. Additionally, in the third stage 1120, the UI 1105 of some embodiments provides an indication if at this stage (e.g., while the user is waiting When the remote user answers his request), the user decides not to enter the video conference, so the user is allowed to cancel entering the video conference, and return to the End (End) button (not shown) of the call state.

The fourth stage 1125 illustrates the UI 1105 in a transitional state after the remote user has accepted the video conference request and established a video conference connection. In this transition state, the display area 1160 displaying the local user's video (in this example, the video captured by the front camera) gradually decreases in size (ie, tapers out), as indicated by arrow 1175 . Display area 1160 (i.e., the local user's video) is zoomed out, enabling UI 1105 to display, behind display area 1160, display area 1170 (e.g., display window 1170) containing video from the remote device's camera. In other words, zooming out of the local user's video 1160 produces a PIP display 1180 with a foreground inset display 1160 of the local user's video, and a background main display 1170 of the remote user. In this example, the background main display 1170 presents a woman whose video is being captured by the remote device's front camera (e.g., Nancy Jones, the user of the remote device), or a woman whose video is being captured by the remote device's back camera (e.g., Nancy Jones video of the woman whose video is being filmed). One of ordinary skill will recognize that the transitional fourth stage shown in FIG. 11 is only one example approach used by some embodiments, and that other embodiments may animate the transitional fourth stage differently.

The fourth stage 1125 also illustrates an optional UI item 1132 in the lower display area 1155 . An optional UI item 1132 (which can be implemented as a selectable button) provides a selectable End Conference option 1132 below the PIP display 1180. The user may select the end meeting option 1132 to end the video conference (eg, with a one-finger tap). Different embodiments may allow the user to end the meeting in different ways, such as by flipping a switch on the mobile device, by issuing a voice command, and so on. Additionally, various embodiments may allow the end meeting option 1132 to fade away during the video conference, thereby allowing the PIP display 1180 to occupy the entire display area 1185 . On subsequent one-finger taps on the bottom of display area 1185 , end meeting option 1132 may reappear, making end meeting option 1132 available to the user. In some embodiments, the layout of the display area 1155 is the same as that of the display area 1155 described in further detail below.

The fifth stage 1130 illustrates the UI 1105 after ending the animation of the fourth transition state 1125. Specifically, the fifth stage 1130 illustrates the PIP display 1180 presented by the UI 1105 during the video conference. As noted above, the PIP display 1180 includes two video displays: a larger background display 1170 from the remote camera, and a smaller foreground inset display 1160 from the local camera.

The PIP display 1180 is just one way of presenting a composite view of the video captured by the remote device and the local device. In addition to this composite view, devices of some embodiments provide other composite views. For example, instead of having a larger background display 1170 of a remote user, the larger background display 1170 could be of the local user, while the smaller foreground inset display 1160 is of the remote user. As described further below, some embodiments allow a user to switch between a local camera and/or a remote camera during a video conference as the camera providing the inset and main views of the PIP display 1180 .

Additionally, some embodiments allow local video and remote video to appear in UI 1105 in two side-by-side display areas (e.g., display windows left and right, or display windows above and below), or in two diagonally arranged display areas. In some embodiments, as described further below, the user may specify the manner in which the PIP display or default display mode is displayed, either through a device preference setting, or through a user-selectable control during a video conference.

When a user of device 1100 of FIG. 11 invites a remote user to a video conference, the remote user can accept or decline the invitation. 12 illustrates the UI 1205 of the remote user's device 1200 in six different stages 1210, 1215, 1220, 1225, 1230, and 1235 representing the operations of presenting and accepting a video conference invitation at the remote user's device sequence. The following description of UI 1205 refers to the user of device 1200 (i.e., the device that received the video conference request) as the invitee recipient, and the user of device 1100 (i.e., the device that sent the video conference request) as the inviter The person (invite requestor). Also, in this example, it is assumed that the invitee's device 1200 is a dual camera device, as is the inviter's device. However, in other examples, one or both of these devices are single camera devices.

The first stage 1210 illustrates the UI 1205 when the invitee receives a video conference invitation from the inviter, John Smith. As shown in FIG. 12, the UI 1205 of the first stage includes a name field 1235, a message field 1240, and two optional UI items 1245 and 1250. Name column 1235 displays the name of the person requesting the video conference. In some embodiments, the name column 1235 displays the phone number of the person requesting the video conference instead of the person's name. The message column 1240 displays the invitation from the inviter to the invitees. In this example, "Video Conference Invitation" in message column 1240 indicates that the inviter is requesting a video conference with the invitees. Optional UI items 1245 and 1250 (which may be implemented as selectable buttons) provide selectable "Deny Request (Deny Request)" and "Accept Request (Accept Request)" for invitees to reject or accept the invitation. "Options 1245 and 1250. Different embodiments may display these options differently and/or display other options.

Upon seeing the "Video Conference Invitation" symbol displayed in the message bar 1240, the invitee can decline or accept the request by selecting the "Deny Request" option 1245 or the "Accept Request" option 1250, respectively, in the UI. The second stage 1215 illustrates that in the example shown in FIG. 12 , the user selects the "Accept Request" option 1250 . In this example, the selection is made by lightly tapping the "Accept Request" option 1250 with the user's finger, and indicated by the highlighting of the option 1250. Other techniques are provided in some embodiments to select the "Accept" or "Deny Request" options 1245 and 1250 (e.g., two consecutive taps, etc.) to indicate the selection (e.g., highlighting the border of a UI item or text).

The third stage 1220 shows the UI 1205 after the invitee agrees to join the video conference. At this stage, the UI 1205 enters a preview mode, which displays a full-screen rendering of the video from the remote device's front camera in the display area 1244. The front-facing camera in this case is directed at the user of the remote device (ie, Nancy Jones in this example). Therefore, her image is displayed in the preview mode. This preview mode allows the invitee to ensure that her video is displayed correctly and that she is happy with how it looks before the video conference starts (for example, before the actual video transmission begins). In some embodiments, a symbol such as a "Preview" symbol may be displayed below the display area 1244 to indicate that the invitee is in preview mode.

Some embodiments allow invitees to select the back camera as the default camera for starting a video conference, or to select either the front or back camera when starting a video conference, as described further below. Additionally, other embodiments display a preview display of invitees differently (eg, in a smaller image placed in a corner of display area 1244). Other embodiments besides this do not include this preview mode, but start the video conference immediately after the invitee accepts the request.

In the third stage, UI 1205 displays two selectable UI items 1275 and 1246. One of them is overlaid on the display area 1244 and the other is below the display area 1244 . An optional UI item 1275 is an "Accept" button 1275 that the user can select to start the video conference. An optional UI item 1246 is an "End" button 1246 that the invitee can select if she decides not to join the video conference at this stage.

The fourth stage 1225 displays the UI 1205 after the invitee selects the "Accept" button 1275 . In this example, the "Accept" button 1275 is highlighted to indicate that the invitee is ready to start the video conference. Such selection may be indicated in different ways in other embodiments.

The fifth stage 1230 illustrates the UI 1205 in a transitional state after the invitee accepts the video conference request. During this transition phase, the display area 1244 showing the invitee's video (in this example, captured by the front camera) gradually decreases in size (ie, tapers out), as indicated by arrow 1260 . The invitee's video is zoomed out, enabling UI 1205 to display, behind display area 1244, display area 1265 (e.g., display window 1265) containing video from the inviter's camera. In other words, zooming out of the invitee's video produces a PIP display 1280 with a foreground inset display area 1244 of the invitee's video, and a main display 1265 of the inviter's background.

In this example, the background main display 1265 presents a video of the man (ie, John Smith, user of the local device 1100) whose video is being captured by the local device's front-facing camera. In another example, the video may be of a man whose video was captured by the local device's back camera (eg, a man whose video was captured by John Smith). Different embodiments may animate the transitional fifth stage differently.

The UI display of the fifth stage 1230 also includes an optional UI item 1285 (e.g., a mute button 1285) to mute another user's audio during the video conference, an optional UI item 1287 (e.g., an end conference button) to end the video conference 1287), and the display area 1155 (eg, tool bar or menu bar) of an optional UI item 1289 (eg, switch camera button 1289) for toggle camera, described further below. Thus, invitees can select any of the selectable UI items 1285-1289 (eg, by single-finger taps) to perform desired actions during the video conference. Different embodiments allow invitees to perform arbitrary actions in different ways, such as toggling a switch on the mobile device, by giving voice commands, and the like.

Although FIG. 12 shows an example layout of the display area 1155, some embodiments provide a different layout of the display area 1155, such as the layout of the display area 1155 of FIG. Option 1132. Other layouts of the display area 1155 may include any of a variety of different selectable UI items for performing different functions. Additionally, fifth stage 1230 represents display area 1155 displayed at the bottom of UI 1205. Different embodiments of display area 1155 may be displayed in different locations within UI 1205, and/or defined as different shapes.

FIG. 12 shows display area 1155 as a static display area (ie, display area 1155 is always displayed). However, in some embodiments, display area 1155 is a dynamic display area. In some of these embodiments, display area 1155 is not normally displayed. Instead, the display area 1155 is displayed only when a triggering event (eg, user selection such as tapping the display area 1280 once, a voice command, etc.) is received. The display area 1155 disappears after receiving a user selection (e.g., selecting the optional mute UI item 985), or a specified amount of time (e.g., 3 seconds), which can be determined by the user via a mobile device or video conferencing. The preference settings specified for the application. In some of these embodiments, the display area 1155 is automatically displayed after the video conference starts, and disappears in the same manner as mentioned above.

The sixth stage 1235 illustrates the UI 1205 after ending the animation of the fifth transition stage. Specifically, the sixth stage illustrates a PIP display 1280 presented by UI 1205 during a video conference. As noted above, the PIP display 1280 includes two video displays: a larger background display 1265 from the local camera, and a smaller foreground inset display 1244 from the remote camera. PIP Display 1280 is just one way of presenting a composite view of the video captured by the remote device and the local device. In addition to this composite view, devices of some embodiments provide other composite views. For example, instead of a background display with a larger invitee, the larger background display could be the inviter's video and the smaller foreground inset display could be the invitee's video. As described further below, some embodiments allow the user to control the inset and main views in the PIP display to switchably display local and remote cameras. In addition, some embodiments allow local video and remote video to appear in two side-by-side display areas in UI 1205 (e.g., display windows left and right, or display windows above and below), or in two display areas arranged diagonally. As described further below, the user may specify the manner in which the PIP is displayed, or the default display mode, through a device preference setting, or through a control that the user can select during a video conference.

Although FIG. 12 represents the sequence of operations for presenting and accepting a video conference invitation in terms of six distinct operational phases, some embodiments may implement the operations in fewer phases. For example, some such embodiments may omit presenting the third stage 1220 and the fourth stage 1225 , thereby proceeding from the second stage 1215 to the fifth stage 1230 after the user selects the "Accept Request" option 1250 . Other embodiments that implement the described operations (i.e., presenting and accepting a videoconference invitation) in fewer stages may omit the first stage 1210 and the second stage 1215, so that when the invitee receives the invitation to the videoconference from the inviter , presenting the third stage 1220 to the user.

FIG. 13 illustrates an example of performing the operation illustrated in FIG. 12 with fewer stages by combining the first and third stages into one stage, and combining the second and fourth stages into one stage. In particular, FIG. 13 illustrates the UI 1205 of the remote user's device 1200 at five different stages 1390, 1392, 1394, 1230, and 1235. The first stage 1390 is similar to stage 1110, except that the name column 1295 displays the name "John Smith" to indicate the name of the person on the other end of the call. That is, a call has been established between the user of the remote mobile device and the user of the local device (ie, John Smith in this example). The second stage 1392 and the third stage 1394 are similar to the first stage 1210 and the second stage 1215 of FIG. ) outside of the preview. The fourth stage 1230 and the fifth stage 1235 are the same as the fifth stage 1230 and the sixth stage 1235 of FIG. 12 .

In addition to activating the video conferencing tool via an optional option during a call, some embodiments allow users of dual camera devices to directly initiate a video conference without having to make a phone call first. Figure 14 illustrates another such alternative method of initiating a video conference. 14 illustrates the UI 1405 at seven different stages 1410, 1415, 1420, 1425, 1430, 1435, and 1440 showing an alternative sequence of operations for initiating a video conference.

In a first stage 1410, the user browses the list of contacts on the mobile device, looking for people with whom he wishes to videoconference, similar to how he looks up contacts for calling. In a second stage 1415, the user selects a person 1455 with whom he would like to videoconference (eg, by a one-finger tap 1460 on that person's name 1455). This selection triggers UI 1405 to display the contact's information and various user selectable options. In this example, Jason's name 1455 is highlighted to indicate that this is the person the user intends to video conference with. Different embodiments may indicate this selection in different ways. While the second stage 1415 allows the user of the device 1400 to select the person with whom the user would like to videoconference via the contacts list, some embodiments allow the user to select the person via the "Recents" call history, which The call history lists specific numbers or names of persons with whom the user of device 1400 has recently videoconferenced or called.

In a third stage 1420, after selecting a character's name 1455, the UI 1405 displays the selected character's information 1462, and various selectable UI items 1468, 1472, and 1470. In this example, one of the various selectable UI items 1472 (which may be implemented as selectable icons or buttons) provides video conferencing tools. Video conferencing option 1472 allows the user to invite persons identified from contacts 1466 to join a video conference. Different embodiments display information 1462 and selectable UI items 1468, 1472, and 1470 differently (eg, in different arrangements).

The fourth stage 1425 represents the user selecting the "Video Conference" option 1472 (eg, with a single finger tap). In this example, the "Video Conferencing" option 1472 is highlighted to indicate that the video conferencing tool 1472 is activated. Such selection may be indicated differently in different embodiments (eg, by highlighting the text or border of the selected icon).

The fifth, sixth and seventh stages 1430, 1435 and 1440 are similar to the third, fourth and fifth stages 1120, 1125 and 1130 illustrated in FIG. Sixth and seventh stages 1430, 1435 and 1440. Briefly, the fifth stage 1430 illustrates an interim hold stage waiting for the remote user to respond to the video conference invitation. The sixth stage 1435 illustrates that after the remote user accepts the video conferencing request, the size of the display area 1480 (showing the local user's video) gradually decreases, enabling the UI 1405 to display a video containing the camera from the remote user behind the display area 1480. Display area 1492 for video. In a seventh stage 1440, the UI 1405 presents a PIP display 1447 during the video conference. In some embodiments, the layout of the display area 1155 in the sixth stage 1435 and the seventh stage 1440 is similar to the layout of the display area 1155 of FIG. 12 described above.

Figures 10, 11, 12, 13 and 14 show several ways of establishing a video conference. In some embodiments, audio data (e.g., voice) is transmitted over one communication channel (over a communication network such as a circuit-switched communication network or a packet-switched communication network) during a call and over another communication channel during a video conference. The communication channel carries audio data. Thus, in such embodiments, audio data (e.g., voice) is communicated over one communication channel prior to establishing the video conference, and once the video conference is established, over a different communication channel (rather than the communication used during the call). channel) to transmit audio.

To provide a seamless transition (eg, handover) of audio data from a call to a video conference, some embodiments do not terminate the call until the video conference is established. For example, some embodiments establish a peer-to-peer video conference connection (eg, after completing the sequence of messages illustrated in FIG. 10 ) before terminating the call and beginning to transmit audio/video data over the peer-to-peer communication session. On the other hand, other embodiments establish a peer-to-peer video conferencing connection (e.g., after completing the message sequence illustrated in FIG. Transfer audio/video data.

The peer-to-peer video conference connection of some embodiments allows mobile devices in a video conference to communicate directly with each other (rather than through, for example, a central server). Some embodiments of peer-to-peer video conferencing allow mobile devices in a video conference to share resources with each other. For example, by sending instructions from one mobile device to another mobile device via a control communication channel of a video conference to instruct the other mobile device to process images differently (i.e., share its image processing resources), such as described in further detail below The exposure adjustment operation, the focus adjustment operation and/or the switching camera operation, the one mobile device can remotely control the operation of the other mobile device in the video conference.

2. Dynamic bit rate setting

In general, mobile devices in a video conference communicate data (e.g., audio) to each other over different types of communication networks, such as communication channels of different private and public wireless communication networks (e.g., cellular networks such as GSM, UMTS). and video images). An example of such a wireless communication network will be described below with reference to FIGS. 91 and 92 .

Due to the ever-changing number of mobile devices accessing the communication network at a particular time, the bandwidth available to the communication network for video conferencing varies from time to time. Even during a video conference, available bandwidth can change. Additionally, flooding the communication network with high bit rates during video conferencing, or extensive signaling in an attempt to figure out the optimal video conferencing bit rate, is not advisable.

For these reasons, some embodiments employ a new method of specifying an initial optimal bit rate for a video conference. In order to identify the initial optimal bit rate for the video conference, the method starts the video conference at a specific bit rate, incrementally at specific time intervals if the embodiments do not detect network conditions that would degrade the quality of the video conference. high bitrate.

An example of such an embodiment is illustrated in FIG. 15 . Figure 15 conceptually illustrates the process 1500 of setting the bit rate for a video conference for some embodiments of the invention. Process 1500 is performed as part of a video conference setup (eg, as part of a video conference setup illustrated in FIG. 10 ) to dynamically determine the bit rate for transferring data (eg, audio and video images) based on various network conditions. In some embodiments, process 1500 is performed by management layer 935 of videoconferencing module 925 described above with reference to FIG. 9 . A more detailed form of the video conferencing module is described below with reference to FIG. 16 .

As shown in Figure 15, process 1500 begins by setting the bit rate to (at 1505) the initial bit rate. In some embodiments, the initial bit rate is the device's default base rate. However, some embodiments allow the user to specify an initial bit rate. At 1505, process 1500 also initiates a video conference by transmitting data (eg, audio and video images) to a remote device at an initial bit rate over one or more communication channels.

Subsequently, process 1500 identifies (at 1510) a set of network condition parameters received from a remote device in the video conference. In some embodiments, the local device receives the set of network condition parameters from the remote device via a real-time transport protocol (RTP) communication session established when starting the video conference. For example, some embodiments provide network condition parameters through extended features of RTP. Furthermore, the RTP extension feature of some embodiments can be used to convey any type of information (e.g., the set of network condition parameters) by noting the presence of extension headers in the RTP packet header, and defining extension headers regarding additional information .

In various embodiments, each device in the video conference transmits different sets of network condition/congestion parameters. In the embodiment described below, the set of network condition parameters includes one-way latency and bandwidth estimation bit rate. In other embodiments, the set of network condition parameters includes packet loss data and round trip time (RTT) delay data. Thus, different embodiments may include any number of different network condition parameters in the set of network condition parameters.

In some embodiments, the set of network condition parameters received from the remote device of the video conference is transmitted from the local mobile device (i.e., the mobile device executing process 1500) during the video conference at the initial bit rate set at operation 1505. based on the data (eg, audio and video) transmitted to the remote device. For example, in some embodiments, the remote device can determine the one-way latency by calculating the time it takes for the audio packet to propagate from the local mobile device to the remote device through the network connection using the timestamp of the audio packet. Specifically, in some embodiments, each audio packet is time stamped. In the absence of packet delay, the remote device should receive audio packets at set intervals equal to the difference in time stamps. However, when there is a one-way latency delay, the remote device receives audio packets at intervals greater than the time stamp difference.

Additionally, in some embodiments, the remote device determines the bandwidth estimate bit rate by examining the time a video packet was received, the time a neighboring video packet was received, and the size of the neighboring video packet. That is, the time difference between receiving two consecutive video packets and the size of the second video packet is used to estimate the available bandwidth of the network connection. Some embodiments determine the bandwidth estimate bit rate by examining pairs of consecutive video packets. The above examples utilize specific types of data (ie, audio data for determining one-way latency, and video data for determining bandwidth estimate bitrate). However, in some embodiments, other types of data transferred over the network connection between the local mobile device and the remote device may also be used.

After identifying (at 1510) the set of network conditions, process 1500 then determines (at 1515) whether the one-way latency has deteriorated beyond a defined threshold amount. In some embodiments, the threshold amount is defined as a specific amount of latency, and the one-way latency is determined if the difference between the current one-way latency and the previous one-way latency exceeds the specified amount of latency Deteriorated beyond a threshold amount. In other embodiments, the threshold amount is defined as a certain rate of change in one-way latency. Thus, when the rate of change of a set of one-way latencies (eg, current one-way latencies and previous one-way latencies) exceeds a certain rate of change, then it is determined that the one-way latencies have deteriorated beyond a threshold amount.

Process 1500 ends when it is determined that the one-way latency has deteriorated beyond a threshold amount. Otherwise, process 1500 determines (at 1520) whether the current bit rate has reached the bandwidth estimated bit rate. In some embodiments, the bandwidth estimate bit rate indicates the amount of bandwidth available for the network connection (eg, 15 kb/s (kbps)). When process 1500 determines that the current bit rate exceeds the bandwidth estimated bit rate, process 1500 ends. When process 1500 determines that the current bit rate does not exceed the bandwidth estimated bit rate, process 1500 proceeds to operation 1525 .

At 1525, process 1500 determines whether the current bit rate reaches the defined maximum bit rate. When process 1500 determines that the current bit rate exceeds the defined maximum bit rate, process 1500 ends. Otherwise, process 1500 proceeds to operation 1530, where the current bit rate is increased by the specified amount. Different embodiments define the amount to increase the bit rate differently. Examples of specified amounts to increase the current bitrate include 32kbps, 64kpbs, and any number of other amounts to increase the bitrate.

Processing then determines (at 1535) whether a specified amount of time has elapsed. The prescribed amount of time may be 1 second, 2 seconds, 5 seconds, or any other possible amount of time, as different embodiments define the amount of time differently. Process 1500 waits for a specified amount of time to elapse so that the remote device can receive data (e.g., audio and video images) transmitted from the local mobile device at the newly increased bit rate (at operation 1530), and The bit rate determines the network condition parameter. If process 1500 determines that the prescribed amount of time has not elapsed, then process 1500 returns to operation 1535 until the prescribed amount of time has elapsed. When process 1500 determines that the prescribed amount of time has elapsed, process 1500 returns to operation 1510 . Operations of process 1500 beginning at 1510 continue as described above until process 1500 ends.

When process 1500 ends (ie, after operations 1515, 1520, or 1525), setting the bit rate for the video conference is complete, and an optimal bit rate has been determined. Since the available bandwidth for a video conference may vary during the video conference, some embodiments continue to adjust the bit rate based on a set of network condition parameters (ie, one-way latency and bandwidth estimate bit rate) received from the remote device. You can adjust the bit rate during a video conference by increasing the bit rate. For example, if process 1500 ends with one-way latency deteriorating beyond a specified threshold amount, and during a video conference, one-way latency improves, some embodiments increase the bit rate. Similarly, if process 1500 ends because the bit rate exceeds the bandwidth estimated bit rate, and during the video conference, the bandwidth estimated bit rate increases, some embodiments increase the bit rate.

Instead, adjust the bit rate during a video conference by reducing the bit rate. For example, if during a video conference, one-way latency continues to deteriorate beyond a specified threshold amount, some embodiments reduce the bit rate. Additionally, some embodiments reduce the bit rate if the bit rate continues to exceed the bandwidth estimated bit rate (eg, the bandwidth estimated bit rate continues to decrease) during the video conference.

Additionally, process 1500 uses one-way latency and bandwidth to estimate the bit rate to determine whether to increase the bit rate. However, one of ordinary skill will recognize that in different embodiments, any number of network condition parameters may be used to determine whether to increase the bit rate. For example, determining whether to increase the bit rate may be based solely on RTT delay data or packet loss data.

C. Video Conferencing Architecture

As noted above, Figure 16 conceptually illustrates the software architecture of the video conferencing and processing module 1600 of a dual camera mobile device of some embodiments. As shown, the video conferencing and processing module 1600 includes a client application 1665, a video conferencing module 1602, a media exchange module 1620, a buffer 1625, a captured image processing unit (CIPU) driver 1630, an encoder driver 1635, and a decoder driver 1640 . In some embodiments, the buffer 1625 is a frame buffer that holds images of the video for display on the display 1645 of the dual camera mobile device.

In some embodiments, client application 1665 is the same as video conferencing client 945 of FIG. 9 . As noted above, client application 1665 may be integrated into another application, or implemented as a stand-alone application. Client application 1665 may be an application that utilizes the videoconferencing functionality of videoconferencing module 1602, such as a videoconferencing application, a voice over IP (VOIP) application (eg, Skype), or an instant messaging application.

The client application 1665 of some embodiments sends instructions to the video conferencing module 1602, such as instructions to start a meeting and end a meeting, receives instructions from the video conferencing module 1602, routes instructions from the user of the dual camera mobile device to the video conferencing module 1602, and generate a user interface that is displayed on a dual camera mobile device and allows the user to interact with the application.

D. Video Conference Manager

As shown in FIG. 16 , videoconferencing module 1602 includes videoconferencing manager 1604 , image processing manager 1608 , networking manager 1614 , and buffers 1606 , 1610 , 1612 , 1616 , and 1618 . In some embodiments, video conferencing module 1602 is the same as video conferencing module 925 illustrated in FIG. 9 , performing some or all of the same functions described above with respect to video conferencing module 925 .

In some embodiments, video conference manager 1604 is responsible for initializing some or all of the other modules of video conference module 1602 (e.g., image processing manager 1608 and networking manager 1614) when the video conference begins, controlling The operation of the video conferencing module 1602, and when the video conference ends, the operation of some or all other modules of the video conferencing module 1602 ceases.

The video conference manager 1604 of some embodiments also processes images received from one or more devices in the video conference, and images captured by one or both cameras of the dual camera mobile device, for display on the dual camera mobile device . For example, video conference manager 1604 of some embodiments retrieves from buffer 1618 decoded images received from another device participating in the video conference, and from buffer 1606 images processed by CIPU 1650 (i.e., images moved by the dual camera images captured by the device). In some embodiments, the video conferencing manager 1604 also scales and composites the images before displaying them on the dual camera mobile device. That is, in some embodiments, video conference manager 1604 generates a PIP or other composite view for display on the mobile device. Some embodiments scale images retrieved from buffers 1606 and 1618 , while other embodiments only scale images retrieved from one of buffers 1606 and 1618 .

Although FIG. 16 diagrammatically shows videoconferencing manager 1604 as part of videoconferencing module 1602 , some embodiments of videoconferencing manager 1604 are implemented as a separate component from videoconferencing module 1602 . Thus, a single videoconferencing manager 1604 can be used to manage and control several videoconferencing modules 1602 . For example, some embodiments will run separate video conferencing modules on local devices to interact with each party in a multi-party conference, each of these video conferencing modules on the local device being managed and controlled by a video conferencing manager.

The image processing manager 1608 of some embodiments processes images captured by the cameras of the dual camera mobile device before the images are encoded by the encoder 1655 . For example, some embodiments of the image processing manager 1608 perform one or more of exposure adjustments, focus adjustments, perspective corrections, dynamic range adjustments, and image scaling on images processed by the CIPU 1650. In some embodiments, the image processing manager 1608 controls the frame rate of encoded images transmitted to another device in the video conference.

Some embodiments of networking manager 1614 manage one or more connections between a dual camera mobile device and another device participating in a video conference. For example, the networking manager 1614 of some embodiments establishes connections between the dual camera mobile device and the other device of the video conference when the video conference begins, and terminates these connections when the video conference ends.

During the video conference, networking manager 1614 transmits images encoded by encoder 1655 to another device in the video conference, and routes images received from another device in the video conference to decoder 1660 for decoding. In some embodiments, the networking manager 1614, rather than the image processing manager 1608, controls the frame rate of the images transmitted to the other device of the video conference. For example, some such embodiments of the networking manager 1614 control the frame rate by dropping (ie, not transmitting) some of the encoded frames that should be transmitted to another device of the video conference.

As shown, the media exchange module 1620 of some embodiments includes a camera source module 1622 , a video compression module 1624 and a video decompression module 1626 . The media exchange module 1620 is the same as the media exchange module 310 shown in FIG. 3, while providing more details. Camera source module 1622 routes messages and media content between videoconferencing module 1602 and CIPU 1650 via CIPU driver 1630, and video compression module 1624 routes messages and media content between videoconferencing module 1602 and encoder 1655 via encoder driver 1635 , and video decompression module 1626 routes messages and media content between videoconferencing module 1602 and decoder 1660 via decoder driver 1640 . Some embodiments implement the TNR module 315 (not shown in FIG. 16 ) included in the media exchange module 310 as part of the camera source module 1622 , while other embodiments implement the TNR module 315 as part of the video compression module 1624 .

In some embodiments, CIPU driver 1630 and encoder driver 1635 are the same as CIPU driver 305 and encoder driver 320 illustrated in FIG. 3 . The decoder driver 1640 of some embodiments acts as a communication interface between the video decompression module 1626 and the decoder 1660 . In such an embodiment, the decoder 1660 decodes images received through the networking manager 1614 from another device of the video conference and routed through the video decompression module 1626 . After the image is decoded, the image is sent back to the video conferencing module 1602 through the decoder driver 1640 and the video compression module 1626 .

In addition to performing video processing during the video conference, the video conference and processing module 1600 of the dual camera mobile device of some embodiments also performs audio processing operations during the video conference. Figure 17 illustrates such a software architecture. As shown, video conferencing and processing module 1600 includes video conferencing module 1602 (which includes video conferencing manager 1604 , image processing manager 1608 and networking manager 1614 ), media exchange module 1620 and client application 1665 . Other components and modules of the video conferencing and processing module 1600 shown in FIG. 16 are omitted in FIG. 17 for simplicity of illustration. Video conferencing and processing module 1600 also includes frame buffers 1705 and 170 , audio processing manager 1715 and audio driver 1720 . In some embodiments, the audio processing manager 1715 is implemented as a separate software module, while in other embodiments, the audio processing manager 1715 is implemented as part of the media exchange module 1620 .

Audio processing manager 1715 processes audio data captured by the dual camera mobile device for transmission to another device in the video conference. For example, the audio processing manager 1715 receives audio data captured by the microphone 1725 through the audio driver 1720, encodes the audio data, and then saves the encoded audio data in the buffer 1705 for transmission to the other device. The audio processing manager 1715 also processes audio data captured by and received from another device in the video conference. For example, the audio processing manager 1715 retrieves audio data from the buffer 1710 , decodes the audio data, and then outputs the decoded audio data to the speaker 1730 through the audio driver 1720 .

In some embodiments, video conferencing module 1602 is part of a larger conferencing module along with audio processing manager 1715 and its associated buffers. When conducting a multiparty audio conference between several devices without exchanging video content, the video conferencing and processing module 1600 only utilizes the networking manager 1614 and the audio processing manager 1715 to enable Audio swapping of layers is easier.

Referring now to FIG. 18, the operation of the video conference manager 1604 of some embodiments is illustrated. FIG. 18 conceptually illustrates a process 1800 performed by a videoconferencing manager of some embodiments, such as videoconferencing manager 1604 illustrated in FIG. 16 . This may equivalently be performed by the management layer 935 of FIG. 9 . In some embodiments, when the user of the dual-camera mobile device accepts (e.g., via a user interface displayed on the dual-camera mobile device) a video conference request, or when a user of another device accepts a request sent by the user of the dual-camera mobile device , videoconferencing manager 1604 performs process 1800.

Process 1800 begins by receiving (at 1805) an instruction to start a video conference. In some embodiments, instructions are received from the client application 1665, or from a user via a user interface displayed on a dual camera mobile device, and forwarded by the client application 1665 to the video conference manager 1604. For example, in some embodiments, when a user of a dual camera mobile device accepts a video conference request, instructions are received through the user interface and forwarded by the client application. On the other hand, when a user of another device accepts a request from the local device, some embodiments receive instructions from the client application without user interface interaction (although there may be a previous user interface interaction that sent the initial request).

Subsequently, process 1800 initializes (at 1810 ) a first module that interacts with videoconferencing manager 1604 . Modules of some embodiments that interact with video conferencing manager 1604 include CIPU 1650, image processing manager 1608, audio processing manager 1715, and networking manager 1614.

In some embodiments, initializing the CIPU 1650 includes instructing the CIPU 1650 to begin processing images captured by one or both cameras of a dual camera mobile device. Some embodiments initialize the image processing manager 1608 by instructing the image processing manager 1608 to begin retrieving images from the buffer 1610, and processing and encoding the retrieved images. To initialize audio processing manager 1715, some embodiments instruct audio processing manager 1715 to begin encoding audio data captured by microphone 1725, and decoding audio data stored in buffer 1710 (audio data received from another device) so that Output to speaker 1730. Initialization of the networking manager 1614 of some embodiments includes instructing the networking manager 1614 to establish a network connection with another device in the video conference.

Process 1800 then determines (at 1815) whether any modules remain to be initialized. When modules remain to be initialized, process 1800 returns to operation 1810 to initialize another module. When all required modules have been initialized, process 1800 generates (at 1820) a composite image for display on the dual camera mobile device (ie, the local display). These composite images may include those shown in Figure 65 described below (i.e., PIP or other composite displays), and may include images from the cameras of the local dual camera mobile device and from another device participating in the videoconference various combinations of camera images.

Next, process 1800 determines (at 1825) whether changes have been made to the video conference. Some embodiments receive changes to the video conference through user interaction with a user interface displayed on a dual-camera mobile device, while other embodiments receive changes to the video conference from another device through the networking manager 1614 (i.e., remote control). In some embodiments, changes to video conferencing settings may also be received from client application 1665 , or other modules in video conferencing module 1602 . Video conferencing settings can also change due to changes in network conditions.

When a change occurs, process 1800 determines (at 1830) whether the change to the video conference is a change to network settings. In some embodiments, the change is either a network setting change or an image capture setting change. When the change to the video conference is a change to network settings, the process modifies (at 1840 ) the network settings, then proceeds to operation 1845 . Network setting changes of some embodiments include changing the bit rate at which images are encoded, or the frame rate at which images are transmitted to another device.

When the change to the video conference is not a change to network settings, process 1800 determines that the change is a change to image capture settings, thereby proceeding to operation 1835 . Process 1800 then performs (at 1835) changes to image capture settings. In some embodiments, changes to image capture settings may include switching cameras (i.e., switching which camera on a dual-camera mobile device captures video), focus adjustments, exposure adjustments, displaying or not displaying an image from a dual-camera mobile device. images from one or both cameras, and zoom in or out on images displayed on dual-camera mobile devices, among other setting changes.

At operation 1845, process 1800 determines whether to end the video conference. When process 1800 determines not to end the video conference, process 1800 returns to operation 1820 . When process 1800 determines that the video conference will end, process 1800 ends. When process 1800 receives an instruction from client application 1665 to end the video conference (i.e., due to an instruction received through the user interface of the native dual camera mobile device, or from another device participating in the video conference), process 1800 Some embodiments of determine to end the video conference.

In some embodiments, when the video conference ends, video conference manager 1604 performs various operations not shown in process 1800 . Some embodiments instruct the CIPU 1650 to stop generating images, instruct the networking manager 1614 to break the network connection with another device in the video conference, and instruct the image processing manager 1608 to stop processing and encoding images.

E. Temporal Noise Reduction

Some embodiments include a special temporal noise reduction module that processes video images to reduce noise in the video. The temporal noise reduction module of some embodiments compares subsequent images in a video sequence to identify and remove unwanted noise from the video.

Figure 19 conceptually illustrates the software architecture of such a temporal noise reduction (TNR) module 190 of some embodiments. Some embodiments implement the TNR module 1900 as part of an application (eg, as part of the media switching module as shown in FIG. 3 ), while other embodiments implement the TNR module 1900 as a stand-alone application used by other applications. Still other embodiments implement the TNR module 1900 as part of an operating system running on a dual camera mobile device. In some embodiments, TNR module 1900 is implemented by a set of APIs that provide some or all of the functionality of TNR module 1900 to other applications.

As shown in FIG. 19 , TNR module 1900 includes TNR manager 1905 , difference module 1910 , pixel averaging module 1915 and motion history module 1920 . Although Figure 19 shows the three modules 1910, 1915, and 1920 as separate modules, some embodiments implement the functionality of these modules described below in a single module. The TNR module 1900 of some embodiments receives as input an input image, a reference image, and a motion history. In some embodiments, the input image is the image currently being processed and the reference image is a previous image in the video sequence to which the input image is compared. The TNR module 1900 outputs an output image (a form of noise-reduced input image) and an output motion history.

The TNR manager 1905 of some embodiments directs the flow of data within the TNR module 1900 . As shown, the TNR manager 1905 receives an input image, a reference image, and a motion history. The TNR manager 1905 also outputs output images and output motion histories. The TNR manager 1905 sends the input image and the reference image to the difference module 1910 and receives the difference image from the difference module 1910 .

In some embodiments, the difference module 1910 processes data received from the TNR manager 1905 and sends the processed data to the TNR manager 1905 . As shown, the difference module 1910 receives an input image and a reference image from the TNR manager 1905 . The difference module 1910 of some embodiments generates a difference image by subtracting pixel values of one image from pixel values of another image. The differential image is sent to the TNR manager 1905 . A difference image of some embodiments indicates the difference between two images in order to identify parts of the input image that have changed compared to a previous image, and parts of the input image that have remained unchanged.

The TNR manager 1905 also sends the input image and the reference image to the pixel averaging module 1915 . As shown, some embodiments also send the motion history to the pixel averaging module 1915. However, other embodiments may only send the input image and the reference image, but not the motion history. In either embodiment, the TNR manager 1905 receives the processed image from the pixel averaging module 1915 .

The pixel averaging module 1915 of some embodiments uses the motion history to determine whether to obtain an average of pixels from the input image and the reference image with respect to a particular location in the image. In some embodiments, the motion history includes a probability value for each pixel in the input image. A particular probability value represents the probability that a corresponding pixel in the input image has changed (ie, a dynamic pixel) relative to a corresponding pixel in the reference image. For example, if a particular pixel in the input image has a probability value of 20, that indicates a probability of 20% that the particular pixel in the input image has changed relative to the corresponding pixel in the reference image. As another example, if a particular pixel in the input image has a probability value of 0, that indicates that the particular pixel in the input image has not changed (ie, a static pixel) relative to a corresponding pixel in the reference image.

Different embodiments save the probability values of the input images differently. Some embodiments may store the probability values for each pixel of the input image in a data array. Other embodiments may store the probability values in a matrix (eg, in a set of arrays) of the same dimensionality as the resolution of the images of the video. For example, if the video's images have a resolution of 320x240, then the matrix will also be 320x240.

When the pixel averaging module 1915 receives motion history from the TNR manager 1905 in addition to the input image and the reference image, the pixel averaging module 1915 reads a probability value for each pixel in the input image. If the probability value of a particular pixel in the input image is below a specified threshold (e.g., 5%, 20%), then the pixel averaging module 1915 determines that there is probably no motion at that particular pixel, thereby distorting between images for that pixel. Assuming that the difference may be due to noise, average that particular pixel value and the corresponding pixel value in the reference image.

If the probability value for a particular pixel in the input image is not below the specified threshold, then the pixel averaging module 1915 does not modify the particular pixel of the input image (ie, the pixel value at that pixel remains the same as in the input image). This is because the motion is likely to be at that particular pixel, so the difference between the images is likely not the result of noise. In some embodiments, when the motion history is not sent to the pixel averaging module 1915, the pixel averaging module 1915 averages each pixel in the input image with the corresponding pixel in the reference image. The processed image output by pixel averaging module 1915 and sent to TNR manager 1905 includes input image pixel values for any pixels that are not averaged, and average pixel values for any pixels that are averaged by pixel averaging module 1915 .

In some embodiments, the motion history module 1920 processes the data received from the TNR manager 1905 and sends the resulting data back to the TNR manager 1905 . The motion history module 1920 of some embodiments receives input images and motion histories from the TNR manager 1905 . Some embodiments feed this data into a Bayesian estimator to generate a new motion history (ie, a set of probability values) that can be used in pixel averaging with respect to the next input image. Other embodiments use other estimators to generate new motion histories.

Referring now to FIG. 20, the operation of the TNR module 1900 is illustrated. Figure 20 conceptually illustrates a process 2000 of some embodiments for reducing temporal noise of images of video. The process 2000 begins by receiving (at 2005 ) an input image, a reference image, and a motion history from the TNR manager 1905 . The input image is the image currently being processed for noise reduction. In some embodiments, the reference image is a previous image in a sequence of images of the video received from the CIPU. However, in other embodiments, the reference image is the output image (ie, the output of the TNR module 1900 ) resulting from processing of a previous input image. The motion history is the output motion history resulting from the processing of the previous input image.

When the input image is the first image of a video, the TNR module 1900 of some embodiments does not process the first image (ie, does not apply TNR to the first image). In other words, the TNR manager 1905 receives the first image, and outputs only the first image. In other embodiments, when the input image is the first image of a video, the first image is used as both the input image and the reference image, and the TNR module 1900 processes the image as described below. Also, when the input image is the first image of the video, and the motion history is empty (ie, null, all zeros, etc.), the TNR manager 1905 simply outputs an empty motion history as the output motion history.

The TNR manager 1905 then determines (at 2010) whether the input image is static. To make this determination, some embodiments send the input image and the reference image to a difference module 1910, from which the difference image is received. Some embodiments classify an input image as a static image when the difference between the two images is below a specified threshold (eg, 5% difference, 10% difference, etc.).

When the input image is a static image, the TNR manager 1905 sends the input image and the reference image to the pixel averaging module 1915 to calculate (at 2015) the average of the pixels of the input image and the pixels of the reference image in order to reduce any noise. Processing then proceeds to operation 2040 described below.

When the input image is not a static image, the TNR manager sends the input image, reference image and motion history to the pixel averaging module 1915 for processing. The pixel averaging module 1915 selects (at 2020) a pixel in the input image. Using the motion history, the pixel averaging module 1915 determines (at 2025) whether the pixel's probability of motion is below a certain threshold, as described above.

If the probability of the selected pixel is below a certain threshold, the pixel averaging module 1915 calculates (at 2030) the average of that pixel of the input image and the corresponding pixel in the reference image. Otherwise, that pixel is not averaged, and the output image will be the same as the input image at that particular pixel. The pixel averaging module 1915 then determines (at 2035) whether there are any unselected pixels left in the input image. If there are any pixels that have not been processed, then processing returns to operation 2020 to select the next pixel. The pixel averaging module 1915 performs operations 2020-2030 until all pixels have been evaluated.

The process then updates (at 2040) the motion history. As shown in FIG. 19 and described above, the motion history module 1920 updates the motion history based on the input image. The new motion history is output by the TNR manager along with the processed image from the pixel averaging module.

F. Image Processing Manager & Encoder

In addition to the temporal noise reduction and image processing operations performed by the CIPU and/or CIPU drivers, some embodiments perform various image processing operations at the image processing layer 930 of the video conferencing module 925 . These image processing operations may include exposure adjustments, focus adjustments, perspective corrections, dynamic range adjustments, and image scaling, among others.

FIG. 21 conceptually illustrates a process 2100 for performing such image processing operations. In some embodiments, some or all of the operations of process 2100 are performed by a combination of image processing manager 1608 and encoder driver 1635 of FIG. 16 . In some such embodiments, the image processing manager 1608 performs pixel-based processing (eg, scaling, dynamic range adjustment, perspective correction, etc.). Some embodiments perform process 2100 during a video conference on an image to be transmitted to another device participating in the video conference.

Referring now to FIG. 16, process 2100 is illustrated. The process begins by retrieving (at 2105 ) an image from buffer 1606 . In some embodiments, the retrieved image is an image of a video (ie, an image in a sequence of images). The video may be captured by a camera of the device on which process 2100 is performed.

Subsequently, process 2100 performs (at 2110) an exposure adjustment on the retrieved image. Some embodiments make exposure adjustments through a user interface displayed on a dual camera mobile device. Figure 22 illustrates an example exposure adjustment operation for such an embodiment.

FIG. 22 illustrates the exposure adjustment operation with reference to three stages 2210, 2215, and 2220 of the UI 2205 of the device 2200. The first stage 2210 illustrates the UI 2205 including the display area 2225 and the display area 1155. As shown, display area 2225 displays an image 2230 of a man with the sun and a dark face and body. The darkened face and body show that the man was not properly exposed. Image 2230 may be a video image taken by a camera of device 2200 . As shown, display area 1155 includes selectable UI item 2250 for ending the video conference. In some embodiments, the layout of the display area 1155 is the same as the layout of the display area 1155 of FIG. 12 described above.

The second stage 2215 illustrates that the user of the device 2200 initiates an exposure adjustment operation by selecting an area of the display area 2225 . In this example, selection is accomplished by placing finger 2235 anywhere within display area 2225 . In some embodiments, the user selects an exposure adjustment from a menu of possible image setting adjustments.

The third stage 2220 shows an image 2240 of the man after the exposure adjustment operation has been completed. As shown, image 2240 is similar to image 2230, except that the man in image 2240 is properly exposed. In some embodiments, the properly exposed image is an image taken after the improperly exposed image. The exposure adjustment operation initiated in the second stage 2215 adjusts the exposure of subsequent images taken by the camera of the device 2200 .

Returning to FIG. 21 , process 2100 then performs (at 2115 ) a focus adjustment on the image. Some embodiments make focus adjustments through a user interface displayed on a dual camera mobile device. FIG. 23 conceptually illustrates an example of such a focus adjustment operation.

FIG. 23 illustrates the focus adjustment operation with reference to three different stages 2310, 2315, and 2320 of the UI 2305 of the device 2300. The first stage 2310 illustrates the UI 2305 including the display area 2325 and the display area 1155. Display area 2325 presents a blurred image 2330 of the man taken by the camera of device 2300 . This blurring indicates that the image 2330 of the man is out of focus. That is, when the camera captured the image 2330 of the man, the lens of the camera was not focused on the man. Also, the image 2330 may be a video image captured by a camera of the device 2300 . As shown, display area 1155 includes selectable UI item 2350 for ending the video conference. In some embodiments, the layout of the display area 1155 is the same as the layout of the display area 1155 of the display area 1155 of FIG. 12 described above.

The second stage 2315 illustrates that a user of the device 2300 initiates a focus adjustment operation by selecting an area of the display area 2325 . In this example, selection is accomplished by placing finger 2335 anywhere within display area 2225 . In some embodiments, the user selects a focus adjustment from a menu of possible image setting adjustments.

The third stage 2320 displays the image 2340 of the man after the focus adjustment operation has been completed. As shown, image 2340 is the same as image 2330, although the man in image 2340 appears sharper. This shows that the lens of the camera is properly focused on the man. In some embodiments, the properly focused image is an image taken after the improperly focused image. The focus adjustment operation initiated in the second stage 2315 adjusts the focus of subsequent images taken by the camera of the device 2300 .

Returning to FIG. 21 , process 2100 performs (at 2120 ) image scaling on the image. Some embodiments perform image scaling on an image to reduce the number of bits used to encode the image (ie, reduce the bit rate). In some embodiments, process 2100 performs image scaling, as described below with reference to FIG. 26 .

Process 2100 then applies (at 2125) perspective correction to the image. In some embodiments, process 2100 performs perspective correction, as described below in FIG. 24 . Such perspective correction involves utilizing data obtained by one or more accelerometer and/or gyroscope sensors that identify the orientation and movement of the dual camera mobile device. The data is then used to modify the image to correct incorrect perspective.

After perspective correcting the image, process 2100 adjusts (at 2130) the dynamic range of the image. In some embodiments, the dynamic range of an image is the range of possible values that each pixel in the image can have. For example, an image with a dynamic range of 0-255 can be scaled to a range of 0-128, or any other range of values. Adjusting the dynamic range of an image can reduce the number of bits that will be used to encode the image (ie, reduce the bit rate), thereby smoothing the image.

Adjusting the dynamic range of an image can also be used for various other purposes. One purpose is to reduce image noise (for example, the image is taken by a noisy camera sensor). To reduce noise, the dynamic range of the image can be adjusted such that dark levels are redefined to include light blacks (ie, crush black). In this way, the noise of the image is reduced. Another purpose of dynamic range adjustment is to adjust one or more colors or color ranges in order to enhance an image. For example, some embodiments may assume that the image captured by the front camera is an image of a human face. Therefore, the dynamic range of this image can be adjusted to enhance reds and pinks to make a person's cheeks appear rosier/more rosy. The dynamic range adjustment operation can also be used for other purposes.

Finally, process 2100 determines (at 2135) one or more rate controller parameters for encoding the image. In some embodiments, such rate controller parameters may include quantization parameters and frame types (eg, predictive frames, bi-directional frames, intra-coded frames). The processing then ends.

Although the various operations of process 2100 are illustrated as being performed in a particular order, one of ordinary skill will recognize that many of these operations (exposure adjustments, focus adjustments, perspective corrections, etc.) can be performed in any order and independently of each other. . That is, the process of some embodiments may perform focus adjustments prior to exposure adjustments, or similar modifications may be made to the process illustrated in FIG. 21 .

1. Perspective Correction

As noted above, some embodiments apply perspective correction to images prior to displaying or transmitting the images. In some cases, one or more cameras on a dual-camera mobile device will not be fully facing its subject, and the subject will appear distorted in the uncorrected image. Images can be processed using perspective correction so that the image will faithfully reflect how the subjects in the image would appear to a human.

FIG. 24 schematically illustrates a perspective correction process 2400 performed by an image processing manager of some embodiments, such as the image processing manager illustrated in FIG. 16 . Process 2400 of some embodiments is performed by image processing layer 930 shown in FIG. 9 (which may include image processing manager 1608). Some embodiments perform process 2400 at operation 2125 of process 2100 to correct the perspective of the most recently captured video image prior to displaying or transmitting the image.

Process 2400 begins by receiving (at 2405) data from an accelerometer sensor, which in some embodiments is part of a dual camera mobile device. The accelerometer sensor of some embodiments measures the rate of change of the velocity of the device (ie, the acceleration of the device) along one or more axes. The process also receives (at 2410) data from a gyroscopic sensor, which in some embodiments may also be part of a dual camera mobile device. The gyroscope and accelerometer sensors of some embodiments may be used alone or in combination to identify the orientation of a dual camera mobile device.

The process 2400 then determines (at 2415) the amount of perspective correction to make based on the data obtained from the accelerometer and gyroscope sensors. In general, more perspective correction is required to produce an optimal image as the orientation is further off-axis. Some embodiments calculate a warp parameter representing the amount of perspective correction based on the orientation of the device.

After determining the amount of perspective correction to make, process 2400 receives (at 2420) an image captured by a camera of the dual camera mobile device. The processing may be performed for each image in the video sequence captured by the camera. Some embodiments may perform separate calculations for each image from each of the two cameras on a dual camera mobile device.

The process then modifies (at 2425) the image according to the determined amount of perspective correction. Some embodiments use a baseline image or other information (eg, user-entered points about which corrections should be made) in addition to warp parameters or other indications of the amount of perspective correction. After modifying the image, process 2400 ends.

Figure 25 conceptually illustrates example image processing operations of some embodiments. 25 illustrates a first image processing operation 2505 performed by the first image processing module 2520 without perspective correction, and a second image processing operation 2550 performed by the second image processing module 2565 with perspective correction.

As shown, a first image processing operation 2505 is performed on a first image 2510 of a block 2515 resulting from spatial perspective looking down at the block at an angle. The top of the block is closer to the perspective than the bottom of the block 2515. Thus, block 2515 appears to be tilted towards the camera that took first image 2510 . FIG. 25 also shows the processed first image 2525 after being processed by the first image processing module 2520 . As shown, the block 2515 in the processed first image 2525 appears the same post-processing because the first image processing module 2520 did not perform any perspective correction.

A second image processing operation 2550 is performed on a second image 2555 of the volume 2560 . Block 2560 is the same as block 2515 in first image 2510 . FIG. 25 also shows the processed second image 2575 after the processing of the second image 2555 by the perspective corrector 2570 of the second image processing module 2565 . Perspective corrector 2570 may use process 2400 in order to correct the perspective of second image 2555 . Based on data from the accelerometer and gyroscope indicating that the camera that took the second image 2555 was tilted at a downward angle (and possibly other data), the perspective corrector 2575 can correct the second image such that in the processed second image In 2575, blocks appear as if viewed straight.

2. Scaling and bitstream processing

Image scaling and bitstream processing are among the functions performed by the image processing layer 930 of some embodiments described above with reference to FIG. 21 . Image scaling (performed at operation 2130) involves, in some embodiments, scaling up or down an image (ie, modifying the number of pixels used to represent the image). In some embodiments, bitstream processing involves inserting into the bitstream data indicating the size of the scaled image. In some embodiments, such scaling and bitstream processing is performed by an encoder driver (eg, driver 1635).

Figure 26 conceptually illustrates the software architecture of such an encoder driver 2600 of some embodiments, and shows example scaling and bitstream processing operations performed by the encoder driver 2600 on an example image 2605. In some embodiments, image 2605 is an image taken by a camera of a dual camera mobile device for video transmission to another device in the video conference. Referring to FIG. 16, in some embodiments video images have been propagated from CIPU 1650 to buffer 1606 via CIPU driver 1630 and camera source module 1622, from which image processing manager 1608 retrieves the video images. After image processing (eg, focus adjustment, exposure adjustment, perspective correction) in the image processing manager 1608 , the image is sent to the encoder driver 1635 through the buffer 1610 and the video compression module 1624 .

As shown, encoder driver 2600 includes processing layer 2610 and rate controller 2645 . An example of a rate controller of some embodiments is illustrated in Figure 30, described below. The processing layer 2610 includes an image scaler 2615 and a bitstream manager 2625 . In some embodiments, these modules perform various operations on the image before and after it is encoded. Although in this example the image scaler is shown as part of the processing layer 2610 of the encoder driver 2600, some embodiments implement the image scaler as part of the image processing manager 1608 rather than as part of the encoder driver 2600 (i.e., in image scaling before sending the image and size data to the encoder driver).

As shown, the image scaler 2615 scales the image before it is sent to the encoder 2650 via the rate controller 2645 . Image 2605 is sent through scaler 2615 and downscaled to image 2630 . In addition to scaling down images, some embodiments are also capable of scaling up images.

As shown in FIG. 26, some embodiments downscale an input image (e.g., image 2605), and then overlay the downscaled image (e.g., 2630) on a spatially redundant image (e.g., image 2635), which The redundant image is the same size (in terms of pixels) as the input image (ie, the image 2605 has the same number of rows and columns of pixels as the spatially redundant image 2635 has the same number of rows and columns of pixels). Some embodiments overlay the downscaled image 2630 on the upper left corner of the spatially redundant image (as shown, to produce composite image 2640), while other embodiments overlay the downscaled image on a different portion of the spatially redundant image ( For example, Center, Top Left, Directly Above, Directly Below, Bottom Right, etc.).

In some embodiments, a spatially redundant image is an image that is substantially all of one color (eg, black, blue, red, white, etc.), or has a repeating pattern (eg, checkered, stripes, etc.). For example, the spatially redundant image 2635 shown in FIG. 26 has a repeating cross-shaped pattern. Due to the repetitiveness, the spatially redundant portion of the composite image 2640 can be easily compressed by the encoder into a small amount of data. Furthermore, if a sequence of images is scaled down using the same spatially redundant images for each image in the sequence, temporal compression can be used to further reduce the amount of data required to represent the encoded image. Heavy.

Some embodiments of the image scaler 2615 also generate size data 2620 indicating the size of the scaled image (eg, the size of the scaled image 2630 ), and send the generated size data 2620 to the bitstream manager 2625 . The size data 2620 of some embodiments indicates the size of the scaled image 2630 in terms of number of rows of pixels and number of columns of pixels (ie, height and width) of the scaled image 2630 . In some embodiments, size data 2620 also indicates the position of scaled image 2630 within composite image 2640 .

After the images are scaled, the composite image 2640 is sent to the encoder 2650 through the rate controller 2645 . In some embodiments, rate controller 2645 controls the bit rate (ie, data size) of the images output by encoder 2650, as described in further detail below. The encoder 2650 of some embodiments compresses and encodes images. Encoder 2650 may use H.264 encoding or another encoding method.

The bitstream manager 2625 of some embodiments receives the bitstream of one or more encoded pictures from the encoder 2650 and inserts size data into the bitstream. For example, in some embodiments, bitstream manager 2625 receives size data 2620 from image scaler 2615 and inserts size data 2620 into bitstream 2655 of encoded composite image 2640 received from encoder 2650 . In this case, the output of bitstream manager 2625 is modified bitstream 2660 including size data 2620 . Different embodiments insert the size data 2620 at different locations in the bitstream 2655 . For example, bitstream 2660 shows that size data 2620 is inserted at the beginning of bitstream 2660 . However, other embodiments insert size data 2620 at the end of bitstream 2655 , in the middle of bitstream 2655 , or at any other location within bitstream 2655 .

In some embodiments, bitstream 2655 is a bitstream comprising a sequence of one or more encoded images of composite image 2640 . In some such embodiments, the images in the series are all scaled to the same size, and size data 2620 indicates the size of these scaled images. After the image is transmitted to the device at the other end of the video conference, the receiving device is able to extract the size information from the bitstream and use it to correctly decode the received image.

Figure 27 conceptually illustrates an image scaling process 2700 performed by an encoder driver of a dual camera mobile device, such as driver 2600. Process 2700 begins by receiving (at 2705) an image (eg, image 2605) captured by a camera of a dual camera mobile device. When a dual camera device captures images with both cameras, some embodiments perform process 2700 on the images from both cameras.

Subsequently, process 2700 scales (at 2710) the received image. As noted above, different embodiments scale image 2605 differently. For example, image 2605 in FIG. 26 is scaled down and overlaid on spatially redundant image 2635 to produce composite image 2640 .

Process 2700 then sends (at 2715 ) the scaled image (eg, composite image 2640 including scaled image 2630 ) to encoder 2650 for encoding. Some embodiments of process 2700 send scaled image 2630 (included in composite image 2640) to encoder 2650 through a rate controller that determines the bit rate at which the encoder encodes the image. The encoder 2650 of some embodiments compresses and encodes the image (eg, using discrete cosine transform, quantization, entropy coding, etc.), and returns a bitstream with the encoded image to the encoder driver 2600.

Subsequently, process 2700 sends (at 2720) the bitstream manager data indicating the size of the scaled image (eg, size data 2620). As shown in Figure 26, in some embodiments, this operation is performed in the encoder driver 2600 (i.e., one module in the encoder driver 2600 sends the size data to another module in the encoder driver 2600) .

After the scaled image is encoded by encoder 2650, process 2700 receives (at 2725) a bitstream from the encoder. As shown, some embodiments receive the bitstream at a bitstream manager that otherwise has received size data. The received bitstream includes the encoded composite image and may include one or more additional images in the video sequence.

Process 2700 then inserts data (eg, size data 2620 ) into the bitstream indicating the size of the scaled image, and then ends. As shown in Figure 26, in some embodiments, this operation is also performed by the bitstream manager. As mentioned above, different embodiments insert large and small data into different parts of the bitstream. In the illustrated example, size data 2620 is inserted at the beginning of bitstream 2655, as shown in resulting bitstream 2660. This bitstream can now be transmitted to another device participating in the video conference where it can be decoded and viewed.

In some embodiments, a decoder driver (eg, driver 1640 ) performs the reverse function of an encoder driver. That is, the decoder driver extracts the size data from the received bit stream, sends the bit stream to the decoder, and scales the decoded image using the size data. Figure 28 conceptually illustrates the software architecture of such a decoder driver 2800 of some embodiments, and represents example bitstream processing and scaling operations performed by the decoder driver 2800 on an example bitstream 2825.

In some embodiments, bitstream 2825 is a bitstream comprising encoded images of video captured by a camera of one of the devices in the video conference (e.g., a bitstream from an encoder driver such as driver 2600), and is transmitted For the device on which the decoder driver 2800 runs. 16, in some embodiments, the bitstream has been received by the networking manager 1614 and sent to the buffer 1616, from which the bitstream is retrieved by the video decompression module 1626 and sent to the decoder driver 1640.

As shown, the decoder driver 2800 includes a processing layer 2805 . The processing layer 2805 includes an image scaler 2810 and a bitstream manager 2820 . In some embodiments, these modules 2810 and 2820 perform various operations on received images before and after the images are decoded. Although in this example the image scaler 2810 is shown as part of the processing layer 2805 of the decoder driver 2800, some embodiments implement the image scaler as part of the image processing manager 1608 rather than as part of the decoder driver (i.e. , after the image is sent from the decoder driver 2800, image scaling is performed).

As shown, the bitstream manager 2820 of some embodiments receives a bitstream of one or more encoded pictures (i.e., pictures in a video sequence), extracts size data from the bitstream, and then sends the bitstream to Decoder 2835 performs decoding. For example, as shown in FIG. 28, the bitstream manager 2820 receives a bitstream 2825 of an encoded image, extracts size data 2815 from the bitstream 2825, and sends the resulting bitstream 2830 (without the size data 2815) to a decoder 2835 to decode. As shown, in some embodiments, bitstream manager 2820 sends extracted size data 2815 to image scaler 2810 .

The size data 2815 for some embodiments is the same as the size data 2620 inserted into the bitstream by the encoder driver 2600. As described above in the description of FIG. 26 , the size data 2815 of some embodiments indicates the size of the sub-image 2845 in terms of the number of rows of pixels and the number of columns of pixels of the sub-image 2845 . Size data 2815 may also indicate the location of sub-image 2845 within larger spatially redundant image 2840 . In this example, bitstream 2825 shows size data 2815 inserted at the beginning of bitstream 2825 . However, as noted above, different embodiments insert the size data 2815 at different locations in the bitstream 2825 .

The image scaler 2810 of some embodiments utilizes the size data received from the bitstream manager 2820 to extract sub-images from the image. For example, FIG. 28 illustrates that image scaler 2810 receives image 2840 from decoder 2835 that includes sub-image 2845 . As shown, image scaler 2810 of some embodiments extracts sub-image 2845 from image 2840 . The extracted images can then be displayed on the dual camera mobile device.

Figure 29 conceptually illustrates an image extraction process 2900 performed by a decoder driver of a device participating in a video conference, such as driver 2800, in some embodiments. The process begins by receiving (at 2905) a bitstream (eg, bitstream 2825) of an encoded image. The bitstream may be sent from another device videoconferencing with the device the decoder driver is running on, or may be saved in memory of the device. When the device receives images from multiple sources, some embodiments process 2900 the images from each source.

Subsequently, process 2900 extracts (at 2910) size data from the bitstream. As mentioned above, the size data can be found in various places in the bitstream. Some embodiments know where to look for the size data, while other embodiments look for a specific flag indicating where the size data is located in the received bitstream. In some embodiments, the size data indicates the size of the sub-image (eg, the number of pixels in each row and the number of pixels in each column) and the position of the sub-image in the encoded image.

Process 2900 then sends (at 2915) the extracted size data to the image scaler. As shown in Figure 28, in some embodiments, this operation is performed within the decoder driver (ie, one module in the decoder driver sends the size data to another module in the decoder driver).

Process 2900 also sends (at 2920) the bitstream to the decoder for decoding. In some embodiments, the decoder decompresses and decodes the bitstream (eg, using inverse discrete cosine transform, inverse quantization, etc.), and returns a reconstructed image to the decoder driver.

After the bitstream is decoded by the decoder, process 2900 receives (at 2925) the decoded image from the decoder. As shown, some embodiments receive decoded images at the image sealer that has also received size data from the bitstream manager. Processing then extracts (at 2930) a sub-image from the decoded image using the received size data. As shown, sub-image 2845 is extracted from the upper left corner of decoded image 2840 as shown in size data 2815 . The extracted sub-images can now be displayed on a display device (eg, the screen of a dual camera mobile device).

3. Rate controller

In some embodiments, the two cameras of the device have different sets of characteristics. For example, in some embodiments, the front camera is a lower resolution camera optimized for capturing motion video images, while the rear camera is a higher resolution camera optimized for capturing still images. Other embodiments may use different combinations of cameras of different characteristics for various reasons such as cost, functionality, and/or geometry of the device.

Cameras with different characteristics can introduce different artifacts. For example, a higher resolution camera will show more noise than a lower resolution camera. Images captured by a higher resolution camera may exhibit a higher level of spatial or temporal complexity than images captured by a lower resolution camera. Additionally, different cameras with different optical properties introduce different gamma values in the captured images. Different light-sensitive mechanisms used by different cameras to capture images also introduce different artifacts.

Some of these camera-specific artifacts hide artifacts generated by other sources. For example, artifacts that are byproducts of the video encoding process become less visible in images captured by high resolution cameras with high levels of noise. When coding noise such as quantization artifacts hides behind camera-specific artifacts, the video coding process can use larger quantization step sizes to achieve smaller bit rates. On the other hand, when the camera introduces fewer artifacts (such as in the case of lower resolution cameras), the video encoding process can utilize finer quantization step sizes in order to avoid unacceptable levels of visual distortion caused by quantization. Thus, a video encoding process optimized to exploit or compensate for these camera-specific characteristics can achieve a better rate-distortion trade-off than a video encoding process that ignores these camera-specific characteristics.

In order to exploit these camera-specific properties to achieve the rate-distortion tradeoff, some embodiments implement two video encoding processes optimized with respect to each of the two cameras, respectively. FIG. 30 illustrates an example of a system with two video encoding processes for two cameras 3060 and 3070 . As shown in FIG. 30 , system 3000 includes encoder driver 3010 , rate controllers 3020 and 3040 , and video encoder 3030 . Encoder 3030 encodes video images captured from video cameras 3060 and 3070 into bitstreams 3080 and 3090 .

In some embodiments, video encoder driver 3010 is a software module running on one or more processing units. It provides the interface between the video encoder 303 and other components of the system, such as video cameras, image processing modules, network management modules and memory buffers. The encoder driver 3010 controls the flow of captured video images from the camera and image processing modules to the video encoder 3030, and it also provides the pipeline for the encoded bitstreams 3080 and 3090 to the storage buffers and network management modules.

As shown in Figure 30, the encoder driver 3010 includes two different instances 3020 and 3040 of the rate controller. These multiple instances can be two different rate controllers for two different cameras, or one rate controller configured in two different ways with respect to two different cameras. Specifically, in some embodiments, the two rate controllers 3020 and 3040 represent two separate rate controllers. On the other hand, in other embodiments, the two rate controllers 3020 and 3040 are two different configurations of a single rate controller.

Figure 30 also shows an encoder driver 3010 that includes a state buffer 3015 that holds encoding state information for rate control operations used during a video conference. Specifically, in some embodiments, the two different rate controllers, or two different configurations of the same rate controller, share the same encoding state information stored in the state buffer 3015 during the video conference. This sharing of state information allows for unified rate controller operation in dual-capture videoconferencing. This sharing also allows for optimal video encoding during switching camera operations in single-video capture videoconferencing (i.e., allows rate control operations on the encoding of the video captured by the current camera to use the rate determined by the encoding of the video captured by the previous camera). Encoded state information maintained by control operations). FIG. 30 shows state buffer 3015 as part of encoder driver 3010 , although other embodiments may implement state buffer 3015 outside of encoder driver 3010 .

In state buffer 3015, different embodiments store different types of data representing encoding state information (eg, different types of encoding parameters). An example of such encoding state information is the current target bit rate for a video conference. One way of identifying the target bit rate is described above in Section III.B. Other examples of such encoding status information include buffer fullness, maximum buffer fullness, bit rate of one or more most recently encoded frames, and other encoding status information.

The rate controller can then use the target bit rate (or another encoding state parameter held in the state buffer) to calculate one or more parameters for use in its rate control operations. For example, as described further below, the rate controller of some embodiments uses the current target bit rate to calculate the quantization parameter QP for a macroblock or frame. For example, some embodiments use the current target bit rate to calculate the quantization adjustment parameter from which the quantization parameter QP of the macroblock and/or frame is derived. Thus, during a camera switching operation in a video conference, sharing the target bit rate between two rate control operations (two rate controllers, or two different configurations of a rate controller) allows The encoding rate control operation benefits from the encoding status data of the previous rate control operation encoding the video captured by the previous camera.

FIG. 30 illustrates an encoder driver 3010 including two different rate controller instances 3020 and 3040 . However, in other embodiments, these rate controller instances 3020 and 3040 are built into the video encoder 3030 . Video encoder 3030 encodes video images captured by cameras 3060 and 3070 into digital bit streams 3080 and 3090 . In some embodiments, the video encoder generates a bitstream conforming to a conventional video encoding standard (eg, H.264 MPEG-4). In some of these embodiments, the video encoder performs encoding operations including motion estimation, discrete cosine transform ("DCT"), quantization, and entropy encoding. A video encoder also performs a decoding operation that is the functional inverse of the encoding operation.

In some embodiments, the encoder 3030 includes a quantizer module 3032 for performing quantization. The quantizer module is controlled by a quantization parameter 3022 or 3042 from a rate controller 3020 or 3040 . In some embodiments, each quantization parameter is set by a corresponding rate controller and is a function of one or more properties of the camera associated with that rate controller, as described further below. The rate controller can reduce the number of bits used for encoding by setting a larger quantization step size, or increase the number of used bits by setting a smaller quantization step size. By controlling the quantization step size, the rate controller also determines how much distortion will be introduced in the encoded video image. Thus, the rate controller enables a trade-off between bit rate and image quality. In achieving the rate-distortion tradeoff, the rate controller monitors the bit rate so as not to overflow the store buffer, underflow the store buffer, or exceed the transmission channel capacity. The rate controller must also control the bit rate in order to provide the best possible image quality and to avoid unacceptable image quality distortions caused by quantization. In some embodiments, each rate controller maintains monitoring data in status buffer 3015 represented as a set of status data values. In some embodiments, rate controllers 3020 and 3040 use camera-specific properties to optimize the rate-distortion tradeoff.

In some embodiments, each rate controller optimizes the rate-distortion tradeoff by directly applying a correction factor to its quantization parameter. In some of these embodiments, the correction factors are predetermined and built into the device along with the camera; the device does not need to calculate these correction factors on the fly. In other embodiments, the system dynamically determines appropriate camera-specific correction factors using input images captured by the camera. In some of these embodiments, the system analyzes a series of input video images captured by the camera over multiple encodings to gather certain statistical data about the camera. The system then uses these statistics to derive correction factors for the camera-optimized quantization parameters.

In some embodiments, these camera-specific correction factors are applied to the quantization parameters via the visual masking properties of the video image. The visual masking property of an image or a portion of an image is an indication of how much coding artifacts can be tolerated in said image or portion of an image. Some embodiments compute a visual masking property that quantifies the luminance energy of the image or image portion, while other embodiments compute a visual masking property that quantifies the activation energy or complexity of the image or image portion. Regardless of how the visual masking properties are calculated, some embodiments utilize the visual masking properties to calculate correction or masking quantization parameters for a video frame. Some such embodiments compute masking quantization parameters as frame-level visual masking attributes and the benchmark visual masking properties The function. In some embodiments, the video masking attribute is used to and The modified quantization parameter is expressed as:

where MQP _frame is the masked or modified quantization parameter of the frame, QP _nom is the initial or nominal quantization value, and β _frame is a constant suitable for local statistics. In some embodiments, the benchmark visual masking properties and the nominal quantization parameter QP _nom are predetermined based on an initial or periodic assessment of network conditions.

In some embodiments, the visual masking property in equation (1) is calculated as

where avgFrameLuma is the frame's average luminance value, and avgFrameSAD is the frame's mean sum of absolute differences. The constants α, β, C, D and E are suitable for local statistics. In some embodiments, these constants are adapted to camera-specific characteristics.

Some embodiments also calculate masked quantization parameters for some portion of the video image, such as a macroblock. In these cases, the masking quantization parameters are computed as macroblock visual masking properties The function:

Wherein, in some embodiments, β _MB is a constant suitable for local statistical data, and MQP _frame is calculated using equations (1) and (2). In some embodiments, the visual masking property in equation (3) is calculated as

where avgMBLuma is the average luminance value of the macroblock, and avgMBSAD is the mean sum of absolute differences of the macroblock. The constants α, β, A, B and C are suitable for local statistics. In some embodiments, these constants are adapted to camera-specific characteristics.

Rather than using multiple camera-specific constants to compute modified quantization parameters as described above, some embodiments implement camera-specific rate control by computing quantization parameters using only a single camera-specific coefficient. For example, the known visual masking properties and As well as the quantization parameter QP _frame , some embodiments utilize coefficients μ specific to a single camera to calculate the quantization parameter for a macroblock:

To compute Equation (5), some embodiments use the frame and macroblock complexity metrics as visual masking attributes, respectively and

Some embodiments apply different camera-specific coefficients in the calculation of QP _MB . For example, in some embodiments, QP _MB is calculated as

where ρ is a coefficient adjusted according to camera-specific characteristics.

As mentioned above, state buffer 3015 holds encoding state information that two different rate controller instances 3020 and 3040 can share during a videoconference in order to obtain better encoding results from their rate control operations. In some embodiments, the target bit rate _RT is an example of such shared state information. This bitrate is the ideal bitrate for encoding a series of frames. Generally, the bit rate is expressed in bits/second and is determined according to processes such as described above in Section III.B.

As described above, the rate controller of some embodiments utilizes the target bit rate to calculate the frame and/or macroblock quantization parameters QP that it outputs to the video encoder 3030 . For example, some embodiments use the current target bit rate to calculate the quantization adjustment parameter, and these embodiments derive the macroblock and/or frame quantization parameter QP from the calculated quantization adjustment parameter. In some embodiments, the quantization adjustment parameter is represented by a fraction calculated by dividing the bit rate of the previous frame, or a running average of the bit rate of the previous frame, by the current target bit rate. In other embodiments, the adjustment parameter is not precisely calculated in this way, but generally (1) is proportional to the bit rate of the previous frame or the running average of the bit rate of the previous frame, and (2) Inversely proportional to the current target bitrate.

After calculating such a quantization adjustment parameter, the rate controller of some embodiments utilizes this parameter to adjust its calculated macroblock and/or frame quantization parameters. One way of doing this adjustment is to multiply the calculated macroblock and/or frame quantization parameter by the quantization adjustment parameter. Another way of doing this adjustment is to calculate an offset quantization parameter value based on the quantization adjustment parameter, and then apply (eg subtract) the offset parameter to the calculated macroblock and/or frame quantization parameter. The rate controller of these embodiments then outputs the adjusted macroblock and/or frame quantization parameters to video encoder 3030 .

In other embodiments, the rate controller uses the target bit rate to calculate other parameters used in its rate control operations. For example, in some embodiments, the rate controller uses the target bit rate to modify the visual masking strength of the macroblock or frame.

G. Networking Manager

Figure 31 conceptually illustrates the software architecture of a networking manager 3100 of some embodiments, such as the networking manager 1614 illustrated in Figure 16 . As described above, the networking manager 3100 manages network connections (eg, connection establishment, connection monitoring, connection adjustments, connection interruptions, etc.) between the dual camera mobile device on which it operates and remote devices in a video conference. During the video conference, the networking manager 3100 of some embodiments also processes data transmitted to the remote device, and processes data received from the remote device.

As shown in FIG. 31, the networking manager 3100 includes a session negotiation manager 3105, a transmitter module 3115, a generic transport buffer 3120, a generic transport buffer manager 3122, a virtual transfer protocol (VTP) manager 3125, a receiver Module 3130 and Media Delivery Manager 3135.

Session negotiation manager 3105 includes protocol manager 3110 . The protocol manager 3110 ensures that during the video conference, the transmitter module 3115 uses the correct communication protocol to transmit data to the remote device, and enforces compliance with the rules of the communication protocol used. Some embodiments of the protocol manager 3110 support multiple communication protocols, such as Real-time Transport Protocol (RTP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Hypertext Transfer Protocol (HTTP), among others.

Session Negotiation Manager 3105 is responsible for establishing connections between the dual camera mobile device and one or more remote devices participating in the video conference, and for terminating those connections after the conference. In some embodiments, the session negotiation manager 3105 is also responsible for establishing a multimedia communication session (e.g., transmitting and receiving video and/or audio streams) between the dual camera mobile device and the remote device in the video conference (e.g., using the session Initiation Protocol (SIP)).

The session negotiation manager 3105 also receives feedback data from the media delivery manager 3135, and determines the operation of the general transmission buffer 3120 (eg, whether to transmit or discard packets/frames) through the general transmission buffer manager 3122 according to the feedback data. In some embodiments, such feedback may include one-way latency and bandwidth estimation bit rate. In other embodiments, the feedback includes packet loss information and round trip delay times (eg, determined from packets sent to and receipt of acknowledgments from remote devices in the video conference). Based on information from the media transfer manager 3135, the session negotiation manager 3105 can determine whether too many packets are being sent, and instruct the general transfer buffer manager 3122 to allow the general transfer buffer 3120 to transfer fewer packets (i.e., as Adjust bitrate as described in Figure 15).

Transmitter module 3115 retrieves encoded images (e.g., in bitrate form) from a video buffer (e.g., buffer 1612 of FIG. 3125, sent to the remote device in the video conference. The manner in which the encoded image is generated and sent to the transmitter module 3115 may be based on instructions or data received from the media delivery manager 3115 and/or the session negotiation manager 3105 . In some embodiments, packetizing the image involves dividing the received bitstream into a set of packets, each packet having a specific size (i.e., a size specified by the session negotiation manager 3105 according to a specific protocol), and adding any required headers (for example, address header, protocol-specific header, etc.).

Generic transmit buffer manager 3122 controls the operation of generic transmit buffer 3120 according to data and/or instructions received from session negotiation manager 3105 . For example, the general transfer buffer manager 3122 may be instructed to instruct the general transfer buffer 3120 to transfer data, stop transferring data, discard data, and so on. As noted above, in some embodiments, when a remote device participating in a conference appears to be missing a packet, this can be identified based on an acknowledgment received from the remote device. To reduce packet loss, the general transmit buffer manager 3122 may be instructed to transmit packets to the remote device at a lower rate.

The general transmit buffer 3120 holds the data received from the transmitter module 3115 and transmits the data to the remote device through the VTP manager 3125. As described above, according to instructions received from general transmission buffer manager 3122, general transmission buffer 3120 may drop packets (eg, images of video).

In some embodiments, RTP is used to communicate data packets (eg, audio packets and video packets) over UDP during a video conference. Other embodiments utilize RTP to communicate data packets over TCP during a video conference. In various embodiments, other transport layer protocols may also be used.

Some embodiments use a pair of port numbers (ie, a source port number and a destination port number) to define a particular communication channel between two mobile devices. For example, one communication channel between mobile devices may be defined with a pair of port numbers (e.g., source port 50 and destination port 100), and a different communication channel between mobile devices may be defined with another different pair of port numbers ( For example, source port 75 and destination port 150) are defined. Some embodiments also utilize a pair of Internet Protocol (IP) addresses to define a communication channel. In some embodiments, different communication channels are used to communicate different types of data packets. For example, video data packets, audio data packets and control signaling data packets may be communicated in separate communication channels. Thus, the video communication channel transfers video data packets, and the audio communication channel transfers audio data packets.

In some embodiments, a control communication channel is used for messaging between a local mobile device and a remote device during a video conference. Examples of such messaging include sending and receiving requests, notifications, and acknowledgments of the requests and notifications. Another example of messaging includes sending remote control instruction messages from one device to another. For example, the remote control operations described below can be performed by sending instructions from the local device to the remote device to remotely control the operation of the remote device via the local device's control communication channel (for example, instructing the device to send only images from a specific camera, or only Capture images with a specific camera). Different embodiments implement the control communication using different protocols, such as Real-time Transport Control Protocol (RTCP), RTP extensions, SIP, and the like. For example, some embodiments utilize RTP extensions to communicate one set of control messages between two mobile devices in a video conference, and use SIP packets to communicate another set of control messages between the two mobile devices during the video conference.

The VTP manager 3125 of some embodiments allows transmission of different types of data packets designated for transmission over different communication channels (e.g., using different pairs of port numbers) over a single communication channel (e.g., using the same pair of port numbers) . One technique for this purpose involves identifying the type of data packet, identifying the communication channel through which the data packet is intended to be transmitted by extracting a pair of designated port numbers of the data packet, and modifying the pair of port numbers of the data packet to the A pair of port numbers of the single communication channel is used to specify that data packets are to be transmitted through the single communication channel (ie, all data packets are transmitted through the same pair of port numbers).

To keep track of the pair of initial port numbers for each type of data packet, some embodiments maintain a mapping of the pair of initial port numbers relative to the data packet type. Some such embodiments then utilize the protocol's packet type field to distinguish between different packets that are multiplexed into one communication channel. For example, some embodiments with a VTP manager multiplex audio, video, and control packets into one RTP stream, using the RTP packet type field to distinguish the Audio, video and control grouping. In some of these embodiments, the VTP manager also routes control messages in SIP packets to the other device.

Some embodiments recognize checking data packet signatures (ie, packet header formats) to distinguish between different packets delivered using different protocols (eg, distinguishing packets delivered using RTP from packets delivered using SIP). In such an embodiment, after determining the data packets of different protocols, the fields of the data packets using the same protocol (eg, audio data and video data using RTP) are inspected to identify the different data types as described above. In this manner, VTP manager 3125 communicates over a single communication channel different data packets intended for transmission over different communication channels.

While the above illustrates one way of combining different types of data over a single communication channel, other embodiments utilize other techniques to multiplex different packet types into one communication stream. For example, one technique of some embodiments involves noting a pair of original port numbers for a data packet, and saving the pair of original port numbers in the data packet itself for later retrieval. There are other techniques that combine different types of data between two videoconference participants into one port-pair channel.

When VTP manager 3125 receives a data packet from a remote device over a virtual communication channel, VTP manager 3125 checks the signature of the data packet to identify a different packet sent using a different protocol. Such signatures can be used to distinguish SIP packets from RTP packets. The VTP manager of some embodiments also demultiplexes the various types of packets (eg, audio, video, and control packets) multiplexed into a single virtualized channel, using the packet type field of some or all of the packets. After identifying these different types of groups, the VTP manager associates each group of different types with its corresponding port pair number according to the mapping between the port pair number and the grouping type it keeps. The VTP 3125 then modifies the pair of port numbers of the data packet with the identified pair of port numbers and forwards the data packet for unpacking. In other embodiments where different packet types are multiplexed into a single channel using different techniques, the VTP manager parses the packets using different techniques.

By utilizing such techniques to multiplex and demultiplex the different packets, the VTP manager 3125 creates a single virtualized communication channel (e.g., a single pair of port numbers) over which the video data, audio data and control signaling data, and audio, video and control packets are received from a remote device over the single virtualized communication channel. Thus, from the point of view of the network, data is transferred through the single virtualized communication channel, while from the point of view of session negotiation manager 3105 and protocol manager 3110, video data, audio data and control signaling data are transmitted through different communication channels.

Images transmitted from a remote device in a video conference are received in packet format similar to the images transmitted to the remote device in the video conference. The receiver module 3130 receives packets and unpacks the packets to reconstruct the image, which is then stored in a video buffer (eg, buffer 1616 of FIG. 16 ) for decoding. In some embodiments, unpacking the image involves removing any headers, and reconstructing a bitstream with only the image data (and possibly size data) from the packets.

Media Delivery Manager 3135 processes feedback data received from the network (e.g., one-way latency, bandwidth estimate bit rate, packet loss data, round-trip delay time data, etc.) to dynamically and adaptively adjust the rate of data transmission ( i.e. bit rate). In some other embodiments, the media delivery manager 3135 also controls fault tolerance according to the processed feedback data, and can also send the feedback data to the video conference manager 1604 to adjust other operations of the video conference module 1602, such as scaling, size Adjust and code. In addition to causing the common transmit buffer to drop packets when the remote device in the conference cannot handle all the packets, the videoconferencing module and encoder can encode images using a lower bit rate so that fewer packets will be sent per image grouping.

In some embodiments, the media delivery manager 3135 may also monitor other variables of the device, such as power consumption and thermal levels that affect how the camera's operating power modes are configured, as described above. This data can also be used as additional input into the feedback data (eg, if the device is getting too hot, the Media Delivery Manager 3135 can try to slow down the process).

Referring now to FIG. 16, several example operations of the networking manager 3100 are illustrated. The transmission of images captured by the cameras of the dual-camera mobile device to the remote device in the video conference will be described first, followed by the description of receiving images from the remote device. The transmitter module 3115 retrieves from the buffer 1612 the encoded image to be transmitted to the remote device in the video conference.

The protocol manager 3110 determines the appropriate protocol to use (eg, RTP for conveying audio and video), and the session negotiation manager 3105 notifies the transmitter module 3115 of the protocol. Afterwards, the transmitter module 3115 packetizes the image, and sends the packetized image to the general transmission buffer 3120 . The general transmission buffer manager 3122 receives an instruction from the session negotiation manager 3105 to instruct the general transmission buffer 3120 to transmit or discard the image. VTP manager 3125 receives packets from general purpose transmit buffer 3120 and processes the packets for transmission to remote devices over a single communication channel.

When receiving images from a remote device, the VTP manager 3125 receives packetized images from the remote device over a virtualized single communication channel and processes the packets to pass through the communication channel (e.g., a video communication channel) assigned to receive the images, The images are directed to the receiver module 3130.

The receiver module 3130 unpacks the packets to reconstruct the image and sends the image to the buffer 1616 to be decoded by the decoder 1660 . The receiver module 3130 also forwards control signaling messages to the media delivery manager 3130 (eg, an acknowledgment of receipt of a packet from a remote device in a video conference).

Several example operations of networking manager 3100 are described above. These are merely illustrative examples, as various other embodiments will utilize different modules, or perform these or different operations with various functions distributed differently among the modules. Additionally, modules of networking manager 3100 or other modules may perform additional operations, such as dynamic bitrate adjustment.

IV. In-session adjustment and control operations

A. Picture-in-picture modification

1. Snap-to-corner

Some embodiments of the invention allow a user of a dual camera mobile device to modify the composite display displayed on the device by moving around one or more display areas that make up the composite display. One such example is moving the inset display area of the PIP display back and forth. Figure 32 illustrates such an example performed during a video conference. In a video conference, a user may want to move the foreground inset display area for various reasons, such as when the foreground inset display area blocks an area of interest in the background display area.

32 illustrates the movement of the insertion display area 3240 in the UI 3205 of the device with reference to five different stages 3210, 3215, 3220, 3225, and 3230 of the UI 3205. The first stage 3210 illustrates the UI 3205 during a video conference between a local user of the device and a remote user of the remote device. UI 3205 in FIG. 32 displays the same PIP display as that displayed in the fifth stage of FIG. 11 after starting the video conference. In this example, video captured by the local user's device is displayed in inset display area 3240 and video captured by the remote user's device is displayed in background display area 3235 . As shown, display area 1155 includes selectable UI item 3245 for ending the video conference. In some embodiments, the layout of the display area 1155 is the same as the layout of the display area 1155 of FIG. 12 described above.

The second stage 3215 illustrates that the user initiates the lock to corner operation by selecting the inset display area 3240 . In this example, selection is made by placing finger 3255 anywhere within inset display area 3240 . This selection is displayed with a thick border 3260 inserted into the display area 3240 as shown. Different embodiments may indicate such selection in different ways, such as by highlighting the display area 3240, by oscillating the display area 3240, and the like.

The third stage 3220 illustrates the UI 3205 after the user begins to move the inset display area 3240 of the PIP display 3250 from one area in the PIP display 3250 to another area in the PIP display 3250. In this example, inset display area 3240 has begun to move from the lower left corner of PIP display 3250 to the lower right corner of PIP display 3250, as indicated by arrow 3265. In this example, after selecting the inset display in the second stage 3215 , the user moves the inset display area 3240 by dragging his finger 3255 towards the lower right corner of the PIP display 3250 . Some embodiments provide other techniques for moving the inset display area 3240 around in the PIP display 3250 .

The fourth stage 3225 illustrates the UI 3205 in a state after the user removes his finger 3255 from the screen of the device 3200. In this state, the inset display area 3240 is still moving towards the lower right corner of the PIP display 3250 as recognized from the user's finger movement in the third stage 3220 . In other words, after the finger 3255 begins to move toward the bottom right corner of the PIP display 3250 into the display area 3240, even if the finger 3255 is removed, the UI 3205 maintains this movement. In order to maintain the movement, the UI 3205 of some embodiments requires the user's drag operation to be greater than a certain threshold amount (e.g., greater than a certain distance, or longer than a certain time) before the user removes their finger 3255; otherwise, these Embodiments may keep the inset display area 3240 in its original bottom left corner position after moving the inset display area 3240 slightly, or may not move the inset display area 3240 at all.

However, while some embodiments allow the inset display to continue moving even if the user stops his dragging operation before the inset display area reaches its new location, other embodiments require the user to keep dragging until the inset display area reaches its new location. its new location. Some embodiments provide other techniques for moving the inset display area. For example, some embodiments require the user to specify where to direct the inset display area 3240 before the inset display area 3240 actually begins to move, etc. Some embodiments also allow the display area to slide and lock into corners by simply tilting the mobile device at different angles.

The fifth stage 3230 illustrates the UI 3205 after the inset display area 3240 has reached its new location in the lower right corner of the PIP display 3250. In the fifth stage 3230, the removal of the thick border 3260 indicates that the lock to corner operation is complete.

To facilitate the movement illustrated in the third, fourth, and fifth stages 3220, 3225, and 3230 described above, the UI 3205 of some embodiments employs that once the user moves the inset display area toward a corner of the PIP display 3250, A snapping rule that allows the insert display to quickly snap to that corner. For example, when a user drags the inset display area 3240 toward a particular corner beyond a threshold amount, the UI 3205 of some embodiments identifies the direction of motion of the inset display area 3240, determines that the movement has exceeded a threshold amount, and then automatically moves the inset display area 3240 without further user input the next grid point in the UI 3205 to which the insertion display area 3240 can be locked. In some embodiments, the only grid points provided for locking inset display area 3240 are the grid points located at the four corners of PIP display 3250 . Other embodiments provide for other grid points in the UI 3205 (e.g., in the PIP display 3250) that the inset display area 3240 can be locked to (i.e., sides or vertices of the inset display area 3240 can be placed on or with it Aligned other grid points).

Alternative embodiments may not employ grid points such that the inset display area may be placed at any point within the PIP display 3250 . Still further embodiments provide a pin to grid point feature that allows the user to toggle the UI on or off. Additionally, various embodiments may allow users to perform pin-to-corner operations on various items, such as icons, in addition to video captured from the device.

FIG. 33 illustrates two other examples 3330 and 3335 of the lock to corner operation in the UI 3205. These additional pin to corner operations display inset display area 3240 that is moved vertically or diagonally in PIP display 3250 according to the user's vertical or diagonal drag operation.

Even though FIGS. 32 and 33 illustrate movement of inset display areas within a PIP display, one of ordinary skill will recognize that other embodiments may utilize similar techniques to move display areas in other types of PIP displays or other types of composite displays. For example, as described further below, the PIP display of some embodiments has two or more foreground inset displays that can be moved within the PIP display by utilizing techniques similar to those described above with reference to FIGS. 32 and 33 . Additionally, some embodiments utilize similar techniques to move display regions around in a composite display (eg, move a display region from the left side of the screen to the right side of the screen via a user drag movement). In addition, movement of one or more display areas of the composite display can result in changes to the image processing operations of the dual camera mobile device, such as causing the videoconferencing manager 1604 to recompose the display areas in the composite display in response to user input. As described further below, some embodiments employ a locking and pushing technique that pushes the first display area away from the first position when the second display area is moved from the third position to the first position.

2. Rotate

When a user of a mobile device used for a video conference rotates the mobile device during the conference, some embodiments rotate the PIP display presented during the video conference. FIG. 34 illustrates the rotation of the UI 1105 of the device 3400 when the device 3400 is rotated from a vertical position to a horizontal position. When the long side of the screen is vertical, the device 3400 is held vertically, and when the long side of the shield is horizontal, the device 3400 is held horizontally. In the example illustrated in FIG. 34 , UI 1105 rotates from a portrait view optimized for a vertical hold of the device to a landscape view optimized for a landscape hold of device 3400. This rotation functionality allows the user to view the UI 1105 displayed in an upright position when the mobile device 3400 is held vertically or horizontally.

FIG. 34 illustrates the rotation of the UI 1105 according to six different stages of operation 3410, 3415, 3420, 3425, 3430, and 3435. The first stage 3410 illustrates the UI 1105 during a video conference between a local user of the device and a remote user of the remote device. UI 1105 in FIG. 34 displays the same PIP display 1180 as that shown in the fifth stage of FIG. 11 after the video conference is established. In this example, the video captured by the local user's device is displayed in inset display area 1160 and the video captured by the remote user's device is displayed in background display area 1170 . Included in display area 1155 below PIP display 1180 is a selectable UI item 3485 (eg, "End Conference" button 3485) that the user can select to end the video conference (eg, by a single-finger tap).

The second stage 3415 illustrates the UI 1105 after the user begins to tilt the device 3400 sideways. In this example, the user begins to tilt device 3400 from being held vertically to being held horizontally, as indicated by arrow 3460 . The appearance of UI 1105 has not changed. In other cases, the user may instead want to tilt the device 3400 from being held horizontally to being held vertically, in which case the UI 1105 switches from a horizontally optimized view to a vertically optimized view.

The third stage 3420 illustrates the UI 1105 after the device 3400 has been tilted from being held vertically to being held horizontally. In this state, the appearance of UI 1105 remains unchanged. In some embodiments, the rotation operation is triggered after tilting the device 3400 more than a threshold amount and holding it for a period of time. In the example illustrated in FIG. 34, it is assumed that the threshold amount and rotation speed will not cause the UI 1105 to rotate until a short time interval after the device has been placed in the horizontal position. Different embodiments have different threshold amounts and wait times for triggering a rotation operation. For example, some embodiments may have such a low threshold for triggering a rotation operation that regardless of the orientation of the device 3400, the UI 1105 appears as if it is always displayed in an upright position. In other embodiments, a user of device 3400 may specify when a rotation operation may be triggered (eg, via a menu preference setting). Additionally, some embodiments may not delay the rotation after the device has been tilted beyond a threshold amount. Furthermore, different embodiments may allow the rotation operation to be triggered in different ways, such as by toggling a switch on the mobile device, by issuing a voice command, upon selection through a menu, and so on.

The fourth stage 3425 illustrates the UI 1105 after starting the rotation operation. Some embodiments animate the rotation of the display area to provide feedback to the user regarding the rotation operation. Figure 34 illustrates an example of such an animation. Specifically, Figure 34 in its fourth stage 3425 shows display areas 1180 and 1155 starting to rotate together. Display areas 1180 and 1155 rotate about axis 3465 (ie, the z-axis) that passes through the center of UI 1105. Display areas 1180 and 1155 are rotated by the same amount, but in a direction opposite to the rotation of device 3400 (eg, by tilting of device 3400 ). In this example, since device 3400 is rotated 90° clockwise (by going from vertical to horizontal holding), the rotating operation will cause display areas 1180 and 1155 to rotate 90° counterclockwise. When display areas 1180 and 1155 are rotated, display areas 1180 and 1155 are scaled down to fit UI 1105 so that display areas 1180 and 1155 can still fully appear on UI 1105. Some embodiments may provide a message indicating the status of the device 3400 (eg, by displaying the word "Rotating").

The fifth stage 3430 illustrates the UI 1105 after display areas 1180 and 1155 have been rotated 90° counterclockwise from portrait view to landscape view. At this stage, display areas 1180 and 1155 have been rotated, but not expanded to the full width of UI 1105. Arrow 3475 indicates that at the end of the fifth stage, display areas 1180 and 1155 will begin to expand sideways to fit the full width of UI 1105. Different embodiments may not include this stage, as the deployment may occur concurrently with the rotation in the fourth stage 3425 .

The sixth stage 3435 illustrates UI 1105 after display areas 1180 and 1155 have been expanded to occupy the entire display of UI 1105. As noted above, other embodiments may achieve this rotation differently. For some embodiments, the rotation operation may be triggered simply by rotating the screen of the device beyond a threshold amount, regardless of the orientation of the device 3400 .

Additionally, other embodiments may provide different animations for indicating rotation operations. The rotation operation performed in FIG. 34 involves rotation of display areas 1180 and 1155 about the center of UI 1105. Alternatively, the display areas may be individually rotated about the central axis of their respective display areas. One such method is shown in FIG. 35 . 35 shows an alternative method of animating the rotation of the display areas 1170 and 1160 of the PIP display 1180 of the UI 1105. The PIP display 1180 illustrated in FIG. 35 is the same as the PIP display 1180 illustrated in FIG. 11 .

FIG. 35 illustrates the rotation of the PIP display 1180 according to six different stages of operation 3410 , 3415 , 3420 , 3525 , 3530 , and 3535 . The operations of the first three stages of the UI 1105 are the same as those illustrated in the UI 1105 in FIG. 34 . In the third stage of Figures 34 and 35, the device 3500 has both been held from portrait to landscape, and the rotation of the UI 1105 has not yet started.

The fourth stage 3525 illustrates an alternative method of animating the rotation. In the fourth stage, the rotation operation has started. Specifically, fourth stage 3525 represents the beginning of rotation of display areas 1170 and 1160 . Display areas 1170 and 1160 rotate about axes 3567 and 3565 (ie, the z-axis), respectively, passing through the center of each display area. Display areas 1170 and 1160 are rotated by the same amount, but in a direction opposite to the rotation of device 3500 (eg, by tilting of device 3500). Similar to what was illustrated above in the fourth stage 3425 of FIG. 34 , since device 3500 is rotated 90° clockwise (by going from vertical to horizontal holding), the rotating operation will cause display areas 1170 and 1160 to rotate in a counterclockwise direction. Rotate 90° clockwise. When display areas 1170 and 1160 are rotated, display areas 1170 and 1160 are scaled down to fit UI 1105 so that display areas 1170 and 1160 can still fully appear on UI 1105.

The fifth stage 3530 illustrates the UI 1105 after both the display areas 1170 and 1160 have been rotated 90° counterclockwise from the portrait view to the landscape view. At this stage, display areas 1170 and 1160 have been rotated, but not expanded to the full width of UI 1105. Furthermore, display area 1160 has not been moved to its final position. The final location of the inset display area 1160 in the PIP display 1180 is determined by the position of the inset display area 1160 in the PIP display 1180 as shown in the first stage 3410 (e.g., the inset display area 1160 is in the lower left corner of the PIP display 1180) . At this stage, the inset display area 1160 is still in the upper left corner of the UI 1105.

Arrow 3580 indicates that at the end of fifth stage 3530, display areas 1170 and 1160 will begin to expand sideways until main display area 1170 fits the full width of UI 1105 of the device held horizontally. Additionally, arrow 3575 indicates that inset display area 1160 will slide to the lower left corner of PIP display 1180 .

Different embodiments may accomplish this differently. In some embodiments, the movement of the inset display area 1160 may occur simultaneously with the expansion of the main display area 1170, or may occur sequentially. Additionally, some embodiments may zoom inset display area 1160 before, during, or after expansion of main display area 1170 to generate new PIP display 1180 . In this example, display area 1155 disappears as display areas 1160 and 1170 rotate. However, in some embodiments, display area 1155 may remain on UI 1105 during rotation and rotate together with display areas 1160 and 1170.

The sixth stage 3535 illustrates the UI 1105 after the inset display area 1160 has reached its new location, and the display areas 1160 and 1170 have been properly expanded to fit the full width of the UI 1105. In this example, inset display area 1160 is now positioned in the lower left corner of PIP display 1180 , overlaid on main display area 1170 . The PIP display 1180 currently has the same display arrangement as the PIP display 1180 of the first stage 3410 . The display area 1155 that appears below the PIP display 1180 in the sixth stage indicates that the rotation operation is complete. As described above, the rotation operation may be triggered by merely rotating the screen of the device beyond a threshold amount, regardless of the orientation of the device 3500 .

In the example described above with reference to Figures 34 and 35, the orientation of the display area 1170 is also changed (ie, from portrait to landscape). That is, after rotating the display area 1170 in the third stage 3420, the orientation of the display area 1170 is changed from portrait to landscape by expanding the PIP display 1180 horizontally so that it fits the entire UI 1105. In some embodiments, when the device 3500 is rotated, the video captured by the remote device rotates, but the orientation of the display area displaying the video captured by the remote device remains the same. One such example is illustrated in Figure 36. Figure 36 is similar to Figure 35, except that the video displayed in display area 1170 is rotated, but display area 1170 remains displayed in portrait orientation.

Also illustrated in FIG. 36 is an example of a rotation operation in which the display area 1155 remains in the same position (rather than rotating and expanding horizontally to fill the PIP display 1180 as shown in FIG. 35 ). Furthermore, FIG. 36 includes the same layout of the display area 1155 as that of the display area 1155 explained above in FIG. 12 . As shown, in stages 3640, 3645, 3650, 3655, 3685, and 3690, when device 3500 is rotated, display area 1155 remains in the same position.

Some embodiments provide a rotate operation where the orientation of the display area showing video captured by the local device changes (instead of remaining at the same orientation as shown in FIG. 35 ) to reflect that the local Orientation of the device. FIG. 36 illustrates an example of such a rotation operation of the UI 1105 with reference to six different stages 3640 , 3645 , 3650 , 3655 , 3685 , and 3690 . In FIG. 36 , the first stage 3640 displays an inset display area 1160 displaying a video captured by a camera of the device 3500 in a portrait direction. The second and third stages 3645 and 3650 are the same as the second and third stages 3415 and 3420 of FIG. 35 in that they represent the tilting of the device 3500 at various stages of rotational operation. At this time, the camera of the device 3500 is capturing an image in a landscape orientation. To indicate this transition, some embodiments provide an animation as shown in the fourth and fifth stages 3655 and 3685, while other embodiments do not provide any animation at all.

In the fourth stage 3655, the image displayed in the inset display area 1160 is rotated, but the inset display area 1160 itself is not rotated because the tilt of the device 3500 in the second and third stages 3445 and 3650 has rotated the inset display area 1160 to landscape orientation. In the fifth stage 3685, the rotated image in the inset display area 1160 is expanded horizontally to fill the inset display area 1160, and the inset display area 1160 begins to move towards the lower left area of the PIP display 1180 to place the inset display area 1160 in and The insertion display area 1160 is in the same relative position as in the PIP display of the first stage 3640 .

In some embodiments, the orientation of the display area displaying video captured by the remote device also changes to reflect the orientation of the remote device after the remote device is rotated. 37 illustrates four different stages of the UI 1105 of the device 3500, where (1) the orientation of the display area (display area 1160 in this example) showing the video captured by the local device changes to reflect when the local device is rotated The orientation of the local device after the operation, and (2) the orientation of the display area displaying the video captured by the remote device (display area 1170 in this example) changes to reflect the orientation of the local device after the rotation operation of the remote device.

In the first stage 3705, the UI 1105 is the same as the UI 1105 in FIG. 36 . Specifically, first stage 3705 displays portrait-oriented display areas 1160 and 1170 because device 3500 is displayed in portrait orientation and the remote device is portrait-oriented (not shown). From the first stage 3705 to the second stage 3710, the local device is rotated by rotating the device 3500 90° from an upright position to a landscape position. The second stage 3710 displays the UI 1105 after completing the rotation operation of the device 3500. In the second stage, the video displayed in display areas 1170 and 1160 has been rotated to an upright position. However, only the locally captured video display area 1160 rotates from a portrait orientation to a landscape orientation because the rotation operation is only performed on the local device (ie, device 3500). Display area 1170 maintains a portrait orientation.

From the second stage 3710 to the third stage 3715, the remote device is rotated by rotating it from an upright position to a landscape position (not shown). The third stage 3715 displays the UI 1105 after completing the rotation operation of the remote device. In the third stage, the video displayed in the display area 1170 and the display area 1170 of the remotely captured video rotates from a portrait orientation to a landscape orientation because only the rotation operation is performed on the remote device. Thus, the third stage of UI 1105 displays display areas 1170 and 1160 for both local and remote capture video in landscape orientation.

From the third stage 3715 to the fourth stage 3720, the local device is rotated by rotating the device 3500 90° from a landscape position to an upright position. The fourth stage 3720 displays the UI 1105 after completing the rotation operation. In the fourth stage 3720, the video displayed in display areas 1160 and 1170 has been rotated to an upright position. However, only the locally captured video display area 1160 rotates from a landscape orientation to a portrait orientation because the rotation operation is only performed on the local device (ie, device 3500). Display area 1170 maintains a landscape orientation.

From the fourth stage 3720 to the first stage 3705, the remote device is rotated by rotating it 90° (not shown) from a landscape position to an upright position. In this case, the first stage 3705 displays the display area 1170 after the rotation operation is completed. Thus, UI 1105 at this stage displays portrait-oriented display areas 1160 and 1170. Although FIG. 37 illustrates a series of different rotation operations, other embodiments may perform any number of rotation operations in any number of different sequences.

Figures 34, 35, 36 and 37 illustrate the rotation of the local and remote devices during a video conference. When the local mobile device is rotated, some embodiments notify the remote device of the rotation so that the remote device can make any modifications to the local device's video (eg, rotate the display area displaying the local device's video). Similarly, when a rotation operation is performed on the remote device, the remote device notifies the local device of the operation, so that the local device can make any modification to the video of the remote device. Some embodiments provide a control communication channel for communicating notifications of rotation operations between a local device and a remote device during a video conference.

Although FIGS. 34, 35, 36, and 37 illustrate different ways in which the animation of rotation can be achieved, one of ordinary skill will recognize that other embodiments can display the animation of rotation in different ways. In addition, the animation of the rotation operation can cause changes in the image processing operations of the local mobile device, such as causing the video conference manager 1604 to recompose the display area at different angles in the UI 1105, and scale the image displayed in the display area.

3. Window resizing

Some embodiments allow a user of a mobile device to adjust the size of an inset display area of a PIP display presented during a video conference. Different embodiments provide different techniques for scaling the inset display area. Figure 38 illustrates one method of scaling the inset display area. In this method, the user of the mobile device adjusts the size of the inset display area by selecting a corner of the inset display area and subsequently expanding or shrinking the inset display area.

In FIG. 38, UI 3800 of mobile device 3825 presents PIP display 3865 during a video conference with a remote user of another mobile device. PIP display 3865 includes two video displays: background main display area 3830 and foreground inset display area 3835 . The background main display area 3830 occupies most of the PIP display 3865 , while the foreground inset display area 3835 is smaller and overlaps the background main display area 3830 . In this example, background main display area 3830 presents a video of a character holding a guitar, which is assumed to be the character whose video was captured by the remote device's front camera, or the remote device's rear camera. The foreground inset display area 3835 presents a video of a person wearing a hat, which in this example is assumed to be the person whose video was captured by the local device's front camera, or the person whose video was captured by the local device's back camera. Below the PIP display 3865 is the display area 1155 that includes a selectable UI item 3860 (e.g., button 3860) labeled "End Conference" that allows the user to end the video conference by selecting the item.

The PIP display 3865 is just a way of presenting a composite view of the video taken by the remote device and the local device. Some embodiments may provide other composite views. For example, instead of having a larger background display for the video from the remote device, the larger background display could be the video from the local device, and the smaller foreground inset display could be the video from the remote device. In addition, some embodiments allow local video and remote video to appear in two side-by-side display areas in UI 3800 (eg, display windows left and right, or display windows above and below), or in two display areas arranged diagonally. In some embodiments, the way of PIP display or the default display mode can be specified by the user. In other embodiments, the PIP display may also include one larger background display, and two smaller foreground inset displays.

FIG. 38 illustrates the zoom operation in terms of the four operational phases of the UI 3800. In the first stage 3805, the foreground inset display area 3835 is significantly smaller than the background main display area 3830. Also in this example, the foreground inset display area 3835 is located in the lower right corner of the PIP display 3865 . In other examples, foreground inset display area 3835 may be a different size, or located in a different area of PIP display 3865 .

In a second stage 3810, a scaling operation is initiated. In this example, the operation is initiated by selecting a corner of inset display area 3835 that the user wants to zoom (eg, by placing finger 3840 on the upper left corner of inset display area 3835). The second stage 3810 of the UI 3800 indicates this selection with a thick border 3845 inserted into the display area 3835. At this stage, the user can expand or shrink inset display area 3835 (eg, by dragging their finger 3840 on PIP display 3865 away from inset display area 3835 or toward inset display area 3835).

The third stage 3815 illustrates that, as indicated by arrow 3850, by moving his finger 3840 away from the inset display area 3835 (i.e., in this example, by moving his finger diagonally towards the upper left corner of the UI 3800), the user begins to expand the inset UI 3800 behind display area 3835. In addition, as indicated by arrow 3855, the movement of finger 3840 expands insertion display area 3835 proportionally in height and width directions. In other examples, the user can zoom out of inset display area 3835 using the same technique (ie, by dragging a finger toward inset display area 3835).

The fourth stage 3820 displays the UI 3800 after zooming inset display area 3835 has completed. In this example, once the inset display area 3835 has reached the desired size, the user completes the zooming of the inset display area 3835 by stopping the dragging of his finger 3840 and removing his finger from the PIP display 3865. As a result of this process, the scaled inset display area 3835 is larger than its original size in the first stage 3805 . The removal of the thick border 3845 indicates that the inset display area scaling operation is now complete.

Some embodiments provide other techniques that allow a user to zoom inset display area 3835 in PIP display 3865 during a video conference. Figure 39 illustrates one such other technique. FIG. 39 illustrates a technique for zooming inset display area 3835 by selecting an edge of inset display area 3835 (ie, on one of the sides of inset display area 3835 ), and then expanding or shrinking inset display area 3835 .

FIG. 39 illustrates the zooming operation in terms of the four operational phases of the UI 3800 of FIG. 38. The first stage 3805 of FIG. 39 is the same as the first stage 3805 of FIG. 38 . Specifically, at this stage, UI 3800 of device 3925 illustrates PIP display 3865 with a larger background main display area 3830 and a smaller foreground inset display area 3835 located in the lower right corner of PIP display 3865. Although FIGS. 38 and 39 illustrate two different techniques for zooming inset display area 3835 in the same UI 3800, one of ordinary skill will recognize that some embodiments will not provide both techniques in the same UI.

The second stage 3910 illustrates the beginning of the scaling operation. In this example, the user initiates a zoom operation by selecting a side of inset display area 3835 that the user wants to zoom (eg, placing finger 3840 on the top or side edge of inset display area 3835). In this example, the user places his finger 3840 on the top edge of the inset display area 3835 to effect the selection. The second stage 3910 indicates this selection with a thick border 3845 inserted into the display area 3835 .

The third stage 3915 illustrates the UI 3800 after the user has begun to expand the inset display area 3835 by moving its finger 3840 away from the inset display area 3835 (i.e., moving vertically toward the top of the PIP display 3865), as indicated by arrow 3950 . Additionally, as indicated by arrow 3955, the movement of finger 3840 expands inset display area 3835 proportionally in both height and width directions. In other examples, the user zooms out of display area 3835 by utilizing the same technique (ie, by dragging finger 3840 toward inset display area 3835).

The fourth stage 3920 displays the UI 3800 after zooming into the display area 3835 is complete. In this example, once the inset display area 3835 has reached the desired size, the user completes the zooming of the inset display area 3835 by stopping the dragging of his finger 3840 and removing his finger 3840 from the device's display screen. As a result of this process, the scaled inset display area 3835 is larger than its original size in the first stage 3805 . The removal of the thick border 3845 indicates that the inset display area scaling operation is now complete.

In response to a drag operation, some embodiments resize the inset display area 3835 proportionally in height and width, as shown in FIGS. 38 and 39 . Other embodiments may allow the user to adjust the height and/or width of the inset display area 3835 without affecting another attribute. Figure 40 illustrates an example of one such scaling process.

Specifically, FIG. 40 illustrates a UI 3800 for a mobile device 4025 that is similar to the UI 3800 of FIG. 38, except that the UI 3800 of FIG. The display area 3835 expands horizontally and/or vertically. 40 illustrates a PIP display 3865 in UI 3800 that is similar to PIP display 3865 of FIG. PIP display 3865 includes two video displays: background main display area 3830 and foreground inset display area 3835 . In this example, background main display area 3830 presents video captured by the remote device's front or back camera. The foreground inset display area 3835 presents video captured by the local device's front camera or background camera.

Similar to FIG. 38 , FIG. 40 illustrates the zoom operation in terms of the four operational phases of the UI 3800. The first stage 4005 is similar to the first stage 3805 of Figure 38, except now the inset display area 3835 is in the upper right corner. The other three stages 4010, 4015, and 4020 are similar to the three stages 3910, 3915, and 3920, except that selection and movement of the bottom edge of the inset display area 3835 only causes the inset display area 3835 to expand vertically without affecting the inset display area 3835 outside of the width.

38 , 39 and 40 provide example embodiments that allow a user to zoom inset display area 3835 of PIP display 3865 by selecting a corner or side of inset display area 3835 . Some embodiments provide other techniques for scaling the inset window 3835. For example, FIG. 41 illustrates that some embodiments allow the inset display area 3835 to be zoomed by selecting the interior of the inset display area 3835 . In this method, the user resizes the inset display area 3835 by placing two fingers 4155 and 4156 on the screen and dragging the two fingers away from or close to each other.

In FIG. 41 , UI 3800 of mobile device 4140 provides PIP display 3865 during a video conference with a remote user of another mobile device. To simplify the description of the UI 3800, FIG. 41 illustrates a PIP display 3865 in the UI 3800 similar to the PIP display 3865 of FIG. 38.

Figure 41 illustrates the zooming operation in terms of the seven operational phases of the UI 3800. The first four stages 3805, 4110, 4115, and 4120 represent the enlargement of the inset display area 3835, while the last three stages represent the reduction of the inset display area 3835. The first stage 3805 in FIG. 41 is the same as the first stage 3805 in FIG. 38 . Specifically, at this stage, UI 3800 illustrates a PIP display 3865 with a larger background main display area 3830 and a smaller foreground inset display area 3835. In this example, background main display area 3830 presents video captured by the remote device's front or back camera. The foreground inset display area 3835 presents video captured by the local device's front or back camera.

The second stage 4110 illustrates the UI 3800 after initiating a zoom operation. In this example, the user initiates a zoom operation by selecting the inset display area 3835 that the user wants to zoom (eg, by placing two fingers 4155 and 4156 within the inset display area 3835). The second stage 4110 of the UI 3800 indicates this selection with a thick border 4190 inserted into the display area 3835.

The third stage 4115 illustrates that, as indicated by arrow 4160, the user moves his fingers 4155 and 4156 away from each other (i.e., moves finger 4155 toward the upper left corner of PIP display 3865, moves finger 4156 toward the lower right corner of PIP display 3865). ), begin to expand the UI 3800 after inserting the display area 3835. As indicated by arrow 4165, the movement of fingers 4155 and 4156 expands insertion display area 3835 proportionally in the height and width directions.

The fourth stage 4120 displays the UI 3800 after zooming into the display area 3835 is complete. In this example, the user completes the zooming of the inset display area 3835 by stopping the dragging of their fingers 4155 and 4156, and removing their fingers 4155 and 4156 from the display screen of the device. As a result of this process, the scaled inset display area 3835 is larger than its original size in the first stage 3805 . The removal of the thick border 4190 indicates that the inset display area scaling operation is now complete.

In the fifth stage 4125, the user reselects the insertion display area 3835 by placing two fingers 4155 and 4156 on the insertion display area 3835. The sixth stage 4130 illustrates the UI 3800 after the user begins zooming out of the inset display area 3835 by moving their fingers 4155 and 4156 in close proximity to each other, as indicated by arrow 4170. As indicated by arrow 4175, the movement of fingers 4155 and 4156 scales down inset display area 3835 in both height and width directions.

The seventh stage 4135 is similar to the fourth stage 4120 of FIG. 41 , except that by the operation described, the size of the inset display area 3835 has been reduced. The removal of the thick border 4190 indicates that the inset display area scaling operation is now complete.

The above descriptions of FIGS. 38-41 illustrate several example user interfaces that allow a user to zoom the inset display area of a PIP display. In some embodiments, scaling of the inset display area causes changes in the image processing operations of the dual camera mobile device, such as causing the video conference manager 1604 to change the scaling and compositing of the inset display area in a PIP display in response to user input. Additionally, in some embodiments, the layout of the display area 1155 in FIGS. 38-41 is the same as the layout of the display area 1155 of FIG. 12 described above.

4. Identify the region of interest

Some embodiments allow a user to identify regions of interest (ROIs) in displayed video in order to modify image processing (e.g., image processing manager 1608 in FIG. 16 ), encoding (e.g., encoding device 1655), the behavior of the mobile device and its camera during the video conference, or a combination thereof. Different embodiments provide different techniques to identify such regions of interest in video. Figure 42 illustrates a user interface of some embodiments for identifying regions of interest of a video in order to improve the image quality of the video.

In FIG. 42, UI 4200 of mobile device 4225 presents PIP display 4265 during a video conference with a remote user of another mobile device. The PIP display in FIG. 42 is basically similar to the PIP display in FIG. 41 . Specifically, the PIP display in FIG. 42 includes two video displays: a background main display 4230 and a foreground inset display 4235. In this example, the background main display 4230 presents a video of a tree and a figure wearing a hat, which is assumed to be the tree and figure whose video was captured by the remote device's front camera, or the remote device's rear camera Its videos of trees and people. The foreground inset display 4235 presents a video of a man, which in this example is assumed to be the man whose video was captured by the local device's front camera, or the person whose video was captured by the local device's back camera. Below the PIP display is a display area 1155 that includes a selectable UI item 4260 (e.g., button 4260) labeled "End Conference" that allows the user to end the video conference by selecting the item.

This PIP display is just one way of presenting a composite view of the video captured by the remote device and the local device. Some embodiments may provide other composite views. For example, instead of having a larger background display for video from a remote device, the larger background display could be the video from the local device and the smaller foreground inset display could be the video from the remote device. In addition, some embodiments allow local video and remote video to appear in two side-by-side display areas in the UI (eg, display windows left and right, or display windows above and below), or in two diagonally arranged display areas. In other embodiments, the PIP display may also include one larger background display, and two smaller foreground inset displays. In some embodiments, the way of PIP display or the default display mode can be specified by the user.

Figure 42 illustrates the ROI identification operation according to the four operational phases of the UI 4200. As shown in the first stage 4205, the video presented in the background display 4230 is of very low quality (ie, the video image is blurry). In this example, the user of the mobile device 4225 intends to identify the area in the background display 4230 where the face 4270 of the person appears as an area of interest.

In a second stage 4210, an operation to identify a region of interest is initiated. In this example, by selecting an area of the video presented in background display 4230 that the user wants to identify as an area of interest (e.g., by displaying a character's face 4270 in background display 4230 on the device's screen Tap your finger 4250) to initiate this operation.

As shown in the third stage 4215, the user's selection of an area causes the UI 4200 to draw a surrounding box 4275 (e.g., a dashed box 4275) around the area selected by the user. The fourth stage 4220 displays the UI 4200 after concluding identification of the region of interest. As a result of this process, the quality of the video within the region of interest has improved significantly compared to the first stage 4205 . The elimination of surrounding box 4275 indicates that the ROI selection operation is now complete. In some embodiments, the ROI identification process also causes the same changes to the same video displayed on the remote device as it does to the local device 4225. For example, in this example, the picture quality in the region of interest of the same video displayed on the remote device is also significantly improved.

In some embodiments, the user may zoom in or out on the surrounding frame 4275 in the third stage 4215 (e.g., by placing the finger 4250 on the display and moving the finger 4250 toward the upper right corner of the screen to zoom in on the surrounding frame 4275, or Move the finger 4250 towards the lower left corner of the screen to shrink the surrounding frame 4275). Some embodiments also allow the user to rotate the surrounding frame 4275 in the third stage 4215 (eg, by placing the finger 4250 on the display and moving the finger 4250 horizontally or vertically on the display). In some other embodiments, the selection of the area does not cause the UI 4200 to draw the surrounding box 4275 in the third stage 4215 at all.

Other embodiments provide different techniques that allow a user to identify regions of interest in a video. Figure 43 illustrates one such other technique. In FIG. 43, the user identifies the region of interest by drawing a shape enclosing the region of interest. In this example, the shape is a rectangle, but could be other shapes (eg, any other polygon, circle, ellipse, etc.). Some embodiments provide the alternative technique of FIG. 43 in a device UI that also provides the technique illustrated in FIG. 42 . However, other embodiments do not provide these two technologies in the same UI.

Figure 43 illustrates this ROI identification operation in terms of the five operational phases of the UI 4200. The first stage 4205 in FIG. 43 is the same as the first stage 4205 in FIG. 42 . Specifically, in a first stage 4205, the UI 4200 illustrates a PIP display 4265 with a larger background main display 4230, and a smaller foreground inset display 4235 located in the lower left corner of the PIP display 4265.

In a second stage 4310, an operation to identify a region of interest is initiated. In this example, a first location defining an area of interest in a video presented in the background main display 4230 is selected by selecting for a period of time (e.g., by holding a finger 4350 on the screen of the device for a period of time on the background display 4230 in the vicinity of the displayed character's face 4270), initiate the operation. In a third stage 4315, the UI 4200 indicates by a dot 4355 adjacent to the selected first location on the background display area 4230 that said first location 4370 has been selected.

The fourth stage 4320 illustrates the UI 4200 after the user has selected a second location 4375 defining an area of interest. In this example, by dragging the finger 4350 within the screen of the device starting from the first position after the dot 4355 appears as shown by the arrow 4360, and stopping at the displayed hat and the displayed hat located in the background display area 430 Between locations in the tree, the user selects a second location 4375. As shown in the fourth stage, this dragging causes the UI 4200 to draw a rectangular bounding box 4365 of the region of interest with first and second positions 4370 and 4375 on its opposing vertices.

The fifth stage 4325 illustrates the UI 4200 after the identification of the region of interest has been completed. In this example, the user completes the identification of the region of interest by stopping the dragging of the finger 4350 and removing the finger 4350 from the display screen of the device once the desired region of interest is identified. The fifth stage 4325 illustrates that through this drawing process, the quality of the video within the region of interest has improved significantly compared to the first stage 4205 . In some embodiments, this dragging process causes the same changes to the display on the remote device as it does to the local device 4225. For example, in this example, the image quality in the region of interest of the same video displayed on the remote device will be significantly improved.

The description of Figures 42 and 43 above exemplifies different ways of identifying regions of interest in a video in order to improve the image quality of the identified regions. In some embodiments, improving the picture quality of identified regions of interest results in changes to the encoding operations of the dual camera mobile device, such as allocating more bits to the identified regions when encoding video.

Some embodiments allow a user to identify areas of interest in a video to make different changes to the mobile device or its camera. For example, FIG. 44 illustrates an example of identifying a region of interest in a video to enlarge or reduce the region of interest on a display. In this method, a user identifies an area of interest in a video by selecting an area on the display as the center of the area of interest and then enlarging or reducing the area of interest.

In FIG. 44, UI 4400 of mobile device 4425 presents PIP display 4265 during a video conference with a remote user of another mobile device. PIP display 4265 in FIG. 44 is substantially similar to PIP display 4265 of FIG. 42, except that foreground inset display 4235 of FIG.

FIG. 44 illustrates the ROI selection operation in terms of the four operational phases of the UI 4400. As shown in the first stage 4405, the background display 4430 presents a video with a man on the left side of the background display 4430, and a tree 4440 on the right side of the background display 4430. Additionally, the tree 4440 is relatively small, occupying only the right side of the background display area 4430 . In this example, the user of mobile device 4425 intends to identify the area on background display area 4430 where tree 4440 appears as an area of interest.

In a second stage 4410, an operation to identify a region of interest is initiated. In this example, by selecting an area 4440 of the video presented in background display 4430 that the user wishes to identify as an area of interest (e.g., by placing two fingers 4445 and 4446 on background display area 4430, displaying a tree 4440), initiate the operation. In a second stage 4410 , by dragging their fingers 4445 and 4446 away from each other, the user can cause the region of interest 4440 to expand and occupy a larger portion of the background display area 4430 . The user can also zoom out the region of interest 4440 by dragging their fingers 4445 and 4446 close to each other so as to occupy a smaller portion of the background display area 4430 .

The third stage 4415 illustrates that, as shown by arrow 4450, the user moves his fingers 4445 and 4446 away from each other (i.e., finger 4445 moves towards the upper left corner of the background display area 4430 and finger 4446 towards the right of the background display area 4430). Move the lower corner), start to expand the ROI 4440 to occupy a larger portion of the UI 4400 behind the background display area 4430 . In some embodiments, finger movement also causes the same changes to the remote device's display as it does to the local device. For example, in this example, the region of interest for the same video would expand to occupy a larger portion of the background display area 4430 of the remote device. In some embodiments, enlargement of the region of interest in the local display and/or the remote display causes one or both mobile devices or their cameras to modify one or more of their other operations, as described further below.

The fourth stage 4420 displays the UI 4400 after the identification of the region of interest has been completed. In this example, once the region of interest has reached the desired proportions in the background display area 4430, the user stops the dragging of the user's fingers 4445 and 4446, and removes the fingers 4445 and 4446 from the display screen of the device. Complete the identification of the region of interest. As a result of this process, the region of interest occupies most of the background display area 4430 . The identification of the region of interest is now complete.

Some of the examples above illustrate how a user can identify a region of interest in a video to improve image quality within the selected region of interest in the video (eg, by increasing the bitrate for encoding the region of interest in the video). In some embodiments, identifying the region of interest in the video results in changes in the mobile device's image processing operations, such as exposure, scaling, focus, and the like. For example, identifying an area of interest in a video may cause videoconferencing manager 1604 to scale and composite images of the video differently (eg, identifying an area of interest to zoom in).

In other embodiments, identifying the region of interest in the video results in a change in the operation of the mobile device's camera (eg, frame rate, zoom, exposure, scaling, focus, etc.). In yet other embodiments, identifying regions of interest in the video results in changes to the mobile device's encoding operations, such as allocating more bits to the identified regions, scaling, and the like. Additionally, while the example ROI identification operations described above cause only one of the aforementioned modifications to the mobile device or its camera, in some other embodiments the ROI identification operation causes more than one of the above modifications to the operation of the mobile device or its camera. A kind of modification. Additionally, in some embodiments, the layout of the display area 1155 of FIGS. 42-44 is the same as the layout of the display area 1155 of FIG. 12 described above.

B. Switch camera

Some embodiments provide a method of switching cameras (ie, changing which camera captures an image) during a video conference. Different embodiments provide different methods of implementing switching camera operations. Some embodiments provide a method performed by a dual-camera mobile device for switching the device's camera (i.e., local switching), while other embodiments provide for a dual-camera mobile device to instruct another dual-camera mobile device in a video conference to switch the other camera. A method for a device's camera (i.e., remote switching). Other embodiments besides this provide methods for both local handover and remote handover. Section IV.B.1 will describe the handling of switching camera operations locally on a dual-camera mobile device. Section IV.B.2 will describe the handling of remote switching camera operations on a dual-camera mobile device.

1. Switch camera locally

Figure 45 illustrates a process 4500 performed by some embodiments at a local dual camera mobile device to switch between the local device's two cameras during a video conference with a remote mobile device that includes at least one camera. In some embodiments, process 4500 is performed by video conference manager 1604 shown in FIG. 16 . For purposes of illustration, the following discussion will refer to one camera of the local dual camera mobile device as camera 1 and the other camera of the local dual camera mobile device as camera 2 .

Process 4500 begins by initiating (at 4505) a video conference between the local dual camera mobile device and the remote mobile device. Process 4500 then sends (at 4510 ) the video image from the currently selected camera (eg, Camera 1 ) of the local dual camera mobile device to the remote mobile device for display on the remote mobile device. At 4510, process 4500 also generates and displays a composite image based on the video image and the video image it received from the remote mobile device.

Process 4500 then determines (at 4515) whether a request to end the video conference has been received. As noted above, in some embodiments, the local dual-camera mobile device user's request (e.g., through the user interface of the local dual-camera mobile device), or at the request of the user of the remote mobile device (e.g., via a remote mobile device user interface), end the video conference. Process 4500 ends when process 4500 receives a request to end the video conference.

When process 4500 does not receive a request to end the video conference, process 4500 then determines (at 4520) whether the user of the local dual camera mobile device has instructed the local device to switch cameras for the video conference. When process 4500 determines (at 4520) that the local device has not been instructed to switch cameras, process 4500 returns to operation 4510. However, when process 4500 determines (at 4520) that the local device has been instructed to switch cameras, process 4500 proceeds to operation 4525.

At 4525, process 4500 sends a notification to the remote mobile device indicating that the local dual camera mobile device is to switch cameras. In some embodiments, process 4500 sends the notification over a videoconferencing control channel multiplexed by VTP manager 3125 along with audio and video channels as described above.

After sending its notification, process 4500 proceeds (at 4530) to switch camera operations. In some embodiments, performing (at 4530) a switch camera operation includes instructing the CIPU to stop capturing video images with camera 1 and begin capturing video images with camera 2. These instructions may simply instruct the CIPU to switch to capturing images from the pixel array associated with camera 2, and to begin processing these images. On the other hand, in some embodiments, the instructions to the CIPU may be accompanied by a set of initialization parameters that instruct the CIPU to: (1) operate camera 2 according to a particular set of settings, (2) operate at a particular frame The rate captures video produced by camera 2, and/or (3) processes video images from camera 2 according to a particular set of settings (eg, resolution, etc.).

In some embodiments, the instruction to switch cameras (at 4530) also includes instructions to switch unused cameras to a fourth operating power mode as described above. In this example, the switch camera command includes a command to camera 2 to switch to its fourth operating power mode. In addition, the switching camera instruction also includes switching from its fourth working power mode to another working power mode, such as the first working power mode, for the camera 1 to save power, or switching to the third working power mode , so that when it is required to take an image, it can quickly switch to the fourth working power mode and start to take the instruction of the image. Switch camera operation 4530 also involves compositing images captured by camera 2 of the local dual camera mobile device (rather than images captured by camera 1 ) and images received from the remote mobile device for display on the dual camera mobile device.

After commanding a switch camera at 4530, process 4500 performs (at 4535) a switch camera animation on the native dual camera mobile device to show the transition between the display of the image from camera 1 and the display of the image from camera 2. Following the switch camera animation on the native dual camera mobile device, process 4500 loops through operations 4510-4520 until an end video conference request or a new switch camera request is received.

Figure 46 illustrates how some embodiments allow a switch camera operation to be requested through the UI 1105 of a dual camera mobile device, and an example of how these embodiments animate the switch camera operation. 46 illustrates switching camera operations in terms of eight different operational stages 4610, 4615, 4620, 4625, 4630, 4635, 4640, and 4645 of the UI 1105 of the device. The first four stages of UI 1105 illustrate an example of receiving a user's request to switch cameras. In some embodiments of the invention, the user of the device has other mechanisms for generating such requests.

The first stage 4610 is the same as the first stage 1130 of the UI 1105 of FIG. 11, which represents the UI 1105 after the video conference is established. At this stage, the UI 1105 displays a PIP display that includes two video displays: a larger background display from the remote camera, and a smaller foreground inset display from the local camera. In this example, the background main display area 1170 presents a video of a woman, who in this example is assumed to be the woman whose video was captured by the remote device, while the foreground inset display area 1160 presents a video of a man, In this example, the man is assumed to be the man whose video was captured by the local device's front camera.

The second stage 4615 then displays selection via the PIP display area 1180 of the UI 1105 to begin switching camera operations. The selection is made by placing the user's finger 4670 on the PIP display 1180 as shown. The third stage 4620 displays the UI 1105 including a selectable UI item 4675 (e.g., switch camera button 4675) for requesting to switch between cameras of the local device 4600 during the video conference. The fourth stage 4625 illustrates the UI 1105 after the user of the local device 4600 selects (e.g., by single-finger tapping) the selectable UI item 4675, and after such selection is indicated by highlighting of the selectable UI item 4675. By selecting this selectable UI item 4675, the user instructs the device 4600 to switch from the front camera of the device 4600 to the back camera of the device 4600 during the video conference. In other examples where the back camera of device 4600 captures video, user selection of selectable UI item 4675 would instruct device 4600 to switch from the back camera of device 4600 to the front camera of device 4600. After the fourth phase, the videoconferencing manager sends an instruction to the CIPU, and the remote device starts switching camera operations.

The last four stages 4630, 4635, 4640, and 4645 of the UI 1105 illustrate examples of switching camera animations on the local device. The animation is intended to give the impression that video captured from the local device's front and back cameras is displayed concurrently on both the front and back sides of the viewing pane, which at any one time only allows the user to see its front and back sides. One of two sides. When a camera switch is requested in the middle of a videoconference, rotate the viewing pane as if about a vertical axis such that a video from a camera presented on a side of the viewing pane that was previously showing video from a camera to the user rotates away from the user , until it is replaced by the other side of the viewing pane, which shows video from another camera. This perceived rotation animation and phenomenon of the viewing pane is achieved by (1) zooming out, in the display area for a camera, of the video image from that camera and applying a perspective correction operation to said video image, Then (2) in the display area, gradually enlarge the video image from the other camera and reduce the effect of the perspective correction operation on the video image.

Thus, the fifth stage 4630 illustrates the initiation of a “rotation of the viewing pane” about the vertical axis 4682 . To give the viewing pane a rotation phenomenon, the UI 1105 reduces the size of the video image of the front camera in the video display area 1160 and applies a perspective operation so that the right side of the video image appears smaller than the left side of the video image The side is further away from the user.

The sixth stage 4635 illustrates that the viewing pane has been rotated 90° such that the user can only see the edges of the pane, as shown by the thin line 4686 displayed in the center of the display area 1160 . The seventh stage 4640 illustrates that the viewing pane continues to rotate such that the back of the viewing pane 4688 now gradually appears in front of the user to display the video captured from the user's back camera. Also, in some embodiments, the rotation animation is performed by reducing the size of the video image of the rear camera in the video display area 4688 and applying a perspective operation so that the video image is larger than the right side of the video image. The left side is further away from the user to achieve.

The eighth stage 4645 illustrates the completion of the animation representing the switch camera operation. Specifically, this stage displays in the display area 1160 a video image of the car captured by the rear camera of the device 4600 .

The example described above with reference to FIG. 46 invokes the switch camera operation through the switch camera user interface. Other embodiments invoke the switch camera operation differently. For example, some embodiments invoke the toggle camera operation by having the toggle camera selectable UI item permanently displayed on a UI during a video conference, such as UI 1105 of FIG. 47 . In FIG. 47 , a switch camera button 1289 is displayed in the display area 1155 together with a mute button 1285 and an end meeting button 1287 . The layout of the display area 1155 is the same as that of the display area 1155 explained above with reference to FIG. 12 .

Figure 47 illustrates the switch camera operation of the UI 1105 in terms of six phases: 4610, 4790, 4630, 4635, 4640, and 4645. The first stage 4610 of FIG. 47 is similar to the first stage 4610 of FIG. 46, except that the layout of the display area 1155 shows a mute button 1285, an end meeting button 1287, and a switch camera button 1289 instead of a single end meeting button. The second stage 4790 illustrates the UI 1105 after a user selection of the local device 4600 (eg, by a single-finger tap with the finger 4670) toggles the camera selectable UI item 1289. In this example, by selecting selectable UI item 1289, the user instructs device 4600 to switch from the front camera of device 4600 to the back camera of device 4600 during the video conference. The last four stages of FIG. 47 are similar to the last four stages of FIG. 46, except that the layout of the display area 1155 is the same as that described above in the first stage 4610, so no further description is given to avoid using unnecessary The details obscure the description of the invention.

In some embodiments, when the remote mobile device receives an image from a different camera of the local dual-camera mobile device (i.e., the local dual-camera mobile device switches cameras), the remote mobile device also performs a switch camera animation to display Camera Transition between the display of an image from one camera of a mobile device and the display of an image from the other camera of a native dual camera mobile device. FIG. 48 illustrates an example of one of such switching camera animations in terms of five operational stages 4810, 4815, 4820, 4825, and 4830 of the UI 4805. FIG. 48 shows an example switching camera animation on a remote mobile device 4800. The various stages of operation are the same as the example animation of FIG. 46 , except that the images displayed in display area 4835 , which is where images from the local dual camera mobile device are displayed on remote mobile device 4800 , are animated. Thus, the image of the man displayed in the display area 4835 is animated as if rotated 180° on the vertical axis 4855 centrally located in the display area 4850 to represent the display of the image of the man in the display area 4835 and the image of the car 4870 to switch between displays. The implementation of the switching camera animation in some embodiments is the same as the animation described above.

The above example illustrates a switching camera animation on a remote device with a specific UI layout. Other embodiments may implement this switching camera animation on remote devices with different user interface layouts. For example, FIG. 49 illustrates one such example of a remote device 4800 having a different user interface layout 4805. In particular, the UI 4805 of FIG. 49 has a mute button 1285, an end meeting button 1287, and a switch camera button 1289 included in the display area 1155 that is permanently displayed to the side of the composite display 4850 during a video conference. . The layout of these three buttons is described above with reference to FIG. 48 . The five stages 4810, 4815, 4820, 4825, and 4830 of FIG. 49 are identical to the five stages 4810, 4815, 4820, 4825, and 4830 of FIG. 48, except for a different user interface layout.

2. Switch cameras remotely

FIG. 50 illustrates a process 5000 of switching the two cameras of a remote dual camera device during a video conference. Process 5000 is performed by a videoconferencing manager of a device that includes at least one camera. In the following discussion, a device through which a user commands to remotely switch cameras is referred to as a local device, and a device that switches its two cameras is referred to as a remote device. Also, in the following discussion, the remote device is considered to be switching between its front camera (or camera 1 ) and its back camera (or camera 2 ).

Referring now to Figures 51, 52, 53 and 54, the process 5000 of Figure 50 will be described. Figure 51 illustrates the UI 5105 of a local device 5100 through which a user requests a remote device to switch between its two cameras during a video conference. FIG. 51 illustrates eight different operational stages 5110, 5115, 5120, 5125, 5130, 5135, 5140, and 5145 of the UI 5105. 54 illustrates a UI 5405 of a remote device 5400 receiving a switch camera request from a local device 5100. FIG. 54 illustrates six different operational stages 5410, 5415, 5420, 5425, 5430, and 5435 of the UI 5405.

As shown in Figure 50, process 5000 begins by initiating (at 5005) a video conference between a local device and a remote device. Process 5000 then receives (at 5010) images from one camera of each device (eg, from each device's front camera) from which a composite image for the videoconference is generated. At 5010, process 5000 also sends video images from the local device to the remote device.

Process 5000 then determines (at 5015) whether a request to end the video conference has been received. As noted above, in some embodiments, a video conference may end at the request of a user of a local or remote device. Process 5000 ends when process 5000 receives a request to end the video conference.

When process 5000 does not receive a request to end the videoconference, process 5000 then determines (at 5020) whether the user of the device on which process 5000 is executing (i.e., the user of the local device) has instructed the device to request that the remote device Switch between cameras for video conferencing. When process 5000 determines (at 5020 ) that it has not been instructed to initiate a remote switch camera, process 5000 returns to operation 5010 . When process 5000 determines (at 5020) that it has been instructed to initiate a remote switch camera, process 5000 proceeds to operation 5025, described further below.

The first four stages 5110, 5115, 5120, and 5125 of the UI 5105 of FIG. 51 illustrate an example of receiving a user's request to switch the camera of a remote device. The first and second stages 5100 and 5115 are the same as the first and second stages 4610 and 4615 of FIG. 46 . The third stage 5120 is the same as the third stage 4620, except that the third stage 5120 includes not only an optional UI item 5175 requesting the local device 5100 to switch cameras, but also an optional UI item 5180 requesting the remote device 5100 to switch cameras. The fourth stage 5125 illustrates the user of the local device 5100 selecting a UI item 5180 requesting the remote device to switch cameras (eg, via a single finger tap 5170 of the optional UI item 5180). The selection is indicated by highlighting selectable UI item 5180 . Figure 51 shows one example of doing this, although other embodiments may do it differently to request the remote device to switch cameras.

The example described above with reference to FIG. 51 invokes the remote switch camera operation through the remote switch camera user interface. Other embodiments invoke the remote switch camera operation differently. For example, some embodiments invoke the toggle camera operation by having the toggle camera selectable UI item permanently displayed on a UI during a video conference, such as UI 5105 of FIG. 52 . In FIG. 52 , a remote switch camera button 5288 is displayed in the display area 1155 along with a mute button 5282 , an end meeting button 5284 and a local switch camera button 5286 .

Figure 52 illustrates the remote switch camera operation of the UI 5105 of the device 5100 in six different stages 5110, 5290, 5130, 5135, 5140, and 5145. The first stage 5110 of FIG. 52 is similar to the first stage 5110 of FIG. 51 , except that the layout of the display area 1155 shows a mute button 5282 , a local toggle camera button 5286 , a remote toggle camera button 5288 and an end meeting button 5284 . The second stage 5290 illustrates the UI 1105 after the user of the local device 5100 selects (e.g., via a single-finger tap 5170) the remote switch camera selectable UI item 5288. The last four stages of FIG. 52 are similar to the last four stages of FIG. 51, except that the layout of the display area 1155 is the same as that described above in the first stage 5110, so no further description is given to avoid unnecessary unnecessary The details obscure the description of the invention.

Some embodiments provide a layout similar to the layout illustrated in FIG. Figure 53 illustrates such a layout 5105. Specifically, FIG. 53 shows a PIP display with a remote toggle camera selectable UI item 5180, and a display area 1155 with only a mute button 5282, a local toggle camera button 5286, and an end meeting button 5284.

As described above, process 5000 proceeds to operation 5025 when a user requests to switch cameras remotely. At operation 5025, process 5000 sends a request to switch cameras to the remote device. In some embodiments, the request is sent over a videoconferencing control channel multiplexed by the VTP manager along with audio and video channels as described above.

After the request to switch cameras is received, process 5000 determines (at 5030) whether the remote device responded to the request to switch cameras. In some embodiments, the remote device automatically sends an acceptance response (ie, sends an acknowledgment) to the local device over the video conference control channel. However, in other embodiments, the user of the remote device must accept the request through the remote device's user interface.

The first two stages 5410 and 5415 of the UI 5405 of FIG. 54 illustrate an example of the remote user accepting a request to switch the camera of the remote device 5400. The first stage 5410 displays (1) a display area 5440 for displaying text notifying the remote user of the request, (2) an optional UI item 5465 for accepting the request to toggle the remote device's camera (e.g., "Allow( "Allow" button 5465), and (3) an optional UI item 5470 for denying a request to switch the remote device's camera (eg, "Reject" button 5470). The second stage 5415 then illustrates the UI 5405 after the user of the remote device selects (e.g., by single-finger tap 5480) the UI item 5465 for accepting the request to switch cameras, by highlighting the optional UI item 5465, instructing the user of said selections.

When process 5000 determines (at 5030) that it has not received a response from the remote device, process 5000 determines (at 5035) whether a request to end the video conference has been received. If yes, then process 5000 ends. Otherwise, the process receives (at 5040) images from the remote device and the currently used cameras of the local device, generates a composite image for the video conference from these images, transmits the local device's video image to the remote device, and then returns to operation 5030.

When process 5000 determines (at 5030) that it has received a response from the remote device, process 5000 determines (at 5045) whether the remote device accepts the request to switch cameras. If not, then process 5000 returns to operation 5010 to continue receiving images from the camera of another device. Otherwise, process 5100 receives (at 5050) an image from another camera of the remote device, followed by (at 5055) a switch camera animation on the local device to show video from the previously utilized remote camera and video from the currently utilized remote camera. Transitions between videos (ie, received images at operation 5050). After operation 5055, processing returns to operation 5010 explained above.

The last four operational stages 5130, 5135, 5140, and 5145 illustrated with respect to UI 5105 in FIG. 51 illustrate one example of such a remote switching camera animation on the local device 5100. This example animation is similar to the example animation illustrated in stages 4815, 4820, 4825, and 4830 of FIG. 48, except that FIG. Outside the animation of the lady's video captured by the device's front camera. The last four stages of FIGS. 52 and 53 illustrate the same animation as that in FIG. 51 , except that display area 1155 of FIGS. 52 and 53 contains different selectable UI items than display area 1155 in FIG. 51 .

In some embodiments, when the remote device switches cameras, the remote device's UI also performs a switch camera animation to show the transition between the two cameras. The last four operational stages 5420, 5425, 5430, and 5435 illustrated with respect to UI 5405 in FIG. 54 illustrate an example of a switch camera animation displayed on remote device 5400 when remote device 5400 switches cameras. This animation is similar to the animation illustrated in stages 4630, 4635, 4640, and 4645 of FIG. The camera captures the video of the lady outside.

As mentioned above, Figures 46, 47, 48, 49, 51, 52, 53 and 54 represent various examples of switching camera animations performed on the user interface. In some embodiments, switching camera animations result in changes in image processing operations of the corresponding dual camera mobile device, such as scaling, compositing, and perspective distortion, which may be performed by video conference manager 1604 and image processing manager 1608 .

C. Exposure adjustment

During a videoconference between a dual camera mobile device and another mobile device, different embodiments provide different techniques for adjusting the exposure of images captured by either mobile device's camera. Some embodiments provide a user of a dual-camera mobile device with techniques to adjust the exposure of an image captured by another device's camera, while other embodiments provide a user with a technique to adjust the exposure of an image captured by the camera of a dual-camera mobile device. Some illustrative examples are described in detail below.

Figure 55 illustrates a process 5500 for performing remote exposure adjustment operations on a dual camera mobile device of some embodiments during a video conference. In the following discussion, the device through which the user instructs the remote device to adjust its exposure is referred to as the local device. In some embodiments, process 5500 is performed by a video conferencing manager of the local device. Additionally, process 5500 will be described with reference to Figures 56, 57, and 58, which illustrate various ways in which a user of a local device may request an exposure adjustment operation from a remote device.

As shown in Figure 55, process 5500 begins by initiating (at 5505) a video conference between a local device and a remote device. Process 5500 then receives (at 5510) the video from the remote device for display on a display screen of the local device. Subsequently, process 5500 determines (at 5515) whether a request to end the video conference has been received. As noted above, some embodiments are capable of receiving a request to end a video conference from a user of a local or remote device. Process 5500 ends when process 5500 receives a request to end the video conference.

However, when process 5500 does not receive a request to end the video conference, process 5500 then determines (at 5520) whether a request to adjust the exposure of the remote device's camera was received. When process 5500 determines that a request to adjust the exposure of the remote device's camera has not been received, process 5500 returns to operation 5510 to receive additional video captured from the remote device. Figures 56, 57 and 58 illustrate three different examples of providing a user with such a request. In Figures 56, 57 and 58, the first stage 5610, 5710 and 5810 all show the PIP displays 5625, 5750 and 5835 of the local devices 5600, 5700 and 5800, and the PIP displays 5625, 5750 and 5835 show two videos: A video captured by the camera of the remote device, and another video captured by the camera of the remote device. In the first stages 5610, 5710 and 5810, the man in the background display 5635, 5760 and 5845 is darker, indicating that the man is not properly exposed.

The second stage 561 of FIG. 56 illustrates one way in which the user of the local device 5600 requests the remote device to make an exposure adjustment by selecting the remote device's video (eg, by clicking on the background display 5635). In this manner, the UI 5605 automatically correlates the user's selection of the region of interest defined by block 5645 with the user's desire to instruct the remote device to make exposure adjustments to the region of interest, thereby instructing the videoconferencing manager of the local device to contact Remote device for exposure adjustment operations. The defined region of interest is used by the remote device for the calculation of exposure adjustments.

Similar to the second stage 5615 of FIG. 56, the second stage 5715 of FIG. 57 represents the local user's selection of the remote device's video, except that the selection instructs the UI 5705 to display selectable UI items 5770 as shown in the third stage 5720 outside. The fourth stage 5725 illustrates the user of the local device selecting selectable UI item 5770 to instruct the remote device to perform exposure adjustment operations as described above.

The second stage 5815 of FIG. 58 is similar to the second stage of FIG. 57, but instead of the user's selection instruction UI displaying a single selectable UI item for the video of the remote device, the user's selection instruction UI 5805 displays selectable UI items 5855, 5860 , 5865 and 580 menus, as shown in the third stage 5820. Selectable UI items include "Auto Focus" item 5855, "Auto Exposure" item 5860, "Switch Camera" item 5865, and "Cancel" item 5870. In some embodiments, the "Toggle Camera" selectable UI item 5865 is used to request a local toggle camera operation, while in other embodiments, the "Toggle Camera" selectable UI item 5865 is used to request a remote toggle camera operation. The fourth stage 5825 illustrates the user selecting the "Auto Exposure," item 5860, to instruct the remote device to perform exposure adjustment operations as described above.

When process 5500 determines (at 5520) that the local user instructs the local device to request an exposure adjustment operation, process 5500 sends (at 5525) a command to the remote device over the video conferencing control channel to adjust the capture by the camera currently capturing and transmitting video to the local device. exposure of the video. After operation 5525, process 5500 returns to operation 5510 explained above.

In some embodiments, the user of the remote device is required to provide permission before the remote device performs the exposure adjustment operation, while in other embodiments, the remote device performs the exposure adjustment operation automatically upon receiving a request from the local device. Additionally, in some embodiments, some videoconferencing functionality is implemented by the videoconferencing manager 1604 . In some of these embodiments, the video conference manager 1604 performs the exposure adjustment operation by instructing the CIPU 1650 to adjust the exposure setting of the sensor of the remote device camera being used.

The final stages 5620, 5730 and 5830 of Figures 56, 57 and 58 show the remote device's video brighter, which indicates that the man is properly exposed. While FIGS. 56, 57, and 58 provide examples of receiving an exposure adjustment request to correct a remote device's exposure, some embodiments provide a way for a user of a local device to request that the local device adjust the exposure of the local device's camera. Such a request may be generated similarly to the manner in which the remote device is requested to adjust the exposure of its camera illustrated in FIGS. 56 , 57 and 58 .

Figures 56-58 described above show several user interfaces for performing exposure adjustment operations. In some embodiments, the exposure adjustment operation can result in a change in the image processing operation of the dual camera mobile device, such as invoking the exposure adjustment process 5900 described in further detail below. Exposure adjustment operations can also result in changes in the operation of the cameras of the dual-camera mobile device capturing video, such as changing the exposure settings of the cameras.

1. Exposure adjustment method

Figure 59 conceptually illustrates an exposure adjustment process 5900 performed by an image processing manager of some embodiments, such as the image processing manager illustrated in Figure 16 . In some embodiments, process 5900 is part of the exposure adjustment operations described above with reference to FIGS. 55 , 56 , 57 and 58 . In some such embodiments, the image processing manager 1608 performs the process 5900 and adjusts the exposure settings of the camera by sending an instruction to the video conferencing manager 1604 which instructs the CIPU 1650 to adjust the camera sensor 405a or 405b, as above.

In some embodiments, process 5900 is performed by image processing layer 930 shown in FIG. 9 , while in other embodiments process 5900 is performed by statistical engine 465 shown in FIG. 4 . Some embodiments perform processing 5900 on images captured by cameras of devices (local or remote) in the video conference, while other embodiments perform processing 5900 as part of process 2100 (e.g., operation 2110) illustrated in FIG. . Some embodiments perform an exposure adjustment operation to expose an image captured by a camera of a dual camera mobile device that is neither too bright nor too dark. In other words, process 5900 is performed to capture images in a manner that maximizes the amount of detail as possible.

Process 5900 begins by receiving (at 5905) an image captured by a camera of a dual camera mobile device. In some embodiments, when the received image is the first frame image captured by a camera of a device in the video conference, the first frame image is not processed 5900 (ie, prior to the first frame image, there is no any image whose exposure value is determined). Process 5900 then reads (at 5910) the pixel values for the specified area in the received image. Different embodiments define the regions differently. Some such embodiments define regions of different shapes, such as squares, rectangles, triangles, circles, etc., while other such embodiments define the regions at different locations in the image, such as in the center, directly above, directly below, etc.

Subsequently, process 5900 calculates (at 5915) an average of pixel values in a specified region of the image. Process 5900 determines (at 5920) whether the calculated average of pixel values is equal to a particular prescribed value. Different embodiments define different specific values. For example, some embodiments define the specific value as the median pixel value of the dynamic range of the image. In some embodiments, ranges of values are defined, rather than individual values. In such embodiments, process 5900 determines (at 5920) whether the calculated average of pixel values is within a specified range of values.

When the calculated average of pixel values is not equal to a particular specified value, process 5900 adjusts (at 5925) the exposure value based on the calculated average. Process 5900 ends when the calculated average of pixel values is equal to the specified specified value. In some embodiments, the exposure value represents the amount of time the camera sensor was exposed. In some embodiments, the adjusted exposure value is used to expose the next frame of image to be captured by the camera that captured the received image. After adjusting the exposure value according to the calculated average, process 5900 ends.

In some embodiments, process 5900 is repeated until the calculated average of pixel values is equal to a certain specified value (or within a specified range of values). Some embodiments perform the processing 5900 continuously during the video conference, while other embodiments perform the processing 5900 at regular intervals (eg, 5 seconds, 10 seconds, 30 seconds, etc.) during the video conference. Additionally, during a video conference, process 5900 of some embodiments dynamically redefines certain pixel values prior to performing process 5900 .

Figure 60 conceptually illustrates an example of an exposure adjustment operation of some embodiments. Examples 6000, 6010, and 6015 all display an image 6020 captured by a camera of a dual camera mobile device on the left. Specifically, image 6020 shows a dark figure facing away from the sun. A dark person indicates that the exposure of the image is not high enough to expose the person's face or body. The right side of each instance 6000, 6010, and 6015 represent images 6025, 6030, and 6035, respectively, taken after image 6020. In some embodiments, image 6020 and the image on the right are images of a video captured by a camera of a dual camera mobile device. In other embodiments, image 620 and the image on the right are still images taken at different times by the cameras of the dual camera mobile device.

The first example 6000 illustrates the operation without any exposure adjustment. Thus, image 6025 appears to be the same as image 6020. Similar to the person in image 6020, the person in image 6025 is still dark because no exposure adjustments have been made.

In a second example 6010, an exposure adjustment operation is performed on an image 6020. In some embodiments, exposure adjustment operations are performed by process 5900 using defined regions 6040 . According to the exposure adjustment operation, the exposure of the camera is adjusted, and the camera captures an image 6030 with the adjusted exposure. As shown in FIG. 60 , the characters in image 6030 are not as dark as in image 6025 . However, the face and body of the person in image 6030 is still unclear.

A third example 6015 represents an exposure adjustment operation performed on an image 6020 . Similar to the second example 6010, the exposure adjustment operation of the example 6015 of some embodiments is performed by the process 5900 using the prescribed area 6045. According to the exposure adjustment operation, the exposure of the camera is adjusted, and the camera captures an image 6035 with the adjusted exposure. As shown in Figure 60, the person in image 6035 is correctly exposed because both the person's face and body are visible.

In some embodiments, the selection of the specified area may be made by the user of the dual camera mobile device. The device itself can automatically adjust its prescribed area for exposure adjustment operation through the feedback loop of exposure adjustment mentioned above in CIPU 400. Statistical engine 465 in FIG. 4 may collect data to determine whether exposure is appropriate for a captured image, and adjust camera sensors accordingly (eg, through a direct connection to sensor module 415).

D. Focus adjustment

FIG. 61 illustrates a process 6100 of adjusting the focus of a dual camera mobile device during a video conference. In the following discussion, the device through which the user instructs the remote device to adjust the focus of the remote device's camera is referred to as a local device. In some embodiments, the process 6100 of Figure 61 is performed by the video conferencing manager 1604 of the local device. Additionally, process 6100 is described below with reference to FIGS. 62 and 63 , which provide two example ways for a user of a local device to request a remote device to perform a focus adjustment operation.

As shown in Figure 61, process 6100 begins by initiating (at 6105) a video conference between a local device and a remote device. Process 6100 then receives (at 6110) the video from the remote device for display on a display screen of the local device. Subsequently, at 6115, process 6100 determines whether a request to end the video conference has been received. As noted above, in some embodiments, a video conference may end at the request of a user of a local or remote device. When process 6100 receives a request to end the video conference, process 6100 ends.

Otherwise, process 6100 determines (at 6120) whether it has received a request to adjust the focus of the remote camera of the remote device. When process 6100 determines that it has not received a request to adjust the focus of the remote camera of the remote device, process 6100 returns to operation 6110 to receive additional video from the remote device. Figures 62, 63, and 64 illustrate three different ways that different embodiments provide users with generating such requests. In Figures 62, 63, and 64, the first stage 6210, 6310, and 6472 all show the PIP displays 6225, 6335, and 6482 of the local devices 6200, 6300, and 6471, and the PIP displays 6225, 6335, and 6482 all show two videos, generated by the local A video taken by the device, and another video taken by the remote device. Display areas 1155 and 1155 in FIGS. 62 and 63 display an end conference button. However, in FIG. 64, the layout of the display area 1155 is the same as that of the display area 1155 of FIG. 12 explained above. Additionally, the switch camera button 6488 shown in display area 1155 may be selected to invoke a local switch camera operation in some embodiments, or a remote switch camera operation in other embodiments. As shown in the first stages 6210, 6310, and 6472, the video of the remote device displayed in the background displays 6235, 6345, and 6480 is blurred.

The second stage 6215 of Figure 62 illustrates the method by which the user of the local device requests a focus adjustment from the remote device by simply selecting the remote device's video (eg, via a single point 6240 on the remote device's video). In this way, the UI 6205 automatically associates the user's selection of the region of interest defined by block 6245 with the user's desire to instruct the remote device to perform an operation (e.g., a focus adjustment operation) on the region of interest, thereby instructing The video conference manager 1604 of the local device 6200 contacts the remote device to perform an adjustment operation (eg, a focus adjustment operation). The defined region of interest is used by the remote device for calculations of focus adjustments.

The second stage 6315 of FIG. 63 similarly represents the local user's selection of the remote video (eg, by the user's tap on the remote device's video). However, unlike the example illustrated in FIG. 62, this selection instruction UI 6305 in FIG. optional button) menu. These selectable UI items include an "Auto Focus" item 6360, an "Auto Exposure" item 6365, a "Switch Camera" item 6370, and a "Cancel" item 6355. In some embodiments, the "Toggle Camera" selectable UI item 6370 is used to request a local toggle camera operation, while in other embodiments, the "Toggle Camera" selectable UI item 6370 is used to request a remote toggle camera operation. The fourth stage 6325 then illustrates the local user selecting an auto-focus item 6360 .

The second stage 6474 of FIG. 64 again similarly represents the local user's selection of the remote video (eg, by the user's tap on the remote device's video). However, unlike the example illustrated in FIG. 63, such selection in FIG. 64 instructs the UI 6478 to request a focus adjustment operation (i.e., in the second stage 6474). After concluding the focus adjustment operation, UI 6478 displays a menu of selectable UI items 6484 and 6486 (i.e., in third stage 6476), which may be implemented as selectable buttons. These selectable UI items include an “Auto Exposure” item 6486 and a “Cancel” item 6484 .

When process 6100 determines (at 6120) that the local user instructs the local device to request a focus adjustment operation, process 6100 sends (at 6140) a command to the remote device, over the video conferencing control channel, to adjust the focus of the camera whose video is currently being captured and transmitted by the remote device. focal length. After 6140, processing returns to operation 6110 explained above.

In some embodiments, the user of the remote device must provide permission before the remote device can perform such operations, while in other embodiments, the remote device performs such operations automatically upon receipt of a request from the local device. Additionally, in some embodiments, the focus adjustment operation adjusts the focus setting of the remote device's camera used during the video conference. In some of these embodiments, some video conferencing functionality is implemented by video conferencing module 1602, as described above. In these embodiments, the videoconferencing manager 1604 instructs the CIPU 1650 to adjust the sensor of the remote device camera being used.

The final stages 6220, 6330 and 6476 of Figures 62, 63 and 64 represent the video of the remote device properly focused. While FIGS. 62, 63, and 64 provide examples of receiving a focus adjustment request to correct the focus of a remote device, some embodiments allow a user of a local device to request that the local device adjust the focus of the local device's camera. Such a request may be generated similar to the method of requesting the remote device to adjust the focus of its camera shown in FIGS. 62 , 63 and 64 .

Figures 62, 63 and 64 illustrate three example user interfaces that allow a user to perform focus adjustment operations. In some embodiments, the focus adjustment operation results in a change in the operation of the camera of the dual camera mobile device capturing the video displayed in the UI, such as changing the focus of the camera.

As described above in FIGS. 56 and 62 , the specified region of interest is used by the remote mobile device for the calculation of exposure adjustment and focus adjustment, respectively, of the video. However, in some other embodiments, a user's selection of an area of interest may be used to instruct a remote device to perform one or more operations. For example, in some embodiments, an exposure adjustment and a focus adjustment may be performed according to a defined region of interest, thereby instructing the remote device to perform both operations.

E. Frame rate control

During a video conference, some embodiments may wish to adjust or maintain the rate at which video images captured by the cameras of the dual camera mobile device (ie, the frame rate) are transmitted to the other device in the video conference. For example, assuming a fixed bandwidth, some such embodiments reduce the frame rate of the video to improve the picture quality of the video image, while other such embodiments increase the frame rate of the video to smooth the video (i.e., reduce jitter).

Different embodiments provide different techniques for controlling the frame rate of video images during a video conference. One example described above adjusts the VBI of the camera's sensor module 415 to control the rate at which images captured by the camera are processed. As another example, some embodiments of the management layer 935 of the video conferencing module 925 shown in FIG. 9 control the frame rate by dropping images. Similarly, some embodiments of the image processing layer 930 control the frame rate by dropping images. Some embodiments provide other techniques besides this to control the frame rate, such as dropping frames in general transmit buffer 3120 .

V. Dual cameras

A. Combined view

1. Picture-in-picture: display of two remote cameras

Some embodiments allow a dual-camera mobile device to display video captured from the mobile device and video captured from another dual-camera mobile device during a video conference in any of several display arrangements. Figure 65 illustrates examples of different display arrangements for video captured from one or more dual camera mobile devices. In FIG. 65, a user of a dual-camera mobile device 6500 (Device A) and another user of another dual-camera mobile device 6505 (Device B) are videoconferencing with each other.

Fig. 65 shows four examples of the display arrangement of the device A on the left side. The four display arrangements for device A are a first view 6510 , a second view 6515 , a third view 6520 and a fourth view 6525 . In addition, FIG. 65 also shows four examples of the display arrangement of the device B on the right side. The four display arrangements for device B are a first view 6565 , a second view 6570 , a third view 6575 and a fourth view 6580 . In this example, device A shows only two videos taken from device A's camera, while device B shows two videos taken from device A's camera, and one or two videos taken from device B's camera.

In a first view 6510, Device A's UI 6585 provides a composite display 6512. Composite display 6512 includes two display areas: display area 6530 that displays video captured from device A's back camera, and display area 6535 that displays video captured from device A's front camera. In this example, display area 6530 is located in the upper half of composite display 6512 and display area 6535 is located in the lower half of composite display 6512 . In the first view 6510, the two display areas are the same size. The upper display area 6530 is displaying a video of a mountain range, which is assumed to be the mountain range being captured by Device A's back camera. Display area 6535 is displaying a tree and man in hat, which is assumed to be the tree and man in hat being photographed by device A's front camera

UI 6585 in second view 6515 provides composite display 6517, which includes the same two display areas as first view 6510, except that display area 6535 (showing video captured from device A's front camera) is being positioned on composite display 6517 Display area 6530 (displaying video captured from device A's back camera) is located outside the lower half of composite display 6517.

In a third view 6520, the UI 6585 provides a PIP display 6595. PIP display 6595 includes two display areas: a display area 6535 as a background display area showing video captured from device A's front camera, and a foreground inset display area displaying video captured from device A's back camera District 6530. In this view, the background display area 6535 occupies most of the PIP display 6595 , while the inset display area 6530 is smaller and overlaps a portion of the background display area 6535 .

UI 6585 in fourth view 6525 also presents PIP display 6598 including display areas 6530 and 6535 as shown in third view 6520. Unlike PIP display 6595, PIP display 6598 includes display area 6530 (photographed from device A's back camera) as the background main display, and display area 6535 (photographed from device A's front camera) as the foreground inset display. Additionally, PIP display 6598 is presented in landscape view (ie, PIP display 598 is wider than it is tall).

The example above illustrates four possible composite views of Device A's UI - two views in which the two display areas 6530 and 6535 displaying video from the first device's two cameras are stacked vertically, and two PIP views. Other views are also possible for device A's UI. For example, two display areas can be stacked horizontally or diagonally, or different PIP views can be used.

The various views illustrated with respect to device B represent that the UI of device B may take different views. These views include video taken from both of Device A's cameras, and one or more of Device B's cameras. In Device B's first view 6565, Device B's UI 6590 provides a PIP display 6568. PIP display 6568 includes the same composite display area 6569 as composite display 6512 displayed on device A, and an inset display area 6550 that displays video captured by one of device B's cameras (eg, the front camera). The composite display area 6569 includes a display area 6531 displaying video captured from the rear camera of device A, and a display area 6536 displaying video captured from the front camera of device B. The composite display 6569 of display device A's video occupies the majority of the PIP display 6568 , while the inset display area 6550 is smaller and overlaps the composite display 6569 . Display area 6550 displays a video of a smiling face, which is assumed to be a smiling face whose video is being captured by the front camera of device B.

The UI 6590 of Device B in the second view 6570 provides a PIP display 6572. PIP display 6572 includes display area 6550 (displaying video captured from device B's front camera), and composite display 6573 having display areas 6531 and 6536 displaying video captured from device A's camera. Composite display 6573 is the same as composite display 6517 in device A's second view 6515 and occupies the majority of PIP display 6572 . As in PIP display 6568 in first view 6565 , display area 6550 is smaller and overlaid on composite display 6573 . Specifically, in both views, display area 6550 overlaps a portion of display area 6531 that displays video captured from device A's back camera.

In third view 6575, UI 6590 provides a PIP display 6577 similar to PIP display 6595 in device A's third view 6520. The PIP display 6577 also includes an additional display area 6550 as a second inset display area that overlays the background display area 6536 . The two inset display areas 6531 and 6550 are horizontally tiled at the bottom of the background main display area 6536 .

UI 6590 in fourth view 6580 provides composite display 6582. Composite display 6582 includes three displays: PIP display 6583, display area 6550, and display area 6540 (eg, for displaying video captured by device B's back camera). PIP display 6583 is the same as PIP display 6598 in device A's fourth view 6525 and occupies most of the composite display area 6582 . Display areas 6540 and 6550 are smaller and are tiled horizontally below PIP display area 6583 .

Although FIG. 65 illustrates four possible views of device B, many other views are possible. The background composite display of device A's video can be tiled horizontally instead of vertically, the inset display area can overlap the front camera display area of device A instead of the back camera display area, and the larger display area can Video from device B's camera is displayed instead of device A's camera video, the inset display area may be in a different location, and so on.

Each set of arrows 6560 originating from each view of device A indicates that there is no requirement for a correlation between the display represented on device A and the display represented on device B. For example, even if device A is displaying zone video in the arrangement of view 6510 (e.g., in the arrangement selected by the user of device A), device B may be in the four arrangements illustrated, or in a variety of arrangements not shown in FIG. Any of the other arrangements displays the video (eg, in an arrangement selected by the user of device B). In other words, device A's display arrangement is independent of device B's display arrangement. Some embodiments do not transfer a display area from one device to another, but only transfer video (eg, in encoded form) that is displayed by a device in its corresponding display area.

2. Dedicated PIP

Some embodiments allow a user of a dual camera mobile device to superimpose the foreground of a video on top of another video in a PIP display during a video conference. In some embodiments, the foreground of a video is blended into another video in such a way that they appear as a display of a single video captured by a single camera. Figure 66 illustrates an example of such superimposition of the foreground of the inserted video on the background video in a PIP display.

Figure 66 illustrates this video overlay operation in terms of seven operational stages 6620, 6625, 6630, 6635, 6640, 6660, and 6665 of UI 6670. The first stage 6620 illustrates the UI 6670 of a dual camera mobile device 6600 with a PIP display 6682 during a video conference with a remote device. As shown in the first stage 6620, the PIP display 6682 includes two video displays: a background main display 6610 and a foreground inset display 6605. The background main display 6610 takes up most of the UI 6670, while the foreground inset display 6605 is smaller and overlaps the background main display 6610.

In this example, background display area 6610 displays a video of a mountain range, which is assumed to be a mountain range captured by one of the remote device's cameras. The foreground inset display area 6605 displays a video of a person wearing a hat, which is assumed to be the person whose video was captured by one of the local device's cameras. Below the PIP display 6682 is an optional UI item 6685 (e.g., a button 6685) labeled "End Conference" that allows the user to select the item (e.g., by clicking or double-clicking the button) to end the video conference.

The second stage 6625 illustrates the invocation of the selectable menu 6675. In some embodiments, by selecting (eg, touching) the PIP display area 6682, a menu of selectable UI items 6675 may be invoked. Instead of, or in combination with, such an invocation operation, some embodiments also allow the user to invoke the optional UI item 6675 through other operations, such as through various touch screen operations or using one or more other physical inputs of the device. menu.

The third stage 6630 displays a UI 6670 with an invoked set of selectable UI items for selecting video overlay operations. In this example, a pop-up menu 6675 is displayed on the PIP display 6682 with several selectable UI items. Menu 6675 of selectable UI items includes "Flip PIP (flip PIP)" selectable UI item 6640 (e.g., button 6640), "Specialized PIP (specialized PIP)" selectable UI item 6645 (e.g., button 6645), and " "Cancel" optional UI item 6690 (eg, button 6690). In this example, selection of the "Flip PIP" button 6640 causes the UI 6670 to swap the background display 6610 and the inset display 6605 (discussed in detail in the next section), and selection of the "Specialized PIP" button 6645 causes the UI 6670 to begin the video overlay operation , and selecting the "Cancel" button 6690 removes the popup menu 6675 from the PIP display 6682. Other embodiments include different or more items in the PIP pop-up menu 6675.

The fourth stage 6635 illustrates the UI 6670 after the user has selected the "Specialized PIP" button 6645 (eg, by tapping the button 6645 with his finger 6695). This selection is indicated by highlighting button 6645 on UI display 6670. Some embodiments use a different indication display (eg, highlighting a border of a selected item or text within a selected item).

The fifth stage 6640 represents the UI 6670 after starting the video overlay operation. At this stage, the UI 6670 allows the user to choose which video he wants to extract from as the foreground, and which video he wants to use as the background in the superimposed video. The UI 6670 provides various options through a pop-up menu 6680 displayed on the PIP display 6682 with several selectable UI items. The popup menu 6680 of selectable UI items includes a "Select Inset" selectable UI item 6655 (e.g., button 6655), a "Select Main" selectable UI item 6650 (e.g., button 6650 ), and a "Cancel" optional UI item 6692 (eg, button 6692).

Selecting the "Select Inset" button 6655 causes the UI 6670 to superimpose the foreground of the inset video 6605 from the local device's camera (i.e., the man in the hat) over the background main video 6610 from the remote device's camera. On the other hand, selecting the "Select Main" button 6650 causes the UI 6670 to superimpose the foreground (i.e., mountains) of the background main video 6610 from the remote device's camera over the insert video 6605 from the local device's camera. In some embodiments, this results in a toggle between the two video feeds such that the video currently in the inset display area 6605 will take up most of the UI 6670 and the video currently in the main display area 6610 will be superimposed on the immediate on the main video. Selecting the "Cancel" button 6692 aborts the video overlay operation and removes the pop-up menu 6680 from the PIP display area 6682.

The sixth stage 6600 illustrates the UI 6670 after the user selects the "Select Inset" button 6655 (e.g., by tapping the button 6655 with his finger 6695). This selection is indicated by highlighting button 6655 on UI display 6670. Some embodiments utilize different indication displays (eg, highlighting a border of a selected item or text within a selected item).

The seventh stage 6665 illustrates the UI 6670 after completion of the video overlay operation. As shown in UI 6670, the foreground inserted into display area 6605 (i.e., a man in a hat) is extracted from display area 6605. The window frame and background of the inset display 6605 are also eliminated from the screen (ie, everything else but the foreground). Finally, the foreground (ie, the man in the cap) is blended into the background video 6610 in such a way that it appears as a single video. There are a variety of different techniques that can be used to remove the background of an inserted video. Some embodiments identify pixels that do not move relative to other pixels, look for a constant pattern or color, use a baseline image compared to an image including the foreground, and subtract the difference, or use a different technique.

While the example of FIG. 66 illustrates that the foreground of the inset display area 6605 remains in the same location in the UI when overlaid over the background display area 6610, this is only one example of how overlaying can be accomplished. Some embodiments move the foreground video to a specific location in the UI 6670 (e.g., the center, one of the corners, etc.). Similar to the features shown in Sections IV.A.1 and IV.A.3, some embodiments allow the user of the local device to drag the overlay foreground video back and forth in the UI, or to change the size of the overlay foreground video.

Different techniques may be used to determine which portion or portions of a video image are "foreground" for a video overlay operation as described above. One such method of some embodiments determines which portion or portions of the video image, if any, are dynamic. The dynamic part is considered "foreground" because the background of a video image is usually static (ie, no motion). In such embodiments, video images are analyzed over a specified period of time. If the difference between the values of a particular pixel is not greater than a specified threshold (eg, 5%, 10%, 15%) within the period of time, then the particular pixel is considered to be a static pixel. After analyzing every pixel in the video image, the dynamic pixels (ie, the pixels that are not static) of the video image are considered to be the "foreground" of the video image.

FIG. 67 illustrates one example of such a technique that can be performed by the video conferencing manager 1604 or the image processing manager 1608 to determine the foreground of a video image. Specifically, FIG. 67 illustrates a series of six images 6705-6730 showing a video of a figure wearing a hat and a tree. In this example, it is assumed that the character is not standing perfectly still and may be speaking. As described above, each pixel in a video image is analyzed to determine whether the pixel is dynamic or static. For example, it is determined whether the difference between the values of pixels 6735 in images 6705-6730 is greater than a specified threshold. Here, since the pixel 6735 represents part of the ground other than the person, the pixel 6735 is considered to be static. After analyzing all the pixels in the images 6705-6730, it is determined that the person in the image is dynamic and the rest of the image is static. Thus, the person is the "foreground" to be extracted by the operation explained above with reference to FIG. 66 .

3. Swap the video in the picture-in-picture display

Some embodiments allow a user of a dual-camera mobile device to swap two display areas in a PIP display (i.e., in a PIP display, the inset display area becomes the background display area, and the background display area becomes the inset display area) during a video conference . Figure 68 illustrates an example of swapping the inset display area 6605 and the background display area 6610 in a PIP display 6682 during a videoconference.

Figure 68 illustrates the exchange PIP operation in terms of eight operational phases of the UI 6670 of the device 6800 in Figure 66. The first three stages in FIG. 68 are the same as the first three stages in FIG. 66 . During these stages, the user causes a menu 6675 to appear within the UI 6670 by making selections using the local device's touch screen.

The fourth stage 6840 of FIG. 68 illustrates the UI 6670 after the user selects the "Flip PIP" button 6640 (e.g., by tapping the button 6640 with his finger 6695). Such selection is indicated by highlighting button 6640 on UI display 6670. Some embodiments utilize different indication displays (eg, highlighting a border of a selected item or text within a selected item).

The fifth stage 6845 illustrates the UI 6670 after starting the exchange PIP operation. Some embodiments animate the swapping of the inset display 6605 and the background display 6610 through a flipping action. Figure 68 illustrates an example of such an animation. In this example, this animation is illustrated by the flipping of the viewing pane with the PIP showing 6682 (before performing the swap operation) on one side and the new PIP showing 6684 (after performing the swap operation) on its other side . The viewing pane rotates 180° about a vertical axis 6686 located at the center of the PIP display 6682. In the fifth stage 6845, the viewing pane begins to rotate about the vertical axis 6686.

At the sixth stage 6850, the viewing pane is indicated to have been rotated approximately 90°. This is indicated by a thin line 6688 displayed in the center of the screen (ie, the edge of the viewing pane). The seventh stage 6855 illustrates that the rotation of the viewing pane is nearing completion. The new PIP shows 6684 starting to emerge from the other side of the viewing pane and expanding horizontally to fill the device's screen. PIP display 6684 includes two display areas 6605 and 6610 after a swap operation has been performed. The display area 6605 showing the video of the man in the hat (from the local device's camera) is now in the background of the PIP display 6684, and the display 6610 showing the video of the mountains (from the remote device's camera) is now in the foreground of the PIP display 6684, so The foreground is overlaid on display 6605. The eighth stage 6860 represents the completion of the swap display operation.

Those of ordinary skill will recognize that the animation shown in Figure 68 is only one of many possible animations for the PIP insert/background swap operation. For example, different embodiments may rotate the viewing pane along a horizontal axis, swap the two display areas instantaneously, expand one display area while shrinking the other, and so on. Some embodiments provide one animation that is always used for the swap operation, while other embodiments allow the user to choose from several animations, or use a different animation (eg, by random selection). Additionally, the swapping operation can result in changes in the image processing operations of the dual camera mobile device, such as causing the video conferencing manager 1604 to change the scaling and compositing of the video in response to user input.

4. Lock to the corner

Some embodiments of the invention allow a user of a dual camera mobile device to modify a composite display by moving around one or more display areas forming the composite display. An example of this movement is described above in Section IV.A.1. Such movement of the inset display is also possible when the PIP display comprises more than one inset display area.

Figure 69 illustrates such an example performed during a video conference. The example illustrated in FIG. 69 is similar to the example illustrated in FIG. 3, except that FIG. 69 illustrates moving back and forth of a PIP display 6965 comprising two inset display areas 6905 and 6910, rather than just one such inset display area. Insert outside display area 6910.

In FIG. 69, UI 6960 of mobile device 6900 presents PIP display 6965 during a video conference with a remote user of another device. PIP display 6965 in FIG. 69 includes three video displays: a background main display 6915 and two foreground inset displays 6905 and 6910. In this example, the background main display 6915 presents a video of a character singing and playing guitar, which is assumed to be video taken by the remote device's back camera. The foreground inset display 6905 presents a video of a character holding a racket, which in this example is assumed to be video captured by the local device's back camera. Another foreground inset shows 6910 presenting a video of a person wearing a hat, which in this example is assumed to be the person whose video was captured by the local device's front camera. Below the PIP display 6965 is an optional UI item 6970 (eg, button 6970) labeled "End Conference," which allows the user to end the video conference by selecting it.

The PIP display 6965 is just a way of presenting a composite view of the video taken by the remote device and the local device. Some embodiments may provide other composite views. For example, instead of a larger background display 6915 with video from a remote device, the larger background display 6915 could be the video from the local device and the smaller foreground inset displays 6905 and 6910 could be the video from the remote device. Additionally, some embodiments allow local and remote video to appear in the UI 6910 with the inset displays 6905 and 6910 on one side, the background display 6915 on the other, or all three displays side-by-side. In other embodiments, the PIP display 6965 may comprise a larger background display 6915 and/or a smaller foreground inset display. In some embodiments, the manner in which the PIP displays 6965 or the default display mode can be specified by the user.

69 illustrates the movement of one of the two inset display areas in the UI 6960 of the device 6900 by referring to five different stages of operation 6920, 6925, 6930, 6935, and 6940. The first stage 6920 illustrates the UI 6960 during a video conference between a local user of the device 6900 and a remote user of the remote device.

The second stage 6925 illustrates that by selecting the inset display area 6910, the user begins the lock-to-corner operation. In this example, by placing finger 6950 anywhere within inset display area 6910, a selection is made. This selection is displayed with a thick border 6962 inserted into the display area 6910 as shown. Different embodiments may indicate such selection in different ways, such as by highlighting the inset display area 6910, by shaking the inset display area 6910, and the like.

The third stage 6930 illustrates the UI 6960 after the user begins moving the inset display area 6910 of the PIP display 6965 from one area in the PIP display 6965 to another area in the PIP display 6965. In this example, inset display area 6910 has begun to move from the lower right corner of PIP display 6965 to the upper right corner of PIP display 6965, as indicated by arrow 6955. After selecting the inset display area 6910 , the user moves the inset display area 6910 by dragging his finger 6950 towards the upper right corner of the PIP display 6965 . Some embodiments provide other techniques for moving the inset display area 6910 around in the PIP display 6965 .

The fourth stage 6935 illustrates the UI 6960 in a state after the user has removed his finger 6950 from the screen of the device 6900. In this state, the inset display area 6910 is still moving towards the upper right corner of the PIP display 6965 identified from the user's finger movement in the third stage. In other words, after finger 6950 initiates movement of inset display area 6910 towards the upper right corner of PIP display 6965, UI 6960 maintains this movement even after finger 6950 is removed. To maintain the movement, the UI 6960 of some embodiments requires that the user's drag operation be greater than a certain threshold amount (e.g., greater than a certain distance, or longer than a certain time) before the user removes their finger; otherwise, these Embodiments may keep the inset display area in its original lower right corner position after moving it slightly, or not move the inset display area at all.

However, while some embodiments allow the inset display to continue moving even if the user stops his dragging operation before the inset display area reaches its new location, other embodiments require the user to keep dragging until the inset display area reaches its new location. its new location. Some embodiments provide other techniques for moving the inset display area. For example, some embodiments require the user to specify where to direct the display area 6910, etc., before the display area 6910 actually begins to move. Some embodiments also allow the display area to slide and lock into corners by simply tilting the mobile device at different angles.

The fifth stage 6940 illustrates the UI 6960 after the inset display area 6910 reaches its new location in the upper right corner of the PIP display area 6965. The elimination of the thick border 6962 in the fifth stage indicates that the lock to corner operation is complete.

To facilitate the movements illustrated in the third, fourth, and fifth stages 6930, 6935, and 6940 described above, the UI 6960 of some embodiments employs , it allows the insertion display area 6910 to quickly lock to the corner locking rules. For example, when a user drags the inset display area 6910 toward a particular corner beyond a threshold amount, the UI 6960 of some embodiments identifies the direction of motion of the inset display area 6910, determines that the movement has exceeded a threshold amount, and then automatically moves the inset display area 6910 without further user input the insertion display area 6910 may be locked to the next grid point in the UI 6960. In some embodiments, the only grid points provided for locking inset display area 6910 are the grid points located at the four corners of PIP display 6965 . Other embodiments provide for other grid points in the UI 6960 (e.g., in the PIP display 6965) that the insertion display area 6910 can be locked to.

Alternative embodiments may not employ grid points such that the inset display area 6910 may be placed at any point in the PIP display. Still further embodiments provide a pin to grid point feature that allows the user to toggle the UI on or off. Additionally, various embodiments may allow users to perform pin-to-corner operations on various items, such as icons, in addition to video captured from the device. As noted above, movement of one or more display areas of the composite display can result in changes to the image processing operations of the dual camera mobile device, such as causing the video conference manager 1604 to recompose the display areas in the composite display in response to user input.

5. Push and lock

The example in FIG. 69 illustrates a pin to corner operation that allows a user of a dual camera mobile device to move one of the two inset display areas from one corner of the PIP display to another corner not occupied by the inset display. Some embodiments enable a push feature that moves the first inset display to the location of the second inset display, and also pushes the second inset display to the new location. Figure 70 illustrates one such example performed during a video conference.

Figure 70 illustrates the movement of an inset display from one corner of the PIP display to another corner of the PIP display not occupied by another inset display by referring to six different stages 7020, 7025, 7030, 7035, 7040, and 7045 of the UI 6960 . The first stage 7020 illustrates the UI 6960 during a video conference between a local user of the device and a remote user of the remote device. UI 6960 in FIG. 70 represents the same PIP display 6965 as that shown in the first stage of FIG. 69 after starting the video conference. In this example, video captured by the local user's device is displayed in inset display areas 6905 and 6910 , and video captured by the remote user's device is displayed in background display area 6915 .

The second stage 7025 illustrates that the user initiates the lock to corner operation by selecting the inset display area 6905 . In this example, selection is made by placing finger 7055 anywhere within inset display area 6905. This selection is displayed with a thick border 7065 inserted into the display area 6905, as shown. Different embodiments indicate such selection in different ways, such as by highlighting the display area 6905, by shaking the display area 6905, and the like.

The third stage 7030 illustrates that, as indicated by arrow 7050, after the user begins to move the inset display area 6905 from the lower left corner of the PIP display 6965 to the lower right corner of the PIP display 6965 (by in the third stage, after selecting the inset display area 6905 , UI 6960 after dragging his finger 7055) towards the bottom right corner of the PIP display 6965. Some embodiments provide other techniques for moving the inset display area 6905 around in the PIP display 6965 .

The fourth stage 7035 illustrates the UI 6960 after the inset display area 6905 contacts the inset display area 6910. When touched, the inset display area 6910 moves towards the next closest corner. In this example, inset display area 6910 begins moving in the direction of the upper right corner of PIP display 6965 (as indicated by arrow 7075). Activation of this push operation is shown with a thick border 7070 inserted into the display area 6910 . Different embodiments may indicate this activation in different ways, such as by highlighting 6910, and so on.

The fifth stage 7040 illustrates the UI in a state after the inset display area 6905 is locked to the lower right corner previously occupied by the inset display area 6910 . In this example, the inset display area is still moving towards the upper right corner of the PIP display 6965. Additionally, the thick border 7065 is no longer displayed. As long as the third stage 7030 user drag operation is greater than the threshold for locking the inset display area 6905 to the right corner, the inset display area 6910 is moved away from its corner and locked all the way to the next lowest corner.

Some embodiments include a set of rules that determine which way to push the second inset display area 6910. In the case illustrated in Figure 70, some embodiments seek to continue to maintain the rotation of the inset display area. That is, since the insertion display area 6905 is moved in the counterclockwise direction, the display area 6910 is also moved counterclockwise. Some embodiments provide a hierarchy of possible positions to which the pushed insert display area 6910 can be moved, and select the first unoccupied position on the list. For example, the upper right corner may be the first position in such a list when an inset display area located in the lower right corner is pushed by an inset display area from the lower left corner. However, if the third inset display area is already in the upper right corner, some embodiments move to the next option on the list (eg, upper left, center, or lower left). Other embodiments may push the third inset display area along with the second inset display area so that the device does not need to determine a new location for the third inset display area.

The sixth stage 7045 illustrates the UI 6960 after the inset display area 6910 has reached its new location in the upper right corner of the PIP display area 6965. At this stage, the removal of the thick border 7070 indicates the lock to the corner - the push operation is complete. Similar to the push-to-corner operation described with reference to FIG. 68 , movement of one or more display areas of the composite display can result in changes to the image processing operations of the dual-camera mobile device, such as causing the videoconferencing manager 1604 to recompose the display in response to user input. The display area in the composite display.

6. Rotate

When a user of a mobile device used for video conferencing rotates the device during the meeting, some embodiments rotate the PIP display presented during the video conference. Figure 71 illustrates the rotation of the device's UI display 7175 as the device 7100 is rotated from a vertical position to a horizontal position. When the long side of the screen is vertical, the device 7100 is held vertically, and when the long side of the screen is horizontal, the device 7100 is held horizontally. In the example illustrated in FIG. 71 , UI display 7175 rotates from a portrait view optimized for a vertical hold of the device to a landscape view optimized for a landscape hold of device 7100 . This rotation functionality enables the user to view the UI 7175 displayed in an upright position when the mobile device 7100 is held vertically or horizontally. The example illustrated in FIG. 71 is similar to the example illustrated in FIG. 34, except that FIG. 71 illustrates a PIP display that rotates to include two inset display areas rather than just one.

In FIG. 71 , the mobile device's UI 7175 presents a PIP display 7180 during a video conference with a remote user of another mobile device. PIP display 7180 in FIG. 71 includes three video displays: a background main display 7115 and two foreground inset displays 7110 and 7160. In this example, the background main display 7115 presents a video of a mountain range, which is assumed to be video taken by the remote device's front or back camera. The foreground inset display 7110 presents a video of a smiling face in the room, assumed to be captured by the local device's front or back camera. Another foreground insert shows 7160 presenting a video of a singing guitar player, presumably the one whose video is being captured by another camera of the local device. Below the PIP display 7180 is an "End Meeting" button 7155, which the user may select to end the video conference (eg, with a one-finger tap). This PIP display is just one way of presenting a composite view of the video captured by the remote device and the local device. Some embodiments may provide other composite views, such as tiled views or different PIP displays.

FIG. 71 illustrates the rotation of the UI 7175 according to six different stages of operation 7120, 7125, 7130, 7135, 7140, and 7145. The first stage 7120 illustrates the UI 7175 during a video conference between a local user of the device and a remote user of the remote device.

The second stage 7125 illustrates the UI 7175 after the user begins to tilt the device 7100 sideways. In this example, device 7100 has begun tilting device 7100 from being held vertically to being held horizontally, as indicated by arrow 7185 . The appearance of UI 7175 has not changed. In other cases, the user may instead want to tilt the device 7100 from being held horizontally to being held vertically, in which case UI display 7175 switches from a horizontally optimized view to a vertically optimized view.

The third stage 7130 illustrates the UI 7175 after the device 7100 has been tilted from being held vertically to being held horizontally. In this state, the UI shows that the appearance of 7175 remains unchanged. In some embodiments, the rotation operation is triggered after the device 7100 is tilted more than a threshold amount and held there for a period of time. In the example illustrated in FIG. 71 , it is assumed that the threshold amount and rotation speed will not cause the UI display 7175 to rotate until a short time interval after the device has been placed in the horizontal position. Different embodiments have different threshold amounts and wait times for triggering a rotation operation. For example, some embodiments may have such a low threshold for triggering a rotation operation that regardless of the orientation of the device 7100, the UI 7175 appears as if it is always displayed in an upright position. In other embodiments, a user of device 7100 may specify when a rotation operation may be triggered (eg, via a menu preference setting). Additionally, some embodiments may not delay the rotation after the device has been tilted beyond a threshold amount. Furthermore, different embodiments may allow the rotation operation to be triggered in different ways, such as by toggling a switch on the mobile device, by issuing a voice command, upon selection through a menu, and so on.

The fourth stage 7135 illustrates the UI 7175 after starting the rotation operation. Some embodiments animate the rotation of the display area to provide feedback to the user regarding the rotation operation. Figure 71 illustrates an example of such an animation. Specifically, Figure 71 in its fourth stage 7135 shows display areas 7110, 7115 and 7160 starting to rotate together. Display areas 7110 , 7115 , and 7160 rotate about axis 7165 (ie, the z-axis) that passes through the center of UI display 7175 . Display areas 7110, 7115, and 7160 are rotated by the same amount, but in a direction opposite to the rotation of device 7100 (eg, by tilting of device 7100). In this example, since device 7100 is rotated 90° clockwise (by going from vertical to horizontal holding), the rotating operation will cause display areas 7110, 7115, and 7160 to rotate 90° counterclockwise. When display areas 7110, 7115, and 7160 are rotated, display areas 7110, 7115, and 7160 are scaled down to fit UI display 7175 such that display areas 7110, 7115, and 7160 can still fully appear within UI 7175. Some embodiments may provide a message indicating the status of the device 7100 (eg, by displaying the word "Rotating").

The fifth stage 7140 illustrates the UI 7175 after display areas 7110, 7115, and 7160 have been rotated 90° counterclockwise from portrait view to landscape view. At this stage, display areas 7110, 7115, and 7160 have been rotated, but not expanded to the full width of UI 7175. Arrow 7170 indicates that at the end of the fifth stage, display areas 7110, 7115, and 7160 will begin to expand sideways to fit the full width of UI 7175. Different embodiments may not include this stage, as the deployment may occur concurrently with the rotation in the fourth stage 7135 .

The sixth stage 7145 illustrates UI 7175 after display areas 7110, 7115, and 7160 have been expanded to occupy the entire display of UI 7175. As noted above, other embodiments may achieve this rotation differently. For some embodiments, the rotation operation may be triggered simply by rotating the screen of the device beyond a threshold amount, regardless of the orientation of the device 7100.

Additionally, other embodiments may provide different animations for indicating rotation operations. The rotation operation performed in FIG. 71 involves rotation of the UI display 7175 about the center of the UI display 7175. Alternatively, the display areas may be individually rotated about the central axis of their respective display areas. One such method is shown in FIG. 72, which shows an alternative method of animating the rotation of the PIP display area 7180 of the UI 7175. The PIP display illustrated in FIG. 72 is the same as the PIP display 7180 illustrated in FIG. 71 .

FIG. 72 illustrates the rotation of the PIP display 7180 according to six different stages of operation 7120 , 7125 , 7130 , 7220 , 7225 , and 7230 . The operation of the first three stages of UI 7175 is the same as that of the first three stages as illustrated in UI 7175 in FIG. 71 . In the third stage of Figures 71 and 72, both devices have changed from vertical to horizontal holding, and the rotation of UI 7175 has not yet started.

The fourth stage 7220 illustrates an alternative method of animating the rotation. In the fourth stage, the rotation operation has started. Specifically, fourth stage 7220 represents the beginning of rotation of display areas 7110 , 7115 , and 7160 . The display areas 7110, 7115, and 7160 rotate about an axis 7250 (ie, a z-axis) passing through the center of each display area, respectively. Display areas 7110, 7115, and 7160 are rotated by the same amount, but in a direction opposite to the rotation of device 7100 (eg, by tilting of device 7100). In this example, since device 7100 is rotated 90° clockwise (by going from vertical to horizontal holding), the rotating operation will cause display areas 7110, 7115, and 7160 to rotate 90° counterclockwise. When display areas 7110, 7115, and 7160 rotate, they also scale down to fit UI display 7175 so that display areas 7110, 7115, and 7160 can still fully appear on UI 7175.

The fifth stage 7225 illustrates the UI 7175 after display areas 7110, 7115, and 7160 have all been rotated 90° counterclockwise from portrait view to landscape view. At this stage, display areas 7110, 7115, and 7160 have been rotated, but have not expanded to the full width of UI 7175, or reached their final positions. The final location of the display area in PIP display 7115 is determined by the position of the display area in PIP display 7115 as shown in first stage 7120 (e.g., inset display 7110 in the lower left corner of PIP display 7180, and inset display 7160 in bottom right corner of PIP display 7180).

Arrow 7170 indicates that at the end of the fifth stage, display areas 7115, 7110, and 7160 will begin to expand sideways until main display area 7115 fits the full width of UI 7175. Additionally, arrow 7255 indicates that inset display areas 7110 and 7160 will move to reach their final positions in PIP display 7180 . In other words, the inset display area 7110 will move down towards the lower left corner of the PIP display 7180 while the other inset display area 7160 will move towards the lower right corner of the PIP display 7180 . Different embodiments may implement this animation differently, for example by utilizing the lock and push operations illustrated in FIG. 71 . The sixth stage 7230 illustrates UI 7175 after display areas 7110, 7115, and 7160 have been expanded to occupy the entire display of UI 7175 and have moved to their final positions.

As noted above, other embodiments may achieve this rotation differently. For example, similar to that illustrated in FIGS. 36 and 37 , some embodiments provide a rotation operation in which the orientation of the display area displaying video captured by the local device changes to reflect the orientation of the local device after the rotation operation is performed on the local device. Orientation, some embodiments provide a rotate operation wherein the orientation of the display area showing the video captured by the remote device changes to reflect the orientation of the remote device after the remote device is rotated, some embodiments provide wherein the display area 1155 remains Rotation operations at the same location, some embodiments provide different layouts in the display area (eg, the layout of display area 1155 of FIG. 12 ), or combinations thereof.

For some embodiments, simply rotating the device's screen beyond a threshold amount may trigger a rotate operation, regardless of the orientation of the device 7100. Also as mentioned above, the local device and the remote device can notify each other of the rotation operation performed on one of the two devices through the control communication channel, so as to allow the other device to make any corresponding modification to the video of the one device. In addition, the animation of the rotation operation can also cause changes in the operation of the camera or the image processing operation of the dual-camera mobile device, such as causing the video conference manager 1604 to recompose one or more display areas in the UI 1105 at different angles, and scale Zooms the image displayed in one or more display areas.

7. Select Watch Remote View

As noted above, some embodiments allow a user of a dual-camera mobile device to select which camera to use for a video conference prior to starting the video conference, or while starting the video conference. Instead of, or in conjunction with, this capability, some embodiments allow the user of the device to choose between two videos displayed in a video conference and either from a remote device. camera, or two video cameras from the user's local device. Figure 73 illustrates the selection of one of two remote videos in a meeting, while Figure 74 illustrates the selection of one of two local videos in a meeting.

73 illustrates the selection of the remote video in terms of six operational stages 7335, 7340, 7345, 7350, 7355, and 7360 of the UI 7375 displayed on the local device 7300. The first stage 7335 illustrates a UI 7375 with an initial PIP display 7390 presented during a video conference with a remote user of a mobile device with two cameras.

As shown in the first stage 7335, the initial PIP display 7390 includes three displays: a background main display 7315 and two foreground inset displays 7305 and 7310. Background display 7315 occupies most of the PIP display area 7390, while foreground inset displays 7305 and 7310 each overlay a portion of background display 7315 on UI 7375. In this example, the background display 7315 presents a video of the person in front of the microphone, which is assumed to be video taken by the remote device's back camera. A first foreground inset display 7305 presents a video of the man's face, which in this example is assumed to be video taken by one of the cameras of the local device 7300. A second foreground inset display 7310 presents a video of a figure wearing a hat, which in this example is assumed to be video taken by the remote device's front camera.

The initial PIP display 7390 is just one way of presenting a composite view of the video captured by the cameras of the local device and the remote device. Some embodiments may provide other composite views. For example, a background display may present video from one of the local device's cameras, and a smaller foreground inset display may present video from the remote device's front and back cameras. Additionally, in some cases, a PIP display may include only one background video display and one foreground video display, both from the remote device. In some embodiments, the way of PIP display or the default display mode can be specified by the user.

The second stage 7340 illustrates the beginning of the video selection operation. In this example, the operation is initiated by invoking a set of selectable UI items to be displayed on the PIP display 7390. The set of selectable UI items presents options for selecting remote video for display. In some embodiments, the set of selectable UI items is invoked by selecting (e.g., by touching) any display area on the UI 7375 that is playing the remote video. In other embodiments, the various items may be invoked by selecting (e.g., by touching) anywhere on the UI 7375. Instead of, or in combination with, this invocation operation, some embodiments also allow the user to invoke the set of available functions through other operations, such as through different touch screen operations, or using one or more other physical inputs of the device. Select the UI item.

The third stage 7345 displays a UI 7375 with a set of optional UI items called 7380 for selecting remote video. In this example, a set of selectable UI items 7380 in the form of a pop-up menu is displayed in the PIP display area 7390, superimposed on the PIP display. The set of selectable UI items 7380 (which may be implemented as selectable buttons) includes a "Select R1" selectable UI item 7320 (e.g., button 7320), a "Select R2" selectable UI item 7325 (e.g., button 7325 ), a "Select Both" selectable UI item 7330 (eg, button 7330), and a "Cancel" selectable UI item 7385 (eg, button 7385). In this example, selecting the "Select R1" button 7320 will cause the UI 7375 to display only the video captured by the remote device's back camera (presented in the background display 7315). Selecting the "Select R2" button 7325 causes the UI 7375 to display only the video captured by the remote device's front camera (presented in the foreground inset display 7310). Selecting the "Select Both" button 7330 causes the UI 7375 to continue to display both videos captured by the remote device's front and back cameras. Selecting the "Cancel" button 7385 cancels the operation. In some embodiments, video captured by the local device is not affected by selections made on this menu.

The fourth stage 7350 illustrates the UI 7375 after the user has selected the "Select R1" button 7320 (e.g., by tapping the button 7320 with his finger 7365). Such selection is indicated by highlighting button 7320 on UI 7375. Some embodiments utilize different indication displays (eg, a border highlighting a selected item, or text within a selected item).

The fifth stage 7355 illustrates the animation of the UI 7375 after the user has selected the video from R1 for display. In this example, UI 7375 removes unwanted foreground inset display area 7310 by sliding it off the right edge of PIP display 7390 as indicated by arrow 7370. Other embodiments eliminate unwanted inset display areas using different animations, such as fading or fading out the inset display area, moving the inset display area in a different direction, or simply eliminating the inset display area immediately.

The sixth stage 7360 displays the UI 7375 during the video conference after the video selection operation has ended. Video display area 7310 is no longer displayed on UI 7375. At this stage, the UI 7375 presents a new PIP display 7395 that includes a video display area 7315 as a background main display, and a video display area 7305 as an inset display.

In some embodiments, this video selection operation also causes the remote device to display only the selected video, although in other embodiments, the video selection operation has no effect on the remote device. In some embodiments, this video selection action causes the remote device to stop transmitting unwanted video to the local device. In fact, in some embodiments, the cover video selection action causes the remote device's camera to stop capturing unwanted video. In some embodiments, the user of the remote device can override these effects on the remote device.

The above example illustrates the case where the selected remote view is the remote view already displayed in the background main display. In some embodiments, when the user selects a remote view displayed in one of the inset displays, the selected remote view is displayed in the background main display. In this case, some such embodiments utilize animations similar to those shown in FIG. 68 . In addition, selection of remote videos can result in changes to the image processing operations of the local dual camera mobile device, such as causing video conference manager 1604 to composite only the selected remote video or videos in a composite display in response to user input.

8. Choose to watch local view

74 illustrates the selection of local video in accordance with six operational stages 7435, 7440, 7445, 745, 7455, and 7460 of the UI 7475 displayed on the local device 7400. The first stage 7435 illustrates a UI 7475 with an initial PIP display 7490 presented during a video conference with a remote user of a mobile device having at least one camera. PIP display 7490 is similar to the PIP display in first stage 7335 in FIG. 73, except that unlike FIG. The foreground inset on the right shows 7405 a video of a man in a hat taken by the front camera of the native mobile device 7400 outside the video of the character holding the guitar captured by the mobile device's back camera. Thus, only one remote video is displayed, while two local videos are displayed.

The second stage 7440 illustrates the beginning of the video selection operation. In this example, the operation is initiated by invoking a set of selectable UI items to be displayed on the PIP display 7490 for selecting remote video for display. In some embodiments, the set of selectable UI items can be invoked by selecting (eg, by touching) any display area on UI display 7475 that is playing local video. In other embodiments, the various items may be invoked by selecting (eg, by touching) anywhere on the UI display 7475 . Instead of, or in combination with, this invocation operation, some embodiments also allow the user to invoke the set of available functions through other operations, such as through different touch screen operations, or using one or more other physical inputs of the device. Select the UI item.

The third stage 7445 displays a UI 7475 with a set of optional UI items called 7480 for selecting local video. In this example, a set of selectable UI items 7480 in the form of a pop-up menu is displayed in a PIP display area 7490, superimposed on the PIP display. The set of selectable UI items 7480 includes a "Select L1" selectable UI item 7420 (e.g., button 7420), a "Select L2" selectable UI item 7425 (e.g., button 7425), a "Select Both" selectable UI item 7430 (eg, button 7430), and a "Cancel" selectable UI item 7485 (eg, button 7485) for canceling the operation. In this example, selecting the "Select L1" button 7420 causes the UI 7475 to display only the video captured by the local device's back camera (rendered in the foreground inset display 7410). Selecting the "SelectL2" button 7425 causes the UI 7475 to display only the video captured by the local device's front camera (rendered in the foreground inset display 7405). Selecting the "Select Both" button 7430 will cause the UI 7475 to continue to display the two videos captured by the local device's two cameras, and selecting the "Cancel" button 7485 will cancel the operation. In some embodiments, video captured by the remote device is not affected by selections made through this menu.

The fourth stage 7450 illustrates the UI 7475 after the user has selected the "Select L2" button 7425 (e.g., by tapping the button 7425 with his finger 7465). Such selection is indicated by highlighting button 7425 on UI 7475. Some embodiments utilize different indication displays (eg, a border highlighting a selected item, or text within a selected item).

The fifth stage 7455 shows an animation of the UI 7475 after the user has selected the video from L2 for display. In this example, UI 7475 removes unwanted foreground inset display 7410 by sliding unwanted foreground inset display 7410 off the left edge of PIP display 7490 as indicated by arrow 7470. Other embodiments eliminate unwanted inset display areas using different animations, such as fading or fading out the inset display area, moving the inset display area in a different direction, or simply eliminating the inset display area immediately.

The sixth stage shows UI 7475 during a video conference after the video selection operation is completed. The video display area 7410 is no longer on the UI 7425. At this stage, the UI 7475 presents a new PIP display 7495 that includes the remote video display 7415 as the background main display, and the local video display 7405 as the inset display. In some embodiments, this video selection operation only affects the local display, since both videos are still transmitted to the remote device. Other embodiments stop taking video from the removed camera.

The above example illustrates the case where the selected local view is the view already displayed in the background main display. In some embodiments, when the user selects a local view displayed in one of the inset displays, the selected local view is displayed in the background main display. In this case, some such embodiments utilize animations similar to those shown in FIG. 68 . Other embodiments will utilize an inset remote view when the local view in the background main display is removed.

Similar to the remote view selection operation described above with reference to FIG. 73 , the selection of the local video results in a change in the image processing operation of the local dual camera mobile device, such as causing the video conferencing manager 1604 to composite only the selection in the composite display in response to user input. One or more remote videos of . Selection of local video may also result in a change in the operation of one or more cameras of the local device. For example, some embodiments cause an unwanted video camera to stop transmitting unwanted video to a remote device, while other embodiments stop the camera from taking unwanted video.

9. Select Send local view

The subsections above provide examples of modifications to the video display in a meeting. Some embodiments also allow a user of a dual camera mobile device to select which camera to use for a video conference before starting the video conference. Figure 75 illustrates the operation of selecting one of the two videos captured by the user's dual camera mobile device for video conferencing prior to the meeting.

Figure 75 illustrates the selection of local video for videoconferencing in terms of eight operational stages of UI 7500. The first stage 7502 illustrates the UI 7500 of a dual camera mobile device 7518 with an initial PIP display 7542 presented after a user requests to start a video conference with a remote user of the mobile device.

As shown in the first stage 7502, the initial PIP display 7542 includes two video displays: a background main display 7520 and a foreground inset display 7522. Background main display 7520 and foreground inset display 7522. Background main display 7520 occupies most of the display screen of the device, while foreground inset display 7522 is smaller and overlaid on background main display. In this example, background display 7520 presents a video of a character holding a guitar, presumably captured by the device's back camera. The foreground inset display 7522 presents a video of a figure wearing a hat, which in this example is assumed to be video taken by the device's front camera.

The initial PIP display 7542 is just one way of presenting a composite view of the video captured by the local device's camera. Some embodiments may provide other composite views. For example, a background display may present video from the device's front camera, and a smaller foreground inset display may present video from the device's back camera. Additionally, some embodiments allow two videos to appear in UI 7500 in two side-by-side display areas (e.g., display windows left and right, or display windows above and below), or in two display areas arranged diagonally. In some embodiments, the way of PIP display or the default display mode can be specified by the user. Below the PIP display is a selectable UI item 7540 (eg, button 7540) labeled "End Conference," which allows the user to end the video conference by selecting this item.

In the first stage 7502, the user of the mobile device 7518 has requested a video conference with a remote user and is waiting for the remote user to respond. The "Preview, Waiting for response..." annotation at the bottom of the display illustrates this waiting period.

The second stage 7504 illustrates the beginning of the video selection operation. In this example, the operation is initiated by invoking a set of selectable UI items to be displayed on the PIP display 7542. The set of selectable UI items presents various options for selecting local video for transmission to the remote device of the video conference. In some embodiments, the set of selectable UI items can be invoked by selecting (eg, touching) anywhere on the UI display 7500 within the pre-meeting time while waiting for the remote user to respond. Instead of, or in combination with, this invocation operation, some embodiments also allow the user to invoke the set of available functions through other operations, such as through different touch screen operations, or using one or more other physical inputs of the device. Select the UI item.

The third stage 7506 illustrates the UI 7500 with a set of selectable UI items 7526 invoked for the user to select a video. In this example, a set of selectable UI items 7526 in the form of a pop-up menu is displayed in the PIP display area 7542, superimposed on the PIP display. In this example, the set of selectable UI items includes: "Transmit L1" item 7528 (e.g., button 7528); "Transmit L2" item 7530 (e.g., button 7530); "Transmit Both" item 7532 (e.g., button 7530); button 7532); and a "Cancel" item 7534 (eg, button 7534). In this example, selecting the "Transmit L1" button 7528 will cause the UI 7500 to transmit only the video captured by the device's back camera to the remote device during the video conference. Selecting the "Transmit L2" button 7530 will cause the UI 7500 to transmit only the video captured by the device's front camera to the remote device during a video conference. Selecting the "Transmit Both" button 7532 will cause the UI 7500 to transmit both videos captured by the device's front and back cameras to the remote user of the video conference, and selecting the "Cancel" button 7534 will cancel the operation.

The fourth stage 7508 illustrates the UI 7500 after the user has selected the "Transmit L1" button 7528 (e.g., by tapping the button 7528 with his finger 7524). This selection is indicated by highlighting button 7528 on PIP display area 7542. Some embodiments utilize different indication displays (eg, highlighting a border of a selected item or text within a selected item).

The fifth stage 7510 illustrates the animation of the UI 7500 after the user has selected the video from the device's back camera for transmission to the remote device. In this example, UI 7500 removes unwanted foreground inset display 7522 by sliding it off the right edge of PIP display 7542 as indicated by arrow 756. In the sixth stage 7512, the inset display 7522 has been completely removed from the PIP display area 7542. Different embodiments utilize different animations to remove unwanted display areas, such as fading or fading the display area, moving the display area in a different direction, or simply eliminating the display area immediately.

The seventh stage 7514 illustrates the animation of the UI 7500 after the remote user has accepted the video conference request. Highlight the acceptance of the video conference request by dismissing the "Preview, Waiting for response..." annotation on the display. At this stage, the background display area 7520, which is video from the device's back camera, tapers down to the lower left corner of the PIP display area 7542, as indicated by arrow 7538. The background display 7520 is zoomed out, enabling the UI 7500 to display a display area 7544 behind the display area 7520 that contains video from the remote user's camera. Some embodiments zoom out the local camera to a different location, utilize a tiled composite display of the two videos being displayed, or make the remote view an inset display area of the PIP display.

The eighth stage 7516 represents the UI 7500 after the video selection operation is completed. UI 7500 presents a new PIP display 7546 that includes an inset display 7520 of video captured from the local device, and a background display 7544 of video transmitted from the remote device.

B. Bandwidth & Frame Rate

In some embodiments, resizing of the remote mobile device's display area during the videoconference causes the local mobile device to reallocate each image captured by the local mobile device's two cameras (i.e., the front camera and the back camera). The bandwidth of the video. Figure 76 illustrates two examples of such bandwidth reallocation between two cameras at the local device.

Each example in FIG. 76 involves the back camera sensor 7605 of the local device, the front camera sensor 7610 of the local device, the video conferencing module 7615 of the local device, and the UI 7635 of the remote mobile device 7620. The back camera sensor 7605 and front camera sensor 7610 capture video from the respective back and front cameras of the local device. The captured video is sent to the video conferencing module 7615, which processes the captured video and transmits them to the remote device for display in the UI 7635.

In FIG. 76, the remote device's UI 7635 presents a composite display. The composite display represents the video captured by the local device's front and back cameras. Video from the front camera captures trees and a man in a hat, while video from the rear camera captures a mountain landscape. As shown in FIG. 76, the two videos can be displayed in the UI 7635 in many different ways depending on the arrangement of the display area used to display the video and the size of the display area. In each instance, the video conferencing module 7615 initially divides the total output bandwidth between each video according to the relative size of the display area in the remote device. Specifically, video displayed in a larger display area in UI 7635 is allocated a greater portion of the total bandwidth, and video displayed in a smaller display area in UI 7635 is allocated a smaller portion of the bandwidth. In some embodiments, when the videos are displayed in the same sized display area, the total output bandwidth is divided equally between the two videos.

The amount of bandwidth allocated to each of the two videos will most affect how each video is processed. For example, a video may require more bandwidth than is allocated for the video. In this case, the frame rate of the video is adjusted, or the size of the video image is scaled down to fit the smaller bandwidth. Reducing the frame rate of a video causes the video to appear "choppy", while scaling down the size of the video image reduces the area in which the video is displayed. Thus, when a video is allocated a certain amount of bandwidth, some embodiments adjust the frame rate of the video, scale down the size of the video image, or a combination of both, in order to ensure that the video can be delivered within the allocated bandwidth. Those of ordinary skill in the art will recognize that adjustments to the frame rate and average frame size can be varied to obtain the best overall video quality while still ensuring that the video can be delivered within the allocated bandwidth.

Example (1) of Figure 76 illustrates a situation of bandwidth reallocation with two phases of operation of the UI 7635. The UI 7635 of the remote device 7620 in the first stage 7670 presents a composite display comprising two displays: one at the top of the UI 7635 and another at the bottom of the UI 7635. In this example, the top display area 7625 displays video captured by the local device's front camera, and the bottom display area 7630 displays video captured by the local device's rear camera. As shown in the first stage 7670 , the top display area 7625 is larger than the bottom display area 7630 . Thus, video from the front camera of the local device is allocated 80% of the bandwidth and video from the back camera of the local device is allocated 20% of the bandwidth. To ensure that the video from the back camera of the local device can be transmitted from the local device to the remote device within the allocated bandwidth, the frame rate and/or scaling of the video is adjusted.

The second stage 7675 illustrates the UI 7635 after the user of the remote device has increased the size of the bottom display area such that the top display area 7625 and the bottom display area 7630 are approximately the same size. As a result, 50% of the total bandwidth is reallocated by the video conferencing module 7615 for each video.

Example (2) of Figure 76 illustrates another situation of bandwidth reallocation with two phases of operation of the UI 7635. In the first stage 7680 of example (2), the UI 7635 presents a PIP display. The PIP display consists of two displays: a background main display area 7650 and a foreground inset display area 7655 . The background main display area 7650 occupies most of the PIP display, while the foreground inset display area 7655 is smaller and overlaps the background main display area 7650 . In this example, background display area 7650 presents video captured by the device's front camera. Inset display area 7655 presents video captured by the device's back camera. As shown at this stage, background display area 7650 is larger than inset display area 7655 . Thus, video from the front camera of the device is allocated 80% of the bandwidth and video from the back camera of the device is allocated 20% of the bandwidth. To ensure that video from the back camera of the local device can be transmitted from the local device to the remote device within the allocated bandwidth, the frame rate and/or scaling of the video is adjusted.

The second stage 7685 illustrates the UI 7635 after the user of the remote device has exchanged the display of the two videos. Specifically, background display area 7660 now presents video captured by the device's back camera, and inset display area 7665 now presents video captured by the device's front camera. Since the size of the display area for these two videos has changed, the video from the device's back camera is allocated 80% of the bandwidth and the video from the device's front camera is allocated 20% of the bandwidth. Thus, the frame rate and/or scale size of the video from the local device's front camera will be reduced. One of ordinary skill in the art will recognize that the bandwidth distribution illustrated in FIG. 76 is an example only and that other techniques for distributing bandwidth between two cameras during a video conference are possible.

1. Frame rate control

Similar to the frame rate control operation in a conference described above, some embodiments may wish to individually adjust or maintain the rate at which video images captured by each camera of a dual camera mobile device are transmitted to the other device in the video conference. Some of these embodiments provide similar techniques to those described above. For example, some embodiments control the frame rate of each camera by adjusting the VBI of each camera's sensor module 415 . Other embodiments also provide additional techniques such as frame dropping that may be performed by each camera's sensor module 415 and/or general transmit buffer 3120 .

2. Bandwidth control via scaling

As noted above, during a video conference between a dual camera mobile device and another device, there is a limited amount of image data that can be transferred over one or more network connections (ie, network connection bandwidth) within a certain amount of time. In order to maximize and maintain the throughput of the network connection, different embodiments of the dual camera mobile device provide different ways to control the amount of image data transmitted over the network connection within a certain amount of time. In some embodiments, throughput is the average rate of successful message delivery over a communication channel (eg, a network connection).

When transferring images captured by both cameras of a dual-camera mobile device, one such method resizes images from one or both cameras of a dual-camera mobile device to control the amount of image data transferred over a network connection . Some embodiments scale down the size of the image captured by the dual camera mobile device to reduce the amount of image data transferred over the network connection, while other embodiments scale up the size of the image to increase the amount of image data transferred over the network connection .

Some embodiments maintain the aspect ratio of the image when scaling (ie, scale evenly). Other embodiments scale the image such that the aspect ratio of the scaled image is different from the aspect ratio of the original image (ie, anamorphic scaling).

In addition, scaling can be performed at different stages of the image processing process. Scaling for some embodiments may be performed by a camera sensor. In such embodiments, the camera sensor may discard rows or columns of data (ie, pixel values) of the image. In some of these embodiments, the remaining image data is interpolated to smooth the shape of the image.

The scaling of other embodiments is performed by the scaler module 455 of the CIPU 400. In some embodiments, scaling is performed by the videoconferencing manager 1604, as described above, and in other embodiments, scaling is performed by an encoder. Thus, different embodiments of dual camera mobile devices scale differently.

3. Bit rate control

Some embodiments provide different mechanisms to manage the bitrate for encoding video captured by the cameras of a dual camera mobile device. In some embodiments, a dual camera mobile device includes a rate controller for each camera. Some embodiments provide a fixed bit rate management scheme. In this scheme, each rate controller is set to a fixed bit rate so that the total bit rate of the video from both cameras on the mobile device is constant. Other embodiments provide a priority scheme where one of the two videos from the device's camera always gets a higher priority than the other when reducing the overall bitrate is required.

In some embodiments, the arbiter module manages two rate controllers for the two cameras. Figure 77 illustrates an example of such an arbiter module. As shown in FIG. 77, the rate controller 7700 sets the bit rate of the front camera, and the rate controller 7705 sets the bit rate of the back camera. The rate controller sends the image from the camera sensor to the encoder 7715. The arbiter module 7710 interfaces with the two rate controllers and controls the ratio of the bit rate of each rate controller 7700 and 7705 in any number of ways based on factors such as available bandwidth, video size of each of the two videos, etc. set up. To ensure that both videos can be sent to the remote device under the available bandwidth. Additionally, the arbiter 7710 may be configured to implement the above-mentioned fixed rate scheme, or a priority scheme.

In some other embodiments, the two rate controllers for the two cameras can communicate with each other. In this scheme, the rate controllers can exchange information about their respective videos and set the video bit rate accordingly. Several examples of rate controller rate management mechanisms are provided. However, many other different mechanisms are possible.

4. Video processing

Some embodiments of a dual camera mobile device process images captured by the two cameras of the dual camera mobile device differently in different situations. For example, when processing a PIP composite image that includes images captured by both cameras of a dual camera mobile device, some embodiments selectively perform TNR processing 2000 on the PIP composite image. Some such embodiments perform TNR processing 2000 only on the main image in the PIP composite image, while other such embodiments perform TNR processing 2000 on only the insert image in the PIP composite image.

As another example of processing images captured by the two cameras of a mobile device, some embodiments respond to various changes in the video conference, such as user adjustments to the display area where the video is displayed (e.g., zooming in on the defining regions of interest in displayed video, swapping main/insert display areas for PIP displays, etc.), changes to the total available bandwidth, etc., scaling images captured by the two cameras of a dual-camera mobile device . Some such embodiments scale images in the manner described above. That is, the image may be scaled by the encoder 1655, the video conference manager 1604, the scaler module 455, and the camera sensor (ie, 405a or 405b) with which the image was captured.

5. Coding

As noted above, some embodiments transmit video from both cameras of a dual camera mobile device. Thus, these embodiments can encode video captured by two cameras for transmission to a remote device during a video conference. Different embodiments provide different ways to encode video for transmission. Figure 78 illustrates a method of processing video for transmission using a multiplexer (MUX) 7815, encoder module 7825, buffer 7830, and combiner module 7835.

According to the selection signal, MUX 7815 obtains an input signal and outputs the selected input signal to encoder 7825. For example, if the select signal instructs the MUX 7815 to get an input signal from C1, then the MUX 7815 selects that input signal and outputs that input signal. The selection signal may be provided in a variety of ways, such as through instructions from the videoconferencing manager 1604 . Through the MUX 7815, the encoder 7825 alternately encodes the images received from the MUX 7815 into a bitstream format and stores the encoded images in the buffer 7830. The combining module 7835 combines (ie, multiplexes) one or more bitstreams held in the buffer 7830 and outputs a single bitstream.

The operation of this encoding method is now described in terms of three stages 7860, 7865 and 7870. In the first stage 7860, the MUX 7815 is configured to receive the image 7805 captured by the camera C1 and output to the encoder 7825 for encoding. The encoder 7825 encodes the received image and generates a bitstream 7850 , which is then saved in the buffer 7830 . The second stage 7865 is similar to the first stage 7860, except that the MUX 7815 is configured to receive the image 7810 captured by the camera C2 and output it to the encoder 7825 for encoding. Likewise, the encoder encodes the received image and generates a bitstream 7855 , which is stored in the buffer 7830 . In the third stage 870, the combining module 7835 retrieves the bitstreams 7850 and 7855 from the buffer 7830 and combines them into one bitstream for transmission to the remote device.

Figure 79 illustrates another method of encoding two videos from a dual camera mobile device for transmission to a remote device during a video conference. In this method, a video frame (i.e., an image) from the first camera of the mobile device and another video frame from the second camera of the mobile device are combined into one video frame, after which the combined video frame is encoded into a bitstream for sending to a remote device. As shown in FIG. 79 , this method includes a combiner 7915 , a buffer 7920 and an encoder 7925 .

As shown, combiner 7915 combines image 7905 from the first camera and image 7910 from the second camera to form composite image 7955 . Different embodiments composite images 7905 and 7910 differently. For example, the combiner 7915 of some embodiments may combine images by arranging two images next to each other (as shown in Figure 80). Composite images 8030 and 8035 illustrate two example composite images utilizing this technique. In composite image 8030, image 7905 from the first camera is arranged over image 7910 from the second camera. And composite image 8035 shows that image 7905 is arranged on the left side of image 7910 .

In some other embodiments, the compositor 7915 may composite the two images 7905 and 7910 by superimposing the two images 7905 and 7910 on top of a larger background image. Composite image 8040 of FIG. 80 illustrates an example composite image utilizing this technique. In composite image 8040, images 7905 and 7910 are arranged diagonally and superimposed on a blank image (ie, image 7905 is in the upper left corner of the background image and image 7910 is in the lower right corner of the background image). In some embodiments, the camera sensors may be of different sizes, thereby capturing images with different pixel resolutions. In such an embodiment, combiner 7915 may combine images 7905 and 7910 in a similar manner as illustrated by combined image 8045 of FIG. 80 . After combining the two images, the combiner 7915 saves the combined image in the buffer 7920. Encoder 7925 retrieves the composite image from buffer 7920, encodes the composite image into a bitstream, and sends it to the remote device of the video conference.

Referring now to the synthesizer 7915, buffer 7920, and encoder 7925 illustrated in FIG. 79, individual operations are described. First, the first camera sends an image 7905 to the compositor 7915 as part of a series of images in the video. At the same time, the second camera sends the image 7910 to the compositor 7915 as part of a series of images in the video. Compositor 7915 then composites images 7905 and 7910 in the manner described above to form composite image 7955 . Afterwards, the synthesizer 7915 sends the synthesized image 7955 to the buffer 7920 . The buffer 7920 then holds the composite image before sending the composite image to the encoder 7925. Finally, the encoder 7925 encodes the composite image into a bitstream and sends it to the remote device of the video conference.

Figure 81 illustrates yet another method of encoding two videos from a dual camera mobile device for transmission to a remote device during a video conference. In this method, two videos from the device are displayed as a composite display, a screenshot of the composite display is taken, and encoded into a bitstream for sending to the remote device. As shown in FIG. 81, this method includes an encoder 8115. In some embodiments, the encoder 8115 encodes the composite image and sends it to the remote device.

Referring now to the encoder 8115 illustrated in FIG. 81, various operations will be described. First, video from both cameras of a dual camera mobile device is displayed in a composite display on the device's screen. The composite display is capable of rendering video in any manner. For example, a composite display in some embodiments can present two videos in a PIP display, such as PIP display 8105 illustrated in FIG. 81 . In other embodiments, the composite display may present the two videos in two side-by-side display areas, or in two diagonally arranged display areas. Take a screenshot of the PIP display 8105, say image 8110, and send it to the encoder 8115. The encoder then encodes the series of screenshots into a bitstream 8120, which is then sent to the remote device in the video conference. Although several different methods of encoding the two videos are described above, other methods are possible.

6. Decoding

Some embodiments of a dual camera mobile device may receive a bitstream encoded in the method described above with reference to Figures 78-81. In such an embodiment, the dual camera mobile device may receive (eg, over a video conference control channel) information indicative of the method used to encode the video. Figure 82 illustrates a method of decoding two video bitstreams received from another device over a communication network for display on a dual camera mobile device during a video conference. Specifically, this method is used to decode a bit stream encoded by the encoding method explained above with reference to FIG. 78 .

As shown in FIG. 82 , this scheme uses a separation module 8235 , buffers 8230 and 8290 , and a decoder module 8225 . The splitting module 8235 splits (ie, demultiplexes) the bitstream into one or more bitstreams and saves the bitstreams in the buffer 8230 . Decoder 8225 retrieves the encoded bitstreams, decodes them to produce video, and then saves the video in buffer 8290.

Referring now to the separation module 8235, buffers 8230 and 8290, and decoder module 8225 illustrated in FIG. 82, the operation of this method is described. First, the dual camera mobile device receives a bitstream 7845 from another device in the video conference (ie, at the networking manager 1614) over the communication network. The split module 8235 splits the received bit stream into two bit streams 8255 and 8260 because the received bit stream is a multiplexed bit stream of the two bit streams. Each encoded bitstream represents video data captured from one of the device's two cameras. Subsequently, the separation module 8235 saves the bitstreams 8255 and 8260 in the buffer 8230 .

Afterwards, decoder 8225 retrieves bitstream 8250 (which is one of two bitstreams 8255 and 8260) from buffer 8230, decoder 8225 decodes bitstream 8250 to generate video 8280, and saves video 8280 in buffer device 8290. Decoder 8225 also decodes the other of bitstreams 8255 and 8260 and saves the resulting video in buffer 8290. Now, both videos can be retrieved from buffer 8290 and saved or displayed on the dual camera mobile device.

FIG. 83 illustrates a method of decoding a bitstream encoded with the method explained with reference to FIG. 79 . As shown in FIG. 83 , the method includes a decoder 8325 , a buffer 8320 and a desynthesizer 8315 .

In some embodiments, decoder 8325 receives a bitstream encoded using the method illustrated in FIG. 79 , decodes the bitstream into one or more composite images, which are then stored in buffer 8320 . Decompositor 8315 extracts the two images from each composite image. To extract the two images from the composite image, the decompositor 8315 also receives information indicating the position of each image within the composite image (e.g., via a videoconferencing communication control channel, from a device in the videoconferencing that composited and encoded the images information received).

Referring now to the decoder 8325, buffer 8320 and desynthesizer 8315 illustrated in FIG. 83, the operation of this method will be described. First, the decoder 325 receives a video bitstream from another mobile device of the video conference, such as the video bitstream generated by the method explained with reference to FIG. 79 . Decoder 8325 decodes the bitstream into one or more composite images including composite image 7955 and stores them in buffer 8320. The buffer 8320 then holds the composited images before sending them to the decompositor 8315. When the decompositor receives composite image 7955 from buffer 8320, it splits composite image 7955 into two images 7905 and 7910 identical to images 7905 and 7910 in FIG. 79 .

When a bitstream is received from a system such as the system illustrated in Figure 81, a decoder, such as decoder 8325 in Figure 83, decodes the bitstream into a series of screenshots. The series of screenshots are displayed as a video on the screen of the device without further processing.

VI. Multiple sources

As mentioned above, video can be captured by both cameras of a dual camera mobile device and sent to another device in a video conference. Rather than delivering video captured from both cameras of a dual camera mobile device, some embodiments may deliver different media content or arbitrary content displayed on a dual camera mobile device along with video captured from one camera of the dual camera mobile device . In other words, these embodiments are capable of delivering content from multiple sources along with video captured by the cameras of the dual camera mobile device.

Figure 84 conceptually illustrates another software architecture for the video conferencing and processing module of a dual camera mobile device of some embodiments. The video conferencing and processing module of FIG. 84 is similar to the video conferencing and processing module 1600 of FIG. 16, except that the video conferencing and processing module includes a display driver 8485 and memory 8475, and the media exchange module 1620 includes a media source module 8470 and a screen capture module 8480 outside.

The media source module 8470 of some embodiments routes media content between the video conferencing module 8402 and the storage 8475. Examples of media content include video, images, documents, and music. Other embodiments store other types of media content in memory 8475. The memory 8475 of some embodiments is internal memory (eg, RAM), while the memory 8475 of other embodiments is external memory (eg, Compact Flash (CF) card, Secure Digital (SD) card, etc.).

In some embodiments, the screen capture module 8480 routes images of the content displayed on the display of the dual camera mobile device through the display driver 8485. In some embodiments, the display driver 8485 is responsible for capturing content on the display and converting the content into images. Different embodiments capture different content displayed on the display. For example, some embodiments capture all content displayed on the display. Other embodiments capture a specific display area of the display (eg, the display area of the currently active window, the display area of a PIP display, etc.).

Referring now to FIG. 84, some example operations of the video conferencing and processing module are illustrated. To deliver media content along with video captured from a camera of a dual camera mobile device, video conferencing module 8402 of some embodiments performs the same operations as video conferencing module 1602 described above in FIG. 16, except instead of retrieving images from CIPU 1650 , the video conference manager 1604 retrieves the media content from the storage 8475 through the media source module 8470 . To transmit images of the content displayed on the display of the dual camera mobile device, some embodiments of the video conferencing manager 1604, through the display driver 8485, retrieve images of the content displayed on the display of the dual camera mobile device. Some embodiments perform similar processing (e.g., perspective correction, scaling, etc.) .

The above discussion describes several examples of delivering content from various sources along with video captured by one camera of a dual camera mobile device. However, other embodiments may deliver other different types of content. For example, in a video conference involving multiple participants, some embodiments transmit video received from one device in the video conference and video captured by a camera of a dual camera mobile device to another device. Thus, any number of different types of content from any number of sources may be delivered along with video captured by one camera of a dual camera mobile device.

VII. Multi-party video conferencing

The above sections dealing with video conferencing describe a video conference with two participants. However, multiparty video conferencing (ie, more than three participants) is also possible with mobile devices of some embodiments. In some embodiments, all participants in a multiparty video conference can see and hear each other. Other embodiments provide a multicast video conference where one participant (e.g., the broadcaster) can see and hear all other participants, and all other participants can see and hear the broadcaster, but Other participants cannot see or hear each other (unless, for example, with the broadcaster's approval).

A. User Interface for Multiparty Video Conferencing

During a multi-party video conference, some embodiments provide a variety of different UIs for displaying the participants of the video conference and selecting one or more specific participants to view. For example, some embodiments of a mobile device provide a UI that simultaneously displays all participants of a multi-party video conference and allows a user of the mobile device to select one of the participants for viewing (eg, by zooming in on the image of the selected participant). Figure 85 illustrates an example of such a UI.

FIG. 85 illustrates the simultaneous display of all participants of a multiparty video conference in the UI 8530 of the mobile device 8500 with reference to five different stages 8505, 8510, 8515, 8515, 8520, and 8525 of the UI 8530, and the selection of one of the participants for viewing. series of operations. The first stage 8505 illustrates the UI 8530 after establishing a multi-party video conference between three other users of other devices. As shown, UI 8530 includes composite display 8535 and display area 1155. Composite display 8535 includes four display areas 8565, 8570, 8575, and 8580 that display images captured by cameras of participants in the multi-party video conference. In this example, display area 8565 represents the user of mobile device 8500 (ie, display area 8565 displays an image captured by the front camera of mobile device 8500). The display area 1155 is the same as the display area 1155 previously described in FIG. 12 .

The second stage 8510 represents the user of the mobile device 8500 beginning a participant selection operation by selecting one of the display areas of the composite display area 8530 . In particular, the second stage 8510 represents the user selecting the display area 870 (eg, by tapping a finger 8550 on the display area 8570).

The third stage 8515 of the UI 8530 illustrates the composite display 8555 after completion of the participant selection operation. Some embodiments provide an animation (not shown) showing the transition between the second stage 8510 and the third stage 8515. Composite display 8555 includes a PIP display 8560 that displays the participant's display area (i.e., display area 8570) selected in the second stage 8510 as a background display area, and a user's display area 8565 as a PIP display 8560 Insert display area. In this example, PIP display 8560 displays an image of the selected display area 8570 stretched horizontally to fit a landscape orientation. In some embodiments, the image is not stretched, and the image of the selected display area maintains its portrait orientation (ie, black bars fill the excess space on either side of the background display area, as shown in FIG. 36 ). In addition, composite display 8555 also includes composite display 8585 that displays scaled-down images of two unselected display regions 8575 and 8580 .

The fourth stage 8520 represents the user of the mobile device 8500 beginning a participant deselection operation by selecting the PIP display 8560 (eg, by tapping a finger 8550 on the PIP display 8560). The fifth stage 8525 illustrates the composite display 8535 after the participant deselection operation is complete.

FIG. 85 shows an exemplary sequence of operations for simultaneously displaying all participants of a multi-party video conference, performing a participant selection operation, and performing a participant deselection operation. Other sequences of operations are also possible. For example, after the third stage 8515, instead of initiating a participant deselection operation, the user can select one of the unselected display areas displayed in composite display 8585 to exchange the newly selected display area in display area 8585 with that of PIP display 8560. The background display area (ie, the previously selected display area). Thus, during a multi-party video conference, the user can swap the display area in display area 8585 and the background display area of PIP display 8560 at any time and any number of times. Additionally, at any time during the multi-party video conference, the user can perform a participant deselect operation to return to composite display 8535 . In addition, different embodiments allow the user to select specific participants in different ways, such as toggling a switch on the mobile device 8500, by issuing voice commands, and the like.

Some embodiments provide techniques for automatically selecting participants, eg, based on voice detection. In such an embodiment, when one of the participants speaks, that participant's display area is automatically selected as the background display area of the PIP display 8560. When a different participant speaks, that participant's display area is automatically selected as the background display area of the PIP display 8560. In some embodiments, the display displays composite display 8535 after a prescribed amount of silence (eg, 3 seconds) when none of the participants in the multi-party video conference is speaking. In some embodiments, when the user of the mobile device 8500 speaks, nothing happens on the UI 8530 of the mobile device 8500.

FIG. 86 illustrates another example sequence of operations for simultaneously displaying all participants of a multi-party video conference and selecting one of the participants for viewing. Figure 86 illustrates this operation in the UI 8645 of the mobile device 850 with reference to seven different stages 8505, 8605, 8610, 8615, 8620, 8625, and 8630 of the UI 8645. The first stage 8505 is the same as the first stage 8505 illustrated in Figure 85 in that it shows the UI 8645 after establishing a multi-party video conference between three other users of other devices.

The second stage 8605 illustrates that the user of the mobile device 8500 initiates a participant selection operation by selecting the display area 8570 (eg, by placing two fingers on the display area 8570). The third stage 8610 represents the transition stage of the participant selection operation. At this stage, the user drags two fingers away from each other, making the display area 8570 larger and filling the display area used to display the composite display 835 . This example shows display area 8570 being selected, but any of the other display areas 8565, 8575, and 8580 may also be selected. In some embodiments, a user of mobile device 8500 is not permitted to select the user's display area (ie, display area 8565 in this example).

The fourth stage 8615 of the UI 8645 shows the PIP display 8635 of the UI 8645 after the participant selection operation is completed. Some embodiments require the user to continue dragging the two fingers away from each other until the display area 8570 fills the background display area 8640 of the PIP display 8635, while other embodiments only require the user to drag more than a certain amount before the user removes the fingers. A threshold amount of (eg, greater than a certain distance, or greater than a certain amount of time). When the user's drag operation meets or exceeds a certain threshold amount, the UI 8645 continues the enlargement of the display area 8570 until it fills the background display area 8640 of the PIP display 8635. Otherwise, the participant selection operation is not complete and the UI 8645 returns to the composite display 8535. As shown, the selected display area (ie, display area 8570 ) is the background display area 8640 of the PIP display 8635 and the user's display area 8565 is the inset display area of the PIP display 8635 . Some embodiments provide an animation (not shown) showing the transition between the third stage 8610 and the fourth stage 8615.

The fifth stage 8620 illustrates that by selecting the background display area 8640 of the PIP display 8635 (eg, by placing two fingers on the background display area 8640 ), the user of the mobile device 8500 initiates a participant deselection operation. The sixth stage 8625 represents the transition stage of the participant deselection operation. This stage illustrates the user dragging the fingers closer to each other to shrink the display area used for the background display area 8640 of the PIP display 8635 . Similar to the operations described in the third stage 8610, some embodiments require that the user's drag operation be greater than a certain threshold amount (eg, greater than a certain distance, or greater than a certain amount of time) before the user removes the finger. Otherwise, the participant deselection operation is not complete and UI 8645 returns to PIP display 8635. The seventh stage 8630 of the UI 8645 represents the composite display 8535 after completion of the participant deselection operation.

FIG. 86 illustrates another example sequence of operations for simultaneously displaying all participants of a multi-party video conference, performing a participant selection operation, and performing a participant deselection operation. However, some embodiments allow the user of mobile device 8500 to repeatedly perform participant selection operations and participant deselection operations. Figure 87 illustrates one such embodiment.

In particular, FIG. 87 illustrates an example sequence of performing participant selection operations and participant deselection operations multiple times with reference to seven different stages 8505, 8705, 8615, 8710, 8715, 8720, and 8725 of the UI 8730. The first stage 8505 is the same as the first stage 8505 of FIGS. 85 and 86 mentioned above. The second stage 8705 is similar to the second stage 8605 of FIG. 86, except that the user selects the display area 8570 by tapping the display area 8570 once (instead of placing two fingers on the display area 8570). The third stage 8615 is the same as the fourth stage 8615 of FIG. 86 in that it represents the PIP display 8635 after completion of the participant selection operation. The fourth stage 8710 is similar to the fifth stage 8620 of FIG. 86, except that the user selects one of the background display areas 8640 of the PIP display 8645 by tapping the background display area 8640 once (instead of placing two fingers on the background display area 8640). outside.

The fifth stage 8715 is the same as the seventh stage 8630 of FIG. 86 in that it represents the composite display 8535 after the participant has completed the deselection operation. The sixth stage 8720 is similar to the second stage 8510 except that the display area 8575 is manipulated for participant selection. Similarly, seventh stage 8725 is similar to third stage 8705 in that it represents the selected display area (ie, display area 8575 ) as background display area 8640 of PIP display 8635 . Although FIG. 87 shows only a few participant selection and participant deselection operations, any number of such operations can be performed during a multiparty video conference.

Additionally, some embodiments provide a UI capable of displaying different numbers of participants during a video conference. For example, the UI of some embodiments displays only some of the participants in a multi-party video conference when the mobile device is held upright (i.e., oriented in portrait orientation) and others when the mobile device is held in landscape orientation (i.e., oriented in landscape). participants. Other embodiments display all participants when the mobile device is held in landscape orientation. Additionally, some embodiments provide animations to indicate transitions between different positions and/or orientations of a mobile device similar to those illustrated in FIGS. 34 , 35 , 36 and 37 . Other different animations are also possible.

As another example of a UI displaying different numbers of participants during a video conference, some embodiments allow a user of a mobile device to select multiple participants to view simultaneously during a video conference. For purposes of illustration, referring to the first stage 8505 of FIG. Corresponding display area in Composite Display 8535). The selected display area can then be displayed in a variety of ways, such as composite display, PIP display, any of the display arrangements illustrated in Figure 65, and other various multi-participant display arrangements. Furthermore, while examples of some embodiments are described, those of ordinary skill will recognize that different embodiments can select and display multiple participants in a multi-party video conference in any number of ways.

B. User Interface for Multicast Videoconferencing

As mentioned above, a multicast video conference allows only one participant to hear and see all other participants, and the other participants cannot hear or see each other. To facilitate multicast video conferencing, some embodiments provide a number of different UIs for displaying the broadcaster and other participants of the multicast video conference. For example, some embodiments provide a student-teacher UI layout, similar to that of the third stage 8515 illustrated in FIG. 85 . Thus, referring now to said third stage 8515, the student-teacher UI layout of some embodiments is illustrated.

In these embodiments, only the broadcaster is displayed in the entire display area of the PIP display 8560 (ie, the inset display area is not displayed). The other participants in the multicast video conference are displayed below the PIP display 8560 , similar to the display area shown in the composite display 8585 . Similar to the above, in some embodiments, a specified number of other participants may be displayed in composite display 8585 when the mobile device is in portrait mode, and additional participants may be displayed in composite display 8585 when the mobile device is in landscape mode. Or all participants. Additionally, other embodiments provide a different UI that displays the broadcaster and other participants of the multicast video conference.

C. Control the audio of a multi-party video conference

Additionally, the mobile device of some embodiments provides different techniques for controlling the audio of multiple participants in a multi-party video conference. For example, some embodiments of a mobile device allow a user of the mobile device to control the audio for each participant in a multiparty video conference through a single set of volume controls (eg, a volume slider) displayed on the UI of such an embodiment. In other embodiments, the mobile device allows the user of the mobile device to individually control the volume of the audio for each participant in the multiparty video conference through a set of volume controls, such as a volume slider displayed in each participant's display area . Some embodiments provide only a mute button, rather than a set of volume controls. Thus, in some such embodiments, the user of the mobile device can only mute or unmute all participants of the multiparty video conference, while in other such embodiments, the user of the mobile device can individually mute or unmute the multiparty video conference each participant. Additionally, other techniques of controlling the audio of participants in a multi-party video conference are possible, such as by toggling a switch on a mobile device, by issuing voice commands, and the like.

VIII. Electronic system

Many of the above-described features and applications are implemented as software processes specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (eg, one or more processors, cores of processors, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, and the like. Computer-readable media do not include carrier waves and electrical signals transmitted wirelessly or through wired connections.

In this specification, the term "software" is intended to include firmware residing in read-only memory, or applications stored in magnetic storage, which can be read into memory for processing by a processor. Additionally, in some embodiments, multiple software inventions may be implemented as sub-parts of a larger program while still being distinct software inventions. In some embodiments, multiple software inventions may also be implemented as separate programs. Finally, any combination of separate programs that together implement the software inventions described herein is also within the scope of the invention. In some embodiments, a software program defines one or more specific machine implementations that execute and implement the operations of the software program when installed to operate on one or more electronic systems.

In environments where calling program code interacts with other program code that is called through one or more interfaces, some embodiments are implemented as a software process that includes one or more application programming interfaces (APIs). Various function calls, messages or other various calls that may further include various parameters can be transmitted between the calling program and the called code through the API. Additionally, the API may provide calling program code with the ability to use data types or classes defined in the API and implemented in the called program code.

At least some embodiments include an environment in which a calling software component interacts with a called software component through an API. One method of operating through the API in this environment includes passing one or more function calls, messages, other various calls or parameters through the API.

In some embodiments, one or more application programming interfaces (APIs) may be used. For example, some embodiments of the media exchange module 310 (or 910) provide a set of APIs to other software components for accessing the various video processing and encoding functions described in FIGS. 3 and 9, such as the TNR described in FIG. Module 1900 functions.

API is an interface implemented by a program code component or hardware component (hereinafter referred to as "API implementing component") that allows different program code components or hardware components (hereinafter referred to as "API calling component") to access and use the API provided by the API implementing component One or more functions, methods, procedures, data structures, classes and/or other services of . The AIP can define one or more parameters passed between the API calling component and the AIP implementing component.

The API allows the developer of the API-calling component (which may be a third-party developer) to take advantage of specified features provided by the API-implementing component. There may be one API calling component, or there may be more than one API calling component. An API can be a source code interface provided by a computer system or program library to support service requests from applications. An operating system (OS) may have multiple APIs to allow applications running on the OS to call one or more of these APIs, and a service (such as a library) may have multiple APIs to allow applications using the service to call one or more Multiple such APIs. The API may be specified with a programming language that can be interpreted or compiled when building an application.

In some embodiments, an API implementing component may provide more than one API, each API providing a different view of, or having a different view of, a different situation accessing a different aspect of the functionality implemented by the API implementing component. For example, an API of the API implementation component can provide the first set of functions and can be exposed to third-party developers, another API of the API implementation component can be hidden (not exposed) and provide a subset of the first set of functions, Another set of functions is also provided, such as testing or debugging functions that are not in the first set of functions. In other embodiments, the API implementation component itself can call one or more other components through the underlying API, so that it is both an API calling component and an API implementation component.

The API defines the language and parameters used by the API call component when accessing and utilizing the specified feature of the API implementation component. For example, an API call component accesses the specified features of the API implementation component through one or more API calls or activations exposed by the API (e.g., embodied by function or method calls), and transmits data and control information using parameters via the API call or activation . The API implementation component can respond to the API call from the API call component, and return a value through the API. Although an API defines the syntax and results of an API call (eg, how to enable the API call, and what the API call does), the API may not expose how the AIP call completes the function specified by the API call. Each API call is communicated through one or more application programming interfaces between the caller (the API calling component) and the API implementing component. Transmitting an API call can include issuing, initiating, enabling, invoking, receiving, returning, or responding to a function call or message; in other words, transmitting can describe the actions of an API calling component or an API implementing component. A function call or other invocation of an API may send or receive one or more parameters via a parameter list or other structure. Parameters can be constants, keys, data structures, objects, object classes, variables, data types, pointers, arrays, lists, or pointers to functions or methods or other references to data or another item to be passed through the API Way.

Additionally, data types or classes may be provided by the API and implemented by the API implementing component. Thus, an API calling component can declare variables, use pointers of this type or class, use or instantiate constants of this type or class, using definitions provided in the API.

In general, an API can be used to access services or data provided by an API-implementing component, or to initiate the execution of operations or computations provided by an API-implementing component. For example, both the API implementation component and the API call component can be any one of the operating system, program library, device driver, API, application program or other modules (it should be understood that the API implementation component and the API call component can be the same type of module, or a type different modules from each other). In some cases, an API implementing component may be embodied at least in part in firmware, microcode, or other hardware logic. In some embodiments, the API may allow client programs to use services provided by software development kit (SDK) libraries. In other embodiments, applications or other client programs may use APIs provided by the application framework. In these embodiments, the application or client program may contain calls to functions or methods provided by the SDK and provided by the API, or use data types or objects defined in the SDK and provided by the API. In these embodiments, the application framework may provide a program with a main event loop that responds to various events defined by the framework. APIs allow applications to specify events and responses to events using the application framework. In some implementations, the API calls can report the capabilities or status of the hardware device to the application, including capabilities or status related to various aspects, such as input capabilities and status, output capabilities and status, processing capabilities, power status, storage capacity and status, Communication capabilities, etc., and APIs may be implemented in part in firmware, microcode, or in part other low-level logic running on hardware components.

The API calling component can be a local component (i.e., on the same data processing system as the API implementation component), or a remote component that communicates with the API implementation component through the network via the API (i.e., on a different data processing system from the API implementation component). superior). It should be understood that the API implementation component can also act as an API call component (that is, it can perform API calls on the AIP exposed by different API implementation components), and by implementing the API exposed to different API call components, the API call component can also act as an API Implement components.

The API also allows multiple API-calling components written in different programming languages to communicate with the API-implementing component (thereby, the API may include features that translate calls and replies between the API-implementing component and the API-calling component); however, specific The programming language implements the API. In one embodiment, the API calling component can call APIs from different providers, such as one set of APIs from an OS provider, another set of APIs from a plug-in provider, and another set of APIs from another provider (e.g., a software library). provider) or another set of APIs from the creator of another set of APIs.

Figure 88 is a block diagram illustrating an example API architecture that may be used in some embodiments of the invention. As shown in FIG. 88, API architecture 8800 includes API implementation components 8810 (e.g., operating systems, libraries, device drivers, APIs, applications, software, or other modules) that implement APIs 8820. The API 8820 specifies one or more functions, methods, classes, objects, protocols, data structures, formats, and/or other characteristics of the API implementing component that can be used by the API calling component 8830. The API 8820 can specify at least one calling convention, and the calling convention specifies how the function in the API implementation component 8810 receives parameters from the API calling component 8830, and how the function returns the result to the API calling component. An API call component 8830 (e.g., an operating system, program library, device driver, API, application, software, or other module) issues an API call through the API 8820 to access and utilize features of the API implementation component 8810 specified by the API 8820. The API implementation component 8810 can respond to the API call, and return a value to the API call component 830 through the API 8820.

It is to be appreciated that the API implementing component 8810 may include additional functions, methods, classes, data structures and/or other features not specified by the API 8820 and available to the API calling component 8830. It should be appreciated that the API calling component 8830 can be on the same system as the API implementing component 8810, or can be located in a remote location and access the API implementing component 8810 via the API 8820 over a network. Although FIG. 88 illustrates a single API call component 8830 interacting with the API 8820, it should be understood that other API call components may use the API 8820, which may be in a different language (or in the same language) as the API call component 8830 write.

API implementing component 8810, API 8820, and API calling component 8830 can be stored on a machine-readable medium, including any mechanism that stores information in a form readable by a machine (eg, a computer or other data processing system). For example, machine-readable media include magnetic disks, optical disks, random access memory, read only memory, flash memory devices, and the like.

89 is an example of a dual camera mobile computing device architecture 8900. Implementations of mobile computing devices may include one or more processing units 8905 , memory interfaces 8910 , and peripherals interfaces 8915 . The components that make up the architecture of the computing device may all be separate components or integrated into one or more integrated circuits. These various components may also be coupled together by one or more communication buses or signal lines.

The peripherals interface 8915 can couple with various sensors and subsystems, including a camera subsystem 8920, a wireless communication subsystem 8925, an audio subsystem 8930, an I/O subsystem 8935, and the like. The peripherals interface 8915 enables communication between the processor and peripheral devices. Peripherals, such as orientation sensor 8945 or acceleration sensor 8950, can be coupled to peripherals interface 8915 to facilitate orientation and acceleration functions.

The camera subsystem 8920 can be coupled with one or more optical sensors 8940, such as charge coupled device (CCD) optical sensors, complementary metal oxide semiconductor (CMOS) optical sensors. A camera subsystem 8920 coupled to a sensor can facilitate camera functions, such as image and/or video data capture. A wireless communication subsystem 8925 may be used to facilitate communication functions. The wireless communication subsystem 8925 may include radio frequency receivers and transmitters, and optical receivers and transmitters. They can be implemented to work over one or more communication networks, such as GSM networks, Wi-Fi networks, Bluetooth networks, etc. Audio subsystem 8930 is coupled with a speaker and a microphone to facilitate voice-enabled functions such as voice recognition, digital recording, and the like.

The I/O subsystem 8935 is concerned with the transfer between input/output peripherals such as displays, touch screens, and the CPU's data bus through the peripheral interface. The I/O subsystem 8935 may include a touch screen controller 8955 and other input controllers 8960 to facilitate these functions. A touch screen controller 8955 can be coupled to the touch screen 8965 and detect contact and movement on the screen using any of a variety of touch sensitive technologies. Other input controls 8960 can be coupled with other input/output devices, such as one or more buttons.

The memory interface 8910 can be coupled with a memory 8970, which can include high-speed random access memory and/or non-volatile memory, such as flash memory. The memory is capable of holding an operating system (OS) 8972 . The OS 8972 may include instructions to handle basic system services and perform hardware-related tasks.

The memory may also include communication instructions 8974 to facilitate communication with one or more additional devices; GUI instructions 8976 to facilitate GUI processing; image/video processing instructions 8978 to facilitate processing and functions involving images/videos; Telephony Instructions 8980 to facilitate processing and functions related to telephony; Media Exchange and Processing Instructions 8982 to facilitate processing and functions related to media communications and processing; Camera Instructions 8984 to facilitate processing and functions related to cameras; and Video Conferencing processing and functions to facilitate Videoconferencing Instruction 8986. The above-described instructions need not be implemented as separate software programs or modules. Various functions of the mobile computing device may be implemented in hardware and/or software, including one or more signal processing and/or application specific integrated circuits.

The above-described embodiments may include a touch I/O device 9001 capable of receiving touch input to interact with a computing system 9003 through a wired or wireless communication channel 9002 as shown in FIG. 90 . Touch I/O device 9001 may be used to provide user input to computing system 9003 instead of or in combination with other input devices, such as a keyboard, mouse, or the like. One or more touch I/O devices 9001 can be used to provide user input to the computing system 9003. Touch I/O device 9001 may be an integral part of computing system 9003 (eg, a touch screen of a laptop computer), or be separate from computing system 9003 .

Touch I/O device 9001 may include a fully or partially transparent, translucent, opaque touch-sensitive panel, or any combination thereof. Touch I/O device 9001 may be embodied as a touch screen, a touch pad, a touch screen functioning as a touch pad (e.g., a touch screen instead of a touch pad of a laptop computer), a touch screen or touch pad combined or incorporated with any other input device (for example, a touchscreen or touchpad placed over a keyboard), or any multi-dimensional object with a touch-sensitive surface for receiving touch input.

In one example, the touch I/O device 9001 embodied as a touch screen may include a transparent and/or translucent touch-sensitive panel partially or fully disposed over at least a portion of the display. According to this embodiment, touch I/O device 9001 is used to display graphical data communicated from computing system 9003 (and/or another source), and is also used to receive user input. In other embodiments, touch I/O device 9001 may be embodied as an integrated touch screen in which a touch-sensitive component/device is combined with a display component/device. In yet other embodiments, the touch screen may be used as a secondary or additional display screen that displays secondary image data, or the same graphical data as the primary display and receives touch input.

Touch I/O Device 9001 may be configured to detect, based on capacitive, resistive, optical, acoustic, inductive, mechanical, or chemical measurements, or any phenomenon measurable with respect to the occurrence of one or more touches or proximity touches in proximity to Device 9001 The location of one or more touches or near-touches on device 9001. Software, hardware, firmware, or any combination thereof may be used to process measurements of detected touches to recognize and track one or more gestures. A gesture may correspond to one or more touches or proximity touches on the touch I/O device 9001, fixed or non-fixed. Move one or more buttons on the touch I/O device 9001 substantially simultaneously, uninterruptedly, or sequentially in a predetermined manner, such as tapping, pressing, swinging, wiggling, twisting, changing orientation, pressing with varying pressure, etc. Multiple fingers or other objects to make gestures. A gesture may be characterized by (but not limited to) a pinching, sliding, swiping, rotating, bending, dragging, or tapping motion between fingers, or any other finger or fingers. A single gesture can be accomplished with one or two hands, by one or more users, or any combination thereof.

The computing system 9003 may drive a display with graphical data to display a graphical user interface (GUI). The GUI can be configured to receive touch input by touching the I/O device 9001. The touch I/O device 9001 embodied as a touch screen can display a GUI. Alternatively, the GUI may be displayed on a display separate from the touch I/O device 9001. A GUI may include graphical elements that are displayed at specific locations within the interface. Graphical elements may include, but are not limited to, various displayed virtual input devices, including virtual scroll wheels, virtual keyboards, virtual knobs, virtual buttons, any virtual UI, and the like. A user may gesture at one or more specific locations of the touch I/O device 9001 that are associated with graphical elements of the GUI. In other embodiments, the user may gesture at one or more locations independent of the location of the graphical elements of the GUI. Gestures made on the touch I/O device 9001 can directly or indirectly manipulate, control, modify, move, actuate, activate, or generally affect graphical elements within the GUI, such as cursors, icons, media files, lists, text, all or part of the image and so on. For example, in the case of a touch screen, a user can directly interact with graphical elements on the touch screen by making gestures on the graphical elements. Touchpads, on the other hand, generally provide indirect interaction. Gestures also affect GUI elements that are not displayed (eg, cause a user interface to appear), or can affect other actions within computing system 9003 (eg, affect the state or mode of a GUI, application, or operating system). Gestures can be made or not made on the touch I/O device 9001 in conjunction with a displayed cursor. For example, in the case of gestures on a touchpad, a cursor (or indicator) can be displayed on a display or touchscreen, and the cursor can be controlled by touch input on the touchpad to interact with graphical objects on the display . In other embodiments in which gestures are made directly on the touch screen, the user can directly interact with objects on the touch screen with or without a cursor or indicator displayed on the touch screen.

Feedback may be provided to the user via communication channel 9002 in response to or in response to a touch or proximity touch on touch I/O device 9001 . Feedback may be conveyed optically, mechanically, electrically, olfactory, acoustically, etc., or any combination thereof, variably or invariably.

These functions described above may be implemented by digital electronic circuits, by computer software, firmware or hardware. Various techniques can be implemented using one or more computer program products. Programmable processors and computers can be included in or packaged as mobile devices. The processes and logic flows can be performed by one or more programmable processors and by one or more programmable logic circuits. General and special purpose computing devices and storage devices can be interconnected by communication networks.

Some embodiments include an electronic component, such as a microprocessor, storing computer program instructions in a machine-readable or computer-readable medium (otherwise referred to as a computer-readable storage medium, machine-readable medium, or machine-readable storage medium). , storage and memory devices. Examples of such computer-readable media include RAM, ROM, compact disc read-only (CD-ROM), compact disc-recordable (CD-R), compact disc rewritable (CD-RW), digital versatile disc (e.g., DVD -ROM, dual-layer DVD-ROM), various recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD cards, etc.), magnetic and/or solid-state hard drives, read-only and recordable Blu-ray Compact discs, ultra-high density discs, any other optical or magnetic media, and floppy disks. The computer-readable medium may store a computer program executable by at least one processing unit, including sets of instructions for implementing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files including high-level code executed by a computer, electronic component, or microprocessor using an interpreter.

Although the above discussion primarily refers to microprocessors or multi-core processors executing software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

As used in the specification and any claims of this application, the terms "computer", "server", "processor" and "memory" all refer to electronic or other technological devices. These terms exclude persons or groups of people. For purposes of this specification, the term "display" means displaying on an electronic device. As used in the specification and any claims of this application, the term "computer-readable medium" is strictly limited to tangible entities that store information in a form readable by a computer. These terms exclude any wireless signals, wired download signals and any other transient signals.

Figure 91 conceptually illustrates an example communication system 9100 for connecting some participants of a video conference, according to some embodiments. As shown, a communication system 9100 includes a number of mobile devices 9115, a number of cellular base stations (or Node Bs) 9110, a number of radio network controllers (RNCs) 9105, and a core network 9125. The cellular base stations and RNCs are collectively known as the Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access Network (UTRAN) 9130 . Each RNC 9105 is connected to one or more cellular base stations 9110 collectively referred to as a Radio Access Network (RAN).

Each cellular base station 9110 covers a service area 9120. As shown, the mobile devices 9115 in each service area are wirelessly connected to the serving cellular base station 9110 in the service area 9120 through the Uu interface. The Uu interface uses a protocol stack with two planes: a control plane and a user plane. The user plane supports circuit-switched, packet-switched, and broadcast data streams. The control plane carries the signaling messages of the network.

Each cellular base station is connected with RNC through Iub interface. Each RNC 9105 is connected to the core network 9125 through Iu-cs and Iu-ps interfaces. The Iu-cs interface is used for circuit switched services (eg voice) and the Iu-ps interface is used for packet switched services (eg data). The Iur interface is used to connect two RNCs together.

Accordingly, communication system 9100 supports circuit-switched services and packet-switched services. For example, a circuit switched service allows a call to be made by transferring call data (eg, voice) through the circuit switched equipment of the communication system 9100 . The packet switching service allows video conferencing by transferring video conferencing data via the packet switching devices of the communication system 9100 using a transport protocol layer such as UDP or TCP on top of an internet protocol layer such as IP. In some embodiments, the call-to-video conference transition (eg, handoff) described above in the video conference setup section utilizes circuit switched services and packet switched services supported by a communication system such as communication system 9100 . That is, in such an embodiment, the call is conducted over the circuit switched equipment of the communication system 9100 and the video conference is conducted over the packet switched equipment of the communication system 9100 .

Although the exemplary communication system in FIG. 91 illustrates a third generation (3G) technology UTRAN wireless mobile communication system, it should be noted that in some embodiments, a second generation (2G) communication system, other 3G communication systems, such as 3GPP2 Evolution Data Optimize or Evolve - Just Data (EV-DO) and 3rd Generation Partnership Project 2 (3GPP2) Code Division Multiple Access 1X (CDMA1X), Fourth Generation (4G) Communication Systems, Wireless Local Area Network (WLAN), and Microwave The Worldwide Interoperability for Access (WiMAX) communication system may be used to connect some of the participants in the conference. Examples of 2G systems include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS) and Enhanced Data Rates for GSM Evolution (EDGE). The 2G communication system architecture is similar to the architecture shown in Figure 91, except that the 2G communication system architecture uses a Base Transceiver (BTS) instead of Node B 9110, and a Base Station Controller (BSC) instead of RNC 9105. In a 2G communication system, the A interface between the BSC and the core network is used for circuit switched services, and the Gb interface between the BSC and the core network is used for packet switched services.

In some embodiments, the communication system 9100 is operated by a service carrier that originally provisioned the mobile device 9115 to allow the mobile device 9115 to utilize the communication system 9100 . Some embodiments provide the mobile device 9115 by provisioning and registering a Subscriber Identity Module (SIM) in the mobile device 9115. In other embodiments, the mobile device 9115 is configured and registered using the memory of the mobile device 9115 instead. Additionally, additional services may be provided (after the customer purchases the mobile device 9115), such as data services like GPRS, Multimedia Messaging Service (MMS) and instant messaging. Once provisioned, the mobile device 9115 is activated by the service operator, allowing use of the communication system 9100.

In some embodiments, communication system 9100 is a dedicated communication network. In such embodiments, the mobile devices 9115 are capable of communicating (eg, conducting voice calls, exchanging data) with each other (eg, the mobile devices 9115 provided for the communication system 9100 ). In other embodiments, the communication system 9100 is a public communication network. Thus, in addition to the mobile device 9115 provided for the communication system 9110 , the mobile device 9115 is also capable of communicating with other devices outside of the communication system 9100 . Some of the other devices external to the communication system 9100 include telephones, computers and other devices connected to the communication system 9100 through other networks, such as the public switched telephone network or another wireless communication network.

The Long Term Evolution (LTE) specification is used to define the 4G communication system. Figure 92 conceptually illustrates an example of a 4G communication system 9200 for connecting some participants of a video conference, in some embodiments. As shown, a communication system 9200 includes a number of mobile devices 9115, a number of evolved Node Bs (eNBs) 9205, a mobility management entity (MME) 9215, a serving gateway (S-GW) 9220, a packet data network (PDN) gateway 9225, and Home Subscriber Server (HSS) 9235. In some embodiments, the communication system 9200 includes one or more MMEs 9215, one or more S-GWs 9220, one or more PDN gateways 9225, and one or more HSSs 9235.

The eNB 9205 provides the air interface for the mobile device 9115. Each eNB 9205 covers a service area 9210 as shown. The mobile device 9115 in each service area 9210 is wirelessly connected to the eNB 9205 in the service area 9210 through the LTE-Uu interface. Figure 92 also shows that the eNBs 9205 are connected to each other through the X2 interface. In addition, the eNB 9205 is connected to the MME 9215 through the S1-MME interface, and connected to the S-GW 9220 through the S1-U interface. The eNBs 9205 are collectively known as the Evolved UTRAN (E-TRAN) 9230.

The eNB 9205 provides various functions such as radio resource management (e.g. radio bearer control, connection mobility control, etc.), routing of user plane data towards the S-GW 9220, signal measurement and measurement reporting, MME selection, etc. The functions of the MME9215 include mobile device tracking and paging in idle mode, radio bearer activation and deactivation, selection of the S-GW 9220 when the mobile device is connected, non-access stratum (NAS) signaling termination, and interaction with the HSS 9235 user authentication, etc.

The functions of the S-GW 9220 include (1) routing and forwarding user data packets, and (2) managing and saving mobile device context, such as parameters of IP bearer services and network internal routing information. Functions of the PDN Gateway 9225 include providing connectivity from the mobile device to external packet data networks (not shown) by being the point of egress and point of entry for the mobile device's traffic. A mobile station may have simultaneous connectivity to more than one PDN gateway in order to access multiple packet data networks. The PDN Gateway 9225 also acts as an anchor for mobility between 3GPP and non-3GPP technologies, such as WiMAXt 3GPP2 (eg, CDMA 1X and EV-DO).

As shown in the figure, MME 9215 is connected to S-GW 9220 through S11 interface, and connected to HSS 9235 through S6a interface. The S-GW 9220 and the PDN gateway 9220 are connected through the S8 interface. The MME 9215, S-GW 9220 and PDN Gateway 9225 are collectively referred to as the Evolved Packet Core (EPC). EPC is a major component of the System Architecture Evolution (SAE) architecture, which is the core network architecture of the 3GPP LTE wireless communication standard. EPC is a pure packet system. For example, an EPC does not have a voice media gateway. Services such as voice and SMS are packet-switched routed and provided by application functions utilizing EPC services. Using the previously described call-to-video conference transition as an example, in some embodiments, both the call and the video conference are conducted through the packet switching device of the communication system 9200 . In some of these embodiments, the packet-switched channel used for the call continues to be used for the audio data of the video conference after the call ends. However, in other such embodiments, a different packet-switched channel is created (eg, when a video conference is established) and audio data is transmitted over the newly created packet-switched channel rather than when the call ends using the packet-switched channel of the call.

Furthermore, the amount of bandwidth provided by these different technologies ranges from 44 kilobits/second (kbps) for GPRS to over 10 megabits/second (Mbps) for LTE. For LTE in the future, the expected download rate is 100Mbps and the upload rate is 50Mbps.

Although the invention has been described with respect to numerous specific details, those skilled in the art will recognize that the invention may be embodied in other specific forms without departing from the spirit of the invention. In addition, the numerous drawings conceptually illustrate various processes. The specific operations of these processes may not be performed in the exact order shown and described. No particular operation may be performed in a continuous series of operations, and particular operations may be performed differently in different embodiments. Furthermore, a process can be implemented with several sub-processes, or as part of a larger macro-process.

Additionally, many of the embodiments are described above with respect to video conferencing between dual camera mobile devices. However, one of ordinary skill in the art will recognize that many of these embodiments can be used between a dual camera mobile device and another device, such as a single camera mobile device, computer, phone with videoconferencing capabilities, etc. in the case of video conferencing. Additionally, many of the embodiments described above can be used in single camera mobile devices and other computing devices with videoconferencing capabilities. Accordingly, those of ordinary skill in the art understand that the present invention is not limited to the details of the above examples, but is only defined by the appended claims.

Claims (36)

1. a method for operation the first mobile device, described method comprises:

During the voice-frequency telephony by cordless communication network and the second equipment, on the first mobile device, present the selectable user interface UI project for be switched to the video conference between the first mobile device and the second equipment from voice-frequency telephony;

After receiving the selection of described optional UI project, show the first video of being taken by the first mobile device; With

Allowing before the first mobile device and the second equipment presents the Voice & Video data that exchange by video conference on the first mobile device by (1), the animation that stops voice-frequency telephony and the switching of (2) display of visually ground instruction from voice-frequency telephony to video conference is initiated video conference, wherein, described animation (i) starts from reducing the size of the first video in the first video and (ii) ending to show that the first video overlay is at least a portion of the second video of being taken by the second equipment showing.

2. in accordance with the method for claim 1, wherein, during presenting optional UI project and being included in voice-frequency telephony, in the viewing area on the display screen that can be presented at the first mobile device, continue to present described optional UI project.

3. also comprise in accordance with the method for claim 1:

In waiting for that the second video arrives the first mobile device, show the first video;

When in the time that the first mobile device has received the second video, show described animation, so that on present the second videometer the first mobile device.

4. in accordance with the method for claim 3, wherein, in reducing the first video big or small, the first video overlay is on the second video.

5. in accordance with the method for claim 3, wherein, once described animation finishes, the first video and the second video just form picture-in-picture and show, wherein, the first video serves as the insertion picture overlapping on the second video.

6. in accordance with the method for claim 1, wherein, allowing before the first mobile device and the second equipment exchange audio frequency and video data by video conference termination voice-frequency telephony.

7. in accordance with the method for claim 1, wherein, after the first mobile device and the second equipment start to exchange audio frequency and video data by video conference, but before the first mobile device and the second equipment start to present the Voice & Video data of exchange, stop voice-frequency telephony.

8. in accordance with the method for claim 1, also comprise the communication port with the second equipment from the first mobile device foundation, to set up the voice-frequency telephony between the first mobile device and the second equipment.

9. in accordance with the method for claim 1, wherein, the first mobile device be with cordless communication network in the mobile communication equipment of another devices communicating.

10. in accordance with the method for claim 1, wherein, the first mobile device is smart phone.

11. in accordance with the method for claim 1, and wherein, the first mobile device provides together with described cordless communication network with the second equipment, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

12. in accordance with the method for claim 1, wherein, the first mobile device provides together with described cordless communication network, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment, and the second equipment is not provided for carrying out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

13. in accordance with the method for claim 12, wherein, described cordless communication network is the first cordless communication network, wherein, the second equipment is provided for by the second cordless communication network that is different from the first cordless communication network, or carries out voice-frequency telephony by another communication network and miscellaneous equipment.

14. in accordance with the method for claim 1, and wherein, communication network is private radio communication network.

15. in accordance with the method for claim 1, and wherein, communication network is public wireless communication network.

16. in accordance with the method for claim 1, wherein

Cordless communication network comprises the circuit switching equipment for route voice-frequency telephony, and for the PSE of route data,

Voice-frequency telephony is by the circuit switching equipment route of communication network, and the Voice & Video data that exchange by video conference be by PSE with the form exchange of grouping,

Grouping, by Internet protocol IP communication link, transmits between the first mobile device and the second equipment.

The method of 17. 1 kinds of operation first mobile devices, comprising:

During the telephone relation between the first mobile device and the second equipment, on the first mobile device, show first user interface UI layout, a UI layout comprises the optional UI project for be switched to video conference from telephone relation;

After receiving the selection of optional UI project, present the 2nd UI layout, the 2nd UI layout comprises the first video of being taken by the first mobile device; And

After receiving the acceptance of video conference request from the second equipment:

Set up the video conference between the first mobile device and the second equipment; And

Present from showing that the 2nd UI layout is switched to the animation that shows the 3rd UI layout by reduce iteratively the size of described the first video in showing the first video, described the 3rd UI layout shows the first video overlay at least a portion of second video of being caught by the second equipment.

18. in accordance with the method for claim 17, and wherein, the first mobile device and the second equipment are to carry out at least one public wireless communication network by a wireless service operators mobile device that telephone relation is supplied with.

19. in accordance with the method for claim 17, and wherein, the 3rd UI layout comprises picture-in-picture PIP layout, and this picture-in-picture layout comprises the main viewing area that shows the second video, and the insertion viewing area of performance the first video, inserts viewing area and be less than main viewing area.

20. in accordance with the method for claim 17, wherein, present from showing that the 2nd UI layout is switched to and show that the animation of the 3rd UI layout comprises the size that reduces the first viewing area that shows the first video, to be emerging in the second viewing area of the first viewing area demonstration the second video afterwards.

21. in accordance with the method for claim 17, and wherein, a UI layout, the 2nd UI layout and the 3rd UI layout are in the identical viewing area of the UI of the first mobile device.

The equipment of 22. 1 kinds of operation first mobile devices, comprising:

For during the telephone relation between the first mobile device and the second equipment, on the first mobile device, show the device of first user interface UI layout, a UI layout comprises the optional UI project for be switched to video conference from telephone relation;

For present the device of the 2nd UI layout after receiving the selection of optional UI project, the 2nd UI layout comprises the first video of being taken by the first mobile device; And

For after receiving the acceptance of video conference request from the second equipment: set up the video conference between the first mobile device and the second equipment; And present from showing that the 2nd UI layout is switched to the device of the animation that shows the 3rd UI layout by reduce iteratively the size of described the first video in showing the first video, described the 3rd UI layout shows the first video overlay at least a portion of second video of being caught by the second equipment.

23. according to the equipment described in claim 22, and wherein, the first mobile device and the second equipment are to carry out at least one public wireless communication network by a wireless service operators mobile device that telephone relation is supplied with.

24. according to the equipment described in claim 22, and wherein, the 3rd UI layout comprises picture-in-picture PIP layout, and this picture-in-picture layout comprises the main viewing area that shows the second video, and the insertion viewing area of performance the first video, inserts viewing area and be less than main viewing area.

25. according to the equipment described in claim 22, wherein, present from showing that the 2nd UI layout is switched to and show that the animation of the 3rd UI layout comprises the size for reducing the first viewing area that shows the first video, to be emerging in the second viewing area of the first viewing area demonstration the second video afterwards.

The equipment of 26. 1 kinds of operation first mobile devices, described equipment comprises:

For during the voice-frequency telephony by cordless communication network and the second equipment, on the first mobile device, present the selectable user interface UI item destination device for be switched to the video conference between the first mobile device and the second equipment from voice-frequency telephony;

For after receiving the selection of described optional UI project, show the device of the first video of being taken by the first mobile device; With

For by (1) before permission the first mobile device and the second equipment present the Voice & Video data that exchange by video conference on the first mobile device, stop the device that voice-frequency telephony and (2) display of visually ground indicate the animation of the switching from voice-frequency telephony to video conference to initiate video conference, wherein, described animation (i) starts from reducing the size of the first video in the first video and (ii) ending to show that the first video overlay is at least a portion of the second video of being taken by the second equipment showing.

27. according to the equipment described in claim 26, wherein, comprises for during voice-frequency telephony for presenting optional UI item destination device, in the viewing area on the display screen that can be presented at the first mobile device, continues to present described optional UI item destination device.

28. according to the equipment described in claim 26, also comprises:

For in waiting for that the second video arrives the first mobile device, show the device of the first video;

For when in the time that the first mobile device has received the second video, show described animation, so that the device on present the second videometer the first mobile device.

29. according to the equipment described in claim 28, and wherein, in reducing the first video big or small, the first video overlay is on the second video.

30. according to the equipment described in claim 28, and wherein, once described animation finishes, the first video and the second video just form picture-in-picture and show, wherein, the first video serves as the insertion picture overlapping on the second video.

31. according to the equipment described in claim 26, wherein, is allowing before the first mobile device and the second equipment exchange audio frequency and video data by video conference termination voice-frequency telephony.

32. according to the equipment described in claim 26, wherein, after the first mobile device and the second equipment start to exchange audio frequency and video data by video conference, but before the first mobile device and the second equipment start to present the Voice & Video data of exchange, stop voice-frequency telephony.

33. according to the equipment described in claim 26, and wherein, the first mobile device provides together with described cordless communication network with the second equipment, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

34. according to the equipment described in claim 26, wherein, the first mobile device provides together with described cordless communication network, to carry out voice-frequency telephony by described cordless communication network and miscellaneous equipment, and the second equipment is not provided for carrying out voice-frequency telephony by described cordless communication network and miscellaneous equipment.

35. according to the equipment described in claim 34, wherein, described cordless communication network is the first cordless communication network, wherein, the second equipment is provided for by the second cordless communication network that is different from the first cordless communication network, or carries out voice-frequency telephony by another communication network and miscellaneous equipment.

36. according to the equipment described in claim 26, wherein

Cordless communication network comprises the circuit switching equipment for route voice-frequency telephony, and for the PSE of route data,

Voice-frequency telephony is by the circuit switching equipment route of communication network, and the Voice & Video data that exchange by video conference be by PSE with the form exchange of grouping,

Grouping, by Internet protocol IP communication link, transmits between the first mobile device and the second equipment.