TW200942305A - Apparatus for streaming databases serving real-time applications used through streaming interactive video - Google Patents

Abstract

An apparatus is described comprising a Power-over-Ethernet interface, a High-Definition Media Interface (HDMI), a Bluetooth interface, a means for decompressing low-latency streaming interactive video and audio from an Internet-based hosting service and outputting the decompressed audio and video through HDMI, and a means for transmitting to the Internet-based hosting service user control information and statistics from received packets. Further, the apparatus is able to receive two interleaved streams of low-latency streaming interactive video.

Description

200942305 VI. Description of the Invention: [Technical Field of the Invention] The present disclosure is generally in the field of a data processing system that improves the ability of a user to manipulate and access audio and video media. This application is the 10th, 315, 460 part of the continuation (CIP) application filed on December 10, 2002, entitled "Apparatus and Method for Wireless Video Gaming". The assignee of the CIP application has been granted. [Prior Art] Since the days of Thomas Edison, recorded audio and film media have become one aspect of society. In the early 20th century, there were widespread releases of recorded audio media (magnetic columns and records) and movie media (coin-operated jukeboxes and movies), but the two technologies are still in their infancy. In the late 1920s, films and audio were combined on the basis of bestsellers, followed by color films and audio. Radio broadcasts have evolved into broadcast-selling audio media that largely support the form of advertising. When the television (TV) broadcast standard was established in the mid-1940s, television and radio were joined in the form of broadcast-selling media to bring previously recorded or live-lived movies to the home. By the middle of the 20th century, most American households had a phonograph record player for playing recorded audio media, a radio for receiving live broadcasts of broadcast audio, and a broadcast audio for live broadcasts. / Video (A/V) media TV. These three "media players" (recorders, radios and TVs) group 139840.doc 200942305 are often combined in a common public speaker cabinet to become the "media center" of the family. Although media choices are limited for consumers, the media “ecosystem” is very stable. Most consumers know how to use the Media Player and enjoy the full range of capabilities. At the same time, publishers of the media (mostly film and television studios, and music companies) are able to distribute their media to both cinemas and families without extensive piracy or "secondary sales" (ie, used media) Resale). Often, publishers do not participate in the second sale of iron and steel, and as a result, the second sale reduces publishers' revenue for new purchases of buyers who would otherwise be able to use the media themselves. It is true that there are already used records sold during the mid-20th century, and the sale of the temple will not have a big impact on the record publishers, because it is different from movies or video programs (which are usually watched once or only several times by adults). Music tracks can be listened to hundreds or even thousands of times. Therefore, music media is much more advanced than movies/body (that is, for adult consumers, it has = 1. Once purchased, if the consumer likes the music, then / for the fee can keep it For a long time. From the middle of the 20th century to today, some body 嘘 system has experienced a series of fundamental changes for consumers and publishing === Loss. In the audio h, 疋 has (5) stereo cassette tape) Introduce a higher degree of convenience. But its _ pan-consumer media practice - the beginning of piracy. Indeed, many/payers use tape cartridges to record their own purely for the convenience of palladium D..... Increased consumers (for example, students in the dorm who are ready to access each other’s album collection) Will enter the "ready access to each other version I. In addition, consumers will record 139840.doc 200942305 by radio playback, η go #9 music instead of self-published materials to buy records or tapes. This will lead to the convenience of the singer 07, because now the VCR can be set to record Tvg$ for seeing it can be watched at a later time, and the establishment of the VCR (four) industry, where the film and the program design can be female, '" access. The rapid development of popular home media devices since the mid-1980s has led to unprecedented choice and convenience for consumers, and has led to a rapid expansion of the media publishing market. Today's consumers are faced with multimedia choices and multimedia devices, the best of which are tied to specific forms of media or to specific publishers. A keen media consumer may connect a stack of devices to a computer in a room to create a "snake" cable to one or more televisions and/or personal computers (PCs) and a group of remote controls. (In the context of this application, the term "personal computer" or "PC" refers to any type of computer suitable for use in a home or office), including desktop computers, Macintosh® or other non-Windows computers, and Windows compatible devices, Unix variant laptops, etc.). Such devices may include video game consoles, VCRs, DVD players, audio surround sound processors/amplifiers, satellite set a, cable TV set top boxes, and the like. In addition, for enthusiastic consumers, there are multiple devices with similar functions due to compatibility issues. For example, consumers may own both HD-DVD and Blu-ray (DVD) DVD players, or both Microsoft Xbox® and Sony Playstation® video game systems. In fact, due to the incompatibility of some games across several versions of the game console, consumers may have both XBox and later versions (such as the 'Xbox 3608'). Frequently, consumers are confused about which video 139840.doc 200942305 input and which remote end to use. Even after placing the disc in a regular player (eg DVD, HD-DVD, Blu-ray, Xbox or Playstation), selecting the video and audio inputs for the device and finding the correct remote control, the consumer still Faced with technical challenges. For example, in the case of a wide-screen DVD, the user may need to first determine the correct aspect ratio (eg, 4:3, full, zoom, wide magnification, cinema wide, etc.) and then on their TV or monitor screen. Set the correct aspect ratio. Similarly, the user may need to first determine the correct audio surround sound system format (e.g., AC-3, Dolby Digital, DTS, etc.) and then set the correct audio surround sound system format. Often, consumers are unaware that they may not have access to the media content of their television or audio system (for example, watching a movie that is squeezed in the wrong aspect ratio, or listening to stereo audio instead of surround sound) . Increasingly, Internet-based media devices have been added to the stack of devices. Audio devices like the Sonos® digital music system stream audio directly from the Internet. Similarly, devices like the 81丨1^13〇乂1^ entertainment player record video and stream it through the home network or via the Internet, where the video can be viewed remotely on the PC. . And Internet Protocol Television (IPTV) services provide cable TV-like services via digital subscriber line (DSL) or other home Internet connections. There have also been recent efforts to integrate multiple media functions into a single device, such as the Moxi® Media Center and the PC running the Windows XP Media Center version. While each of these devices provides a little convenience for the functions they perform, each lacks universal and simple access to most media. In addition, these devices are often costly manufactured in large quantities due to the need for processing and/or local storage of the 139840.doc 200942305. In addition, these modern consumer electronic devices typically consume a large amount of power even when idle, which means that it is expensive and wastes energy over time. For example, if the consumer forgets to cut a device or switch it to a different video input, the device may continue to operate. In addition, 'because none of these devices are a complete solution', it must be integrated with other devices in the family. This still leaves the mouse with a mouse line and many remote controls. In addition, when many of the newer Internet-based devices work properly, they typically provide a more general form of media (compared to the form that might otherwise be available). For example, devices that stream video over the Internet often only stream video material, and often do not accompany dvd interactive "extra billing" streams, such as video "production" and travel comments. This is due to the fact that interactive materials are often produced in the specific format of a particular device that is intended to handle interactivity at the local end. For example, each of DVD, HD-DVD, and Blu-ray Disc has its own specific interactions. (iv) Home media devices or local computers (which may be developed to support all popular formats) will require some degree of sophistication and flexibility that may be too expensive and complicated for consumer operations. ° This problem is exacerbated. If a new format is introduced in the future, the local device may not have the hardware capability to support the new format. This means that the consumer must buy an upgraded local media device. For example, if a higher resolution video or stereoscopic video is introduced in the later period after the manuscript (for example, each - 'more sleep 139840.doc 200942305 stream), the local device may not have decoded the video. Computing power, or it may not have hardware for outputting the video in a new format (eg, false-twisting to achieve stereo vision by 120 fps video synchronization with shuttered glasses), where 60 fps is delivered To each eye, if the consumer's video hardware can only support 6 〇 fps video, this option will not be available in the absence of a k-level hardware purchase. When it comes to cutting-edge interactive media (especially video games), the problem of media device obsolescence and complexity is a serious problem. The modern video game application is basically divided into four main non-portable hardware platforms: Sony PlayStation® 1, 2 and 3 (PS1, PS2, and PS3);

Microsoft Xbox® and Xb〇x are 8; and Nintend〇 8 and • WllTM; and pC-based games. Each of these platforms is different from the others, so that games written to be executed on one platform are typically not executed on another platform. There may also be compatibility issues between a generation of devices and next-generation devices. Even though most software game developers build software games designed to be independent of a specific platform, in order to execute a specific game on a specific platform, a proprietary software layer (which is often referred to as a "game development engine") is needed to adapt the game. Used on a specific platform. Each platform is sold to the consumer or itself as a PC in the form of a "console" (i.e., a separate box attached to the TV or monitor/speaker). Typically, video games are sold on optical media such as Blu-ray DVD, DVD_R〇M4CD_R〇M2, which contain video games embodied in sophisticated real-time software applications. With the increase in the speed of the wide band of the court, video games are becoming increasingly available for download. 139840.doc 200942305 Due to the immediacy and high computing requirements of high-end video games, the special requirements for achieving platform compatibility with video game software are extremely demanding. For example, we may expect full game compatibility from a generation of video games to next-generation video games (for example, from XBox to XBox 360, or from piaystati〇n 2 ("PS2") to Playstation 3 ("PS3")). Sex, as there is a general productivity of productivity (eg, Microsoft Word) from one PC to another PC with a faster processing unit or core. However, for the δίΐ game is not the case. Because when a generation of video games is released, video game manufacturers often seek the highest possible performance for a given price point, so dynamic system changes are often made to the system so that many games written for the previous generation system are later The first generation system does not work. For example, Lu's XBox is based on the χ86 series processor, while the χΒ〇χ360 series is based on the PowerPC series. Techniques can be used to simulate the previous architecture, but assuming video games are instant applications, it is often impractical to achieve the exact same behavior in the simulation. This is a loss to consumers, video game console manufacturers and video game software publishers. For consumers, it means keeping the old generation of video game consoles and the new generation of video game consoles. Both are connected to τν to be able to play all the games. For console manufacturers, the idea is related to the cost associated with the simulation and slower adoption of the new console. And ten; publishers and έ 'which means that multiple versions of new games may have to be released. To cover all potential consumers - not only for each version of the video game, for example, XBox, piaystati〇n), Also, versions for giving each version of the logo (for example, pS2 and pS3) are often issued. Example 139840.doc •10- 200942305 For the purpose of developing a separate version of the electronic art "Crazy Rugby 08" for XBox, XBox 36〇, pS2, pS3, Gamecube, wii and pc platforms and other platforms. • Portable devices such as cellular phones and portable media players also challenge game developers. Increasingly, these devices are connected to a wireless data network and are capable of downloading video games. However, there are a variety of cellular phones and media reference devices in the market that have a variety of display resolutions and computing capabilities. Also, because such devices typically have power consumption, cost, and weight constraints, they typically lack high-order graphics ("Gpu")-like acceleration hardware (such as that manufactured by NVIDIA of Santa Clara, CA). Device). Therefore, game software developers typically develop a given game title that is used for many different types of portable devices. The user can find that the given game title is not available for its particular cellular phone or portable media player. In the case of a home game console, hardware platform manufacturers typically ask software game developers for a version of the tax on their ability to publish games on their platform. Honeycomb telephony wireless communication companies often also ask game publishers for royalties to download games to cellular phones. In the case of PC games, there is no royalties paid for the release of games, but game developers often face high costs due to the high consumer service burden for supporting the PC configuration and possible installation problems. . Moreover, PCs often have fewer barriers to piracy of game software because they can be easily reprogrammed by technically savvy users and games can be more easily pirated and more easily assigned (eg, via the Internet) road). Therefore, for the software game developer 139840.doc 200942305, there are costs and disadvantages found in the game console, cellular phone and Pc. For game publishers of consoles and PC software, the cost is not limited to this. In order to distribute the game via the retail channel, the publisher asks the retailer for a wholesale price that is lower than the sale price to give the retailer a profit margin. Publishers must also pay for the cost of manufacturing and distributing physical media for the game. Retailers often also ask publishers for "price protection fees" to cover possible unforeseen expenses (such as games not being sold, or the price of the game is reduced, or the retailer must refund some or all of the wholesale price and / or recover from the purchaser game). In addition, retailers often ask publishers for help in selling games in advertising flyers. In addition, retailers are increasingly purchasing back games from users who have completed the game, and then the games are sold as used games, and the revenue of the used games is typically not shared with the game publisher. The following facts add to the cost burden imposed on game publishers: games are often pirated and distributed via the Internet for users to download and make free copies. As the Internet broadband speed increases and broadband connectivity becomes more widespread in the United States and around the world (specifically, to the home and to the "Internet cafes" that lease PCs connected to the Internet), games are increasingly being Assigned to a PC or console via download. Moreover, broadband connections are increasingly being used to play multiplayer and massively multiplayer online games (both referred to in the present disclosure by the acronym "MMOG")^ These changes mitigate the distribution associated with retail distribution Some of the costs and problems. Downloading online games solves the disadvantages of game publishers because distribution costs are usually small and there are fewer or no 139840.doc -12- 200942305 There are unsold media, costs. But the downloaded games are still subject to piracy, and, size (the size is often a lot of ten-digit bytes) and you ~ Gan spends a very long day and a tenth (heart position group) makes it possible - such as Together with the portable text 1 additional 'multiple games can fill up the small disk * 々, 1 type computer - or together with the video game console / with the disc player. S, as far as the game or MMOG needs to be connected online, the piracy problem can be alleviated in terms of the level of playability, because the user needs to have a valid user account. It is different from the video that can be used to capture the screen by camera Φ or the linear media (such as video and music) from the sound of crying through the microphone. Each view (4): system-only, and can not make (4) single video / Audio recording to copy. Therefore, even in areas where copyright law is not enforced and piracy is promoted, it is also possible to protect mm〇g from piracy and thus support business. For example, the "World of Warcraft" of VivencU SA has been successfully deployed, and the world has not been pirated. And many online or MM〇G games (such as,

Lab's "Second Life" MM〇G) generates the revenue of the game operator through the economic model built into the game. The assets can be sold and even established using online tools. Therefore, mechanisms other than the purchase or subscription of the game software can be used to pay for the use of online games. Although piracy is often mitigated due to the nature of the online or MMOG, 1 online game operators still face the remaining challenges. Many games require a lot of

The local end (i.e., within the home) processes resources for proper work on the line *mm〇g. If the user has a low-performance local computer (for example, a computer without a GPU, such as a low-end laptop), it may not be able to play the game. In addition, as the game console ages, it lags far behind the current state of the art and may not be able to handle higher-order games. Even if it is assumed that the user's local PC can handle the calculation requirements of the game, there is often installation complexity. There may be driver incompatibility (for example, if a new game is downloaded, it is possible to install a new graphics driver, which causes the game that is dependent on the old version of the graphics driver to be installed. The console may run out of local disk space. When bugs are found and fixed, or if the game is modified (for example, if the game developer finds the game level too difficult to play or too easy to play), complex games usually The downloaded patches are received from the game developer over time. These patches require new downloads, but sometimes not all users complete the download of all patches. At other times the 'downloaded patches are introduced Other compatibility or disk space consumption issues. x Also during the game play, large data downloads may be required to provide graphical or behavioral information to the local PC or console. For example, who?

If you enter a scene or a person in the MMOG that encounters an action that is not available on the local machine or consists of the user, you must download the material of the scene or person. If the internet connection is not fast enough, this can result in substantial delays during game play. In addition, if the scene or character encountered needs more than the storage space or computing capacity of the local PC or console, the storage space or computing power can generate a situation: the user cannot continue in the game, or must Continue with the graphic of reduced quality. Therefore, green or MMOG games often limit their storage and/or computational complexity requirements. In addition, it often limits the amount of data transfer during the game. Online or MMOG games can also narrow the market for users of playable games. S 139840.doc •14- 200942305 In addition, 'technically erudite users are increasingly reversing the native copy of the game and modifying the game to make it cheating. Cheating can be as simple as repeating button presses that are faster than possible with human power (for example, to shoot very quickly). In a game that supports in-game asset trading, 'cheating can reach the level of fraud that leads to fraudulent transactions involving assets of real economic value. When the online or MMOC® model is based on such asset transactions, this can result in substantial harmful consequences for the game operator.

The cost of developing new games grows as pCs and consoles produce increasingly sophisticated games (eg, with more realistic graphics (such as instant ray tracing), and more realistic behaviors (such as real-time physics simulations). In the early days of the video game industry, video game development was very similar to the development of application software; that is, most of the development costs were in the development of software (as opposed to the development of graphics, audio and behavioral elements or "assets"). ), such as the software development that can be developed for use in movies with a wide range of special effects. Today, many cutting-edge video game development efforts are more like software developments with special effects than software development. For example, many video games offer simulations of the 3-D world' and produce characters, props, and environments that are increasingly real (i.e., computer graphics that appear to be realistic with live action images taken by photography). One of the most challenging aspects of photo-realistic game development is to create a human face that can't be distinguished from a living person's face. Facial capture techniques (such as the capture of the precise geometry of the performer's face by CAiSan Francisc) and tracking the performer's face with high resolution while the performer is in motion Geometry. This technology allows 139840.doc 200942305 3D face to be reproduced on a PC or game console. This 3D face can't actually be distinguished from the captured live action face. Accurately capture and reproduce the "photo-realistic" face. Useful in several ways. First, highly recognizable celebrities or athletes are often used in video games (often hired at high cost), and imperfections may be obvious to the user, making the viewing experience distracting or unpleasant. Frequently, high detail is required to achieve a high degree of photorealistic sensation - potentially needed to reproduce a large number of polygons and high resolution textures (changes in polygons and / or textures as the face moves over the frame of the frame) B) Supports the PC or game console of a game when the high-polygon count scene with detailed texture changes rapidly Can not have enough ram to store enough polygons and texture data for the required number of moving frames generated in the game segment. In addition, usually a single CD player or a single disk on a poetry PC or game console Machines are usually much slower than RAM, and usually cannot keep up with the maximum data rate that the GPU can accept in rendering polygons and textures. When 2 ships often load most polygons and textures into RAM, this means, "疋" The hall, complexity and duration are largely limited by the capacity of the ram. In the case of, for example, facial animation, this may limit the PC or game control σ to a low resolution without realism. Degrees, or limited to realistic faces that can be animated in a limited number of frames before the game is paused and planted into polygons and textures (and other tributes) for more frames. When the station displays a message similar to "Loading...", the viewing bar 5 moves more slowly on the screen. The AI of today's complex video games is inherently flawed. The next scene is from the disk (unless there is another bar 13) 9840.doc -16- 200942305 poetry, disk "refers to non-volatile optical media or magnetic" such as non-magnetic media such as semiconductor "flash" memory). The delay of loading % can take several seconds or even A few minutes. This I can make gamers quite frustrated. As discussed earlier, a large or all delay may be due to the loading of polygons, textures, or other data from the magnetic disk. : When the processor and / or gpu in the PC or console are ready to use the data for the scene, it takes a part of the loading time.

❹ Example, Lu, English; i-ball video game allows players to choose among a large number of players, groups, sports fields and weather conditions. Therefore, depending on the choice of which combination, it may be necessary to use different polygons, textures, and other materials (collectively referred to as objects) for the scene (for example, different groups have different colors on their uniforms: and patterns). It is possible to enumerate many or all of the various arrangements: pre-calculate many or all of the objects and store the objects on the game material. However, if the number of permutations is large, then all objects: and the inaccuracies may be too large to be mounted on the disk (or too impossible to download). As a result, existing (10) and console systems are often constrained in both the complexity and playback duration of the scene, and the scene suffers from long load times. Another limitation of the prior art video tour (4) system and the application software system is that it is increasingly using (for example) a large database of 3D objects (such as 'polygons and textures'), which need to be manned. Go to the game console for processing. As described above, when the database is stored on the disk at the local end, the library can take a long time to load. d and * the database is stored in a remote location and stored via the Internet. 139840.doc •17- 200942305 Take, the load time is usually much more serious. In such cases, downloading a large repository can take minutes, hours, or even days. In addition, these J-stock libraries often generate a large amount of expenses (such as '3 £> model for detailed high-altitude sailing boats used in games, movies, or historical documentaries) and are intended for sale to local end users. . However, once the database is downloaded to the local end, it is at risk of being pirated. In many cases, the user only uses the estimated database to see if it is suitable for the user (eg, whether the 3D garment used for the game character has a satisfactory appearance or appearance when the user performs a particular movement). And hope to download the database. For users who estimate the 3Df library before deciding to make a purchase, long manned time can be a hindrance. Similar issues arise in MMOG (specifically, games that allow users to leverage the benefits of customizing characters). For pcs or game consoles that display characters, they need to be able to access data with 3D geometry (polygons, textures, etc., and the behavior of the characters (for example, if the character has a shield, then the shield is strong enough to deflect the spear) In general, when a user is playing for the first time, a large database of characters is available under the initial copy of the game. The initial copy is available on the local disc or downloaded to the disk. But as the game progresses, if the user encounters a character or object that the database is not available on the local end (for example, if another user has created a custom character), then the user or object must be downloaded before the character or object can be displayed. Its database. This can lead to a substantial delay in the game. Given the sophistication and complexity of video games, another challenge for video game developers and publishers in the case of prior art video game consoles is at 139840.doc - 18· 200942305 Yu. Development video tour iron often yuan. 偬定如, 日-卞王』平, cost in tens of millions of US speed (four) into the game console platform is roughly In the five-years, the game developers need to start the new game in the new game _ A # >> 1 & The game is open for several years... The game before the game is available at the same time. When you make it look at the same time, sometimes the console is ^^Α in years or two years, but it is still a bit of a blindness (for example, which console will produce the maximum)

7 game software sales). For example, in the recent console loop. Please ox 360, sony playstation uNintendo is expected to be introduced in about the same -fli. 4ϊϊ 3! J^ time box. But in the years before these introductions, 'game developers must essentially "wear" which console platforms will be more successful than others, and invest in their development resources accordingly. Film production companies must also allocate their limited production resources for a long time before the film is released based on their estimated success. Given the degree of investment required for video games, game production is becoming more and more similar to film production and game production companies routinely invest their production resources based on their estimates of the future success of a particular video game. However, unlike film companies, this bet is not based solely on the success of the production itself; the truth is that it is based on the success of the game console on which the game is intended to execute. Simultaneously launching games on multiple consoles mitigates the risk, but this extra effort increases costs and often delays the actual release of the game. The application software and user environment on the PC are becoming more computationally intensive, dynamic and interactive' not only making it more visually appealing to the user, but also making it more useful and intuitive. For example, the new Windows VistaTM operating system 139840.doc -19· 200942305 and both successive versions of the Macintosh® operating system have visual animation effects. Good-looking graphical tools, such as MayaTM from Aut〇desk, offer very cutting-edge 3D rendering and dynamic production capabilities that push the limits of current state of the art CPUs and GPUs. However, the calculation requirements for these new tools create many practical problems for users and software developers of such products. · Because the visual display of the operating system (OS) must work on a variety of computers (including those that are no longer sold but can still be upgraded with the new OS), the graphics requirements are largely intended by OS. The minimum commonality limit for computers, which typically include computers that do not include GPUs. This severely limits the graphics capabilities of 〇8. In addition, battery-powered portable computers (eg, laptops) limit visual display capabilities because high computing activity in CPUs or GPUs typically results in more horsepower consumption and shorter battery life. Portable computers are usually included. Software that automatically reduces processor activity when using the processor to reduce power consumption. In some computer models, the user can manually reduce the activity of the device. For example, the VGN-SZ280P laptop 3 of s〇ny has the side marked "Stamina" on the side (for low performance, longer battery life) and the other side marked "Speed". (for high performance, short battery life) exchangers. The 〇s performed on a portable computer must also be able to function, even if it is executed at a fraction of its peak performance capabilities. As a result, QS graphics performance often remains at a much lower computing power than current state of the art. Frequently selling high-end computing-intensive applications (such as Maya), it is expected that applications such as ° Xu will be used on high-performance PCs. This usually yields a much higher efficiency 139840.doc -20- 200942305, and is more expensive and less portable, with the least commonality. As a result, these applications have a much lower target audience than a general-purpose OS (or generic productivity application like Microsoft 〇ffice) and are typically sold in much lower quantities than general-purpose OS software or general-purpose application software. Potential audiences are further limited because it is often difficult for prospective users to try out these highly intensive applications in advance. For example, suppose students want to know how to use Maya or know that potential buyers of such apps are buying

I hope that before investing in _Maya (this may involve buying a high-end computer that can execute Maya). When a student or potential purchaser can download a demo version of the handsome version or receive a physical media copy of the Maya demo version, it will not be able to perform a computer that can perform Maya to its full potential (for example, to handle complex deduction scenarios). Comprehensive evaluation of the product. This essentially limits the audience for these high-end applications. It also makes the sale price higher because development costs are usually paid off through purchases that are much smaller than the number of purchases by the generic application. The Reward Application & Style also generates more stimuli for individuals and businesses that use pirated copies of the application software. As a result, high-end application software has been hunted, although publishers of the software have made a lot of efforts to pirate through various techniques. However, even the high-end application users of the ❹ version are unlikely to rule out the need to invest in expensive PCs with current state of the art. Therefore, although the software application is used at a rate of one of the available sales prices, the user still needs to buy or obtain an expensive PC to utilize the application. ]39840.(j〇c -21 200942305 This is also true for users of high-performance pirated video games. Although the second edition can get the game at a rate of the actual price of the game, it still needs to build a proper game. Expensive computing hardware required (for example, Gpu_Enhanced PC, or a high-end video game console like XBox 360). Assuming that video games are usually entertainment for consumers, the extra cost for high-end video game systems can be too high. Expensive. This situation is worse in countries where the current average annual income of workers is relatively low (relative to the average annual income of current workers in the US) (for example, China). Therefore, a much smaller percentage of people have high-end video. Game systems or high-end PCs. In these countries, "Internet cafes" where users can pay ◎ fees to use computers connected to the Internet are quite common. Often, these Internet cafes have features that are not highly efficient (such as An older model or a low-end PC that would otherwise allow players to play Gpu, which computes intensive video games. This is the key to the success of games executed on low-end PCs. (eg, Vivendi's "World of Warcraft", which is highly successful in China, and is through. * is played in Chinese Internet cafes.) In contrast, computationally intensive games (such as First Life) are more It is impossible to play on pC installed in Chinese Internet cafes. These games can't actually reach users who can only access low-performance PCs in Internet cafes. For considering buying video games and first willing to use via the Internet There are also obstacles to users who download the roadshow/beetlet to their trial version and try out the demo version of the game. Video game demos are often the all-around version of the game, some of which are used specifically or impose restrictions on the amount of game play. It involves a long process (possibly several hours) of downloading billions of bytes of data before installing the game on a PC or console and executing it on a PC or console. On the PC 139840.doc -22- In 200942305, it may also involve calculating which special drivers (such as DirectX or 0penGL drivers) the game needs, downloading the correct version, installing the correct version, and then determining if the PC can play the This latter step can be used to determine if the PC has sufficient processing (CPU and GPU) power, sufficient RAM, and compatible OS (for example, some games are executed on Windows XP without being executed on Vista). Therefore, after trying to perform the long process of video game presentation, the user may find that the video game demo is not playable (given the user's PC configuration). Worse, once the user has downloaded the new drive to try it out In the demonstration, these drive versions may be incompatible with other games or applications that the user is accustomed to using on the PC, so the installation of the demo may render the previously operable game or application inoperable. These obstacles not only frustrated users, but they also created obstacles for video game software publishers and video game developers to sell their games. Another problem that leads to non-economic benefits is related to the fact that a given pc or game console is typically designed to meet the performance requirements of the application and/or game. For example, some PCs have more or less RAM, a slower or faster CPU and are slower or faster (if they have a GPU). Some games or applications take advantage of the full computing power of a given pc or console, while some games or applications do not take advantage of the full computing power of a given pc or console. If the user's game or application selection does not reach the peak performance capability of the local PC or console, the user may be due to the unused features. (: or the console is wasting money. In the case of the console, (4) the manufacturer may pay more than the cost of subsidizing (four). 139840.doc -23- 200942305 exists in the sales and enjoyment of video games In addition, the problem involves allowing the user to watch other people play the game before the user implements the purchase of the game. There are several prior art methods for recording the video game for playback at a later time. For example, US Patent No. 5,558,339 teaches Game state information (including game controller actions) is recorded during the game play on the video game client computer (owned by the same or different). This status information can be used at a later time in the video game client. Replay some or all of the game actions on a computer (eg, a PC console). Significant shortcomings of this method

In order to view the user of the recorded game, the user must have a video game client computer capable of playing the game and must have a video game application executed on the computer so that when the recorded game is played back Recording time and frequency playback materials are the same. In addition, the video game application must be written in such a way that there is no measurable execution difference between the recorded game and the played back game. For example, the game graphics system is calculated based on the frame of the frame.

=Multi-games' depending on whether the scene is particularly complex or other delays in the subscription (eg 'on the PC, another - the process may be stubborn, so that the game application m takes away the CPU ·), the game logic sometimes This is a graph that is shorter than a frame time or displayed in a frame. In the case where such a cursor is calculated to be slightly less (for example, a value of a few c 乂t frames), the frame may eventually appear. Λ2: Pulse cycle) The calculation of the threshold is the same as the same as the field μ. The game state information and then the poor day τ 'may be easy to spend _ (10) the time of the cycle (for example: the frame time is a few more right 円. 卩 CPU bus is slightly different from the external 139840.doc -24 - 200942305 DRAM bus Phase, and a few milliseconds of CPU time - over ^ not / child in the game from the game

Delay in cycle time). Therefore, Λ delay, which also introduces several CPUs, when playing back the game, the frame becomes calculated in two frame times instead of single-frame time. In this case, Sun Mei calculates the frequency of the new frame in the game ^ & For example, if the game is sampled from the input of the game control benefits). When the play ^ ^ stamp is played, the deviation in the time reference for different behaviors does not affect the age of the game, but it can lead to the playback of the game.

The play produces different results. For example, the orbit of the right ball is calculated at a steady 60 fps rate, but the game slam dunk input is sampled based on the calculated frame rate 'When the record age ^ avoids the drama' The rate of the juice frame may be 53 fps, and the rate of the replayed game may be 52 fps. This may cause the basketball to be prevented from entering the basket.

This leads to different results. Therefore, Yoshida, in consultation with the machine A, uses a game state recording video game that requires a very cautious game software design, τ to ensure that the same game state is used to reproduce the same result. Another prior art method for recording video games is to record only the video output of a PC or video game system (e.g., to a VCR, dvd recorder, or to a video capture board on a PC). The video can then be replayed and replayed, or alternatively, the recorded video can be uploaded to the Internet (usually after video compression). The disadvantage of this method is that when the playback is performed 31) the game sequence N. The user is limited to only the self-test viewpoint (the sequence is recorded from it) to view the sequence. In other words, the user cannot change the view point of the scene. In addition, when the compressed video sequence of the recorded game sequence played on the home PC or the game console via the Internet is available to other users, 139840.doc -25- 200942305, even if the video system is compressed immediately It is not possible to upload compressed video to the Internet in real time. The reason is because many homes connected to the Internet in the world have highly asymmetric broadband connections (for example, fox and power (4) machines typically have a much lower downlink bandwidth than the upload bandwidth). Compressed high resolution video sequences often have a higher bandwidth than the network's upload bandwidth, making it impossible to upload on the fly. Therefore, after playing the game sequence (possibly for a few minutes or even hours), there will be a significant delay before another user on the Internet can view the picture. Although this delay can be tolerated in the form of a trait (eg, viewing the outcome of a game player that appeared at a previous time), it eliminates the ability to watch a live game of the game (eg, a μ ball tournament played by a winning player) or The "immediate playback" capability when playing a game live. Another - prior art method allows a viewer with a TV pure device to watch a video game live broadcast, but only under the control of the television producer. Some TV channels in the United States and other countries provide video game detection video channels. Among them, TV viewers (4) watch certain video game users on the video channel (such as 'the top players participating in the tournament. This is because the video game system (PC and / or the console) video transmission line is sent to the video distribution and processing equipment for the TV channel to complete. This is the case when the TV channel broadcasts a live basketball game. 'There is a difference between the camera and the blue court. The degree provides a live broadcast feed. The TV channel can then use its video/audio processing and effects devices to manipulate the rounds from various video game systems. For example, 'TV channels can be overlaid on video from video games. The text of the status of different players (as it can be broadcast live in the blue 139840.d〇 ( -26- 200942305 during the ball game)) and the TV channel can be added from the commentator (its ° profile during the game) The sound of the action). In addition, the camera that outputs the video game to the actual player of the game (for example, One of the problems with this method is that the live video feed must be instantly available for the video distribution and processing device of the TV channel so that it has the stimulation of live broadcasts. However, as previously discussed, when the video game system is implemented from the home (especially when a broadcast packet is included in the live broadcast of the camera from the real world video camera that is capturing the game player), this is often In addition, in the case of tournaments, the focus is on the game in the home where the player can modify the game and cheat, as previously described. For these reasons, the video game broadcasts on the TV channel are often configured to be gathered in a common location. A player and video game system (eg, at a television studio or in an arena), wherein the television production device can accept video feeds from multiple video game systems and potentially live broadcast cameras. Technical video game TV channels can provide very exciting performances for TV viewers (its Live broadcast sports events of the same kind (for example, the same kind of video game players as "athletes"), not only based on their actions in the video game world, but also according to their actions in the real world), but such video Game systems are often limited to situations where players are physically close to each other. In addition, because the TV channels are broadcast, each broadcast channel can only display a video stream selected by the producer of the TV channel. Due to these restrictions and broadcast time , production equipment and production staff 139840.doc -27· 200942305 and cost, players. shai and other e-channels usually only show the top level of the top tournaments, Bu, Xiang: TV viewers broadcast video game full screen The image of the TV channel is only displayed for the video channel - video game. This is a serious choice of the viewer. For example, today, electricity, 』 叩. TV viewers may not be interested in the game shown at a given time. Another viewer may only be interested in watching a particular player's game play that is not shown by the video-track at a given time. In other situations, the viewer may only be interested in seeing how the insider handles the level of amnesty in the game. Other viewers may wish to control the view point (the video game is from its point of view that the view point is different from the view point selected by the production team, etc. In short' TV viewers may have countless preferences in watching video games (even # ; _ π m . , 丨 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右 右There are significant limitations in presenting video games. Another disadvantage of prior art video game systems and application software systems is that they are complex and often suffer from errors, crashes, and/or undesired and *❹1 ( Collectively referred to as "defects") ^Although games and applications are released: 刖通书 undergoes debugging and tuning processes. Often referred to as "software quality assurance" or SQA) 'But almost unchanged is · once the game or application Being, to the broad audience in the field, the defects will suddenly appear. Unfortunately, it is difficult for software developers to identify and trace many defects after the release. Software developers may have difficulty recognizing defects. Even when it understands - defects, there may be only a limited number of things that can be used to identify what caused the defect. 139840.doc •28- 200942305

Information on quantity. For example, [the reader can call the game developer's consumer service hotline and leave a message stating that when playing the game, the start of the game starts f(4), then I changes to solid blue and Pc; East knot. It provides the sqa team with very little information that is useful in tracking defects. One of the online connections Some games or applications sometimes provide more information under certain conditions. For example, the "Watchdog" process can sometimes be used to monitor whether a game or application "crashes". The watchdog process collects statistics about the state of the game or application process (eg, the state of use of the memory stack, the extent to which the game or application progresses, etc.) when the game or application crashes, and then proceeds via the Internet. Upload his information to the SQA team. However, in a complex game or application, the information can take a very long time to decrypt, in order to determine what the user is doing during the crash. Despite this, it is impossible to determine what sequence of events caused the crash. The problem associated with PCs and game consoles is that they suffer from service problems that are extremely inconvenient for consumers. Service Issues (4) The manufacturer of the game console, because it usually needs to send a special box to safely ship a damaged PC or console' and thus incur the cost of repair ((4) or the console is under warranty). Game or application software publishers may also be affected by sales losses (or online service usage) caused by PCs and/or consoles in the repair state. Figure 1 illustrates a prior art video game system such as S〇ny Playstati〇n8 3, Micr〇s〇ft xb〇x 3_, Nintendo WiiTM, wind_based PC or Apple Macintosh. Each of these systems (4) is used to execute the code processing unit (mj) (usually 139840.doc -29- 200942305 for the graphics processing unit (GPU) for performing high-level graphics operations), and for Multiple forms of input/output (1/〇) in communication with external devices and users. For the sake of simplicity, these components are shown as being combined into a single unit 100. The prior art video game system of FIG. 1 is also shown to include an optical media drive 104 (eg, a DVD_ROM drive); a hard drive 103 for storing video games, code and data; for playing multiplayer games, for a network connection 105 containing games, patches, demos, or other media; a random access memory (RAM) 101 for storing code currently being executed by the CPU/GPU 100; used for receiving during use during game play The game controller 106 that inputs the command; and the display device 1〇2 (for example, sdtv/HDTV or computer monitor).

The prior art system shown in Figure 1 suffers from several limitations. First, the optical disk drive 1 & 4 and the hard disk drive tend to have a much slower access speed than the RAM 101 access speed. When working directly via RAM 1〇1, since RAM 101 typically has a much higher bandwidth and does not suffer from the relatively long search delay of the disc mechanism, Cpu/Gpu 1〇〇 can be processed in practice than direct When the program code and data are read from the hard disk drive 103 or the optical disk drive 1G4, the number of polygons per second can be much larger. However, only a limited amount of Ram is provided in such prior art systems (e.g., 256-512 megabytes). Therefore, a sequence of "Loading..." is often required, in which rami〇i is periodically filled with τ for a video game - H-systems attempt to simultaneously overlap the manned and game play of the code, but This can only be done in the presence of a known sequence of events (for example, if driving along a road, the geometry of the building that is approaching the roadside 139840.doc -30. 200942305 can be loaded while driving the car. ). This type of stack usually does not work for complex and/or fast scene changes. For example, if the user is in the middle of the campaign and the rAM 101 is completely filled with the information indicating the object in the view of the time, if the user quickly moves the view to the left to view when the ^ is not loaded The object in the RAM 101 will cause a discontinuity in motion because there is not enough time to load the new object from the hard disk drive 1 or the optical disk drive into the RAM 101.

Another problem with the system of Figure 1 is due to the limitations of the hard disk drive 1 and the storage capacity of the optical media. Although the disk storage device can be manufactured to have a relatively large storage capacity (for example, 500 million bytes or more than 500 million bytes), it is not provided for use in current video games. Sufficient storage capacity for a particular situation. For example, as previously described, a British 2-ball video game may allow a user to select among many groups, players 1 in the world. For each-group, per-player, and per-sports, a large amount of texture mapping and environment mapping is required to characterize the 3D surface of the world (for example, each __ group has a -only - jersey, each _ need - unique texture mapping ). One of the techniques used to solve this latter problem: For games, once the user has selected texture and environment mapping, the texture and environment are pre-computed: shot. This can involve a number of computationally intensive processes, including decompressing images, observing, adding shadows, organizing data structures, and the like. Therefore, when the video game performs such calculations, there may be a delay for the user. The method of reducing this delay is in principle: all such calculations are performed when the game is initially developed _ including the group, the player's roster, and each of the sports fields. The release version of the game thus 139840.doc • 31 · 200942305 will include all such pre-processed material stored on optical media 104 or on one or more servers on the Internet, when the user makes a selection, only via The Internet downloads selected pre-processed data for a given group, player roster, and sports field selection to the hard drive 1〇3. However, as a practical matter, the pre-loaded data for each of the possible permutations in the game play* is easily a terabyte of data far exceeding the capacity of today's optical media devices. In addition, the information used for a given group, = family roster, and sports field selection may easily be hundreds of millions of bytes of information or hundreds of millions of bytes of information. In the case of a home network connection (for example, 10 Mbps), downloading this data via the network connection 1〇5 takes longer than calculating the data on the local side. Thus, the prior art gaming architecture shown in Figure 1 allows the user to experience significant delays between large scene transitions of complex games. Another problem with prior art methods such as the prior art method shown in Figure 1 is that video games tend to become higher order and require more CPU/GPU processing power over the years. Therefore, even with an unlimited amount of RAM, video game hardware requirements exceed the peak levels of processing power available in such systems. Therefore, users are required to upgrade their game hardware every few years to stay in sync (or play newer games at a lower quality level). One consequence of the trend of higher-order video games over the past is that machines that play video games for home use are often not economical because their cost is usually determined by the requirements of the highest-performance games they can support. For example, you might use XBox 360 to play a game like "Gears of War," which requires a high-performance CPU, GPU, and hundreds of millions of bytes of ram, or 139840.doc -32· 200942305 You might use XBox 360 to play Pac Man, a game from the 1970s that requires only a few thousand bytes of RAM and a very low-performance CPU. In fact, the XBox 360 has enough computing power to host many of the same Pacman games at the same time. • The video game console is usually cut off during most of the week. According to a July 2006 Nielsen Entertainment study of active gamers aged 13 and over, on average, active players spend 14 hours a week or only 12% of all hours in a φ week to play the console. Video game. This means that the average video game console is idle 88% of the time, which is an inefficient use of expensive resources. It is particularly meaningful to assume that the video game console is often funded by the manufacturer to lower the purchase price (expected that the grant will be earned back from the royalties of future video games). • Video game consoles also incur costs associated with virtually any consumer electronics device. For example, the electronics and mechanisms of the system need to be housed in a housing. Manufacturers need to provide service guarantees. Selling the retail of the system Φ Merchants are required to receive sales of the system and/or Lihu for sales of video game software. All of these factors add to the cost of the video game console, which must be funded by the manufacturer, delivered to the consumer, or funded by both the manufacturer and the consumer. In addition, piracy is a big problem in the video game industry. In fact, the security agencies used on each of the larger video game systems have been "broken" over the years, resulting in unauthorized copying of video games. For example, the Xbox 360 security system ruptured in July of the next 2 years and users can now download illegal copies online. Downloadable games (for example, games for 叱 or ·) 139840.doc -33- 200942305 = vulnerable to piracy. In a specific region of the world (where piracy control is not illusory 'substantially* there is a viable market for independent video game software, because users can easily purchase pirated copies at a fraction of the cost with legitimate copies. X, in the world In many places, the cost of the game console is a percentage of revenue, so that even if piracy is controlled, few people can afford the game system of the current state of the art. In addition, the market for games used reduces the revenue of the video game industry. When the user becomes bored with the game, he can sell the game to the store that sells the game to other users. This unauthorized but common practice significantly reduces the revenue of the game publisher. Similarly, 'when each When there is a platform change every few years, there is usually a reduction of about 5G% of sales. This is due to the fact that when the user knows that a newer version of the platform will be released, the user stops buying games for the older platform (eg ' #后发布3, the user stopped buying the Playstation 2 game). Combined, the loss of sales and the increase of the "new flat mouth" Development costs may have on the profitability of game developers of the very significant adverse effects. ❹ new game console is also very expensive responsibility .Xbox 360, Nintendo Wii and

Sony Playstation 3 is retailed for hundreds of dollars. The PC gaming system can cost up to $8. This represents a significant investment by the user, in particular, considering the fact that the hardware became obsolete in a few years and many systems were purchased for the child. The above problem: 3⁄4 method is an online game in which the game code and data host are hosted on the feeding device and delivered to the client machine as required, and the compressed video and audio are serialized via a digital broadband network. flow. Some of the 139840.doc •34·200942305 divisions (such as 'G-Cluster of Finland', which is now a subsidiary of SOFTBANK Broadmedia in Japan) currently offer such services online. Similar gaming services become available in local networks such as hotels and their networks provided by dsl. and cable TV providers. The disadvantage of these systems is the delay, that is, the time it takes for the signal to travel to the game console and travel from the game server. The game server is usually located in the "head end" of the operator. Fast motion video games (also known as "twitch" video games) require very low latency between when the user performs the action by the game controller and when the display screen is updated to show the result of the user action. . A low latency is required to make the user feel the game respond "immediately". The type of visual game and the proficiency of the user - and the user is not required to be delayed. For example, for slow • informal games (like backgammon) or slow action role-playing games, a 100 millisecond delay may be tolerable, but in fast action games, over 70 milliseconds or 80 milliseconds. The delay can cause the user to perform worse in the game and is therefore unacceptable. For example, in a game that requires fast response time, there is a sharp drop in accuracy when the delay is increased from 5 〇 milliseconds to 100 milliseconds. When the game or application server is installed in a nearby controlled network environment or when the user's network path is predictable and/or tolerant of peak bandwidth, in terms of maximum latency and latency consistency, The control delay is easy to pass (for example, the user observes a steady motion from the digital video capture via the network). This level of control can be achieved either between the cable TV network head end and the cable TV user's home, or from the DSL Central Office 139840.doc •35· 200942305 to the user's home, or from the server or user The commercial office area network (LAN) environment is likely to obtain a targeted, point-to-point private connection between businesses with guaranteed bandwidth and latency. But in the game or application system that hosts the game console in a server center connected to the universal Internet and then streams the shrunk video to the user via a broadband connection, many factors cause delays, resulting in A severe limitation in the deployment of prior art systems. In a typical broadband wideband family, the user can have a DSL_data machine for broadband service. These broadband services typically incur up to 25 milliseconds of travel between the user's home and the universal Internet (and sometimes even later). In addition, there is a round trip delay due to routing to the server center via the Internet. The delay through the Internet changes based on the delay in the route of the given data and the data being routed. In addition to routing delays, tripping delays are also incurred due to the speed of light passing through the fibers interconnecting most of the Internet. For example, for every 1000 miles, a round trip delay of about 22 milliseconds is incurred due to the speed of light passing through the fiber and other _. The extra delay can be incurred due to the data rate of the tribute of the rushing stream through the Internet. For example, if the user has a DSL service sold under "(4) hunger service, in practice, the latter will most likely have = at least 5 Mbps under the throughput" and will likely see the factors periodically (such as during the peak load time during the digital subscriber line access multiplex. (DSLAM) congestion) resulting in the degradation of the connection. If the coaxial power is shared by the local end of the adjacent == device, there is congestion or electrical data. Machine System Network: 139840.doc •36·200942305 There is congestion elsewhere, a similar problem can arise, reducing the data rate of the cable modem used for the sale of the "6 Mbps Cable Data Server Service" to Far less than the data rate. If the data packet under the stable rate of 4 Mbps is unidirectionally self-serving in the User Datagram Protocol (UDP) format, the server center streams through these connections, and if everything works properly, the data packet Will pass without incurring additional delays, but if there is congestion (or other obstruction) and only 3.5 Mbps can be used to stream data to the user, then in the case of 'Xin', the packet will be discarded' resulting in lost data, or The packet will be queued at the point of congestion until it can be sent 'by taking an extra delay. Different congestion points have different capacity for storing the delayed packets, so in some cases, the congestion will not be successfully resolved immediately. In other cases, millions of bits of data are sorted into • and finally sent. However, in almost all cases, the queue at the point of congestion has a capacity limit, and once it exceeds these limits, the queue will overflow and the packet will be discarded. Therefore, in order to avoid incurring additional delays (or worse, loss of packets), it is necessary to avoid exceeding the data rate capacity from the game or application server to the user. Delays are also incurred due to the time required to compress the video in the server and decompress the video in the client device. When the video game _i executed on the server is calculating the frame to be displayed, the step is delayed. Currently available video compression algorithms suffer from high data rates or high-tech. For example, the term “sports” is a (four) lossy compression algorithm characterized by a low extension of $. Each frame of the video is compressed independently of each other frame of the video. When the client device receives a frame of compressed motion ( (} video, its 139840.doc •37- 200942305 can immediately decompress the frame and display the frame, resulting in a very low latency. Each frame is compressed separately, so the algorithm cannot take advantage of the similarity between successive frames' and therefore only the in-frame video compression algorithm suffers from very high data rates. For example, 6 〇 fps (per second) Frame number) 64〇><480 motion review video may require 4 Mbps (million bits per second) or 40 Mbps (megabits per second) or more. For these low solutions · The high data rates of the resolution video window will be too expensive in many broadband applications (and indeed for most consumer Internet-based applications). Also, because each frame Independently compressed, so artifacts in the frame due to lossy compression may appear at different locations in successive frames. This can result in the viewer being seen as moving when decompressing the video. Visual artifacts. Other pressures The bedding method (such as 'mpeg2, H.264 or VC9 from Micr〇s〇ft') can be used in prior art configurations to achieve high compression ratios, but at the expense of high latency. The algorithm utilizes inter-frame compression and in-frame dust reduction. Periodically, these algorithms perform only intra-frame compression of the frame. This type of frame is called a key frame (usually called ". Frame followed by this The equal-width algorithm usually compares the I frame with both the previous frame and the successive frame. Instead of compressing the previous frame and the subsequent frame independently, the algorithm determines the image from the I frame to the previous frame and What changes have been made to successive frames, and then their changes are stored as: "B" frame (for changes before i frame) and "P" frame (for changes after I frame). This leads to comparison The frame compresses a much lower data rate.

It is usually at the expense of higher latency. The I frame is usually much larger (usually 10 times larger) than the framed circle 1 consolidating AP frame, and therefore, 139840.doc -38. 200942305 is proportionally longer at a given data rate transmission. time. Consider, for example, a case where the I frame is 10 times the size of the B frame and the p frame and there are 29 B frames + 3 for each single I frame (^Ρ图. Box=59 Between boxes, or for each "frame group" (GOP) a total of 60 frames. Therefore 'at 60 fps, there is one 60 frame GOP per second. Suppose the transmission channel has a maximum data rate of 2 Mbps. In order to achieve the most m3 video in the channel, the compression algorithm will generate a 2 Mbps stream and given the above ratio 'this will result in 2 million bits per frame (Mb) / (59 + 10) = 30,394 bits and 303, 935 bits per I frame. When receiving a compressed video stream by a decompression algorithm, in order to play video stably, it needs to be decompressed at regular intervals (for example, 60 fps). In order to achieve this result, if any frame is subject to transmission delay, all frames need to be delayed - at least one delay, so the worst case frame delay will be defined for each view frame. Delay. Because the frame is the largest, the frame introduces the longest transmission delay, and the entire I frame will be Received before the decompressible and display frame (or between any frames depending on the I frame). Assuming a channel data rate of 2 Mbps 'transfers an I frame will cost 3〇3,935/2 Mb=145 milliseconds An inter-frame video compression system (as described above) that uses a large percentage of the bandwidth of the transmission channel will experience long delays due to the large size of the j-frame relative to the average size of the frame. Or, in other words, when the prior art box The inter-compression algorithm achieves a lower average data rate per frame than the in-frame compression algorithm (eg, 2 MbPs versus 40 Mbps), which still suffers from high peak per frame data rates due to large I-frames (eg, 303, 935 * 60 = 18.2 Mbps). But the kiss has lived. 'The above analysis assumes that both the p-frame and the B-frame are much smaller than the I-frame. 139840.doc -39- 200942305 Although this is generally true, but for the previous and Frames, high motion, or scenes with unrelated high image complexity are not valid. In these cases, the P or B frame can become larger than the ! frame (if p frame or ^ The box becomes larger than the I frame, then the tip compression algorithm passes Replace the P-frame or the B-frame with the I-frame. Therefore, the frame size can be peaked at any time in the digital video stream. Therefore, Compressed video, when the average video data rate is close to the data rate capacity of the transmission channel (often for this situation, given the high data rate requirement for video) 'from the I frame or the large P frame or B frame The high peak data rate results in a high frame delay. Of course, the above discussion only characterizes the compression algorithm delays generated by the large 6-frame, p-frame or I-frame in G〇p. If you use the 8 frame, the delay will be higher. The reason for this is because all b frames and I frames after the B frame must be received before the 3 frame can be displayed. So in such as

In the picture group (G〇p) sequence of BBBBBIPPPPPBBBBBIPPPPP, there are 5 B frames before each frame, and only the video decompressor can display the first after receiving the subsequent /8 frame and I frame. B frame. Therefore, if the video is streamed at 60 fps (ie, 16.67 milliseconds/frame), then before the first frame can be decompressed, no matter how fast the channel bandwidth is, five frames and thumbnails are received. The box will cost 16.67*6 = 1〇〇 milliseconds, and this is the case for only 5 8 frames. A compressed video sequence with 30 frames is quite common. In addition, at a low channel bandwidth such as 2 Mbps, the delay effect due to the small I of the j frame is largely added to the delay caused by waiting for the B frame to arrive. Therefore, on the 2 Μ-channel, in the case of 139840.doc -40· 200942305, it is quite easy to use the delay of the prior art video compression technology over 5 milliseconds or more than 500 milliseconds. If the B-frame is not used (at the expense of a lower compression ratio for a given quality level), then the B-frame delay is not incurred, but the delay due to the peak frame size described above is still incurred. The problem is exacerbated by the nature of many video games. The video compression algorithm utilizing the GOP structure described above is largely optimized for use with live video or movie material that is intended to be used for passive viewing. Usually, the camera (real camera, or virtual camera in the state of the animation generated by the computer) and the scene are relatively stable, just because if the camera or scene moves too far and bumpy, the video or movie material (a) usually looks at the order. People are unpleasant, and (b) if they are being watched, when the camera suddenly bumps back and forth, the viewer is usually unable to follow the action closely (for example, if the camera is disturbed while shooting the child who blows out the candle on the birthday cake) And suddenly bumping back and forth between the cakes, the viewers usually focus on the children and the cake, and ignore the brief interruption when the camera suddenly moves. In the case of video conferencing or video conferencing, the camera can be held in a fixed position and not moved at all, resulting in very little data spikes. But 3D high-motion video games are characterized by constant motion (for example, considering 3D competitions, where the entire frame is in fast motion during the duration of the competition, or the first-person shooter game where the virtual camera is constant) Moving back and forth bumpyly • These video games can produce a sequence of frames with large and frequent peaks, where the user may need to clearly see what happened during their sudden movements. Therefore, in 3D high motion In video games, compression artifacts 139840.doc 41 200942305 are far intolerable. Therefore, the video output of many video games (due to their nature) produces a compressed video stream with very sharp and frequent peaks. The user has a small tolerance for high latency and gives all of the above delay reasons, so far there are limitations on video games hosted by server hosts that stream video over the Internet. In addition, if highly interactive applications are required The program host is hosted on the universal Internet and the video strings & 'these applications are made The user suffers from similar restrictions. 5 Xuan and other services require network configuration, and the host hosting server is directly set at the head end (in the case of wide-band environment) or the central office (in the case of digital subscriber line (DSL)). In the middle, or in the LAN in the commercial context (or on a specially graded private connection), to control the route and distance from the client device to the server to minimize latency and adapt to peaks without incurring delays. (Typically rated at 1 〇〇 still (4) and leased lines with sufficient bandwidth usually support peak bandwidth requirements (for example, 18 Mbps peak bandwidth is one of the 100 Mbps LAN capacity fractions). Peak bandwidth requirements can also be accommodated by residential broadband infrastructure. For example, on an electrical τν system, a dedicated bandwidth can be given for digital video traffic, which can handle peaks such as large r frames. In addition, 'on the DSL system, a higher speed muscle data machine (considering high peaks) can be supplied, or a special J-knife connection that can handle higher data rates can be supplied. However, attached to the general purpose The familiar network around the data machine and DSL infrastructure is far from tolerant for peak bandwidth requirements for compressed video. Therefore, online services (hosting video games or application hosts to the client) The device is long-distance in the server center, and 139840.doc •42- 200942305 residential broadband connections cause compressed delays and peak bandwidth requirements via the Internet. _ Especially for shooting games and other Use de-private (for example, first-person _, interactive action game, or application that requires quick return/time). [Embodiment] The present disclosure will be further described in the following [Embodiment] and the accompanying drawings. It is to be understood that the invention is not intended to be limited Specific details (such as device type, system group method, etc.) are set forth in the description to provide a thorough understanding of the present disclosure. However, '@...the skilled technicians should be aware of the implementations described in the practice. Examples may not require such specific details. 4❻ Figure 2a to Figure 2 provides a high-level architecture of two embodiments, in which the video game and software application are hosted by the host hosting service 21 and hosted by the user service under the subscription service (note, "Use "where" means that wherever the user is located, if the mobile device is used, including the outdoor device, the client device 205 is via the Internet 2〇6 (or other public or private network). access. The client device 2〇5 can be a general-purpose computer having an internal or external display device 222 with a wired or wireless connection to the Internet (such as Microsoft Corporation or Microsoft-based Macintosh computers). ), or it may be a dedicated client device (such as a wired or wireless connection to the Internet) that will turn video and audio out to a monitor or television 222, such as a set-top box, or it may be speculated to be 139840.doc • 43· 200942305 Comes with mobile devices with wireless connectivity to the Internet. Any of these devices may have their own user input devices (eg, 'keyboard, buttons, touch screens, track pads or mertia丨-sensing wands, video capture cameras and / or motion tracking camera, etc., or it can use an external input device 221 that is connected by wire or wirelessly (eg, keyboard, mouse, game controller, inertial sensor, video capture camera, and/or motion tracking) Cameras, etc.) As described in more detail below, the colocation service 21 includes servers of various performance levels (including those with high capability CPU/GPU processing capabilities). During playback of the game or use of the application on the colocation service 210, the home or office client device 205 receives the keyboard and/or controller input from the user' and then enters the controller via the Internet 2〇6 Transfer to the host agency service 210, in response to which the host hosting service 21 executes the game code and generates successive frames for the video output (video image sequence) of the game or application software (eg, if the user presses A button that directs the person on the screen to move to the right, the game program will then produce a sequence of video images showing the person moving to the right. The video image sequence is then compressed using a low latency video compressor and the colocation service 2 then transmits the low latency video stream over the internet 206. The home or office client device then decodes the compressed video stream and reproduces the decompressed video image on a monitor or τν. Therefore, the computational and graphics hardware requirements of the client device 205 are significantly reduced. The client 205 only needs to have the processing capability for forwarding the keyboard/controller input to the Internet 206 and decoding and decompressing the compressed video stream received from the Internet 2〇6, actually 139840 .doc -44 - 200942305 Any personal computer can do this in software on its CPU (for example, Inte's dual-core cpu running at about 2 GHz can decompress compressors using H.264 and Windows Media VC9 72〇p hdtv). In addition, in the case of any client device, the dedicated chip can also be executed at a much lower cost than the general-purpose CPU and much less power consumption than the general-purpose CPU (such as required by modern PCs). Used for video decompression of these standards. Notably, in order to perform the transfer controller input and decompression view φ (four) function 'home client device 2 〇 5 does not require any special graphics processing unit (GPU), CD player or hard drive (such as Figure 1 The prior art video game system shown in the middle). As games and application software become more complex and more photo-realistic, they will require higher performance CPUs, GPUs, more RAM, and larger • faster disks. The computing power at host A pipe service 210 is continuously upgraded' but the end user will not need to upgrade the home or office client platform 205 because the home or office client will be enabled by a given video decompression algorithm. The processing requirements of platform 205 remain constant for display resolution and frame rate. Therefore, there are no hardware limitations and compatibility issues seen today in the systems illustrated in Figures 2a through 2b. Because the game and application software is only executed in the console service of the colocation service 21〇+, so in the user's home or office (unless there are conditions other than ^), the "office" as described in this article will include There is never a copy of the game or application software (in the form of optical media, or downloaded software) in any non-domestic background including, for example, a classroom. This significantly reduces the possibility of illegal copying (piracy) of games or application software and the possibility of pirating valuable libraries that can be used by games or application software. In fact, if you need a specialized server (for example, a very expensive, large or noisy device) to play games or application software that is not feasible for home or office use, even if you get a game or application software. A copy of the pirated copy will not be operated in the home or office. Micro-management service 210 provides software development to design video games in a consistent application or application software developer (which is generally referred to as a software development company, a game or movie studio, or a game or application software publisher). Tools so that it can be designed to be able to play games on the colocation service 21〇. These tools allow developers to take advantage of the features of colocation services (the features are usually not available in a standalone pc or game console) (eg 'very fast access to a very large database of complex geometries (unless there is another #) $ then "geometry" will be used herein to refer to the polygons, textures, .a ^ and parameters) that define the 3D dataset. t Lighting, ^ and other components Ο Under this architecture, different business models are possible. Under the model, the host escrow service 210 pays royalties from the end user and pays royalties to the developer 220 as shown in Fig. 2a. + ϋ , _, exhibition is not in the alternative implementation (shown in Figure 2b), the developer 220 directly receives π 1 person from the user. The fee is charged to the host and the spear force 210 is paid for the host to manage the content of the application. This proprietary principle is not limited to (4) to provide any specific business model for the line. The video features that are compressed by the host computer on behalf of the host I39840.doc -46- 200942305 As discussed earlier, one of the significant problems with online video game services or application software services is latency. The delay of 7 〇 _ 8 〇 milliseconds (3) the time when the input is actuated by the user until the time when the response is displayed on the display device is the upper limit for games and applications that require fast response time. • However, this is very difficult to achieve in the case of the architecture shown in Figures 2a and 2b due to a large number of actual and physical constraints. As indicated in Figure 3, when the user subscribes to the Internet service, the connection φ is often rated to the nominal maximum data rate 301 to the user's home or office. Depending on the provider's policy and routing device capabilities, the maximum data rate may be more or less strictly enforced, but the actual available data rate is typically lower due to one of many different reasons. For example, there may be too many network traffic at the DSL central office or on the local cable modem loop, or there may be noise on the cable, causing the dropped packets, or the provider may establish a per-user The maximum number of digits per month. Currently, the maximum downlink data rate for cable and DSL services is often in the range of hundreds of kilobits per second (KbP to 30 Mbps. Honeycomb services are typically limited to hundreds of Kbps of data. However, broadband The number of users of service ^ speed and subscription broadband services will increase dramatically over time. Currently, some analysts estimate that 33% of US broadband users have data rates below 2 Mbps or above 2 Mbps. For example Some analysts predict that by 2010, more than 85% of US broadband users will have a data rate of 2 Mbps or more. As indicated in Figure 3, the actual maximum data rate 302 available can follow Time fluctuates. Therefore, in the case of low latency, online games or application software 139840.doc -47- 200942305, it is sometimes difficult to predict the data rate for a particular video stream. The scene complexity and motion of the temple are given at a given number ((4) with a given resolution (for example, the data rate required for the 64 〇χ 4 ^ sec frame quality level 3. 3 liters °? ° (4) maintains the feed rate 302 ( In Figure 3, the use of the majority of the poor and the peak of the peak of the ^ Τ 则 则 ' ' 则 则 则 则 则 则 则 则 则 则 则 则 ' ' ' ' ' ' έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ έ Thus, the lost data on the screen 1 and the distorted/lost shadows, his service will temporarily buffer (ie, rank (4)) extra packets and provide the packets at the available data rate, resulting in an increase in latency. - Unacceptable results for many video games and applications. Finally, some Internet service providers increase the data rate: for malicious attacks (such as whether to cut service attacks (used by computer hackers) A well-known technique for deactivating a network connection, and will ( - 4) and will disconnect the user's Internet connection during a special time period. Thus the embodiment described herein seeks to ensure that the video game is used. The required data rate does not exceed the maximum available data rate. The hosted service architecture. Figure 4a illustrates the architecture of the hosted service 21〇 according to an embodiment. The hosted service 21G can be located on a single— Multiple server centers in the server center are scattered (^ knife dare (thinking that users with lower latency than others to the path of the specific feeder center provide low latency connections to provide load balancing between users) And providing redundancy in the event of one or more server center failures. The host generation f service 21() can eventually include thousands or even millions of servers 4〇2, so the servo is very large The use of 139840.doc -48· 200942305 user base. The hosted service control system 4.1 provides overall control over the hosted service 210 and directs routers, servers, video compression systems, billing and Accounting system, etc. In one embodiment, the colocation: the service control system 401 is implemented on a Linux-based distributed processing system that is coupled for storage for user information, server latency, and system statistics. A RAID array of data databases. In the above description, various actions implemented by the colocation service 21 are initiated and controlled by the colocation service control system 401 unless attributed to other specific systems. The colocation service 210 includes a number of servers 4, such as those currently available from Intel, IBM, and Hewlett Packard, among others. Alternatively, the server 402 can be assembled into a custom component configuration, or at most, the server can be integrated to implement the entire server as a single wafer. Although this figure shows a small number of servers 4〇2 for illustrative purposes, in actual deployments, there may be as few as one server 4〇2 or as many as millions or tens of millions of servers 402. server. Server 402 can be configured in the same way (as an example of some configuration parameters, with the same cpu type and performance; with or without GPU, and with GPU, with the same GPU type and performance; with the same number of CPUs and GPIJ; RAM with the same amount and the same type/speed; and with the same RAM configuration), or various subsets of the server 402 can have the same configuration (for example, 25% of the servers • can be configured in a specific way) 50% of the servers are configured in a different way, and 25% of the servers are configured in yet another way, or each server 4〇2 can be different. 139840.doc •49· 200942305 ^-In the embodiment, the server has no disk, that is, its own large-capacity storage is crying. It is optical or magnetic storage, or semiconductor-based. The storage device, the flash memory or the other large storage unit of the servo-like power system, each accessing the shared mass storage device via the fast backplane or the network connection. In the real (four), 2 fast catch connection is connected to the independent redundant disk array (raid) milk series of the wrong area network (SAN), there is a connection between devices using ultra-high-speed Ethernet implementation . As is known to those skilled in the art, the certificate can be used to combine many RAID arrays together, resulting in extremely high bandwidth - close to or possibly exceeding the Ram available for use in current game consoles and PCs. bandwidth. Moreover, while (10) hopping based on rotating media such as magnetic media often has significant seek time access latency, a RAID array based on semiconductor memory can be implemented with much lower access latency. In another configuration, some or all of the servers 4〇2 provide some or all of their own mass storage at the local end. For example, the server 402 can store frequently accessed information (such as a copy of its operating system and video game or application) on a low-latency local flash memory-based storage, but A SAN is used to access a RAID array 405 with a higher seek latency based on rotating media to access a large database of geometry or game state information less frequently. Additionally, in one embodiment, the co-hosting service 21 uses the low latency video compression logic 404 described in detail below. Video compression logic 404 can be implemented in software, hardware, or any combination thereof (described in its specific embodiments below). Video compression logic 404 includes logic for compressing audio and visual material 139840.doc -50- 200942305.

In operation, when playing a video game via a keyboard, mouse, game controller or other input device 421 or using an application at the user location 211, the control signal logic 413 on the client 41 5 will be represented by the user. The actuated button press (and other types of user input) control signals 4〇6b (usually in the form of a UDP packet) are transmitted to the colocation service 21〇. In the future, the control signals from a given user are routed to the appropriate server (or to multiple servers if multiple servos respond to the user's input device) 402. As illustrated in Figure 4a, the control signal 4 can be routed to the server 402 via the SAN. Alternatively or additionally, control signals may be routed directly to server 402 via a hosted service network (e.g., an Ethernet based local area network). Regardless of how the control signals 4〇6a_b are transmitted, the or the servers execute the game or application software in response to the control signals 4〇6a_b. It is not illustrated in Figure 4a that various types of connection components (such as firewalls and/or gateways) can handle the host hosting service 21 (for example, the colocation service 21〇 and the Internet 41〇). Incoming and outgoing traffic between the edge of the user premises 211 (between the Internet 41 and the home or office client 415). The graphics and audio output of the executed game or application software (ie, the new video image sequence) is provided to the low-latency video compression logic 4G4, and the low-latency video compression logic 4〇4 is based on the low-latency video compression technology (such as this article) The techniques described in the above) compress the video image sequence and via the Internet 41 (or, as described below, via the best high speed network service bypassing the universal Internet): compressed video stream (usually with compressed or uncompressed audio) transmitted back to 139840.doc -51 - 200942305 to client 415. Next, the low latency video decompression logic 412 on the client 415 decompresses the video and audio streams and reproduces the decompressed video stream, and typically plays the decompressed audio stream on the display device 422. Alternatively, the audio may be played on a separate speaker from the display device 422 or not at all. Note that although input device 421 and display device 422 are shown as stand-alone devices in Figures 2a and 2b, they can be integrated into a client device such as a portable computer or mobile device. The home or office client 415 (described previously as home or office client 205 in Figures 2a and 2b) can be a very low cost and low power device with very limited computational or graphical performance and possibly a very limited basis. The end of the mass storage or does not have the local mass storage. In contrast, each server 402 that is integrated into the SAN 403 and the plurality of RAIDs 405 can be an extraordinarily high performance computing system, and in fact, if multiple servers are cooperatively used in a side-by-side configuration configuration, then There is almost no limit to the amount of calculation that can be tolerated and the Bb force to be processed. In addition, due to the low-latency video compression 4〇4 and the low-latency video decompression 412 (perceived by the user), the server 402 is supplied to the user. When the user presses the input device 421 When the button is connected, the image on the display 422 is updated in response to the button press (with no meaningful delay in perception), as if the game or application soft system is executing on the local end. Therefore, for a very inefficient computer or only The home or office client 415, which implements the low-latency video decompression and control signal logic 413 of the low-cost chip, effectively provides the computing capability for the latter at the remote location available at the local end. The ability to play the highest-end, intimate (usually new) video games and applications with the highest performance of 139840.doc -52· 200942305. The show is very basic and inexpensive for home or office clients. Device 465. - This device is an embodiment from the home or office client 415 of Figure and Figure. It is about 2 inches long. It has and is powered by Ethernet - (P E) The Ethernet cable 462 of the Ethernet cable is connected to the Ethernet socket 462 to obtain its power and its connectivity to the Internet. The device can support the network. Address translation (NAT) network execution ❿ NAT纟 office to the environment, many new Ethernet switches have P host E and bring P〇E directly to the Ethernet port in the office Under this type of 幵y, it is the Ethernet wall mirror from the wall jack to the client 465. If the available Ethernet connection does not carry power (for example, in a messy • Cable data) In the case of a home without a PqE, there is an inexpensive wall-type "brick J" (ie, power supply) that will accept a powerless Ethernet cable and output a P with a p〇E The network 465 3 has a control signal logic 413 (Fig. 4a) coupled to the Bluetooth wireless interface, such as a keyboard, mouse, game control and/or microphone and/or earphone. The Bluetooth input device 479 is interfaced. Again, one embodiment of the client 465 is coupled to the display device 468. The video is output at 12 〇 fps, and the display device 468 is capable of supporting 12 〇 fps video and signals a pair of opaque glasses 466 (typically via infrared) to alternately mask one eye and then the other for each successive pattern. The effect perceived by the user is to "jump out" the stereoscopic 3D image of the display screen. One of the operations is supported. The display device 468 is a Samsung HL-T5076S. Since the video stream for each eye is separate, in one embodiment where the two independent video streams are compressed by the host 139840.doc • 53· 200942305 pipe service 210, the frames are interleaved in time, and The frame is decompressed in two separate decompression processes at the client side 465. The client 465 also includes low-latency video decompression logic 4丨2, which decompresses incoming video and audio and outputs 'HDMI (High Definition Multimedia Interface), connection via HDMI (High Definition Multimedia Interface), connector 463 The device 463 is inserted in SDTV (Standard Definition Television) or HDTV (High Definition Television) 468 to provide video and audio to τν, or to be inserted into HDMI-enabled monitor 468. If the user's monitor 468 does not support HDMI, you can use HDM]^DVI (Digital Visual Interface), but the audio will be lost. Under the HDMI standard, display capabilities (eg, supported resolution 'frame rate') 464 are conveyed from display device 468 and then transmitted back to colocation service 21 via Internet connection 462, thus The host agency service 21 enables the compressed video to be streamed in a format suitable for the display device. - Figure 4d shows a home or office client device 475, which is identical to the home or office client device 465 shown in the figure, except that the client device 475 has more external interfaces. Again, the client 475 can accept p〇e Ο to power, or it can occupy an external power adapter (not shown) that is plugged into the wall. The video camera 477 provides the compressed video to the client 475 using the client 475 USB input, and the compressed video is uploaded by the client 475 to the colocation service 210 for use as described below. The low latency compressor will be built into the camera 477 using the compression techniques described below. • In addition to the Ethernet connector for its Internet connection, the User 475 also has an 802.11 g wireless interface to the Internet. Both interfaces 139840.doc -54- 200942305 are capable of using NAT within a network that supports NAT. In addition, in addition to the HDMI connector for outputting video and audio, the client 475 also has a dual-link DVI-I connector, and the dual-link DVI-I connector includes an analog output (and has a VGA output). Standard adapter 'cable'. It also has an analog output for composite video and S video. For audio, the client 475 has a left/right analog stereo RCA jack and has a TOSLINK output for digital audio output. In addition to the Bluetooth wireless interface to input device 479, it also has a USB jack for interfacing to the input device. Figure 4e shows an embodiment of the internal architecture of the client 465. All or some of the devices shown in this figure can be implemented in a field programmable logic array. Columns, custom ASICs, or several discrete devices (custom design or off-the-shelf). 乙Ethernet 497 with PoE Attached to the Ethernet interface 48 1 . Power 499 is derived from Ethernet 497 with PoE and is connected to the rest of φ in user terminal 465. Bus 480 is a common bus for communication between devices. Control CPU 483 that executes the small client control application from flash memory 476 (almost any small CPU is suitable, such as MIPS R4000 series CPU at 100 MHz with embedded RAM) for network ' That is, the protocol of the Ethernet interface stacks and also communicates with the colocation service 2 10 - and configures all devices in the client 465. It also processes the interface with the input device 469 and sends the packet (along with the user controller data subject to forward error correction protection) back to the colocation service 210. 139840.doc -55- 200942305 Again, control CPU 483 monitors packet traffic (e.g., packet loss or delay) and the timestamp of its arrival. This information is sent back to the colocation service 210' so that it can constantly monitor the network connection and adjust the content it sends accordingly. The flash memory 476 is initially loaded with a control program for controlling the CPU 483 and a serial number unique to a particular client terminal 5 unit at the time of manufacture. This sequence number allows the colocation service 21 to uniquely identify the user 465 unit. The Bluetooth interface 484 wirelessly communicates to the input device 469 via its antenna (within the client 465). Video decompressor 486 is a low latency video decompressor configured to implement the video decompression described herein. A large number of video decompression devices exist, either as off-the-shelf products or as intellectual property (Ip) with a design that can be integrated into an fpga or custom ASIC. One is provided for Η.%* decoding H ^ a] ^Nsw Australian Ocean Logic of Manly 〇 The advantage of using IP is that the compression technique used in this article does not match the compression standard. Some standard decompressors are flexible enough to be adapted to the compression techniques of this article, but some standard decompressors may not. However, in the case of IP, it is completely flexible in redesigning the solution as needed. The output of the video decompressor is coupled to a video output subsystem 487, which combines the video plane to the video interface of the H_Interface. The sound decompression subsystem 4 8 8 or the flyer uses the available standard audio decompressor to implement 'or install *5】·杳·, with a poor, or can be implemented, for example, 139840.doc 200942305

The audio decompression is implemented in the control processor 483 of the Vorbis audio decompressor. The device that implements the audio decompression is coupled to an audio output subsystem 489 that couples the audio to the audio output of the HDMI interface 490. Figure 4f shows an embodiment of the internal architecture of the client 475. As can be seen, in addition to the additional interface and optional external DC power from the power adapter plugged into the wall (and if so, the optional external DC power replacement will be from the Ethernet PoE 497), The architecture is the same as that of the client 465. The functionality common to the client 465 will not be repeated below, but the additional functionality will be described below. The CPU 483 communicates with additional devices and configures additional devices.

The WiFi subsystem 482 provides wireless internet access via its antenna as an alternative to Ethernet 497. The WiFi subsystem is available from a number of manufacturers, including Atheros Communications of Santa Clara, CA. USB subsystem 485 provides an alternative to Bluetooth communication for wired USB input device 479. The USB subsystem is fairly standard and can be easily used in FPGAs and ASICs, and is often built into existing devices that perform other functions such as video decompression. The video output subsystem 487 produces a wider range of video outputs than the video output in the client 465. In addition to providing HDMI 490 video output, it provides DVI-I 491, S-Video 492 and Composite Video 493. Also, when the DVI-I 491 interface is used for digital video, the display capability 464 is passed back from the display device to the control CPU 483 so that it can notify the host escrow service 210 of the capabilities of the display device 478. The 139840.doc -57- 200942305 interface provided by the video output subsystem 487 is a fairly standard interface and is readily available in many forms. The audio output subsystem 489 digitally outputs audio via the digital interface 494 (S/PDIF and/or Toslmk) and outputs an analog form of audio via the stereo analog interface 495. Round trip delay analysis Of course, in order to realize the benefit of the previous paragraph, the delay between the user's action using the input device 421 and the effect of seeing the action on the display device 42〇 should be no more than 7 milliseconds _8 milliseconds. This delay must account for all factors in the path from the input device 421 in the user premises 211 to the colocation service 21A and back to the user premises 2 to the display device 422. Figure 4b illustrates the various components and networks through which signals must travel, and above these components and the network is a timetable that lists exemplary delays that can be expected in actual implementation. Note that the graph is simplified to show only important path routes. Other routes for information about other features of the system are broadcast below. A double-headed arrow (e.g., arrow 453) indicates a round-trip delay and a single-headed arrow (e.g., arrow 457) indicates a one-way delay, and "~" indicates an approximate detection. It should be pointed out that there will be real-world situations in which the listed delays are unachievable. In the United States, the use of the user's premises 211 = electricity, (4) machine connection, these delays can be in the next paragraph Description I II: Cheng... Note that although the cellular wireless connectivity to the Internet will be broken in the system shown in the nested data system (such as _): non-two big, American bees far enough (four)...) The money (four) face material will not show the delay. However, these basic principles can be implemented on future cellular technologies that can implement this level of delay. 139840.doc -58- 200942305 Beginning with the input device 421 at the user premises 211, once the user actuates the input device 421, a user control signal is sent to the client 415 (which may be a standalone device such as a set top box, or It can be a software or hardware implemented in another device such as a pc or a mobile device, and it is packetized (in one embodiment in UDP format) and given a destination address for the packet to reach the host generation. The management service is 21〇. The packet will also contain information indicating which user the control signal is from. The control signal packet is then forwarded to the w n interface 442 via a firewall/router/❷ N AT (Network Address Translation) device 4 4 3 . The WAN interface 442 is an interface device provided to the user premises 211 by the user's ISP (Internet Service Provider). The WAN interface 442 can be a cable or DSL modem, a wiMax transceiver, a fiber optic transceiver, a cellular • data interface, an Internet Protocol- 〇ver-Powerline interface, or many interfaces to the Internet. Any other interface in . Additionally, firewall/router/NAT device 443 (and (possibly) WAN interface 442) can be integrated into client 415. An example of this would be a mobile phone 'which includes software for implementing the functionality of the home or office client 415' and for wirelessly routing and connecting to the Internet via a standard (eg, 802.11 g) Components. The WAN interface 442 then routes the control signals to a "presence point" 441 for the user's Internet service provider (ISSP), which is a WAN transporter that provides connectivity to the user premises 211. A facility with an interface between a universal Internet or a private network. The nature of the point of existence will vary depending on the nature of the Internet service provided. For DSL, it will typically be the central office of the telephone company to which the DS LAM is located. For the 139840.doc -59- 200942305 cable modem, it will typically be the cable multi-system operator (MS〇) headend. For a honeycomb system, it will typically be the control room associated with the honeycomb tower. But regardless of the nature of the point, it will then route the control signal packet to the universal internet 410. The control signal packet is then routed to the WAN interface 444 to the host escrow service 210 via the most likely fiber optic receiver; WAN 444 will then route the control signal packets to the routing logic 4〇9 (which can be implemented in many different ways, including Ethernet switches and routing cages), routing logic_estimating the user's address and going to The nickname is routed to the correct server 4〇2 for a given user. The server 402 then treats the control signals as inputs to the game or application software executing on the server 4〇2 and uses the control signals to process the next frame of the game or application. Once the next frame is generated, the video and audio are output from the servo to the video. The video and audio self-feeding device 402 can be output to the ink retractor 4〇4 via various components. First, the first retractor 404 can be placed in the feeding machine, so the compression can be performed at the local end within 4 〇 2 of the feeding device. Alternatively, it may be connected via a network to a network (either a private network between the server 402 and the video compressor 4〇4, or a network via a shared network such as the SAN 403) ( For example, an Ethernet connection) outputs video and/or audio in a packetized form. Alternatively, video self-feeding benefit 402 can be output via a binary output connector (such as a VGA connector) and then captured by video compressor 404. Alternatively, the audio-server 402 can be digitally encoded (e.g., via an S:IF connector) or analog audio' analog audio to be digitized and encoded by audio compression logic within the video compressor 404. 139840.doc •60· 200942305 Once the video compressor 404 has captured the video frame from the server 4〇2 and the audio generated during the frame time, the video compressor will compress the video using the technique described below. And audio. Once the video and audio are compressed, they are packetized by a single address to send it back to the user's client terminal 415 and routed to the WAN interface 444, which then passes through the WAN interface 444. The Internet 410 routes the video and audio packets. The universal Internet 410 then routes the video and audio packets to the user's presence point 441. The presence point 441 routes the video and audio packets to the WAN interface 442 at the user's premises. The WAN interface 442 routes the video and audio packets to the firewall/router/NAT device 443, and the firewall/router/NAT device 443 then routes the video and audio packets to the client 415. • The client 415 decompresses the video and audio, and then displays the video on the display device 422 (or the built-in display device on the user side) and sends the audio to the display device 422 or to a separate amplifier/speaker or to the user. The amplifier/speaker in the end. 〇 In order for the user to feel that the entire process just described is perceptually free of lag, the round trip delay needs to be less than 70 milliseconds or 80 milliseconds. Some of the delay delays described in the return trip path are controlled by the colocation service 210 and/or the user, while other delay delays are not controlled by the colocation service 21 and/or the user. Nonetheless, based on the analysis and testing of a large number of real-world situations, the following are approximate measurements. • The one-way transmission time 45 1 for transmitting control signals is typically less than i milliseconds, and the round-trip routing 452 via the user premises is typically performed using an easy-to-use consumer firewall/router/NAT switch on the Ethernet. The appointment was completed within 139840.doc -61 - 200942305 seconds. The user ISP extensively changes its round trip delay 453, but in the case of DSL and cable modem providers, it is typically seen between 1 and 25 milliseconds. The back-to-back delay of the Universal Internet 41 is how the routing is routed and whether there are any faults on the route (and these issues are discussed below), but the Universal Internet provides the best. The routing and delay is largely determined by the speed of light passing through the fiber (given the distance to the destination). As discussed further below, the commercial grade fiber optic high speed interface is ignored. Therefore, the Universal Internet Network Delay 454 is similar to the one after the user. For example, λ古μ has determined 1000 miles as the approximate farthest distance from which the host escrow service 21 is expected to be placed away from the user premises 211. At 1 mile (round trip ❹ 2000 miles), the actual transmission time for signals via the Internet is approximately 22 milliseconds. The WAN interface 444 to the host hosting service is usually available.

The round trip delay of route 452, so the rest is video decompression 458 extension 139840.doc -62- 200942305 'video decompression 458 delay depends on whether video decompression 458 is implemented in dedicated hardware or implemented in user device 415 In software such as PCs or mobile devices, the size of the visual display and the decompression of cpU2 performance vary. Typically, decompression 458 takes between 1 millisecond and 8 milliseconds. Thus, the worst-case back-and-forth delay that the user of the system shown in Figure 4a can expect to experience can be determined by adding together all the worst-case delays seen in practice. It is: 1 + 1+25+22+1 + 16+6+8=8〇 milliseconds. In addition, in practice, in practice (with the misunderstandings discussed below) 'This is roughly a prototype version of the system shown in Figure ( (using an off-the-shelf Windows PC in the US as a client device and home DSL and Cable data machine connection) See the round trip delay. Of course, the worst case can lead to much shorter delays, but you can't rely on it to develop widely used business services. In order to achieve the delay listed in the servant via the universal internet, the video compressor 404 and the video decompressor 412 (from Figure 4a) in the client 415 are required to generate a packet stream having very specific characteristics, so that The sequence of packets generated via the entire path from the colocation service 210 to the display device 422 is not subject to delay or excessive packet loss, and in detail, consistently falls over the Internet connection via the user (via the WAN interface) Material 2 and firewall / router / NAT 433) can be used within the constraints of the user's bandwidth. In addition, the video compressor must generate a robust packet stream so that it can't be used in normal Internet and network transmissions; see the inevitable packet loss and packet reordering. Low-latency video compression 139S40.doc -63· 200942305 To accomplish the above objectives, an embodiment employs a new video compression method that reduces the delay and peak bandwidth used to transmit video. Prior to describing such embodiments, an analysis of current video compression techniques will be provided with respect to Figures 5 and 6a through 6b. Of course, if the user has a bandwidth sufficient to handle the data rate required by such techniques, then such techniques can be used according to the basic principles. Note that this article does not address audio compression, but rather states that audio compression is implemented simultaneously and synchronously with video compression. Prior art audio compression techniques that meet the requirements for this system exist. Figure 5 illustrates a particular prior art for compressing video in which each individual video frame 501-503 is compressed by compression 逻辑 logic 520 using a particular compression algorithm to produce a series of compressed frames 5 11-5 13 . One embodiment of this technique is "transport tUPEG" in which each frame is compressed based on a discrete cosine transform (DCT) according to the Joint Image Experts Group (jpEG) compression algorithm. A variety of different types of compression algorithms can be used, however, these basic-principles are still adhered to (eg, wavelet-based compression algorithms such as JPEG-2000). One of the problems with this type of compression is that it reduces every The data rate of a frame is ©: 'But it does not use the similarity between successive frames to reduce the data rate of the total video stream. For example, as illustrated in Figure 5, assume that the frame "Ox·* bit/pixel = 640*480*24/8/1024=9〇〇 kilobytes/frame (κβ/frame) Rate, then for a given quality image, the motion jpEG may only compress the stream (10)' to produce a 9 〇 KB/frame data stream. At 6 frames/sec, this would require 90 KB*8 bits. *6〇 Frame/Second 2... The channel bandwidth, for almost all of the US home Internet today, 139840.doc -64· 200942305 β ❹ will be the same bandwidth' and for many office networks In terms of network connectivity, it is too high a bandwidth. In fact, it is assumed that it requires 6 mussel streams in such a high frequency range and it will only bribe - use I, even in an office LAN environment. Will consume the bandwidth of just Mbps Ethernet LAN and the heavy Ethernet switch that supports the LAN. So 'when compared to other compression technologies (such as those described below) , time, compression inefficiency for motion video. In addition, single-frame compression algorithm using lossy compression algorithm (5) pE G and jpEG_ are the uncompressed compression artifacts in still images (for example, artifacts in dense foliage in a scene may not appear as artifacts, because the eye does not know exactly how dense leaves should be Rendering. However, once the scene is in motion, the artifact may be prominent because the eye detects artifacts that change from frame to frame, although the artifact is in the area of the scene (in this area, false This may be unobtrusive in still images. This leads to the perception of "background noise" in the sequence of frames. The appearance of the "background noise" is similar to the "snow" noise visible during TV reception. . Of course, this type of compression can still be used in the specific embodiments described herein, but in general, in order to avoid background noise in the scene, a high data rate (ie, a low compression ratio) is required for a given perceived quality. ). Other types of compression (such as H.264, or Windows Media VC9, MPEG2, and MPEG4) are more efficient in compressing video streams because they exploit the similarities between successive frames. These techniques all rely on the same general techniques for compressing video. Thus, although the H.264 standard will be described, the same general principles apply to a variety of other compression algorithms. A large number of h.264 compressions 139840.doc •65· 200942305 Retractors and decompressors available' include the FFmpeg open source software library for compressing the H.264ix264 open source software library and for decompressing H.264. 6a and 6b illustrate an exemplary prior art compression technique in which a series of uncompressed video frames 5〇1_5〇3, 559_561 are compressed by compression logic 620 into a series of "I frames" 611, 671; Boxes 612-613; and "B-frames" 670. The vertical axis in Figure 6a generally represents the resulting size of each of the encoded frames (although the frames are not drawn to scale). As described above, those skilled in the art are well aware of the video writing codes using the I frame, the B frame, and the P frame. Briefly, block I 611 is a DCT-based compression of frame 501 that is completely uncompressed (similar to a compressed JPEG image as described above). The size of P-frame 612·613 is typically significantly smaller than the size of I-frame 611 because it utilizes data from the previous I-frame or frame; that is, it contains an indication between the previous I-frame or ρ-frame. Changed information. The frame 670 is similar to the frame except that the frame is subsequently referenced to the frame of the frame and (possibly) to the frame of the previous reference frame. For the following discussion, it will be assumed that the desired frame rate is 6 frames per second, each I frame is about 160 Kb' average frame and the beta frame is 16 Kb and a new I is generated every second. Frame. Under this set of parameters, the average data rate will be: 160 Kb + 16 Kb * 59 = l.l Mbps. This data rate falls appropriately within the maximum data rate for many current broadband Internet connections to homes and offices. This technique also tends to avoid background noise problems from intra-frame coding only. Because the difference between the P-frame and the B-frame tracking frame, the compression artifact tends to appear and disappear from the frame to the frame. To reduce the background noise problem described above. 139840.doc -66 - 200942305 One of the problems with compression of the above type is that although the average data rate is relatively low (for example, 1 _ 1 Mbps), a single I frame may take several frame times to transmit. For example, using prior art, a 2.2 Mbps network connection (eg, a DSL or cable modem with a 2.2 Mbps peak from the largest available data rate 302 of Figure 3&) would typically be sufficient to stream video at M Mbps, One 160 Kbps I frame per 60 frames. This will be done by having the decompressor put a 1 second video into the queue before decompressing the video. At 1 second e

Within this, 1.1 Mb of data will be transmitted, which will be easily accommodated by the 2.2 Mbps maximum available data rate, even though it is assumed that the available data rate may periodically drop by as much as 50%. Unfortunately, this prior art method would result in a leap second delay in video due to the 1 second video buffer at the receiver. This delay is sufficient for many prior art applications (e.g., playback of linear video), but it is an extremely long delay for fast motion video games that cannot tolerate delays greater than 7 milliseconds _ 8 milliseconds. If an attempt is made to eliminate the 1 second video buffer, it will still not result in a sufficient delay reduction for fast motion video games. For example, as previously described, the use of a B-frame will require receipt of all 8 frames and ^ frames before the 1 frame. If it is assumed that approximately 59 non-1 frames are split between the P frame and the B frame, then at least 29_frames will be present and the I frame received before any B frames can be displayed. Therefore, regardless of the available bandwidth of the channel, it is necessary to delay the duration of the frame by 1/6G seconds, or the delay of the milliseconds. Obviously, his time is extremely long. The other method will be to eliminate the B frame and only use the frame and p frame ° (this - the consequence is H to the H level) data rate will increase by 139840.doc -67- 200942305 in this instance For the sake of this, continue to assume that each -! frame: small is (10) Kb and the average P frame size is _, and therefore the data speed is still M Mbps). This method eliminates the inevitable delay of the frame, because the decoding of each ρ frame depends only on the frame received before the frame. The problem with this method is still:! Frame ratio is average? The frame is much larger, so that on low-frequency wide channels (as is typical in most homes and many offices), the transmission of the frame adds a real f delay. This is explained in _. The video stream data rate 624 is lower than the available maximum data rate 621 (except for the I frame)' where! The peak data rate required by the frame (2) far exceeds the maximum available data rate of 622 (and even exceeds the rated maximum data rate of 62 1). The data rate required for the P frame is less than the maximum data rate available. Even if the peak maximum data rate peak at 2.2 Mbps is steadily maintained at its 2.2 Mbps peak rate, it will take 16 〇Kb/22 Mb=7i milliseconds to transmit the work frame, and if the maximum available data rate 622 drops by 〇% ( 11 Mbps), then 142 milliseconds will be used to transmit the I frame. Therefore, the delay in the transmission I frame will fall somewhere between 71 milliseconds and 142 milliseconds. This delay is added to the delay identified in the servant (the delays are up to 7 〇 milliseconds in the worst case), so this will result from the moment the user actuates the input device 421 until the image is presented on the display device 422. The total round trip delay of 141-222 milliseconds is extremely high. And if the maximum available data rate drops below 2.2 Mbps, the delay will increase further. It is also noted that ‘blocking" by ISPs that exceed the peak data rate 623 of the available data rate 622 typically has serious consequences. Devices in different ISPs will behave differently 'but when the packet is received at a data rate 139840.doc -68 - 200942305 that is much higher than the available data rate 622', the following behavior is quite common among the data and isp. a) delay the packet (introduction delay) by placing the packet into the queue, (8) discard some or all packets, (4) stop the delivery time-time period (most likely because the ISP is concerned about malicious attacks, such as "denial service" "attack). Therefore, transmitting packet streams at full data rate (having characteristics such as those shown in the figure) is not a viable option. Peak 623 can be queued at host hosting service 210 and below the maximum available data rate

The data rate of the rate is sent, which leads to the unacceptable delay described in the previous paragraph. In addition, the video stream data rate queue 624 shown in FIG. 6b is a very "brushed (t_)" video stream data rate sequence, and will be expected to be generated due to compression of video from the video sequence. Rate sequence The video sequence does not change very much and has very little motion (eg, as would be common in X video teleconferences, in a video conference call, the camera is in a fixed position with very little motion, and Objects in a scene (for example, a seated person's conversation) show less movement). The video stream data rate sequence shown in Figure 6c is a typical sequence that is expected to be visible from video with much more motion, such as may be produced in a movie or video game or in an application software. Note that in addition to the work frame modifier 633, there are also quite a large number of p-frame peaks (such as 635 and 636) that exceed the available maximum data rate in many cases. Although the peaks of these P-frames are not quite as large as the I-frame peaks, they are still so high that they cannot be carried by the channel at the full data rate, and like the peaks of the 1 frame, the peaks of the P-frame must be transmitted slowly ( This increases the delay). 139840.doc -69- 200942305 on a high frequency wide channel (for example, 100 Mbps LAN, or high frequency wide

On a Mbps private connection, the network will be able to tolerate large peaks such as 1 frame peak ο] or p frame peak 636, but in principle, low latency can be maintained. / However, these networks are often shared among many users (for example, in office environments), and the "peak" data will affect the performance of the LAN, especially when the network traffic is routed to a private shared connection. (for example, from a remote data/center to the office). First, remember that this example is an example of a relatively low resolution video stream with 640x480 pixels added. 6 〇 之 狮 狮 X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X X Using H.264 compression, 6〇 is as follows. The 192” X-face high motion video sequence may require 4.5 Mbps to obtain a reasonable time level. If the frame peak is assumed to be the nominal data rate, it will produce a 45 Mbps peak and a small but still quite large P (four) peak. If several users are receiving video streams on the same network—for example, the private network connection between the office and the data center, it is easy to see how the peak of the video stream from several users can happen. Alignment, thereby drowning the bandwidth of the network, and may overwhelm the bandwidth of the backplane on the network that supports the user's exchange benefits. Even in the case of ultra-high-speed Ethernet, if enough users have simultaneous alignment or network switches. In addition, the == value can submerge the network. The 2560X1440 resolution video becomes more I. The video stream data rate may be 9.5 MbpS, which may result in a second-peak data rate. Needless to say, the 1〇0 connection between the data center and the office (which is now an extraordinarily fast connection) will be completely defeated by the peak traffic of 139840.doc -70· 200942305 from a single user. Therefore, even if LAN and private network connections are more tolerant for peak-stream video, streaming video with high peaks is not required and may require special planning and adaptation by the IT department of the office. _ Ο Of course 'for standard linear video applications, these issues are not a problem' because the data rate is "smoothed" at the transmission point and the data for each frame is below the maximum available data rate 622 and is decompressed. Before the frame, P frame and B frame sequence, the buffer in the client side stores the I frame, the P frame and the B frame sequence. Therefore, the data rate on the network remains close to the average data rate of the video stream. Unfortunately, this introduction delay, even if the B-frame is not used, is unacceptable for low-latency applications such as video games and applications that require fast response times. A prior art solution for mitigating video streams with south peaks is to use techniques commonly referred to as "constant bit rate" (CBR) encoding. The CBR seems to imply that all frames are compressed to have the same bit rate (ie, size), but it is often referred to as the I-negative example, in the compression example _, allowing a certain number of frames to be spanned (in my case, _ frame) the maximum bit rate. For example, in the case of a graph, if a CBR constraint is applied to the encoding that limits the bit rate to, for example, 7 G% of the nominal maximum data rate 621, then the compression algorithm will limit each of the frames. The compression of one will be such that any frame that would normally be compressed using more than 70% of the nominal maximum data rate 621 will be compressed with fewer bits. The result of this is that it will require more bits to maintain the "extremely lacking" of the frame of a given quality level and the image quality of these frames will be less than 139840.doc •71 · 200942305 The other frame products with a maximum data rate of 621 of 7621% are fixed: bad. This method is for a particular type of compressed video (where (4): period is better than motion or scene change and (b) user can accept periodic quality degradation: the result of a readable text. A good example of a suitable CBR application is Depending on the π phone will 4' because there are fewer peaks and the quality is temporarily degraded: (for example, if the camera is swept, resulting in significant scene motion and large peaks), there may not be enough bits during the glance. For high quality (four) compression 'which will result in downgrade (four) image quality), most users can accept: Unfortunately, CBR is not well suited for scenes with high complexity or large amounts of motion and / or need a reasonably constant quality level Many other applications. The low latency compression logic 404 used in the embodiment uses several different techniques to solve many of the problems of streaming low latency compressed video while maintaining high quality. First, the low latency compression logic 4 〇 4 only produces j maps. Block and p-frame, thereby alleviating the need to wait for a number of frame times to decode each B-frame. Additionally, as illustrated in Figure 7a, in one embodiment, low-latency The compression logic 404 divides each uncompressed frame 7〇1_76 into a series of "tiles" and individually encodes each image block into an I-frame or a p-frame. The grouped compressed I and P frames are referred to as "r frames" 711-770. In the particular example shown in Figure 7a, each uncompressed frame is subdivided into 16 images of a 4x4 matrix. However, these basic principles are not limited to any particular sub-dividing mechanism. In one embodiment, 'low latency compression logic 404 divides the video frame into a number of image blocks and encodes one of the image blocks from each frame ( That is, 139840.doc 72· 200942305 compression) is the i frame (that is, the image block is compressed as if it were 1/16 of the size of the full image, and used for this "mini" The compression of the frame is I frame compression) and the remaining image blocks are encoded into p frames (ie, compressed by each frame of the "mini" frame to P frame compression). Compressed • 1 The image block of the frame and the image block compressed into the p frame will be referred to as I image block and "p shadow" respectively. Block. The image block to be encoded as an I image block is changed with each successive video frame. Therefore, in a given frame time, only the image block in the video frame is an I image. Block, and the remainder of the image blocks are p-image blocks. For example, in Figure 7a, image block 0 of frame 701 that has not been shrunk is encoded as image block 10 and the remaining 1-15 The image block is encoded as a p-image block (Ρι to Pi5) to generate an R frame 711. In the next uncompressed video frame 7〇2, the image block 1 of the uncompressed frame 701 is encoded as an I image. Block 1 and the remaining image blocks 2 and 2 to 15 are encoded as P image blocks (P 〇, and h to U to generate R frame 712. Therefore, the I image block and the P image block for the image block are successively The frames are gradually staggered in time. The process continues until R image block 770 is generated (the last image block in the matrix is encoded as an image block (i.e., the process is then restarted, resulting in, for example, frame 711 (i.e., for image block, Another R frame of the encoded I image block), etc. Although not illustrated in Figure 7a, in one embodiment, the -R frame of the video sequence of the R frame contains only the image block (i.e., such that The p-frame has reference image data (from the beginning of the ten-count motion). Or, in one embodiment, the start-up sequence uses the same image block type as the hanging, but does not include the image that has not been used together with the I image. Blocks the P image blocks of the image blocks of the horses. In other words, the specific image blocks are encoded without any data before the arrival of the first I image 139840.doc -73- 200942305 block, thereby avoiding Figure 9a. The start peaks in the video stream data rate 934 are described in more detail below. Further, as described below, various different sizes and shapes can be used for the image blocks while still complying with these basic principles.

The video decompression logic 412 executing on the client 415 decompresses each image block as if it were a separate video sequence of the small I frame and the p frame, and then reproduces each image block to the map of the drive display device 422. Frame buffer. For example, I〇APA from the frame 711 to 77〇 decompresses and reproduces the image block of the video image. Similarly, use the reconstructed block from R frame 7ΐι to 77〇. As described above, the de-shuffling of the 10-frame and p-frames is well known to the fourth, and the decompression of the If彡 block and the (4) block can be achieved by having multiple instances of the video decompressor in the user. Execution on end 415 is complete. Although the multiplication process seems to increase the burden of user-side computing, it does not increase the burden on the client 415, because the image block itself is proportionally smaller (relative to the extra processing d).

The heart and the number of pixels in the '4' (4) are the same, as if there is _ processing and use the conventional full-size 1 frame and ρ frame. This R-frame technique significantly reduces the brother and the filial usually with the I frame. The associated bandwidth peak 4 (illustrated in Figures 6b and 6c), Α } because any given frame is mainly composed of a frame that is smaller than the I frame. For example, 'assuming another typical I picture I is 1 60 Kb, then the ΒΒ > m in Figure 7a says that the I image of each of the moon frames will be roughly 1/16 or 丨^ υ Kb. It is assumed that the typical p-frame j 16 Kb 'is used for the p-picture of each of the moon-shaped blocks in Fig. 7a to be approximately 1 Kb. The final society wants to be inflammatory.... Kb+bH Kb=25 Kb of R礓139840.doc •74» 200942305 box. Therefore, each 60 frame sequence will be 25 ΚΜ60=1·5 Mbps. So 'at 60 frames per second, this will need A channel capable of maintaining a bandwidth of 丨5 Mbps', but with a much lower peak due to the dispersion of the I-picture block throughout the 6-frame interval.

Note that in the previous example, at the same assumed data rate for both the frame and the p frame, the average data rate is! i Mbp" This is because in the previous example, only a new i frame is introduced every 60 frame time, and in this example, 16 image blocks constituting the I frame are cycled in 16 frame times. And, therefore, an equalization of one frame is introduced every 16 frame times, resulting in a slightly higher average data rate. As such, in practice, the introduction of more frequent I frames does not increase linearly. Data rate. This is due to the fact that the p-frame (or P-picture block) mainly encodes the difference from the previous frame to the next frame. Therefore, if the previous frame is quite similar to the next frame, then the P-picture The box will be very small, Mang I will contact the country to do π, and the right first frame is not the same as the next frame.

In the same way, the frame will be very large.田田& ρ闻4 L 彳—Because the p-frame is largely derived from the previous frame, not from the actual frame, ^ * so the coded frame

There may be more errors than the frame with a sufficient number of 夕A π 疋 数目 ( (eg, visual artifacts). In addition, when the -ρ frame follows the other -ρ frame, error accumulation can occur (when there is a long frame sequence, it gets worse now, the cutting-edge §fl compressor will be said to be stupid 2 σ @ Factory V like the 00 quality in the - sequence ρ frame after the fact of degrading, and if necessary, the complex eight will be assigned to the subsequent ρ frame to mention the South quality, or even! S > In order to take a valid action process, replace the P frame with a frame. Therefore, when using a sequence with a longer P frame (for example, 59 P frames as in the previous example above), ;f special S's scene has a large number of complex 139840.doc -75· 200942305 Hybrid and / or motion 'Generally, more bits are needed for the P frame (because it becomes farther from the I frame). Or ' Looking at the P-frame from the relative view point, the p-frame that closely follows the I-frame tends to require fewer bits than the p-frame farther from the I-frame. Therefore, in the example shown in Figure 7a, The p-free frame is farther apart from the I frame by 15 frames (before the I frame). In the previous example, the p-frame can separate 59 frames from the I frame. Therefore, in the case of more frequent I frames, the p-frame is smaller. Of course, the exact relative size will change based on the nature of the video stream, but in the example of Figure 7a, if the I-image block is 1〇Kb, Then the size of the image block can be an average of only 0.75 kb, resulting in 1〇Kb+15*0.75 Kb=21.25 Kb, or at 60 frames/sec, the data rate will be 21.25 Kb*60 = l.3 Mbps, Or about 16% higher than the data rate of an I-frame followed by 59 p-frames at 1.1 Mbps. Again, the relative result between the two methods for video compression will be video. The sequence changes, but in general, we have empirically found that for a given quality level, using the R frame requires about 20% more bits than using the I/p frame sequence. However, of course, the R frame is drastically reduced. Peak, which makes the gfl sequence available at a much slower delay than the I/p frame sequence. The nature of the video sequence, the reliability of the channel, and the available data rate allow the R frame to be configured in a number of different ways. In the example, a number of image blocks other than 16 are used in the 4χ4 configuration. For example, 2 image blocks are used in 2χ i or 1x2 configurations, 4 image blocks can be used in 2χ2, 4χ1 or 1χ4 configurations, and one image block can be used in 3x2, 2x3, όχΙ or 1x6 configurations or can be used in 4χ2 8 image blocks are used in the 2χ4, 8χ1 or 1χ8 configuration (as shown in Figure 7b). Note that the image block does not need to be a square video image 139840.doc •76- 200942305 The frame does not have to be square, or even rectangular. The image block can be broken down into whatever shape best fits the video stream and application used.另一 ❹ In another embodiment, the loop of the 'IM image block and the P image block is not locked to the number of image blocks. For example, in an 8 image block 4 X 2 configuration, a 丨6 cycle sequence can still be used as illustrated in Figure 7b. The sequential uncompressed frames 721 '722, 723 are each divided into 8 image blocks 〇_7, and each image block is individually compressed. From R frame 731, only the image block is compressed to! The image block, and the remaining image blocks are compressed into P image blocks. For the subsequent R frame 732, all 8 image blocks are compressed into p image blocks, and then for the subsequent R frame 733, the image blocks are compressed into image blocks and the other image blocks are compressed into P image blocks. . In addition, the 16 frames continue to be sorted so that only one image block is generated every other frame, so during the 15th frame time (not shown in FIG. 7b) and during the 16th frame time The last I image block for the shirt image block 7 is generated (using all p image block compression r frames 780). Then, the sequence is again compressed into image blocks by the image block and the other image blocks are compressed into! > Image block begins. As in the previous embodiment, the entire frame of the f sequence will generally be [image block to provide a reference for P blocks that are forward from the other point. 1 The loop of the image block image block does not even need to be an even multiple of the number of image blocks. For example, in the case of 8 image blocks, before using the other-I image block, each frame having the called image block may be followed by two frames of all p-image blocks. In yet another example, right (for example) a particular area of the known screen has more motion: a more frequent image block), while other areas are more static (eg, displaying a number of swims) (requiring a less frequent image block) , and the other image blocks can be sorted together with the i image blocks more often than other image blocks 139840.doc -77- 200942305. Furthermore, although each frame has a single image block as illustrated in Figures 7a-7b However, multiple I image blocks can be encoded in a single frame (depending on the transmission frequency bandwidth). Conversely, a particular frame or frame sequence can have no j image blocks (ie, only p image blocks). Transfer in case. The reason why the 刖-segment method works properly is that although the I-image block that does not have a single frame across the parent seems to cause a large bee value, the behavior of the system is not so simple. Because each image block is compressed separately from other image blocks, the coding of each image block can become less effective when the image block becomes smaller, because the money of a given county block cannot utilize other image blocks. Similar to image features and similar sports. Therefore, dividing the screen into 16 image blocks compared to dividing the screen into 8 image blocks will often result in less efficient coding. However, if the screen is divided into 8 image blocks and it causes every 8 frames (rather than every 16 frames) to introduce the data of the full I frame, it results in a much higher data rate overall. . Thus, by introducing a full I frame every 16 frames (rather than every 8 frames), the total data rate is reduced. Again, by using eight larger shadows: blocks (rather than 16 smaller image blocks), the total data rate is reduced, which also reduces the peak of the data caused by the larger image blocks to some extent. In another embodiment, the low-latency video compression logic 4〇4 in FIG. 7a and FIG. 7b is configured by pre-configuration or based on images in each image block by based on known characteristics of the video sequence to be compressed. The ongoing analysis of quality automatically controls the allocation of bits to each image block in the R frame. For example, in some competitive video games, the player's car (which is relatively unmoved in the scene 139840.doc •78· 200942305) occupies the screen below the screen. Most, and almost two: completely filled with roads, buildings, and landscapes that are approaching, allocating 仏: moving. If the shrink logic 404 will match an equal number of bits, the mother-image block, the image block (image block 4_7) in the uncompressed frame 721 in the main frame under the screen will usually be In contrast to the uncompressed frame 721 in Figure 7b, the image block in the half (image block 1❿, and | shrink. If this particular game or game is known) specific

Ο With these features, the operator of the colocation service 210 can set up the compression logic 404 to distribute the image blocks at the top of the screen (with the short, and the bottom of the screen). Compared to the bit of the image block.) Alternatively, the contraction =_ can estimate the compression quality of the image block after compressing the frame (using one or more of the reduced quality metrics, such as the peak signal noise ratio -R)), and if it is determined at a specific time On the window, a particular image block consistently produces better quality results, which gradually assigns more bits to the image block that produces the low quality result until the various image blocks reach a similar level of quality. In an alternate embodiment, compressor logic 4〇4 assigns bits to achieve higher quality in a particular image block or image block group. For example, it can provide a better overall perceived appearance to have a higher quality at the center of the screen than at the edges. In an embodiment, to improve the resolution of a particular region of the video stream, video compression logic 404 uses a smaller image block to encode a region of the video stream that has relatively more scene complexity and/or motion (with video stream). Relatively small compared to the complexity of the scene and/or the area of motion). For example, as shown in FIG. 8, an area of an R frame 811 (possibly followed by a series of R frames (not shown) having the same image block size 139840.doc -79-200942305 small) Smaller image blocks are used around the moving characters 8〇5. Next, when the character mail moves to a new area of the image, a smaller image block is used around the new area in the other ~_box 812, as illustrated. As described above, various sizes and shapes can be used as "image blocks" while still complying with these basic principles. Although the cyclic I/P image blocks described above substantially reduce peaks in the data rate of the video stream, they do not completely eliminate peaks, especially in rapidly changing or highly complex video images (such as in movies, video games). And under the condition of an application software. For example, during a sudden scene transition, a complex frame may be followed by another complex frame that is completely different. Even if several image blocks can lead the scene transition by only a few frame times, it does not help in this case because the material of the new frame is independent of the previous I image block. In such a situation (and in other situations where a large number of images change, even if not everything changes), the video compressor 4〇4 will determine that many, if not all, P-picture blocks are more efficiently coded as 1 image. The block, and what is caused is the very large peak in the data rate of the frame. As previously discussed, it is only for the condition of most consumer-grade Internet connections (and many office connections) that it can only be "blocked" beyond the maximum data rate available and the maximum data rate shown in Figure 6c as 622. 621 information. Note that the nominal maximum data rate of 62 ι (e.g., "6 Mbps DSL") is essentially a sales figure for a user considering purchasing an internet connection, but typically it does not guarantee a performance level. For the purposes of this application, 'it's not the same' because we only focus on the maximum data rate 622 available when streaming video over a connection. Thus, in Figures 9a and ,, when the solution to the peak problem is described in 139840.doc -80-200942305, the nominal maximum data rate is omitted from the graph and only the maximum data rate 922 is shown. The video streaming data rate must not exceed the maximum available data rate of 922. To solve this problem, the first thing the video compressor 404 does is to determine the peak data rate 941, which is the data rate that the channel can process steadily: this rate can be determined by a multi-technology. One technique is to gradually increase the data rate test stream from the hosted service 21 to the client 415 (in Figures 4a and 4b) and cause the client to address the packet loss and delay. The quasi-feedback is provided to the colocation service. When the packet is lost and the delay begins to show a sharp increase, it is indicative of the maximum data rate 922 available. Thereafter, the colocation service 21 can gradually reduce the data rate of the test stream until the client 415 reports that the test stream has been received in a reasonable period of time (the packet loss and the acceptable level of the delay are close to the minimum) is ^ . This determines the peak maximum data rate 941, which will then be used as the bee data rate for the string & video. Over time, the peak data rate 94i will fluctuate (for example, if the other in the family - make = network connection) - need "two: = rate 941 to see if the packet is lost or the delay is 拎 -, the material rate 922T drops Below the previously determined value; and if so, then the peak data rate 94 is similarly shifted, and the terminal 415 finds that the packet is lost and the delay 曰, then it can request that the attenuator slowly increase the data rate. Whether the large data rate increases (for example, if the family: - sees the most available to the Internet, the connection is severe, so the person has stopped - the person waits until the packet is lost 139840.doc 200942305 Loss and / or higher delay indication has exceeded The maximum data rate 922 is up, and the lower bit _ of the 9-peak data rate 941 can be found again, but the comparison may be higher than the level before the test of the increased f-rate rate. Therefore, by using this technique ( And other techniques similar thereto, the peak data rate 941 is found and periodically adjusted as needed. The bee data rate state is determined by the video compressor 4G4 to stream the video to the user. Maximum data rate. The logic used to determine the peak data rate can be implemented at the user premises 2U and/or at the colocation service 21〇. At the user premises 211, the client device 415 performs calculations to determine peak data. Rate and transmit this information back to the colocation service 21; at the colocation service 210, the cooperating server's server 4〇2 performs calculations based on statistics received from the client 415 (eg, Peak data rate is determined by peak loss, delay, maximum data rate, etc. Figure 9a shows an example video stream data rate 934 'with substantial scene complexity and/or motion using the previously described and in the figures, figures and The loop Ι/P image block compression technique is illustrated in Figure 8. The video compressor 404 is configured to output compressed video at an average data rate lower than the peak data rate 941, and note that most of the time, video The stream data rate ^ is kept below the peak data rate 941. The data rate 934 is the video stream data rate 634 shown in Figure 6c (which is produced using I/P/B or I/p frames). The comparison shows that the cyclic P/P image block compression produces a much smoother data rate, but in the frame 2 times the peak 952 (which is close to 2 times the peak data rate 942) and the frame 4 times the peak 954 (which is close to 4 times the peak) At data rate 944), the f-rate still exceeds the material data rate of 941, which is unacceptable. In practice, even for 139840.doc • 82· 200942305, the peak of the high-speed 941 from fast-changing video games is also A peak of less than 2% of the image rate 942 is rarely seen as a peak. However, motion video, which exceeds the peak data rate box, exceeds 2 times the peak value transition period, and the required resource is low. A good quality video image must be produced. Now and exceeds 3 times the peak data rate 943 ’ when it does appear (for example, in the scene)

One way to solve this problem is to simply configure the video compressor 404 to have its large data rate output as the peak data rate. Unfortunately, the resulting video output quality during the peak frame is poor because the compression algorithm is "extremely lacking". It’s also a sigh of misunderstanding~ Listening to the phenomenon of compression artifacts when there is a sudden change or rapid movement, and in time, the user begins to realize that when there is a sudden change or rapid movement, the fake shadow is always a sudden heart. And it can become quite annoying. Although the human visual system is quite sensitive to visual artifacts that occur during sudden changes or rapid motions, it is very sensitive to the reduction in frame rate in such situations (4). In fact, when these sudden changes occur, it seems that the human visual system is focused on tracking the changes, and if the frame rate temporarily drops from 60 fpS to 3 〇 fps and then immediately returns to 6 〇, then the person The visual system will not notice. In addition, in the case of a very sharp transition (such as a sudden scene change), if the frame rate drops to 2 〇 or even 15 fps and then immediately returns to 6 〇 fps, the human visual system will not notice. As long as the frame rate reduction occurs only occasionally, it seems that the video viewer is constantly performing at 60 fps for the human observer. This feature of the human visual system is utilized by the technique illustrated in Figure 9b ❹ 139840.doc • 83- 200942305 ❹. Server 402 (from Figures 4a and 4b) produces an uncompressed video output stream at a stable frame rate (in the embodiment, at 60 fpS). The timetable shows the output of each frame 961_97 per 1/60 second. Starting from frame 961, the per-uncompressed video frame is output to the low-latency video compressor, and the low-latency video compressor 4〇4 compresses the frame for less than one frame time, resulting in The compressed frame of the first frame 丨 981. The data generated for the compressed frame 981 can be larger or smaller depending on many factors as previously described. If the data is small enough to transmit the peak data rate 941 to the client 415 at frame time (1/6 () seconds) or less than one frame time, then at the transmission time (xmh time) 991 (indication) During the round of the duration of the arrow between the length of the t), it will be passed #. In the next frame time, the server 402 generates an uncompressed frame 2 962, which is compressed into a compressed frame 2 982, and at a peak data rate during a transit time 992 less than the frame time. 941 transmits it to the client.

Next, in the next frame time, the feeder 402 produces a frame 3 963 that is uncompressed. When the compressed image is compressed by the video compressor 4〇4, the resulting reduced frame 3 983 is more than the data that can be transmitted in the frame time than the peak data $ rate 941. data. Therefore, it is transmitted during the transmission time (2 times peak) 993, which occupies all the frames and one part of the next frame time. Now, at the next frame time, the gate servo 402 generates another uncompressed frame 4 964 and will go to the video compressor 4〇4' but the data is ignored and is illustrated by 974 because the video compressor 404 It is configured to ignore other un-attenuated video frames that arrive when they are still transmitting previously compressed frames. Of course, the video decompressor at the client 139840.doc •84· 200942305 415 will not receive Go to block 4, but simply continue to display frame 3 over the display device 422 for 2 frame times (i.e., temporarily reduce the frame rate from 60 fps to 3 〇 fps). For the next frame 5, the server 402 outputs uncompressed frame 5 • 965, compresses it into compressed frame 5 985 and passes it through the frame 1 during transmission time 995. The video decompressor of client 415 decompresses frame 5 and displays it on display device (2). Then, the feeding device 4〇2 outputs the uncompressed frame 6 966 'video compressor 4〇4 to compress it into compressed frame 6 986' but the data obtained at this time is very good at the transmission time (4 times peak) During the 996 period, the compressed frame is transmitted at the peak data rate 941, but it takes almost 4 frame times to transmit the frame. During the next three frame times, the video compressor 404 ignores the three frames from the server 4〇2, and the decompressor of the client 415 stably holds the frame 6 on the display device 422 for a duration. 4 frame times (ie, temporarily reduce the frame rate from 6〇fps by J to 15 fps). Finally, the server 4〇2 outputs a frame, which is compressed by the video compressor 404 into a compressed frame 10 987, which is transmitted during transmission time 997, and decompressor decompressed by the client 415. Frame 10 is displayed on display device 422 and again video resumes at 6〇邙3. Note that although the video compression device 404 discards the video frame from the video stream generated by the feeder 4〇2, it does not discard the audio material (regardless of the form of the audio system), and when discarding During the video frame, the video compressor continues to compress the audio data and transmits it to the client 415. The client 415 continues to decompress the audio data and provides the audio to the user for playback of the sound 139840.doc -85- 200942305 §fl No matter what device. Therefore, during the period in which the frame is discarded, the audio continues without being attenuated. Compressed audio consumes a relatively small percentage of the bandwidth compared to compressed video and therefore does not have a significant impact on the overall data rate. Although not illustrated in any of the data rate maps, there is always a data rate capacity reserved for the compressed audio stream within peak = material rate 941. The example just described in Figure 9b is chosen to illustrate how the frame rate drops during the peak data rate, but the undescribed system uses the previously described

In the case of loop P/P image block technology, the peak of these data rates and the consequent discarded frames are few, even in high scene complexity/high motion sequences (such as in video games, movies, and an application). The same is true during the period in which the software appears. Sub, the reduced frame rate is rare and temporary, and the human visual system does not measure them.

If the frame rate reduction mechanism just described is applied to the video stream data rate as shown in Figure 2, the resulting video stream rate is illustrated in Figure 9c. In this example, the 2x peak 952 has been reduced to a flattened 2x value 953' and the 4x peak 955 has been reduced to a flattened 4x peak 955, 9 the entire video stream data rate 934 remains at or below Peak data rate Therefore, using the technique described above, high motion video streams can be transmitted with low latency via a universal internet and ΪΓ=Γ connection. A connection)::r 'data center and office... The office to the environment can transmit high 139840.doc • 86 - 200942305 motion video stream without peaks, so that multiple users (for example, to 4 5 Mbps transmission of 1920x1 〇8〇 under 60 fPs) can be connected using LAN or shared private data without flooding the network or network switch backplane. • Data Rate Adjustment • : In this embodiment, the colocation service 210 initially evaluates the channel's maximum available > material rate 622 and delay to determine the appropriate data rate for the video stream and then dynamically adjusts the data in response thereto. rate. To adjust the data rate, the colocation service 210 can, for example, modify the image resolution and/or the number of frames per second of the video stream to be sent to the client 415. In addition, the colocation service can adjust the quality level of the compressed video. When changing the resolution of the video stream (e.g., from 1280 720 to 64 〇χ 36 〇), the video decompression logic 412 on the client 415 can scale the image to maintain the same image size on the display screen. In one embodiment, the colocation service 210 suspends the game in the event that the channel is completely exited. In the case of a multiplayer game, the colocation service Ο reports to other users that the user has quit the game and/or paused the game for other users. The discarded or delayed packet is lost in one embodiment if the data is lost due to blunt or packet loss between the video compressor _404 and the client 415 in the servant, or cannot be decompressed due to being too late to arrive. And the packet that satisfies the delay requirement of the decompressed frame • is lost without being received in order, and the video decompression logic 412 can mitigate visual artifacts. In the stream; [/ Ρ frame implementation, if there is a lost/delayed = package, the entire screen is affected, which may cause the glory to be exactly like the calendar J39840.doc •87· 200942305 ❹ a time period or display Other screen wide visual artifacts. For example, if a lost/delayed packet causes the I frame to be lost, the decompressor will lack a reference for all P frames to follow before receiving the new frame. If the P frame is lost, it will affect the p frame that is used for the entire screen. How long before the visual frame appears, this will have a longer or shorter visual impact. Using the interleaved I/P blocks shown in Figures 7a and 7b, the lost/delayed packets are less likely to affect the entire screen because it only affects the image blocks contained in the affected packets. If the data of each image block is sent in an individual packet, if the packet is lost, it only affects the image block. Of course, the duration of the visual artifact will depend! Whether the image block packet is lost or not in the case of the p image block! How many f will be spent before the image block appears = Assume that the different image blocks on the screen are used! The frame is updated very frequently (perhaps every frame), and even if one of the image blocks on the screen is affected, other image blocks may not be affected. In addition, if an event causes several packets to be lost at the same time (for example, a spike in the power of the adjacent cable that temporarily interrupts the lean stream), some image blocks will be subjected to ~θ more than other image blocks but because some image blocks will It is quickly updated by the new I image block' so it is only temporarily affected. In addition, in the case of the implementation of the stream " frame, not only the 1 frame is the most critical frame, but also! The frame is extremely large, so if the event of the discarded/delayed packet is caused, there is a higher probability of having a much smaller image block and then there is a higher probability (i.e., if any part of the I frame is lost, then It is impossible to solve the I-I frame). Due to all such originals

The use of the ', / block results in a much smaller visual artifact than when the _ packet is discarded/delayed compared to the L/P frame. 139840.doc -88* 200942305 - Embodiments attempt to reduce the effect of lost packets by intelligently encapsulating compressed video blocks in TCP (Transmission Control Protocol) packets or UDp (User Datagram Protocol) packets. For example, in one embodiment, as long as • m aligns the scene block with the packet boundary. How wiGa(4) can encapsulate the image block in the -Series packet 1001_1005 with the implementation of this feature. Specifically, in Figure (10), the image block crosses the packet boundary and is inefficiently packaged so that the loss of the single-packet results in the loss of multiple frames. For example, if the packet 1〇〇3 or 1004 is lost, then three blocks are lost, resulting in visual artifacts. In contrast, Figure 10b illustrates image block packaging logic 1010 for intelligently encapsulating image blocks within a packet to reduce the effects of packet loss. First, the image block encapsulation logic 1010 aligns the image block with the packet boundary. Therefore, the image blocks T1, T3, T4, T7, and T2 are aligned with the boundaries of the packets 1001_1005, respectively. Image block encapsulation logic also attempts to combine image blocks into packets in a potentially more efficient manner without crossing the packet boundaries. Combining the image blocks Τ1 and Τ6 in one package 1001 based on the size of each one in the image block; combining Τ3 and Τ5 in one package 1002; combining the image blocks and several in one package 1003; The image block D8 is added to the packet 1〇〇4; and the image environment τ2 is added to the packet 1005. Therefore, under this scheme, the loss of a single packet will result in the loss of no more than 2 image blocks (rather than the three image areas as illustrated in Figure 1a). An additional benefit of the embodiment shown in Figures 1a is that the image blocks are transmitted in a different order in which they are displayed within the image. This method produces less noticeable artifacts on the display if the adjacent packets are lost due to the same event that interferes with the transmission (which would affect areas that are not in close proximity to each other on the screen 139840.doc -89 - 200942305). An embodiment uses forward error correction (FEC) techniques to protect the video stream's knives from channel errors. As is known in the art, FEC techniques such as Reed-Solomon and Viterbi generate error correction information and attach it to the material transmitted via the communication channel. If an error occurs in the basic data (for example, j frame), FEC can be used to correct the error. The FEC code increases the data rate of the transmission, so ideally, it is better to use the FEC code to protect the data only if the 1 material is being sent when it is most needed, and it will not cause very noticeable visual artifacts. . For example, a p-image block immediately before a lost I-picture block will only produce a 1/60 second visual artifact on the screen (ie, the image block on the screen will not be updated). This visual artifact can hardly be detected by the human eye. As the p-image block is further backwards from the I-image block, the loss of the p-image block becomes more and more noticeable. For example, if the image block loop pattern is used again in the image block, the Liyi I image block is followed by 15! > image block, if the P image block immediately after the 1 image block is lost, it causes the image block to display the incorrect image for 15 frame time (at 6〇fps, it will be 25〇 milliseconds) ). The human eye will detect the interruption of the flow of 250 milliseconds. Therefore, the more the p image block is from the new I image block (that is, the closer the P image block follows the image block), the more noticeable the artifact is. As discussed previously, nevertheless, in general, the P-picture block follows the I-view> the closer the block is, the smaller the data for the block of the image. Therefore, the 1> image block following the I image block is not only more critical for protection from loss 139840.doc -90- 200942305, but also its size. In addition, in general, the smaller the information that needs to be protected, the smaller the FEC code needed to protect it. Thus, as illustrated in Figure 11a, in one embodiment, only the image block has an FEC code due to the importance of the j image block in the video stream. Therefore, fec 1101 contains an error correction code for I picture block 1100 and 1104 contains an error correction code for I picture block 1103. In this embodiment, no FEC is generated for the p-picture block.

In one embodiment illustrated in Figure Ub, FECU^ is also generated for p-image blocks that are most likely to cause visual artifacts when lost. In this embodiment, FEC 1105 provides error correction codes for the first 3 p-image blocks but not for the following p-picture blocks. In another embodiment, an FEC code is generated for the p-image block with the smallest data size (which would tend to be the P-block that was originally generated after the Z-image block, which is most critical for protection). In another embodiment, instead of transmitting the FEC code along with the image block, the image block is transmitted twice twice each time in a different packet. If one packet is lost/delayed, another packet is used. In one embodiment shown in Figure Uc, audio packets 1111 and (1) 3, respectively, for audio packets (1) transmitted from the colocation service simultaneously with the video are generated. Maintaining the integrity of the audio in the video stream is particularly important because distorted audio (e.g., 'clicks or clicks') will result in a special direct experience code that helps ensure that the audio content is reproduced without distortion at the client computer 415. In the other seven embodiment, the FEC code is not transmitted together with the audio: (4), and the audio data is transmitted twice, each time in a different packet. 139840.doc -91 · 200942305 If a packet is lost/delayed, another packet is used. In addition, in one embodiment illustrated in Figure lid, the 'keys 1121 and 1123' are used for user input commands (e.g., button presses) 112 and 1122 for uplink transmission from the client 415 to the colocation service, respectively. This is important because missing button presses or mouse movements in a video game or application can result in an undesirable user experience. In another embodiment, rather than transmitting the FEC code along with the user input command data, the user input command data is transmitted twice, each time in a different packet. If a packet is lost/delayed, an additional packet is used. In one embodiment, the colocation service 210 evaluates the quality of the communication channel with the client 415 to determine whether to use it! ^]^, and if used, determines which part of the video, audio, and user commands should be applied for FEC. The "quality" of the evaluation channel may include functions such as estimating packet loss, delay, etc. as described above. If the channel is particularly unreliable, the colocation service 21 can apply fec to all I video blocks, p video blocks, audio and user commands. In contrast, if the channel is reliable, the colocation service 21 can apply FEC' only to audio and user commands or fec to audio or video, or FEC may not be used at all. Various other arrangements of FEC applications can be used while still adhering to these basic principles. In an embodiment, the colocation service 210 constantly monitors the status of the channel and changes the FEc policy accordingly. In another embodiment, referring to FIG. 4a and FIG. 4b, when the packet is lost/delayed, resulting in loss of image block data, or if the FEC is unable to correct the lost image block data due to the particularly bad packet loss, The 139840.doc -92- 200942305 server 415 evaluates how many frames remain before the new image block will be received and compares it to the round trip delay from the client 415 to the colocation service 21 . If the round trip delay is less than the number of frames before the new image block should arrive, the client 415 sends a message to the colocation service 21 to request a new I block. This message is routed to the video compressor 4〇4, and instead of generating a p-image block for the image block in which the data has been lost, a work image block is generated. Assuming that the system shown in Figures 4a and 4b is designed to provide a material that is typically less than a leap second (four), this causes the material (10) to be corrected within 8 milliseconds (at 60fPs, the frame has a duration of 16.67 milliseconds). Therefore, the 80mm delay in the full frame time will result in a corrected image block within 83 to 33 milliseconds, and 83.33 milliseconds for 5 frame times, which is an attractive interruption, but far less than (for example) for 15 A frame of 25 〇 milliseconds interrupts attention). When the compressor 404 is out of its normal cyclic order to generate such a work image block, if the I image block will cause the bandwidth of the frame to exceed the available bandwidth, the compressor 404 will delay the cycle of the other image blocks, so that The other image blocks receive the P image block during the frame time (even if the image block will normally be an I image block during the frame), and then the usual loop will continue from the next frame' and will usually The image block that received the (3) image block in the previous frame will receive the I image block. Although this action temporarily delays the order of the 11 frame loops, it will usually be visually unobtrusive. Video and Audio Compressor/De-Finishing Apparatus Implementation Figure 12 illustrates a particular embodiment in which a multi-core and/or multi-processor 1200 is used to compress 8-bit blocks in parallel. The -invention towel uses a dual-core processor running at 2.66 GHz or higher, a four-computer 139840.doc-93-200942305 system, and each core implements an open source x264 H.264 compressor as a separate process. However, a variety of other hardware/software configurations can be used while still adhering to these basic principles. For example, each of the CPU cores can be replaced by an H.264 compressor implemented in an FPGA. In the example shown in Figure 12, cores 1201-1208 are used to process I and P blocks simultaneously as eight separate threads. As is well known in the art, current multi-core and multi-processor computer systems are inherently integrated with multi-threaded processing operating systems such as Microsoft Windows XP Professional (64-bit or 32-bit) and Linux. Ability to perform multi-thread processing. In the embodiment illustrated in Figure 12, since each of the eight cores is only responsible for one image block, it operates largely independently of the other cores, each performing a separate instantiation of x264. Use a PCI Express xl-based DVI capture card (such as the Sendero Video Imaging IP Development Board from Microtronix of Oosterhout in the Netherlands) to capture uncompressed video at 640x480, 800x600 or 1280x720 resolution, and the FPGA on the card Direct memory access (DMA) is used to transfer the captured video to the system RAM via the DVI bus. The image blocks are configured in a 4x2 configuration 1205 (although this is illustrated as a square image block, in this embodiment it has a 160x240 resolution). Each instantiation of x264 is configured to compress one of the eight 160x240 image blocks and is synchronized such that each core enters a loop after initial I image block compression, each frame and another The frames are out of phase to compress an I image block followed by seven P image blocks, as illustrated in FIG. At each frame time, the resulting compressed 139840.doc -94 - 200942305 thumbnail blocks are combined into a packet stream using the techniques previously described, and the compressed image blocks are then transmitted to the destination client 41 5 . Although not illustrated in Figure 12, if the data rate of the combined 8 image blocks exceeds the private peak data rate 941, all 8 χ264 processes will temporarily wait for the necessary frame time until the transmission is completed. Up to the data of the combined 8 image blocks. In one embodiment, the client 415 is implemented as a software executing the 8 _ instantiated pcs of FFmpeg. The receiving process receives eight image blocks and routes each image block to FFmpeg for instantiation, and FFmpeg instantiates the decompressed image block and reproduces it to the appropriate image block location on display device 422. The client 415 receives the keyboard, mouse, or game controller input from the input device driver of the PC and transmits it to the server 402. The server 402 is connected to the input device data received by the application and applied to the game or application executed on the server 402. The server 4〇2 is a PC that executes Windows using the Intel 2 16 GHz dual-core CPU. Server 402 then generates a new frame φ box and via its DVI output, the new frame is output from the motherboard-based graphics system or via the DVI output of the NVIDIA 8800GTX PCI Express card. At the same time, the server 402 outputs audio generated by the game or application via its digital audio output (eg, S/PDIF), which is coupled to a dual quad-core Xeon-based PC that implements video compression. Digital audio input. Vorbi's open source audio compressor is used to compress audio simultaneously with video, regardless of the core that can be used to process the thread. In the example, the core of the compressed image block is first compressed to perform the audio pressure 139840.doc -95- 200942305. The compressed audio is then transmitted along with the compressed video and the compressed audio is decompressed at the client 415 using a Vorbis audio decompressor. The colocation service server center distributes light traveling through the glass (such as an optical fiber) at a fraction of the speed of the light in the vacuum, and thus the exact propagation speed of the light in the fiber can be determined. However, in practice, considering the routing delay, transmission inefficiency, and other elapsed time, we have observed that the optimal delay on the Internet reflects a transmission speed that is closer to 50% of the speed of light. Therefore, the optimal 1 mile round trip delay is about 22 milliseconds, and the optimal 3000 mile round trip delay is about milliseconds. Therefore, a single server on the US coast will be too far apart to serve the client on the other coast (which may be up to 3 miles away) with the desired delay. However, as illustrated in Figure 13a, if the colocation service 21/server center 1300 is located at the center of the United States (e.g., Kansas, Nebraska, etc.), the distance to any point in the continental United States is about 15 miles. Or below 1500 miles, the round-trip internet latency can be as low as 32 milliseconds. Referring to Figure 4b, note that although the worst case delay allowed by the user ISP 453 is 25 milliseconds, in general, in the case of DSL and cable modem systems, we observe a delay of approximately 10-15 milliseconds. Further, Fig. 4b assumes that the maximum distance from the user premises 211 to the colocation center 210 is 1 mile. Thus, in the case of a typical 15 millisecond user ISP round trip delay used and a maximum internet distance of 1500 miles for a 32 millisecond round trip delay, the time from when the user actuates the input device 42 1 The total round trip delay for seeing a response on display device 422 is 139840.doc • 96· 200942305 1 + 1 + 15+32+1 + 16+6 + 8 = 80 milliseconds. As a result, 80 millisecond response time is typically achieved over an Internet distance of 〇〇5〇〇 miles. This will allow any user location in the US mainland with a sufficiently short user ISP delay of 453 to be accessed in a centrally located single server center.

In another embodiment illustrated in Figure 13b, the colocation service 2i〇feeder center HS1-HS6 is strategically located around the United States (or other geographic area), a larger hosted service server The center is located close to the high population center (for example, HS2 and HS5). In one embodiment, server hubs hS1-HS6 exchange information via network 13()1, which may be an internet or private network or a combination of both. In the case of multiple server centers, it is possible to provide a service to a user with a high user Isp delay 453 at a lower delay. The distance on the network is indeed affected by the round trip delay through the Internet. Factors, but sometimes the two factors that are largely unrelated to the delay also work. Sometimes the packet flow is routed via the Internet to the distance © 遢 and returned again, resulting in a delay from the long loop. Sometimes there are routing devices that are not properly operated on the road technology, resulting in delays in transmission. Sometimes there are traffic overloading the path, which leads to delays. In addition, there is a fundamental reason for preventing the user's Isp from routing to a given destination. Therefore, although the universal Internet usually provides a connection from one point to another with fairly reliable and optimal routing and latency', this fairly reliable and optimal route and delay is largely determined by distance (especially It is in the case of a long-distance connection that leads to the outside of the user's local area.) The guilty and delay is not guaranteed and often cannot be used 139840.doc •97· 200942305 User's place to general Achieved by a given destination on the Internet. In one embodiment, when the user client 415 is initially connected to the colocation service 210 to play a video game or use an application, the client initiates and each of the available colocation service server centers HS1_HS6 at startup. Communication (eg, using the techniques described above). If the delay is low enough for a particular connection, use the connection. In one embodiment, the user =. communicates with a subset of all of the hosted service server centers or the hosted service server centers, selecting the hosted service server center with the lowest latency connection. The client can select the service with the lowest latency connection, or the server can identify the server center with the lowest latency connection and provide this information (for example, in the form of an Internet address) to the client. , = π 哎 哎 哎 / / / 3⁄4 user tour right specific host escrow service service money program can tolerate to another ... less manned host escrow service server pirate center delay, then the user terminal 415 Redirect to the other-hosted service server hub. In this case, the game or application being executed by the user will be paused on the server 402 at the user's overloaded server center and the game or application status data will be transferred to the other server. The server at the server center is 4〇2. The game or application will then be replayed. In a 眚 丨 丨 ° ° Λ 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在 在The transfer is only performed after the "Save" operation. In yet another implementation -

It will be reserved until the user's activity is stopped for a specified period of time (eg, W 139840.doc • 98· 200942305) and then the transmission will begin at that time. In the embodiment described above, the colocation service (4) subscribes to the Internet bypass service 440 in an attempt to provide a guaranteed delay to its client. The Internet bypass service system, which is used in this article, provides a guaranteed feature (such as 'delay, data rate, etc.') from one point on the Internet to another point. Use the Internet route service. For example, if the colocation service 210 is using 纟"Μη*. The provided squad service receives a large amount of traffic from the user (rather than routing to the San FranC1SC as the base central office), The colocation service (10) will be rented between the central office based in SanFrancisc〇 and one or more of the feeding center 10 for the colocation service 210 (4) the service provider (possibly (10) itself or another - Provider's high-capacity private data connection. Then, Mang's from all the hosted service servers in the heartbeat via the Internet to the (10) mountain ^ user's road material caused a high delay, Instead of using private resources #连,° although private data connections are usually more expensive than routes via the universal Internet, as long as they maintain a small percentage of the colocation service 21% connected to the user, the total cost impact It's low, and users will experience a more consistent service experience. 纟In the case of power failures, the 'feeder center often has two backup power layers. The first layer is usually from the battery (or from the replacement) Ready-to-use • & the source of backup power such as the flywheel that keeps running and attached to the generator, which provides power immediately when the mains fails and keeps the server center running. If the power (four) is riding, And the power dry material speed returns (eg 139840.doc -99· 200942305 such as 'in one minute), then the battery is kept listening to the center of the server. But the right power failure lasts longer. , usually the lightning machine (for example, diesel powered) is activated to replace the battery and the power can be operated. These generators are extremely expensive, because the eight-fuel mussels must be able to generate how far the server center usually gets from the mains. In the second example, the colocation service, each of which uses the user's poor material, in order to have a power failure in a server center, it can suspend the ongoing games and applications, and The game or application status data is then transmitted from each server 4 〇 2 to the feeders 402' at the center of the other feeders and then the user 415 of each user is notified Instruct it to communicate to the new server 4Q2. If the situation occurs occasionally, then the user will be transferred to the host server service server that cannot provide the best delay (ie, the user will only have to tolerate higher delays). The duration of the power failure may be acceptable, which would allow for a much wider range of options for transferring the user. For example, given the time difference across the United States, the user on the East Coast is at n : 3QPM may be going to sleep, and users on the West Coast are starting to peak in video games at 8:3 〇pM. If there is a power failure in the hosted service server center on the West Coast, then other hosts will There may not be enough West Coast servers 4〇2 for handling all users at the server service server center. In such a case, some users may be transferred to a colocation service server center with an available server 402 on the East Coast, and the only consequence for the user will be a higher latency. Once the user has transferred from the center of the feeder that lost power, the server center can then begin its server and set the 139840.doc •100- 200942305 for an orderly cut-off so that the battery (or other immediate power reserve) Turn off all equipment before exhausting. In this way, the cost of the motor for the dragon's heart can be avoided. In an embodiment, during the time of severe loading of the colocation service 21 (or due to peak user loading, or because one or more servant centers fail) based on the game being used by the user or The application's delay requires the user to be transferred to another center. Because & will be

Users of games or applications with low latency give preference to available low latency server connections with limited supply. Host Hosting Service Feature Figure 15 shows an embodiment of the component in the server for the host hosting service used in the following special government description, and the host hosting service 210 illustrated in the figure. The components of this server center are controlled and coordinated by the host hosting service 210 control system 401 unless otherwise conditional. The incoming Internet traffic 15〇1 from the user client 415 is directed to the incoming bee route 1502. Usually, the Internet traffic 15〇1 will enter the server center via the high-speed fiber connection to the Internet, but any network connection component with sufficient bandwidth, sin and low latency will be sufficient. . Inbound routing 1502 is a network (which can be implemented as an Ethernet, fiber channel network, or via any other transport component) switch and a system that supports the routing servers of the switches, which arrives The packets are packetized and routed to the appropriate application/game server 1521·ΐ525. In one embodiment, the packet delivered to the particular application/game server represents a subset of the data received by the client 139840.doc 200942305 and/or may be other components within the data center (eg, a network connection component) , such as gateways and routers, to translate/change. In some cases, the packet is routed to more than one serving benefit 1521-1525 each time, for example, if the game or application is executing concurrently on multiple servers. The RAID array 15U_1512 is coupled to the incoming routing network 1502 such that the application/game server 1521-1525 can read the RAID array 15; the Π-1512 and the write RAm array 1511_15i2. In addition, a raid array 1515 (which can be implemented as a plurality of shakuhachi 11} arrays) is also coupled to the inbound routing 1502, and data from the RAID array 1515 can be read from the application/game servers 1521-1525. Incoming routing 15〇2 can be implemented in a variety of prior art 'circumferential architectures (including tree-structured switches, inbound Internet traffic 1 1 at its root); in a mesh structure interconnecting all of the various devices A solid yoke, or as a sub-network sequence of interconnects (concentrated traffic among interworking devices and centralized traffic isolation among other devices). One type of network, and the heart is S AN, which is commonly used for storage devices, but it can also be used for general high-speed data transfer between devices. Also, the application/game servers 1521-1525 can each have multiple network connections to the incoming route 15〇2. For example, servers 1521-1525 may have a network connection to a subnet attached to arrays 1511-1512 and another network connection to a subnetwork attached to other devices. The application/game servers 1521-1525 can be configured identically, somewhat differently, or all differently, as previously described with respect to the server 402 in the embodiment illustrated in the drawings. In one embodiment, each user typically uses at least one application/game servo when using the colocation service 139840.doc • 102- 200942305

1521-1525. For the sake of simplicity of the description, it will be assumed that a given user is using the application/game server 1521, but multiple servers can be used by one user, and multiple users can share a single application/game server 1521 -1525. The user's control input sent from the client 415 (as previously described) is received as an incoming Internet service 15 5 and routed to the application/game server 1521 via the incoming route 1502. The application/game server 1521 uses the user's control input as a control input to the game or application executing on the server and calculates the video and the next frame of the audio associated therewith. The application/game server 1521 then outputs the uncompressed video/audio 1529 to the shared video compression 1530. The application/game server can output uncompressed video via any component (including one or more ultra-high speed Ethernet connections), but in an embodiment, the video is output via a DVI connection, and the audio and Other compression and communication channel status information is output via a universal serial bus (U s B ) connection. The eight-video compression i530 compresses the uncompressed video and audio from the program/game pad i^i_ 1525. This compression can be implemented entirely in hardware or in hardware that executes the software. There may be a special dust reducer for the parent application/game server 1521-1525 with _, j, or if the compressor is fast enough, the (four) reducer can be used to compress the application from more than one application. Platinum 4 of the 1521-1525 is viewed as / a矾. For example, when the 6〇 plus frame time is 16 67奁#, the right compressor can compress 1 frame within 1 millisecond, then the compressor can be used for Yishan % ^ ^ to obtain the servo from one connection. The thief's input is compressed by up to 16 applications/game servers 139840.doc -103- 200942305 1521-1525 video/audio, which stores the state of each video/audio compression process and when Switch background when looping through the video/audio stream from the server. This results in substantial cost savings for the collapsed hardware. Because different servers will complete the frame at different times' so in one embodiment, the compressor resources are in a shared storage component (eg, RAM, flash memory) with a state for storing each compression process. In the shared pool 153, and when the server 1521-1525 is complete and ready to be compressed, the control component determines which compressed resource is available at the time, provides the state of the compression process of the server and the uncompressed Compressed video/audio frame. Note that one of the states of the compression process for each server includes information about the compression itself, such as the decompressed frame buffer data of the previous frame (which can be used as a reference for the p image block), Resolution of video output; quality of compression; image block structure; allocation of bits per image block; compression quality, audio format (eg stereo, surround sound, Dolby® AC-3). However, the compression process state also includes information about the following communication channel status: peak data rate 941, and whether the previous frame (as illustrated in the figure) is currently being output (and therefore the current frame should be ignored), and potentially There are channel characteristics that should be considered in compression (such as excessive packet loss) that affect compression decisions (eg, in terms of the frequency of I-picture blocks, etc.). Since the peak data rate 941 or other channel characteristics change over time 'as determined by the application/game server 1521-1525 that supports each user monitoring the data sent from the client 415, the application/game The server 1521-1525 sends the relevant information to the shared hardware compression 139840.doc • 104· 200942305 1530. The shared hardware compression 1530 also encapsulates the compressed video/audio using components such as those previously described, and, where appropriate, applies a fec code, copies specific material, or takes other steps to adequately ensure video. The audio data stream is received by the client 4丨5 and is decompressed with a viable high quality and reliability. Some applications (such as those described below) need to be given the video/audio output of (4)/game ship n1521_1525 (4) at multiple resolutions (or in multiple formats). If the application/game server 1521-1525 thus informs the shared hardware to compress 153 resources, the uncompressed video of the application/game server 1521_1525

❷ 1529 will be compressed simultaneously in different formats, different resolutions, and/or in different packet/error configurations. In some cases, some of the compressed resources may be shared among multiple compression processes that compress the same video/audio (eg, in many S-shoulder algorithms) to have the image scaled (four) to multiple sizes before applying compression. Step. If you need to output images of different sizes, this step can be used to simultaneously servo several compression processes). In other cases, separate compression resources will be required for each format. In any case, all of the various resolutions and formats of compressed video/audio (10) for a given application/game server ΐ52ι_ΐ525 (one or more) are simultaneously output to the outbound routing. In the embodiment, the compressed video/audio 1539 is transmitted (four), in the qing format, so it is a unidirectional stream 0 outbound routing network 154 〇 comprising a series of routing feeders and switches 139840.doc - 105- 200942305 Series Routing Servers and Switches direct each shrinking video/audio stream to the desired network via the Internet Protocol 1599 interface, which is typically connected to the Internet's fiber interface Or other destinations and/or return to the delay buffer 1515, and/or return to the incoming route 15〇2, and/or output via a private network (not shown) for video distribution. Note (as described below): Outbound Routing 1540 can simultaneously output a given video/audio stream to a destination. In one embodiment, this is implemented using Internet Protocol (IP) multicast, where the broadcast is intended to simultaneously stream to a given WDP capture of multiple destinations, and the broadcast is routed by the outgoing route 154 The server and the switch are repeated. Multiple destinations of the broadcast may be via the Internet to the user terminal 415 of the plurality of users, to the plurality of application/game servos IU52M525 via the incoming route 丨5 〇2, and/or to - or multiple Delay buffer 1515. Thus, the output of a given server 1521_1522 is compressed into one or more formats and each compressed stream is directed to one or more destinations. Additionally, in another embodiment, if multiple application/game servers 1521-1525 are simultaneously used by a user (eg, in a parallel processing configuration for generating 3D output with complex scenes) and each If the server generates a part of the obtained image, the video output of the plurality of servers 1521-1525 can be combined into a combined frame by the shared hardware compression 153, and the combined frame is processed as described above from the other side, as if It comes from a single application/game server 1521-1525. Note that in one embodiment, a copy of all of the video generated by the application/game server 1521_ 1525 (at least at the resolution of the video viewed by the user or higher resolution) is recorded in the delay buffer 1515 for a duration of 139840.doc •106- 200942305 Less than a certain number of minutes (i 5 minutes in one embodiment). This allows each user to "rewind" the video from each session to check previous work or performance (in the case of the game). Thus, in one embodiment, each compressed video/audio output 1539 stream routed to user client 415 is also multicast to delay buffer 1515. When the video/audio is stored on the delay buffer 1515, the directory on the delay buffer 1515 provides the network address and delay buffer 1515 of the application/game server (which is the source of the delayed video/audio). A reference between the location of the delayed video/audio can be found. The live, instantly viewable, instantly playable game application/game server 1521-1525 can be used not only to execute a given application or video game of the user, but also to establish support for hosting via the host The user interface application of the hosted service 210 of the navigation and other features of the service 210. A screen shot of the user interface application is shown in Figure 16 ("Game Viewfinder" screen. This particular user interface screen allows the user to view 15 games played (or delayed) by other users. Each of the thumbnails video windows (such as 1600) is a live video window in motion that displays a video from a user's game. The view shown in the thumbnail can be positive for the user. Look at the same view, or it can be a delayed view (for example, if the user is playing a wrestling game, the user may not want other users to see it. It is hidden where and it can choose to delay any view of its game play A time period (eg, 10 minutes). The view can also be a camera view of the game that is different from the view of any user. The user can select via the 139840.doc -107· 200942305 menu based on multiple criteria (not in this description) As shown in the figure), the selection of the game to be simultaneously viewed is selected. As a small sample of the exemplary selection, the user can select a random selection of the game (such as FIG. 16). The games shown in the game), all categories of games (both played by different players), only the top players of the game, players of a given level in the game, or lower level players (for example, if the player is learning Basic), a player who is a "partner" (or a competitor), a game with the largest number of viewers, etc. Note that 'generally, each user will decide whether the video from their game or application can be viewed by others, and If so, it is determined by which other people the video can be viewed and when it can be viewed by others, and whether the video can only be viewed with a delay. The application/game server 1521 that produces the user interface screen shown in FIG. -1525 obtains the 15 videos by sending a message to the application/game server 1521-1525 of each user (the application/game server 1 521-1 525 is requesting a game from the user) The audio feed is sent via the incoming route 1502 or another network. The message will include the size and format of the requested video/audio and will be identified. User of the user interface screen. A given user may choose to select the "Piracy" mode and not allow any other user to view the video/audio of their game (from their view point or from another view point), or as in the previous paragraph As described in the above, the user can choose to allow viewing of video/audio from their game, but delay the video/audio being viewed. User application/game server Μ] 1525 (its receiving and receiving text allows their video/ The request for the audio to be viewed will therefore be acknowledged to the requesting server, and it will also notify the shared hardware compression. 53) J39840.doc -108- 200942305 is required to generate the requested format or screen size (assuming the format and screen size are different) The additional compressed video stream of the format and screen size that has been generated, and which also shows 4a as the destination of the compressed view (ie, the request server). If the requested video/audio is only delayed, the application/game server is requested. 1521 1525 will be notified as such, and it will look up the location of the video/audio in the directory on the delay buffer 1515 and The delay buffer 15 15 acquires delayed video/audio for the network address of the application/game server 1521_1525 from which the video/audio source is delayed. Once all such requests have been generated and processed, up to 15 live broadcast thumbnail-sized video streams are routed from the outgoing route 1540 to the incoming route 15〇2, to the application/game server that generates the user interface screen. The 1521_1525 will be decompressed and displayed by the server. The delayed video/audio stream may be at an excessive screen size, and if so, the application/game server 1521_ 15 5 will decompress the streams and scale the video stream down to the thumbnail size. In one embodiment, the request for audio/visual is sent to a central "management" service (not shown in Figure 15) similar to the hosted service control system of Figure (and managed by a central "management" service The central "management" service then redirects the requests to the appropriate application/game server 1521-1525. Moreover, in an embodiment, a request may not be required because the thumbnails are "pushed" to the user of the user who is allowed to. All of the simultaneous mixing of audio from 15 games can produce harsh sounds. The user can choose to mix all the sounds together in this way (perhaps to get the "beep" feeling from all the actions being viewed), or the user can choose to listen to the audio from one game at a time. I39840.doc -109- 200942305

The choice of play is done by moving the yellow selection box 1 to a given game (the yellow frame can be moved by using the arrow keys on the keyboard, by moving the mouse, by moving the joystick or by pushing The direction button on the other device of the mobile phone to complete P. Once a single game is selected, only the audio game of the game is played. Also, the game information 1602 is displayed. In the case of the game, for example, the trademarks are published ("EA And the game logo "Need for Speed Card Canyon" and the color strips indicate the number of people who play games or view games at a particular time under relative conditions (in this case, many, therefore: the play is "hot") In addition, a "status" is provided indicating that there are 80 different instantiations in which 145 players are actively playing Need for Speed games (ie, the game can be played by individual player games or multiplayer games), and there are 680 Viewer (this user is one of them). Note that these statistics (and other statistics) are collected by the hosted service control system 401 and stored in the RAID array 1511·15. 12, in order to maintain the courier service 21〇 operation and to properly charge the user and pay the publisher who provides the content. Some statistics are due to the service control system 4〇ι The H statistics are reported by the individual application/game server 1521-1525 to the service control system 4G1. For example, when the rhyme is being viewed (and when the game is stopped viewing), the game is executed. The application/game servo II 1521-1525 of the viewfinder application sends a message to the host escrow service control (4) to enable the host escrow service control system 4〇1 to update how many games are in the view of the statistics. These statistics can be used by user interface applications such as this game viewfinder application. 139840.doc • U0· 200942305 If the user clicks the start button on their input device, they will see the yellow box Thumbnail video zooming while thumbnail video is live broadcast to full screen size. This effect is shown in the process in Figure 17. Note that the window is 1700. In order to implement this effect, the application/game server I521·1525 requests the full screen size of the game routed from the application/game server 1521-1525 executing the selected game (by the user) A copy of the video stream of the resolution of the display device 422. The game/game server 1521-1525 is notified to notify the shared hardware compressor 153 that a copy of the thumbnail size of the game is no longer needed (unless another application The program/game server 1521-1525 requires such a thumbnail), and then it directs the hard memory compressor 1 5 3 0 to send a copy of the full screen size of the video to the magnified application/game server 1521-1525. The user playing the game may or may not have a display device 422 having the same resolution as the display device of the user who zooms in on the game. In addition, other viewers of the game may or may not have display device 422 that has the same resolution as the user who zoomed in on the game (and may have different audio playback components, such as stereo or surround sound). Therefore, the shared hardware compressor 153 determines whether a suitable compressed video/audio stream that satisfies the requirements of the user requesting the video/audio stream has been generated, and if the appropriate compressed video/audio stream does exist, the shared hard The volume compressor 1530 notifies the routing route 154 to route the replica of the stream to the application/game server 1521_1525 that amplifies the video, and if the appropriate compressed video/audio stream does not exist, the compressed video is suitable for Another copy of the user and directs the routing to send the stream back to the incoming route 1502 and the application/game server 1521·1525 that amplifies the video. I39840.doc -111 - 200942305 This server that now receives the full screen version of the selected video will decompress the full screen version and gradually scale it up to full size. Figure 18 illustrates how the screen looks after the game is fully enlarged to full screen and the game is displayed at full resolution of the user's display device 422 (as indicated by the image pointed by arrow 1 800). The application/game server 1521_1525 executing the game viewfinder application sends a message to the other application/game server 1521_1525 that provides the thumbnails to indicate that the thumbnails are no longer needed and host the service control server 4 to the host. 1 Send a message to indicate that no other games will be viewed. At this point, the only display that appears is the overlay 1801 of the top of the screen, which provides information and menu controls to the user. Note that as the game progressed, the audience grew to 2,503 viewers. In the case of so many viewers, there must be a number of viewers with display devices 422 that have the same or close resolution (each application/game server 1521_1525 has a proportionally adjusted video to · Ability to adjust fit). Since the game shown is a multiplayer game, the user can decide to join the game at some point. Host hosting services 21 or © may not allow users to join the game for a variety of reasons. For example, the user may have to pay for the game and choose not to pay, the user may not have enough levels to join the particular game (eg, for other players, it will not be competitive), or use The Internet connection may not have enough delays to allow the user to play (for example, there is no delay constraint for viewing the game, so the view can be viewed remotely without delay attention (actually 'Playing on another continent', but for games played 139840.doc • 112· 200942305, the delay must be low enough for the user to (a) enjoy the game and (b) be and possibly Other players with lower latency connections are equal). If the user is not allowed to play, the user is provided with a game viewfinder, and the user interface application/game server 1521-1525 will request the host • the representative service control server 401 to start (ie, locate and start) via The application/game server 152i 1525, which is suitably configured to play a particular game, loads the game from the RAID array 1511_15丨2, and then the host participates in the service control server 401 to guide the incoming route 1 5〇2 will transfer the user's control number 5 to the current hosted game application/game server and now the hosted game application/game server will guide the shared hardware compression 1530 self-compression From the hosted game viewfinder application. The video/audio of the application/game server is switched to the video/audio from the application/game server of the host endorsement game. The vertical synchronization of the game viewfinder application/game service with the new application/game server of the colocation game is not synchronized, and thus there may be a time lag between the two Φ syncs. Since the shared video compression hardware will start compressing the video after the application/game server 1521_1525 completes the video frame, the first frame from the new server can be completed earlier than the full frame time of the old server. The first frame from the new server may be before the transmission is completed in the previously compressed frame (eg 'considering the transmission time outside the picture 992: if the uncompressed frame 3 963 is completed earlier than the frame time - Half, then it will impact transmission time 992). In this case, the shared video compression hardware 1530 will ignore the first frame from the new server ' (eg, as ignored (974) frame 4 964), and the client 415 will come from the old servo J39840.doc - The last frame of the 113-200942305 device remains - extra frame time, and the shared video compression hardware 1530 will begin compressing the next frame time video from the new application/games feeder of the colocation game. For the user, the transition from the application-game/game word service H to the other-application/game server will be seamless. The colocation service control server then switches the application/game server i52i_i525 hosting the game viewfinder to the idle state until it is needed again. The user can then play the axis. In addition, the exceptional game will be played instantly on the sensation (because the game has been loaded from the Raid array to the app/game server 1521_1525 at a rate of one billion bits per second) and will be driven by the ideal drive. The scratchpad configuration (in the case of wind〇ws2) loads the game along with the operating system configured for the game to the server that is exactly suitable for the game, and is not possible with the game Other applications that compete for content are executed on this server. Also, as the user progresses in the game, each of the segments of the game will load the servo from the RAID array bh-bu at a rate of one billion bits per second (ie, the next billionth byte loaded). And because of the huge storage capacity of the raid arrays 1511-512 (because it is a shared resource for many users, it can be very large, but still cost effective), allowing pre-computation of geometry settings or other game segment settings It is stored on the RAID array 1511_1512 and loaded very quickly. In addition, since the hardware configuration and computational capabilities of each application/game server 1521·1525 are known, pixel and vertex shading can be pre-computed. As a result, the game can be launched almost instantly, it will be executed in an ideal environment, 139840.doc -114- 200942305 and the subsequent clips will be loaded almost instantly. In addition to these advantages, the user will be able to view other people playing the game (via the previously described game viewfinder, and other components), and both determine whether the game is interesting, and if so, then watch others And study skills. In addition, the user will be able to present the game instantly without having to wait for a large download and/or installation, and the user will be able to play the game instantly (perhaps on a less expensive trial basis or on a longer term basis) . In addition, the user is able to play the game on the PC, on the television, at home, while traveling, and even on the mobile phone by a wireless connection of sufficiently low latency. In addition, this can be done without having a copy of the game.

As previously stated, the user may decide not to allow their game play to be viewable by others, allow their game to be viewed after a delay, allow their game to be viewed by a selected user, or allow their game to be viewed by all users. How, in one embodiment, the video/audio is stored in the delay buffer 1515 for 15 minutes, and the user will be able to "rewind" and view his previous game play, and pause the game to slowly play back the game. Fast forward the game, etc., as it can be seen while watching τν with a digital video recorder (DVR). Although the user is playing the game in this example, the same "DVR" capability is available if the user is using the application. This may be useful in verifying previous work and in other applications as detailed below. In addition, if the game is designed to have the ability to fall back based on the game state information so that the camera view can be changed, the "3D DVR" capability will also be supported, but it will need to design the game to support "3D 139840. Doc -115- 200942305 DVI Capabilities. Using the Delay Buffer (5) The ability to "dvr" will work with any game or application (of course 'limited to video generated when using a game or application'), but with 3D(10) In the case of a game of abilities, the user can control the 3D "traversing J" of the previously played clip, and cause the delay buffer 1515 to record the video and record the game state of the game segment. Therefore, the specific "crossing" is recorded as compressed. Vision - A 'But because the game state will also be recorded, different traversals will likely be at a later date in the same segment of the game. As described below, users on the colocation service 210 will each have a single user page' in which the user can post information about himself and other materials. One of the things that the user will be able to announce is a video clip from a game that has been saved. For example, if the user has overcome the particular difficulty of the game, the user can just "return" to the location where he or she gets the big result in the game, and then guides the colocation service 210 A video clip of a certain duration (for example, 30 seconds) is saved on the user's user page for viewing by other users. To do this, the application/game server 152 that is used by the user is only required to play back the video stored in the delay buffer 55.1 to the RAID array 1511·1512 and then The video clip is indexed on the user's user page. If the game has the ability to have a 3D DVR, as described above, the user can also record the game state information required by the 3D DVR and make it available to the user's user page. In the case where the game is designed to have a "bystander" in addition to an active player, 139840.doc -116- 200942305 'that is, a user who can travel in the world without observing (31) and observes the action) The game viewfinder app will enable the user to join the game as a bystander and player. ^ ^ g inspection implementation point See = in the colocation system 210, the user is a bystander rather than a live 2豕 there is no difference. The game will be loaded on the app/game pad and the user will control the game (eg, control the virtual camera of the world). Only - the difference will be the user's gaming experience. ❹ 多个 Multiple User Cooperation = Another feature of the escrow service 210 is the ability of multiple users to collaborate while viewing live video (even if they are viewed using a different device). This is useful when playing games and when using an application. ^Multi-pc and mobile phones are equipped with video cameras and have the ability to perform instant video (especially when the image is small). Also, a small camera is available, can be attached = visually and is implemented in software or using one of many hardware compressors for I-Video. It is not difficult to implement instant compression. Many Pc and mobile phones have microphones. And the headset is available with a microphone. These cameras and/or microphones combined with native video/audio compression capabilities (specifically, using the low-latency video compression techniques described herein) will enable users to share video and/or audio with the device. The control data is transmitted from the user premises 211 to the colocation service 21G. When used, the ability illustrated in Figure 19 can be achieved: the user can cause Wei = 1900 to appear on the screen of another user's game or application. This example is a multiplayer game where teammates collaborate in the car. Video/audio using 139840.doc 117· 200942305 can only be selectively viewed/heard by their teammates. Moreover, because there is effectively no delay in using the techniques described above, the player will be able to talk to each other or exercise at once without an appreciable delay. This video/audio integration is accomplished by having compressed video and/or audio from the user's camera/microphone as incoming Internet traffic. Next, the routing route 1502 routes the video and/or audio to the application/game server 1521-1525 that is permitted to view/hear video and/or audio. Then, the user who selects the respective application/game server 152 1-1 525 using the video and/or audio decompresses the video and/or audio and

It is integrated as needed to appear in a game or application, such as illustrated by 1900. S The example in Figure 19 shows how to use this collaboration in a game but the collaboration can be an extremely powerful tool for applications. Consider a scenario where a large building is being designed by New York City for a New York-based real estate developer by an architect in Chicago, but the decision involves a financial investor who is on the move and happens to be in Miami, and needs Decisions are made regarding the specific design elements of the building (how it relates to the buildings in the vicinity) to satisfy both the technical and real estate developers. The fake construction company has a high-resolution monitor with a camera attached to the PC in Chicago. The real estate developer has a laptop with a camera in New York, and the investor has a mobile phone with a camera in Miami. The building company can use the colocation service 210 to provide a powerful architectural design application that can be highly realistically reproduced by the host's and can take advantage of the large database of buildings in New York City, as well as the positive. Also - a database of ten buildings. The architectural design application will be executed in the application / 139840.doc 200942305 game feeder 1521-1525 (or on several if it requires a lot of computing power). Each of the three users at the disparate location will be connected to the colocation service 21〇, and each will have a simultaneous view of the video output of the design application, but it will be targeted .#—The user has the given device and network connection characteristics and is properly sized by the shared hardware. The dust reduction 1530 is appropriately sized (for example, the construction company can see the 2560xl yang 60 fps display via the Internet connection via 2〇鸠(4). The real estate developer of the group can use his lap to attack Lei Mu Dan to check the 孓罨 Mbps DSL connection to see β —..., and investors can use their mobile power = 25 〇 KbpS cellular data connection See 32〇χΐ8〇6〇, such as each side of the image will hear the other party's voice (will be handled by any of the many widely available conference call suite software in the application/game server (5)) 1525. The wire is triggered by the user's button on the device, and the user will be able to use his local camera to make the video appear. As the meeting progresses, the architect will be able to use the photorealistic sense of reality. Buckle reproduction display The same visuality is visible when the building rotates and causes it to adjoin the other building in the area, and all parties will see the same video at the resolution of the display devices of the parties. Any of the native devices used can handle 3D animation with this realism without problems, not to download .9 downloads or even store the huge amount of money needed to reproduce the buildings in New York City (4). From the point of view of each, although the distance is very long, and • #官系异异端端' but it will simply have a seamless experience at an incredible level. In addition, when one wants its face to be seen = better conveyed In the emotional state, it can be done. In addition, if the real estate 139840.doc -119- 200942305 = business or investors want to control (four) program money (four) its own input, for the keyboard, mouse, keypad or touch screen), It can be so, and it can respond without a perceived delay (assuming its network connection does not have a reasonable delay). For example, in the case of a mobile phone, if the action is connected to the WiFi network of the airport, it will have a very low latency. However, if it uses the cellular data network available in the United States today, it is likely to suffer a noticeable lag. However, most of the objectives of the conference (where investors are watching architects controlling building crossings or talking about videoconferencing = meetings), and even honeycomb delays should be acceptable. Finally, at the end of the σ conference, the real estate developer and investment cut their comments and the hosted service ceased, and the construction company will be able to rewind the video of the conference that has been recorded on the delay buffer 1515 and verify. Comments, facial expressions, and/or actions applied to the 3〇 model of the building during the meeting. If there are specific segments that they wish to save, they can move the video/audio from the delay buffer 1515 to the RAm array 1511-1 512 for archival storage and later playback. And from the cost point of view, if the construction (four) only need to use the computing power of New York City and the large database for 15 minutes of conference calls, then it only needs to pay the time when the resources are used, and ^ buy, You don't have to have a high-performance workbench and you don't have to buy a big copy of the big database. Video-rich community services Hosting services 210 are an unprecedented opportunity to build video-rich community services on the Internet. Figure 2G shows an illustrative user page for a game player on the colocation service 21〇. Like the game viewfinder application 139840.doc • 120· 200942305 The same program as the 'user page' is an application executed on one of the application/game servers 1521-1525. All thumbnails on this page and the §fL window show a constantly moving video (if the clip is short, it loops). Using a video camera or by uploading a video, the user (the user name is "KILLHAZARD") can post his own video 2〇00 (other users can view the video). The video is stored on the RAID array 15 11 -15 12 . In addition, when other users come to the user page of KILLHAZARD, if KILLHAZARD is using the colocation service 2 10 at this time, it will display the video of the live broadcast of whatever is being performed 2〇〇1 (assuming it Users who are allowed to view their user page are allowed to view it). This will be requested by the host/host server application application/game server 1521_1525 from the service control system 401 to determine if the KILLHAZARD is active (and if so, • request the application/game server 1521- 1525) to complete. The compressed video stream of the appropriate resolution and format is then sent to the application/game server 1521-1525 executing the user page application using the same method used by the game viewfinder application and displayed. If the user selects a window with a live broadcast of KILLHAZARD and then clicks on their input device appropriately, the window will be enlarged (using the same method as the game viewfinder application again) and the live video will be The resolution of the display device 422 of the user (suitable for viewing the characteristics of the user's internet connection) is filled in the screen. The key advantage of this prior art approach is that the user viewing the user page can see the live broadcast of the game that the user does not own, and may not have a local computer or game console capable of playing the game. It is 139840.doc -121· 200942305 users have an excellent opportunity to see the users of the "active" users playing games on the user page, and it is an opportunity to understand the games that users may wish to try or are good at. .

Video clips recorded or uploaded from the camera of KILLHAZARD's HD 2002 are also displayed on the (4) page, and below each video clip is the text of whether the partner is playing online or not (for example, "_" is playing the game. "Eragon" and MrSnuggles99 offline, etc. By clicking on the menu item (not shown), the framing video clip self-exhibition* recorded or uploaded video is switched to the currently playing colocation service (10) The game partner is in the video of the live broadcast of the beta material that he is currently carrying. So it becomes a game viewfinder for the partner group. If you choose the partner game and the user clicks on the game, the game will zoom in. To the full screen, and the user will be able to watch the game played live on the full screen. Again, the user of the game viewing the partner does not own the copy of the game and does not have the local computing/game console resources for playing the game. The game view is effective instantaneous.

As described earlier, when the user plays the colocation service 2, the player can "rewind" the game and find the photo slave he wishes to save, and then save the video clip to his user. page. This is called "Brag Clip". The video clips are all KII^LHAZARD self-reward clips saved from the previous game they played. The number of times the 2004 clip shows that the clip has been viewed, and the self-review clip is checked, the user has its rating. Opportunity to read the color of the spoon: The number of the shape of the figure 2005 indicates how high the rating is. When the user views the 1 139840.doc • 122- 200942305 user page, the self-review clip 2003 is continuously cycled along with the rest of the video on the page. If the user selects and clicks one of the self-review clips 2〇〇3, it is zoomed in to present the self-reward clip 20〇3, and the DVR that allows the clip to be played, paused, rewinded, fast forwarded, stepped, etc. control. The self-reward clip 2003 playback is loaded with the compressed view g-segment stored on the RAID array 1511-1552 by the application/game server ι521_ 1525 and decompressed by the application/game server ι521_ 1525 and Playback to implement. #

The self-reward clip 2003 can also be a 3D DVR" video clip from a 3D DVR capable game (i.e., a sequence of game states from a game that can be played back and allows the user to change the camera view point). In this case, game state information is also stored in addition to the particular "crossing" compressed video recordings that the user performs while recording the game segment. When the user page is being viewed and all thumbnails and video windows are constantly looped, the 3d self-reward clip 2' will be recorded as a self-reward clip of the compressed video when the user records the "travel" of the game segment. Ground loop. However, in addition to the user's choice of 3D DVR self-rewarding and smashing 3D, the user will be able to click to give it aside, except for the 峨 control that allows the playback of the video clip. A backup for the 3D DVR capabilities of the game clip. It will be able to independently control the camera during the game segment "crossing, and if it wishes (and has the user page allows it)" the basin will be able to record the alternative self-reward clip in the form of a thumbnail video; "Alternative self-reward | 轾 " 空勒 ^ JJ J traverse" will then be available to other viewers of the user (immediately, or the owner of the page on the user page 1 139840) .doc -123· 200942305 After verifying the opportunity to edit the clip). This 3D DVR Self-Camp Clip 2003 capability is enabled by launching a game that will replay the recorded game state slot on another application/game server 1521-1525. Because the game can be launched almost instantaneously (as previously described), so it is activated (its playback is limited to the game state recorded by the self-reward clip) and then the user is allowed to record the compressed video to the delay buffer 15 15 It is not difficult to use the camera to "cross" at the same time. Once the user completes the "crossing", the game is deactivated. From the user's point of view, it is not difficult to start the dvr control with the 3D DVR self-reward clip 2003. It does not know the game or even know how to play the game. It is only a virtual camera operator staring at the 3D world during the game segment recorded by another operator. The user will also be able to add their own audio to the self-reward clip (or • recorded or uploaded from the microphone 5). In this way, you can use the self-appreciation to create custom movements using characters and actions from the game. This animation technique is often referred to as "machinhna". A ❹ As the user progresses through the game, they will reach different skill levels. The played game reports the results to the service control system 4〇1 and these skill levels will also be displayed on the user page. Interactive animated ads Online ads have evolved from text to still images, video, and now to = dynamic clips, usually implemented using an Ad〇beFlash2 animated streamlined user. The reason for using the thin client is: user 139840.doc -124- 200942305

' 禹 I : The privilege of selling products or services to them is delayed. The streamlined client is executed on a very low-performance PC, and therefore, can have a degree of confidence' · interactive advertising will work properly . Unfortunately.. The animated streamlined client like Ad〇be Fhsh is limited in the degree of interactivity and the duration of the experience (to reduce download time). Figure 1 illustrates an interactive advertisement in which the user selects the exterior and interior colors of the car as it rotates in the showroom, while the instant ray is 縱 纟 : : : : : : 。 。 。 。 。 。 。 。 。 。 。 。 。 。 The user then selects the avatar to drive the car, and then the user can use the car for driving on a race track or through a country other than Monaco. The user can select a larger engine or a better wheel L and then see how the changed configuration affects the car's ability to accelerate or maintain stability. • Of course 'Advertising is effectively a cutting-edge 3D video game. But for such an ad that can be played on a PC or video game console, it will need to be 100 MB downloaded' and in the case of a pc, it may need to install a special drive, and possibly? (: The lack of sufficient CPU or GPU computing power is not performed at all. Therefore, 'These advertisements are impractical in the prior art configuration. In the colocation service 21〇, the advertisements are placed almost instantly, and preferably Execution, regardless of the user's client 415 capabilities, so it delivers more quickly than a streamlined user-side interactive ad, and the experience is more plentiful' and two-times reliable. • Streaming geometry RAID array 1511 during instant animation The -1512 and the incoming route 15〇2 can provide such a fast data rate with such a low latency that the design is likely to depend on 139840.doc •125· 200942305 RAID Array 1511-1512 λ 阜 阜 Route 1502 is on the fly Animation (for example, φ#, #越的) with a complex database in the middle of game play or in an application, can directly deliver a few video games and applications in a shape. j Technology, 4 ( For example, the mass video storage devices available under the video game system shown in Figure i (especially in practical home devices are so slow that they cannot be played in the game - Streaming Geometry (4) ( For example, in the presence of a specified road, the driving pleasure φ, - ΓΑ can properly predict the geometry of the building used to enter the field of view and A large number of storage devices can be used to search for the location of the upcoming geometry in advance. In a complex scene with unpredictable changes (for example, in a battle scene with complex characters around), if the RAM on a PC or video game system Fully filled with the geometry of the object currently in view, and then after the user suddenly turns his character to view his character, if the geometry is not preloaded into the RAM, the geometry may be displayed There is a delay before. In the colocation service 210, the RAID array 1511·1512 can exceed the speed data of the ultra-high speed Ethernet speed, and under the SAN network, it is possible to achieve better than 10 billion bits. Ethernet is better than other network technologies at a rate of 10 billion bits per second. 100 million bits per second will load 1 billion bytes of data in less than - seconds. At 60 fps In the frame time (16 67 sec), about 170 megabytes (21 MB) can be loaded. Of course, even in the RAID configuration, the rotating media will still cause a delay of more than one frame time, but Flash memory-based RAID storage will eventually be larger than the rotating 139840.doc -126- 200942305 media RAID array and will not incur this high latency. In one embodiment, the use of large RAM writes is fast. Taken to provide very low latency access. Therefore, at a high enough network speed and a large amount of low enough latency, the geometry can be streamed to the CPU and/or GPU to process 3D data. In the application/game server 1521-1525. Thus, in the previously given example, where the user suddenly turns his character and looks backwards, the geometry of all the characters behind him can be loaded before the character completes the rotation, and thus, for the user, It seems that it is in a world that is as realistic as a real-life photo of live action. As discussed earlier, one of the last boundaries in a photo-realistic computer animation is the human face, and due to the sensitivity of the human eye to incompleteness, the slightest error from a photo-realistic face can result from viewing. The negative reaction of the person. Figure 22 shows the use of 〇0111; 0111*1^ authenticity capture technique (the subject of the same application in the application: September 10, 2004, application No. 10/942,609, "Apparatus and method for capturing the motion of a Performer"; September 15th, 2004, pp. 10/942, 413, "Apparatus and method for capturing the expression of a performer"; February 25, 2005 曰 Application No. 11/066,954 "Apparatus and method for improving marker identification Within a motion capture system"; March 10, 2005 App Application No. 11/077,628 "Apparatus and method for performing motion capture using shutter synchronization"; October 20, 2005 曰 Application No. 1 1/255, 854 "Apparatus and 139840.doc -127- 200942305 method for performing motion capture using a random pattern on capture surfaces"; "System and method for performing motion capture using phosphor application techniques", 2006, No. 11/449,131, filed on June 7, 2006 Application No. 11/449,043 "System and m" on June 7 "ethod for performing motion capture by strobing a fluorescent lamp"; "System and method for three dimensional capture of stop-motion animated characters", filed on June 7, 2006, each of these applications The live broadcast performance captured by the assignee of this CIP application has resulted in a very smooth capture surface that achieves a high polygon count tracking surface (i.e., the polygon motion accurately follows the movement of the face). Finally, photo-realistic results are produced when the live broadcast of the live video is mapped onto the tracking surface to create a textured surface. Although current GPU technology is capable of reproducing many polygons in the tracking surface and texture and illuminating the surface in real time, if the polygon and texture each frame time changes (which will produce the most photorealistic result), it will quickly All available RAM of a modern PC or video game console is consumed. Using the stream geometry technique described above, the geometry is continuously fed into the application/game server 1521-1525 so that it can continually create a photo-realistic face to allow for almost indistinguishable The video game of the facial face of the live action becomes practical. Integration of linear content and interactive features 139840.doc -128- 200942305 Movies, TV shows and audio materials (collectively "Linear Content") are widely available in many forms for home and office users. Linear content is available on physical media such as CD, DVD, HD-DVD and Blu-ray media. It can also be recorded by a DVR broadcast from satellite and cable τν. In addition, it can be used by the Pay-As-You-See (PPV) content of the Hygiene and Cable Tv and the Video on Demand (VOD) on the cable τν'. Increasingly linear content is available via the Internet for downloaded content and cross-talk. Today, there is really no place to experience all the features associated with linear media. For example, Qing and other video optical media often have interactive features that are not available at other locations (such as the director's » Ping _, "Flock" short films, etc.). Online music sites have cover art and song information that are not normally available on CDs, but not all CDs are available. • Websites associated with TV shows often have additional features, blogs, and sometimes comments from actors or creators. In addition, in the case of many electric events or sports events, there are often frequent (four) sexual media-issued (in the case of movies) or (in the case of sports) can be closely linked to real-world events. (For example, the player's video game. The colocation service 21 is very suitable for delivering linear content when linking together disparate forms of content. Indeed, delivering a movie is not as good as delivering a faceted interactive video game. And colocation services 21. Deliver linear content to a variety of devices in a home or office, or to a mobile device. Figure 23 shows an exemplary user interface page for a colocation service 21' that exhibits linear content 139840.doc -129. 200942305 However, unlike most linear content delivery systems, colocation service 210 is also capable of delivering relevant interactive components (eg, menus and features on DVD, interactive on HD-DVD) Overlay, and Adobe Flash animation on the site (as explained below). Therefore, the client device 415 limits no longer on which In addition, the colocation system 210 can dynamically and instantly link linear content to video game content. For example, if a user is watching a Quidditch game in a Harry Potter movie, and decides If you are willing to try Quidditch, you can just click the button and the movie will pause and it will be immediately sent to the Quidditch segment of the Harry Potter video game. After playing the Quidditch game, another click on the button and the movie will be revisited immediately In the case of photo-realistic graphics and production techniques, the video captured by the camera cannot be distinguished from the live action characters, when the user conducts a live broadcast of the action movie Quidditch game to the colocation service. In the transition of the Quidditch game in video games (as described in this article), the two scenes are virtually indistinguishable. This provides a new creative option for directors of both linear and interactive (eg, 'video game') content. Because the lines between the two worlds become indistinguishable. Using the host shown in Figure 14. The service architecture provides control of the virtual camera in the 3D movie to the viewer. For example, in the % view that occurs within the train, it will be possible to allow the viewer to control the virtual camera and look around the train as the story progresses. This assumes that all 3D objects ("assets") in the train and enough 139840.doc -330 - 200942305 computing power levels are available to instantly reproduce these scenes as well as the original movie. And even for non-electrical entertainment, there is It can provide (4) interactive features of constant stimulation. For example, *5, "The arrogant and more beautiful old British building," the eight-day scene. For a specific building scene, the user can pause the video and then control the camera. To patrol the generosity, or possibly the surrounding area. In order to implement this, a camera with a fisheye lens can be carried through the building. When it is chasing its location, it is very similar to implementing the prior art.

AP_Company's QuickTime VR. The various frames will then be converted so that the image is not distorted' and then stored along with the movie on the ruler array 1 5 11 15 12 and played back when the user chooses to continue the virtual tour 0 in the event of a sport event A live event (such as a basketball game) can be streamed via the colocation service 210 for viewing by the user (as it would be for a typical TV). After the user views a particular play, the video game of the game (finally the basketball player looks as realistic as the real player's photo) can catch up with the player starting in the same position, and the user (possibly controlling one player each) can replay To see if it can do better than these players. The colocation service described in this article is extremely suitable for supporting this future world because it can withstand the computing power and large-capacity storage resources that are impractical to be installed in the home or in most office settings, and . Ten resources are always up-to-date (with the latest computing hardware available), but in the home background, there will always be families with older generations of pCs and video games. In addition, 'in the colocation service 21〇, the use of 139840.doc • 131 · 200942305 is hidden in this computational complexity, so even if the user uses a very sophisticated system, from the user's point of view, it is like changing TV: The frequency of Siam is as simple as that. In addition, the user will be able to access all of the computing power and computing power that will result from any client 415. The multiplayer game to the extent that the game is a multiplayer game, then (4) will be communicated not only to the application/game server (1) M525 via the incoming route 1502 but also by the bridge to the hosted service 21 . The Internet (not shown) of the server or game machine being executed. When playing multiplayer games on a computer on a universal Internet, the application/game server '1521-1525 will have extremely fast access to the Internet (with the game system running on the server in the home) In contrast, but it will be limited by the capabilities of other computers playing games on slower connections, and potentially by the network: the game consoles of the Internet j are designed to accommodate the least commonalities (these games) Benefits will be a factual limitation of household electricity on a relatively slow consumer Internet connection. Tian 70 King can make a huge difference when playing multiplayer games in the host agency service 21 〇 server center. Each user-app/game server 1521_1525 will be hosted with other application/game servos 1§ 1521.1 525 and by extremely high speed, very low latency connectivity and large, very fast storage array colocation Any server interconnection for the central control of multiplayer games. For example, if ultra-high-speed Ethernet is used for 15Q2 network, then the application/game server 1521-1525 will be in the field. 1 Communicates to any server that hosts a centrally controlled multiplayer game at a time rate of 1 billion 139.doc 200942305 milliseconds or less than 1 millisecond per second. In addition, RAID array 1511- The 1512 will be able to respond very quickly and then transmit the data at a rate of one billion bits per second. As an example, if the user customizes the character in appearance and clothing, the character has a large amount of geometry unique to the character. Shape and behavior, under the prior art system limited to the game client executing on the PC or game console in the court, if the character will enter the view of another user, the user ❹ will have to wait until the long Slow download is complete to load all geometry and behavior data into its computer. In the colocation service 210, the same download can be better than the RAID array 15 11 · 1512 servo at a rate of 1 billion bits per second. Ultra-high-speed Ethernet. Even if the home user has 8 Mbps • Internet connection (which is extremely fast according to current standards), the ultra-high-speed Ethernet is also 100 times faster. Therefore, a one-minute work on a fast Internet connection will take less than a second on a super-fast Ethernet network. Top Player Grouping and Tournament e The Hosting Service 210 is ideal for tournaments because there is no game system. Executed in the local client, so there is no chance for the user to cheat. Moreover, due to the ability of the output route 1540 to multicast the UDP stream, the colocation service 210 can simultaneously broadcast a larger tournament to thousands of viewers. In fact, when there are so popular that thousands of users are receiving the same stream of specific video streams (for example, showing a larger tournament view), the video stream is sent to the content delivery network (cdn) (such as , Akamai or

Lmehght) may be more effective for large distribution to many client devices 415. 139840.doc 133· 200942305 Fields using CDNs to show top-level player grouping game viewfinder pages yield similar levels of efficiency. For larger tournaments, live celebrity commentators can be used to provide comments during specific competitions. Although a large number of users will be watching a larger game and a relatively small number will be played in the tournament. An app/game server 1521-1525 that can route any celebrity mode replicas from a celebrity solution, a member's 0° to a host hosted in a tournament and hosted a tournament towel game. 'And you can add audio to the game audio. You can cover the celebrity's video on the game (or just on the bystander's view). Web page loading accelerates the World Wide Web's main delivery agreement, hypertext transfer protocol (htt_ is conceived and defined in the era of commercial high-speed Internet connectivity, and online consumers using dial-up modems or ISDN. At this time, the "page gold standard" for the Shaanxi speed connection is the T1 line, which symmetrically provides a 1 5 PS data rate (that is, an equal data rate in both directions). ❹ Today, the situation is completely different. The average home connection speed in the world connected via a 胤 or electric iron modem has a much higher data rate than the τι line. In fact, in some parts of the world, fiber-to-the-curb It is bringing the rate of up to 5 〇 〇 ^ ^ 引入 。 。 。 。 π 用 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 The server just says 'HTTP request _ file, the collection of files on the list is waiting to download the file 139840.doc • 134· 200942305 f ask for the second file 'waiting to download the « etc. In fact, HTTP allows More than one "open connection" (that is, each request _ more than one broadcast), a; negotiated 'quasi (and the desire to prevent the network server from being overloaded) so that only a very small number of open connections are permitted. In addition, because web pages are structured in a way that is often unaware of multiple simultaneous pages that can be used for immediate downloading (ie, 'only becomes apparent after scraping a page: you need to download: a new file for the shirt image). Therefore, the Tanru case on the website is essentially one by one, and because of the request and response agreement used by HTTp, there is a general connection with each of the loaded cases (access to the typical US web server). ) 1 〇〇 millisecond delay. In the case of relatively low speed connections, this does not introduce a lot of problems, because the download time for the file itself determines the waiting time of the web page. However, the connection speed increases (especially complex web pages) In the case of), it starts to cause problems. ❹ In the example shown in 圊24, a typical commercial website (this particular website is from a larger sports shoe trademark) is displayed. There are M files on the website. t ^^HTML ^ CSS ^ JPEG > PHP > JavaScript A Flashy and includes video content. Before the live web page (that is, «and (4) use it), 'must load a total of 15M bits 2 Γ mass 楷There are many reasons for this. First, it is a complex and sophisticated network and secondly it is a web page that is dynamically combined based on information about the users accessing the page (eg, from which country the user came from, what: ^user Whether it was before buying, etc.), and depending on all these factors, but different files. But 'it is still a very typical commercial web page. 139840.doc •135· 200942305 Figure 24 shows the increase in connection speed before the live webpage The amount of time that has passed away. At 1.5 Mbps connection speed 2401, using a conventional web server with a conventional web browser, it takes 135 seconds before the live web page is live. At 12 Mbps connection speed 2402, the load time is reduced to 6.5 seconds, or approximately doubled. However, at a connection speed of 2403 at 96 Mbps, the load time is only reduced to approximately 5.9 seconds. The reason for this is that at this high download speed, the time to download the file itself is the smallest, but the delay of approximately 1 millisecond per file is maintained, resulting in a delay of 54 files * 100 milliseconds = 5 4 seconds. Therefore, no matter how fast the connection to the court is, the site will spend at least 5.4 seconds before the live broadcast. Another factor is that the server side is queued up; each HTTP request is in the back of the queue & &force; because on the busy server 'this will have a significant impact' because for the network feeder For each small item that gets, the HTTP request needs to wait for it to return. · One way to solve this problem. After the cover & go & β ^ -----

Slow U speed, and HTTP does work well at slow speeds

Or on the local library). By the RAM on the type 1/game server 1521-1 525, because of the very fast interconnection via the routing route 15〇2 (or to the local storage 139840.doc -136· 200942305 (5), not every Ηττρ is used The file has a delay of 〇〇 (4), but there will be a minimum delay per audit when using HTTP. Then, instead of allowing the user in the family to access the webpage via ΐΗττρ, the user can access the webpage via the client 415. Then, even in the Μ

Ps is connected (because this page does not require a lot of bandwidth for its video), = will also be live broadcast in less than 1 second per line 2400. There will be no delay before the web browser executing on the application/game server 1521_1525 displays the live broadcast page, and will not exist until the user displays the video output from the web browser. Only the measured delay. When the user uses the mouse to search the webpage and/or type a word on the webpage, the user's input information is sent to the web browser executed on the application/game server 1521·1525, and the network Liu The browser will respond accordingly.不利 The disadvantage of this method is that if the device is transmitting video resources in a positive manner: the bandwidth is used, even if the web page becomes static. This can be remedied by the group reducer to transmit data only when (and if) the web page changes and then only transmit the poor to the changed page portion. When there are web pages that have a constantly changing flash slogan, etc., the web pages tend to be ostentatious, and unless there is a reason to move something (e.g., a video clip), the web page is typically static. For such web pages, there may be situations where the colocation service 210 will use less traffic (compared to the conventional web server) because only the actual displayed image will be transmitted, no thin client executable code And there are no large objects that may never be viewed (moving images). ^ J39840.doc -137· 200942305 Therefore, using the colocation service 210 to host the old webpage can reduce the webpage loading time to the extent that opening the webpage is similar to changing the channel on the TV: effectively live broadcast instantly The page. Promoting the Debugging of Games and Applications As mentioned earlier, video games and applications with instant graphics are very complex applications and often have defects when they are released into the field. Although the software developer will get feedback on the defect from the user and it may have some tricks to return the machine state after the crash, but exactly identify what caused the game or instant application to crash or inappropriately Execution is very difficult. Although the game or application is executed in the colocation service 21〇

The video/audio output of the play or application is constantly recorded on the delay buffer 1.515. In addition, the watchdog process executes each application/game server 1521-1525. The watchdog process will report to the hosted service control system 401 that the application/game server bum is smoothly smoothing. The right watchdog process fails to report', then the server control system communicates with the application/game server 1521_1525, and if successful, will collect whatever machine state is available. Regardless of what is available:: The business is sent to the delay by the record-to-record: 21='Λ Game or application soft (4) Provider from the host escrow service map knows the reason for its reason The box is connected to the value. ,money. This information is missing (4) to fix it in the price of 139840.doc • 138· 200942305 Also, the main idea is that although the app/game server 1 52 1 -1 525 crashes, the most recent restartable Apologize for technical difficulties by restarting the feeder and providing the message to the user. Resource Sharing and Cost Savings • The systems shown in ® 4a and Figure 4b provide multiple benefits for both end users and both game and application 帛 developers. For example, in general, home and office to client systems (e.g., pcs or game consoles) are only in use for a small percentage of hours of the week. According to Nielsen Entertainment "Active Player Benchmark Study" (http://www.prnewswire c〇m/cgi_ bin/stories.pl?ACCT=104&STORY=/www/story/10-05-2006/ 〇〇 〇4446115&EDATE=) 2nd, 6th, 1st, 5th, 5th, communication, active game's average of 14 hours a week to play on the video game console and about 17 hours a week on the handheld device play. The report also states that for all game play activities (including console, handheld devices and pc game play) active players average - 13 hours per week. Consider the higher number of console video game play time, there is a week 24 *7=168 hours, he suggested that in the active gamer's home, the video game console is only in use during the hours of 17/ΐ68 = ι〇% of the week. Or, at 9〇% of the time, the video game console It is idle. Given the high cost of video game consoles and the fact that manufacturers are responsible for devices such as (4), this is a very inefficient use of expensive resources. PCs in business are usually only one hour a week. Used in non-portable desktop PCs, especially for applications such as Autodesk Maya. Although some businesses operate on all hours and holidays and some pcs (for example, take them home for use) The portable system that works in the evening is used at 139840.doc • 139· 200942305 for all hours and holidays, and the time zone is concentrated in the week; most commercial activities tend to be given in the business to Friday. 9 ΛΜ to 5 PM, less false 曰 and disconnection time (such as 牛 ^ b ^ 夕假1 * noon name)' and because most pc usage occurs when the user actively uses the PC ^ ^ Use, ^, see, so it follows: Desktop PCs tend to follow these hours of operation.

5 , ρλ/ί .Ε Right false 疋—The five days of the week are constantly used 9 from 9 AM to 5 PM, then he will imply that the PC is in the hour t 40/168=24% of the hour ▲ &# ^ ^ θ ^ Utilization. The still-powered desktop PC is a very expensive investment for business, and the computer is low (four) degrees. The school that teaches on the computer on the table can be m, PV* ^ ^ - the use of electricity in the week and even the smaller rate is changed and the number of hours is changed by the teachings, but most of the teaching is in Appears during the hours of the week from Friday to Friday. Therefore, in general, the PC and the sfL game console are only utilized in _ 〇 at a small fraction of 31 hours. Passing attention to 'because many people learn and play during the Tuen Mun period from Monday to Friday, so these people usually do not play video games during these hours, a gentleman and therefore when they do play In video games, the k-hang is free during the other hours, weekends, and fakes. - The configuration of the main building s service shown in Figure 4a, the usage patterns described in the above two paragraphs lead to the use of the 十. Obviously, there is a limit on the number of people who can be servoed by the colocation service 210 by the #Μ## eroded field at a given time', especially if the user needs to use it for complex applications. Game) The case of instant responsiveness. However, no .^ city 』0 or a commercially used PC (which is usually placed idle most of the time), can be used by different users.

Re-use at the same time. For example, t, a, A

° Dual-CPU with dual CPU and dual GPU 139840.doc •140· 200942305 and high-performance RAM 4〇2 can be used by commercial and school in non-false 9 9 AM to 5 PM, but by playing cutting-edge video games The player uses it at night, on weekends and on holidays. Similarly, low-performance applications can be utilized by businesses and schools during business hours on low-performance servers with Celeron CPUs, GPUs without GPUs (or low-end GPUs), and limited RAM. The game may utilize the low performance server 402 during non-commercial hours. In addition, in the case of the colocation service configuration described herein, the resource is effectively shared among thousands (if not millions) of users. In general, online services only use a small percentage of their total user base to use the service at a given time. If you consider the Nielsen video game usage statistics listed previously, it's easy to understand why. If active players play console games for only 17 hours a week, and if the peak usage time of the game is assumed to be • at night (5-12 AM, 7*5 days = 35 hours/week) and weekends (8 am_12 AM' 16 *2=32 hours/week for a typical non-working, non-commercial hour, for a 17-hour game, there are 35+32=65 peaks in a week. It is difficult to estimate the exact peak user load on the system for a number of reasons: some users will play during the extra-peak time, there may be a cluster peak of the user at a particular daytime, and the peak time may be affected by the type of game being played ( For example, a child's game will probably be affected at an early time in the evening. Time to play). However, assuming that when the player is likely to play the game, the average number of hours played by the player is much less than the number of hours between the players, then only the number of users of the host escrow service 210 will use the host at a given time. Agency service 210. For this analysis, the peak load is assumed to be 12 . Therefore, only 12.5% of the computing, compression and bandwidth resources are used at 139840.doc -141 - 200942305 at a given time, resulting in only 12 5% of the hardware cost to support a given user due to resource reuse. Play one of the performance games given levels. In addition, assuming that some games and applications require more computing power than others, resources can be dynamically allocated based on the game being played by the user or by the application executed by the user. Therefore, users who choose a low performance game or application will be assigned a low performance (lower cost) server 402, and the selection will be high. The user of the performance game or application will be assigned a high performance (more expensive) server 402. In fact, a given game or application may have a lower performance and higher performance area for the game or application, and the user can switch from a server 4〇2 to a game or application area. Another server 4〇2 is executed to keep the user on the lowest cost server 402 that satisfies the needs of the game or application. Note that a RAID_column 405 that is much faster than a single disk will be used by even the low performance server 4〇2, which has the benefit of a faster disk. Therefore, the average cost per server for all games played or used is much less than the cost of most expensive servers 402 playing the highest performance games or applications, however, even low performance The server 402 will also obtain the disk performance benefit from the RAID array 405. In addition, the server 402 in the colocation service 210 may be only a pc motherboard without a disk or a peripheral interface (different from the network interface), and just can be integrated down to a fast network just to the SAN 403. The single interface of the road interface. Also, the RAID array 405 may be shared among a much larger number of users than in the presence of a disk, so the disk cost per active user will be 'far less than a disk drive. All of this equipment will likely reside in an environmentally controlled bracket in the environment of the 139840.doc -142. 200942305. If the feeder 4〇2 fails, it can be easily repaired or replaced at the colocation service 210. Compared to 2 'Pc or game console in the home or office must be sturdy, must be · · _ free from reasonable wear and tear to prevent being hit or (4) independent piano II. - The disk drive 'has to survive unfavorable environmental conditions (for example, being forced into a heated AV cabinet with other appliances). The brother needs a service guarantee, must be packaged and shipped, and is purchased by a retailer who may charge zero (4) In addition, the PC or game console must be configured to meet the peak performance of the most computationally intensive game or application that will be used at some point in the future, even for lower performance games or applications (or games or applications). The program area may also play most of the time. In addition, if the console fails, it will be repaired and the process will be time-consuming (adversely affecting manufacturers, users and software developers). In this case, the system shown in Figure 4a provides the experience of the experience of the local computing resource to the user for the user to experience in the home, office or school. Computational power, which provides much lower computing power through the architecture shown in Figure 4a. Eliminates the need for upgrades. In addition, users no longer have to worry about upgrading PCs and/or consoles to play new games or handle higher performance. New applications. Any game or application on the Hosting Service 21 (regardless of the type of server 4〇2 required by their games or applications) can be used by users and all games and applications Proximity to execute (ie, 'fastly load from the RAID array 4〇5 or the local storage on the server 402) and properly have the latest updates and defects 139840.doc 143· 200942305 〇 repair (ie, software The developer will be able to select the ideal server configuration for executing the server 402 of a given game or application, and then configure the feeder 402 with the best driver, and then over time, the developer will Ability to simultaneously provide updates, bug fixes, etc. to all copies of the game or application hosted by the hosted service 210. In effect, the user begins to use the colocation service After 210, the user may find that the game and the application private continue to provide a better experience (eg, via updates and/or bug fixes) and may be in a situation where the user discovers a new game or application available to utilize computing technology after one year. (for example, a higher-performance GPU) (which did not even exist a year ago) on the 21G service, so for the user, it will be impossible for Lin to buy the game or execute the application in one year. The former (4). Because the computing resources for playing games or executing applications are invisible to the user σ (that is, 'from the user's point of view, the user only chooses the game that starts to execute immediately or (4) the program _ more like The user's TV channel), so the user's hardware will be "upgraded" even if the user does not even know the upgrade.消除 Eliminate the need for backups. Another problem with (4) users in schools, schools, and homes is injury. The right disk is faulty, or it is hard

Or the video game console (10) stored in the local PC (for example, in the case of the console, the game results and levels) may be lost. There are many applications available for backup of PCs, and the game can be controlled: the state is on the line for backup, but usually it must be stored in another location that must be stored in a safe and organized place. 139840.doc • 144·200942305, his non-volatile storage device), and the backup for online services is often limited due to the slow upstream speed available via the typical low-cost Internet connection. In the colocation service 21 of Figure 4a, the stored in the RAm array can be used as a prior art well known to those skilled in the art; fine configuration techniques are configured to cause the disk to fail. At the time, the device will not be discarded and will notify the technician at the server center that contains the failed disk and then the technology will replace the disk, which will then be automatically updated 0 to make the suppression The array is once again tolerated and faulty. In addition, since all the drives are close to each other with a fast local network via the programming, all the disk systems are configured in the server center to periodically injure the human-level storage (which can be stored) It is not difficult at the center of the feeding machine or by the heavy new location. From the point of view of the user of the colocation service 210 - see that its data is always completely secure and it never has to be considered for backup. Access to the presentation Users often want to make a game or an application before purchasing a game or application. As previously described, there is a demonstration (the "verb" verb form means the trial demo version, the demo version is also called the "demo", but as a - noun) the prior art components of the game and the application, but each of them - Subject to restrictions and/or inconvenience. Using the colocation service 21〇, . It is easy and convenient for the user to try out the demo. In fact, what the user does is select the demo and try the demo via a user interface (such as the user described below). The presentation will load almost instantly onto the feeder 402 suitable for the presentation, and it will be executed exactly like any other game or application. Regardless of whether the presentation requires a very efficient server 139840.doc -145. 200942305 402 is still a low-performance server 402, and regardless of the type of home or office client 415 used by the user, from the user's point of view, the presentation is Will work. The software publisher of the game demo or application demo will be able to precisely control how long the demo and trials are allowed for the user to try, and of course, the demo may include providing the user with the full version of the game or application being presented. The user interface element of the access opportunity. Because the presentation may be below cost or free of charge, some users may try to use repeated presentations (especially repeating playable game demos that may be interesting). The colocation service 210 can use various techniques to limit the presentation usage for a given user. The most straightforward method is to establish a user ID for each user and limit the number of times a given user m can play the presentation. However, the user can set multiple user IDs, especially if they are free. One technique for addressing this problem is to limit the number of times a given client 415 can play a presentation. If the client is a standalone device, the device_ will have a sequence number and the colocation service 21 can limit the number of times the presentation can be accessed by the client with the serial number. If the client 415 is executing on pc or other software on n pieces, then the serial number can be assigned by the host hosting service 21 and the sequence number is saved on the PC and the serial number is used to limit the presentation use but assume that the PC can be used. To re-programize, the item number is erased or changed, then the other-option host hosting service 210 maintains the Pc network adapter media access control (MAC) address (and/or other machine-specific identifiers such as Record the hard drive number, etc.) and limit the use of the demo to the mac address. It is assumed that the MAC address of the network adapter can be changed. However, this is not an extremely simple method. The other-method limit (four) shows the number of times that can be played to a given green. 139840.doc -146- 200942305 Although the address can be reassigned periodically by cable modems and DSL providers, it does not occur very frequently in practice and can be determined if (eg, by contacting an ISP) IIMf, in a block of IP addresses for residential DSL or cable data 2 access, typically establishes a small number of presentations for a given home. Also, there may be multiple devices in the home after the Nat router sharing the same IP address, but typically in a residential setting, there will be a limited number of such devices. If the IP address is in the Servo Business Area block, a larger number of presentations for the business can be established. However, in the end, all combinations of the previously described methods are the best way to limit the number of presentations on the PC. While there may not be a very simple way to make the determined and technically skilled user likely to be limited in the number of replayed presentations, creating a number of obstacles can create enough obstacles to make most PCs worthwhile. The gods are going to abuse the deductive system, and instead, they use the demo when they want to try out new games and applications. Benefits to Schools, Businesses and Others 〇 Significant evidence is particularly evident in commercial, school and other institutions that utilize the systems shown in Figure 4a. Businesses and schools have substantial costs associated with installing, maintaining, and upgrading PCs, especially when talking about PCs such as high-performance applications. As stated previously, pcs are typically only used in one-hours of an hour, and as in the home, the cost of a PC with a given amount of efficiency is much higher in an office or school environment than in a servo. The cost in the heart environment. In the case of larger businesses or schools (eg, large universities), the IT department of these entities sets up the server center and maintains the computer connected via the LAN level and the remote 139840.doc -147- 200942305 Actual. There are many solutions for remotely accessing a computer via a LAN or via a private high-frequency wide connection between office and office, either by Microsoft's Windows terminal server or by a virtual network computing application ( For example, VNC from RealVNC Ltd. or a remote client component from Sun Microsystems can obtain remote access to a PC or server with a range of qualities in graphical response time and user experience. In addition, these self-managed feeding machine centers are usually dedicated to single-business or school, and (4) this cannot be used in disparate applications (for example, entertainment and business applications) at different times of the week - calculation (four) The overlap of possible uses. Therefore, 'many businesses and schools lack the scale, resources or expertise to independently set up a ship center with a network connection to each user's speed. In fact, a large percentage of schools and businesses have the same Internet as the family. Network connection (for example, DSL, cable modem). ❹ However, these organizations may still have the need for very high performance calculations (either periodically or periodically). For example, a small construction company may only have a small number of (4) divisions. When designing I, it has a relative computational need of 5, but it may periodically require very efficient 3D diversification (9) such as 'when establishing The system of the new building design at the user end: the system that is not displayed in 4a is extremely suitable for the network connection of the group ==^home^-type (for example,

As the user terminal..., or completely two: can use the low-cost PC signal logic (4) and low latency, f $ can also be implemented, and only implement the control _ time video decompression 412 low-cost special device 139840.doc • 148· 200942305 Pieces. These features are particularly attractive for the problem of damage to the device, the theft of the PC, or the special group within the pc. The configuration solves many of the problems that are also used for general purpose calculations (and many of these operational costs (which ultimately must be: / shared by the user). For example, there are viable businesses): Recursively return to the user for

: Other applications that use time share, (8) These groups are obtained only when they are needed (and incur costs). The storage of high-performance computing resources does not have to be provided for backup or otherwise maintain efficient and efficient. Resources. Elimination of piracy ▲In addition, games, applications, interactive movies, etc. may no longer be pirated as they are today. Since the game is executed at the server center, the user does not have the # for the basic code, because there is no piracy. Even if the user is going to copy the source code, the user cannot execute the code on the standard game console or the court computer. This opens up markets around the world where standard video games are not available, such as China. Resale of games that have already been used is also impossible. For game developers, there are fewer market discontinuities as they are today. With the new generation of technology forcing users and developers to upgrade and game developers depending on the current situation of timely delivery of the hardware platform, the colocation service 210 can be gradually updated as the game requirements change over time. Streaming Interactive Video 139840.doc • 149· 200942305 The above description provides a low-latency streaming interactive §fl based on the Universal Internet (which implicitly also includes audio along with video, as described in this article A variety of applications that enable the use of novel basic concepts. It is known via the Internet that prior art systems for streaming video have only those applications that can be implemented by high latency interactions. For example, for the basics of linear video, playback control (e.g., pause, rewind, fast forward) works properly at high latency, and it is possible to select among the online video feeds. Moreover, as previously stated, the nature of some video games allows them to be played with high latency. However, the high latency (or low compression ratio) of prior art methods for streaming video severely limits the potential application of streaming video or narrows its deployment to a specialized network environment, and even in such environments. The prior art also introduced a substantial burden on the Internet. The techniques described in this article open up the large number of applications that are likely to be available under low-latency streaming interactive video over the Internet, especially those enabled via consumer-grade Internet connections. . In fact, under the user-side device that is generally small with the user terminal 465 of the heart, it is sufficient to use any amount of computing power, any amount of fast storage, and extremely fast network connection between powerful feeders. And provide an enhanced user experience' that enables a new computing era. Because the bandwidth requirement does not increase with the calculation of the system for @ i & μ eb force (material, because the bandwidth requirement is only about the display resolution, the quality of the product and the frame (4) W/fm The problem is that the 生 曰 曰 ( 例如 ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( 经由 经由 经由 经由 经由 经由 经由 经由 经由 经由 经由 经由 经由Typical consumers and 139840.doc •150· 200942305 Business applications are necessary for heavy clients (such as executing Wind〇ws,

Linux, PC or mobile phones such as SX, or even thin client (such as 'Adobe Flash or Java').

The emergence of streaming interactive video has led to a reconsideration of assumptions about the structure of the computing architecture. One such example is the colocation service 210 server center embodiment shown in FIG. A video path feedback loop for the delay buffer buffer and/or the group video 1550, wherein the multicast φ stream interactive video output of the application/game server 1521-1525 is either via the path or via path 1551 The optional delay is then fed back to the application/game server 1521_1525. This can be achieved by a number of practical applications where prior art feeders or local computing architectures are not possible or feasible (e.g., such applications as illustrated in Figures 16, 圃 Figure 17 and Figure 20). However, 疋 'as a more general 牟 拉 r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r Video infinitely loops m^ This soft moon has never been available for multiple application possibilities. The characteristics of the sputum and sputum in the 71 soap to the udp flow. This effect enables any degree of multicasting of streaming interactive video (in contrast, the two-way flow of the (10) streams will generate more traffic on the incoming network as the number of users increases. Stagnant). Multicast suits | § The key capabilities within the center are / that allow the system to respond to the growing needs of Internet users (actually, the world's population) to communicate on a one-to-one basis. Again, the descriptions discussed in this article are examples of the use of both deportation and multicast (eg, Figure 139840.doc)

• 15U 200942305 I6) is only the tip of a very large iceberg with the possibility. In one embodiment, the various functional modules and associated steps described herein may be comprised of specific hardware components (such as special application integrated circuits ("ASICs") that contain the fixed-line logic for performing the steps. )) or by any combination of programmed computer components and custom hardware components. In an embodiment, the modules can be implemented in a programmable digital signal processor ("Dsp") such as TMS320X architecture (eg, TMS32〇C6〇〇〇, tms32〇c5嶋, 〇, etc.). ")on. A variety of different DSPs can be used while still adhering to these basic principles. Embodiments may include various steps as set forth above. These steps may be embodied in a machine executable finger 7 that causes a general purpose or special purpose processor to perform particular steps. Various components (such as computer memory, hard disk drives, and input devices) that are not related to these basic principles have been omitted from the figure to avoid mixing the relevant aspects. The elements of the <RTI ID=0.0>>><>> The machine-readable medium can include "the day is not limited to (four) flash memory, optical disc, CD_R〇M, DVD r〇m, ram, EPROM, EEPROM, magnetic or optical card, propagation medium or other type of machine suitable for storing electronic instructions. Readable medium. For example, the present invention can be downloaded as a computer program that can be accessed via a communication link (10) such as a 'data machine or network connection' by means of a carrier or other propagating medium. It is transmitted from a remote computer (for example, a server) to a requesting computer (for example, a client). It should also be understood that the elements of the @μ &> Λ ± sentence can also be used as computer programs. 139840.doc -152- 200942305 The product can be included, and the instructions can be used to program the electric cat (for example, to handle crying or other electronic devices). Execution-sequence operation. Alternatively, the operations are performed by a combination of hardware and software. The machine-readable medium may include "not limited to" a flexible disk, a compact disc, a CD_R〇M, and a magneto-optical disc, a job, MM. , EPROM, E EPR〇M 'Magnetic or optical card, media or other type of media/machine readable medium suitable for storing electronic instructions. Example Φ The elements of the subject matter disclosed by Lu's can be downloaded as a computer program product. The program can be transmitted via a communication link (for example, a data modem or a network connection) by means of a data signal embodied in a carrier wave or other communication medium. And from the remote computer or electronic device to the request process. The subject matter disclosed is described in connection with the specific embodiments, and numerous modifications and variations are Therefore, the description and drawings are to be regarded as illustrative and not restrictive. Figure 1 illustrates the architecture of a prior art video game system. 2a-2b illustrate a high order system architecture in accordance with an embodiment. Figure 3 illustrates the actual, rated, and required data rate for communication between the client and the server. The figure illustrates a colocation service and user used in accordance with the embodiment. Figure 4b illustrates an exemplary delay associated with communication between the client and the colocation service. The figure illustrates a client device in accordance with an embodiment. 139840.doc • 153- 200942305 Figure 4d illustrates a client device in accordance with another embodiment. Figure 4e illustrates an example block diagram of the client device of Figure 4C. Figure 4f illustrates an example block diagram of the client device of Figure 4d. Figure 5 illustrates an example of video compression that may be used in accordance with an embodiment. Figure 6a illustrates an example of video compression that can be used in another embodiment. Figure 6b illustrates the peak in the data rate associated with transmitting a low complexity, low motion video sequence. Figure 6c illustrates the peak in the data rate associated with transmitting a high complexity, high motion video sequence. Figures 7a through 7b illustrate an example video compression technique used in an embodiment. Figure 8 illustrates an additional example video compression technique used in an embodiment. Figures 9a through 9c illustrate example techniques for use in mitigating data rate peaks in an embodiment. Figures 10a through 1B illustrate an embodiment of efficiently packaging a block of image images within a packet. Figures 11a through 11d illustrate an embodiment using a forward error correction technique. Figure 12 illustrates an embodiment of compression using a multi-core processing unit. Figures 13a through 13b illustrate geolocation and communication between colocation services in accordance with various embodiments. Figure 14 illustrates an exemplary delay associated with communication between the client and the colocation service. 139840.doc -154· 200942305 Figure 15 illustrates an example host hosting service server central architecture. Figure 16 illustrates an example screen shot of an embodiment of a user interface including a plurality of live video windows. • Figure 17 illustrates the user interface of Figure 16 after selecting a particular video window. • Figure 1 8 illustrates the user interface of Figure 17 after zooming in on a particular video window. Figure 196 shows an example collaborative user video material overlaid on a multiplayer game screen. Figure 20 illustrates an example user page for a game player on a colocation service. Figure 21 illustrates an example 3D interactive advertisement. • Figure 22 illustrates an example sequence of steps for surface capture from a live broadcast performance to produce a photo-realistic image with a textured surface. Figure 23 illustrates an example user interface page that allows selection of linear media content. φ Figure 24 is a graph showing the amount of time elapsed and the connection speed before the live web page is live. [Main component symbol description] 100 CPU/GPU 101 Random access memory (RAM) 102 Display device (SDTV/HDTV or 1^ brain monitor;) 103 Hard disk drive 104 Optical media drive/Optical media/CD player 105 Road connection 139840.doc -155- 200942305 106 205 206 210 211 220 221 222 301 302 303 401 402 403 404 405 406 406a 406b 409 410 412 413 Game controller client device / client / client platform Internet host generation Management Service/Host Hosting System User Location Game or Application Software Developer Input Device Display Device/Monitor or TV Nominal Maximum Data Rate Data Rate Required for Maximum Data Rate Hosting Service Control System/Host Generation Tube Service Control Server Servo Storage Area Network (SAN) Video Compression Logic / Video Compressor Independent Redundant Array (RAID) Control Signal Control Signal Control Signal Routing Logic Network Low Latency Video Decompression Logic Control Signal Logic 139840.doc -156.

200942305 415 Client/Client Device/Customer Computer 420 Display Device 421 Input 422 Display Device/Display » 440 Internet Bypass Service* 441 Presence Point 442 WAN Interface e 443 Firewall/Router/NAT (Network Address Translation) Device 444 WAN Interface 451 One-way transmission time for transmitting control signals 452 Round trip routing/user field via user location - Route 453 Arrow / Round trip delay / User ISP 454 Universal Internet Delay 455 One-way routing 456 456 Maximum one-way delay 457 arrow/compression 458 Video decompression 462 Ethernet jack/internet connection 463 HDMI (High Definition Multimedia Interface), connector* 464 Display capability* 465 Client device / Client 466 Eyewear 468 Display Device / SDTV (Standard Definition TV) or HDTV 139840.doc -157- 200942305 (High Definition TV) / Monitor 469 Input Device 475 Client Device 476 Flash Memory 477 Video Camera 478 Display Device 479 Bluetooth Input Device / Wired USB Input Device 480 Bus 481 B 482 WiFi network interface subsystem 483 to control CPU / processor 484 controls the Bluetooth subsystem 485 USB interface 486 video decompressor 487 video subsystem 488 outputs the compressed audio subsystem 489 Solutions Audio output subsystem 490 HDMI interface

491 DVI-I 492 S-Video 493 Composite Video 494 Digital Interface 495 Stereo Analog Interface 497 Ethernet 139840.doc -158- 200942305 499 Power 501 Uncompressed Video Frame 502 Uncompressed Video Frame. 503 Uncompressed video frame 511 • Compressed frame 512 Compressed frame 513 Compressed frame 520 Compressed logic A w 559-561 Uncompressed video frame 611 I Frame 612-613 P Box. 620 Compression 621 Rated Maximum Data Rate 622 Available Maximum Data Rate / Available Data Rate 623 I Required Peak Data Rate / Peak Data Rate ^ 624 Video Stream Data Rate / Video Stream Data Rate Sequence 633 I Frame Peak 634 Video Stream Data Rate Sequence / Video Stream Data Rate 635 P Frame Peak 636 P Frame Peak ' 670 B Frame • 671 I Frame 701-760 Uncompressed Frame 711-770 R Frame 139840.doc • 159· 200942305 721-770 Uncompressed frame 731-780 R frame 805 Mobile character 811 R frame 812 R frame 922 Available maximum data rate 934 Video stream data rate 941 Peak value Material rate/peak maximum data rate 942 2 times peak data rate 943 3 times peak data rate 944 4 times peak data rate 952 frame 2 times peak 953 flattened 2 times peak 954 frame 4 times peak 955 4 times flattened Peak 961 uncompressed frame 1 962 Uncompressed frame 2 963 Uncompressed frame 3 964 Uncompressed frame 4 965 Uncompressed frame 5 966 Uncompressed frame 6 967 Uncompressed frame 7 968 Uncompressed frame 8 969 Uncompressed frame 9 139840.doc - 160- 200942305 970 Uncompressed frame 1 ο 981 Compressed frame 1 982 Compressed Box 2. 983 Compressed frame 3 985 • Compressed frame 5 986 Compressed frame 6 987 Compressed frame 10 991 Transmission time (xmit time) ® 992 Transmission time 993 Transmission time (2 times Peak) 995 Transmission time 996 Transmission time (4 times peak) 997 Transmission time 1001-1005 Packet 1010 Image block package package logic _ 1100 I image block 1101 Forward error correction code (FEC) 1103 I image block 1104 Forward error correction code (FE C) 1105 Forward Error Correction Code (FEC) 1110 Audio Packet - 1111 FEC Code 1112 Audio Packet 1113 FEC Code 139840.doc -161 - 200942305 1120 1121 1122 1123 1200 1201- 1205 1300 1301 1501 1502 1511 1512 1515 1521 1529 1530 1539 1540 1550 1551 1552 1599 139840.doc User input command FEC code User input command FEC code multi-core and / or multi-processor 1208 core ' 4x2 configuration / uncompressed map pivot. Server center / colocation service network Incoming Internet Traffic _ Incoming Routing / Incoming Routing Network RAID Array RAID Array RAID Array / Delay Buffer. 1525 Application / Game Server Uncompressed Video / Audio Shared Video Compression / Shared Pool / Shared Hardware Compression / Common® Hard Compressor / Shared Video Compression Hardware Compressed Video / Audio Output Routing / Routing Routing Network Delay Buffer and / or Grouped Video / Feedback Loop · Path - Path Out Internet Service•162· 200942305 1600 “Thumbnail” Video Window 1601 Yellow Selection Box 1602 Game Information, 1700 Video Window 1800 t Arrow 1801 Overlay 1900 Video and Audio 2000 User's Video® 2001 Live Video 2002 Partner 2003 Video Clip/Self-review Clip/3D DVR Self-Puzzle 2004 Digital 2005 2400 Line 2401 1.5 Mbps Connection Speed _ 2402 ❹ 12 Mbps Connection Speed 2403 96 Mbps connection speed HS1-HS6 Colocation Service Server Center 139840.doc 163 -

Claims (1)

200942305 VII. Patent Application Range: 1. A device comprising: one or more servers supporting a server center; and an independent redundant disk array (RAID) storing the geometry of a complex scene body Shape, the RAID is coupled to one or more applications or servers and can be manipulated to interact in a live animation while streaming through the geometry = the geometry of the instant animation is related to the "batch -> lu A Servers 执 之 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -