JP2020017803A - Information processing unit, method, and program - Google Patents

Abstract

To generate a virtual viewpoint image of high picture quality, while suppressing expansion cost of transmission band.SOLUTION: Cameras are grouped for each photography region so that the transfer cost allocated to two transmission routes is standardized. In the case of a photography region 30, cameras 120 and 122 belong to group 1, and cameras 121 and 123 and 124 belong to group 2 for allocating the transfer cost as uniform as possible. Similarly, a photography region 31 is grouped to group 3 and group 4 so as to allocate the transfer cost as uniform as possible. Thereafter, a determination is made for each photography region, to which of the transmission routes A, B each group is allocated. As a result, the cost 405 of each transmission route aggregating the transfer costs by combining the groups allocated to the same transmission route of the photography regions 30, 31 is standardized and equalized.SELECTED DRAWING: Figure 4

Description

  The present invention relates to transmission of image data.

  In recent years, a technique of installing a plurality of cameras at different positions and generating a virtual viewpoint image using a plurality of viewpoint images obtained from each camera has been attracting attention. The virtual viewpoint image generated from the multiple viewpoint image is an image generated by an end user and / or an appointed operator or the like freely operating the position and orientation of a camera (virtual camera) corresponding to the virtual viewpoint, It is also called a free viewpoint image or an arbitrary viewpoint image. The virtual viewpoint image may be a moving image or a still image. The virtual viewpoint image can give the user a higher sense of realism as compared with a normal image.

  A virtual viewpoint image is typically generated by collecting image data obtained by photographing a subject (hereinafter referred to as a foreground object) from a plurality of directions on a server and performing image processing using the collected image data. Is done. Patent Literature 1 discloses a technique in which content data including a plurality of image data from different viewpoints and viewpoint information is stored in a storage medium, the image data is read from the storage medium, and a playback apparatus generates a virtual viewpoint image. ing.

  Further, there is a high demand for generating a virtual viewpoint image at a timing near real time in watching sports. In order to meet such demands, it is necessary to generate a virtual viewpoint image in a short time from shooting. In order to generate a high-quality virtual viewpoint image at a timing close to real time, it is necessary to transfer a plurality of image data from a plurality of cameras to a server without delay and aggregate them.

JP 2014-215828 A

  On the other hand, as the resolution of each image, a higher resolution such as 8K has been used as the performance of the camera has improved, and the size of image data to be transferred has been increased. Therefore, when transferring high-resolution image data from a plurality of cameras to a server, the transmission bandwidth of some transmission paths may be insufficient, and a virtual viewpoint image or a free viewpoint image may not be stably generated in a short time from shooting. there were. As a method of improving the shortage of the transmission band, it is conceivable to extend the band of the transmission path or reduce the data size by reducing the resolution and / or the frame rate of the image data to be transferred.

  However, there is a problem that extending the transmission band is costly, and lowering the resolution and / or frame rate of the image from each camera degrades the image quality of the virtual viewpoint image.

  Therefore, an object of the present invention is to be able to generate a virtual viewpoint image with high image quality while suppressing the expansion cost of the transmission band.

  The present invention relates to a transmission path used for data transfer of each of the image data in a system for transferring a plurality of image data obtained by performing predetermined image processing to photographed image data of a plurality of cameras to a server device through a plurality of transmission paths. An information processing apparatus for determining, for each of the plurality of cameras, a first calculation unit that calculates a data size when the predetermined image processing is performed on the captured image data, based on the data size. For each camera group that outputs the photographed image data whose data size changes in synchronization with the predetermined image processing, the usage rate of the plurality of transmission paths is leveled so that the use rates of the plurality of transmission paths are equalized. An information processing apparatus comprising: a determination unit that determines a transmission path used for data transfer.

  According to the present invention, it is possible to generate a virtual viewpoint image with high image quality while suppressing the cost of expanding the transmission band.

FIG. 1 is a configuration diagram of an image data aggregation system according to a first embodiment. FIG. 3 is a functional block diagram of a data aggregation server A100. FIG. 4 is a diagram illustrating the principle of a method for calculating the data size of a foreground object area. 9 is a table for explaining a process of determining a route from a data size ratio of a foreground object area according to the first embodiment. FIG. 2 is a functional block diagram of the image processing apparatus. Shows the position of the foreground object at time t _1. Shows the position of the foreground object at time t _2. FIG. 6 is a diagram showing a time series of a change in transfer data size per unit time for each shooting area of foreground image data transferred from the image processing apparatus. 9 is a flowchart illustrating a route selection process. FIG. 9 is a configuration diagram of an image data aggregation system according to a second embodiment. 9 is a table for explaining a process of determining a route from a data size ratio of a foreground object area according to the second embodiment. FIG. 9 is a configuration diagram of an image data aggregation system according to a third embodiment. 16 is a table for explaining a process of determining a route from a data size ratio of a foreground object area according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The following embodiments do not limit the present invention, and all combinations of features described in the present embodiments are not necessarily essential to the solution of the present invention.

Embodiment 1

  FIG. 1 shows a configuration diagram of an image data aggregation system according to the first embodiment of the present invention. The present embodiment is a system in which a plurality of cameras 120 to 124 and 140 to 143 are installed, and image data captured by the plurality of cameras 120 to 124 and 140 to 143 are collected in a data collection server A100 and a data collection server B101. Examples of a place where the cameras 120 to 124 and 140 to 143 are installed include facilities such as a stadium (stadium) and a concert hall.

  The link portion of the network 10 is a communication line, and may be, for example, Ethernet (registered trademark), a wireless LAN, an optical fiber channel, or the like. The link portion of the network 10 connects the data aggregation server A100, the data aggregation server B101, the image processing apparatuses 110 to 114, 130 to 133, and the storage 102, which will be described later, in a ring connection, and enables data transfer between the nodes. In the present embodiment, the image processing apparatuses 110 to 114 and 130 to 133 are connected by a ring, but may be connected by a bus, a daisy chain, a star, a tree, or a mesh, or are not shown in FIG. May be connected via the relay device.

  The foreground objects 20 and 21 are shooting targets for generating a virtual viewpoint image. In FIG. 1, the number of foreground objects is two.

  The photographing regions 30 and 31 are photographing regions photographed by the cameras 120 to 124 and 140 to 143. The first camera groups 120 to 124 face the same gazing point in the shooting area 30, and similarly, the second camera groups 140 to 143 face the same gazing point in the shooting area 31. Here, the gazing point indicates an arbitrary point (a point in the line of sight of the camera) in the imaging space that the camera gazes at. Further, in the present embodiment, two shooting areas are shown, but the number of shooting areas may be three or more, and an arbitrary place in the stadium other than the illustrated place may be set as the shooting area.

  The data aggregation server A100 and the data aggregation server B101 are information processing devices such as PCs and server devices, and are server devices that aggregate data for creating virtual viewpoint images from image processing devices 110 to 114 and 130 to 134 described below. It is. It also has a function of creating a virtual viewpoint image using the aggregated data. In the present embodiment, two data aggregation servers are installed, but the number of installed data aggregation servers may be one or three or more.

  The storage 102 is a device that stores data aggregated by the data aggregation server A100 and the data aggregation server B101. The data aggregation server A100 and the data aggregation server B101 can write data to the storage 102 and read data from the storage 102.

  The cameras 120 to 124 and 140 to 143 are digital cameras, digital video cameras, and surveillance cameras, and are devices that capture images, videos, and sounds. Further, in the present embodiment, the cameras 120 to 124 and 140 to 143 are installed on the outer periphery of the stadium so as to surround the imaging areas 30 and 31. However, the positions at which the cameras 120 to 124 and 140 to 143 are arranged are not limited to the outer periphery of the stadium, but may be installed at any locations where the photographing areas 30 and 31 can be photographed. Further, in the present embodiment, as described above, the first camera groups 120 to 124 image the imaging region 30, and the second camera groups 140 to 143 image the imaging region 31. In the present embodiment, the image data constituting the image is aggregated, but it is also possible to aggregate the audio data.

  The image processing apparatuses 110 to 114 and 130 to 134 transmit data obtained by subjecting the captured image data of the cameras 120 to 124 and 140 to 143 to predetermined processing to the data aggregation server A100 or / and the data aggregation server B101 via the network 10. Device to transfer to Further, it has a function of extracting data necessary for a virtual viewpoint image from captured image data of a camera. For example, only the foreground object area is extracted from the captured image data, and only the foreground image data, which is the image data obtained by extracting only the foreground object area, is transferred to the data aggregation server A100 and / or the data aggregation server B101. Similarly, for voice, audio data near the shooting area 30 and the shooting area 31 is extracted, and only the extracted voice data near the shooting area 30 and the shooting area 31 is transferred to the data aggregation server A100 or / and the data aggregation server B101.

  Next, FIG. 2 shows functional blocks showing the internal structure of the data aggregation server A100 and the data aggregation server B101. The data transmission / reception unit 200 has a function of connecting to the network 10 and transmitting / receiving data to / from the image processing apparatuses 110 to 114 and 130 to 133 and the storage 102.

  The bottleneck specifying unit 201 specifies a bottleneck point on the transmission path of data transferred from the image processing apparatuses 110 to 114 and 130 to 133, and determines a bottleneck throughput (a data size that can be processed per unit time). ) Is measured. For example, a location that is likely to be a bottleneck is a device (for example, a specific image processing device, a HUB, a specific cable, or the like) having a low data transfer capability, that is, a low throughput or transmission band. In addition, in the configuration shown in FIG. 1, in addition to the above-described devices, specific portions of the network 10, such as between the image processing device 133 and the data aggregation server A100 or between the image processing device 110 and the data aggregation server B101. Is likely to be a bottleneck. This is because the data transferred from the image processing apparatuses 110 to 114 and 130 to 133 is aggregated around the data aggregation server A100 of the network 10, so that the transmission band is likely to be tight. For example, the bottleneck specifying unit 201 may transmit predetermined data from the image processing apparatuses 110 to 114 and 130 to 133 and specify the bottleneck based on a response to the data.

  The photographing area information acquiring unit 202 has a function of acquiring photographing area information indicating which of the photographing areas 30 and 31 each of the cameras 120 to 124 and 140 to 143 is photographing.

  The camera information acquisition unit 203 has a function of acquiring camera information and the like regarding the cameras 120 to 124 and 140 to 143. The camera information may include the resolution and size of the image sensor of the camera, the focal length of the lens, the distance from the camera to the foreground object, the ISO sensitivity, and the like.

  Note that the photographing area information acquiring unit 202 and the camera information acquiring unit 203 may be configured to read out the photographing area information or camera information stored in the built-in storage device, or an external device that stores the photographing area information or camera information. A configuration in which data is read from a storage device may be adopted.

  The data size calculation unit 204 has a function of calculating a data size (for example, the number of pixels) of a foreground object area in an image captured by each camera. The data size of the foreground object area is calculated based on foreground image data acquired from the image processing apparatuses 110 to 114 and 130 to 133 or information acquired by the imaging area information acquisition unit 202 and the camera information acquisition unit 203. FIG. 3 is a diagram illustrating the principle of the method for calculating the data size of the foreground object area. The data size of the foreground object area can be calculated from the number of pixels of the foreground object area included in the captured image data of the cameras 121 and 124. The captured image 301 is a captured image of the camera 121, and the captured image 302 is a captured image of the camera 124. Since the camera 121 is closer to the shooting area 30 than the camera 124, the number of pixels of the foreground object area 304 shown in the shot image 301 of the camera 121 is larger than that of the foreground object area 305 shown in the shot image 302 of the camera 124. The data size is also large.

  The number of pixels in the foreground object area that determines the data size of the foreground image data can be calculated from the foreground image data received from the image processing device, but can also be calculated based on the camera information. The proportion of the foreground object area in the captured image can be calculated from the size of the foreground object in the imaging space, the distance from the camera to the foreground object, and the angle of view of the camera. Then, the number of pixels of the foreground object area can be calculated from the ratio of the foreground object area in the captured image and the number of pixels of the entire captured image. Since the angle of view of the camera can be calculated from the size of the image sensor of the camera and the focal length of the lens of the camera, the number of pixels of the foreground object area, that is, the data size of the foreground object area can be calculated based on the camera information.

  In the present embodiment, it is assumed that the gradations of the images are all the same and have the same predetermined gradation, for example, RGB 16 bits. Further, since the size of the foreground object in the imaging space is the same if the photographing area is the same, it is not necessary to calculate the data size ratio of the foreground object area described later. However, when the ratio of the data size of the foreground object area can be compared between the imaging areas, the size of the foreground object in the imaging space needs to be the same between the imaging areas.

  Further, the data size calculation unit 204 calculates the ratio of the data size of the foreground object region of the same foreground object between cameras from the ratio of the foreground object region in the captured image of each camera or the number of pixels thereof. In the case of FIG. 3, the foreground object area 304 in the captured image 301 of the camera 121 and the foreground object area 305 in the captured image 302 of the camera 124 have, for example, 100 and 50 pixels, respectively. In this case, the data size ratio of the foreground object area is 2: 1. The data size ratio of the foreground object area may be calculated based on the foreground image data received from the image processing apparatus, or may be calculated based on the data size of the foreground object area calculated based on the camera information. Similarly, the cameras 120, 122 and 123 for photographing the photographing area 30 and the cameras 140 to 143 for photographing the photographing area 31 also calculate the data size ratio of the foreground object area. When calculating the ratio of the data size of the foreground object area based on the proportion of the foreground object area in the captured image, it is assumed that the resolutions of the captured images of all cameras are equal and the number of pixels of the captured images is the same. Note that a captured image may include a camera having a different number of pixels, such as a camera having different resolution settings, for example, a system in which a 4K camera and an 8K camera are mixed. In that case, the ratio of the data size of the foreground object region is calculated based on the number of pixels of the foreground object region instead of the ratio of the foreground object region in the captured image.

  When the image processing apparatuses 110 to 114 and 130 to 133 perform image compression processing on the foreground image data and transfer the data, the compression ratio may vary depending on the brightness in the stadium. For example, in some cases, light is applied to a foreground object from a specific direction, and a camera in which the light is forward light and a camera in which the light is backlight is generated. In this case, even with the captured image of the same foreground object, the captured image of the camera at the backlight position has more noise than the captured image of the camera at the front light position, and the compression ratio is reduced. Therefore, when the brightness of the foreground object area differs for each camera, the data size ratio of the foreground object area is multiplied by the compression efficiency of each foreground image data by the ratio or the number of pixels of the foreground object area in the image captured by each camera. It may be a ratio of the calculated values.

  The transmission path determination unit 205 has a function of determining a transmission path through which the image processing apparatuses 110 to 114 and 130 to 133 transfer foreground image data via the network 10. The transmission path determination unit 205 determines a transmission path to be used for each image processing apparatus such that the usage rate (transfer data size per unit time / transmission band) of a plurality of transmission paths is equalized. At this time, the transmission path determination unit 205 calculates the usage rate of the transmission path based on the data size ratio of the foreground object area calculated by the data size calculation unit 204. However, when the data size calculation unit 204 calculates the data size ratio of the foreground object area from the foreground image data, the first foreground image data for calculating the data size ratio of the foreground object area is the transmission data specified in advance in the initial setting. Transferred using a route.

  FIG. 4 shows a table for explaining a process of determining a transmission path from the data size ratio of the foreground object area. In the present embodiment, there are two transmission paths for transferring image data to the data aggregation server A100 clockwise or to the data aggregation server B101 counterclockwise on the network 10 shown in FIG. The clockwise direction is the transmission path A, and the counterclockwise direction is the transmission path B.

  First, as shown in a column 400, the data size ratio of the foreground object area for each camera calculated by the data size calculation unit 204 is listed. In this case, the transfer cost 403 of the shooting area 30 is “32” which is a value obtained by adding the data size ratios of the foreground object areas of the first camera groups 120 to 124 shooting the shooting area 30. The transfer cost 404 of one shooting area 31 is “26” which is a value obtained by adding the data size ratios of the foreground object areas of the second camera groups 140 to 143 shooting the shooting area 31.

  Next, cameras are grouped in a column 401 for each imaging region so that the transfer costs allocated to the two transmission paths are equalized. In the case of the photographing area 30, the camera 120 and the camera 122 are group 1 and the camera 121, the camera 123, and the camera 124 are group 2 in order to allocate transfer costs as evenly as possible. Similarly, the photographing areas 31 are grouped into groups 3 and 4 so that the transfer costs are allocated as evenly as possible.

  Although a bottleneck device that reduces the transfer rate on the transmission path is not described here, if there is a bottleneck device on the transmission path, the bottleneck throughput is reduced by the transmission bandwidth of the transmission path. And When the transmission band differs for each transmission path, the transfer cost is weighted for each transmission path, and when the weighted transfer costs become equal, the usage rate of each transmission path is equalized. For example, if the throughput of the bottleneck device existing on the transmission path A is 3/4 of the transmission band of the other transmission path B, the transmission cost of the transmission path A is 4% of the data size ratio of the foreground object area. A value obtained by multiplying by / 3 may be used. By grouping the transmission paths A and B based on the weighted transfer costs so that the transmission costs of the transmission paths A and B are equal, the usage rates of the transmission paths A and B can be equalized.

  Next, as shown in a column 402, it is determined which of the transmission paths A and B each group is assigned to for each imaging region. As a result, the cost 405 for each transmission path obtained by combining the groups assigned to the same transmission path in the imaging regions 30 and 31 and summing the transfer costs is equalized. In the case shown in FIG. 4, the group 1 of the shooting area 30 and the group 3 of the shooting area 31 are allocated to the transmission path A, and the group 2 of the shooting area 30 and the group 4 of the shooting area 31 are allocated to the transmission path B. .

  Note that the method of determining the transmission path described here is an example, and the transmission path may be determined or specified using another method. For example, the transmission path length and transmission delay between each image processing apparatus and each data aggregation server may be considered. For example, the image processing apparatus 133 may be preferentially assigned to the transmission path A because the transmission path length with the data aggregation server A100 is short.

  The virtual viewpoint image generation unit 206 has a function of generating virtual viewpoint contents using the image data received from the image processing device or / and the storage 102 and virtual viewpoint information specified by the user.

  In the present embodiment, the various function processing units 200 to 206 are integrated into the data aggregation server A100. However, the present invention is not limited to this, and even if these function processing units 200 to 206 are distributed to a plurality of servers. Good.

  Next, FIG. 5 shows functional blocks illustrating the internal structure of the image processing apparatuses 110 to 114 and 130 to 133.

  The image acquisition unit 501 has a function of connecting to a camera via a cable or the like and receiving image data from the camera. Further, it may have a function of changing various parameters of the camera and controlling a shooting operation. An SDI (Serial Digital Interface) cable can be used as the cable. The cable may be an HDMI (registered trademark) (High-Definition Multimedia Interface) cable, a CoaXPress (registered trademark) cable, or the like.

  The foreground object area extraction unit 502 has a function of extracting a foreground object area from the captured image data acquired by the image acquisition unit 501 and generating foreground image data.

  The data transmission / reception unit 503 has a function of transmitting metadata such as foreground image data and the data size of the foreground image data to the data aggregation server. In addition, a command for designating a transmission path for transferring data from the data aggregation server is received, and the image data is transferred on the designated transmission path.

  The data relay unit 504 has a relay function of transferring image data received from another image processing device to the next image processing device or data aggregation server.

Next, a problem in transmitting image data will be described with reference to FIGS. 6, 7, and 8. FIG.
6 and 7 are views showing the positions of the foreground object at times t ₁ and t ₂ , respectively. FIG. 8 is a diagram showing, in chronological order, a change in the transfer data size per unit time for each shooting area of the foreground image data transferred from the image processing apparatus. At time t ₁ shown in FIG. 6, there are five foreground objects in the shooting area 31 and one foreground object in the shooting area 30. At this time, the transfer data amount per unit time for each shooting area is 13 Gbps (800) for the shooting area 31 and 1 Gbps (801) for the shooting area 30. On the other hand, at time t ₂ shown in FIG. 7, there is one foreground object in the photographing area 31 and five foreground objects in the photographing area 30. At this time, the transfer data amount per unit time for each shooting area is 12 Gbps (802) for the shooting area 30 and 3 Gbps (803) for the shooting area 31.

Here, a case is considered where the transmission bands of the transmission paths A and B are 10 Gbps and 8 Gbps, respectively, and only one of the transmission paths is used for each imaging region. In such a case, at time t ₁ , since the transfer data size per unit time of the imaging region 31 is 13 Gbps, the transmission band is insufficient when either of the transmission paths A and B is selected. Not all image data can be transferred in real time. Similarly, at time t ₂ , since the transfer data size per unit time of the image capturing area 30 is 12 Gbps, the transmission bandwidth of both the transmission paths A and B is insufficient, so that all the image data of the image capturing area 30 are transferred in real time. Cannot be forwarded to

  In such a case, in order to transfer all the image data for each shooting area in real time, the transfer data size per unit time is adjusted according to the transmission band of the transmission path, for example, by lowering the resolution or lowering the frame rate. There is a way to make it smaller. However, this method has a problem that the image quality of the virtual viewpoint image is reduced.

Further, if the transmission paths A and B are dynamically switched for each photographing area, if the foreground object moves between the photographing area 30 and the photographing area 31 violently, the transmission path switching cannot be made in time and one of the transmission paths A May be congested. For example, all image data size for each area to be photographed, a case where the photographing region 30 time t ₃ is 10Gbps, next imaging region 31 7 Gbps, imaging region 30 at time t ₄ is 7 Gbps, imaging region 31 is 10Gbps . The time t ₅ in the photographing region 30 is 10 Gbps, consider a case where the imaging area 31 is 7 Gbps. At this time, to avoid congestion, at time t ₃ , the transmission path A in the imaging area 30, the transmission path B in the imaging area 31, the transmission path B in the imaging area 30 at time t ₄ , and the transmission path B in the imaging area 31. A needs to be allocated. Furthermore, the transmission path A to the imaging region 30 at time t _5, it is necessary to allocate a transmission path B the imaging area 31. If the assignment of the transmission path for each imaging region is not performed with good timing, the transmission path B will be congested.

  As described above, if much of the data allocated to one transmission path changes synchronously, the fluctuation of the usage rate of the transmission path becomes large, and a situation occurs in which the transmission band tends to be insufficient.

On the other hand, in the present invention, data whose data size changes synchronously is allocated to the plurality of transmission paths so that the usage rates of the plurality of transmission paths are equalized. Therefore, in the present embodiment, as described above, foreground image data is allocated to the two transmission paths A and B for each shooting region. Thus, in the present embodiment, even when a large deviation occurs in the data size of foreground image data for each camera group having the same shooting area, the usage rates of a plurality of transmission paths are leveled, and the entire transmission path is used efficiently. be able to. Explaining the situation at time t ₁ shown in FIG. 8 as an example, the transfer data size per unit time of the imaging area 31 is significantly larger than the transfer data size per unit time of the imaging area 30. However, in the present embodiment, it is possible to prevent the usage rate of only one transmission path from increasing. Therefore, both transmission bands of the transmission paths A and B can be used without waste. As a result, the data size that can be transferred to the data aggregation server in real time can be increased without lowering the resolution or frame rate and without congesting the transmission paths A and B.

Also in the time t ₃ ~t _5, the transmission path A, the change in the transfer data size per unit time B, the magnitude corresponding to the difference between the change in the transfer data size per unit time of the imaging region 30, 31 It can be suppressed. Therefore, stable transfer is enabled without congestion in either transmission path.

  Next, FIG. 9 shows a flowchart illustrating the route selection processing. In step S901, the bottleneck identification unit 201 identifies a bottleneck existing on a route of the network 10. In the present embodiment, since the network 10 is configured as a ring type, data from the image processing apparatuses 110 to 114 and 130 to 133 are sequentially aggregated and transferred to the data aggregation server A100 and the data aggregation server B101. Therefore, when the transmission bandwidth is insufficient between the image processing apparatus 133 and the data aggregation server A100 and between the image processing apparatus 110 and the data aggregation server B101, these portions become bottlenecks. Also, when the data relay unit 504 of each image processing apparatus, the HUB installed between the image processing apparatuses 110 to 114 and 130 to 133, the data aggregation server A100, and the data aggregation server B101 have a lower throughput than the network 10, Becomes a bottleneck. Hereinafter, “step S〜” is abbreviated as “S〜”.

  Next, in S902, the data size calculation unit 204 determines the foreground image data from the image processing apparatuses 110 to 114 and 130 to 133 and / or the camera information of the cameras 120 to 124 and 140 to 143 acquired by the camera information acquisition unit 203. To get. Then, the data size calculation unit 204 estimates the data size ratio of the foreground object area from the acquired foreground image data and / or camera information. Further, the data size calculation unit 204 obtains, from the shooting area information obtaining unit 202, shooting area information to be shot by each camera. In the present embodiment, the first camera groups 120 to 124 image the imaging region 30, and the second camera groups 140 to 143 image the imaging region 31.

  Next, in S903, it is determined whether the information acquired in S901 and S902 has been updated. If none of the information has been updated, the process returns to step S901. If any information has been updated, the process proceeds to step S904.

  Next, in step S904, the transmission path determination unit 205 uses the information acquired in steps S901 and S902 to transfer the foreground image data to the data aggregation server A100 and the data aggregation server B101 as described above. Is determined. In other words, the usage rates of a plurality of transmission paths are leveled for each camera group that outputs captured image data that is the source of data whose data size changes synchronously, and in this embodiment, for each camera group that has the same imaging area. The transmission path used for the transfer of each data is determined.

  Next, in step S905, it is determined whether the setting of the route determined in step S904 can be changed for each image processing apparatus. If the setting cannot be changed, the process returns to step S901 or waits for a certain period of time to confirm whether the setting can be changed again. If the setting can be changed, the process advances to step S906.

  Next, in S906, an instruction to change the transmission path determined in S904 is issued to each image processing apparatus. When a change instruction is issued, an instruction including a time for performing the change is issued to each image processing apparatus. Each image processing device changes the route at the designated time. A change instruction may be issued to each image processing apparatus immediately without including the time for making the change. In this case, it is desirable to start from a location where the influence on the bottleneck location is small.

  In this embodiment, the bottleneck identification unit 201, the imaging region information acquisition unit, the camera information acquisition unit 203, the data size calculation unit 204, and the transmission path determination unit 205 are configured by the data aggregation server A100 and / or the data aggregation server B101. The configuration was provided. However, some or all of these functions may be implemented in an information processing device different from the data aggregation server A100 and the data aggregation server B101. The information processing device may be a server device such as the data aggregation server A100 and / or the data aggregation server B101 or a control device that controls the server device, or a general-purpose information processing device such as a PC separate from the server device. There may be.

Embodiment 2

  The second embodiment is an image data aggregation system when the topology of the network 10 is different from that of the first embodiment.

  FIG. 10 is a configuration diagram of an image data aggregation system according to Embodiment 2 of the present invention, and FIG. 11 is a diagram illustrating a process of determining a transmission path. The network 10 of the present embodiment is composed of four cascade networks, and is composed of four transmission paths (transmission path C, transmission path D, transmission path E, and transmission path F). The transmission path C connects the data aggregation server B101, the image processing device 111, and the image processing device 131 in a daisy chain type. The transmission path D connects the data aggregation server B101, the image processing device 110, and the image processing device 130 in a daisy chain type. The transmission path E connects the data aggregation server A100, the image processing device 132, and the image processing device 112 in a daisy chain type. The transmission path F connects the data aggregation server A100, the image processing device 133, the image processing device 114, and the image processing device 113 in a daisy chain type. In this embodiment, the wiring route between the image processing apparatuses 110 to 114 and 130 to 133, the data aggregation server A100, and the data aggregation server B101 is determined based on the transmission routes C to F determined in advance by the transmission route determination unit 205. Things. Specifically, as shown in FIG. 11, based on the data size ratio of the foreground object area shown in the column 1100, the cameras 120 to 124 and 140 to 143 are divided into four groups for each shooting area as shown in the column 1111. Grouping. Then, as shown in a column 1112, one of the groups of the photographing areas 30 and one of the groups of the photographing areas 31 are combined, and at this time, it is preferable that the transmission paths C to F be equalized so that the transfer costs of the transmission paths C to F are equalized. Assign to one of F. If there is room in the transmission bands of the transmission paths C to F, the image data may not be leveled, and the neighboring image processing apparatuses may be connected in consideration of ease of installation.

Embodiment 3

  Embodiment 3 is an image data aggregation system in the case where the topology of the network 10 is different from Embodiments 1 and 2.

  FIG. 12 is a diagram illustrating a system configuration according to the third embodiment, and FIG. 13 is a diagram illustrating a process of determining a transmission path. The network 10 of the present embodiment includes two ring networks (10a, 10b), and the network 10a includes a clockwise transmission path G and a counterclockwise transmission path H. Similarly, the network 10b includes a clockwise transmission path I and a counterclockwise transmission path J. In the present embodiment, one of the groups of the imaging regions 30 and one of the groups of the imaging regions 31 are combined without considering the cable cost. Specifically, as shown in FIG. 13, based on the data size ratio of the foreground object area shown in column 1300, cameras 120 to 124 and 140 to 143 are grouped into four groups for each shooting area as shown in column 1311. Grouping. Then, as shown in a column 1312, one of the groups of the photographing areas 30 and one of the groups of the photographing areas 31 are combined. At this time, it is preferable that the transmission paths G to J be equalized so that the transfer costs of the transmission paths G to J are equalized. Assign to one of J. If there is a margin in the transmission bandwidth of the transmission paths G to J, the transmission paths are not leveled, and the transmission paths are determined in consideration of the distances between the image processing apparatuses 110 to 114 and 130 to 133 and the data aggregation server A100 and the data aggregation server B101. You may decide.

Embodiment 4

  In the first to third embodiments, the data whose data size changes synchronously is foreground image data obtained by extracting a foreground object region from photographed image data from a group having the same photographing region. On the other hand, in the present embodiment, captured image data in which the illuminance of the foreground object changes in synchronization or compressed image data obtained by compressing the foreground image data may be used. When the illuminance of the foreground object changes, the noise amount changes and the compression ratio also changes, so that the data size changes. Therefore, the captured image data in which the illuminance of the foreground object changes synchronously, the data size changes synchronously when compressed. Compressed image data may be allocated to each camera group that outputs captured image data in which such a compression ratio changes synchronously so that the usage rates of a plurality of transmission paths are equalized. The compression ratio may be calculated from the data size of the compressed image data before and after compression, or may be calculated from the ISO sensitivity of the camera. Note that the compressed image data whose compression ratio changes in synchronization may be data obtained by compressing foreground image data or data obtained by compressing photographed image data.

(Other embodiments)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

100, 101 Data aggregation server 204 Data size calculation unit 205 Transmission route determination unit

Claims (13)

Information for determining a transmission path to be used for data transfer of each of the image data in a system for transferring a plurality of image data obtained by performing predetermined image processing on captured image data of a plurality of cameras to a server device through a plurality of transmission paths A processing device,
A first calculating unit that calculates a data size when the predetermined image processing is performed on the captured image data for each of the plurality of cameras;
Based on the data size, for each camera group that outputs the captured image data that changes in synchronization with the data size when the predetermined image processing is performed, so that the usage rates of the plurality of transmission paths are leveled, Determining means for determining a transmission path used for data transfer of each of the image data;
An information processing apparatus comprising: Further comprising an area information acquisition means for acquiring imaging area information indicating the imaging area of each of the plurality of cameras,
The information processing apparatus according to claim 1, wherein the camera group is configured by cameras that photograph the same photographing region from different directions based on the photographing region information. The plurality of cameras includes a plurality of the camera group different in the shooting area,
The information processing apparatus according to claim 2, wherein the predetermined image processing is image processing for extracting a foreground object area from the captured image data.   4. The information processing apparatus according to claim 3, wherein the first calculator calculates the data size based on a ratio of the foreground object area to the captured image data.   5. The information processing apparatus according to claim 3, wherein the first calculation unit calculates the data size based on the number of pixels of the foreground object area occupying the captured image data. 6. For each of the plurality of cameras, further comprising camera information acquisition means for acquiring camera information including information on the resolution and size of the image sensor, the focal length of the lens, and the distance to the foreground object,
The information processing apparatus according to claim 2, wherein the first calculation unit calculates the data size using the camera information. The predetermined image processing is image compression processing,
2. The information processing apparatus according to claim 1, wherein the camera group includes cameras that output the captured image data in which a compression ratio of the image data changes synchronously. 3. A camera information acquisition unit for acquiring camera information including information on ISO sensitivity for each of the plurality of cameras,
The information processing apparatus according to claim 7, wherein the first calculation unit calculates the data size using the camera information.   The information processing apparatus according to claim 1, wherein the first calculation unit calculates the data size using the image data. The apparatus further includes a second calculating unit that measures a throughput of the bottleneck of the plurality of transmission paths and calculates a transmission band of the plurality of transmission paths based on the throughput of the bottleneck,
The said determination means determines the transmission path used for the data transfer of each said image data based on the said data size and the said transmission band, so that the usage rate of the said several transmission path may be equalized. Item 10. The information processing device according to any one of items 1 to 9.   The determining unit is configured such that, when a camera constituting a camera group that outputs the captured image data whose data size changes in synchronization with the predetermined image processing is changed, each of the changed camera groups is The information processing apparatus according to any one of claims 1 to 10, wherein a transmission path used for data transfer of the image data is determined. A method for determining a transmission path to be used for data transfer of each of the image data in a system for transferring a plurality of image data obtained by performing predetermined image processing to image data captured by a plurality of cameras to a server device through a plurality of transmission paths And
For each of the plurality of cameras, calculating a data size when the predetermined image processing is performed on the captured image data,
Based on the data size, for each camera group that outputs the captured image data that changes in synchronization with the data size when the predetermined image processing is performed, so that the usage rates of the plurality of transmission paths are leveled, Determining a transmission path to be used for data transfer of each of the image data;
A method comprising: A function of determining a transmission path to be used for data transfer of each of the image data in a system that transfers a plurality of image data obtained by performing predetermined image processing to image data captured by a plurality of cameras to a server device via a plurality of transmission paths. A program that realizes
A first calculating unit that calculates a data size when the predetermined image processing is performed on the captured image data for each of the plurality of cameras;
Based on the data size, for each camera group that outputs the captured image data that changes in synchronization with the data size when the predetermined image processing is performed, so that the usage rates of the plurality of transmission paths are leveled, Determining means for determining a transmission path used for data transfer of each of the image data;
A program characterized by functioning.

Images