JP7277173B2 - Imaging device with video distribution function - Google Patents

Description

The present invention relates to a technique for streaming distribution of moving images captured by a camera.

2. Description of the Related Art In recent years, streaming technology has been widely used in which video and audio data are decoded while being downloaded and played back. Also, as one of the streaming technologies, an IP video transmission technology is known in which a distributor encodes moving images and audio in real time and distributes them via an IP (Internet Protocol) network.

In IP video transmission technology, it is common to use RTP (Realtime Transport Protocol) using UDP (User Datagram Protocol) with an emphasis on real-time performance, and RTSP, which enables distribution control from the IP video receiving side. (Real Time Streaming Protocol) is used together to systematize distribution control. Patent Literature 1 listed below discloses IP video distribution technology using RTSP.

In addition, the web browser and application software of terminals such as PCs and tablets can be used to remotely operate the camera using API (Application Programming Interface) conforming to HTTP (HyperText Transfer Protocol), and to monitor the status by sending and receiving camera information. Remote operation technology for performing is generally known.

JP 2016-100651 A

When incorporating a remote control function into an imaging device that has a video distribution function by streaming so that remote control can be performed from another terminal, it is necessary to have a mechanism that allows both functions to be compatible and appropriately controls video distribution. Become.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to enable remote control from another terminal in an imaging apparatus having a moving image distribution function, and to appropriately control moving image distribution.

According to one aspect of the present invention, there is provided an imaging apparatus comprising imaging means capable of capturing moving images, communication means, and the imaging apparatus according to a remote control signal received from an external device via the communication means. a remote control function for operating a video image, setting means for performing settings related to a video distribution function for streaming video to an external device via the communication means, and control means, wherein the control means uses the setting means to When both the remote control function and the moving image distribution function are set to be activated, control is performed so that streaming distribution is set to OFF as the initial setting of the moving image distribution function.

To enable remote control from another terminal and appropriately control moving image distribution in an imaging device having a moving image distribution function.

It is a figure which shows an example of the moving image delivery system which concerns on embodiment of this invention. It is a figure showing an example of an imaging device concerning an embodiment of the present invention. It is a figure which shows an example of the remote control terminal which concerns on embodiment of this invention. It is a figure which shows an example of a screen display of the imaging device which concerns on embodiment of this invention. 4 is a flow chart showing an example of connection to an external device and distribution start processing of the imaging device according to the embodiment of the present invention. 6 is a flow chart showing an example of processing for connecting the remote control terminal to the imaging device and starting the operation screen according to the embodiment of the present invention. FIG. 4 is a diagram showing an example of screen display of the remote control terminal according to the embodiment of the present invention;

(Embodiment 1)
Exemplary embodiments of the invention will now be described in detail with reference to the drawings.

In the following description, a digital camera capable of transmitting moving image streaming data to an external device is used as an example of an imaging device having a moving image distribution function. can be applied. For example, mobile phones including smartphones, game consoles, tablet terminals, information terminals in the form of watches and glasses, medical equipment, cameras and servers for monitoring systems, in-vehicle equipment, server equipment and virtual machines that operate in data centers, etc. can do. A tablet terminal is used as an example of a remote control device for remotely controlling an imaging device, and a PC capable of receiving and reproducing streaming data is used as an example of an IP video receiving device. However, the devices described above may include, for example, smartphones, head-mounted displays, televisions, and devices applicable to imaging devices having the distribution function described above.

(Configuration of Streaming Distribution System 1000)
FIG. 1 is an example of a streaming distribution system (video distribution system) according to this embodiment.

First, the imaging device 2000 is a digital camera as an example. Also, the remote operation terminal 3000 is a tablet terminal as an example, and the IP video receiving terminal 4000 is a PC as an example.

The imaging device 2000 can be connected to a remote control terminal 3000 and an IP video receiving terminal 4000 via an access point (AP) 5000 via an IP network. The communication connection may be either wireless or wired.

(Configuration of imaging device 2000)
FIG. 2 is a functional block diagram showing an example of the configuration of the imaging device 2000. As shown in FIG.

Each functional block shown in FIG. 2 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or a programmable processor such as a CPU (Central Processing Unit) or MPU (Micro Processing Unit) may be implemented by software. may be implemented by executing It may also be implemented by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as main entities, the same hardware can be implemented as the main entities.

The control unit 2001 includes, for example, a CPU, and executes programs stored in the nonvolatile memory 2002 or recording medium 2012 to control each unit of the imaging apparatus 2000 . Also, the control unit 2001 controls each unit of the imaging device 2000 according to an operation signal from an operation unit 2005 that receives operations from the user.

The nonvolatile memory 2002 includes a ROM (Read Only Memory) composed of, for example, flash memory, and stores various programs and constants for the operation of the control unit 2001 . The volatile memory 2003 includes, for example, a RAM (Random Access Memory) such as a DRAM, and stores various content data such as moving image data, still image data, and audio data, or programs, variables, and working data necessary for the operation of the imaging apparatus 2000 . Store temporary data.

The input processing unit 2004 generates a control signal according to the user's operation received by the operation unit 2005 and supplies the control signal to the control unit 2001 . An operation unit 2005 is an input device for accepting user operations, and includes, for example, a character information input device such as a keyboard, and a pointing device such as a mouse and a touch panel. Note that the operation unit 2005 is not limited to being attached to the imaging device, and may be a remote control device such as an infrared remote controller.

The display control unit 2006 outputs a display signal for displaying on the display unit 2007 based on display data such as a GUI (Graphical User Interface) generated by the control unit 2001 according to a program. A display unit 2007 is a display panel such as liquid crystal or organic EL, and performs display according to a display signal output from the display control unit 2006 . Note that when a touch panel is used as the operation unit 2005, the display unit 2007 and the operation unit 2005 can be configured integrally.

The communication control unit 2008 includes, for example, a circuit for communication control, and communicates with an external device via the connector/antenna 2009 and the access point 5000 . As a communication method, public wireless communication such as 3G, LTE, and 5G, IEEE802.11 including wireless LAN, Bluetooth (registered trademark), wired IEEE802.3, and the like can be used.

An internal bus 2010 electrically connects each unit of the imaging device 2000 to enable data transmission/reception between each unit. A recording medium control unit 2011 reads and writes data from a recording medium 2012 . The recording medium 2012 includes a magnetic disk such as an HDD and a semiconductor memory such as a memory card, and in addition to recording video data obtained by shooting, can record information necessary for controlling each unit executed by the control unit 2001 . can. Note that the recording medium 2012 may be configured to be detachable via a socket (not shown) or the like.

An optical system 2013 includes a photographing lens including focus, diaphragm, zoom mechanisms, etc., and forms an optical image of a subject. The imaging element 2014 is composed of, for example, a CCD or CMOS element, and converts an optical image formed on the imaging element into an electric signal, and then converts the analog electric signal into a digital signal by an internal A/D converter. Convert. By doing so, the imaging element 2014 can capture a moving image.

The camera signal processing unit 2015 includes, for example, an image processing circuit, and performs predetermined pixel interpolation, resize processing such as reduction, and various correction processing such as color conversion on the image data output from the image sensor 2014 .

The encoding/decoding processing unit 2016 compresses and encodes the digital signal processed by the camera signal processing unit 2015 at a predetermined bit rate and in a predetermined format, and decodes the video compression-encoded data. Although not shown, the imaging apparatus 2000 includes a microphone, an A/D converter for digitizing an analog audio signal, and a configuration for encoding digital data when sequentially streaming moving images (video) with audio. will include In this case, audio is recorded together with video when video is recorded, and video and audio are multiplexed by the encoding/decoding processing unit 2016 to generate video data with audio.

(Configuration of remote control terminal 3000)
FIG. 3 is a functional block diagram showing an example of the configuration of the remote control terminal 3000. As shown in FIG.

Each configuration from the control unit 3001 to the recording medium 3012 of the remote operation terminal 3000 can have the same functions as the configuration from the control unit 2001 to the recording medium 2012 described for the imaging device 2000 . Therefore, description of each configuration from the control unit 3001 to the recording medium 3012 is omitted. The encoding/decoding processing unit 3013 decodes the video compression encoded data and, if necessary, re-encodes the decoded data under the control of the control unit 3001 .

The remote control terminal 3000 can perform a remote control function of remotely controlling the imaging device 2000 by transmitting a remote control signal to the imaging device 2000 according to a user's operation.

(Configuration of IP video receiving terminal 4000)
Also, the configuration of the IP video receiving terminal 4000 is the same as that of the remote control terminal 3000, so the description thereof will be omitted. Note that the remote operation terminal 3000 and the IP video reception terminal 4000 can be the same terminal, in which case one terminal performs remote operation and IP video reception.

(Control Sequence of Imaging Apparatus 2000 at Start of Streaming)
An example of the sequence of the imaging device when streaming distribution is started will be described with reference to FIGS. 4 to 7. FIG.

FIG. 4 is a diagram showing an example of a setting screen displayed on the display unit 2007 of the imaging device 2000 when streaming settings are made before streaming distribution.

First, the user operates the operation unit 2005 to set the streaming function. FIG. 4A shows a function selection screen 4010 displayed on the display unit 2007 of the imaging apparatus 2000. FIG. A display unit 2007 functions as an operation interface.

A function selection screen 4010 displays a selection button 4011 for activating only the "streaming function" and a selection button 4012 for activating both the "streaming function and remote operation function". An OK button 4013 for confirming the selection and a cancel button 4014 for canceling the selection are displayed on the function selection screen 4010 .

The user can operate the operation unit 2005 on the function selection screen 4010 to select and set whether to activate only the “streaming function” or to activate both the “streaming function and the remote operation function”.

Then, when the user presses the streaming function selection button 4011 and confirms with the OK button 4013, detailed settings regarding streaming can be set in a wizard format.

Detailed settings related to streaming include, for example, selection of communication protocol such as RTP/UDP and RTSP, selection of IP address and port number of streaming distribution destination; 264 and H.264. H.265, video format selection, video resolution selection, audio channel number selection, FEC (Forward Error Correction) port and interval selection, and the like.

After the user makes detailed settings related to streaming and confirms the function settings, the screen advances to the connection selection screen 4020 . FIG. 4B shows a connection selection screen 4020 displayed on the display unit 2007. FIG.

A connection selection screen 4020 displays communication interfaces to which the imaging apparatus 2000 can be connected. By selecting a Wi-Fi selection button 4021 and an Ethernet selection button 4022, a communication interface to be used for communication connection can be selected. . The connection selection screen 4020 also displays an OK button 4023 for confirming the selection and a cancel button 4024 for canceling the selection.

Here, the subsequent processing will be described assuming that the Wi-Fi selection button 4021 is selected. However, even when the Ethernet selection button 4022 is selected, the only difference is the communication interface, and the same processing is performed.

When the user operates the operation unit 2005 to select the Wi-Fi selection button 4021 and press the OK button 4023, detailed settings regarding Wi-Fi connection can be set in a wizard format. Since this embodiment does not depend on the detailed connection settings of the Wi-Fi connection, description of the detailed settings will be omitted.

When the user makes detailed settings regarding the communication interface and confirms the connection settings, the screen advances to a SET (function + connection) selection screen 4030 . FIG. 4C shows a SET (function + connection) selection screen 4030. FIG.

On the SET (function+connection) selection screen 4030, a pair of the function setting and the connection setting that have been confirmed earlier can be selected, and a SET of functions to be provided to the external device to be connected can be created.

On the SET (function + connection) selection screen 4030, as an example, a SET1 button 4031 for performing streaming distribution via the Wi-Fi interface and a SET2 button 4032 for simultaneously starting streaming distribution and remote operation via the Wi-Fi interface are already created. ing.

Also, on the SET (function+connection) selection screen 4030, an OK button 4033 for confirming a selection item and a cancel button 4034 for canceling the selection are displayed.

Here, when the user operates the operation unit 2005 to select the SET1 button 4031 and press the OK button 4033, the control unit 2001 executes the communication control sequence from S501 to S505 in FIG. 5A. do.

FIG. 5 is a flow chart showing an example of a communication connection with an external device and a streaming distribution start process. Part of the processing in FIG. 5A will be described in detail with reference to FIG. 5B.

First, in S501, the control unit 2001 discovers other devices on the network by “device discovery and control” via the communication control unit 2008. FIG.

Next, in S502, when the control unit 2001 discovers the access point 5000, which is a counterpart device, via the communication control unit 2008, it connects to the network formed by the access point 5000 in S503.

Next, in S504, the control unit 2001 decodes the video data from the camera signal processing unit 2015 and the audio data from the microphone in the encoding/decoding processing unit 2016, and creates a TS packet (Transport Stream Packet) used for streaming. start the process to Further, the control unit 2001 activates communication services such as UDP required for streaming via the communication control unit 2008 .

The start sequence of streaming distribution is as described above, but when remote operation is also to be activated, in S504, communication service start processing for remote operation can be performed.

Next, in S505, the control unit 2001 performs distribution initial setting processing, which will be described later, and determines whether or not to transmit the TS packet generated by the encoding/decoding processing unit 2016. FIG.

(Sequence of delivery initial value setting processing)
The distribution initial setting process will be described with reference to FIG. 5(B).

First, in S5051, the control unit 2001 confirms whether the functions to be activated are streaming distribution only or streaming distribution and remote operation.

In S5052, if the function to be activated is only streaming distribution, the process proceeds to S5053, and if the function to be activated is streaming distribution and remote operation, the process proceeds to S5054.

In S5053, the control unit 2001 sets the initial streaming distribution setting to ON, permits transmission of TS packets, and starts streaming. In S5054, the control unit 2001 sets the initial streaming distribution setting to OFF, disallowing the transmission of TS packets.

As a result of executing the processing according to the control sequence of FIG. 5, the display displayed on the display unit 2007 becomes a network setting screen 4040 shown in FIG. 4(D). The streaming distribution selection button 4042, which is grayed out and inoperable until the above process is executed, is set to the streaming distribution initial settings determined in S5053 and S5054, and is enabled in an operable state. be done.

Further, when an activated selection button is operated or a streaming distribution ON/OFF operation is performed by the IP video receiving terminal 4000, which will be described later, the control unit 2001 sets the streaming distribution state according to the operation. It can be changed accordingly to control the transmission/stop of TS packets.

By the above control, when both streaming and remote operation are set to be activated, initial setting is made so that streaming distribution is not started when each function is activated. In the case of this setting, it is assumed that streaming distribution is started after the user performs camera settings of the imaging device 2000 by remote operation, so streaming distribution is not started immediately, but streaming is started by user operation. I'm trying

Note that the IP video receiving terminal 4000 connects to the network formed by the access point 5000 either before or after the sequence for starting the streaming distribution described above, starts the IP video receiving function, and receives the TS packet from the imaging device 2000. Streaming video can be played by receiving

Further, when the imaging device 2000 has RTSP set as a communication protocol, the IP video receiving terminal 4000 transmits a message requesting video distribution in RTSP, thereby ON/OFF of streaming distribution to the imaging device 2000. (on/off) can be controlled remotely.

(Control Sequence of Imaging Apparatus 2000 at Simultaneous Activation of Streaming and Remote Operation)
Next, focusing on the case where the functions of streaming distribution and remote operation are activated at the same time, the control sequence in that case will be described in detail.

As described above, the control unit 2001 executes the processing from S501 to S503 and connects to the network formed by the access point 5000. FIG. Next, in S504, the control unit 2001 decodes the video data from the camera signal processing unit and the audio data from the microphone with the encoding/decoding processing unit 2016 to start processing to generate TS packets. Then, the control unit 2001 activates a communication service such as UDP required for streaming via the communication control unit 2008 .

Furthermore, in S504, the control unit 2001 activates the HTTP server service for remote operation. Then, the control unit 2001 executes the distribution initial setting processing sequence of S505 to set the streaming distribution initial setting to OFF. In other words, it is temporarily set to stop the transmission of TS packets.

As a result, the streaming distribution selection button 4042 on the network setting screen 4040 is enabled from a grayed-out state to an operable state, and turned off with reference to the streaming distribution initial settings.

As described above, when the streaming distribution ON/OFF setting is changed by user operation, the control unit 2001 changes the setting of the streaming distribution state accordingly, and controls transmission/stop of TS packets. .

Further, when simultaneous activation of streaming distribution and remote operation is set on the selection screen 4010, RTSP cannot be selected as a communication protocol in the detailed setting of functions set by the subsequent wizard. In other words, the setting of the communication protocol used for control from the external device to which the streaming is distributed is disabled.

By doing so, since the RTSP request message from the IP video receiving terminal 4000 is not received, conflict with the ON/OFF operation of streaming distribution from the remote control terminal 3000 can be avoided. In this case, the state determination processing of the stream setting button 7011, which will be described later, is unnecessary.

Furthermore, the control unit 2001 can receive requests for remote operations relating to camera operations and requests for providing camera status information from the remote operation terminal 3000 via the network, and can return responses corresponding to these requests.

Further, although the details will be described later, this camera status information may include the selection status of the communication protocol in the function settings set by the wizard. Alternatively, by changing the content of the operation browser provided to the remote control terminal 3000 according to the selected state of the communication protocol in the function settings set by the wizard, the stream setting button 7011 on the remote control terminal 3000 side can be changed. It is possible to prevent command conflicts from occurring when RTSP is selected.

(Control sequence of remote control terminal 3000)
FIG. 6 is a flowchart showing an example of a process of establishing a communication connection with an imaging device and starting a remote operation of the remote operation terminal. Part of the processing in FIG. 6A will be described in detail with reference to FIG. 6B.

Either before or after the imaging apparatus 2000 executes the processing at the start of streaming, the remote control terminal 3000 executes the processing from S601 to S603 in FIG. Further, the Web browser is activated on the remote control terminal and the processing from S604 to S605 is executed.

First, in S<b>601 , the control unit 3001 discovers other devices on the network via the communication control unit 3008 . Next, in S602, when the control unit 3001 discovers the access point 5000, which is a counterpart device, via the communication control unit 3008, it connects to the network formed by the access point 5000 in S603.

Then, a sequence in which the remote operation terminal 3000 remotely operates the imaging device 2000 will be described. Here, as the remote operation of the imaging device 2000 by the remote operation terminal 3000, remote operation by a Web browser using WebAPI will be described as an example.

As shown in FIG. 7, the remote control terminal 3000 activates a web browser and inputs a URL corresponding to the IP address and port number of the imaging device. The imaging device 2000 returns page data according to the input URL. A web browser 7010 displayed on the remote control terminal 3000 includes a stream setting button 7011 and displays the transmission/stop state of streaming distribution of the imaging device 2000 .

By graying out the stream setting button 7011, it is possible to indicate that the setting button is disabled. In S604, the state of the stream setting button 7011 is determined according to the processing from S6041 to S6044. In this case, the processing of the imaging device 2000 may be executed by notifying the remote control terminal 3000 to invalidate the streaming delivery setting in the remote control terminal 3000 .

(State decision processing of stream setting button 7011)
First, in S<b>6041 , the control unit 3001 checks whether streaming from the imaging device 2000 to the IP video receiving terminal 4000 can be controlled by the IP video receiving terminal 4000 . For example, confirm whether RTSP is set as the communication protocol in the detailed settings of the streaming function set by the wizard.

Specifically, the remote control terminal 3000 issues a request for camera status information to the imaging device 2000 using the Web API, and according to the response, the activation status of the streaming function and the details of the streaming settings selected. This is achieved by referring to the protocol.

In S6042, if IP video receiving terminal 4000 can control streaming delivery (for example, RTSP is set as a communication protocol), or if stream delivery itself is not performed, proceed to S6043; For example, if the data is distributed by RTP/UDP), the process proceeds to S6044.

In S6043, the control unit 3001 sets the stream setting button 7011 to be grayed out. Also, in S6044, the control unit 3001 sets the streaming distribution initial setting to valid display, and permits setting change.

By executing the processing from S6041 to S6044, it is possible to avoid conflict between the operation from the remote control terminal 3000 and the streaming distribution operation from the IP video receiving terminal 4000. FIG.

Then, in S605, all buttons are drawn like the web browser screen 7010. FIG.

When the user instructs the operation unit 3005 to change the setting of the stream setting button 7021 while the stream setting button 7011 is activated, a request for ON/OFF change of stream distribution is transmitted by the Web API. Similarly, the display of the setting change is updated by referring to the response to the camera status acquisition request by the Web API.

By executing the above processing, ON/OFF operation of streaming distribution is performed, and conflict with ON/OFF operation of streaming distribution from IP video receiving terminal 4000 issued when RTSP is used is avoided. It is possible to avoid.

In the above-described method, the web browser side makes a determination, but in response to the web page data request issued by the remote control terminal 3000, the imaging device 2000 determines the above-described conditions, changes the web page data, and provides the web page data. This can also be achieved by

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Also, although the network configuration relaying the access point 5000 has been described as an example, other IP networks may be used, and the connection form does not depend on Ethernet, Wi-Fi, or other communication methods.

2000 imaging device 2001 control unit 2005 operation unit 2006 display control unit 2008 communication control unit 2014 imaging device 3000 remote operation terminal 4000 IP video reception terminal

Claims (8)

An imaging device,
an imaging means capable of capturing moving images;
means of communication;
Settings are made for a remote operation function for operating the imaging device according to a remote operation signal received from an external device via the communication means and a moving image distribution function for streaming a moving image to an external device via the communication means. setting means;
a control means;
When the setting means sets both the remote operation function and the moving image distribution function to be activated, the control means performs control such that streaming distribution is set to OFF as an initial setting of the moving image distribution function. An imaging device characterized by: 2. The imaging apparatus according to claim 1, wherein after said moving image distribution function is activated by said initial setting, streaming distribution is not started until streaming distribution is set to ON by said setting means. Display means for displaying an operation interface for accepting user operations,
3. The imaging apparatus according to claim 1, wherein said display means displays said operation interface in a state in which said streaming distribution can be switched on after said initial setting. When both the remote control function and the video distribution function are set to be activated, the setting means selects the communication protocol used for the remote control function as a communication protocol used for control from the external device to be streamed. 4. The imaging apparatus according to any one of claims 1 to 3, wherein is not settable. 5. The imaging apparatus according to claim 4, wherein the communication protocol is RTSP (Real Time Streaming Protocol). When both the remote control function and the video distribution function are set to be activated, the control means disables the streaming distribution setting by an external device that remotely controls the imaging device according to the remote control signal. 6. The imaging apparatus according to any one of claims 1 to 5, wherein the notification is given as follows. an imaging means capable of capturing moving images;
means of communication;
Settings for setting a remote operation function for operating an imaging device in response to a remote operation signal received from an external device via the communication means and a video distribution function for streaming video images to an external device via the communication means. means and
A control method for an imaging device comprising:
When the setting means sets both the remote control function and the moving image distribution function to be activated, the imaging apparatus controls the moving image distribution function so that streaming distribution is turned off as an initial setting of the moving image distribution function. control method. an imaging device;
a first external device that remotely operates the imaging device;
A video distribution system including a second external device for streaming distribution of video images from the imaging device,
The imaging device includes imaging means capable of capturing moving images;
means of communication;
a remote control function for operating the imaging device in response to a remote control signal received from the first external device via the communication means; and streaming delivery of moving images to the second external device via the communication means. a setting means for performing settings related to the video distribution function to be used;
a control means;
When the setting means sets both the remote operation function and the moving image distribution function to be activated, the control means performs control such that streaming distribution is set to OFF as an initial setting of the moving image distribution function. A video distribution system characterized by:

Images