JP6683864B1 - Content control system, content control method, and content control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance the visual effect of expression using an avatar. A content control system according to an embodiment includes at least one processor. The content control system acquires original image data showing a scene where a teacher conducts a lesson, generates motion data indicating a teacher's action based on the original image data, and based on the motion data, an avatar corresponding to the teacher. By determining the specifications of the above and placing an avatar based on the determined specifications at a position different from that of the teacher, educational content data for the students taking the lesson is generated, and the generated educational content data is output. . [Selection diagram] Fig. 4

Description

  One aspect of the present disclosure relates to a content control system, a content control method, and a content control program.

  Avatar, which is an example of a virtual object, is used in various computer systems. For example, Patent Literature 1 describes a learning system that displays an avatar of a teacher. The instructor's device is an HMD that is displayed on the display mounted on the head as if the screen were floating in the air, and an input device that wraps around the hand and converts the position and movement of the finger into an electric signal. The glove device (cyber glove) is connected. Input signals from the glove device, joypad, keyboard, mouse, etc. control the movement of the avatar drawn as the incarnation of the instructor in the virtual space.

JP, 2009-145883, A

  It is desired to enhance the visual effect of expressions using avatars in order to attract viewers to the content.

  A content control system according to one aspect of the present disclosure includes at least one processor. At least one of the at least one processor acquires original image data showing a scene where a teacher conducts a lesson. At least one of the at least one processor generates motion data indicating the motion of the teacher based on the original image data. At least one of the at least one processor determines a specification of the avatar corresponding to the teacher based on the motion data. At least one of the at least one processor arranges the avatar based on the determined specifications at a position different from the position of the teacher to generate educational content data for the student taking the lesson. At least one of the at least one processor outputs the generated educational content data.

  In such an aspect, educational content data expressing corresponding teachers and avatars is generated. By using this educational content data having a configuration not described in Patent Document 1, it is possible to enhance the visual effect of the content, and it can be expected that the viewer can be attracted to the content.

  According to an aspect of the present disclosure, it is possible to enhance the visual effect of expression using an avatar.

It is a figure showing an example of application of a content distribution system (contents control system) concerning an embodiment. It is a figure which shows an example of the hardware constitutions relevant to the content delivery system which concerns on embodiment. It is a figure which shows an example of a usage scene of a teacher terminal (distributor terminal). It is a figure which shows an example of a functional structure relevant to the content delivery system which concerns on embodiment. It is a flow chart which shows an example of selection of a display mode of educational contents. It is a flow chart which shows an example of output of educational contents data in compound mode and virtual mode. It is a figure which shows an example which determines operation | movement of an avatar. It is a figure which shows an example of the educational content in a live-action mode. It is a figure showing an example of educational contents in compound mode. It is a figure which shows another example of the educational content in compound mode. It is a figure which shows an example of the educational content in virtual mode. It is a sequence diagram which shows various examples of provision of content. It is a figure which shows the example of the auxiliary image displayed on a teacher terminal.

  Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same or equivalent elements will be denoted by the same reference symbols, without redundant description.

[System overview]
The content control system according to the embodiment is a computer system that controls content distributed to users. Content refers to information that is provided by a computer or computer system and that can be recognized by a person. Electronic data indicating content is called content data. The expression format of the content is not limited, and for example, the content may be expressed by an image (for example, photograph, video, etc.), document, sound, music, or a combination of any two or more of these elements. The content can be used for various modes of information transmission or communication, and can be used in various situations or purposes such as news, education, medical care, games, chats, commerce, lectures, seminars, and training. Content control refers to processing executed to provide content to a user. Content control may include at least one of generation, editing, storage, and distribution of content data, or may include processing other than these.

  In the present embodiment, the content is expressed using at least an image. An image showing content is called a "content image". A content image is an image that allows a person to visually recognize some information. The content image may be a moving image (video) or a still image. Electronic data indicating a content image is referred to as content image data.

  The content control system provides the content image data to the viewer to support the information transmission from the distributor to the viewer. A distributor is a person who tries to convey information to a viewer, that is, a sender of content. A viewer is a person who wants to obtain the information, that is, a user of the content. In one example, the distributor is located remotely to the viewer. The distributor can distribute the content by himself / herself. For example, the distributor shoots an area including the distributor for the distribution. The content control system acquires data (image data) of an image of a distributor, analyzes the image data to identify the distributor's action, and generates content image data including an avatar expressing the action. To do. In the present disclosure, an image analyzed in order to identify the action of the distributor (that is, an image showing the distributor) is referred to as “original image”, and electronic data indicating this original image is referred to as original image data. The original image can be said to be a material for generating content.

  In one example, the avatar may appear in the content image on behalf of the distributor, in which case the viewer viewing the content image sees the avatar rather than the distributor. In another example, the avatar may be rendered in the content image with the publisher. A viewer can experience augmented reality (Augmented Reality (AR)), virtual reality (Virtual Reality (VR)), or mixed reality (Mixed Reality (MR)) by viewing the content image. In this embodiment, the content control system can switch the display mode of the content in response to the viewer's request or automatically. The display mode refers to a method or format for displaying content.

  Since the content control system identifies the behavior of the distributor by analyzing the image data, the distributor does not need to wear a device for motion capture such as a body strap or gloves.

  An avatar is a user's alter ego represented by a computer. An avatar is a kind of virtual object that is an object that does not actually exist in the real world and is represented only on a computer system. The avatar is represented by two-dimensional or three-dimensional computer graphics (CG) using the image material independent of the original image, not the person who is photographed (that is, not the user itself shown in the original image). It The expression method of the avatar is not limited. For example, the avatar may be expressed by using an animation material, or may be expressed as if it is close to the real thing based on a photographed image. The avatar may be freely selected by the user of the content control system (eg, teacher or student).

  In one example, the content image represents the virtual space in which the avatar resides. The virtual space refers to a virtual two-dimensional or three-dimensional space represented by an image displayed on a computer. From a different point of view, it can be said that the content image is an image showing the scenery seen from the virtual camera set in the virtual space. The virtual camera is set in the virtual space so as to correspond to the line of sight of the user who views the content image.

  In one example, the content control system may deliver the content to a viewer. Distribution means a process of transmitting information to a user via a communication network or a broadcast network. In this disclosure, delivery is a concept that may include broadcasting. In the present disclosure, a content control system having a function of delivering content is also referred to as a content delivery system.

  The method of generating and delivering the content by the content control system is not limited. For example, the content control system may control live content. In this case, the content control system processes the real-time video provided from the distributor terminal to generate content data, and transmits the content data to the viewer terminal in real time. It can be said that this is one aspect of live Internet broadcasting. Alternatively, the content control system may generate content data by processing an image captured in the past. This content data may be transmitted to the viewer terminal, or may be temporarily stored in a storage device such as a database. The content control system may be used for time shifting, where content can be viewed in a given period after real-time distribution. Alternatively, the content control system may be used for on-demand distribution in which the content can be viewed at any timing. As described above, since the content image may be a still image, the content control system (content distribution system) may be used to deliver the content of the still image in real time or later.

  In this disclosure, the expression "transmitting data or information to a computer" means transmission for finally delivering the data or information to the computer. It should be noted that this expression is meant to include the case where another computer or communication device relays data or information in the transmission.

  As described above, the purpose and usage of the content are not limited. In the present embodiment, educational content is shown as an example of content, and the content control system controls educational content data. Educational content is content used by teachers to teach classes to students. A teacher is a person who teaches academics and arts, and a student is a person who receives the instruction. A teacher is an example of a distributor and a student is an example of a viewer. The teacher may be a person with a teacher's license or may be a person without a teacher's license. Classes mean that a teacher teaches students about academic work and arts. The age and affiliation of each teacher and student are not limited, and thus the purpose and usage of educational content are not limited. For example, educational content may be used in various schools such as a nursery school, kindergarten, elementary school, junior high school, high school, university, graduate school, vocational school, preparatory school, online school, etc. May be. In this regard, educational content may be used for a variety of purposes such as early childhood education, compulsory education, higher education, lifelong learning and the like.

[System configuration]
FIG. 1 is a diagram showing an example of application of a content distribution system (content control system) 1 according to an embodiment. In this embodiment, the content distribution system 1 includes a server 10. The server 10 is a computer that generates and distributes content image data. The server 10 is connected to at least one student terminal 20 via the communication network N. Although FIG. 1 shows two student terminals 20, the number of student terminals 20 is not limited in any way. Furthermore, the server 10 may be connected to at least one of the teacher terminal 30, the original image database 40, and the content database 50 via the communication network N. The configuration of the communication network N is not limited. For example, the communication network N may be configured to include the Internet or may be configured to include an intranet.

  The student terminal 20 is a computer used by a student and is an example of a viewer terminal (computer used by a viewer). The student terminal 20 has a function of accessing the content distribution system 1 to receive and display content data. The student terminal 20 may have a function of capturing and transmitting an image. The type and configuration of the student terminal 20 are not limited. For example, the student terminal 20 may be a high-performance mobile phone (smartphone), a tablet terminal, a wearable terminal (for example, a head mounted display (HMD), smart glasses, etc.), a laptop personal computer, a mobile terminal such as a mobile phone. Alternatively, the student terminal 20 may be a stationary terminal such as a desktop personal computer. Alternatively, the student terminal 20 may be a classroom system including a large screen installed in the room.

  The teacher terminal 30 is a computer used by a teacher and is an example of a distributor terminal (computer used by a distributor). In one example, the teacher terminal 30 is located remote from the student terminal 20. The teacher terminal 30 has a function of shooting a video and a function of accessing the content distribution system 1 and transmitting electronic data (video data) showing the video. The teacher terminal 30 may have a function of receiving and displaying an image or content. The type and configuration of the teacher terminal 30 is not limited. For example, the teacher terminal 30 may be an image capturing system having a function of capturing, recording, and transmitting an image. Alternatively, the teacher terminal 30 may be a high-performance mobile phone (smartphone), a tablet terminal, a wearable terminal (for example, a head mounted display (HMD), smart glasses, etc.), a laptop personal computer, a mobile terminal such as a mobile phone. Alternatively, the teacher terminal 30 may be a stationary terminal such as a desktop personal computer.

  The administrator of the classroom or the student operates the student terminal 20 to log in to the content distribution system 1, whereby the student can view the educational content. The teacher operates the teacher terminal 30 to log in to the content distribution system 1, and thereby can provide his class to the students. In the present embodiment, it is assumed that the user of the content distribution system 1 has already logged in.

  The original image database 40 is a device that stores original image data. The original image data indicates a video or a still image. The original image data is stored in the original image database 40 by the server 10, the teacher terminal 30, or another computer such as another computer. The original image database 40 can be said to be a library that stores original images captured in the past.

  The content database 50 is a device that stores educational content data. The educational content data indicates a video or a still image. It can be said that the content database 50 is a library of educational content.

  The installation locations of the original image database 40 and the content database 50 are not limited. For example, the original image database 40 or the content database 50 may be provided in a computer system different from the content distribution system 1 or may be a component of the content distribution system 1. One database may function as both the original image database 40 and the content database 50.

  FIG. 2 is a diagram showing an example of a hardware configuration related to the content distribution system 1. FIG. 2 shows a server computer 100 functioning as the server 10 and a terminal computer 200 functioning as the student terminal 20 or the teacher terminal 30.

  As an example, the server computer 100 includes a processor 101, a main storage unit 102, an auxiliary storage unit 103, and a communication unit 104 as hardware components.

  The processor 101 is a computing device that executes an operating system and application programs. Examples of the processor include a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), but the type of the processor 101 is not limited thereto. For example, the processor 101 may be a combination of sensors and dedicated circuitry. The dedicated circuit may be a programmable circuit such as an FPGA (Field-Programmable Gate Array) or another type of circuit.

  The main storage unit 102 is a device that stores a program for realizing the server 10, a calculation result output from the processor 101, and the like. The main storage unit 102 includes at least one of a ROM (Read Only Memory) and a RAM (Random Access Memory), for example.

  The auxiliary storage unit 103 is generally a device capable of storing a larger amount of data than the main storage unit 102. The auxiliary storage unit 103 is composed of a non-volatile storage medium such as a hard disk or a flash memory. The auxiliary storage unit 103 stores a server program P1 for causing the server computer 100 to function as the server 10 and various data. For example, the auxiliary storage unit 103 may store data regarding at least one of a virtual object such as an avatar and a virtual space. In this embodiment, the content control program is implemented as the server program P1.

  The communication unit 104 is a device that executes data communication with other computers via the communication network N. The communication unit 104 is composed of, for example, a network card or a wireless communication module.

  Each functional element of the server 10 is realized by causing the processor 101 or the main storage unit 102 to read the server program P1 and causing the processor 101 to execute the program. The server program P1 includes code for realizing each functional element of the server 10. The processor 101 operates the communication unit 104 according to the server program P1 to read and write data in the main storage unit 102 or the auxiliary storage unit 103. By such processing, each functional element of the server 10 is realized.

  The server 10 may be composed of one or more computers. When a plurality of computers are used, these computers are connected to each other via a communication network to logically form one server 10.

  As an example, the terminal computer 200 includes a processor 201, a main storage unit 202, an auxiliary storage unit 203, a communication unit 204, an input interface 205, an output interface 206, and an imaging unit 207 as hardware components.

  The processor 201 is a computing device that executes an operating system and application programs. The processor 201 may be, for example, a CPU or GPU, but the type of the processor 201 is not limited to these.

  The main storage unit 202 is a device that stores a program for realizing the student terminal 20 or the teacher terminal 30, a calculation result output from the processor 201, and the like. The main storage unit 202 is composed of, for example, at least one of ROM and RAM.

  The auxiliary storage unit 203 is generally a device capable of storing a larger amount of data than the main storage unit 202. The auxiliary storage unit 203 is configured by a non-volatile storage medium such as a hard disk or a flash memory. The auxiliary storage unit 203 stores a client program P2 for causing the terminal computer 200 to function as the student terminal 20 or the teacher terminal 30 and various data. For example, the auxiliary storage unit 203 may store data regarding at least one of a virtual object such as an avatar and a virtual space.

  The communication unit 204 is a device that executes data communication with other computers via the communication network N. The communication unit 204 is composed of, for example, a network card or a wireless communication module.

  The input interface 205 is a device that receives data based on a user's operation or action. For example, the input interface 205 includes at least one of a keyboard, operation buttons, pointing device, microphone, sensor, and camera. The keyboard and operation buttons may be displayed on the touch panel. Since the type of the input interface 205 is not limited, the input data is not limited. For example, the input interface 205 may accept data input or selected by a keyboard, operation buttons, or pointing device. Alternatively, the input interface 205 may accept voice data input by the microphone. Alternatively, the input interface 205 may accept image data (for example, video data or still image data) captured by a camera.

  The output interface 206 is a device that outputs data processed by the terminal computer 200. For example, the output interface 206 includes at least one of a monitor, a touch panel, an HMD, and a speaker. A display device such as a monitor, a touch panel, or an HMD displays the processed data on the screen. The speaker outputs the sound represented by the processed sound data.

  The image capturing unit 207 is a device that captures an image of the real world, and is specifically a camera. The imaging unit 207 may shoot a moving image (video) or a still image (photo). When capturing a moving image, the image capturing unit 207 acquires a series of frame images arranged in time series as a moving image by processing the video signal based on a given frame rate. The imaging unit 207 can also function as the input interface 205.

  Each functional element of the student terminal 20 or the teacher terminal 30 is realized by loading the client program P2 onto the processor 201 or the main storage unit 202 and executing the program. The client program P2 includes codes for realizing each functional element of the student terminal 20 or the teacher terminal 30. The processor 201 operates the communication unit 204, the input interface 205, the output interface 206, or the imaging unit 207 according to the client program P2 to read and write data in the main storage unit 202 or the auxiliary storage unit 203. By this processing, each functional element of the student terminal 20 or the teacher terminal 30 is realized.

  At least one of the server program P1 and the client program P2 may be provided after being fixedly recorded in a tangible recording medium such as a CD-ROM, a DVD-ROM, or a semiconductor memory. Alternatively, at least one of these programs may be provided via a communication network as a data signal superimposed on a carrier wave. These programs may be provided separately or together.

  FIG. 3 is a diagram showing an example of a usage scene of the teacher terminal 30. In this example, the teacher terminal 30 includes a computer main body 210 that houses a processor 201, a main storage unit 202, an auxiliary storage unit 203, a communication unit 204, an imaging unit 207 and a microphone 211 that function as an input interface 205, and an output interface 206. And a monitor 212 functioning as. The teacher 90 gives a lesson while writing or displaying characters or figures on a board (eg, whiteboard, blackboard, electronic whiteboard, electronic blackboard, etc.) 91 as needed. The imaging unit 207 obtains an original image by shooting the scene of the lesson. The voice (utterance) of the teacher 90 is recorded by the microphone 211. The teacher terminal 30 can acquire video data in which the sound is associated with the captured video. The teacher 90 may give a lesson while watching an image (for example, an auxiliary image described later) displayed on the monitor 212.

  FIG. 4 is a diagram showing an example of a functional configuration related to the content distribution system 1. The server 10 includes a content management unit 11, an image acquisition unit 12, a motion identification unit 13, a content generation unit 14, an output unit 15, and an auxiliary image generation unit 16 as functional elements. The content management unit 11 is a functional element that manages generation and output of educational content, and includes an image acquisition unit 12, a motion identification unit 13, a content generation unit 14, and an output unit 15. The image acquisition unit 12 is a functional element that acquires original image data. The motion identification unit 13 is a functional element that identifies the teacher's action from the original image data. The content generation unit 14 is a functional element that generates educational content data including an avatar corresponding to a teacher. The output unit 15 is a functional element that outputs the educational content data. The auxiliary image generation unit 16 is a functional element that generates auxiliary image data, which is electronic data of an auxiliary image showing a state of a student, and transmits the auxiliary image data to the teacher terminal 30. The auxiliary image may be a moving image (video) or a still image. The auxiliary image allows the teacher to visually recognize the students in class.

  The student terminal 20 includes a request unit 21, a reception unit 22, a display control unit 23, and a transmission unit 24 as functional elements. The request unit 21 is a functional element that requests the server 10 to switch the display mode of the educational content. The receiving unit 22 is a functional element that receives educational content data. The display control unit 23 is a functional element that processes the educational content data and displays the educational content on the display device. The transmission unit 24 is a functional element that transmits the image data generated by the imaging unit 207 to the server 10.

  The teacher terminal 30 includes a transmitting unit 31, a receiving unit 32, and a display control unit 33 as functional elements. The transmission unit 31 is a functional element that transmits the image data generated by the imaging unit 207 to the server 10. The receiving unit 32 is a functional element that receives the auxiliary image data. The display control unit 33 is a functional element that processes the auxiliary image data and displays the auxiliary image on the display device.

[System operation]
The operation of the content distribution system 1 (more specifically, the operation of the server 10) will be described, and the content control method (or content distribution method) according to the present embodiment will be described. Hereinafter, the image processing will be described in detail, and the detailed description of the audio data processing will be omitted.

  FIG. 5 is a flowchart showing an example of the selection of the display mode of the educational content as the processing flow S1. In step S11, the content management unit 11 specifies the next display mode (that is, the display mode after switching). The display mode switching method is not limited, and in connection with this, the content management unit 11 may specify the next display mode by an arbitrary method. For example, the display mode may be switched in response to an operation on the student terminal 20. In this case, the student performs an operation for switching the display mode of the educational content on the student terminal 20. In the student terminal 20, the request unit 21 transmits a switching request to the server 10 in response to the operation. The switching request is a data signal requesting switching of the display mode, and indicates the next display mode. The content management unit 11 receives the switching request and specifies the next display mode. Alternatively, the display mode may be automatically switched without receiving a request from the user. For example, the content management unit 11 may identify the next display mode by referring to a preset scenario for controlling the educational content.

  As shown in step S12, subsequent processing is changed depending on the next display mode. The number of display modes and the specific format are not limited, and may be set by an arbitrary policy. In the present embodiment, as an example, it is assumed that the content distribution system 1 can provide three types of display modes: a live-action mode, a composite mode, and a virtual mode. The live-action mode is an expression format in which the original image, which is a live-action image, is displayed as it is. Therefore, in this case, the educational content does not include an avatar. The composite mode is an expression format in which an image that is visible to both the teacher and the avatar is displayed, and in this case, the avatar is displayed at a position different from that of the teacher. “Both the teacher and the avatar are visually recognizable” means that a person who views the educational content can visually and clearly recognize both the teacher and the avatar. The virtual mode is an expression format that displays an image that the teacher cannot see but the avatar can see. "Teachers are invisible, but avatars are visible." Means that the person watching the educational content cannot visually or clearly recognize the teacher's appearance, but the appearance of the avatar is visually It is something that can be clearly recognized.

  If the live-action mode is designated, the process proceeds to step S13. In step S13, the content management section 11 outputs the original image data as it is as educational content data. Specifically, the image acquisition unit 12 acquires original image data, and the output unit 15 outputs the original image data as educational content data. The acquisition method of the original image data is not limited. For example, the image acquisition unit 12 may receive the image data sent from the teacher terminal 30 as the original image data. Alternatively, the image acquisition unit 12 responds to a request signal from the student terminal 20 and outputs an image corresponding to the request signal (for example, at least a part of the video of the lesson desired by the student) from the original image database 40. May be read as The output method of the educational content data is also not limited. For example, the output unit 15 may transmit the educational content data to one or more student terminals 20, or may store the educational content data in the content database 50. Alternatively, the output unit 15 may transmit the educational content data to the student terminal 20 and store the educational content data in the content database 50.

  If the composite mode is designated, the process proceeds to step S14. In step S14, the content management unit 11 outputs educational content data that is visible to both the teacher and the avatar.

  If the virtual mode is designated, the process proceeds to step S15. In step S15, the content management unit 11 outputs educational content data in which the teacher is not visible but the avatar is visible.

  Details of steps S14 and S15 will be described with reference to FIG. FIG. 6 is a flowchart showing an example of outputting educational content data in the composite mode and the virtual mode. The difference between the composite mode and the virtual mode is the arrangement method of the avatars, and since there are many parts common to both processes, both the composite mode and the virtual mode will be described with reference to FIG.

  In step S101, the image acquisition unit 12 acquires original image data. Similar to step S13, the method of acquiring the original image data is not limited. Therefore, the image acquisition unit 12 may receive the original image data from the teacher terminal 30 or may read it from the original image database 40.

  In step S102, the motion identifying unit 13 identifies the teacher's action based on the original image data. A person's movement refers to a person's posture, facial expression, or body movement, and may also include vocalization accompanied by mouth movement. In one example, the motion identification unit 13 identifies the area in which the teacher is displayed from the original image, identifies the two-dimensional movement (for example, posture, facial expression) of the teacher in the area, and determines a plurality of joints corresponding to the movement. Position and estimate the depth of each joint. The joint that defines the posture corresponds to the body part of the teacher. For example, the motion specifying unit 13 may set a joint at a joint and a main part of the face (eyebrows, eyes, chin, etc.), or may set a joint at a position different from these.

  The motion specifying unit 13 determines each bone (adjacent joints) with respect to the lens center of the camera based on the positional relationship between adjacent joints and a rule (motion rule) predetermined based on rationality and consistency of body movement. Estimate the direction and angle of the imaginary line connecting). The motion specifying unit 13 can specify the three-dimensional motion of the teacher by this estimation. Rationality and coherence of physical movement refer to the possible movements of humans. For example, its rationality and consistency is due to the constraint that the elbows and knees bend in one direction but not the other, the range of head movements relative to the neck, the range of upper arm movements relative to the shoulders, and finger movements. It may include a movable range and the like. The method for specifying the teacher's motion is not limited to the above, and the motion specifying unit 13 may specify the teacher's motion by another method.

  As shown in step S103, the motion identifying unit 13 may identify the state of the physical object (this is referred to as a "related physical object") related to the motion of the teacher (distributor) based on the original image data. The physical object refers to something that can be perceived by a person, and may include various objects such as an object, a person, and a voice. For example, the physical object may be an object shown by the original image or a sound associated with the original image. It can be said that the teacher is also an example of a real object. The state of a real object is a state of the real object that can be grasped by human perception, and the state is, for example, at least one of shape, position, movement (motion), sound, and voice of the real object. May be included. Step S103 may be omitted. Related reality objects are reality objects that change, operate, appear, or disappear in relation to the behavior of the distributor. However, in the present disclosure, the related reality object does not include a distributor (teacher). The type of related reality object is not limited. Examples of related reality objects that correspond to a teacher include textbooks that the teacher picks up or puts on a desk and descriptions by the teacher (for example, letters, strings, symbols, or characters that the teacher writes or erases on the board. At least one of the picture) and the utterance of the teacher. The motion identifying unit 13 may identify one or more related physical objects by an arbitrary image analysis method.

  In step S104, the motion specifying unit 13 generates raw data. Raw data refers to electronic data that indicates at least the operation of the identified teacher (distributor), and further indicates the state of one or more related real objects when step S103 is executed. In the present embodiment, the data indicating the motion of the teacher (distributor) is also referred to as motion data. Raw data includes motion data. The data structure of both raw data and motion data is not limited and may be designed arbitrarily. For example, the motion identifying unit 13 may include information about a plurality of joints and a plurality of bones that indicate a three-dimensional movement (for example, posture, facial expression) of the teacher and the teacher identifier (ID) in the motion data. Examples of information about joints and bones include three-dimensional coordinates of individual joints and combinations of adjacent joints (that is, bones), but the configuration of this information is not limited to this and may be arbitrarily designed. The motion specifying unit 13 may convert at least one of the teacher's utterance and description into text and include the text in raw data or motion data. The configuration of the information indicating the related reality object is not limited, and for example, the motion identification unit 13 sets the identifier (ID) and the information indicating the state (eg, shape, position, character string, etc.) for each related reality object as raw data. May be included.

  In step S105, the content generation unit 14 acquires model data of the avatar corresponding to the teacher. The method of acquiring model data is not limited. For example, the content generation unit 14 may read model data of a preset avatar or an avatar designated by a user (for example, a teacher or a student) of the content distribution system 1 from the auxiliary storage unit 103.

  Model data refers to electronic data used to define the specifications of virtual objects. A virtual object specification refers to an agreement or method for controlling a virtual object. For example, the specifications include at least one of a virtual object's configuration (eg, shape and size), behavior, and audio. The data structure of the avatar model data is not limited and may be arbitrarily designed. For example, the model data may include information about a plurality of joints and a plurality of bones that make up the avatar, graphic data indicating an appearance design of the avatar, an attribute of the avatar, and an avatar identifier (ID). Examples of information about joints and bones include three-dimensional coordinates of individual joints and a combination of adjacent joints (that is, bones), but the structure of this information is not limited to this and may be arbitrarily designed. . An avatar's attributes are any information set to characterize the avatar, and may include, for example, nominal size, voice quality, or personality.

  As shown in step S106, the content generation unit 14 may acquire model data of a virtual object (this is referred to as a “related virtual object”) corresponding to each of the one or more related real objects. When step S103 is not executed, step S106 is also omitted. The associated virtual object may represent any object. For example, the related virtual object may represent an object that does not exist in the real world (for example, a fictional character), or may represent a natural object or an artificial object that exists in the real world. Alternatively, the related virtual object may be a representation for giving a visual effect to the related real object. For example, the related object may be a book or a character corresponding to a textbook used by the teacher, a graphic expression for decorating a character string written on the board by the teacher, or a graphic expression related to the text of the utterance of the teacher. . The data structure of the model data of the related virtual object is not limited and may be arbitrarily designed according to the intended expression. For example, if the related virtual object is a character, its model data may have the same data structure as that of the avatar. Alternatively, the model data of the related virtual object may include only graphic data indicating the appearance design.

  In step S107, the content generation unit 14 generates educational content data including an avatar. When steps S103 and S106 are executed, the content generation unit 14 can generate educational content data that further includes one or more related virtual objects in addition to the avatar.

  In one example, the content generation unit 14 sets the virtual space based on the original image data. The setting of the virtual space may include a process of specifying the position of the virtual camera in the virtual space, and a process of specifying the position and the size of each of the one or more real objects shown in the original image. The content generation unit 14 may calculate the position of each real object in the optical axis direction of the virtual camera or the positional relationship between the real objects, and set the virtual space based on the calculation result. Alternatively, the content generation unit 14 may set the virtual space by analyzing the original image by a method such as machine learning. In one example, the scene shown in the original image may be set in the virtual space like a two-dimensional screen.

  After setting the virtual space, the content generation unit 14 executes control related to the avatar and the teacher according to the display mode. The content generation unit 14 arranges an avatar in the virtual space and further arranges each related virtual object when one or a plurality of related virtual objects exist. “Arranging an object (such as an avatar or a related virtual object)” means placing the object at a predetermined position, and is a concept that includes changing the position of the object.

  When the next display mode is the composite mode, the content generation unit 14 generates educational content data that can be visually recognized by both the avatar and the teacher. Specifically, the content generation unit 14 arranges the avatar at a position different from that of the teacher, so that both the avatar and the teacher can be visually recognized. “Arranging the avatar at a position different from that of the teacher” means arranging so that both the avatar and the teacher can be visually recognized in the content image when the educational content is displayed. The content generation unit 14 may arrange the avatar so that the avatar does not overlap the teacher on the content image. Alternatively, the content generation unit 14 may arrange the avatar so that the student can visually recognize the teacher although the avatar overlaps a part of the teacher on the content image.

  In one example, the content generation unit 14 may arrange the avatar at a position different from the teacher in the virtual space so that both the avatar and the teacher can be captured from the virtual camera.

  In another example, the content generation unit 14 generates a screen including a window in which an avatar appears (an avatar window or a first window) and a window in which a teacher appears (a teacher window or a second window), so that the avatar becomes a teacher. You may arrange | position in a different position. For example, the content generation unit 14 may divide the screen into an avatar window showing a virtual space including an avatar and a teacher window showing the real world in which the teacher is shown.

  The content generation unit 14 may represent the avatar window by a combination of a real image area and a virtual object, or may represent the avatar window by a virtual object and a virtual background without using the real image area. The content generation unit 14 may set the original image as it is as the teacher window. The content generation unit 14 may set both windows so that the area of the avatar window is larger than the area of the teacher window. Alternatively, the content generation unit 14 may arrange the original image (teacher window) in the virtual space as a part of the background to arrange the avatar at a position different from that of the teacher. In this case, since the original image (teacher window) is arranged as one object in the virtual space, it is possible to three-dimensionally define the positional relationship between the avatar and the original image (teacher window) showing the teacher. it can. For example, the content generation unit 14 may position the original image (teacher window) behind the avatar in the virtual space. The content generation unit 14 may set both windows so that the area of the teacher window is larger than the area of the avatar window.

  The content generation unit 14 may use any technique for screen division, and may perform screen division by, for example, picture-in-picture (Picture-in-Picture (PinP)).

  When using the avatar window and the teacher window, the content generation unit 14 may include the visual effect related to the display of the avatar or the window in the educational content data. For example, the content generation unit 14 may include a visual effect in which the avatar window pops out of the teacher window in the educational content data. Alternatively, the content generation unit 14 may include a visual effect in which the avatar pops out of the teacher window or the teacher's body into the educational content data.

  The content generation unit 14 determines the specifications of the avatar corresponding to the teacher by setting the position of each joint of the avatar placed at a position different from that of the teacher based on raw data (more specifically, motion data). . “Specification of an avatar corresponding to the teacher” means that the specifications of the avatar follow or almost follow the actions of the teacher. Determining the specification of the determined avatar may include the action of the avatar, in which case mirroring of the action is achieved. By setting the position of each joint of the avatar so as to match the movement of the teacher, the orientation and angle of each bone of the avatar reflect the posture of the teacher. The content generation unit 14 may adjust the position of each joint of the avatar so that the size of the avatar is the same as or substantially the same as the size of the teacher. By making the size of the avatar the same as the size of the teacher, it is possible to further emphasize visually that the avatar is the alter ego of the teacher. The content generation unit 14 may make the size of the avatar different from that of the teacher.

  When the next display mode is the virtual mode, the content generation unit 14 generates educational content data in which the avatar is visible but the teacher is not. In one example, the content generation unit 14 may arrange the avatar so as to be superimposed on the teacher. This placement process means placing the avatar so that the teacher hides it when the educational content is displayed on the display device. More specifically, in other words, “arranging the avatar so as to overlap the teacher” means arranging the avatar so as to overlap the teacher in the scene shown in the original image. It is noted that "teacher is hidden by an avatar" means not only when the teacher's body is completely hidden by the avatar, but also when the teacher's body is not completely hidden by the avatar. I want to. For example, the teacher's body may appear to protrude from the avatar on the educational content due to factors such as the physical constitution difference between the teacher and the avatar. May be included. In any case, by superimposing the avatar on the teacher, the student cannot see the teacher on the content image.

  The content generation unit 14 arranges the avatar in the virtual space so that the avatar is displayed on behalf of the teacher in the two-dimensional image. The content generation unit 14 determines the specifications of the avatar corresponding to the teacher by adjusting the position of each joint of the avatar to the corresponding part (joint) of the teacher based on the raw data (more specifically, motion data). To do. By aligning the joints of the avatar with the joints of the teacher, the orientation and angles of the individual bones of the avatar reflect the posture of the teacher and the dimensions of the avatar are adjusted to be approximately or nearly the same as the teacher. It

  In the virtual mode, the content generation unit 14 arranges the avatar at a position different from that of the teacher, sets the motion and size of the avatar as in the composite mode, and then makes the teacher invisible in the image by the blurring process. Good. The blurring process is a process of making the contours of individual parts of an object unclear so that the existence of an object (a teacher in this example) can be recognized but the appearance of the object cannot be clearly recognized visually. Say. Mosaic processing, blurring processing, and fog processing are examples of the blurring processing, but other techniques may be used. Alternatively, the content generation unit 14 may make the teacher invisible by deleting the teacher from the image and restoring the background of the teacher's portion (that is, the portion hidden by the teacher). This process is also called retouching.

  In both the composite mode and the virtual mode, the content generation unit 14 may further arrange one or more related virtual objects. The method of arranging each related virtual object is not limited. For example, the content generation unit 14 may arrange the related virtual object so as to be superimposed on the corresponding real object. Alternatively, the content generation unit 14 may arrange the related virtual object without superimposing it on the corresponding real object or almost without superimposing it. In any case, the content generation unit 14 determines the specification of each related virtual object based on the raw data. For example, the content generator 14 sets the position, size, (and motion, if any) of the associated virtual object.

  In both the composite mode and the virtual mode, the content generation unit 14 generates educational content data indicating a virtual space in which an avatar (and related virtual object) is placed in the virtual space. The educational content data may include audio data corresponding to the original image data. The educational content data generation method and data structure are not limited. For example, the content generation unit 14 may generate educational content data including virtual space data indicating the virtual space and the positions, dimensions, and movements (postures) of individual objects. Alternatively, the content generation unit 14 may generate the educational content data by executing rendering based on the set virtual space. In this case, the educational content data indicates the content image itself including the avatar (and related virtual object). In one example, the content generation unit 14 generates educational content data by combining a captured image area obtained from an original image and a virtual object (avatar and related virtual object, if any). This educational content data represents a composite image of the real world represented by the original image and a virtual object (avatar and related virtual object, if any).

  In step S108, the output unit 15 outputs the educational content data. Similar to step S13, the method of outputting the educational content data is not limited. Therefore, the output unit 15 may transmit the educational content data to one or more student terminals 20 or may store the educational content data in the content database 50 and execute both the transmission and the storage.

  When the output unit 15 transmits the educational content data to the student terminal 20, the receiving unit 22 of the student terminal 20 receives the educational content data, and the display control unit 23 transmits the educational content data. Process and display educational content on a display device. When the server 10 has not executed the rendering, the display control unit 23 displays the content image by executing the rendering based on the educational content data. When the educational content data indicates the content image itself, the display control unit 23 displays the content image as it is. The student terminal 20 outputs sound from the speaker in accordance with the display of the content image.

  When the educational content is live content or when the video content in the original image database 40 is processed, step S14 or S15 is repeatedly executed. Step S14 or S15 may be executed for each frame image or may be executed for a series of a plurality of frame images. Of course, the teacher moves over time, and the avatar in the educational content moves correspondingly. Further, in some cases, the related virtual object is displayed in association with the movement of the teacher.

  FIG. 7 is a diagram showing an example of determining avatar specifications in the composite mode and the virtual mode. The motion specifying unit 13 specifies a three-dimensional motion of the teacher 90 by estimating a plurality of joints 501 and a plurality of bones 502 corresponding to the movement (for example, posture, facial expression) of the teacher 90 based on the original image. (Step S12). Then, the motion specifying unit 13 generates raw data (motion data) indicating the specified motion (step S14). It should be noted that the center of FIG. 7 is a convenient drawing for helping understanding of the joint 501 and the bone 502, and it is not intended that the drawing is essential in the content distribution system 1. The content generation unit 14 sets an avatar 92 that performs the same operation as the teacher 90 based on the raw data (motion data) and the avatar model data (step S17).

  An example of educational content will be described with reference to FIGS. In either example, the original image 401 indicates a scene in which the teacher 90 stands in front of the board 91.

  FIG. 8 is a diagram showing an example of educational contents in the live-action mode. In the live-action mode, the content distribution system 1 outputs the original image data as it is as the educational content data, so that the original image 401 is provided as it is as the educational content 402.

  FIG. 9 is a diagram showing an example of educational contents in the composite mode. In this example, the content distribution system 1 provides the educational content 403 in which both the avatar 92 and the teacher 90 are visible through the avatar window 404 and the teacher window 405. The area of the avatar window 404 is larger than the area of the teacher window 405, and the teacher window 405 is displayed at the edge of the avatar window 404 (upper right in the example of FIG. 9). The content distribution system 1 (content generation unit 14) may set the teacher window 405 as a screen area different from the avatar window 404. Alternatively, the content distribution system 1 (content generation unit 14) may arrange the original image 401 in the virtual space as one element of the background of the avatar, and set an image showing the virtual space as the avatar window 404. In this case, the original image 401 arranged in the virtual space is displayed as the teacher window 405.

  FIG. 10 is a diagram showing another example of educational contents in the composite mode. In this example, the content distribution system 1 arranges the avatar 92 and the teacher 90 at different positions in the virtual space based on the original image 401, and provides the educational content 406 that copies this virtual space.

  FIG. 11 is a diagram showing an example of educational contents in the virtual mode. In this example, the content distribution system 1 arranges the avatar 92 so as to be superimposed on the teacher 90, thereby providing the educational content 408 in which the teacher 90 is invisible but the avatar 92 is visible.

  In all of the educational contents 403, 406, and 408 shown in FIGS. 9 to 11, the specifications of the avatar 92 correspond to the teacher 90, so that the student can see the avatar 92 performing the same operation as the teacher 90.

  In the examples of FIGS. 9 to 11, the motion specifying unit 13 specifies the handwritten word “This” written on the board 91 by the teacher 90 as the related reality object (step S13), and sets the motion data including this word. It may be generated (step S14). The motion identifying unit 13 may include the handwritten word “This” as text data (character string data) in the motion data, or may extract the feature points of each handwritten character and collect the coordinates of the feature points. May be set as the motion data of the word “This”. The content generation unit 14 generates educational content data including a related virtual object corresponding to the handwritten word "This" based on such motion data (step S17). The student terminal 20 displays the educational content data so that the student can see the word "This" with a new or additional visual effect (eg, decorated handwriting "This", handwriting replaced with CG. "This").

  As described above, the method of generating and delivering the content is not limited. FIG. 12 is a sequence diagram showing various examples of providing educational content by the content distribution system 1. In the example (a) of FIG. 12, content distribution in the case of delivering educational content in real time, that is, live distribution or live Internet broadcasting is shown as a processing flow S2. In the processing flow S2, the teacher terminal 30 shoots the scene in which the teacher gives a lesson by the imaging unit 207 (step S21), and the transmission unit 31 directs the video data (original image data) obtained by the shooting to the server 10. And transmits (step S22). The server 10 executes the processing flow S1 for the video data, and transmits the educational content data to the student terminal 20 (step S23). The student terminal 20 receives and displays the educational content data (step S24).

  In the example (a) of FIG. 12, the processing flow S2 is repeatedly executed while the teacher terminal 30 is photographing the lesson (in other words, the processing flow S2 is executed for each frame image forming the video data). ). In the virtual mode, the student can view the class as if the avatar is teaching on behalf of the teacher in real time, and in the composite mode, the class as the teacher and the avatar are teaching together can be viewed in real time. Since the movement of the avatar is determined by analyzing the original image, the teacher can carry out the lesson in his / her usual clothes without wearing a device for motion capture.

  The example (b) of FIG. 12 shows a case where the video captured in the past is processed and the educational content is distributed as a processing flow S3. In the processing flow S3, the server 10 reads video data (original image data) showing a lesson taken in the past from the original image database 40 (step S31), executes the processing flow S1 on the video data, and uses it for educational purposes. The content data is transmitted to the student terminal 20 (step S32). The student terminal 20 receives and displays the educational content data (step S33). The timing at which the server 10 transmits the educational content data to the student terminal 20 is not limited. For example, the server 10 may transmit the educational content data after executing the processing flow S1 for all the frame images forming the video data. Alternatively, the server 10 may transmit the educational content data corresponding to each frame image every time the processing flow S1 is executed.

  The example (c) of FIG. 12 shows a case where the video captured in the past is processed and the educational content is stored as a processing flow S4. In the processing flow S4, the server 10 reads out video data (original image data) showing a lesson taken in the past from the original image database 40 (step S41), executes the processing flow S1 on the video data, and uses it for educational The content data is stored in the content database 50 (step S42). For example, the server 10 may store the educational content data after executing the processing flow S1 for all the frame images forming the video data. The student terminal 20 can access the content database 50 at any timing to receive and display the educational content (steps S43 and S44).

  The method of providing the educational content to the student terminal 20 is not limited. For example, the educational content may be provided to the student terminal 20 via the server 10, or may be provided via a computer or computer system different from the server 10. When the server 10 provides the educational content, the student terminal 20 transmits a content request, which is a data signal for acquiring the educational content, to the server 10 in response to the operation of the student. The server 10 receives the content request, reads out the educational content data indicated by the request from the content database 50, and transmits the educational content data to the student terminal 20. The method of transmitting the educational content data is not limited, and may be streaming distribution or download, for example.

  It can be said that each of the examples (b) and (c) of FIG. 12 is the use or reuse of the video content photographed or used in the past. There are many educational video contents in the world that show teachers teaching lessons. By using the content distribution system 1, it becomes possible to convert the enormous amount of video content into more attractive video content using an avatar.

  The method of content generation and distribution is not limited to the example of FIG. 12, and another processing flow may be adopted. In any case, the content distribution system 1 can be applied to various distribution methods such as live distribution (Internet live broadcasting), time shift distribution, and on-demand distribution.

  FIG. 13 is a diagram showing an example of the auxiliary image 410 displayed on the teacher terminal 30. In the example of FIG. 3, this auxiliary image 410 is displayed on the monitor 212. The auxiliary image 410 shows three students (viewers). The configuration of the auxiliary image 410 is not limited. For example, the auxiliary image 410 may be composed of a set of images or photographs of the students photographed by the individual student terminals 20. In FIG. 13, the auxiliary image 410 is a set of images 411, 412, 413 of three students corresponding to the three student terminals 20. Alternatively, the auxiliary image 410 may be a single image or image obtained by combining images or photographs of individual students. Alternatively, the auxiliary image 410 may be a single image or picture showing multiple students in a room. Each student may be represented by a live-action image, an avatar that moves in the same manner as the student, or a still image that is not linked to the movement of the student.

  The method of generating the auxiliary image is not limited. The auxiliary image generation unit 16 of the server 10 may generate auxiliary image data based on the student image data transmitted from one or more student terminals 20. Student image data refers to electronic data of an image of a student. The auxiliary image generation unit 16 may generate the auxiliary image data by embedding the student image data from each student terminal 20 in the auxiliary image as it is, or may set the student image data as it is as the auxiliary image data. Alternatively, the auxiliary image generation unit 16 may generate the auxiliary image data without acquiring the student image data from the student terminal 20. The auxiliary image generation unit 16 transmits the generated auxiliary image data to the teacher terminal 30. The auxiliary image generation unit 16 may receive voice data indicating a voice recorded by the student terminal 20 from the student terminal 20 and associate the voice data with the auxiliary image data. In the teacher terminal 30, the receiving unit 32 receives the auxiliary image data, and the display control unit 33 processes the auxiliary image data and displays the auxiliary image. The auxiliary image may be a live video showing the current situation of the student. In this case, the auxiliary image generation unit 16 generates and transmits each frame image of the auxiliary video, and the teacher terminal 30 causes the individual frame to be transmitted. Display images in sequence. The teacher terminal 30 may process the voice data received from the server 10 and output the voice of the student.

[effect]
As described above, the content control system according to one aspect of the present disclosure includes at least one processor. At least one of the at least one processor acquires original image data showing a scene in which the teacher gives a lesson. At least one of the at least one processor generates motion data indicating the motion of the teacher based on the original image data. At least one of the at least one processor determines a specification of the avatar corresponding to the teacher based on the motion data. At least one of the at least one processor arranges the avatar based on the determined specifications at a position different from the position of the teacher to generate educational content data for the student taking the lesson. At least one of the at least one processor outputs the generated educational content data.

  A content control method according to one aspect of the present disclosure includes a step of acquiring original image data showing a scene where a teacher conducts a lesson, a step of generating motion data indicating a teacher's action based on the original image data, By determining the specification of the avatar corresponding to the teacher based on the motion data, and by arranging the avatar based on the determined specification in a position different from the teacher, educational content data for students taking classes can be obtained. It includes a step of generating and a step of outputting the generated educational content data.

  A content control program according to one aspect of the present disclosure includes a step of acquiring original image data showing a scene where a teacher conducts a lesson, a step of generating motion data indicating a teacher's action based on the original image data, By determining the specification of the avatar corresponding to the teacher based on the motion data, and by arranging the avatar based on the determined specification in a position different from the teacher, educational content data for students taking classes can be obtained. The computer is caused to execute the step of generating and the step of outputting the generated educational content data.

  In such an aspect, educational content data expressing corresponding teachers and avatars is generated. By using this educational content data having a configuration not described in Patent Document 1, the visual effect of the content can be expected to be improved, and as a result, the interest of students can be attracted to this content. In other words, the interest in educational content can be enhanced.

  The visual effect of the content can be enhanced by providing educational content in which both the teacher and the avatar are reflected, instead of simply replacing the teacher with the avatar. As a result, it can be expected that the students will find the educational content familiar and interesting, which in turn can lead to maintaining or improving the motivation of the students who take the lessons. On the other hand, from the standpoint of a distributor such as a teacher, it is not necessary to wear a device for motion capture, so classes are conducted as usual without incurring the costs necessary to purchase or use that special device. be able to.

  Furthermore, because educational content data can be generated from past images that were not intended to replace teachers with avatars, huge live-action images of the past were converted to educational content using avatars, and the Can be used or reused.

  In the content control system according to another aspect, at least one of the at least one processor transmits the generated educational content data to the student terminal of the student, so that the educational content is displayed on the student terminal. The data may be displayed. By this processing, the educational content data including the avatar can be shown to the students.

  In the content control system according to another aspect, the action of the avatar corresponding to the teacher may be the same action as the teacher. By making the avatar perform the same action as the teacher, the visual effect of educational content can be enhanced.

  In the content control system according to another aspect, at least one of the at least one processor generates a screen including a first window in which the avatar appears and a second window in which the teacher appears, thereby making the avatar the teacher. You may arrange | position in a different position. By adopting a technique such as screen division, the visual effect of educational contents can be enhanced.

  In a content control system according to another aspect, at least one of the at least one processor makes the area of the first window larger than the area of the second window and arranges the second window at the edge of the first window. May be. By configuring the educational content data such that the first window showing the avatar is the main window and the second window showing the teacher is the sub window, the visual effect of the educational content can be enhanced.

  In the content control system according to another aspect, at least one of the at least one processor may position the second window behind the avatar in the virtual space. By thus three-dimensionally defining the positional relationship between the avatar and the second window, the second window can be displayed as a part of the background of the avatar. With such a visual effect, the visual effect of the educational content can be enhanced.

  In the content control system according to another aspect, the educational content indicated by the educational content data is in a composite mode in which both the teacher and the avatar can see, and a virtual mode in which the teacher cannot see but the avatar can see. It may be switchable between. At least one of the at least one processor may place the avatar based on the determined specifications in a different position than the teacher in the combined mode. Providing such two kinds of display modes can enhance the visual effect of educational contents.

  In the content control system according to another aspect, at least one of the at least one processor makes the teacher invisible by arranging the avatar based on the determined specification so as to be superimposed on the teacher in the virtual mode. Good. In this case, it is possible to create a visual effect as if the teacher replaced the avatar.

  In the content control system according to another aspect, at least one of the at least one processor arranges the avatar based on the determined specification in a position different from the teacher in the virtual mode, and The teacher may be made invisible by executing the blurring process. In this case, in this case, the visual effect of displaying the avatar while leaving the presence of the teacher on the content can be produced.

[Modification]
The foregoing has been a detailed description based on the embodiment of the present disclosure. However, the present disclosure is not limited to the above embodiment. The present disclosure can be variously modified without departing from the gist thereof.

  Although the content distribution system 1 is configured using the server 10 in the above-described embodiment, the content control system may be applied to direct distribution between user terminals that do not use the server 10. In this case, each functional element of the server 10 may be installed in any user terminal, for example, may be installed in either one of a distributor terminal and a viewer terminal. Alternatively, the individual functional elements of the server 10 may be installed separately in a plurality of user terminals, for example, in the distributor terminal and the viewer terminal. In this regard, the content control program may be implemented as a client program. The content control system may be configured using a server or may be configured without using the server.

  In the above-described embodiment, the content control system sets a virtual space and arranges avatars (and related virtual objects, if any) in the virtual space to generate educational content data. However, the use of virtual space is not essential. For example, the content control system may generate content data (for example, educational content data) by arranging a two-dimensional representation avatar on a two-dimensional image.

  In the above embodiment, the content control system provides three types of display modes: a live-action mode, a composite mode, and a virtual mode, but the content display mode is not limited. For example, the content control system may provide composite mode and virtual mode without providing live-action mode. Alternatively, the content control mode may provide only the composite mode. The content control system has two types of composite modes: a first composite mode using screen division (composite mode as shown in FIG. 9) and a second composite mode not using screen division (composite mode as shown in FIG. 10). Educational content may be switched between.

  As mentioned above, the content control system may control any type of content other than educational content. For example, the content control system may control any content to support any communication or communication between users.

  In the present disclosure, the expression “at least one processor executes a first process, a second process, ... An nth process”, or a corresponding expression is the following. This is a concept including a case where the execution subjects (that is, processors) of n processes from the first process to the nth process change in the middle. That is, this expression is a concept including both the case where all of the n processes are executed by the same processor and the case where the processors change in an arbitrary process in the n processes.

  The processing procedure of the method executed by at least one processor is not limited to the example in the above embodiment. For example, some of the steps (processes) described above may be omitted, or each step may be executed in a different order. Further, any two or more of the steps described above may be combined, and some of the steps may be modified or deleted. Alternatively, other steps may be executed in addition to the above steps.

  DESCRIPTION OF SYMBOLS 1 ... Content distribution system, 10 ... Server, 11 ... Content management part, 12 ... Image acquisition part, 13 ... Motion specification part, 14 ... Content generation part, 15 ... Output part, 16 ... Auxiliary image generation part, 20 ... Student terminal , 21 ... requesting unit, 22 ... receiving unit, 23 ... display controlling unit, 24 ... transmitting unit, 30 ... teacher terminal, 31 ... transmitting unit, 32 ... receiving unit, 33 ... display controlling unit, 40 ... original image database, 50 ... content database, 90 ... teacher (distributor), 92 ... avatar, 401 ... original image, 402, 403, 406, 408 ... educational content, 404 ... avatar window (first window), 405 ... teacher window (second) Window), 410 ... auxiliary image, P1 ... server program, P2 ... client program.

Claims (11)

With at least one processor,
At least one of the at least one processor acquires original image data showing a scene where a teacher conducts a lesson,
At least one of the at least one processor, based on the original image data, to generate motion data indicating the motion of the teacher,
At least one of the at least one processor determines a specification of an avatar corresponding to the teacher based on the motion data,
At least one of the at least one processor generates a screen including a first window in which the avatar appears and a second window in which the teacher appears, so that the teacher can obtain the avatar based on the determined specifications. Is arranged at a position different from that of the first window, and the area of the first window is made larger than the area of the second window to generate educational content data for a student taking the lesson,
At least one of the at least one processor outputs the generated educational content data.
Content control system. At least one of the at least one processor transmits the generated educational content data to the student terminal of the student to display the educational content data on the student terminal,
The content control system according to claim 1. The action of the avatar corresponding to the teacher is the same action as the teacher,
The content control system according to claim 1. At least one of the at least one processor, arranging the second window to the edge of the front Symbol first window,
Content control system according to any one of claims 1 to 3. At least one of the at least one processor positions the second window behind the avatar in a virtual space,
The content control system according to claim 4 . The educational content indicated by the educational content data can be switched between a composite mode in which both the teacher and the avatar are visible and a virtual mode in which the teacher is invisible but the avatar is visible. And
At least one of the at least one processor arranges the avatar based on the determined specification at a position different from that of the teacher in the composite mode.
Content control system according to any one of claims 1-5. At least one of the at least one processor renders the teacher invisible in the virtual mode by arranging the avatar based on the determined specifications so as to be superimposed on the teacher.
The content control system according to claim 6 . In the virtual mode, at least one of the at least one processor arranges the avatar based on the determined specification at a position different from that of the teacher and blurs the teacher indicated by the image data. Making the teacher invisible by executing a process,
The content control system according to claim 6 . At least one of the at least one processor identifies a region in which the teacher is displayed from the original image data, identifies a two-dimensional motion of the teacher in the region, and determines a plurality of joints corresponding to the motion. Of the joint, and the motion data is generated based on a positional relationship between the joints adjacent to each other and a predetermined motion rule based on rationality and consistency of body movement,
Content control system according to any one of claims 1-8. A content control method executed by a content control system comprising at least one processor,
A step of acquiring original image data showing a scene in which the teacher gives a lesson,
Generating motion data indicating the motion of the teacher based on the original image data;
Determining specifications of an avatar corresponding to the teacher based on the motion data;
By generating a screen including a first window in which the avatar appears and a second window in which the teacher appears, the avatar based on the determined specifications is arranged at a position different from that of the teacher, and the first window is displayed. Generating educational content data for a student taking the lesson by making the area of the window larger than the area of the second window ;
Outputting the generated educational content data. A step of acquiring original image data showing a scene in which the teacher gives a lesson,
Generating motion data indicating the motion of the teacher based on the original image data;
Determining specifications of an avatar corresponding to the teacher based on the motion data;
By generating a screen including a first window in which the avatar appears and a second window in which the teacher appears, the avatar based on the determined specifications is arranged at a position different from that of the teacher, and the first window is displayed. Generating educational content data for a student taking the lesson by making the area of the window larger than the area of the second window ;
A content control program for causing a computer to execute the step of outputting the generated educational content data.

Images