JP7578209B1 - Image generation system, image generation method, and image generation program - Google Patents

Abstract

The present invention obtains video data in response to a query input by a user.
[Solution] The video generation system disclosed herein comprises an acquisition unit that acquires an input query related to video generation from a user, a scenario generation unit that generates scenario data related to video generation based on the input query, a code generation unit that generates code for constructing 3D data based on the scenario data, and a video data acquisition unit that acquires video data based on the code.
[Selected figure] Figure 3

Description

This disclosure relates to an image generation system, an image generation method, and an image generation program.

Technology has been provided for automatically generating images (also called "videos"). For example, technology has been provided for estimating the three-dimensional posture of a virtual person from a two-dimensional line drawing and generating a video (for example, Patent Document 1).

JP 2003-058906 A

However, there is room for improvement in the conventional technology. For example, in the conventional technology, two-dimensional line drawings, i.e., images, are required to generate videos, and preparing images such as two-dimensional line drawings places a large burden on the user, making it difficult to generate videos when the user is unable to prepare images. For this reason, it is desirable to provide a video generation service that places less burden on the user and has high usability, and it is desirable, for example, to obtain video data in response to a query input by the user.

Therefore, this disclosure proposes a video generation system, a video generation method, and a video generation program that can acquire video data in response to a query input by a user.

In order to solve the above problems, a video generation system according to one embodiment of the present disclosure includes an acquisition unit that acquires an input query related to video generation from a user, a scenario generation unit that generates scenario data related to video generation based on the input query, a code generation unit that generates code for constructing 3D data based on the scenario data, and a video data acquisition unit that acquires video data based on the code.

FIG. 1 illustrates an example of an image generation system according to the present disclosure. FIG. 1 is a diagram illustrating an example of a hardware configuration of an image generation system according to the present disclosure. FIG. 1 is a diagram showing an example of a flow of an image generation process according to the present disclosure. FIG. 11 is a diagram showing another example of the flow of the image generation process of the present disclosure. FIG. 11 is a diagram showing an example of a flow of an evaluation process according to the present disclosure. FIG. 11 is a diagram illustrating an example of a generation process of scenario generation information. FIG. 11 is a diagram illustrating an example of a generation process of scenario data. FIG. 11 is a diagram illustrating an example of a process for generating information for code generation. FIG. 11 is a diagram illustrating an example of image quality improvement processing. FIG. 4 illustrates an example of a user interface. FIG. 4 illustrates an example of a user interface. FIG. 4 illustrates an example of a user interface. FIG. 13 illustrates an example of a user interface. FIG. 13 illustrates an example of a user interface. FIG. 4 illustrates an example of a user interface. FIG. 4 illustrates an example of a user interface. FIG. 11 is a diagram illustrating an example of a generation process of information for sound generation. FIG. 2 is a diagram showing an example of a USD file. 11 is a flowchart showing a processing procedure executed by the video production system. FIG. 4 illustrates an example of a user interface. FIG. 4 illustrates an example of a user interface. FIG. 4 illustrates an example of a user interface. FIG. 11 illustrates an example of an editing process. FIG. 11 illustrates an example of an editing process. FIG. 11 illustrates an example of an editing process. FIG. 11 illustrates an example of an editing process. FIG. 11 illustrates an example of an editing process. FIG. 11 illustrates an example of an editing process. FIG. 11 illustrates an example of an editing process. FIG. 13 is a diagram showing an example of a confirmation operation during editing. 13 is a flowchart showing a processing procedure executed by the video production system. FIG. 13 is a diagram illustrating an example of an evaluation process according to a range selection. FIG. 11 is a conceptual diagram showing an example of processing according to the depth of field. FIG. 13 is a diagram showing an example of an image during a confirmation operation. FIG. 13 is a diagram showing an example of highlighting of a changed portion. FIG. 13 is a diagram showing an example of presentation of the relationship between cuts. FIG. 13 is a diagram illustrating an example of object selection. FIG. 13 illustrates an example of object selection. FIG. 13 is a diagram illustrating an example of processing using reference data. 10 is a flowchart showing a flow of processing in response to a user operation. FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing device.

Embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the image generation system, image generation method, and image generation program according to the present application are not limited to these embodiments. In addition, in the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

The present disclosure will be described in the following order.
1. Embodiment 1-1. Overview of the configuration of the video generation system of the present disclosure 1-2. Processing by the video generation system of the present disclosure 1-3. User interface 1-4. Processing examples 1-4-1. Sound generation example 1-4-2. Text logo generation example 1-4-3. Re-learning example 1-4-4. USD update example 1-4-5. Evaluation example 1-4-6. Selection example of multiple cuts 1-4-7. Use example of time information 1-4-8. Response example according to degree of verbalization 1-4-9. Use example of qualitative values 1-4-10. Rendering processing example during input 1-4-11. Confirmation work example during editing 1-4-12. Processing example according to range selection 1-4-13. Processing example according to depth of field 1-4-14. Playback processing example during confirmation work 1-4-15. Highlighting example 1-4-16. Example of presenting relationships between cuts 1-4-17. Example of selecting objects 1-4-18. Example of using 3D models 1-4-19. Example of using reference data 1-4-20. Example of advantages of having 3D data 1-5. Example of processing flow from the user's perspective 1-6. About AI models 2. Other embodiments 2-1. Other configuration examples 2-2. Others 3. Effects of the present disclosure 4. Hardware configuration

1. Embodiment
1-1. Overview of the configuration of the image generation system of the present disclosure
Fig. 1 is a diagram showing an example of an image generation system according to the present disclosure. The image generation system 1 includes an image generation module 100, an information acquisition module 200, a sensor unit 300, and a client UI display unit 400. Although Fig. 1 shows only one of each configuration, the image generation system 1 may include a plurality of image generation modules 100, a plurality of information acquisition modules 200, a plurality of sensor units 300, and a plurality of client UI display units 400.

First, the configuration of the image generation module 100 that performs the image generation process will be described. The image generation module 100 has an input text analysis unit 110, a sensor analysis unit 120, a prompt generation unit 130, an image generation unit 140, a sound generation unit 150, a text/logo generation unit 160, a composite editing unit 170, an evaluation unit 180, a client UI module 190, etc.

The input text analysis unit 110 analyzes the input text. For example, the input text analysis unit 110 analyzes the text input from the information acquisition module 200. The sensor analysis unit 120 analyzes the input sensor information. For example, the sensor analysis unit 120 analyzes the sensor information acquired from the information acquisition module 200.

The prompt etc. generation unit 130 generates various information required for generating a video (video) including a prompt etc. to be input to an AI (Artificial Intelligence) model (also simply referred to as a "model"), which is a machine learning model described later. For example, the prompt etc. generation unit 130 generates a prompt using a user's input and a pre-stored prompt (template, etc.). Note that a prompt is merely one example of information to be input to an AI model (model input information), and the model input information to be input to an AI model is not limited to a prompt, and any form of model input information can be adopted, and the "prompt etc. generation unit" may be read as a "model input information etc. generation unit." In FIG. 1, the prompt etc. generation unit 130 has a scenario-oriented generation unit 131, a video-oriented generation unit 132, a sound-oriented generation unit 133, and a text/logo-oriented generation unit 134.

The scenario-oriented generation unit 131 generates various information related to the generation of a scenario. The scenario-oriented generation unit 131 generates input information to be input to a model that outputs a scenario. For example, the scenario-oriented generation unit 131 is a scenario generation unit that generates scenario data related to video generation based on an input query. For example, the scenario-oriented generation unit 131 is a first output unit that outputs scenario generation information used by the scenario generation unit to generate scenario data based on an input query.

The video generation unit 132 generates various information related to the generation of videos. The video generation unit 132 generates input information to be input to a model that outputs code for constructing 3D (three-dimensional) data. For example, the video generation unit 132 is a code generation unit that generates code for constructing 3D data based on scenario data. For example, the video generation unit 132 is a second output unit that outputs code generation information used by the code generation unit to generate code for constructing 3D data based on scenario data.

The sound-oriented generation unit 133 generates various information related to the generation of sound information (audio information). The sound-oriented generation unit 133 generates input information to be input to a model that outputs sound. The text/logo-oriented generation unit 134 generates various information related to the generation of text and logos. The text/logo-oriented generation unit 134 generates input information to be input to a model that outputs at least one of text and logos.

The video generation unit 140 executes processing related to the generation of video. The video generation unit 140 is a video acquisition unit that acquires video data based on a code. The video generation unit 140 generates video using various information generated by the prompt generation unit 130. For example, the video generation unit 140 is a video generation unit that generates video data based on a code. Note that the video generation unit 140 may acquire video data in any manner. For example, the video generation unit 140 may acquire video data by transmitting data used to generate video data to an external service providing device (vendor, etc.) that provides a video data generation service, and receiving the video data generated by the service providing device from the service providing device. In FIG. 1, the video generation unit 140 has a USD generation unit 141, a rendering unit 142, and a video refinement unit 143.

The USD generation unit 141 generates various information related to the USD (Universal Scene Description). For example, the USD generation unit 141 uses a prompt obtained by the video prompt generation in the video generation unit 132 to generate USD-Python, etc., with an AI model such as the Large Language Model (hereinafter also referred to as "LLM").

The rendering unit 142 executes various processes related to rendering. The rendering unit 142 executes a process of rendering the USD generated by the USD generation unit 141.

The image refinement unit 143 executes various processes to refine the image. The rendering unit 142 improves the quality of the generated image through image refinement processing. For example, the image refinement unit 143 is an image quality improvement unit that executes image quality improvement processing to improve the image quality of video data.

The sound generation unit 150 executes a process of generating sounds. Using the prompts obtained by the sound-oriented prompt generation in the sound-oriented generation unit 133, the sound generation unit 150 generates sound information such as background music (BGM), sound effects (SE), narration, and dialogue, based on an AI model such as a contrastive learning model.

The text/logo generator 160 executes a process for generating at least one of text and a logo. The text/logo generator 160 generates at least one of text and a logo using the information generated by the text/logo generator 134.

With the above-described configuration, the image generation module 100 generates a prompt for scenario generation by combining a pre-stored prompt with a user's input. The image generation module 100 generates a scenario by inputting the generated prompt into an AI model such as an LLM. The image generation module 100 also generates a prompt for generating images, sounds, and text/logos from the generated scenario. The image generation module 100 performs image generation, sound generation, and text/logo generation using prompts, scenarios, etc. for generating images, sounds, and text/logos.

The composite editing unit 170 executes processes related to editing. For example, the composite editing unit 170 executes a process of combining (combining) the generated video, sound, and text/logo into one video.

The evaluation unit 180 executes an evaluation process to evaluate various targets. The evaluation unit 180 evaluates the information generated by the above-mentioned configuration. For example, the evaluation unit 180 generates information indicating an evaluation of at least one of the scenario data and the video data.

The client UI module 190 executes processes related to output on the client-side UI (User Interface). For example, the client UI module 190 generates various information related to output on the client-side UI. In this case, the client UI module 190 executes processes to generate a UI to be displayed on the user side. The client UI module 190 generates various information to be displayed on the client UI display unit 400.

In addition, the information acquisition module 200 acquires various information. The information acquisition module 200 has an input text acquisition unit 210, a sensor acquisition unit 220, etc. The input text acquisition unit 210 acquires text information input via the keyboard 320 or the microphone 330. For example, the input text acquisition unit 210 acquires text information input by the user via the keyboard 320 or the microphone 330. For example, the input text acquisition unit 210 is an acquisition unit that acquires an input query related to video generation from the user.

The sensor acquisition unit 220 acquires information (also called "sensor information") detected by a sensor such as the camera 340 or a motion capture device. The information acquisition module 200 provides (transmits) the acquired various information to the image generation module 100. Note that the information acquisition module 200 may be integrated with the image generation module 100.

The sensor unit 300 has various sensors. The sensor unit 300 senses user input. The sensor unit 300 accepts operations by the user. For example, the sensor unit 300 is a reception unit that accepts operations related to video editing from the user. For example, the sensor unit 300 has a mouse 310, a keyboard 320, a microphone 330, a camera 340, an IMU 350 which is an inertial measurement unit, and the like. Thus, in addition to the mouse 310 and the keyboard 320, the sensor unit 300 includes a user terminal (such as a smartphone) equipped with the microphone 330, the camera 340, and the IMU 350, sensors such as motion capture, and the like, and senses user input.

The client UI display unit 400 displays various information to be presented to the client (user). The client UI display unit 400 displays the UI generated by the client UI module 190 on the client's display (display device). For example, the client UI display unit 400 is a display control unit that displays a storyboard based on scenario data. The storyboard is configured to display video data for each cut of the video.

The video generation system 1 may have a hardware configuration as shown in FIG. 2. FIG. 2 is a diagram showing an example of a hardware configuration of the video generation system of the present disclosure. In FIG. 2, the video generation system 1 has, as its hardware configuration, a cloud-side computer 10, a client-side computer 20, and a camera/sensor 30 including various sensors such as a camera. The video generation system 1 may also include an information providing device (computer) that provides information resources 40 such as learning data and an AI model 50 to the computer 10.

The hardware configuration shown in FIG. 2 is merely an example, and the video generation system 1 can adopt any hardware configuration as long as it can execute the desired processing. For example, the computer 10 and the computer 20 may be integrated. Furthermore, the information resource 40 and the AI model 50 may be stored inside the computer 10.

The computer 10 includes a CPU (Central Processing Unit) 11, a GPU (Graphics Processing Unit) 12, a communication device 13, and a memory/storage 14. For example, the computer 10 corresponds to the image generation module 100 and the information acquisition module 200 in FIG. 1. The computer 10 may be a service providing device (server device) that provides an image generation service. The CPU 11 and the GPU 12 are so-called processors, and execute calculation processes (arithmetic operations) related to various processes such as image generation.

The communication device 13 is a communication device having a communication function for transmitting and receiving information between the computer 20, an information providing device, etc., and may be, for example, a communication circuit, a NIC (Network Interface Card), etc. The communication device 13 communicates with other devices such as the computer 20 and the information providing device via a predetermined network (such as the Internet). For example, the communication device 13 is connected to the predetermined network by wire or wirelessly, and transmits and receives information between the computer 20, the information providing device, etc.

The memory/storage 14 is a storage device that stores various types of information. For example, the memory/storage 14 is a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The memory/storage 14 stores various types of information used for processing by processors such as the CPU 11 and the GPU 12. The memory/storage 14 may also store information resources 40, AI models 50, etc.

The computer 20 includes a CPU 21, a GPU 22, a communication device 23, a memory/storage 24, and an IO interface 25. For example, the computer 20 corresponds to the client UI display unit 400 in FIG. 1. The computer 20 may be a terminal device (such as a PC (Personal Computer), a mobile device such as a smartphone, etc.) used by a user who uses the video generation service. The CPU 21 and the GPU 22 are so-called processors, and perform calculation processing (arithmetic processing) related to various processes such as video display. Note that the above is only an example, and the computer 20 can adopt any configuration as long as it is capable of performing the desired processing. For example, the computer 20 may perform calculation processing (arithmetic processing) related to various processes such as video display using a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The computer 20 may also be configured to directly incorporate a program into the circuit of the processor instead of storing the program in the memory (such as the memory/storage 24). In this case, the processor realizes the function by reading and executing the program incorporated in the circuit. In addition, each processor in this embodiment is not limited to being configured as a single circuit for each processor, but may be configured as a single processor by combining multiple independent circuits to realize its functions. Also, like computer 20, computer 10 can adopt any configuration as long as it is capable of performing the desired processing.

The communication device 23 is a communication device having a communication function for transmitting and receiving information between the computer 10, the sensor 30, etc., and may be, for example, a communication circuit, a NIC, etc. The communication device 23 communicates with other devices such as the computer 10, the sensor 30, etc. via a predetermined network (such as the Internet). For example, the communication device 23 is connected to the predetermined network by wire or wirelessly, and transmits and receives information between the computer 10, the sensor 30, etc., and other devices.

The memory/storage 24 is a storage device that stores various types of information. For example, the memory/storage 24 is a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The memory/storage 24 stores various types of information used for processing by processors such as the CPU 21 and the GPU 22.

The IO interface 25 is an input/output interface device. The computer 20 receives input from the sensor 30 via the IO interface 25. For example, the computer 20 receives input from an input device such as a keyboard or a mouse via the IO interface 25. The computer 20 also outputs information from a display (display device) and a speaker (audio output device) via the IO interface 25. For example, the computer 20 plays video on a display and a speaker via the IO interface 25.

The various sensors 30, such as a camera, sense user input. The various sensors 30, such as a camera, accept operations by the user. For example, the sensor 30 corresponds to the sensor unit 300 in FIG. 1. The information resource 40 includes various information such as learning data. For example, the information resource 40 includes learning data used to learn various AI models such as LLM. The AI model 50 includes information on an AI model used in processing related to image generation such as LLM. For example, the AI model 50 includes information on various AI models such as models M1 to M3 described below. As described above, the image generation system 1 may have a configuration other than that shown in FIG. 2.

1-2. Processing by the image generation system of the present disclosure
From here, the processing by the video generation system will be described. First, an example of the flow of the video generation processing shown in Fig. 3 will be described. Fig. 3 is a diagram showing an example of the flow of the video generation processing of the present disclosure. Note that the processing described below with the video generation system 1 as the processing subject may be performed by any device capable of executing the processing, depending on the device configuration included in the video generation system 1.

User input information UIN1, denoted as "User's Input" in FIG. 3, corresponds to information input by the user for image generation (also referred to as "input query"). Note that the input query is not limited to text (character information) and any information can be used. The input query may be any information including at least one of text, image, audio, and 3D data.

The video generation system 1 uses the user input information UIN1 to generate scenario generation information (also referred to as "first input information") that is used as input for model M1, denoted as "LLM" in FIG. 3, which will be described later. For example, model M1 is a first model that outputs scenario data in response to the input of the first input information. Any AI model, such as an LLM (large-scale language model), can be used for model M1, so long as it is capable of producing the desired output in response to the input. AI models such as model M1 will be described later.

The video generation system 1 generates a scenario FD1 by inputting first input information to the model M1 and having the model M1 output a scenario FD1, which is scenario data. Then, the video generation system 1 generates information required for USD generation SD1, which is code generation information (also referred to as "second input information") used as input for the model M3, using the scenario FD1, the output of the model M2 which uses the scenario FD1 as input, and user input information UIN2, etc. For example, the model M3 is a second model which outputs a code in response to the input of the second input information. The model M3 can be any AI model, such as an LLM (large scale language model), as long as it is capable of producing the desired output in response to the input.

Note that, although only one piece of information SD1 required for generating USD is illustrated in FIG. 3, there may be multiple pieces of information SD1 required for generating USD, for example, depending on the number of USD files to be generated. For example, there may be multiple pieces of information SD1 required for generating USD, depending on the number of USD files to be generated corresponding to the data structure shown in FIG. 18.

For example, model M2 may be a model that outputs a template or the like corresponding to scenario data in response to the input of that scenario data. Any AI model can be adopted for model M2 as long as it is capable of providing the desired output in response to the input. For example, user input information UIN2 may be information for specifying constraint conditions for image generation. Note that image generation system 1 may generate second input information using scenario FD1 and template input information, but this point will be described later.

The video generation system 1 generates Python code OD1 by inputting USD generation necessary information SD1 to the model M3 and having the model M3 output Python code OD1, indicated as "python" in FIG. 3. For example, the Python code OD1 is (program) code that generates USD format data (also called a "USD file") when executed. Note that Python is only an example, and any code format can be used, not limited to Python, as long as it can generate desired 3DCG data. Also, USD is only an example, and any format, such as FBX (Film Box), can be used as long as the data is for 3DCG. The video generation system 1 executes the Python code OD1 to generate a USD file OD2, indicated as "USD" in FIG. 3.

The video generation system 1 generates video data MV1, which is indicated as "PreMovie" in FIG. 3, by executing a rendering process PS1, which is indicated as "Renderer" in FIG. 3. For example, the video data MV1 is data (also referred to as "first video data") before executing a refinement process PS2, which will be described later.

The video generation system 1 generates video data MV2, denoted as "RefinedMovie" in FIG. 3, by executing a refinement process PS2, denoted as "Refiner" in FIG. 3. For example, the refinement process PS2 is a picture quality improvement process that improves the picture quality of the video data. The video data MV2 is data (also referred to as "second video data") obtained after the video data MV1, which is the first video data, is updated by the refinement process PS2.

The video production system 1 generates video data MV3, indicated as "FinalMovie" in FIG. 3, by executing composite editing PS3 using video data MV2, user input information UIN3, etc. For example, the composite editing PS3 executes a process of updating (editing) video data MV2 in response to a user's editing instruction indicated by user input information UIN3, thereby generating video data MV3 in which video data MV2 has been updated.

Note that the flow of the image generation process shown in FIG. 3 is merely an example, and the image generation system 1 can adopt any processing mode as long as it can generate video data from a user's input query. For example, in FIG. 3, the model M1 outputs code (Python code), but the model M1 may be a model that outputs data for 3DCG such as a USD file. In addition, the image generation system 1 may perform various types of image generation processing, not limited to the processing shown in FIG. 3. An example of this point will be described with reference to FIG. 4. FIG. 4 is a diagram showing another example of the flow of the image generation process of the present disclosure. FIG. 4 differs from FIG. 3 in that sound generation necessary information SD2 and text logo necessary information SD3 are generated and image generation processing is performed using them. Note that the same points as those described in FIG. 3 will not be described as appropriate.

In FIG. 4, the video generation system 1 uses a scenario FD1, the output of a model M2 that uses the scenario FD1 as input, and user input information UIN2 to generate information required for sound generation SD2, which is information for sound generation used as input for a model M4, denoted as "AI" in FIG. 3. For example, model M4 is a model that outputs various sound data in response to input of information required for sound generation SD2 and video data MV2. Any AI model can be used for model M4 as long as it is capable of producing the desired output in response to the input. Note that model M4 may be a model that uses only information required for sound generation SD2 as input.

The video production system 1 generates sound data corresponding to a video by inputting information SD2 required for sound production to the model M4 and having the model M4 output sound data AD1 for background music (BGM), sound data AD2 for sound effects, sound data AD3 for narration, etc. The video production system 1 may also generate sound data AD1, AD2, AD3 using user input information UIN4. For example, if the model M4 outputs sound data AD1, AD2, AD3 as one piece of sound data, the video production system 1 may extract sound data AD1, AD2, AD3 from the one piece of sound data output by the model M4 based on the specification in the user input information UIN4, and generate the sound data AD1, AD2, AD3.

In FIG. 4, the video generation system 1 uses a scenario FD1, the output of a model M2 that uses the scenario FD1 as input, and user input information UIN2 to generate information required for text logo generation SD3, which is information for generating a text logo to be used as input for model M5. For example, model M5 is a model that outputs at least one of text and a logo in response to the input of information required for text logo generation SD3. Any AI model can be used for model M5 as long as it is capable of producing the desired output in response to the input.

Video generation system 1 inputs information SD3 required for text logo generation to model M5, and generates text logo data corresponding to the video by having model M5 output text logo data DI1 for Text, text logo data DI2 for Logo, etc.

The video generation system 1 generates video data MV3 by executing composite editing PS3 using video data MV2, audio data AD1, AD2, AD3, text logo data DI1, DI2, user input information UIN3, etc. For example, the composite editing PS3 executes a process of combining (combining) the video data MV2, audio data AD1, AD2, AD3, text logo data DI1, DI2, etc. into one and generating video data MV3 as a single image.

The above-mentioned Figures 3 and 4 show an example of processing in an initial state where there is no scenario, information required for generating USD, USD, PreMovie, RefinedMovie, etc. As described above, the video generation system 1 generates a prompt for generating a scenario based on a user's input query, provides the prompt to a natural language model to generate a scenario, generates a prompt for outputting code constituting a video from text information described in the scenario, provides the prompt to the natural language model to generate code constituting the video, and generates a video. In this way, when creating a video, the video generation system 1 inputs what the user wants to create or the purpose, and generates an effective video storyboard and video even without knowledge of 3DCG or video production. In addition, the video generation system 1 makes it easier to edit afterwards by creating a storyboard. These points will be described in detail later.

The video production system 1 may also perform various processes related to video production. For example, the video production system 1 may perform evaluation processing on the generated information. In this regard, an example of the flow of the evaluation processing will be described with reference to FIG. 5. FIG. 5 is a diagram showing an example of the flow of the evaluation processing of the present disclosure.

In FIG. 5, the video generation system 1 inputs at least one of the scenario FD1, information required for USD generation SD1, information required for sound generation SD2, and information required for text logo generation SD3, and causes the model M10 to output evaluation text information EV1 indicating an evaluation of the input information, thereby evaluating the generated information. For example, the model M10 is a model that outputs an evaluation of input information in response to the input of information. For example, the model M10 outputs evaluation text indicating an evaluation of the input scenario FD1 in response to the input of a scenario FD1. Note that the model M10 may be a model that accepts input of each of the scenario FD1, information required for USD generation SD1, information required for sound generation SD2, and information required for text logo generation SD3, or may be a model that accepts input of a combination of these pieces of information. The model M10 may also be a model that, in response to the input of information indicating a video (such as a caption), outputs an evaluation of the video corresponding to the input information.

From here, specific examples of each process executed by the video production system 1 will be described for the above-mentioned process flow. Note that explanations of points similar to those described above will be omitted as appropriate.

For example, the video production system 1 generates scenario generation information (first input information) as shown in FIG. 6. FIG. 6 is a diagram showing an example of a generation process of scenario generation information. In FIG. 6, the video production system 1 acquires user input information IDT1 and IDT2 input by the user to content CT1 as user input information. Content CT1 is content for accepting user input information for each of the questions "What kind of video do you want to create?" and "Style."

For example, the client UI display unit 400 displays the content CT1, and the sensor unit 300 accepts the user input information IDT1 and IDT2 as user input information. For example, the user input information IDT1 and IDT2 correspond to the user input information UIN1 in Figures 3 and 4.

In FIG. 6, the client UI display unit 400 displays the question "What kind of video do you want to make?". In response to the question "What kind of video do you want to make?", the sensor unit 300 accepts user input information IDT1, "a 15-second sneaker commercial video." The client UI display unit 400 also displays the question "style." In response to the question "style," the sensor unit 300 accepts user input information IDT2, "cinematic."

The video production system 1 may accept user input information in any manner, or may accept a user selection from multiple candidates. For example, the video production system 1 may convert information input by the user from a keyboard or microphone into text information and accept it as user input information. The video production system 1 may also accept, in addition to free text, settings such as the number of seconds for the entire video, style, and camerawork, as well as other files such as images and videos, as user input information.

The video production system 1 generates a prompt PT1, which is scenario generation information (first input information), using the user input information IDT1, IDT2, and a template TP1, which is template input information. For example, the template TP1 may be preset, or may be selected from a plurality of template candidates. For example, the video production system 1 may select a template that corresponds to the user's input information from among the plurality of template candidates. For example, the video production system 1 may select a template TP1 related to a movie-style advertisement from among the plurality of template candidates based on the contents indicated by the user input information IDT1, IDT2.

For example, the video generation system 1 generates a prompt PT1 by reflecting the user input information IDT1 and IDT2 in the template TP1. In FIG. 6, the video generation system 1 generates the prompt PT1 by adding "cinematic" indicated by the input information IDT2 to the style item of the constraint condition, and adding "15-second sneaker commercial video" indicated by the input information IDT1 to the input sentence. In this way, the video generation system 1 generates a prompt for generating a scenario based on the user's input information. Note that the user's input information may be input on a single screen, or may be input by answering several questions, and examples of these points will be described later.

The video production system 1 also generates scenario data as shown in FIG. 7. FIG. 7 is a diagram showing an example of a scenario data generation process. In FIG. 7, the video production system 1 generates scenario data SN1 using a prompt PT1. For example, the scenario data SN1 corresponds to the scenario FD1 in FIG. 3 and FIG. 4. The scenario data SN1 includes information such as the number of seconds for each scene, such as the opening scene, the scene where sneakers are put on, and an explanation of the cut.

For example, the video generation system 1 generates scenario data SN1 by inputting a prompt PT1 into a model M1, such as an LLM, and having the model M1 output scenario data SN1. In this way, the video generation system 1 generates a scenario by inputting the generated prompt into an AI (such as an LLM). In addition to the information shown in FIG. 7 (also called "scenario information"), the scenario data SN1 also includes information on the environment, characters, motion, camera work, lighting, color, and the like. For example, by inputting the user's past experience learning data or learning data of a specific director or person into the model M1, etc. using RAG (Retrieval-Augmented Generation) or fine tuning so that the scenario generation has characteristics, the video generation system 1 can generate various scenario variations.

The video production system 1 also generates code generation information (second input information) as shown in FIG. 8. FIG. 8 is a diagram showing an example of a process for generating code generation information. The video production system 1 generates a prompt PT2, which is code generation information (second input information), using scenario data SN1 and a template TP2, which is template input information. For example, the template TP2 may be preset or may be selected from a plurality of template candidates. For example, the video production system 1 may select a template corresponding to a scenario from among the plurality of template candidates. For example, the video production system 1 may select a template TP2 related to a commercial from among the plurality of template candidates based on the contents indicated by the scenario data SN1.

For example, the video production system 1 generates a prompt PT2 by reflecting the scenario data SN1 in the template TP2. In FIG. 8, the video production system 1 generates a prompt PT2 by adding information indicated by the scenario data SN1 to the input sentence. In this way, the video production system 1 generates a prompt for a video based on a scenario generated by AI.

For example, FIG. 8 shows an example of prompt generation for converting a scenario related to a person into USD-Python. Conversion into USD-Python is merely one example of a conversion format, and the conversion is not limited to USD-Python and may be any conversion format. For example, the conversion format may be Python for Blender, USD, or other formats. Furthermore, the video generation system 1 may generate prompts individually for targets such as people, environments, and camerawork, or may generate prompts collectively. Furthermore, the generated prompt may include paths to assets and motions to be used, or may include source code or APIs (Application Programming Interfaces) for throwing (inputting) the assets/motions into an AI generation algorithm.

Then, the video generation system 1 generates a USD-Python file by inputting the generated prompt into an AI (such as LLM). Note that the file format is not limited to Python format, and files may be generated in other formats such as USD. The video generation system 1 then converts the file into a format that can be rendered, such as a USD file, and generates a PreMovie (a video file such as mp4) by performing rendering.

The video production system 1 may perform composite editing using PreMovie, but may also perform refinement processing, which is an example of image quality improvement processing, on the PreMovie, as shown in FIG. 9. FIG. 9 is a diagram showing an example of image quality improvement processing. In FIG. 9, the video production system 1 generates a second video OT1 from a first video IN1 by refinement processing using a model M11, which is a diffusion model that receives a first video IN1, which is a PreMovie, as input, and outputs a second video OT1, which is a RefinedMovie.

The AI model (such as model M11) used in the refiner process is not limited to the diffusion model, and any AI model such as the latent diffusion model (LDM) or latent consistency model (LCM) can be used. In addition, techniques such as AnimateDiff (time directional stabilization) and ControlNet (line art control) may be used in the refiner process. Through this refiner process, the video production system 1 can improve the quality of the video while maintaining the consistency of the characters, backgrounds, props, etc.

In addition, the refiner process may use prompts in addition to videos. For example, the model M11 may input prompt IN2 in addition to the first video IN1. For example, when a target such as a woman in her 30s is specified by prompt IN2, the model M11 outputs a second video OT1 in which the parts of the first video IN1 that represent the woman in her 30s have been improved. This allows the video generation system 1 to generate a second video OT1 in which the image quality, etc., of the target specified by prompt IN2 in the first video IN1 has been improved.

From the user's perspective, the above-mentioned processing of the video production system 1 appears to generate a scenario (storyboard) and video for each cut after input, and the processing in between is closed within the system. This processing may be performed in one go from the user's input text, from the scenario to all the cuts and refinement processing, or the user may input preferences during the processing. For example, the video production system 1 may generate several scenarios with only the outline, then the user may select one, and detailed scenarios and video generation processing may be performed based on the selected outline scenario. Also, the video production system 1 may generate several patterns of characters for the video to be generated before video generation after scenario generation, and after the user selects one, the video may be rendered and refined.

The components of a scenario may include videos of cuts, a representative image (such as the first frame of a video), a description of the cut, characters (visuals, setting, etc.), the motions of each character, lighting, camera work, background environmental information, transitions between cuts, dialogue, narration, etc. Some of these are presented to the user, while others are not presented to the user and are kept for processing purposes. The scenario is arranged in chronological order by cut.

Currently, various existing video generation services are provided, such as Pika, Runway Gen-2, Lumiere, and Stable Video Diffusion. These generate videos using a diffusion model that moves vectors in the spatial and temporal directions from an image. These generate videos using only 2D images. On the other hand, video generation system 1 stores 3D information internally. For example, with existing video generation services, it is possible to modify only a specified (X, Y) area in a video, but there is an issue that even if you want to change only the color of the clothes, the motion also changes. On the other hand, video generation system 1 stores 3D information internally, so it is possible to modify only the targeted parts, such as only the motion, only the lighting, or only the color of a person's clothes.

In addition, existing video generation services only generate videos for each cut, and users must ensure the consistency of each cut themselves. However, video generation system 1 can consistently carry out everything from the scenario (storyboard) to video generation and editing, and can generate videos with consistency in terms of actors, backgrounds, color grading, etc.

<1-3. User interface>
From here, we will describe the user interface (UI) for users who use the image generation system 1. Note that explanations of points similar to those described above will be omitted where appropriate.

As shown in FIG. 10, the video generation system 1 provides the user with content CT11. FIG. 10 is a diagram showing an example of a user interface. The content CT11 is a display screen (content) for receiving user input information. For example, the client UI display unit 400 displays the content CT11. The user inputs text (the "Prompt" column in FIG. 10) instructing what kind of video to create and a style selection (the "Style" column in FIG. 10) as user input information via the content CT11. In this way, the user inputs text and style selection as user input information. For example, the user inputs text information in the "Prompt" column with reference to example sentences included in the content CT11. For example, the user selects a style to use from multiple style candidates displayed by pressing (clicking, etc.) the downward triangle in the "Style" column.

When the user has completed inputting the user's input information, the user selects the button labeled "Ask AI Director" in FIG. 10 to instruct the video production system 1 to generate a video in accordance with the user's input information. This causes the video production system 1 to execute the process of generating a video in accordance with the user's input information.

As shown in FIG. 14, the video generation system 1 provides the user with content CT15 related to the generated video. FIG. 14 is a diagram showing an example of a user interface. The content CT15 is a storyboard screen (content) for accepting user operations (instructions) for the generated video. As shown in FIG. 14, the content CT15 is a storyboard screen that displays video data for each cut of the generated video. For example, the client UI display unit 400 displays the content CT15. In this way, the video generation system 1 provides a UI that outputs a storyboard and video according to information input by the user. The user sets the video, content, narration, dialogue, camera work, background music, lighting, color, etc. for each cut on the storyboard screen.

The video generation system 1 may accept user input information while asking the user a question. For example, when the button labeled "Ask AI Director" in FIG. 10 is selected, the video generation system 1 accepts user input information through a conversation (dialogue) with the user, as shown in FIGS. 11 to 13. FIGS. 11 to 13 are diagrams showing an example of a user interface. Content CT12 in FIG. 11 is a display screen (content) that presents samples generated in response to the user's input information entered in FIG. 10, and asks the user whether there is any that is close to the image they have in mind. For example, the client UI display unit 400 displays the content CT12.

Content CT13 in FIG. 12 is a display screen (content) that requests (asks) detailed targets, etc. in response to the user's response (input information) that none of the samples presented in content CT12 in FIG. 11 match the image. For example, the client UI display unit 400 displays content CT13.

Content CT14 in FIG. 13 is a display screen (content) that presents samples that have been regenerated in response to a user's answer (input information) that specifically specifies a target, etc., and asks the user whether there is one that is close to the image. For example, the client UI display unit 400 displays content CT14. FIG. 13 shows a case in which the user places the mouse cursor over the leftmost sample video of the four samples and performs a designation operation such as clicking, thereby designating that the leftmost sample video of the four samples is the video that is close to the image. As a result, the video generation system 1 executes the process of generating a video in response to the user's input information that designates the leftmost sample video of the four samples.

In this case, the video generation system 1 provides the user with content CT15 related to the generated video, as shown in FIG. 14. For example, the client UI display unit 400 displays the content CT15. In this way, when generating a storyboard and video using the initial input content, if the video generation system 1 does not have enough necessary information, it may work out the details while collecting the necessary information through conversation (dialogue) with the user.

As shown in Figs. 15 and 16, the video generation system 1 may also have the user input a short sentence about the video they wish to create, generate several video stories based on the short sentence, and allow the user to select the one they like best. Figs. 15 and 16 show an example of a user interface.

As shown in FIG. 15, the video generation system 1 provides the user with content CT21. Content CT21 is a display screen (content) for accepting information input by the user. For example, the client UI display unit 400 displays the content CT21. The user inputs a sentence (short sentence) indicating what kind of video they want to create as user input information via the content CT21. For example, the user inputs text information into an input field in the content CT21.

When the user has finished inputting the user's input information, the user selects the button marked "Start" on the right side of the input field in content CT21 to instruct video production system 1 to generate a video story according to the user's input information. This causes video production system 1 to execute the process of generating a video story according to the user's input information.

As shown in FIG. 16, the video generation system 1 provides the user with content CT22 relating to the story of the generated video. The content CT22 in FIG. 16 is a display screen (content) that presents a sample of the story of the video generated in response to the user's input information entered in FIG. 15. For example, the client UI display unit 400 displays the content CT22. For example, if the user likes one of the video story samples, the user selects a button labeled "Continue" on the right edge of the display area for that sample, thereby instructing the video generation system 1 to generate a video corresponding to the selected sample.

If the user does not like any of the video story samples, the video generation system 1 will generate a different pattern again. For example, if the user does not like any of the video story samples, the user can select the button labeled "Continue" on the right edge of the short sentence display area to instruct the video generation system 1 to generate a different pattern of video story sample again. This causes the video generation system 1 to generate a different pattern of video story sample.

<1-4. Processing example>
From here, in addition to the specific examples described above, specific examples of each process executed by the video production system 1 will be described. Note that explanations of points similar to those described above will be omitted as appropriate. Below, specific examples of the generation process of sound, etc., and the evaluation process in the processing of the video production system 1 described above will be described. Note that explanations of points similar to those described above will be omitted as appropriate.

<1-4-1. Sound generation example>
For example, the video production system 1 generates information for sound generation (corresponding to the information required for sound generation SD2 in FIG. 3 and FIG. 4) as shown in FIG. 17. FIG. 17 is a diagram showing an example of a process for generating information for sound generation. The video production system 1 generates a prompt PT3, which is information for sound generation, using scenario data SN1 and a template TP3, which is template input information. For example, the template TP3 may be preset or may be selected from a plurality of template candidates. For example, the video production system 1 may select a template corresponding to a scenario from among the plurality of template candidates. For example, the video production system 1 may select a template TP3 related to a commercial from among the plurality of template candidates based on the contents indicated by the scenario data SN1.

For example, the video production system 1 generates a prompt PT3 by reflecting the scenario data SN1 in the template TP3. In FIG. 17, the video production system 1 generates a prompt PT3 by adding information indicated by the scenario data SN1 to an input sentence. For example, the video production system 1 generates a prompt for extracting several key phrases from the scenario in order to generate a sound (BGM, etc.) that matches the scenario. The video production system 1 may input the generated prompt to an AI (LLM, etc.) to obtain keywords and text information required for sound generation.

The video generation system 1 generates sound data using the generated prompt PT3. For example, the video generation system 1 generates sounds (sound data) such as background music, sound effects, narration, and dialogue based on the generated video and text information written on the storyboard (scenario data SN1, etc.). Furthermore, if necessary words (text information) such as dialogue or narration have already been extracted from the scenario, the video generation system 1 may not generate a prompt and may instead save the words (text information) as sound data for dialogue, narration, etc.

For example, the video generation system 1 generates sound data from the necessary information for sound generation (information for sound generation), video, and audio data input by the user (sound source, user's voice, humming, etc.). For background music and sound effects, the video generation system 1 may generate sound from text or video using a transformer (model) such as text to music generation. The video generation system 1 may also search for contrastively learned sound sources such as text to music estimation from natural text.

In addition, for lines and narration, the video generation system 1 may generate audio using text to speech (such as a diffusion model or flow matching) based on the words themselves and text information about the characters obtained from the scenario. In addition, when linking the generated video and audio, the video generation system 1 may incorporate meta information into the video or audio file to indicate the start and end times, volume, etc. of the audio.

<1-4-2. Example of text logo generation>
For example, video production system 1 generates information for generating a text logo (corresponding to information required for text logo generation SD3 in FIGS. 3 and 4). Based on the generated storyboard (scenario data SN1, etc.), video production system 1 generates text and logo information (captions, titles, logos, descriptions, etc.) to be displayed on the video.

For example, the text sentence to be displayed may be clearly stated in the generated scenario, but if it is not clearly stated, the video generation system 1 generates a prompt to generate text information (text logo data, etc.) and sends it to an AI (such as an LLM) to generate the text information. In addition, if the user inputs the text to be displayed, the video generation system 1 may use the information input by the user as text information (text logo data, etc.).

The font, size, and position of the text display may be determined by any method. For example, the video generation system 1 may determine the font, size, and position of the text display using any AI such as a diffusion model, a variational auto-encoder (VAE), generative adversarial networks (GAN), DALL E, StyleGAN, StyleGAN2, Pix2Pix, TransGAN, or LLM. The font, size, and position of the text display may also be manually set by the user.

Regarding logos and images, the user may input images or videos in jpeg or mp4 format, etc. Also, regarding logos and images, the video generation system 1 may generate a prompt for image generation and send it to any AI such as a Diffusion model, VAE, GAN, DALL E, StyleGAN, StyleGAN2, Pix2Pix, TransGAN, or LLM to generate logo information.

In addition, when linking text or logo information with a video, the video generation system 1 may incorporate meta information into the video or into the text or logo itself to clearly indicate (meta information) the start and end times, position, and size of the text or logo.

<1-4-3. Re-learning example>
Furthermore, when generating a scenario or image, the image generation system 1 may generate a scenario or image that can only be produced based on specific replaceable learning data. For example, it is possible to re-learn as learning data based on images and videos produced in the past. By re-learning with RAG or fine tuning, it is possible to change the scenario or image to be generated. In other words, the image generation system 1 can re-learn the data (history) of an individual user produced in the past, and can also generate images and images using a model re-learned as learning data using the works of a specific movie director.

The learning data can also be re-learned on an individual's PC or on a server. The learning data is used to train a number of models, such as the LLM and Diffusion model. In cases such as when it is desired to generate an overall scenario using director A, but to generate color grading for the video color using a different director B, the video generation system 1 may re-learn specific parts using different learning data.

<1-4-4. USD update example>
As described above, the video production system 1 modifies the 3DCG assets and rendering method that are the source of the existing video, based on the user's input information and the text information of the generated storyboard. The video production system 1 then , a prompt for outputting a code for constructing a new video is output, and the prompt is provided to the natural language model to output the code for constructing the video, thereby generating the video. In this way, the video generation system 1 However, even if you have no knowledge of 3DCG or video production, you can modify the image according to the user's input.

After generating a video (video), video generation system 1 stores scenario information, video information, and the like as text and video. This allows the user to modify this information with input information (text, sensors, etc.).

For example, in the video production system 1, USD files are stored separately for each asset and motion, as shown in FIG. 18. FIG. 18 is a diagram showing an example of a USD file. For example, in the data structure shown in FIG. 18, a USD in a higher hierarchical level may include a path (file path) to a USD in a lower hierarchical level.

For example, the entire USD includes paths to the environment assets USD, human assets UDS, camera USD, etc. The environment assets USD also include paths to the building assets USD and prop assets USD. The building assets USD also include mesh information of the building itself, etc. The human assets USD also include paths to human mesh information and motion USD. In this way, data for 3DCG (3D data) such as a USD file may include multiple data sets. Note that the configuration (data structure) of the USD file shown in FIG. 18 is merely an example, and any configuration can be adopted, and the entire USD (USD file) may be configured as one block (one data set).

When modifications are made, the video production system 1 analyzes the user's modification information using AI (such as LLM) according to the processing flow shown in FIG. 19, and the processing differs depending on whether the USD is replaced or part of the USD is modified. After the modification, rendering processing and refinement processing are executed. FIG. 19 is a flowchart showing the processing procedure executed by the video production system. As a specific example, FIG. 19 is a flowchart showing the processing procedure related to rewriting a USD file.

First, the video generation system 1 accepts correction information input by the user (step S101). For example, the sensor unit 300 accepts input information instructing the user to make corrections. The video generation system 1 analyzes the input using AI (step S102). For example, the video generation module 100 analyzes the content of the input information instructing the user to make corrections using various models, etc.

The video production system 1 recognizes the format of the existing USD file (step S103). For example, the video production module 100 recognizes the format of the USD file before modification. The video production system 1 determines whether or not to modify a part of the USD (step S104). For example, the video production module 100 determines whether or not to modify a part of the USD based on the content of the input information instructing the user to modify and the format of the existing USD file.

When a portion of the USD is to be modified (step S104: Yes), the video generation system 1 generates a prompt for generating a modified USD-Python (step S105). For example, when a portion of the USD is to be modified, the video generation module 100 generates a prompt for generating the modified USD-Python using a template or the like for generating the modified USD-Python.

The video generation system 1 generates USD-Python using AI (LLM, etc.) (step S106). For example, the video generation module 100 generates USD-Python by inputting a prompt into a model for generating USD-Python.

The video production system 1 replaces the USD file to be modified (step S107). For example, the video production module 100 replaces the USD file to be modified by reflecting the generated USD-Python in the USD file to be modified. In this manner, the video production system 1 updates at least one of the multiple data sets of the USD file. For example, the video production system 1 executes a process of updating a portion of the multiple data sets of the USD file.

The video generation system 1 executes the rendering process using the updated USD file (step S108). For example, the video generation module 100 executes the rendering process using the rewritten, i.e., modified, USD file.

On the other hand, if the image generation system 1 does not modify any part of the USD (step S104: No), it generates a prompt for generating the USD-Python for creation (step S109). For example, if the image generation module 100 does not modify any part of the USD, i.e., if it creates (generates) a new USD, it generates a prompt for generating the USD-Python for creation using a template or the like for generating the USD-Python for creation.

The video generation system 1 generates USD-Python and USD using AI (LLM, etc.) (step S110). For example, the video generation module 100 generates USD-Python and USD by inputting a prompt into a model for generating USD-Python.

The video production system 1 replaces the USD file to be modified (step S111). For example, the video production module 100 replaces the generated USD with the USD file to be modified. In this manner, the video production system 1 executes a process to update the USD file. For example, the video production system 1 executes a process to update the entire multiple data sets of the USD file. Then, the video production system 1 executes the process of step S108. For example, the video production module 100 executes a rendering process using the replaced, i.e., modified, USD file.

Now, the user interface (UI) related to the above-mentioned modifications will be described. As shown in FIG. 20, the video generation system 1 provides the user with content CT31. FIG. 20 is a diagram showing an example of a user interface. The content CT31 is a display screen (content) for presenting information related to the generated video, such as a storyboard, and for accepting modifications from the user. For example, the client UI display unit 400 displays the content CT31.

The user inputs information instructing corrections to be made to the generated video via content CT31. For example, when the user clicks on a specific part of the video and inputs corrections in text, the video generation system 1 updates the USD and updates (changes) the video to reflect the corrections.

Figure 20 shows a case where the user selects the top cut (thumbnail) image and instructs a modification to "child skips and runs to mother." In this case, the video production system 1 updates the USD for the part of the generated video that corresponds to the top cut (thumbnail) image based on the modification instruction to "child skips and runs to mother," and generates a video that reflects the modification content.

Note that the above UI is merely an example, and the video generation system 1 may accept the user's correction instructions in various ways. For example, as shown in FIG. 21, the video generation system 1 may provide the user with content CT32 and accept the user's correction instructions. FIG. 21 is a diagram showing an example of a user interface. The content CT32 is a display screen (content) for accepting the user's correction instructions. For example, the client UI display unit 400 displays the content CT32.

Also, as shown in FIG. 22, the video generation system 1 may provide the user with content CT33 and accept the user's correction instructions. FIG. 22 is a diagram showing an example of a user interface. The content CT33 is a display screen (content) for accepting the user's correction instructions. For example, the client UI display unit 400 displays the content CT33.

The user inputs information instructing corrections to the generated video via content CT32 or content CT33. As a result, the video generation system 1 accepts the correction instructions from the user, updates the USD based on the correction instructions, and generates a video that reflects the corrections. For example, the user can select a person or object that appears in the video, and edit the motion settings for the selected target in text. The user can also change the assets of the selected target. The user can also edit the camera work in text. In addition to the above, the user can also edit the background and lighting settings of the video in text.

Although the above description is given as an example of a moving image, the video generation system 1 may also perform correction processing on sound information (BGM, SE, narration, dialogue, etc.) and text logo information, etc., based on correction instructions from the user.

<1-4-5. Evaluation example>
Furthermore, the video production system 1 executes an evaluation process. For example, the video production system 1 generates an evaluation text for scenario data using an AI model such as model M10. For example, the video production system 1 may generate scenario data again based on an instruction for editing (correction, etc.) made by a user based on the evaluation text generated for the scenario data. The video production system 1 presents the generated evaluation text. This allows the user to create a video while viewing the evaluation.

For example, the video generation system 1 presents evaluation text about the scenario data to the user, and accepts editing instructions for the scenario data from the user who has confirmed the presented evaluation text. The video generation system 1 generates scenario data based on the editing instructions accepted from the user. For example, the video generation system 1 changes (updates) the contents of the scenario data based on the editing instructions accepted from the user. The video generation system 1 generates code based on the scenario data generated based on the evaluation text. The video generation system 1 generates video data using the code generated based on the evaluation text.

The video generation system 1 may automatically generate (update) at least one of the scenario data or the code based on the evaluation text. For example, the video generation system 1 may change the content of the scenario data to correspond to the content indicated by the evaluation text for the scenario data. For example, when the evaluation text for the scenario data indicates that the orientation of a certain character is not good, the video generation system 1 generates scenario data in which the orientation of the character is changed. Note that the above is merely one example, and the video generation system 1 may generate at least one of the scenario data or the code by appropriately using the evaluation text.

In the video generation system 1, after generating a video (video), scenario information, video information, sound information, text/logo information, etc. are stored in text, video files, sound files, image files, etc. Therefore, the video generation system 1 can perform evaluation processing using these information. The video generation system 1 modifies the 3DCG assets and rendering method that are the source of the existing video based on the user's input information, the text information of the generated storyboard, and the evaluation text, outputs a prompt for outputting the code that constitutes the new video, provides the prompt to the natural language model, outputs the code that constitutes the video, and generates the video. For example, the video generation system 1 may output a prompt for generating evaluation text (corresponding to the evaluation text information EV1 in FIG. 5) based on the user's input information and the text information of the generated storyboard, and provide the prompt to the natural language model to generate the evaluation text. For example, the video generation system 1 may generate an evaluation prompt based on the scenario information and input it to the AI (LLM, etc.) to generate an evaluation text that indicates the evaluation of the scenario.

In addition, when evaluating the composition of a video, the video generation system 1 acquires a caption for the video by inputting one frame of the video into AI (Contrastive Captioner Model, Image Captioning Model, etc.). Then, the video generation system 1 generates an evaluation text indicating an evaluation of the composition of the video by inputting the acquired caption and scenario text into AI (LLM, etc.). For example, the video generation system 1 may evaluate whether the image is as per the caption by comparing the acquired caption and scenario text with AI (LLM, etc.). The video generation system 1 may also accept a specification of the evaluator type that the user desires to be the one to evaluate. For example, the video generation system 1 may perform an evaluation from the perspective of a market strategy, copywriter, video director, a specific person, etc.

<1-4-6. Example of multiple cut selection>
From here, some examples of the editing process will be described. In the conventional technology, there is a problem that multiple cuts cannot be simultaneously corrected on a storyboard. In this way, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video production system 1 may select multiple cuts and perform the editing process (correction process) as shown in FIG. 23. FIG. 23 is a diagram showing an example of the editing process. As a specific example, FIG. 23 is a diagram showing an example of the editing process based on the selection of multiple cuts.

As shown in FIG. 23, the video production system 1 may provide the user with content CT41 and accept correction instructions according to the user's selection of multiple cuts. The content CT41 is a display screen (content) for accepting the user's correction instructions for a video including multiple cuts (scenes) such as cuts CU1 to CU4. For example, the client UI display unit 400 displays the content CT41.

The user selects multiple cuts from among cuts CU1 to CU4 via content CT41, and inputs information instructing corrections to the selected multiple cuts. For example, the user selects cuts CU1 to CU3 by performing an operation to select the range in which cuts CU1 to CU3 are displayed (such as by surrounding it with a line) among cuts CU1 to CU4 in content CT41, or by clicking on each of cuts CU1 to CU3.

After selecting cuts CU1 to CU3, the user inputs a prompt (text information, etc.) indicating a correction instruction as user input information, thereby instructing the video production system 1 to correct cuts CU1 to CU3. The video production system 1 executes corrections for cuts CU1 to CU3 in response to the correction instruction from the user. This allows the user to select multiple cuts and correct the configuration and content of the selected cuts by inputting a prompt. For example, the scenario information for each cut includes the content of the cut, the characters, the time period during which the cut was shot, etc., and the video production system 1 regenerates the scenario by providing the user's input information and scenario information to the AI (such as LLM). Note that the video production system 1 executes any correction process based on the content of the user's correction instruction. For example, the video production system 1 may update the USD file of the video as necessary, or may change only the order of the cuts while leaving the USD file as is. The above-mentioned process allows the video production system 1 to improve usability.

<1-4-7. Examples of using time information>
In the conventional technology, there is a problem that after a specific cut is corrected, other scenes affected by the correction are not corrected (changed). Thus, the conventional technology has a problem with usability, and there is room for improvement in usability. Therefore, the video production system 1 may perform an editing process using time information as shown in FIG. 24. FIG. 24 is a diagram showing an example of the editing process. As a specific example, FIG. 24 is a diagram showing an example of an editing process based on the correction content determined according to the time information. For example, the video production system 1 puts date and time information (also called "time information" or "date information") generated by AI into each cut, and determines the cutting of the video and the correction content based on the information.

In FIG. 24, the video production system 1 manages each cut by associating it with date and time information (time information) corresponding to that cut. For example, cut CU11 is associated with time information TI11 indicating 10:31 on December 21, 2023. Cut CU12 is associated with time information TI12 indicating 10:41 on December 21, 2023. In this way, cut CU11 and cut CU12 are cuts that are close in time (close to each other). In this case, if cut CU11 is modified, the video production system 1 also reflects the modification in cut CU12. Time information (date information) is associated with each cut in the video data.

For example, when the clothing of person X in cut CU11 is modified, the video production system 1 also modifies the clothing of person X in cut CU12 in the same way as the clothing of person X in cut CU11. For example, the video production system 1 compares the time information between each cut, and when there is a cut (also called an "affected cut") whose time difference with the modified cut (also called a "cut to be modified") is within a predetermined range, it determines that the modification based on the modification content in the cut to be modified should also be reflected in the affected cut.

Then, the video production system 1 performs corrections on the affected cuts based on the correction content in the cut to be corrected. Note that the video production system 1 may perform corrections (changes) on the affected cuts based on the influence between cuts not only on human assets, but also on environmental assets (weather, lighting, props such as candles that change over time, etc.).

For example, cut CU21 is associated with time information TI21 indicating 10:31 on December 21, 2023. Also, cut CU22 is associated with time information TI22 indicating 18:41 on December 21, 2023. In this way, cut CU21 and cut CU22 are cuts that are distant in time (separate). In this case, when cut CU21 is modified, the video production system 1 does not reflect the modification in cut CU22.

For example, when the clothing of person X in cut CU21 is modified, the video production system 1 does not modify the clothing of person X in cut CU22 in response to the modification of the clothing of person X in cut CU21. For example, the video production system 1 compares time information between each cut, and if there is no cut (affected cut) whose time difference with the modified cut (cut to be modified) is within a predetermined range, it determines that the modification based on the modification content of the cut to be modified will not be reflected in other cuts.

For example, when generating a scenario, the video production system 1 generates date and time information for each cut using AI (such as LLM) and saves it as meta information for each cut. This date and time information is used to consider the relationship with the previous and next cuts and to understand the season, etc. The generated fictitious date and time information is included to maintain the content of each cut and the relationships between each cut. For example, if the shot is taken on a morning when it is snowing, the video production system 1 may set it to 6:30 AM on January 24th. Also, if the shot has a strong relationship with the previous and next cuts, the video production system 1 may set it to nearby time periods on the same day, such as 6:30 AM on January 24th and 7:00 AM on January 24th.

For example, the video generation system 1 changes the clothes worn by the user, the sun's lighting settings, the haze of the air, etc. depending on the season and time. Also, if the date and time information of the target cut and the previous cut are close, changing the target cut will also affect the previous and next cuts. For example, if the same person appears in another cut and the time is close to that of the target cut, changing the clothes in the target cut will also change the clothes in the other cut. Through the above-mentioned processing, the video generation system 1 can improve usability.

<1-4-8. Response examples according to the degree of verbalization>
In the conventional technology, when the user cannot specifically verbalize the content of the correction, it is difficult to make corrections according to the user's intention. Thus, the conventional technology has problems with usability, and there is room for improvement in usability. Therefore, the image generation system 1 may enable corrections according to the user's intention by the process described below, even when the user cannot specifically verbalize the content of the correction.

For example, the video production system 1 may change the response method depending on the input text. As shown in FIG. 25, the video production system 1 may vary the response depending on the level of abstraction and verbalization of the user's instructions. FIG. 25 is a diagram showing an example of an editing process. As a specific example, FIG. 25 is a diagram showing an example of an editing process depending on the level of verbalization.

Response example AP1 in FIG. 25 shows a case where a user issues a purpose-based correction instruction such as "Please make the video so that the viewer feels like they are in an everyday situation." In this case, image generation system 1 generates an image with a specific instruction text attached. For example, in response to a purpose-based correction instruction such as "Please make the video so that the viewer feels like they are in an everyday situation," image generation system 1 adds a text such as "Lower your hands at a natural angle and move them in accordance with the direction of your body," and presents the video corrected based on the text to the user by displaying it on a display or the like.

Response example AP2 in FIG. 25 shows a case where the user instructs to correct an abstract sentence such as "Keep both arms in a natural position." In this case, the image generation system 1 presents a number of concrete sentences and allows the user to select one. For example, the image generation system 1 presents to the user a number of sentences such as "Put your hands down and move them in the direction of the sky," "Scratch your head with your hands and then lower your hands," and "Put your hands in your pockets" by displaying them on a display or the like. The image generation system 1 then presents to the user an image that has been corrected based on the sentence selected by the user from the multiple sentences by displaying them on a display or the like.

Response example AP3 in FIG. 25 shows a case where the user gives a specific instruction to correct a sentence, such as "Please look at the car in front of you on the right in 2 seconds." In this case, the image generation system 1 presents the image to the user by displaying on a display or the like an image that has been changed (corrected) according to the sentence entered by the user. For example, the difference between a purpose-based sentence, an abstract sentence, and a specific sentence may be determined by an AI (such as an LLM), and the image generation system 1 may change the prompt it generates accordingly and pass the processing to the AI (such as an LLM).

For example, the video production system 1 may allow the user to specify motion or camera movement using a sensor. As shown in FIG. 26, the video production system 1 may superimpose (overlap) motion information obtained from an arbitrary sensor such as a web camera on the video and accept user corrections. FIG. 26 is a diagram showing an example of editing processing. As a specific example, FIG. 26 is a diagram showing an example of editing processing by superimposing a display on the video.

For example, the video generation system 1 displays motion information MT obtained from any sensor such as a mobile motion capture or a web camera superimposed on the video MV11 and accepts correction instructions from the user. In this way, the video generation system 1 displays motion information including facial expressions etc. superimposed on the current video, and accepts correction instructions from the user based on a visualized state of the difference between the current video and the motion information. For example, if it is a four-second cut, the four-second cut will always continue to be played, and the user can change the motion information themselves as many times as necessary until they are satisfied. For example, if there is motion information that the user likes, the video generation system 1 may finally generate natural motion information to change the motion of the video.

The user's motion is tracked using a mobile motion capture device, a web camera, etc. The user may also select in advance from the UI with the mouse which character's motion they wish to modify. The size and position of the portion of the user's motion to be superimposed is determined based on the size and orientation of the head and body in the video. The user may adjust the size, position, and orientation in detail using the mouse and keyboard. Motion may be recorded by repeating the cuts shown in the image above, or by starting recording several seconds after the start button is pressed and the user adjusts their position. After filming, the user may press the motion generation button, and the input motion may be applied to the specified character.

In addition, since the captured motion itself may be unnatural, the video generation system 1 may use a motion-to-motion AI to estimate or generate the captured motion, convert it into a more natural motion, and then adapt it to the motion of the characters. The video generation system 1 may also estimate or generate the motion using the input motion and natural text. For example, after inputting the motion, the video generation system 1 may send it to an AI (such as an LLM or a text&motion to motion generation model) for processing along with the natural text input by the user, such as "Make the movement lively like this."

The user may also change the camerawork by moving their hand as if it were a camera. For example, the video production system 1 acquires the movement of the user's hand using a sensor such as a web camera. As shown in FIG. 27, the video production system 1 presents (superimposed) the changed camerawork as a frame superimposed on the video. FIG. 27 is a diagram showing an example of editing processing. As a specific example, FIG. 27 is a diagram showing an example of superimposed display of camerawork on video.

For example, the video production system 1 displays a frame CW1 indicating the camerawork before the change and a frame CW2 indicating the camerawork after the change, superimposed on the video MV12. Then, when the user instructs the video production system 1 to use the changed camerawork, the video production system 1 executes a rendering process based on the changed camerawork. In this way, when the user approves the changed camerawork, the video is rendered with the changed camerawork.

The user may also specify the camerawork using an IMU or ImageSLAM of the user's mobile device (such as a smartphone). For example, the image generation system 1 may present the main character by placing it in real space (such as on a real desk) using AR (Augmented Reality). The image generation system 1 displays the changed camerawork by superimposing a frame on the image, in the same way as in the case shown in FIG. 27.

As described above, in the video generation system 1, the user may use the movement of his/her hand or smartphone to determine the motion and camera movement. For example, when the user determines the camerawork with the movement of his/her hand, the user may imagine a person designated with the other hand and determine the relative position of the camera from the distance between the left hand (camera) and the right hand (person). In addition, the video generation system 1 displays the camerawork changed by the user's hand or camera input as a frame superimposed on the video, but the user may then fine-tune the frame and its movement by operating the mouse. In addition, since the input camerawork is entered by hand and may be unnatural, the video generation system 1 may correct the camerawork to a more natural one using a motion to motion generation model, a text&motion to motion generation model, etc. Through the above-mentioned processing, the video generation system 1 can improve usability.

<1-4-9. Examples of using qualitative values>
In the conventional technology, there are cases where consideration is not given to how the system determines the current state and the change history. For example, there are cases where a user wants to make adjustments by comparing with the current state, such as "stand a little further back" or "make it a little brighter," and there is a problem with how the system understands (grasps) the current state. Thus, the conventional technology has a problem with usability, and there is room for improvement in usability. Therefore, the video generation system 1 may be capable of appropriately determining the current state and the change history by the process shown below.

For example, the video production system 1 may store each point, such as the speed of motion and brightness, as a quantitative numerical value and modify the video by comparing the values. The video production system 1 may also perform editing processing using qualitative values, as shown in FIG. 28. FIG. 28 is a diagram showing an example of editing processing. As a specific example, FIG. 28 is a diagram showing an example of editing processing based on modification contents determined according to qualitative values.

In FIG. 28, value information VL21 indicates a quantitative value associated with video MV21. For example, value information VL21 includes qualitative values such as the brightness of the entire town, people's walking speed, the speed at which people's faces move, people's positions, car speeds, and car positions, and indicates values associated with cuts in video MV21. In this way, video production system 1 stores all values that can be changed as quantitative values, and changes the video by comparing them with those values.

In FIG. 28, the user gives a modification instruction for video MV21, "move your face a little slower," and video production system 1 generates video MV22 in which the speed at which the face moves in video MV21 is slowed down. Based on the modification instruction, "move your face a little slower," video production system 1 generates video MV22 with value information VL22 in which the value of the speed at which the person's face moves, in the value information VL21 of video MV21, is reduced from "21" to "10."

In FIG. 28, value information VL22 indicates quantitative values associated with the corrected video MV22. For example, value information VL22 indicates values associated with cuts in video MV22, including qualitative values such as the brightness of the entire town, people's walking speed, the speed at which people's faces move, people's positions, car speeds, and car positions, etc.

In this way, the video production system 1 may perform video editing processing using qualitative values. For example, the video production system 1 may store quantitative values for each point (item), such as movement speed and brightness, and compare the values to modify the video. As described above, classifications such as the brightness of the entire city and people's walking speed, for which qualitative values are stored, are items that are set in advance. For example, the video production system 1 may set and modify each of these using AI (such as LLM) from natural language, or may directly modify the set values. Through the above-mentioned processing, the video production system 1 can improve usability.

An example of a method for acquiring each value is shown below, but the acquisition method is not limited to the following and may be another method. For example, for lighting, the image generation system 1 acquires the setting values (position, rotation, strength, color, etc.) of the light in the 3DCG. For color grading, the image generation system 1 acquires the white balance, color temperature, color cast correction, saturation, exposure, contrast, highlights, shadows, white level, black level, color, LUT settings, etc. set in composite editing. For walking speed, the image generation system 1 acquires the position movement speed of the hip bone of the target 3D model. For face movement speed, the image generation system 1 acquires the rotation speed of the head bone. For position, the image generation system 1 acquires the position of the 3D model.

<1-4-10. Example of rendering process during input>
The conventional technology has an issue in that it is difficult for the user to wait for the rendering time (low usability). Thus, the conventional technology has an issue with usability, and there is room for improvement in usability. Therefore, the image generation system 1 may improve the usability of rendering by the process described below.

For example, the video production system 1 may start the rendering process in the middle of text input. As shown in FIG. 29, the video production system 1 may perform the rendering process in the middle of user input. FIG. 29 is a diagram showing an example of the editing process. As a specific example, FIG. 29 is a diagram showing an example of the editing process based on the rendering process in the middle of input.

In FIG. 29, the video generation system 1 executes rendering processing and the like in response to the user's input of text TX31, "scratching his head with his hand," and displays video MV31. For example, the "|" at the end of text TX31 indicates that the user is in the middle of inputting a correction instruction. Once the video generation system 1 is able to understand the text as a sentence, it starts processing in the background. For example, the video generation system 1 starts rendering processing and the like in the background once the user has input up to "scratching his head with his hand." In this way, the video generation system 1 may start rendering processing while the user is in the middle of inputting text.

In FIG. 29, the user changes the sentence from text TX31 to text TX32, which reads "Put your hands down." For example, the "|" at the end of text TX32 indicates that the user is in the middle of inputting a correction instruction. The video production system 1 executes processing in response to the change from text TX31 to text TX32. For example, when the sentence is changed, the video production system 1 stops the processing being performed at that time and starts the processing again. For example, the video production system 1 ends the processing that was being performed based on text TX31, and executes rendering processing, etc. based on text TX32 to display video MV32.

In FIG. 29, the user changes text TX32 to text TX33, "Keep your hands down, in a natural way," completes the sentence, and instructs execution of processing by pressing a start button or the like. The video generation system 1 ends the background processing that has begun, and displays video MV33, which has been rendered based on text TX33. Through the above-mentioned processing, the video generation system 1 can improve usability.

<1-4-11. Example of confirmation work when editing>
In the conventional technology, there is a problem that the usability is low in the confirmation work at the time of editing, and there is a room for improvement. Therefore, the video production system 1 may perform the confirmation work at the time of editing as shown in Fig. 30. Fig. 30 is a diagram showing an example of the editing process. As a specific example, Fig. 30 is a diagram showing an example of the confirmation work at the time of editing.

Checking task CP1 in FIG. 30 shows an example of a checking task when it is necessary to check changes in the time axis direction, such as motion or camera work. For example, in checking task CP1, the user repeatedly selects a video that is close to ideal. Checking task CP2 in FIG. 30 shows an example of a checking task other than when it is necessary to check changes in the time axis direction. For example, in checking task CP2, the user repeatedly selects an image that is close to ideal.

For example, the video production system 1 may render images of all cuts and then render a video starting with the one to be edited. For example, the video production system 1 may present the generated video to the user and allow the user to decide whether or not to edit the input information. For example, as shown in confirmation tasks CP1, CP2, etc., when presenting the user with the generated results of several patterns of video with variations in response to the user's input information, there is an issue of waiting time for rendering multiple videos. As such, the conventional technology has issues with usability and there is room for improvement in usability.

Therefore, in order to reduce this waiting time, for tasks other than the confirmation task that requires viewing changes along the time axis (corresponding to confirmation task CP2), the video generation system 1 renders only one or a few frames from the video and presents candidates to the user as images. This enables the video generation system 1 to reduce the rendering waiting time.

Furthermore, when the user edits the motion or camerawork, the video generation system 1 presents candidates in the form of video. For example, methods for selecting a few frames from a video to be used for rendering include simply rendering only the first and last two frames of the video, rendering a few frames with large changes in animation in the USD file, and having the AI select the frames to be displayed as highlights from among all the frames. For example, when multiple images are rendered in the video generation system 1, the user can check the generated results in the form of a flip book by hovering the mouse cursor over them.

In addition, when generating videos, the video generation system 1 can reduce waiting time during video preview by starting video generation for each pattern in sequence once image generation for candidate selection is completed. As for the order of video generation for each pattern, in addition to a method of generating them in a random order, other methods include a method in which the user selects the rendering order by pressing a button on the UI, and a method in which images are rendered in order of the length of time the mouse cursor is hovered over them. Through the above-mentioned processing, the video generation system 1 can improve usability.

Here, an example of a processing flow related to the above-mentioned confirmation work will be described with reference to FIG. 31. FIG. 31 is a flowchart showing the processing procedure executed by the video production system. As a specific example, FIG. 31 is a flowchart showing the processing procedure related to editing processing.

In FIG. 31, the video production system 1 generates several patterns of USD files from the user's settings (step S201). If image writing of all patterns of USD files has been completed (step S202: Yes), the video production system 1 ends the process of writing images of the files (e.g., steps S202 to S204).

If image writing of all patterns of USD files has not been completed (step S202: No), the video production system 1 writes out images of the USD files for which writing has not been completed (step S203). The video production system 1 displays the written images on the UI (step S204). After that, the video production system 1 starts the process from step S205 onwards, and repeats the process of steps S202 to S204 until image writing of the files is completed.

When video export of all patterns of USD files has been completed (step S205: Yes), the video production system 1 ends the process of exporting the files (e.g., steps S205 to S207).

If video export of all patterns of USD files has not been completed (step S205: No), the video production system 1 exports videos of USD files for which export has not been completed (step S206). The video production system 1 displays the exported videos on the UI (step S207). After that, the video production system 1 repeats the processes of steps S205 to S207 until video export of the files is completed.

<1-4-12. Examples of processing according to range selection>
In the conventional technology, when a person (also called an "amateur") who has no experience (knowledge) in video (moving image) generation tries to create a video, it is difficult for the person to judge what is good and what is bad, and the person often makes an erroneous judgment. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may evaluate the generated video, as shown in FIG. 32. FIG. 32 is a diagram showing an example of evaluation processing according to range selection.

In FIG. 32, the video production system 1 performs evaluation processing in response to the range selection. Video MV40 in FIG. 32 shows a moving image generated in response to user input. Video MV41 in FIG. 32 shows a state in which the user specifies a part of a person as the range they want evaluated, and text TX41 indicating the evaluation by the video production system 1 of that range is superimposed. In FIG. 32, the video production system 1 makes an evaluation (suggests a correction) of the part of the person specified by the user, as indicated by the text TX41, "it would be easier to convey emotions to the user if you showed this person's facial expression."

Video MV42 in FIG. 32 shows a state in which text TX42 indicating an evaluation by video production system 1 is further superimposed. In FIG. 32, video production system 1 performs an evaluation on the part of the person specified by the user, as indicated by text TX42, saying, "From a production perspective, it would be better to capture the face from directly in front."

For example, if the user thinks that the evaluation indicated by text TX42 is good, video production system 1 accepts from the user an instruction for correction based on that evaluation. Then, video production system 1 generates and displays multiple candidate videos MV43, MV44, and MV45 that correspond to the evaluation indicated by text TX42. In this way, video production system 1 presents to the user multiple candidate videos MV43, MV44, and MV45 that correspond to the evaluation indicated by text TX42.

For example, a user specifies a range with a mouse, and video generation system 1 evaluates the specified range and starts a dialogue (discussion) with the user. In response to the range selected with the mouse, video generation system 1 starts an AI evaluation of that range. For example, video generation system 1 performs an evaluation (presentation of correction suggestions) once every N seconds until the user instructs correction. When the user clicks with the mouse when they think it is good, video generation system 1 presents multiple correction suggestions based on the dialogue (discussion) up to that point. Through the above-mentioned processing, video generation system 1 can improve usability.

<1-4-13. Example of processing according to depth of field>
In the conventional technology, when the number of polygons in the environmental asset is high or the texture resolution is high, and the asset is heavy, it is difficult to suppress the increase in the time required for rendering. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may perform processing according to the depth of field as shown in FIG. 33. FIG. 33 is a conceptual diagram showing an example of processing according to the depth of field.

In FIG. 33, the video generation system 1 uses a high polygon count and high resolution texture for a specified range in front of and behind the subject, and a low polygon count and low resolution texture for a specified range in front of and behind the subject, depending on the depth of field. In FIG. 33, objects with a high polygon count and high resolution texture (circles close to the subject) are shown with dark hatching, and objects with a low polygon count and low resolution texture (circles farther from the subject) are shown with light hatching. In this way, the video generation system 1 changes the number of polygons and texture resolution depending on the depth of field. The video generation system 1 calculates the depth of field using equations (1) to (3) as follows.

Equation (1) is a function for calculating the front depth of field (mm). The image generation system 1 calculates the front depth of field using equation (1). For example, in FIG. 33, the front depth of field corresponds to the front side of the subject (the side closer to the camera). Furthermore, equation (2) is a function for calculating the rear depth of field (mm). The image generation system 1 calculates the rear depth of field using equation (2). For example, in FIG. 33, the rear depth of field corresponds to the rear side of the subject (the side farther from the camera). Equation (3) is a function for calculating the depth of field. The image generation system 1 calculates the depth of field by adding together the front depth of field and the rear depth of field using equation (3).

For example, the video generation system 1 replaces parts closer to the camera than the forward depth of field and parts farther from the camera than the rear depth of field with parts with a lower number of polygons and texture resolution before rendering. For example, when creating a blur in terms of depth of field, the video generation system 1 reduces (lowers) the number of polygons and texture resolution. In this way, the video generation system 1 determines the number of polygons and texture resolution based on criteria such as focal length, F-number, and subject distance. For example, the video generation system 1 may make the above determination when generating a USD file. Through the above-mentioned processing, the video generation system 1 can improve usability.

<1-4-14. Example of playback process during confirmation work>
In the conventional technology, when all the generated videos are played back for review, it is difficult to suppress an increase in the time required for the reviewing work. Thus, the conventional technology has a problem regarding usability, and there is room for improvement in usability. Therefore, the video generation system 1 may perform playback suitable for the reviewing work. The video generation system 1 may change the mode of playback in response to a user's operation. For example, the video generation system 1 may play back the video in a mode like a flip book in response to a user's operation.

The video production system 1 may advance the playback of the video by a predetermined number of seconds (e.g., 0.5 seconds) for each click or rotation of the mouse wheel by the user. For example, the video production system 1 advances frames within a cut according to the movement or position of the mouse or mouse wheel. The video production system 1 may advance the playback of the video by the number of seconds corresponding to the position of the mouse, as shown in FIG. 34.

In FIG. 34, the video generation system 1 may provide the user with content CT41 and accept an instruction on the degree to which the video should be advanced (number of seconds, number of frames, etc.). Content CT41 is a display screen (content) for accepting the degree to which the user should advance the video. Content CT41 arranges information for specifying the number of seconds to advance the video by superimposing it on the video. FIG. 34 shows a case in which a value between 0 and 3.5 seconds can be specified, with the number of seconds to advance the video increasing from left to right. For example, the client UI display unit 400 displays content CT41.

Through content CT41, the user inputs information specifying the number of seconds to advance the video when playing the video. In FIG. 34, the user operates the mouse to position the mouse cursor MS in the area marked 1.0 to specify 1.0 seconds as the number of seconds to advance the video. In this case, the video production system 1 advances the playback of the video at 1.0 second intervals in response to the user's specification of the number of seconds to advance the video. Note that the user may operate the mouse to position the mouse cursor MS in the area marked 1.0 and click, etc. to specify 1.0 seconds as the number of seconds to advance the video. Through the above-described processing, the video production system 1 can improve usability.

<1-4-15. Highlighting example>
In the conventional technology, it may be difficult to understand the parts of the video that have been changed by editing or the like, and it is difficult to prevent the increase in time required for the confirmation work. However, there are problems with usability and there is room for improvement in usability. Therefore, the image production system 1 may perform highlighting as shown in Fig. 35. Fig. 35 shows an example of highlighting the changed portion. For example, the image generation system 1 highlights the part that has changed from the previous generation result. In Fig. 35, a case where a woman in the image has been changed will be described as an example.

Video MV51 in FIG. 35 shows a first highlighting mode. Video MV51 shows a mode in which the changed parts are highlighted by darkening parts other than the changed parts (reducing brightness, etc.). Video production system 1 generates video MV51 and displays video MV51 to highlight the parts that have been changed by editing, etc.

Video MV52 in FIG. 35 shows a second highlighting mode. Video MV52 shows a mode in which the changed parts are highlighted (by adding color, etc.). Video production system 1 generates video MV52 and displays video MV52 to highlight the parts that have been changed by editing, etc. In this way, videos MV51 and MV52 show a case in which parts of people are highlighted when they have been changed.

Video MV53 in FIG. 35 shows a third highlighting mode. Video MV53 shows a mode in which the changed portion is highlighted by indicating the time when the change occurred on the seek bar. Video production system 1 generates video MV53 including a seek bar on which colored points HL531 and HL532 are positioned corresponding to the time when the change occurred, and displays video MV53 to highlight the portion that has been changed by editing, etc.

Video MV54 in FIG. 35 shows a fourth highlighting mode. Video MV54 shows a mode in which the changed portion is highlighted by indicating the time when the change occurred on the seek bar. Video production system 1 generates video MV54 including a seek bar with a colored bar HL54 positioned in a range corresponding to the time period when the change occurred, and displays video MV54 to highlight the portion that has been changed by editing, etc.

As described above, when a user configures settings related to video generation and regenerates a video, the generated video needs to be checked. In a UI that presents multiple generation results, the user has to check each video one by one, which places a heavy burden on the user. Therefore, in order to reduce the burden of checking the videos, the video generation system 1 presents the user with the differences from the results of the previous video generation. For example, the video generation system 1 presents to the user only the parts of the video that have been changed by highlighting them or by displaying the time when the change occurred in the video on a seek bar. This allows the user to check only the parts that have been changed, and the video generation system 1 can reduce the burden of the checking work. Through the above-mentioned processing, the video generation system 1 can improve usability.

For example, methods for extracting changed parts of a video include detecting differences from a generated USD file, and detecting differences by comparing a rendered video with a previously rendered video frame by frame. For example, in a method for detecting differences from a generated USD file, taking advantage of the characteristic of generating USD files each time and performing video rendering, the video generation system 1 compares the contents of the USD files generated before and after the user updates the video generation settings, and detects objects that have changed and the time at which the changes occurred. Also, in a method for detecting differences by comparing a rendered video with a previously rendered video frame by frame, the video generation system 1 compares the videos generated before and after the user updates the video generation settings, and detects changed pixels and the time at which the changes occurred.

<1-4-16. Example of showing the relationship between cuts>
In the conventional technology, there is a problem that when each cut is corrected, the relationship between the previous and next cuts may become unclear. Thus, the conventional technology has a problem with usability, and there is room for improvement in usability. Therefore, the video production system 1 may present the relationship between cuts as shown in Fig. 36. Fig. 36 is a diagram showing an example of the presentation of the relationship between cuts.

Figure 36 shows a case where cut CU61 is a modified cut (also called the "target cut"). A previous relationship bar TR60 is superimposed on cut CU60 that precedes cut CU61, the target cut, to indicate that the cut precedes the target cut. For example, the previous relationship bar TR60 is a triangle with its base on the right and extending to the left. The previous relationship bar TR60 is displayed in such a way that it extends further to the left the further away it is from the target cut.

In addition, a subsequent relationship bar TR62 is superimposed on cut CU62 that is after cut CU61, the target cut, indicating that the cut is a cut that comes after the target cut. For example, the subsequent relationship bar TR62 is a triangle that extends to the right with its base on the left side. A subsequent relationship bar TR63 is superimposed on cut CU63 that is after cut CU62 that is after cut CU61, the target cut, indicating that the cut is a cut that comes after the target cut. For example, the subsequent relationship bar TR63 is a triangle that extends to the right with its base on the left side.

The later relationship bars TR62, TR63 are displayed in such a way that the further they extend to the left the further they are from the target cut. In FIG. 36, cut CU63 is later than cut CU62, so later relationship bar TR63 is displayed in such a way that it extends further to the right than later relationship bar TR62. Note that the triangular display mode is only one example of a display mode, and any display mode can be used as long as the time relationship and its amount can be presented.

The video production system 1 plays a video including cut CU60 to CU63 in response to a user's operation. For example, when displaying cut CU60 in response to a user's operation, the video production system 1 superimposes a previous relationship bar TR60. For example, when displaying cut CU62 in response to a user's operation, the video production system 1 superimposes a next relationship bar TR62. For example, when displaying cut CU63 in response to a user's operation, the video production system 1 superimposes a next relationship bar TR63. In this way, when playing cuts before and after a target cut, the video production system 1 presents the amount of distance from the target cut. In this way, when playing cuts before and after the target cut, the video production system 1 superimposes information indicating the amount and direction according to the number of seconds away from the target cut on the screen, allowing the user to recognize whether the cut is before or after the target cut and how far it is from the target cut. Through the above-mentioned processing, the video production system 1 can improve usability.

<1-4-17. Example of object selection>
The conventional technology has an issue that it is difficult for a user to select an object during editing (low usability). Thus, the conventional technology has an issue regarding usability, and there is room for improvement in usability. Therefore, the image generation system 1 may select an object as shown in Fig. 37. Fig. 37 is a diagram showing an example of object selection.

In video MV71 of FIG. 37, the user operates the mouse to select the right-hand person of the three people by positioning the mouse cursor MS71 in a range that includes the right-hand person. In this case, video production system 1 accepts an operation to select the right-hand person where the mouse cursor MS71 is positioned as the target object. For example, video production system 1 identifies the segmentation (range) corresponding to the right-hand person where the mouse cursor MS71 is positioned as the range selected by the user.

In video MV72 of FIG. 37, the user operates the mouse to select the central person by positioning the mouse cursor MS72 in a range that includes the central person among the three people. In this case, video production system 1 accepts an operation to select the central person where the mouse cursor MS72 is positioned as the target object. For example, video production system 1 identifies the segmentation (range) corresponding to the central person where the mouse cursor MS72 is positioned as the range selected by the user.

In video MV73 of FIG. 37, the user operates the mouse to select the left person of the three people by positioning the mouse cursor MS73 in a range that includes the left person. In this case, video production system 1 accepts an operation to select the person on the left where the mouse cursor MS73 is positioned as the target object. For example, video production system 1 identifies the segmentation (range) corresponding to the person on the left where the mouse cursor MS73 is positioned as the range selected by the user.

In this way, the video generation system 1 recognizes the selection of the target object within the range recognized by segmentation. This allows the user to select the object they want to specify with just a click. For example, when the user wants to change the motion or assets of an object displayed in a video, it is necessary to select the object to be changed. If the user can directly click on the object in the video, rather than using a UI that allows the user to select the object to be changed from a list of objects displayed in the video, the user can more intuitively select the object to be changed. The click range of an object in a video can be achieved by segmenting the pixel area of each object for each frame of the video.

The click range of an object may be calculated in the following manner. For example, because object and camera information for each frame of a video is stored in the USD, it is possible to map which object a pixel seen from the camera in that frame points to, and the video generation system 1 can then work backwards from that to map the coordinates where the user clicked on the video to the object to be edited.

Methods for determining the click range of an object include linking the target USD to a person selected by segmenting the object pixel by pixel calculated backwards from the camera for each frame of the video. For example, the video itself may contain information about what is located in a particular area of x, y (coordinates, etc.). For example, the video generation system 1 may select an object as shown in FIG. 38.

Frame FR in FIG. 38 shows one frame of a video rendered from a camera. USD data OB in FIG. 38 shows a 3D object and a camera on the USD. Because information on the 3D object and the camera is included in the USD, the video production system 1 can map which location on the 3D object a pixel viewed from a camera (such as camera CM in FIG. 38) points to. This enables the video production system 1 to map the coordinates clicked by the user on the video to an object. Through the above-mentioned processing, the video production system 1 can improve usability.

<1-4-18. Example of using 3D model>
The above-described process is merely an example, and the image production system 1 may execute various processes other than the above-described process. In this regard, several examples will be described below.

For example, the video production system 1 may perform a process of importing characters and props (such as products). As a method of importing characters and props (such as products) in the video production system 1, for example, a 3D model or three-dimensional drawing that is to be specified and appear in the video may be registered and used in the video. For example, the video production system 1 may register a 3D model input by a user and use it in the video. For example, the video production system 1 may be capable of including photos of a product taken from various angles. In this case, the video production system 1 may use technology such as NeRF (Neural Radiance Fields) to generate a 3D model of the product from photos of the product taken from various angles and use it in the video.

For example, the video generation system 1 accepts a user operation to specify an object to be changed in the video data, and generates code in which the 3D data of the object specified by the user operation has been changed. For example, when a character in a video (also referred to as "character A") is specified by the user as the object to be changed, and the user selects another character (also referred to as "character B") from a registered 3D model as the changed character, the video generation system 1 generates code in which the 3D data of character A in the video has been changed to the 3D data of character B. This allows the video generation system 1 to generate video data in which character A in the video has been changed to character B indicated by the registered 3D model.

Note that the above-mentioned process is merely an example, and the video production system 1 may generate a code in which the 3D data of the object indicated by the user's operation has been changed by any process. For example, the video production system 1 may generate a changed code by making a change to the 3D data itself of the object indicated by the user's operation in the video data. For example, the video production system 1 may generate a changed code by changing the external shape (height, etc.) of the 3D data of the object indicated by the user's operation in the video data. Furthermore, the video production system 1 may not need to partially perform refinement processing after rendering for registered people and props. Furthermore, the video production system 1 may prepare a marketplace within the video production service for selling characters, props, etc.

<1-4-19. Examples of using reference data>
Furthermore, the video production system 1 may refer to a video and a story from a previously created (video) project. When a sequel to a previously created project is to be used, the video production system 1 may receive an input of the previously created project as reference data, as shown in Fig. 39. Fig. 39 is a diagram showing an example of processing using reference data.

For example, the video production system 1 acquires user input information regarding references to other projects that the user has input to content CT51. Content CT51 is content for accepting user input information regarding items such as an item for specifying with a check mark whether or not to refer to other projects, an item for specifying with a check mark the projects to refer to, and an item for specifying with a check mark which information of the referred projects to refer to.

For example, the client UI display unit 400 displays the content CT51, and the sensor unit 300 accepts user input information. In FIG. 39, the user checks "Refer to other projects" to select to refer to other projects. The user also checks "Product X CM video" to select to refer to the project for the CM video for Product X.

The user also checks "Characters" and "Visual Style," choosing to use the characters and visual style of the commercial video project for Product X as a reference. The user also does not check "Story," "Content Style," or "BGM," choosing not to use the story, content style, or BGM of the commercial video project for Product X as a reference.

The video production system 1 uses the user input information received in the content CT51 to generate a prompt PT51, which is information for scenario generation (first input information). Note that although not illustrated in FIG. 39, the video production system 1 may generate the prompt PT51 using template input information such as the template TP1 shown in FIG. 6.

For example, the video production system 1 generates prompt PT51 by reflecting the characters and visual style in the commercial video project for product X. In FIG. 39, the video production system 1 generates prompt PT51 including a constraint specifying the use of "Mike", a character in the commercial video for product X, and the use of the "cinematic" visual style in the commercial video for product X. In this way, the video production system 1 generates a prompt for generating a scenario based on reference data for past projects in response to the user's selection. This allows the video production system 1 to generate video data for character A in the video based on the past project specified by the user.

<1-4-20. Examples of advantages of having 3D data>
Since the image generation system 1 has 3D data inside, it has the following functions and advantages. The image generation system 1 has the following functions and advantages regarding the images (moving images) it generates. For example, the image generation system 1 can create more natural images by using physical simulation. For example, the image generation system 1 can place an object on a platform, bounce it, roll it, and reproduce the natural swaying of cloth.

For example, the image generation system 1 can realistically reproduce the reflection of light depending on the lighting and object material. When it comes to a bonnet or mirror that is highly reflective, people or objects in front of it will be brighter due to the reflected light. When it comes to fabric that is less reflective, people or objects in front of it will not receive much reflected light.

For example, because the video generation system 1 can fix lighting and objects, it is possible to reduce the possibility of time-series breakdowns. For example, the video generation system 1 can express the lighting position without breakdowns even if it is changed during the video. For example, the video generation system 1 can output video in real time by setting a simple light source. In this case, the video generation system 1 does not need to perform refinement processing.

For example, by inputting product data and characters as 3D data, the video generation system 1 can faithfully reproduce the product itself within the video. For example, because the video generation system 1 knows the depth, global position, normal, etc., it can reduce the possibility of failure during refinery processing. For example, the video generation system 1 can fix the three-dimensional position of a speaker or other sound source, making it easy to create interaction with the sound. For example, the video generation system 1 can reduce rendering time by performing lighting processing such as Differential Rendering.

In addition, the video production system 1 has the following functions and advantages when it comes to editing (correcting) video (moving images). For example, the video production system 1 can maintain consistency of the surrounding environment and lighting, even when the camera position, angle, and camerawork are significantly modified. For example, the video production system 1 can specify the position and angle three-dimensionally within the video.

For example, even after a certain amount of video has been generated, the video generation system 1 can change only specific things, such as only the characters or only the props, while maintaining the video for the rest of the video. For example, the video generation system 1 can change a character from a realistic human-like figure to a two-headed character. For example, the video generation system 1 can change a signboard from a blackboard type to a plastic board. For example, the video generation system 1 can change only a part of the logo on a product package.

For example, by viewing the captured scene in three dimensions, the image generation system 1 can correct the image of parts that are not shown in the video frames. For example, the image generation system 1 can place a light source in a part that is not shown. For example, the image generation system 1 can place a person or object in a part that is not shown and display only a shadow in the video frame. For example, the image generation system 1 can place a person or object in a part that is not shown and specify that the person who is shown looks into the eyes of the person who is not shown.

In addition to the above, the image generation system 1 has the following functions and advantages. For example, the image generation system 1 can easily convert content into 3D device content such as AR, VR (Virtual Reality), SRD (Spatial Reality Display), and 3D displays.

<1-5. Example of processing flow from the user's perspective>
Next, a procedure of information processing by the image production system 1 in response to a user operation will be described as an example of a processing flow seen from the user's perspective with reference to Fig. 40. Fig. 40 is a flowchart showing the flow of processing in response to a user operation.

As shown in FIG. 40, in the video production system 1, the user presses a project creation button (step S1). Then, in the video production system 1, the user inputs the necessary information for video generation (step S2). For example, the user inputs information including the purpose of creating the video, the message to be conveyed through the video, the target users, the functional features of the product/service to be conveyed, the length of the video, the aspect ratio, etc.

Then, in the video production system 1, the user presses the storyboard, video, sound, and text logo creation buttons (step S3). For example, the video production system 1 may generate everything at once (for example, up to the video data) in response to user operation, or may present a storyboard and allow the user to make some modifications in response to instructions from the user before generating the video, sound, and text logo.

Then, the video production system 1 modifies the storyboard, video, sound, and text logo in response to user operations (step S4). For example, the video production system 1 may accept modifications to each of them in the order of the user's preference.

Then, the video production system 1 performs export in response to a user operation (step S5). For example, the video production system 1 may perform export in a video file format such as mp4, avi, or mov in response to a user operation. Also, for example, the video production system 1 may perform export in a file format of any video editing software such as Premiere Pro, After Effects, or Davinci Resolve.

<1-6. About the AI model>
In addition, each of the AI models used in each of the above-mentioned processes is not limited to the examples described in each section, and any internal structure can be adopted as long as it is possible to output desired information in response to the input. Any combination of input, output, and internal structure of the AI model can be adopted as long as it is possible to output desired information.

The input of the AI model may be text, images, audio, 3D data, etc., or a combination thereof. The output of the AI model may be text, images, audio, 3D data, etc. Note that the above-mentioned inputs and outputs are merely examples, and the above-mentioned AI model may have any input and output.

The internal structure of the AI model can be any structure depending on the combination of input and output. In other words, the internal structure of the AI model can be any structure that can produce the desired output for the input.

For example, the AI model may have a structure related to Transformer. For example, the AI model may have a structure related to Transformer and perform processing that takes into account context such as the context within data, such as text and time-series data. For example, the AI model may have a self-attention mechanism. For example, the AI model may have any attention mechanism, such as Single-Head Attention or Multi-head Attention. Note that the AI model does not have to have an attention mechanism.

The AI model may have a mechanism for extracting features from an input. For example, the AI model may have an encoder. The AI model may have a mechanism for generating information based on the extracted features. For example, the AI model may have a decoder.

The AI model may have a structure related to a CNN (Convolutional Neural Network). For example, when performing processing on an image, the AI model may have a structure related to a CNN. For example, the AI model may have at least one of a convolutional layer, a pooling layer, a fully connected layer, etc.

Note that the above-mentioned internal structure is merely an example, and the above-mentioned AI model may have any internal structure. For example, the AI model may have a skip connection. In addition, the AI model may have a structure related to a diffusion model.

The AI model described above may be generated (learned) by any learning process. The AI model may be a machine learning model learned using any machine learning method. For example, the AI model may be a model generated based on a so-called Foundation Model by fine-tuning the Foundation Model to apply it to a specific task (e.g., scenario data generation, code generation, etc.). For example, an AI model such as the LLM described above may be a model generated by fine-tuning the Foundation Model to apply it to a specific task.

The base model here is a model that has been trained to be applicable to various tasks, for example to be able to perform a wide variety of tasks. For example, the base model is a neural network that has been pre-trained with a large amount of unlabeled data set. Note that the base model may have any structure, such as a Transformer-based architecture. For example, the base model is generated by self-supervised learning using data without correct answer labels. As described above, the base model is fine-tuned so that it can be adapted to a wide range of downstream tasks.

For example, when applied to a task of scenario data generation, the base model is fine-tuned so that it can be adapted to the task of scenario data generation, and an AI model (model M1, etc.) applied to the task of scenario data generation is generated. Also, when applied to a task of code generation, the base model is fine-tuned so that it can be adapted to the task of code generation, and an AI model (model M3, etc.) applied to the task of code generation is generated. Also, when applied to a task of sound data generation, the base model is fine-tuned so that it can be adapted to the task of sound data generation, and an AI model (model M4, etc.) applied to the task of sound data generation is generated. Also, when applied to a task of text logo generation, the base model is fine-tuned so that it can be adapted to the task of text logo generation, and an AI model (model M5, etc.) applied to the task of text logo generation is generated.

For example, an AI model (such as model M1) applied to a task of generating scenario data is trained using training data that includes a combination of input information corresponding to the AI model and scenario data (also called "correct answer information") that is the correct output when the input information is input. The training data, such as the input information and the correct answer information, may be data created by a person, or may be data automatically generated by a computer that generates the training data. For example, the scenario data that is the correct answer information may be data created by a person. Below, model M1 will be briefly described as an example. For example, model M1 is trained to output correct answer information corresponding to each input information when each piece of input information in the training data is input. For example, model M1 is trained by adjusting (correcting) parameters (connection coefficients) using a method such as backpropagation (error backpropagation method) so that the error between the output of model M1 when certain input information is input and the correct answer information corresponding to that input information is reduced. In addition, other AI models such as an AI model applied to a code generation task (e.g., model M3), an AI model applied to a sound data generation task (e.g., model M4), an AI model applied to a text logo generation task (e.g., model M5), an AI model applied to an evaluation task (e.g., model M10), and an AI model applied to an image improvement processing task (e.g., model M11) may also be trained using a similar learning process.

Note that the above-mentioned learning process is merely an example, and the above-mentioned AI model is learned by any learning process according to the input, output, and internal structure of the AI model. For example, the AI model may be learned by an unsupervised learning method such as GAN (Generative Adversarial Network). The AI model may also be learned in a distributed state without aggregating data such as federated learning. In this case, a local model collected by each video generation service device (server, etc.) may be generated in the service, and a server (aggregation server) that aggregates information (parameters, etc.) of the local models generated by each video generation service device (server, etc.) may generate a global model using the information of the local model. In this case, the video generation system 1 may receive the global model generated by the aggregation server from the aggregation server, and use the received global model as an AI model for processing.

In this way, the AI models described above may be generated (learned) by any computer. In other words, the learning process to generate the AI models may be performed by any device (computer, etc.) of the video production system 1, or may be performed by a device outside the video production system 1. For example, when a device outside the video production system 1 generates at least one of the AI models described above, the video production system 1 acquires the AI model from the device outside the video production system 1 and performs processing using the acquired AI model.

2. Other embodiments
The processing according to each of the above-described embodiments may be implemented in various different forms (variations) other than the above-described embodiments and variations.

<2-1. Other configuration examples>
The above-described configuration of the image generation system 1 is merely an example, and any manner of division of functions in the image generation system 1 can be adopted. That is, the above-described configuration is merely an example, and the image generation system 1 may have any manner of division of functions and any configuration as long as it can provide the above-described service related to image generation. For example, the image generation system 1 may be configured by one device (such as a computer) that performs the above-described processing. In this case, one device of the image generation system 1 may have the functions of the image generation module 100, the information acquisition module 200, the sensor unit 300, and the client UI display unit 400. For example, the image generation service provided by the image generation system 1 may be provided to the user as a program such as a tool (AI Assist Creation Tool) that operates on a terminal device (such as the computer 20) used by the user.

<2-2. Others>
Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

Furthermore, the above-mentioned embodiments and variations can be combined as appropriate to the extent that they do not cause any contradiction in the processing content.

Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

3. Effects of the Present Disclosure
As described above, the video generation system (video generation system 1 in the embodiment) according to the present disclosure includes an acquisition unit (input text acquisition unit 210 in the embodiment), a scenario generation unit (scenario-oriented generation unit 131 in the embodiment), a code generation unit (video-oriented generation unit 132 in the embodiment), and a video data acquisition unit (video generation unit 140 in the embodiment). The acquisition unit acquires an input query related to video generation from a user. The scenario generation unit generates scenario data related to video generation based on the input query. The code generation unit generates code for constructing 3D data based on the scenario data. The video data acquisition unit acquires video data based on the code.

In this way, the video generation system according to the present disclosure generates code for constructing 3D data based on scenario data generated based on a query input from a user, and obtains video data based on the code, thereby obtaining video data in response to a query input from a user.

The video generation system also includes an image quality improvement unit (in this embodiment, an image refinement unit 143). The image quality improvement unit executes an image quality improvement process that improves the image quality of the video data. In this way, the video generation system can obtain high-quality videos by improving the image quality of the video data.

The image quality improvement unit also improves the image quality of the video data by performing an image quality improvement process based on the text prompt. In this way, the video generation system can obtain high-quality video by improving the image quality of the video data based on the text prompt.

The image quality improvement unit also performs image quality improvement processing based on a text prompt that specifies the target of the image quality improvement processing in the video data. In this way, the video production system can obtain high-quality video by improving the image quality of the video data for the specified target.

The image quality improvement unit also performs image quality improvement processing based on a text prompt that specifies the target determined to require improvement. In this way, the video generation system can obtain high-quality video by improving the image quality of the video data for the target determined to require improvement.

The video generation system also includes a display control unit (client UI display unit 400 in this embodiment). The display control unit displays a storyboard based on the scenario data. In this way, the video generation system can present information in a manner that is highly convenient for the user by displaying a storyboard based on the scenario data.

The storyboard is also configured to display video data for each cut of the video. In this way, the video production system is able to present information in a manner that is highly convenient for the user, by configuring the storyboard to display video data for each cut of the video.

The video generation system also includes a sound generation unit (sound generation unit 150 in this embodiment). The sound generation unit generates sound data corresponding to the video data based on the scenario data and the video data. In this way, the video generation system can obtain a video including sound by generating sound data corresponding to the video data.

The video generation system also includes a text generation unit (in this embodiment, a text/logo generation unit 160). The text generation unit generates text data indicating text to be displayed on the video displayed by the video data, based on the scenario data. In this way, the video generation system can obtain a video including text by generating text data corresponding to the video data.

The video generation system also includes a logo generation unit (in this embodiment, a text/logo generation unit 160). The logo generation unit generates logo data indicating a logo to be displayed on a video displayed by the video data, based on the scenario data. In this way, the video generation system can obtain a video including a logo by generating logo data corresponding to the video data.

The input query also includes at least one of text, images, audio, and 3D data. In this way, the video generation system can obtain video data in response to the input query from the user by the input query including at least one of text, images, audio, and 3D data.

The video generation system also includes a first output unit (in this embodiment, a scenario-oriented generation unit 131). The first output unit outputs scenario generation information used by the scenario generation unit to generate scenario data based on the input query. The scenario generation unit generates scenario data based on the scenario generation information. In this way, the video generation system can acquire video data in response to an input query from a user by generating scenario data based on the scenario generation information generated based on the input query.

The first output unit also generates a first prompt for generating scenario data based on the input query as scenario generation information. The scenario generation unit generates scenario data based on the first prompt. In this way, the video generation system can obtain video data in response to the input query from the user by generating scenario data based on the first prompt.

Furthermore, the first output unit generates, based on the input query, first input information used as an input of a first model for generating scenario data as scenario generation information. The scenario generation unit inputs the first input information generated using the input query to the first model and causes the first model to output scenario data, thereby generating scenario data. In this way, the video generation system can obtain video data in response to an input query from a user by generating scenario data using the first model.

The video generation system also includes a second output unit (video generation unit 132 in this embodiment). The second output unit outputs code generation information used by the code generation unit to generate code for constructing 3D data based on the scenario data. The code generation unit generates code based on the code generation information. In this way, the video generation system can acquire video data in response to an input query from a user by generating code based on the code generation information generated based on the scenario data.

The second output unit also generates, as code generation information, a second prompt for outputting a code for constructing 3D data based on the scenario data. The scenario generation unit generates the code based on the second prompt. In this way, the video generation system can acquire video data in response to an input query from a user by generating a code based on the second prompt.

Furthermore, the second output unit generates, based on the input query, second input information used as an input of a second model for generating code as information for code generation. The scenario generation unit generates code by inputting the second input information generated using the scenario data to the second model and causing the second model to output code. In this way, the video generation system can obtain video data in response to an input query from a user by generating code using the second model.

The video generation system also includes a reception unit (sensor unit 300 in this embodiment). The reception unit receives operations related to video editing from a user. The code generation unit generates code from editing based on the operations. In this way, the video generation system can appropriately acquire video corresponding to the user's editing by generating code in response to the video editing operations from the user.

The receiving unit also receives an operation specifying a motion or camera movement using a sensor. The code generating unit generates a code corresponding to the motion or camera movement indicated by the operation. In this way, the video generation system can appropriately acquire video corresponding to the user's editing by generating a code in response to a user's operation specifying a motion or camera movement.

The receiving unit also receives an operation to select multiple cuts from the video data. The code generating unit generates code in which the portions corresponding to the multiple cuts indicated by the operation have been modified. In this way, the video generating system can appropriately acquire video corresponding to the user's editing by generating code in response to the user selecting multiple cuts.

Furthermore, date information is associated with each cut of the video data. The code generation unit determines the content of the edit indicated by the operation based on the date information of each cut of the video data. In this way, the video generation system can appropriately acquire video corresponding to the user's editing by determining the content of the edit based on the date information of each cut of the video data.

The receiving unit also receives an operation to specify an object of the video data to be changed. The code generating unit generates a code in which the 3D data of the object specified by the operation has been changed. In this way, the video generating system can appropriately acquire video corresponding to the user's edits by generating a code in which the 3D data specified by the user as the object to be changed has been changed.

The video generation system also includes an evaluation unit (evaluation unit 180 in this embodiment). The evaluation unit generates information indicating an evaluation of at least one of the scenario data and the video data. In this way, the video generation system can generate information indicating an evaluation of at least one of the scenario data and the video data, thereby evaluating the generated information.

The code generation unit also generates a code based on the evaluation. In this way, the video generation system can appropriately acquire information according to the evaluation by generating a code based on the evaluation.

The scenario generation unit also generates scenario data based on the evaluation. In this way, the video generation system can acquire information appropriately according to the evaluation by generating scenario data based on the evaluation.

The scenario generation unit also generates a code based on the scenario data generated based on the evaluation. In this way, the video generation system can appropriately acquire information according to the evaluation by generating a code based on the scenario data generated based on the evaluation.

4. Hardware Configuration
An information processing device (information equipment) having the image generation module 100, the information acquisition module 200, the client UI display unit 400, and the like according to each of the above-described embodiments is realized by a computer 1000 having a configuration as shown in FIG. 41, for example. FIG. 41 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device. The image generation module 100 according to the embodiment will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input/output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

The CPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400 and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

The ROM 1300 stores boot programs such as the Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000.

HDD1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU1100 and data used by such programs. Specifically, HDD1400 is a recording medium that records the image generation program according to the present disclosure, which is an example of program data 1450.

The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input/output interface 1600. The input/output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. The media may be, for example, optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when computer 1000 functions as image generation module 100 according to an embodiment, CPU 1100 of computer 1000 executes an image generation program loaded onto RAM 1200 to realize functions of control unit 1301, etc. Also, HDD 1400 stores the image generation program according to the present disclosure and data in storage unit 1302. Note that CPU 1100 reads and executes program data 1450 from HDD 1400, but as another example, these programs may be obtained from other devices via external network 1550.

The present technology can also be configured as follows.
(1)
An acquisition unit that acquires an input query regarding video generation from a user;
a scenario generation unit that generates scenario data related to video generation based on the input query;
a code generating unit that generates a code for constructing 3D data based on the scenario data;
a video data acquisition unit that acquires video data based on the code;
An image generation system comprising:
(2)
an image quality improvement unit that executes an image quality improvement process to improve the image quality of the video data;
The image generation system according to (1) further comprising:
(3)
The image quality improvement unit includes:
The image generation system according to any one of claims 2 to 6, further comprising: a display unit for displaying an image of the moving image data based on a text prompt;
(4)
The image quality improvement unit includes:
The image generation system according to (2) or (3), further comprising: a text prompt that specifies a target of the image quality improvement process in the video data;
(5)
The image quality improvement unit includes:
The image generation system according to claim 4, further comprising: a text prompt that specifies the object determined to require improvement, and the image quality improvement process is performed based on the text prompt.
(6)
a display control unit that displays a storyboard based on the scenario data;
The image generation system according to any one of (1) to (5), further comprising:
(7)
The video production system according to (6), wherein the storyboard is configured to display the video data for each cut of the video.
(8)
a sound generating unit that generates sound data corresponding to the video data based on the scenario data and the video data;
The image generation system according to any one of (1) to (7), further comprising:
(9)
a text generating unit that generates text data indicating text to be displayed on a video displayed based on the video data, based on the scenario data;
The image generation system according to any one of (1) to (8), further comprising:
(10)
a logo generating unit that generates logo data indicating a logo to be displayed on a video image displayed based on the moving image data, based on the scenario data;
The image generation system according to any one of (1) to (9), further comprising:
(11)
The image generation system according to any one of (1) to (10), wherein the input query includes at least one of text, image, audio, and 3D data.
(12)
a first output unit that outputs scenario generation information used by the scenario generation unit to generate the scenario data based on the input query;
Further comprising:
The scenario generation unit includes:
The image generation system according to any one of (1) to (11), further comprising: generating the scenario data based on the scenario generation information.
(13)
The first output section is
generating a first prompt for generating the scenario data based on the input query as the scenario generation information;
The scenario generation unit includes:
The image production system according to any one of claims 12 to 17, further comprising: a display unit configured to display a display screen of the image production system according to the first prompt;
(14)
The first output section is
generating, based on the input query, first input information to be used as an input of a first model for generating the scenario data, as the scenario generation information;
The scenario generation unit includes:
The image generation system according to (12) or (13), further comprising: inputting the first input information generated using the input query into the first model; and causing the first model to output the scenario data, thereby generating the scenario data.
(15)
a second output unit that outputs code generation information used by the code generation unit to generate a code for configuring the 3D data based on the scenario data;
Further comprising:
The code generation unit
The image generation system according to any one of (1) to (14), further comprising: generating the code based on the code generation information.
(16)
The second output section is
generating, as the code generation information, a second prompt for outputting a code for configuring the 3D data based on the scenario data;
The scenario generation unit includes:
The image production system of any one of claims 1 to 15, further comprising: generating the code based on the second prompt.
(17)
The second output section is
generating second input information as the code generation information based on the input query, the second input information being used as an input of a second model for generating the code;
The scenario generation unit includes:
The video generation system according to (15) or (16), wherein the code is generated by inputting the second input information generated using the scenario data into the second model and causing the second model to output the code.
(18)
a reception unit that receives operations related to video editing from a user;
Further comprising:
The code generation unit
The image generation system according to any one of (1) to (17), further comprising: generating the code by editing based on the operation.
(19)
The reception unit is
Accepting the operation specifying a motion or camera movement using a sensor;
The code generation unit
The image generation system according to claim 18, further comprising: generating the code corresponding to the motion or camera movement indicated by the operation.
(20)
The reception unit is
accepting the operation of selecting a plurality of cuts from the video data;
The code generation unit
The video generation system according to (18) or (19), further comprising: generating the code in which the portions corresponding to the plurality of cuts indicated by the operation have been changed.
(21)
Date information is associated with each cut of the video data,
The code generation unit
The video production system according to any one of (18) to (20), further comprising: determining the content of the editing indicated by the operation based on date information of each cut of the video data.
(22)
The reception unit is
accepting the operation of designating an object to be changed from among the video data;
The code generation unit
The image generation system according to any one of (18) to (21), further comprising: generating the code in which 3D data of the object indicated by the operation has been changed.
(23)
the 3D data comprises a plurality of data sets;
The code generation unit
The image generation system according to any one of (18) to (22), further comprising: generating the code corresponding to at least one of the plurality of data sets by editing based on the operation.
(24)
The code generation unit
The video generation system according to any one of the above (23) and (24), further comprising: a process for updating a part of the plurality of data sets; and a process for updating the entirety of the plurality of data sets, depending on the editing content indicated by the operation.
(25)
an evaluation unit that generates information indicating an evaluation of at least one of the scenario data and the video data;
The image generation system according to any one of (1) to (24), further comprising:
(26)
The code generation unit
The image generation system according to any one of claims 25 to 30, further comprising: generating the code based on the evaluation.
(27)
The scenario generation unit includes:
The image generation system according to (25) or (26), further comprising: generating the scenario data based on the evaluation.
(28)
The code generation unit
The image generation system according to any one of claims 27 to 31, further comprising: generating the code based on the scenario data generated based on the evaluation.
(29)
Obtaining an input query for video generation from a user;
generating scenario data related to movie generation based on the input query;
generating a code for constructing 3D data based on the scenario data; and
and acquiring video data based on the code.
(30)
Obtaining an input query for video generation from a user;
generating scenario data related to movie generation based on the input query;
generating a code for constructing 3D data based on the scenario data; and
and acquiring video data based on said code.

1 Video generation system 100 Video generation module 110 Input text analysis unit 120 Sensor analysis unit 130 Prompt generation unit 131 Scenario generation unit 132 Video generation unit 133 Sound generation unit 134 Text/logo generation unit 140 Video generation unit 141 USD generation unit 142 Rendering unit 143 Video refinement unit 150 Sound generation unit 160 Text/logo generation unit 170 Composite editing unit 180 Evaluation unit 190 Client UI module 200 Information acquisition module 210 Input text acquisition unit 220 Sensor acquisition unit 300 Sensor unit 400 Client UI display unit

Claims (30)

An acquisition unit that acquires an input query regarding video generation from a user;
a scenario generation unit that generates scenario data related to video generation based on the input query;
a code generation unit that generates code, which is an editable computer program, for configuring 3D data and defining at least one of a scene and an object based on the scenario data;
a video data acquisition unit that acquires video data based on the code;
An image generation system comprising: an image quality improvement unit that executes an image quality improvement process to improve the image quality of the video data;
The video production system of claim 1 further comprising: The image quality improvement unit includes:
The image generation system of claim 2 , further comprising: a display unit configured to display a display image of the video data based on the display image of the video data; The image quality improvement unit includes:
The video production system according to claim 2 , wherein the image quality improvement process is performed based on a text prompt that specifies a target of the image quality improvement process in the video data. The image quality improvement unit includes:
The image production system of claim 4 , further comprising: a text prompt that specifies the object determined to require improvement, and the image quality improvement process is performed based on the text prompt that specifies the object determined to require improvement. a display control unit that displays a storyboard based on the scenario data;
The video production system of claim 1 further comprising: The video production system according to claim 6 , wherein the storyboard is configured to display the video data for each cut of the video. a sound generating unit that generates sound data corresponding to the video data based on the scenario data and the video data;
The video production system of claim 1 further comprising: a text generating unit that generates text data indicating text to be displayed on a video displayed based on the video data, based on the scenario data;
The video production system of claim 1 further comprising: a logo generating unit that generates logo data indicating a logo to be displayed on a video image displayed based on the moving image data, based on the scenario data;
The video production system of claim 1 further comprising: The video generation system of claim 1 , wherein the input query includes at least one of text, image, audio, and 3D data. a first output unit that outputs scenario generation information used by the scenario generation unit to generate the scenario data based on the input query;
Further comprising:
The scenario generation unit includes:
The image production system according to claim 1 , wherein the scenario data is generated based on the scenario generation information. The first output section is
generating a first prompt for generating the scenario data based on the input query as the scenario generation information;
The scenario generation unit includes:
The image production system of claim 12 , further comprising: a display unit configured to display a screen displaying the screen image data based on the first prompt; The first output section is
generating, based on the input query, first input information to be used as an input of a first model for generating the scenario data, as the scenario generation information;
The scenario generation unit includes:
The image generation system according to claim 12, wherein the scenario data is generated by inputting the first input information generated using the input query into the first model and causing the first model to output the scenario data. a second output unit that outputs code generation information used by the code generation unit to generate a code for configuring the 3D data based on the scenario data;
Further comprising:
The code generation unit
The video generation system according to claim 1 , wherein the code is generated based on the information for code generation. The second output section is
generating, as the code generation information, a second prompt for outputting a code for configuring the 3D data based on the scenario data;
The scenario generation unit includes:
The video production system of claim 15 further comprising: generating the code based on the second prompt. The second output section is
generating second input information as the code generation information based on the input query, the second input information being used as an input of a second model for generating the code;
The scenario generation unit includes:
The video generation system according to claim 15 , wherein the code is generated by inputting the second input information generated using the scenario data into the second model and causing the second model to output the code. a reception unit that receives operations related to video editing from a user;
Further comprising:
The code generation unit
The video generation system according to claim 1 , wherein the code is generated by editing based on the operation. The reception unit is
Accepting the operation specifying a motion or camera movement using a sensor;
The code generation unit
The image generation system of claim 18 , further comprising: generating the code corresponding to the motion or camera movement indicated by the operation. The reception unit is
accepting the operation of selecting a plurality of cuts from the video data;
The code generation unit
The video production system according to claim 18 , further comprising: generating the code in which portions corresponding to the plurality of cuts indicated by the operation have been changed. Date information is associated with each cut of the video data,
The code generation unit
The video production system according to claim 18 , further comprising: determining the content of the editing operation indicated by the operation based on date information of each cut of the video data. The reception unit is
accepting the operation of designating an object to be changed from among the video data;
The code generation unit
The image generation system according to claim 18 , further comprising: generating the code in which 3D data of the object indicated by the operation has been changed. the 3D data comprises a plurality of data sets;
The code generation unit
The video production system of claim 18 , further comprising: generating the code corresponding to at least one of the plurality of data sets through editing based on the operation. The code generation unit
The video generation system according to claim 23 , further comprising: a process for updating a part of the plurality of data sets, or a process for updating all of the plurality of data sets, depending on editing content indicated by the operation. an evaluation unit that generates information indicating an evaluation of at least one of the scenario data and the video data;
The video production system of claim 1 further comprising: The code generation unit
The video production system of claim 25 , further comprising: generating said code based on said evaluation. The scenario generation unit includes:
The image generation system according to claim 25 , further comprising: generating the scenario data based on the evaluation. The code generation unit
The image generation system according to claim 27, further comprising: generating the code based on the scenario data generated based on the evaluation. Obtaining an input query for video generation from a user;
generating scenario data related to movie generation based on the input query;
generating, based on the scenario data, code that is an editable computer program for constructing 3D data and defining at least one of a scene and an object ;
and acquiring video data based on the code. Obtaining an input query for video generation from a user;
generating scenario data related to movie generation based on the input query;
generating, based on the scenario data, code that is an editable computer program for constructing 3D data and defining at least one of a scene and an object ;
and acquiring video data based on said code.

Images