WO2019187719A1

WO2019187719A1 - Information processing device, information processing method, and program

Info

Publication number: WO2019187719A1
Application number: PCT/JP2019/005042
Authority: WO
Inventors: 元戸村; 修祐佐伯; 琢麿馬渕; 健太多田; 佐藤　和美; 寛飯渕
Original assignee: ソニー株式会社
Priority date: 2018-03-28
Filing date: 2019-02-13
Publication date: 2019-10-03

Abstract

The present invention achieves a structure wherein non-real-time tasks are performed in a real-time core to increase processing efficiency, and the real-time property of real-time tasks is guaranteed. The present invention has: a plurality of cores that are structured from a real-time core performing real-time tasks and a non-real-time core performing non-real-time tasks; and a task scheduler that schedules tasks for the plurality of cores. The task scheduler performs task assignment to cause the real-time core to perform not only real-time tasks but also non-real-time tasks, and in response to the occurrence of a previously prescribed event, performs a migration to transfer a non-real-time task assigned to the real-time core to a different core.

Description

Information processing apparatus, information processing method, and program

The present disclosure relates to an information processing apparatus, an information processing method, and a program. More specifically, in an apparatus that can execute a plurality of processes in parallel using a plurality of cores (processors) such as a multi-core system, an information processing apparatus that realizes real-time performance of a real-time task, and information processing The present invention relates to a method and a program.

In systems that require parallel processing of a large number of tasks (processes), each task (process) is executed in parallel, so information with multiple processors, such as a multicore system capable of parallel processing by multiple cores (processors) A processing device is used.

Tasks executed in such an information processing apparatus include tasks that require completion of processing up to a specified time, that is, real-time tasks that require real-time performance (real-time processing), and other tasks, that is, identification Non-real-time tasks (non-real-time processing) that do not require completion of processing until the specified time are mixed.

In order to achieve the completion of processing up to a specified time for a real-time task, that is, as a technology for ensuring real-time properties, there is a shielding technology for setting a part of a plurality of cores of an information processing device as a core dedicated to a real-time task.

This shielding technique is set so that a dedicated core (real-time core) does not execute non-real-time processing, so that a dedicated core can reliably execute a real-time task when a real-time task occurs.
However, this configuration has a problem that it is necessary to set a dedicated core that executes only a real-time task, and is difficult to realize when the number of cores is small.

For example, Patent Document 1 (Japanese Patent Laid-Open No. 2005-157955) discloses a technique for disclosing a specific real-time task by a specified time without providing a dedicated core for executing a real-time task.

This Patent Document 1 calculates the processing time of a real-time task required by a processor, and if there is a surplus time before the specified completion time of the real-time task, causes the processor to execute another process and there is no room Is configured to execute only real-time tasks.

This configuration realizes a configuration that guarantees the real-time performance of the real-time task even in a configuration where the number of cores (processors) is not sufficient and a real-time dedicated core cannot be set.

However, this configuration has a problem that a lot of additional processing occurs, for example, it is necessary to calculate the processing time of the real-time task, and it is also necessary to set an interrupt for calculating the processing time. . Another problem is that it is difficult to deal with real-time tasks that occur at unpredictable timing.

JP 2005-157955 A

The present disclosure has been made in view of the above-described problems, for example, in a multi-core system or the like that uses a plurality of cores (processors) to execute a plurality of processes in parallel, and some cores are processed in real time. Information processing apparatus, information processing method, and program capable of ensuring real-time performance of a real-time task without performing processing such as pre-calculating the processing time of each task The purpose is to provide.

The first aspect of the present disclosure is:
Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The task scheduler
A task allocation process for causing the real-time core to execute not only a real-time task but also a non-real-time task;
The information processing apparatus executes migration for moving a non-real-time task assigned to the real-time core to another core in response to occurrence of a predetermined event.

Furthermore, the second aspect of the present disclosure is:
An information processing method executed in an information processing apparatus,
The information processing apparatus includes:
Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The task scheduler is
A task allocation process for causing the real-time core to execute not only a real-time task but also a non-real-time task;
According to the information processing method, migration is performed in which a non-real-time task assigned to the real-time core is moved to another core in response to occurrence of a predetermined event.

Furthermore, the third aspect of the present disclosure is:
A program for executing information processing in an information processing apparatus;
The information processing apparatus includes:
Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The program is stored in the task scheduler.
Causing the real-time core to execute a task assignment process for executing not only a real-time task but also a non-real-time task;
In accordance with the occurrence of a predefined event, there is a program for executing a migration that moves a non-real-time task assigned to the real-time core to another core.

Note that the program of the present disclosure is a program that can be provided by, for example, a storage medium or a communication medium provided in a computer-readable format to an information processing apparatus or a computer system that can execute various program codes. By providing such a program in a computer-readable format, processing corresponding to the program is realized on the information processing apparatus or the computer system.

Further objects, features, and advantages of the present disclosure will become apparent from a more detailed description based on embodiments of the present disclosure described below and the accompanying drawings. In this specification, the system is a logical set configuration of a plurality of devices, and is not limited to one in which the devices of each configuration are in the same casing.

According to the configuration of an embodiment of the present disclosure, a configuration in which non-real-time tasks are executed in the real-time core to improve processing efficiency and the real-time property of the real-time task is ensured is realized.
Specifically, for example, it has a real-time core that executes a real-time task, a plurality of cores configured by a non-real-time core that executes a non-real-time task, and a task scheduler that schedules the tasks of the plurality of cores. The task scheduler executes task assignment that allows the real-time core to execute not only real-time tasks but also non-real-time tasks, and migrates non-real-time tasks assigned to real-time cores to other cores in response to the occurrence of a predefined event. Do.
With this configuration, a non-real-time task is executed in the real-time core to improve processing efficiency, and a configuration that ensures the real-time property of the real-time task is realized.
Note that the effects described in the present specification are merely examples and are not limited, and may have additional effects.

It is a figure explaining a real-time task (real-time processing). It is a figure explaining task movement processing (migration) between cores. It is a figure explaining the shield process which sets the exclusive core for performing a real-time process. It is a figure explaining the conventional strict shielding process. It is a figure about the structure of this indication which performs a loose shield structure. It is a figure explaining the assignment of the task with respect to the core which a kernel performs, and the movement (migration) process of a task. It is a figure which shows the flowchart explaining the process sequence which a task scheduler performs. It is a figure explaining the specific example of the process (real-time inhibition process) which influences ensuring of the real-time property (process completion certainty to regulation time) of the real-time process currently performed by the real-time core. It is a figure which shows the structural example of an endoscope system. It is a figure explaining the processing sequence of the core 1 (CPU1) 131 which is a real-time core, and the device A (GPU) 141. FIG. FIG. 25 is a diagram for describing an example hardware configuration of an information processing device.

Hereinafter, the details of the information processing apparatus, the information processing method, and the program of the present disclosure will be described with reference to the drawings. The description will be made according to the following items.
1. 1. About real-time tasks (real-time processing) 2. Task migration processing (migration) between cores 3. Execution example of real-time processing by shield processing 4. Configuration that performs real-time processing with a specific core and improves the processing efficiency of the entire device. 5. Task scheduler that controls execution tasks in each core 6. Specific system configuration and processing example to which the processing of the present disclosure is applied 7. Configuration example of information processing apparatus Summary of composition of this disclosure

[1. About real-time tasks (real-time processing)
First, a real-time task (real-time processing) will be described with reference to FIG.
As described above, an information processing apparatus having a plurality of cores (processors) such as a multi-core system in which a plurality of cores (processors) can be operated in parallel in order to execute a plurality of different tasks (processes) in parallel is used. .
For processing executed by an information processing apparatus having a plurality of cores (processors),
(A) Processing that requires completion of processing up to a specified time, that is, a real-time task that requires real-time performance (real-time processing),
(B) Non-real-time task (non-real-time processing) that does not require completion of processing up to a specified time,
These two types of tasks (processes) are mixed.

Specifically, for example, when an image is taken with a video camera and the work is performed while viewing the photographed image displayed on the monitor (display unit), the time from the camera photographed image to the monitor (display unit) output is determined. If the delay occurs, the work may be delayed or the display may be distorted. At worst, it is impossible to work while looking at the front image of the monitor.
Image processing in this case basically needs to be executed as real-time processing.

An information processing apparatus (image processing apparatus) that executes such image processing may perform, for example, a monitor output of a captured image and a process of recording the captured image in a storage unit such as a hard disk in parallel.
In many cases, it is not necessary to refer to the recorded image for the storage means in real time. In other words, there is no problem in the image recording process even if a slight time delay occurs. Such processing is allowed to be executed as a non-real-time task (non-real-time processing).

In a multi-core system in which a plurality of cores (processors) can be operated in parallel, real-time processing and non-real-time processing are mixed, and a plurality of different processing is performed by each of the plurality of cores.

The real-time task (real-time processing) further includes the following two types of processing.
(A1) Periodic processing (a2) Aperiodic processing

(A1) Periodic processing is processing that occurs periodically in advance. The above-described image processing for outputting the captured image of the camera to the monitor (display unit) is also periodic processing. In the image processing for outputting the captured image of the camera to the monitor, an image captured at a predetermined frame rate is cycled. It is necessary to perform processing periodically, which is a periodic process.

For example, the camera captures images at various frame rates, such as 60 f / s for capturing an image of 60 frames per second and 120 f / s for capturing an image of 120 frames per second.
The core (processor) needs to perform periodic processing on each of the continuous images at the prescribed frame intervals.

On the other hand, (a2) non-periodic processing is to execute event-corresponding processing within a predetermined response time when a sudden event occurs.
For example, focus adjustment is required when the focal length changes due to movement of a camera or subject during moving image shooting. If the focus adjustment cannot be completed within a predetermined time, only a blurred image can be obtained and a clear moving image cannot be taken. Similarly, in the camera shake correction process or the like, if the correction process cannot be completed within a predetermined time after the camera shake is detected, the correction cannot be made in time and the purpose cannot be achieved.
Such processing is an aperiodic real-time task.

A periodic real-time task is required to execute one cycle of processing at a predetermined time, and an interrupt response-type real-time task is executed within a predetermined response time from the occurrence of an event such as the above-mentioned doctor operation. To complete the process.

With reference to FIG. 1, an example of an execution sequence of a real-time task by a core (processor) will be described.
FIG. 1 shows three examples of real-time task execution sequences by one core (processor) in an information processing apparatus having a plurality of cores (processors).
For real-time tasks,
Event occurrence time = Ts
Processing completion specified time = Te,
And
In order to satisfy the real-time property of the real-time task, it is necessary to complete the processing by the predetermined processing completion time (Te) after the predetermined time (Te-Ts) after the event occurrence time.

(1) Processing example 1 is an example in which processing within a specified time for a real-time task is successfully completed.
In Processing Example 1, after the event occurrence time = Ts, the processing by the core is started at time T1 after a predetermined delay time (latency), and the processing is ended at time T2.
The processing end time T2 is a time before the processing completion specified time Te, and this processing example 1 is a successful example of real-time processing.

(2) Processing example 2 is a failure example of processing completion within a specified time of a real-time task.
In the processing example 2, the processing by the core is started at a time T1 after a predetermined delay time (latency) after the event occurrence time = Ts, and so far, the processing is the same as the processing example 1. However, the processing end time T2 is a time after the processing completion specified time Te, and this processing example 2 is an example of failure of real-time processing.

This processing example 2 is an example in which the processing time of the real-time task has become longer. This is because, for example, resources and data required for real-time task processing compete with resources and data used by other tasks (processing) executed by other cores, and there is a waiting time for acquiring resources and data. This is an example that can occur.

(3) Processing example 3 is also an example of a process completion failure within a specified time of the real-time task.
In the processing example 3, the processing by the core is started at the time T1 after a predetermined delay time (latency) after the event occurrence time = Ts, but the delay time from the event occurrence time Ts to the processing start time T1 is started. In comparison with the processing examples 1 and 2, it is longer.
Although the time from the process start time T1 to the process end time T2 is shorter than the process example 1, as a result, the process end time T2 is a time after the process completion specified time Te, and this process example 3 is also real time. This is an example of task processing failure.

In this processing example 3, during the period from the event occurrence time Ts to the processing start time T1, another processing, for example, another real-time task or non-real-time task is being executed in this core, and the task (processing) ends. This is an example that occurs when it is necessary to wait.

[2. Task migration processing (migration) between cores]
Next, task movement processing (migration) between cores will be described with reference to FIG.
In an information processing apparatus having a plurality of cores (processors) such as a multi-core system, each of the plurality of cores executes a task (processing) assigned thereto.

The tasks executed in each core are various different tasks, and the required processing time is also different.
Therefore, for example, in a configuration in which various tasks are executed using two cores, the balance of processing assigned to each core may deteriorate.
Specifically, there is a case where the processing amount of the task assigned to one core is large, the processing amount of the task assigned to the other core is small, and an idle time occurs in the other core.

In such a case, the task assigned to one core may be moved to the other core for processing.
Such task movement processing between cores is called migration.

A specific example of migration will be described with reference to FIG.
FIG. 2 shows an example of migration in a configuration in which tasks (processes) are executed using two cores (CPUs).
Time elapses in the order of time zones T10 to T20, T20 to T30, and T30 to T40 from the left to the right shown in the figure. The figure shows the task execution status of the two cores 1 (CPU 1) and 2 (CPU 2) in each time zone.

In time zone T10-T20,
Core 1 (CPU 1) executes task A, task B, and task C. The core 1 (CPU 1) executes these three tasks as, for example, time division processing.
The size of the task A to C area of the core 1 (CPU 1) shown at (time T10 to T20) corresponds to the distribution ratio of the processing time of each task in the core 1 (CPU 1).
On the other hand, the core 2 (CPU 2) executes only the task D and there is a free time during which no processing is performed.

When a processing imbalance occurs such that tasks concentrate on one core and idle time occurs on the other core, the total processing time, for example, the time until all tasks of tasks A to D are completed. Will become longer.

In such a case, task movement between cores, that is, migration is effective.
During time T20 to T30 in FIG. 2, the task B assigned as the execution process of the core 1 (CPU 1) is moved to the core 2 (CPU 2).
Through the task movement processing between the cores, that is, migration, the processing amounts of the core 1 and the core 2 can be made substantially equal.

At time T30 to T40 after the migration process,
Core 1 (CPU 1) executes only task A and task C.
On the other hand, the core 2 (CPU 2) executes task D and task B.
Note that arrows in the figure at times T30 to T40 mean that the distribution of processing time of each task in each core can be expanded.

That is, the core 1 (CPU 1) can use the processing time of the task B, which has been executed during the time T10 to T20, as the processing time of the task A and the task C.
In addition, the core 2 (CPU 2) can use all of the free time in the times T10 to T20 as the processing time of the task B that has moved by migration.

By performing such migration, it is possible to increase the total processing efficiency in the multi-core system.

[3. Example of real-time processing using shield processing]
Next, shield processing for setting a dedicated core for performing real-time processing will be described with reference to FIG.

A core isolation process (technology) that allocates a part of a core to a specific process and prevents disturbances that affect the process is called a shield process or a shielding technique.
FIG. 3 shows an example of shield processing for a multi-core system having four cores, that is, four cores, core 1 (CPU 1) to core 4 (CPU 4).

In the 4-core configuration shown in FIG. 3, the second core 2 (CPU 2) from the left is set as a real-time core dedicated to real-time processing. That is, the core 2 (CPU 2) is shielded (isolated) and set as a real-time processing-dedicated core.
Other cores other than the core 2 (CPU 2) are non-real-time cores that execute non-real-time processing.

Only a real-time task is assigned to the core 2 (CPU 2) which is a real-time core.
Other cores (core 1 (CPU 1), core 3 (CPU 3), and core 4 (CPU 4)) other than core 2 (CPU 2) execute various applications (App) that are non-real-time tasks.
Each core also executes a kernel thread.
The kernel thread corresponds to a part of kernel processing that executes processing such as management of resources and memory necessary for task execution in each core and task switching.
The kernel is software that is the core of the OS (operating system).

Core 2 (CPU 2), which is a real-time core, receives only an interrupt from a device necessary for processing a real-time task, device B shown in the figure.
Interrupts to the real-time core from other devices are prohibited.
On the other hand, other cores (core 1 (CPU 1), core 3 (CPU 3), core 4 (CPU 4)) other than core 2 (CPU 2) are allowed to interrupt from various devices.

The device is a device that requires a processing result by the core, and is configured by various devices such as a GPU (graphic processing unit), a hard disk (HD), and a USB.

Note that in the example shown in FIG. 3, only one device B is the interrupt-permitting device for the core 2 (CPU 2), which is a real-time core, but there may be a plurality of devices.
For example, it is assumed that the real-time task executed in the core 2 (CPU 2) is processing for monitor output of a captured image, as in the above-described example.
In this case, the device B is, for example, a GPU (graphic processing unit).

The core 2 (CPU 2), which is a real-time core, executes generation of image processing parameters necessary for an image processing process in a GPU, for example. Core 2 (CPU2). Provide the generated parameters to the GPU. The GPU uses the generation parameters of the core 2 (CPU 2) to generate an image for monitor output.
These series of processes need to be executed without time delay in order to immediately output the camera-captured image to the monitor.
That is, the image parameter generation process executed in the core 2 (CPU 2) is a real-time task.

In order to guarantee the real-time property of the real-time task in the core 2 (CPU 2), which is the real-time core shown in FIG. 3, that is, task completion (processing completion) up to a specified time, The following processing is performed.
(1) Only real-time tasks are assigned to real-time cores.
(2) The real-time core allows only interrupts from devices necessary for processing real-time tasks, and prohibits interrupts from other devices.

The shield process with the two processes (1) and (2) above allows the core 2 (CPU 2), which is a real-time core, to execute a real-time task without being interrupted by other tasks or interrupts. Can be completed in time. That is, processing that guarantees real-time performance is possible.

[4. About a configuration that performs real-time processing with a specific core and improves the processing efficiency of the entire device]
Next, a configuration of the present disclosure in which real-time processing is performed with a specific core and processing efficiency of the entire apparatus is improved will be described.

The configuration of the present disclosure is a configuration that executes a process to which a new shielding technique is applied.
Specifically, the shielded core that executes real-time processing is also configured to execute a non-real-time task as long as the real-time task executed in the core can be completed within a specified time.

That is, in the configuration of the present disclosure, the shield process of the core that executes the real-time task is a gentle shield process. That is, the real-time task execution core is also configured to execute a non-real-time task depending on the situation.
As described above, by executing the non-real-time task even in the execution core of the real-time task, it becomes possible to improve the processing efficiency of the entire apparatus even in a configuration without a sufficient number of cores.

Hereinafter, with reference to FIG. 4 and FIG. 5, a difference between the conventional strict shield configuration and the gentle shield configuration of the present disclosure will be described.
FIG. 4 is a diagram illustrating a conventional strict shield configuration, and FIG. 5 is a diagram illustrating a gradual shield configuration of the present disclosure.

First, a conventional strict shield configuration will be described with reference to FIG.
The configuration shown in FIG. 4 is the same as the configuration described above with reference to FIG. 3, and is a multi-core system configuration having four cores, that is, four cores, core 1 (CPU 1) to core 4 (CPU 4). is there.

In the four-core configuration shown in FIG. 4, conventional strict shielding (isolation) processing is executed for the second core 2 (CPU 2) from the left, and the core 2 (CPU 2) is made a real-time core dedicated to real-time tasks. Set.
Other cores other than the core 2 (CPU 2) are non-real-time cores that execute non-real-time tasks.

Only the real-time task (RT) is assigned to the core 2 (CPU 2), which is a conventional real-time core with a strict shield.
Other cores (core 1 (CPU 1), core 3 (CPU 3), core 4 (CPU 4)) other than core 2 (CPU 2) execute a non-real-time task (non-RT).

Although not shown in FIG. 4, each core also executes a kernel thread, as described above with reference to FIG.
As described above, the kernel thread is a process corresponding to the kernel, which is the core software of the OS (operating system), management of resources and memory necessary for task execution in each core, and processing such as task switching Is included.

The four cores shown in FIG. 4, that is, core 1 (CPU 1) to core 4 (CPU 4), are processes corresponding to various devices such as a GPU (graphic processing unit), a hard disk (HD), and a USB. Execute.

However, the core 2 (CPU 2), which is a real-time core isolated by strict shielding processing, receives only an interrupt from a device necessary for processing a real-time task, and executes only processing corresponding to the device.
Interrupts from other devices are prohibited for the core 2 (CPU 2), which is a real-time core isolated by strict shielding processing.

Interrupts from various devices are allowed for cores other than core 2 (CPU 2) (core 1 (CPU 1), core 3 (CPU 3), core 4 (CPU 4)).

As described above, the core 2 (CPU 2), which is a real-time core isolated by the strict shielding process shown in FIG. 4, has the following strict shielding in order to guarantee the real-time property of the real-time task, that is, the task completion up to a specified time. Processing is performed.
(1) Only real-time tasks are assigned to real-time cores.
(2) The real-time core allows only interrupts from devices necessary for processing real-time tasks, and prohibits interrupts from other devices.

With the strict shielding process involving the two processes (1) and (2) above, the core 2 (CPU 2), which is a real-time core, can execute a real-time task without being interrupted by other tasks or interrupts. Can be completed within a specified time. That is, processing that ensures real-time performance is possible.

However, if such a strict shielding process is performed, only the real-time task is executed in the core 2 (CPU 2), which is the real-time core, and all non-real-time cores other than the real-time core must perform processing. I must.

There is no problem if the number of cores is large and many non-real-time tasks can be executed at the same time. For example, when setting the number of cores to two or three, the non-real-time cores other than the shielded real-time core The processing load of the core increases, the time until completion of the non-real-time task becomes long, and there arises a problem that the processing efficiency of the entire apparatus is remarkably lowered.

As shown in FIG. 4, when strict shielding processing is performed, the real-time core 2 (CPU 2) can execute only the real-time task (RT). Unassigned and cannot be executed.
As a result, when many non-real-time tasks occur, a small number of non-real-time cores must execute all these non-real-time tasks, which causes a problem that the processing delay of the non-real-time tasks increases.

A configuration of the present disclosure that solves such a problem, that is, a configuration of the present disclosure that performs gentle shielding processing will be described with reference to FIG. 5.
The configuration shown in FIG. 5 is the same as the configuration described with reference to FIG. 4, and is a multi-core system configuration having four cores, that is, four cores of core 1 (CPU 1) to core 4 (CPU 4).

In the four-core configuration shown in FIG. 5, the gentle shield (isolation) processing of the present disclosure is executed for the second core 2 (CPU 2) from the left, and the core 2 (CPU 2) is changed to the real-time task (RT). Set to the real-time core to be performed.
In the configuration of the present disclosure, the core 2 (CPU 2), which is a real-time core, executes not only a real-time task (RT) but also a non-real-time task (non-RT).

However, non-real-time tasks (real-time inhibition processing) that are running in the real-time core may not be processed (real-time inhibition processing) until the specified time (real-time performance) of the real-time task (RT) running in the real-time core is hindered. If it is set as an execution scheduled process of (non-RT), the non-real-time task (non-RT) is moved (migrated) to another core.

Other cores other than the core 2 (CPU 2) are non-real time cores that execute non-real time tasks (non-RT).

The conventional real-time core (core 2 (CPU 2)) to which the strict shield described with reference to FIG. 4 is assigned only the real-time task (RT), but the gentle shield of the present disclosure shown in FIG. The real-time core (core 2 (CPU 2)) for which notation is executed executes not only a real-time task (RT) but also a non-real-time task (non = RT).

However, as described above, the non-real-time task (non = RT) being executed by the real-time core (core 2 (CPU 2)) is moved to the non-real-time core based on the occurrence of a predetermined condition.

This pre-defined condition means that processing (real-time inhibition processing) that may prevent processing completion (real-time performance) up to the specified time of the real-time task (RT) being executed in the real-time core is This is the detection that the non-real-time task (non-RT) being executed is set as the scheduled execution process.
When this condition occurs, the non-real-time task (non-RT) is moved (migrated) to another core.

Although not shown in FIG. 5, each core also executes a kernel thread as described above with reference to FIG. 3.
As described above, the kernel thread is a process corresponding to the kernel, which is the core software of the OS (operating system), and manages resources and memory necessary for task execution in each core, and also performs processing such as task switching. is there.

The four cores shown in FIG. 5, that is, core 1 (CPU 1) to core 4 (CPU 4), are processes corresponding to various devices such as GPU (graphic processing unit), hard disk (HD), USB, and the like. Execute.

Core 2 (CPU 2), which is a real-time core isolated by gentle shield processing, receives not only interrupts from devices necessary for real-time task processing but also non-real-time processing interrupts without being prohibited.

The core 2 (CPU 2), which is a real-time core isolated by the strict shielding process shown in FIG. 4 described above, is prohibited from interrupting from other devices, but is isolated by the gentle shielding process shown in FIG. The core 2 (CPU 2), which is a real-time core, is allowed to interrupt from other devices.
Other cores (core 1 (CPU 1), core 3 (CPU 3), core 4 (CPU 4)) other than core 2 (CPU 2) are allowed to interrupt from various devices.

The core 2 (CPU 2), which is a real-time core isolated by the gentle shield process shown in FIG. 5, performs a shield process with the following settings in order to guarantee the real-time property of the real-time process, that is, the completion of the process up to a specified time. .
(1) A real-time task is assigned to a real-time core.
(2) A non-real time task is also assigned to the real time core.
(3) Non-real-time tasks (non-real-time processing) that are running in the real-time core (non-real-time inhibition processing) that may interfere with processing completion (real-time performance) of the real-time task that is running in the real-time core. If it is set to the execution schedule process of (RT), the non-real-time task (non-RT) is moved (migrated) to another core.

The core 2 (CPU 2), which is a real-time core, can complete the real-time task within a specified time by the gentle shielding process involving the three processes (1), (2), and (3). That is, it is possible to process a real-time task that ensures real-time performance.

By performing such a gentle shield process, for example, when the number of cores is set to 2 or 3 and the number of non-real-time processing increases, both the real-time core and the non-real-time core are used. Non-real-time processing can be executed, the time until completion of non-real-time processing can be shortened, and the processing efficiency of the entire apparatus can be improved.

[5. About the task scheduler that controls the execution tasks in each core]
As described with reference to FIG. 5, in the apparatus of the present disclosure, a gentle shield process is executed for a real-time core, and a real-time task and a non-real-time task are also assigned to the real-time core.

However, non-real-time tasks (non-RT) running in the real-time core may not be processed (real-time inhibition processing) that may prevent processing completion (real-time performance) of the real-time task running in the real-time core. ) Is set as an execution schedule process, the non-real-time task (non-RT) is moved (migrated) to another core.
By performing these processes, it is possible to improve the processing efficiency of the entire apparatus.

Task assignment to a plurality of cores and task migration (migration) are executed in a kernel which is a partial function of the OS.
With reference to FIG. 6, task assignment to a core executed by the kernel and task migration (migration) processing will be described.

FIG. 6 illustrates a multi-core 20 configured by a plurality of cores (CPUs) configuring an information processing apparatus of the present disclosure and a device group 10 configured by a plurality of devices.
The device group 10 includes various devices. Specifically, it is a device that requires a processing result by any one of the cores in the multi-core 20, and is configured by various devices such as a GPU (graphic processing unit), a hard disk (HD), and a USB.

The multi-core 20 includes two or more cores (CPUs) as hardware.
FIG. 6 shows a software configuration (software stack) executed by a core (CPU) that is hardware in the multi-core 20.

In the figure, a kernel (OS) 30 as a base stack and a plurality of tasks as its upper stack are shown.
The plurality of tasks include a number of tasks such as a real-time task (RT) 52 in which a process completion deadline and a processing start time are defined, and other non-real-time tasks (non-RT) 51, 53, and 54.
These are executed in each of a plurality of cores constituting the multi-core 20.

A kernel thread is executed in each core constituting the multi-core 20. This is the kernel thread described above with reference to FIG.
The kernel thread is a process corresponding to the kernel that is the core software of the OS (operating system), and includes processing such as management of resources and memory necessary for task execution in each core, and task switching.

As shown in FIG. 6, the kernel also has a function of a task scheduler 31.
The task scheduler 31 executes task scheduling processing such as task assignment to each core constituting the multi-core 20 and task movement processing between each core.
These task scheduling processes are also part of the kernel thread executed in each core.

The task scheduler 31 of the kernel (OS) 30 shown in FIG. 6 executes processing such as task assignment to each core constituting the multi-core 20 and task movement between the cores.

The task scheduler 31 also executes the task movement process (migration) described above with reference to FIG.
In other words, processing (real-time inhibition processing) that may interfere with processing completion (real-time performance) of a real-time task being executed in the real-time core is prevented from being processed in the real-time core (non-RT ) Is set as an execution schedule process, the non-real-time task (non-RT) is moved (migrated) to another core.

The sequence of processing executed by the task scheduler 31 will be described with reference to the flowchart shown in FIG.

Processing of each step in the flow shown in FIG. 7 will be described.
(Step S101)
The task scheduler 31 first executes a normal task scheduling process in step S101.
Specifically, task scheduling is executed in which real-time tasks are assigned to real-time cores, and non-real-time tasks are assigned according to free times of all cores including real-time cores and non-real-time cores.

(Step S102)
Next, in step S102, the task scheduler 31 performs processing (real-time inhibition processing) that affects the real-time property of the real-time task being executed in the real-time core (the processing completion certainty up to the specified time) by the real-time core. It is determined whether or not the non-real-time task (non-RT) being executed is set as a scheduled execution process.

The task scheduler 31 monitors the processes being executed in all the cores, and further monitors and manages the schedules of all processes such as process requests from all devices and interrupt process requests.

As part of the process monitoring process, the task scheduler 31 includes a process newly scheduled as an execution scheduled process of a non-real-time task (non-RT) being executed in the real-time core.
“Processing that affects the real-time performance of the real-time processing being executed in the real-time core (process completion certainty up to the specified time) (real-time inhibition processing)”
The process which determines whether it is is performed is performed.

“Processing that affects the real-time performance of the real-time processing being executed in the real-time core (process completion certainty up to the specified time) (real-time inhibition processing)”
For example, a process that causes an interruption of a real-time process being executed in the real-time core or the like.

This “Process that affects the real-time performance of the real-time processing being executed in the real-time core (the certainty of processing completion up to the specified time) (real-time inhibition processing)”
A specific example will be described with reference to FIG.

In FIG. 8, “Processing that affects the real-time performance of the real-time processing being executed in the real-time core (process completion certainty up to the specified time) (real-time inhibition processing)”
As an example, the following three processes are shown.

(1) System call (2) Interrupt inhibition processing (3) Preemption inhibition processing

(1) The system call is an output process of instructions and functions issued by the OS (kernel) (for example, instructions and functions for providing and using functions for tasks).
(2) The interrupt prohibition process is an interrupt prohibition process forcibly executed by interrupting a process being executed.
(3) The preemption prohibition process is a preemption prohibition process for temporarily interrupting a task being executed.

When the processes (1) to (3) shown in FIG. 8 (real-time inhibition process) are set as scheduled execution processes for a non-real-time task (non-RT) being executed in the real-time core, they are executed in the real-time core. There is a possibility of interruption of the real-time task during.

In step S102, the task scheduler 31 executes a non-real-time task (non-RT) in which a real-time inhibition process such as the processes (1) to (3) shown in FIG. It is determined whether or not the processing is set.

If the determination in step S102 is Yes, that is, if it is determined that the real-time inhibition process is set as the execution scheduled process for the non-real-time task (non-RT) being executed in the real-time core, the process proceeds to step S103.
Otherwise, the process returns to step S101, and normal task scheduling processing is continued.

(Step S103)
If the determination in step S102 is Yes, that is, if it is determined that the real-time inhibition process is set as the execution scheduled process for the non-real-time task (non-RT) being executed in the real-time core, the process proceeds to step S103.
In step S103, the task scheduler 31 performs processing for moving (migrating) the non-real-time task (non-RT) set as the execution scheduled processing to the real-time inhibition processing to another core.

By this task movement process (migration), the real-time core that is executing the real-time task does not need to execute the non-real-time task that is scheduled to execute the real-time inhibition process, without causing an unexpected interruption of the real-time task, Real-time tasks can be completed by the scheduled time. That is, a real-time task that secures real-time performance can be executed.

[6. Specific system configuration and processing example to which the processing of the present disclosure is applied]
Next, a specific system configuration and processing example to which the processing of the present disclosure is applied will be described.
The embodiment described below is an endoscope system that captures 4K images.
The 4K image is a high-quality image having high density pixels of 3840 × 2160 pixels, for example.
For example, the endoscope system captures 120 frames per second of a 4K image and performs a process of executing a process of displaying the frame on a monitor.

That is, an image processing apparatus that processes an image photographed by an endoscope must continuously input a 120 f / s image and continuously generate an image for output to a monitor. Don't be.

FIG. 9 shows a configuration example of the endoscope system 100.
The endoscope system 100 includes a camera device 110, an image processing device 120, and a monitor 140.
The camera 110 captures a 4K image at a frame rate of 120 f / s and inputs the captured image to the image processing apparatus 120.
The image processing apparatus 120 performs image processing on a 120 f / s image input from the camera 110, that is, processing on each of 120 captured images per second, and generates an output image to be output to the monitor 140. .
The output image generated by the image processing apparatus 120 is output to the monitor 140.
For example, a doctor performs an appropriate treatment while viewing the output image of the monitor 140.

The image output to the monitor 140 needs to be a real-time image, and a large time delay is not allowed.
Therefore, the image processing apparatus 120 needs to input an image with a frame rate of 120 f / s from the camera 110, perform image processing for each frame without time delay, and generate an output image of 120 frames per second. is there.
This image generation process is a real-time task.

As shown in FIG. 9, the image processing apparatus 120 includes a multi-core 130 having a plurality of cores (CPUs) and a device group 140 composed of a plurality of devices.

The multi-core 130 has two cores, a core 1 (CPU 1) 131 and a core 2 (CPU 2) 132.
The device group 140 includes a plurality of devices A (GPU) 141, device B (camera) 142, device C (USB) 143,. These devices are devices that require a processing result by the core, and are constituted by various devices such as a GPU (graphic processing unit), a camera, and a USB.

When an image with a frame rate of 120 f / s is input from the camera 110 and image processing of each frame is executed to generate an output image of 120 frames per second, the following processing is performed for each frame. Is required.
(Process 1) Processing by a real-time core in the multi-core 30, for example, the core 1 (CPU 1) 131,
(Process 2) Process by device A (GPU) 141 which is one device of the device group,
It is necessary to execute these

processes

1 and 2 sequentially.

These series of processes need to be executed without time delay in order to immediately output the camera-captured image to the monitor.
The core 1 (CPU 1) 131 that is a real-time core executes generation of image processing parameters necessary for an image processing process in the device A (GPU) 141, for example. The core 1 (CPU 1) 131 is. The generated parameter is provided to the device A (GPU) 141. The device A (GPU) 141 generates an image for monitor output using the generation parameters of the core 1 (CPU 1).

These processes need to be executed without time delay in order to immediately output the camera-captured image to the monitor, and the image parameter generation process executed in the core 1 (CPU 1) 131 must be executed as a real-time task.

The processing sequence of the core 1 (CPU 1) 131 and the device A (GPU) 141, which are real-time cores, will be described with reference to FIG.
The processing sequence is shown at the bottom of FIG.
Time elapses from left to right along the time axis (t).
Times T1 and T2 are the period of the input image from the camera (V period = 1 frame interval).
In this example, the input image has a frame rate of 120 f / s,
Since 1/120 = 8.3 ms,
V cycle = T1 to T2 = 8.3 ms.

The core 1 (CPU 1) 131 and the device A (GPU) 141, which are real-time cores, need to repeatedly perform image processing in units of one frame at intervals of 8.3 ms.

The sequence diagram shown in FIG. 10 shows the processing sequence of one frame image.
First, at time t11 after time T0, an image input interrupt corresponding to an image processing request is input from the device B (camera) 142 to the core 1 (CPU 1) 131 which is a real-time core.
Note that control for inputting an image input interrupt to the core 1 (CPU 1) 131, which is a real-time core, is executed by a kernel task scheduler.

In response to this interrupt input, the core 1 (CPU 1) 131, which is a real-time core, generates image processing parameters used for actual processing, specifically, image processing in the GPU, from time t12 after a predetermined delay time. Start processing.
This process is a real-time task, and the time until the process is completed is defined.

The core 1 (CPU 1) 131, which is a real-time core, completes the image processing parameter generation process at time t13 and outputs the generated image processing parameter to the device A (GPU) 141.
Thereafter, the device A (GPU) 141 generates an output image and outputs it to the monitor 140 until time T2.

When the conventional strict shielding process is executed, the core 1 (CPU 1) 131, which is a real-time core, is set to be allowed to execute only real-time tasks, and the task is executed for a time from time t13 to time T2. Therefore, it was necessary to wait until the next frame was processed.

However, in the configuration of executing the gentle shielding process of the present disclosure, the core 1 (CPU 1) 131 that is a real-time core is allowed to execute not only a real-time task but also a non-real-time task.
Therefore, after the time t13, it is possible to execute the non-real-time task processing for the time up to the time T2.

As shown in FIG. 9, the configuration having two cores enables parallel processing of non-real-time tasks using two cores, and the processing efficiency jumps compared to a configuration in which only one core executes non-real-time tasks. Can be increased.

Note that the core 1 (CPU 1) 131, which is a real-time core, can also execute a non-real-time task in a time-sharing manner in addition to the real-time task that is the image processing parameter generation process even during the period of time t12 to t13. .

However, if real-time obstruction processing occurs in the non-real-time task that is being executed at this time, that is, processing completion (real-time performance) up to the specified time of the real-time task (RT) being executed in the real-time core may be hindered. When there is a process that has a characteristic (real-time inhibition process), the non-real-time task (non-RT) is moved (migrated) to another core.

By performing these processes, it is possible to reliably execute a process that ensures the real-time property of the real-time task executed in the real-time core.

[7. Configuration example of information processing apparatus]
Next, a hardware configuration example of an information processing apparatus that executes processing according to the above-described embodiment will be described with reference to FIG.
For example, the image processing apparatus 120 described with reference to FIGS. 9 and 10 also has the hardware configuration of the information processing apparatus 300 shown in FIG.

Hereinafter, each component which comprises the information processing apparatus shown in FIG. 11 is demonstrated.
The multi-core 301 includes a plurality of cores (CPU: Central Processing Unit). As shown in FIG. 11, the multi-core 301 has at least two cores such as a core 1 (CPU 1) 351, a core 2 (CPU 2) 352, a core 3 (CPU 3) 353, and the like.

In these plural cores (CPUs), for example, various processes according to programs stored in the ROM (Read Only Memory) 303 or the storage unit 309 are executed.

Note that at least one of the cores in the multi-core 301 is set as a real-time core whose processing is limited by the gentle shield processing described in the above-described embodiment.
As described above, in the configuration of the present disclosure, the shield process of the core that executes the real-time task is a gentle shield process, and the execution of the non-real-time task is allowed even in the real-time core depending on the situation.

However, non-real-time tasks (non-RT) running in the real-time core may not be processed (real-time inhibition processing) that may prevent processing completion (real-time performance) of the real-time task running in the real-time core. ) Is set to the scheduled execution process, a process of moving (migrating) the non-real-time task (non-RT) to another core is executed.

The real-time core can complete the real-time task within the specified time by performing a gentle shield process that is set to perform task migration (migration) under the above conditions for the real-time core. In other words, processing that ensures real-time performance of the real-time task is possible.

A GPU (Graphic Processing Unit) 302 is an image-dedicated processor that executes image processing. As described above, image processing using a parameter generated by the CPU is performed.

A ROM (Read Only Memory) 303 is used as a storage area for programs and parameters executed by the multi-core 301 and the GPU 302.
A RAM (Random Access Memory) 304 is used as a work area for processing executed by the multi-core 301 and the GPU 302, a parameter storage area, a recording area for other data, and the like.
These multi-core 301, GPU 302, ROM 303, and RAM 304 are mutually connected by a bus 305.

The multi-core 301 and the GPU 302 are connected to an input / output interface 306 via a bus 305. The input / output interface 306 includes various switches, a keyboard, a touch panel, a mouse, a microphone, and a data acquisition unit such as a sensor and a camera. An input unit 307, a display such as a monitor, and an output unit 309 including a speaker are connected.

The multi-core 301 receives commands, status data, and the like input from the input unit 307, executes various processes, and outputs the processing results to the output unit 308, for example.
The storage unit 309 connected to the input / output interface 306 includes, for example, a hard disk and stores programs executed by the multi-core 301 and various data. The communication unit 310 functions as a data transmission / reception unit via a network such as the Internet or a local area network, and communicates with an external device.

The drive 311 connected to the input / output interface 306 drives a removable medium 312 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory such as a memory card, and executes data recording or reading.

[8. Summary of composition of the present disclosure]
As described above, the embodiments of the present disclosure have been described in detail with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiments without departing from the gist of the present disclosure. In other words, the present invention has been disclosed in the form of exemplification, and should not be interpreted in a limited manner. In order to determine the gist of the present disclosure, the claims should be taken into consideration.

The technology disclosed in this specification can take the following configurations.
(1) a plurality of cores for executing data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The task scheduler
A task allocation process for causing the real-time core to execute not only a real-time task but also a non-real-time task;
An information processing apparatus that executes migration for moving a non-real-time task assigned to the real-time core to another core in response to occurrence of a predetermined event.

(2) The task scheduler
Real-time inhibition processing, which is a process that may prevent processing completion (real-time performance) of a real-time task being executed in the real-time core from being executed, is scheduled to be executed for a non-real-time task being executed in the real-time core. The information processing apparatus according to (1), wherein migration is performed to move the non-real-time task from the real-time core to another core.

(3) The real-time inhibition process is:
The information processing apparatus according to (2), wherein the information processing apparatus is one of a system call, an interrupt prohibition process, a preemption prohibition process, a CPU exception process, or a cache lock.

(4) The task scheduler
Assign a real-time task only to the real-time core and execute it in the real-time core,
The information processing apparatus according to any one of (1) to (3), wherein a non-real-time task is assigned to the non-real-time core and the real-time core, and task assignment processing is executed by a plurality of cores.

(5) The information processing apparatus according to any one of (1) to (4), wherein each of the plurality of cores is a processor.

(6) The information processing apparatus
It is a configuration that executes image processing for inputting an image captured by a camera and generating an output image to be displayed on the display unit,
The real-time core is
The image processing device according to any one of (1) to (5), wherein generation of a parameter provided to a GPU (Graphic Processing Unit) that executes the image processing is executed as the real-time task.

(7) The real-time core is
The information processing apparatus according to (6), wherein the parameter generation processing corresponding to one image frame is executed as a real-time task, and the non-real-time task is executed in a free time thereafter.

(8) An information processing method executed in the information processing apparatus,
The information processing apparatus includes:
Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The task scheduler is
A task allocation process for causing the real-time core to execute not only a real-time task but also a non-real-time task;
An information processing method for executing migration for moving a non-real-time task assigned to the real-time core to another core in response to occurrence of a predetermined event.

(9) A program for executing information processing in an information processing device,
The information processing apparatus includes:
Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The program is stored in the task scheduler.
Causing the real-time core to execute a task assignment process for executing not only a real-time task but also a non-real-time task;
A program for executing migration to move a non-real-time task assigned to the real-time core to another core in response to occurrence of a predetermined event.

Further, the series of processes described in the specification can be executed by hardware, software, or a combined configuration of both. When executing processing by software, the program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware and executed, or the program is executed on a general-purpose computer capable of executing various processing. It can be installed and run. For example, the program can be recorded in advance on a recording medium. In addition to being installed on a computer from a recording medium, the program can be received via a network such as a LAN (Local Area Network) or the Internet and installed on a recording medium such as a built-in hard disk.

In addition, the various processes described in the specification are not only executed in time series according to the description, but may be executed in parallel or individually according to the processing capability of the apparatus that executes the processes or as necessary. Further, in this specification, the system is a logical set configuration of a plurality of devices, and the devices of each configuration are not limited to being in the same casing.

As described above, according to the configuration of an embodiment of the present disclosure, a configuration in which a non-real-time task is executed in a real-time core to improve the processing efficiency and the real-time property of the real-time task is ensured is realized.
Specifically, for example, it has a real-time core that executes a real-time task, a plurality of cores configured by a non-real-time core that executes a non-real-time task, and a task scheduler that schedules the tasks of the plurality of cores. The task scheduler executes task assignment that allows the real-time core to execute not only real-time tasks but also non-real-time tasks, and migrates non-real-time tasks assigned to real-time cores to other cores in response to the occurrence of a predefined event. Do.
With this configuration, a non-real-time task is executed in the real-time core to improve processing efficiency, and a configuration that ensures the real-time property of the real-time task is realized.

10 device group 20 multi-core 30 kernel (OS)
31

Task scheduler

51, 53, 54, Non-real time task 52 Real time task 100 Endoscope system 110 Camera device 120 Image processing device 130 Multi-core 131, 132 Core (CPU)
140 device group 141-143 device 301 multi-core 302 GPU
303 ROM
304 RAM
305 Bus 306 Input / output interface 307 Input unit 308 Output unit 309 Storage unit 310 Communication unit 311 Drive 312 Removable media 351 to 353 Core (CPU)

Claims

Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The task scheduler
A task allocation process for causing the real-time core to execute not only a real-time task but also a non-real-time task;
An information processing apparatus that executes migration for moving a non-real-time task assigned to the real-time core to another core in response to occurrence of a predetermined event.
The task scheduler
Real-time inhibition processing, which is a process that may prevent processing completion (real-time performance) of a real-time task being executed in the real-time core from being executed, is scheduled to be executed for a non-real-time task being executed in the real-time core. The information processing apparatus according to claim 1, wherein migration is performed to move the non-real-time task from the real-time core to another core.
The real-time inhibition process is:
The information processing apparatus according to claim 2, wherein the information processing apparatus is one of a system call, an interrupt prohibition process, a preemption prohibition process, a CPU exception process, or a cache lock.
The task scheduler
Assign a real-time task only to the real-time core and execute it in the real-time core,
The information processing apparatus according to claim 1, wherein a non-real time task is assigned to the non-real time core and the real time core, and task assignment processing is executed in which the non real time task is executed by a plurality of cores.
2. The information processing apparatus according to claim 1, wherein each of the plurality of cores is a processor.
The information processing apparatus includes:
It is a configuration that executes image processing for inputting an image captured by a camera and generating an output image to be displayed on the display unit,
The real-time core is
The image processing apparatus according to claim 1, wherein generation of a parameter to be provided to a GPU (Graphic Processing Unit) that executes the image processing is executed as the real-time task.
The real-time core is
The information processing apparatus according to claim 6, wherein the parameter generation processing corresponding to one image frame is executed as a real-time task, and the non-real-time task is executed in a free time thereafter.
An information processing method executed in an information processing apparatus,
The information processing apparatus includes:
Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The task scheduler is
A task allocation process for causing the real-time core to execute not only a real-time task but also a non-real-time task;
An information processing method for executing migration for moving a non-real-time task assigned to the real-time core to another core in response to occurrence of a predetermined event.
A program for executing information processing in an information processing apparatus;
The information processing apparatus includes:
Multiple cores that perform data processing;
A task scheduler for scheduling tasks executed in the plurality of cores;
The plurality of cores includes a real-time core that executes a real-time task and a non-real-time core that executes a non-real-time task,
The program is stored in the task scheduler.
Causing the real-time core to execute a task assignment process for executing not only a real-time task but also a non-real-time task;
A program for executing migration to move a non-real-time task assigned to the real-time core to another core in response to occurrence of a predetermined event.