JP7599905B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to a technique for prediction based on captured images.

In recent years, there have been many efforts to use IT to solve a variety of problems in agriculture, such as predicting yields, predicting optimal harvest times, controlling the amount of pesticide sprayed, and planning field restoration.

For example, Patent Document 1 discloses a method for quickly determining the growth status and harvest forecast by appropriately referring to sensor information acquired from the field where agricultural crops are grown and a database that stores that information, and for quickly detecting and dealing with abnormal growth conditions.

Patent document 2 also discloses a method of field management that reduces variation in the quality and yield of agricultural crops by making arbitrary inferences based on information acquired from a wide variety of agricultural crop-related sensors and referring to registered information.

JP 2005-137209 A JP 2016-49102 A

However, the methods proposed so far have been based on the assumption that a sufficient number of past cases for the fields for which predictions are to be performed are stored and that adjustments have been made so that predictions can be estimated with high accuracy based on information about those cases.

On the other hand, the success or failure of agricultural crops is generally greatly influenced by environmental fluctuations such as weather and climate, and can also vary greatly depending on how workers spray fertilizers and pesticides. If all conditions due to external factors remained constant from year to year, there would be no need to predict yields or harvest times, but unlike industry, agriculture has many external factors that workers cannot control, making predictions extremely difficult. Also, when trying to predict yields when unprecedented weather conditions continue, it is difficult to make accurate predictions using estimation systems that have been adjusted from past examples as mentioned above.

The most difficult case to predict is when the above prediction system is introduced to a new field. For example, consider the case of predicting yield in a specific field or detecting non-productive areas for the purpose of repairing poorly growing areas (dead branches, lesions). In such tasks, images and parameters of crops collected in the past in the field are usually stored in a database. Then, when actually carrying out predictions on the field, images taken in the current observed field and data related to growth information obtained from other sensors are cross-referenced and adjusted to make accurate predictions. However, as mentioned above, when these prediction systems and non-productive area detectors are introduced to a different new field, they cannot be applied immediately because the conditions (of the field) often do not match. In such cases, it was necessary to collect a sufficient amount of data in the new field and make adjustments.

In addition, if the above-mentioned prediction system or non-productive area detector is adjusted manually, it takes a lot of time because the parameters related to crop growth are high-dimensional. Even when deep learning or similar machine learning methods are used, manual labeling (annotation) is usually required to achieve good performance for new input, resulting in high work costs.

Ideally, when introducing a new prediction system or in the case of unprecedented natural disasters or weather conditions, it is desirable to be able to make good predictions and estimates with simple settings that place a low burden on the user.

The present invention provides technology that enables processing using a learning model according to the situation, even when processing is difficult using only previously collected information or when no previously collected information is available.

According to one aspect of the present invention, there is provided a method for selecting one or more candidate learning models from a plurality of learning models trained in different learning environments based on information related to photographing an object;
A second selection means for selecting one or more candidate learning models from the candidate learning models selected by the first selection means based on a result of the object detection process using the candidate learning models;
A detection means for performing an object detection process on a photographed image of the object by using at least one of the candidate learning models selected by the second selection means ;
A means for predicting a yield of a crop and detecting a repair area in a farm field based on an object detection area obtained as a result of the object detection process;
The present invention is characterized by comprising:

The configuration of the present invention makes it possible to process using a learning model according to the situation even when processing is difficult using only previously collected information or when no previously collected information exists.

FIG. 1 is a diagram showing an example of a system configuration. 4 is a flowchart of a process performed by the system. 10 is a flowchart showing details of the process in step S23. 10 is a flowchart showing details of the process in step S233. 3 is a diagram showing an example of a method for photographing a farm field using a camera 10; Diagram showing a difficult case. FIG. 13 is a diagram showing the results of annotation work performed on a captured image. FIG. 13 is a diagram showing a display example of a GUI. FIG. 13 is a diagram showing a display example of a GUI. 4 is a flowchart of a process performed by the system. 10 is a flowchart showing details of the process in step S83. 10 is a flowchart showing details of the process in step S833. FIG. 4 is a diagram showing a detection example of a detection region. FIG. 13 is a diagram showing a display example of a GUI. 1A is a diagram showing an example of the configuration of a query parameter, FIG. 1B is a diagram showing an example of the configuration of a parameter set of a learning model, and FIG. 1C is a diagram showing an example of the configuration of a query parameter.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

[First embodiment]
In this embodiment, a system is described that performs analysis processing of a farm field, such as predicting the yield of agricultural crops in the field and detecting areas that need repair, based on an image of the field captured by a camera.

First, an example of the configuration of a system according to this embodiment will be described with reference to FIG. 1. As shown in FIG. 1, the system according to this embodiment has a camera 10, a cloud server 12, and an information processing device 13.

First, the camera 10 will be described. The camera 10 captures moving images of the field and outputs the images of each frame in the moving images as "captured images of the field". Alternatively, the camera 10 captures still images of the field periodically or irregularly and outputs the captured still images as "captured images of the field". In order to accurately perform the predictions described below from the captured images, it is desirable that images captured in the same field are captured in the same environment and conditions as much as possible. The captured images output from the camera 10 are transmitted to a cloud server 12 or an information processing device 13 via a communication network 11 such as a LAN or the Internet.

The method of photographing the field using the camera 10 is not limited to a specific photographing method. An example of the method of photographing the field using the camera 10 will be described with reference to FIG. 3(A). In FIG. 3(A), cameras 33 and 34 are used as the camera 10. In a typical field, rows of crop trees are planted in a planned manner by a farmer. For example, as shown in FIG. 3(A), rows of crop trees are planted side by side, such as row 30 of crop trees and row 31 of crop trees. The farm tractor 32 is provided with a camera 34 that photographs the row 31 of crop trees on the left side in the traveling direction indicated by the arrow, and a camera 33 that photographs the row 30 of crop trees on the right side. Therefore, when the farm tractor 32 moves between the rows 30 and 31 in the traveling direction indicated by the arrow, the camera 34 will take multiple images of the crop trees in row 31, and the camera 33 will take multiple images of the crop trees in row 30.

In many farm fields that are designed for agricultural tractors 32 to enter and where crop trees are planted at equal intervals, it is relatively easy to photograph a large number of crop trees at a constant height and at a constant distance from the crop trees by photographing the crop trees with cameras 33, 34 installed on the agricultural tractor 32 as shown in FIG. 3(A). This makes it possible to take images of the entire target field under almost the same conditions, making it easy to take images under desirable conditions.

Note that other photographing methods may be used as long as it is possible to photograph the field under roughly the same conditions. An example of a method of photographing a field using a camera 10 will be described with reference to FIG. 3(B). In FIG. 3(B), cameras 38 and 39 are used as the camera 10. As shown in FIG. 3(B), in a field where the distance between the rows of crop trees 35 and the rows of crop trees 36 is narrow and travel by a tractor is not possible, photographing may be performed using cameras 38 and 39 attached to a drone 37. The drone 37 is provided with a camera 39 that photographs the row of crop trees 36 on the left side in the traveling direction indicated by the arrow, and a camera 38 that photographs the row of crop trees 35 on the right side. Therefore, when the drone 37 moves between the rows 35 and 36 in the traveling direction indicated by the arrow, the camera 39 will take multiple photographs of the crop trees in the row 36, and the camera 38 will take multiple photographs of the crop trees in the row 35.

Also, a camera installed on the self-propelled robot may be used to capture images of crop trees. Although the number of cameras used for capturing images is two in Figures 3(A) and (B), this is not limited to a specific number.

Regardless of the method used to capture the image of the crop tree, the camera 10 outputs the captured image with the shooting information at the time of the image capture (Exif information that records the shooting location (e.g., the shooting location measured by GPS), the shooting date and time, information related to the camera 10, etc.) attached to the captured image.

Next, the cloud server 12 will be described. The captured image and Exif information transmitted from the camera 10 are registered in the cloud server 12. In addition, a plurality of learning models (detectors/settings) for detecting an image area related to agricultural products from the captured image are registered in the cloud server 12, and each learning model is a model trained in a different learning environment. Then, the cloud server 12 selects a candidate learning model to be used when detecting an image area related to agricultural products from the captured image from among the plurality of learning models held by the cloud server 12, and presents the candidate learning model to the information processing device 13.

The CPU 191 executes various processes using computer programs and data stored in the RAM 192 and the ROM 193. As a result, the CPU 191 controls the operation of the entire cloud server 12, and executes or controls various processes that will be described as being performed by the cloud server 12.

RAM 192 has an area for storing computer programs and data loaded from ROM 193 or external storage device 196, and an area for storing data received from the outside via I/F 197. RAM 192 also has a work area used by CPU 191 when executing various processes. In this way, RAM 192 can provide various areas as needed.

ROM 193 stores configuration data for cloud server 12, computer programs and data related to starting up cloud server 12, computer programs and data related to the basic operation of cloud server 12, and the like.

The operation unit 194 is a user interface such as a keyboard, mouse, or touch panel, and the user can operate it to input various instructions to the CPU 191.

The display unit 195 has a screen such as an LCD screen or a touch panel screen, and can display the results of processing by the CPU 191 using images, text, etc. The display unit 195 may also be a projection device such as a projector that projects images and text.

The external storage device 196 is a large-capacity information storage device such as a hard disk drive device. The external storage device 196 stores an OS (operating system) and computer programs and data for causing the CPU 191 to execute or control various processes described as being performed by the cloud server 12. The data stored in the external storage device 196 also includes data related to the learning model described above. The computer programs and data stored in the external storage device 196 are loaded into the RAM 192 as appropriate under the control of the CPU 191, and become the subject of processing by the CPU 191.

I/F 197 is a communication interface for performing data communication with the outside, and cloud server 12 transmits and receives data with the outside via I/F 197. CPU 191, RAM 192, ROM 193, operation unit 194, display unit 195, external storage device 196, and I/F 197 are all connected to system bus 198. Note that the configuration of cloud server 12 is not limited to the configuration shown in FIG. 1.

The captured image and Exif information output from the camera 10 may be temporarily stored in the memory of another device, and the captured image and Exif information may be transferred from the memory to the cloud server 12 via the communication network 11.

Next, the information processing device 13 will be described. The information processing device 13 is a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device. The information processing device 13 presents candidates for learning models presented by the cloud server 12 to the user, accepts the user's selection of a learning model, and notifies the cloud server 12 of the learning model selected by the user. The cloud server 12 uses the learning model notified by the information processing device 13 (the learning model selected by the user from the candidates) to detect image areas related to agricultural products from the image captured by the camera 10 (object detection processing), and performs the above-mentioned analysis processing.

The CPU 131 performs various processes using computer programs and data stored in the RAM 132 and the ROM 133. As a result, the CPU 131 controls the operation of the entire information processing device 13, and also executes or controls various processes that will be described as being performed by the information processing device 13.

RAM 132 has an area for storing computer programs and data loaded from ROM 133, and an area for storing data received from camera 10 and cloud server 12 via input I/F 135. RAM 132 also has a work area used by CPU 131 when executing various processes. In this way, RAM 132 can provide various areas as appropriate.

ROM 133 stores setting data for information processing device 13, computer programs and data related to the startup of information processing device 13, computer programs and data related to the basic operation of information processing device 13, etc.

The output I/F 134 is an interface used by the information processing device 13 to output/transmit various information to the outside.

The input I/F 135 is an interface used by the information processing device 13 to input/receive various information from the outside.

The display device 14 has an LCD screen or a touch panel screen, and can display the results of processing by the CPU 131 as images, text, etc. The display device 14 may also be a projection device such as a projector that projects images and text.

The user interface 15 includes a keyboard and a mouse, and can be operated by the user to input various instructions to the CPU 131. Note that the configuration of the information processing device 13 is not limited to the configuration shown in FIG. 1, and may include, for example, a large-capacity information storage device such as a hard disk drive device, in which computer programs and data such as a GUI described below are stored. The user interface 15 may also include a touch sensor such as a touch panel.

Next, we will explain the task flow of predicting the yield of agricultural crops to be harvested in a farm field at a stage earlier than the harvest time, based on images of the farm field captured by the camera 10. When predicting the yield by simply counting the fruits to be harvested at the harvest time, the objective can be achieved by simply detecting the target fruits from the captured images using a classifier using a method called specific object detection. Since the fruits themselves have a very distinctive appearance, this method detects them using a classifier that has learned this distinctive appearance.

In this embodiment, when the crop is fruit, the system does not only count the fruits after they have ripened, but also predicts the yield of the fruit at an earlier stage than the harvest time. For example, the system detects inflorescences that will later become fruit and predicts the yield from their number, detects dead branches or diseased areas that are unlikely to bear fruit, or predicts the yield from the state of leaf growth. To make such predictions, a prediction method that can handle the fact that the growth conditions of agricultural crops differ depending on the time of photography and the weather is required. In other words, it is necessary to select a prediction method with good prediction performance depending on the condition of the agricultural crops. In this case, it is expected that the above predictions will be made appropriately using a learning model that matches the field to be predicted.

Here, various objects captured in captured images are classified into classes such as crop tree trunk class, branch class, dead branch class, and support class, and yield is predicted based on the class. Universal prediction is difficult because the appearance of objects belonging to classes such as tree trunk class and branch class changes depending on the time of capture. An example of such a difficult case is shown in Figure 4.

4(A) and 4(B) show examples of images captured by the camera 10. These captured images show crop trees at approximately equal intervals, but the fruits to be harvested have not yet borne fruit, so the task of detecting fruit cannot be performed from the captured images. The tree in the captured image of FIG. 4(A) is a crop tree captured at a relatively early stage of the season, and the tree in the captured image of FIG. 4(B) is a tree captured at a stage when the leaves have grown to a certain extent. In the captured image of FIG. 4(A), all the branches have the same amount of leaves, so it can be determined that there are no areas of poor growth, and all areas can be determined as harvestable. On the other hand, in the captured image of FIG. 4(B), the leaves of the branches near the central region 41 in the captured image are clearly different from the others, so it is easy to determine that they are poorly grown. However, the appearance of the central region 41 (an area with few leaves) can also be found as a similar pattern near the region 40 in the captured image of FIG. 4(A). These two examples show that abnormal areas in agricultural trees cannot be identified by local patterns. In other words, a judgment cannot be made by inputting only local patterns, as in the specific object detection method described above, and it is necessary to reflect the context obtained from the entire image.

In other words, unless the specific object detection is performed using a learning model trained on images of crops in similar growth conditions taken in the past, sufficient performance will not be achieved.

In order to handle all kinds of cases, such as when an image is input that was taken in a new field where no images have been taken before, when an image is input that was taken under conditions different from those previously taken due to some external factor such as prolonged drought or extremely heavy rainfall, or when an image is input that was taken at a time convenient for the user, it is necessary to acquire a learning model that is trained under conditions close to those of the input image each time.

Here, we will explain what kind of annotation work is required when annotation work and deep learning are performed every time a photograph of a farm field is taken. For example, the results of annotation work performed on the photographed images in Figures 4(A) and 4(B) are shown in Figures 5(A) and 5(B), respectively.

Rectangular areas 500-504 in the captured image in Figure 5 (A) are image areas designated by annotation work. Rectangular area 500 is an image area designated as an area of normal branches, and rectangular areas 501-504 are image areas designated as areas of the tree trunk. Rectangular area 500 is an image area that represents a normal state of tree growth, and therefore this image area is an area that is highly relevant to yield prediction. Hereinafter, areas that represent a normal state of tree growth, such as rectangular area 500, and areas where fruit, etc. can be harvested, are referred to as production areas.

Rectangular regions 505-507 and 511-514 in the captured image in Figure 5 (B) are image regions designated by annotation work. Rectangular regions 505 and 507 are image regions designated as normal branch regions, and rectangular region 506 is an image region designated as an abnormal dead branch region. Regions that indicate abnormal conditions, such as rectangular region 506, and regions where fruit or the like cannot be harvested are called non-productive regions. Rectangular regions 511-514 are image regions designated as tree trunk regions. Rectangular regions 505 and 507 are image regions that are determined to be regions where fruit or the like can be harvested (productive regions), and therefore rectangular regions 505 and 507 are regions that are closely related to yield prediction.

It is very costly to perform such annotation work on a large number of captured images (e.g., hundreds to thousands) each time a field is photographed. Therefore, in this embodiment, good prediction results are obtained without performing such more cumbersome annotation work. In this embodiment, a learning model is acquired by deep learning. However, the method of acquiring the learning model is not limited to a specific acquisition method. Also, various object detectors may be applied instead of the learning model.

Next, the process performed by the system according to this embodiment to perform analytical processes such as predicting the yield in a field and calculating the non-productive rate for the entire field based on the image of the field captured by the camera 10 will be described with reference to the flowchart in FIG. 2A.

In step S20, the camera 10 captures an image of the field while a mobile object such as an agricultural tractor 32 or a drone 37 is moving, thereby generating a captured image of the field.

In step S21, the camera 10 attaches the above-mentioned Exif information (shooting information) to the captured image generated in step S20, and transmits the captured image with the Exif information attached to the cloud server 12 and the information processing device 13 via the communication network 11.

In step S22, the CPU 131 of the information processing device 13 acquires information about the field and crops photographed by the camera 10 (such as the variety and age of the crops, and the cultivation and pruning methods of the crops) as photographed field parameters. For example, the CPU 131 displays a GUI (graphical user interface) shown in FIG. 6(A) on the display device 14 and accepts input of the photographed field parameters from the user.

In the GUI of FIG. 6(A), a map of the entire field is displayed in area 600. The map of the field displayed in area 600 is divided into multiple sections, and each section is displayed with its own unique identifier (ID). The user operates user interface 15 to specify a location in area 600 that corresponds to the section photographed by camera 10 (i.e., the section on which the above-mentioned analysis process will now be performed), or inputs the identifier of the section into area 601. When the user operates user interface 15 to specify a location in area 600 that corresponds to the section photographed by camera 10, the identifier of the section is displayed in area 601.

The user can operate the user interface 15 to input a crop name (the name of the agricultural product) in area 602. The user can also operate the user interface 15 to input a variety of the agricultural product in area 603. The user can also operate the user interface 15 to input Trellis in area 604. For example, if the agricultural product is grapes, Trellis is a method of designing grape vines for growing grapes in a grape field. The user can also operate the user interface 15 to input a planted year in area 605. For example, if the agricultural product is grapes, the planted year indicates the time when the grape vines were planted. Note that it is not necessary to input the photography field parameters for all of these items.

When the user operates the user interface 15 to select the registration button 606, the CPU 131 of the information processing device 13 transmits the photographed field parameters for each of the above items input in the GUI of FIG. 6(A) to the cloud server 12. The CPU 191 of the cloud server 12 stores (registers) the photographed field parameters transmitted from the information processing device 13 in the external storage device 196.

In addition, when the user operates the user interface 15 to select the modification button 607, the CPU 131 of the information processing device 13 enables modification of the photographed field parameters that have already been input in the GUI of FIG. 6(A).

The GUI in FIG. 6(A) is a GUI for inputting photographed field parameters that are intended to be used in particular to manage grape fields, but even if the purpose is the same, the photographed field parameters that the user is prompted to input are not limited to those shown in FIG. 6(A). Similarly, even if the crop is not grapes, the photographed field parameters that the user is prompted to input are not limited to those shown in FIG. 6(A). For example, when the crop name entered in area 602 is changed, the titles of areas 603-605 and the photographed field parameters that the user is prompted to input may be changed.

The photographed field parameters entered in the GUI in FIG. 6(A) can basically be used as fixed parameters once they have been determined, so for example, if photographing the field every year to predict yields, the already registered photographed field parameters can be called up and used. If photographed field parameters have already been registered for a desired division, from the next time, the photographed field parameters corresponding to the desired division can be displayed in regions 609-613 by specifying the location in region 600 that corresponds to the desired division, as shown in FIG. 6(B).

Here, it is desirable to input all correct photographed field parameters in order to select a learning model later, but even if there are photographed field parameters that the user is unable to input because they are unknown, subsequent processing can be carried out while still remaining unknown.

In step S23, a process is performed to select candidates for a learning model to be used to detect objects such as agricultural crops from the captured image. Details of the process in step S23 will be described with reference to the flowchart in FIG. 2B.

In step S230, the CPU 191 of the cloud server 12 generates query parameters from the Exif information attached to each captured image acquired from the camera 10 and the captured field parameters (the captured field parameters for the section corresponding to the captured image) registered in the external storage device 196.

An example of the configuration of a query parameter is shown in FIG. 11(A). The query parameter in FIG. 11(A) is generated when the photographed field parameters in FIG. 6(B) are input.

The "Query Name" is set to "F5" input in area 609. The "Variety" is set to "Shiraz" input in area 611. The "Trellis" is set to "Scott-Henry" input in area 612. The "Age" is set to the number of years elapsed from "2001" input in area 613 to the shooting date and time (year) included in the Exif information, which is the age of the tree "19". The "Shooting Date" is set to "Oct 20", the shooting date and time (month and day) included in the Exif information. The "Shooting Time Zone" is set to "12:00-14:00", the time zone between the oldest shooting date and time (time) and the most recent shooting date and time (time) in the Exif information attached to each captured image received from camera 10. The "latitude, longitude" is set to the shooting position "35°28'S, 149°12'E" contained in the Exif information.

The method of generating query parameters is not limited to the above method. For example, data that a farmer of a certain crop is already using for field management can be read, and a set of parameters that match the above items can be used as query parameters.

In some cases, information about some items may be unknown. For example, if information about the planted year or variety is unknown, it will not be possible to fill in all items as shown in FIG. 11(A). In this case, some of the query parameters will be left blank, as shown in FIG. 11(C).

Next, in step S231, the CPU 191 of the cloud server 12 selects M (1≦M<E) candidate learning models (candidate learning models) from the E (E is an integer equal to or greater than 2) learning models stored in the external storage device 196. In this selection, learning models that have been trained based on an environment similar to the environment indicated by the query parameters are selected as candidate learning models. The external storage device 196 stores, for each of the E learning models, a parameter set that indicates the environment in which the learning model was trained. An example of the configuration of the parameter sets for each learning model in the external storage device 196 is shown in FIG. 11 (B).

"Model name" is the name of the learning model, "variety" is the variety of the crop that the learning model has learned, and "Trellis" is the "method of designing grape trees for growing grapes in a grape field" that the learning model has learned. "Tree age" is the age of the crop that the learning model has learned, and "photographed date" is the date and time of the image of the crop that the learning model used for learning. "Photographed time period" is the period from the oldest date and time of the image of the crop that the learning model used for learning to the most recent date and time of the image, and "latitude, longitude" is the location "35°28'S, 149°12'E" of the image of the crop that the learning model used for learning.

Some learning models learn by mixing data sets collected from multiple field blocks. For this reason, there may be models whose parameter sets are configured to include multiple settings (variety, tree age, etc.), such as the learning models with the model names "M004" and "M005".

The CPU 191 of the cloud server 12 therefore calculates the similarity between the query parameters and the parameter set for each learning model shown in FIG. 11(B), and selects the top M learning models in descending order of similarity as candidate learning models.

If the parameter sets of the learning models with model names = M001, M002, ... are denoted as _M1 , _M2 , ..., the CPU 191 of the cloud server 12 calculates the similarity D(Q, _Mx ) between the query parameter Q and the parameter set _Mx by calculating the following formula (1).

Here, q _k represents the k-th element from the top in the query parameter Q. In the case of Fig. 11A, the query parameter Q includes six elements, namely "variety", "Trellis", "age", "photographed date", "photographed time zone", and "latitude, longitude", so k = 1 to 6.

m _{x, k} represents the k-th element from the top in the parameter set M _x . In the case of Fig. 11B, the parameter set includes six elements, "variety", "Trellis", "age", "photographed date", "photographed time zone", and "latitude, longitude", so k = 1 to 6.

_fk ( _ak , _bk ) is a function for finding the distance between elements _ak and _bk , and is set in advance. _fk ( _ak , _bk ) may be set carefully in advance by experiment, but since the distance definition in the above formula (1) basically only needs to be a larger value for learning models with different properties, it can be set simply as follows:

In other words, elements are basically divided into two types: classification elements (variety, Trellis) and continuous value elements (age of tree, date of photo, etc.). Therefore, the function that specifies the distance between classification elements is defined as in the following formula (2), and the function that specifies the distance between continuous value elements is defined as in the following formula (3).

Functions for all elements (k) are implemented in advance as a rule base. In addition, α _k is determined according to the degree of influence of each element on the final inter-model distance. For example, since differences due to "type" (k=1) do not appear much in the differences in the images, α ₁ is set as close to 0 as possible, and since differences in "Trellis" (k=2) have a large influence, α ₂ is set large in advance.

In addition, in the case of a learning model in which multiple settings are registered for "variety" or "age," such as the learning models with model names "M004" and "M005" in Figure 11 (B), for example, in the case of "variety," the distance is found for each setting registered for "variety," and the average distance is taken as the distance corresponding to "variety." Similarly, in the case of "age," the distance is found for each setting registered for "age," and the average distance is taken as the distance corresponding to "age."

Note that the selection method is not limited to a specific method as long as the CPU 191 of the cloud server 12 selects M learning models as candidate learning models based on the above similarity. For example, the CPU 191 of the cloud server 12 may select M learning models that have a similarity equal to or greater than a threshold value.

However, if all elements in the query parameters are empty, the processing in step S231 is not performed, and as a result, subsequent processing is performed with all learning models as candidate learning models.

The effects of selecting candidate learning models are manifold. First, by eliminating learning models that are unlikely to be prior knowledge in this step, the processing time required for subsequent steps such as scoring learning models to create rankings can be significantly reduced. Also, even in rule-based scoring of learning models, including learning models that do not actually need to be compared as candidates can reduce the accuracy of the learning model selection, but this possibility can be minimized.

Next, in step S232, the CPU 191 of the cloud server 12 selects P (P is an integer equal to or greater than 2) captured images from the captured images received from the camera 10 as images for model selection. The method for selecting P captured images from the captured images received from the camera 10 is not limited to a specific selection method. For example, the CPU 191 may randomly select P captured images from the captured images received from the camera 10, or may select them according to some criteria.

Next, in step S233, a process is performed to select one of the M candidate learning models as a selected learning model using the P captured images selected in step S232. Details of the process in step S233 will be described with reference to the flowchart in FIG. 2C.

In step S2330, the CPU 191 of the cloud server 12 performs an object detection process for each of the M candidate learning models, which is a process for detecting an object from each of the P captured images using the candidate learning model.

As a result, for each of the P captured images, the "result of the object detection process for that captured image" for each of the M candidate learning models is obtained. In this embodiment, the "result of the object detection process for the captured image" is the position information of the image area (rectangular area, detection area) of the object detected from the captured image.

In step S2331, the CPU 191 obtains a score for each of the M candidate learning models for the "result of the object detection process for each of the P captured images." The CPU 191 then ranks the M candidate learning models based on the scores, and selects N (N≦M) candidate learning models from the M candidate learning models.

In this case, since the captured images do not contain annotation information, accurate evaluation of detection accuracy is not possible. However, for objects that are systematically designed and maintained, such as farms, it is possible to predict and evaluate the accuracy of the object detection process using the following rules. The score for the results of the object detection process using the candidate learning model is calculated, for example, as follows:

In a typical farm field, crops are planted at equal intervals, as shown in Figures 3(A) and (B). Therefore, when detecting an object such as the annotation (rectangular area) shown in Figures 5(A) and (B), a normal detection state is when a rectangular area is always detected continuously and evenly from the left edge to the right edge of the image.

For example, as in FIG. 5(A), if the entire captured image from the left to the right edge is detected as an area where fruit or the like can be harvested, then a production area should be detected as rectangular area 500. Also, as in FIG. 5(B), if the captured image contains rectangular area 506, which is a non-production area, then rectangular areas 505, 506, and 507 should be detected from the left to the right edge of the captured image. If an object detection process is performed on the captured image using a learning model that does not match the conditions of the captured image, there is a possibility that some of the above rectangular areas will not be detected. The more the learning model corresponds to conditions that are farther from the conditions of the captured image, the higher the possibility of this happening. Therefore, the following method, for example, can be considered as the simplest scoring method for evaluating candidate learning models.

The candidate learning model of interest detects detection areas of multiple objects from the photographed image of interest. Therefore, the detection area is searched for in the vertical direction of the photographed image of interest, and the number of pixels Cp in the area where the detection area is not present is counted, and the ratio of the number of pixels Cp to the number of pixels in the width of the photographed image of interest is set as the penalty score of the photographed image of interest. In this way, a penalty score is calculated for each of the P photographed images in which object detection processing is performed using the candidate learning model of interest, and the sum of the calculated penalty scores is set as the score of the candidate learning model of interest. By performing such processing for each of the M candidate learning models, the score of each candidate learning model is determined. Then, the M candidate learning models are ranked in order of decreasing score, and the top N candidate learning models are selected in order of decreasing score. When making this selection, a condition that "score is less than a threshold value" may be added.

In addition, the score of the candidate learning model may be calculated based on the detection area of the trunk of a tree, which is usually planted at equal intervals. As shown in FIG. 5(A), tree trunks should be detected at approximately equal intervals as rectangular areas 501, 502, 503, and 504, so the expected number of "detected tree trunk areas" for the width of the captured image is fixed. Images with fewer/more than the expected number are likely to have detection errors, so this detection number may be reflected in the score.

Then, the CPU 191 transmits to the information processing device 13 the P captured images, the "results of the object detection process on the P captured images" of each of the N candidate learning models selected from the M candidate learning models, and information about the N candidate learning models (such as model names). As described above, in this embodiment, the "results of the object detection process on the captured images" are positional information of the image area (rectangular area, detection area) of the object detected from the captured image, and such positional information is transmitted to the information processing device 13 as data in a file format such as json format or txt format.

Next, the user is prompted to select one of the selected N candidate learning models. At the end of the processing in step S2331, N candidate learning models still remain, and since the output on which the performance comparison is based is the result of the object detection processing on the P captured images, the user must compare the results of the object detection processing on the N x P captured images. In such a situation, it is difficult to appropriately select one candidate learning model as the selected learning model (narrow it down to one).

Therefore, in step S2332, the CPU 131 of the information processing device 13 performs scoring (display image scoring) for the P captured images in order to present information that is easy for the user to subjectively compare. In the display image scoring, a larger score is determined for each of the P captured images as the detection area arrangement patterns differ more significantly among the N candidate learning models. Such a score can be obtained, for example, by calculating the following formula (4).

Here, Score(z) is the score for the captured image Iz. T _Iz (Ma, Mb) is a function for obtaining a score based on the difference between the result of the object detection process (arrangement pattern of the detection area) performed by the candidate learning model Ma on the captured image Iz and the result of the object detection process (arrangement pattern of the detection area) performed by the candidate learning model Mb on the captured image Iz. Various functions can be applied to such a function, and it is not limited to a specific function. For example, for each detection area Ra detected by the candidate learning model Ma from the captured image Iz, a function that obtains the difference between the position of the detection area Rb' (e.g., the position of the upper left corner and the position of the lower right corner) closest to the detection area Ra among the detection areas Rb detected by the candidate learning model Mb from the captured image Iz and the position of the detection area Ra (e.g., the position of the upper left corner and the position of the lower right corner) and returns the sum of the obtained differences may be set as T _Iz (Ma, Mb).

The results of object detection processing using the top N candidate learning models are often similar, so even if randomly selected images are compared, there is often little difference, and this does not provide a basis for selecting a learning model. Therefore, by looking at only the top captured images scored using formula (4) above, it becomes easier to judge the quality of the learning model.

In step S2333, the CPU 131 of the information processing device 13 causes the display device 14 to display (display control) the top F (a specified number from the top) captured images of the P captured images received from the cloud server 12 in descending order of score for each of the N candidate learning models, and the results of the object detection process for the captured images received from the cloud server 12. At that time, the F captured images are displayed in descending order of score from the left.

Figure 7(A) shows an example of a GUI displaying the captured images and the results of the object detection process for each candidate learning model. Figure 7(A) shows the case where N = 3 and F = 4.

The top row displays the model name "M002" of the candidate learning model with the highest score along with a radio button 70, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area of the object detected from the captured image by the candidate learning model with model name "M002" is superimposed on the captured image.

In the middle row, the model name "M011" of the candidate learning model with the second highest score is displayed along with a radio button 70, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area of the object detected from the captured image by the candidate learning model with model name "M011" is superimposed on the captured image.

In the bottom row, the model name "M009" of the candidate learning model with the third highest score is displayed along with a radio button 70, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area of the object detected from the captured image by the candidate learning model with model name "M009" is superimposed on the captured image.

In addition, in this GUI, images in the same row are displayed as if they were the same photograph, so that the results of the object detection process using each candidate learning model can be easily compared at a glance.

Then, the user visually checks the differences in the results of the object detection process for the F captured images using the N candidate learning models, and selects one of the N candidate learning models using the user interface 15.

In step S2334, the CPU 131 of the information processing device 13 accepts a candidate learning model selection operation (user operation, user input) by the user. In step S2335, the CPU 131 of the information processing device 13 determines whether or not a candidate learning model selection operation (user input) has been performed by the user.

In the case of FIG. 7(A), when the user wants to select a candidate learning model with the model name "M002", the user selects the radio button 70 in the top row using the user interface 15. When the user wants to select a candidate learning model with the model name "M011", the user selects the radio button 70 in the middle row using the user interface 15. When the user wants to select a candidate learning model with the model name "M009", the user selects the radio button 70 in the bottom row using the user interface 15. In FIG. 7(A), because the radio button 70 corresponding to the model name "M002" is selected, a frame 74 is displayed indicating that the candidate learning model with the model name "M002" has been selected.

When the user operates the user interface 15 to select the decision button 71, the CPU 131 determines that "the user has performed a candidate learning model selection operation (user input)," and selects the candidate learning model corresponding to the selected radio button 70 as the selected learning model.

If the result of this determination is that the user has performed a selection operation (user input) of a candidate learning model, processing proceeds to step S2336; if the user has not performed a selection operation (user input) of a candidate learning model, processing proceeds to step S2334.

In step S2336, the CPU 131 of the information processing device 13 checks whether only one learning model was ultimately selected. If only one learning model was ultimately selected, the process proceeds to step S24, and if not, the process proceeds to step S2332.

Here, if the user is unable to narrow it down to one just by looking at the display in FIG. 7(A), multiple radio buttons 70 may be selected to select multiple candidate learning models. For example, if the user operates the user interface 15 to select the radio button 70 corresponding to the model name "M002" and the radio button 70 corresponding to the model name "M011" in FIG. 7(A) and presses the decision button 71, the number of selected radio buttons 70, "2", is set to N, and the process proceeds to step S2332 via step S2336. In this case, from step S2332 onwards, the same process is performed for N=2 and F=4. In this way, the process is repeated until the number of learning models finally selected becomes "1".

The user may also select a learning model using the GUI of FIG. 7(B) instead of the GUI of FIG. 7(A). The GUI of FIG. 7(A) is a GUI that allows the user to directly select which learning model is better. In contrast, in the GUI of FIG. 7(B), a check box 72 is provided for each captured image, and the user operates the user interface 15 to specify and turn on (put a check mark on) the check box 72 of the captured image that is determined to have a better result of the object detection process among the captured images in the captured image sequence, for each captured image sequence that is lined up vertically. Then, when the user operates the user interface 15 to instruct the decision button 75, the CPU 131 of the information processing device 13 selects, as the selected learning model, the candidate learning model with the largest number of captured images with the check box 72 turned on, from among the candidate learning models with the model names "M002", "M011", and "M009". In the example of FIG. 7(B), three check boxes 72 are checked out of the four captured images of the candidate learning model with the model name "M002", one check box 72 is checked out of the four captured images of the candidate learning model with the model name "M011", and none of the check boxes 72 are checked out of the four captured images of the candidate learning model with the model name "M009". In this case, the candidate learning model with the model name "M002" is selected as the selected learning model. This method of selecting a selected learning model using a GUI is effective, for example, when the value of F increases and it is difficult for the user to determine which candidate learning model is best.

If there are candidate learning models with the same or a small difference in the "number of captured images with check boxes 72 checked," step S2336 determines that "not only one learning model was ultimately selected," and processing proceeds to step S2332. From step S2332 onward, processing is performed on candidate learning models with the same or a small difference in the "number of captured images with check boxes 72 checked." Even in such cases, processing is repeated until the number of learning models ultimately selected becomes "1."

In addition, since the captured image displayed further to the left is an image for which the difference in the results of the object detection process between the candidate learning models is greater, a larger weight value may be assigned to the captured image displayed further to the left. In this case, for each candidate learning model, the sum of the weight values of the captured images for which the check boxes 72 are checked may be calculated, and the candidate learning model with the largest calculated sum may be selected as the selected learning model.

Then, regardless of the method used to select the selected learning model, the CPU 131 of the information processing device 13 notifies the cloud server 12 of information indicating the selected learning model (e.g., the model name of the selected learning model).

In step S24, the CPU 191 of the cloud server 12 performs object detection processing on the captured image (the captured image transmitted by the camera 10 to the cloud server 12 and the information processing device 13) using the selected learning model identified by the information notified from the information processing device 13.

In step S25, the CPU 191 of the cloud server 12 performs analysis processing such as predicting the yield of the target field and calculating the non-productive rate for the entire field from the detection area obtained as a result of the object detection processing in step S24. This calculation is performed taking into account both the production area rectangle detected from all captured images and the non-productive areas determined to be dead branch areas, diseased areas, etc.

Note that the learning model in this embodiment is a model learned by deep learning, but various object detection techniques such as rule-based detectors defined by various parameters, fuzzy inference, genetic algorithms, etc. may also be used as the learning model.

Second Embodiment
In the following embodiments, differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In the present embodiment, a system for performing visual inspection on a production line in a factory will be described as an example. The system according to the present embodiment detects abnormal areas in industrial products that are the objects of inspection.

Traditionally, in visual inspections on factory production lines, the imaging conditions of the inspection equipment (equipment that photographs and inspects the appearance of products) were carefully adjusted for each production line, and it was common to spend time adjusting the settings of the inspection equipment each time a production line was started up. However, in recent years, manufacturing sites are expected to respond immediately to diversifying customer needs and changes in the market. There is a growing need for speedy responses, such as setting up a line in a short period of time, even for small lots, to produce an amount that meets demand, and then immediately dismantling the line once sufficient supply has been met to prepare for the next production line.

In this case, if the settings for visual inspections are set up each time based on the experience and intuition of experts on the manufacturing site, as in the past, it will not be possible to respond to a rapid start-up. If similar products have been inspected in the past, the related setting parameters can be stored and those past setting parameters can be called up when a similar inspection is to be performed, allowing anyone to set up the inspection equipment without relying on the experience of experts.

As in the first embodiment, the above objective can be achieved by assigning an already-held learning model to the inspection target image of a new product. Therefore, the above information processing device 13 can also be applied to the second embodiment.

The setting process for the inspection device by the system according to this embodiment (setting process for visual inspection) will be described with reference to the flowchart in FIG. 8A. Note that the setting process for visual inspection is assumed to be performed when starting up an inspection step in the production line.

Multiple learning models (appearance inspection models/settings) for performing appearance inspections on captured images are registered in the external storage device 196 of the cloud server 12, and each learning model is a model that has been trained in a different learning environment.

Camera 10 is a camera for photographing a product that is the subject of visual inspection (product under inspection). As in the first embodiment, camera 10 may be a camera that photographs periodically or irregularly, or may be a camera that photographs moving images. In order to accurately detect abnormal areas in the product under inspection from the photographed images, when a product under inspection containing an abnormal area enters the inspection process, it is desirable to photograph the product under conditions that highlight the abnormal area as much as possible. Camera 10 may be a multi-camera if it photographs the product under inspection under multiple conditions.

In step S80, the camera 10 captures an image of the product to be inspected to generate a captured image of the product to be inspected. In step S81, the camera 10 transmits the captured image generated in step S80 to the cloud server 12 and the information processing device 13 via the communication network 11.

In step S82, the CPU 131 of the information processing device 13 acquires information about the product to be inspected photographed by the camera 10 (part name and material of the product to be inspected, manufacturing date, imaging system parameters at the time of photographing, lot number, temperature, humidity, etc.) as parameters of the product to be inspected. For example, the CPU 131 displays a GUI on the display device 14 and accepts input of the product to be inspected parameters from the user. When the user operates the user interface 15 to input a registration instruction, the CPU 131 of the information processing device 13 transmits the product to be inspected parameters of each of the above items input in the GUI to the cloud server 12. The CPU 191 of the cloud server 12 stores (registers) the product to be inspected parameters transmitted from the information processing device 13 in the external storage device 196.

In step S83, a process is performed to select a learning model to be used to detect the above-mentioned inspection target product from the captured image. Details of the process in step S83 will be described with reference to the flowchart in FIG. 8B.

In step S831, the CPU 191 of the cloud server 12 selects M candidate learning models (candidate learning models) from the E learning models stored in the external storage device 196. The CPU 191 generates query parameters from the product parameters to be inspected registered in the external storage device 196 in the same manner as in the first embodiment, and selects a learning model that has learned about an environment similar to the environment indicated by the query parameters (a learning model used in a similar past inspection).

If the query parameters include "circuit board" as the "part name," a learning model that was used in past circuit board inspections is more likely to be selected, and if the query parameters include "glass epoxy" as the "material," a learning model that was used in the inspection of glass epoxy circuit boards is more likely to be selected.

In step S831, as in the first embodiment, M candidate learning models are selected using the learning model parameter set and the query parameters, and in this case, the above formula (1) is used, as in the first embodiment.

Next, in step S832, the CPU 191 of the cloud server 12 selects P images from the captured images received from the camera 10 as images for model selection. For example, products that flow through the main inspection process of the main production line are randomly selected, and P images are obtained from the images captured by the camera 10 with the same settings as during actual operation. Since the number of defective products that occur on a production line is usually small, the processing in the subsequent steps does not function well if the number of products photographed in the process is small. Therefore, as a guideline, it is desirable to photograph several hundred or more products.

Next, in step S833, a process is performed to select one of the M candidate learning models as a selected learning model using the P captured images selected in step S832. Details of the process in step S833 will be described with reference to the flowchart in FIG. 8C.

In step S8330, the CPU 191 of the cloud server 12 performs "object detection processing, which is processing for detecting an object from each of the P captured images using the candidate learning model" for each of the M candidate learning models. In this embodiment, too, the result of the object detection processing for the captured images is position information of the image area (rectangular area, detection area) of the object detected from the captured image.

In step S8331, the CPU 191 obtains a score for the "result of the object detection process for each of the P captured images" for each of the M candidate learning models. The CPU 191 then ranks the M candidate learning models based on the scores (creates a ranking) and selects N candidate learning models from the M candidate learning models. The score for the result of the object detection process using the candidate learning models is obtained, for example, as follows.

For example, in a task to detect anomalies on a printed circuit board, object detection processing is performed on various specific local patterns on a fixed print pattern. Here, it is assumed that in a specific learning model, detection areas 901 to 906 as shown in FIG. 9(A) are obtained from a captured image of a normal product. Since the frequency of occurrence of abnormalities in products produced on a manufacturing line is extremely low, a good learning model for performing the above task is one that can output stable results for expected variations in captured images. For example, a slight change in the appearance of the captured image of the product due to environmental fluctuations on the area sensor side may make it impossible to detect detection area 906 out of detection areas 901 to 906 as shown in FIG. 9(B). In such a case, a penalty should be imposed on the evaluation score of a learning model in which the detection area changes in response to an input with only a slight difference.

Therefore, for example, the CPU 191 of the cloud server 12 determines a higher score for each of the M candidate learning models, the greater the difference in the arrangement pattern of the detection area by the candidate learning model between the P captured images. Such a score can be obtained, for example, by calculating the above formula (4). The M candidate learning models are then ranked in ascending order of score, and the top N candidate learning models are selected in descending order of score. When making this selection, a condition that "the score is less than a threshold value" may be added.

In step S8332, the CPU 131 of the information processing device 13 performs scoring (display image scoring) for the P captured images to present information that is easy for the user to subjectively compare, in the same manner as in the first embodiment (step S2332).

In step S8333, the CPU 131 of the information processing device 13 causes the display device 14 to display, for each of the N candidate learning models selected in step S8331, the top F captured images among the P captured images received from the cloud server 12 in descending order of score, and the results of the object detection process for the captured images received from the cloud server 12. At that time, the F captured images are displayed in descending order of score from the left.

Figure 10(A) shows an example of a GUI displaying the captured images and the results of the object detection process for each candidate learning model. Figure 10(A) shows the case where N = 3 and F = 4.

The top row displays the model name "M005" of the candidate learning model with the highest score along with a radio button 100, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area detected from the captured image by the candidate learning model with the model name "M005" is superimposed on the captured image.

In the middle row, the model name "M023" of the candidate learning model with the second highest score is displayed along with a radio button 100, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area detected from the captured image by the candidate learning model with the model name "M023" is superimposed on the captured image.

In the bottom row, the model name "M014" of the candidate learning model with the third highest score is displayed along with a radio button 100, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area detected from the captured image by the candidate learning model with model name "M014" is superimposed on the captured image.

In addition, in this GUI, images in the same row are displayed as if they were the same photograph, so that the results of the object detection process using each candidate learning model can be easily compared at a glance.

In this case, the difference in the arrangement pattern of the detection area is as shown in FIG. 10(A) because the product appearance is almost fixed and most of the products are normal. The F captured images are displayed from the left in descending order of score, and images with high scores tend to be those in which the shooting conditions at the time of individual shooting were significantly different or those containing abnormal areas. Therefore, compared to the conventional method in which the user performs annotation work for abnormal areas in the captured images of the product in advance and manually searches for defective products from a large number of products before setting up the inspection device, the user can preferentially present captured images of products that may contain abnormal areas by simply looking at this GUI without performing any of the above work, thereby saving labor. The user can compare the results of the object detection process in the GUI in FIG. 10(A) and select a learning model that can correctly detect abnormal areas.

Then, the user visually checks the differences in the results of the object detection process for the F captured images using the N candidate learning models, and selects one of the N candidate learning models using the user interface 15.

In step S8334, the CPU 131 of the information processing device 13 accepts a candidate learning model selection operation (user input) by the user. In step S8335, the CPU 131 of the information processing device 13 determines whether or not a candidate learning model selection operation (user input) has been performed by the user.

In the case of FIG. 10(A), when the user selects a candidate learning model with the model name "M005", the user selects the radio button 100 in the top row using the user interface 15. When the user selects a candidate learning model with the model name "M023", the user selects the radio button 100 in the middle row using the user interface 15. When the user selects a candidate learning model with the model name "M014", the user selects the radio button 100 in the bottom row using the user interface 15. In FIG. 10(A), because the radio button 100 corresponding to the model name "M005" is selected, a frame 104 is displayed indicating that the candidate learning model with the model name "M005" has been selected.

When the user operates the user interface 15 to select the decision button 101, the CPU 131 determines that "the user has performed a candidate learning model selection operation (user input)," and selects the candidate learning model corresponding to the selected radio button 100 as the selected learning model.

If the result of this determination is that the user has performed a selection operation (user input) of a candidate learning model, processing proceeds to step S8336; if the user has not performed a selection operation (user input) of a candidate learning model, processing proceeds to step S8334.

In step S8336, the CPU 131 of the information processing device 13 checks whether the "number of learning models desired by the user" has been selected. If the "number of learning models desired by the user" has been selected, the process proceeds to step S84, and if the "number of learning models desired by the user" has not been selected, the process proceeds to step S8332.

Here, the "number desired by the user" is determined mainly based on the time (takt time) that can be spent on visual inspection. For example, if the "number desired by the user" is two, it may be possible to detect a wider range of defects by changing the tendency of the detection target, such as using one learning model to detect low-frequency abnormal areas and the other learning model to detect high-frequency defects.

Here, if the user is unable to narrow down the number to "the number desired by the user" simply by looking at the display in FIG. 10(A), multiple candidate learning models may be selected by selecting multiple radio buttons 100. For example, if the "number desired by the user" is "1" and the number of selected radio buttons 100 is 2, N=2 and the process proceeds to step S8332 via step S8336. In this case, from step S8332 onwards, similar processing is performed for N=2 and F=4. In this way, the process is repeated until the number of learning models finally selected is "the number desired by the user."

The user may also select a learning model using the GUI of FIG. 10(B) instead of the GUI of FIG. 10(A). The GUI of FIG. 10(A) is a GUI that allows the user to directly select which learning model is better. In contrast, in the GUI of FIG. 10(B), a check box 102 is provided for each captured image, and the user operates the user interface 15 to specify and turn on (put a check mark on) the check box 102 of the captured image that is determined to have a good result of the object detection process among the captured images in the captured image sequence, for each captured image sequence that is lined up vertically. Then, when the user operates the user interface 15 to specify the decision button 1015, the CPU 131 of the information processing device 13 selects, as the selected learning model, the candidate learning model with the largest number of captured images with the check box 102 turned on, from among the candidate learning models with the model names "M005", "M023", and "M014". In the example of FIG. 10(B), two check boxes 102 are checked out of four captured images of a candidate learning model with a model name of "M005", one check box 102 is checked out of four captured images of a candidate learning model with a model name of "M023", and one check box 102 is checked out of four captured images of a candidate learning model with a model name of "M014". In this case, the candidate learning model with the model name "M005" is selected as the selected learning model. This method of selecting a selected learning model using a GUI is effective, for example, when the value of F increases and it is difficult for the user to determine which candidate learning model is best.

The easiest way to narrow down the learning models to the "number desired by the user" in the GUI of Figure 10 (B) is to select the "number desired by the user" from the top in order of the number of checked check boxes.

If there are candidate learning models with the same or a small difference in the "number of captured images with check boxes 102 turned on," it is determined in step S8336 that "the number of learning models selected is not the "number desired by the user" in the end," and the process proceeds to step S8332. Then, from step S8332 onward, processing is performed on candidate learning models with the same or a small difference in the "number of captured images with check boxes 102 turned on." Even in such a case, the process is repeated until the number of learning models finally selected is the "number desired by the user."

Then, regardless of the method used to select the selected learning model, the CPU 131 of the information processing device 13 notifies the cloud server 12 of information indicating the selected learning model (e.g., the model name of the selected learning model).

In step S84, the CPU 191 of the cloud server 12 performs object detection processing on the captured image (the captured image transmitted by the camera 10 to the cloud server 12 and the information processing device 13) using the selected learning model identified by the information notified from the information processing device 13. The CPU 191 of the cloud server 12 then performs final settings for the inspection device from the detection area obtained as a result of the object detection processing. Inspection is performed using the learning model and various parameters set here when the production line is actually launched.

Note that the learning model in this embodiment is a model learned by deep learning, but various object detection techniques such as rule-based detectors defined by various parameters, fuzzy inference, genetic algorithms, etc. may also be used as the learning model.

<Modification>
Each of the above embodiments is an example of a technique for reducing the cost of learning a learning model and adjusting the settings each time a detection/identification process is performed on a new target in a task that performs a target detection/identification process. Therefore, the technique described in each of the above embodiments is not limited to application to prediction of crop yields, detection of repair areas, detection of abnormal areas in industrial products that are inspected, etc. The target is applied to agriculture, industry, fisheries, and a wide range of other fields.

The above radio buttons and check boxes are displayed as an example of a selection section for the user to select a target, and other display items may be displayed instead as long as they have a similar effect. In the above embodiment, a configuration has been described in which a learning model to be used in the object detection process is selected based on a user operation (S24). However, this is not limited to this, and a configuration may be adopted in which a learning model to be used in the object detection process is automatically selected. For example, a configuration may be adopted in which a candidate learning model with the highest score is automatically selected as the learning model to be used in the object detection process.

Furthermore, the subject of each process in the above description is just an example. For example, some or all of the processes described as being performed by the CPU 191 of the cloud server 12 may be performed by the CPU 131 of the information processing device 13. Furthermore, some or all of the processes described as being performed by the CPU 131 of the information processing device 13 may be performed by the CPU 191 of the cloud server 12.

In the above explanation, the system of each embodiment is described as performing the analysis process. However, the subject of the analysis process is not limited to the system of each embodiment, and for example, the analysis process may be performed by another device/system.

In addition, the numerical values, processing timing, processing order, processing subject, data (information) structure/destination/source, etc. used in the above embodiments and variations are given as examples to provide a concrete explanation, and are not intended to be limiting.

Furthermore, any or all of the embodiments and variations described above may be used in appropriate combination.Furthermore, any or all of the embodiments and variations described above may be used selectively.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

10: Camera 11: Communication network 12: Cloud server 13: Information processing device 14: Display device 15: User interface 131: CPU 132: RAM 133: ROM 134: Output I/F 135: Input I/F 191: CPU 192: RAM 193: ROM 194: Operation unit 195: Display unit 196: External storage device 197: I/F 198: System bus

Claims (12)

a first selection means for selecting one or more learning models as candidate learning models from a plurality of learning models trained in different learning environments based on information related to photographing an object;
A second selection means for selecting one or more candidate learning models from the candidate learning models selected by the first selection means based on a result of the object detection process using the candidate learning models;
A detection means for performing an object detection process on a photographed image of the object by using at least one of the candidate learning models selected by the second selection means ;
A means for predicting a yield of a crop and detecting a repair area in a farm field based on an object detection area obtained as a result of the object detection process;
An information processing device comprising: The information processing device according to claim 1, characterized in that the first selection means generates a query parameter based on the information, and selects, from among the multiple learning models, one or more learning models that have been trained in an environment similar to the environment indicated by the query parameter as candidate learning models. The information processing device according to claim 1 or 2, characterized in that the second selection means obtains a score for each candidate learning model selected by the first selection means based on the result of the object detection process using the candidate learning model, and selects one or more candidate learning models from the candidate learning models selected by the first selection means based on the score. Furthermore,
4. The information processing apparatus according to claim 1, further comprising a display control means for displaying a result of the object detection process using the candidate learning model selected by the second selection means. The information processing device according to claim 4, characterized in that the display control means determines a score that is larger for each of the multiple captured images for which the candidate learning model selected by the second selection means has performed an object detection process, the larger the difference in the results of the object detection process between the candidate learning models, and displays, for each candidate learning model selected by the second selection means, a GUI including the results of the object detection process by the candidate learning model for a specified number of captured images, starting from the highest score. the GUI includes a selection portion for selecting a candidate learning model;
The information processing device according to claim 5, characterized in that the detection means sets a candidate learning model corresponding to a selection section selected in response to a user operation on the GUI as a selected learning model, and performs object detection processing using the selected learning model. The information processing device according to claim 5, characterized in that the detection means selects the candidate learning model that has the most object detection process results selected in response to a user operation from among the object detection process results displayed for each candidate learning model by the display control means as a selected learning model, and performs object detection processing using the selected learning model. 8. The information processing device according to claim 1 , wherein the information includes Exif information of the captured image, information relating to a field in which the captured image was captured, and information relating to objects included in the captured image. The information processing device according to any one of claims 1 to 7, characterized in that the information includes information related to an object included in the captured image. The information processing device according to any one of claims 1 to 9, characterized in that the detection means performs the object detection processing on the captured image of the object using a candidate learning model selected from the candidate learning models selected by the second selection means based on a user operation. An information processing method performed by an information processing device,
a first selection step in which a first selection means of the information processing device selects one or more learning models as candidate learning models from a plurality of learning models trained in different learning environments based on information related to photographing an object;
a second selection step in which a second selection means of the information processing device selects one or more candidate learning models from the candidate learning models selected in the first selection step based on a result of an object detection process using the candidate learning models;
a first detection step in which a first detection means of the information processing device performs an object detection process on a photographed image of the object by using at least one of the candidate learning models selected in the second selection step ;
a second detection step in which a second detection means of the information processing device predicts a yield of a crop and detects a repair area in the field based on a detection area of the object obtained as a result of the object detection process;
An information processing method comprising: A computer program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10 .

Images