IT202300015324A1 - IMAGE RECOGNITION SYSTEM AND METHOD USING ARTIFICIAL INTELLIGENCE AND APPLICATION OF REWARD CRITERIA - Google Patents

Description

?Image recognition system and method using artificial intelligence and applying reward criteria?

DESCRIPTION

FIELD OF TECHNOLOGY

The field of using artificial intelligence and image recognition to create cultural games, also known as cultural games, represents a fascinating fusion of technology and culture, offering an engaging and educational gaming experience. This discipline focuses on the application of advanced artificial intelligence techniques to analyze and recognize visual details within works of art, historical monuments, cultural sites, and cultural symbols.

In this field, image recognition technologies are leveraged to allow the game to "see" and "understand" the cultural world through detailed image analysis. Convolutional neural networks (CNNs) and other deep learning techniques are trained on vast datasets containing images of artworks, historical sites, and more, to recognize shapes, contours, colors, textures, and other specific visual features. This allows the game to identify objects, artistic styles, historical periods, and more, enabling the game to provide detailed and precise information about the culture represented.

STATE OF THE PRE-EXISTING TECHNOLOGY

In the field of image recognition and the use of artificial intelligence in cultural games, several advanced technologies are used to ensure accurate and engaging performance. Some of the key technologies used in this area include:

Convolutional Neural Networks (CNNs): CNNs are one of the fundamental technologies in image recognition. These specialized neural networks can automatically recognize and classify visual features in images, such as shapes, edges, textures, and colors, enabling detailed analysis of artworks and cultural sites.

Deep Learning: Deep learning is an area of artificial intelligence that uses deep neural networks to learn and identify complex patterns within images. This technology is particularly useful for recognizing details and creating immersive gaming experiences.

Image Feature Extraction: Image feature extraction technology allows us to identify and analyze specific visual attributes of artworks or cultural sites, such as geometric details, texture, color, and perspective. This allows for a deeper understanding of images and their unique characteristics.

Text-Based Search Engines: To transform image recognition results into useful, contextual information, text-based search engines are used, including those based on natural language processing (NLP) algorithms. These engines translate the recognized information into detailed and precise textual descriptions.

Cloud Computing: Using cloud services allows you to process large amounts of data and manage complex image recognition tasks efficiently. The scalability and flexibility offered by cloud services are particularly advantageous for cultural games, as they can support a growing number of users and cultural content.

Mobile Computing: The integration of image recognition into mobile devices allows users to use their smartphone or tablet cameras to explore and interact with the cultural world in real time. This offers a more immediate and immersive experience.

Augmented Reality (AR) Techniques: AR combines real-world elements with virtual ones, allowing users to interact directly with artworks or cultural sites through their mobile devices. This technology adds a level of immersion and realism to the enjoyment of art and culture.

Gamification Techniques: Gamification techniques transform cultural exploration into a playful and fun experience, encouraging user engagement and active participation.

Cultural Database Integration: Access to cultural databases containing detailed information on artworks, monuments, and historical sites is essential to enriching the Cultural Game experience and providing accurate and up-to-date information.

These are just some of the technologies used in image recognition in cultural games. The evolution of artificial intelligence and image analysis technologies will continue to improve and enrich the cultural discovery and learning experience through these engaging games.

SUMMARY OF THE INVENTION

The present invention relates to both single images of works of art and sequences of images or videos depicting pictorial works, where the context or content is determined automatically through analysis by a hardware system supported by a method integrated into a software.

The invention focuses on a system and method that uses automatic image recognition to create playful cultural experiences. The system's hardware analyzes the entire image, identifying and subdividing the individual characterizing pixels, initially using existing software-based image analysis methods. Subsequently, through subsequent steps, artificial intelligence systems, such as neural networks, are used to automatically recognize the image content through association with a textual knowledge base.

First, a matrix or network analysis is performed on several hierarchical classes of individual image element features, general image context problems, and automatic image and content recognition problems. Significant groups of grouped image elements (meta-elements) and geometrically related image objects (objects) increase exponentially, requiring a vast knowledge base to train an AI-implemented system. This knowledge base must be quickly accessible when using the automatic recognition system.

Once the image has been analyzed, an image detail (such as an object, a person, or an animal) is introduced. This detail, thanks to the analysis of the meta-elements present in the image (chromatic, dimensional, and structural), is harmoniously integrated into the image itself. The observer can have fun trying to identify the detail and possibly receive a reward once identified and reported (for example, with a tap on a smartphone).

According to the invention, an advantageous image recognition system is obtained thanks to the method described in the preceding claim.

The next steps in the process involve analyzing an image or video through a combination of image feature analysis and text analysis. Individual elements are modeled based on basic shapes, textures, colors, natural or artificial, material, and other characteristics. A separate textual knowledge base is used to verify the elements' characteristics and determine which objects they belong to or whether they are independent objects.

The context between individual elements, meta-elements, and the overall image is analyzed, thus improving recognition accuracy. Comprehensive image analysis can be applied to a wide range of scenarios or areas of life.

The detailed description and practical example illustrate the stages of the process, as in the case of the "De Chirico" example illustrated in the digital image in Figure 1.

DESCRIPTION OF FIGURES

- Figure 1A Hardware device underlying the present system and method

- Figure 1B highlights the total image to be analyzed - Figure 1C highlights the comparison between the analyzed image and the images present in an image database to identify the artwork and its author: e.g. De Chirico ?Enigma of a Day? - Figure 2 The selection of image elements from FIG. 1B then leads to separate image components, shown here.

- Figure 3 Now all image components are further processed in parallel, and even pixels that occur multiple times are considered in parallel with their exact number of occurrences in the overall image.

- Figure 4 All individual elements of the overall image, which are now considered in parallel, as well as simultaneously the overall image containing all these individual elements, are then subjected to further analysis of the image characteristics and provided with more precisely descriptive textual attributes.

- Figure 5 Here, the parallel analysis and evaluation of the overall image is illustrated, taking into account individual image elements, which have already been characterized and classified more carefully in textual terms, with the example of deciding whether there is a perspective in the overall image.

- Figure 6 Here one of the individual elements of the image in Fig. 4 is highlighted again, namely the mostly white surface of the ground area of the overall image, which is again analyzed pixel by pixel.

- Figure 7 Here another of the individual image elements from Fig. 4 is selected again, namely the large, dominant arc in the overall image, which is again analyzed pixel by pixel as an example.

- Figure 8 Here the next single pixel from Rg is taken.

4, or a less complex "smaller" element, namely the arched window, which appears several times in the overall image.

- Figure 9. All individual image elements recognized as similar in this figure are listed here. According to the invention, all these individual image elements recognized by the machine system up to this point should now be combined as a so-called meta-element into a common higher class of identified image objects and provided with one or more descriptive textual terms that are initially considered by the machine as working hypotheses. These working hypotheses are then successively compared with the overall image as well as with all separate meta-elements with the aim of ranking the recognized objects in order of importance based on the least degree of contradiction. Here, classifications such as Castle, College, Palace, Church could (and should) be made using such comparisons.

- Figure 10 In this figure the insertion of a detail is highlighted (in the example a star) which is well camouflaged thanks to the analysis of the metaelements contained in the image in fig. 1B

- Figure 11 This figure highlights the possibility of clicking on the added detail once it has been identified. In the drawing, the example hypothesis is identification via a tap on a touchscreen.

DETAILED DESCRIPTION OF THE INVENTION

2.1 First phase: Given a hardware system, a device as shown in fig. 1A, the system will use image recognition to detect the distinctive elements of an image representing a work of art and will deduce the author and title by comparing them with images of works of art contained in a database located in the memory of the system shown in fig. 1A.

Next, the essential individual elements of the image are extracted. The elements are selected based on shape, edges, true color areas and gradients, contrasts, and changes in image parameters (such as resolution, color, contrast, brightness, etc.). Existing methods are used at this stage.

Examples: Fig. 1B shows the white surface of the terrain with its irregular structure resulting from the color surface analysis, as well as the facade with its two-tone cladding (it could also be composed of the light and ochre modules). The two targets can be selected as elements with clear contours.

This leads (in a simplified way) for example to the elements shown as an example in Figure 2: Selecting the image elements from Figure 1B then leads to separate individual image components, which are shown here in Figure 2.

As shown in Figure 1C, the artwork image is compared with images in a database to deduce the title and author of the artwork.

2.2 Second Phase: identification of the characteristics

First, we extract the essential aspects of the image. Elements are selected based on their shape, contours, true color areas, gradients, contrasts, and variations in image parameters such as resolution, hue, brightness, etc. At this stage, existing techniques are used.

As a result, we obtain the elements shown as an example in Figure 2, which constitute the essential contents extracted from the main image.

Next, we move on to the feature determination phase. Here, the entire image and all elements are analyzed simultaneously to identify specific features (Figure 3). Elements are analyzed individually, also taking into account their repetitions in the image and the exact number of occurrences in the entire image.

At this stage, a possible sequence of analysis steps for both the elements and the overall image could be the following:

Natural or artificial?

Present perspective?

Shape/Structure

Color, gradient

Surface, texture

... other significant features

During this process, deterministic-analytical image analysis methods can be combined with trained AI systems like Apollo, which can distinguish, for example, whether an object is of natural or artificial origin, whether the image has a perspective and therefore a horizon, allowing for the identification of a terrain area. Distinctions are also made regarding geometric shapes and structures, symmetries, curves, arcs, circles, polygons, hues, shading, brightness, color gradients, textures, etc. Any extensions beyond color and gradient are taken into account and whether the object is above or below the horizon line, whether it has a regular or irregular shape (FFT analysis), and whether it has regular or irregular textures, identifying the respective areas.

In steps 1 and 2, a binary (yes/no) assessment is obtained. In steps 3, 4, and 5, verbally described results are provided.

Examples:

a) Natural or artificial? The analysis is performed on both the overall image and each element. Four elements are identified with positive results: the three people and the snow. Regarding the overall image, a unequivocal answer is not currently possible. However, this distinction can be learned by the system through training (Apollo), allowing the software to autonomously learn the features relevant for the distinction through training games.

b) Perspective yes/no? This characteristic is determined in the overall image, for example through alignment lines (see Figure 5). The following conclusion is reached:

Perspective is present, and a horizon dividing the image is identified. This result will be evaluated later (for example, by defining the lower part as the ground). Perspective is not recognizable in the elements (except for the second arch, where it may be difficult to identify due to the small size and image quality).

2.3 Third Phase: Meta-element Detection

In this phase, the recognized features are processed, specifically determining the relationships between elements and the organization of elements within the context of the overall image. The textual outputs from phases 3, 4, and 5 for the elements are fed into an associative textual system with a different "taxonomy," processed in a textual knowledge base, thus providing meaningful elements. A) Let's return to the arched windows (see Figure 8): Here in Figure 8, the next single pixel from Figure 4 is now removed, and it is a less complex "smaller" one, namely the arched window, which the system has yet to recognize as such; so far (yes) it only knew it based on its specific geometric shape and size (still as in Figure 7, a rectangle with a semicircle above it, but this time smaller), color and brightness (dark), characteristic texture (screen), symmetry (asymmetric), which is a single image element that appears multiple times in the overall image.

Result of the previous analysis (based on basic knowledge of the structures...):

Symmetrical, half-arched above, grid-like, dark. Now, contextual analysis of both the elements and their relationship to the overall image and their position within the overall image: Comparison with other elements, determining the frequency of the most similar elements. Result: At least four very similar objects appear in the image. b) Verification of the regularity of the arrangement. Result: structured, regular, arranged in two rows one above the other, forming three axes each, separated by regular vertical elements.

c) Checking for further objects within the regular structure leads to two more very similar figures in the middle of the image (central window and small arched window) and two less similar ones (lower windows on the left and right, see Figure 9 below).

Here, in Figure 9, all individual image elements recognized as similar in this figure are listed, which were recognized as similar because they all have the same symmetry (axis), have approximately the same shape and texture, while the size of the individual image elements was not initially used to assess the degree of similarity.

These window elements are then combined as a so-called meta-element into a common higher class of identified image objects and assigned one or more descriptive textual terms. To determine this characterizing generic term (here, of course, "window" should be included among other things), all previously determined attributes of all similarly identified pixels are then entered as "search terms" or "keywords" into an associative text search engine. In this case, the "keywords" for the (textual) search engine would be approximately: square, half-arch, grid, dark, two rows, three axes (of symmetry), etc. Now, objects automatically associated with the textual knowledge base that closest match the "sum" of these pixel attributes or that can be most meaningfully associated with these attributes are searched on-site. This degree of similarity or significance of the correlation between attributes and the found objects, as the first working hypothesis of the text search/object suggestion engine, is now further examined so that a ranking of the object suggestions initially hypothesized by the search engine can be determined. To this end, two statistical methods, particularly those based on averaging, are used: first, to control the largest possible number and highest possible quality of matches, and, second, to control the smallest possible number and lowest possible significance of contradictions.

Accordingly, the text search engine might return, for example, the following (superordinate) terms: Painting, Italian Painter, Square, Statue, Day, De Chirico, Tower, etc. Furthermore, this search engine, based on an associative textual knowledge base classified in a hierarchical taxonomy, will systematize the found terms, particularly with the help of a KL system such as a neural network, for example, into hierarchically structured levels of abstraction, such as actual size, and/or even after detailed integration into a larger, superordinate structure. For example: Window Tower > Tower Types (Watchtower, Bell Tower, Water Cistern). All these tentatively recognized meta-elements are now hypotheses for the image content or for components of the overall image content. These working hypotheses are then subsequently compared with the overall image as well as with all meta-elements, with the aim of sorting the objects according to the least degree of contradiction in a relevance ranking. Here they could (and should) be classified by such comparisons, as for example Tower ?Bell Tower? Water Cistern ?. Now follows an analysis of the color areas (Figure 4):

In the next step, for example, the color surfaces were reanalyzed and also included in these comparison operations to minimize contradictions, further clarifying the meaning of the ranking of hypotheses relevant to the recognized image content. For example, the attributes found for the image region identified as the lower area (white, partially smooth, partially irregularly structured, irregularly delimited) are again entered into the taxonomies, and these hierarchically associatively classified textual knowledge bases (search engines) could then with a high probability yield meaningful correlations to objects such as marble, snow, carpet, which in turn are hypotheses, this time for the lower part of the image. These hypotheses for the ground are now compared again with the overall image and all meta-elements, with the aim of ranking the objects according to the least contradictory or greatest similarity. Due to the previously recognized natural ground and the irregular surface structure, and due to the snowfall in the overall image, which can be traced back to the external environment, there is a strong possibility of a correlation. This contradicts the "carpet" hypothesis. The lack of structure and the irregular or demarcated surface contradict the "marble" hypothesis. This is confirmed by the irregularity of the ground area for a snow surface and the upper test for snowfall in the image (small white lines across the entire image, or at least a large part of it, suggest snowflakes, especially in front of the dark areas in the half of the image above the horizon). This leads to a classification like this: snow "marble" carpet. Optionally, during the training phase of the KL system, manual intervention can be provided in the selection of such selectively selected image sections. c) group of people (to be performed)

Keywords: silhouette and head shape suggest people. No recognizable faces from behind. d) Close-up: arch (to be continued)

Keywords: From the established perspective, it can be deduced that it is a sort of tunnel with recognizable structures on the ceiling (which become clearer in brighter images), a very bright part at the opening...

The sequence of analysis phases is simplified. It is subject to feedback and therefore variable and controlled by the system itself. In general, each hypothesis can be compared with the intermediate results of other phases, resulting in a matrix or network-type procedure. This procedure is essentially analogous to a video. Let's add the analysis of changes over time.

2.4 Fourth phase: use of collected image metadata to create cultural games with reward criteria?

After determining the metadata for each element of the image, the system and method described so far involves inserting a further detail (e.g., a shoe, man, animal, thing, etc.) into the overall image. This insertion will take place in such a way that the detail can blend in as much as possible within the overall image. This is to provoke a desire for discovery in the user/player of the cultural game. This insertion, as described in figure 11, is the element that, once found and followed by an identification process (for example, a tap on a touchscreen), will give rise to the reward experience.

In the context of cultural games, analyzing metadata through image recognition opens up several opportunities to engage players in creative and educational ways. Cultural games can leverage the information extracted from the analysis to provide clues or hints to players, encouraging them to explore and discover hidden cultural aspects in the image.

For example, a game might require players to identify recognizable historical objects or monuments within an image and provide detailed information about them. Other cultural games might challenge participants to solve puzzles based on the features identified in the image, stimulating their cultural knowledge.

Furthermore, metadata analysis can be used to create dynamically generated game levels, ensuring unique and different gameplay experiences with every playthrough. This approach promotes the longevity and repeatability of cultural games, making them engaging and captivating for a wide and diverse audience.

Overall, the use of metadata analysis via image recognition in cultural games enables the creation of engaging, educational, and fun experiences, fostering learning and discovery of cultural heritage in an interactive and engaging way.

This system and method for applying Artificial Intelligence and Image Recognition can be applied in the industrial sector, especially for the creation of digital games aimed at learning and education. Furthermore, the application of reward criteria offers the opportunity to increase the engagement of those to whom the digital game is aimed.

Claims (6)

CLAIM'S 1. Image recognition system and method, through the following steps: - Image acquisition and comparison with a pre-existing database for image metadata. - Subdivision and classification into details or image elements via database and image recognition analysis for geometric shapes, textures, colors, and perspective. - Recognition of element features through artificial intelligence (neural network) and assignment of descriptive names. - Feeding the names into a textual knowledge base, with further analysis of the relationships between the elements and the image using a text search engine (neural network) to determine the context of the image. - Inserting details retrieved from a pre-existing image database into the image. This insertion, thanks to the use of artificial intelligence, tends to camouflage the detail in the source image in order to facilitate the search for the same detail within the source image. 2. A method according to claim 1, wherein the recognition accuracy is increased through repeated analysis and feedback: - of different sections of the image - of the entire image where correspondences are maximized and/or contradictions are minimized between those initially considered by the image recognition method as a working hypothesis, the image elements or image objects that characterize partially or completely with descriptive textual words and/or generic terms between them, especially within the same phase of the pixel and/or image object hierarchies and between these hierarchies on the upper part, so that there is a matrix and/or network type approach. 3. A method according to claims 1-2, wherein substantially detailed image elements are extracted on the basis of an analysis of the characteristics of the pixels, in particular with regard to their geometric shape, edges, color areas and gradients, contrasts, textures, degree of resolution, brightness, black and white, perspective and the like, taking into account existing deterministic analytical software engineering methods of image processing and analysis, in particular mathematical and numerical methods such as the threshold value method, gradient and extremal value determinations with the aid of the Hessian matrix for determining the structural features, in particular the image elements to be extracted, edge sampling, bubble analysis, Fourier methods for determining regularities and roughnesses. and average particle sizes in textures, cross-correlation method regarding rotation and translation of image elements, image objects, image parts or even the overall image for determining symmetries and/or periodicities, color distribution histograms and the like. 4. Method according to claim 3, wherein all determined image elements are further processed in parallel to determine further characteristics of the image element and/or image object, such as natural/artificial question, perspective yes/no, shape, structure, material, color, gradient, surface, texture or the like, this second stage of the image feature analysis of the image elements is already exposed in text form, which in particular is already used by a trained Kl system. 5. A process according to claims 1-4, whereby image elements described in more detail in text form are fed into a reverse taxonomic associative text system, i.e., a text-based search engine, to determine the relationships of the image elements to one another and the organization and classification of these image elements within the image context, whereby the text-based image feature processing of the pixels in a textual knowledge base provides these pixels with textually meaningful content, the image elements are then subdivided and classified into higher-level groups of meta-elements and provided with a textual generic term that the computer vision system according to the invention initially uses as a working hypothesis for in further steps to specify the accuracy of the respective image content. 6. A process according to claims 1-5, wherein all the most descriptive generic terms of the recognized/identified meta-elements and/or meta-objects are in turn entered into the associative textual knowledge base, in particular a neural network-based text search engine, in particular as "search terms" and/or "keywords", in order to determine a textual characterization of the overall image content, wherein each guess for the generic term characterizing a meta-element and/or meta-object is compared with intermediate results from other steps concerning other recognized or yet to be recognized meta-elements and/or meta-objects, and then a matrix-type structure is generated.