CN113614700A

CN113614700A - Image display monitoring method, device and equipment

Info

Publication number: CN113614700A
Application number: CN202080001518.0A
Authority: CN
Inventors: 李华; 薛林; 文长春
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-03-03
Filing date: 2020-03-03
Publication date: 2021-11-05
Anticipated expiration: 2040-03-03
Also published as: WO2021174407A1; CN113614700B

Abstract

The application discloses a method, a device and equipment for monitoring image display, wherein the image display monitoring equipment can comprise a first display controller and a neural network processor, wherein the first display controller can generate a first image by utilizing first data, the first image is used for displaying on display equipment, and actually, the first image has the problem of generating errors possibly, so that the first image cannot correctly embody the characteristics of the first data. Therefore, in the embodiment of the present application, at least a part of the first image may be obtained by using the neural network processor, and whether at least a part of the first image is correctly displayed is determined by using the neural network model, so as to obtain the monitoring result of at least a part of the first image.

Description

Image display monitoring method, device and equipment

Technical Field

The present application relates to the field of image display technologies, and in particular, to a method, an apparatus, and a device for monitoring image display.

Background

At present, an image to be displayed may be generated by a display controller, and the display device displays the image, so that a user may acquire necessary information. However, the image to be displayed generated by the display controller may have an error, so that the display apparatus displays the wrong image, causing the user to acquire the wrong information.

Therefore, it is necessary to monitor the image to be displayed generated by the display controller. Currently, a monitoring chip is connected between the display controller and the display device, and the monitoring chip is a hardware logic calculation unit, and may perform operations on pixel data, and may perform weighted sum operation or average calculation on red-green-blue (RGB) values of all pixels in a specific region of an image frame, or calculate a Cyclic Redundancy Check (CRC) value of a pixel data set, or calculate a contrast between a pattern lighting part and a background part in a region, for example, an RGB difference value. In this way, when the comparison result of the calculated value and the preset threshold value does not satisfy the preset condition, it may be considered that the image to be displayed generated by the display controller is incorrect.

However, in this way, once the functions of the hardware circuit are deployed, the functions cannot be manually changed, and a new monitoring algorithm cannot be manually added, so that the method is not flexible enough, the characteristics capable of monitoring are not comprehensive enough, and the monitoring accuracy is not high.

Disclosure of Invention

The technical problem to be solved by the application is to provide an image display monitoring method, device and equipment, which can monitor a generated first image by using a neural network model, and improve the flexibility and accuracy of image display.

In a first aspect, an embodiment of the present application provides an image display monitoring apparatus, including: a first display controller and a neural network processor; the first display controller is used for generating a first image by using the first data, the generated first image is used for displaying on the display device, and actually, the first image may have the problem of generating errors, so that the first image cannot correctly embody the characteristics of the first data. Therefore, in the embodiment of the present application, the neural network processor is configured to obtain at least a portion of the first image, and determine whether the at least a portion of the first image is displayed correctly by using the neural network model to obtain the monitoring result of the at least a portion of the first image.

In some possible embodiments, the neural network processor is further configured to: when the monitoring result is that at least one part of the first image is displayed incorrectly, sending an image generation instruction to a second display controller; the second display controller is to generate a second image to be displayed on the display device in place of the first image in response to the image generation instruction.

In the embodiment of the application, when the neural network processor judges that at least one part of the first image is displayed incorrectly, the second display controller can be used for generating the second image, so that the second image can be used for replacing the display of the first image on the display equipment, and on the premise of not displaying errors, the correct image can be displayed, thereby further improving the display correctness and improving the user experience and the system safety.

In some possible embodiments, the second display controller has a higher safety integrity level than the first display controller.

In the embodiment of the application, the second image can be generated by using the second display controller, so that the second image is displayed on the display device in place of the first image, and therefore, the safety integrity level of the second display controller can be higher than that of the first display controller, so that the accuracy of the second image generated by the second display controller is higher than that of the first image, and the accuracy of image display can be improved.

In some possible embodiments, the first display controller comprises at least one of a first image processor GPU or a first display subsystem DSS, and the second display controller comprises at least one of a second image processor GPU, a second display subsystem DSS, or a micro control unit MCU.

In the embodiment of the application, the first display controller and the second display controller may both include a GPU and/or a DSS, so that the generation process of the first image and the second image is more complete, and the generated images can better meet the requirements of users.

In some possible embodiments, at least a portion of the first image comprises an indicator icon; the apparatus further comprises: and the MCU is used for determining whether the indication icon is an indication icon needing to be displayed or not when the monitoring result is that at least one part of the first image is displayed correctly.

In the embodiment of the present application, the correct display of the indication icon is important for the functional safety of the system, so that at least a part of the first image may be the indication icon, and the monitoring of the indication icon is beneficial to improve the functional safety of the system. Therefore, when the monitoring result is that at least one part of the first image is displayed correctly, whether the correctly displayed indication icon is the indication icon needing to be displayed or not can be further judged through the MCU, and the accuracy of image display is further improved.

In some possible embodiments, at least a portion of the first image comprises an indicator icon; the neural network processor is further configured to: and when the monitoring result is that at least one part of the first image is displayed correctly, judging whether the indication icon is an indication icon needing to be displayed or not by utilizing a neural network model and indication information from the MCU.

In the embodiment of the application, when the monitoring result is that at least one part of the first image is correctly displayed, the neural network processor can judge whether the indication icon is the indication icon needing to be displayed by using the neural network model and the indication information from the MCU, so that the accuracy of image display is further improved.

In some possible embodiments, the image display monitoring apparatus further includes: a main processor for generating the first data; the first display controller is specifically configured to acquire the first data from the main processor.

In the embodiment of the application, the first display controller may acquire the first data from the main processor, so that the first image may be generated based on the data acquired from the main processor, so as to display information that the main processor needs to display, thereby improving user experience.

In some possible embodiments, the main processor is further configured to: and acquiring an instruction from the MCU, and responding to the instruction to generate the first data.

In the embodiment of the present application, the main processor may generate the first data based on the instruction acquired from the MCU, so that the reliability of the first data may be improved.

The embodiment of the application also provides an image display monitoring method, which comprises the following steps: generating, by a first display controller, a first image using first data, the first image for display on a display device; acquiring at least a portion of a first image; and judging whether at least one part of the first image is displayed correctly by utilizing a neural network model so as to obtain a monitoring result of at least one part of the first image.

In some possible embodiments, the method further comprises: when the monitoring result is that at least one part of the first image is displayed incorrectly, sending an image generation instruction to a second display controller; generating, by the second display controller in response to the image generation instruction, a second image for display on the display device in place of the first image.

In some possible embodiments, at least a portion of the first image includes an indicator icon, the method further comprising:

when the monitoring result is that at least one part of the first image is displayed correctly, determining whether the indication icon is an indication icon needing to be displayed.

An embodiment of the present application further provides an image display monitoring apparatus, including: a display control module to generate a first image with first data by a first display controller, the first image for display on a display device; a processing module for acquiring at least a portion of a first image; and judging whether at least one part of the first image is displayed correctly by utilizing a neural network model so as to obtain a monitoring result of at least one part of the first image.

In some possible embodiments, the processing module is further configured to: when the monitoring result is that at least one part of the first image is displayed incorrectly, sending an image generation instruction to a second display controller; the display control module is further configured to: generating, by the second display controller in response to the image generation instruction, a second image for display on the display device in place of the first image.

In some possible embodiments, at least a portion of the first image includes an indicator icon, then the processing module is further configured to: when the monitoring result is that at least one part of the first image is displayed correctly, determining whether the indication icon is an indication icon needing to be displayed.

An embodiment of the present application further provides a neural network processor, including: a processor and a memory; the memory for storing computer programs or instructions; the processor is configured to execute the computer program or the instructions in the memory to implement the method for monitoring image display provided in the embodiment of the present application.

Embodiments of the present application further provide a computer-readable storage medium, which includes a computer program or instructions, and when the computer program or instructions runs on a computer, the computer is enabled to implement the method for monitoring image display provided by the embodiments of the present application.

The embodiment of the application provides a method, a device and equipment for monitoring image display, wherein the image display monitoring equipment can comprise a first display controller and a neural network processor, wherein the first display controller can generate a first image by using first data, the first image is used for displaying on display equipment, and actually, the first image may have the problem of generation error, so that the first image cannot correctly embody the characteristics of the first data. Therefore, in the embodiment of the present application, at least a part of the first image may be obtained by using the neural network processor, and the neural network model is used to determine whether at least a part of the first image is displayed correctly, so as to obtain the monitoring result of at least a part of the first image.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art according to the drawings.

FIG. 1 is a schematic illustration of an indicator icon during monitoring of an electrical and electronic device in the automotive field;

fig. 2 is a schematic diagram of a system framework of image monitoring according to an embodiment of the present application;

FIG. 3 is a schematic diagram of another system for displaying images provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of another image monitoring system provided in an embodiment of the present application;

fig. 5 is a flowchart of an image monitoring method according to an embodiment of the present application;

fig. 6 is a block diagram of an apparatus for monitoring image display according to an embodiment of the present disclosure.

Detailed Description

The embodiment of the application provides an image display monitoring method, device and equipment, which can monitor a generated first image by using a neural network model, and improve the flexibility and correctness of image display.

Before describing the method provided by the embodiments of the present application, the following description is made. The terms "first," "second," "third" or "fourth," etc., in the description and claims of this application and in the above-described figures, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

"at least one" means one or more, "a plurality" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone, wherein A and B can be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of the singular or plural items. For example, at least one (one) of a, b, and c, may represent: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, wherein a, b and c can be single or multiple.

In the embodiment of the invention, the image to be displayed can be generated based on the display data through the display controller, and then the display device displays the image, so that the user can acquire the required information through the displayed image. For example, in the industrial field, the automotive field, the medical field, and the like, in order to improve functional safety (functional safety) of the system, the electronic and electrical equipment is often required to have a series of characteristics such as real-time self-detection, post-fault-detection alarm, post-fault-detection processing, and the like, so as to ensure that a warning effect can be provided for a user when a fault occurs, or certain functions and operability can be provided even when a part of circuits have faults, thereby avoiding injury to personal safety. Among them, the functional safety design of industrial control electrical and electronic equipment is described in IEC 61508 standard, and the functional safety design of electrical and electronic equipment in the automotive field is described in ISO 26262 standard. Taking an Automobile Safety Integrity Level (ASIL) as an example, ASIL refers to a level of functional safety requirements on a module system obtained by a module supplier and an integrator according to a risk and hazard analysis process, and may be divided into 5 levels, namely QM, a, B, C and D, from the level QM to ASIL-D, the system is required to have higher and higher fault self-detection and self-processing capabilities.

The method for improving the system function safety includes that a warning can be given after the system is abnormal, specifically, an indication icon can be generated according to abnormal information, the indication icon is displayed, and a user obtains specific content of the abnormal information through the indication icon. Specifically, in the monitoring process of the electric power control center on the electronic and electrical equipment in the industrial control scene, or in the monitoring process of the electronic and electrical equipment in the automobile field, or in the monitoring process of the electronic and electrical equipment in the medical field, if the working state of the monitored equipment is abnormal, an indication icon can be generated according to the abnormal information, and the indication icon is displayed, so that a user can timely acquire the abnormal information of the monitored equipment, corresponding operation is performed according to the abnormal information, and the functional safety of the system is improved.

In the field of automobiles, abnormal information of automobiles is indicated through an indication icon, including fault display when the automobiles cannot be started or the whole automobiles have faults, and the corresponding ASIL is usually in a B-level mode. Referring to fig. 1, a schematic diagram of an indication icon in a monitoring process of an electronic and electrical device in the automotive field is shown, and the indication icon may include: (1) the automatic control system comprises a constant-speed cruise control indicator lamp, (2) a power indicator lamp, (3) an anti-lock braking system (ABS) indicator lamp, (4) an economic mode indicator lamp, (5) a parking indicator lamp, (6) an engine self-checking lamp, (7) a high-beam auxiliary function indicator lamp, (8) a coolant temperature indicator lamp, (9) a Tire Pressure Monitoring System (TPMS) indicator lamp, (10) a reversing radar warning indicator lamp, (11) a high-beam indicator lamp and (12) a fuel indicator lamp.

The on-off states of the indication icons can reflect whether the vehicle has a fault corresponding to the indication icon, and it should be noted that when the indication icon is not lighted, the indication icon is not considered to exist in the image to be displayed. For example, the battery indicator lamp is an indicator lamp for displaying the working state of the storage battery, and if the generator or the circuit fails, the battery indicator lamp is not on or is not on for a long time, so that a driver can acquire the state of the generator or the circuit by judging whether the displayed image comprises the battery indicator lamp or not and whether the battery indicator lamp flickers or not.

That is to say, in the industrial field, the automotive field, the medical field, and the like, whether an image to be displayed generated by a display controller is correct, that is, whether an indication icon in the image to be displayed is displayed correctly, directly determines whether a user can acquire abnormal information of a vehicle through the displayed image, so that the accuracy of image display is particularly important, and especially in the case of relating to personal safety, if the image to be displayed is incorrect, the user cannot timely know the fault condition of a monitored device, so as to perform reasonable operation, and the error may pose a certain threat to the human body, and the functional safety of the system is also reduced accordingly. For example, when the generator of the vehicle is failed, the abnormal information of the vehicle is the generator failure, however, an icon representing the generator abnormality is not normally displayed in the image to be displayed, for example, the icon is blocked or displayed incompletely, and the driver may not accurately obtain the information of the generator failure based on the incorrectly displayed icon, which is dangerous for the driver.

The solutions described in the background art are not flexible enough and the monitoring characteristics are not comprehensive enough, resulting in limited monitoring accuracy. Particularly, with the advance of automobile electronization, intellectualization, networking and home furnishing, the instrument panel of some new concept vehicle types has evolved from the scheme of controlling the alarm indicator lamp by the traditional motor control pointer and level signal to the scheme of ' one screen ', at this time, all the information of the pointer and the alarm lamp ' is displayed by the liquid crystal display screen, and simultaneously, on the liquid crystal display screen, the gorgeous background and atmosphere light, real-time navigation information, real-time media playing information, real-time weather and other network information can be displayed, the image of the night vision camera for enhancing vision and the like can be displayed, the image monitoring by utilizing the hardware circuit is interfered to a certain extent, the accuracy is further reduced, and the actual requirements can not be met.

Based on the foregoing technical problem, embodiments of the present application provide a method, an apparatus, and a device for monitoring image display, where the image display monitoring device may include a first display controller and a neural network processor, where the first display controller may generate a first image using first data, and the first image is used for displaying on a display device, and actually, the first image may have a problem of generation error, so that the first image may not correctly embody a feature of the first data. Therefore, in the embodiment of the present application, at least a part of the first image may be obtained by using the neural network processor, and the neural network model is used to determine whether at least a part of the first image is displayed correctly, so as to obtain the monitoring result of at least a part of the first image.

Referring to fig. 2, a schematic diagram of a system framework for image monitoring provided by an embodiment of the present application may include an image display monitoring apparatus and a display apparatus 200. Among them, the image display monitoring apparatus may include a first display controller 101 and a neural network processor 102.

The first display controller 101 may include a display subsystem (DSS) or a display driver, and may be a high-end media processor having a function of performing complex drawing. The first display controller 101 may further include a Graphics Processing Unit (GPU). Of course, the first display controller 101 may also include both a DSS and a GPU, for example, the first display controller 101 may be a GPU with DSS embedded therein, or may be an integrated component including both a GPU and a DSS.

Among other things, GPUs may be used for rendering and drawing of images. The DSS may be configured to perform layer overlay processing, and send an image formed after the layer overlay to the display device 200 for display, and optionally, the DSS may also be configured to perform processing such as image flipping, image method, or image reduction, which is not limited in this embodiment of the present application. The layer overlay process includes, but is not limited to, overlaying the image rendered by the GPU with other images, such as with a background image or a window. The background image may include gorgeous background and atmosphere light, real-time navigation information, real-time media playing information, real-time weather and other network information, and a picture of the enhanced-vision night vision camera, and these contents may be used as the background image of the current indication icon to enhance the visual effect of the user or to expand the display content of the display device to meet the user's needs.

In this embodiment of the application, the first display controller 101 may generate the first image based on the first data, specifically, when the first display controller 101 generates the first image, the first image may be formed by adopting a layered drawing and a graph superposition manner, for example, the plurality of indication icons may be located in different layers, the first display controller 101 may perform a layered drawing on the plurality of indication icons and the background image located in a different layer from the indication icons, obtain a multilayer graph, and generate the first image by graph superposition.

Since the display of the first image is of great significance to improve the functional safety of the system, functional safety enhancement designs may be made for the first display controller 101 to improve the functional safety of the first display controller 101, including but not limited to: setting a dual-core backup, enabling a memory connected with the first display controller 101 to support verification codes for error detection and error correction, setting a signal detection strategy for the first display controller 101, performing self-detection by the first display controller 101, and providing hardware logic guarantee for the first display controller 101 to power down the first display controller 101 or to light up a reset device indication icon by default, and the like.

The first data used for generating the first image is data indicating image features in the first image, and may be, for example, a detection result generated based on detection data obtained by detecting a device to be detected, and the detection result may include an abnormal result and/or a normal result, where in the vehicle field, the device to be detected may be an electronic and electrical device on an automobile, such as an engine, a tire, or a reverse radar, and in the industrial control field or the medical field, the device to be detected may be other electronic and electrical devices, which are not illustrated herein. When the first image includes the indication icon, the first data includes data indicating status information of the indication icon in the first image, and the status information of the indication icon may include on or off, generally, if the first data indicates that the status of the indication icon in the first image is on, the first image generated based on the first data includes the illuminated indication icon, and if the first data indicates that the status of the indication icon in the first image is off, the first image generated based on the first data does not include the illuminated indication icon.

Referring to fig. 1, the indication icon in the monitoring process of the electric and electronic device in the automobile field may include: (1) the device comprises a constant-speed cruise control indicator lamp, (2) a power indicator lamp, (3) an ABS indicator lamp, (4) an economic mode indicator lamp, (5) a parking indicator lamp, (6) an engine self-checking lamp, (7) a high beam auxiliary function indicator lamp, (8) a coolant temperature indicator lamp, (9) a TPMS indicator lamp, (10) a reversing radar warning indicator lamp, (11) a high beam indicator lamp and (12) a fuel indicator lamp. The on-off state of the indication icons reflects whether the equipment to be detected corresponding to the indication icons in the vehicle has faults or not, for example, the battery indicator lamp is an indicator lamp for displaying the working state of the storage battery, and if the battery indicator lamp is not on or is not on for a long time, the generator or circuit faults are indicated.

For each first image, the corresponding first data is determined, so that the on-off state of the indication icon is also determined, and the on-off state of the indication icon accurately corresponds to the first data in content. In actual operation, the first display controller 101 may sequentially generate a plurality of first images based on an image display order, thereby macroscopically embodying a dynamic effect, such as a long-lighting or blinking effect of an indication icon in the first image, when the plurality of first images are sequentially displayed.

In the embodiment of the present application, the first display controller 101 may be connected to the main processor 300, and acquire the first data from the main processor 300, thereby generating the first image based on the first data. The main processor 300 may generate the first data, for example, the first data may be generated according to the acquired detection data or instruction. Specifically, the main processor 300 may include a Central Processing Unit (CPU), a Field Programmable Gate Array (FPGA), a Network Processor (NP), a digital signal processing circuit (DSP), a Programmable Logic Device (PLD), a Micro Control Unit (MCU), an Application Specific Integrated Circuit (ASIC), a system on chip (SoC), or the like.

Generally speaking, since the first image is generated based on the first data, the first image may represent the content of the first data, that is, the indication icon in the first image and the data indicating the state of the indication icon in the first data should be consistent, and at this time, the generated first image may be considered to be correct, so that the display apparatus 200 may display the first image, and the user may more vividly obtain the content of the first data corresponding to the first image from the first image.

However, the first image may have a problem of generation error, such as the indicating icon in the generated first image has problems of being incomplete, being covered, and being wrong in color. This is because, in practical operation, the safety integrity level of the first display controller 101 is not very high due to cost considerations, and for example, in the vehicle field, the ASIL of the first display controller 101 only meets the quality standard of QM, so the extra workload of the first display controller 101 is generally low, and the cost is also low. Therefore, the first display controller 101 may generate an error when generating the first image, and if the error first image is displayed, the user may not obtain information included in the accurate first data in time.

Therefore, in this embodiment of the application, the neural network processor 102 may be used to monitor the display of the first image, and the neural network processor 102 may obtain at least a portion of the first image and determine whether the at least a portion of the first image is displayed correctly by using the neural network model, so as to obtain a monitoring result for the at least a portion of the first image.

The neural network processor 102 may be a processing device with a neural network model operation capability, such as a neural-Network Processing Unit (NPU), an Artificial Intelligence (AI) processor, or a Bionic (Bionic) device, and the model operation performed by the neural network processor may be a neural network operation, such as embedding a neural network model trained in advance in the neural network processor 102, thereby implementing the model operation. For example, the neural network processor 102 may be a stand-alone chip, such as a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), or other types of neural processing units.

In this embodiment, the neural network processor 102 may monitor at least a portion of the first image, so as to obtain a monitoring result of at least a portion of the first image. Specifically, the neural network processor 102 may input at least a part of the first image into the neural network model to obtain a determination result of correctness of the at least a part of the first image, and as a monitoring result of the at least a part of the first image, the process may be regarded as matching the at least a part of the first image with a preset image to obtain a matching degree, where if the preset image is a correct image, the higher the matching degree is, the higher the probability that the at least a part of the first image is correct is, and if the preset image is an incorrect image, the higher the matching degree is, the higher the probability that the at least a part of the first image is incorrect is.

Of course, in order to obtain the determination result, a first threshold may be set for the probability of correctness of at least a portion of the first image, and when the probability of correctness is higher than the first threshold, at least a portion of the first image may be considered to be correct, otherwise, at least a portion of the first image may be considered to be incorrect. For example, it may be obtained that at least a part of the first image is correct with a probability of 20%, wrong with a probability of 80%, and the first threshold value is 80%, when at least a part of the monitored first image may be considered wrong, i.e. the first image is generated in error.

Specifically, the neural network processor 102 may obtain at least a portion of the first image from the memory 104, and the memory 104 may be connected to the first display controller 101 and the neural network processor 102, respectively, so that after the first display controller 101 generates the first image, the first image may be stored in the memory 104, and the neural network processor 102 may obtain at least a portion of the first image from the memory 104. Wherein the first image may be stored in an image buffer area in the memory 104, and the first display controller 101 may send a storage location of at least a portion of the first image to the neural network processor 102 after generating the first image, so that the neural network processor 102 acquires at least a portion of the first image according to the storage location of at least a portion of the first image. Of course, the memory 104 may also be connected to the display device 200, so that the display device 200 may obtain the first image in the image buffer area, and further perform the display of the first image; of course, the memory 104 may not be directly connected to the display device 200, and the display device 200 may acquire the first image through the first display controller 101 to display the first image.

As can be seen from the above description, in the first image, the correct display of the indication icon is important for the functional safety of the system, so in the embodiment of the present application, at least a part of the first image may be the indication icon, and the monitoring of the indication icon is beneficial to improving the functional safety of the system. That is, the neural network processor 102 may not monitor all image features in the first image, but monitor the indicator icon in the first image, reducing the computational workload of the neural network processor 102. For example, at least a part of the first image is an indication area image including an indication icon, wherein the indication area image may be a rectangle or other shapes.

In a specific implementation, a preset storage area may be set in the memory 104 for at least a part of the first image, so that after the first image is generated, the first display controller 101 may store image data of at least a part of the first image in the preset storage area, and the neural network processor 102 may also read the part of the data from the preset storage area. Specifically, when the first image is stored in the determined data structure, the image data of at least a part of the first image may be mapped to a physical address of the storage area, that is, to the preset storage area, so that the neural network processor 102 reads the pixel data of the preset storage area, that is, obtains the image data of at least a part of the first image.

It should be noted that, when at least a portion of the first image has a fixed display position in the first image, the preset storage area mapped by at least a portion of the first image may be determined in the design stage; when at least a portion of the first image does not have a fixed display position in the first image but dynamically changes according to actual conditions, the neural network processor 102 may obtain a physical address of a preset storage area mapped by at least a portion of the first image, so as to obtain data of the portion, specifically, the neural network processor 102 may obtain the physical address of the preset storage area from the main processor 300 or the MCU400 with a security authentication function connected to the neural network processor 102, and the obtaining may be through various inter-core communication methods, such as an interrupt and a specific message mechanism.

In the embodiment of the present application, since the generation of the first image has a certain periodicity, for example, a video image with 50 Frames Per Second (FPS) outputs one frame only for about 20ms, the neural network processor 102 may determine the period for acquiring the first image according to the generation period of the first image, for example, the neural network processor 102 may acquire the first image once every 20ms, so that the neural network processor 102 in the embodiment of the present application is not exclusive to the neural network model, and may also have the capability to perform other data processing, for example, the neural network processor 102 may further have an audio recognition model, and these different models are not affected by each other.

The neural network model in the neural network processor 102 may be obtained by training in advance using the historical display information and the historical images, so that the neural network model has the capability of recognizing the images, wherein the neural network model may be obtained by training in the design stage or may be obtained by training in the later stage before monitoring the first image. The history images may include correct (Truth) history images and/or incorrect (False) history images, where the incorrect history images may be occluded, incomplete to display, incorrect in color, incorrect in shape, and the like, for example, the indication icons in the history images are occluded by the upper graphics, and the color errors, for example, the indication icons in the history images are not correctly lighted, for example, the indication icons in the history images should be red in color and actually green, and the like.

Since the neural network processor 102 may only acquire image data of at least a portion of the first image, the historical image may only include image data of the portion of the image corresponding to at least a portion of the first image to reduce the amount of computation of data training and the accuracy of image recognition. For example, at least a portion of the first image includes the indicator icon in the first image, the historical image may also include only the correct indicator icon and/or the incorrect indicator icon, so that the trained neural network model has the capability of recognizing the indicator icon.

In the embodiment of the present application, in the generating process of the first image, a background image may be superimposed, for example, a gorgeous background and network information such as atmosphere light, real-time navigation information, real-time media playing information, and real-time weather are superimposed, and a picture of the night vision camera is enhanced, instead of the black background indicator icon in fig. 1, so as to enhance a visual effect of a user or expand display content of a display device to meet a user requirement. In this way, even if the background image is superimposed on the first image, the recognition of the first image by the neural network model is not affected, so that in the process of generating the first image in actual operation, more types of background images can be superimposed without arranging a black rectangular frame outside the indication icon, which is beneficial to diversification and beautification of the first image. The superposition of the indication icon and the historical background image can be performed by DSS superposition, which is not described in detail herein.

After monitoring the later part of the first image, if the neural network model is used to determine that at least one part of the first image is displayed incorrectly, it indicates that the first image has an error in the generation process, and at this time, if the first image is displayed, the displayed image cannot truly reflect the information of the first data, thereby reducing the functional safety of the system. Therefore, the neural network processor 102 may send the determination result to the display device 200 to stop the display device 200 from displaying the first image, and the neural network processor 102 may also send the determination result to the MCU400 or the main processor 300 connected to the neural network processor to cause the MCU400 or the main processor 300 to generate a stop display instruction to stop the display device 200 from displaying the first image, and in addition, the MCU400 or the main processor 300 may also generate and display an alarm icon and alarm information such as an alarm voice to remind the user of a fault, so that the user may not rely on the display content of the display device, avoid being misled by the wrong display content, and improve the display security to a certain extent.

Of course, in order to obtain accurate display content, the neural network processor 102 may also send an image generation instruction to the second display controller, the image generation instruction is used to instruct the second display controller to generate a second image, the second image is used to replace the first image and is displayed by the display device, the second data used to generate the second image is generated by the MCU400 connected to the neural network processor 102, and the second data and the first data are obtained based on the same detection data, that is, the first data and the second data have the same data source. While the first image is stored in the image buffer area in the memory 104, the generated second image may be stored in the image buffer area in the memory 104 in place of the first image. In this way, the second display controller can be used to generate the image corresponding to the information to be displayed again, so that the display device 200 can acquire and display the second image in the image buffer area, and the problem caused by the display device 200 directly displaying the incorrect first image is avoided, thereby improving the display accuracy of the image.

The second display controller 103 may be the above MCU400 with function security authentication capability or a high-end media processor with complex drawing function, and both may have different image generation capabilities, and refer to fig. 3 and 4, which are schematic diagrams of two other image display monitoring devices provided in this embodiment of the present application. The high-end media processor serving as the second display controller 103 may include a DSS or a GPU, and certainly, the second display controller 103 may also include both a GPU and a DSS, and in order to distinguish the first display controller 101 from the second display controller 103, the GPU in the first display controller 101 may be used as the first GPU, the DSS may be used as the first DSS, the GPU in the second display controller 103 may be used as the second GPU, and the DSS may be used as the second DSS. Wherein the second display controller 103 may be connected to the memory 104 such that the second display controller 103 may store the generated second image into the image buffer area.

In this embodiment, the main processor 300 connected to the first display controller 101 and the MCU400 connected to the neural network processor 102 may obtain the detection data at the same time, for example, the detection data may be obtained at the same time through another bus or module in the vehicle, so that the main processor 300 may obtain the first data based on the detection data to generate the first image, and the MCU400 may realize backup of the detection data and then generate the second image using the backup detection data, thereby improving the accuracy of image generation.

In fact, the neural network processor 102 and the second display controller 103 are used for monitoring the first image output from the first display controller 101 and performing generation of the second image, and therefore the second display controller 103 and the neural network processor 102 are often required to have a higher safety integrity level than the main processor 300 and the first display controller 101. For example, in the vehicle field, according to the functional safety standard such as ISO 26262, the neural network processor 102 and the second display controller 103 need to implement the quality standard of ASIL-B, while the first display controller 101 and the main processor 300 need only implement the quality standard of QM, which can improve the overall safety integrity of the system without increasing the cost of the main processor 300 and the first display controller 101. Therefore, in this embodiment of the application, the MCU400 may also acquire the detection data, then obtain an instruction based on the detection data, and send the obtained instruction to the main processor 300, the main processor 300 generates the first data in response to the received instruction from the MCU400, and then the first display controller 101 acquires the first data through the main processor 300.

When the safety integrity level of the second display controller 103 is higher than that of the first display controller 101, the reliability of the second image generated by the second display controller 103 is higher than that of the first image generated by the first display controller 101, and the probability of the second image being erroneous is lower than that of the first image, so that the reliability of displaying the second image is higher than that of displaying the first image. Of course, after the second image is stored in the image buffer area, the neural network processor 102 may also be used to monitor the second image, and the monitoring process may refer to the monitoring process for the first image.

For convenience of understanding, the image display monitoring device provided in the embodiments of the present application is exemplarily described below with reference to specific scenarios.

Referring to fig. 3, the NPU may serve as the neural network processor 102, the MCU400 may serve as the second display controller 102 at the same time, and the CPU may serve as the main processor 300, so the system includes the CPU 300, the first display controller 101, the NPU 102, the MCU400, the memory 104, and the display device 200, where the arrow direction may embody the information flow direction between the respective devices, that is, the first display controller 101 may obtain the first data from the CPU 300, and generate the first image using the first data and store the first image in the image buffer area in the memory 104, the NPU 102 may obtain at least a part of the first image from the image buffer area in the memory 104, so as to monitor at least a part of the first image using the neural network model therein, and when it is determined that at least a part of the first image is incorrect, the NPU 102 may send an image generation instruction to the MCU400, this may instruct MCU400 to generate a second image based on the image generation instruction and store the generated second image in the image buffer area in memory 104, and display apparatus 200 may display the second image.

Referring to fig. 4, the NPU may serve as the neural network processor 102, and the CPU may serve as the main processor 300, so the system includes the CPU 300, the first display controller 101, the NPU 102, the MCU400, the second display controller 103, the memory 104, and the display device 200, where the direction of the arrow may embody the flow direction of information between the respective devices, that is, the first display controller 101 may obtain the first data from the CPU 300, generate the first image using the first data and store the first image in the image buffer area in the memory 104, the NPU 102 may obtain at least a portion of the first image from the image buffer area in the memory 104, so as to monitor at least a portion of the first image using the neural network model therein, and when it is determined that at least a portion of the first image is incorrect, the NPU 102 may send an image generation instruction to the second display controller 103 via the MCU400, this may instruct the second display controller 103 to generate a second image based on the image generation instruction and store the generated second image to the image buffer area in the memory 104, and the display apparatus 200 may display the second image.

The above description is only an exemplary description, and those skilled in the art can freely combine other forms of image monitoring systems according to the functions of the respective modules, which are not illustrated herein.

In the embodiment of the present application, the first display controller 101, the neural network processor 102, and the second display controller 103 may be respectively disposed in a plurality of chips, for example, the first display controller 101 may be integrated with the main processor 300 on the same chip, and the neural network processor 102 may be integrated with the MCU400 on the same chip; or the first display controller 101 may be integrated on the same chip as the neural network processor 102; or the first display controller 101 and the second display controller 103 may be integrated on the same chip; or the first display controller 101 may be integrated with the neural network processor 102 and the second display controller 103 on the same chip; or the neural network processor 102 may be integrated on the same chip as the main processor 300 and the MCU400, and the first display controller 101 and the second display controller 103 are integrated on the same chip. It is understood that the more functional modules integrated on the same chip, the more advantageous the overall hardware size is, and the more advantageous the cost is.

Based on the above description, the neural network processor 102 may monitor at least a portion of the first image generated by the first display controller, where the monitored content is mainly whether the first image is correctly displayed, for example, whether the first image is displayed completely, whether the first image is blocked, whether a color error exists, and the like, and in actual operation, at least a portion of the first image may also have another problem, that is, the displayed icon is not a desired icon, for example, the first data indicates that a certain indication icon is displayed, and the indication icon is not included in the first image, but another indication icon is generated, and the neural network processor 102 in the previous embodiment cannot monitor whether the indication icon is an icon that needs to be displayed, which also easily causes a display error of the first image, thereby affecting the safety integrity of the system.

Therefore, as a possible implementation manner, the neural network processor 102 in this embodiment of the application may further obtain the indication information from the MCU400 in advance, and then monitor the indication icon in the first image based on the indication information and the neural network model, so as to further determine whether the correctly displayed indication icon is an indication icon that needs to be displayed when at least a part of the first image is correctly displayed as a result of the monitoring, thereby further improving the accuracy of image display. That is, the indication information may indicate an indication icon that needs to be displayed in the first image, so that the indication icon in the first image may be further monitored. The difference is that the first data is obtained by the main processor 300, and the indication information is obtained by the MCU400, because the MCU400 usually has a higher safety integrity level than the main processor 300, the indication information has higher reliability than the first data.

As another possible implementation manner, the neural network processor 102 in the embodiment of the present application may only perform monitoring on the content of the indication icon, and the MCU400 may determine whether the indication icon is an indication icon that needs to be displayed. Specifically, the MCU400 may determine whether the correctly displayed indication icon is an indication icon that needs to be displayed based on the obtained detection data when the monitoring result of the neural network processor 102 is that at least a portion of the first image is correctly displayed, where the determination may be by using a neural network model, or may be in another image recognition mode, and is not limited again.

The embodiment of the application provides an image display monitoring device, which can comprise a first display controller and a neural network processor, wherein the first display controller can generate a first image by using first data, the first image is used for displaying on a display device, and actually, the first image has the problem of generating errors possibly, so that the first image cannot correctly embody the characteristics of the first data. Therefore, in the embodiment of the present application, at least a part of the first image may be obtained by using the neural network processor, and the neural network model is used to determine whether at least a part of the first image is displayed correctly, so as to obtain the monitoring result of at least a part of the first image.

For ease of understanding, the following specifically illustrates an image monitoring method provided in the embodiments of the present application. Referring to fig. 5, the method may include the steps of:

s101, generating a first image by a first display controller by using first data. In this embodiment, the first display controller 101 may generate the first image based on first data, where the first data used for generating the first image is data indicating image features in the first image, for example, may be a detection result generated based on detection data obtained by detecting a device to be detected, and the detection result may include an abnormal result and/or a normal result. When the first image includes the indication icon, the first data includes data indicating status information of the indication icon in the first image, and the status information of the indication icon may include on or off, generally, if the first data indicates that the status of the indication icon in the first image is on, the first image generated based on the first data includes the illuminated indication icon, and if the first data indicates that the status of the indication icon in the first image is off, the first image generated based on the first data does not include the illuminated indication icon. The manner in which the first display controller generates the first image based on the first data may refer to the description of the foregoing system embodiment, which is not repeated herein.

S102, at least one part of the first image is acquired. In this embodiment, the neural network processor 102 may monitor at least a portion of the first image, so as to obtain a monitoring result of at least a portion of the first image. Specifically, the neural network processor 102 may input at least a part of the first image into the neural network model to obtain a determination result of correctness of the at least a part of the first image, and as a monitoring result of the at least a part of the first image, the process may be regarded as matching the at least a part of the first image with a preset image to obtain a matching degree, where if the preset image is a correct image, the higher the matching degree is, the higher the probability that the at least a part of the first image is correct is, and if the preset image is an incorrect image, the higher the matching degree is, the higher the probability that the at least a part of the first image is incorrect is.

In the embodiment of the present application, at least a portion of the first image may be an indication icon, and monitoring of the indication icon is beneficial to improve the functional safety of the system, so that at least a portion of the first image is an indication area image including the indication icon, wherein the indication area image may be a rectangle or other shapes. The manner of acquiring at least a portion of the first image may refer to the description of the foregoing system embodiment, and is not repeated herein.

S103, judging whether at least one part of the first image is displayed correctly by using the neural network model so as to obtain a monitoring result of at least one part of the first image. Generally speaking, since the first image is generated based on the first data, the first image may represent the content of the first data, that is, the indication icon in the first image and the data indicating the state of the indication icon in the first data should be consistent, and at this time, the generated first image may be considered to be correct, so that the display apparatus 200 may display the first image, and the user may more vividly obtain the content of the first data corresponding to the first image from the first image.

However, the first image may have a problem of generation error, such as the indicating icon in the generated first image has problems of being incomplete, being covered, and being wrong in color. Therefore, the first display controller 101 may generate an error when generating the first image, and if the error first image is displayed, the user may not obtain the information included in the accurate first data in time. Therefore, in this embodiment of the application, the neural network processor 102 may be used to monitor the display of the first image, and the neural network processor 102 may obtain at least a portion of the first image and determine whether the at least a portion of the first image is displayed correctly by using the neural network model, so as to obtain a monitoring result for the at least a portion of the first image.

In this embodiment, the neural network processor 102 may monitor at least a portion of the first image, so as to obtain a monitoring result of at least a portion of the first image. Specifically, the neural network processor 102 may input at least a part of the first image into the neural network model to obtain a determination result of correctness of the at least a part of the first image, and as a monitoring result of the at least a part of the first image, the process may be regarded as matching the at least a part of the first image with a preset image to obtain a matching degree, where if the preset image is a correct image, the higher the matching degree is, the higher the probability that the at least a part of the first image is correct is, and if the preset image is an incorrect image, the higher the matching degree is, the higher the probability that the at least a part of the first image is incorrect is. The training process of the neural network model in the neural network processor 102 may refer to the description in the foregoing system embodiment, and is not described herein again.

Of course, in order to obtain accurate display content, the neural network processor 102 may also send an image generation instruction to the second display controller 103, the image generation instruction is used to instruct the second display controller 103 to generate a second image, the second image is used to replace the first image and is displayed by the display device, the second data used to generate the second image is generated by the MCU400 connected to the neural network processor 102, and the second data and the first data are obtained based on the same detection data, that is, the first data and the second data have the same data source. While the first image is stored in the image buffer area in the memory 104, the generated second image may be stored in the image buffer area in the memory 104 in place of the first image. In this way, the second display controller can be used to generate the image corresponding to the information to be displayed again, so that the display device 200 can acquire and display the second image in the image buffer area, and the problem caused by the display device 200 directly displaying the incorrect first image is avoided, thereby improving the display accuracy of the image.

As a possible implementation manner, the neural network processor 102 in this embodiment of the application may further obtain the indication information from the MCU400 in advance, and then monitor the indication icon in the first image based on the indication information and the neural network model, so as to further determine whether the correctly displayed indication icon is an indication icon that needs to be displayed when at least a part of the first image is correctly displayed as a result of the monitoring. That is, the indication information may indicate an indication icon that needs to be displayed in the first image, so that the indication icon in the first image may be further monitored. The difference is that the first data is obtained by the main processor 300, and the indication information is obtained by the MCU400, because the MCU400 usually has a higher safety integrity level than the main processor 300, the indication information has higher reliability than the first data.

The embodiment of the application provides an image display monitoring method, wherein a first display controller can generate a first image by using first data, the first image is used for displaying on a display device, and actually, the first image may have the problem of generation error, so that the first image cannot correctly embody the characteristics of the first data. Therefore, in the embodiment of the present application, at least a part of the first image may be obtained by using the neural network processor, and the neural network model is used to determine whether at least a part of the first image is displayed correctly, so as to obtain the monitoring result of at least a part of the first image.

The method for monitoring image display provided by the embodiment of the present application is described in detail above with reference to fig. 5. Hereinafter, the image display monitoring apparatus according to the embodiment of the present application will be described in detail with reference to fig. 6. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments, and therefore, for brevity, details are not repeated here, since the details that are not described in detail may be referred to the above method embodiments.

A display control module 110 for generating a first image by a first display controller using first data, the first image for display on a display device; a processing module 120 for acquiring at least a portion of the first image; and judging whether at least one part of the first image is displayed correctly by utilizing a neural network model so as to obtain a monitoring result of at least one part of the first image.

The image display monitoring apparatus may implement the steps or flows corresponding to those executed by the processing device in the method according to the embodiment of the present application, and the apparatus may include a unit for executing the method executed by the processing device in the method in fig. 5. Also, the units in the apparatus and the other operations and/or functions described above are respectively for implementing the corresponding flows of the method in fig. 5.

It should be understood that the specific processes of the units for executing the corresponding steps are already described in detail in the above method embodiments, and therefore, for brevity, detailed descriptions thereof are omitted. Any of the above modules may be implemented in software, hardware, or a combination of both. The hardware may include various electronic circuits such as a digital logic circuit or an analog circuit, which is not limited in this embodiment. The modules, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or a magnetic or optical disk.

According to the image display monitoring method provided by the embodiment of the application, the embodiment of the application also provides a neural network processor, which comprises a processor and a memory; the memory for storing computer programs or instructions; the processor is configured to execute the computer program or the instruction in the memory to implement the image display monitoring method according to any one of the embodiments shown in fig. 5.

According to an image display monitoring method provided by an embodiment of the present application, the present application further provides a computer program product containing a computer program or an instruction, the computer program product including: computer program code which, when run on a computer, causes the computer to implement the image display monitoring method of any one of the embodiments shown in fig. 5.

According to the image display monitoring method provided by the embodiment of the present application, the present application also provides a computer-readable medium, in which a program code is stored, and when the program code runs on a computer, the computer is caused to implement the image display monitoring method of any one of the embodiments shown in fig. 5.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The above embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions in the embodiments of the present application.

Claims

An image display monitoring apparatus, comprising:

a first display controller for generating a first image using first data, the first image for display on a display device;

and the neural network processor is used for acquiring at least one part of the first image, and judging whether the at least one part of the first image is displayed correctly by using a neural network model so as to obtain a monitoring result of the at least one part of the first image.
The apparatus of claim 1, wherein the neural network processor is further configured to: when the monitoring result is that at least one part of the first image is displayed incorrectly, sending an image generation instruction to a second display controller; the second display controller is to generate a second image to be displayed on the display device in place of the first image in response to the image generation instruction.
The device of claim 2, wherein the second display controller has a higher safety integrity level than the first display controller.
The device of claim 2 or 3, wherein the first display controller comprises at least one of a first image processor GPU or a first display subsystem DSS, and the second display controller comprises at least one of a second image processor GPU, a second display subsystem DSS, or a micro control unit MCU.
The device of any of claims 1-4, wherein at least a portion of the first image comprises an indicator icon;

the apparatus further comprises: and the MCU is used for determining whether the indication icon is an indication icon needing to be displayed or not when the monitoring result is that at least one part of the first image is displayed correctly.
The device of any of claims 1-4, wherein at least a portion of the first image comprises an indicator icon;

the neural network processor is further configured to: and when the monitoring result is that at least one part of the first image is displayed correctly, judging whether the indication icon is an indication icon needing to be displayed or not by utilizing a neural network model and indication information from the MCU.
The apparatus of any of claims 1-6, further comprising:

a main processor for generating the first data;

the first display controller is specifically configured to acquire the first data from the main processor.
The apparatus of claim 7,

the main processor is further configured to: and acquiring an instruction from the MCU, and responding to the instruction to generate the first data.
A method of image display monitoring, comprising:

generating, by a first display controller, a first image using first data, the first image for display on a display device;

acquiring at least a portion of a first image;

and judging whether at least one part of the first image is displayed correctly by utilizing a neural network model so as to obtain a monitoring result of at least one part of the first image.
The method of claim 9, further comprising:

when the monitoring result is that at least one part of the first image is displayed incorrectly, sending an image generation instruction to a second display controller;

generating, by the second display controller in response to the image generation instruction, a second image for display on the display device in place of the first image.
The method of claim 10, wherein the second display controller has a higher safety integrity level than the first display controller.
The method of any of claims 9-11, wherein at least a portion of the first image comprises an indicator icon, the method further comprising:

when the monitoring result is that at least one part of the first image is displayed correctly, determining whether the indication icon is an indication icon needing to be displayed.
An image display monitoring apparatus, comprising:

a display control module to generate a first image with first data by a first display controller, the first image for display on a display device;

a processing module for acquiring at least a portion of a first image; and judging whether at least one part of the first image is displayed correctly by utilizing a neural network model so as to obtain a monitoring result of at least one part of the first image.
A computer-readable storage medium comprising a computer program or instructions which, when run on a computer, cause the computer to carry out the image display monitoring method of any one of the preceding claims 9-12.