KR102904535B1 - An electronic device for proccesing an image and a method thereof - Google Patents

Abstract

Various embodiments of the present disclosure relate to an electronic device and a control method thereof that selectively applies an image processing model for processing an image of a region of interest based on deep learning.
To this end, the electronic device may set a region of interest to be image-quality processed based on deep learning from an input image, and designate one of a plurality of image processing models with different batch sizes as the selected image processing model, taking into account the memory usage and the size of the region of interest. The electronic device may perform an operation of performing image-quality processing based on deep learning by applying the selected image processing model to the image of the region of interest. In addition, various other embodiments may be possible.

Description

{AN ELECTRONIC DEVICE FOR PROCCESING AN IMAGE AND A METHOD THEREOF}

Various embodiments of the present disclosure relate to an electronic device for processing an image of a region of interest (RoI) based on deep learning and a control method thereof.

Communication technologies are rapidly evolving to enable the acquisition of desired information and data without temporal or spatial constraints. One example is the recently commercialized new radio (NR) communication system (e.g., ^5th- generation, or pre-5G, communication system). NR communication systems were developed and commercialized to meet the increasing demand for wireless traffic compared to previous communication systems (e.g., 4th ^- generation, or 4G, communication systems). For this reason, NR communication systems are also referred to as "beyond 4G networks" or "post-LTE" systems.

Advances in communications technology, coupled with the proliferation of the digital economy, have enabled the comprehensive collection of diverse data formats, including numbers, text, and images, creating a foundation for a "big data" environment. Furthermore, this big data environment has enabled artificial intelligence (AI) and deep learning through the processing and utilization of information and data.

Deep learning can be used to cluster or classify objects or data. Deep learning can also be applied to image processing technologies. Image processing technologies based on deep learning can use either the entire image as input or fixed-size blocks to produce output. Therefore, electronic devices that process images may reach limits in processing speed or memory usage when using one or two image processing engines.

According to various embodiments of the present disclosure, an electronic device and a control method thereof can be provided that selectively applies an image processing model for processing an image of a region of interest based on deep learning.

The technical task to be achieved in this document is not limited to the technical task mentioned above, and other technical tasks may be anticipated within a scope that can be clearly understood by a person of ordinary skill in the technical field to which the various embodiments proposed in the description below belong.

According to various embodiments of the present disclosure, an electronic device includes a memory, a camera module, a communication module, and at least one processor that performs image quality processing based on deep learning on an input image provided from at least one of the memory, the camera module, or the communication module, wherein the memory may store instructions that, when executed, cause the at least one processor to set a region of interest for image quality adjustment from the input image, determine one of a plurality of image processing models having different batch sizes as a selected image processing model in consideration of the usage amount of the memory and the size of the region of interest, and perform image quality processing based on deep learning by applying the selected image processing model to an image of the region of interest.

According to various embodiments of the present disclosure, a method for performing image quality processing based on deep learning in an electronic device may include an operation of setting a region of interest to be image quality processed based on deep learning from an input image, an operation of determining one of a plurality of image processing models having different batch sizes as a selected image processing model by considering the amount of memory used and the size of the region of interest, and an operation of performing image quality processing based on deep learning by applying the selected image processing model to an image of the region of interest.

According to various embodiments of the present disclosure, when processing an image based on deep learning in an electronic device, an image processing model having a batch size corresponding to an optimal batch processing area for an image of a region of interest is applied, thereby ensuring efficient memory usage or image processing speed for image processing.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects may be predicted within a range that can be clearly understood by a person having ordinary skill in the art to which the various embodiments proposed in the description below belong.

FIG. 1 is a block diagram illustrating a configuration of an electronic device (101) within a network environment according to various embodiments of the present disclosure;
FIG. 2 is a block diagram illustrating a camera module (180) according to various embodiments of the present disclosure;
FIG. 3 is a diagram illustrating examples of image processing models (300) applicable to image processing based on deep learning in an electronic device (101), according to various embodiments of the present disclosure;
FIG. 4 is a diagram illustrating an example of selecting an image processing model to be applied to deep-learning-based image processing in an electronic device (101) according to various embodiments of the present disclosure;
FIG. 5 is a block diagram (500) illustrating a model selection module (410) according to various embodiments of the present disclosure;
FIG. 6 is a diagram illustrating a control flow (600) for performing image processing based on deep learning in an electronic device (101) according to various embodiments of the present disclosure;
FIG. 7 is a diagram illustrating an example of performing image processing on a region of interest in an electronic device (101) according to various embodiments of the present disclosure;
FIG. 8 is a diagram illustrating another example of performing image processing on a region of interest in an electronic device (101) according to various embodiments of the present disclosure; and
FIG. 9 is a diagram illustrating another example of performing image processing on a region of interest in an electronic device (101) according to various embodiments of the present disclosure.

Hereinafter, various embodiments proposed in the present disclosure will be described in detail with reference to the attached drawings. However, for convenience of explanation, the sizes of components in the drawings may be exaggerated or reduced. For example, the sizes and thicknesses of each component shown in the drawings are arbitrarily shown for convenience of explanation, and it should be noted that the various embodiments proposed in the present disclosure are not necessarily limited to those illustrated.

FIG. 1 is a diagram illustrating a block configuration of an electronic device (101) within a network environment according to various embodiments of the present disclosure.

Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input device (150), an audio output device (155), a display device (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the display device (160) or the camera module (180)), or may have one or more other components added. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, a sensor module (176) (e.g., a fingerprint sensor, an iris sensor, or an ambient light sensor) may be implemented embedded in a display device (160) (e.g., a display).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may load a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) into a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor), and a secondary processor (123) (e.g., a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together therewith. Additionally or alternatively, the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part of it.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display device (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)).

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input device (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input device (150) can include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

The audio output device (155) can output audio signals to the outside of the electronic device (101). The audio output device (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

A display device (160) can visually provide information to an external party (e.g., a user) of an electronic device (101). The display device (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display device (160) may include touch circuitry configured to detect a touch, or a sensor circuit (e.g., a pressure sensor) configured to measure the intensity of a force generated by the touch.

The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through an input device (150), or output sound through an audio output device (155), or an external electronic device (e.g., an electronic device (102)) (e.g., a speaker or headphone)) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104)), or server (108), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) may use subscriber information stored in the subscriber identification module (196) (e.g., an international mobile subscriber identity (IMSI)) to identify and authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199).

The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include one antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from among the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and the external electronic device via the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the electronic devices (102, 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices receiving the request may execute at least a portion of the requested function or service, or additional functions or services related to the request, and transmit the results of the execution to the electronic device (101). The electronic device (101) may process the results, either as is or additionally, and provide them as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, or client-server computing technologies may be utilized, for example.

FIG. 2 is a diagram illustrating a block configuration (200) exemplifying a camera module (180) according to various embodiments of the present disclosure.

Referring to FIG. 2, the camera module (180) may include a lens assembly (210), a flash (220), an image sensor (230), an image stabilizer (240), a memory (250) (e.g., a buffer memory), or an image signal processor (260). The lens assembly (210) may collect light emitted from a subject that is a target of image capturing. The lens assembly (210) may include one or more lenses. According to one embodiment, the camera module (180) may include a plurality of lens assemblies (210). In this case, the camera module (180) may form, for example, a dual camera, a 360-degree camera, or a spherical camera. Some of the plurality of lens assemblies (210) may have the same lens properties (e.g., angle of view, focal length, autofocus, f-number, or optical zoom), or at least one lens assembly may have one or more lens properties that are different from the lens properties of the other lens assemblies. A lens assembly (210) may include, for example, a wide-angle lens or a telephoto lens.

The flash (220) can emit light used to enhance light emitted or reflected from a subject. According to one embodiment, the flash (220) can include one or more light-emitting diodes (e.g., red-green-blue (RGB) LED, white LED, infrared LED, or ultraviolet LED), or a xenon lamp. The image sensor (230) can acquire an image corresponding to the subject by converting light emitted or reflected from the subject and transmitted through the lens assembly (210) into an electrical signal. According to one embodiment, the image sensor (230) can include one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, a plurality of image sensors having the same property, or a plurality of image sensors having different properties. Each image sensor included in the image sensor (230) can be implemented using, for example, a CCD (charged coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor.

The image stabilizer (240) can move at least one lens or image sensor (230) included in the lens assembly (210) in a specific direction or control the operating characteristics of the image sensor (230) (e.g., adjusting the read-out timing, etc.) in response to the movement of the camera module (180) or the electronic device (101) including the same. This allows compensating for at least some of the negative effects of the movement on the captured image. In one embodiment, the image stabilizer (240) can detect such movement of the camera module (180) or the electronic device (101) using a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module (180). In one embodiment, the image stabilizer (240) can be implemented as, for example, an optical image stabilizer. The memory (250) can temporarily store at least a portion of the image acquired through the image sensor (230) for the next image processing task. For example, when image acquisition is delayed due to the shutter, or when multiple images are acquired at high speed, the acquired original image (e.g., a Bayer-patterned image or a high-resolution image) is stored in the memory (250), and a corresponding copy image (e.g., a low-resolution image) can be previewed through the display device (160). Thereafter, when a specified condition is satisfied (e.g., a user input or a system command), at least a portion of the original image stored in the memory (250) can be acquired and processed, for example, by the image signal processor (260). According to one embodiment, the memory (250) can be configured as at least a portion of the memory (130) or as a separate memory that operates independently therefrom.

The image signal processor (260) can perform one or more image processing operations on an image acquired through an image sensor (230) or an image stored in a memory (250). The one or more image processing operations may include, for example, depth map generation, 3D modeling, panorama generation, feature point extraction, image synthesis, or image compensation (e.g., noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening). Additionally or alternatively, the image signal processor (260) may perform control (e.g., exposure time control, read-out timing control, etc.) for at least one of the components included in the camera module (180) (e.g., image sensor (230)). An image processed by the image signal processor (260) may be stored back in the memory (250) for further processing or provided to an external component of the camera module (180) (e.g., memory (130), display device (160), electronic device (102), electronic device (104), or server (108)). According to one embodiment, the image signal processor (260) may include at least one of the processors (120). It may be configured as a separate processor that is configured as a part of the processor (120) or operates independently of the processor (120). If the image signal processor (260) is configured as a separate processor from the processor (120), at least one image processed by the image signal processor (260) may be displayed through the display device (160) as is or after undergoing additional image processing by the processor (120).

According to one embodiment, the electronic device (101) may include a plurality of camera modules (180), each having different properties or functions. In this case, for example, at least one of the plurality of camera modules (180) may be a wide-angle camera, and at least another may be a telephoto camera. Similarly, at least one of the plurality of camera modules (180) may be a front camera, and at least another may be a rear camera.

FIG. 3 is a diagram illustrating examples of image processing models (300) applicable to image processing based on deep learning in an electronic device (101) (e.g., the electronic device (101) of FIG. 1), according to various embodiments of the present disclosure.

Referring to FIG. 3, image processing models (310, 320, 330, 340) applicable to image processing based on deep learning as an example may have different numbers of image pixels that can be batch processed. The number of image pixels that can be batch processed may define a batch size corresponding to the size of the image that can be batch processed, for example. The batch size may define a batch region. The number of image pixels of the image processing model may be defined by the number of horizontal and vertical pixels. The image processing models (310, 320, 330, 340) may have, for example, 4032 3024 Model (310), 2016 1564 Model (320), 1008 782 Model (330), 252 It may include 189 models (340). For example, 4032 The 3024 model (310) has a number of pixels within the layout area of 4032. It could be an image processing model with a batch size of 3024 (4032 horizontal pixels multiplied by 3024 vertical pixels). Another example is 252 189 Model (340) has 252 pixels within the deployment area. It could be an image processing model with a batch size of 189 (252 horizontal pixels multiplied by 189 vertical pixels).

According to one embodiment, the electronic device (101) may selectively apply all or some of the image processing models (310, 320, 330, 340) having various batch sizes, taking into account signal processing capability or memory size, etc. In FIG. 3, three image processing models (e.g., 4032 3024 Model (350), 2016 1564 Model (360), 252 An example of a 189 model (340) (hereinafter referred to as a 'candidate image processing model') being applied to an electronic device (101) for image processing based on deep learning is shown.

According to one embodiment, the electronic device (101) considers at least one of the following factors: a region of interest, memory usage in use by the electronic device (101), available memory for image processing, or signal processing speed to select candidate image processing models 4032 3024 Model (350), 2016 1564 Model (360), 252 Among 189 models (340), the optimal image processing model can be designated as the selected image processing model. The region of interest may be, for example, a partial region in which detailed image quality, etc. is to be changed among the entire region of the input image. The electronic device (101) may process an image included in the region of interest (hereinafter referred to as a “region of interest image”) using deep learning to which the selected image processing model is applied. The processing of the region of interest image may be, for example, image processing for adjusting the image quality of the region of interest image. The image quality may be changed by at least one of the three major attributes (e.g., HSV (hue saturation value)): hue (a characteristic of color), saturation (a pure degree of color), and value (a degree of brightness of color). The image quality may also be adjusted by, for example, indirectly changing brightness, contrast, color temperature, etc.

According to one embodiment, the electronic device (101) may increase or decrease the image processing time or memory usage based on deep learning in proportion to the batch size of the image processing model. That is, as the batch size of the image processing model increases, the area of the image that can be processed by a single image processing (e.g., the batch area) may expand or the number of pixels to be processed by a single image processing may increase. However, as the batch size of the image processing model increases, the time required for image processing or the memory usage may increase. For example, 4032 The time required for image processing using the 3024 model (350) is 2016 The time required for image processing using the 1564 model (360) may be doubled. Therefore, the electronic device (101) should be able to select an image processing model with a batch size that is expected to optimize the time required or memory usage, taking into account the size of the region of interest.

According to one embodiment, the image signal processor (260) may set a region of interest for image quality adjustment from an input image, and designate one of a plurality of image processing models (350, 360, 370) having various batch sizes as a selected image processing model by considering at least one of the size of the region of interest, the amount of memory used, the processing speed, or the image processing capability. The image signal processor (260) may perform image quality processing on an image of the region of interest based on deep learning that applies the selected image processing model.

According to one embodiment, the image signal processor (260) may determine as the selected image processing model one image processing model having a batch size equal to or similar to the size of the region of interest, one image processing model having a batch size larger than the size of the region of interest, or one or more image processing models having a batch size smaller than the size of the region of interest.

According to one embodiment, when there are multiple image processing models among the plurality of image processing models (350, 360, 370) having a batch size larger than the size of the region of interest, the image signal processor (260) may designate one image processing model (e.g., an image processing model with the smallest residual area) having a batch size most similar to the size of the region of interest among the image processing models having the large batch sizes as the selected image processing model. The one image processing model most similar to the size of the region of interest may be, for example, an image processing model in which the remaining area (e.g., residual area) excluding the region of interest in the batch area of the image processing model is the smallest. When the selected image processing model is designated among the image processing models having a batch size larger than the region of interest, the image signal processor (260) may, for example, apply the batch area of the selected image processing model so that the residual area is uniformly distributed around the region of interest.

According to one embodiment, when there are multiple image processing models among the plurality of image processing models (350, 360, 370) having a batch size smaller than the size of the region of interest, the image signal processor (260) may designate a selected image processing model among the plurality of image processing models having smaller sizes by considering the number of uses or the overlapping area. For example, the image signal processor (260) may designate one image processing model that generates the minimum overlapping area as the selected image processing model when processing an image of the region of interest among the image processing models having smaller sizes. As another example, the image signal processor (260) may determine one image processing model that has the minimum number of uses as the selected image processing model when processing an image of the region of interest among the image processing models having smaller sizes.

FIG. 4 is a diagram illustrating an example of selecting an image processing model to be applied to image processing based on deep learning in an electronic device (101) (e.g., the electronic device (101) in FIG. 1) according to various embodiments of the present disclosure.

Referring to FIG. 4, an electronic device (101) according to an embodiment can set a region of interest by a function such as zoom-in, region of interest (ROI), or crop in the entire region (420) of an input image. For example, when a user selects a partial region (421) from the entire region (420) of an input image by using a specific function, the electronic device (101) can set the partial region (421) selected by the user as a region of interest. An image of the region of interest included in the region of interest can be used as an input image (430) for image processing based on deep learning.

According to one embodiment, when a region of interest image (430) is input for image processing based on deep learning, a model selection module (410) included in an electronic device (101) (e.g., an image signal processor (260) in FIG. 2) may designate a selected image processing model having a batch size that is predicted to be able to efficiently process the region of interest image (430) from among a plurality of candidate image processing models (e.g., a first model (451), a second model (453), a third model (455)). The plurality of candidate image processing models (e.g., a first model (451), a second model (453), a third model (455)) may have different batch sizes.

For example, the first model (451) is 4032 in Fig. 3 It may be the 3024 model (350), and the second model (453) may be the 2016 in FIG. 3 It may be the 1564 model (360), and the third model (455) may be the 252 in FIG. 3. It may be a 189 model (340). The first model (451) may have, for example, a batch size larger than the size of the region of interest image (430), the second model (453) may have, for example, a batch size similar to the size of the region of interest image (430), and the third model (455) may have, for example, a batch size smaller than the size of the region of interest image (430).

According to one embodiment, the model selection module (410) may preferably select an image processing model having a batch size that can ensure optimized memory usage or processing speed from among a plurality of candidate image processing models (451, 453, 455) by considering information about the region of interest image (430). The information about the region of interest image (430) may be, for example, information about the size of the region of interest image (430). For example, the model selection module (410) may designate an image processing model having a batch size suitable for processing the region of interest image (430) based on deep learning as the selected image processing model (440) by evaluating at least one of the processing speed, memory usage, or memory availability in the electronic device (101). The model selection module (410) may, for example, designate as the selected image processing model (440) one of an image processing model having a batch size similar to the size of the region of interest (e.g., the second model (453)), an image processing model having a batch size smaller than the size of the region of interest but generating a minimum surplus area when applied (e.g., the third model (455)), or an image processing model having a batch size larger than the size of the region of interest (e.g., the first model (451)).

The above selection model (440) may be provided as an image processing module (not shown) that performs image processing based on deep learning. The image processing module may be provided, for example, within an image signal processor (260).

FIG. 5 is a block diagram illustrating a configuration (500) of a model selection module (410) (e.g., the model selection module (410) in FIG. 4) according to various embodiments of the present disclosure.

Referring to FIG. 5, a model selection module (410) according to an embodiment may include a storage medium (520), a memory inspection module (530), a speed inspection module (540), or a position normalization module (550). The model selection module (410) may, for example, select an image processing model based on input region of interest information (510) and output a model index (560) indicating the selected image processing model (e.g., the selected model (440) in FIG. 4). The selected image processing model may be, for example, an optimal image processing model for processing an image included in a region of interest indicated by the region of interest information (510) (a region of interest image (430) in FIG. 4) using deep learning. The optimal image processing model may be, for example, an image processing model that is predicted to have fast processing and/or minimal memory usage and/or efficient memory usage when processing a target image (e.g., an image included in a region of interest) using deep learning.

According to one embodiment, the storage medium (520) may store information about a plurality of image processing models that can be applied in image processing based on deep learning, information about processing speed for each image processing model, and/or information about memory usage for each image processing model. The storage medium (520) may provide, for example, information about one of the plurality of image processing models and information about processing speed and/or memory usage for selecting an image processing model. The information about the processing speed and/or memory usage may include, for example, an exec time or an open time. The information about the processing speed and/or memory usage may include, for example, information about memory usage and/or processing speed for each candidate image processing model. The information about the memory usage and/or processing speed for each candidate image processing model may be obtained, for example, by performing image processing for each candidate image processing model and measuring the memory usage and/or processing speed during the image processing.

According to one embodiment, the memory inspection module (530) may perform an evaluation of memory usage for each image processing model for a region of interest recognized by the region of interest information (510). The memory inspection module (530) may, for example, predict memory usage for each image processing model when processing an image of a region of interest based on deep learning that applies each of a plurality of candidate image processing models (e.g., the first, second, and third models (451, 453, 455) in FIG. 4).

According to one embodiment, the memory inspection module (530) may classify at least one usable image processing model (e.g., the selection model (440) in FIG. 4) among candidate image processing models, considering the memory usage or memory availability. The memory usage or the memory availability may be an important requirement for determining, for example, the memory availability that can be used for image processing in the electronic device (101). The memory availability may change from time to time depending on one or more functions driven by the electronic device (101). For example, if the memory usage by one or more functions driven by the electronic device (101) increases, the memory availability that can be used for image processing will decrease in response thereto. If the memory availability decreases, the batch size of the selected image processing model will also decrease.

According to one embodiment, the memory inspection module (530) may designate an image processing model among candidate image processing models as the selected image processing model if one of the candidate image processing models satisfies the memory availability. However, if multiple image processing models among the candidate image processing models satisfy the memory availability, the memory inspection module (530) may designate an image processing model with the smallest surplus area among the batch areas as the selected image processing model. For example, the selected image processing model may designate an image processing model that has an image processing size most similar to the size of the region of interest image among the multiple image processing models that satisfy the memory availability as the selected image processing model. The image processing size may be, for example, a batch size of one image processing model (e.g., see the example illustrated in FIG. 7) or a size corresponding to an area in which a region of interest image can be processed by using the batch area of one image processing model multiple times (e.g., see the example illustrated in FIG. 8).

According to one embodiment, the speed test module (540) may further consider the processing speed that can determine the time required for image processing in order to designate the selected image processing model. The speed test module (540) may, for example, evaluate the processing speed for each image processing model for the region of interest recognized by the region of interest information (510). The speed test module (540) may, for example, predict the processing speed for each image processing model when processing the image of the region of interest based on deep learning that applies each of a plurality of image processing models. According to one embodiment, the speed test module (540) may test the processing speed for processing the region of interest image based on deep learning for one or more selected image processing models that are tested as being applicable for processing the region of interest image by the memory test module (530). The speed test module (540) may, for example, select an image processing model for processing the region of interest image by considering the image processing capability of the electronic device (101).

According to one embodiment, the speed inspection module (540) may additionally consider the rotation of the region of interest image when inspecting the processing speed for each candidate image processing model. The speed inspection module (540) may, for example, compare the processing speed when processing an unrotated region of interest image using deep learning that applies the same candidate image processing model with the processing speed when processing a rotated region of interest image, and generate rotation information indicating whether the region of interest image is rotated based on the comparison result.

According to one embodiment, the position normalization module (550) may designate a selected image processing model to be applied to deep learning among a plurality of candidate image processing modules by considering at least one of the predicted processing speed and/or memory usage for each candidate image processing model. Once the selected image processing model to be applied to deep learning is determined, the position normalization module (550) may output an index (560) indicating the determined image processing model.

According to one embodiment, the position normalization module (550) may output rotation information (560) in addition to the model index if the rotation of the region of interest image has been taken into account by the velocity check module (540).

FIG. 6 is a diagram illustrating a control flow (600) for performing image processing based on deep learning in an electronic device (101) (e.g., the electronic device (101) in FIG. 1) according to various embodiments of the present disclosure. The operations according to the control flow illustrated in FIG. 6 may be performed, for example, by at least one processor (e.g., the processor (120) in FIG. 1 or the image signal processor (260) in FIG. 2).

Referring to FIG. 6, in operation 610 according to an embodiment, the electronic device (101) may determine a region of interest (e.g., a region of interest (ROI) or a zoom region (e.g., a region of interest (421) in FIG. 4)) such as a ROI region or a zoom region from a target image (e.g., an input image (420) in FIG. 4) input for image processing. The region of interest may be, for example, a region to be image-quality-processed based on deep learning from an input image, and may be designated or selected by a user from the target image.

In operation 620 according to an embodiment, the electronic device (101) may designate one or more selected image processing models to be applied for image processing based on deep learning among a plurality of candidate image processing models. The electronic device (101) may, for example, predict memory usage and/or processing speed when performing image processing on a region of interest image based on deep learning by applying each of the plurality of candidate image processing models having different batch sizes, and designate a selected image processing model to be applied to deep learning in consideration of the memory usage and/or processing speed predicted for each candidate image processing model. The plurality of candidate image processing models may include image processing models that are similar in size to, larger than, or smaller than the region of interest image.

According to one embodiment, when there are multiple image processing models among a plurality of candidate image processing models (e.g., image processing models 350, 360, 370 in FIG. 3) having a batch size larger than the size of the region of interest, the electronic device (101) may designate as a selected image processing model one image processing model among the image processing models having the large batch size a batch size most similar to the size of the region of interest (e.g., an image processing model having the smallest residual area). The one image processing model most similar to the size of the region of interest may be, for example, an image processing model having a batch area of the image processing model in which the remaining area excluding the region of interest (e.g., residual area) is the smallest.

According to one embodiment, when there are multiple image processing models among a plurality of candidate image processing models having a batch size smaller than the size of the region of interest, the electronic device (101) may designate a selected image processing model among the plurality of image processing models having a smaller size by considering the number of uses or the overlapping area. For example, the electronic device (101) may designate one image processing model having a minimum overlapping area as the selected image processing model among the image processing models having a smaller size when processing an image of the region of interest. As another example, the electronic device (101) may determine one image processing model having a minimum number of uses as the selected image processing model among the image processing models having a smaller size when processing an image of the region of interest.

According to one embodiment, the electronic device (101) may vary the method of applying the image processing model to the region of interest depending on the correlation between the size of the image processing model and the size of the region of interest.

For example, when the batch size of the selection image processing model is similar to the size of the region of interest, the selection image processing model can be applied to the region of interest (e.g., see (a) of FIG. 7). As another example, when the size of the selection image processing model is smaller than the size of the region of interest, the selection image processing models can be applied in an overlapping manner so that a minimum overlapping area occurs for the region of interest (e.g., see (a) of FIG. 8). When a required overlapping area is determined when processing a region of interest image using a selection image processing model having a small size, the electronic device (101) can arrange the selection image processing models in an overlapping manner so as to cover the region of interest while satisfying the required overlapping area. As another example, when the batch size of the selection image processing model is larger than the size of the region of interest, the selection image processing model can be applied so that a certain surplus area exists in the border area of the region of interest image (e.g., see (a) of FIG. 9).

According to one embodiment, in operation 630, the electronic device (101) may perform image quality processing on an image of a region of interest based on deep learning by applying a previously determined image processing model.

For example, if the size of the image processing model is similar to the size of the image of the region of interest, the electronic device (101) can position the image of the region of interest so that it is located at the center of the image processing model and perform an image processing operation on the image of the region of interest based on this. In this case, the image processing result may be as illustrated in (b) of Fig. 7.

As another example, if the size of the image processing model is smaller than the image size of the region of interest, the electronic device (101) may place the image processing model on the image of the region of interest so that the overlapping area can be minimized (e.g., see (a) of FIG. 8). The electronic device (101) may perform image processing on the image of the region of interest based on deep learning by applying the image processing model placed so that the overlapping area occurs. In this case, the image processing result may be as illustrated in (b) of FIG. 8.

As another example, if the size of the image processing model is larger than the size of the image of the region of interest, the electronic device (101) may place the image of the region of interest at the center of the image processing model so that the surplus area is uniformly distributed around the image of the region of interest (e.g., see (a) of FIG. 9). The electronic device (101) may perform image processing on the image of the region of interest based on deep learning by applying the image processing model placed so that the surplus area is uniformly distributed around the region of interest. In this case, the image processing result may be as illustrated in (b) of FIG. 9.

FIG. 7 is a diagram illustrating an example of performing image processing on a region of interest in an electronic device (101) (e.g., the electronic device (101) of FIG. 1) according to various embodiments of the present disclosure, assuming that the image processing model has a size similar to the image size of the region of interest.

Referring to FIG. 7, (a) shows an input image (710) before performing image processing on a region of interest (720) according to an embodiment, and (b) shows an input image (750) after performing image processing using deep learning with an image processing model that has a placement area (730) of a similar size to the region of interest (720). Therefore, it can be confirmed that the image quality of the region of interest image (740) included in the region of interest (720) of (a) is the same as the image quality of the image of the remaining region excluding the region of interest (720) from the entire region (710). However, in (b) where image processing is performed on the placement area (730) of the image processing model including the region of interest (720), it can be confirmed that the image quality of the region of interest image (760) is improved compared to the image quality of the image of the remaining region excluding the region of interest (720) from the entire region (750). That is, the region of interest image (760) in (b) has relatively good image quality compared to the region of interest image (740) in (a).

FIG. 8 is a diagram illustrating another example of performing image processing on a region of interest in an electronic device (101) according to various embodiments of the present disclosure.

Referring to FIG. 8, (a) shows an input image (810) before performing image processing on a region of interest (820) according to an embodiment, and (b) shows an input image (860) after performing six iterations of image processing using deep learning with an image processing model having placement areas (831, 832, 833, 834, 835, 836) smaller in size than the region of interest (820). Accordingly, it can be confirmed that the image quality of the region of interest image (840) included in the region of interest (820) of (a) is the same as the image quality of the remaining area in the entire area (810) excluding the region of interest (820). However, in (b), where image processing is performed by considering the placement area (830) of the image processing model including the area of interest (820), it can be confirmed that the image quality of the area of interest image (860) is improved compared to the image quality of the remaining area excluding the area of interest (820) in the entire area (850). That is, the area of interest image (860) in (b) has relatively better image quality than the area of interest image (840) in (a).

In (a) and (b) according to one embodiment, a part of the entire area (810, 850) of the input image is set as the area of interest (820), and image processing models (812, 813, 814, 815, 816, 817) having a smaller batch size than the area of interest (820) can be arranged to slightly overlap.

In (c) according to one embodiment, an example is shown in which image processing models (831, 832, 833, 834, 835, 836) having a batch size smaller than the region of interest (820) are arranged to overlap each other in predetermined regions (a, b, c, d, e, f, g, h, i). In this case, the electronic device (101) can perform image processing for adjusting the image quality of an image (840) of the region of interest (820) for each part based on deep learning applied to each of the image processing models (822, 823, 824, 825, 826, 827) distributed to overlap the batch areas.

In (d) according to one embodiment, when processing an image using deep learning that applies an image processing model having a smaller batch size than the region of interest (820), it may be preferable to apply it in cases where an image processing model having a similar size to the region of interest (820) should not be applied due to processing of a border portion (e.g., a border portion of the region of interest (820)) (870). The border portion (870) may correspond to, for example, an area of the processing region (830) that is not included in the region of interest (820) due to the processing region (830) being larger than the region of interest (820).

FIG. 9 is a diagram illustrating another example of performing image processing on a region of interest in an electronic device (101) according to various embodiments of the present disclosure.

Referring to FIG. 9, (a) shows an input image (910) before performing image processing on a region of interest (920) according to an embodiment, and (b) shows an input image (950) after performing image processing using deep learning with an image processing model having a placement area (930) larger than the region of interest (920). Therefore, it can be confirmed that the image quality of the region of interest image (940) included in the region of interest (920) of (a) is the same as the image quality of the image of the remaining region excluding the region of interest (920) from the entire region (910). However, in (b) where image processing is performed on the placement area (930) of the image processing model including the region of interest (920), it can be confirmed that the image quality of the region of interest image (960) is improved compared to the image quality of the image of the remaining region excluding the region of interest (920) from the entire region (950). That is, the region of interest image (960) in (b) has relatively good image quality compared to the region of interest image (940) in (a).

In (c) according to one embodiment, when processing an image using deep learning that applies an image processing model having a large batch size (930) compared to the region of interest (920), it may be desirable to apply it in cases where an image processing model having a similar size to the region of interest (920) should not be applied due to processing of a border portion (e.g., a border portion of the region of interest (920)) (970). The border portion (970) may correspond to an area of the processing region (930) that is not included in the region of interest (920), for example, due to a larger batch area (930) compared to the region of interest (920).

According to one embodiment, the placement area (930) of the image processing model can be placed so that the border portion (970) in (c) is maintained constantly at the edge of the region of interest (920). For example, if the placement area (930) of the image processing model is larger than the region of interest (920), the electronic device (101) can set the input coordinates (input X, Y) to be entered into the placement area (930) of the image processing model to be smaller than the coordinate values (X, Y coordinates) that can indicate a specific pixel at the upper left corner centered on the region of interest (920). This is because, when processing the image of the region of interest (920) based on deep learning, padding can be processed with 0 or a surrounding value if padding is required. In this case, loss or error in the board portion (970) can be prevented.

According to various embodiments, an electronic device (e.g., the electronic device (101) of FIG. 1) includes: a memory; a camera module; a communication module; and at least one processor that performs image quality processing based on deep learning on an input image provided from at least one of the memory, the camera module, or the communication module, wherein the memory may store instructions that, when executed, cause the at least one processor to set a region of interest for image quality adjustment from the input image, determine one of a plurality of image processing models having different processing sizes as a used image processing model in consideration of the usage amount of the memory and the size of the region of interest, and perform image quality processing based on deep learning by applying the used image processing model to an image of the region of interest.

According to various embodiments, the instructions may cause the at least one processor to determine the one image processing model among the plurality of image processing models by further considering the processing speed of the image of the region of interest.

According to various embodiments, the instructions may cause the at least one processor to determine the memory availability for image processing by considering the usage of the memory, and classify one or more image processing models that can be processed by the determined memory availability among the plurality of image processing models.

According to various embodiments, the instructions may cause the at least one processor to position the placement area of the selected image processing model such that a surplus area is constantly present in a border portion of the area of interest when the placement area of the selected image processing model is larger than the area of interest.

According to various embodiments, the instructions may cause the at least one processor to designate, when there are multiple image processing models having a layout area larger than the region of interest, an image processing model having a minimum surplus area with respect to the region of interest among the image processing models having a layout area larger than the region of interest as the selected image processing model.

According to various embodiments, the instructions may cause the at least one processor to apply the selected image processing model so that a surplus area is uniformly distributed in a border portion of the region of interest when the selected image processing model is determined from among image processing models having a placement area larger than the region of interest.

According to various embodiments, the instructions may cause the at least one processor to position the placement area of the selected image processing model such that an overlapping area is minimized in the region of interest when the placement area of the selected image processing model is smaller than the region of interest.

According to various embodiments, the instructions may cause the at least one processor to designate the selected image processing model from among the image processing models having a layout area smaller than the region of interest, taking into account the number of uses or the overlapping area, when there are multiple image processing models having a layout area smaller than the region of interest.

According to various embodiments, the instructions may cause the at least one processor to determine, as the selected image processing model, one image processing model having a minimum overlapping area when processing an image of the region of interest among image processing models having a layout area smaller than the region of interest.

According to various embodiments, the instructions may cause the at least one processor to determine, as the selected image processing model, one image processing model having a minimum number of uses when processing an image of the region of interest among image processing models having a layout area smaller than the region of interest.

According to various embodiments, a method for performing image quality processing based on deep learning in an electronic device (e.g., the electronic device (101) of FIG. 1) may include: setting a region of interest to be image quality processed based on the deep learning from an input image; determining one of a plurality of image processing models having different processing sizes as a used image processing model by considering the amount of memory used and the size of the region of interest; and performing image quality processing based on deep learning by applying the used image processing model to an image of the region of interest.

According to various embodiments, the operation of determining the image processing model to be used may be an operation of determining one image processing model among the plurality of image processing models by further considering the processing speed of the image of the region of interest.

According to various embodiments, the operation of determining the selected image processing model may further include an operation of determining the memory capacity for image processing by considering the memory usage, and classifying one or more image processing models that can be processed by the determined memory capacity among a plurality of image processing models.

According to various embodiments, the operation of performing the image quality processing may further include an operation of positioning the placement area of the selected image processing model so that a surplus area is constantly present in a border portion of the area of interest when the placement area of the selected image processing model is larger than the area of interest.

According to various embodiments, the operation of determining the image processing model to be used may be an operation of designating, as the selected image processing model, an image processing model having a minimum surplus area for the area of interest among the image processing models having a layout area larger than the area of interest, when there are multiple image processing models having a layout area larger than the area of interest.

According to various embodiments, when the selected image processing model is determined from among image processing models having a placement area larger than the region of interest, the selected image processing model can be applied so that a surplus area is uniformly distributed in the border portion of the region of interest.

According to various embodiments, if the placement area of the selected image processing model is smaller than the region of interest, the method may further include positioning the placement area of the selected image processing model so that an overlapping area in the region of interest is minimized.

According to various embodiments, the operation of determining the image processing model to be used may be an operation of designating the selected image processing model from among the image processing models having a placement area smaller than the region of interest, taking into account the number of uses or the overlapping area, when there are multiple image processing models having a placement area smaller than the region of interest.

According to various embodiments, the operation of determining the image processing model to be used may determine, as the selected image processing model, one image processing model that produces the minimum overlapping area when processing an image of the region of interest among image processing models having a smaller layout area than the region of interest.

According to various embodiments, the operation of determining the image processing model to be used may be an operation of determining, as the selected image processing model, one image processing model that has a minimum number of uses when processing an image of the region of interest among image processing models having a smaller layout area than the region of interest.

Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, laptops, PDAs, portable multimedia devices, and portable medical devices. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "or at least one of B," "A, B, or C," "at least one of A, B, and C," and "at least one of B or C" can each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish the corresponding component from other corresponding components, and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" as used herein may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including at least one instruction stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims (20)

In electronic devices,
memory;
camera module; and
At least one processor that performs image quality processing based on deep learning on an input image provided from the camera module,
Here, the memory is such that, when executed, the at least one processor,
Set a region of interest for image quality adjustment from the above input image,
Considering the usage of the above memory and the size of the region of interest, one of the multiple image processing models having different batch sizes is designated as the selected image processing model.
Perform image quality processing based on deep learning by applying the above-mentioned selection image processing model to the image of the above-mentioned area of interest,
An electronic device storing instructions for positioning the placement area of the selected image processing model so that a surplus area is constantly present in the border portion of the area of interest when the placement area of the selected image processing model is larger than the area of interest.

In the first paragraph,
The above instructions cause at least one processor to:
An electronic device that determines the selected image processing model among the plurality of image processing models by further considering the processing speed of the image of the region of interest.
In the first paragraph,
The above instructions cause at least one processor to:
An electronic device that determines the memory capacity for image processing by considering the usage of the above memory, and classifies one or more image processing models that can be processed by the determined memory capacity among a plurality of image processing models.
delete In the first paragraph,
The above instructions cause at least one processor to:
An electronic device that, when there are multiple image processing models having a placement area larger than the region of interest, designates an image processing model having the smallest surplus area for the region of interest among the image processing models having a placement area larger than the region of interest as the selected image processing model.
In paragraph 5,
The above instructions cause at least one processor to:
An electronic device that, when the selected image processing model is determined from among image processing models having a placement area larger than the region of interest, applies the selected image processing model so that a surplus area is uniformly distributed in the border portion of the region of interest.
In the first paragraph,
The above instructions cause at least one processor to:
An electronic device that positions the placement area of the selected image processing model so that the overlapping area in the area of interest is minimized when the placement area of the selected image processing model is smaller than the area of interest.
In the first paragraph,
The above instructions cause at least one processor to:
An electronic device that, when there are multiple image processing models having a placement area smaller than the region of interest, designates the selected image processing model among the image processing models having a placement area smaller than the region of interest by considering the number of uses or the overlapping area.
In paragraph 8,
The above instructions cause at least one processor to:
An electronic device that determines, when processing an image of the region of interest among image processing models having a placement area smaller than the region of interest, one image processing model that produces a minimum overlapping area as the selected image processing model.
In paragraph 8,
The above instructions cause at least one processor to:
An electronic device that determines, as the selected image processing model, one image processing model that has a minimum number of uses when processing an image of the region of interest among image processing models having a smaller layout area than the region of interest.
A method for performing image quality processing based on deep learning in an electronic device,
An operation of setting a region of interest for image quality processing based on the deep learning from an input image;
An operation of determining one of a plurality of image processing models having different batch sizes as a selected image processing model, taking into account the memory usage and the size of the region of interest;
An operation of performing image quality processing based on deep learning by applying the selected image processing model to the image of the above region of interest; and
A method comprising an operation of positioning the placement area of the selected image processing model so that a surplus area is constantly present in the border portion of the area of interest when the placement area of the selected image processing model is larger than the area of interest.

◈Claim 12 was waived upon payment of the registration fee.◈ In Article 11,
The operation determined by the above selection image processing model is:
A method for determining the selected image processing model among the plurality of image processing models by further considering the processing speed of the image of the region of interest.
◈Claim 13 was waived upon payment of the registration fee.◈ In Article 11,
The operation determined by the above selection image processing model is:
A method further comprising an operation of determining the memory capacity for image processing by considering the usage of the above memory, and classifying one or more image processing models that can be processed by the determined memory capacity among a plurality of image processing models.
delete ◈Claim 15 was waived upon payment of the registration fee.◈ In Article 11,
The operation determined by the above selection image processing model is:
A method for designating, when there are multiple image processing models having a placement area larger than the region of interest, an image processing model having a minimum surplus area for the region of interest among the image processing models having a placement area larger than the region of interest as the selected image processing model.
◈Claim 16 was waived upon payment of the registration fee.◈ In Article 15,
A method in which, when the selected image processing model is determined from among image processing models having a placement area larger than the region of interest, the selected image processing model is applied so that a surplus area is uniformly distributed in the border portion of the region of interest.
◈Claim 17 was waived upon payment of the registration fee.◈ In Article 11,
A method further comprising an action of positioning the placement area of the selected image processing model so that an overlapping area in the area of interest is minimized when the placement area of the selected image processing model is smaller than the area of interest.
◈Claim 18 was waived upon payment of the registration fee.◈ In Article 11,
The operation determined by the above selection image processing model is:
A method for designating a selected image processing model among image processing models having a placement area smaller than the region of interest, taking into account the number of uses or the overlapping area, when there are multiple image processing models having a placement area smaller than the region of interest.
◈Claim 19 was waived upon payment of the registration fee.◈ In Article 18,
The operation determined by the above selection image processing model is:
A method for determining, as the selected image processing model, one image processing model that produces the minimum overlapping area when processing an image of the region of interest among image processing models having a smaller layout area than the region of interest.
◈Claim 20 was waived upon payment of the registration fee.◈ In Article 18,
The operation determined by the above selection image processing model is:
A method for determining, as the selected image processing model, one image processing model that has a minimum number of uses when processing an image of the region of interest among image processing models having a smaller layout area than the region of interest.