CN120856979A - Image processing method and electronic device - Google Patents

Abstract

The embodiments of the present application provide an image processing method and an electronic device, which relate to the field of terminal technology. The problem of the difference between taking photos and previewing during the period when the non-ZSL camera mode is enabled in an electronic device is solved. The specific solution is: when the electronic device displays the first preview frame in the preview stream, the user's instruction to take photos is detected; the display content of the first preview frame is the same as the first image area in the first image captured by the electronic device, and the first image area is located at the first position of the first image; in response to the user's instruction to take photos, the electronic device captures a second image and stores the first photographed image corresponding to the second image; wherein the display content of the first photographed image is the same as the second image area in the second image, and the second image area is located at the second position of the second image; the offset between the second position and the first position is the target offset, and the target offset is the offset between the image areas presenting the same target object in the first image and the second image.

Description

Image processing method and electronic equipment

Technical Field

The embodiment of the application relates to the technical field of terminals, in particular to an image processing method and electronic equipment.

Background

With the development of electronic technology, a shooting function becomes a necessary function of an electronic device. The need for users to take photographs (photograph and/or camera) of electronic devices and their experience is also continually increasing. The electronic device may take an image using digital zoom (digitalzoom). Digital zooming refers to cropping and/or magnifying an image by a software algorithm.

Under a shooting scene with a non-zero second delay (ZSL) camera mode and a high zoom magnification, after digital zooming processing, the problem that the difference between the display content of a shooting image and the display content of a preview frame of the last frame before shooting is large easily occurs, so that a user repeatedly shoots again and again, and the man-machine interaction efficiency of the electronic equipment is affected.

Disclosure of Invention

The embodiment of the application provides an image processing method and electronic equipment, which are used for solving the problem of man-machine interaction efficiency of photographing in a non-ZSL camera mode.

In order to achieve the above purpose, the embodiment of the present application adopts the following technical scheme:

In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device.

When the electronic equipment foreground runs an application program with a shooting function, a ZSL camera mode or a non-ZSL camera mode can be adopted for shooting according to actual service requirements.

In a partial shooting scene, for example, a scene of shooting a night scene, or a scene of which the preview stream and the shooting stream have different picture formats, the electronic device can only enable the non-ZSL camera mode.

In the case of enabling the non-ZSL camera mode, the electronic device may display a preview stream collected in real time when an operation indicating photographing is not detected.

The preview stream is composed of a plurality of continuously acquired preview frames, for example, a first preview frame is included. In a scene where the zoom magnification of the electronic device is high (greater than the base magnification), the first preview frame may be an image obtained by performing digital zoom processing on the first image, that is, the display content of the first preview frame is the same as the first image area in the first image acquired by the electronic device. In addition, the first image region is located at a first position of the first image.

It can be appreciated that the first position is used to describe the position information of the first image area under the image coordinate system corresponding to the first image, and may be the image coordinate corresponding to the center point of the first image area, the image coordinate corresponding to the vertex of the first image area, and/or the image coordinate corresponding to the edge point of the first image area.

Exemplary scenario, while the electronic device displays a first preview frame in the preview stream, an operation is detected that the user indicates shooting. In this scenario, the first preview frame is the last preview frame displayed before photographing. Then, in response to the shooting operation instructed by the user, the electronic device acquires a second image and stores a first shot image, wherein the first shot image is an image obtained by performing digital zooming processing on the second image, that is, the display content of the first shot image is the same as that of a second image area in the second image, the second image area is located at a second position in the second image, and the second position may also be an image coordinate corresponding to a center point of the second image area, an image coordinate corresponding to a vertex of the second image area, an image coordinate corresponding to an edge point of the second image area, or the like. In addition, the size between the first image area and the second image area is the same.

In the embodiment of the application, the offset of the second position relative to the first position is a target offset, and the target offset is an offset between image areas of the first image and the second image which present the same target object.

The target object may be a shooting object appearing in both the first image and the second image. It will be appreciated that positional deviations of the same subject on different image data may characterize the positional transformation relationship between the first image and the second image. In the process of carrying out digital zooming on the second image, the determined second image area and the determined first image area in the first image also meet the position transformation relation, so that the display content in the second image area and the display content in the first image area are ensured to be similar, the problem that the difference between a photographed image and a preview frame of the last frame before photographing is large is solved, the consistency of pictures is ensured, and the man-machine interaction efficiency of photographing is improved.

In some embodiments, after the electronic device acquires the second image, the electronic device performs image registration based on the first image and the second image, and determines a third image area in the first image and a fourth image area in the second image, where the third image area and the fourth image area are used to display the same target object. And the electronic equipment determines the target offset according to the positions of the third image area and the fourth image area. The electronic device determines a second location based on the target offset and the first location.

In the above embodiment, the target offset between the first image and the second image is determined by means of image registration. Therefore, in the process of carrying out digital zooming on the second image, the position for actually cutting can be determined through the target offset and the first position, instead of simply and practically cutting at the first position, the obtained photographed image can be closer to the display content of the first preview frame, and the man-machine interaction efficiency of photographing is improved.

In some embodiments, the target object includes any one of the following:

(1) The object included in the first preview frame.

For example, if the shot object tracked by the lens of the electronic device is in a motion state and/or only one shot object is included in the first preview frame, the target object is the shot object included in the first preview frame. It is understood that the shot object tracked by the lens of the electronic device may be a shot object that appears in each of the consecutive multi-frame preview frames in the preview stream. Wherein the continuous multi-frame preview frame comprises a first preview frame.

(2) The occupied image area contains the subject of the center point of the first preview frame. For example, if the shot object tracked by the lens of the electronic device is in a motion state and/or the first preview frame includes only a plurality of shot objects, the target object is a shot object whose occupied image area includes the center point of the first preview frame.

(3) The object having the largest image area occupied in the first preview frame. For example, if the shot object tracked by the lens of the electronic device is in a motion state and/or only a plurality of shot objects are included in the first preview frame, the target object is the shot object with the largest image area occupied in the first preview frame.

(4) The position of the shooting object is fixed. For example, if the photographic subject tracked by the lens of the electronic device is static, the target subject is a photographic subject whose position is fixed. Such as buildings, facilities, road signs, scenery, mountains, forests, etc.

(5) The first image and the second image are both present in the still photographic subject. If the shot object tracked by the lens of the electronic equipment is static, the target object is a shot object which appears in the first image and the second image and is static. For another example, a still subject may be a subject whose relative position is unchanged in consecutive multi-frame preview frames. The relative position refers to a positional relationship with the photographic subject whose position is fixed.

In the above embodiment, the target objects selectable by the electronic device are more diversified, and in addition, different target objects can be selected according to different scenes, so that the selected target objects are more suitable for calculating the target offset, and the accuracy of the determined target offset is improved.

In some embodiments, when the electronic device acquires a second image, the electronic device acquires first posture information, inputs the first posture information and the second image into an electronic anti-shake EIS algorithm model, and combines a first position corresponding to the first image and second posture information to obtain the second position, wherein the second posture information can be the posture information acquired when the first image is acquired.

The gesture information may include gyroscope information collected by a gyroscope sensor and/or acceleration information collected by an acceleration sensor.

In the above embodiment, the second image used for making the photographed image is equivalent to an image with a later acquisition time than the first image in the preview stream, and the position of the second image area in the second image is determined by using the manner of processing the preview stream by using the EIS algorithm model.

It can be understood that the EIS algorithm model is an algorithm for processing stream data, and the second image is equivalent to an image with acquisition time later than that of the first image in the preview stream, so that the EIS algorithm model can combine related information (second pose information and first position) of the first image, determine a second position corresponding to a second image area in the second image, and enable the offset between the determined second position and the first position to be a target offset value.

In some embodiments, the electronic device clips the first view from the second image according to the second location.

Illustratively, when the second location includes edge coordinates of the second image region, the second image region is cropped in the second image to obtain the first view according to the edge coordinates in the second location.

Further illustratively, when the second location includes specific point coordinates of the second image area, the second location further includes size information of the second image area, and the electronic device may locate the second image area in the second image according to the specific point coordinates and the size information in the second location, and crop the second image area to obtain the first view.

In some embodiments, after cropping the first view, the electronic device generates a first photographed image based on the first view. For example, the electronic device may zoom in on the first view to obtain a first photographed image having the same image size as the second image. The first view map may be enlarged by upsampling the first view map.

In some embodiments, with the ZSL camera mode enabled, the electronic device captures a third image and buffers the third image. The electronic equipment displays a second preview frame corresponding to the third image, the display content of the second preview frame is the same as the fifth image area in the third image, the electronic equipment responds to the shooting operation instructed by the user, and generates and stores a second shooting image according to the cached third image, wherein the display content of the second shooting image is the same as the fifth image area in the third image.

In the embodiment, the electronic device can be compatible with scenes of photographing by enabling the ZSL camera mode and enabling the non-ZSL camera mode, and the adaptability of the scheme is improved.

In some embodiments, before storing the first photographed image corresponding to the second image, it is determined that the current zoom magnification is greater than or equal to a preset magnification.

In some embodiments, after storing the first photographed image corresponding to the second image, the electronic device configures the zoom magnification to a value smaller than the preset magnification in response to a user operation.

The electronic device then continues to display the preview stream. When the electronic device displays the third preview frame in the preview stream, the operation of the user to instruct shooting is detected again. The display content of the third preview frame is the same as a sixth image area in the fourth image, and the sixth image area is located at a third position in the fourth image.

And responding to the shooting operation instructed by the user, the electronic equipment acquires a fifth image and stores a third shooting image corresponding to the fifth image. The display content of the third photographed image is the same as that of a seventh image area in the fifth image, and the seventh image area is located at a third position in the fifth image.

Therefore, in a scene with a zoom magnification larger than a preset magnification, for example, in a scene of shooting a first shooting image, a corresponding second position of the first shooting image in the second image has an offset compared with a corresponding first position of a previous frame preview frame (for example, a first preview frame) in the first image, and the offset can ensure that the display contents of the first shooting image and the first preview frame are similar, and ensure the quality of the shooting image.

In a scene where the zoom magnification is smaller than the preset magnification, that is, in a scene where the second photographed image is photographed, the position of the second photographed image corresponding to the fifth image is the same as the position of the previous frame preview frame (e.g., the third preview frame) corresponding to the fourth image, and since the zoom magnification is relatively low, the quality of the photographed image is not affected although the offset processing is not performed.

In the above embodiment, the offset processing of the sub-scene is realized, and unnecessary processing links are reduced while the shooting quality is ensured.

In some embodiments, before the electronic device captures the second image, the electronic device further needs to configure a target camera parameter for capturing a photographed image into the camera sensor, where the target camera parameter is different from the camera parameter for capturing the preview stream.

In a second aspect, an embodiment of the application provides an electronic device comprising a processor and a memory for storing code instructions, the processor being for executing the code instructions to cause the electronic device to perform a method as described in the first aspect and any implementation thereof.

In a third aspect, embodiments of the present application provide a computer-readable storage medium storing instructions that, when executed, cause a computer to perform a method as described in the first aspect and any implementation thereof.

In a fourth aspect, embodiments of the present application provide a computer program product comprising a computer program which, when run, causes a computer to perform the method as described in the first aspect and any implementation thereof.

It should be understood that the second aspect to the fourth aspect of the embodiment of the present application correspond to the technical solutions of the first aspect of the embodiment of the present application, and the beneficial effects obtained by each aspect and the corresponding possible implementation manner are similar and are not repeated.

Drawings

Fig. 1 is a schematic structural diagram of an electronic device according to an embodiment of the present application;

fig. 2 is a schematic diagram of a software and hardware structure of an electronic device according to an embodiment of the present application;

fig. 3 is an exemplary diagram of preview stream generation and sending in ZSL camera mode;

Fig. 4 is an exemplary diagram of generation and transmission of a photographing stream in ZSL camera mode;

Fig. 5 is a diagram of an example of a scene for previewing and photographing in a non-ZSL camera mode;

fig. 6 is one of exemplary diagrams of a zoom scene provided in an embodiment of the present application:

Fig. 7 is a second exemplary diagram of a zoom scene according to an embodiment of the present application:

Fig. 8 is an example diagram of a large difference between a photographed and a previewed map caused by digital zooming in a non-ZSL camera mode in some embodiments;

FIG. 9 is a flowchart showing steps of an image processing method according to an embodiment of the present application;

fig. 10 is a diagram illustrating an example of determining offset information between image data a and image data b according to an embodiment of the present application;

FIG. 11 is an exemplary diagram of determining a position of a view field crop box for cropping a photographed image using the offset information of FIG. 10 according to an embodiment of the present application;

Fig. 12 is a second flowchart of a step of an image processing method according to an embodiment of the present application.

Detailed Description

The terms "first" and "second" are used below for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more such feature. In the description of the present embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

The implementation of the present embodiment will be described in detail below with reference to the accompanying drawings.

The embodiment of the application provides an image processing method which is applied to electronic equipment with a shooting function and is used for solving the difference between a preview image and a shooting image. The preview image is an image collected and displayed before the electronic device detects the shooting operation instructed by the user. The photographed image refers to an image acquired by the electronic device in response to an operation of the user to instruct photographing.

By way of example, the electronic device may be a desktop, laptop, tablet, handheld, cell phone, notebook, ultra-mobile Personal Computer, UMPC, netbook, and cellular telephone, personal digital assistant (Personal DIGITAL ASSISTANT, PDA), television, VR device, AR device, or the like having a camera.

As shown in fig. 1, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, keys 190, a motor 191, an indicator 192, a camera 193, a display 194, a subscriber identity module (subscriber identification module, SIM) card interface 195, and the like.

The sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

It is to be understood that the structure illustrated in the present embodiment does not constitute a specific limitation on the electronic apparatus 100. In other embodiments, electronic device 100 may include more or fewer components than shown in FIG. 1, or certain components may be combined, certain components may be split, or a different arrangement of components. The components shown in fig. 1 may be implemented in hardware, software, or a combination of software and hardware.

The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (IMAGE SIGNAL processor, ISP), a controller, a memory, a video codec, a digital signal processor (DIGITAL SIGNAL processor, DSP), a baseband processor, and/or a neural network processor (neural-network processing unit, NPU), etc. Wherein the different processing units may be separate devices or may be integrated in one or more processors.

The controller may be a neural hub and command center of the electronic device 100. The controller can generate operation control signals according to the instruction operation codes and the time sequence signals to finish the control of instruction fetching and instruction execution.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to reuse the instruction or data, it can be called directly from the memory. Repeated accesses are avoided and the latency of the processor 110 is reduced, thereby improving the efficiency of the system.

In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an integrated circuit (inter-INTEGRATED CIRCUIT, I2C) interface, an integrated circuit built-in audio (inter-INTEGRATED CIRCUIT SOUND, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver transmitter (universal asynchronous receiver/transmitter, UART) interface, a mobile industry processor interface (mobile industry processor interface, MIPI), a general-purpose input/output (GPIO) interface, a subscriber identity module (subscriber identity module, SIM) interface, and/or a universal serial bus (universal serial bus, USB) interface, among others.

It should be understood that the connection relationship between the modules illustrated in this embodiment is only illustrative, and does not limit the structure of the electronic device 100. In other embodiments, the electronic device 100 may also employ different interfaces in the above embodiments, or a combination of interfaces.

The electronic device 100 implements display functions through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display images, videos, and the like. The display 194 includes a touch layer and a display panel. The touch layer is used for sensing an interaction between a user and the display 194, and the display panel may be a Liquid Crystal Display (LCD) CRYSTAL DISPLAY, an organic light-emitting diode (OLED), an active-matrix organic LIGHT EMITTING diode (AMOLED), a flexible light-emitting diode (flex), miniled, microLed, micro-oLed, a quantum dot (QLED), or the like. The electronic device 100 may implement photographing functions through an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

The ISP is used to process data fed back by the camera 193. For example, when photographing, the shutter is opened, light is transmitted to the camera photosensitive element (image sensor) through the lens, the optical signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing, so that the electric signal is converted into an image visible to the naked eye. ISP can also optimize the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in the camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image onto the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a Complementary Metal Oxide Semiconductor (CMOS) phototransistor, such as referred to as a camera sensor. The photosensitive element converts the optical signal into an electrical signal, which is then transferred to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format. In some embodiments, electronic device 100 may include N cameras 193, N being a positive integer greater than 1.

The digital signal processor is used for processing digital signals, and can process other digital signals besides digital image signals. For example, when the electronic device 100 selects a frequency bin, the digital signal processor is used to fourier transform the frequency bin energy, or the like.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record video in a variety of encoding formats, such as moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The NPU is a neural-network (NN) computing processor, and can rapidly process input information by referencing a biological neural network structure, for example, referencing a transmission mode between human brain neurons, and can also continuously perform self-learning. Applications such as intelligent recognition of the electronic device 100, for example, image recognition, face recognition, voice recognition, text understanding, etc., can be realized through the NPU.

Fig. 2 is a schematic diagram of an architecture (including a software system and a portion of hardware) in which embodiments of the application are implemented. As shown in fig. 2, the application architecture is divided into several layers, each with distinct roles and branches. The layers communicate with each other through a software interface. In some embodiments, the application architecture may be divided into five layers, from top to bottom, an application layer, an application framework layer, a Hardware Abstraction Layer (HAL), a driver layer, and a hardware layer, respectively.

As shown in fig. 2, the application layer includes a camera application and a gallery application. It will be appreciated that some applications are shown in fig. 2, and that in fact, the application layer may include other applications, as the application is not limited in this respect. Such as applications including information, alarm clocks, weather, stopwatches, compasses, timers, flashlights, calendars, payment devices, etc.

As shown in fig. 2, the application framework layer includes a camera access interface. Camera management and camera devices are included in the camera access interface. The hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. Wherein the camera hardware abstraction layer comprises a plurality of camera nodes. The camera algorithm library comprises a post-processing algorithm module, a decision module, an image registration algorithm module, an offset processing module and an electronic anti-shake (EIS) algorithm module.

It can be understood that the image registration algorithm module, the offset processing module, and the electronic anti-shake (EIS) algorithm module are algorithm modules required for implementing the method provided by the embodiments of the present application, and the operation principle and logic thereof will be described in detail in the following embodiments, which are not repeated herein. In addition, the decision module, the image registration algorithm module, the offset processing module and the EIS algorithm module may also be at an application layer or an application framework layer, which is not particularly limited in the embodiment of the present application.

The driving layer is used for driving hardware resources. The driving layer may include a plurality of driving modules therein. As shown in fig. 2, the driving layer includes a camera device driver, a digital signal processor driver, a graphic processor driver, and the like.

The hardware layer includes a plurality of camera sensors, ISPs, digital signal processors, graphic processors, and gyro sensors (not shown in fig. 2). It will be appreciated that other hardware modules may also be included in the hardware layer, which is not specifically limited.

Wherein each camera node in the camera hardware abstraction layer corresponds to one camera sensor of the hardware layer. Illustratively, the camera application of the application layer may invoke the camera node through the camera access interface. The camera node is used for managing the corresponding camera sensor, such as setting configuration parameters of the camera sensor, controlling the camera sensor to collect images, and the like.

Exemplary scenarios, the user may click on the camera application and the electronic device may run the camera application in the foreground. The camera application sends control instructions to the camera hardware abstraction layer through the camera access interface, wherein the control instructions can carry information such as an enabled camera mode (e.g. a photographing mode, a video recording mode, a portrait mode, etc.) and zoom magnification. The camera hardware abstraction layer calls a camera algorithm library in response to the control instruction. The decision module in the camera algorithm library can determine the camera sensor (such as a target camera sensor) and configuration parameters (including a sensor image mode, parameter configuration for an ISP module and parameter configuration of a post-processing algorithm module) required to be started according to zoom magnification, camera mode and ambient light. The decision module may pass the identity and configuration parameters of the target camera sensor to the camera hardware abstraction layer.

The camera hardware abstraction layer may create a target camera node corresponding to the target camera sensor. And issuing configuration parameters to a camera device driver through the target camera node. The camera device driver sends the configuration parameters to the hardware layer, for example, sends the sensor pattern to the target camera sensor, and sends the parameter configuration of each ISP module to the ISP. Thereafter, the target camera sensor may make a map based on the sensor map. The ISP may perform the corresponding processing based on the parameter configuration for the ISP module. It will be appreciated that the configuration parameters transferred to the target camera sensor may be referred to as camera parameters, and may include Auto Exposure (AE) policies, map sizes, etc. in addition to the sensor map modes exemplified in the foregoing embodiments, and are not particularly limited herein.

The camera algorithm library is also used for sending digital signals to the digital signal processor driver in the driving layer so that the digital signal processor driver calls the digital signal processor in the hardware layer to process the digital signals. The digital signal processor may drive the processed digital signal back to the camera algorithm library through the digital signal processor.

The camera algorithm library is also used for sending digital signals to the graphic signal processor driver in the driving layer so that the graphic signal processor driver calls the graphic processor in the hardware layer to process the digital signals. The graphics processor may return the processed graphics data to the camera algorithm library via the graphics processor driver.

In addition, when the camera application foreground runs, the camera application can send a preview drawing instruction or a photo drawing instruction to the camera hardware abstraction layer through the camera access interface. Wherein the photographing drawing instruction is an instruction of the camera application to respond to the user instruction photographing operation. The preview drawing instruction is an instruction which displays a shooting preview interface provided by the camera application and is periodically generated by the camera application during an operation of not detecting the shooting instruction of the user.

After the camera hardware abstract layer receives the preview drawing instruction or the photo drawing instruction, a target camera node in the camera hardware abstract layer can be driven by a camera device to transmit the preview drawing instruction or the photo drawing instruction to a target camera sensor. The target camera sensor responds to the preview image command or the photo image command, acquires original RAW image data, transmits the data to the ISP, and converts the data into an image visible to the naked eyes of the user. The image output by the ISP may be sent to the camera device driver. The camera device driver may send the image output by the ISP to the camera hardware abstraction layer. The camera hardware abstraction layer can send the image into a post-processing algorithm module for further processing, and can also send the image into a camera access interface. The camera access interface may send the image returned by the camera hardware abstraction layer to the camera application. In this way, the camera application may draw the image and trigger the display screen to display the image.

For clarity and conciseness in the description of the embodiments below, a brief description of the related art will be given first.

(1) A zero-second delay (ZSL) is a camera mode that can make photographing more instant and responsive by reducing the delay time of a camera shutter, thereby reducing the difference between a preview image and a photographed image.

In an exemplary scenario, where the electronic device foreground runs a camera application and the ZSL camera mode is enabled, a take preview interface provided by the camera application may be displayed for displaying a preview stream from a target camera sensor, the preview stream consisting of successively acquired multiple frames of preview frames (or preview images).

Fig. 3 shows a process of generating and transmitting a preview stream in ZSL camera mode. Taking the preview stream as an example, preview frame 1, preview frame 2, preview frame 3, preview frame 4, preview frame 5, preview frame 6, and preview frame 7. As shown in fig. 3, the target camera sensor of the electronic device acquires the original image data (such as raw image) of the preview frame 1, the preview frame 2, the preview frame 3, the preview frame 4, the preview frame 5, the preview frame 6 and the preview frame 7, and each time the target camera sensor acquires one frame of original image data (such as the original image data of the preview frame 1), the original image data (such as the original image data of the preview frame 1) can be transferred to the ISP, and after the ISP processing, the preview frame (such as the preview frame 1) which can be displayed is obtained.

It will be appreciated that the electronic device may transfer the original image data of preview frame 1, preview frame 2, preview frame 3, preview frame 4, preview frame 5, preview frame 6 and preview frame 7 to the ISP frame by frame in the order of acquisition. After ISP processing, the corresponding preview frame is obtained. In this way, the electronic device can display preview frame 1, preview frame 2, preview frame 3, preview frame 4, preview frame 5, preview frame 6, and preview frame 7 one by one on the photographing preview interface.

Wherein the ISP may include an ISP first module, an ISP second module, and an ISP third module, and different modules may perform different processes on the image data. For example, the ISP first module may also include one or more of the following processes, such as binning, HDR fusion, etc. The ISP second module also includes one or more of bad point correction (bad pixelcorrection, BPC), black level correction (black level correct, BLC), lens shading correction (lens shadecorrection, LSC), automatic white balance (automatic white balance, AWB), bayer domain noise reduction (noisereduce, NR), demosaic, and the like. The ISP third module also includes one or more of Color Correction (CC), YUV domain noise reduction NR, color enhancement (color enhancer, CE), sharpening (Sharpen), tone mapping (ToneMapping), and the like.

Illustratively, after the original image data is input to the ISP, the ISP first module is used to perform Remosaic processes on the first image data. The ISP second module is used for carrying out Bayer domain processing and outputting data in RGB format. And the ISP third module is used for carrying out RGB domain or YUV domain processing and outputting YUV format data.

The above description of the ISP first module, the ISP second module and the ISP third module is only an exemplary description, and embodiments of the present application are not limited thereto.

As shown in fig. 3, the raw image data of the preview frames acquired by the target camera sensor may also be cached in the cache queue, for example, in a scenario where the target camera sensor has acquired the raw image data of preview frame 1, preview frame 2, preview frame 3, preview frame 4, preview frame 5, preview frame 6, and preview frame 7, preview frame 1, preview frame 2, preview frame 3, preview frame 4, preview frame 5, preview frame 6, and preview frame 7 may be cached in the cache queue.

As shown in fig. 4, when the electronic apparatus displays the preview frame 5, an operation of the user to instruct photographing is detected. The electronic device may fetch the raw image data of preview frame 5 from the buffer queue. Then, the original image data of the preview frame 5 is processed by an image processing algorithm module (e.g., a post-processing algorithm module) in the ISP and/or camera algorithm library, so as to obtain a photographed image corresponding to the preview frame 5, and the photographed image is stored.

The photographed image and the preview frame may multiplex the same original image data such that a difference between the preview frame displayed when the electronic device detects an operation of the user to instruct photographing and the photographed image obtained by the electronic device in response to the operation of the user to instruct photographing is small. In other words, the effect of shortening the camera shutter delay is achieved.

ZSL camera mode is a common camera mode, and in most scenarios the electronic device defaults to enabling ZSL camera mode. However, in some scenarios, the electronic device cannot enable ZSL camera mode, i.e. needs to enable non-ZSL camera mode.

For example, in a scene where the AE policies enabled by the preview and photo graphs are different, the non-ZSL camera mode needs to be enabled.

Taking a scene of night scenes of the electronic equipment as an example, in order to ensure shooting effect, the automatic exposure time is long when shooting images are acquired, for example, the automatic exposure time is 100ms. In addition, the frequency of collecting preview frames by the electronic equipment is 30fps, and accordingly, the automatic exposure time when collecting the preview frames is not more than 33ms. Obviously, during preview and photograph, AE policies configured for target camera sensors of the electronic device are different, and in this scenario, the preview frame and the photograph cannot multiplex the same original image data.

Also exemplary, the electronic device may also need to enable a non-ZSL camera mode to preview scenes in which the picture format and the photo picture format are different. It can be understood that when the preview image format and the photographing image format are different, when the target camera sensor collects the original image data of the photographing image and the preview frame, the set camera parameters are different, and the preview frame and the photographing image cannot multiplex the same original image data.

Still further exemplary, the target camera sensor is a specified type of camera sensor and the electronic device also needs to enable the non-ZSL camera mode. The camera parameters required to be configured when the camera sensor of the specified type collects the preview frames are different from the camera parameters required to be configured when the camera sensor collects the photographed images.

In some embodiments, the electronic device foreground runs a camera application and enables a non-ZSL camera mode, the target camera sensor collects original image data (such as a raw image) of the preview frame 1, the preview frame 2, the preview frame 3, the preview frame 4, the preview frame 5, the preview frame 6 and the preview frame 7, the target camera sensor transmits the collected original image data to the ISP, and after the processing of the ISP, the target camera sensor obtains a preview frame which can be transmitted and displayed, and transmits the preview frame to a display screen of the electronic device for displaying. Unlike the ZSL camera mode, in the non-ZSL camera mode, the electronic device does not buffer the original image data of the preview frame.

As shown in fig. 5, in the case where the electronic device foreground runs a camera application, the camera parameter 1 may be set in the target camera sensor. The camera parameter 1 is a camera parameter required for capturing a preview frame. Thereafter, the electronic device displays the preview frames from the target camera assembly from frame to frame. It will be appreciated that after the target camera sensor sets the camera parameter 1, the target camera sensor may collect the raw image data based on the camera parameter 1, and after the raw image data collected by the target camera sensor is processed by the image processing module such as ISP, a preview image that can be sent is obtained, for example, preview frame 1, preview frame 2, preview frame 3, preview frame 4, preview frame 5, preview frame 6 and preview frame 7.

As shown in fig. 5, in the case where the electronic apparatus displays the preview frame 7, an operation of the user to instruct shooting is detected. In response to an operation of the user to instruct shooting, a camera parameter 2 (which may also be referred to as a target camera parameter) is set in the target camera sensor. The camera parameters 2 are required for capturing a photographed image. Thereafter, the electronic device stores the captured photographic image 8. It will be appreciated that after the target camera sensor sets the camera parameters 2, the target camera sensor may collect raw image data based on the camera parameters 2, and the raw image data collected by the target camera sensor may be processed by the ISP and/or the image processing algorithm module in the camera algorithm library to obtain the photographed image 8.

After the electronic device stores the photographed image 8, the electronic device may set the camera parameters 1 again into the target camera sensor, as shown in fig. 5. Thereafter, the electronic device may continue to display preview frames from the target camera component, such as preview frame 9, preview frame 10, and so forth.

In addition, the electronic device detects an operation of the user to instruct photographing at a time point a, and the electronic device collects and generates a photographed image 8 at a time point b. Between time point a and time point b, the electronic device stops collecting preview frames, and changes the camera parameters of the target camera sensor, e.g., updates from camera parameter 1 to camera parameter 2. Then, based on the original image data acquired by the target camera sensor according to the camera parameters 2, a corresponding photographed image 8 is generated, for example, the original image data is processed by using the ISP and the camera algorithm library, and the photographed image 8 is obtained and stored. The time period between the above-described time point a and time point b, i.e., the shutter delay.

The above preview frame 7 is the last frame preview frame displayed by the electronic device before the photographed image 8 is collected, and may also be referred to as the target preview frame corresponding to the photographed image 8. Since the acquisition time of the preview frame 7 is different from that of the photographed image 8 and the interval is long, there may be a difference between the contents displayed by the preview frame 7 and the photographed image 8, and particularly, the difference may be very obvious in a scene where both the preview frame 7 and the photographed image 8 are images obtained by digital zooming.

(2) Zooming refers to changing the focal length of a lens. The zooming mode includes various modes such as optical zooming and digital zooming.

The digital zooming is to cut and/or enlarge the image by a software algorithm to achieve the same effect as changing the focal length. The optical zooming is realized by means of an optical lens structure, and the focal length of a target camera sensor is changed in a lens moving mode.

(3) The zoom magnification may indicate a change in a focal length range achieved by the target camera sensor under the optical zoom and digital zoom functions. When the imaging distance between the target camera sensor and the imaging subject is not changed, the larger the zoom magnification is, the larger the area occupied by the imaging subject on the imaging screen is. The smaller the zoom magnification, the smaller the area occupied by the subject in the photographed screen.

The base magnification is the maximum zoom magnification achievable by optical zooming. In an exemplary case, when the zoom magnification of the electronic device is less than or equal to the base magnification, the focal length of the electronic device may be adjusted by means of optical zooming. Under the condition that the zoom magnification of the electronic equipment is larger than the basic magnification, the focal length of the electronic equipment can be adjusted in a digital zooming mode.

Exemplary scenario, the base magnification of the electronic device is 2.5X. As shown in fig. 6, the electronic device displays a main interface 701. The main interface 701 includes an application icon 702 of the camera application. In response to the user clicking on the application icon 702, a shooting preview interface 703 is displayed. The shooting preview interface 703 also displays that the current zoom magnification is 1X. In the shooting preview interface 703, a preview frame acquired by the target camera sensor at a zoom magnification of 1X is displayed.

As shown in fig. 6, the shooting preview interface 703 further includes a zoom bar 704. A sliding window 705 is displayed on the zoom bar 704. It will be appreciated that different points in the zoom bar 704 correspond to different zoom magnifications. The zoom magnification indicated by the position point where the sliding window 705 overlaps the zoom bar 704 is the currently selected zoom magnification. In addition, the sliding window 705 may also display a value of the currently selected zoom magnification.

During the display of the photographing preview interface 703, the electronic device receives an operation of a user to instruct zooming. Such as a sliding operation on the zoom bar 704. The sliding operation may instruct the sliding window 705 to adjust a position point overlapping the zoom bar 704, thereby instructing to modify the zoom magnification used, and after the sliding operation by the user is finished, the electronic device may obtain the modified zoom magnification. For example, as shown in fig. 6, where the user indicates to change the selected zoom magnification from 1X to 2X, after determining that the user indicates that the enabled zoom magnification is changed to 2X, the electronic device may display the capture preview interface 706. The current zoom magnification is displayed as 2X in the shooting preview interface 706.

Correspondingly, the target camera sensor can acquire images according to the zoom magnification of 2X, and the original image data acquired by the target camera sensor is processed to obtain corresponding preview frames. In this way, the electronic device may display preview frames from the target camera sensor at the capture preview interface 706.

It will be appreciated that sliding on the zoom bar 704 is merely an example of a type of zoom operation, and embodiments of the present application are not particularly limited in terms of the form of zoom operation.

As shown in fig. 7, during the display of the shooting preview interface 706, the electronic device may also receive an operation of the user to instruct zooming. As shown in fig. 7, the electronic device may display a shooting preview interface 801 after determining that the user instructs the enabled zoom magnification to change from 2X to 5X, as indicated by the user's instruction to change the selected zoom magnification from 2X to 5X. The current zoom magnification is displayed at 5X in the shooting preview interface 801.

In addition, fig. 7 also shows a process of digital zooming. As shown in fig. 7, after configuring the zoom magnification to 5X, the target camera sensor may acquire the original image data 1 at the base magnification (2.5X), and then perform one or more image processes with respect to the original image data 1, resulting in an image 1 to be zoomed. This image to be zoomed 1 may also be referred to as an image acquired at a base magnification. In addition, the one or more image processes may include image processes performed by image processing algorithm modules in the ISP and/or camera algorithm library, such as noise reduction, RGB domain or YUV domain processing, and the like.

Then, the position of a field crop box (crop region) in the image 1 to be zoomed is determined. The view field clipping frame is used for clipping a view field image corresponding to 5X, and an image area occupied by the view field clipping frame on the image 1 to be zoomed corresponds to the view field image corresponding to 5X.

The above position may describe the position information of the view field crop frame under the image coordinate system corresponding to the image to be zoomed 1, and may represent the image area occupied by the view field crop frame on the image to be zoomed 1. For example, the position may be defined by a set of position coordinate information, such as image coordinates corresponding to a set of edge points of the field of view crop frame. Also by way of example, the above-mentioned positions may include the image coordinates of a specific point of the field of view crop frame in the image coordinate system corresponding to the image 1 to be zoomed and the size of the field of view crop frame. For example, the particular point may be the center point or vertex of the field of view crop box.

In addition, the corresponding view map under 5X can be cut out from the image 1 to be zoomed by calling a crop function. Then, the corresponding view under 5X is enlarged, so that the size of the enlarged view is the same as that of the image 1 to be zoomed, and a preview frame under 5X is obtained. The view map may be enlarged by processing the view map using an upsampling algorithm.

In a possible embodiment, the image to be zoomed 1 may also be original image data acquired by the target camera sensor, and after the electronic device cuts out the view field map under 5X from the image to be zoomed 1, one or more image processes are performed on the view field map, so as to obtain a preview frame under 5X.

In some embodiments, if the electronic device enables the ZSL camera mode under the condition that the zoom magnification of the electronic device exceeds the basic magnification, the electronic device may acquire gesture information indicating the gesture of the electronic device in the process of acquiring the original image data of the preview frame. Wherein the gesture information may include gyroscope information and/or acceleration information.

For example, corresponding gyroscope information is acquired from the gyroscope sensor, wherein the corresponding gyroscope information can include a deflection angle, a deflection speed, a deflection time point and the like of the electronic device, and the corresponding gyroscope information can be used for representing the gesture of the electronic device when the electronic device acquires the original image data of the preview frame.

For another example, corresponding acceleration information is obtained from the acceleration sensor, and the corresponding acceleration information can also be used for representing the gesture of the electronic device when the electronic device collects the original image data of the preview frame.

In addition, in the process of processing the original image data of the preview frame, the electronic device can determine the size of the view field cutting frame according to the zoom multiplying power, and input the posture information of the original image data into the EIS algorithm module to obtain the position of the view field cutting frame corresponding to the original image data.

For example, if the original image data is the first frame image acquired under the current zoom magnification, the EIS algorithm module may determine an offset 1 based on pose information of the original image data, where the offset 1 may be an offset of a center point of a field of view crop frame corresponding to the original image data relative to the center point of the original image data. Then, based on the offset 1 and the center point of the original image data, the position of the view field crop frame corresponding to the original image data can be determined.

If the original image data is not the first frame image acquired under the current zoom magnification, the EIS algorithm module may determine an offset 2 based on the pose information of the original image data, where the offset 2 may be an offset of a view field crop frame of the original image data relative to a view field crop frame corresponding to the original image data of an adjacent previous frame. Then, the position of the field of view crop frame of the original image data is determined based on the offset 2 and the positions of the field of view crop frames of the adjacent previous frame of original image data.

In this way, the electronic device performs one or more image processes on the raw image data, resulting in a processed image to be zoomed. And then obtaining a preview frame under the current zoom magnification from the image to be zoomed according to the view field clipping frame. It can be appreciated that, in the case where the current zoom magnification is greater than the base magnification, the principle of generating the preview frame at the current zoom magnification may refer to fig. 7, which is not described herein again.

In addition, the electronic device can correspondingly store the view field cutting frame position and the original image data. When the electronic device displays the preview frame, if an operation indicating shooting is received, the original image data of the preview frame can be obtained from the buffer queue, and the view field cutting frame position corresponding to the original image data is obtained. Then, based on the original image data and the field of view crop frame, a photographed image at the current zoom magnification is generated.

For example, in the case of enabling the ZSL camera mode, the electronic device captures a third image and buffers the third image. And then, the electronic equipment displays a second preview frame corresponding to the third image, wherein the display content of the second preview frame is the same as the fifth image area in the third image, in other words, the second preview frame is an image obtained by carrying out digital zooming based on the third image. The fifth image region is a region determined based on the field of view crop box position.

In addition, the third image may be raw image data acquired by the target camera sensor. Or the third image may be an image obtained after one or more image processes are performed on the original image data. And the electronic equipment responds to the shooting operation instructed by the user, and generates and stores a second shooting image according to the cached third image. The display content of the second photographed image is the same as the fifth image area in the third image. Since the second photographed image and the second preview frame are images obtained based on the same original image data (third image) and the field of view crop frame at the same position, the display contents of the two are the same.

It can be appreciated that, in the case where the current zoom magnification is greater than the basic magnification, the principle of generating a photographed image under the current zoom magnification may refer to the principle of generating a preview frame under the current zoom magnification, which is not described herein.

In some embodiments, if the electronic device enables the non-ZSL camera mode under the condition that the zoom magnification of the electronic device exceeds the basic magnification, the electronic device may also determine the size of the view field crop frame according to the zoom magnification during processing of the original image data of the preview frame, and input the pose information of the original image data into the EIS algorithm module to obtain the position of the view field crop frame corresponding to the original image data. After receiving an operation of a user to instruct shooting, the electronic device acquires original image data corresponding to a shot image, and then acquires a position of a field of view crop frame of a target preview frame (a preview frame displayed by the electronic device when the shooting operation is detected), which may also be referred to as a target position 1. The view field crop frame of the target preview frame may be a view field crop frame corresponding to original image data of the target preview frame.

In some embodiments, the electronic device may perform one or more image processing on the original image data of the photographed image, to obtain an image to be zoomed corresponding to the photographed image. Then, according to the image to be zoomed and the target position 1, a view field diagram under the current zoom magnification is cut out, and after the view field diagram is enlarged, a photographed image under the current zoom magnification is obtained.

As shown in fig. 8, the electronic device sequentially displays preview 5, preview 6, and preview 7. In the case of displaying the preview frame 7, an operation of the user to instruct shooting is detected. In response to an operation of the user to instruct photographing, the photographed image 8 is acquired and stored.

As shown in fig. 8, the preview frame 5 is a preview frame generated from the view field fig. 5. The view field fig. 5 is an image area cut out from the image 5 to be zoomed according to the view field cutting frame 5. In addition, the field of view crop box 5 is located at a position a on the image 5 to be zoomed.

The image to be zoomed 5 may be, for example, original image data acquired by a target camera sensor, and, for example, the image to be zoomed 5 may also be image data obtained by processing the original image data, for example, an image obtained by processing in an RGB domain or a YUV domain.

As shown in fig. 8, the preview frame 6 is a preview frame generated from the view field fig. 6. The view field fig. 6 is an image area cut out from the image 6 to be zoomed according to the view field cutting frame 6. In addition, the field of view crop box 6 is located at position b on the image 6 to be zoomed. The image to be zoomed 6 is similar to the image to be zoomed 5, and the acquisition time is later than the image to be zoomed 5, and the description is omitted here.

As shown in fig. 8, the preview frame 7 is a preview frame generated from the view field fig. 7. The view field fig. 7 is an image area cut out from the image 7 to be zoomed according to the view field cutting frame 7. In addition, the field of view crop box 7 is located at a position c on the image 7 to be zoomed. The image to be zoomed 7 is similar to the image to be zoomed 5, and the acquisition time is later than the image to be zoomed 5 and the image to be zoomed 6, which are not described in detail herein.

As shown in fig. 8, the photographed image 8 is an image generated from the view field fig. 8. The view field fig. 8 is an image area cut out from the image 8 to be zoomed according to the view field cutting frame 7. The field of view crop box 7 is located at position c on the image 8 to be zoomed.

It will be appreciated that in the scenario shown in fig. 8, the field crop box corresponding to the photographed image inherits the target preview (i.e., the last frame preview frame displayed before the photographing operation is detected, e.g., preview frame 7). In a scene in which the non-ZSL camera mode is started, due to the longer acquisition time interval between the image 8 to be zoomed and the image 7 to be zoomed, the larger difference of the picture content is caused, and the larger difference exists between the view field diagram 8 cut out by the view field cutting frame 7 and the view field diagram 7 cut out by the view field cutting frame 7, so that the larger difference between the preview and the actual shooting diagram is caused.

In order to solve the above problems, an embodiment of the present application provides an image processing method applied to an electronic device. Under the condition that the electronic equipment starts a non-ZSL camera mode and the zoom magnification exceeds the basic magnification, the difference between a photographed image and a target preview frame can be improved through the method.

In some embodiments, as shown in fig. 9, the above image processing method may include the following steps:

s101, when the current zoom magnification is larger than the base magnification, displaying a preview stream from the target camera sensor.

In some embodiments, when the electronic device foreground runs a camera application (or other application program with shooting function), a shooting preview interface provided by the application program can be displayed. Wherein the shot preview interface may be used to display a preview stream from the target camera sensor. The preview stream includes a plurality of frames of preview frames collected in succession. In the case where the current zoom magnification is larger than the base magnification, the preview frame may be an image obtained by digital zooming. The electronic device may cache the image to be zoomed corresponding to the preview frame and corresponding related information.

For example, the related information may include gesture information when a high-resolution zoom image is acquired. For another example, the related information may further include position information of a view field cropping frame used in the digital zooming process based on the image to be zoomed.

S102, in the case of displaying the preview frame a, an operation of the user to instruct shooting is detected.

For example, the above-described operation of indicating photographing may be an operation in which the user clicks a photographing control in the photographing preview interface. Also, for example, the above-described operation of instructing photographing may also be a voice instruction by which the user speaks the instruction of instructing photographing. Still further, for example, the above-described operation of instructing photographing may also be that the user makes a gesture or the like of instructing photographing.

The preview frame a (for example, referred to as a first preview frame) is an image obtained by performing digital zoom processing based on the image data a (first image). The image data a may be an image to be zoomed corresponding to the preview frame a.

The image data a may be raw image data acquired by a target camera sensor, or may be an image obtained by processing one or more images of the raw image data. It is to be appreciated that the one or more image processing may include noise reduction, RAW domain processing, YUV domain processing, and the like, which are not particularly limited in embodiments of the present application. In the subsequent embodiments, description will be made with the image data a referring to an image after image processing.

In addition, the preview frame a is a preview frame displayed by the electronic device when an operation of the user to instruct photographing is detected, and may also be referred to as a target preview frame of a photographed image.

S103, in response to the shooting operation instructed by the user, image data b for making a shooting image is acquired.

In some embodiments, the electronic device may configure the target camera sensor with camera parameters suitable for capturing a photographed image in response to a user indicating a photographed operation. The camera parameters may include AE policy, graph format, graph size, and other parameters. The electronic device then obtains image data b (which may also be referred to as a second image) via the target camera sensor. The image data b may be an image to be zoomed for producing a first frame photographed image. For example, if the image data a is raw image data, the image data b may be first frame raw image data acquired by the target camera sensor in response to a photographing operation by the user. Also, for example, if the image data a is a processed image, the image data b may be an image obtained by performing one or more image processes on the first frame of original image data. In the subsequent embodiments, description is made with the image data b being an image obtained through image processing.

And S104, performing image registration on the image data a corresponding to the preview frame a and the acquired image data b to obtain offset information.

In some embodiments, the position of the same object of interest (also may be referred to as target object) between image data a and image data b is determined by image registration. Offset information is determined from the position of the object of interest in the image data a and the position of the object of interest in the image data b.

As shown in fig. 10, by image registration, it is determined that an image region 1101 (which may also be referred to as a third image region) in image data a and an image region 1102 (which may also be referred to as a fourth image region) in image data b are image regions presenting the same object of interest. Then, the image region 1101 is projected into the image coordinate system 1103 according to the image coordinates of the image region 1101 in the image data a. The image region 1102 is projected into an image coordinate system 1103 according to the image coordinates of the image region 1102 in the image data b. In the image coordinate system 1103, offset information between the image region 1101 and the image region 1102 is determined. For example, in the image coordinate system 1103, a vector 1104 between the center point of the image region 1101 and the center point of the image region 1102 is used as the corresponding offset information.

The above object of interest may be, for example, a photographic object appearing in the preview frame a. For example, in the case where a plurality of subjects appear in the preview frame a, the subject having the largest image area among the subjects may be determined as the subject of interest. For another example, in the case where a plurality of shot objects appear in the preview frame a, the object of interest may be a shot object whose occupied image area contains the center point of the preview frame a. For another example, in the case where a plurality of photographic subjects appear in the preview frame a, the object of interest may be a photographic subject that appears in each of the consecutive multi-frame preview frames.

Also exemplary, the object of interest may also be a still photographic object in the image data a. For example, the still photographic subject may be a photographic subject whose position is fixed, such as a building, a facility, a road sign, a scene, a mountain peak, a forest, or the like. For another example, a still subject may be a subject whose relative position is unchanged in consecutive multi-frame preview frames. The relative position refers to a positional relationship with the photographic subject whose position is fixed.

In some embodiments, if the electronic device recognizes that the shot object tracked by the lens is in a motion state, the shot object in the preview frame a may be determined as the object of interest. The electronic device may identify, as the shot object for shot tracking, the shot objects that appear in the preview frame a and the multi-frame preview frame that is continuous with the preview frame a. In other embodiments, if the electronic device recognizes that the shot object tracked by the lens is stationary, any still shot object in the image data a may be determined as the object of interest.

In some embodiments, S104 may be performed using an image registration algorithm module in a camera algorithm library.

As one implementation, the image registration algorithm module may be feature-based registration. For example, by detecting feature points (such as corner points, edges, and the like) in the image data a and the image data b, an image area indicating the same photographic subject between the image data a and the image data b is determined by matching these feature points. Offset information, such as a target offset, may then be determined based on the display positions of the same object of interest in image data a and image data b.

Feature detection may be implemented, for example, by SIFT (scale invariant feature transform), SURF (speeded up robust features), or (oriented fAST and rotated brief, ORB) algorithms.

As another implementation, the image registration algorithm module may be region-based registration (Area-Based Registration). For example, a similarity measure such as Cross-Correlation (Cross-Correlation) or mutual information (Mutual Information) is generally used to evaluate the similarity between image areas, thereby determining an image area between image data a and image data b indicating the same photographic subject. This method does not require the feature points to be detected in advance and can be operated on over the entire image area.

As another implementation, the image registration algorithm module may be a Model-based registration (Model-Based Registration). For example, the image is assumed to be an instance of some kind of parameterized model, such as affine transformation, perspective transformation, etc. The alignment of the images is achieved by optimizing the model parameters.

As another implementation, the image registration algorithm module may be a depth learning based registration (DEEP LEARNING-Based Registration).

As another implementation, the image registration algorithm module may be a Multi-point registration (Multi-Point Registration). When there are a plurality of corresponding points in the image data a and the image data b, the multi-point registration method determines an image area between the image data a and the image data b indicating the same photographic subject by considering these points at the same time.

As another implementation, the image registration algorithm module may employ rigid (Rigid) registration, affine (Affine) registration, and Non-rigid (Non-Rigid) or Elastic (Elastic) registration. Where rigid registration assumes that the display position transformation of the same subject between images involves only rotation and scaling, affine and non-rigid registration may handle more complex transformations such as translation, rotation, scaling, warping, etc. The choice of image registration algorithm depends on the specific application scenario, image characteristics and the required accuracy. In practical applications, the most suitable registration method may need to be selected from the rigid (Rigid) registration, affine (Affine) registration, non-rigid (Non-Rigid) registration, and Elastic (Elastic) registration, according to the content, quality, transformation type, and other factors of the image.

S105, acquiring the target position 1 of the view field cutting frame corresponding to the preview frame a.

It will be appreciated that the target position 1 (also referred to as a first position) may be used to indicate the position of the field of view crop frame on the image data a, through which the view map corresponding to the preview frame a (target preview frame) may be cropped from the image data a. The image area covered by the view field crop frame on the image data a may be referred to as a first image area, and the size and display content of the first image area are the same as those of the view field map of the preview frame a. In the following embodiment, the target position 1 may also be referred to as the position of the field of view crop frame for cropping the preview frame a.

In some embodiments, the electronic device may cache the target position 1 after determining the target position 1 and cropping out the view field map corresponding to the preview frame a.

In one implementation manner, in the process of manufacturing the preview frames, the electronic device determines the position information of the view field cutting frame corresponding to each frame of preview frame, and stores the position information of the view field cutting frame into a preset queue. The manner of determining the position of the view field cutting frame may refer to the foregoing embodiment, which is not described herein. In addition, the position information in the preset queue is arranged according to the sequence of the storing time points. When the user shooting operation is detected, the position information with the latest storing time can be taken out from the preset queue as the target position 1.

In one implementation manner, in the process of manufacturing the preview frames, the electronic device determines the position information of the view field cutting frame corresponding to each frame of preview frame, and writes the position information into a preset storage position. In the preset storage location, the newly written location information replaces the original location information. In this way, upon detecting an operation of shooting by the user, the position information can be taken out from the preset storage position as the target position 1.

S106, determining the target position 2 according to the target position 1 and the offset information.

In some embodiments, both S105 and S106 may be performed by an offset processing module. It will be appreciated that the target position 2, which may also be referred to as a second position, is the determined position of the field of view crop box on the image data b. The field of view crop box located at the target location 2, the image area covered on the image data b may be referred to as a second image area.

For example, if the target position 1 includes a set of position coordinate information, each position coordinate information corresponding to the target position 1 may be moved according to the offset information, to obtain position coordinate information corresponding to the target position 2. The edge of the image area to be cut out on the image data b can be determined from the position coordinate information contained in the target position 2.

For example, if the target position 1 includes the center point coordinate (or the vertex coordinate) and the size of the view clipping frame, the electronic device may move the center point coordinate (or the vertex coordinate) of the target position 1 according to the offset information to obtain the center point coordinate (or the vertex coordinate) corresponding to the target position 2. In addition, target location 2 also includes a size of a field of view crop box that is related to the current zoom magnification, in some examples, the size of the field of view crop box in target location 2 may be the same as the size of the field of view crop box in target location 1. Thus, by the center point coordinates (or vertex coordinates) included in the target position 2 and the size of the field of view trimming frame, the image area to be trimmed on the image data b can be determined.

As shown in fig. 11, the target position 1 of the field crop box 1201 corresponding to the preview frame a is acquired. The manner of acquiring the target position 1 may refer to S104, which is not described herein.

It is to be understood that the preview frame a is a preview frame displayed when a user photographing operation is detected, and may also be referred to as a target preview frame. The location of the field crop box 1201 corresponding to preview frame a may also be referred to as target location 1. The view map a can be cropped from the image data a by a view crop box 1201 located at the target position 1. After the up-sampling process is performed on the view map a, a preview frame a having the same size as the image data a is obtained.

As illustrated in fig. 11, the target position 2 of the field of view crop box 1202 is determined from the target position 1 of the field of view crop box 1201 and the offset information (vector 1104 as determined in fig. 10).

S107, a photographed image is generated from the target position 2 and the image data b.

In some embodiments, S107 described above may be performed by an ISP.

As shown in fig. 11, a view b (e.g., referred to as a first view) is cropped from image data b according to a view cropping box 1202. Wherein the size and display content of the view field map b are the same as those of the second image area. After the up-sampling process is performed on the view b, a photographed image, for example, a first photographed image, having the same size as the image data b is obtained.

After determining the offset information between the image data a and the image data b through image registration, as shown in fig. 11, the position of the view field clipping frame for clipping the photographed image is determined on the image data b according to the position of the view field clipping frame corresponding to the image data a and the offset information, so that the difference between the finally obtained photographed image and the preview image a (target preview frame) is reduced.

It will be appreciated that, when the preview frame a is displayed, the user makes an operation of instructing photographing, which means that the photographed image the user desires to collect is the same as the preview frame a. However, in the case where the non-ZSL camera mode is enabled, the acquisition time of the original image data a for making the preview frame a and the original image data b for making the photographed image are different. Not only the subject position (e.g., the subject is moving) but also the position of the electronic device (e.g., the user holds the electronic device with a shake) may change during the time interval between the acquisition of the raw image data a and the raw image data b. Of course, whether the shooting object or the position of the electronic equipment changes, consistency between the shooting picture and the preview frame a can be ensured through an image registration mode, probability of re-shooting of a user is reduced, and under the condition of enabling a non-ZSL camera mode, man-machine interaction efficiency of shooting is improved, and user experience is improved.

In other scenes, when the shooting operation is an operation for indicating to shoot a dynamic picture, the electronic equipment responds to the shooting operation and can continuously acquire and generate a plurality of frames of shooting images to form the dynamic picture.

In this scenario, first, the electronic device generates a first frame of photographed image according to S103 to S107.

Secondly, after detecting the shooting operation, the electronic device collects the second frame of original image data through the target camera sensor, and after performing one or more image processes on the second frame of original image data, image data c, that is, an image to be zoomed corresponding to the second frame of shooting image, can be obtained. In addition, the electronic device may acquire the posture information 1 of the electronic device when acquiring the second frame of original image data. Similarly, when the electronic device collects the first frame of original image data, the corresponding gesture information 2 is also acquired and stored. Then, the image data c and the corresponding pose information 1 are input to the EIS algorithm module. The EIS algorithm module determines a target position 3 of the view field cutting frame on the image data c according to the target position 2, the gesture information 2, the image data c and the gesture information 1. Then, a view field map c is cut out from the image data c according to a view field cutting frame located at the target position 3, and a second frame photographed image is generated based on the view field map c.

And after detecting the shooting operation, the electronic device continues to acquire the third frame of original image data, and performs one or more image processes on the third frame of original image data to obtain image data d, namely an image to be zoomed corresponding to the third frame of shooting image.

When the third frame of original image data is acquired, the corresponding posture information 4 is acquired. The image data d and the pose information 4 are input to an EIS algorithm module. The EIS algorithm module determines a target position 5 based on the image data d, the pose information 4, the pose information 1, and the target position 3. Based on the view field clipping frame corresponding to the target position 5, a view field map d is clipped from the image data d. And acquiring a third frame of photographed image according to the view field d, and so on, after the electronic device acquires the original image data of the ith frame, the same manner can be adopted to acquire the ith frame of photographed image, wherein i can be a positive integer greater than 2. The generated multi-frame photographed image can constitute a dynamic photograph.

In some embodiments, as shown in fig. 12, the above image processing method may include the following steps:

s201, when the current zoom magnification is larger than the base magnification, a preview stream from the target camera sensor is displayed.

S202, in the case of displaying the preview frame a, an operation of the user to instruct shooting is detected.

S203, in response to an operation of the user to instruct photographing, image data b for making a photographed image is acquired.

In some embodiments, the implementation details of S201 to S203 may refer to S101 to S103, which are not described herein.

S204, the posture information 2 corresponding to the image data b is acquired.

The gesture information 2 may be referred to as first gesture information, and includes gyroscope information collected by a gyroscope sensor and/or acceleration information collected from an acceleration sensor when the target camera sensor collects original image data of the image data b.

S205, inputting the image data b and the posture information 2 into an EIS algorithm module, and combining the related information corresponding to the image data a to obtain the target position 4 of the view field cutting frame for cutting the photographed image.

In some embodiments, in S205 described above, image data b is equivalent to the next frame of image in the preview stream that is acquired later in time than image data a. In this way, even if a certain acquisition time interval exists between the image data b and the image data a, the EIS algorithm module can determine the target position 4 of the view clipping frame based on the image data b, the posture information 2 and the related information corresponding to the image data a.

The above-described related information may include, for example, pose information 3 (also may be referred to as second pose information) of the image data a and the target position 1. The gesture information 2 may represent a gesture 1 when the electronic device collects the image data b, and the gesture information 3 may represent a gesture 2 when the electronic device collects the image data a. The EIS algorithm module determines a jitter condition (e.g., a jitter amplitude) between the electronic device from the captured image data a to the captured image data b through the difference between the pose 1 and the pose 2, and then, the EIS algorithm module may estimate the target position 4 (i.e., the second position determined by the EIS algorithm module) according to the target position 1 and the jitter condition.

S206, generating a photographed image according to the target position 4 and the image data b.

In some embodiments, the implementation details of S206 may refer to S107 in the foregoing embodiments, which is not described herein.

In addition, in a scene where the photographing operation is an operation indicating photographing of a moving picture, after the photographing operation is detected, first-frame original image data is acquired, and one or more image processing operations are performed on the first-frame original image data, resulting in image data b. The electronic equipment generates a first frame photographing image according to the image data b according to S204-S205. The electronic device acquires the second frame of original image data through the target camera sensor, and performs one or more image processes on the second frame of original image data to obtain image data c. The electronic device may acquire the posture information 1 indicating the posture of the electronic device when acquiring the second frame of original image data. The electronic device inputs the image data c and the corresponding pose information 1 into an EIS algorithm module. And the EIS algorithm module determines a target position 3 corresponding to the view field cutting frame according to the target position 4, the gesture information 2, the image data c and the gesture information 1. Then, according to the view field cutting frame positioned at the target position 3, a view field map c is cut out from the image data c, a second frame of photographed image is generated based on the view field map c, and the like, so that a plurality of frames of photographed images can be obtained to form a dynamic photo.

In some embodiments, a preset magnification is preconfigured in the electronic device, the preset magnification being greater than the base magnification. In a scenario where the electronic device enables the non-ZSL camera mode, the method illustrated in fig. 9 or 12 may be performed in a case where the current zoom magnification is greater than or equal to a preset magnification.

In the case where the current zoom magnification is smaller than the preset magnification, after the image data b is acquired, the target position 1 of the field of view clipping frame for clipping the preview frame a is acquired. A field of view crop box at the target position 1 is determined on the image data b. And cutting out a view field diagram b from the image data b according to the view field cutting frame of the target position 1, and then processing the view field diagram b to obtain a photographed image with the same image size as the image data b.

For example, in a scenario in which the non-ZSL camera mode is enabled, the electronic device may reconfigure the zoom magnification of the electronic device in response to a user indicating a zoom operation. If the zoom magnification of the electronic equipment is smaller than the preset magnification, the electronic equipment collects a fourth image and displays a third preview frame corresponding to the fourth image. The display content of the third preview frame is the same as the sixth image area in the fourth image, and the sixth image area is located at a third position in the fourth image, in other words, the view corresponding to the third preview frame is cropped from the fourth image based on the view cropping frame at the third position.

During the display of the third preview frame, in response to an operation of the user to instruct photographing, the electronic device acquires a fifth image and stores a third photographed image corresponding to the fifth image. The display content of the third photographed image is the same as that of a seventh image area in the fifth image, and the seventh image area is located at a third position in the fifth image. That is, the view corresponding to the third photographed image is cropped from the fifth image based on the view cropping frame at the third position.

Therefore, when the zoom magnification is smaller than the preset magnification, the position of the view field cutting frame corresponding to the target preview frame can be used to obtain a photographed image from the fifth image.

It can be appreciated that the larger the zoom magnification, the smaller the size of the field of view crop box. The smaller the zoom magnification, the larger the size of the field of view crop box. In the non-ZSL camera mode, the smaller the size of the view field crop frame, the greater the influence of the position of the view field crop frame on the content displayed by the photographed image, so that the difference between the target preview frame and the photographed image may be greater, exceeding the user acceptable level. The larger the size of the view field cutting frame is, the smaller the influence on the content displayed by the photographed image is, and the smaller the difference between the target preview frame and the photographed image is. The preset multiplying power corresponds to a target size. If the view field cutting frame is smaller than the target size, the position of the view field cutting frame can obviously influence the content displayed by the photographed image. If the field of view crop box is greater than or equal to the target size, the location of the field of view crop box affects what is displayed in the photographed image to an acceptable extent. The preset multiplying power can be obtained through an empirical value obtained through testing, and the corresponding preset multiplying powers of the electronic devices with different basic multiplying powers can be different.

The methods shown in fig. 9 and 12 are described in the foregoing embodiments as applied to a scene where a single photographed image is photographed and a moving picture is photographed. The above method may also be applied to a scene of capturing video, and the specific implementation process may refer to the scene of capturing and generating a moving picture described in the foregoing embodiment, which is not described herein.

Some embodiments of the application also provide an electronic device that may include a memory and one or more processors. The memory is coupled to the processor. The memory is for storing computer program code, the computer program code comprising computer instructions. When the processor executes the computer instructions, the electronic device may perform the various functions or steps performed by the electronic device in the method embodiments described above.

The embodiment of the application also provides a computer readable storage medium, which comprises computer instructions, when the computer instructions run on the electronic device, the electronic device is caused to execute the functions or steps executed by the mobile phone in the embodiment of the method.

The embodiment of the application also provides a computer program product which, when run on an electronic device, causes the electronic device to execute the functions or steps executed by the mobile phone in the above method embodiment.

It will be apparent to those skilled in the art from this description that, for convenience and brevity of description, only the above-described division of the functional modules is illustrated, and in practical application, the above-described functional allocation may be performed by different functional modules according to needs, i.e. the internal structure of the apparatus is divided into different functional modules to perform all or part of the functions described above.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical functional division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another apparatus, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and the parts displayed as units may be one physical unit or a plurality of physical units, may be located in one place, or may be distributed in a plurality of different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or a part contributing to the prior art or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, including several instructions for causing a device (may be a single-chip microcomputer, a chip or the like) or a processor (processor) to perform all or part of the steps of the method described in the embodiments of the present application. The storage medium includes a U disk, a removable hard disk, a Read Only Memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.

The foregoing is merely illustrative of specific embodiments of the present application, but the scope of the present application is not limited thereto, and any changes or substitutions within the technical scope of the present application should be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims (13)

1. An image processing method, characterized in that the method further comprises: When a non-zero-second delay ZSL camera mode is enabled, the electronic device displays a preview stream; When the electronic device displays a first preview frame in the preview stream, detecting a user instruction to shoot; the display content of the first preview frame is the same as a first image area in a first image captured by the electronic device, and the first image area is located at a first position of the first image; In response to a user-instructed shooting operation, the electronic device captures a second image and stores a first photographic image corresponding to the second image; wherein the display content of the first photographic image is the same as the second image area in the second image, and the second image area is located at the second position of the second image; the offset between the second position and the first position is the target offset, and the target offset is the offset between the image areas presenting the same target object in the first image and the second image. 2. The method according to claim 1, wherein after the electronic device captures the second image, the method further comprises: The electronic device performs image registration based on the first image and the second image to determine a third image area in the first image and a fourth image area in the second image, where the third image area and the fourth image area are used to display the same target object; The electronic device determines the target offset according to positions of the third image area and the fourth image area; The electronic device determines the second position according to the target offset and the first position. 3. The method according to claim 1 or 2, wherein the target object includes any one of the following: a photographic subject included in the first preview frame; the photographed object whose occupied image area includes the center point of the first preview frame; the photographed object occupying the largest image area in the first preview frame; The photographed subject is fixed in position; The photographed object appears in both the first image and the second image and is in a static state. 4. The method according to claim 3, characterized in that If the subject tracked by the lens of the electronic device is in motion and the first preview frame contains only one subject, the target object is the subject contained in the first preview frame; If the subject tracked by the lens of the electronic device is in motion and the first preview frame contains multiple subjects, the target subject is the subject whose image area includes the center point of the first preview frame, or the subject that occupies the largest image area in the first preview frame; If the subject tracked by the lens of the electronic device is static, the target object is the subject whose position is fixed, or the subject that appears in both the first image and the second image and is static. 5. The method according to claim 1, wherein when the electronic device captures the second image, the method further comprises: The electronic device acquires first posture information; The electronic device inputs the first posture information and the second image into an electronic image stabilization (EIS) algorithm model, and obtains the second position by combining the first position and the second posture information corresponding to the first image, wherein the second posture information may be the posture information obtained when the first image is captured. 6. The method according to claim 2 or 5, further comprising: The electronic device crops a first field of view image from the second image according to the second position; The electronic device generates the first photographic image based on the first field of view image. 7. The method according to any one of claims 1 to 6, further comprising: When the ZSL camera mode is enabled, the electronic device captures a third image and caches the third image; The electronic device displays a second preview frame corresponding to the third image, where display content of the second preview frame is the same as the fifth image area in the third image; In response to a user-instructed shooting operation, the electronic device generates and stores a second photographic image based on the cached third image; wherein the display content of the second photographic image is the same as the fifth image area in the third image. 8. The method according to any one of claims 1 to 6, characterized in that before storing the first photographed image corresponding to the second image, the method further comprises: Make sure the current zoom ratio is greater than or equal to the preset ratio. 9. The method according to claim 8, characterized in that after storing the first photographed image corresponding to the second image, the method further comprises: The electronic device configures the zoom ratio to a value smaller than the preset ratio in response to a user operation; The electronic device displays the preview stream; When the electronic device displays a third preview frame in the preview stream, a user instruction to shoot is detected; wherein the display content of the third preview frame is the same as a sixth image area in a fourth image captured by the electronic device, and the sixth image area is located at a third position in the fourth image; In response to a user-instructed shooting operation, the electronic device captures a fifth image and stores a third photographic image corresponding to the fifth image; wherein the display content of the third photographic image is the same as the seventh image area in the fifth image, and the seventh image area is located at the third position in the fifth image. 10. The method according to claim 1, wherein before the electronic device captures the second image, the method further comprises: The electronic device configures target camera parameters for capturing photographic images into a camera sensor, wherein the target camera parameters are different from camera parameters for capturing a preview stream. 11. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10. 12. A computer-readable storage medium having a computer program/instruction stored thereon, wherein the computer program/instruction, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10. 13. A computer program product, comprising a computer program/instruction, wherein the computer program/instruction implements the steps of the method according to any one of claims 1 to 10 when executed by a processor.