CN116049868A - Privacy protection method, device and storage medium - Google Patents

Abstract

The application provides a privacy protection method, privacy protection equipment and a storage medium. According to the method, the collected image data are subjected to face recognition, identity recognition and living body detection, and when the living body face exists in the surrounding environment, the user is reminded of paying attention to privacy risks, so that leakage of user information can be reduced, frequent prompt of the face in surrounding posters, photos, videos and the like can be avoided, and user experience is improved.

Description

Privacy protection method, device and storage medium

technical field

The present application relates to the field of electronic technology, and in particular to a privacy protection method, device and storage medium.

Background technique

With the development of science and technology, terminal equipment has more and more functions, and terminal equipment occupies an increasingly important position in people's daily life and work. Taking mobile phones as an example, users can not only watch videos and play games for entertainment through mobile phones, but also can send and receive emails and handle work through mobile phones, and conduct instant messaging activities through mobile phones.

But at the same time, the leakage of user privacy information is becoming more and more serious, especially in public places. When users use their mobile phones to perform some privacy and security-related operations, such as viewing work files and personal accounts, if they do not pay attention, the mobile phone The content displayed on the screen will be seen by other people, causing information leakage and causing unnecessary trouble and distress to users.

Contents of the invention

In order to solve the above technical problems, this application provides a privacy protection method, device and storage medium, which aims to remind the user to pay attention to privacy risks only when a living human face is identified in the surrounding environment, which can not only reduce the leakage of user information, but also Avoid frequent reminders of faces in surrounding posters, photos, videos, etc., thereby improving user experience.

In a first aspect, the present application provides a privacy protection method. The method is applied to a terminal device including a front camera, and the method includes: when a set trigger mechanism is satisfied, controlling the front camera to collect image data; performing face recognition on the image data, and identifying all faces included in the image data ;Identify each identified face to determine whether there is a face of a non-registered user, and the non-registered user indicates a user who has not obtained the authorization to use the terminal device; when there is a face of a non-registered user, according to the image data Liveness detection is performed on the face of each non-registered user, and the liveness detection is used to determine whether the face is the face of a person with vital signs; when there is a face of a non-registered user who is alive, the terminal device is controlled to take privacy protection risks Prompt, the privacy protection risk prompt is used to prompt the user that there is a risk of privacy leakage.

Among them, the trigger mechanism is, for example, any one or more of the trigger mechanism 1, trigger mechanism 2, and trigger mechanism 3 mentioned below. For the selection requirements of trigger mechanism 1, trigger mechanism 2, and trigger mechanism 2, please refer to the following. I won't repeat them here.

Wherein, the front camera may be integrated in the terminal device, or external to the terminal device. In this way, for a terminal device without a front-facing camera, the privacy protection method provided by the present application can also be implemented through an external camera, thereby realizing an anti-peeping scene.

Therefore, by performing face recognition, identity recognition, and liveness detection on the collected image data, the user is reminded to pay attention to privacy risks when a live face is recognized in the surrounding environment, thereby reducing the leakage of user information and ensuring Avoid frequent prompts due to faces in surrounding posters, photos, videos, etc., to improve user experience.

According to the first aspect, the front camera is an infrared depth camera, the image data includes two-dimensional image data and three-dimensional image data, the two-dimensional image data is an infrared image, and the three-dimensional image data is a depth image.

In this way, the processor of the terminal device does not need to process the image data, and processes the image data including only two-dimensional image data into three-dimensional image data.

According to the first aspect, or any implementation of the above first aspect, the front camera is a D-RGB camera, the image data includes two-dimensional image data and three-dimensional image data, the two-dimensional image data is an RGB image, and the three-dimensional image data is depth image.

In this way, the processor of the terminal device does not need to process the image data, and processes the image data including only two-dimensional image data into three-dimensional image data.

According to the first aspect, or any implementation of the above first aspect, the front camera is an RGB camera, the image data includes two-dimensional image data, and the two-dimensional image data is an RGB image; The image is processed to generate a depth image, and the depth image is used as three-dimensional image data.

In this way, even if the front camera of the terminal device can only collect RGB images, depth images can also be obtained through deep learning processing, thereby ensuring that subsequent liveness detection can be performed, so that the privacy protection method provided by this application can cover more terminals The device, such as the terminal device that has been shipped but not equipped with a D-RGB front camera, or an infrared depth front device.

According to the first aspect, or any implementation of the above first aspect, face recognition is performed on the image data, and all faces included in the image data are identified, including: performing face recognition on the two-dimensional image data, and identifying All faces included in the image data; identify each recognized face and determine whether there is a face of a non-registered user, including: identify each recognized face based on the two-dimensional image data , determine whether there is a face of a non-registered user; perform liveness detection on the face of each non-registered user according to the image data, including: perform liveness detection on the face of each non-registered user based on two-dimensional image data and three-dimensional image data .

According to the first aspect, or any implementation method of the above first aspect, according to the two-dimensional image data, the identification of each recognized face is performed, including: extracting the person of each face from the two-dimensional image data Face feature information; match the face feature information of each face with the pre-registered user's face and physique information, and the registered user indicates the user who has obtained the right to use the terminal device; if it does not match, it is determined that there is a non-registered user human face.

According to the first aspect, or any implementation of the above first aspect, liveness detection is performed on the face of each non-registered user according to the two-dimensional image data and the three-dimensional image data, including: extracting each face from the two-dimensional image data The reflectivity information of the face of the non-registered user; extract the three-dimensional information of the face of each non-registered user from the three-dimensional image data; when the three-dimensional information and the reflectance information meet the living body standard, determine that there is a living non-registered user human face.

According to the first aspect, or any implementation of the above first aspect, the image data includes two-dimensional image data and three-dimensional image data, the two-dimensional image data is an RGB image or an infrared image, and the three-dimensional image data is a depth image; Before performing liveness detection on the face of each non-registered user, the method also includes: determining the distance between the face of each non-registered user and the screen of the terminal device according to the three-dimensional image data; when the distance is less than the set safety distance, Execute the step of performing liveness detection on the face of each non-registered user according to the image data.

In this way, before liveness detection, a distance detection link is introduced. If the non-registered user is farther than the set safe distance from the screen of the terminal device, it is considered that the current usage scenario is safe and there is no risk of information leakage. When there are more than one, the range of liveness detection can be narrowed based on the distance, thereby reducing the occupation of terminal device resources, reducing power consumption and improving efficiency.

According to the first aspect, or any implementation of the above first aspect, before controlling the terminal device to make a privacy protection risk prompt, the method further includes: performing eye movement tracking on the registered user who is a living body according to the image data; When the gaze point of the registered user is on the screen of the terminal device, the step of controlling the terminal device to make a privacy protection risk prompt is executed.

In this way, before the terminal device is controlled to give a privacy protection risk reminder, it is further detected whether the user who is determined to be alive is looking at the screen of the terminal device. If not, there is no need to control the terminal device to make a privacy protection risk reminder. At the same time, the number of prompts is further reduced, which greatly improves the user experience.

According to the first aspect, or any implementation manner of the above first aspect, controlling the terminal device to make a privacy protection risk prompt includes: popping up the privacy protection risk prompt information on the screen of the terminal device in the form of a pop-up window.

According to the first aspect, or any implementation of the above first aspect, the pop-up window includes a first control and a second control, the first control is for the user to close the pop-up window, and the second control is used to control the terminal device to take anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: when receiving an operation on the second control, adjusting the font size of the content displayed on the screen of the terminal device, so that according to the font The safe distance determined by the size is greater than the distance between a living non-registered user and the screen.

According to the first aspect, or any implementation of the above first aspect, the pop-up window includes a first control and a second control, the first control is for the user to close the pop-up window, and the second control is used to control the terminal device to take anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: when receiving an operation on the second control, adjusting the brightness of the screen of the terminal device so that the viewing distance of the screen is greater than that of a living body. of non-registered users are located from the screen.

According to the first aspect, or any implementation of the above first aspect, the pop-up window includes a first control and a second control, the first control is for the user to close the pop-up window, and the second control is used to control the terminal device to take anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: when receiving an operation on the second control, performing a screen lock operation, so that the screen is in a screen-off state.

According to the first aspect, or any implementation of the above first aspect, the pop-up window includes a first control and a second control, the first control is for the user to close the pop-up window, and the second control is used to control the terminal device to take anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: when no operation on the second control is received within a preset time, or when an operation on the first control is received, execute Lock screen operation, so that the screen is in an off-screen state.

In this way, if the user is not currently in front of the terminal device, the screen lock operation can also be automatically performed, thereby avoiding information leakage.

According to the first aspect, or any implementation manner of the above first aspect, controlling the terminal device to make a privacy protection risk prompt includes: playing the privacy protection risk prompt information in voice form.

According to the first aspect, or any implementation method of the above first aspect, the privacy protection risk prompt information is played in voice form, including: identifying whether the terminal device is connected to the earphone; when the terminal device is connected to the earphone, playing the privacy Protect risk prompt information; when the terminal device is not connected to the headset, the privacy protection risk prompt information will be played through the microphone of the terminal device.

According to the first aspect, or any implementation of the above first aspect, before playing the privacy protection risk prompt information through the earphone, the method further includes: determining whether the earphone is worn on the registered user's ear; When it is on the ear, perform the step of playing the privacy protection risk prompt information through the earphone; when the earphone is not worn on the registered user’s ear, play the privacy protection risk prompt information through the microphone of the terminal device, or in the form of a pop-up window, on the terminal device A privacy protection risk warning message pops up on the screen.

According to the first aspect, or any implementation of the above first aspect, before playing the privacy protection risk prompt information through the microphone of the terminal device, the method further includes: detecting whether the microphone is registered for external playback; when the microphone does not support external playback, In the form of a pop-up window, the privacy protection risk prompt information is popped up on the screen of the terminal device; when the microphone supports external playback, the step of playing the privacy protection risk prompt information through the microphone of the terminal device is performed.

According to the first aspect, or any implementation method of the above first aspect, playing the privacy protection risk prompt information in voice form includes: determining the volume of playing the privacy protection risk prompt information according to the surrounding environment; playing the privacy protection risk prompt information according to the volume Prompt information.

In a second aspect, the present application provides a terminal device. The terminal device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the first aspect or any possible implementation manner of the first aspect method directive.

The second aspect and any implementation manner of the second aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For technical effects corresponding to the second aspect and any implementation manner of the second aspect, reference may be made to the technical effects corresponding to the above-mentioned first aspect and any implementation manner of the first aspect, and details are not repeated here.

In a third aspect, the present application provides a computer-readable medium for storing a computer program, where the computer program includes instructions for executing the method in the first aspect or any possible implementation manner of the first aspect.

The third aspect and any implementation manner of the third aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For the technical effects corresponding to the third aspect and any one of the implementation manners of the third aspect, refer to the above-mentioned first aspect and the technical effects corresponding to any one of the implementation manners of the first aspect, which will not be repeated here.

In a fourth aspect, the present application provides a computer program, where the computer program includes instructions for executing the method in the first aspect or any possible implementation manner of the first aspect.

The fourth aspect and any implementation manner of the fourth aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For the technical effects corresponding to the fourth aspect and any one of the implementation manners of the fourth aspect, refer to the above-mentioned first aspect and the technical effects corresponding to any one of the implementation manners of the first aspect, and details are not repeated here.

In a fifth aspect, the present application provides a chip, and the chip includes a processing circuit and transceiving pins. Wherein, the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive signals, so as to Control the send pin to send signal.

The fifth aspect and any implementation manner of the fifth aspect correspond to the first aspect and any implementation manner of the first aspect respectively. For the technical effects corresponding to the fifth aspect and any one of the implementation manners of the fifth aspect, refer to the technical effects corresponding to the above-mentioned first aspect and any one of the implementation manners of the first aspect, and details are not repeated here.

Description of drawings

FIG. 1 is a schematic diagram of an exemplary anti-peeping scene using an anti-peeping film;

Fig. 2 is the schematic diagram that adopts two-dimensional image to realize anti-peeping scene;

Figures 3a to 3c are schematic diagrams of anti-peeping scenarios implemented using the privacy protection method provided by the embodiment of the present application;

FIG. 4 is a schematic diagram of a hardware structure of a terminal device that implements the privacy protection method provided by the embodiment of the present application;

FIG. 5 is a schematic diagram of a software structure of a terminal device that exemplarily shows a privacy protection method provided by an embodiment of the present application;

Figures 6 to 11 are exemplary illustrations of interfaces for enabling the privacy protection mode through the setting application and performing related settings;

FIG. 12 is a schematic diagram of an interface showing that the privacy protection mode is enabled by pulling down the notification bar;

FIG. 13 is an exemplary interactive schematic diagram of functional modules involved in implementing the privacy protection method provided by the embodiment of the present application;

Fig. 14a and Fig. 14b are timing diagrams of the privacy protection method provided by the embodiment of the present application;

FIG. 15 is an exemplary interface schematic diagram showing a privacy protection risk prompt based on the privacy protection method provided by the embodiment of the present application;

FIG. 16 is an exemplary interface diagram showing a privacy protection risk prompt based on the privacy protection method provided by the embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations.

The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of objects. For example, the first target object, the second target object, etc. are used to distinguish different target objects, rather than describing a specific order of the target objects.

In the embodiments of the present application, words such as "exemplary" or "for example" are used as examples, illustrations or illustrations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application shall not be interpreted as being more preferred or more advantageous than other embodiments or design schemes. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete manner.

In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

With the development of science and technology, terminal equipment has more and more functions, and terminal equipment occupies an increasingly important position in people's daily life and work. At present, users can not only watch videos and play games for entertainment through these terminal devices, but also send and receive emails, process work, and conduct instant messaging activities through these terminal devices.

But at the same time, the leakage of user privacy information is becoming more and more serious, especially in public places, when users use terminal devices to perform some privacy-related and security-related operations, such as viewing work files and personal accounts, if they do not pay attention, The content displayed on the screen will be seen by others, causing information leakage and causing unnecessary trouble and distress to users, such as money loss and loss of customer resources.

For the above problems, in some implementations, a solution is to stick an anti-peep film on the screen of the terminal device to prevent others from peeping.

It is understandable that the so-called anti-peeping film is actually a piece of screen glass. Specifically, the refractive index of the glass is used to reduce the refraction angle of light so as to prevent people from peeping.

The following takes the personal computer (Personal Computer, PC) as an example of the terminal device where the anti-spy film is posted to describe the anti-peeping scene realized by the anti-spy film.

As shown in Figure 1, a privacy film is pasted on the screen of the PC. After posting the anti-peeping film, take the central axis of the PC perpendicular to the horizontal plane as the axis of symmetry, and the PC screens on both sides of the axis of symmetry are divided into visible ranges and invisible ranges. Wherein, the visible ranges on both sides of the symmetry axis are, for example, 25°, and the invisible ranges outside the visible range and the horizontal plane are 65° respectively.

Continue referring to FIG. 1, if user A is the user (owner, or authorized user) of the PC, user B and user C are non-authorized users, that is, the above-mentioned bystanders. If user A is currently using a PC to view privacy-related files, user B and user C on both sides of user A can still see the file content displayed on the PC screen through the visible range.

From the above description, it can be known that the screen of the terminal device with the anti-peeping film has a visible range and an invisible range. Therefore, if bystanders are within sight, there is still a risk of leakage of the content displayed on the screen.

It should be understood that the above is only a specific implementation scenario of posting a privacy film with a visible range of 50° and an invisible range of 130°, in order to better understand the anti-spying scene of the anti-spy film The listed examples are not intended to be the only limitation on the anti-peeping film.

In addition, after posting the anti-peeping film, the touch sensitivity of the screen will decrease for touch-sensitive terminal equipment, which will affect the user experience, and after posting the anti-peeping film, the screen brightness of the terminal equipment will also decrease, especially When the sun shines on the screen, the user cannot see the content displayed on the screen even if it is within the visible range, and the user's eyes will be damaged to a certain extent in a low-brightness environment for a long time.

In addition, the anti-peeping film is only a passive anti-peeping method, and cannot remind the user that someone else is currently peeping at the screen. As shown in Figure 1, when user B and user C peek at the screen through the visible range, if user A does not Arrived, there was no way to take action at all.

Obviously, this privacy protection method is not ideal enough. Based on this, in order to solve the above problems, in other implementations, the solution adopted is to use the front camera of the terminal device to collect images in real time, and when the face of a stranger (unauthorized user) is recognized in the image, The user is prompted that there is currently a stranger peeping at the screen of the terminal device. However, this method only extracts face information from two-dimensional images for comparison. Therefore, when the user's environment includes posters, photos, video playback devices, etc., the images collected by the front camera of the terminal device often include posters. , photos, and videos, which will cause the terminal device to frequently prompt the user that there are strangers peeping at the screen of the terminal device, seriously affecting the user experience.

The following still uses the PC as an example of the terminal device to describe the anti-peeping scene that uses the face information in the two-dimensional image to identify strangers, and then prompts the user to implement.

As shown in Figure 2, user A hangs a photo frame on his back, and the photos in the photo frame include many faces, most of which are not registered as authorized users of the PC. When user A sits at the desk and uses the PC, the front camera of the PC with the anti-peeping function enabled will collect images in real time. The collected images include many faces in the photos in the photo frame. Because there are faces registered as authorized users, the PC will frequently prompt the user "There are strangers looking at your screen, please pay attention to safety."

That is to say, in the above method, no matter whether the recognized face is a live face, or a face in a poster, photo, or video, a reminder will be given.

Obviously, this method of privacy protection is not ideal either. In view of this, this application provides a privacy protection method, which aims to remind the user to pay attention to the privacy risk in time when there is a living human face watching the screen in the surrounding environment, which can not only reduce the leakage of user information, but also avoid Faces in posters, photos, videos, etc. are frequently prompted to improve user experience. That is, the privacy protection method provided by this application only reminds when the recognized human face is a living human face, not a human face in an image or video, and the living human face is watching the screen, so as to prevent peeping , and avoid frequent prompts, ensuring user experience.

The following still uses the terminal device as a PC to describe the anti-peeping scenario implemented by using the privacy protection method provided by this application.

When user A is using a PC while sitting at a desk, after turning on the privacy protection mode/anti-peeping function provided by this application, based on the privacy protection method provided by this application, by processing the image captured by the front camera, the image including the human face is obtained. The characteristic infrared image/RGB image, and the depth image including depth information and stereoscopic information, and then determine whether the collected image includes a living human face according to the facial features and stereoscopic information, and determine the living user and PC according to the depth information. screen distance.

As shown in Figure 3a, since there is no live user in the scene shown in Figure 3a, it is determined that there is currently no bystander peeping at the PC screen. In this case, based on the privacy protection method provided by this application, there is no need to prompt the user , so that the user's use will not be interrupted, and the user experience is guaranteed.

As shown in Figure 3b, User B (unauthorized living user) is standing behind User A. If it is determined according to the depth information that the distance between User B and the PC screen is less than the safe distance for the fonts displayed on the current screen to be visible, that is, as shown in Figure 3b In the scenario, user B can clearly see the content of the file displayed on the PC screen, and there is a risk of information leakage. In order to allow user A to take timely measures, a pop-up window can be displayed on the PC screen to remind user A that "there is a stranger Looking at your screen, please stay safe."

As shown in Figure 3c, if it is determined according to the depth information that the distance between user B and the PC screen is greater than the safe distance for the fonts displayed on the current screen to be visible, that is, in the scene shown in Figure 3b, user B cannot see clearly, or even see the PC screen. For the content of the file displayed on the screen, in this case, based on the privacy protection method provided by this application, there is no need to prompt the user, so that the user's use will not be interrupted, and the user experience is guaranteed.

In addition, it should be noted that, based on the privacy protection method provided by this application, when it is determined that there is a living human face, and the distance between the living user and the PC screen is less than the safe distance for the fonts displayed on the current screen to be visible, if the living user There is no need to look at the PC screen, and there is no need to prompt the user, thereby further reducing the number of prompts and making the user's use of the physical examination better.

In order to better understand the technical solution provided by this application, before explaining the technical solution of this application, first, in conjunction with the accompanying drawings, the hardware structure of the applicable terminal equipment (such as mobile phone, tablet computer, touch PC, etc.) of this application is carried out. illustrate.

Understandably, since the technical solution provided by this application also requires image acquisition, the terminal device to which this application is applicable needs to have a camera.

For ease of description, the hardware structure of the mobile phone will be described in detail below by taking the terminal device as an example.

4, the terminal device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, Antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, A display screen 194, and a subscriber identification module (subscriber identification module, SIM) card interface 195, etc.

Exemplarily, in some implementations, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, and an ambient light sensor. , bone conduction sensors, etc., are not listed here one by one, and this application is not limited thereto.

In addition, it should be noted that the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), image signal processor (image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural -network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.

Understandably, the controller may be the nerve center and command center of the terminal device 100 . In practical applications, the controller can generate operation control signals according to the instruction opcode and timing signals, and complete the control of fetching and executing instructions.

In addition, it should be noted that a memory may also be set in the processor 110 for storing instructions and data. In some implementations, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.

Exemplarily, in some implementation manners, the processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous transmitter (universal asynchronous receiver/transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and / Or a universal serial bus (universal serial bus, USB) interface, etc.

Continuing to refer to FIG. 4 , for example, the charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some implementations of wired charging, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 . In some implementations of wireless charging, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100 . While the charging management module 140 is charging the battery 142 , it can also supply power to the terminal device through the power management module 141 .

Continue referring to FIG. 4 , for example, the power management module 141 is used to connect the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other implementation manners, the power management module 141 may also be disposed in the processor 110 . In other implementation manners, the power management module 141 and the charging management module 140 may also be disposed in the same device.

Continuing to refer to FIG. 4 , for example, the wireless communication function of the terminal device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, and a baseband processor.

It should be noted that the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other implementations, the antenna may be used in conjunction with a tuning switch.

Continuing to refer to FIG. 4 , for example, the mobile communication module 150 may provide a wireless communication solution including 2G/3G/4G/5G applied on the terminal device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves and radiate them through the antenna 1 . In some implementations, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some implementations, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.

In addition, it should be noted that the modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . In some implementations, the modem processor can be a stand-alone device. In other implementation manners, the modem processor may be independent of the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.

Continuing to refer to FIG. 4, exemplary, the wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions plan. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.

Specifically, in the technical solution of the present application, the terminal device 100 can communicate with a cloud server or other servers through the mobile communication module 150 or the wireless communication module 160 .

Exemplarily, in some implementations, the above-mentioned cloud server or other servers are, for example, servers for face recognition models, identity recognition models, and living body detection models obtained based on massive facial feature data and depth information training. . For the scene where the face recognition model, identity recognition model and living body detection model are trained by the server, the terminal device 100 can send a request to the cloud server to obtain the above models through the mobile communication module 150, and then obtain each model obtained by the server training. To the local, so that after the privacy protection mode/peep prevention function is turned on later, the terminal device can implement the privacy protection method provided by this application based on the above model. Exemplarily, the cloud may be a server cluster composed of multiple servers.

Exemplarily, in other implementation manners, the various models mentioned above may also be directly preset into the terminal device 100 before the terminal device 100 leaves the factory, or be obtained by the terminal device 100 through training based on collected data.

It should be understood that the above description is only an example for better understanding of the technical solutions of the present application, and is not intended as the only limitation to the present application. For the convenience of description, this application takes the above-mentioned various models preset in the terminal device 100 in advance as an example, followed by self-learning and improvement according to the user's usage.

In addition, it should be noted that the terminal device 100 implements a display function through a GPU, a display screen 194, an application processor, and the like. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

Continuing to refer to FIG. 4 , for example, the display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light emitting diode or an active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some implementation manners, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

In addition, it should be noted that the terminal device 100 may realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 , and the application processor.

In addition, it should be noted that the ISP is used to process data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some implementations, the ISP can be provided in the camera 193 .

In addition, it should be noted that the camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some implementation manners, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

Specifically in the technical solution provided by this application, considering that in the privacy protection method provided by this application, depth information, stereo information, and facial features need to be obtained, so in some implementations, the camera in the terminal device can be a depth camera, for example. , that is, the RGB-D camera, so that the depth image and the RGB image can be obtained directly in one acquisition.

Furthermore, in order to prevent the RGB image from being unable to accurately extract facial features due to overexposure, etc., in other implementations, the camera in the terminal device can be an infrared depth camera, so that the infrared image and depth can be obtained directly in one acquisition. image. Therefore, infrared images are used to extract face features, which effectively solves the above-mentioned problems in RGB images.

In addition, in order to make the privacy protection method provided by this application applicable to more terminal devices, for example, terminal devices that have been put into the market without a depth camera and/or an infrared depth camera. The privacy method provided by this application can also be based on a monocular or multi-camera depth estimation algorithm to process RGB images of a camera that can only collect RGB images, thereby converting a depth image, based on the above logic for face detection, liveness detection, etc.

It should be understood that the above description is only an example for better understanding of the technical solutions of the present application, and is not intended as the only limitation to the present application.

In addition, it should be noted that the digital signal processor is used for processing digital signals, and can process other digital signals in addition to digital image signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

In addition, it should be noted that the video codec is used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

Continuing to refer to FIG. 4 , for example, the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, so as to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

Continuing to refer to FIG. 4 , for example, the internal memory 121 may be used to store computer-executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by executing instructions stored in the internal memory 121 . The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the terminal device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.

Specifically, in the technical solution provided by this application, the various models mentioned above can be stored in the internal memory 121 of the terminal device 100 in advance, so as to facilitate operations such as face recognition, identity recognition, and living body detection.

In addition, it should be noted that the terminal device 100 may implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

In addition, it should be noted that the audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some implementation manners, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .

Continuing to refer to FIG. 4 , for example, the key 190 includes a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100 .

Continuing to refer to FIG. 4 , for example, the motor 191 may generate a vibration prompt. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

Continuing to refer to FIG. 4 , for example, the indicator 192 may be an indicator light, which may be used to indicate the charging status, the change of the battery capacity, and may also be used to indicate messages, missed calls, notifications and the like.

This concludes the introduction of the hardware structure of the terminal device 100. It should be understood that the terminal device 100 shown in FIG. few parts, two or more parts may be combined, or may have different part configurations. The various components shown in FIG. 1 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and/or application specific integrated circuits.

In order to better understand the software structure of the terminal device 100 shown in FIG. 4 , the software structure of the terminal device 100 will be described below. Before describing the software structure of the terminal device 100, the architecture that can be adopted by the software system of the terminal device 100 will be described first.

Specifically, in practical applications, the software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.

In addition, it is understandable that software systems used by current mainstream terminal devices include but are not limited to Windows systems, Android systems, and iOS systems. For ease of description, the embodiment of the present application uses an Android system with a layered architecture as an example to illustrate the software structure of the terminal device 100 .

In addition, in the specific implementation of the privacy protection scheme provided by the embodiment of the present application, the privacy protection scheme provided by the embodiment of the present application is also applicable to other systems.

Referring to FIG. 5 , it is a block diagram of a software structure of a terminal device 100 according to an embodiment of the present application.

As shown in FIG. 5 , the layered architecture of the terminal device 100 divides the software into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some implementations, the Android system is divided into four layers, which are the application program layer, the application program framework layer, the Android runtime (Android runtime) and system library, and the kernel layer from top to bottom.

Wherein, the application program layer may include a series of application program packages. As shown in FIG. 5 , the application program package may include application programs such as map, WLAN, Bluetooth, camera, application, privacy protection, setting, etc., which are not listed here and are not limited in this application.

Regarding the privacy protection application, in this application, it is specifically an application for enabling the privacy protection mode (or anti-peeping mode) and setting corresponding information about the privacy protection mode.

Understandably, in practical applications, this function can also be integrated into the setting application, that is, the privacy protection application is not provided separately, and related settings can be performed through the function entry provided in the setting application.

Further, in order to facilitate the user to enable the privacy protection mode, the anti-peeping scene is implemented based on the privacy protection method provided by this application. A shortcut can also be provided to enable the privacy protection mode. For example, by providing a user entry in the drop-down notification bar, or by specifying the number of taps, tap position, key, voice, etc. to open.

It should be understood that the above description is only an example for better understanding of the technical solutions of the present application, and is not intended as the only limitation to the present application.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. In some implementations, these programming interfaces and programming frameworks can be described as functions. As shown in Figure 5, the application framework layer can include functions such as trigger detection module, face recognition module, identity recognition module, distance detection module, living body detection module, eye tracking module, prompt module, view system, notification manager, etc. Here, they are not listed one by one, and the present application does not limit them.

Exemplarily, in this embodiment, the trigger detection module is used to detect whether the terminal device satisfies the condition of triggering the camera to capture images after the user turns on the privacy protection mode (hereinafter referred to as: trigger mechanism).

Regarding the trigger mechanism, in some implementations, for example, after the privacy protection mode is turned on, the screen of the terminal device is unlocked; in other implementations, for example, after the privacy protection mode is turned on, the specified privacy/sensitive application is accessed, Or in a specified privacy scene; in other implementations, for example, after turning on the privacy protection mode, a specified action command, voice command, etc. are made.

It should be understood that the above description is only an example for better understanding of the technical solutions of the present application, and is not intended as the only limitation to the present application.

Exemplarily, in this embodiment, the face recognition module is used to perform face recognition on the infrared image/RGB image obtained by processing the image data captured by the camera, so as to determine whether there is a human face in the current image data.

Exemplarily, in this embodiment, the identity recognition module is used to compare the face features based on the face with the stored face features of the registered user (authorized user) when there is a face in the current image data, To determine whether the face existing in the current image data is a registered user.

Exemplarily, in this embodiment, the distance detection module is used to perform distance detection based on the depth image obtained by processing the image data captured by the camera when the face existing in the current image data is not a registered user.

Exemplarily, in this embodiment, the live body detection module is used to perform live body detection on the faces of non-registered users based on depth images and infrared images (RGB images) when the distance determined by the distance detection module is less than the set safety distance. To determine whether the non-registered user in the image data is alive.

Exemplarily, in this embodiment, the eye-tracking module is used to detect the non-registered user's eyes, eyes, and gaze when the live body detection module determines that the non-registered user is alive, so as to determine whether the non-registered user is currently Look at the screen of the end device.

Exemplarily, in this embodiment, when a non-registered user looks at the screen of the terminal device, that is, when the user is currently in an open-eyed state, and the gaze point is on the screen of the terminal device, the user of the prompt module triggers a privacy protection risk prompt, such as sending a notification The manager sends the set prompt content so that the notification manager can respond, so that the terminal device can display prompt information in the status bar or user interface, or give voice prompts through speakers or earphones inserted.

It can be understood that the division of the above functional modules is only an example for better understanding the technical solution of the present application, and is not the only limitation on the application. In practical applications, the above functions may also be integrated into one functional module, which is not limited in this application.

In addition, it should be noted that the above-mentioned view system located in the application framework layer includes visual controls, such as controls for displaying text, controls for displaying pictures, and the like. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.

In addition, it should be noted that the above-mentioned notification manager located in the application framework layer enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction . For example, the notification manager is used to notify the download completion, message reminder, etc. Specifically, in the technical solution provided by this application, when the user is prompted in the form of a notification message that a stranger is watching the screen of the terminal device he is currently using, the prompt module can send the set prompt content to the notification manager, so that the notification management The controller can respond, so that the terminal device can display prompt information in the status bar or user interface.

Android Runtime includes core library and virtual machine. Android Runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc.

Understandably, the 2D graphics engine mentioned above is a graphics engine for 2D graphics.

In addition, understandably, the kernel layer in the Android system is a layer between hardware and software. The kernel layer includes at least display driver, camera driver, microphone driver, bluetooth driver, sensor driver, etc. Exemplarily, the camera driver is used to drive the camera to capture images when the privacy protection module is turned on, and the trigger detection module determines that the above-mentioned trigger mechanism is satisfied, and then obtain the images that need to be processed when implementing the privacy protection method provided by this application data.

The software structure of the terminal device 100 is introduced here. It can be understood that the layers in the software structure shown in FIG. 2 and the components included in each layer do not constitute a specific limitation on the terminal device 100 . In other embodiments of the present application, the terminal device 100 may include more or fewer layers than shown in the figure, and each layer may include more or fewer components, which are not limited in the present application.

In order to better understand the privacy protection method provided by this application, the terminal device is a mobile phone as an example, and the specific operations of enabling the privacy protection mode through the setting application and pulling down the notification bar and performing related settings are described in conjunction with the accompanying drawings.

1. Turn on the privacy protection mode through the setting application and make related settings

Referring to the interface 10a shown in (1) in FIG. 6, for example, the current interface 10a of the mobile phone may include one or more controls. Controls include, but are not limited to: network controls, battery controls, application icon controls, and the like.

Continuing to refer to the interface 10a shown in (1) in FIG. 6 , exemplary application icon controls include but are not limited to: clock application icon control, calendar application icon control, gallery application icon control, memo application icon control, file management application icon control, email application icon control, music application icon control, calculator application icon control, video application icon control, voice recorder application icon control, weather application icon control, browser application icon control, settings application icon control 10a-1, etc. No longer enumerate them one by one here, and the present application does not limit them.

Continuing to refer to the interface 10a shown in (1) in FIG. 6, for example, after the user clicks the control 10a-1, the mobile phone starts the interface 10b shown in (2) in FIG. 6 in response to the user's operation behavior.

Referring to the interface 10b shown in (2) in FIG. 6 , for example, the interface 10b may include one or more controls. The controls include but are not limited to: a control 10b-1 for exiting the interface 10b, a control for setting the sound and vibration modes of the mobile phone, a control for setting notifications, a control 10b-2 for setting the privacy protection mode, and a control 10b-2 for setting the privacy protection mode. The controls for viewing the installed applications on the mobile phone, the controls for viewing the battery information of the mobile phone, the controls for viewing the current storage space of the mobile phone, the controls for viewing the security information of the mobile phone, etc., are not listed here, and this application does not make any reference to them. limit.

Continuing to refer to the interface 10b shown in (2) in FIG. 6, for example, after the user clicks the control 10b-2, the mobile phone starts the interface 10c shown in (1) in FIG. 7 in response to the user's operation behavior.

Referring to the interface 10c shown in (1) in FIG. 7 , for example, the interface 10c may include one or more controls. The controls include but are not limited to: a control 10c-1 for exiting the interface 10c, and a control 10c-2 for turning on or off the privacy protection mode.

Exemplarily, in this embodiment, the state of the control 10c-2 in the interface 10c shown in (1) in FIG. 7 indicates that the privacy protection mode is not turned on, that is, the privacy protection mode is turned off; ) shows that the state of the control 10c-2 in the interface 10c indicates that the privacy protection mode is on, that is, the privacy protection mode is on.

Continue to refer to the interface 10c shown in (1) in FIG. 7. Exemplarily, when the user clicks the control 10c-2, the mobile phone responds to the user's operation behavior, and the control 10c-2 changes from the control 10c-2 shown in (1) in FIG. The state switches to the state shown in (2) in FIG. 7 .

Referring to the interface 10c shown in (2) in Figure 7, for example, after the control 10c-2 is switched from the state shown in Figure 7 (1) to the state shown in Figure 7 (2), in the interface 10c Also displayed is a control 10c-3 to enter the trigger mechanism setting interface.

Continuing to refer to the interface 10c shown in (2) in FIG. 7, for example, after the user clicks the control 10c-3, the mobile phone starts the interface 10d shown in (1) in FIG. 8 in response to the user's operation behavior.

Referring to the interface 10d shown in (1) in FIG. 8 , for example, the interface 10d may include one or more controls. The controls include but are not limited to: a control 10d-1 for exiting the interface 10d, a control 10d-2 for selecting the trigger mechanism 1, a control 10d-3 for selecting the trigger mechanism 2, and a control 10d for selecting the trigger mechanism 3 -4.

Understandably, in some implementations, at the same moment, only one of the controls 10d-2, 10d-3, and 10d-4 can be selected; in other implementations, at the same moment, the controls 10d- Only one of 2 and control 10d-3 can be selected, only one of control 10d-2 and control 10d-4 can be selected, and control 10d-3 and control 10d-4 can be selected at the same time.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment. For ease of description, this embodiment takes a scene where only one of the controls 10d-2, 10d-3 and 10d-4 can be selected at the same time as an example for illustration.

In addition, it is understandable that in some implementations, when the user first turns on the privacy protection mode and enters the interface 10d, the trigger mechanism may default to trigger mechanism 1, that is, the control 10d-2 is in the selected state.

Continue to refer to the interface 10d shown in (1) in FIG. 8 . Exemplarily, when the user needs to change the trigger mechanism, for example, after clicking the control 10d-3, the mobile phone responds to the user's operation behavior, and the control 10d-3 is selected. The states of the control 10d-2, the control 10d-3 and the control 10d-4 are shown as (2) in FIG. 8 .

Referring to the interface 10d shown in (2) in FIG. 8, for example, after the control 10d-3 is selected, the interface 10d also displays a control 10d-5 for entering the privacy application setting interface corresponding to the trigger mechanism 2.

Continuing to refer to the interface 10d shown in (2) in FIG. 8, for example, after the user clicks the control 10d-5, the mobile phone starts the interface 10e shown in (1) in FIG. 9 in response to the user's operation behavior.

Referring to the interface 10e shown in (1) in FIG. 9, for example, the interface 10e may include one or more controls. The controls include but are not limited to: a control 10e-1 for exiting the interface 10e, and a control 10e-2 for displaying a list of applications installed on the mobile phone.

Referring to the interface 10e shown in (1) in FIG. 9, for example, the list 10e-2 includes one or more controls. The controls include, but are not limited to: a control 10e-21 corresponding to each application, and a control 10e-22 for the applications displayed in the sliding list 10e-2.

Understandably, when the user slides the control 10e-22, the applications displayed in the list 10e-2 and the controls 10e-21 corresponding to each application will slide up and down, thereby displaying applications not displayed in the current list and Corresponding controls 10e-21.

Exemplarily, in some implementation manners, the user can click on the control 10e-21 corresponding to the application that he wants to set as a private application, so that the application can be set as a private application, that is, when the user uses the application, When the privacy protection mode is turned on, the privacy protection method provided in this embodiment will be executed.

For example, when the user clicks the control 10e-21 corresponding to the gallery, the mobile phone responds to the user's operation behavior, and the control 10e-21 corresponding to the gallery will switch from the state in the 10e interface shown in (1) in Figure 9 to Figure 9 The state shown in the 10e interface shown in (2) indicates that the gallery application is set as a privacy application. When the privacy protection mode is turned on, when the user accesses the gallery application, the application provided by this embodiment will be executed. Privacy protection methods.

Exemplarily, when the user clicks the control 10e-21 corresponding to the memo, the control 10e-21 corresponding to the file management, the control 10e-21 corresponding to the email, and the control 10e-21 corresponding to the address book, the mobile phone will also make a response. The control 10e-21 corresponding to the memo, the control 10e-21 corresponding to the file management, the control 10e-21 corresponding to the email, and the control 10e-21 corresponding to the address book are switched from the state in the 10e interface shown in (1) in Figure 9 It is the state shown in the 10e interface shown in (2) in Figure 9, which shows that the memo application, file management application, email application, and address book application are also set as privacy applications. When the privacy protection mode is turned on , when the user accesses these applications, the privacy protection method provided in this embodiment will be executed.

Exemplarily, in some other implementation manners, the interface 10e may also display controls for selecting the controls 10e-21 corresponding to all application programs, and controls for unselecting the controls 10e-21 corresponding to all application programs. In this way, the user can realize the selection of the controls 10e-21 corresponding to all applications by operating and selecting the controls 10e-21 corresponding to all applications, so that all applications are set as privacy applications. By operating and unselecting the controls 10e-21 corresponding to all the applications, the controls 10e-21 corresponding to all the selected applications can be deselected, which greatly facilitates the user's operation and further improves the user experience.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

Continuing to refer to the interface 10e shown in (2) in FIG. 9, after the user finishes setting the privacy application, if the control 10e-1 is clicked, the mobile phone will return to the interface 10d in response to the user's operation behavior.

Referring to the interface 10d shown in (1) in Figure 10, or the interface 10d shown in (2) in Figure 8, for example, if the user wants to select the trigger mechanism 3, when the user clicks the control 10d-4, the mobile phone responds Due to the user's operation behavior, the control 10d-4 is selected, and the states of the control 10d-2, the control 10d-3 and the control 10d-4 are shown in (2) in FIG. 10 .

Referring to the interface 10d shown in (2) in FIG. 10 , for example, after the control 10d-4 is selected, the interface 10d also displays a control 10d-6 for entering the privacy scene setting interface corresponding to the trigger mechanism 3 .

It should be noted that in some implementations, privacy scenarios can be divided in advance according to business needs and application types, or a custom entry can be provided for users to set.

This embodiment takes the pre-determined privacy scene as an example. Wherein, the predetermined privacy scene includes, for example, a game scene, a video scene, a social scene, an office scene, and the like.

It is understandable that the applications included in the above scenarios are specifically determined according to the application type. For example, WeChat belongs to the social scenario, email belongs to the office scenario, audio and video playback applications belong to the video scenario, and various game applications Then enter the game scene.

Continuing to refer to the interface 10d shown in (2) in FIG. 10, for example, after the user clicks the control 10d-6, the mobile phone starts the interface 10f shown in (1) in FIG. 11 in response to the user's operation behavior.

Referring to the interface 10f shown in (1) in FIG. 11 , for example, the interface 10f may include one or more controls. The controls include but are not limited to: a control 10f-1 for exiting the interface 10f, a control 10f-2 for setting the game scene as a private scene, a control 10f-3 for setting the video scene as a private scene, and a control 10f-3 for setting the game scene as a private scene. A control 10f-4 for setting the social scene as a private scene, and a control 10f-5 for setting the office scene as a private scene.

Exemplarily, in this embodiment, the state of the controls 10f-2, 10f-3, 10f-4, and 10f-5 in the interface 10f shown in (1) in FIG. 11 indicates that the corresponding scene is not set as privacy Scene: The state of the controls 10f-4 and 10f-5 in the interface 10f shown in (2) in FIG. 11 indicates that the corresponding scene is set as a privacy scene. That is to say, when the user clicks the control 10f-4 in the interface 10f shown in (1) in Figure 11, the mobile phone will respond to the user's operation behavior, and will display from the interface 10f shown in (1) in Figure 11 The state is switched to the state displayed on the interface 10f shown in (2) in FIG. 11 .

This is the end of the introduction to the operation of enabling the privacy protection mode through the settings application and making related settings. It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

2. Turn on the privacy protection mode by pulling down the notification bar and make related settings

It should be noted that by pulling down the notification bar to enable the privacy protection mode, the user can perform the action of pulling down the notification bar on any interface when the phone is on with the screen on. This embodiment takes the action of pulling down the notification bar when the mobile phone is in the above-mentioned interface 10a, and then pulling down the notification bar, enabling the privacy protection mode by pulling down the notification bar, and performing related settings as an example.

Referring to the interface 10a shown in (1) in FIG. 12, for example, when the user slides down from the upper edge of the mobile phone in the direction of the arrow, the mobile phone will display in the upper edge area of the interface 10a in response to the user's operation behavior Pull down the notification bar 10a-2 as shown in (2) in FIG. 12 .

Referring to the drop-down notification bar 10a-2 shown in (2) in FIG. 12, the drop-down notification bar 10a-2 may include one or more controls. Controls include, but are not limited to: Wi-Fi setting options, Bluetooth setting options, shortcut entries 10a-21 for privacy applications and/or shortcut entries 10a-22 for setting applications, etc.

Exemplarily, after the user clicks the shortcut entry 10a-22, the mobile phone can directly start the interface 10b shown in (2) in FIG. Find the control 10a-1 for setting the application in the interface 10a, pull down the notification bar, and click the shortcut entry 10a-22 to quickly start the interface 10b. Afterwards, the process of enabling the privacy protection module and performing related settings is similar to the above-mentioned operation of directly launching the setting application through the control 10a-1, and will not be repeated here.

Exemplarily, if the user clicks on the shortcut entry 10a-21, the mobile phone may start an interface 10c similar to that shown in (2) in FIG. 8 in response to the user's operation behavior. Afterwards, the process of enabling the privacy protection module and performing related settings is similar to the above-mentioned operation of directly launching the setting application through the control 10a-1, and will not be repeated here.

This is the end of the introduction for enabling the privacy protection mode by pulling down the notification bar and making related settings. It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

In order to better understand the privacy protection method provided by this embodiment, the interactive processing logic when the user implements the privacy protection method by each functional module in the mobile phone after turning on the privacy protection mode according to any of the above and completing the relevant settings will be described in detail .

Referring to Figure 13, the privacy protection method flow provided by this embodiment specifically includes:

After the privacy protection mode is turned on, the trigger mechanism detection module located at the application framework layer will detect whether the terminal device currently satisfies the set trigger mechanism in real time in the background or according to the preset cycle, and if it is detected that the set trigger mechanism is satisfied, then execute the steps S101, notify the front-facing camera driver at the kernel layer to invoke the hardware of the front-facing camera to start collecting image data.

After receiving the notification sent by the trigger mechanism detection module in the application framework layer, the front camera driver at the kernel layer executes step S102 to call the front camera for image acquisition.

It is understandable that the front camera can continuously capture multiple pieces of image data at intervals, and can also continuously collect video streams at a set time.

Correspondingly, after the front camera collects the image data/video stream according to the set requirements (this embodiment takes image data as an example), step S103 will be executed to transmit the collected image data to the front camera driver at the kernel layer .

The front camera driver at the kernel layer transmits the image data collected by the front camera to the face recognition module at the application framework layer, that is, step S104 is performed.

It should be noted that if the front camera is only an RGB camera, the image data collected by the front camera is an RGB image. In this case, if there are dual front cameras, the image data collected by the dual RGB cameras is used to generate the depth If the image is a single camera, use a single image data to perform deep learning to generate a depth image. For the specific processing method of converting an RGB image to a depth image, please refer to the current standard, and details will not be repeated here.

Based on the above situation, the application framework layer can also integrate an image processing module for processing image data to obtain RGB images for face recognition, or infrared images, and depth images for distance detection (not shown in the figure). For ease of description, this embodiment takes the D-RGB or infrared depth camera as an example of the front camera of the terminal device, that is, the front camera can directly collect RGB images and depth images, or infrared images and depth images.

The face recognition module in the application framework layer performs face recognition on the RGB images or infrared images collected by the front camera, recognizes all the faces included in the image, and then executes step S105 to transmit the recognized faces to the identity module.

The identity recognition module of the application framework layer identifies each face, such as matching the face feature information of each face with the pre-stored face feature information of registered users, and then determines whether these faces include non- Registered users, that is, the faces of strangers.

Correspondingly, if the stranger's face is included, step S106 is executed to transmit the stranger's face data to the distance detection module.

The distance detection module of the application framework layer determines the distance between the stranger's face and the screen of the terminal device according to the depth image collected by the front camera. The face data of strangers at a safe distance are transmitted to the liveness detection module.

It should be noted that the depth image used by the distance detection module can be directly obtained from the front camera driver, or it can be transmitted from the face recognition module to the identity recognition module, and then transmitted to the distance detection module by the identity recognition module , it can also be driven by the front camera to cache the depth image, RGB image/infrared image collected by the front camera to the designated buffer area, and when each subsequent module needs to process the image data collected by the front camera, directly from the The designated cache area acquires the corresponding image data, and only the corresponding instructions and processed results are transmitted between each module.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

Next, the liveness detection module of the application framework layer performs liveness detection on the faces of strangers whose distance is less than the set safety distance according to the depth image and infrared image collected by the front camera, or according to the depth image and RGB image.

Correspondingly, if it is determined that the stranger meeting the above conditions is a living body, step S108 is executed to transmit the face data of the living stranger whose distance is less than the set safety distance to the eye tracking module.

Understandably, the face data transmitted above and the face data of strangers are all RGB images/infrared images. In another feasible implementation manner, the image data may be transmitted, but the coordinate information of the face in the RGB image/infrared image/depth image that meets the above conditions is transmitted. Because they include the same coordinate information, and one-to-one correspondence. Therefore, by transmitting the coordinate information, the functional modules receiving the coordinate information can obtain the corresponding image data from the designated buffer area to realize their respective functions.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

The eye-tracking module of the application framework layer detects whether the faces of strangers who meet the above conditions are open, closed, and gazed.

Correspondingly, if the stranger's eyes are open and the gaze point is on the screen of the terminal device, step S109 is executed, and the notification and prompting module makes a privacy protection risk prompt.

As can be seen from the above description, the reminder module can transmit the set privacy protection risk reminder information to the notification manager, and the notification manager will prompt the user in the form of notification information, or call the microphone driver to drive the microphone to play the privacy protection in the form of voice. Protect risk warning information.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

For the specific implementation details of face recognition, identity recognition, distance detection, liveness detection, eye tracking, etc. involved in the above processing flow, you can refer to the documents of these technologies, and will not repeat them here.

As a result, the user is reminded to pay attention to the privacy risk only when it recognizes that there is a living human face watching the screen in the surrounding environment, thereby reducing the risk of leakage of user information and prompting the user as little as possible, ensuring the safety of the user. Use experience.

In order to better understand the processing performed by each functional module in FIG. 13 when implementing the privacy protection method in this embodiment, the specific implementation process of the privacy protection method provided in this embodiment will be described below in conjunction with FIG. 14a and FIG. 14b.

Referring to Figure 14a and Figure 14b, the privacy protection method provided by this embodiment specifically includes:

S201. When the trigger mechanism detection module detects that the terminal device satisfies the set trigger mechanism, it sends an image acquisition instruction to the front camera driver.

For example, when the user turns on the privacy protection mode, the trigger mechanism selected is the trigger mechanism 3 mentioned above, and the social scene and the office scene are set as the privacy scene. Wherein, the social scene includes, for example, various instant messaging applications, such as WeChat, and the office scene, for example, includes applications such as e-mail and memo.

When the user unlocks the terminal device and opens any installed application, the trigger mechanism detection module detects the user's operation behavior, obtains the attribute information of the application program that the user wants to open, and can determine whether the application program opened by the user is Belongs to applications that are set to privacy scenarios.

Correspondingly, when it is determined that the application program opened by the user is an application program set as a privacy scenario, the trigger mechanism detection module determines that the terminal device meets the set trigger mechanism 3, and at this time sends an image acquisition instruction to the front camera driver.

Correspondingly, after receiving the image acquisition instruction sent by the trigger mechanism detection module, the front camera driver will transmit the pending image acquisition instruction to the front camera, that is, drive the front camera to automatically perform the image acquisition operation according to the image acquisition instruction.

For the convenience of description, this embodiment takes the front camera as an example of an infrared depth camera, that is, the infrared image and the depth image can be collected directly by using the front camera.

Understandably, in some implementations, the image data collected by the infrared depth camera can be one piece, that is, the image data that integrates the infrared image and the depth image; in other implementations, it can also be two types of images Data, that is, an infrared image and a depth image are respectively obtained. This embodiment takes the latter as an example.

S202, the infrared depth front camera responds to the image acquisition instruction, collects image data according to the set requirements, and obtains a depth image and an infrared image.

Exemplarily, the requirement mentioned above is, for example, to continuously shoot multiple images at intervals, or to shoot a video stream within a preset time, and subsequently use the frame corresponding to each frame of the video stream as a piece of image data.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

Exemplarily, in some implementation manners, after the infrared image and the depth image are collected, the front camera may directly transmit the collected infrared image and depth image to the face recognition module via the front camera drive. In this way, there is no need to designate a special cache area in the terminal device, which saves space on the terminal device.

Exemplarily, in some other implementation manners, after collecting the infrared image and the depth image, the front camera can directly store the collected infrared image and depth image in a designated buffer area, and notify the front camera driver that the The image acquisition operation is completed, and then the front camera driver notifies the face recognition module that the front camera has completed the image acquisition operation, and the infrared image can be taken from the designated buffer area for face recognition operation. Subsequent processing of infrared images and depth images by other functional modules can be directly taken out from the cache area, so that infrared images and depth images do not need to be transmitted between modules, which can effectively avoid the loss of infrared images and depth images caused by the transmission process. Frame, interference and other abnormalities. This embodiment takes this method as an example.

S203, the face recognition module takes out the captured infrared image from the cache area, performs face recognition on the infrared image based on the face recognition technology, recognizes all faces, determines the coordinate information of each face, and The coordinate information of the face is transmitted to the identification module.

For specific implementation details of recognizing a face contained in an infrared image based on the face recognition technology, refer to the related introduction of the face recognition technology, which will not be repeated here.

S204. After the identity recognition module receives the coordinate information of each face transmitted by the face recognition module, it takes out the captured infrared image from the cache area, and identifies the face at each coordinate information in the infrared image based on the identity recognition technology. Recognition, when it is determined that there is a non-registered user, transmit the coordinate information of the non-registered user to the distance detection module.

Understandably, for the sake of security, the owner of the terminal device/user authorized to use the terminal device usually enters his own face feature information (hereinafter referred to as: the face feature information of the registered user), that is, in step S204 in When identifying each face included in the infrared image, for example, the face feature information of the face at each coordinate information in the infrared image is matched with the face feature information of the registered user.

Correspondingly, when the two match completely, or the matching degree satisfies the set confidence level, it is determined that the face is the face of a registered user; otherwise, it is the face of a non-registered user, that is, a stranger.

Through the above method, after identifying the face at each coordinate information, if it is determined that there is a non-registered user in the infrared image, there may be one or more, then the coordinates of the face of each non-registered user The information is transmitted to the distance detection module.

S205. After the distance detection module receives the coordinate information transmitted by the identity recognition module and is determined to be a non-registered user, it takes out the captured depth image from the cache area, and according to the three-dimensional depth of the location of the coordinate information of the non-registered user in the depth image The feature information determines the distance of the non-registered user from the screen of the terminal device.

Understandably, since the pixel value corresponding to each pixel in the depth image can represent the distance between a certain point in the captured scene and the camera, the camera in this embodiment is a front-facing camera, which is usually connected to the screen of the terminal device. In one plane, according to the three-dimensional depth feature information of the location of the coordinate information of the non-registered user in the depth image, the distance from the non-registered user to the screen of the terminal device can be determined.

Regarding the determination of the distance between two points (the point at the location of the non-registered user in the depth image and the camera) based on the 3D depth feature information in the depth image, please refer to the relevant literature on determining the distance based on the depth image, which will not be repeated here.

S206. When the determined distance is less than the set safety distance, the distance detection module transmits coordinate information of non-registered users whose distance is less than the set safety distance to the living body detection module.

Since the distance from the text displayed on the screen of the terminal device to the naked eye is related to the size of the text (each word occupies a pixel of the screen), even if the non-registered user is a living person, that is, a person with vital signs, if the distance from the screen When the distance is greater than the distance from which the text displayed on the screen can be clearly seen, that is, the safe distance mentioned above, the non-registered user cannot see clearly, or even see the content on the screen. In this case, there is no need to trigger privacy protection risks prompt, that is, no follow-up procedures are required. Therefore, the distance detection module determines whether the distance between the non-registered user and the screen is smaller than the safe distance corresponding to the font displayed on the current screen.

Correspondingly, when it is determined that the distance is less than the set safety distance, the coordinate information of the non-registered user whose distance is less than the set safety distance is transmitted to the living body detection module.

The determination of the safety distance can be realized based on the following formula:

Among them, PPI is pixels per inch, which refers to the number of pixels per inch; PPD is angular resolution, which refers to the number of pixels filled in an average angle of 1° in the field of view; X is the number of length pixels, which refers to the length of the screen The number of pixels included; Y is the number of pixels in the width, referring to the number of pixels included in the width of the screen; Z is the screen size, referring to the length of the diagonal, in inches; R is the safety distance to be determined, in units of Inches; font_pixels is the font display size currently set on the screen, referring to the number of pixels occupied by each word; α is the minimum resolution angle of the human eye corresponding to the font of this size.

Since X, Y, Z and font_pixels can all be obtained from the terminal device, according to the known four parameters and the above four formulas, the corresponding safety distance of the font currently set by the terminal device can be determined. For example, for a terminal device with a screen size of 6.81 inches, a resolution of 2848*1312 (X=2848, Y=1312), and fonts occupying 50 pixels, according to the above four formulas, the final calculated safety distance is about 59 inches ( approximately equal to 1.5 meters). That is, if the determined distance is less than 1.5 meters, step S207 is performed; otherwise, no privacy protection risk prompt is triggered.

S207, after the living body detection module receives the coordinate information of the non-registered user whose distance transmitted by the distance detection module is less than the set safety distance, it takes out the captured infrared image and depth image from the buffer area, and according to the coordinate information of the non-registered user in the depth image The three-dimensional depth feature information of the location and the feature information of the coordinate information of the non-registered user in the infrared image are used for live detection. When the non-registered user is determined to be alive, the transmission distance to the eye tracking module is less than the set safety distance And the coordinate information of non-registered users who are determined to be living.

It should be noted that, in this embodiment, the living body detection module is specifically used to determine the stereoscopic degree of the non-registered user according to the three-dimensional depth feature information of the coordinate information of the non-registered user in the depth image; The characteristic information of the location where the user's coordinate information is located determines the skin reflectance of the non-registered user.

Understandably, since the reflectance of the skin of a person with vital signs is different from that of portraits in posters, photos, and videos, and when the camera captures a person with vital signs, the non-registered user is Sensitive, but the portraits in the posters, photos, and videos taken have no three-dimensional effect. Therefore, based on the above information, it can be determined whether the non-registered user is a living body.

For the specific details of realizing the living body detection based on the depth image and the infrared image, please refer to the relevant literature, and will not repeat them here.

S208. After the eye tracking module receives the coordinate information of a non-registered user whose distance transmitted by the living body detection module is less than the set safety distance and is determined to be a living body, take out the captured infrared image from the buffer area, and based on the eye tracking technology according to The characteristic information of the location of the coordinate information of the non-registered user in the infrared image is used to track the eye movement of the non-registered user.

It can be understood that the so-called eye tracking (eye tracking) is a process of automatically realizing the positioning of the eye pupil center and fixation point. Since the external information obtained by humans is mainly through the visual information perceived by the human eye, eye movement can intuitively reflect the point and time of human gaze. Therefore, by using the eye tracking module to locate the eye pupil center and fixation point of the non-registered user who meets the aforementioned judgment, it can be determined whether the non-registered user is in an eye-open state, which is the fixation point on the screen of the terminal device.

S209. When the eye-tracking module determines that the non-registered user is in an eye-open state and the gaze point is on the screen of the terminal device, send a privacy protection risk prompt instruction to the prompt module.

Understandably, since the non-registered user is in the state of eyes open and the gaze point is on the screen of the terminal device, it appears that the non-registered user is staring at the screen of the terminal device. If the screen of the terminal device displays some private content at this time, such as When documents involving commercial secrets are likely to be seen by non-registered users, and then stolen. Therefore, in order to avoid this situation, when it is recognized that the non-registered user is in an open-eyed state and the gaze point is on the screen of the terminal device, the eye-tracking module immediately sends a privacy protection risk prompt instruction to the prompt module, so that the prompt module quickly In response, non-registered users are prevented from obtaining the content of the file displayed on the screen of the terminal device.

In addition, it should be noted that the execution order of the above steps S204 to S209 can be adjusted according to actual business requirements, which is not limited in this embodiment.

S210, the prompting module triggers a privacy protection risk prompt in response to the privacy protection risk prompt instruction sent by the eye movement tracking module.

Exemplarily, in an implementation manner, the prompt module can directly send the preset triggering privacy protection risk prompt information, such as "a stranger is looking at your screen, please pay attention to safety" to the notification manager, and then the notification The manager displays the above-mentioned triggering privacy protection risk prompt information on the screen of the terminal device in the form of a pop-up window, as shown in FIG. 15 for example.

Referring to FIG. 15 , for example, the pop-up window 10 may include one or more controls in addition to displaying preset risk prompt information that triggers privacy protection. The controls include but are not limited to: a control 10-1 for closing the pop-up window 10, and a control 10-2 for activating anti-peeping.

Exemplarily, in an implementation manner, after the user clicks the control 10-2, the terminal device may reduce the font size of the file displayed on the screen in response to the user's operation behavior, so that the distance between the non-registered user and the The distance of the terminal is greater than the set safety distance, so that the user can still use the terminal device to view the file and perform corresponding processing, while the non-registered user B and non-registered user C in Figure 15 cannot see the content on the screen.

Exemplarily, in another implementation manner, after the user clicks the control 10-2, the terminal device may, in response to the user's operation behavior, lower the brightness of the screen to a level that only the registered user can see clearly at a distance from the screen. Brightness, so that users can still use the terminal device to view the file and perform corresponding processing, while non-registered user B and non-registered user C in Figure 15 cannot see the content on the screen.

Exemplarily, in another implementation manner, after the user clicks the control 10-2, the terminal device can directly lock the screen in response to the user's operation behavior, even if the terminal device enters the state shown in FIG. 16 .

In addition, considering that in practical applications, the registered user of the terminal device may not be currently in front of the terminal device, so after a privacy protection risk reminder is given in the form of a pop-up window 10, the terminal device cannot receive the operation of the registered user on the control 10-2. To avoid information leakage, the terminal device can be set to automatically perform a screen lock operation when it does not receive an operation on the control 10-2 within 3 seconds, even if the terminal device enters the state shown in Figure 16, so that non-registered user B and non-registered user C You cannot see the contents of the screen.

In addition, it should be noted that in practical applications, in addition to using the above-mentioned pop-up window to give the privacy protection risk reminder, the privacy protection risk reminder can also be given in the form of voice.

Specifically, when the prompt module receives the privacy protection risk prompt instruction sent by the eye tracking module, in response to the instruction, it can directly play the preset privacy protection risk prompt information through the microphone of the terminal device, such as "a stranger is watching Your screen, stay safe."

Furthermore, taking into account environmental factors, in order to ensure that registered users can hear the privacy protection risk prompt information played by the microphone, and then take corresponding measures, such as pressing the lock screen button to turn off the screen of the terminal device, or reducing the brightness of the screen, or reducing the font size etc. It is also possible to collect the sound information of the surrounding environment through the microphone, and then determine the volume for playing the privacy protection risk prompt information, and finally play the privacy protection risk prompt information at a determined volume.

In addition, in some other implementation manners, the privacy protection risk prompt information may also be played through an earphone connected to the terminal device. For this scenario, the prompt module can, for example, call the earphone driver to determine whether the currently connected earphone is connected and/or call the bluetooth driver to determine whether the currently connected bluetooth earphone is connected. Play privacy protection risk reminder information.

Furthermore, considering that in an actual usage scenario, the terminal device is connected to an earphone, but the user does not wear the earphone on the ear, so the status information of the earphone can be further obtained to determine whether the earphone is worn on the ear.

Correspondingly, when the earphone is worn on the earphone, the privacy protection risk prompt information is played through the earphone, otherwise, the privacy protection risk prompt information is played through the microphone of the terminal device.

It is understandable that when playing the privacy protection risk prompt information through the earphone, the surrounding environmental factors can also be considered, the volume of the earphone can be adjusted, and then the privacy protection risk prompt information can be played at the adjusted volume.

Furthermore, in order to achieve double protection, the above two methods can be combined, that is, the pop-up 10 shown in FIG. 15 pops up on the screen of the terminal device, and at the same time, the earphone/microphone is used to play the privacy protection risk prompt information.

In addition, in other implementations, other terminal devices associated with the terminal device, such as mobile phones, smart bracelets, smart watches, etc., can also be obtained, and then privacy protection risk reminders can be given through the associated terminal devices. In this way, for the scenario where the registered user is in front of the terminal device, based on other associated terminal devices, the privacy protection risk reminder can be made, so that the registered user can rush back to the terminal device as soon as possible, and let the non-registered user open the terminal device to steal information.

In addition, the terminal device can also store the face image of the non-registered user staring at the screen, so that if the user causes trouble due to privacy leakage, the user can hold the accountable through the stored face image of the non-registered user.

It should be understood that the above description is only an example for better understanding of the technical solution of this embodiment, and is not the only limitation to this embodiment.

Therefore, the privacy protection method provided by this embodiment, when it is determined through face recognition and identity recognition that there is a risk of non-registered users peeping at the screen of the terminal device, further passes distance detection, liveness detection, eye opening and closing interception, eye movement Tracking and other dimensions can be processed to reduce the probability of false prompts as much as possible while ensuring privacy and security, thereby prompting the user's experience.

In addition, based on information such as the minimum resolution angle of the human eye and screen parameters, the safe distance suitable for the font currently displayed on the screen is automatically calculated, so that privacy protection and user experience can be considered at the same time.

In addition, by providing users with settings for privacy applications, privacy scenarios, and selection of different trigger mechanisms, privacy protection can be more in line with the actual needs of users, so that it varies from person to person, not stereotyped.

In addition, it can be understood that, in order to realize the above functions, the terminal device includes hardware and/or software modules corresponding to each function. Combining the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions in combination with the embodiments for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

In addition, it should be noted that in actual application scenarios, the privacy protection methods provided by the above-mentioned embodiments implemented by the terminal device may also be performed by a chip system included in the terminal device, wherein the chip system may include processor. The system-on-a-chip can be coupled with a memory, so that the system-on-a-chip invokes a computer program stored in the memory when running, so as to realize the above-mentioned steps performed by the terminal device. Wherein, the processor in the chip system may be an application processor or a non-application processor.

In addition, the embodiment of the present application also provides a computer-readable storage medium, the computer storage medium stores computer instructions, and when the computer instructions are run on the terminal device, the terminal device executes the above-mentioned related method steps to implement the above-mentioned embodiment. privacy protection methods.

In addition, an embodiment of the present application also provides a computer program product, which, when running on a terminal device, causes the terminal device to perform the above-mentioned related steps, so as to implement the privacy protection method in the above-mentioned embodiment.

In addition, embodiments of the present application also provide a chip (which may also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein, the transceiver pins and the The processing circuits communicate with each other through internal connection paths, and the processing circuits execute the above-mentioned relevant method steps to implement the privacy protection method in the above-mentioned embodiment, to control the receiving pin to receive signals, and to control the sending pin to send signals.

In addition, it can be seen from the above description that the terminal device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding method provided above, therefore, the beneficial effects it can achieve can be With reference to the beneficial effects in the corresponding method provided above, details will not be repeated here.

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions described in each embodiment are modified, or some of the technical features are replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the various embodiments of the application.

Claims (21)

1. A privacy protection method, characterized in that it is applied to a terminal device, the terminal device includes a front camera, and the privacy protection method comprises: When the set trigger mechanism is satisfied, the front camera is controlled to collect image data; Perform face recognition on the image data, and identify all faces included in the image data; Perform identity recognition on each of the recognized faces, and determine whether there is a face of a non-registered user, and the non-registered user indicates a user who has not obtained the authorization to use the terminal device; When there is a face of a non-registered user, according to the image data, a live body detection is performed on each face of the non-registered user, and the live body detection is used to determine whether the face is a person with vital signs Face; When there is a live non-registered user's face, the terminal device is controlled to make a privacy protection risk prompt, and the privacy protection risk prompt is used to prompt the user that there is currently a privacy leakage risk. 2. The method according to claim 1, wherein the front camera is an infrared depth camera, the image data includes two-dimensional image data and three-dimensional image data, and the two-dimensional image data is an infrared image, so The three-dimensional image data is a depth image. 3. The method according to claim 1, wherein the front camera is a D-RGB camera, the image data includes two-dimensional image data and three-dimensional image data, and the two-dimensional image data is an RGB image, The three-dimensional image data is a depth image. 4. The method according to claim 1, wherein the front camera is an RGB camera, the image data includes two-dimensional image data, and the two-dimensional image data is an RGB image; The method also includes: The RGB image is processed based on a deep learning algorithm to generate a depth image, and the depth image is used as three-dimensional image data. 5. The method according to any one of claims 2 to 4, wherein performing face recognition on the image data to identify all faces included in the image data includes: Perform face recognition on the two-dimensional image data, and identify all faces included in the image data; The identification of each of the recognized faces to determine whether there is a face of a non-registered user includes: According to the two-dimensional image data, perform identity recognition on each of the recognized faces, and determine whether there is a face of a non-registered user; The live detection of the face of each of the non-registered users according to the image data includes: Liveness detection is performed on the face of each of the non-registered users according to the two-dimensional image data and the three-dimensional image data. 6. The method according to claim 5, wherein the identifying each of the recognized faces according to the two-dimensional image data comprises: Extracting facial feature information of each of the human faces from the two-dimensional image data; Matching the facial feature information of each of the faces with the pre-entered facial constitution information of the registered user, the registered user indicating the user who has obtained the right to use the terminal device; If there is no match, it is determined that there is a non-registered user's face. 7. The method according to claim 5, characterized in that, performing liveness detection on the face of each of the non-registered users according to the two-dimensional image data and the three-dimensional image data, comprising: extracting the reflectance information of each of the faces of the non-registered users from the two-dimensional image data; Extracting the three-dimensional information of the human face of each of the non-registered users from the three-dimensional image data; When the stereoscopic information and the reflectance information satisfy the living body standard, it is determined that there is a human face of a non-registered user who is living. 8. The method according to claim 1, wherein the image data comprises two-dimensional image data and three-dimensional image data, the two-dimensional image data is an RGB image or an infrared image, and the three-dimensional image data is a depth image ; Before performing liveness detection on the faces of each of the non-registered users according to the image data, the method further includes: determining the distance between the face of each of the non-registered users and the screen of the terminal device according to the three-dimensional image data; When the distance is less than the set safety distance, the step of performing liveness detection on the faces of each of the non-registered users according to the image data is executed. 9. The method according to claim 1 or 8, wherein before controlling the terminal device to make a privacy protection risk prompt, the method further comprises: perform eye-tracking on the registered user who is a living body according to the image data; When the gaze point of the registered user of the living body is on the screen of the terminal device, the step of controlling the terminal device to make a privacy protection risk prompt is executed. 10. The method according to claim 1, wherein the controlling the terminal device to make a privacy protection risk reminder comprises: In the form of a pop-up window, the privacy protection risk prompt information pops up on the screen of the terminal device. 11. The method according to claim 10, wherein the pop-up window includes a first control and a second control, the first control user closes the pop-up window, and the second control is used to control the The terminal equipment takes anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: When an operation on the second control is received, adjust the font size of the content displayed on the screen of the terminal device, so that the safe distance determined according to the font size is greater than the distance from the screen for a non-registered user who is a living body distance. 12. The method according to claim 10, wherein the pop-up window includes a first control and a second control, the first control is used for the user to close the pop-up window, and the second control is used to control the The terminal equipment takes anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: When an operation on the second control is received, the brightness of the screen of the terminal device is adjusted, so that the visible distance of the screen is greater than the distance of a non-registered user who is a living body from the screen. 13. The method according to claim 10, wherein the pop-up window includes a first control and a second control, the first control is used for the user to close the pop-up window, and the second control is used to control the The terminal equipment takes anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: When an operation on the second control is received, a screen lock operation is performed, so that the screen is in an off-screen state. 14. The method according to claim 10, wherein the pop-up window includes a first control and a second control, the first control is used for the user to close the pop-up window, and the second control is used to control the The terminal equipment takes anti-peeping measures; In the form of a pop-up window, after the privacy protection risk prompt information pops up on the screen of the terminal device, the method includes: When an operation on the second control is not received within a preset time, that is, an operation on the first control is received, a screen lock operation is performed so that the screen is in an off-screen state. 15. The method according to claim 1, wherein the controlling the terminal device to make a privacy protection risk reminder comprises: In the form of voice, play the privacy protection risk reminder information. 16. The method according to claim 15, wherein the playing the privacy protection risk prompt information in voice form comprises: identifying whether the terminal device is connected to an earphone; When the terminal device is connected to an earphone, play the privacy protection risk prompt information through the earphone; When the terminal device is not connected to an earphone, the privacy protection risk prompt information is played through the microphone of the terminal device. 17. The method according to claim 16, wherein before playing the privacy protection risk prompt information through the earphone, the method further comprises: determining whether said headset is worn on the registered user's ear; When the earphone is worn on the ear of the registered user, performing the step of playing the privacy protection risk prompt information through the earphone; When the earphone is not worn on the registered user's ear, the privacy protection risk prompt information is played through the microphone of the terminal device, or the privacy protection risk prompt information is popped up on the screen of the terminal device in the form of a pop-up window. Prompt information. 18. The method according to claim 16, wherein before playing the privacy protection risk prompt information through the microphone of the terminal device, the method further comprises: Detect whether the microphone is registered for external playback; When the microphone does not support external playback, a privacy protection risk prompt message pops up on the screen of the terminal device in the form of a pop-up window; When the microphone supports external playback, the step of playing the privacy protection risk prompt information through the microphone of the terminal device is performed. 19. The method according to claim 15, wherein the playing the privacy protection risk prompt information in voice form comprises: According to the surrounding environment, determine the volume of playing the privacy protection risk prompt information; Play the privacy protection risk prompt information according to the volume. 20. A terminal device, characterized in that the terminal device includes: a memory and a processor, the memory is coupled to the processor; the memory stores program instructions, and the program instructions are executed by the processor , causing the terminal device to execute the privacy protection method according to any one of claims 1 to 19. 21. A computer-readable storage medium, characterized by comprising a computer program, which, when the computer program is run on a terminal device, enables the terminal device to perform the privacy protection according to any one of claims 1 to 19. method.

Images