KR102323650B1 - Image/sound acquiring or editing apparatus for creating an original image/sound file or a deepfake modified file having metadata related to file creating history, hashbank server for receiving and storing hash values related to an original image/sound file or a deepfake modified file, and server and mathod for receiving and processing an original image/recorded sound file or a deepfake modified file - Google Patents

Abstract

An image file processing method performed by an image server for receiving and storing images uploaded from a remote device through a communication network, the method comprising the steps of: receiving an image file; determining whether the image file includes metadata, wherein the metadata indicates that a hash value for the image file is stored in an external hash bank server, the metadata also includes identification information about the hash value stored in the hashbank server, and the metadata indicates whether the image file is an original image file obtained by photographing a subject through a camera or a modulated image file generated by modulating at least a part of a predetermined original image file; and when the image file includes the metadata and the metadata indicates that the image file is the modulated image file, storing the image file together with a label indicating that the image file is a modulated image file. Reliable and easy identification is possible.

Description

A video/audio acquisition or editing device that generates an original video/audio file or deepfake modulation file including metadata related to the video/audio creation history, receiving and storing hash values related to the original video/audio file or deepfake modulation file Hashbank server that receives and processes original video/audio files or deepfake-modulated files HISTORY, HASHBANK SERVER FOR RECEIVING AND STORING HASH VALUES RELATED TO AN ORIGINAL IMAGE/SOUND FILE OR A DEEPFAKE MODIFIED FILE, AND SERVER AND MATHOD FOR RECEIVING AND PROCESSING AN ORIGINAL IMAGE/RECORDED SOUND FILE OR A DEEPFAKE FILE ORIGINAL IMAGE/RECORDED SOUND FILE

The present disclosure discloses, when generating an original (actual) image file by shooting or artificially generating a deepfake-modulated image file by computer work, the generation history of each image (eg, in shooting by a camera) generation history, etc. indicating whether it is a raw original image directly acquired by It relates to a method of making it possible to easily identify whether a corresponding image file distributed after creation is an original image file or a tampered image file based on the above metadata by allowing metadata about the image to be included in each generated image file. . The present disclosure, more specifically, when generating an original image file or a deepfake-modulated image file, each image-related hash value is generated and stored in an external hashbank server, and the corresponding image-related hash value stored in the hashbank server By including the identification information on the hashbank server of the original image file or the tampered image file in the metadata of the image file being created, whether the image file circulated after creation is the original image file or the tampered image file is reliable based on the above metadata and a method for making it easily identifiable.

The development of deep learning technology has also brought great development in image synthesis and modulation technology, and in recent years, a lot of images synthesized or modulated so precisely that it is difficult to distinguish them from actual captured images are being produced and used. Such deep learning-based image synthesis or modulation technology is widely used to manipulate the face of a person in a video, for example, to change a person's facial expression or characteristics, and to change the person's face itself to another person's face. is being utilized As described above, a technique for modulating a person in a given image based on a deep learning algorithm is generally called a deepfake technique, and an image generated by the deepfake technique is called a deepfake image.

Deepfake technology, for example, applies an image modulation model built according to a deep learning algorithm based on other learning images including specific facial expressions to be applied to an original image actually obtained by shooting, A new modulated image with a changed facial expression is created, and an image modulation model built according to a deep learning algorithm is applied to the original image actually obtained by shooting, based on the images of other specific people to be applied, to the original image. It also creates a new modulated image in which the face of the included person is changed to that of another specific person. In particular, when a video about a certain person is given, the deepfake technology acquires a large number of still image information for each frame unit extracted from the video, and applies each of the still image information to a deep learning-based image modulation model. A completely new image is created by modulating and then synthesizing the image, and also, a large number of still image information in each frame unit extracted from a moving picture about an arbitrary person is used as learning data for building an image modulation model.

Recently, deepfake images synthesized or modulated using such deepfake technology are increasingly being abused. For example, by using deepfake technology, a person who manipulates the face of a celebrity to make and distribute a deepfake video that makes it appear as if he has said something that the person did not actually do so to harm the celebrity or deceive those who saw the video In some cases, a specific person's face is synthesized into a malicious video such as pornography to insult the corresponding person. The malicious creation and distribution of such deepfake images often causes fatal damage to those whose faces are stolen for the purpose of creating the deepfake images or to those who believe that such images are actually filmed original images. In particular, as video sharing among the public through video sharing servers becomes more common, a synthetic or modulated deepfake video is a video sharing server without revealing that it is a deepfake video (as if it was an original video that was actually filmed). It can be widely distributed among the public through

Therefore, there is a need for a method for easily and reliably identifying each image file from the image file itself whether the image file is an original image captured by a real camera or a modulated image obtained by modulation of the original image. do. In addition, when an image is shared through the image sharing server, users who play the image from the server notify that it is a deepfake-modulated image or an image with such a possibility, so that the deep-fake-modulated image is misunderstood as the original image. Measures are needed to prevent possible damage in advance.

According to one feature of the present disclosure, as an image editing apparatus, an image file acquisition unit configured to acquire an original image file- The original image file includes original image data obtained by photographing a subject through a camera, and the original image data. Includes original video metadata about -; an image editing unit, a hash generating unit configured to generate modulated image data based on the original image file, wherein the modulated image data is generated by modulating at least a portion of the original image data of the original image file; configured to obtain a predetermined modulated image hash value by applying a predetermined hash function to the modulated image data -, a communication unit configured to communicate with an external hash bank server through a communication network - The communication unit is configured to transmit the modulated image hash value to the hash bank transmitted to a server and configured to receive identification information about the modulated image hash value stored in the hashbank server -, and an image file generating unit- The image file generating unit is configured to include the modulated image data and stored in the hashbank server and generating and storing a modulated image file including identification information regarding the modulated image hash value and modulated image metadata including at least a portion of the original image metadata of the original image file. A device is provided.

According to an embodiment of the present disclosure, the hash bank server is configured to store an original image hash value derived from the original image data, and the original image metadata includes the photographing for obtaining the original image data. It may include at least one of identification information of an image photographing device, an acquisition time of the original image data on the image photographing device, and identification information about the original image hash value stored in the hashbank server.

According to an embodiment of the present disclosure, the image editing unit obtains derived original image data by extracting a part of the original image data of the original image file, and the hash generator includes a predetermined hash on the derived original image data. It is further configured to obtain a derived original image hash value by applying a function, wherein the communication unit transmits the derived original image hash value to the hash bank server, and relates to the derived original image hash value stored in the hash bank server further configured to receive identification information, wherein the image file generating unit includes: the derived original image data; identification information about the derived original image hash value stored in the hashbank server; and the original image metadata of the original image file and generate and store a derived original image file including at least a portion of the derived original image metadata, wherein the image editing unit is further configured to generate the modulated image data based on the derived original image file. can

According to an embodiment of the present disclosure, the original image data may include video data, and the derived original image data may include still image data obtained from arbitrary image frames constituting the video data.

According to an embodiment of the present disclosure, the image editing unit may be configured to generate the modulated image data based on the original image file according to an image modulation model built by a deep learning-based algorithm.

According to an embodiment of the present disclosure, the deep learning-based algorithm may include a GAN algorithm or an Auto-Encoder algorithm.

According to an embodiment of the present disclosure, the modulated image metadata may further include at least one of identification information of the image editing apparatus and a generation time of the modulated image data on the image editing apparatus.

According to another feature of the present disclosure, as a server device, a communication unit configured to communicate with any one of the above-described image editing apparatuses through a communication network, the original image hash value from the image editing device through the communication unit , an information storage configured to receive and store the derived original image hash value or the modulated image hash value, the original image hash value, the derived original image hash value or the modulated image hash received and stored by the information storage unit and an identification information generating unit that generates identification information on the server device regarding a value, wherein the communication unit transmits the identification information generated by the identification information generation unit to each corresponding image editing device through the communication unit. A server device further configured to transmit to

According to another feature of the present disclosure, there is provided an image file processing method performed by an image server for receiving and storing an image uploaded from a remote device through a communication network, the method comprising: receiving an image file, the image file including metadata Whether the metadata indicates that the hash value for the image file is stored in an external hashbank server, and the metadata also includes identification information about the hash value stored in the hashbank server, the Determining whether the image file is an original image file obtained by photographing a subject through a camera or a modulated image file generated by modulating at least a part of a predetermined original image file; and Provided is an image file processing method comprising metadata and, when the metadata indicates that the image file is the modulated image file, storing the image file together with a label indicating that the image file is a modulated image file.

According to an embodiment of the present disclosure, the method includes receiving a request for the image file, and in response to receiving the request, providing the image file, wherein the image file indicates that the image file is a modulated image file. The method may further include providing a label together.

According to an embodiment of the present disclosure, the method includes marking the image file as an original image file when the image file includes the metadata and the metadata indicates that the image file is the original image file. storing the image file together, receiving a request for the image file, and in response to receiving the request, providing the image file, together with the cover indicating that the image file is an original image file may include more.

According to an embodiment of the present disclosure, in the method, when the image file includes the metadata, and the metadata indicates that the image file is the original image file, the metadata is included in the metadata. Based on the identification information about the hash value of the image file stored in the hashbank server, accessing the hashbank server, the hash value stored on the hashbank server about the image file from the hashbank server obtaining a hash value from image data included in the image file, and when the hash value obtained from the image data and the hash value obtained from the hash bank server match each other, The method may further include storing the image file together with a cover called the original image file.

According to an embodiment of the present disclosure, the method includes: when the image file does not include the metadata, inquiring whether the image file is an original image file or a modulated image file; The method may further include the steps of receiving, and when the reply is that the image file is a modulated image file, storing the image file together with a label indicating that the image file is a modulated image file.

According to an embodiment of the present disclosure, the method includes, when the reply is that the image file is an original image file, along with a label indicating that it is not confirmed whether the image file is a modulated image file or an original image file It may further include the step of storing.

According to another feature of the present disclosure, as a server device, there is provided a server device configured to perform any one of the image file processing methods among the above-described image file processing methods.

According to another feature of the present disclosure, there is provided an image photographing apparatus, a camera unit configured to acquire original image data regarding the subject by photographing the subject, and an original image hash value by applying a predetermined hash function to the original image data A hash generator configured to obtain, transmit the original image hash value to an external hashbank server through a communication network, and receive identification information on the hashbank server regarding the original image hash value from the hashbank server A communication unit that operates, and a file creation/storage unit for generating and storing an original image file including the original image data and original image metadata including identification information on the hash bank server regarding the original image hash value , an image capturing apparatus is provided.

According to an embodiment of the present disclosure, the identification information on the hashbank server regarding the original image hash value may include storage location information on the hashbank server of the original image hash value.

According to an embodiment of the present disclosure, the original image metadata may further include at least one of identification information of the image capturing apparatus and an acquisition time of the original image data.

According to an embodiment of the present disclosure, when the original image file is copied to an external device, the original image data and the original metadata may be combined and copied.

According to the embodiment of the present disclosure, it is possible to easily and reliably determine whether each image file is an original image captured by an actual camera or a modulated image obtained by modulation of the original image from the image file itself. have. According to an embodiment of the present disclosure, when an image is shared through an image sharing server, by notifying users who play images from the server that it is a deepfake-modulated image or an image with such a possibility, a deepfake-modulated image It is possible to prevent social confusion and damage that may occur due to misunderstanding that it is an original image, and ultimately, it can promote well-intentioned development research in the field of deep learning-based image modulation research.

1 is a diagram illustrating a schematic configuration of an image generation/sharing system according to an embodiment of the present disclosure.
FIG. 2 is a diagram illustrating a schematic generation process of an original image file by the image photographing apparatus 110 of FIG. 1 and a schematic configuration of the original image file according to an embodiment of the present disclosure.
FIG. 3 is a diagram illustrating a schematic generation process of a derived original image file by the image editing apparatus 130 of FIG. 1 and a schematic configuration of the derived original image file according to an embodiment of the present disclosure.
4 is a diagram illustrating a schematic generation process of a modulated image file by the image editing apparatus 130 of FIG. 1 and a schematic configuration of the modulated image file according to an embodiment of the present disclosure.
5 is a flowchart schematically illustrating a process of uploading an image file by the image sharing server 150 of FIG. 1 according to an embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Hereinafter, when it is determined that there is a risk of unnecessarily obscuring the subject matter of the present disclosure, detailed descriptions of already known functions and configurations will be omitted. In addition, it should be understood that the contents described below are only related to one embodiment of the present disclosure, and the present disclosure is not limited thereto.

The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure. For example, an element expressed in a singular should be understood as a concept including a plurality of elements unless the context clearly means only the singular. It should be understood that the term "and/or" as used in this disclosure encompasses any and all possible combinations by one or more of the enumerated items. Terms such as 'comprise' or 'have' used in the present disclosure are only intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the present disclosure exist, and the terms It is not intended to exclude the possibility of addition or existence of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present disclosure, a 'module' or 'unit' means a functional part that performs at least one function or operation, and may be implemented as hardware or software, or a combination of hardware and software. In addition, the plurality of 'modules' or 'units' may be integrated into at least one software module and implemented by at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware. have.

In an embodiment of the present disclosure, the term "original image" is a live-action image obtained by optically photographing a subject using a camera and converting it into an electrical signal, artificial manipulation using computer graphics on image data for image display It shows the image without adding . Here, the term “artificial manipulation” refers to, for example, image transformation, modulation, etc. (eg, facial expression manipulation, face swapping, etc.) to the extent that it harms the identity of the image based on a deep learning-based algorithm, and the range of the extent to which the identity of the image is not impaired. It should be noted that it does not include minor corrections or corrections made in . In an embodiment of the present disclosure, the original image file may include original image data expressed as an actual image, and may also include original image metadata including various types of information related to the original image file.

In an embodiment of the present disclosure, the original image may include an original original image obtained and generated by being directly photographed by a camera or the like as described above, and a derived original image obtained from the original original image. In an embodiment of the present disclosure, the derived original image is a moving image in which a part is extracted for each time frame from the original moving image data or a moving image formed by bundling the original still image data into each time frame, etc. It may be an image obtained by editing the original image data (however, the value of image data displaying an image in one frame is not changed). In an embodiment of the present disclosure, the derived original image file includes the derived original image data extracted from the original image data of the aforementioned original image file, and also includes various information related to the derived original image file. It should be noted that image metadata is included, wherein the derived original image metadata may include all or at least part of the original image metadata of the original image file on which it is based. Hereinafter, the term derived image and the term derived original image may be used interchangeably, all of which are obtained through extraction or combining operation for each frame from the live-action raw image obtained from the camera, and are used for image display within one time frame. It should be noted that it refers to a thing different from a modulated image, which will be described later, in that it does not change the image data.

In an embodiment of the present disclosure, the term modulated image denotes an image generated by applying artificial modulation to at least a portion of original (or derived original) image data of the aforementioned original (or derived original) image file. Here, the term artificial manipulation refers to image transformation, modulation, etc. (e.g., facial expression manipulation, face swapping, etc.) to the extent that it harms the identity of the image, for example, based on a deep learning-based algorithm, as described above. It should be noted that it does not include minor corrections or corrections (eg, correction of skin expression in the case of a face photo, correction of screen display tone, etc.) made within the range of the degree of harm. In an embodiment of the present disclosure, the modulated image file includes modulated image data generated by applying artificial modulation to at least a portion of the original (or derived original) image data of the aforementioned original (or derived original) image file. and modulated image metadata including various information related to the modulated original image file, wherein the modulated image metadata is an original (or derived original) image of the original (or derived original) image file on which it is based. It should be noted that it may include all or at least part of the metadata.

Hereinafter, with reference to the accompanying drawings, an embodiment of the present disclosure will be described in detail.

1 is a diagram illustrating a schematic configuration of an image generation/sharing system 100 according to an embodiment of the present disclosure. As shown, the image creation/sharing system 100 includes a reward image capturing device 110 , a communication network 120 , a plurality of image editing devices 130 , a hashbank server 140 , and an image sharing server 150 . ) is included.

According to an embodiment of the present disclosure, each of the plurality of image photographing apparatuses 110 optically acquires image information about the subject by photographing the subject through a camera, and converts it into an electrical signal to obtain original (actual) image data can be obtained. According to an embodiment of the present disclosure, each of the image photographing apparatuses 110 may obtain a hash value by applying the original (actual) image data obtained through a camera to a predetermined hash function when the obtained image data is obtained. It may be transmitted to the hashbank server 140 through the communication network 120 . According to an embodiment of the present disclosure, each image photographing device 110 transmits a hash value for each acquired image data to the hash bank server 140 , and then, through the communication network 120 , the hash bank server 140 . ) from the response, for example, identification information that can identify the location where the above-mentioned hash value is stored on the hash bank server 140 can be received. According to an embodiment of the present disclosure, each image photographing device 110 receives the above-described hash value related response from the hash bank server 140, and then the original (actual) image data ( That is, a corresponding original image file may be generated by combining the actually acquired data value for expression of the image) and related metadata. According to an embodiment of the present disclosure, the original image data may include moving image data or still image data. According to an embodiment of the present disclosure, the metadata of the original image file is, for example, information indicating that the image file is a file related to the original original (actual) image data, the hash value received from the hashbank server 140 described above. Related identification information, unique identification information assigned to the corresponding image photographing device 110 (eg, may include, but not limited to, a device serial number, IP information, etc.), and acquisition time information of the corresponding original image data, etc. It may include a variety of related information, including According to an embodiment of the present disclosure, when an original (actual) image file is transmitted or copied to another external device or the like, metadata may be combined with the original image data of the file to be copied.

According to an embodiment of the present disclosure, the communication network 120 may be a predetermined communication network supporting any communication protocol. According to an embodiment of the present disclosure, the communication network 120 is a plurality of image photographing devices 110 , a plurality of image editing devices 130 , and a hash bank server 140 according to any communication protocol such as TCP/IP protocol. ) and information transmission/reception between the image sharing server 150 may be supported.

According to an embodiment of the present disclosure, each of the plurality of image editing apparatuses 130 may receive and store an original (actual) image file generated by the image photographing apparatus 110 . According to an embodiment of the present disclosure, each image editing apparatus 130 may receive an original image file from the image photographing apparatus 110 through, for example, the communication network 120 , and may receive the original image file directly from the image photographing apparatus 110 . You can also copy and save video files. In the drawings and description, each image capturing apparatus 110 and each image editing apparatus 130 are illustrated and described as separate apparatuses separated from each other, but it should be understood that the present disclosure is not limited thereto. According to another embodiment of the present disclosure, it should be noted that the same or a combined device may operate as the image capturing apparatus 110 and the image editing apparatus 130 .

According to an embodiment of the present disclosure, each image editing apparatus 130 may receive an original image file including video data. According to an embodiment of the present disclosure, the image editing apparatus 130 may generate partial information among the original image data of the original image file, for example, image data (ie, still image data) related to each frame of the original video image data. It can be obtained by extraction, but in the present disclosure, the image data extracted from the original image data is called the derived original image data (that is, the actual image data obtained by shooting, but image data extracted from another file and generated). something to do.

According to an embodiment of the present disclosure, the image editing apparatus 130 may obtain a derived original image hash value by applying the derived original image data extracted from the original image data to a predetermined hash function, and the communication network 120 ) through the hashbank server 140 can be transmitted. According to an embodiment of the present disclosure, each image editing device 130 transmits a hash value related to each obtained derived original image data to the hashbank server 140 and then, through the communication network 120 , the hashbank server It is possible to receive a response from the 140, for example, identification information that can identify the location where the above-mentioned hash value is stored on the hash bank server 140, and the like. According to an embodiment of the present disclosure, each image editing device 130 receives the above-described hash value related response from the hash bank server 140, and then combines the previously obtained derived original image data with the related metadata to generate a corresponding original image file. According to an embodiment of the present disclosure, the metadata of the derived original image file includes the original image metadata of the original original image file that is the basis of the derived original image file (eg, the hash value of the original image data of the original image file is It may include all or at least a portion of the stored, hashbank server 140, which may include identification information that can identify a storage location, etc.). According to an embodiment of the present disclosure, the metadata of the derived original image file is information indicating that the image file is a file related to the derived original (actual) image data, received from the above-mentioned hashbank server 140, Acquisition of hash value related identification information, unique identification information (eg, device serial number, IP information, etc., but not limited thereto) given to the corresponding image editing device 130 , and the corresponding derived original image data It may include various related information including time information and the like. According to an embodiment of the present disclosure, when a derived original (actual) image file is transmitted or copied to another external device, or the like, metadata may be copied together with the derived original image data of the file in a combined form.

According to an embodiment of the present disclosure, the image editing apparatus 130 may include original image data from an original image file received from the image capturing apparatus 110 or a derived original image file extracted/generated from the received original image file. A modulated image file may be generated by modulating the original image data derived from the . According to an embodiment of the present disclosure, the image editing apparatus 130 may change the value of the original image data or the derived original image data. According to an embodiment of the present disclosure, the image editing apparatus 130 is, for example, a deep learning-based algorithm (eg, including but not limited to a GAN algorithm, an Auto-Encoder algorithm, etc.) According to the fake image modulation model, modulated image data based on the original image file or the derived original image file may be generated. According to an embodiment of the present disclosure, the image editing apparatus 130 may change the face information of, for example, original or derived original image data including a specific person's face, so that the same person makes a completely different expression or Modulated image data may be generated by modulating to have completely different characteristics or changing/modulating the face of a corresponding person to that of another person.

According to an embodiment of the present disclosure, the image editing apparatus 130 may obtain a modulated image hash value by applying the generated modulated image data to a predetermined hash function, and the hash bank server through the communication network 120 It can be transmitted to (140). According to an embodiment of the present disclosure, each image editing device 130 transmits a hash value related to each obtained modulated image data to the hash bank server 140, and then, through the communication network 120, the hash bank server ( 140) may receive a response, for example, identification information that can identify the location where the above-described hash value is stored on the hash bank server 140, and the like. According to an embodiment of the present disclosure, each image editing device 130 receives the above-described hash value related response from the hash bank server 140, and then combines the previously acquired modulated image data with the related metadata to A corresponding modulated image file may be generated. According to an embodiment of the present disclosure, the metadata of the modulated image file includes the original image metadata of the original original image file that is the basis of the modulated image file (eg, a hash value of the original image data of the original image file is stored, It may include all or at least part of the hash bank server 140 (which may include identification information that can identify the storage location, etc.). According to an embodiment of the present disclosure, the metadata of the modulated image file includes the derived original image metadata of the derived original image file that is the basis of the modulated image file (eg, a hash value of the derived original image data of the derived original image file). It may include all or at least a part of the stored, hashbank server 140, which may include identification information that can identify the storage location, etc.). According to an embodiment of the present disclosure, metadata of a modulated image file includes information indicating that the image file is a file related to modulated image data, hash value-related identification information received from the above-described hash bank server 140 . , unique identification information (eg, may include, but not limited to, device serial number, IP information, etc.) given to the corresponding image editing device 130, and various information including acquisition time information of the corresponding modulated image data. Relevant information may be included. According to an embodiment of the present disclosure, when a modulated image file is transmitted or copied to another external device, etc., the modulated image data and metadata of the file may be copied in a combined form.

According to an embodiment of the present disclosure, the hash bank server 140, as described above, through the communication network 120, from the video photographing device 110 and/or the video editing device 130, each generated raw Each hash value of the original image data, the derived original image data, and/or the modulated image data may be received, and each received hash value may be stored. According to an embodiment of the present disclosure, the hash bank server 140 generates identification information in the hash bank server 140 of each hash value received after the storage of each hash value, for example, hash value storage location information, etc. , and the generated information may be returned to each corresponding image capturing apparatus 110 and/or image editing apparatus 130 through the communication network 120 . According to an embodiment of the present disclosure, the hash bank server 140, when receiving a request for a predetermined hash value stored on the hash bank server 140 using identification information, searches for the requested hash value can be provided. According to an embodiment of the present disclosure, for example, with respect to a predetermined image file distributed as an original (actual) image file, an apparatus for checking whether the image file is in the original state may include a related hash value of the image file (eg, corresponding You may want to check whether the hash value of the image data included in the image file) matches the hash value stored on the hash bank server 140. It is possible to request a hash value stored in the hash bank server 140 based on the. According to an embodiment of the present disclosure, the device compares the hash value received from the hash bank server 140 with the hash value derived from the current image file, and according to whether the match, as an original (actual) image file It can be checked whether the video file being distributed is really a video file about the original (actual) video data.

According to an embodiment of the present disclosure, the video sharing server 150 may be a server device that supports video sharing between a plurality of devices through the communication network 120 . According to an embodiment of the present disclosure, the image sharing server 150 may receive a predetermined image and an image upload request from a device having an arbitrary image, and upload and store the image to the server according to the received request. can According to an embodiment of the present disclosure, the image sharing server 150 may also receive, from any device, a download or real-time playback request for an image uploaded and stored on the server 150, and the reception Upon request, the video may be provided for download or real-time playback.

Specifically, according to an embodiment of the present disclosure, the video sharing server 150 is, through the communication network 120, from an arbitrary device, for example, the video photographing device 110 and/or the video editing device 130, a predetermined of the video file can be received along with the upload request. According to an embodiment of the present disclosure, the image sharing server 150 provides information indicating whether the received image file is original image data or modulated image data, and the image of the image file. It may be determined whether the hash value for the data includes metadata including identification information that can be used to retrieve the hash value from the hash bank server 140 .

According to an embodiment of the present disclosure, the image sharing server 150 may store the image file together with a label of the modulated image file when the metadata of the image file indicates that the image file is a modulated image file. . According to an embodiment of the present disclosure, the video sharing server 150, when a playback request for the corresponding video file is received thereafter, reproduces and provides the data of the video file along with a cover of a modulated video file, thereby viewing it. It is possible to allow a user to view the corresponding image while recognizing that it is a tampered image file.

According to an embodiment of the present disclosure, the image sharing server 150, when the metadata of the image file indicates that the image file is an original (derived original) image file, a hashbank obtained from the metadata of the image file You can access the hash bank server 140 using the hash value identification information on the server 140, and receive a corresponding hash value therefrom. According to an embodiment of the present disclosure, the image sharing server 150 may compare the hash value received from the hash bank server 140 with the hash value directly derived from the image data of the corresponding image file, and whether the Accordingly, the originality of the video file can be checked. According to an embodiment of the present disclosure, the image sharing server 150 may store the image file together with a label of the original image file after confirming the originality of the image file through the hash value comparison process described above. According to an embodiment of the present disclosure, the image sharing server 150, when a request to play a corresponding image file is received thereafter, reproduces and provides the data of the image file together with a cover of the original image file, thereby viewing it. It is possible to allow a user to view the corresponding image while recognizing that it is a tampered image file. On the other hand, for example, in the above-described hash value comparison process, when the hash value received from the hash bank server 140 does not match the hash value directly derived from the image data of the image file, the image sharing server 150 is , the corresponding image file may be stored together with a warning sign indicating that it may be a tampered image. According to an embodiment of the present disclosure, the video sharing server 150, when a playback request for the corresponding video file is received thereafter, sends a warning that the data of the video file may be a forged video file. By providing playback along with the cover, the user who watches the video can watch the video while recognizing the situation.

According to an embodiment of the present disclosure, the image sharing server 150, when it is determined that the image file requested to be uploaded does not include the above-mentioned metadata indicating whether the file is original or tampered with, the image sharing server 150 transmits the upload request. With respect to the sent device, it is possible to inquire whether the corresponding image file is an original image file or a modulated image file. According to an embodiment of the present disclosure, the image sharing server 150, in response to the above-mentioned query, when receiving a response that the corresponding image file is a modulated image file from the corresponding device, converts the corresponding image file into a modulated image. It can be treated as a file, and the corresponding video file can be stored together with a label called a tampered video file. According to an embodiment of the present disclosure, the video sharing server 150, when a playback request for the corresponding video file is received thereafter, reproduces and provides the data of the video file along with a cover of a modulated video file, thereby viewing it. It is possible to allow a user to view the corresponding image while recognizing that it is a tampered image file. On the other hand, if the video sharing server 150 does not obtain a response to the above query within the time limit or receives a response that the corresponding video file is the original video file in response to the query, the corresponding video file is the original video It is regarded as whether it is a file or a tampered image file, and the image file may be stored with a mark indicating that it is a possible tampered image file. According to an embodiment of the present disclosure, the image sharing server 150, when a playback request for the corresponding image file is received thereafter, displays the data of the image file with a mark indicating that the file is likely to be a tampered image file. By providing playback together, a user who watches the video can watch the video while recognizing such a situation.

FIG. 2 is a diagram illustrating a schematic generation process of an original original image file by the image photographing apparatus 110 of FIG. 1 and a schematic configuration of the original image file, according to an embodiment of the present disclosure.

As illustrated, first, in step 210 , the image photographing apparatus 110 may acquire a corresponding still image or video image data by photographing a specific subject through a camera provided in the apparatus. According to an embodiment of the present disclosure, in step 220 , the image photographing apparatus 110 applies a predetermined hash function to the image data obtained in step 210 to obtain a predetermined hash value. can be obtained The hash value obtained here can be said to be a kind of electronic fingerprint for the corresponding image data. According to an embodiment of the present disclosure, the image photographing apparatus 110 may transmit the obtained hash value to the external hashbank server 140 through the communication network 120 in step 230, step ( In 240), it is possible to receive a predetermined response from the hashbank server 140, for example, the identification information of the hash value on the hashbank server 140. According to an embodiment of the present disclosure, the hashbank server 140 may be any predetermined external server device. According to an embodiment of the present disclosure, in operation 250 , the image photographing apparatus 110 includes information obtained in operation 240 and other information about corresponding image data (eg, the image photographing apparatus 110 ). It is possible to generate metadata including identification information such as a device ID of a , information about a capture time of the corresponding image, but is not limited thereto). Then, in operation 260 , the image photographing apparatus 110 may generate and store an original image file including the previously acquired image data and the metadata generated in operation 250 . An exemplary schematic configuration of the original image file generated in step 260 is shown in (A).

FIG. 3 is a diagram illustrating a schematic generation process of a derived original image file by the image editing apparatus 130 of FIG. 1 and a schematic configuration of the derived original image file according to an embodiment of the present disclosure.

As illustrated, first, in operation 310 , the image editing apparatus 130 may acquire derived image data from a predetermined original image file. According to an embodiment of the present disclosure, the original image file may be, for example, obtained by the image editing apparatus 130 from another device through a wired/wireless communication method or other connection method, and the present disclosure is not limited thereto. . According to an embodiment of the present disclosure, the image editing apparatus 130 may obtain, as derivative image data, still image data extracted from each frame of the moving image data, for example, when a given original image file includes moving image data. Also, video data obtained by extracting some time sections from the corresponding video data may be obtained as derived video data. According to an embodiment of the present disclosure, the image editing apparatus 130 may obtain, for example, moving image data obtained by combining respective image data of a plurality of given still image files as derivative image data. According to an embodiment of the present disclosure, the original image file serving as the source from which the image editing apparatus 130 acquires the derived image data in step 310 may be an original original image file obtained by photographing, or another It should be noted that it may also be a derived original image file.

According to an embodiment of the present disclosure, in step 320 , the image editing apparatus 130 applies a predetermined hash function to the derived image data obtained in operation 310 to obtain a predetermined hash value. can be obtained. According to an embodiment of the present disclosure, the image editing apparatus 130 may transmit the obtained hash value to the external hashbank server 140 through the communication network 120 in step 330 . According to an embodiment of the present disclosure, the image editing device 130, in step 330, transmits the obtained hash value to the external hashbank server 140 through the communication network 120, but a derivative image thereof It is possible to transmit identification information of the original image hash value extracted from the original image file of the original image that became the source of . Then, in step 340 , the image editing device 130 may receive a predetermined response from the hashbank server 140 , for example, identification information of the corresponding hash value on the hashbank server 140 . According to an embodiment of the present disclosure, in operation 350 , the image editing apparatus 130 includes information acquired in operation 340 and other information about corresponding image data (eg, the image editing apparatus 110 ). (which may include, but is not limited to, identification information such as device ID of the corresponding image, derivative image data acquisition time information, etc.) You can create metadata including Then, in operation 360 , the image editing apparatus 130 may generate and store a derived original image file including the previously acquired image data and the metadata generated in operation 350 . An exemplary schematic configuration of the derived original image file generated in step 360 is shown in (B).

4 is a diagram illustrating a schematic generation process of a modulated image file by the image editing apparatus 130 of FIG. 1 and a schematic configuration of the modulated image file according to an embodiment of the present disclosure.

As illustrated, first, in step 410 , the image editing apparatus 130 may acquire modulated image data by modulating image data of a predetermined original image file. According to an embodiment of the present disclosure, the original image file may be, for example, obtained by the image editing apparatus 130 from another device through a wired/wireless communication method or other connection method, and the present disclosure is not limited thereto. . It should be noted that, according to an embodiment of the present disclosure, the original image file may be an original original image file directly acquired by camera shooting, or a derived original image file obtained from the original original image file. According to an embodiment of the present disclosure, the image editing apparatus 130 is, for example, a deep learning-based algorithm (eg, including but not limited to a GAN algorithm, an Auto-Encoder algorithm, etc.) According to the fake image modulation model, deep fake modulation image data based on the original image file or the derived original image file may be obtained. According to an embodiment of the present disclosure, the image editing apparatus 130 may change the face information of, for example, original or derived original image data including a specific person's face, so that the same person makes a completely different expression or Modulated image data may be generated by modulating to have completely different characteristics or changing/modulating the face of a corresponding person to that of another person.

According to an embodiment of the present disclosure, in step 420 , the image editing apparatus 130 applies a predetermined hash function to the modulated image data obtained in operation 410 to obtain a predetermined hash value. can be obtained. According to an embodiment of the present disclosure, the image editing apparatus 130 may transmit the obtained hash value to the external hashbank server 140 through the communication network 120 in step 430, and in step ( In 440 , it is possible to receive a predetermined response from the hashbank server 140 , for example, identification information of the hash value on the hashbank server 140 . According to an embodiment of the present disclosure, in operation 450 , the image editing apparatus 130 includes information acquired in operation 440 and other information about corresponding image data (eg, the image editing apparatus 110 ). identification information such as device ID of the corresponding image, information on the acquisition time of modulated image data of the corresponding image, algorithm information used to generate the modulated image data, etc.), and the original source of the modulated image data Metadata including all or part of the metadata of the image file may be generated. Then, in operation 460 , the image editing apparatus 130 may generate and store a modulated image file including the previously obtained modulated image data and the metadata generated in operation 450 . An exemplary schematic configuration of the modulated image file generated in step 460 is shown in (C).

5 is a flowchart schematically illustrating a process of uploading an image file by the image sharing server 150 of FIG. 1 according to an embodiment of the present disclosure.

First, in step 502 , the image sharing server 150 requests a predetermined image file and an upload request for the file from the image capturing device 110 and/or the image editing device 130 through the communication network 120 . can receive In step 504 , the image sharing server 150 may determine whether the received image file includes predetermined metadata related to a hash value of the image file. According to an embodiment of the present disclosure, the hash value related metadata that is determined to be included in the image file here indicates that, for example, a hash value related to the file is stored in an external hash bank server, and also in the hash bank server. It includes identification information about the stored hash value, and through the metadata, whether the corresponding image file is an original image file obtained by photographing a subject through a camera or a modulation generated by modulating at least a part of a predetermined original image file It may be metadata that can determine whether it is an image file.

If it is determined in step 504 that the image file includes certain metadata related to a hash value, the procedure proceeds to step 506, where the image sharing server 150 determines that the metadata indicates that the image file is a tampered image file. It can be determined whether When the metadata indicates that the file is a modulated image file in step 506 , in step 508 , the image sharing server 150 may store the image file together with a label indicating that it is a modulated image file. Although not shown in this figure, when the video sharing server 150 receives a download or real-time playback request for the above-described image file from any device, the image file is provided for download or real-time playback. , a mark indicating that the corresponding image file is a modulated image file may be provided together, so that a user who receives, reproduces and views the corresponding image file can view the image while recognizing that it is a modulated image file.

If the metadata does not indicate that the file is a tampered image file in step 506, in step 510, the image sharing server 150 includes a hash stored in the hashbank server, which is included in the metadata of the image file. Based on the identification information about the value, it is possible to access the hash bank server to obtain a corresponding hash value, compare the obtained hash value with a hash value derived from the image data of the image file to determine whether they match. In step 510, if the hash value on the hash bank server and the hash value derived from the image file match, in step 512, the image sharing server 150 considers that the originality of the image file is confirmed, The image sharing server 150 may store the image file together with a cover called the original image file. Although not shown in this figure, when the video sharing server 150 receives a download or real-time playback request for the above-described image file from any device, the image file is provided for download or real-time playback. , a mark indicating that the corresponding image file is an original image file may be provided together, and thereby, a user who receives, reproduces and views the corresponding image file can view the image while recognizing that it is the original image file. In step 510, if the hash value on the hash bank server and the hash value derived from the image file do not match, in step 514, the image sharing server 150 determines that the originality of the image file is not confirmed. See, the image sharing server 150 may store the image file, but may store the image file together with a cover indicating that it is not confirmed whether the image file is an original image file or a forged image file. Although not shown in this figure, when the video sharing server 150 receives a download or real-time playback request for the above-described image file from any device, the image file is provided for download or real-time playback. , a mark indicating that it is not confirmed whether the image file is an original image file or a forged image file may be provided together, so that a user who receives, reproduces, and views the image file may find that the image file is a tampered image file. You can watch the video while recognizing that there is a possibility.

Again, returning to step 504, if it is determined that the image file does not include certain metadata related to a hash value, the procedure proceeds to step 516, where the image sharing server 150 receives the device requesting upload of the file. , it is possible to inquire whether the image file is an original image file or a modulated image file, and receive an answer to the inquiry. In step 518, the reply to the query in step 516 determines whether the image file is a tampered image file, and if so, the procedure proceeds to step 520 where the image sharing server 150 determines the corresponding image file. A video file can be saved with a label called a tampered video file. In step 518 , if a reply to the query in step 516 is not received within a predetermined time period, or if the image file in question is the original image file, the procedure may proceed to step 514 . As described above, in step 514 , the image sharing server 150 may store the image file, but may store the image file together with a label indicating whether it is an original image file or a modulated image file. Although not shown in this figure, when the video sharing server 150 receives a download or real-time playback request for the above-described image file from any device, the image file is provided for download or real-time playback. , a mark indicating that it is not confirmed whether the image file is an original image file or a forged image file may be provided together, so that a user who receives, reproduces, and views the image file may find that the image file is a tampered image file. You can watch the video while recognizing that there is a possibility.

Although the present specification has been mainly described with respect to editing or modulation of an image file (which may or may not include an audio), the present disclosure is not limited thereto. It should be noted that the present disclosure can also be used for editing or altering audio files that do not include images. That is, the principle of the present disclosure is, in relation to an audio file that does not include an image, when generating an original audio file or a modulated audio file, each audio-related hash value is generated and stored in an external hash bank server, and the hash value is generated. By including the identification information on the hash bank server of the corresponding voice-related hash value stored in the bank server in the metadata of the original voice file or the modulated voice file being created, whether the voice file circulated after creation is the original voice file or the altered voice file It should be noted that recognition may be implemented as a way to enable reliable and easily identifiable based on the above metadata.

As will be appreciated by those skilled in the art, the present invention is not limited to the examples described in the present disclosure, and various modifications, reconstructions and substitutions may be made without departing from the scope of the present disclosure. It should be understood that the various techniques described herein may be implemented by hardware or software, or a combination of hardware and software.

Claims (19)

delete delete delete delete delete delete delete delete An image file processing method performed by a video sharing server operable to receive and store an image file uploaded from a remote device through a communication network, and provide the image file to the other remote device based on a request from another remote device As,
Receiving an image file including image data including a face of a person;
determining whether the image file contains metadata;
The metadata includes identification information for identifying information related to the image file stored on the external server on the external server,
The metadata is determined whether the image data is original image data obtained by photographing the face of the person through a camera, or the original image data obtained by photographing the person's face through a camera by a deep learning-based algorithm. On the basis of the constructed image modulation model, the face of the person included in the original image data is replaced with the face of another person having no identity, or the expression of the face of the person included in the original image data is replaced with another expression having no identity. Indicates whether it is deepfake-modulated image data generated by modulating in a way that replaces -,
When the image file includes the metadata and the metadata indicates that the image data is the deepfake-modulated image data, storing the image file together with a mark of a deepfake-modulated image file; and
When the image file includes the metadata and the metadata indicates that the image data is the original image data, the image file is sent to the external server based on the identification information related to the external server included in the metadata. access, and confirm whether the image data is the original image data based on the information stored in the external server, and when it is confirmed that the image data is the original image data, the image file is steps to save
Image file processing method comprising a. 10. The method of claim 9,
receiving the request for the image file from the other remote device; and
In response to receiving the request, providing the image file to the other remote device, together with the image file, a mark of the deepfake modulation image file associated with the image file or a mark of the original image file Image file processing method further comprising the step. delete delete 10. The method of claim 9,
When the image file does not include the metadata, whether the image data is original image data obtained by photographing the person's face through a camera, or original image data obtained by photographing the person's face through a camera to replace the face of the person included in the original image data with the face of another person having no identity based on the image modulation model built by the deep learning-based algorithm, or Inquiring whether it is deepfake-modulated image data generated by replacing the expression of a face with another expression that does not have the same identity;
receiving a reply to the query; and
When the reply is that the image data is deepfake-modulated image data, the method further comprising the step of storing the image file together with a mark of a deepfake-modulated image file. 14. The method of claim 13,
If the reply is that the image data is the original image data, further comprising the step of storing the image file together with a label indicating that it is not confirmed whether the image file is an original image file or a deepfake-modulated image file Way. As a server device,
An image sharing server device, configured to perform the image file processing method according to any one of claims 9, 10, 13, and 14. delete delete delete delete

Images