KR20240085268A - Method for generating creative emoticon that communicate user's emotion more effectively and creative emoticon-based content service system for performing the same - Google Patents

Abstract

Embodiments include obtaining emoticon object data related to an emoticon object sequence image from a shooting sequence image, requesting a purchase of a background sticker from a market site to a server operating the market site, and receiving the corresponding background sticker data. Recognizing the motion of the emoticon object in the sequence image by inputting the emoticon object sequence image into a previously learned motion recognition model; Obtaining semantic information of background visual elements included in the background sticker from the background sticker data; Generating creative emoticon data to more effectively convey the user's emotions, comprising generating creative emoticon data combining a background sticker and the emoticon object sequence image based on the recognition result of the motion and semantic information of the background visual element. It relates to a method of doing this and a creative emoticon-based content service system that performs this.

Description

{Method for generating creative emoticon that communicate user's emotion more effectively and creative emoticon-based content service system for performing the same}

Embodiments of the present application relate to a system that provides content services, and more specifically, to a method of generating a creative emoticon using a user's image and a creative emoticon-based content service system that performs this.

As IT technology, including information and communications, develops, social networks that form relationships and communicate online have developed. Through social networks, people can easily accept content without much difficulty.

Recently, a true content society is emerging in which consumers go beyond unilaterally accepting corporate content and instead create their own content and communicate through content. Therefore, it has become a popular culture for content creators to express and reveal themselves. These content creators are active on video platforms such as YouTube and TikTok or image-based social networking platforms such as Instagram and Facebook.

Meanwhile, the messenger field is also an area where social networking activities are active. SNS and mobile platform services are being actively implemented centering on mobile messengers installed on smart phones.

In chatting (conversation) using mobile messengers, unlike the conventional text service (SMS), there are many attempts to convey the speaker's emotions more effectively by actively using emoticons.

However, in the messenger field, content creators still rarely create and utilize self-produced content to express themselves, and the environment that can support this is also insufficient.

Patent Registration Publication No. KR 10-2007506 (2019.07.30.)

According to embodiments according to one aspect of the present application, a method for generating a creative emoticon using a user's image is intended to be provided.

In addition, a creative emoticon-based content service system that performs the above method is provided.

A method of generating a creative emoticon that can be transmitted through a messenger application from the electronic device of a user registered as a member of an emoticon service according to an aspect of the present application includes a captured sequence image of the user continuously photographing the user implementing one or more actions. and acquiring emoticon object data related to the emoticon object sequence image from the captured sequence image, wherein the captured sequence image includes a subset according to each operation, and the emoticon object sequence image includes at least a partial region within the captured sequence image. Each contains a sequence image in which an emoticon object appears; A step of requesting a purchase of the background sticker from a market site where content products including background stickers and the creative emoticon are traded to a server operating the market site and receiving the corresponding background sticker data - the background sticker includes one or more background stickers an element having an animation sequence image represented by animation, the animation sequence image including one or more subsets corresponding to each background visual element, the background visual element being a background text or background object; Inputting the emoticon object sequence image into a previously learned motion recognition model to recognize the motion of the emoticon object in the sequence image; Obtaining semantic information of background visual elements included in the background sticker from the background sticker data; and generating creative emoticon data combining a background sticker and the emoticon object sequence image based on a recognition result of the motion and semantic information of the background visual element.

In one embodiment, based on the recognition result of the motion and the semantic information of the background visual element, creative emoticon data is generated by combining an emoticon object sequence image in which an emoticon object appears with at least a portion of a background sticker and a shooting sequence image, respectively. The step includes, when the frame size of the purchased background sticker and the frame size of the emoticon object sequence image are different, adjusting the frame size of the emoticon object sequence image to match the frame size of the purchased background sticker; calculating a sub-output section according to the motion recognized by the motion recognition model from the entire output section of the captured sequence image; calculating a sub-output section according to the meaning of the background visual element shown in each animation sequence image in the entire output section of the background sticker; Implementing an emoticon object sequence image in the first layer using the emoticon object data associated with the emoticon object sequence image with the frame size adjusted; For each recognized motion, if the meaning of a background visual element shown in each animation sequence image corresponds to a meaning previously associated with the corresponding motion, mapping the animation sequence image and the corresponding motion; adjusting the animation sequence image data of the background sticker data so that the starting point of the sub-output section of the mapped background visual element matches the starting point of the sub-output section of the corresponding motion; It may also include implementing an animation sequence image of the background sticker in the second layer using the background sticker data including adjusted animation sequence image data.

In one embodiment, when a single action is recognized and the background sticker includes animation sequence images in different output orders, if the meaning of the background visual element appearing in each animation sequence image corresponds to a meaning previously associated with the action, Mapping the animation sequence image and the corresponding motion includes mapping the plurality of background visual elements to the recognized single motion. The step of adjusting the animation sequence image data of the background sticker data includes dividing the start point of the sub-output section of the animation sequence image with the earliest output order among the plurality of animation sequence images into the start point of the sub-output section of the corresponding operation. The animation sequence image data of the background sticker data may be adjusted to match.

In one embodiment, when the plurality of consecutive actions are recognized and the background sticker includes a single animation sequence image, if the meaning of the background visual element appearing in the single animation sequence image corresponds to the meaning previously associated with each action , the step of mapping the animation sequence image and the corresponding motion may include mapping each of a plurality of motions to the single animation sequence image. The step of adjusting the animation sequence image data of the background sticker data includes matching the starting point of the sub-output section of the animation sequence image with the starting point of the sub-output section of the operation having the earliest output order among the plurality of consecutive operations. The animation sequence image data of the background sticker data may be adjusted to do so.

In one embodiment, when a plurality of consecutive actions are recognized and the background sticker includes a plurality of animation sequence images, if the meaning of the background visual element appearing in the single animation sequence image corresponds to the meaning previously associated with each action , The step of mapping the animation sequence image and the corresponding motion may include mapping each of the plurality of motions with the plurality of animation sequence images. Adjusting animation sequence image data of the background sticker data may include specifying an output order among the plurality of animation sequence images; Specifying an output order between a plurality of recognized operations; and adjusting the animation sequence image data of the background sticker data so that the starting point of the sub-output section of each animation sequence image matches the starting point of the sub-output section of the operation that is mapped to each other and has the same output order.

In one embodiment, the unit of the background visual element for determining whether the meaning of the background visual element corresponds to a meaning previously associated with the corresponding action may be a text group or an object group to which the same animation effect is applied. The meaning of the background visual elements is classified into emotion types. The meaning of the background visual element corresponds to the meaning previously associated with the corresponding action, which means that one of the emotion types previously associated with the action recognized using the action semantic table recording the emotion type previously associated with each action and The emotion type classified by the background visual element may match.

In one embodiment, the method includes obtaining the text of a message of the user or another user, wherein the message is transmitted to the electronic device of the other user by executing any messenger application installed on the electronic device. It may further include a message or a message from the other user received from the other user's electronic device. The creative emoticon is created to be transmitted to the other user's electronic device through the messenger application. The step of generating creative emoticon data combining a background sticker and the emoticon object sequence image based on the recognition result of the motion and the semantic information of the background visual element includes at least some of the messages obtained before transmitting the creative emoticon message. Inputting the text of the message into a pre-trained natural language processing model to calculate each embedding vector obtained by embedding the text of each message; inputting a set of embedding vectors for the at least some messages into a previously learned text-based emotion recognition model to calculate the user's emotional intensity level reflected in the at least some messages; and adjusting the animation effect or the size of the background visual element according to the calculated emotional intensity level. The step of implementing an animation sequence image of the background sticker in the second layer using the background sticker data includes the step of implementing the animation sequence image of the background sticker in the second layer using the background sticker data adjusted according to the calculated emotional intensity level. It may be implementing an animation sequence image.

In one embodiment, the step of adjusting at least one of the animation effect of the background sticker data and the background visual element according to the calculated emotional intensity level may include adjusting the animation speed to a lower emotional intensity level as the calculated emotional intensity level is higher. Adjust the background sticker data to increase the animation speed in or change the size of the background visual element according to the animation effect as the calculated emotional intensity level is higher. The background visual element according to the animation effect at a lower emotional intensity level. It may also include adjusting the background sticker data to increase the size change.

In one embodiment, the text-based emotion recognition model is a natural language processing model that calculates an embedding vector by natural language processing the input text, and an emotion predicted to be held by the user at the time of generating the input text by processing the input embedding vector. It may also include a classification model that calculates the intensity level.

The natural language processing model, when a group of a series of messages is input, natural language processes each of the series of messages continuously input, calculates a series of embedding vectors for the series of messages, and combines the contents of the series of messages in order. Obtaining text for the entire content of the series of messages, processing the entire content of the series of messages in natural language to calculate an embedding vector for the entire content of the series of messages, and a series of embedding vectors for the series of messages, and By concatenating the embedding vectors for the entire content of a series of messages, a single embedding vector for the group of input messages is produced. Calculating the user's emotional intensity level reflected in the at least some messages by inputting a set of embedding vectors for the at least some messages into a pre-learned text-based emotion recognition model includes selecting a single embedding vector for the group of the input messages. It may be input to the text-based emotion recognition model to calculate the emotional intensity level analyzed in terms of content and context of the group of the input message.

A computer-readable recording medium according to another aspect of the present application includes a program that performs a method of generating a creative emoticon that can be transmitted through a messenger application on the electronic device of a user registered as a member of the emoticon service according to the above-described embodiments. Record it.

Embodiments of the present application allow the user to create his or her own creative emoticon using an image showing the user's own face and send it as a message.

Additionally, the creative emoticon may be configured to convey emotions through emoticons more effectively.

In various embodiments, the creative emoticon is created to provide an animation effect that appears at the start of motions where the animated sequence image in the background sticker has a corresponding meaning. As a result, the action is provided first to the other person who received the creative emoticon, and animation effects with the same meaning are provided in duplicate, thereby further reducing the impact of the animation effect, or the animation effect is provided first, thereby reducing the impact of the emoticon object to take. It is possible to prevent the problem that the influence of motion transmission is further reduced by spoiling the motion.

Additionally, the animation effect of the creative emoticon is adjusted according to the emotional intensity level, thereby enriching the level of emotion conveyed by the creative emoticon.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

In order to more clearly explain the technical solutions of the embodiments of the present invention or the prior art, drawings necessary in the description of the embodiments are briefly introduced below. It should be understood that the drawings below are for illustrative purposes only and not for limiting purposes of the present specification. Additionally, for clarity of explanation, some elements may be shown in the drawings below with various modifications, such as exaggeration or omission.
Figure 1 is a block diagram showing a creative emoticon-based content service system according to one aspect of the present application.
Figure 2 is a schematic diagram of the operation of the system of Figure 1, according to various embodiments of the present application.
Figure 3 is a flowchart of a process for granting permission to create emoticons according to various embodiments of the present application.
Figure 4 is a flowchart of a creative emoticon creation process according to various embodiments of the present application.
Figure 5 is a schematic diagram of the creative emoticon creation process of Figure 4.
Figure 6 shows the structure of a creative emoticon according to various embodiments of the present application.
Figure 7 shows a screen of an electronic device for accessing the location of the creation storage space, according to various embodiments of the present application.
Figure 8 shows a screen for storing a creative emoticon in a second storage space according to various embodiments of the present application.
FIG. 9 shows a screen for transmitting a creative emoticon stored in the second storage space of FIG. 8 through the corresponding messenger application.
Figure 10 shows a screen for product registration request according to various embodiments of the present application.
Figure 11 shows a screen showing the registration result according to the product registration request of Figure 10.
Figure 12 is a flowchart of the voting event progress process according to various embodiments of the present application.
13 illustrates a voting screen according to various embodiments of the present application.
Figure 14 shows a voting message screen according to various embodiments of the present application.
FIG. 15 is a flowchart of a process for generating a creative emoticon based on semantic information of a background visual element and semantic information of an action of an emoticon object in a captured image, according to various embodiments of the present application.
FIG. 16 illustrates a creative emoticon in which a single action and a single animation sequence image are mapped to each other, according to various embodiments of the present application.
FIG. 17 illustrates a creative emoticon in which a plurality of actions and a single animation sequence image are mapped to each other, according to various embodiments of the present application.
FIG. 18 illustrates a creative emoticon in which a plurality of actions and a plurality of animation sequence images are mapped to each other, according to various embodiments of the present application.
Figure 19 is a flowchart of a process for generating a creative emoticon based on the emotional intensity of a previously obtained message, according to various embodiments of the present application.
Figure 20 is a schematic diagram of a learning process of a text-based emotion recognition model according to various embodiments of the present application.
Figure 21 shows a creative emoticon in which the size change of the background visual element is adjusted according to the intensity of emotion, according to various embodiments of the present application.
Figure 22 is a schematic diagram of a content service system configured to issue NFTs for content products, according to another aspect of the present application.
Figure 23 is a flowchart of the process of issuing a creative emoticon NFT according to various embodiments of the present application.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

However, this disclosure is not intended to limit the disclosure to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of the disclosure. . In connection with the description of the drawings, similar reference numbers may be used for similar components.

In this specification, expressions such as “have,” “may have,” “includes,” or “may include” refer to the relevant features (e.g., numerical values, functions, operations, steps, parts, elements, and/or components). It refers to the presence of components such as etc.) and does not exclude the presence or addition of additional features.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Expressions such as “first,” “second,” “first,” or “second” used in various embodiments may modify various elements regardless of order and/or importance, and limit the elements. I never do that. The above expressions can be used to distinguish one component from another. For example, the first component and the second component may represent different components, regardless of order or importance.

Embodiments of singular expressions used in this specification also include embodiments of plural expressions, unless phrases related to the singular expression clearly indicate the contrary.

The expression “configured to” used in this specification may mean, for example, “suitable for,” “having the capacity to,” depending on the situation. ,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured (or set) to” may not necessarily mean “specifically designed to” in terms of hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” refers to a processor dedicated to performing those operations (e.g., an embedded processor), or executing one or more software programs stored on a memory device. By doing so, it may mean a general-purpose processor (eg, CPU or application processor) capable of performing the corresponding operations.

Terms including technical or scientific terms used in the present invention may have the same meaning as generally understood by a person of ordinary skill in the technical field described in the present invention. Among the terms used in the present invention, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in the present invention, have an ideal or excessively formal meaning. It is not interpreted as In some cases, even terms defined in the present invention cannot be interpreted to exclude embodiments of the present invention.

FIG. 1 is a block diagram showing a creative emoticon-based content service system according to one aspect of the present application, and FIG. 2 is a schematic diagram of the operation of the system of FIG. 1 according to various embodiments of the present application.

Referring to FIG. 1, a creative emoticon-based content service system 1 (hereinafter, “content service system”) according to various embodiments of the present invention includes an electronic device 110, a telecommunication network 120, and a server 130. Includes. In various embodiments, the content service system 1 is linked with a third-party system 2 that exchanges messages between users through a messenger application.

The content service system 1 according to embodiments may be entirely hardware, or may have aspects that are partly hardware and partly software. For example, a system may collectively refer to hardware equipped with data processing capabilities and operating software for running it. In this specification, terms such as “unit,” “system,” and “device” are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device that includes a Central Processing Unit (CPU), Graphics Processing Unit (GPU), or other processor. Additionally, software may refer to a running process, emoticon object, executable, thread of execution, program, etc.

The telecommunication network 120 provides a wired/wireless telecommunication path through which the electronic device 110, the server 130, and the third-party system 2 can transmit and receive data with each other. A telecommunication network is not limited to a communication method according to a specific communication protocol, and an appropriate communication method may be used depending on the implementation. For example, when configured as an Internet Protocol (IP)-based system, the telecommunication network may be implemented as a wired and/or wireless Internet network. Alternatively, when the electronic device 110 and the server 130 are implemented as mobile communication terminals, the telecommunication network may be implemented as a wireless network such as a cellular network or a wireless local area network (WLAN) network.

Third-party system 2 may be one or more types of servers, one or more data stores, one or more interfaces, including but not limited to APIs, one or more web services, one or more content sources, one or more networks, or e.g. It may include any other suitable components. Third-party system 2 may be operated by a user different from the user operating server 130. However, in various embodiments, server 130 and third-party system 2 may operate in conjunction with each other to transmit creative emoticons in messages.

In this sense, another system, such as a third-party system 2, may provide the server 130 with a platform or backbone that can be used to perform messenger functions.

The third-party system 2 includes a messenger application. The messenger application supports sending messages between users by communicating with the electronic device 110 or the server 130. The messenger application may be, for example, Facebook Messenger, Kakao Talk, Line, WeChat, or any other mobile messenger application capable of sending emoticons.

Third-party systems 2 may also include social networking applications. To this end, the content service system 1 may be linked with a third-party system 2 that exchanges messages between users through a social network application. The creative emoticon can also be used to convey a specific user's emotions to other users through a messenger application or social network application.

Hereinafter, for clarity of explanation, the present invention will be described in more detail through embodiments using a messenger application.

The electronic device 110 is a client terminal device that communicates with the server 130. Additionally, the electronic device 110 may allow its user to communicate with other users of other electronic devices 110 . To this end, the electronic device 110 may allow a network user in the electronic device 110 to access the telecommunication network 120 .

The electronic device 110 may be a computing system that includes hardware, software, or embedded logic components, or a combination of two or more of these components, and is capable of performing appropriate functions implemented or supported by the electronic device 110. The electronic device 110 may include, for example, a desktop computer, laptop computer, netbook, tablet computer, e-book reader, GPS device, camera, personal digital assistant (PDA), portable electronic device, cellular phone, smartphone, It may include computer systems such as other computing devices, other mobile devices, other wearable devices, other suitable electronic devices, or any suitable combination thereof.

Users 100_1 and 100_2 use the electronic device 110 to acquire the right to sell creative content, create creative content, transmit the creative content to other users through a messenger application, and send the creative content to a market site. refers to a person who implements one or more of the following: registering and selling the creative content, purchasing the creative content on a market site, and performing various activities related to the creative content.

In this specification, a user may be a person, a group of people, or an organization such as a company.

Even if the electronic device 110 is the same, the electronic device 110 is configured to perform different functions more intensively depending on the users 100_1 and 100-2 of the electronic device 110. Hereinafter, for clarity of explanation, the electronic device 110 will be classified into the electronic device 110_1 of the first user 100_1 and the electronic device 110_2 of the second user 100_2.

In the content service system 1, the first user 100_1 is a user who creates creative content and may also be referred to as a content creator. The first user 100_1 may have content creation rights to sell creative content as a product through a market site operated by the server 130. The first user 100_1 may use his/her electronic device 110_1 to transmit the creative content he/she created to another user as a message through a messenger application.

The first user 100_1 may create creative content representing himself or may create creative content representing another person, animal, or plant.

In the content service system 1, the second user 100_2 may purchase a content product created by the first user 100_1 through the market site. The second user 100_2 may use his/her electronic device 1102 to transmit the purchased creative content of the first user 100_1 to another user as a message through a messenger application.

In various embodiments of the present application, the first user 100_1 and the second user 100_2 may have a relationship between a star and a fan. When the star (100_1) creates creative content, the fan (100_2) can purchase the creative content created by the star (100_1) and use it in the messenger application.

Here, a star refers to a person someone prefers or supports, and is not limited to traditional celebrities such as singers, actors, comedians, politicians, and other broadcasters, but also relatively ordinary people such as YouTubers and Instagram influencers. It should be understood as a concept that includes famous people close to .

In the content service system 1, the first user 100_1 and the second user 100_2 are not fixed. The specific user may be a first user (100_1) who creates original content and may also be a second user (100_2) for another user.

In the content service system 1, users 100_1 and 100_2 may receive points by performing various activities on the content service system 1. In some embodiments, the activity may be a voting event that measures the popularity of creative content.

In this specification, creative content may be a creative emoticon. The creative emoticon is an emoticon created using an image taken by the first user (100_1) and an image purchased by the first user (100_1) through a market site. The first user 100_1 may create a creative emoticon based on the captured image and the purchased image, or may create a creative emoticon based only on the purchased image.

Server 130 may be a unitary server, or may be a distributed server spanning multiple computers or multiple data centers. In various embodiments, the server 130 is connected to a lower-level device that can communicate with another network server through a computer network such as a private intranet or the Internet, receives a request to perform a task, performs the task, and provides a performance result. means computer systems and computer software (network server programs). However, the server 130 should be understood as a broad concept that includes, in addition to the network server program, a series of application programs operating on the server and, in some cases, various databases built within the server. Server 130 may include, by way of example and without limitation, a web server, news server, mail server, message server, advertising server, file server, application server, exchange server, database server, proxy server, etc. that perform the functions or processes described herein. It may be of various types, such as another server suitable for use, or any combination thereof. In certain embodiments, server 130 may include hardware, software, or embedded logic elements, or a combination of two or more such elements, to perform appropriate functions implemented or supported by server 130.

In various embodiments, the content service system 1 may include one or more data stores 134 that store data of the server 130. Data store 134 may be used to store various types of information. In some embodiments, information stored in data store 134 may be structured according to a particular data structure. Additionally, in some embodiments, each data store 134 may be a relational, columnar, correlational, or other suitable database. Although this specification describes or illustrates a specific type of database, this specification contemplates any suitable type of database. In some embodiments, the server 130 may provide an interface that allows the electronic device 110 to manage, search, change, add, or delete information stored in the data store 134.

In the content service system 1, the manager 100_3 accesses the server 130 using a third electronic device 110_3 and operates a market site that sells creative emoticons as products through the server 130, One or more of the following: granting permission to sell creative content, permitting the use of specific marks while creating creative content, permitting registration of creative content on a market site, and holding various events related to creative content. It means the person who implements it. The manager 100_3 may be a person with the authority to operate the server 130 below.

As shown in FIG. 2, the server 130 is configured to support the creation of creative emoticons or to support transactions. Various user activities are possible through emoticon creation services, market services, voting events, and other event provision services supported by the server 130, and compensation may be paid to each user depending on the ripple effect of user activities.

In certain embodiments, the server 130 includes a member management module 131, a market management module 133, a point management module 135, and an event management module 137.

The member management module 131 is an administrator member (manager 100_3 in FIG. 1), general member (second user 100_2 in FIG. 1), and creative member (first user in FIG. 1) for the content service (or emoticon service). User (100_1)) is configured to perform various member management operations such as checking each member's information, checking system usage history, or changing user status.

The member information includes user identification information and user status information. The user's identification information includes one or more of the user's offline identification information, online identification information, and identification information of the electronic device 110 obtained when the user registers as a member. Additionally, the member information may further include user profile information. Profile information is information that describes the user.

The offline identification information is the user's identity information and may include, for example, one or more of name, address, email account, phone number, resident registration number, driver's license, and other ID numbers.

The online identification information may be a member ID of the content service system 1.

The identification information of the electronic device 110 may include one or more of a device identifier and a network identifier. The network identifier may be, for example, a MAC address (Media Access Control Address) or an IP address (internet protocol address), but is not limited thereto.

The status information indicates the user's authority. The user's state may be a first state indicating a first user (100_1) with content creation authority such as emoticon creation authority, a second state indicating an administrator (100_3) with management authority, or a second state indicating neither content creation authority nor management authority. It may also be a third status referring to general members.

The system usage details include content product creation details, content product purchase details, payment details, point settlement details, and event activity details. In various embodiments, the event activity details include voting event participation details.

In various embodiments, the member management module 131 may grant permission to sell emoticons to the first user 100_1 using an electronic signature. To this end, the content service system 1 may be connected to the authentication system 3.

In this specification, an electronic signature is digital information attached to a document to be signed, and is used for the purpose of verifying the identity of the user who created the electronic signature and indicating the user's approval of the electronic document.

A document subject to signature refers to an electronic document that requires the user's electronic signature. The electronic device 110 can authenticate that the user 100 approves the document to be signed by electronically signing the document to be signed. In the content service system 1, the document to be signed is generated by the server 130 and includes an electronic contract document for granting sales authority. In some embodiments, the document to be signed is generated by the electronic device 110 and may further include a creative emoticon for registering a new product on a market site.

The signature target data is data to be derived from the signature target document. As an example, the signature target data may be data processed from the original data of the signature target document using a predetermined algorithm. In one example, the signature target data may be data obtained by processing the original data of the signature target document using a hash function. In another example, the signature target data may be data processed from the original data of the signature target document using a hash function and a padding algorithm. Padding refers to a technique of adjusting the size of data by adding blank data or meaningless symbol data to the end of the data when the data size is smaller than the target fixed size.

The data processing algorithms described above are merely illustrative, and the embodiment is not limited thereto. Additionally, the data to be signed may be the same as the original data of the document to be signed.

As an example, the authentication system 3 may be configured to issue a pair of asymmetric keys for electronic signatures in a public key infrastructure (PKI). The asymmetric key is the private key of the first user 100_1 and the public key corresponding to the private key.

The authentication system 3 transmits the private key or asymmetric key pair of the first user 100_1 to the first electronic device 110_1 and transmits the public key of the first user 100_1 to the server 130. .

These authority operations are described in more detail with reference to Figure 3 below.

The market management module 133 is configured to manage a market site where content products are traded. The content product may be transmitted together with text through a messenger application or may be transmitted alone.

The content product includes a creative emoticon and a background sticker used to create the creative emoticon. In some embodiments, the content product may further include an emoticon object used to create a creative emoticon along with a background sticker.

The background sticker is an emoticon component that provides a background image for a creative emoticon. The background sticker is placed on the emoticon object. A portion of the emoticon object overlaid on the background sticker may be obscured on a two-dimensional plane. When the background sticker is replaced, the background image of the creative emoticon changes. Because of this arrangement structure and replacement effect, emoticon backgrounds for creative emoticons are referred to as background stickers.

The background sticker includes only a background image, or is configured to have text and/or sound as background elements in addition to the image. The background image consists of background objects with shapes, colors, etc. For example, a background sticker may be implemented as multi-media data, including a background image in which the background object appears, background text, and background sound. The voice expresses content related to the text and image in the background sticker as a voice signal.

The background sticker may be a still image or a sequence image consisting of a series of images showing continuously moving background elements such as background text and background objects. If the background sticker is a sequence image, the background sticker provides an animation effect. The background sticker may have a structure of combining animation sequence images implemented on different layers for each group of background visual elements. In some embodiments, the background sequence image may be implemented as Graphics Interchange Format (GIF) image data. Then, the background sequence image may be sequentially output from the image in the first output order to the image in the last output order, and then the image in the first output order may be output again.

The emoticon object may be an emoticon object representing the first user 100_1. The emoticon object may be acquired through a market site that sells the emoticon object as a content product, or may be obtained through a captured image of the emoticon object as a subject.

The captured image may be a still image or a series of images showing the emoticon object moving continuously. If the captured image of the emoticon object is a sequence image, the emoticon object image provides an animation effect.

In various embodiments, the emoticon object may be an image showing the face of the first user 100_1. However, the emoticon object is not limited to this, and the emoticon object may be an image showing another person, animal, or plant, such as the face of the first user 100_1 that the second user 100_2 likes.

The market management module 133 may approve or reject a product registration request from the first user 100_1 who is granted sales authority. This will be described in more detail with reference to Figure 9 below.

Additionally, the market management module 133 may manage the main screen of the market site. The main screen may be the site screen displayed when first accessing the market site.

In various embodiments, the market management module 133 may register, modify, or delete a product banner image. The market management module 133 may form product categories and product groups and expose them on the main screen. The market management module 133 may generate a mini-chart related to product trading and display it on the main screen.

The market management module 133 may be configured to check sales history, check the sales amount or total sales amount for a certain period of time for each seller, or cancel the purchase of a product for which payment has been requested incorrectly by the seller or buyer. there is.

The market management module 133 may manage the list of products for sale and the sales details of products within the list. The product for sale may also be a content product. Product sales details may include product description, sales amount, etc.

The market management module 133 may also generate transaction details. The transaction details may include transaction time, transaction product, payment method, product price, etc. Additionally, the market management module 133 may further create sales details and purchase details.

The market management module 133 may store the generated details in the store 134.

The point management module 135 may check the point payment target and pay the target points to the payment target. The point is a means of payment that can be used to trade content products on the market site of the content service system 1, and may be electronic money of the content service system 1. The points are paid in the amount corresponding to the amount paid when the user 100 pays with real currency, or when the user 100 engages in an activity in the content service system 1, a pre-designated number of points for the activity is provided. It may be paid.

The point management module 135 may generate and view payment details recording the fact that the server 130 has paid points to all users.

The point management module 135 may also check the points held by the user 100 in the point wallet. Additionally, the point management module 135 may generate accumulation details for each user for each user 100 and modify the accumulation details. The accumulation details may be generated whenever an accumulation event occurs in which a pre-designated accumulation point amount is paid if the accumulation conditions are met. The point management module 135 may check the accumulation history of the user 100 and the amount of accumulated points within the accumulation history. Additionally, in some embodiments, if points have an expiration date, the expiration date of the corresponding accumulated point amount may be inquired.

The point management module 135 may store payment details and accumulation details in the store 134.

The accumulation event may be associated with various activities of the content service system 1. In various embodiments, the accumulation event may be a voting event. In the voting event, the accumulation condition may be whether or not to participate in voting.

Point accumulation through participation in the voting event will be described in more detail with reference to Figure 12 below.

The event management module 137 may generate voting details and provide a voting list consisting of the generated voting events.

When the electronic device 110 of the user 100 or the manager 100_3 inputs the voting period, voting title, and voting candidate, a voting event creation request including the input voting content is transmitted to the server 130. When the event management module 137 receives a request for creating a voting event, it may generate voting details based on the voting content in the request for creating a voting event. The voting details are a group of voting-related information and may include the content of the votes received and information obtained during the voting process. The voting details include, for example, voting period, voting title, voting candidate, number of votes for each candidate, list of voting users for each candidate, voting-related messages, preference for the message (e.g., number of likes), detailed information, tags, etc. It may also be included. The event management module 137 may generate voting screen data used to display a voting screen based on the voting details and transmit the voting screen data to the electronic device 110.

The voting list includes voting events for which voting has ended and/or voting events for which voting is in progress. For example, the server 130 may provide a voting list consisting only of voting events whose voting period has not expired, or a voting list including voting events whose voting period has expired.

The event management module 137 may provide some or all of the information of the voting details to the point management module 135 as voting results to provide voting users with points according to voting participation activities and voting results.

The event management module 137 may store voting details in the store 134.

The operation of the event management module 137 will be described in more detail with reference to FIG. 12 below.

Agreement for permission to create emoticons

The member management module 131 may grant permission to sell emoticons to a specific user (the first user 100_1 in FIG. 1) among the users 100 registered as members for the content service. Only the first user (100_1) who has been granted permission to sell emoticons can register his or her creative emoticons as trading products on the market site.

Figure 3 is a flowchart of a process for granting permission to create emoticons according to various embodiments of the present application.

The process of FIG. 3 may also be performed by the member management module 131.

Referring to FIG. 3, the process of granting permission to sell emoticons to the first user (100_1) includes identification information and sales range information of the first user (100_1) stored in the first electronic device (110_1). Receiving a permission request (S310); Searching for pre-registered member information of the first user (100_1) based on the identification information of the first user (100_1) (S320); Generating an electronic contract document for the first user (100_1) by applying the searched membership information of the first user (100_1) and the sales range information to a preset contract document template (S330); transmitting the electronic contract document to the first electronic device 110_1 to request an electronic signature of the first user 100_1 (S340); Verifying whether the electronic signature of the electronic contract document received from the first electronic device 110_1 satisfies a preset signature standard (S350); and granting permission to sell emoticons to the first user 100_1 whose electronic signature has been verified (S360).

The sales range information may be information describing the content product that the first user 100_1 wants to sell. In some embodiments, the sales range information may include product type. The product type may be one or more content types among emoticon objects, background stickers, and creative emoticons. Additionally, in some embodiments, the sales range information may further include product details to be sold. The product content may include the type of emoticon object to be sold, such as a person or an animal.

The member management module 131 stores member information of the user 100 who has signed up for the emoticon service in the data store 134. The member management module 131 searches for pre-registered member information of the first user (100_1) based on the identification information of the first user (100_1) in the sales authorization request (S320). The identification information of the first user 100_1 in the sales authorization request may include all or part of the registered member information of the first user 100_1. If the sales authorization request includes part of member information, the part of member information may be insufficient to create the electronic contract document to be created in step S330 below. Then, the member management module 131 may use some member information to obtain the remaining information for creating the electronic contract document to be created in step S330 from the pre-registered member information. For example, when a member ID is included in a sales authorization request, the member management module 131 may search for member information with the member ID to obtain the remaining member information.

The member management module 131 may also store contract document templates in advance. The contract document template has a template structure consisting of preset contract items. The member management module 131 applies the member information of the first user 100_1 retrieved in step S320 and the sales range information obtained in step S310 to the contract document template to obtain a member information value corresponding to the contract item. By inserting , an electronic contract document for the first user (100_1) is created.

In step S340, when the first electronic device 110_1 receives the electronic contract document generated in step S330, it electronically signs the electronic contract document with the private key of the first user 100_1 and transmits it to the server 130. do.

The signature standard set in advance in step S350 includes verification using a public key. The server 130 verifies the electronic signature of the received electronic contract document using the pre-stored public key of the first user 100_1. If verification is successful, the validity of approval of the contract contents of the first user (100_1) is verified (S350). The server 130 grants permission to sell emoticons to the first user 100_1 whose verification was successful (S360).

Additionally, in some embodiments, the preset signature standard may further include verification of the user's identity other than an electronic signature. In this case, the first electronic device 110_1 may insert an identity verification image of the face of the first user 100_1 and an ID card showing the user's own face into an electronic signature document and transmit it to the server 130.

Then, the step (S350) includes detecting the user's face area and ID card area in the received electronic contract document; Checking whether the face part in the identification area and the face part in the user's face area match; checking whether the values of at least some of the plurality of personal information items displayed in the ID area match the values of the same items among the pre-stored member information of the first user; It may also include verifying the validity of the electronic signature using a pre-stored public key of the first user after the match of the face part and the personal information item are confirmed.

To this end, the member management module 131 may include a region detection model configured to detect a preset region of interest in the input image. As an example, the region detection model may be a machine learning model that extracts features from an input image and detects a preset region of interest. The region detection model is, for example, based on Haar, Convolution Neural Network (CNN), Scale Invariant Feature Transform (SIFT), Histogram of Gradients (HOG), Neural Network (NN), Support Vector Machine (SVM), and Gabor method. It may be a model, but is not limited to this.

The area of interest includes a face area where the face of the first user 100_1 is photographed, and a face area within the ID card. The face area is an area containing a portion of the first user's face photographed along with the identification card. Additionally, the area of interest may further include some or all of the personal information items displayed on the ID card. The personal information items may include, for example, one or more of name, address, resident registration number, and driver's license number.

In the step S350, the member management module 131 inputs the identity verification image inserted in the received electronic contract document into the area detection model to detect the face area of the first user 100_1 and the face area in the ID card, It is checked whether the face part in the ID card area and the face part in the face area of the first user match. The member management module 131 calculates the similarity between the first feature set extracted to detect the face region in the region detection model and the second feature set extracted to detect the face region in the ID card, and matches the calculated similarity to the first feature set extracted to detect the face region in the region detection model. Based on this, the match of facial parts can also be judged. The similarity may be calculated through, for example, Euclidean, cosine-based similarity calculation methods, or other vector similarity calculation methods.

Additionally, in step S350, the member management module 131 inputs the identity verification image inserted in the received electronic contract document into the area detection model and detects the area detection model among a plurality of personal information items appearing on the identification card. detects an area where each of at least some items preset to be displayed is displayed, recognizes text or numbers in the detected area, extracts the corresponding item value, and extracts the extracted item value from the pre-stored member information of the first user (100_1). Check whether it matches the value of the item.

The member management module 131 sets signature criteria including matching the photographed face part with the face part in the ID card, matching the ID card contents with member information, and verifying the validity of the electronic signature to the first user ( If 100_1) is satisfied, permission to sell emoticons is granted to the first user (100_1) (S360). When permission to sell emoticons is granted, the status information of the first user 100_1 is changed from a general member to a creative member and the member information of the first user 100_1 is updated (S360).

The member management module 131 may record the first user(s) who have granted permission to sell emoticons and create a list of people authorized to sell. The selling authority list includes at least part of the member information of the first user 100_1 to whom selling authority has been granted, and the public key of the first user 100_1.

Additionally, when the member management module 131 grants the first user 100_1 the authority to sell emoticons, it may designate the creation storage space of the first user 100_1 in the storage space of the server 130. Then, the creative emoticon of the first user 100_1 created in the first electronic device 110_1, which will be described below, is stored in the creation storage space of the first user 100_1.

This creation storage space will be described in more detail with reference to FIG. 7 below.

Create creative emoticons

If the first user 100_1 is granted permission to sell emoticons in the content service system 1, a creative emoticon that can be transmitted through a messenger application can be created.

The first electronic device 110_1 may create a creative emoticon using an emoticon object in a captured image.

FIG. 4 is a flowchart of a creative emoticon creation process according to various embodiments of the present application, and FIG. 5 is a schematic diagram of the creative emoticon creation process of FIG. 4 .

Referring to FIGS. 4 and 5 , in the first electronic device 110_1 of the first user 100_1 registered as a member of the emoticon service, a captured image is obtained by photographing an emoticon object, and at least a partial area of the photographed image is captured. Obtaining emoticon object data related to the emoticon object image (S410); The first electronic device 110_1 requests the server 130, which operates the market site, to purchase the background sticker at a market site where background stickers and content products including the creative emoticons are traded, thereby sending the corresponding background sticker data. Receiving (S430); and generating, in the first electronic device 110_1, creative emoticon data of the first user 100_1 based on the background sticker data and emoticon object data (S450).

In various embodiments of the present application, the emoticon object includes the face and/or body parts of the first user 100_1.

In step S410, the first electronic device 110_1 may acquire a face image by photographing the face with a camera, or may obtain an upper body image by photographing the upper body including the face. Hereinafter, for clarity of explanation, the present invention will be described in more detail with examples in which facial images are obtained as captured images.

In step S410, the first electronic device 110_1 extracts an emoticon object image in which an emoticon object appears in at least a partial area from the captured image and obtains emoticon object data related to the extracted emoticon object image. For example, the entire frame area of the captured image may be used as an emoticon object image.

To this end, the first electronic device 110_1 is configured to provide an interface screen that allows selecting an emoticon object image using a captured image. The first user 100_1 may directly select a captured image as an emoticon object image, or select a region of interest, such as a face part, from the captured image and select the region of interest image as an emoticon object image.

The emoticon object data may include two-dimensional image data representing the emoticon object and/or a set of parameters for rendering the emoticon object in two dimensions.

The set of parameters of the emoticon object includes the frame size of the emoticon object, the size, position, and color of the emoticon object on the frame, identification information of the emoticon object, and other parameters for implementing the emoticon object.

The identification information of the emoticon object may be a creator's device identifier, a network identifier, or a service system identifier such as a member ID.

When the captured image is a sequence image consisting of a plurality of images, the emoticon object data includes a set of data (e.g., individual emoticon object data) describing the emoticon object appearing in each emoticon object image constituting the sequence image, It further includes sequence data such as the output order and output speed of each image that makes up the entire time series section of the image.

As shown in FIG. 5, in step S430, the first electronic device 110_1 may display a background sticker sold through a market site.

The background sticker is classified into one of a plurality of preset emotion types. In various embodiments, the emotion type may include two or more of joy, anxiety, anger, embarrassment, pain (or hurt), sadness, and neutral. The market site may provide a screen displaying all background stickers, background stickers already purchased by the first user (100_1), or background sticker groups classified by each emotion type.

When a user input for selecting one of a plurality of background stickers is input, the first electronic device 110_1 transmits a purchase request for the selected background sticker to the server 130 to download background sticker data. It may be possible. The purchased background sticker may be stored in the purchased product storage box of the first user 100_1 in the server 130 or in the first electronic device 110_1 (S430).

In various embodiments of the present application, the background sticker may be a background sticker related to the emotion of the first user 100_1 shown in the captured image. The first electronic device 110_1 or the server 130 analyzes the emoticon object part of the captured image, that is, the face part of the first user, extracts the emotion predicted to be possessed by the first user 100_1, and extracts the extracted emotion. A background sticker within the group of the emotion type corresponding to the emotion may be recommended (S430).

The step (S430) includes detecting a facial area in which the face of the user of the electronic device is expressed from the captured image data in order to recommend a background sticker; Recognizing the user's emotion by inputting a patch image of the detected face area into a pre-trained image-based emotion recognition model; searching the market site for one or more background stickers associated with the user's recognized emotion; generating a recommendation list for the user based on the sales range information of the searched background sticker and transmitting it to the electronic device to be displayed on the electronic device; And it may also include selecting a specific background sticker from the recommendation list and requesting a purchase of the selected background sticker.

The first electronic device 110_1 may include an area detection model. The first electronic device 110_1 may detect a face area by inputting a captured image into the area detection model. Since the area detection model has already been described in relation to the member management module 131 above, detailed description will be omitted.

The first electronic device 110_1 extracts a face area from a captured image and generates a region of interest image consisting of the face area.

The image-based emotion recognition model may be a neural network-based machine learning model configured to recognize the emotion of the input image by extracting features of the input image to infer the correlation between the facial part of the input image and the emotion. The image-based emotion recognition model has, for example, a model structure based on CNN (Convolutional Neural Networks) and other NNs.

The image-based emotion recognition model is pre-trained to recognize emotion types using a training data set consisting of a plurality of training samples.

The training data set may be subset into each of a plurality of emotion types. For example, the training data set is configured so that the image-based emotion recognition model recognizes any one of joy, anxiety, anger, embarrassment, pain (or hurt), sadness, and neutral as the emotion type. It consists of multiple training samples, subsetted into seven emotion types.

Each training sample has training data and label data. The training data may be a sample emoticon object image, such as a sample face image. The label data indicates the type of emotion possessed by the sample emoticon object shown in the sample emoticon object image. The label data is expressed as a value indicating one of a plurality of emotion types that define a subset of the training data set.

The image-based emotion recognition model is designed to minimize the error between the output value (i.e., predicted emotion type) calculated by inputting training data of the training data set and the label data (i.e., actual emotion type) corresponding to the input training data. It is learned. As learning progresses and more training data is input, the values of model parameters (eg, weight, bias, etc.) in the image-based emotion recognition model are updated so that the error gradually decreases. Model parameters for which learning has been completed may have values that minimize errors.

In various embodiments, learning of the image-based emotion recognition model may be performed in the server 130 or the electronic device 110. When performed on the server 130, the electronic device 110 may receive a trained image-based emotion recognition model and use it to create a creative emoticon.

When the region of interest image is input to the image-based emotion recognition model learned in this way and the emotion type value predicted to be possessed by the first user 100_1 is calculated, the first electronic device 110_1 determines the emotion type predicted through the market site. You can also display a group of background stickers associated with . For example, if the emotion type predicted from the captured image is “joy,” the market site may provide a recommendation list in which background sticker(s) previously associated with “joy” are arranged.

The first electronic device 110_1 may select a specific background sticker from the recommendation list and request the server 130 to proceed with purchasing the selected background sticker. When the purchase process is completed, the first electronic device 110_1 may receive background sticker data. The background sticker is data that implements a background sticker image. In various embodiments, the background sticker data may include two-dimensional image data displaying the background sticker and/or a set of parameters for rendering the background sticker in two dimensions.

The set of parameters of the background sticker includes the frame size of the background sticker, the size of the background components (e.g., background text, background objects) that make up the background sticker, the position on the frame, the color, the type of emotion associated with the background sticker, and the background sticker. Includes other parameters to implement.

When the background sticker is a sequence image, the parameters of the background sticker may further include sequence data such as the output order as a sequence parameter.

In step S450, the first electronic device 110_1 selects the emoticon object image selected as at least a portion of the captured image of the first user 100_1 obtained in step S410 and the first electronic device 110_1 purchased in step S430. A creative emoticon is created by combining the background stickers of the user (100_1).

Figure 6 shows the structure of a creative emoticon according to various embodiments of the present application.

Referring to FIG. 6, the creative emoticon consists of a first layer 51 and a second layer 52. The second layer 52 is disposed on the first layer 51.

The step (S450) includes implementing an emoticon object image in the first layer using the emoticon object data; and implementing a background sticker image in the second layer using the background sticker data.

The first electronic device 110_1 may implement an emoticon object image in the first layer 51 using the emoticon object data. When the emoticon object data includes two-dimensional image data, the first electronic device 110_1 displays the two-dimensional image on the first layer 51. When the emoticon object data is a parameter set of the emoticon object, the first electronic device 110_1 renders a two-dimensional image of the emoticon object in the first layer 51 based on the parameter set.

The first electronic device 110_1 may implement a background sticker image in the second layer 52 using the background sticker data. When the background sticker data includes two-dimensional image data, the first electronic device 110_1 displays the two-dimensional image of the background sticker on the second layer 52. When the background sticker data is a parameter set of the background sticker, the first electronic device 110_1 renders a two-dimensional image of the background sticker in the second layer 52 based on the parameter set.

Additionally, when the frame size of the background sticker and the frame size of the emoticon object image do not match, the first electronic device 110_1 adjusts the frame size of the background sticker to match the frame size of the emoticon object image. In this case, in step S450, if the frame size of the background sticker and the frame size of the emoticon object image do not match each other, the frame size of the purchased background sticker and the frame size of the emoticon object image including the emoticon object portion are adjusted. Adjusting the frame size of the purchased background sticker or emoticon object image to the remaining frame size to match each other; Implementing an emoticon object image in the first layer using the emoticon object data - when the frame size of the emoticon object image is adjusted, the emoticon object data includes the adjusted frame size; and implementing a background sticker image in the second layer using the background sticker data - when the frame size of the background sticker is adjusted, the background sticker data includes the adjusted frame size.

For example, the background sticker data may be adjusted so that the background sticker has a frame size that matches the frame size of at least a portion of the captured image. Then, the first electronic device 110_1 implements an emoticon object image in the first layer using the emoticon object data, and implements an emoticon object image in the second layer using background sticker data with an adjusted frame size. Implements a background sticker image.

Here, the frame size is adjusted to create a creative emoticon that more harmoniously combines the emoticon object part expressed in the captured image (e.g., the captured face part) and the purchased background sticker, and the background visual elements included in the background sticker (e.g., This includes adjusting the size or position of an emoticon object included in a background object or background text) or emoticon object image. Here, a more harmonious combination is at least a relative decrease in the ratio difference between the size of the emoticon object part and the background object in the background sticker before and after the adjustment operation, and a relative decrease in the overlap between the emoticon object part and the background sticker. The appearance of one effect means that the sense of heterogeneity due to proportional incongruity in the combination relationship between the emoticon object part and the background sticker within the combined creative emoticon is relatively reduced.

In one embodiment, the step of adjusting the frame size of the purchased background sticker or emoticon object image to the remaining frame size includes, when the frame size of the emoticon object image is adjusted, the frame size of the emoticon object image before adjustment and the emoticon object image after adjustment. adjusting the size of the emoticon object portion based on a change in the ratio of the frame size of the image; When the frame size of the background sticker is adjusted, the method may include adjusting the size of the background object based on a change in the ratio between the frame size of the background sticker before adjustment and the frame size of the background sticker after adjustment.

Additionally, the first electronic device 110_1 adjusts the size of the background object or the size of the emoticon object portion to generate creative emoticon data of the electronic device based on the background sticker data and emoticon object data. In the emoticon object part, the area of the exposed part that is not covered by the background object is calculated, and the ratio of the exposed part is calculated based on the total area of the emoticon object part and the area of the exposed part, and the exposed part If the ratio is less than a preset threshold exposure ratio, it may be further configured to further reduce the size of the emoticon object part or move the position of the emoticon object part so that an area greater than the threshold exposure ratio is exposed. This can prevent the emoticon object part from being displayed because it is excessively covered by the background object.

In some embodiments, the first electronic device 110_1 may store a minimum size that limits further reduction of the emoticon object portion. If the emoticon object is reduced too much, the user 100 has limitations in recognizing the emoticon object. In this case, the size of the emoticon object part is further reduced so that the area greater than the critical exposure ratio is exposed, and if the ratio of the exposed part is still less than the critical exposure ratio even though the emoticon object part is reduced to the minimum size, the emoticon object is removed from the creative emoticon image. It may be configured to search the remaining space adjacent to the object part and move the emoticon object part of the minimum size to the adjacent remaining space.

When the first electronic device 110_1 creates a creative emoticon, it transmits the creative emoticon data to the server 130 and stores the creative emoticon data in the creation storage space of the first user 100_1 already formed in the storage space in the server 130. Save.

The creative emoticon data may include two-dimensional image data that displays the creative emoticon in two dimensions and/or a parameter set of the creative emoticon.

The parameter set of the creative emoticon includes a parameter set of the emoticon object and a parameter set of the background sticker. If the size, position, etc. of the emoticon object and background sticker are adjusted during the creation of the creative emoticon, the parameter set of the creative emoticon includes a parameter set with changed values for the adjusted emoticon object and background sticker.

In various embodiments, when the first user 100_1 is granted permission to sell emoticons, a creative emoticon creation mode is activated in the first electronic device 110_1. In a state where permission to sell emoticons is not granted, the creative emoticon creation mode is deactivated, and the electronic device 110 may not execute the creative emoticon creation mode even if it receives an execution input for the creative emoticon creation mode.

In some embodiments, the first electronic device 110_1 may generate a creative emoticon with a watermark 53 inserted on the second layer 52. The watermark 53 is associated with the first user and may be composed of at least one of text and a logo describing the first user. The first electronic device 110_1 may receive watermark information indicating the identity of the first user 100_1, or may store it in advance.

Additionally, in some embodiments, the content service system 1 may require a deduction of a certain amount of points to insert the watermark 53 into the creative emoticon. To this end, in step S450, the first electronic device 110_1 requests the use of a watermark including the identifier of the first user in order to insert the watermark 53 on the second layer 52. transmitting to the server; When the server receives a request to use the watermark, deducting a preset amount of points for the mark from the point wallet of the first user and transmitting permission to use the mark to the first electronic device; and, when permission to use the watermark is received by the first electronic device, placing the watermark on the second layer to create a creative emoticon with the watermark inserted.

Additionally, in some embodiments, the content service system 1 analyzes the emoticon object part of the captured image, that is, the face part of the first user, and sets the emotion type predicted to be possessed by the first user 100_1 to the upper emotion parameter. and may be further configured to set lower emotion parameters based on the shape of the emoticon object portion of the captured image.

The shape of the emoticon object portion of the captured image includes the facial expression of the first user 100_1 and/or the shape of the still or continuous motion of the first user 100_1. The first electronic device 110_1 or the server 130 is configured to recognize the motion of the first user 100_1 according to stationary or continuous motion.

In one embodiment, the first electronic device 110_1 may include a motion recognition model and a secondary emotion recognition model.

The motion recognition model may be a neural network-based machine learning model configured to recognize the motion of an emoticon object appearing in the input image by extracting features of the input image to infer the correlation between the object movement of the input image and the motion indicated by the motion. there is. The image-based emotion recognition model has, for example, a model structure based on CNN (Convolutional Neural Networks) and other NNs.

In various embodiments, the motion recognition model may obtain data about joint points by extracting features from each frame of the input sequence image. Data about the joint point may be expressed as a scale value, a translation value (e.g., a joint coordinate value), or a rotation value (e.g., a movement value). In some embodiments, the size, position, and rotation values may be expressed as three-dimensional vectors. In various embodiments, the recognition of the action may be recognition of a type of action.

The motion recognition model is trained in advance to recognize motion types using a training data set consisting of a plurality of training samples. Each training sample has a sample sequence image showing the continuous motion of an emoticon object as training data and the motion type indicated by the corresponding motion as label data. The training data set may be subset into each of a plurality of operation types. In some embodiments, the label data of the motion type may be composed of data about a joint point that implements a reference motion indicating the corresponding motion type.

The motion recognition model is trained to minimize the error between the output value (i.e., predicted motion type) calculated by inputting the training data of the training data set and the label data (i.e., actual motion type) corresponding to the input training data. . As learning progresses and more training data is input, the values of model parameters (eg, weight, bias, etc.) in the motion recognition model are updated so that the error gradually decreases. Model parameters for which learning has been completed may have values that minimize errors.

In various embodiments, learning of the motion recognition model may be performed in the server 130 or the electronic device 110. When performed in the server 130, the electronic device 110 may receive a learned motion emotion recognition model and use it to create a creative emoticon.

By inputting an emoticon object sequence image into the learned motion recognition model, a motion type value predicted to be taken by the first user 100_1 may be obtained.

The motion type value of the motion recognition model is provided as a secondary emotion recognition model.

The secondary emotion recognition model is a secondary emotion prediction model predicted to have under the motion and facial expression taken by the user 100 when the emotion type value calculated from the image-based emotion recognition model and the motion type value calculated from the motion recognition model are input. It is configured to calculate the emotion type.

Unlike the previous seven emotion types classified based on the facial expression of the user 100, the secondary emotion type is an emotion type that can be classified based on the facial expression and motion of the user 100, and is referred to as an auxiliary emotion type. It could be.

The secondary emotion type may be set to have a different value from the previous seven emotion types. For example, the secondary emotion type may be an emotion type of another aspect such as positivity, negativity, exercise, or love.

The secondary emotion recognition model includes the emotion type value of one of the seven emotion types classified based on the facial expression of the user (100) calculated from the image-based emotion recognition model and the user (100) calculated from the motion recognition model. ) is a machine learning model configured to predict the corresponding secondary emotion type by processing the input combination to infer the correlation between the combination of the action type value and the preset secondary emotion type.

The secondary emotion recognition model has a neural network structure. The neural network structure may be, for example, a fully connected layer structure, a multi-layer perceptron (MLP) model structure, or other NN structure.

The secondary emotion recognition model is trained in advance to recognize secondary emotion types using a training data set consisting of a plurality of training samples.

The training data set may be subset into each secondary emotion type. For example, the training data set is subset into the four emotion types so that the secondary emotion recognition model recognizes any one of positive, negative, exercise, and love as the secondary emotion type. It consists of a plurality of training samples.

Each training sample has training data and label data. The training data may be a sample emotion type and sample motion type value calculated from a sample captured image. The label data indicates the secondary emotion type possessed by the sample emoticon object shown in the sample emoticon object image. The label data is expressed as a value indicating one of a plurality of secondary emotion types that define a subset of the training data set.

Since the learning process of the secondary emotion recognition model is similar to the learning process of the image-based emotion recognition model described above, detailed description is omitted.

Using the secondary emotion recognition model, the first electronic device 110_1 may obtain a secondary emotion type value for the emoticon object. Then, the first electronic device 110_1 may generate parameters of the emoticon object including the secondary emotion type value. In this way, system 1 can extract various emotions other than the seven emotions as attributes related to the emoticon object.

Save Creative Emoji

The creative emoticon data created in this way is transmitted to the server 130 and stored in the creation storage space of the first user 100_1. The creation storage space has already been created when the first user (100_1) is granted permission to sell emoticons.

Figure 7 shows a screen of an electronic device for accessing the location of the creation storage space, according to various embodiments of the present application.

Referring to FIG. 7 , the first electronic device 110_1 may check the creative emoticon stored in the created storage space of the first user 100_1 by selecting the icon 60 indicating the created storage space.

In some embodiments, the first electronic device 110_1 may store creative emoticon data in a storage space within the first electronic device 110_1.

The first electronic device 110_1 forms a first storage space in the storage space within the first electronic device 110_1 linked to the created storage space of the first user 100_1 within the server 130 to store the first information. The creative emoticon data may be stored in the space, and/or the creative emoticon data may be stored in one or more second storage spaces.

After the first storage space is formed, if a user input to store a creative emoticon in the storage box is input after a creative emoticon is created in the first electronic device (110_1), the corresponding creative emoticon data is stored in the storage box of the first user (100_1), and then the creative emoticon data is stored in the storage box of the first user (100_1). In response to being stored in the storage box of the first user 100_1, the corresponding creative emoticon data is stored in the first storage space of the first electronic device 110_1.

FIG. 8 shows a screen for selecting a messenger application for transmitting a creative emoticon stored in a second storage space according to various embodiments of the present application, and FIG. 9 shows a screen for selecting a creative emoticon stored through the messenger application selected in FIG. 8. Shows the screen for transmitting.

Additionally, the first electronic device 110_1 may store the creative emoticon data in one or more second storage spaces formed in the storage space of the first electronic device 110_1. Each second storage space is a messenger storage space that stores emoticons of the corresponding messenger application 70, and the one or more second storage spaces are the corresponding application created when the corresponding messenger application 70 is installed on the electronic device. This is a dedicated storage space for (70).

The emoticon of the messenger application 70 is stored in the second storage space. The plurality of second storage spaces are formed in a different location from the first storage space in the first electronic device 110_1. The first electronic device 110_1 may store the creative emoticon in each of a plurality of second storage spaces in response to storing the creative emoticon in the first storage space. In some embodiments, storing the creative emoticon in the second storage space may be the location of the first storage space where the creative emoticon is stored. As a result, it is possible to minimize storage capacity while expanding the range of applications that can use creative emoticons.

As shown in FIG. 8, the plurality of messenger applications 70 corresponding to the plurality of second storage spaces are self-messenger applications operated by the operator of the server 130 and/or messenger applications not operated by the operator of the server 130. It may also include third-party messenger applications.

When the first electronic device 110_1 stores the creative emoticon data in the second storage space, the message sender's emotions can be conveyed through a messenger emoticon as shown in FIG. 9 using the stored creative emoticon.

When the location of the first storage space where the creative emoticon is stored is stored in the second storage space, the first electronic device 110_1 obtains the creative emoticon stored in the first storage space through the location of the first storage space and displays the creative emoticon in FIG. 9 Provides a preview screen or sends the selected creative emoticon.

In some embodiments, when transmitting a creative emoticon to another electronic device 110, the first electronic device 110_1 may transmit a parameter set of the target creative emoticon instead of transmitting an image of the creative emoticon.

Another electronic device 110 that has received the parameter set can create a creative emoticon using the parameter set. Then, the time required to send/receive emoticons can be minimized compared to when transmitting creative emoticons in image format.

In addition, in some embodiments, creative emoticon image data is stored in the first storage space or the second storage space, or a creative emoticon image and a parameter set of the creative emoticon are stored, and the first user 100_1 in the server 130 ), the parameter set of the creative emoticon may be transmitted to the server 130 to store the parameter set of the creative emoticon in the storage space of ). The data capacity of the parameter set of the creative emoticon is relatively smaller than the data capacity of the creative emoticon image. As a result, the speed and success probability of transmitting the creation result of the creative emoticon from the first electronic device 110_1 to the server 130 is increased, further minimizing the storage space of the server 130, and at the same time, the original of the creative emoticon The image can be stored in the first electronic device 110_1.

Additionally, the first electronic device 110_1 may select a creative emoticon to be deleted from the first user's library and transmit a request for deletion of the selected creative emoticon to the server 130. The server 130 deletes the creative emoticon data from the first user's library and transmits a deletion notification to the first electronic device 110_1. When receiving a deletion notification, the first electronic device 110_1 may delete the creative emoticon data from one or more second storage spaces in response to deletion from the first user's storage box.

In this way, the first electronic device 110_1 can check whether messages are linked through the application for creating creative emoticons, thereby eliminating the user's inconvenience in managing emoticons directly in each of the multiple messenger applications 70.

Product registration and trading of creative emoticons

The first user 100_1 may register the creative emoticon he created as a product for sale on the market site through the first electronic device 110_1. The created emoticon of the first user (100_1) may be additionally registered in the list of products for sale on the market site.

To this end, the first electronic device 110_1 transmits a product registration request including creative emoticon data of the first user 100_1 to the server 130.

FIG. 10 shows a screen for requesting product registration according to various embodiments of the present application, and FIG. 11 shows a screen showing a registration result according to the product registration request of FIG. 10 .

Referring to FIG. 10, the first electronic device 110_1 selects a creative emoticon that is to be registered as a new product among the creative emoticons stored in the creation storage space of the first user 100_1, thereby generating the first user 100_1. A product registration request including creative emoticon data is generated and the product registration request is transmitted to the server 130.

Upon receiving the product registration request, the market management module 133 may add the first user's creative emoticon to the list of products for sale on the market site using the received creative emoticon data of the first user.

In various embodiments of the present application, when the creative emoticon requested for product registration satisfies preset registration criteria, the market management module 133 uses the creative emoticon data of the first user 100_1 to sell the market site. The first user's (100_1) creative emoticon can also be added to the product list.

The registration criteria determines whether the emoticon object of the creative emoticon belongs to the sales range information, and/or whether the emoticon object of the creative emoticon corresponds to a filter product in a pre-stored filter list based on the obtained emoticon object data. Includes.

The market management module 133 obtains emoticon object-related information, such as the type of emoticon object (eg, person, animal, etc.), from the creative emoticon data in the product registration request.

The market management module 133 determines that if the type information of the emoticon object acquired in advance and included in the sales range information matches the type of the emoticon object acquired through a product registration request, the emoticon object of the creative emoticon is included in the sales range information. decided to belong. If the type information of the emoticon object acquired in advance and included in the sales range information does not match the type of the emoticon object acquired through a product registration request, the market management module 133 determines that the emoticon object of the creative emoticon is changed to the sales range information. Decide that it does not belong to

Additionally, the market management module 133 checks whether the emoticon object of the creative emoticon is not searched in the pre-stored filter list. When an emoticon object is searched in a pre-stored filter list, it is determined that the emoticon object of the creative emoticon corresponds to a filter product in the pre-stored filter list. If the emoticon object is not searched in the pre-stored filter list, it is determined that the emoticon object of the creative emoticon does not correspond to the filter product in the pre-stored filter list. The filter list is a list of permission-denied emoticon objects, and may also be a list of disapproved emoticon objects. In some embodiments, the filter emoticon object included in the filter list is an emoticon object with copyright issues and includes, for example, pre-designated character(s) and picture(s).

When it is determined that the emoticon object of the creative emoticon belongs to the sales range information and/or that the emoticon object of the creative emoticon does not correspond to a filter product in the pre-stored filter list, the market management module 133 1 The user's creative emoticon can also be added to the list of products for sale.

Additionally, in some embodiments, the first electronic device 110_1 may generate a product registration request by processing creative emoticon data with an electronic signature. The market management module 133 may verify the validity of the electronic signature of the creative emoticon data in the product registration request using the public key of the first user in the sales authority list. When the validity of the electronic signature is verified, the market management module 133 may add the first user's creative emoticon to the list of products for sale.

Then, as shown in FIG. 11, the creative emoticon created in step S450 is exposed as a new product on the market site. The creative emoticon is assigned a product category based on product attributes, and may be arranged together with other products belonging to the same product category. For example, if the product emotion is joy, it is arranged together with other products that have the attribute of joy, such as “If you smile, you will be blessed,” as shown in FIG. 11.

Another user 100, such as the second user 100_2, may purchase a creative emoticon by paying a purchase price for the creative emoticon of the first user 100_1 through a market site.

When the second user (100_2) completes purchasing the creative emoticon of the first user (100_1), the creative emoticon data of the first user (100_1) is stored in the purchase storage space of the second user (100_2) created in the server 130. do.

In some embodiments, the second electronic device 110_2 of the second user 100_2 may store purchased creative emoticon data in a storage space within the second electronic device 110_2. Additionally, in some embodiments, the second electronic device 110_2 may use a third storage space linked to the purchase storage space of the second user 100_2 in the server 130 in the storage space of the second electronic device 110_2. The creative emoticon data may be stored in the third storage space, and/or the creative emoticon data may be stored in one or more fourth storage spaces. The third storage space corresponds to the first storage space described with reference to FIG. 8, and the fourth storage space corresponds to the second storage space. The fourth storage space is a messenger storage space that stores emoticons of the corresponding messenger application 70, and the one or more fourth storage spaces store the corresponding application 70 formed when the corresponding messenger application 70 is installed on the electronic device. This is a dedicated storage space for The plurality of fourth storage spaces are formed in a different location from the third storage space in the second electronic device 110_2.

When a creative emoticon of the first user 100_1 is purchased, the second electronic device 110_2 may transmit the purchased creative emoticon through a messenger application.

Additionally, the market management module 133 may generate sales history data of the first user 100_1 and purchase history data of the second user 100_2, and perform a settlement operation accordingly. Due to the purchase of the second user 100_2, the first user 100_1 acquires real currency (eg, cash) or points.

voting event

The user 100 may acquire points by performing various activities in the content service system 1. The activity may be participating in a voting event.

The server 130 creates a voting event through the event management module 137, manages voting details of the corresponding voting event, and pays points according to voting participation and results through the point management module 135.

Figure 12 is a flowchart of a voting event progress process according to various embodiments of the present application.

Referring to FIG. 12, the voting event progress process includes generating a voting event by the event management module 137 (S1210).

The voting event is implemented to update the number of participants who voted for the voting target during the voting period. The voting event has a group of voting event-related information.

In the step (S1210), the event management module 137 forms a voting group consisting of at least some of the creative emoticons included in the product sales list and counts the popularity of each creative emoticon included in the voting group. You can also create voting events. To this end, the server 130 may generate the voting event by inputting the voting period, information related to a plurality of creative emoticons, and other voting contents through the electronic device 110 of the manager 100_3.

The voting group includes different emoticon objects. Here, the emoticon object of a specific person and the emoticon object of the group included in the specific person are treated as different emoticon objects.

In addition, the voting event progress process of FIG. 12 includes determining the number of votes for each creative emoticon in the voting group based on the number of second users (100_2) who selected the creative emoticon in the voting group during the voting period (S1220) ); It includes determining a voting ranking according to the number of votes and generating voting details of the voting event recording the number of votes, voting ranking, and users 100 who participated in the voting event (S1230).

13 illustrates a voting screen according to various embodiments of the present application.

Referring to FIG. 13, the event management module 137 transmits voting screen data to the second electronic device 110_2 to encourage voting participation after the voting event is created and the second electronic device 110_2 displays the voting screen of the voting event. If a voting event is created in step S120, the voting screen data may be provided.

As shown in FIG. 13, the voting screen is configured to arrange the creative emoticons for voting included in the voting group and select which creative emoticon you want to vote for among the arranged creative emoticons for voting. The arrangement list of the voting group may include a creative emoticon image and emoticon object information of the creative emoticon (eg, name ### in FIG. 13).

In step S1220, the second electronic device 110_2 generates a request to participate in voting according to the selection and transmits it to the server 130. Then, the event management module 137 updates the number of votes for the selected creative emoticon based on the voting participation request by the number of second users (100_2) who selected and voted for the creative emoticon and votes for the second user (100_2) who voted. Update the participant list. When the voting period expires, the event management module 137 determines the voting ranking by counting the number of votes for each creative emoticon.

In various embodiments of the present application, the event management module 137, in addition to selecting a creative emoticon in a voting group, collects votes for the selected creative emoticon based further on the purchase history of the selected creative emoticon and/or input of a voting message. It may also be configured to update the number. The event management module 137 recognizes the second user (100_2), who satisfies the counting conditions including selection of the creative emoticon, purchase details of the creative emoticon, and input of a voting-related message, as a voting participant.

Specifically, the event management module 137 determines whether a specific creative emoticon subject to voting in the voting participation request has been purchased from the purchase history of the second user who has received the voting participation request, and the second user The number of votes for the specific creative emoticon may be updated according to the voting participation request of the second user, who has confirmed that the user has purchased the specific creative emoticon subject to the vote.

Figure 14 shows a voting message screen according to various embodiments of the present application.

Referring to FIG. 14, the event management module 137 may input a voting-related message of the second user 100_2 and provide a voting message screen in which the input voting-related messages of the participants are arranged.

The voting-related message may have creative emoticons and/or text as message content. The participant who entered the voting-related message may be at least some of all second users (100_2) who participated in the voting event.

The event management module 137 may generate voting message screen data and transmit it to the second electronic device 110_2 so that the voting message screen is displayed on the second electronic device 110_2.

The second electronic device 110_2 may display the voting message screen and transmit the voting-related message of the second user 100_2 entered into the input window to the server 130. The event management module 137 may transmit the received voting-related message to the electronic device 110 that displays the voting message screen so that it is additionally displayed on the voting message screen.

The event management module 137 may update the number of votes for the creative emoticon based on the voting-related message displayed on the voting message screen. Specifically, when the event management module 137 receives the voting participation request from the second electronic device, it relates to the voting participation of the second user who sent the voting participation request among the voting-related messages arranged on the voting message screen. Search for messages. If the voting-related message of the second user (100_2) is entered, it is searched on the voting message screen.

Then, the number of votes for the selected creative emoticon is updated according to the voting participation request of the second user (100_2) whose voting-related message was retrieved. In other words, voting participants must use the creative emoticon for the vote to be reflected in the number of votes.

In some embodiments, the event management module 137 allows the second user 100_2 to use a creative emoticon based on an emoticon object that has the same identity but a different pose from the emoticon object displayed in the creative emoticon included in the voting-related message. In this case, it may be further configured to reflect the number of votes. Specifically, the event management module 137 checks whether the searched message contains a creative emoticon that is the same as the creative emoticon of the voting target or contains another creative emoticon based on the same emoticon object as the emoticon object in the creative emoticon of the voting target. And, the voting participation of a second user identified among the retrieved messages as containing a creative emoticon identical to the creative emoticon of the voting target or containing another creative emoticon based on an emoticon object identical to the emoticon object in the creative emoticon of the voting target. Upon request, the number of votes for the specific creative emoticon may be updated.

The event management module 137 checks whether the searched message contains a creative emoticon identical to the creative emoticon of the voting target or contains another creative emoticon based on the same emoticon object as the emoticon object in the creative emoticon of the voting target. To this end, the voting group detects emoticon object data in the creative emoticon data of the voting target included in the voting participation request, detects an emoticon object identifier in the emoticon object data in the creative emoticon data of the voting target, and creates the searched message. Detect emoticon object data in the emoticon data, detect an emoticon object identifier from the emoticon object data in the creative emoticon data of the searched message, and if the detected emoticon object identifiers match each other, they include the same creative emoticon as the creative emoticon for voting. Alternatively, it may be configured to confirm that the voting target's created emoticon includes another created emoticon based on the same emoticon object as the emoticon object.

As an example, the same first user (100_1) creates the 1-1 creative emoticon using the first captured image taken in the first pose and creates the second created emoticon using the 1-2 captured image taken in the second pose. After creating and registering the 1-1 creative emoticon and the 1-2 creative emoticon in the product, the 1-1 creative emoticon is arranged in the voting group, and the second user (100_2) can purchase the 1-2 creative emoticon. . When the second user (100_2) selects the 1-1 creative emoticon from the voting group and enters a message including the already purchased 1-2 creative emoticon in the voting message screen, the 1st-1 creative emoticon of the second user (100_2) -1 The voting act of selecting a creative emoticon is recognized and reflected in the number of votes. This is because when different creative emoticons have the same emoticon object, the identity of the emoticon object is maintained, so it can be viewed as a message related to the object for which the second user (100_2) voted.

In various embodiments of the present application, the electronic device 110 may be configured to simultaneously display the voting screen of FIG. 13 and the voting message screen of FIG. 14 as a first sub-screen and a second sub-screen within a single screen. Additionally, in some embodiments, the second sub-screen may be displayed in the form of a pop-up window on a single screen.

Through these operations, the participation of the second user 100_2 in voting during the voting period in step S1220 is reflected as information of the voting event.

When the voting period ends, the event management module 137 counts the number of votes for each creative emoticon in the voting group and assigns a voting ranking. In order to generate voting details of the voting event, the event management module 137 receives a voting participation request from a second user including the identifier of a specific creative emoticon in the voting group selected by the second electronic device as a voting target. When received, the number of votes of the specific creative emoticon is counted, and for each creative emoticon, the creative emoticon with a higher number of counted votes may be given a higher voting rank than the creative emoticon with a lower number of counted votes (S1230) .

If the voting event is related to point payment, the event management module 137 provides the point management module 135 with a list of the second users (100_2) who participated in voting during the voting period or manages the voting details for points after the voting period ends. It may also be provided as a module 135.

The voting event progress process includes obtaining, by the event management module 137, a list of second users (100_2) who participated in the corresponding voting event (S1240); And it may also include a step (S1250) of paying a preset amount of points for the corresponding voting event as voting participation points to the second user (100_2) in the list of second users acknowledged to have participated.

In the above step (S1240), when the voting participation of the second user 100_2 is acknowledged, the event management module 137 provides information on the acknowledged second user 100_2 to the point management module 135 in real time, or By providing voting details to the point management module 135 after the end of the voting period, the point management module 135 can obtain a list of second users 100_2 whose voting participation is acknowledged.

In some embodiments, the voting participation points may be an equal amount for all second users 100_2 whose voting participation is acknowledged.

In addition, in some embodiments, the voting event progress process is performed by the event management module 137 to give the second user 100_2, who voted for the creative emoticon with a higher voting rank, a creative emoticon with a lower voting rank. A step (S1260) of paying a larger number of points as voting compensation points than that of the other second users (100_2) who voted may be further included.

In this way, by paying voting reward points in proportion to the voting ranking, the frequency of use of creative emoticons can be further increased by encouraging more users to purchase and enter messages for the creative emoticons for which they want to vote.

Creation of creative emoticons based on the meaning of the background and the meaning of emoticon object actions

Additionally, the first electronic device 110_1 controls the motion of the emoticon object and the background visual of the background sticker based on the semantic information of the background visual element to which the animation effect is applied in the background sticker and the semantic information of the motion of the emoticon object in the captured image. The animation effects of the elements may be further configured to create more naturally coordinated creative emoticons.

These creative emoticons may be created in step S450. Since the creation of such a creative emoticon is similar to the creation of a creative emoticon in step S450 described with reference to FIGS. 4 and 6, the differences will be mainly explained.

FIG. 15 is a flowchart of a process for generating a creative emoticon based on semantic information of a background visual element and semantic information of an action of an emoticon object in a captured image, according to various embodiments of the present application.

Referring to FIG. 15, the process of creating a creative emoticon based on the semantic information is: acquiring a shooting sequence image of a user who continuously photographs a user implementing one or more operations in the first electronic device 110_1, and capturing the shooting sequence image Obtaining emoticon object data related to the emoticon object sequence image from (S1510); A step of requesting a purchase of the background sticker from a market site where content products including the background sticker and the creative emoticon are traded to a server operating the market site and receiving the corresponding background sticker data (S1520); Inputting the emoticon object sequence image into a previously learned motion recognition model to recognize the motion of the emoticon object in the sequence image (S1530); Obtaining semantic information of background text included in the background sticker from the background sticker data (S1540); and generating creative emoticon data combining a background sticker and the emoticon object sequence image based on the recognition result of the motion and the semantic information of the background text (S1550).

The shooting sequence image is a set of images taken sequentially of the user, and includes a subset according to each operation.

The emoticon object sequence image in step S1510 is a sequence image in which the emoticon object images described above in step S310 are sequentially photographed. The emoticon object sequence image consists of a set of emoticon object images in which an emoticon object appears in at least a partial region in each captured image forming the captured sequence image.

In some embodiments, the emoticon object sequence image may be implemented as GIF (Graphics Interchange Format) image data. Then, the emoticon object sequence image may be sequentially output from the image in the first output order to the image in the last output order, and then the image in the first output order may be output again.

In step S1520, the background sticker has an animation sequence image in which one or more background visual elements are expressed as animation. The background visual elements are background text and/or background objects. For example, a background sticker may be purchased with animation effects applied to the background object shape and/or background text. The animated sequence image includes a subset of one or more background visual elements to which the same animation effect is applied.

In the step (S1530), the motion recognition model is a neural network configured to recognize the motion of the emoticon object appearing in the input image by extracting features of the input image to infer the correlation between the object movement of the input image and the motion indicated by the motion. It may be a machine learning model based. The image-based emotion recognition model has, for example, a model structure based on CNN (Convolutional Neural Networks) and other NNs.

In various embodiments, the motion recognition model may obtain data about joint points by extracting features from each frame of the input sequence image. Data about the joint point may be expressed as a scale value, a translation value (e.g., a joint coordinate value), or a rotation value (e.g., a movement value). In some embodiments, the size, position, and rotation values may be expressed as three-dimensional vectors. In various embodiments, the recognition of the action may be recognition of a type of action.

The motion recognition model is trained in advance to recognize motion types using a training data set consisting of a plurality of training samples. Each training sample has a sample sequence image showing continuous movements of an emoticon object as training data, and data describing the type of movement indicated by the corresponding movement as label data. The training data set may be subset into each of a plurality of operation types. In some embodiments, the label data of the motion type may be composed of data about a joint point that implements a reference motion indicating the corresponding motion type.

The motion recognition model is trained to minimize the error between the output value (i.e., predicted motion type) calculated by inputting the training data of the training data set and the label data (i.e., actual motion type) corresponding to the input training data. . As learning progresses and more training data is input, the values of model parameters (eg, weight, bias, etc.) in the motion recognition model are updated so that the error gradually decreases. Model parameters for which learning has been completed may have values that minimize errors.

In various embodiments, learning of the motion recognition model may be performed in the server 130 or the electronic device 110. When performed in the server 130, the electronic device 110 may receive a learned motion emotion recognition model and use it to create a creative emoticon.

By inputting an emoticon object sequence image into the learned motion recognition model, a motion type value predicted to be taken by the first user 100_1 may be obtained.

In some embodiments, when the server 130 includes a motion recognition model and a secondary emotion recognition model, the motion recognition task of step S1530 may be performed by the motion recognition model.

In the step S1540, the meaning of the background visual element included in the background sticker data may be an emotion type. The first electronic device 110_1 may obtain emotion type information previously associated with the background text in the background sticker data as the meaning of the text. The emotion type information is associated with the corresponding visual element at the time of creating the background sticker.

In various embodiments of the present application, the step (S1550) is performed so that, when the frame size of the purchased background sticker and the frame size of the emoticon object sequence image are different, the emoticon object sequence image is matched to the frame size of the purchased background sticker. adjusting the frame size; calculating a sub-output section according to a recognized motion in the entire output section of the captured sequence image; calculating a sub-output section according to the meaning of each background text from the entire output section of the background sequence image; Implementing an emoticon object sequence image in the first layer (51) using the emoticon object data associated with the emoticon object sequence image with the frame size adjusted; For each recognized action, mapping the background text and the corresponding action if the meaning of the background text corresponds to a meaning previously associated with the corresponding action; adjusting background sequence image data of the background sticker data so that the starting point of the sub-output section of the mapped background text matches the starting point of the sub-output section of the corresponding operation; It may also include implementing a background sequence image of the background sticker in the second layer 52 using the background sticker data including adjusted background sequence image data.

In step S1550, the first electronic device 110_1 may calculate a sub-output section according to the motion of the emoticon object recognized by the motion recognition model in the entire output section of the captured sequence image. When a shooting sequence image is captured while the first user 100_1 performs a plurality of actions, a sub-output section for each of the plurality of actions is calculated. The sub-output section is defined as the range from the output order of the image in which the motion first appears to the output order of the image in which the motion appears last.

In step S1550, the first electronic device 110_1 may calculate a sub-output section according to the meaning of the background visual element in the background sequence image. When a single background sequence image provides multiple animation effects by multiple groups of background visual elements, each sub-output section from the start to the end of each animation effect is calculated.

In the first layer 51 of FIG. 5, the emoticon object sequence image is implemented so that the frame size matches the frame size of the background sticker.

For each recognized motion, the first electronic device 110_1 determines whether the meaning (i.e., emotion type) of the background visual element shown in each animation sequence image corresponds to a meaning previously associated with the corresponding motion. To this end, the first electronic device 110_1 may store an operation semantic table in advance.

The action semantic table records emotion types previously associated with each action. For example, the action semantic table may include “joy” as the emotion type associated with the action type “hurray.” One action type may have multiple emotion types associated with it. Additionally, multiple action types may be associated with one emotion type.

The first electronic device 110_1 selects one of the emotion types previously associated with the motion recognized using an action semantic table recording the emotion types previously associated with each motion and the emotion type into which the background visual element is classified. If this matches, it is determined that the meaning of the background visual element corresponds to the meaning previously associated with the corresponding action.

FIG. 16 illustrates a creative emoticon in which a single action and a single animation sequence image are mapped to each other, according to various embodiments of the present application.

In the emoticon object sequence image used to create the creative emoticon in Figure 16, the action type of “cover mouth” is recognized. The background sticker used to create the creative emoticon of FIG. 16 includes an animation sequence image 1610 showing a concentrated line and an animation sequence 1620 showing a background shape expressing surprise. The animation sequence image 1610 is classified as a normal emotion type, and the animation sequence image 1620 is classified as an emotion type of “joy.”

If the emotion type of the recognition operation result or animation sequence image is normal, the mapping operation may not be performed.

The first electronic device 110_1 searches for the “cover mouth” action in the action semantic table and determines whether the emotion type associated with the “cover mouth” action has the emotion type of “joy.” If the emotion type of "joy" is previously associated with the "cover mouth" action, the first electronic device 110_1 determines that the "cover mouth" action and the animation sequence image 1620 of FIG. 16 have corresponding meanings. and map them to each other. The background sequence image data included in the background sticker data is modified so that the starting point of the sub-output section of the mapped background text matches the starting point of the sub-output section of the corresponding operation. For example, the output start time of the animation sequence image 1620 may be modified to be synchronized with the start time of the movement of the “mouth covering” motion. Then, the creative emoticon is created so that the animation sequence image 1620 provides an animation effect that appears at the start of the mouth covering motion.

As a result, the action is provided first to the other person who received the creative emoticon, and animation effects with the same meaning are provided in duplicate, thereby further reducing the impact of the animation effect, or the animation effect is provided first, thereby reducing the impact of the emoticon object to take. It is possible to prevent the problem that the influence of motion transmission is further reduced by spoiling the motion.

Additionally, in various embodiments of the present application, the first electronic device 110_1 may generate a creative emoticon by combining an emoticon object sequence image showing one or more actions and an animation sequence image providing one or more animation effects.

In various embodiments of the present application, when a single action is recognized and the background sticker includes a plurality of background visual elements with different output orders, in step S1550, the first electronic device 110_1 displays each animation sequence. It may be determined that the meaning of the background visual element appearing in the image corresponds to the meaning previously associated with the corresponding action. Then, multiple animation sequence images may be mapped to a single action.

Then, the first electronic device 110_1 sets the earliest output order among the plurality of background animation sequence images to match the start point of the sub-output section of the animation sequence image. The animation sequence image data of the background sticker data can also be adjusted.

FIG. 17 illustrates a creative emoticon in which a plurality of actions and a single animation sequence image are mapped to each other, according to various embodiments of the present application.

Referring to FIG. 17, when a plurality of consecutive actions are recognized and the background sticker includes a single animation sequence image, in step S1550, the first electronic device 110_1 selects the background time shown in the single animation sequence image. It may be determined that the meaning of an element corresponds to a meaning previously associated with each of a plurality of operations. Then, multiple actions may be mapped to a single animation sequence image.

For example, the animation sequence image 1710 provides an animation effect in which the letters "Going" appear in a sequence at specified positions in the frame, and the recognized action is the emoticon object making a hurrah gesture with its elbows folded followed by an uppercut motion with one hand. It may be. Then, the first electronic device 110_1 determines that the hurrah motion and uppercut motion of FIG. 17 have the meaning of an emotional type corresponding to the visual element (character “fighting”) shown in the animation sequence image 1710, and performs a plurality of motions. and a single animation sequence image 1710 may be mapped to each other.

Then, the first electronic device 110_1 selects the starting point of the sub-output section of the background visual element as the sub-output of the operation with the earliest output order among the plurality of consecutive operations (e.g., the hurray operation in FIG. 17). The animation sequence image data of the background sticker data may be adjusted to match the starting point of the section. That is, the start of output of the animation sequence image 1710 coincides with the start of output of the operation with the earliest output order (the hurray operation in FIG. 17).

FIG. 18 illustrates a creative emoticon in which a plurality of actions and a plurality of animation sequence images are mapped to each other, according to various embodiments of the present application.

Referring to FIG. 18, when a plurality of consecutive actions are recognized and the background sticker includes a plurality of animation sequence images, in step S1550, the first electronic device 110_1 adds a plurality of animation sequence images to each of the plurality of animation sequence images. It may be determined that the meaning of the displayed background visual element corresponds to a meaning previously associated with each of the plurality of actions. Then, multiple actions may be mapped to multiple animation sequence images.

For example, animation sequence image 1810 provides an animation effect in which multiple "knob" characters appear sequentially at specified positions in a frame, and animation sequence image 1820 provides an animation effect in which multiple speech bubbles containing an It provides animation effects that appear in order, and recognized movements may be crossing both hands to form an X or shaking the head.

Then, the first electronic device 110_1 converts the hand crossing motion and head shaking motion of FIG. 18 into emotional types corresponding to the visual elements (letter “knob”, letter “X”) shown in the animation sequence images 1810 and 1820. It is possible to judge that it has a meaning and map a plurality of actions and a plurality of animation sequence images (1810, 1820) to each other.

Then, the first electronic device 110_1 specifies the output order between animation sequence images, specifies the output order between recognized actions, and maps the start points of the sub-output sections of each animation sequence image to each other and in the same output order. The animation sequence image data of the background sticker data may be adjusted to match the starting point of the sub-output section of the operation having . The first electronic device 110_1 designates the output order of the hand crossing motion and the head shaking motion in FIG. 18 as motion 1 and motion 2, and selects the animation sequence image 1810 as animation number 1 and animation sequence image ( 1820) is designated as animation number 2. The first electronic device 110_1 starts outputting the animation sequence image 1810 and the hand crossing motion at the same time, which are mapped to each other and have the same output order, and the animation sequence image 1810 and the hand cross motion are mapped to each other and have the same output order. The output of the crossing motion starts simultaneously, and an animation sequence image 1820 that is mapped to each other and has the same output order and a creative emoticon in which the output of the head shaking motion starts simultaneously are created.

The created creative emoticon may be transmitted to another electronic device 100 through a messenger application, or may be requested to be registered as a new sales product on a market site (S1560).

The transmission of the creative emoticon and the registration request to the market site in the step (S1560) have been described above, and detailed explanations will be omitted.

Creation of creative emoticons based on the emotional intensity of previous messages

Additionally, the first electronic device 110_1 acquires the text of the message of the first user 1101 or another user 100, and the first electronic device 110_1 transmits the creative emoticon through the messenger application. It may be further configured to calculate the emotional intensity of the message obtained in advance, generate a creative emoticon in which the calculated emotional intensity is reflected in the animation effect of the background sticker, and transmit the created creative emoticon through the messenger application.

The message may be the user's message transmitted to another user's electronic device by running a messenger application installed on the electronic device and/or the other user's message received from the other user's electronic device.

These creative emoticons may be created in step S450. In some embodiments, the creative emoticon is generated according to meaning in the step (S1550) of generating creative emoticon data combining a background sticker and the emoticon object sequence image based on the recognition result of the motion and semantic information of the background visual element. It can also be created by modifying the animation effect based on the emotional intensity after the output section is mapped.

Since the creation of such a creative emoticon is similar to the creation of a creative emoticon in step S450 described with reference to FIGS. 4 and 6, the differences will be mainly explained.

Figure 19 is a flowchart of a process for generating a creative emoticon based on the emotional intensity of a previously obtained message, according to various embodiments of the present application.

Referring to FIG. 19, the text of at least some of the messages obtained before transmitting the creative emoticon message is input into a pre-trained natural language processing model to calculate an embedding vector obtained by embedding the text of each message ( S1910); Inputting a set of embedding vectors for the at least some messages into a previously learned text-based emotion recognition model to calculate the user's emotional intensity level reflected in the at least some messages (S1920); and adjusting the size of the animation effect and/or background visual element according to the calculated emotional intensity level (S1930). Additionally, in some embodiments, the process of FIG. 19 further includes implementing an animation sequence image of the background sticker in the second layer using background sticker data adjusted according to the calculated emotional intensity level (S1940). do. Additionally, in some embodiments, the first electronic device 1101 uses the background sticker data of step S1550 to implement an animation sequence image of the background sticker in the second layer, such that the calculated emotional intensity level An animation sequence image of the background sticker may be implemented in the second layer using background sticker data adjusted according to (S1940).

Figure 20 is a schematic diagram of a learning process of a text-based emotion recognition model according to various embodiments of the present application.

The text-based emotion recognition model processes the input embedding vector to infer the correlation between the embedding vector corresponding to the natural language data and the emotional intensity, and calculates the emotional intensity of the user 100 at the time of generating the natural language data as a level value. It is a machine learning model designed to predict. The emotional intensity level may be classified as, for example, low (or calm), normal, or high (or amplified), with higher level values indicating higher emotional intensity and lower level values indicating lower emotional intensity. However, it is not limited to this

Referring to FIG. 20, the text-based emotion recognition model includes a preprocessing model, a natural language processing model, and a classification model.

Messages used to create creative emoticons or similar sample messages are input into the text-based emotion recognition model as natural language data. The natural language data includes a plurality of messages exchanged between conversing users 100. The message may be a group of messages in which the speaker is the same (eg, input messages from the first user 100_1).

The preprocessing model is configured to preprocess sentences and phrases in the input natural language data and perform various text preprocessing operations such as label error correction and extracting labeled sentences or phrases.

The natural language processing model is configured to process the pre-processed input text into natural language and calculate the results of numerical analysis of the contents expressed in the input text as an embedding vector. For example, the natural language processing model may include an encoder. The encoder calculates an embedding vector obtained by processing the input text into natural language, and may be implemented with an artificial neural network. The artificial neural network may include one or more hidden layers that segment the input text into text units (eg, words, morphemes), extract features of subtext for each text unit, and calculate a hidden state vector. The values calculated through the hidden layer are concatenated and output as an embedding vector.

The classification model is a machine learning model configured to calculate the emotional intensity level of the user 100 that is predicted to have been present at the time the text corresponding to the input embedding vector was created by processing the input embedding vector. The machine learning model has a neural network structure. For example, the neural network structure may be composed of a fully connected layer or other NN structure including a multi-layer perceptron (MLP).

The text-based emotion recognition model calculates probability values to be classified in the input vector for each of a plurality of preset emotional intensity levels, and the user at the time the corresponding natural language data is generated based on the probability values calculated for each emotional intensity level. The predicted emotional intensity level of (100) may be determined. For example, the emotion recognition model of the neural network structure is configured to determine which emotion intensity level at which text data is most likely to be classified based on the correlation between vector data obtained by embedding text data and the emotion intensity level. do

The text-based emotion recognition model may be trained in advance using a training data set consisting of a plurality of training samples. Each training sample has a sample embedding vector obtained by processing the sample message into natural language as training data, and has the emotional intensity level specified by the user 100 at the time of entering the sample message as label data.

When each training data is input, the text-based emotion recognition model calculates the result value (i.e., predicted emotional intensity level) by processing the input training data with internal parameters (e.g., weight, bias), and produces the calculated output. It is learned to minimize the error between the value and the label data (i.e., actual emotional intensity level) corresponding to the input training data. As learning progresses and more training data is input, the values of model parameters (eg, weight, bias, etc.) in the text-based emotion recognition model are updated so that the error gradually decreases. Model parameters for which learning has been completed may have values that minimize errors.

In various embodiments, learning of the text-based emotion recognition model may be performed in the server 130 or the electronic device 110. When performed on the server 130, the electronic device 110 may receive a text-based emotion recognition model that has been trained and use it to create a creative emoticon.

When a previous message written by the first user (100_1) is input to the text-based emotion recognition model learned in this way, an embedding vector that quantifies the content of the previous message is calculated after preprocessing, and the calculated embedding vector is input into the classification model and corresponding The emotional intensity level of the first user (100_1) shown in the message is calculated.

In some embodiments, the text-based emotion recognition model is inputted with the message immediately preceding the creation of the creative emoticon, and an embedding vector is calculated as a result of numerically analyzing the contents of the most recent message. The emotional intensity level of the first user 100_1 shown in may be calculated. Here, the emotional intensity level is an emotional intensity level analyzed in terms of message content.

In some other embodiments, the text-based emotion recognition model includes the message immediately preceding the creation of the creative emoticon and the group of previous message(s), and the emotion of the first user 100_1 that appears in the group of the input messages is input to the text-based emotion recognition model. An intensity level may be calculated. Here, the emotional intensity level is an emotional intensity level based on content and context. To this end, the natural language processing model may be further configured to calculate an embedding vector for the entire content of the target message for which the embedding vector is to be calculated and at least one previous message. The at least one previous message includes the message immediately preceding the target message (eg, the immediately preceding input message of the first user 100_1).

A group of a series of messages (or preprocessed messages) may be input to the natural language processing model. The natural language processing model processes each series of continuously input messages into natural language to calculate a series of embedding vectors for the series of messages, and combines the contents of the series of messages in order to provide text for the entire contents of the series of messages ( or preprocessed text). Since the context of a conversation is based on the flow of message content before and after, in order to numerically analyze the context of the message, the entire content of the series of messages is processed in natural language to calculate an embedding vector for the entire content of the series of messages. . The natural language processing model calculates a single embedding vector for a group of input messages by concatenating a series of embedding vectors for a series of messages and an embedding vector for the entire content of the series of messages. The single embedding vector is the result of numerical analysis of the text of a series of messages in terms of content and contexts. When the single embedding vector is input to the classification model, a predicted emotional intensity level is calculated considering content and context.

For this purpose, the classification model is learned in advance using a training data set consisting of a plurality of training samples. Each training sample has as training data a single sample embedding vector for a group of a series of input messages, calculated from a series of sample messages. The process of learning a classification model using this training data set is similar to the learning process described above with reference to FIG. 19, so detailed description is omitted.

Referring again to FIG. 19, in step S1930, the existing animation effect may be maintained or the animation effect may be adjusted depending on the emotional intensity level. The animation effect may be adjusted when the emotional intensity level is higher than normal. Adjusting the size of the animation effect and/or background visual element includes adjusting the output speed of the animation sequence image and/or adjusting the change in size of the background visual element according to the animation effect according to the emotional intensity level. .

In some embodiments, in order to adjust the animation effect, the first electronic device 110_1 adjusts the background sticker data so that the animation speed increases as the calculated emotional intensity level increases compared to the animation speed at lower emotional intensity levels. Or, as the calculated emotional intensity level is higher, the background sticker data is adjusted to increase the size change of the background visual element according to the animation effect compared to the size change of the background visual element according to the animation effect at a lower emotional intensity level. It may be configured to do so. The increase in size change is implemented by decreasing the size of the background visual element at the start of the animation effect and/or increasing the size of the background visual element at the end of the animation effect.

The size of the background visual element may be increased to satisfy the above-described critical exposure ratio.

The size of the background visual element may be reduced to satisfy a preset minimum size for the background visual element.

Figure 21 shows a creative emoticon in which the size change of the background visual element is adjusted according to the intensity of emotion, according to various embodiments of the present application.

The creative emoticon in FIG. 18 is output when the emotional intensity level is normal, and the creative emoticon in FIG. 21 is output when the emotional intensity level is higher.

Referring to FIG. 21, the final size of the background text providing the animation effect increases compared to FIG. 18, so the range of change in size of the animation effect may further increase as the emotional intensity increases. Ultimately, by using the emoticon of Figure 21, emotional excitement can be better conveyed through animation effects.

NFT issuance and transfer for content products

Additionally, the content service system 1 may be further configured to issue NFTs for individual content (eg, background stickers, creative emoticons, etc.) used as content products. The content service system 1 can convert individual content into NFT and use it as WEB 3.0 independent content that is not dependent on a specific platform and can be used as long as network environment conditions are met.

In this specification, a non-fungible token (NFT) refers to a digital asset that has scarcity. NFT is based on blockchain, so all information related to the content product described below is stored in blockchain, so proof of originality is possible and there is no risk of tampering with data such as owner information (e.g., creator information or buyer information) and transaction history. is impossible, so integrity can be secured.

Because the metadata and information of digital assets are distributed and stored in participating nodes through the blockchain network, forgery and alteration are impossible as they are distributed and stored through the blockchain network, and all transaction details from the original issuer to the current owner can be traced. However, being tamper-proof does not mean that the token's data (e.g., metadata) cannot be changed. The token data may be changed depending on the agreement of participants in the blockchain network. Then, the data of the changed token is distributed and stored again through the blockchain network, and eventually the data of the changed token is updated to be impossible to forge or falsify.

Using the metadata and information of these digital assets, it is possible to trace all transaction details that have not been forged or altered from the original issuer to the current owner.

Figure 22 is a schematic diagram of a content service system configured to issue NFTs for content products, according to another aspect of the present application.

Since the content service system 1 of FIG. 22 has similar components to the content service system 1 of FIG. 1, the differences will be mainly described.

Referring to FIG. 22, the content service system 1 configured to issue the NFT includes an electronic device 110 and a server 130. Additionally, in some embodiments, the content service system 1 may further include a file DB system 150.

The content service system 1 may be configured to operate in conjunction with the blockchain network 20. Below, we will first look at the configuration of the content service system 1 of FIG. 22, and then describe the linked blockchain network 20.

The electronic device 110 operates as a terminal device for uploading NFT objects, that is, content products, to the system or searching for other users' NFTs.

The first user (100_1) acts as a supplier who issues and sells NFTs for his/her content products, such as background stickers and creative emoticons, on the system.

The second user (100_2) performs the role of a buyer who purchases other users' NFTs on the system.

In some cases, the user 100 may perform both the supplier role and the buyer role.

The roles of these users (100_1, 100_2) are implemented in the system through electronic devices (110_1, 110_2).

In addition, when the server 130 receives an NFT issuance request or an NFT purchase request from the electronic device 110, it is further configured to perform an NFT-related operation according to the request. Additionally, the server 130 is linked to the blockchain network 20 to perform NFT-related operations.

In certain embodiments, the server 130 performs operations of generating blockchain data for content product data, registration, verification, and inquiry of blockchain information, transaction and transfer of ownership information through the blockchain network 20. It may also be configured to perform. Content products as objects of NFT include background stickers and creative emoticons uploaded to the market site by the first user (100_1).

The blockchain network 20 is linked to the content service system 1 through a network and can process data generated from the content service system 1. The blockchain network 20 is a distributed group of blocks containing information that records and tracks data between secure groups of systems or computers in a verifiable manner. The blockchain network 20 can be understood as a P2P network system that implements a distributed database with a blockchain structure.

The blockchain network 20 can be understood as a peer-to-peer network including a plurality of interconnected computing devices (hereinafter referred to as 'nodes'), and each node may include one or more processors to perform calculations.

Blockchain data within the blockchain network 20 is treated safely and trustworthy. To this end, the blockchain network 20 distributes the ledger publicly so that at least some users of the network can view all content in each block and users can view changes.

Each block has a hash value (digital fingerprint) that links it to the previous block, making it very easy to notice changes. The technology that guarantees the integrity of blockchain data based on this hash value is treated as the original technology of blockchain, and the original technology of blockchain can be viewed as a consensus mechanism between network participants. The characteristics of the consensus mechanism are separated into public blockchains and private blockchains. The consensus mechanisms of the former blockchain include Proof of Work (PoW), which relies on computing power, and equity, which relies on cryptocurrency holdings. It includes Proof of Stake (PoS) and Delegated Proof of Stake (DPoS), which is a method of forming a kind of National Assembly through reputation and voting, and a representative private blockchain consensus method is solving the Byzantine general problem. This may include the PBFT (Practical Byzantine Fault Tolerance) method, which is a solution that does this. In addition, the blockchain according to an embodiment of the present disclosure is based on BIX (Blue IndeX token) and NFT (Non-Fungible Token), which provide unique characteristics and value, and applies reliability securing technology, where reliability is mutually exclusive. It refers to the integrity of the ledger (immutability) between untrusted network participants. In other words, it guarantees data integrity and provides confidentiality, authentication, integrity, and non-repudiation services.

A public key and a private key are issued to the electronic device 110 through the server 130.

Public keys are generated only for senders using specific blockchain technology, confirming their origin to the blockchain platform (e.g., exchange).

Blockchain account information may include private keys and public keys. Users participating in the content service system 1 may have blockchain account information registered in the blockchain network 20.

The secret key is supplied to every user within the blockchain network 20 and is used by everyone within the system to accurately identify a unique user. The secret key is a transaction in the blockchain network 20 (e.g., a transaction that causes a 'digital payment method' to be transferred from a first user account to a second user account, a transaction that distributes a smart contract on the blockchain network 20). , transactions that execute smart contracts, etc.) can function as a digital signature of the user. The public key can function as the user's account address (hereinafter referred to as 'account').

Wallets that can store NFTs operate using private keys. In various embodiments, the wallet may include a human-readable string or number string.

Each NFT has a specific group of blocks containing information that proves ownership of the digital asset and allows users to essentially own a digital receipt. In this specification, digital assets include some or all of the information describing the content product. For example, the digital asset may be part or all of background sticker data or creative emoticon data.

The ‘digital payment method’ described in the above example may include cryptocurrency. Cryptocurrency is a means of payment recorded in a distributed system within the blockchain network 20, and is sometimes used as a term for a concept including digital tokens (hereinafter referred to as 'tokens'), but is understood in the description of this document. For convenience, ‘cryptocurrency’ and ‘token’ will be used separately as follows.

In the description of this specification, cryptocurrency refers to a means of payment used as a means of compensation for data recording and trust maintenance from the moment a blockchain network (20) is created. When referring to cryptocurrency in this sense, it refers to a native password. It may also be referred to by the name currency. However, the term 'cryptocurrency' described in the claims of this document does not necessarily include only the concept of 'native cryptocurrency', and may also include the concept of 'token'.

In the description of this specification, the token conceptually implemented in cryptocurrency may have a certain purpose or value as a means of digital exchange additionally generated in one blockchain network 20. Tokens can be voluntarily generated by nodes participating in the blockchain network 20, and token generation, transfer, and usage information can be recorded in a distributed system within the blockchain network 20. When using the Ethereum-based blockchain network (20) as an example to explain the concept of the terms described above, cryptocurrency means Ethereum, and tokens are used in the Ethereum blockchain network (20) according to the ERC-721 protocol. This may mean an issued ERC-721 token.

A transaction may be created or executed by a node (e.g., server 130, electronic device 110) using the blockchain network 20. Transactions may include information related to digital assets, information regarding transfers of NFTs/cryptocurrencies, or information regarding smart contracts. Nodes can confirm transaction processing results through blockchain.

Transactions that occur on the blockchain network 20 are cryptographically signed with the secret key of the blockchain account of the subject who created the transactions, so they can be recorded as cryptographic proof data that cannot be forged or altered on the blockchain network 20. there is.

When a transaction occurs, the nodes of the blockchain network 20 verify the integrity of the transaction that occurred, and the consensus algorithm implemented in the blockchain network 20 (e.g., Proof of Work (POW), Proof of Stake (POS), Based on DPOS (Delegated Proof of Stake), etc.), a new block is created to follow the already created block, and the newly created block can be propagated to other nodes and execute transactions. A block may contain multiple transaction information.

Information about blocks can be stored in a transaction database shared by nodes. A transaction database can be understood as a public ledger in which multiple nodes share the same information.

The blockchain network 20 can distribute and execute various smart contracts provided by various nodes. A smart contract is a rule written in digital commands, and when an event that follows the conditions of the rule occurs in the blockchain network 20, it can cause a specific transaction to occur or execute another smart contract according to the set rule. Prescription and grandfather storage operations can be performed more easily using the smart contract.

In certain embodiments, the blockchain network 20 may correspond to the Ethereum network or other Proof of Stake (POS)-based blockchain networks.

The server 130 may operate as a node of the blockchain network 20. For this reason, the server 130 may store content product data in blockchain data shared between a plurality of nodes of the blockchain network 20 or search for previously stored data.

In some embodiments, NFTs for background stickers (hereinafter referred to as background sticker NFTs) and NFTs for creative emoticons (hereinafter referred to as creative emoticon NFTs) may be issued together with a smart contract. The NFT's smart contract may have cryptocurrency-based trading operations as a condition for performing ownership transfer operations.

The operation of this server 130 will be described in more detail with reference to FIG. 23 below.

Figure 23 is a flowchart of the process of issuing a creative emoticon NFT according to various embodiments of the present application.

The first user 100_1 who creates a creative emoticon may, as an issuance requester, request the issuance of a creative emoticon NFT through the first electronic device 110_1. The second user 100_2 may be both a primary buyer and a secondary seller.

Referring to FIG. 23, the process of issuing the creative emoticon NFT is: the server 130 receives a request for issuance of a creative emoticon NFT including the user's own creative emoticon identifier from the first electronic device 110_1 of the issuing requester. Step S2010, and the server 130 has a unique token ID corresponding to the created emoticon of the first user 100_1 and converts the account address (or wallet address) of the first user 100_1 to the owner address. It includes a step (S2030) of generating a creative emoticon NFT of the first user (100_1) on the blockchain network (20).

In step S2010, the first electronic device 110_1 of the issuance requester may generate an issuance request for an NFT including a creative emoticon identifier. Since the creative emoticon data is stored in the first storage space 60 of the server 130, an issuance request including creative emoticon data is not necessary.

The first electronic device 110_1 may provide a screen for NFT creation. The screen is configured to select a target creative emoticon to issue an NFT from among the creative emoticons that have already been created. When the target creative emoticon is selected, the first electronic device 110_1 starts an operation to generate an NFT for the selected creative emoticon (S2010).

The request for issuance of the NFT may include a creative emoticon identifier as the subject of NFT issuance, and an issuance amount corresponding to the number of divided ownership of the corresponding creative emoticon data. In some embodiments, the issuance amount may be preset to 1.

The server 130 issues a creative emoticon NFT of the first user 100_1 based on the NFT issuance request (S2030). The server 130 has a unique token ID corresponding to the creative emoticon data using a smart contract previously distributed on the blockchain network 20 and has the account address of the first user 100_1 as the owner address. NFTs can also be created on the blockchain network 20. The creative emoticon data is mapped to the token ID.

In some embodiments, the smart contract may be a smart contract with a first protocol. The server 130 may generate information related to a smart contract with a first protocol for issuing a predetermined number of NFTs to a designated account and disseminate the smart contract with the first protocol to the blockchain network 20. The server 130 may spread the smart contract and issue a creative emoticon NFT.

Referring to the above example, in a system (1) based on the Ethereum network, the smart contract may be a smart contract written according to the ERC-1155 or ERC-721 token standard. In some embodiments, the issuance quantity of the first protocol may be 1. In other words, one NFT may be issued per NFT issuance request.

In step S2030, the initial owner of each NFT equal to the issued amount created on the blockchain network 20 is designated as the requesting user. When the user's NFT is first issued in the content service system 1, the NFT equivalent to the initially issued amount may be sent directly to the user's account address (S2030).

In some embodiments, the step (S2030) includes storing the creative emoticon data of the first user 100_1 included in the issuance request in a file DB system 150 that exists outside the blockchain network 20, It may also include generating metadata containing information related to the creative emoticon and then storing the metadata in the file DB system 150.

The server 130 may use all of the information in the creative emoticon data as metadata items, or may use some pre-designated items among the information in the creative emoticon data as metadata items. For example, metadata is classified into owner-related items, transaction history-related items, and creative emoticon-related items, and the creative emoticon-related items are some of the parameters of the creative emoticon data (e.g., creator, emoticon object identifier, background). sticker identifier, size, file format, etc.) or all of them.

In certain embodiments, the file DB system 150 may be an InterPlanetary File System (IPFS).

Additionally, the request for issuance of the NFT may further include information related to the first user (100_1). The information related to the first user 100_1 may further include the first user's name, a profile for the first user, and/or the first user's electronic signature. To this end, the first electronic device 110_1 may be further configured to receive this information.

Then, when the server 130 generates an NFT equal to the issuance amount, the creative emoticon data included in the issuance request is stored in the file DB system 150. In addition, the server 130 calculates a hash value for the creative emoticon data and then includes the name of the first user, the profile for the first user, the account address of the first user, the electronic signature of the first user, and the hash. Metadata including value and/or path data accessible to the creative emoticon data stored in the file DB system 150 may be created and stored in the file DB system 150. The server 130 may additionally store various information about the first user 100_1 and the creative emoticon included in the creative emoticon data and corresponding metadata in an internal database.

Since the metadata for the creative emoticon includes the user's path data (e.g., URL, other path address), as described later, the NFT of the first user is transferred and the NFT owned by the first user is 0. Even in the case of a dog, the fact that it is this user who entered/registered the creative emoticon data is continuously stored and maintained in the metadata associated with the sold NFT.

In this way, the content service system 1 stores creative emoticon data and metadata about the creative emoticon data in the file DB system 150 that exists outside of the blockchain network 20 and provides an accessible path to the metadata. By having a storage structure that stores data and the token ID of the NFT in the blockchain network (20), the time and cost required to store creative emoticon data or metadata can be reduced, and the creative emoticon data or metadata can be saved. You can access data faster. Furthermore, the connection between token ID and metadata is clearly guaranteed.

In some embodiments, the content of the metadata stored in the file DB system 150 is changed, thereby disconnecting or changing the connection between the creative emoticon NFT and the source data (i.e., creative emoticon data) created in the blockchain network 20. To prevent this from happening, the server 130 may additionally store at least one piece of information included in the metadata in the blockchain network 20 (S2030).

For example, the server 130 may additionally store path data for creative emoticon data included in metadata in the blockchain network 20. Then, the connection between the creative emoticon NFT and the source data (creative emoticon data) is further strengthened.

In addition, the process of issuing the creative emoticon NFT of FIG. 23 includes: transferring the creative emoticon NFT of the first user (100_1) issued in step (S2030) through a transaction with the second user (100_2) (S2050). Includes.

In the step S2050, the content service system 1 has the right to view the creation emoticon NFT of the first user 100_1 issued in step S2030 through a transaction between the first user 100_1 and the second user 100_2. It may be provided to users other than the issuer.

In certain embodiments, the transaction of the NFT may be performed based on cryptocurrency. In this case, the content service system 1 may further include an exchange server 140.

The exchange server 140 is configured to perform an NFT transaction process using cryptocurrency accepted by the exchange. In some embodiments, the exchange server 140 may be operated by an operator of the server 130.

The step of trading the user's creative emoticon NFT (S2050) includes receiving a creative emoticon NFT listing request from the electronic device 110 at the server 130; And the server 130 may include a step of listing (going public) the NFT on the exchange server 140 for free trading of the NFT corresponding to the issuance amount for the user's created emoticon.

The electronic device 110 of step S2050 may be the first electronic device 110_1 of the requester for issuance of the NFT in step S2010. The server 130 may list an already issued NFT on the exchange server 140 in response to the listing request received from the first electronic device 110_1.

When the NFT is listed on the exchange server 140, the exchange server 140 receives the sale price for the NFT from the first electronic device 110_1 that owns one or more NFTs, or a combination of the sale price and the number of sales. You may also receive . When one NFT is issued, it is unnecessary for the exchange server 140 to receive information on the number of items sold.

Additionally, the exchange server 140 may receive a purchase price for the corresponding NFT from the second electronic device 110_2 of one or more second users 100_2 or a combination of the purchase price and the number of purchases. When one NFT is issued, it is unnecessary for the exchange server 140 to receive information on the number of purchases.

When the sales price and the purchase price match for the same NFT, the exchange server 140 determines the first user 100_1 of the first electronic device 110_1, which transmitted the matching sales price, as the seller, and performs matching. The second user 100_2 of the second electronic device 110_1 that transmitted the purchase price may be determined as the purchaser. The smaller or equal number of the sell number and the buy number may be determined as the transaction number.

In some embodiments, the exchange server 140 may also be linked to the blockchain network 20 and operate as a node of the blockchain network 20. Then, the exchange server 140 transmits the cryptocurrency(s) corresponding to the transaction amount multiplied by the sales price by the number of transactions from the buyer's account address to the seller's account address, and distributes them on the blockchain network 20. Using a smart contract, you can also transfer creative emoticon NFTs equivalent to the transaction amount from the seller's account address to the buyer's account address.

The smart contract is a transfer that transfers a creative emoticon NFT corresponding to the transaction amount to the account address of the transferor (i.e., buyer) when the user transfers cryptocurrency corresponding to the transaction amount to a pre-designated account (e.g., owner account). It is configured to generate transactions.

In addition, the exchange server 140 may store a transaction history in the blockchain network 20 in which the ownership of an NFT equivalent to the transaction amount has changed from the seller's account address to the buyer's account address.

In some embodiments, the exchange server 140 may simultaneously perform a transaction to transfer ownership of a creative emoticon NFT equal to the transaction amount according to the transfer of cryptocurrency and a transaction to store the changed history in the blockchain network 20. . For example, the exchange server 140 may simultaneously perform the ownership transfer transaction and the storage transaction using a transfer single event or transfer batch event that can be implemented according to the ERC-1155 or ERC-721 token standard.

In some other embodiments, the exchange server 140 may supply some or all of seller information, buyer information, sales price, and transaction amount to the server 130. In this case, the server 130 may perform the ownership transfer transaction and storage transaction. That is, in system 1, the exchange server 140 mainly performs the operation of determining the seller and buyer, and the server 130 may be configured to perform the actual transfer of ownership of blockchain data.

A buyer who has purchased at least one of the NFTs of a specific user (eg, a seller) in the system 1 may view some or all of the creative emoticon data associated with the purchased NFT through his or her electronic device 110. Since the creative emoticon NFT includes path data for accessing the creative emoticon data as source data, it may be treated as a viewing right for the creative emoticon data. That is, only the user terminal of the user or another user who at least partially owns the user's creative emoticon NFT can access the creative emoticon data corresponding to the creative emoticon NFT.

The buyer's electronic device 110 may transmit a request to view creative emoticon data previously stored in the server 130. The viewing request includes the corresponding creative emoticon NFT.

The server 130 provides some or all of the information related to the creative emoticon already stored in the file DB system 150 and the information related to the creator based on the path data included in the NFT in the viewing request to the purchaser who sent the viewing request. It can also be transmitted to the electronic device 110.

In other specific embodiments, the NFT transaction may be performed through a market site based on service points. In other words, the market site's content products further include NFTs based on other content products on the market site, such as creative emoticon NFTs and background sticker NFTs. The content service system 1 is configured to perform an NFT transaction process using service points accepted as a means of payment, and the market management module 133 may form a product category made of NFT. That is, the exchange server 140 is unnecessary, and at least some of the operations of the exchange server 140 may be implemented in the server 130.

In this case, the step of trading the user's created emoticon NFT (S2050) is to upload the NFT owned as a content product to the market site (e.g., in the market management module 133) by the server 130. Receiving a sale price for the NFT or a combination of the sale price and the number to be sold from the first electronic device 110_1 that owns the NFT or more; It may also include receiving a purchase price for the corresponding NFT uploaded to the market site from the second electronic device 110_2 of one or more second users 100_2 or receiving a combination of the purchase price and the number of purchases.

When one NFT is issued, it is unnecessary for the server 130 to receive information on the number of items sold. Additionally, when one NFT is issued, it is unnecessary for the server 130 to receive information on the number of copies.

When the sales price and the purchase price match for the same NFT, the server 130 determines the first user 100_1 of the first electronic device 110_1, which transmitted the matching sales price, as the seller, and makes the matching purchase. The second user 100_2 of the second electronic device 110_1 that transmitted the price may be determined as the buyer. The smaller or equal number of the sell number and the buy number may be determined as the transaction number.

As described above, the server 130 is linked to the blockchain network 20 and operates as a node of the blockchain network 20. The server 130 uses a smart contract distributed on the blockchain network 20. Thus, creative emoticon NFTs equivalent to the transaction amount can be transferred from the seller's account address to the buyer's account address. In addition, the server 130 simultaneously transmits the creative emoticon NFT and transmits points corresponding to the transaction amount multiplied by the number of transactions by the sales price from the buyer's point wallet to the seller's point wallet.

Additionally, the content service system 1 may be configured to specifically issue NFTs for some creators' creative emoticons.

In some embodiments, the process of issuing the creative emoticon NFT may further include: generating a list of NFT issuing authorities by the server 130 (S2020).

The list of NFT issuing authorities is a list of creators with NFT issuing authority. Creators with the authority to issue the NFT may be selected based on the voting results in FIG. 12.

In some embodiments, the server 130 may select creators with the authority to issue NFTs from the creator who received the most votes to a preset threshold based on the voting results of FIG. 12. The critical ranking may be, for example, 10th or 100th, but is not limited thereto.

Then, in the step (S2030), the server 130 checks whether the issuance requester is an NFT issuance authority included in the list of NFT issuance authorities in step (S2020), and if it is confirmed that the NFT issuance authority is the creation It could be issuing emoticon NFTs.

The transaction price of the creative emoticon NFT may increase through secondary and tertiary transactions after initial issuance. The transaction price is determined by reflecting the artistry, symbolism, and other values of the created emoticon NFT. As mentioned above, if creative emoticons that have received a high number of votes are converted into NFTs, the price may be higher as the transaction progresses. The final purchaser of a creative emoticon NFT will retain the original data of these popular creative emoticons.

Meanwhile, the target of NFT issuance according to the issuance process of Figure 23 is not limited to creative emoticons. The object of issuance of the NFT may be a background sticker. In this case, the first electronic device 110_1 may generate background sticker data and request issuance of a background sticker NFT. In the process of FIG. 23, the operation using the creative emoticon may be performed by replacing it with background sticker data. As such, the issuance process of the background sticker NFT is similar to the issuance process of the emoticon NFT in Figure 23, so the differences will be mainly described.

In some embodiments, the issuance quantity of the background sticker NFT may be set to one or more. For example, the issuance quantity of the background sticker NFT may be set to 50 or less.

The background sticker NFT can also be traded. In some embodiments, the first purchase of the background sticker NFT may be performed through an auction. The total number of successful bidders may be less than the number of background sticker NFTs issued. One successful bidder may receive multiple background sticker NFTs.

The server 130 may provide various benefits to buyers who purchase the creator's background sticker NFT. If there are N buyers for a single background sticker NFT, the benefit will be provided to the buyer who owns the background sticker NFT at the time the benefit is provided.

In one embodiment, when a creative emoticon created using the background sticker NFT is uploaded to a market site for a buyer who purchases a background sticker NFT, the server 130 sells the uploaded creative emoticon for an amount less than the sales price. Purchase authority to purchase the creative emoticon may be provided. The purchase authority includes a discount rate or discount amount. The discount rate may be 100%.

In this way, if you purchase one background sticker NFT, you can have multiple additional creative emoticons, which can increase the need for purchasing background sticker NFT.

Additionally, in one embodiment, the server 130 may provide a purchaser who owns the creator's background sticker NFT with pre-purchase rights for the creator's subsequent background sticker NFT. The server 130 uses the background sticker NFT as proof of rights to search for a buyer who owns the background sticker NFT, confirms the purchase intention to the user 100 who has pre-purchase authority for the subsequent background sticker NFT, and , After the purchase of the user 100 with pre-purchase rights is completed, an auction for the first purchase of the subsequent background sticker NFT may be performed with the remaining issuance amount.

Additionally, the content service system 1 may be further configured to issue a service representative NFT.

The service representative NFT may be paid to some creators in the form of a PFP (profile picture). The number of people paying for the service representative NFT may be variable depending on the scale of the service. The scale of the service depends on the number of users, the number of creators, the total number of products, and the number of types of content products. The number of payers may gradually increase as the scale of the service grows. For example, the number of paying people may start with 100 people and then increase to 10,000 people.

In some embodiments, issuance of the service representative NFT may be initiated in response to satisfying preset issuance conditions without an issuance request from the electronic device 110. The preset issuance conditions may be the number of users for which the service scale is preset, the number of creators for which the service is preset, the number of content products for which the preset is set, and the number of types of content products for which the service scale is preset. The server 130 may store the Nth issuance conditions in advance. The N-order issuance condition has a larger condition value as the number increases.

The number of payments for the service representative NFT may be calculated based on the voting history for the creator accumulated up to the time of issuance of the service representative NFT.

The server 130 searches for the voting details in which each creator participated as a voting candidate during the service period from the start of the service to the time of issuance of the service representative NFT, and retrieves the number of votes obtained by the corresponding creator from the voting details, Based on the total value of the searched votes, creators corresponding to the preset number of payments can be selected. The larger the total number of votes, the more priority you will be selected for payment.

If the creator selected as the payer has already received the service representative NFT, the creator is excluded from the payer. The next ranked creator in the total by a controlled number is re-selected as a payout.

For example, if 100 people have already received service representative NFTs at the time of initial issuance and the service scale satisfies the conditions at the time of secondary issuance and an additional 1,000 service representative NFTs are issued, the server 130 records the voting details. Through this, creators can be ranked cumulatively according to the total results of votes received. If the above cumulative ranking includes creators who received the first issued service representative NFT, those creators are excluded and the number of creators excluded from the creators ranked 1001st or lower is additionally selected.

The server 130 may provide various benefits to creators who have received the service representative NFT. For example, the server 130 may pay points or cash to the creator who received the service representative NFT, or may provide additional compensation.

It will be clear to those skilled in the art that the content service system 1 may also include other components. For example, the content service system 1 may include other hardware elements necessary for the operations described herein, including input devices for data entry and output devices for printing or other data presentation.

When implementing embodiments of the present invention using hardware, application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), and programmable logic devices (PLDs) configured to perform the present invention. , FPGAs (field programmable gate arrays), etc. may be provided in the processor of the present invention.

Meanwhile, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable medium. Additionally, the data structure used in the above-described method can be recorded on a computer-readable storage medium through various means. Program storage devices, which may be used to describe a storage device containing executable computer code for performing various methods of the present invention, should not be understood to include transient objects such as carrier waves or signals. do. The computer-readable storage media includes storage media such as magnetic storage media (eg, ROM, floppy disk, hard disk, etc.) and optical readable media (eg, CD-ROM, DVD, etc.).

The embodiments described above combine the components and features of the present invention in a predetermined form. Each component or feature should be considered optional unless explicitly stated otherwise. Each component or feature may be implemented in a form that is not combined with other components or features. Additionally, it is possible to configure an embodiment of the present invention by combining some components and/or features. The order of operations described in embodiments of the invention may be changed. Some features or features of one embodiment may be included in another embodiment or may be replaced with corresponding features or features of another embodiment. It is obvious that claims that do not have an explicit reference relationship in the patent claims can be combined to form an embodiment or included as a new claim through amendment after filing.

It will be clear to those skilled in the art that the present invention can be embodied in other forms without departing from the technical spirit and essential features of the present invention. Accordingly, the above embodiments should be considered in all respects as illustrative rather than restrictive. The scope of rights of the present invention should be determined by reasonable interpretation of the appended claims and all possible changes within the equivalent scope of the present invention.

Claims (10)

In the method of generating a creative emoticon that can be transmitted through a messenger application from the electronic device of a user registered as a member of the emoticon service,
Obtaining a shooting sequence image of a user who continuously captures a user performing one or more actions and acquiring emoticon object data related to an emoticon object sequence image from the shooting sequence image - the shooting sequence image is a subset according to each action Includes, wherein the emoticon object sequence image includes a sequence image in which an emoticon object appears, each having at least a partial area within the captured sequence image;
A step of requesting a purchase of the background sticker from a market site where content products including background stickers and the creative emoticon are traded to a server operating the market site and receiving the corresponding background sticker data - the background sticker includes one or more background stickers an element having an animation sequence image represented by animation, the animation sequence image including one or more subsets corresponding to each background visual element, the background visual element being a background text or background object;
Inputting the emoticon object sequence image into a previously learned motion recognition model to recognize the motion of the emoticon object in the sequence image;
Obtaining semantic information of background visual elements included in the background sticker from the background sticker data; and
Comprising the step of generating creative emoticon data combining a background sticker and the emoticon object sequence image based on the recognition result of the motion and semantic information of the background visual element,
method. The method of claim 1, wherein creative emoticon data is generated by combining an emoticon object sequence image in which an emoticon object appears with at least a portion of a background sticker and a shooting sequence image, respectively, based on the recognition result of the motion and the semantic information of the background visual element. The steps are:
If the frame size of the purchased background sticker and the frame size of the emoticon object sequence image are different, adjusting the frame size of the emoticon object sequence image to match the frame size of the purchased background sticker;
calculating a sub-output section according to the motion recognized by the motion recognition model from the entire output section of the captured sequence image;
calculating a sub-output section according to the meaning of the background visual element shown in each animation sequence image in the entire output section of the background sticker;
Implementing an emoticon object sequence image in the first layer using the emoticon object data associated with the emoticon object sequence image with the frame size adjusted;
For each recognized motion, if the meaning of a background visual element shown in each animation sequence image corresponds to a meaning previously associated with the corresponding motion, mapping the animation sequence image and the corresponding motion;
adjusting the animation sequence image data of the background sticker data so that the starting point of the sub-output section of the mapped background visual element matches the starting point of the sub-output section of the corresponding motion;
Comprising the step of implementing an animation sequence image of the background sticker in the second layer using the background sticker data including adjusted animation sequence image data,
method. According to paragraph 2,
When a single action is recognized and the background sticker includes animation sequence images with different output orders, if the meaning of the background visual element appearing in each animation sequence image corresponds to a meaning previously associated with the corresponding action, the animation sequence image and The steps to map that action are:
comprising mapping a single recognized action with the plurality of background visual elements,
The step of adjusting the animation sequence image data of the background sticker data is,
Adjusting the animation sequence image data of the background sticker data so that the starting point of the sub-output section of the animation sequence image with the earliest output order among the plurality of animation sequence images matches the starting point of the sub-output section of the corresponding operation. Characterized by,
method. According to paragraph 2,
When a plurality of consecutive actions are recognized and the background sticker includes a single animation sequence image, if the meaning of the background visual element appearing in the single animation sequence image corresponds to the meaning previously associated with each action, the animation sequence image and The steps to map that action are:
comprising mapping each of a plurality of actions to the single animation sequence image,
The step of adjusting the animation sequence image data of the background sticker data is,
Characterized by adjusting the animation sequence image data of the background sticker data so that the starting point of the sub-output section of the animation sequence image matches the starting point of the sub-output section of the operation with the earliest output order among the plurality of consecutive operations. to,
method. According to paragraph 2,
When a plurality of consecutive movements are recognized and the background sticker includes a plurality of animation sequence images, if the meaning of the background visual element appearing in the single animation sequence image corresponds to the meaning previously associated with each movement, the animation sequence image The steps to map the corresponding actions are
comprising mapping each of the plurality of actions with a plurality of animation sequence images,
The step of adjusting the animation sequence image data of the background sticker data is,
Specifying an output order between the plurality of animation sequence images;
Specifying an output order between a plurality of recognized operations; and
And adjusting the animation sequence image data of the background sticker data so that the starting point of the sub-output section of each animation sequence image matches the starting point of the sub-output section of the operation that is mapped to each other and has the same output order. to,
method. According to paragraph 1,
The unit of the background visual element for determining whether the meaning of the background visual element corresponds to a meaning previously associated with the corresponding action is a text group or object group to which the same animation effect is applied,
The meaning of the background visual elements is classified into emotion types,
The meaning of the background visual element corresponds to the meaning previously associated with the corresponding action, which means that one of the emotion types previously associated with the action recognized using the action semantic table recording the emotion type previously associated with each action and Characterized in that the background visual element matches the classified emotion type,
method. The method of claim 2, wherein
Obtaining the text of the user's or another user's message - the message is sent to the other user's electronic device by executing any messenger application installed on the electronic device or from the other user's electronic device It is a received message from the other user,
The creative emoticon is created to be transmitted to the other user's electronic device through the messenger application,
The step of generating creative emoticon data combining a background sticker and the emoticon object sequence image based on the recognition result of the motion and semantic information of the background visual element,
Inputting the text of at least some of the messages obtained before transmitting the creative emoticon message into a pre-trained natural language processing model to calculate an embedding vector obtained by embedding the text of each message;
inputting a set of embedding vectors for the at least some messages into a previously learned text-based emotion recognition model to calculate the user's emotional intensity level reflected in the at least some messages; and
Further comprising: adjusting animation effects or adjusting the size of background visual elements according to the calculated emotional intensity level,
Implementing an animation sequence image of the background sticker in the second layer using the background sticker data,
Characterized in that an animation sequence image of the background sticker is implemented in the second layer using background sticker data adjusted according to the calculated emotional intensity level.
method. In clause 7,
The step of adjusting at least one of the animation effect of the background sticker data and the background visual element according to the calculated emotional intensity level,
As the calculated emotional intensity level is higher, the background sticker data is adjusted to increase the animation speed compared to the animation speed at a lower emotional intensity level, or as the calculated emotional intensity level is higher, the background visual element according to the animation effect is adjusted. Characterized in that it includes; adjusting the background sticker data to increase the size change greater than the size change of the background visual element according to the animation effect at a lower emotional intensity level.
method. The method of claim 7, wherein the text-based emotion recognition model is a natural language processing model that processes the input text in natural language to calculate an embedding vector, and calculates the input embedding vector to calculate the input text that the user predicted to have at the time of generating the input text. Includes a classification model that calculates the emotional intensity level,
The natural language processing model is,
When a group of a series of messages is input, each series of continuously input messages is processed into natural language to calculate a series of embedding vectors for the series of messages,
Combining the contents of a series of messages in order to obtain text for the entire contents of the series of messages, processing the entire contents of the series of messages in natural language to calculate an embedding vector for the entire contents of the series of messages, and
Concatenating the series of embedding vectors for the series of messages and the embedding vectors for the entire contents of the series of messages to yield a single embedding vector for the group of input messages,
Inputting a set of embedding vectors for the at least some messages into a pre-learned text-based emotion recognition model to calculate the user's emotional intensity level reflected in the at least some messages includes:
Characterized by inputting a single embedding vector for the group of input messages into the text-based emotion recognition model to calculate the emotional intensity level analyzed in terms of content and context of the group of input messages.
method. A computer-readable record recording a program that performs a method of generating a creative emoticon transmittable through a messenger application on the electronic device of a user registered as a member of the emoticon service according to any one of claims 1 to 9. media.

Images