KR102248706B1 - System for intergrated education management based on intelligent image analysis technology and method thereof - Google Patents

Abstract

An integrated education management system and method thereof are provided. In the integrated education management system according to some embodiments of the present disclosure, a collection unit for collecting images of an education place from one or more photographing devices and a training participant included in the collected image through facial recognition are identified, and the identification result It may include an attendance management unit to determine whether the attendance of the training participant based on.

Description

Integrated education management system and method based on intelligent video analysis technology {SYSTEM FOR INTERGRATED EDUCATION MANAGEMENT BASED ON INTELLIGENT IMAGE ANALYSIS TECHNOLOGY AND METHOD THEREOF}

The present disclosure relates to an integrated education management system and method based on intelligent video analysis technology. More specifically, it relates to an integrated education management system capable of performing attendance management, analysis of education concentration, and detection of abnormal situations occurring in the course of education using intelligent video analysis technology, and a method performed in the system. .

The development of information and communication technology and the spread of smartphones laid the foundation for creating a smart education environment, and various educational institutions are currently promoting the introduction of a system for smart education. A representative example of a smart education system is a smart attendance management system. Recently, some universities have introduced a system that automatically checks attendance through electronic cards, beacons, and Bluetooth functions of smartphones.

Such a smart attendance management system enables education officials such as students and instructors to quickly check attendance status through the Internet, thereby improving the efficiency and convenience of education management.

However, the above-mentioned attendance confirmation method only identifies the participant through a physical device such as a terminal or card possessed by the training participant, and thus cannot solve the problem of proxy attendance. In addition, since the beacon or Bluetooth-based attendance check method checks attendance based on distance, attendance can be recognized by communicating at a place other than an educational place. Therefore, even in this technique, it is difficult to rule out the possibility of irregular attendance.

In addition, there is a hassle of having to carry a terminal or a card separately, and there may be cases where the training participants are restricted by environmental factors such as humidity and temperature, and there is a risk of equipment loss.

In addition, most of the current research related to the creation of a smart education environment is focused on smart attendance management, and improves the effectiveness of education through analysis of the concentration of participants during training, or detects abnormal situations during training in real time to ensure the safety of the training place. There is hardly any research on the technology to be secured. For this reason, the establishment of a system capable of integrating attendance management, attitude management, and safety management of training venues of training participants is still inadequate.

Korean Patent Publication No. 10-2016-0145384 (published on December 20, 2016)

A technical problem to be solved through some embodiments of the present disclosure is to provide an integrated education management system capable of creating a smart education environment and a method performed in the system.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a system capable of solving a proxy attendance problem based on intelligent video analysis technology and automating attendance management for training participants, and a method performed in the system. It is to do.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a system capable of improving training effect and a method performed in the system by measuring the concentration of training participants based on intelligent video analysis technology. will be.

Another technical problem to be solved through some embodiments of the present disclosure is a system capable of creating a safe educational environment by detecting an abnormal situation occurring in an educational process based on an intelligent image analysis technology, and a system performed in the system. Is to provide a way.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a system capable of improving the convenience and accuracy of attendance management by improving the accuracy of facial recognition, and a method performed in the system.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a system capable of solving a problem in that the accuracy of facial recognition decreases according to wearing of jewelry, and a method performed in the system.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a system capable of securing real-time capability of detecting an abnormal situation by performing intelligent image analysis at high speed, and a method performed in the system.

Another technical problem to be solved through some embodiments of the present disclosure is to provide a system capable of mitigating a privacy invasion problem caused by video data leakage, and a method performed in the system.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the technical problem, the integrated education management system according to some embodiments of the present disclosure includes a collection unit that collects images of an education place from one or more photographing devices, and education included in the collected images through facial recognition. It may include an attendance management unit that identifies the participant and determines whether the training participant attends or not based on the identification result.

In some embodiments, at least one of the one or more photographing devices may further include a processing unit including a fisheye lens and performing a dewarping process on an image captured by the fisheye lens.

In some embodiments, at least one of the one or more photographing devices may photograph the entrance of the educational venue.

In some embodiments, the attendance management unit detects a facial area in a first direction and a facial area in a second direction from the collected image, extracts a first feature point from the facial area in the first direction, and the second A second feature point may be extracted from the facial area of the direction, and the training participant may be identified based on the first feature point and the second feature point.

In some embodiments, the attendance management unit analyzes the collected image to determine whether the training participant wears jewelry, and in response to the determination that the training participant has worn jewelry, through an image conversion model based on Generative Adversarial Networks (GAN). The collected image may be converted into an image from which ornaments have been removed, and the face recognition may be performed on the converted image.

In some embodiments, further comprising a concentration measurement unit for measuring the concentration of the training participant by analyzing the collected image based on at least one of a degree of departure from the training place, a gaze movement, and a degree of use of the personal terminal of the training participant can do.

In some embodiments, an abnormal situation detector configured to detect an abnormal situation occurring in the educational place by analyzing the collected image, and an action unit configured to perform an action corresponding to the detected abnormal situation may be further included.

In some embodiments, further comprising a storage unit for processing and storing the collected image, wherein the processing includes detecting a facial area in the collected image, and concealing at least a portion of the detected facial area, It may include a process of injecting the identification code of the training participant into the concealed facial area.

In the integrated education management method according to some embodiments of the present disclosure for solving the above technical problem, in the integrated education management method performed by a computing device, the steps of collecting an image of an education place from one or more photographing devices, It may include identifying the training participant included in the collected image through facial recognition, and determining whether the training participant attends or not based on the identification result.

A computer program according to some embodiments of the present disclosure for solving the above-described technical problem is combined with a computing device to collect an image of an education place from one or more photographing devices, and to the collected image through facial recognition. It may be stored in a computer-readable recording medium to execute the step of identifying the included training participant and determining whether the training participant attends or not based on the identification result.

According to various embodiments of the present disclosure described above, an integrated education management system capable of all attendance, attitude, and safety management, etc., may be constructed using intelligent video analysis technology, and a smart education environment may be created through the system. .

In addition, by checking attendance through intelligent video analysis, proxy attendance is blocked at the source and dropout during class can be prevented. In addition, the accuracy and efficiency of attendance management can be improved by automating attendance verification.

In addition, facial recognition may be performed using facial feature points of various angles. Accordingly, the accuracy of facial recognition and attendance confirmation may be improved.

In addition, by removing the ornaments on the image through the image conversion model, facial recognition can be accurately performed even when the training participant wears the ornaments.

In addition, by using intelligent video analysis technology, the degree of concentration of training participants may be measured, and the measured concentration information may be provided to various education related persons. Accordingly, the overall educational effect, including the degree of participation of training participants, can be improved.

In addition, by using intelligent video analysis technology, various abnormal situations that may occur during education, such as violence and fire, can be detected in real time, and appropriate actions can be automatically taken according to the detected abnormal situation. Accordingly, convenience of safety management can be improved, and a safe educational environment can be created.

In addition, image data analyzed to detect an abnormal situation through downsampling and excluding color difference components may be lightened. Accordingly, real-time detection of abnormal situations can be secured, and a safer educational environment can be created.

In addition, abnormal situation detection may be performed based on various features through feature map merging or synthesis, and accordingly, the accuracy of the abnormal situation detection may be improved.

In addition, since the masking process, which is a concealment process for the facial area, is performed before the image of the training place is stored in the system, the problem of privacy invasion of the training participant due to data leakage can be prevented in advance.

The effects according to the technical idea of the present disclosure are not limited to the above-mentioned effects, and other effects that are not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is an exemplary configuration diagram illustrating an integrated education management system according to some embodiments of the present disclosure.
2 illustrates a photographing position of a photographing apparatus using a standard lens according to some embodiments of the present disclosure.
3 and 4 are views for explaining a photographing position and a distortion correction method of a photographing apparatus using a fisheye lens or a wide-angle lens according to some other exemplary embodiments of the present disclosure.
5 is an exemplary block diagram illustrating an integrated education management server according to some embodiments of the present disclosure.
6 is an exemplary flowchart illustrating an attendance management method according to some embodiments of the present disclosure.
7 is an exemplary diagram for describing a face recognition method according to the first exemplary embodiment of the present disclosure.
8 to 11 are diagrams for describing a face recognition method according to a second exemplary embodiment of the present disclosure.
12 is an exemplary flowchart illustrating a method of measuring education concentration according to some embodiments of the present disclosure.
13 is an exemplary flowchart illustrating a method of detecting an abnormal situation according to some embodiments of the present disclosure.
14 to 16 are diagrams for explaining a detailed process of the abnormal situation detection step S320 shown in FIG. 13.
17 is an exemplary diagram for describing a privacy protection method according to some embodiments of the present disclosure.
18 illustrates an exemplary computing device capable of implementing devices according to various embodiments of the present disclosure.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Advantages and features of the present disclosure, and a method of achieving them will be apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the technical idea of the present disclosure is not limited to the following embodiments, but may be implemented in various different forms, and only the following embodiments complete the technical idea of the present disclosure, and in the technical field to which the present disclosure pertains. It is provided to completely inform the scope of the present disclosure to those of ordinary skill in the art, and the technical idea of the present disclosure is only defined by the scope of the claims.

In adding reference numerals to elements of each drawing, it should be noted that the same elements are assigned the same numerals as possible, even if they are indicated on different drawings. In addition, in describing the present disclosure, when it is determined that a detailed description of a related known configuration or function may obscure the subject matter of the present disclosure, a detailed description thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which this disclosure belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically. The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present disclosure. In this specification, the singular form also includes the plural form unless specifically stated in the phrase.

In addition, in describing the constituent elements of the present disclosure, terms such as first, second, A, B, (a) and (b) may be used. These terms are for distinguishing the constituent element from other constituent elements, and the nature, order, or order of the constituent element is not limited by the term. When a component is described as being "connected", "coupled" or "connected" to another component, the component may be directly connected or connected to that other component, but another component between each component It will be understood that elements may be “connected”, “coupled” or “connected”.

As used in this disclosure, "comprises" and/or "comprising" refers to the recited component, step, operation and/or element. It does not exclude presence or addition.

Components included in one embodiment and components including common functions may be described using the same name in other embodiments. Unless otherwise stated, the description in any one embodiment may be applied to other embodiments, and a detailed description will be omitted within the overlapping range or the range that can be clearly understood by a person skilled in the art. I can.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is an exemplary configuration diagram illustrating an integrated education management system according to some embodiments of the present disclosure. In the following description, when referring to any photographing apparatus (1-1 or 1-2, ?? or 1-n) or collectively referring to all photographing apparatuses (1-1 to 1-n), reference numeral "1" Try to use.

As shown in FIG. 1, the integrated education management system may include one or more photographing devices 1 and an integrated education management server 10. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some components may be added or deleted as necessary. For example, the integrated education management system may further include an education manager's terminal (not shown), an education participant's terminal (not shown), and the like. Hereinafter, for convenience of description, the integrated education management server 10 will be abbreviated as "management server".

It should be noted that each of the constituent elements of the integrated education management system shown in FIG. 1 represents functional elements that are functionally divided, and a plurality of constituent elements may be implemented in a form in which they are integrated with each other in an actual physical environment. In addition, in an actual physical environment, each of the constituent elements may be implemented in a form that is separated into a plurality of detailed functional elements. For example, a first function of the management server 10 may be implemented in a first computing device, and a second function may be implemented in a second computing device. Hereinafter, each of the above components will be described.

In the integrated education management system, the management server 10 is a computing device that provides an integrated education management function. The integrated education management may include, for example, attendance management of training participants, education concentration management (ie, education attitude management), safety management, and the like. However, the present invention is not limited thereto, and various types of management functions may be further included.

As mentioned above, the management server 10 may be implemented with one or more computing devices. Here, the computing device may be a notebook, a desktop, a laptop, etc., but is not limited thereto, and may include all types of devices equipped with a computing function and a communication function. However, in an environment in which real-time management is to be performed through intelligent image analysis, the management server 10 may be preferably implemented as a high-performance server-class computing device. Referring to FIG. 18 for an example of the computing device.

In various embodiments of the present disclosure, the management server 10 may collect various images of an education place from one or more photographing devices 1 and analyze the collected images based on intelligent image analysis technology. In addition, the management server 10 may perform integrated education management based on the analysis result. For example, the management server 10 may check the attendance of an education subject through intelligent image analysis, measure an education concentration, and detect an abnormal situation. According to the present embodiment, since the overall education management function can be automated using the intelligent image analysis technology, a smart education environment can be created. A more detailed description of the present embodiment will be described later with reference to the accompanying drawings in FIG. 5.

In the integrated education management system, the photographing device 1 is a device that photographs an education place and generates an image to be analyzed. The image may include various types/types of images such as a still image, a video image (ie, a moving image) composed of a plurality of frame images, an infrared image, and a visible light image.

The photographing device 1 may be installed at various locations in an educational place. For example, as illustrated in FIG. 2, the first photographing device 1-1 is installed to photograph the first entrance 21 of the education place, and the second photographing device 1-2 is of the education place. It may be installed to photograph the second entrance 23. In such a case, since people entering and exiting the training place can be accurately identified, integrated training management (e.g. attendance management) can be performed more effectively. Here, the first photographing apparatus 1-1 and the second photographing apparatus 1-2 may be devices having a standard lens (ie, a general camera lens other than a fisheye lens or a wide-angle lens).

The photographing device 1 may be implemented with various types of camera devices such as CCTV (Closed-Circuit Television), and any device may be implemented as long as it can generate an image. In addition, in some embodiments, some or all of the intelligent image analysis function may be implemented on the photographing device 1 side.

In some embodiments, one or more photographing apparatuses 1 having a fisheye lens or a wide-angle lens may photograph an educational site. In such a case, a processing process of correcting/removing image distortion (eg converting a circular panoramic image taken with a fisheye lens into a flat image) is performed through a dewarping technique, and the processed image is based on Intelligent image analysis can be performed. Hereinafter, the present embodiment will be further described with reference to FIGS. 3 and 4.

As illustrated in FIG. 3, the photographing apparatus 1 provided with a fisheye lens is located between the instructor's position 25-1 and the training participant's position 25-3 at the front of the training place rather than the entrance 25-5. It may be desirable to be placed in the vicinity. Due to the nature of the fisheye lens capable of 360-degree shooting, the photographing device 1 can take an image of the entire educational venue at the corresponding location, and this makes it possible for both instructors and participants to be intelligent, such as facial recognition, behavior recognition, and abnormal situation recognition. This is because an image technique can be applied.

In the above-described embodiment, the reason for using the fisheye lens is to reduce the cost burden while minimizing the security blind spot. CCTV security management technology using a conventional general-purpose camera (ie, a general camera that does not use a fisheye lens) has arranged a plurality of cameras in one space so as not to create a blind spot in a place where the camera is photographed. However, if a plurality of cameras are arranged in one space, there is a problem that installation and maintenance costs increase, and efficiency is lower than expected. In order to solve such a problem, it may be understood that a fisheye lens is used in the above-described embodiment.

Since the fisheye lens has a short focal length of 7 to 15 mm, a field of view of 180 degrees or more can be secured even with a single photographing device (e.g. 1 in FIG. 3). Therefore, by using the photographing apparatus (e.g. 1 in Fig. 3) equipped with a fisheye lens, it is possible to photograph a wide area without a blind spot at low installation cost and low maintenance cost. However, an image photographed with a fisheye lens may be generated in a very distorted form because refraction becomes severe as it goes toward the edge region. Accordingly, it is necessary to correct and use image distortion. In some embodiments of the present disclosure, a processing process for correcting image distortion may be performed based on Equation 1 below. However, since other equations may be used, the technical scope of the present disclosure is not limited thereto.

In Equation 1, f denotes the focal length of the camera, and P _f denotes the distance from the optical center of the camera to the (x, y) coordinates of the _{input image} (ie, (X input image, Y _{input image} )). In addition, the X _{input image} and the Y _{input image} are the (x, y) coordinate values of the input image, and the X _{distortion correction image} and the Y _{distortion correction image} refer to the (x, y) coordinate values of the image on which the distortion correction has been performed.

It is necessary to know the focal length f and the distance to the optical center P _f to obtain a distortion correction image by applying Equation 1 above. Therefore, when correcting distortion of an image formed by a fisheye lens, parameters such as a focal length or an optical center position have very important meanings.

An example in which distortion is corrected by Equation 1 is shown in FIG. 4. As shown in FIG. 4, the input image 27 may be corrected into an image 29 in a flatly unfolded form by Equation 1 above. For a description of the image distortion correction technique, reference is made to Korean Patent Publication No. 10-2011-0068375, and the entire disclosure of the patent may be incorporated herein by reference.

In various embodiments of the present disclosure, the photographing device 1 and the management server 10 may communicate through a network. Here, the network is all types of wired/wireless networks such as a local area network (LAN), a wide area network (WAN), a mobile radio communication network, and Wibro (Wireless Broadband Internet). Can be implemented.

So far, the integrated education management system according to some embodiments of the present disclosure has been described with reference to FIGS. 1 and 2. Hereinafter, a detailed configuration and operation of the management server 10 will be described in detail with reference to the accompanying drawings in FIG. 5.

5 is an exemplary block diagram illustrating a management server 10 according to some embodiments of the present disclosure.

As shown in FIG. 5, the management server 10 may include a collection unit 11, a processing unit 19, a management unit 13, an action unit 15, and a storage unit 17. However, only the components related to the embodiment of the present disclosure are shown in FIG. 5. Accordingly, those of ordinary skill in the art to which the present disclosure pertains can recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 5. In addition, it should be noted that each of the components of the management server 10 shown in FIG. 5 represents functional components that are functionally divided, and a plurality of components may be implemented in a form in which they are integrated with each other in an actual physical environment. Hereinafter, each component will be described.

The collection unit 11 may collect various types of information for integrated education management. For example, the collection unit 11 may collect an image of an educational place from the photographing device 1 in real time. However, the collection unit 11 may collect various types of monitoring data (e.g. Internet access information, voice data, etc.) from various monitoring devices (e.g. AP) in addition to the photographing device 1.

Next, the processing unit 19 may perform a dewarping process on the image acquired by the collection unit 11 (e.g. an image photographed by a fisheye lens). When the photographing apparatus 1 includes a fisheye lens or a wide-angle lens, as described above, distortion may occur in the image due to a short focal length and a wide angle of view. Accordingly, the processing unit 19 may perform a dewarping process that corrects distortion of the original image and converts it into a panorama form in order to apply an intelligent image technique such as facial recognition. If the collection unit 11 collects only images photographed with a standard lens, the processing unit 19 may be omitted.

Next, the management unit 13 may perform integrated management for training participants and/or training places through intelligent image analysis. The management unit 13 according to some embodiments may include an attendance management unit 13-1, a concentration measurement unit 13-2, and an abnormal situation detection unit 13-3.

The attendance management unit 13-1 may determine and manage the attendance of training participants through intelligent video analysis. For example, the attendance management unit 13-1 may perform facial recognition based on the collected image to identify the training participant included in the image. In addition, the attendance management unit 13-1 may determine whether the training participant attends or not based on the identification result. In order to eliminate redundant descriptions, a detailed description of the operation of the attendance management unit 13-1 will be described later with reference to FIGS. 6 to 11.

Next, the concentration measurement unit 13-2 may measure the concentration of the training participant through intelligent image analysis. For example, the concentration measurement unit 13-2 includes the degree of the education participant leaving the education place, the participant's gaze movement, the direction of the gaze, the degree of using his personal terminal (eg smartphone), the degree of body movement, etc. The concentration of each participant can be measured based on various indicators. In order to exclude redundant descriptions, a detailed description of the operation of the concentration measurement unit 13-2 will be described later with reference to FIG. 12.

Next, the abnormal situation detection unit 13-3 may detect an abnormal situation occurring in the educational place in real time through intelligent image analysis. The abnormal situation may include, for example, violence, fire, intrusion of a big player, and the like, but various types of abnormal situations may be further included, so that the technical scope of the present disclosure is not limited to the examples listed above. In order to exclude redundant descriptions, a detailed description of the operation of the abnormal situation detection unit 13-3 will be described later with reference to FIGS. 13 to 16.

Next, the action unit 15 performs a corresponding action according to the attendance check result, concentration measurement result, abnormal situation occurrence, etc. (ie, the intelligent video analysis result of the management unit 13). The specific action method may vary depending on the embodiment.

For example, when it is confirmed that a specific participant is late or absent, the action unit 15 is sent to the terminal (not shown) of an education official (eg, an education supervisor such as a parent or a supervisor, the education participant or acquaintance, education manager). You can provide a notification.

As another example, when the concentration of a specific training participant is less than the first standard value (eg poor class attitude) or more than the second standard value (eg excellent class attitude), the action unit 15 notifies the terminal (not shown) of the person concerned with education. Can provide. In addition, the action unit 15 may provide concentration status information or concentration measurement information to a terminal (not shown) of an education person in real time or non-real time.

As another example, in the event of an abnormal situation such as violence, fire, or intrusion of a large number of people, the action unit 15 provides a notification to a terminal (not shown) of an education person, or an emergency exit to a terminal (not shown) of an education participant Measures such as providing information or notifying an abnormal situation through broadcasting may be performed. In addition, the action unit 15 may provide a notification to an emergency dispatch agency (e.g. police station, fire station, hospital) and request emergency rescue.

Next, the storage unit 17 may store and manage various data and/or information related to integrated education management. For example, the storage unit 17 can manage training participant information DB 17-1, attendance information DB 17-2, instructor information DB 17-3, and other information DB 17-3. I can. The other information DB 17-3 may include raw image data, action policies, and intelligent image analysis results.

The education participant information DB 17-1 may store information such as identity information of the education participant, image or feature point information for facial recognition, and the education application status of the education participant.

The attendance information DB 17-2 may store information such as the attendance status (e.g. attendance, absence, lateness, etc.) of each training participant.

Instructor information DB 17-3 may store information such as an instructor's identity information and a professor's subject.

In some embodiments of the present disclosure, the storage unit 17 may perform concealment processing on at least a portion of the facial area before storing the image including the training participant. According to this embodiment, even if data leakage occurs, the privacy of training participants can be protected. This embodiment will be described in more detail with reference to FIG. 17.

Each component shown in FIG. 5 may mean software or hardware such as a processor (ie, a computing device), a Field Programmable Gate Array (FPGA), or an Application-Specific Integrated Circuit (ASIC). . However, the components are not limited to software or hardware, and may be configured to be in an addressable storage medium, or may be configured to execute one or more processors. The functions provided in the components may be implemented by more subdivided components, or may be implemented as one component that performs a specific function by combining a plurality of components.

Meanwhile, according to various embodiments of the present disclosure, an application operating in conjunction with the integrated education management server 10 may be provided, and various intelligent image analysis information (eg, attendance of instructors and training participants) may be provided through the application. , Concentration, abnormal situation information, etc.) may be provided. The application may be installed on the terminals of various educational personnel. Through the above application, instructors can more easily manage attendance and grades of training participants, and training participants can more easily check their attendance and grades.

The main function of the above application is a real-time attendance check, attendance management function including public attendance and consolidation processing, class notification function that provides information about classes in real time such as absences and reinforcement, schedules for each lecture and project, task management, and progress scale. It may include a community management function that provides a place for communication through the community, such as a schedule management function provided, meetings and information by department or subject.

Hereinafter, methods according to various embodiments of the present disclosure will be described in detail with reference to FIGS. 6 to 17. Each step of the above methods may be performed by a computing device. In other words, each step of the methods may be implemented with one or more instructions executed by the processor of the computing device. All of the steps included in the above methods may be performed by one physical computing device, but may be distributed and performed by a plurality of physical computing devices. For example, first steps of the method may be performed by a first computing device, and second steps of the method may be performed by a second computing device. In the following, for convenience of understanding, description will be continued on the assumption that each step of the above methods is performed by the management server 10. Accordingly, in the following description, when the subject of each operation is omitted, it will be understood that it can be performed by the illustrated apparatus 10.

First, an attendance management method according to some embodiments of the present disclosure will be described with reference to FIGS. 6 to 11. The following attendance management method may be performed by the attendance management unit 13-1 and the action unit 15.

6 is an exemplary flowchart illustrating an attendance management method according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some steps may be added or deleted as necessary.

As shown in FIG. 6, the attendance management method may begin in step S100 of collecting an image of an education place. For example, an image including visitors (e.g. training participants, beasts, etc.) may be collected from a photographing device (e.g. 1 in FIG. 1) that photographs the entrance of an education place.

In some embodiments, the collected image may be an image captured by a fisheye lens or a wide-angle lens (e.g. a circular panoramic image). In this case, before analyzing the collected image, a processing process of correcting/removing the distortion of the collected image (eg converting a circular panoramic image into a flatly unfolded image) may be performed using a dewarping technique. have. In addition, by analyzing the processed image, training participants can be identified.

In step S120, the training participant may be identified by analyzing the collected image. For example, the identity of the training participant entering or leaving the training place, the training participant leaving the training place, and the training participant returning to the training place after leaving may be automatically identified through image analysis.

In step S120, a specific method of identifying a training participant may vary according to embodiments.

In some embodiments, training participants may be identified based on facial recognition results for the collected images. A facial recognition method according to various embodiments of the present disclosure will be described in detail with reference to FIGS. 7 to 11. According to the present embodiment, since the attendance of the training participant is confirmed through facial recognition, the proxy attendance may be blocked at the source.

In some other embodiments, in addition to the facial recognition result, the training participant may be further identified based on at least one of an accessory feature, a dress feature, and a voice recognition result extracted from the collected image. For example, the training participant information DB 17-1 may store appearance characteristic information about clothes and accessories that the training participant frequently wears. Then, the training participant may be identified by comprehensively considering the first similarity obtained as a result of facial recognition and the second similarity between the accessory feature and the clothing feature extracted from the collected image and the pre-stored appearance feature information. For another example, the training participant may be identified by comprehensively considering a first similarity obtained as a result of facial recognition and a second similarity obtained as a result of voiceprint recognition. The voiceprint recognition may be performed in various ways, such as a method of obtaining a similarity between a voice feature of the currently collected training participants and pre-stored voice feature information, and a method using a deep learning-based voiceprint recognition model. According to the present embodiment, since the identity of the training participant is identified by further considering appearance characteristics or voice characteristics other than the face, the accuracy of participant identification and attendance determination may be further improved.

In addition, in some embodiments, when a training participant is not accurately identified from the collected first image (or processed first image), an object corresponding to the training participant on the image is tracked to provide a second image containing the object. Is detected, and the training participant may be identified based on the second image. Through this, the accuracy of identification of training participants and determination of attendance may be further improved.

In step S140, it is determined whether to attend or not based on the identification result. For example, if the identified training participant is a person who has applied for a corresponding training course, it may be determined that they have attended the corresponding training course. For another example, when the identified training participant is a pre-applicant but the identification time has passed the training start time, it may be determined that they are late for the corresponding training course.

In some embodiments, the attendance information obtained as a result of the determination in step S140 may be recorded in the attendance information DB 17-2, and the attendance status information, attendance status information, etc. Or it may be provided as a terminal of acquaintances, education managers).

In addition, in some embodiments, the above-described attendance determination process (ie, steps S100 to S140) may be continuously performed during training. According to the present embodiment, various behaviors that hinder the educational atmosphere, such as unauthorized leaving of an educational place during training, can be prevented, and an overall educational effect can be improved.

So far, the attendance management method according to some embodiments of the present disclosure has been described with reference to FIG. 6. According to the above-described method, since attendance management can be automated using an intelligent image analysis technology such as facial recognition, the convenience of attendance management can be greatly improved.

Hereinafter, a facial recognition method according to various embodiments of the present disclosure will be described with reference to FIGS. 7 to 11. Facial recognition methods to be described later may be utilized in the above-described training participant identification step S120.

First, FIG. 7 is an exemplary diagram for describing a face recognition method according to a first embodiment of the present disclosure.

The first embodiment is designed to improve the accuracy of facial recognition itself regardless of whether or not an accessory is worn, and relates to a method of performing facial recognition using feature points extracted from facial regions in various directions.

More specifically, facial regions corresponding to a plurality of directions from one or more training participant images may be detected, and feature points may be extracted from each facial region. For example, as shown in FIG. 7, in the training participant image, a front face area 35, a left face area 34, a right face area 36, an upper face area 32, and a lower side The directional facial region 38, the upper left facial region 31, the lower left facial region 37, the upper right facial region 33 and the lower right facial region 39 are detected, and each facial region ( 31 to 39) can be extracted. In some embodiments, when the facial area in the first direction is not detected, the facial area image in the first direction may be generated using the previously detected facial area image in the second direction and the head model. In this case, the head model may be selected from among a plurality of head models based on demographic information (e.g. race, age, gender, etc.) of a corresponding training participant.

The feature points extracted from the facial areas in multiple directions are compared with the previously stored feature points of the training participant (i.e., feature points extracted from the facial areas in the multiple directions), and the identity of the training participant can be identified based on the similarity obtained as a result of the comparison. have. Here, the feature points to be compared may be previously stored in the training participant DB (e.g. 17-1 in FIG. 5).

In some embodiments, different weights may be assigned to at least some of the similarities for each direction (e.g. the similarity between the feature points extracted from the first direction facial area and the previously stored first direction feature points). For example, the highest weight may be given to the front direction. For another example, when a corresponding training participant wears an ornament, the weight may be determined based on the degree to which the ornament is included in the facial area in each direction (ie, the degree to which it interferes with facial recognition). That is, the higher the proportion of the ornaments visible, the lower the weight may be given to the similarity in the corresponding direction. In this embodiment, the facial recognition result may be derived based on the final facial similarity, and the final facial similarity may be calculated based on the weight assigned to the similarity for each direction (e.g. sum of weights).

So far, a face recognition method according to the first embodiment of the present disclosure has been described with reference to FIG. 7. According to the above-described method, the accuracy of facial recognition can be greatly improved by utilizing feature points extracted from various directions/angles.

Hereinafter, a facial recognition method according to a second exemplary embodiment of the present disclosure will be described with reference to FIGS. 8 to 11.

The second embodiment is designed to solve a problem in that the accuracy of facial recognition is deteriorated due to accessories worn by training participants. The ornaments may include, for example, hats, masks, glasses, earrings, necklaces, and the like, but may include all kinds of tools that can cause changes in appearance. Hereinafter, the second embodiment will be described.

8 is an exemplary flowchart illustrating a method for recognizing a face according to a second exemplary embodiment of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some steps may be added or deleted as necessary. This will be described with reference to FIG. 8.

In step S122, it is determined whether or not to wear jewelry from the image of the training participant. For example, a facial area of a corresponding training participant may be detected and extracted from the image, and it may be determined whether or not an accessory exists in or near the extracted facial area.

A specific method of determining the presence of jewelry may vary according to embodiments. For example, if a feature corresponding to a specific facial area is not detected in the image, a method of determining that there is an accessory in the area (eg, if a mouth, nose, etc. is not detected, it is determined that a mask exists), an image of a specific accessory A method of directly comparing the image of the training participant with the corresponding training participant, and a method of determining the presence of an accessory based on whether an object having the characteristics of a specific accessory (eg shape, color, etc.) is detected in a specific facial part of the image. I will be able to. However, in addition to this, the presence or absence of accessories may be determined in various ways, so the technical scope of the present disclosure is not limited to the examples listed above.

If it is determined that the training participant is not wearing jewelry, step S124 may be performed. In the opposite case, step S126 may be performed.

In step S124, facial recognition is performed with the original image. Here, the original image refers to an image that has not been subjected to a conversion process (that is, the processing process of step S126) for removing ornaments. Therefore, in order to increase the accuracy of facial recognition, it goes without saying that even if a predetermined image processing is performed, it may be included in the range of the original image.

The facial recognition may be performed by the facial recognition method according to the first embodiment described above, but is not limited thereto, and may be performed in other ways, such as a method using a deep learning-based facial recognition model.

In step S126, the original image is converted into an image from which the ornaments have been removed. This may be understood as improving the accuracy of facial recognition through the removal of jewelry. In this step, image conversion may be performed through an image conversion model based on Generative Adversarial Networks (GAN) (e.g. 40 in FIG. 9). In more detail, as shown in FIG. 9, the first domain image 41 or the second domain image 43 is faked from another domain through the image conversion model 40 performing domain transfer. It can be converted into images 45 and 47. Here, the first domain may be associated with an image wearing a specific accessory (e.g. a mask), and the second domain may be associated with an image not wearing the specific accessory. Hereinafter, in describing the present embodiment, the first domain and the second domain will be continuously used in the aforementioned meaning.

The image conversion model 40 may be implemented based on various types of GANs equipped with a domain conversion (or image conversion) function. For example, the image conversion model 40 may be implemented based on various GANs such as Disco-GAN, cycle-GAN, UNIT (Unsupervised Image-to-Image Translation), and MUNIT (Multimodal Unsupervised Image-to-Image Translation). I can.

Some examples of image transformation model 40 are shown in FIGS. 10 and 11. 10 illustrates that the image conversion model 40 is implemented based on cycle-GAN, and FIG. 11 illustrates that the image conversion model 40 is implemented based on the UNIT. Hereinafter, for convenience of description, the image conversion model 40 is abbreviated as "conversion model", the image conversion model illustrated in FIG. 10 is referred to as "50", and the image conversion model illustrated in FIG. 11 is a reference number. It should be referred to as "60".

Hereinafter, the structure of the transformation model 50 and a learning method will be described with reference to FIG. 10.

As shown in FIG. 10, the transformation model 50 may include a first generator 51, a second generator 52, a first discriminator 53 and a second discriminator 54.

The first generator 51 is a module that converts the image 55-1 of the first domain into a fake image 55-2 of the second domain. In addition, the second discriminator 54 is a module that determines the actual image 56-1 and the fake image 55-2 of the second domain. The first generator 51 and the second discriminator 54 may be complementarily learned through hostile learning (that is, the weights are updated).

In addition, the first generator 51 may be further learned using _{a first consistency loss (L CONST A).} The first consistency error (L _CONST A) is calculated based on the difference between the actual image 55-1 input to the first generator 51 and the fake image 55-3 converted through the second generator 52. Can be. In this case, the fake image 55-3 refers to an image obtained by converting the fake image 55-2 converted through the first generator 51 into an original domain through the second generator 52 again. By learning the first consistency error L _CONST A, the first generator 51 can accurately perform image conversion even if the training image set is not composed of image pairs. The first consistency error (L _CONST A) may be calculated based on, for example, a Euclidian distance, a cosine similarity, and the like, but the technical scope of the present disclosure is not limited to the examples listed above. .

Next, the second generator 52 is a module that converts the image 56-1 of the second domain into a fake image 56-2 of the first domain. In addition, the first discriminator 53 is a module that determines the actual image 55-1 and the fake image 56-2 of the first domain. The second generator 52 and the first discriminator 53 may also be complementarily learned through hostile learning.

Also, the second generator 52 may be further learned using _{the second consistency error L CONST B.} Since the learning process of the second generator 52 is similar to that of the first generator 51, further description will be omitted.

Hereinafter, the structure of the transformation model 60 and a learning method will be described with reference to FIG. 11.

As shown in Fig. 11, the transform model 60 includes a first encoder 61, a first generator 63, and a first discriminator 65 associated with a first domain, and a second encoder associated with a second domain. 62, a second generator 64 and a second discriminator 66 may be included.

The first encoder 61 is a module that encodes the image 68-1 of the first domain into the encoded data 67, and the second encoder 62 converts the second domain image 69-1 into the encoded data 67. It is a module that encodes with ). The encoded data 67 may be understood as data on a specific space potentially shared between the first domain and the second domain image (ie, between different domains).

The first generator 63 is a module that generates a fake image 69-2 of the first domain based on the encoded data 67. Similarly, the second generator 64 is a module that generates a fake image 69-2 of the second domain based on the encoded data 67.

The first discriminator 65 is a module that performs a discrimination operation on a first domain image, and the second discriminator 66 is a module that performs a discrimination operation on a second domain image. Hostile learning may be performed between the first discriminator 65 and the first generator 63, and hostile learning may also be performed between the second discriminator 66 and the second generator 64.

In addition, learning may be performed according to the flow shown in FIG. 11 for image conversion. For example, when encoding data 67 for the image 69-1 of the second domain is input to the first generator 63, the image 69-1 of the second domain is a fake image of the first domain ( 68-2) can be performed. Similarly, when the encoding data 67 for the image 68-1 of the first domain is input to the second generator 64, the image 68-1 of the first domain is a fake image 69 of the second domain. Learning can be performed to transform into -2). For specific learning methods, refer to a document called "UNIT; Unsupervised Image-to-Image Translation". Actual image conversion may also be performed according to the flow shown in FIG. 11.

The description will be continued with reference to FIG. 8 again.

In step S128, facial recognition may be performed with the converted image. As described above, the facial recognition may be performed according to the facial recognition method according to the first embodiment described above, but may be performed in a different manner.

Meanwhile, in some other embodiments of the present disclosure, the identity of the training participant may be identified as an original image wearing jewelry without an image conversion process. For example, identification factors such as facial recognition results (i.e., facial similarity), jewelry characteristics, clothing characteristics, motion recognition results (i.e., motion similarity), and voiceprint recognition results (i.e., voice similarity) are comprehensively integrated. In consideration of this, the identity of the training participant may be identified. In this case, the same weight may be assigned to each identification element, or differential weights may be assigned to each other. For example, since the accuracy of the facial recognition result may be slightly degraded due to jewelry, a lower weight may be given than other identification factors.

So far, a face recognition method according to a second embodiment of the present disclosure has been described with reference to FIGS. 8 to 11. According to the above-described method, face recognition can be performed with high accuracy even when the training participant wears jewelry by using the GAN-based image conversion model. Accordingly, the accuracy of attendance management may also be improved.

Hereinafter, a method of measuring education concentration according to some embodiments of the present disclosure will be described with reference to FIG. 12.

12 is an exemplary flowchart illustrating a method of measuring education concentration according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some steps may be added or deleted as necessary. The following education concentration measurement method may be performed by the concentration measurement unit 13-2 and the action unit 15.

As illustrated in FIG. 12, the method of measuring the education concentration may begin in step S200 of collecting images of participants in education. For example, images photographed during education may be collected from one or more photographing devices (e.g. 1 in FIG. 1) installed in an education place. In addition, in this step, various information that can be monitored, such as voice data and Internet access information through an AP (Access Point), may be further collected.

In step S220, the concentration of the training participant may be measured by analyzing the collected image based on a predetermined interaction index. Here, the interaction indicator is, for example, the degree of departure from the training place, the direction of the gaze of the training participant, the movement of the gaze, the degree of use of the personal terminal of the training participant, the number of times the instructor raises his hand, the number of answers to the inquiry, It may include the accuracy of the answer, the posture, emotion, and facial expression of the training participant. However, in addition to the various interaction indicators may be further included, the technical scope of the present disclosure is not limited to the examples listed above. Specific examples of measuring concentration in this step are as follows.

In some examples, concentration may be measured based on the number of times and/or the time of departure from the educational venue. That is, the greater the number of times of departure or the longer the departure time, the lower the concentration may be measured. Of course, in the opposite case, concentration can be measured high.

As another example, as an education participant's gaze direction is toward the instructor and the gaze retention time increases, the degree of concentration may be measured higher. Of course, in the opposite case, concentration can be measured low.

As another example, the greater the number of times or longer the training participant uses the personal terminal, the lower the degree of concentration may be measured. Of course, in the opposite case, concentration can be measured high.

As another example, as the number of times the training participant raises his or her hand or the number of answers increases, the degree of concentration may be measured higher. Of course, in the opposite case, concentration can be measured low.

As another example, if the training participant is not staring straight ahead or is in a specific posture (e.g. tight posture), the degree of concentration may be measured to be low. Of course, in the opposite case, concentration can be measured high.

In some examples, speech recognition may be performed on the voices of training participants collected during training, and the degree of concentration may be measured based on a result of the voice recognition and a degree of relevance to the educational topic. For example, if the degree of relevance is high, the degree of concentration may be regarded as an accurate answer to a query, and the degree of concentration may be measured as high.

In addition, in some examples, the concentration of the specific training participant may be measured based on a difference between the emotions or facial expressions of the plurality of training participants recognized in the collected images and the emotions or facial expressions of the specific training participant. For example, when a number of training participants are insensitive or expressionless, and a specific training participant has a pleasant emotion or a smiling expression, the concentration of the specific training participant may be measured low.

Further, in some examples, the degree of concentration of the training participant may be measured based on various combinations of the aforementioned interaction indicators. For example, by taking the weight of each interaction index into account and synthesizing the concentration measured according to each interaction, the final concentration of the training participant can be determined. In this case, the weight of each interaction index may be determined or adjusted based on the result of statistical analysis using real data. Specifically, it is based on the actual feedback of the training participants (eg survey results on education satisfaction, etc.), educational outcomes (eg test scores of training participants, etc.), and the instructor's profile (eg achievements, teaching period, number of times, etc.). The effectiveness of education can be measured quantitatively. In addition, a relationship between the measured educational effect and each interaction index may be derived through statistical analysis such as regression analysis, and the weight of each interaction index may be determined or adjusted according to the derived relationship. For example, when a specific interaction index has a great influence on improving the educational effect (ie, a high degree of relevance), the weight of the specific interaction index may be adjusted to a higher value.

On the other hand, in some embodiments, the concentration measurement result of step S220 may be recorded in the training participant information DB 17-1 or other information DB 17-4, and the education official (eg, supervisor, It may be provided as a terminal of the training participant or acquaintance, training manager). According to the present embodiment, the overall educational effect may be improved by preventing a decrease in the concentration of training participants.

In addition, in some embodiments, the above-described concentration measurement process (ie, steps S200 and S220) may be continuously performed during training. According to the present embodiment, the overall educational effect may be improved by preventing a decrease in the concentration of training participants.

So far, a method of measuring education concentration according to some embodiments of the present disclosure has been described with reference to FIG. 12. According to the above-described method, the degree of concentration of the training participant may be measured by using the intelligent image analysis technology, and various education personnel may be provided with the measured concentration information. Accordingly, the overall educational effect, including the degree of participation of training participants, can be improved.

Hereinafter, a method of detecting an abnormal situation according to some embodiments of the present disclosure will be described with reference to FIGS. 13 to 16. The following abnormal situation detection method may be performed by the abnormal situation detection unit 13-3 and the action unit 15.

13 is an exemplary flowchart illustrating a method of detecting an abnormal situation according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some steps may be added or deleted as necessary.

As illustrated in FIG. 13, the method for detecting an abnormal situation may begin in step S300 of collecting an image of an educational place. The image may include a plurality of frame images.

In step S320, an abnormal situation may be detected by analyzing the collected image. In this step, to detect an abnormal situation, an abnormal situation detection model based on a convolutional neural network may be used. In this regard, it will be described later with reference to FIGS. 14 to 16.

However, according to some other embodiments of the present disclosure, an abnormal situation may be detected using various intelligent image analysis techniques in addition to the convolutional neural network. For example, a violent situation may be detected in real time by detecting an object from a plurality of frame images and analyzing the degree of movement of the detected object. For example, when motion of more than a reference value is detected between two or more objects adjacent to each other, it may be determined that a violent situation has occurred.

In step S340, an appropriate action may be performed according to the detected abnormal situation. For example, measures such as providing a notification to an education official or requesting an emergency rescue from an emergency response organization may be performed. For specific measures, refer to the description of the action unit 15 described above.

Hereinafter, a detailed process of the abnormal situation detection step S320 described above will be described with reference to FIGS. 14 to 16. 14 and 15 are for explaining an embodiment in which image data to be analyzed is lightened in order to detect an abnormal situation in real time, and FIG. 16 is for explaining an embodiment designed to improve the accuracy of detection of an abnormal situation.

14 is an exemplary flowchart illustrating a detailed process of an abnormal situation detection step S320 according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some steps may be added or deleted as necessary.

As shown in FIG. 14, in step S322, a plurality of luminance images may be extracted from each of the plurality of frame images by excluding a color difference component. For example, as illustrated in FIG. 15, when a specific image in an RGB format is converted into a YUV format and color difference components (U, V) are removed, a luminance image may be extracted from the specific image. This step may be understood as a step of performing weight reduction by removing data corresponding to a color difference component.

In some embodiments, step S322 may be performed only when the abnormal situation to be detected is a violent situation. This is because the violent situation will be detected based on features such as movement, and therefore abnormal situations will be detected with high accuracy even if the color difference component is excluded.

In step S324, a three-dimensional image may be generated by accumulating a plurality of extracted luminance images. For example, as shown in FIG. 15, a predetermined number of luminance images 71 may be stacked to generate a 3D image 73.

In step S326, downsampling is performed on at least a part of the generated three-dimensional image, and as a result, an input image may be configured. The input image may mean an image input as an abnormal situation detection model. For example, as shown in FIG. 15, the input image 75 may be configured by down-sampling the 3D image 73.

The downsampling may be performed by a method of extracting part of a plurality of luminance images included in a three-dimensional image (eg 73) (eg, in the case of detection of a violent situation, extracting only the circular image with the number of motion vectors equal to or greater than the reference value). There may be, but it can be done in a different way.

In step S328, the abnormal situation may be detected by inputting the input image to the abnormal situation detection model. For example, as shown in FIG. 15, when the input image 75 is input to the abnormal situation detection model, feature maps 77, 79, etc. from the input image 75 through convolution and pooling operations, etc.; map) may be extracted, and the abnormal situation detection model may output the abnormal situation detection result based on the feature maps 77, 79, etc. For example, if the abnormal situation detection model is a binary classification model that detects fire (e.g. a model classified into two classes of fire or normal), the abnormal situation detection model may output confidence scores of two classes. Then, whether or not a fire occurs may be determined based on whether the confidence score of the fire class is equal to or greater than a reference value.

Meanwhile, according to some embodiments of the present disclosure, the abnormal condition detection model may generate a new feature map by synthesizing a plurality of feature maps, and detect the abnormal condition using the generated feature map. By doing so, the accuracy of detecting an abnormal situation can be improved. Hereinafter, the present embodiment will be further described with reference to FIG. 16.

As shown in FIG. 16, the abnormal situation detection model extracts a feature map containing high-dimensional features from the input image 75 through a plurality of convolution operations. For example, a convolution operation is again performed on the first feature map 77, the second feature map 79 is extracted, and the third feature map 81 and the fourth feature map 83 are similar in a similar manner. This is extracted, and the fourth feature map 83 includes higher-dimensional features than the first feature map 77.

However, low-dimensional features can also play an important role in detecting abnormal situations, and the abnormal situation detection model synthesizes a plurality of feature maps (eg 79, 83) to generate a fifth feature map 85 containing various features. And, based on the fifth feature map 85, it may be determined whether or not an abnormal situation has occurred. For example, the abnormal situation detection model may generate the fifth feature map 85 by synthesizing the feature values of the second feature map 79 and the feature values of the fourth feature map 83 associated with the same frame image. . That is, in the synthesis process, the feature values of the second feature map 79 associated with the first frame image and the feature values of the fourth feature map 83 are synthesized, and the second feature map 79 associated with the second frame image. ) And the feature values of the fourth feature map 83 may be combined. Through the above synthesis process, since low-dimensional features and high-dimensional features are mixed or important features are amplified, the accuracy of detecting an abnormal situation can be further improved.

So far, a method of detecting an abnormal situation according to some embodiments of the present disclosure has been described with reference to FIGS. 13 to 16. According to the above-described method, various abnormal situations that may occur during education, such as violence and fire, may be detected in real time by using intelligent image analysis technology, and appropriate measures may be automatically taken according to the detected abnormal situation. Accordingly, convenience of safety management can be improved, and a safe educational environment can be created. In addition, since the image data analyzed for detecting an abnormal situation through downsampling and excluding the color difference component can be reduced in weight, real-time capability of detecting an abnormal situation can be secured. In addition, abnormal situation detection may be performed based on various features through feature map merging or synthesis, and accordingly, the accuracy of the abnormal situation detection may be improved.

Hereinafter, a privacy protection method according to some embodiments of the present disclosure may be described with reference to FIG. 17. The following privacy protection method may be performed by the storage unit 17, but may be performed by other modules. In addition, the storage unit 17 may perform the privacy protection method in conjunction with other modules.

17 is an exemplary diagram for describing a privacy protection method according to some embodiments of the present disclosure.

As shown in FIG. 17, the image 91 collected from the photographing device (e.g. 1 in FIG. 1) may include various aspects of the training participant (95, etc.). If such an image 91 is leaked, privacy may be infringed, and therefore, in some embodiments of the present disclosure, an appropriate processing process may be performed before storing the image 91.

The processing may include concealing at least a portion of the facial area (e.g. 93-1). For example, when a facial area (e.g. 93-1) is detected in the collected image 91, masking or mosaic processing may be performed on all or part of the facial area (e.g. 93-1). An example of the concealed image 94 is shown at the bottom of FIG. 17.

In addition, the processing may further include a process of injecting an identification code for the training participant (e.g. 95) into the concealed area (e.g. 93-2). For example, an appropriate identification code may be inserted in the concealed area (e.g. 93-2) to identify the training participant (e.g. 95). Of course, the identification code may be inserted in another location or stored as separate metadata. In addition, the form of the identification code may be any form. The reason for inserting the identification code is that when analyzing the stored image 94 later, the corresponding training participant (e.g. 95) needs to be identified. In some embodiments, the identification code may be encrypted and stored.

So far, a privacy protection method according to some embodiments of the present disclosure has been described with reference to FIG. 17. According to the above-described method, since concealment processing is performed on the image including the face of the training participant, the problem of invading the privacy of the training participant due to data leakage can be prevented in advance.

Hereinafter, an exemplary computing device 100 capable of implementing devices (e.g. the integrated management server 10 of FIG. 1) according to various embodiments of the present disclosure will be described with reference to FIG. 18.

18 is an exemplary hardware configuration diagram illustrating the computing device 100.

As shown in FIG. 18, the computing device 100 is a memory for loading a computer program executed by one or more processors 110, a bus 150, a communication interface 170, and the processor 110 ( 130) and a storage 190 for storing the computer program 191. However, only the components related to the embodiment of the present disclosure are shown in FIG. 18. Accordingly, those of ordinary skill in the art to which the present disclosure belongs may recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 18. That is, the computing device 100 may further include various components in addition to the components illustrated in FIG. 18.

The processor 110 controls the overall operation of each component of the computing device 100. The processor 110 includes at least one of a CPU (Central Processing Unit), a Micro Processor Unit (MPU), a Micro Controller Unit (MCU), a Graphic Processing Unit (GPU), or any type of processor well known in the technical field of the present disclosure. It can be configured to include. In addition, the processor 110 may perform an operation on at least one application or program for executing methods/operations according to various embodiments of the present disclosure. The computing device 100 may include one or more processors.

The memory 130 stores various types of data, commands, and/or information. The memory 130 may load one or more programs 191 from the storage 190 to execute methods/operations according to various embodiments of the present disclosure. For example, when the computer program 191 is loaded in the memory 130, a module as shown in FIG. 5 may be implemented on the memory 130. The memory 130 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

The bus 150 provides a communication function between components of the computing device 100. The bus 150 may be implemented as various types of buses such as an address bus, a data bus, and a control bus.

The communication interface 170 supports wired/wireless Internet communication of the computing device 100. In addition, the communication interface 170 may support various communication methods other than Internet communication. To this end, the communication interface 170 may be configured to include a communication module well known in the art.

The storage 190 may non-temporarily store the one or more programs 191. The storage 190 is a nonvolatile memory such as a ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, etc., a hard disk, a removable disk, or well in the technical field to which the present disclosure belongs. It may be configured to include any known computer-readable recording medium.

The computer program 191 may include one or more instructions that when loaded into the memory 130 cause the processor 110 to perform methods/operations according to various embodiments of the present disclosure. That is, the processor 110 may perform methods/operations according to various embodiments of the present disclosure by executing the one or more instructions.

For example, the computer program 191 may include an operation of collecting an image of an education place from one or more photographing devices, an operation of identifying an education participant included in the collected image through facial recognition, and the identification result. It may include one or more instructions for performing an operation to determine whether the training participant attends or not. In this case, the integrated education management server 10 according to some embodiments of the present disclosure may be implemented through the computing device 100.

So far, the technical idea and various embodiments of the present disclosure have been described with reference to FIGS. 1 to 18. For convenience of understanding, the technical idea of the present disclosure has been described as an example in which the technical idea of the present disclosure is applied to integrated management of an educational place and environment, but the technical idea of the present disclosure may be applied to any environment in which intelligent image analysis can be performed.

According to the technical idea and various embodiments of the present disclosure described with reference to FIGS. 1 to 18 so far, the following effects may be further achieved.

First, the conventional attendance management was performed in such a way that the instructor directly calls the names of participants in the training to confirm attendance. In this method, part of the lecture time was used for attendance confirmation, and accurate attendance confirmation could not be expected for substitute attendance. In addition, the recently proposed beacon-based attendance management system is a method in which training participants directly check attendance using a terminal near the beacon, so it is possible to be a proxy attendance and check irregular attendance outside the classroom, and beacon is searched. There were areas that were not efficient, such as failures or connection errors due to network problems. On the other hand, the attendance confirmation method according to the exemplary embodiment of the present disclosure can completely eliminate the possibility of proxy attendance and irregular attendance by accurately recognizing educational participants through intelligent image analysis. In addition, since attendance is managed through software, the convenience of attendance confirmation can be improved because training participants do not need to use a separate terminal.

In addition, in the conventional case, class information and alarms were posted on a bulletin board through the department office of each school, or could be checked through the university information system homepage of each school. This method has limitations in checking class information and alarms in real time for reasons such as the need for students to visit the department office or use a computer. On the other hand, through the integrated education management server 10 according to an embodiment of the present disclosure and an application linked thereto, changes in class schedules, such as information on classes, or cancellation and reinforcement, may be shared in real time between the instructor and the training participant. .

In addition, in the conventional case, in order to check the schedule, task management, and progress scale for each lecture and project, professors and training participants had to check face-to-face or access a management system (e.g. university information system) to check. For example, students had to go to the website of the university information system and check in order to see what kind of progress their lectures are currently taking. Due to the cumbersomeness of such an approach process, conventionally, there has been a problem that most of the students do not check the progress of the lecture. On the other hand, the integrated education management server 10 and an application linked thereto according to an embodiment of the present disclosure may visually show a progress measure of a lecture currently being taken. Accordingly, the training participant can easily check whether the lecture is a good lecture by comparing the lecture plan of the instructor with the current progress scale. In addition, the instructor registers information about the assignment in the application, so that training participants can easily check the assignment.

In addition, in the conventional case, a place for communication through a community, such as meetings and information by topic, was formed as a school club, an Internet cafe, or a single application. In addition, there is a problem in that students do not access the cafe except for special notices due to the hassle of accessing the Internet cafe. On the other hand, the integrated education management server 10 according to an embodiment of the present disclosure and an application linked thereto may provide excellent accessibility. That is, students can easily access the community through the application, and new clubs and places of communication can be easily and smoothly formed. In addition, information about a club or a place of communication can be shared among students without meeting in person through the file attachment function of the application.

The technical idea of the present disclosure described with reference to FIGS. 1 to 18 so far may be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium is, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). I can. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed in the other computing device, thereby being used in the other computing device.

In the above, even if all the constituent elements constituting the embodiments of the present disclosure have been described as being combined into one or operating in combination, the technical idea of the present disclosure is not necessarily limited to these embodiments. That is, as long as it is within the scope of the object of the present disclosure, one or more of the components may be selectively combined and operated.

Although the operations are illustrated in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order shown or in a sequential order, or all illustrated operations must be executed to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the above-described embodiments should not be understood as necessitating such separation, and the program components and systems described are generally integrated together into a single software product or may be packaged into multiple software products. It should be understood that there is.

Although the embodiments of the present disclosure have been described with reference to the accompanying drawings above, those of ordinary skill in the art to which the present disclosure pertains may implement the present disclosure in other specific forms without changing the technical idea or essential features. I can understand that there is. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. The protection scope of the present disclosure should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the technical ideas defined by the present disclosure.

Claims (18)

A collection unit that collects an image of an educational place from one or more photographing devices; And
It includes an attendance management unit that identifies the training participant included in the collected image through facial recognition, and determines whether the training participant attends or not based on the identification result,
The attendance management unit extracts a feature point from the facial area in multiple directions from the collected image, and identifies the identity of the training participant based on the similarity obtained by comparing the extracted feature point with the previously stored feature point of the training participant, For the facial regions in the plurality of directions, a different weight is given to the similarity for each direction by determining the degree of interference with facial recognition, and a final facial similarity is calculated based on the weight given to the similarity for each direction,
The feature points are the feature points of the front face area, the feature points of the left face area, the feature points of the right face area, the feature points of the upper face area, the feature points of the lower face area, the feature points of the upper left face area, the lower left face Including the feature points of the region, the feature points of the facial region in the upper right direction and the feature points of the facial region in the lower right direction
Integrated education management system. The method of claim 1,
At least one of the one or more photographing devices includes a fisheye lens,
Further comprising a processing unit for performing a dewarping (dewarping) processing on the image photographed by the fisheye lens,
Integrated education management system. The method of claim 1,
At least one of the one or more photographing devices is to photograph the entrance and exit of the educational place,
Integrated education management system. The method of claim 1,
The attendance management unit,
In addition to the facial recognition result, identifying the training participant further based on the accessory feature and the dress feature extracted from the collected image,
Integrated education management system. delete delete The method of claim 1,
The attendance management unit,
By analyzing the collected image, it is determined whether the training participant wears jewelry,
In response to the determination that the ornaments were worn, the collected image is converted into an image from which the ornaments have been removed through an image conversion model based on Generative Adversarial Networks (GAN),
Performing the facial recognition on the converted image,
Integrated education management system. The method of claim 7,
The image conversion model includes a first generator that converts an image of a first domain to a fake image of a second domain, and a second generator that converts the image of the second domain to a fake image of the first domain. A generator, a first discriminator for discriminating an image of the first domain and a fake image of the first domain, and a second discriminator for discriminating an image of the second domain and a fake image of the second domain, ,
The first domain is associated with an image including the ornament,
The second domain is associated with an image that does not include the ornament,
The attendance management unit,
The first training image is converted into a second training image through the first generator, the second training image is converted into a third training image through the second generator, and the first training image and the third training image Update the first generator based on the difference between,
Adversarial learning is performed between the second generator and the first discriminator,
Learning the image transformation model by performing hostile learning between the first generator and the second discriminator,
Integrated education management system. The method of claim 7,
The image conversion model uses a first encoder for encoding an image of a first domain, a second encoder for encoding an image of a second domain, and encoding data output from the first encoder or the second encoder. A first generator for generating a fake image of the first domain and a second generator for generating a fake image of the second domain by using the encoded data,
The first domain is associated with an image including the ornament,
The second domain is associated with an image that does not include the ornament,
The attendance management unit,
Obtaining first encoding data for the collected image through the first encoder,
Generating the converted image by inputting the first encoded data to the second generator,
Integrated education management system. The method of claim 1,
Further comprising a concentration measurement unit for measuring the concentration of the training participant by analyzing the collected image based on at least one of a degree of departure from the training place, a gaze movement, and a degree of use of the personal terminal of the training participant,
Integrated education management system. The method of claim 10,
The concentration measurement unit,
Performing speech recognition on the voice of the training participant collected during training,
Measuring the degree of concentration further based on a degree of relevance between the result of the speech recognition and the subject of education,
Integrated education management system. The method of claim 10,
The concentration measurement unit,
Measuring the concentration of the specific training participant further based on a difference between the emotions or facial expressions of the plurality of training participants recognized in the collected image and the emotions or facial expressions of the specific training participant,
Integrated education management system. The method of claim 1,
An abnormal situation detector configured to analyze the collected image to detect an abnormal situation occurring in the educational place; And
Further comprising an action unit for performing an action corresponding to the detected abnormal situation,
Integrated education management system. The method of claim 13,
The collected image includes a plurality of frame images,
The abnormal situation detection unit,
Extracting a plurality of luminance images by excluding a color difference component from the plurality of frame images,
Accumulating the extracted plurality of luminance images to generate a three-dimensional image,
Downsampling at least a part of the generated three-dimensional image to construct an input image,
Inputting the configured input image to an abnormal situation detection model based on a convolutional neural network to detect the abnormal situation,
Integrated education management system. The method of claim 13,
The collected image includes a plurality of frame images,
The abnormal situation detection unit,
Accumulating the plurality of frame images to form an input image,
The configured input image is input to an abnormal situation detection model based on a convolutional neural network to detect the abnormal situation,
The abnormal situation detection model,
A first convolution layer extracting a first feature map through a first convolution operation and a second convolution layer extracting a second feature map through a second convolution operation,
A third feature map is generated by aggregating feature values of the first feature map and the feature values of the second feature map associated with the same frame image,
Detecting the abnormal situation based on the third feature map,
Integrated education management system. The method of claim 1,
Further comprising a storage unit for processing and storing the collected image,
The processing treatment,
Detecting a facial area in the collected image,
At least a part of the detected facial area is concealed,
Including the process of injecting the identification code of the training participant in the concealed facial area,
Integrated education management system. delete delete

Images