KR102090109B1 - Learning and inference apparatus and method - Google Patents

Abstract

Disclosed is a learning and reasoning apparatus and method. The learning and inference apparatus according to an embodiment of the present invention includes an input layer, a hidden layer, and a main network processing unit including an output layer outputting a final output value, and an error based on output values of the input layer and the hidden layer, respectively The gradient value is calculated, and the local network modules of the next stage are updated based on the calculated training loss value.

Description

LEARNING AND INFERENCE APPARATUS AND METHOD}

The present invention relates to model parallelization deep learning, and more particularly, to a learning and inference apparatus and method for simultaneously performing feed forward and update by adding a local network module.

In recent years, deep learning has advanced significantly and has been successfully applied in many fields. There is an important mechanism behind the success of deep neural networks to extract useful information through a hierarchical structure. More and more complex deep neural network structures are being developed to solve challenging real-world problems. However, complex network structures require a significant amount of computation in learning.

A common way to alleviate this difficulty is data parallelization. In other words, a plurality of identical models are trained independently using different learning date sets.

Another method called model-parallelism was studied. The backpropagation learning paradigm is dominant in neural network learning, and basically works in a continuous fashion. Each layer uses update information from the upper layer and is updated one by one in reverse order. On the other hand, in the model parallel learning approach, the network model can be divided into parts that can be learned simultaneously and independently in separate computational units.

Existing methods for model parallel learning can be classified into two approaches.

The first is layer-wise sequential training, and the sequential learning method for each layer is that multiple groups of operation operations in each layer are identified and separated so that only operations in the same group have a dependency. These methods can be regarded as a scheduling technique by efficient implementation of backpropagation learning, not a learning algorithm that is distinguished by itself.

Second, the dependency of one layer on the preceding layer can be removed so that the other layers can be separated and learned separately from other computational units. Back propagation algorithms are not suitable for this approach. Therefore, a method of auxiliary coordinates (MAC) has been proposed to make this possible. This replaces the existing least-squares loss minimization problem with an equal-constrained optimization problem by introducing auxiliary variables for each data and each hidden unit.

Then, solving this problem is formulated by repeatedly solving the sub-problem. A similar method (a method using the alternating direction method of multiplipers (ADMM)) is proposed. It adopts an equal-constrained optimization, but has no other auxiliary variables, so the subproblem has a closed solution. However, these methods are not extensible for deep learning architectures such as convolutional neural networks. The decoupled neural interface (DNI) synthesizes synthetic gradients by directly using additional small neural networks for weight training of the estimation layer. As long as the composite gradient is close to the actual back propagated gradient, each layer does not have to wait for errors in the output layer to propagate back through the preceding layer. In other words, this allows each layer to be learned independently. However, this method causes performance degradation compared to back propagation. In addition, the idea of using additional models to support the main model hierarchy is used in Sobolev training. Here, additional networks are trained by approximating the output of the main model instead of the error gradient. For this reason, the Sobolev training method is not suitable for parallel learning.

Korean Patent Publication No. 10-2001-0047163, "Multilayer Perceptron Neural Network Learning Method" (2001.06.15)

The present invention is to provide a learning and reasoning apparatus and method for performing an update without a complete feedforward and backward propagation process in model parallelization deep learning.

In addition, the present invention is to provide a learning and reasoning apparatus and method for unlocking the hierarchical dependency of backpropagation learning.

In addition, the present invention is to provide a learning and reasoning apparatus and method for unlocking the dependency of a local network module on an upper layer.

In addition, the present invention is to provide a learning and reasoning apparatus and method for outputting a classification value without a complete feedforward process.

The learning and inference apparatus according to an embodiment of the present invention for achieving the above object includes a main network processing unit including an input layer, a hidden layer, and an output layer outputting a final output value, and the input layer and the hidden layer The error slope value is calculated based on each output value, and local network modules are updated based on the training loss value calculated by the local network module in the next stage.

The local network modules calculate the approximate value approximated to the final output value using the output values of the input layer and the hidden layer, and calculate the training loss value by the loss function using the approximate value, and the training The error slope value can be calculated by the loss value.

The loss function may be a min-absolute error function, a min-square error function, or a cross-entropy function.

The input layer and the hidden layer may be updated by the error slope value, respectively.

The input layer and the hidden layer may be updated by a gradient descent method.

The local network module of the input layer calculates a local training loss value by a loss function using the training loss value calculated by the local network module of the hidden layer, and calculates a local error slope value by the local training loss value. And can be updated by the local error slope value.

The loss function may be a min-absolute error function, a min-square error function, or a cross-entropy function.

The network module of the input layer may be updated by a gradient descent method.

A local network module of the input layer and the input layer is configured as a first sub-model unit, and a local network module of the input layer, the hidden layer and the hidden layer is configured as a second sub-model unit, and the input layer and the hidden The layer and the output layer are configured as a main model unit, and the reliability of the sub-models is sequentially determined to output the approximation value as an inference value when the reliability has an approximation value equal to or greater than the threshold, and the reliability threshold When there is no abnormal value above, a reasoning unit that outputs the final output value as an inference value may be further included.

In the learning and inference method according to an embodiment of the present invention, the local network modules corresponding to each of the input layer and the concealment layer calculate an error slope value based on the output value of each layer, and the input layer and the concealment Each layer is updated by the error slope value, and the local network module of the input layer is updated based on the training loss value calculated by the local network module of the hidden layer.

The calculating of the error slope value may include: calculating, by the local network modules, an approximate value approximating the final output value using the output values of the input layer and the hidden layer, and the local network modules performing the calculation. The method may include calculating a training loss value by a loss function using an approximate value, and calculating the error slope value by the local network modules using the training loss value.

The step of updating the local network module of the input layer may include calculating, by the local network module of the input layer, a local training loss value by a loss function using the training loss value calculated by the network module of the hidden layer. The local network module of the input layer may include calculating a local error slope value based on the local training loss value, and the local network module of the input layer being updated by the local error slope value.

A local network module of the input layer and the input layer is configured as a first sub-model unit, and a local network module of the input layer, the hidden layer and the hidden layer is configured as a second sub-model unit, and the input layer and the hidden Comprising a layer and the output layer as a main model unit, the inference unit sequentially determines the reliability of the approximate values of the sub-models, and the inference unit infers the approximate value when the reliability has an approximation value equal to or greater than a threshold value. And outputting the final output value as an inference value when there is no approximate value above the threshold.

The learning and reasoning apparatus and method according to an embodiment of the present invention can perform updates without a complete feedforward and backward propagation process.

In addition, the learning and inference apparatus and method according to an embodiment of the present invention can unlock the hierarchical dependency of backpropagation learning.

In addition, the learning and inference apparatus and method according to an embodiment of the present invention can unlock the dependency of the local network module on the upper layer.

In addition, the learning and inference apparatus according to an embodiment of the present invention, and the method can output classification values without a complete feedforward process.

1 is a block diagram of a learning and reasoning apparatus according to an embodiment of the present invention.
2 is a view for explaining a learning operation of the learning and reasoning apparatus according to an embodiment of the present invention.
3 is a view for explaining the reasoning operation of the learning and reasoning apparatus according to an embodiment of the present invention.
4 is a flowchart illustrating a learning and reasoning method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and the contents described in the accompanying drawings, but the present invention is not limited or limited by the embodiments.

The terminology used herein is for describing the embodiments and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, "comprises" and / or "comprising" refers to the components, steps, operations and / or elements mentioned above, the presence of one or more other components, steps, operations and / or elements. Or do not exclude additions.

As used herein, "example", "example", "side", "example", etc. should be construed as any aspect or design described being better or more advantageous than other aspects or designs. It is not done.

In addition, the term 'or' refers to the inclusive 'inclusive or' rather than the exclusive 'exclusive or'. That is, unless stated otherwise or unclear from the context, the expression 'x uses a or b' means any of the natural inclusive permutations.

Also, the singular expression (“a” or “an”) used in the specification and claims generally means “one or more” unless the context clearly indicates otherwise that it is related to the singular form. It should be interpreted as.

Further, terms such as first and second used in the present specification and claims may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings commonly understood by those skilled in the art to which the present invention pertains. In addition, terms defined in the commonly used dictionary are not ideally or excessively interpreted unless specifically defined.

On the other hand, in the description of the present invention, when it is determined that a detailed description of related known functions or configurations may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms used in the present specification (terminology) are terms used to properly represent an embodiment of the present invention, which may vary according to a user, an operator's intention, or a custom in a field to which the present invention pertains. Therefore, definitions of these terms should be made based on the contents throughout the present specification.

1 is a block diagram of a learning and reasoning apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the learning and inference device 100 includes an input layer 110, a hidden layer 120, an output layer 130, and local network modules 140 and 150.

In order to artificially implement a biological neural network, similar structures and computation methods have been proposed in various forms, and the methodology of constructing the artificial neural network is called a neural network model.

In the neural network model, artificial neurons are connected by directional connecting lines to form a network, and each neuron has its own output value and affects adjacent neurons by passing the value through the connecting line.

Each connection line between a neuron and a neuron has its own property value and controls the strength of the signal it transmits. The property value of the connecting line is a weight value indicating the connection strength of the connecting line connecting the neuron and the neuron.

The input layer 110 may be composed of input neurons that receive input values.

The output layer 130 is composed of output neurons that are delivered to the outside as a result of a neural network.

The hidden layer 120 exists between the input layer and the output layer and may be composed of a plurality of hidden neurons.

Only between neurons in adjacent layers, connecting lines are connected from the input layer to the output layer.

The input layer 110, the hidden layer 120, and the output layer 130 may constitute a main network processing unit.

The input layer 110 receives an input value and outputs a first output value h _i .

At this time, the input value may include a learning value and a target value.

The learning value refers to a data value to be learned by the learning and inference device 100, and the target value refers to a correct answer value for the corresponding learning value.

For example, in the case of the learning and inference device 100 that classifies a puppy picture and a cat picture, the learning value means a picture data value to be learned, and the target value means a dog data value when the corresponding learning value is a dog. You can.

The input layer 110 receives an input value and outputs a first output value h _i calculated by neurons constituting the input layer 110.

The input layer 110 may be composed of a plurality of connected neurons.

At this time, neurons can multiply the learning value by the weight value, and calculate the output value by the active function by adding the bias value. The output values of the calculated neurons may be input to adjacent neurons and output the first output value through the same process.

 The present invention can be updated so that the final output value output by the weight values constituting the input layer 110, the hidden layer 120, and the output layer 130 has a target value. This is called learning of the learning and reasoning apparatus 100.

The hidden layer 120 receives a first output value (h _i ) and outputs a second output value (h _{i + 1} ).

The hidden layer 120 may be composed of a plurality of neurons.

The hidden layer 120 receives the first output value h _i and outputs the second output value h _{i + 1} through the same neuron operation process as the input layer 110.

The output layer 130 receives a second output value (h _{i + 1} ) and outputs a final output value (h _N ).

The output layer 130 may be composed of a plurality of neurons.

The output layer 120 receives the first output value h _i and outputs a final output value h _N through a process of calculating the same neurons as the hidden layer 120.

The local network modules 140 and 150 include a first local network module 140 corresponding to the input layer 110 and a second local network module 150 corresponding to the hidden layer 120.

)을 계산한다.The first local network module 140 receives the first output value (h _i ) as shown in Equation 1 below and approximates the final output value (h _N ) (

).

[Equation 1]

)을 이용하여 손실 함수(l)에 의해 입력 레이어(110)에 대한 제1 트레이닝 로스 값(L_i)을 계산한다.The first local network module 140 may have a first approximation value (

) To calculate the first training loss value (L _i ) for the input layer 110 by the loss function (l).

[Equation 2]

)을 이용하였으므로 물결무늬 등호로 표시하였다.Here, y is a learning target value, and a first approximation value approximated to the final output value h _N (

), So it is indicated by the wave pattern equal sign.

At this time, the loss function (l) may be a min-absolute error function, a cross-entropy function, or a min-squared error function.

)을 계산한다.The first local network module 140 differentiates the first training loss value Li into the first output value hi as shown in Equation 3 below, so that the first error gradient value for the input layer 110 (

).

[Equation 3]

Here, since the first training loss value L _i is an approximation of the training loss value of the input layer 110, it is represented by a wave pattern equal sign.

)을 입력 레이어(110)로 출력한다.The first local network module 140 calculates the first error gradient value (

) Is output to the input layer 110.

)에 기초하여 입력 레이어(110)의 가중치 값(

)을 업데이트한다.The input layer 110 has a first error gradient value (

Weight value of the input layer 110 based on)

).

[Equation 4]

는 학습율(learning rate)이다.here,

Is the learning rate.

)을 업데이트 하는 것을 수식으로 표현하였으나, 이외 다른 방식을 사용하여 가중치 값(

)을 업데이트 할 수 있다.Equation 4 is the weight value by the gradient-descent rule (

) Is expressed as a formula, but the weight value (

) Can be updated.

Therefore, the update of the input layer 110 does not have to wait for the first output value h _i to propagate to the output layer 130, and the error gradient value is reverse propagated.

The weight value of the first local network module 140 is updated by the second training loss value calculated by the second local network module 150.

The second local network module 150 calculates the second approximation value using the second output value h _{i + 1 that} is the output value of the hidden layer 120 in the same process as the first local network module 140.

The second local network module 150 calculates the second training loss value in the same process as the first local network module 140 based on the second approximation value.

The second local network module 150 outputs the second training loss value to the first local network module 150.

The first local network module 140 receives a second training loss value from the second local network module 150.

The first local network module 140 calculates the local training loss value L _mi using the second training loss value L _{i + 1} as shown in Equation 5 below.

[Equation 5]

Here, L _i is a first training loss value, and l is a loss function.

The loss function (l) may be a mean-absolute error function, a cross-entropy function, or a min-squared error function.

The first local network module 140 updates the weight value of the first local network module 140 using the local training loss value L _mi .

Alternatively, the first local network module may calculate the local training loss value Lmi as shown in Equation 6 below.

[Equation 6]

In FIG. 1, although one hidden layer 130 is shown between the input layer 110 and the output layer 130, the learning and inference device 100 has one between the input layer 110 and the output layer 130. The above-mentioned hidden layer may be included.

At this time, the number of local network modules corresponding to the added hidden layer may increase.

As described above, the learning and inference apparatus 100 according to an embodiment of the present invention may unlock hierarchical dependencies on upper layers generated when back-propagation learning of layers constituting the main network processing unit.

In addition, the learning and inference device 100 according to an embodiment of the present invention may unlock the dependency on the upper layer of the local network module.

Accordingly, the present invention can significantly improve the learning speed since there is no need to go through the back-forward process after the feed-forward process.

2 is a view for explaining a learning operation of the learning and reasoning apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the first local network module 212 corresponding to the first layer 211 and the first layer 211 is configured as a first node 210 to describe a learning operation of the learning and inference device And, the second local network module 222 corresponding to the second layer 221 and the second layer 221 is configured and illustrated as a second node 220.

The first layer 211 and the second layer 221 may be input layers and hidden layers, respectively.

Alternatively, the first layer 211 and the second layer 221 may be both hidden layers when there are multiple hidden layers.

Looking at the operation of the learning and inference device, the first layer 211 outputs a first output value to the second layer 221 and the first local network module 212 (step ①).

Next, the first local network module 212 outputs the first error gradient value to the first layer 211, and at the same time, the second layer 221 outputs the second to the second local network module 222 and the next layer. The value is output (step ②). At this time, the weight value of the first layer 221 may be updated by a gradient descent method using a first error gradient value.

Next, the second local network module 222 outputs the second error gradient value to the second layer 211, and at the same time, the next layer outputs the output value to the next local network module (step ③). At this time, the weight value of the second layer 221 may be updated by a gradient descent method using a second error gradient value.

Next, the second local network module 222 outputs the calculated second training loss value of the second layer 211 to the first local network module 212 (step ④). At this time, the weight value of the first local network module 212 calculates the local training loss value using the second training loss value, calculates the local error gradient value by differentiating the calculated local training loss value, and calculates the calculated local value. It can be updated by gradient descent using the error gradient value.

In this way, the learning and inference device can improve the learning speed by performing feed forward and update simultaneously through a parallel network.

The detailed operation of the learning and reasoning apparatus illustrated in FIG. 2 is the same as the operation of the learning and reasoning apparatus described with reference to FIG. 1, and detailed description thereof will be omitted.

3 is a view for explaining the reasoning operation of the learning and reasoning apparatus according to an embodiment of the present invention.

Referring to FIG. 3, the learning and inference device receives and classifies the input value X and outputs the inference value.

The learning and reasoning device of FIG. 3 may be a device learned by the learning operation described with reference to FIGS. 1 and 2.

In the learning operation, the input value means a data value for classification.

For example, when the learning and inference device performs learning to classify a dog picture and a cat picture, the input value may be a picture to be classified (or inferred).

The learning and inference device includes an input layer 311, a first hidden layer 321, a second hidden layer 331, an output layer 341, a first local network module 312, and a second local network module 322. And a third local network module 332.

The first sub-model unit 310 receives the input value X and performs an operation to output a first approximation value.

The first sub-model unit 310 includes an input layer 311 and a first local network module 312.

The input layer 311 receives the input value X and performs an operation to output the first output value to the first local network module 312.

The first local network module 312 receives a first output value and performs an operation to output a first approximation value.

The second sub-model unit 320 receives the first output value and performs an operation to output the second approximation value.

The second sub-model unit 320 includes an input layer 311, a first local network module 312, a first hidden layer 321, and a second local network module 322.

The first hidden layer 321 receives a first output value from the input layer 311.

The first hidden layer 321 receives the first output value and performs an operation to output the second output value.

The second local network module 322 receives the second output value and performs an operation to output the second approximation value.

The third sub-model unit 330 receives the second output value and performs an operation to output the third approximation value.

The third sub model unit 330 includes an input layer 311, a first local network module 312 and a first hidden layer 321, a second local network module 322, a second hidden layer 331, and a third It consists of 3 local network modules (332).

The second concealment layer 331 receives a second output value from the first concealment layer 321.

The second concealment layer 331 receives the second output value and performs an operation to output the third output value.

The third local network module 332 receives the third output value and performs an operation to output a third approximation value.

The main model unit 340 receives the input value X and outputs the final output value.

The main model unit 340 includes an input layer 311, a first hidden layer 321, a second hidden layer 331, and an output layer 341.

The main model unit 340 is fed forward in the order of the input layer 311, the first hidden layer 321, the second hidden layer 331, and the output layer 341, and outputs a final output value.

Operations of the input layer 311, the first hidden layer 321, the second hidden layer 331, and the output layer 341 are the same as those of the layers described with reference to FIGS. 1 and 2, so a detailed description is omitted. do.

Operations of the first local network module 312, the second local network module 322, and the third local network module 332 are the same as those of the local network modules described with reference to FIGS. Omitted.

The learning and reasoning device may further include a reasoning unit (not shown).

The inference unit receives approximate values output from the sub-model units 310, 320, and 330 sequentially output while being fed forward.

The inference unit determines whether the reliability of the first approximation value is greater than or equal to the threshold.

The inference unit outputs the first approximation value as a reasoning value when the reliability of the first approximation value is greater than or equal to the threshold value, and determines the reliability of the second approximation value when the reliability is less than the threshold value.

The inference unit outputs the second approximation value as a reasoning value when the reliability of the second output value is greater than or equal to the threshold value, and determines the reliability of the third approximation value when the threshold value is less than the threshold value.

The inference unit outputs the third approximation value as the inference value when the reliability of the third approximation value is greater than or equal to the threshold value, and outputs the final output value as the inference value when both the first to third approximation values are less than the threshold value.

At this time, the threshold may be preset by the user, and the first to third approximate values and the final output value may be softmax output values.

That is, the learning and inference apparatus sequentially determines the reliability values of the approximate values of the sub-model units, and when the threshold value is greater than or equal to a threshold, the output layer 341 undergoes a complete feedforward process and does not have to wait until the output layer 341 outputs the final output value. The inference value of the reliability can be obtained. At this time, when the reliability values of the first to third sub-model units 310, 320, and 330 are less than the threshold, the final output value of the main model unit 340 may be output as an inference value.

For example, the learning and inference device that has trained a dog picture and a cat picture may output a first approximation value through a first sub-model. In this case, the first sub-model may output a first approximation value of 0.6 for a puppy and 0.4 for a kitten. Assuming that the preset threshold is 0.9, the inference unit receives a first approximation value, determines that it is less than the threshold, and inputs a second approximation value of a probability of 0.85 as a puppy and a probability of 0.15 as a cat from the second sub-model. Can receive Similarly, the inference unit may receive a third approximation value, which is a probability that the puppy is 0.91 and a probability that the cat is 0.09, by determining that it is less than a threshold from the third submodel. At this time, the reasoning unit determines that the probability of being a puppy is 0.91 or more, and outputs a third approximation value as a reasoning value. That is, the learning and reasoning device determines the puppy and outputs a reasoning value.

Accordingly, the learning and inference apparatus and the method according to an embodiment of the present invention can output a classification value (inference value) without a complete feedforward process when the reliability of the submodel unit is greater than or equal to a threshold. Accordingly, the operation speed can be significantly improved.

Table 1 below may represent an algorithm for implementing a reasoning unit of a learning and reasoning apparatus according to an embodiment of the present invention.

[Table 1]

Input : data x, threshold t

Model : sub-model m _i , main-model f

Initialize: classification = 0.

for I = 1 to N-1 do

if max softmax (m _i (x))> then

classification = argmax softmax (m _i (x))

break

end if

end for

if classification == 0 then

  #if all sub-models are not confident

  Classification = argmax softmax (f (x))

end if

The learning and inference device illustrated in FIG. 3 is composed of two hidden layers, but may be composed of one or more hidden layers and corresponding local network modules.

4 is a flowchart illustrating a learning and reasoning method according to an embodiment of the present invention.

Referring to FIG. 4, in the learning and inference device, in step S410, local network modules corresponding to each of the input layer and the hidden layer calculate the error slope value based on the output value of each layer.

In the learning and inference device, in step S420, the input layer and the hidden layer are updated by error slope values, respectively.

In step S430, the learning and inference device is updated based on the training loss value calculated by the local network module of the hidden layer in the local network module of the input layer.

The learning and reasoning method of FIG. 4 is the same as the operation method of the learning and reasoning device described with reference to FIGS. 1 to 3, and detailed descriptions thereof will be omitted.

The device described above may be implemented with hardware components, software components, and / or combinations of hardware components and software components. For example, the devices and components described in the embodiments include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors (micro signal processors), microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, a processing device may be described as one being used, but a person having ordinary skill in the art, the processing device may include a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that may include. For example, the processing device may include a plurality of processors or a processor and a controller. In addition, other processing configurations, such as parallel processors, are possible.

The software may include a computer program, code, instruction, or a combination of one or more of these, and configure the processing device to operate as desired, or process independently or collectively You can command the device. Software and / or data may be interpreted by a processing device, or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodied in the transmitted signal wave. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments or may be known and usable by those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and magnetic media such as floptical disks. -Hardware devices specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language codes produced by a compiler. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described by a limited embodiment and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques are performed in a different order than the described method, and / or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form from the described method, or other components Alternatively, even if replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

A main network processor including an input layer, a hidden layer, and an output layer outputting the final output value; And
An error slope value is calculated based on the output value of each of the input layer and the hidden layer, and the local network modules are updated based on the training loss value calculated by the local network module in the next stage.
Learning and reasoning devices. According to claim 1,
The local network modules calculate the approximate value approximated to the final output value using the output values of the input layer and the hidden layer, and calculate the training loss value by the loss function using the approximate value, and the training Calculate the error slope value by Loss value
Learning and reasoning devices. According to claim 2,
The loss function is a min-absolute error function, a min-square error function, or a cross-entropy function.
Learning and reasoning devices. According to claim 3,
The input layer and the hidden layer are each updated by the error slope value.
Learning and reasoning devices. According to claim 3,
The input layer and the hidden layer is a learning and inference device that is updated by the gradient descent method. According to claim 2,
The local network module of the input layer calculates a local training loss value by a loss function using the training loss value calculated by the local network module of the hidden layer, and calculates a local error slope value by the local training loss value. And updated by the local error slope value
Learning and reasoning devices. The method of claim 6,
The loss function is a min-absolute error function, a min-square error function, or a cross-entropy function.
Learning and reasoning devices. The method of claim 6,
The network module of the input layer is a learning and inference device that is updated by the gradient descent method. According to claim 2,
A local network module of the input layer and the input layer is configured as a first sub-model unit, and a local network module of the input layer, the hidden layer and the hidden layer is configured as a second sub-model unit, and the input layer and the hidden The layer and the output layer are configured as a main model unit,
The reliability of the approximate values of the sub-models is sequentially determined to output the approximate value as an inference value when the reliability has an approximation value equal to or higher than a threshold value, and infers the final output value when the reliability does not have an approximation value equal to or higher than a threshold value. Further comprising a reasoning unit to output as a value
Learning and reasoning devices. In the learning and inference method of the learning and inference device comprising an input layer, a hidden layer and an output layer outputting the final output value,
Local network modules corresponding to each of the input layer and the hidden layer calculate an error slope value based on the output value of each layer;
Updating the input layer and the hidden layer by the error slope value, respectively; And
And updating the local network module of the input layer based on the training loss value calculated by the local network module of the hidden layer.
Learning and reasoning methods. The method of claim 10,
The step of calculating the error slope value,
Calculating, by the local network modules, an approximate value approximated to a final output value of the output layer by using output values of the input layer and the hidden layer;
Calculating, by the local network modules, a training loss value by a loss function using the approximation value; And
And the local network modules calculating the error slope value based on the training loss value.
Learning and reasoning methods. The method of claim 11,
The loss function is a min-absolute error function, a min-square error function, or a cross-entropy function.
Learning and reasoning methods. The method of claim 10,
The input layer and the hidden layer are updated by a gradient descent method
Learning and reasoning methods. The method of claim 10,
In the step of updating the local network module of the input layer,
Calculating a local training loss value by a loss function using the training loss value calculated by the network module of the hidden layer by the local network module of the input layer;
A local network module of the input layer calculating a local error slope value based on the local training loss value; And
And updating the local network module of the input layer by the local error slope value.
Learning and reasoning methods. The method of claim 14,
The loss function is a min-absolute error function, a min-square error function, or a cross-entropy function.
Learning and reasoning methods. The method of claim 10,
The local network module of the input layer is updated by the gradient descent method.
Learning and reasoning methods. The method of claim 11,
A local network module of the input layer and the input layer is configured as a first sub-model unit, and a local network module of the input layer, the hidden layer and the hidden layer is configured as a second sub-model unit, and the input layer and the hidden The layer and the output layer are configured as a main model unit,
Sequentially determining the reliability of the sub-models for the approximate values of the sub-models; And
The inference unit further includes outputting the approximation value as an inference value when the reliability has an approximation value equal to or greater than the threshold, and outputting the final output value as an inference value when the reliability does not have an approximation value equal to or higher than the threshold value.
Learning and reasoning methods.

Images