EP3671574B1

EP3671574B1 - Device and method to improve the robustness against adversarial examples

Info

Publication number: EP3671574B1
Application number: EP18213925.3A
Authority: EP
Inventors: Zico KOLTER; Frank Schmidt; Eric Wong
Original assignee: Robert Bosch GmbH; Carnegie Mellon University
Current assignee: Robert Bosch GmbH; Carnegie Mellon University
Priority date: 2018-12-19
Filing date: 2018-12-19
Publication date: 2024-07-10
Anticipated expiration: 2038-12-19
Also published as: CN113302630A; JP7264410B2; WO2020126372A1; US20210326647A1; JP2022515756A; EP3671574A1

Description

The invention concerns a method for assessing robustness of an image classifier according to claim 1, a method for training an image classifier according to claim 8, a computer program according to claim 11, a machine-readable storage medium according to claim 12, and a system according to claim 13

Prior art

US10007866 BB discloses a method comprising: accessing, from a memory, a neural network image classifier, the neural network image classifier having been trained using a plurality of training images from an input space, the training images being labeled for a plurality of classes;

computing a plurality of adversarial images by, for each adversarial image, searching a region in the input space around one of the training images, the region being one in which the neural network is linear, to find an image which is incorrectly classified into the plurality of classes by the neural network; applying the training image to the neural network and observing a response of the neural network;
computing a constraint system which represents the input space using the observed response; and
further training the neural network image classifier to have improved accuracy using at least the adversarial images.

"Universal Adversarial Perturbations Against Semantic Image Segmentation", arXiv preprint arXiv:1704.05712v3, Jan Hendrik Metzen, Mummadi Chaithanya Kumar, Thomas Brox, Volker Fischer, disclose a method for generating adversarial perturbations.
A. C. Serban, E. Poll, "Adversarial examples - a complete characterisation of the phenomenon", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853 provides an overview on how to generate and use adversarial examples.

Advantages of the invention

Classifiers, like e.g. neural network classification systems can easily be fooled. It is well known that classifiers which may be based on deep learning may be sensitive to small perturbations. In order to deploy such systems in the physical world it is important to provide a proof about the system's robustness.
It is possible to compute robust classifiers with respect to adversarial noise that lies within a small L_p ball. Nonetheless, adversarials with respect to more natural perturbations are not necessarily covered by these robustness statements. More natural perturbations include partial translations, rotations and motion blur. Moving for example a dark object by one pixel will lead to a very large L _∞ -distance if the background is very bright, but will usually be considered as a small change in the physical world. As a consequence, these small physical changes are not covered by L _∞ --robustness.
The methods of the claim set have the advantage to improve robustness with respect to such perturbations.

Disclosure of the invention

A first embodiment according to the invention is defined by claim 1.
The at least approximate Wasserstein distance is a Sinkhorn distance which differs from a Wasserstein distance by an entropic term, wherein for any pair of first distribution (P) and second distribution (Q), said entropic term characterizes an entropy of a distribution Π that satisfies Π1 _n =P, Π^T 1 _n =Q. If P and Q are distributions defined over the same domain Ω, then Π is a distribution over the domain Ω×Ω with P and Q as its marginals.
It has been discovered that the inclusion of said entropic term enables an approximate solution to the projection on a Wasserstein ball that is a lot faster to compute.
It should be noted that, as shown in "Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances", arXiv preprint arXiv:1306.0895v1, Marco Cuturi (2013), a Sinkhorn distance is in fact not a metric in the mathematical sense, since it is possible to have a zero distance between two distributions that are not the same. Instead, in a mathematical sense, it is a pseudo-metric.
In fact, it has been discovered that a good way for determining the projected input signal (x^proj) by solving a convex optimization corresponding to said minimization under said constraints. This is described in detail in the description corresponding to figure 14.
In one further aspect to the invention, the adversarial input signal (x^adv) may be provided by a targeted attack, i.e. provided to cause the classifier to classify it as belonging to a predefined second class. An efficient way for doing so can be provided if said classifier when provided with said input signal (x), is configured to output a first classification value (f _{l 0}) corresponding to said first class (ℓ₀) and a second classification value (f_l ) corresponding to said a predefined second class (ℓ). In this sense, it may be said that said input signal (x) causes said first and/or second classification value. A robust way to generate said targeted misclassification is by determining said modified input signal (x^mod) such as to cause a difference (g) between said first classification value (f _{l 0}) and said second classification value (f_l ) to be smaller than the difference (g) caused by said original input signal (x^org). Conveniently, this may be achieved by determining said modified input signal (x^mod) depending on a gradient (∇g) of said difference (g).
In alternative embodiment to said targeted attack, said adversarial input signal (x^adv) may be provided by an untargeted attack, i.e. provided to cause said classifier to classify it as belonging to any different second class. In this case, conveniently said modified input signal (x^mod) is provided such as to cause said first classification value (f _{l 0}) to be smaller than said first classification value (f _{l 0}) caused by said original input signal (x^org). Conveniently, this may be achieved by determining said modified input signal (x^mod) depending on a gradient (∇f _{l 0}) of said first classification value (f _{l 0}).
In a further aspect, the steps of modifying said original input signal (x^org) to yield said modified input signal (x^mod) and projecting said modified input signal (x^mod) onto said predefined subset to yield said projected input signal (x^proj) are carried out iteratively by using said projected input signal (x^proj) of a preceding iteration as original input signal (x^org) a subsequent iteration, wherein said step projecting said modified input signal (x^mod) onto said predefined subset is carried out after each step of modifying said original input signal (x^org). Such an iterative method is preferable, because it ensures that intermediate modified input signal (x^mod) remain close to a boundary of the at least approximate Wasserstein ball, thus enhancing convergence of the method.
Embodiments of the invention will be discussed with reference to the following figures in more detail. The figures show:

Figure 1: a control system having a classifier controlling an actuator in its environment;
Figure 2: the control system controlling an at least partially autonomous robot;
Figure 3: the control system controlling a manufacturing machine;
Figure 4: the control system controlling an automated personal assistant;
Figure 5: the control system controlling an access control system;
Figure 6: the control system controlling a surveillance system;
Figure 7: the control system controlling an imaging system;
Figure 8: a training system for controlling the classifier;
Figure 9: a flow-chart diagram of a training method carried out by said training system;
Figure 10: a flow-chart diagram illustrating a method for operating said control system;
Figure 11: an embodiment of a structure of said classifier;
Figure 12: a flow-chart diagram illustrating a method for determining said adversarial input signal x^adv;
Figure 13: a flow-chart diagram illustrating a method for projecting a modified input signal x^mod onto a Wasserstein ball;
Figure 14: a flow-chart diagram illustrating a method for projecting a modified input signal x^mod onto a Sinkhorn ball;
Figure 15: a flow-chart diagram illustrating a method for computing Π as defined in equation (2) from the maximizing values Φ*, Ψ*, ρ* that solve equation (5).

Description of the embodiments

Shown in figure 1 is one embodiment of an actuator 10 in its environment 20. Actuator 10 interacts with a control system 40. Actuator 10 and its environment 20 will be jointly called actuator system. At preferably evenly spaced distances, a sensor 30 senses a condition of the actuator system. The sensor 30 may comprise several sensors. Preferably, sensor 30 is an optical sensor that takes images of the environment 20. An output signal S of sensor 30 (or, in case the sensor 30 comprises a plurality of sensors, an output signal S for each of the sensors) which encodes the sensed condition is transmitted to the control system 40.
Thereby, control system 40 receives a stream of sensor signals S. It then computes a series of actuator control commands A depending on the stream of sensor signals S, which are then transmitted to actuator 10.
Control system 40 receives the stream of sensor signals S of sensor 30 in an optional receiving unit 50. Receiving unit 50 transforms the sensor signals S into input signals x. Alternatively, in case of no receiving unit 50, each sensor signal S may directly be taken as an input signal x. Input signal x may, for example, be given as an excerpt from sensor signal S. Alternatively, sensor signal S may be processed to yield input signal x. Input signal x comprises image data corresponding to an image recorded by sensor 30. In other words, input signal x is provided in accordance with sensor signal S.
Input signal x is then passed on to an image classifier 60, which may, for example, be given by an artificial neural network.
Classifier 60 is parametrized by parameters φ, which are stored in and provided by parameter storage St ₁.
Classifier 60 determines output signals y from input signals x. The output signal y comprises information that assigns one or more labels to the input signal x Output signals y are transmitted to an optional conversion unit 80, which converts the output signals y into the control commands A. Actuator control commands A are then transmitted to actuator 10 for controlling actuator 10 accordingly. Alternatively, output signals y may directly be taken as control commands A.
Actuator 10 receives actuator control commands A, is controlled accordingly and carries out an action corresponding to actuator control commands A. Actuator 10 may comprise a control logic which transforms actuator control command A into a further control command, which is then used to control actuator 10.
In further embodiments, control system 40 may comprise sensor 30. In even further embodiments, control system 40 alternatively or additionally may comprise actuator 10.
In still further embodiments, it may be envisioned that control system 40 controls a display 10a instead of an actuator 10.
Furthermore, control system 40 may comprise a processor 45 (or a plurality of processors) and at least one machine-readable storage medium 46 on which instructions are stored which, if carried out, cause control system 40 to carry out a method according to one aspect of the invention.
Figure 2 shows an embodiment in which control system 40 is used to control an at least partially autonomous robot, e.g. an at least partially autonomous vehicle 100.
Sensor 30 may comprise one or more video sensors and/or one or more radar sensors and/or one or more ultrasonic sensors and/or one or more LiDAR sensors and or one or more position sensors (like e.g. GPS). Some or all of these sensors are preferably but not necessarily integrated in vehicle 100. Alternatively or additionally sensor 30 may comprise an information system for determining a state of the actuator system. One example for such an information system is a weather information system which determines a present or future state of the weather in environment 20.
For example, using input signal x, the classifier 60 may for example detect objects in the vicinity of the at least partially autonomous robot. Output signal y may comprise an information which characterizes where objects are located in the vicinity of the at least partially autonomous robot. Control command A may then be determined in accordance with this information, for example to avoid collisions with said detected objects.
Actuator 10, which is preferably integrated in vehicle 100, may be given by a brake, a propulsion system, an engine, a drivetrain, or a steering of vehicle 100.
Actuator control commands A may be determined such that actuator (or actuators) 10 is/are controlled such that vehicle 100 avoids collisions with said detected objects. Detected objects may also be classified according to what the classifier 60 deems them most likely to be, e.g. pedestrians or trees, and actuator control commands A may be determined depending on the classification.
In further embodiments, the at least partially autonomous robot may be given by another mobile robot (not shown), which may, for example, move by flying, swimming, diving or stepping. The mobile robot may, inter alia, be an at least partially autonomous lawn mower, or an at least partially autonomous cleaning robot. In all of the above embodiments, actuator command control A may be determined such that propulsion unit and/or steering and/or brake of the mobile robot are controlled such that the mobile robot may avoid collisions with said identified objects.
In a further embodiment, the at least partially autonomous robot may be given by a gardening robot (not shown), which uses sensor 30, preferably an optical sensor, to determine a state of plants in the environment 20. Actuator 10 may be a nozzle for spraying chemicals. Depending on an identified species and/or an identified state of the plants, an actuator control command A may be determined to cause actuator 10 to spray the plants with a suitable quantity of suitable chemicals.
In even further embodiments, the at least partially autonomous robot may be given by a domestic appliance (not shown), like e.g. a washing machine, a stove, an oven, a microwave, or a dishwasher. Sensor 30, e.g. an optical sensor, may detect a state of an object which is to undergo processing by the household appliance. For example, in the case of the domestic appliance being a washing machine, sensor 30 may detect a state of the laundry inside the washing machine. Actuator control signal A may then be determined depending on a detected material of the laundry.
Shown in figure 3 is an embodiment in which control system 40 is used to control a manufacturing machine 11, e.g. a punch cutter, a cutter or a gun drill) of a manufacturing system 200, e.g. as part of a production line. The control system 40 controls an actuator 10 which in turn control the manufacturing machine 11.
Sensor 30 may be given by an optical sensor which captures properties of e.g. a manufactured product 12. Classifier 60 may determine a state of the manufactured product 12 from these captured properties. Actuator 10 which controls manufacturing machine 11 may then be controlled depending on the determined state of the manufactured product 12 for a subsequent manufacturing step of manufactured product 12. Or, it may be envisioned that actuator 10 is controlled during manufacturing of a subsequent manufactured product 12 depending on the determined state of the manufactured product 12.
Shown in figure 4 is an embodiment in which control system 40 is used for controlling an automated personal assistant 250. Sensor 30 may be an optic sensor, e.g. for receiving video images of a gestures of user 249. Alternatively, sensor 30 may also be an audio sensor e.g. for receiving a voice command of user 249.
Control system 40 then determines actuator control commands A for controlling the automated personal assistant 250. The actuator control commands A are determined in accordance with sensor signal S of sensor 30. Sensor signal S is transmitted to the control system 40. For example, classifier 60 may be configured to e.g. carry out a gesture recognition algorithm to identify a gesture made by user 249. Control system 40 may then determine an actuator control command A for transmission to the automated personal assistant 250. It then transmits said actuator control command A to the automated personal assistant 250.
For example, actuator control command A may be determined in accordance with the identified user gesture recognized by classifier 60. It may then comprise information that causes the automated personal assistant 250 to retrieve information from a database and output this retrieved information in a form suitable for reception by user 249.
In further embodiments, it may be envisioned that instead of the automated personal assistant 250, control system 40 controls a domestic appliance (not shown) controlled in accordance with the identified user gesture. The domestic appliance may be a washing machine, a stove, an oven, a microwave or a dishwasher.
Shown in figure 5 is an embodiment in which control system controls an access control system 300. Access control system may be designed to physically control access. It may, for example, comprise a door 401. Sensor 30 is configured to detect a scene that is relevant for deciding whether access is to be granted or not. It may for example be an optical sensor for providing image or video data, for detecting a person's face. Classifier 60 may be configured to interpret this image or video data e.g. by matching identities with known people stored in a database, thereby determining an identity of the person. Actuator control signal A may then be determined depending on the interpretation of classifier 60, e.g. in accordance with the determined identity. Actuator 10 may be a lock which grants access or not depending on actuator control signal A. A non-physical, logical access control is also possible.
Shown in figure 6 is an embodiment in which control system 40 controls a surveillance system 400. This embodiment is largely identical to the embodiment shown in figure 5. Therefore, only the differing aspects will be described in detail. Sensor 30 is configured to detect a scene that is under surveillance. Control system does not necessarily control an actuator 10, but a display 10a. For example, the machine learning system 60 may determine a classification of a scene, e.g. whether the scene detected by optical sensor 30 is suspicious. Actuator control signal A which is transmitted to display 10a may then e.g. be configured to cause display 10a to adjust the displayed content dependent on the determined classification, e.g. to highlight an object that is deemed suspicious by machine learning system 60.
Shown in figure 7 is an embodiment of a control system 40 for controlling an imaging system 500, for example an MRI apparatus, x-ray imaging apparatus or ultrasonic imaging apparatus. Sensor 30 may, for example, be an imaging sensor. Machine learning system 60 may then determine a classification of all or part of the sensed image. Actuator control signal A may then be chosen in accordance with this classification, thereby controlling display 10a. For example, machine learning system 60 may interpret a region of the sensed image to be potentially anomalous. In this case, actuator control signal A may be determined to cause display 10a to display the imaging and highlighting the potentially anomalous region.
Shown in figure 8 is an embodiment of a training system 140 for training classifier 60. A training data unit 150 determines input signals x, which are passed on to classifier 60. For example, training data unit 150 may access a computer implemented database St ₂ in which a set T of training data is stored. Set T comprises pairs of input signal x and corresponding desired output signal y_s. Training data unit 150 selects samples from set T, e.g. randomly. Input signal x of a selected sample is passed on to classifier 60. Desired output signal y_s is passed on to assessment unit 180.
Classifier 60 is configured to compute output signals y from input signals x. These output signals x are also passed on to assessment unit 180.
A modification unit 160 determines updated parameters φ' depending on input from assessment unit 180. Updated parameters φ'are transmitted to parameter storage St ₁ to replace present parameters φ.
For example, it may be envisioned that assessment unit 180 determines the value of a loss function
depending on output signals y and desired output signals y_s. Modification unit 160 may then compute updated parameters φ' using e.g. stochastic gradient descent to optimize the loss function
.
Furthermore, modification unit 160 may compute an adversarial dataset T' comprising modified input signals x^adv based on original input signals x taken, for example, from training set T and their respective desired output signals y_s.
Furthermore, training system 140 may comprise a processor 145 (or a plurality of processors) and at least one machine-readable storage medium 146 on which instructions are stored which, if carried out, cause control system 140 to carry out a method according to one aspect of the invention.
Shown in figure 9 is a flow-chart diagram of an embodiment of the method for training classifier 60, what may be implemented by training system 140.
First (901), classifier 60 is trained with training data of set T in a conventional manner, as discussed above.
Then (902), one or more adversarial input signals x^adv and corresponding desired output signals y_s are generated with the method according illustrated in figure 11 by modifying input signals from data set T and leaving the corresponding desired output signal y_s unchanged. These one or more pairs of adversarial input signal x^adv and corresponding desired output signals y are added to adversarial dataset T'.
Now (903), classifier 60 is trained with training data of set adversarial dataset T'. The trained classifier 60 may then (904) be used for providing an actuator control signal A by receiving sensor signal S comprising data from sensor 30, determining the input signal x depending on said sensor signal S, and feeding said input signal x into classifier 60 to obtain output signal y that characterizes a classification of input signal x. Actuator 10 or 10a may then be controlled in accordance with provided actuator control signal A. This concludes the method.
Shown in figure 10 is a flow-chart diagram of an embodiment of the method for operating classifier 60, which may be implemented by control system 40.
First (911), parameters φ that characterize the operation of classifier 60 are provided. Conventionally, they are obtained by a training method for training classifier 60, e.g. by supervised training as outlined above.
The trained classifier 60 may then (912) be used for providing a first output signal y1 by receiving sensor signal S comprising data from sensor 30, determining the input signal x depending on said sensor signal S, and inputting said input signal x into classifier 60 to obtain first output signal y1 that characterizes a classification of input signal x.
Then (913), an adversarial input signal x^adv is generated with the method according illustrated in figure 11 by modifying input signal x.
This adversarial input signal x^adv is then (914) inputted into classifier 60 to obtain a second output signal y2 that characterizes a classification of adversarial input signal x^adv.
Next (915), a parameter vu indicating a vulnerability of classifier 60 is computed based on said first output signal y1 and said second output signal y2. For example, it is possible to set said parameter vu to a first value (for example "1") indicating a vulnerability, if said first output signal y1 is not equal to said second output signal y2, and equal to a first value (for example "0") indicating a non-vulnerability, if said first output signal y1 is equal to said second output signal y2.
An actuator control signal (A) may then (916) be determined in accordance with said parameter vu, and actuator (10) may be controlled in accordance with said actuator control signal (A). For example, if said parameter vu indicates a non-vulnerability, said actuator control signal (A) may then be determined to correspond to normal operation mode, whereas, if said parameter vu indicates a vulnerability, said actuator control signal (A) may then be determined to correspond to a fail-safe operation mode, by e.g. reducing a dynamics of a motion of said actuator (10).
Figure 11 illustrates schematically a structure of one embodiment of classifier 60. Input signal x is inputted into processing unit 61, which may, for example, be given by all but the last layer of a neural network. Processing unit 61 is configured to output a vector f comprising preferably at least one entry z_ℓ for each of the possible classes ℓ for classification. Said vector f is inputted into a selector 62, which may be given, e.g., by an implementation of an argmax function. Selector 62 is configured to output signal y, which corresponds to the class corresponding to that one of the entries f_ℓ of vector f that has the highest value. To highlight the dependence on input signal x, vector f will also be denoted f(x).
Figure 12 illustrates a method for determining an adversarial input signal x^adv based on a given input signal x, which will also be called original input signal x^org. This method may be implemented by control system 40 or by training system 140. In an initialization step (1000), a counter variable may be initialized as counter = 0, a step size τ may be initialized as e.g. τ = 20 and a modified input signal x^mod may be initialized as x^mod=x. Original input signal x^org is inputted into classifier 60 and resulting output signal y is determined. A correct classification ℓ₀ is initialized as ℓ₀ = y. A target classification ℓ ≠ ℓ₀ may be selected either randomly or e.g. by setting it to a predefined value, or by selecting it as that classification ℓ ≠ ℓ₀ that is closest to correct classification ℓ₀. For example, target classification ℓ may be determined as that classification that corresponds to the second largest entry f _ℓ of vector f. May be targeted or untargeted attack
Then (1100), modified input signal x^mod is inputted into classifier 60 and corresponding vector f(x^mod) is determined. Then, a scalar function g(x) = f(x)_ℓ - f(x) _{ℓ 0} is evaluated and its gradient ∇g(x)|_{_x=x mod} is determined. Modified input signal x^mod may be updated as $x^{\mod} = x^{\mod} + τ \cdot {\nabla g (x) |}_{x = x^{\mod}} .$
Next (1200), a projected input signal x^proj is be determined by projecting modified input signal x^mod onto a Wasserstein ball with a predefined radius ε centered around original input signal x^org. This projection may be carried out with one of the methods illustrated in figures 13 and 14.
Then (1300), the counter is incremented counter ← counter + 1 and it is checked (1400), if the counter is a multiple of a predefined number, e.g. 20. If that is the case (1500), the counter is reset to counter = 0 and step size τ is increased by a predefined factor, e.g. τ ← τ · 1.1.
Both of steps (1400) and (1500) are followed by checking (1600) whether the counter is less than a predefined maximum counter counter_max, i.e. if counter < counter_max. Furthermore, modified input signal x^mod is set equal to projected input signal x^proj, and scalar g(x^mod) is evaluated. If counter < counter_max and if g(x ^mod) ≤ ub with an upper bound ub which may be set to any non-negative number, e.g. ub = 0 (i.e. classification has not changed from correct classification ℓ₀ to target classification ℓ), the method iterates back to step (1200). If not (1600), adversarial input signal x^adv is provided as equal to modified input signal x^mod. Optionally, if g(x^adv) ≤ ub an error message may be provided indicating that no adversarial has been found with the desired confidence. This concludes the method.
Figure 13 illustrates a method for determining projected input signal x^proj from modified input signal x^mod. This projection involves the computation of a Wasserstein distance W_D (P, Q) between two n-dimensional vectors P and Q which are given as P = (P ₁, ... , P_n ) and Q = (Q ₁, ... , Q_n ). Distances between indices i and j are stored in a matrix D_ij ∈ R ^n×n (where D_ij = ∥i - ∥ ^p for some pre-defined value of p) and the Wasserstein distance W_D (P, Q) can be computed as $W_{D} (P, Q) = \min_{\begin{matrix} R^{n \times n} ∋ Π \geq 0 \\ Π 1 = P \\ Π^{T} 1 = Q \end{matrix}} tr (Π^{⊤} D) = \max_{\begin{matrix} Φ, Ψ \in R^{n} \\ \forall i, j : Φ_{i} \leq Ψ_{j} + D_{ij} \end{matrix}} P^{⊤} Φ - Q^{⊤} Ψ .$
(Here, the 1 in Π1 = P, Π^T1 = Q denotes an n-dimensional vector of ones). Determining said projected input signal x^proj then corresponds to solving equation $\min_{W_{D} (x^{\mod}, x^{org *}) \leq ϵ} L_{2} (x^{proj}, x^{org}) .$
(Of course, the L ₂-metric may be replaced by any other metric).
First (1310), it is determined whether the Wasserstein distance W_D (x^mod,x^org ) between modified input signal x^mod and original input signal x^org is not larger than predefined radius ε, i.e. if $W_{D} (x^{\mod}, x^{org}) \leq ϵ .$
If that is the case (1320), projected input signal x^proj is set equal to modified input signal x^mod, and the method ends.
If that is not case (1330), and denoting P = x^mod and Q = x^org, equation $\max_{\begin{matrix} Φ, Ψ \in R^{n}, ρ \in R \\ \forall i, j : Φ_{i} \leq Ψ_{j} + ρ \cdot D_{ij} \end{matrix}} P^{⊤} Φ - Q^{⊤} Ψ - [\frac{1}{2} {‖ Ψ ‖}^{2} + ρ \cdot ϵ]$
is solved with e.g. projected gradient ascent to yield maximizing values Φ*, Ψ*, ρ*. It will be appreciated that equation (5) is a dual formulation to a primal problem given by equations (3) and (2).
Next (1340), Π as defined in equation (2) is determined from the maximizing values Φ*, Ψ*,ρ* using e.g. the method illustrated in figure 15.
Then (1350), projected input signal x^proj is set equal to x^proj = Π ^T 1. This concludes the method.
Figure 14 illustrates another, approximate but more efficient method, wherein instead of using a Wasserstein distance W_D (P,Q) as defined in equation (2) for defining the ball on which to project, one uses a Sinkhorn distance $W_{D}^{λ} (P, Q)$
by subtracting an entropic term E_T, i.e. $W_{D}^{λ} (P, Q) = \min_{\begin{matrix} R^{n \times n} ∋ Π \geq 0 \\ Π 1 = P \\ Π^{T} 1 = Q \end{matrix}} tr (Π^{⊤} D) + E_{T},$
$E_{T} = \frac{1}{λ} \sum_{Π_{ij} > 0} Π_{ij} \log (Π_{ij})$
with a predefined variable λ ≠ 0, e.g. λ = 1.
First (1311), it is determined whether the Sinkhorn distance $W_{D}^{λ} (x^{\mod}, x^{org})$
between modified input signal x^mod and original input signal x^org is not larger than predefined radius ε, i.e. if $W_{D}^{λ} (x^{\mod}, x^{org}) \leq ϵ .$
If that is the case (1321), projected input signal x^proj is set equal to modified input signal x^mod, and the method ends.
If that is not case (1331), and denoting P = x^mod and Q = x^org, a variable ρ is initialized as ρ = 1 and two n-dimensional vectors R, S are initialized by setting each of their components equal to e.g. R_i = S_i = 1/n.
Then (1341), an exponential n × n-dimensional matrix K is computed as $K = \exp (- λ \cdot ρ \cdot D) .$
Next (1351), components R_i of matrix R are updated as $R_{i} = P_{i} / {(K \cdot S)}_{i}$
and components S_i of matrix S are updated as $S_{i} = W_{0} (\exp ({λQ}_{i} - \frac{1}{2}) \cdot {tmp}_{i}) / {tmp}_{i}$
with $tmp = λ \cdot K^{T} R .$
Then (1361), scalars g and h are computed as $g = 〈 R, DKS 〉 - ε$
$h = - λ 〈 R, DDKS 〉$
where 〈...〉 denotes a scalar product, i.e. entry-wise multiplication followed by a sum over all multiplied entries. A value α is set to a positive value, e.g. α = 1.
Then (1371), α is set such that $ρ \approx α \frac{g}{h}$
but also $ρ \geq α \frac{g}{h}$
, e.g. by updating $α \leftarrow \frac{α}{2}$
as long as $ρ < α \frac{g}{h}$
.
Now (1381), ρ is updated as $ρ \leftarrow ρ - α \frac{g}{h}$
.
Next (1391), it is checked whether the method as converged, e.g. if changes to R and/or S over the last iteration are sufficiently small (e.g. less than a predefined threshold). If that is not the case, the method iterates back to step (1341). If the method has converged, however, step (1392) follows.
In this step, one sets $Π_{ij} = 〈 R_{i}, K \cdot S_{i} 〉$
$x^{proj} = Π^{T} 1$
This concludes the method.
Figure 15 illustrates an embodiment to compute Π as defined in equation (2) from the maximizing values Φ*, Ψ*, ρ* as obtained from the solution of equation (5) in step (1330). First (2000), variable i is initialized as i = 1. Then (2010), all values of j ∈ {1, ..., n} are identified for which $Φ_{i}^{*} < Ψ_{j}^{*} + ρ^{*} \cdot D_{ij}$
holds. If it does, the corresponding component Π _ij is set to Π _ij = 0. Then (2020), all values of j ∈ {1, ..., n} are identified for which $Φ_{i}^{*} = Ψ_{j}^{*} + ρ^{*} \cdot D_{ij}$
holds (i.e. all remaining values of j), and stored in a set J. Next (2030), the number of elements in J is counted and denoted sz(J). Then (2040), for all j ∈ J, the corresponding component Π _ij is set to Π _ij = p_i /sz(j). It is then checked (2060), if i < n holds. If this is the case (2070), i is incremented i ← i + 1 and the method iterates back to step (2010). If not, the method concludes and continues in step (1350).
The term "computer" covers any device for the processing of pre-defined calculation instructions. These calculation instructions can be in the form of software, or in the form of hardware, or also in a mixed form of software and hardware.
It is further understood that the procedures cannot only be completely implemented in software as described. They can also be implemented in hardware, or in a mixed form of software and hardware.

Claims

A computer-implemented method for assessing a robustness of an image classifier (60), wherein the image classifier (60) detects where objects are located in a vicinity of an at least partially autonomous robot from data received from one ore more video sensors or the classifier (60) determines a state of a manufactured product (12) from captured properties by an optical sensor (30) which captures properties of the manufactured product (12), wherein the method comprises the steps of:
- receiving a sensor signal (S) comprising data from the one or more video sensors or from the optical sensor (30),

- determining an original input image (x^org) which depends on said sensor signal (S),

- determining, by the image classifier (60), a first class (y1) that characterizes a classification of said original input image (xorg);

- determining an adversarial input image (x^adv) with a method of obtaining an adversarial input image (x^adv) to the image classifier (60),

- determining, by the image classifier (60), a second class (y2) that characterizes said adversarial input image (x^adv) and

- determining a robustness value (vu) indicating a vulnerability of the classifier (60) depending on said first class (y1) and on said second class (y2), wherein the robustness value (vu) is set to a first value, for example "1", indicating a vulnerability, if said first class (y1) is not equal to said second class (y2), and the robustness value (vu) is set to a second value, for example "0", indicating a non-vulnerability, if said first class (y1) is equal to said second class (y2);

wherein the method for obtaining the adversarial input image (x^adv) to the image classifier is obtained from an original input image (x^org), and wherein said adversarial input image and said original input image cause the image classifier (60) to classify said original input image (x^org) as belonging to a first class (ℓ₀) and said adversarial input image (x^adv) as belonging to a second class (ℓ), comprising the steps of:
- modifying said original input image (x^org) to yield a modified input image (x^mod);

- projecting said modified input image (x^mod) onto a metric ball around said original input image (x^org) to yield a projected input image (x^proj); and

- obtaining said adversarial input image (x^adv) depending on said projected input image (x^proj),

characterized in that

said metric is an at least approximate Wasserstein distance (W_D , $W_{D}^{λ}$
) and that said at least approximate Wasserstein distance is a Sinkhorn distance ( $W_{D}^{λ}$
) which differs from said Wasserstein distance (W_D ) by an entropic term (E_T ), and for any pair of first distribution (P) and second distribution (Q), said entropic term (E_T ) characterizes an entropy of a distribution Π that satisfies Π1 _n =P, Π^T 1 _n =Q.
The method according to claim 1, wherein said projected input image (x^proj) is determined by minimizing a distance to said modified input image (x^mod) under a constraint that a distance, according to said at least approximate Wasserstein distance (W_D, $W_{D}^{λ}$
) is not larger than a predefined radius (ε) of said metric ball.
The method according to claim 2, wherein said minimization is obtained by maximizing a dual problem corresponding to a primal problem that is given by said minimization under said constraints.
The method according to claim 1 or 2, wherein said projected input image (x^proj) is determined by solving a convex optimization corresponding to said minimization.
The method according to any one of the above claims, wherein said image classifier (60), when provided with an input image (x), is configured to output a first classification value (f _{l 0}) corresponding to said first class (ℓ₀) and a second classification value (f_l ) corresponding to said a predefined second class (ℓ), and wherein said modified input image (x^mod) causes a difference (g) between said first classification value (f _{l 0}) and said second classification value (f_l ) to be smaller than the difference (g) caused by said original input image (x^org).
The method according to any one of claims 1 to 5, wherein said image classifier (60), when provided with an input image (x), is configured to output a first classification value (f _{l 0}) corresponding to said first class (ℓ₀), and wherein said modified input image (x^mod) causes said first classification value (f _{l 0}) to be smaller than said first classification value (f _{l 0}) caused by said original input image (x^org).
The method according to any one of the above claims, wherein the steps of modifying said original input image (x^org) to yield said modified input image (x^mod) and projecting said modified input image (x^mod) onto said predefined subset to yield said projected input image (x^proj) are carried out iteratively by using said projected input image (x^proj) of a preceding iteration as original input image (x^org) a subsequent iteration, wherein said step projecting said modified input image (x^mod) onto said predefined subset is carried out after each step of modifying said original input image (x^org).
A computer-implemented method for training an image classifier (60), wherein the classifier (60) detects where objects are located in a vicinity of an at least partially autonomous robot from data received from one or more video sensors or the classifier (60) determines a state of a manufactured product (12) from captured properties by an optical sensor which captures properties of the manufactured product (12), comprising the steps of:
- accessing, from a memory (146), the image classifier (60), the image classifier (60) having been trained using a plurality of training input signals, the training images being labeled for a plurality of classes;

- receiving a sensor signal (S) comprising data from the one or more video sensors or from the optical sensor (30),

- determining an original input image (X_org) which depends on said sensor signal (S),

- determining an adversarial input image (X_adv) with a method of obtaining an adversarial input image (X_adv) to the image classifier (60), wherein the method for obtaining the adversarial input image (X_adv) to the image classifier is obtained from an original input image (X_org), and wherein said adversarial input image (X_adv) and said original input image (X_org) cause the image classifier (60) to classify said original input image (X_org) as belonging to a first class (ℓ_o) and said adversarial input image (X_adv) as belonging to a second class (ℓ) different from said first class (ℓ_o), comprising the steps of:

- modifying said original input image (X_org) to yield a modified input image (X_mod);

- projecting said modified input image (X_mod) onto a metric ball around said original input image (X_org) to yield a projected input image (X_proj); and

- obtaining said adversarial input image (X_adv) depending on said projected input image (X_proj), characterized in that said metric is an at least approximate Wasserstein distance (W_Dλ, W_D ^λ) and that said at least approximate Wasserstein distance is a Sinkhorn distance a (W_D ^λ) which differs from said Wasserstein distance (W_D ) by an entropic term (E_T), and for any pair of first distribution (P) and second distribution (Q), said entropic term (E_T) characterizes an entropy of a distribution Π that satisfies Π1_n =P, Π^T 1_n =Q;

- further training the image classifier (60) to have improved accuracy using at least the adversarial input image (x^adv).
A computer-implemented method for providing an actuator control signal (A) for controlling an actuator (10) depending on an output signal (y) of an image classifier (60), comprising the steps of:
- assessing whether said image classifier (60) is robust or not using the method according to claim 1 to 7, and determining said actuator control signal (A) in accordance with a result of said assessment, in particular by determining said actuator control signal (A) to cause said actuator (10) to operate in a safe mode if said image classifier (60) is deemed not robust as a result of said assessment.
The method according to claim 9, in which said actuator (10) controls an at least partially autonomous robot (100) and/or a manufacturing machine (200)).
Computer program that is configured to cause a computer to carry out the method according to any one of claims 1 to 10 with all of its steps if the computer program is carried out by a processor (45, 145).
Machine-readable storage medium (46, 146) on which the computer program according to claim 11 is stored.
System (140) that is configured to carry out the method according to claim 1 to 10.