[WIP] REFACTOR: dict_learning and dict_learning_online #9036

dohmatob · 2017-06-07T11:20:55Z

Addresses issue #9009.

Proposal

Have a single function dict_learning_online(batch_size=some_default) which reproduces full-batch mode (the current dict_learning function) when batch_size == n_samples.

dohmatob · 2017-06-07T11:22:13Z

@agramfort: could you review this ? Thanks.

vene · 2017-06-07T12:57:51Z

I'm looking at this, unfortunately I have zero memory of the old code, I need to re-read it and figure things out.

Depending on how simple you think the changeset is, we might want to split it into the projection bugfix and the refactoring.

vene · 2017-06-07T12:59:16Z

sklearn/decomposition/dict_learning.py

        Data matrix.

-    code : array of shape (n_components, n_samples)
+    A : array of shape (n_components, n_components)


Was the old version of this just plain wrong?

nevermind, I see what is happening. This needs some documentation in terms of what this function is solving

No, sorry, I don't understand the math in here. Would you mind explaining the changes a bit? The shapes are not the same as before. If the math does the same thing, it is not obvious to me why.

Could you please be precise about what part of the computations you don't understand. Is it normalization by A[k, k], etc ?

Also, in a moment I'll push a modified docstring with more detailed comments in it.

I added a more detailed comment below after thinking about it. No, the normalization makes sense to me. It just seems like the old computation does not correspond to what is in the paper.

(to be clear, the old code is what is confusing here, not yours!)

Ya, indeed it looks like the old _dict_update code is incorrect in the online case, due to the extra dictionary[:, k] = np.dot(R, A[k, :].T) on line 357.

does A denote atoms? if so can we just have it as atoms?

vene · 2017-06-07T14:08:56Z

sklearn/decomposition/dict_learning.py

+            dictionary[:, k] /= A[k, k]
+        _proj_l2(dictionary[:, k])
+        # R <- -1.0 * U_k * V_k^T + R
+        R = ger(-1.0, dictionary[:, k], A[k, :], a=R, overwrite_a=True)


This went down one level of indentation, is it correct?

It's intentional. The old indentation appeared incorrect to me: it would lead to incorrect modifications (at least unjustified) in R, whenever a random atom was inserted into the dictionary (in case the old one was close to zero). Agreed ?

vene · 2017-06-07T14:13:30Z

sklearn/decomposition/dict_learning.py

-            # R <- -1.0 * U_k * V_k^T + R
-            R = ger(-1.0, dictionary[:, k], code[k, :], a=R, overwrite_a=True)
+            dictionary[:, k] /= A[k, k]
+        _proj_l2(dictionary[:, k])


Your fix of the l2 projection here agrees with eqn 10 in http://www.di.ens.fr/sierra/pdfs/icml09.pdf. Nice catch!

vene · 2017-06-07T15:32:43Z

sklearn/decomposition/dict_learning.py

-        R = ger(1.0, dictionary[:, k], code[k, :], a=R, overwrite_a=True)
-        dictionary[:, k] = np.dot(R, code[k, :].T)
+        R = ger(1.0, dictionary[:, k], A[k, :], a=R, overwrite_a=True)
+        dictionary[:, k] = np.dot(R, A[k, :])


I tried working through this on paper and I think the batch update ends up different by a factor of CCt.

The old implementation seems incorrect to me; your implementation corresponds to the online update in the paper, applied over the entire dataset.

Previously we had:

R = X - DCt # dimensions here are transposed compared to sklearn conventions D[:, k] = R * C[k] = (X - DCt) C[k]

Now, by forming A and B, we have

R = B - DA = XC - DCtC = (X - DCt)C D[:, k] = R Ak = R CtC[k] = (X - DCt) CCt C[k]

Am I missing something or was this indeed a bug previously?

I did a quick check and indeed I seem to get slightly different dictionary atoms before and after this PR with the batch version. They seem equal at the first decimal but differ at the second or the third, so they're close.

Your current implementation seems to match the paper correctly, at least in the online case. Unfortunately, the dict_learning_online function doesn't print the obj value at each iteration, so I can't compare that easily, but for the old implementation the objective does seem to decrease at least.

even with n_iter=10000 and tol=1e-20 a difference persists. I computed the residual by hand with

import numpy as np rng = np.random.RandomState(0) X = rng.randn(30, 20) U, V, _ = dict_learning(X, n_components=3, alpha=0.1, random_state=0, verbose=2, max_iter=10000, tol=1e-20) print(0.5 * np.sum((X - np.dot(U, V)) ** 2) + 0.1 * np.sum(np.abs(U)))

On master I get 188.766637, and on the current branch I get 188.785850, so slightly more...

Also it seems quite faster on master.

Ah, looks like i actually introduced a bug: the line dictionary[:, k] = np.dot(R, A[k, :]) shouldn't be there anymore. This additional buggy computation should account for some of the speed loss you're experiencing. Thanks for the catch. Fixed

BTW, this is what I'm doing:

Minimize .5 * ||Xt - DC||_F^2 = .5 * tr(DtDA) - * tr(DtB) + .5 * tr(XXt), as a function of the dictionary (V), where B = XtCt, A = CCt, R = B - DA. Note that the update of the jth atom is given by (see eqn 10 of ref paper below): BCD for kth atom: R = R + D[:, k]A[k, :] # rank-1 update D[:, k] = R[:, k] / A[k, k] # = D[:, k] + R[:, k] / A[k, k] D[:, k] = proj(D[:, k]) # eqn 10 R = R - D[:, k]A[k, :] # rank-1 update

I just checked with with your code (modified)

import time import numpy as np from sklearn.decomposition.dict_learning import dict_learning rng = np.random.RandomState(0) X = rng.randn(30, 20) t0 = time.time() U, V, _ = dict_learning(X, n_components=3, alpha=0.1, random_state=0, verbose=2, max_iter=10000, tol=1e-20) print(0.5 * np.sum((X - np.dot(U, V)) ** 2) + 0.1 * np.sum(np.abs(U))) print("Time elapsed: %.2f" % (time.time() - t0))

On master:

Iteration 647 (elapsed time: 16s, 0.3mn, current cost 198.777) Iteration 648 (elapsed time: 16s, 0.3mn, current cost 198.777) --- Convergence reached after 648 iterations 198.776663706 Time elapsed: 16.46

On PR:

--- Convergence reached after 647 iterations Learning code... done (total time: 10s, 0.2mn) 198.776663706 Time elapsed: 10.38

Thus the results (and trajectory of cost function, convergence point, etc.) are exactly the same, but my PR is faster than the master branch. Can you confirm. Thanks.

Confirm that your PR is faster and reaches the same error.

With tol=1e-20 the number of iterations needed for convergence fluctuates (both on master and here), but I think (although odd) it's a numerical issue. With reasonable tol 1e-6 the number of iterations is exactly the same. The returned U and V have the same norm as well, so I think you're right in saying the paths are identical.
master

[...] Iteration 166 (elapsed time: 3s, 0.1mn, current cost 198.781) --- Convergence reached after 166 iterations 198.780858248 Time elapsed: 3.13 U norm 14.4716684728 V norm 1.73205080757

here

[...] Iteration 150 (elapsed time: 1s, 0.0mn) --- Convergence reached after 166 iterations Learning code... done (total time: 1s, 0.0mn) 198.780765235 Time elapsed: 1.99 U norm 14.4722062044 V norm 1.73205080757

Wait, the dctionary is the same, but the code is a bit different now...

master

Iteration 1 (elapsed time: 0s, 0.0mn, current cost 279.778010084) Iteration 2 (elapsed time: 0s, 0.0mn, current cost 199.544246461) Iteration 3 (elapsed time: 0s, 0.0mn, current cost 199.522729734) [...] Iteration 165 (elapsed time: 2s, 0.0mn, current cost 198.781266815) Iteration 166 (elapsed time: 2s, 0.0mn, current cost 198.781055910) --- Convergence reached after 166 iterations 198.780858248 Time elapsed: 2.13 U norm 14.4716684728 V norm 1.73205080757

head

Iteration 0 (elapsed time: 0s, 0.0mn, current cost 0.000000000 Iteration 1 (elapsed time: 0s, 0.0mn, current cost 279.778014383 Iteration 2 (elapsed time: 0s, 0.0mn, current cost 199.544247482 Iteration 3 (elapsed time: 0s, 0.0mn, current cost 199.522729073 [...] Iteration 165 (elapsed time: 2s, 0.0mn, current cost 198.781266718 Iteration 166 (elapsed time: 2s, 0.0mn, current cost 198.781055820 --- Convergence reached after 166 iterations Learning code... done (total time: 2s, 0.0mn) 198.780765048 Time elapsed: 2.16 U norm 14.4722067347 V norm 1.73205080757 I think the paths are indeed the same up to numerical precision. Other than that, with the new version, the code is updated once more at the end. (previously, in the batch mode the dict was the last block updated.)

Hm, I'd expect things to be numerically a bit different due to at least one of the following

the modified projection formula

the indentation issue on the second call to ger(...)

I'd have proposed to open a separate issue on these things and then address them in a separate PR, but then this would be a really minor change (in terms of patch size). So i'd rather address them as part of this PR, except there are compelling reasons to do otherwise. What do you think ?

vene · 2017-06-08T06:14:24Z

sklearn/decomposition/dict_learning.py

+
+    .5 * ||Xt - DC||_F^2 = .5 * tr(DtDA) - 2 * tr(DtB) + tr(XXt),
+
+    as a function of the dictionary (V), where


the dictionary D?

We should get the variable names consistent throughout. I would not be opposed to using C/D for code/dictionary in the whole file, and B/A with the meanings from the paper.

vene · 2017-06-08T07:14:41Z

sklearn/decomposition/dict_learning.py

-            theta = float(batch_size ** 2 + ii + 1 - batch_size)
-        beta = (theta + 1 - batch_size) / (theta + 1)
-
+            beta = 0.
        A *= beta
        A += np.dot(this_code, this_code.T)
        B *= beta
        B += np.dot(this_X.T, this_code.T)


I think it's better to move the moving average update inside the if online case and do

else: A = np.dot(...) B = np.dot(...)

just because it's less subtle

vene · 2017-06-08T07:17:08Z

sklearn/decomposition/dict_learning.py

    else:
-        return dictionary.T
+        return dictionary.T, errors


unfortunately this is public API that we cannot break, we need to return the exact same signature as before. We can add a default argument (return_errors=False), but this logic is going to get messy

vene · 2017-06-08T07:21:51Z

sklearn/decomposition/dict_learning.py


-    if n_jobs == -1:
-        n_jobs = cpu_count()
+def _compute_residuals_from_code(X, V, U):


this helper is currently only called in one place, we could inline it

vene · 2017-06-08T07:58:44Z

sklearn/decomposition/dict_learning.py

-                         method='lars', iter_offset=0, random_state=None,
-                         return_inner_stats=False, inner_stats=None,
-                         return_n_iter=False):
+                         batch_size=None, verbose=False, shuffle=True,


batch_size should stay 3 by default, otherwise this breaks the public API.

And even beyond the fact that the API breaks, current behavior is counterintuitive: the function name is "dict_learning_online", but it does batch learning with the default arguments!

Agreed. Fixed.

vene · 2017-06-08T08:03:07Z

sklearn/decomposition/dict_learning.py

-                         return_inner_stats=False, inner_stats=None,
-                         return_n_iter=False):
+                         batch_size=None, verbose=False, shuffle=True,
+                         n_jobs=1, method='lars', iter_offset=0, tol=0.,


the new tol argument is unused in the online case, right? This is confusing from an API standpoint.

Maybe a better idea is to have a private function _dict_learning that does what you're currently doing here, and wrap it with dict_learning and dict_learning_online which implement the appropriate API.

Wouldn't this lead to a messier code base due to the proliferation of *dict_learning* functions ? Seems there might be a comprise to be made here.

A less invasive solution would be to enrich the documentation of the tol param, explaining that it only takes effect in batch mode.

functions starting with "_" are not documented and not visible to the user normally, so it will be just a way for us to organize the im F987 plementation.

I think that's preferable to having parameters that only work some of the time.

OK, fixed. "dict_learning_online" no longer has "tol" param. Now, the APIs should be as before.

vene · 2017-06-08T08:04:09Z

sklearn/decomposition/tests/test_dict_learning.py

@@ -13,6 +13,7 @@
 from sklearn.utils.testing import assert_raises
 from sklearn.utils.testing import ignore_warnings
 from sklearn.utils.testing import TempMemmap
+from sklearn.utils.testing import assert_almost_equal


vene · 2017-06-08T08:11:11Z

In batch mode the new implementation is nearly identical to the old, up to (I suspect) numerical precision and an extra fit of the code at the end.

In online mode there are differences. This is on the digits dataset:

code:

import time
import numpy as np
from sklearn.decomposition.dict_learning import dict_learning_online
from sklearn.datasets import load_digits

X, _ = load_digits(return_X_y=True)

t0 = time.time()
res = dict_learning_online(X, n_components=3, alpha=0.1, random_state=0,
                           batch_size=32)
U = res[0]
V = res[1]  # this ugly unpacking is needed because of the API differences in this PR

print(0.5 * np.sum((X - np.dot(U, V)) ** 2) + 0.1 * np.sum(np.abs(U)))
print("Time elapsed: %.2f" % (time.time() - t0))
print("U norm", np.linalg.norm(U, 'fro'))
print("V norm", np.linalg.norm(V, 'fro'))

this pr:
final obj 755227.180842
Time elapsed: 2.04
U norm 2391.27139755
V norm 1.73205080757

master:
756096.103391
Time elapsed: 2.23
U norm 2556.1895028
V norm 1.73205080757

@dohmatob, I remember you mentioned yesterday that the old online version might have been incorrect, can you double-check the math? If you are right and can write a test that fails on master but passes here, it will motivate the differences in results and justify merging.

dohmatob · 2017-06-08T08:58:28Z

OK, great. I'll look at it this afternoon.

dohmatob · 2017-06-08T11:30:51Z

I've double checked the math in the dict update on master, and indeed:

1- The batch mode is correct
2 - The online mode is wrong

Why there is an error on the master branch
From our previous discussion, it's clear that the dictionary update (without the recaling and projection steps):

R = R + D[:, k]A[k, :]
D[:, k] = R[:, k]
R = R - D[:, k]A[k, :]

where

R := B - DA, B := XCt, A := CCt

Now, if we rewrite this in terms of the input data X and the codes C, we get:

D[:, k] = (R - D[:, k]A[k, :])[:, k] = (B - DA + D[:, k]A[k, :])[:, k]
          = ((X - DC)Ct + D[:, k](CCt)[:, k])[:, k]
          = ((X-DC)Ct)[:, k] + (D[:, k](CCt)[:, k])[:, k]
          = (X-DC)C[k, :] + DCC[:, k]
          = (X - DC + D[:, k]C[k, :])C[:, k]
          = formula for dict update on master branch running batch mode

Why the bug doesn't manifest ?
Roughly, the "bug" is equivalent to post-multiplying the a (correctly updated) dictionary with an invertible linear transformation. This transformation get's scaled-away during the code update stage, since (D, C) and (DG, G^{-1}C) are equivalent DL solutions, for any invertible G. Thus the "bug" is absorbed. Unfortunately, there is practical F438 ly no numerical way (i.e tests) to reveal this issue.

dohmatob · 2017-06-08T11:34:08Z

BTW, i've fixed the "API unpacking" problem you complained about. The API dict_learning and dict_learning_online APIs should be same as before.

raghavrv · 2017-06-08T12:32:59Z

sklearn/decomposition/dict_learning.py

-                 random_state=None):
-    """Update the dense dictionary factor in place.
+def _proj_l2(v, copy=False):
+    """Projects v unto l2 unit ball in-place.


--> in-place (if copy=False) ;)

hm, that's what happens when docstring and func prototype follow different evolutions and become out-of-sync. Good catch. Thx :)

raghavrv · 2017-06-08T12:33:15Z

sklearn/decomposition/dict_learning.py

-def _update_dict(dictionary, Y, code, verbose=False, return_r2=False,
-                 random_state=None):
-    """Update the dense dictionary factor in place.
+def _proj_l2(v, copy=False):


can you rename it to _project_l2

raghavrv · 2017-06-08T12:35:52Z

sklearn/decomposition/dict_learning.py

+
+    B = XtCt, A = CCt, R = B - DA.
+
+    Note that the update of the jth atom is given by (see eqn 10 of


you mean kth atom?

ya, fixed. Thanks.

raghavrv · 2017-06-08T12:36:44Z

sklearn/decomposition/dict_learning.py

+    R = R + D[:, k]A[k, :]  # rank-1 update
+    D[:, k] = R[:, k] / A[k, k]  # = D[:, k] + R[k] / A[k, k]
+    D[:, k] = proj(D[:, k])  # eqn 10
+    R = R - D[:, k]A[k, :]  # rank-1 update


Can you add a note on what the different alphabets represent?

A, B are standard notation from the reference paper (and this notation is already used on master).
Please see eqn 11 of ref paper http://www.di.ens.fr/sierra/pdfs/icml09.pdf.

raghav makes a good point, we need to document A and B. No, A are not atoms, D[:, k] are the atoms.

Actually scratch that. This is already documented at line 332, @raghavrv, is it easy to miss?

we could rename dictionary to D in the entire file

Hm, how about instead making clear (by means of docstrings, comments, etc.) that D means dictionary (which is already the case). If we set out to rename D, we'd have to do something similar for code, this_code, etc. I think the labels D, C, etc. are only interesting when stating the formula being implemented (as done in the docstrings). But I may be wrong.

raghavrv · 2017-06-08T12:36:51Z

sklearn/decomposition/dict_learning.py

        Data matrix.

-    code : array of shape (n_components, n_samples)
+    A : array of shape (n_components, n_components)


does A denote atoms? if so can we just have it as atoms?

raghavrv · 2017-06-08T12:39:01Z

sklearn/decomposition/dict_learning.py

+    This function has two modes:
+
+    1. Batch mode, activated when batch_size is None or
+       batch_size >= n_sampels. This is the default.


n_samples

raghavrv · 2017-06-08T12:42:18Z

sklearn/decomposition/dict_learning.py

@@ -593,14 +582,18 @@ def dict_learning_online(X, n_components=2, alpha=1, n_iter=100,
    n_iter : int,
        Number of iterations to perform.

+    tol : float,
+        Tolerance for the stopping condition. Used in batch mode.
+
    return_code : boolean,
        Whether to also return the code U or just the dictionary V.

    dict_init : array of shape (n_components, n_features),
        Initial value for the dictionary for warm restart scenarios.

    callback :


callback : callable, optional default None

The docstring issue you'r pointing to already exists in master. Fixed anyways.

…gle function

vene · 2017-06-10T16:10:20Z

We could pose the bugfix as an optimization, because we avoid the spurious multiplication with a rotation matrix that was present before. Maybe this way @GaelVaroquaux will like it :)

vene · 2017-06-10T16:14:06Z

sklearn/decomposition/dict_learning.py

@@ -571,12 +577,19 @@ def dict_learning_online(X, n_components=2, alpha=1, n_iter=100,

        (U^*, V^*) = argmin 0.5 || X - U V ||_2^2 + alpha * || U ||_1
                     (U,V)
-                     with || V_k ||_2 = 1 for all  0 <= k < n_components
+                     with || V_k ||_2 <= 1 for all  0 <= k < n_components

    where V is the dictionary and U is the sparse code. This is


the U, V here is also inconsistent with D/C as used above, and with D/alpha as used in the paper... but consistent with other decompositions algos in sklearn. Maybe we should use U/V in the other docstrings and maybe in code too?

vene · 2017-06-10T16:14:23Z

sklearn/decomposition/dict_learning.py

@@ -600,7 +613,8 @@ def dict_learning_online(X, n_components=2, alpha=1, n_iter=100,
        Initial value for the dictionary for warm restart scenarios.

    callback :
-        Callable that gets invoked every five iterations.
+        Callable that gets invoked every five iterations. If it returns


it should be customizable how often it gets invoked.

This was the old logic. But yeah, I can add that to the PR.

vene · 2017-06-10T16:14:44Z

sklearn/decomposition/dict_learning.py

@@ -600,7 +613,8 @@ def dict_learning_online(X, n_components=2, alpha=1, n_iter=100,
        Initial value for the dictionary for warm restart scenarios.

    callback :
-        Callable that gets invoked every five iterations.
+        Callable that gets invoked every five iterations. If it returns
+        non-True, then the main loop (iteration on data) is aborted.


i guess the term is "falsish"?

haha, ya sure

dohmatob · 2017-06-11T10:12:04Z

@vene: the spurious multiplication bug that we found, came as a by-product after looking at the code. My main objective with this PR was / is to collapse dict_learning and dict_learning_online into a single function (as they really should be!) :) Along the road, this fixes the spurious multiplication bug too.

vene · 2017-06-12T07:23:30Z

sklearn/decomposition/dict_learning.py

-    R = -np.dot(dictionary, code)
-    R += Y
+    R = -np.dot(dictionary, A)
+    R += B


is this actually faster than R = B - np.dot(dictionary, A)? The latter is more readable imo

vene · 2017-06-12T07:25:41Z

sklearn/decomposition/dict_learning.py

    R = np.asfortranarray(R)
-    ger, = linalg.get_blas_funcs(('ger',), (dictionary, code))
+    ger, = linalg.get_blas_funcs(('ger',), (dictionary, A))
    for k in range(n_components):
        # R <- 1.0 * U_k * V_k^T + R


V_k -> A_k

U_k should be dictionary or the other way around, as long as consistent in the entire file

jeremiedbb · 2022-03-30T09:23:40Z

Some refactoring has been done in #18975. Now dict_learning_online calls MiniBatchDictionaryLearning instead of the opposite, so the proposed refactoring here is no longer relevant. Instead, the same refactoring should be done for DictionaryLearning. The fix for the loss has also been done in a separate PR. Closing

dohmatob changed the title ~~REFACTOR: dic_learning and dict_learning_online refactored into a sin…~~ REFACTOR: dict_learning and dict_learning_online Jun 7, 2017

dohmatob mentioned this pull request Jun 7, 2017

DictionaryLearning should support more general constraints on the components #8994

Open

vene reviewed Jun 7, 2017

View reviewed changes

dohmatob force-pushed the dict-update branch from 4d6cab8 to 208ac71 Compare June 7, 2017 15:00

vene reviewed Jun 7, 2017

View reviewed changes

dohmatob force-pushed the dict-update branch from 208ac71 to 420a54c Compare June 7, 2017 16:56

vene reviewed Jun 8, 2017

View reviewed changes

vene changed the title ~~REFACTOR: dict_learning and dict_learning_online~~ [WIP] REFACTOR: dict_learning and dict_learning_online Jun 8, 2017

dohmatob force-pushed the dict-update branch 2 times, most recently from e5dcffa to bb2cddc Compare June 8, 2017 08:51

raghavrv reviewed Jun 8, 2017

View reviewed changes

dohmatob force-pushed the dict-update branch 5 times, most recently from 584b955 to 5c39eb5 Compare June 8, 2017 13:23

dohmatob force-pushed the dict-update branch 8 times, most recently from c4cd31e to c4b1b28 Compare June 9, 2017 09:10

dohmatob mentioned this pull request Jun 9, 2017

It seams "dict_learning" doesn't use the "code_init" param at all #9081

Closed

dohmatob force-pushed the dict-update branch 5 times, most recently from 9339fc0 to c4a2297 Compare June 10, 2017 12:29

REFACTOR: dic_learning and dict_learning_online refactored into a sin…

6591716

…gle function

vene reviewed Jun 10, 2017

View reviewed changes

vene reviewed Jun 12, 2017

View reviewed changes

dohmatob force-pushed the dict-update branch from c4a2297 to 6591716 Compare February 2, 2018 01:08

amueller added the Stalled label Aug 5, 2019

github-actions bot added the module:decomposition label Mar 2, 2020

jeremiedbb mentioned this pull request Jan 19, 2021

FIX Make dict learning loss match the paper #19210

Merged

Base automatically changed from master to main January 22, 2021 10:49

jeremiedbb closed this Mar 30, 2022

jeremiedbb mentioned this pull request Mar 30, 2022

REFACTOR: dict_learning and dict_learning_online should be merged into a single func #9009

Closed


		.5 * \|\|Xt - DC\|\|_F^2 = .5 * tr(DtDA) - 2 * tr(DtB) + tr(XXt),

		as a function of the dictionary (V), where


		B = XtCt, A = CCt, R = B - DA.

		Note that the update of the jth atom is given by (see eqn 10 of

Uh oh!

[WIP] REFACTOR: dict_learning and dict_learning_online #9036

[WIP] REFACTOR: dict_learning and dict_learning_online #9036

Uh oh!

Conversation

Proposal

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!