Boost Haar feature calculation for meeting training and real-time inference request #4399
Conversation
2. Propose new Pytest procedure for Haar feature 3. Correct Haar feature bug
|
Hello @WeiChungChang! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found: There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻 Comment last updated at 2020-09-03 03:54:49 UTC |
| h_l = self.__pad_h - basic_dim[0] | ||
| w_l = self.__pad_w - basic_dim[1] | ||
| if direction == 'h' or direction == 'd': | ||
| basic_fmap = |
There was a problem hiding this comment.
the syntax is invalid here (missing parenthesis or line continuation), causing the tests to fail.
There was a problem hiding this comment.
Hey just a few small comments.
Looks like a large amount of code. Are you in a position to give scikit-image a license or will we always have to attribute Amazon?
(Sigh, github seems to wait until all comments are completed. on the phone it isn't always obivous that this is happening, sorry for not pushing send on this before.)
skimage/feature/haar_fast.py
Outdated
| } | ||
|
|
||
| def pad_zeros_to_imgs(imgs): | ||
| l= len(imgs.shape) |
|
|
||
| def integral_array_horizontal(imgs): | ||
| l= len(imgs.shape) | ||
| r = np.zeros(imgs.shape) |
There was a problem hiding this comment.
This declaration is actually never used
skimage/feature/haar_fast.py
Outdated
| def integral_array_vertical(imgs): | ||
| l= len(imgs.shape) | ||
| r = np.zeros(imgs.shape) | ||
| if l == 3 or l == 4: # "NHW" or "NHWC" |
There was a problem hiding this comment.
Again, this allocation is never used
skimage/feature/haar_fast.py
Outdated
|
|
||
| def __init__(self, label): | ||
| self.__children = [] | ||
| self.__start = 0 |
There was a problem hiding this comment.
I think double underscores have done name mangling features l. Do you want to use them? If not, please use single underscore
| b3_rb = [coord[0] + h - 1, coord[1] + w - 1] | ||
| return [[b1_lt, b1_rb], [b2_lt, b2_rb], [b3_lt, b3_rb]] | ||
|
|
||
| def __cal_type_4_coord(coord, assem_pat): |
There was a problem hiding this comment.
Is this supposed to be a static method? If not, why not use the self name?
| @@ -0,0 +1,232 @@ | |||
| # File name: test_haar_new.py | |||
| # Contributor: Amazon Lab126 Multimedia team | |||
There was a problem hiding this comment.
Are these tests compatible with the old haar features? If so, maybe it makes sense to have them in a seperate PR so as to enable a distinc review for these additions?
Description
[Purpose]
[Summarize current issues]
We summarize current issues here first.
1.as fix boundary condition when calculating Haar features #4076
2.as sample code shows.
3.Currently pytest only has limited & fixed test patterns.
1.as 1.2.
2.too slow when inference.
1.too slow when train.
We give a classification of how Haar feature will be applied for common use cases.
For inference, assume we have trained a model to detect human face by detection window = 24 * 24.
Given an image, ex 640*480, we will
This figure shows the original image and the places having strong response for first stage.
This figure shows the candidate windows after the first stage. The black holes have been filtered out as they are possibly NOT as human faces.
This figure shows typical inference flow of cascaded classifiers when doing inference.
Assume we have trained a set of features which work well.
At 1st(initial) stage, we apply the set of features to all detection window. The purpose is to discard as many regions where seems to have no faces within it as possible.
At 2nd(refine) stage, the inputs now are blocks survive from initial classifier. We apply the set of features within it to refine the result.
The procedures keep on going until we have reached the output (final) classifier where all identified results are output.
Current implementation is good for refine classifiers with few blocks but very slow and hard to be used at initial classifiers. Also, Current implementation has bug in detection window.
To understand why "current implementation has bug in detection window."
Please refer to the sample code here:
the result shows:
Notice that the answer should be [ 1 0 1 0 -1 -1].
However by applying detection window, the result is [1, 0, -1, 0, -4, -4]
The problem is with offset (x=1, y=0), current flow calculates result from partial integral image as below.
[3, 6]
[8, 12]
The partial integral image with wrong padding will create wrong answer.
please check the figure above; for 1st detection window it is correct.
For 2nd detection window, notice current padding values are of all zeros. However, we should pad with the actual neighboring values accordingly.
[Algorithm description]
There are two algorithms here to boost speed.
1. apply numpy matrix operation to replace calculate individual feature respectively.
2. apply the algorithm to assemble feature map to save calculation.
We explain the algorithms in this section.
This figure outlines in this proposal where the speedup comes from.
Reference: https://californiaconsultants.org/wp-content/uploads/2018/04/CNSV-1806-Patterson.pdf
page 37
This figure shows the fast algorithm of:
[Performance]
The experiment shows ~ x50 speedup compared to current method for calculating all features of a single frame by 25x25.
The experiment shows ~ x250 speedup compared to current method for calculating all features of a batch consists of 10 frames by 25x25.
The experiment shows ~ x1000 speedup compared to current method for calculating all features of a batch consists of 500 frames by 25x25.
For training with large batch (typically training need more than ten thousands images), it implies, originally we need 1~2 days to complete the calculation of all features but now can be done within 10 minutes.
[To be done]
Checklist
./doc/examples(new features only)./benchmarks, if your changes aren't covered by anexisting benchmark
For reviewers
later.
__init__.py.doc/release/release_dev.rst.@meeseeksdev backport to v0.14.x