10000 feature selection: remove duplicated features · Issue #114 · feature-engine/feature_engine · GitHub
[go: up one dir, main page]
Skip to content

feature selection: remove duplicated features #114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
solegalli opened this issue Sep 9, 2020 · 7 comments
Closed

feature selection: remove duplicated features #114

solegalli opened this issue Sep 9, 2020 · 7 comments
Labels
new transformer New feature or request
Milestone
Feature Selection

Comments

@solegalli
Copy link
Collaborator 
# check for duplicated features in the training set

# create and empty dictionary, where we will store 
# the duplicate groups
duplicated_feat_pairs = {}

# create an empty list to collect features
# that were found to be duplicated
_duplicated_feat = []


# iterate over every feature in our dataset:
for i in range(0, len(X_train.columns)):
    
    # this bit helps me understand where the loop is at:
    if i % 10 == 0:  
        print(i)
    
    # choose 1 feature:
    feat_1 = X_train.columns[i]
    
    # check if this feature has already been identified
    # as a duplicate of another one. If it was, it should be stored in
    # our _duplicated_feat list.
    
    # If this feature was already identified as a duplicate, we skip it, if
    # it has not yet been identified as a duplicate, then we proceed:
    if feat_1 not in _duplicated_feat:
    
        # create an empty list as an entry for this feature in the dictionary:
        duplicated_feat_pairs[feat_1] = []

        # now, iterate over the remaining features of the dataset:
        for feat_2 in X_train.columns[i + 1:]:

            # check if this second feature is identical to the first one
            if X_train[feat_1].equals(X_train[feat_2]):

                # if it is identical, append it to the list in the dictionary
                duplicated_feat_pairs[feat_1].append(feat_2)
                
                # and appended to our monitor list for duplicated variables
                _duplicated_feat.append(feat_2)
                
                # done! and run!

X_train = X_train[duplicated_feat_pairs.keys()]
X_test = X_test[duplicated_feat_pairs.keys()]
@Tejash-Shah
Copy link
Contributor

Hi @solegalli, I was wondering if anyone is working on this one? I'm willing to work on this one if you are ok...

@solegalli
Copy link
Collaborator Author

@Tejash-Shah you are more than welcome!! A few things to consider:

  1. please checkout the develop branch (not master) and make a PR to develop with your changes. Reason is, I am currently re-organising the files into submodules to improve readability and remove technical debt. So master is not the latest view of the project.

  2. Instead of the loops that I show in this issue, we should use itertools. I've learned recently that is much faster than the loops and better coding practice. Could you have a look at that as well?

Thanks a lot and look forward to working together again :)

@Tejash-Shah
Copy link
Contributor

Yes, that was my idea of using itertools where we can create feature pairs and iterate over-pairs. Please check the below screenshot for the implementation. However, only after writing tests, we can be assured about it, below is just a screenshot of my idea. Also, let me know if you were thinking about another implementation using itertools, would be interesting to know. Thank you :)

Screen Shot 2020-10-01 at 11 34 45 PM
Screen Shot 2020-10-01 at 11 35 02 PM
Screen Shot 2020-10-01 at 11 35 08 PM

@solegalli
Copy link
Collaborator Author

@Tejash-Shah looks good! look forward to the PR.

@solegalli solegalli mentioned this issue Oct 2, 2020
Closed
@solegalli solegalli added new transformer New feature or request help wanted labels Oct 2, 2020
@Tejash-Shah
Copy link
Contributor
Tejash-Shah commented Oct 3, 2020 •
edited
Loading

Hi @solegalli is the develop branch has an error in stylechecks? Tox is throwing off error at

feature_engine/base_transformers.py:40:89: E501 line too long (101 > 88 characters)
ERROR: InvocationError for command 'C:\hide_path\feature_engine\.tox\unit_tests\Scripts\flake8.EXE' feature_engine/base_transformers.py (exited with code 1)

EDIT:
Just saw CircleCI: test_style is failing at same place. Let me know if still I can push to develop branch? Thanks.

@solegalli
Copy link
Collaborator Author

yes, my bad. I need to fix that.

Please ignore that error for the moment, and push as usual. I will let you know when I fixed it. Sorry :(

@solegalli
Copy link
Collaborator Author

@Tejash-Shah fixed

@Tejash-Shah Tejash-Shah mentioned this issue Oct 3, 2020
Merged
1 task
solegalli added a commit that referenced this issue Oct 9, 2020
@Tejash-Shah @solegalli
Drop duplicate features #114 (#144)
5c632bd 
* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* create drop duplicate transformer

* delete extra fixture

Co-authored-by: Soledad Galli <solegalli@protonmail.com>
solegalli added a commit that referenced this issue Oct 9, 2020
@Tejash-Shah @solegalli
Drop duplicate features #114 (#144)
4628d84 
* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* create drop duplicate transformer

* delete extra fixture

Co-authored-by: Soledad Galli <solegalli@protonmail.com>
solegalli added a commit that referenced this issue Oct 9, 2020
@solegalli @NicoGalli @Tejash-Shah
reformat code style with black (#153)
1a4ab46 
* style formatting base scripts

* reformat style creation modules

* reformat style discretisers

* reformat style imputers

* rewords strings, minor changes

* reformat codestyle outliers

* reformat code style selection

* reformat style transformers

* reformat style wrappers

* separate woe and ratio encoder closes issue  #143 (#149)

* Issue 143

* doc issue 143

documentation  issue 143

* Update PRatioEncoder.rst

'C' Removed from  RareLabelCEncoder whch was causing and error. Also enconder_dict_ ratio results added

* Changes in docstrings #143

Changes requested by Sole in docstrings, after first pull request related to this issue.

* More docstring changes related to #143

minor updates in docstrings

* separate woe and ratio tests into functions #147

separate woe and ratio tests into functions #147

* move PRatioEncoder under WoEencoder in list

as these are related transformers

* add ' to log_ratio in docstring

* fix docstring with intro to transformer funcion

* add more detail about the encoding in docstrings

* renamed test functions

* reword tests

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reorganised folders with jupyter notebooks (#155)

* Drop duplicate features #114 (#144)

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* create drop duplicate transformer

* delete extra fixture

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reformat code style encoders

* shorten dosctrings with flake8

Co-authored-by: NicoGalli <72278140+NicoGalli@users.noreply.github.com>
Co-authored-by: Tejash Shah <stejash15@gmail.com>
8000 solegalli added a commit that referenced this issue Oct 9, 2020
@Tejash-Shah @solegalli
Drop duplicate features #114 (#144)
4af3a5b 
* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* create drop duplicate transformer

* delete extra fixture

Co-authored-by: Soledad Galli <solegalli@protonmail.com>
solegalli added a commit that referenced this issue Oct 9, 2020
@solegalli @NicoGalli @Tejash-Shah
reformat code style with black (#153)
48bdf41 
* style formatting base scripts

* reformat style creation modules

* reformat style discretisers

* reformat style imputers

* rewords strings, minor changes

* reformat codestyle outliers

* reformat code style selection

* reformat style transformers

* reformat style wrappers

* separate woe and ratio encoder closes issue  #143 (#149)

* Issue 143

* doc issue 143

documentation  issue 143

* Update PRatioEncoder.rst

'C' Removed from  RareLabelCEncoder whch was causing and error. Also enconder_dict_ ratio results added

* Changes in docstrings #143

Changes requested by Sole in docstrings, after first pull request related to this issue.

* More docstring changes related to #143

minor updates in docstrings

* separate woe and ratio tests into functions #147

separate woe and ratio tests into functions #147

* move PRatioEncoder under WoEencoder in list

as these are related transformers

* add ' to log_ratio in docstring

* fix docstring with intro to transformer funcion

* add more detail about the encoding in docstrings

* renamed test functions

* reword tests

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reorganised folders with jupyter notebooks (#155)

* Drop duplicate features #114 (#144)

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* create drop duplicate transformer

* delete extra fixture

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reformat code style encoders

* shorten dosctrings with flake8

Co-authored-by: NicoGalli <72278140+NicoGalli@users.noreply.github.com>
Co-authored-by: Tejash Shah <stejash15@gmail.com>
@solegalli solegalli added this to the Feature Selection milestone Oct 9, 2020
solegalli added a commit that referenced this issue Oct 20, 2020
@solegalli @NicoGalli
@Tejash-Shah @Okroshiashvili
refactor code v1.0.0 (#128)
e40457c 
* sort modules in subfolders, separate classes in individual modules, rename encoding classes, rename few other classes

* fix WoE name in doc

* sort selection in subfolder, separate class into modules

* fix selection imports

* update test_selection

* update VERSION to 1.0.0

* match doc name with submodule name, fix imports and class names (#129)

* reorganise test submodules as per package submodules, create individual test files for each class (#130)

* rename class init params and defo values (#131)

* reorganise base transformers and functions, clean imports (#133)

* fix style check (#141)

* improve code style throughout (#142)

* remove iterable from init MathematicalCombination

* replace listcomp by genexp in arbitrary discretiser

* create abstraction decision tree discretiser

* create abstraction in base cat encoder

* replace listcomp by genexp

* create abstraction base numerical imputer

* create baseOutlier

* final cleanup of code

* replace listcomp by genexp in randomsampleimputer

* split tests into individual functions (#147)

* split imputation tests

* fix typo, unify test names in math combination test

* reformat line space in sklearwrapper test

* remove comment in drop constant tests

* split discretisation tests

* split encoding tests

* split test outliers

* split transformer tests

* separate woe and ratio encoder closes issue  #143 (#149)

* Issue 143

* doc issue 143

documentation  issue 143

* Update PRatioEncoder.rst

'C' Removed from  RareLabelCEncoder whch was causing and error. Also enconder_dict_ ratio results added

* Changes in docstrings #143

Changes requested by Sole in docstrings, after first pull request related to this issue.

* More docstring changes related to #143

minor updates in docstrings

* separate woe and ratio tests into functions #147

separate woe and ratio tests into functions #147

* move PRatioEncoder under WoEencoder in list

as these are related transformers

* add ' to log_ratio in docstring

* fix docstring with intro to transformer funcion

* add more detail about the encoding in docstrings

* renamed test functions

* reword tests

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reorganised folders with jupyter notebooks (#155)

* Drop duplicate features #114 (#144)

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* create drop duplicate transformer

* delete extra fixture

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reformat code style with black (#153)

* style formatting base scripts

* reformat style creation modules

* reformat style discretisers

* reformat style imputers

* rewords strings, minor changes

* reformat codestyle outliers

* reformat code style selection

* reformat style transformers

* reformat style wrappers

* separate woe and ratio encoder closes issue  #143 (#149)

* Issue 143

* doc issue 143

documentation  issue 143

* Update PRatioEncoder.rst

'C' Removed from  RareLabelCEncoder whch was causing and error. Also enconder_dict_ ratio results added

* Changes in docstrings #143

Changes requested by Sole in docstrings, after first pull request related to this issue.

* More docstring changes related to #143

minor updates in docstrings

* separate woe and ratio tests into functions #147

separate woe and ratio tests into functions #147

* move PRatioEncoder under WoEencoder in list

as these are related transformers

* add ' to log_ratio in docstring

* fix docstring with intro to transformer funcion

* add more detail about the encoding in docstrings

* renamed test functions

* reword tests

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reorganised folders with jupyter notebooks (#155)

* Drop duplicate features #114 (#144)

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* add DropDuplicateFeatures in init

* add fixture for duplicate features

* add DropDuplicateFeatures functionality

* add test for DropDuplicateFeatures

* create drop duplicate transformer

* delete extra fixture

Co-authored-by: Soledad Galli <solegalli@protonmail.com>

* reformat code style encoders

* shorten dosctrings with flake8

Co-authored-by: NicoGalli <72278140+NicoGalli@users.noreply.github.com>
Co-authored-by: Tejash Shah <stejash15@gmail.com>

* reformat test code style (#157)

* expand style check in tox, add black to test_req (#154)

* update docs, shorten lines, test code (#158)

* update docs, shorten lines, test code

* include v1.0.0 changes in changelog

* fix linebreaks in changelog

* Add type hints, docstrings, and expand test. Introduce bug fix in _define_variables (#159)

* Add .vscode in gitignore

* add type hints and docstrings

* Add type hints, docstrings, and introduce bug fix

* add type hints and docstrings

* add type hints, docstrings, and stylistic modifications

* add type hints, docstrings, and stylistic modifications

* some stylistic modifications

* add test to check null values in dataframe

* remove redundant docstring

* stylistic modification

* add new test and expanded existing one

* fix flake8 suggestions

* fix some indentation errors

* fix indention error

* add type hints and docstrings in boxcox transformer

* add type hints and docstrings in log transformation

* add type hints, docstrings, and recaftor constructor in power transformation

* add type hints and docstrings in reciprocal transformer

* add type hints

* add type hints and docstrings

* add test case to test the type of math operation argument

* remove extra blank line

* add test cases for parameter_checks

* black it files

* update type hints and docstring

* black them and update type hints and docstrings

* update type hints and docstrings

* black it

* update type hints

* tidy imports

Co-authored-by: NicoGalli <72278140+NicoGalli@users.noreply.github.com>
Co-authored-by: Tejash Shah <stejash15@gmail.com>
Co-authored-by: Nodar Okroshiashvili <n.okroshiashvili@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
new transformer New feature or request
Projects
None yet
Milestone
Feature Selection
Development

No branches or pull requests
2 participants
@solegalli @Tejash-Shah
0