8000 [MRG+1] Add text vectorizers benchmarks by rth · Pull Request #9086 · scikit-learn/scikit-learn · GitHub
[go: up one dir, main page]
Skip to content

[MRG+1] Add text vectorizers benchmarks #9086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jun 28, 2017
Merged

[MRG+1] Add text vectorizers benchmarks #9086

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
455c287
Add benchmark_text_vectorizers.py
rth Jun 9, 2017
fcdec51
Add encoding information and remove comment
rth Jun 11, 2017
ce66d1e
Review comments
rth Jun 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 76 additions & 0 deletions benchmarks/bench_text_vectorizers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
"""

To run this benchmark, you will need,

* scikit-learn
* pandas
* memory_profiler
* psutil (optional, but recommended)

"""

from __future__ import print_function

import timeit
import itertools

import numpy as np
import pandas as pd
from memory_profiler import memory_usage

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import (CountVectorizer, TfidfVectorizer,
HashingVectorizer)

n_repeat = 3


def run_vectorizer(Vectorizer, X, **params):
def f():
vect = Vectorizer(**params)
vect.fit_transform(X)
return f


text = fetch_20newsgroups(subset='train').data

print("="*80 + '\n#' + " Text vectorizers benchmark" + '\n' + '='*80 + '\n')
print("Using a subset of the 20 newsrgoups dataset ({} documents)."
.format(len(text)))
print("This benchmarks runs in ~20 min ...")

res = []

for Vectorizer, (analyzer, ngram_range) in itertools.product(
[CountVectorizer, TfidfVectorizer, HashingVectorizer],
[('word', (1, 1)),
('word', (1, 2)),
('word', (1, 4)),
('char', (4, 4)),
('char_wb', (4, 4))
]):

bench = {'vectorizer': Vectorizer.__name__}
params = {'analyzer': analyzer, 'ngram_range': ngram_range}
bench.update(params)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What am I missing, why not simply

bench = {
    'vectorizer': ...,
    'analyzer': ...,
    'ngram_range': ...,
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bench are results, params are input parameters (so I can do e.g. run_vectorizer(Vectorizer, text, **params)), couldn't find a way to use a single dictionary for both here...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right

dt = timeit.repeat(run_vectorizer(Vectorizer, text, **params),
number=1,
repeat=n_repeat)
bench['time'] = "{:.2f} (+-{:.2f})".format(np.mean(dt), np.std(dt))

mem_usage = memory_usage(run_vectorizer(Vectorizer, text, **params))

bench['memory'] = "{:.1f}".format(np.max(mem_usage))

res.append(bench)


df = pd.DataFrame(res).set_index(['analyzer', 'ngram_range', 'vectorizer'])

print('\n========== Run time performance (sec) ===========\n')
print('Computing the mean and the standard deviation '
'of the run time over {} runs...\n'.format(n_repeat))
print(df['time'].unstack(level=-1))

print('\n=============== Memory usage (MB) ===============\n')
print(df['memory'].unstack(level=-1))
0