8000 ENH: Add encoding option to numpy text IO by juliantaylor · Pull Request #4208 · numpy/numpy · GitHub
[go: up one dir, main page]
Skip to content

ENH: Add encoding option to numpy text IO #4208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 27 commits into from
Closed

ENH: Add encoding option to numpy text IO #4208

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
298748a
attempt to salvage loadtxt and genfromtxt
juliantaylor Apr 3, 2017
2c821c0
update some internal tests
juliantaylor Apr 4, 2017
749dbe7
update datasource
juliantaylor Apr 8, 2017
e6fceb7
less ugly dtype=S hack
juliantaylor Apr 8, 2017
0d066e8
use special value encoding="bytes" for converter value
juliantaylor Apr 8, 2017
49d143d
windows test fix
juliantaylor Apr 8, 2017
3fb3794
add gzip line separator test
juliantaylor Apr 8, 2017
c612568
try to keep genfromtxt backward compat
juliantaylor Apr 8, 2017
1f44b3c
add decoding converter tests
juliantaylor Apr 8, 2017
03e827d
support unicode user dtype in genfromtxt
juliantaylor Apr 8, 2017
9cdd203
add unicode support to fromregexp
juliantaylor Apr 9, 2017
760c1a1
add docstrings
juliantaylor Apr 9, 2017
dcfc2c7
move line decoding to a function
juliantaylor Apr 14, 2017
1b23544
add binary stream decode test
juliantaylor Apr 14, 2017
67cd094
only use writewrap in savetxt when necessary
juliantaylor Apr 14, 2017
38fa80f
don't convert data to lists unnecessarily in genfromtxt
juliantaylor Apr 14, 2017
06725fd
avoid the file encoding workaround if encoding is provided by user
juliantaylor Apr 14, 2017
eea935d
add xz support and add tests
juliantaylor Apr 14, 2017
7f0d6f7
cleanup compressed file handling in datasource
juliantaylor Apr 16, 2017
088f4b3
remove two now unnecessary abstractions
juliantaylor Apr 16, 2017
097f7c0
fix encoding argument not being passed to Linesplitter
juliantaylor Apr 16, 2017
053449d
move decoding into Linesplitter's handyman function
juliantaylor Apr 16, 2017
3aba208
cleanup
juliantaylor Apr 16, 2017
bec193e
ENH: change loadtxt to use a generator to load data
juliantaylor Apr 16, 2017
1fe69f3
DOC: add release notes for text IO changes
juliantaylor Jul 11, 2017
e9ae400
DEPR: add a deprecation warning when reading strings without encoding
juliantaylor Jul 11, 2017
c482a5b
add test for savetxt into StringIO
juliantaylor Jul 11, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix encoding argument not being passed to Linesplitter
  • Loading branch information
@juliantaylor
juliantaylor committed Nov 6, 2017
commit 097f7c0d9c5aecc31a4e1f7d9399b5e21b30247f
2 changes: 1 addition & 1 deletion numpy/lib/npyio.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1651,7 +1651,7 @@ def genfromtxt(fname, dtype=float, comments='#', delimiter=None,
"or generator. Got %s instead." % type(fname))

split_line = LineSplitter(delimiter=delimiter, comments=comments,
autostrip=autostrip)._handyman
autostrip=autostrip, encoding=encoding)._handyman
validate_names = NameValidator(excludelist=excludelist,
deletechars=deletechars,
case_sensitive=case_sensitive,
Expand Down
6 changes: 6 additions & 0 deletions numpy/lib/tests/test_io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1867,6 +1867,12 @@ def test_latin1(self):
assert_equal(test[1, 0], b"test1")
assert_equal(test[1, 1], b"testNonethe" + latin1)
assert_equal(test[1, 2], b"test3")
test = np.genfromtxt(TextIO(s),
dtype=None, comments=None, delimiter=',',
encoding='latin1')
assert_equal(test[1, 0], u"test1")
assert_equal(test[1, 1], u"testNonethe" + latin1.decode('latin1'))
assert_equal(test[1, 2], u"test3")

test = np.genfromtxt(TextIO(b"0,testNonethe" + latin1),
dtype=None, comments=None, delimiter=',')
Expand Down
0