8000 ENH: Allow compression in NDFrame.to_csv to be a dict with optional arguments (#26023) by drew-heenan · Pull Request #26024 · pandas-dev/pandas · GitHub
[go: up one dir, main page]

Skip to content

ENH: Allow compression in NDFrame.to_csv to be a dict with optional arguments (#26023) #26024

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Aug 26, 2019
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
4e73dc4
ENH/BUG: Add arcname to to_csv for ZIP compressed csv filename (#26023)
drew-heenan Apr 8, 2019
ab7620d
DOC: Updated docs for arcname in NDFrame.to_csv (#26023)
drew-heenan Apr 8, 2019
2e782f9
conform to line length limit
drew-heenan Apr 8, 2019
83e8834
Fixed test_to_csv_zip_arcname for Windows paths
drew-heenan Apr 8, 2019
d238878
Merge remote-tracking branch 'upstream/master' into issue-26023
drew-heenan Apr 8, 2019
b41be54
to_csv compression may now be dict with possible keys 'method' and 'a…
drew-heenan Apr 9, 2019
60ea58c
test_to_csv_compression_dict uses compression_only fixture
drew-heenan Apr 9, 2019
8ba9082
delegate dict handling to _get_compression_method, type annotations
drew-heenan Apr 10, 2019
0a3a9fd
fix import order, None type annotations
drew-heenan Apr 10, 2019
a1cb3f7
compression args passed as kwargs, update relevant docs
drew-heenan Apr 14, 2019
af2a96c
style/doc improvements, change arcname to archive_name
drew-heenan Apr 15, 2019
5853a28
Merge branch 'master' into issue-26023
drew-heenan Apr 15, 2019
789751f
Merge branch 'master' into issue-26023
drew-heenan Apr 22, 2019
5b09e6f
add to_csv example, no method test, Optional types, tweaks; update wh…
drew-heenan Apr 22, 2019 8000
68a2b4d
remove Index import type ignore
drew-heenan Apr 22, 2019
c856f50
Revert "remove Index import type ignore"
drew-heenan Apr 22, 2019
8df6c81
Merge remote-tracking branch 'upstream/master' into issue-26023
drew-heenan Apr 23, 2019
40d0252
Merge branch 'master' into issue-26023
drew-heenan Apr 26, 2019
18a735d
Improve docs/examples
drew-heenan May 5, 2019
103c877
Merge branch 'master' into issue-26023
drew-heenan May 5, 2019
b6c34bc
Merge remote-tracking branch 'upstream/master' into issue-26023
WillAyd Jun 8, 2019
969d387
Added back missed Callable import in generic
WillAyd Jun 8, 2019
abfbc0f
Merge remote-tracking branch 'upstream/master' into issue-26023
WillAyd Jun 9, 2019
04ae25d
Address comments
WillAyd Jun 9, 2019
9c22652
Typing cleanup
WillAyd Jun 9, 2019
56a75c2
Cleaned up docstring
WillAyd Jun 9, 2019
bbfea34
Merge remote-tracking branch 'upstream/master' into issue-26023
WillAyd Jun 23, 2019
7717f16
Merge remote-tracking branch 'upstream/master' into issue-26023
WillAyd Jul 15, 2019
779511e
blackify
WillAyd Jul 15, 2019
780eb04
Merge remote-tracking branch 'upstream/master' into issue-26023
WillAyd Jul 16, 2019
6c4e679
Added annotations where feasible
WillAyd Jul 16, 2019
1b567c9
Black and lint
WillAyd Jul 16, 2019
9324b63
Merge remote-tracking branch 'upstream/master' into issue-26023
WillAyd Jul 17, 2019
7cf65ee
isort fixup
WillAyd Jul 17, 2019
29374f3
Docstring fixup and more annotations
WillAyd Jul 17, 2019
6701aa4
Merge remote-tracking branch 'upstream/master' into issue-26023
WillAyd Aug 24, 2019
0f5489d
lint fixup
WillAyd Aug 24, 2019
e04138e
mypy fixup
WillAyd Aug 24, 2019
6f2bf00
whatsnew fixup
WillAyd Aug 25, 2019
865aa81
Annotation and doc fixups
WillAyd Aug 25, 2019
8d1deee
mypy typeshed bug fix
WillAyd Aug 25, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge remote-tracking branch 'upstream/master' into issue-26023
  • Loading branch information
WillAyd committed Aug 24, 2019
commit 6701aa436cec03ed74a9d419f6f51b211d802c36
1 change: 0 additions & 1 deletion doc/source/whatsnew/v0.25.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,6 @@ Other enhancements
- :class:`RangeIndex` has gained :attr:`~RangeIndex.start`, :attr:`~RangeIndex.stop`, and :attr:`~RangeIndex.step` attributes (:issue:`25710`)
- :class:`datetime.timezone` objects are now supported as arguments to timezone methods and constructors (:issue:`25065`)
- :meth:`DataFrame.query` and :meth:`DataFrame.eval` now supports quoting column names with backticks to refer to names with spaces (:issue:`6508`)
- :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` now support dicts as ``compression`` argument with key ``'method'`` being the compression method and others as additional compression options when the compression method is ``'zip'``. (:issue:`26023`)
- :func:`merge_asof` now gives a more clear error message when merge keys are categoricals that are not equal (:issue:`26136`)
- :meth:`pandas.core.window.Rolling` supports exponential (or Poisson) window type (:issue:`21303`)
- Error message for missing required imports now includes the original import error's text (:issue:`23868`)
Expand Down
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.0.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ ExtensionArray
Other
^^^^^
- Trying to set the ``display.precision``, ``display.max_rows`` or ``display.max_columns`` using :meth:`set_option` to anything but a ``None`` or a positive int will raise a ``ValueError`` (:issue:`23348`)
- :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` now support dicts as ``compression`` argument with key ``'method'`` being the compression method and others as additional compression options when the compression method is ``'zip'``. (:issue:`26023`)


.. _whatsnew_1000.contributors:
Expand Down
1 change: 1 addition & 0 deletions
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
Set,
Union,
)
from typing import Callable, Dict, FrozenSet, List, Optional, Set
import warnings
import weakref

Expand Down
13 changes: 8 additions & 5 deletions pandas/io/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import mmap
import os
import pathlib
from typing import Any, Dict, Optional, Tuple, Type, Union
from typing import IO, Any, AnyStr, BinaryIO, Dict, Optional, TextIO, Tuple, Type, Union
from urllib.error import URLError # noqa
from urllib.parse import ( # noqa
urlencode,
Expand All @@ -35,6 +35,8 @@

from pandas._typing import FilePathOrBuffer

lzma = _import_lzma()

# gh-12665: Alias for now and remove later.
CParserError = ParserError

Expand Down Expand Up @@ -287,7 +289,9 @@ def _get_compression_method(
return compression, compression_args


def _infer_compression(filepath_or_buffer, compression) -> Optional[str]:
def _infer_compression(
filepath_or_buffer: FilePathOrBuffer, compression: Optional[str]
) -> Optional[str]:
"""
Get the compression method for filepath_or_buffer. If compression='infer',
the inferred compression method is returned. Otherwise, the input
Expand Down Expand Up @@ -387,13 +391,12 @@ def _get_handle(
handles : list of file-like objects
A list of file-like object that were opened in this function.
"""
need_text_wrapping = (BytesIO,) # type: Tuple[Type[BytesIO], ...]
try:
from s3fs import S3File

need_text_wrapping = need_text_wrapping + (S3File,)
need_text_wrapping = (BufferedIOBase, S3File)
except ImportError:
pass
need_text_wrapping = BufferedIOBase

handles = list()
f = path_or_buf
Expand Down
You are viewing a condensed version of this merge commit. You can view the full changes here.
0