Description
I have choosen Pandas for an ETL project and encountered data type detection problems.
I already posted on StackOverflow and responses were pointing the fact that missing values can make data type detection error in Pandas.
Here is the post containing code example and behaviour :
http://stackoverflow.com/questions/39298989/python-pandas-dtypes-detection-from-sql
Briefly, it appears that while querying a database, if tuple has missing fields, then the whole column type is affected : dates are not correctly interpreted or integers turn into float.
I understand than working with flat csv files can be tricky for type detection.
However, as Pandas works with a whole database layer (SqlAlchemy, cx_Oracle, DB API), when working with a database, it should have access to metadata that describes columns type.
Am I missing something? Is this a bug? Or a function still not implemented?
INSTALLED VERSIONS
commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.18-238.el5
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
pandas: 0.18.1
nose: None
pip: 8.1.1
setuptools: 21.2.1
Cython: None
numpy: 1.11.0
scipy: None
statsmodels: None
xarray: None
IPython: 4.2.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.4
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.13
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None
Sincerely,
Guillaume