8000 Add speedup module for CPython · Issue #297 · PyMySQL/PyMySQL · GitHub
[go: up one dir, main page]
Skip to content

Add speedup module for CPython #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
methane opened this issue Feb 2, 2015 · 8 comments
Open

Add speedup module for CPython #297

methane opened this issue Feb 2, 2015 · 8 comments

Comments

@methane
Copy link
Member
methane commented Feb 2, 2015

There are two bottlenecks.

  1. Reading packet
  2. Parsing packet

1 is resolved by using io module.

To resolve issue 2, we should remove classes for parsing packets, and use simple flat function and
builtin types. It make PyMySQL on PyPy faster.
Then, we can easily replace them with speedup module written in C.
@methane methane added this to the 0.7 milestone Feb 2, 2015
@methane methane removed this from the 0.7 milestone May 17, 2016
@methane methane mentioned this issue Jun 15, 2016
Closed
@methane methane modified the milestones: 0.7, 0.8 Jul 14, 2016
@methane methane modified the milestones: 1.0, 0.8 LTS Aug 30, 2016
@zcattacz
Copy link
zcattacz commented Dec 2, 2016

I have script periodically pump excel data into mariadb using pymysql.
each time it needs to send ~50k records. it takes about 5.5 min.
After compiling pymysql module with Nuitka, the time is lowered to 3.2min.
I consider it a good speed gain.

@groutr
8000 Copy link
groutr commented Jan 19, 2017

What is the status of this speedup module?

@ghost
Copy link
ghost commented Jun 20, 2017

I was able to speed up reading/parsing packets by roughly 28% just by reducing the number of function calls. Reading ~70k rows (x45 cols) dropped from 8s to 5.7s on my machine. This quick edit however duplicates some code. See my attempt here: https://github.com/horacekmi/PyMySQL/commit/14125ef1026553c88e28ef3d7bdb849676edd102

@byaka
Copy link
byaka commented Jun 20, 2017

micro-optimisations, that converts source from hierarchical to linear really can burst performance near 20% in some cases, but it bad way. much better to rewrite thats parts of code to cython...
anyway i agree with TS, but who has time for full rewrite of parser? :(

@groutr
Copy link
groutr commented Jan 16, 2018 •
edited
Loading

I'm willing to make an initial attempt at isolating and simplifying the parsing code (issue 2 from above).

@groutr groutr mentioned this issue Jan 23, 2018
Closed
@groutr groutr mentioned this issue May 9, 2018
Closed
@rspadim
Copy link
rspadim commented Jun 22, 2020

i was using aiomysql, and found a point to speed up

when i was executing a insert ignore, or something that could report a known warning, it was executing a show warnings (2 queries)

force removing the show warnings changed my app to run from ~7kqps to ~10kqps (multi process)

i didn't checked pymysql code, but this could be a nice parameter at cursor.execute() function

@rspadim
Copy link
rspadim commented Jun 22, 2020

and maybe we could use numba+cython, when possible?

@kesmit13
Copy link
kesmit13 commented Sep 6, 2022

I opened another similar issue (#1066) not realizing this one was here. I have a fork of PyMySQL with a C extension that replaces the row data reader method and adds some tuned cursors that gives significant speed improvements. The issue was opened to start a conversation about a possible plugin for PyMySQL, and to collaborate on further improvements.

@methane methane removed this from the 1.0 milestone May 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@methane @byaka @zcattacz @rspadim @kesmit13 @groutr
0