Fix threading.Condition with monkey-patching on Python 3.3 and newer #187
Conversation
|
Is it ok to change Semaphore.acquire() behaviour for negative timeout? Any issue with the backward compatibility? The idea is to have a behaviour to the Python stdlib. |
With eventlet 0.16, it becomes possible to run Oslo Messaging tests on Python 3 with eventlet. This change ports the zmq driver to Python 3: * encode the topic explicitly to UTF-8 * use a list comprehension instead of map() to also get a list on Python 3 (not a generator) The following eventlet change is needed to run tests: eventlet/eventlet#187 Related eventlet issue: eventlet/eventlet#185 I will propose a different change to enable tests with eventlet enabled when a release of eventlet including this fix will be available. Change-Id: Ic8fec515cfa757e08ffb9604e3bfb2e87d08f3d8
Project: openstack/oslo.messaging ae009a4ed99caa414104bc06829ac5a4f4cc9d2c Port zmq driver to Python 3 With eventlet 0.16, it becomes possible to run Oslo Messaging tests on Python 3 with eventlet. This change ports the zmq driver to Python 3: * encode the topic explicitly to UTF-8 * use a list comprehension instead of map() to also get a list on Python 3 (not a generator) The following eventlet change is needed to run tests: eventlet/eventlet#187 Related eventlet issue: eventlet/eventlet#185 I will propose a different change to enable tests with eventlet enabled when a release of eventlet including this fix will be available. Change-Id: Ic8fec515cfa757e08ffb9604e3bfb2e87d08f3d8
Project: openstack/oslo.messaging ae009a4ed99caa414104bc06829ac5a4f4cc9d2c Port zmq driver to Python 3 With eventlet 0.16, it becomes possible to run Oslo Messaging tests on Python 3 with eventlet. This change ports the zmq driver to Python 3: * encode the topic explicitly to UTF-8 * use a list comprehension instead of map() to also get a list on Python 3 (not a generator) The following eventlet change is needed to run tests: eventlet/eventlet#187 Related eventlet issue: eventlet/eventlet#185 I will propose a different change to enable tests with eventlet enabled when a release of eventlet including this fix will be available. Change-Id: Ic8fec515cfa757e08ffb9604e3bfb2e87d08f3d8
|
Ping? Is someone available to review this change? Is it ok? |
|
@Haypo I'll review this tomorrow, sorry for the wait |
|
@Haypo The negative timeout value seems to be working already, do you have a test case showing a failure? |
My first commit changes the behaviour of eventlet.semaphore.Semaphore.acquire() when you pass a negative timeout. Before, Semaphore.acquire(timeout=-1) was non-blocking. After, Semaphore.acquire(timeout=-1) becomes blocking, same behaviour than Python 3 lock. In Python 2, threading.Lock.acquire() has no timeout parameter. So RLock only calls acquire() (on its internal lock) with one parameter: blocking. In Python 3, threading.Lock.acquire() got a new timeout parameter: def acquire(blocking=True, timeout=None). RLock calls aquire() (on its internal lock) with two parameters: blocking and timeout. In Python 3, threading.Lock.acquire(timeout=-1) blocks, whereas eventlet.semaphore.Semaphore.acquire(timeout=-1) is currently non-blocking. eventlet monkey-patching is not supposed to modify the behaviour of patched functions. I wasn't the case in Python 2, since Lock.acquire() had no timeout parameter. It's now the case, since Python 3 also uses the second timeout parameter. In short, my changes fix two different Python 3 bugs:
|
|
FYI I failed to find a call to Semaphore.acquire() which uses the timeout parameter in eventlet source code. In eventlet, the timeout doesn't look to be used. The timeout parameter is documented: The behaviour on negative timeout is not documented. |
|
Ping? |
|
@Haypo I've rebased it on recent master, now |
Cool!
It's not a regression of recent changes in eventlet. I'm not sure, but I guess that I used Python 3.3 to work on my patch, whereas Python 3.4 requires extra changes. Python 3.4 adds a new threading.Thread._tstate_lock attribute which doesn't work with monkey-patching, because it relies on private C code which releases the new lock at thread exit (when its Python thread state is destroyed). Well, see the pull request #211 for the details, I proposed a fix. |
For the Python implementation of threading.RLock because the new C implementation of threading.RLock of Python 3.3 is not compatible with eventlet monkey patching. Fix the issue #185.
|
The changeset "Disable the thread state lock" fixes the threading support on Python 3.4 when eventlet is used with monkey-patching. Python 3.4 introduces threading.Thread._tstate_lock to fix a race condition, see: I'm not really proud of this change: patching a Thread instance after its creation is not the most elegant fix. I tried to create a Thread subclass in eventlet.green.threading, but I got a lot of new issues. If the subclass is declared in eventlet.green.threading, the parent Thread class uses functions of the original threading module, instead of using patched functions by eventlet. It's unclear to me how symbols are added and patched in modules. I noticed that the threading module of the standard library is imported multiple times. The implementation of monkey-patching is really complex :-/ Currently, the best solution is to patch Thread instance in the patched thread.start_new_thread() function. The problem is to retrieve the instance. My heuristic checks if the wrapped function is the bounded Thread.run() method, and I retrieve the instance from function.self. I'm not yet sure that I really understood the problem. At least, "tox -e py34-selects" pass. |
|
I rewrite the first commit: in fact, we don't need to support negative timeout, we only need to accept timeout=-1 for Semaphore.accept(). If the timeout value is None or -1, the timeout is ignored, only the blocking parameter is used:
|
| if timeout == -1: | ||
| timeout = None | ||
| if timeout is not None and timeout < 0: | ||
| raise ValueError("timeout value must be strictly positive") |
There was a problem hiding this comment.
Exception says "strictly positive" ( x > 0 ), while code allows timeout=0 just fine.
There was a problem hiding this comment.
Exception says "strictly positive" ( x > 0 ), while code allows timeout=0 just fine.
Hum... yes... I copied code from the C implementation of threading.Lock, to try to be as close as possible to threading.Lock.
Example:
$ python3
Python 3.4.1 (default, Nov 3 2014, 14:38:10)
>>> import threading
>>> l=threading.Lock()
>>> l.acquire(timeout=-2)
...
ValueError: timeout value must be strictly positive
>>> l.acquire(timeout=0)
True
>>> l.acquire(timeout=0)
False
|
temodo wrote "Exception says "strictly positive" ( x > 0 ), while code allows timeout=0 just fine." but I copied the error message from Python. Do you expect that I change my patch, or are you ok to mimick Python lock? |
|
Quite surprised that Python code has this code/comment mismatch. Yeah, leave it same as stdlib. |
Oh, so did you review my patch? Does it look ok? Should I change something? |
|
LGTM, running last round of tests now. |
|
I took the liberty of sqashing PEP-8 fixes into your commits. |
|
I took the liberty of sqashing PEP-8 fixes into your commits.
No problem. I only care of having eventlet running on python 3.3 ;-)
|
|
It's merged into master 390e71d thank you! |
|
Thanks @Haypo! |
It fixes monkey-patching threading.Condition on Python 3.3 and newer:
fix issue #185.