Closed
Description
[REQUIRED] Step 2: Describe your environment
- Operating System version: macOS 10.13.6
- Firebase SDK version: 2.13.0
- Firebase Product: database
[REQUIRED] Step 3: Describe the problem
Steps to reproduce:
- Have some Firebase Realtime Database path containing a significant amount of data (>200KB, say).
- Call
db.reference('/path').listen(callback).
As the first update, the server will send the entire contents of that path. On my machine this pegs a CPU core for minutes before the callback is finally called. The issue is the following code:
firebase-admin-python/firebase_admin/_sseclient.py
Lines 93 to 96 in abd3980
Since
self.resp_iterator iterates through the stream one character at a time and self._event_complete() runs a regular expression over the entire buffer, this is an O(N^2) operation, which takes a long time when N is large.
Separately, there is a copyright violation here. I discovered that `_sseclient.py, despite having a Google copyright header, is largely copied from https://github.com/btubbs/sseclient/blob/master/sseclient.py.
Relevant Code:
Run the following code to reproduce:
def listen_issue():
path = '/sandbox/listen'
print("Inserting data...")
start = time.time()
for i in range(10):
long_data = ''.join(random.choice(string.ascii_uppercase) for _ in range(20000))
db.reference(os.path.join(path, str(i))).set(long_data)
end = time.time()
print("...done (took %f seconds)" % (end - start))
done_event = threading.Event()
listener = db.reference(path).listen(lambda db_event: done_event.set())
start = time.time()
print("Waiting for first message from listen()...")
done_event.wait() # CPU usage is at 100% here
end = time.time()
print("...done (took %f seconds)" % (end - start))
listener.close()Example output on my machine:
Inserting data...
...done (took 1.678544 seconds)
Waiting for first message from listen()...
...done (took 109.180737 seconds)
Metadata
Metadata
Assignees
Labels
No labels