8000 Concurrency performance question · Issue #110 · python-kasa/python-kasa · GitHub
[go: up one dir, main page]
Skip to content

Concurrency performance question #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pcwalden opened this issue Nov 5, 2020 · 3 comments
Closed

Concurrency performance question #110

pcwalden opened this issue Nov 5, 2020 · 3 comments

Comments

@pcwalden
Copy link
pcwalden commented Nov 5, 2020

I am converting my home brew web app from pyHS100 to Kasa. I thought the asyncio interface for Kasa would speed it up, but so far I do not see much improvement and in some cases it is slower.

In the initial page render, I get updates from a list of all devices by IP address. The pyHS100 code has to do each one sequentially. With Kasa I am using asyncio.gather() to get the updates concurrently, or so I think. The klights() function below is run from the main thread using asyncio.run(). The question is whether the asyncio.gather is really concurrent or am I mistaken? It does not appear to be much faster than using pyHS100 in a similar sequential fashion.

from kasa import SmartDevice, SmartPlug, SmartStrip, SmartDeviceException
async def update_wrapper(smartdevice):
    global logger
    try:
        await smartdevice.update()
    except BaseException as bx:
        logger.error("Unable to initialize SmartDevice {}: {}".format(smartdevice.host,str(bx)))
        print(bx)
        pass

async def klights():
    devices = db(db.devices).select(db.devices.host_ip_addr,db.devices.device_type,distinct=True)
    smartdevs = []
    keys = []
    dev_names = []
    for device in devices:
        if device.device_type == 'plug':
            smartdevs.append(SmartPlug(device.host_ip_addr))
        else:
            smartdevs.append(SmartStrip(device.host_ip_addr))
    sdaws = []
    for sdev in smartdevs:
        sdaws.append(update_wrapper(sdev))
    devs = asyncio.gather(*sdaws)
    return smartdevs

Lastly, the same web app takes ajax calls to the server to turn devices on/off. In this case the server has to connect to a device, change its state, and query its resulting state. Here kasa definitely seems slower as it must perform extra update()'s to perform the same function with pyHS100. The question is how to reduce the multiple round trips to the device?

pyHS100:
dev = SmartPlug(ip_addr)
dev.turn_on()
state = dev.is_on

Kasa:
dev = SmartPlug(ip_addr)
asyncio.run(dev.update())
asyncio.run(dev.turn_on())
asyncio.run(devupdate())
state = dev.is_on
@rytilahti
Copy link
Member
rytilahti commented Nov 6, 2020 •
edited
Loading

Hi @pcwalden, and thanks for your interesting questions! The performance really piqued my interest, so I had to do some benchmarking... :-)

I thought the asyncio interface for Kasa would speed it up, but so far I do not see much improvement and in some cases it is slower.

How many devices are we talking about? And how fast are the updates already? How did you benchmark the performance?

Anyway, asyncio does something called cooperative concurrency, where a task can yield and let other tasks to run while it is waiting for I/O (or something else to finish). This is mostly useful in cases where an operation is taking some time to finish, so if the devices are already fast to respond, it doesn't really matter that much.

The question is whether the asyncio.gather is really concurrent or am I mistaken?

Awaiting on asyncio.gather() will wait for all coroutines to finish before returning, so even if some tasks are already finished it is still blocking. So doing updates with it could be faster, especially if some devices are less responsive than others (that is, sequential accesses would wait for responses). The total execution time does not really change that much if you wait all requests to finish, I'll add a test script & some results from the devices I have at the end of this comment.

You usually use asyncio in asynchronous context, which allows you to do updates as soon as the results come in. Check out asyncio.as_completed() or use add_complete_callback() to handle the results as soon as they come in. As you mentioned using ajax, you should simply signal your web app to do updates as they come.

The question is how to reduce the multiple round trips to the device?

You can assume that the action has completely successfully if no exception is raised. Alternatively, you could check the return value of methods like turn_on, which will contain the payload from the device.

Now, for the tests I did. Thanks for raising the question, it made me wonder why I'm also having sometimes troubles with my light bulb. For these tests I used 10 rounds of updates:, both concurrently and sequentially. The test devices were two HS110s, a KL130 and a KP303.

Note that it's not uncommon for devices (or their chipsets) to go to some sort of sleep mode if they are not actively communicating, so if you really want to get more conclusive results, you should adjust the sleeps to be longer to simulate a more realistic use case.

These are only done using python-kasa, but the query itself is (mostly) the same as used by pyhs100. The only real difference is that also emeter statistics are being queried for devices supporting it on python-kasa.

=== Testing using gather on all devices ===
              took                                                                      
             count      mean       std       min       25%       50%       75%       max
type                                                                                    
concurrently  10.0  0.196667  0.068511  0.044919  0.154346  0.229221  0.237399  0.269855
sequential    10.0  0.272287  0.084113  0.136539  0.249556  0.250960  0.260461  0.454781

So based on this, executing the tasks "concurrently" yields better results on average (the median doesn't matter that much considering the sample size), but it seems still to perform better.

I also did do another test round to demonstrate what happens if you wouldn't do a gather but simply handle the results as soon as they come:

=== Testing per-device performance ===
                           took                                                                      
                          count      mean       std       min       25%       50%       75%       max
id                                                                                                   
139882552852288-KP303(UK)  10.0  0.022219  0.007838  0.011652  0.018958  0.020973  0.023152  0.041773
139882552994688-HS110(EU)  10.0  0.023627  0.008106  0.014368  0.018894  0.021746  0.025552  0.043705
139882553034304-KL130(EU)  10.0  0.190850  0.105943  0.042239  0.083322  0.246528  0.251389  0.336146
139882553158912-HS110(EU)  10.0  0.020193  0.009985  0.012975  0.016302  0.017394  0.018336  0.047909

So in my 8000 case, it looks like KL130 is consistently an order of magnitude slower than others..

Here's the test script I used, to try it out simply fill the addrs array and let it run for a while:

import asyncio
import time
import pandas as pd
from kasa import SmartPlug, SmartBulb, SmartStrip


async def update(dev, lock=None):
    if lock is not None:
        await lock.acquire()
        await asyncio.sleep(2)
    try:
        start_time = time.time()
        #print("%s >> Updating" % id(dev))
        await dev.update()    
        #print("%s >> done in %s" % (id(dev), time.time() - start_time))
        return {"id": f"{id(dev)}-{dev.model}", "took": (time.time() - start_time)}
    finally:
        if lock is not None:
            lock.release()


async def update_concurrently(devs):
    start_time = time.time()
    update_futures = [asyncio.ensure_future(update(dev)) for dev in devs]
    done = await asyncio.gather(*update_futures)
    return {"type": "concurrently", "took": (time.time() - start_time)}


async def update_sequentially(devs):
    start_time = time.time()

    for dev in devs:
        await update(dev)

    return {"type": "sequential", "took": (time.time() - start_time)}


async def main():
    devs = [
        # SmartPlug("127.0.0.1"),
    ]

    data = []
    rounds = 10
    test_gathered = True
    
    if test_gathered:
        print("=== Testing using gather on all devices ===")
        for i in range(rounds):
            data.append(await update_concurrently(devs))
            await asyncio.sleep(2)


        await asyncio.sleep(5)

        for i in range(rounds):
            data.append(await update_sequentially(devs))
            await asyncio.sleep(2)


        df = pd.DataFrame(data)
        print(df.groupby("type").describe())
    
    
    print("=== Testing per-device performance ===")
    
    futs = []
    data = []
    locks = {dev: asyncio.Lock() for dev in devs}
    for i in range(rounds):
        for dev in devs:
            futs.append(asyncio.ensure_future(update(dev, locks[dev])))

    for fut in asyncio.as_completed(futs):
        res = await fut
        data.append(res)
        
    df = pd.DataFrame(data)
    print(df.groupby("id").describe())

if __name__ == "__main__":
    asyncio.run(main())

@pcwalden
Copy link
Author
pcwalden commented Nov 6, 2020

Thank you for the extensive set of answers. It will take me awhile to process these as I am relatively new to asyncio.

I have one KP400, four HS100's, six HS200's and two HS210's. All plugs or strips and no bulbs.

@rytilahti
Copy link
Member

I think we can close this issue now. There have been various improvements to the I/O handling in this library, the most recent being keeping the connection open to avoid TCP setup&teardown (#213). Please give a test to the 0.4.0 release, it will probably be much more performant if you are doing multiple requests on the device :-)

@pcwalden pcwalden mentioned this issue Oct 11, 2021
Closed
@rytilahti rytilahti mentioned this issue Oct 12, 2021
Merged
rytilahti added a commit to rytilahti/python-kasa that referenced this issue Jan 14, 2022
@rytilahti
Prepare 0.4.1
3189cb6 
This minor release fixes issues that were found after homeassistant integration got converted over from pyhs100.

[Full Changelog](python-kasa/python-kasa@0.4.0...0.4.1)

**Implemented enhancements:**

- Add --type option to cli [\python-kasa#269](python-kasa#269) ([rytilahti](https://github.com/rytilahti))
- Minor improvements to onboarding doc [\python-kasa#264](python-kasa#264) ([rytilahti](https://github.com/rytilahti))
- Add fixture file for KL135 [\python-kasa#263](python-kasa#263) ([ErikSGross](https://github.com/ErikSGross))
- Add KL135 color temperature range [\python-kasa#256](python-kasa#256) ([rytilahti](https://github.com/rytilahti))
- Add py.typed to flag that the package is typed [\python-kasa#251](python-kasa#251) ([rytilahti](https://github.com/rytilahti))
- Add script to check supported devices, update README [\python-kasa#242](python-kasa#242) ([rytilahti](https://github.com/rytilahti))
- Add perftest to devtools [\python-kasa#236](python-kasa#236) ([rytilahti](https://github.com/rytilahti))
- Add KP401 US fixture [\python-kasa#234](python-kasa#234) ([bdraco](https://github.com/bdraco))
- Add KL60 US KP105 UK fixture [\python-kasa#233](python-kasa#233) ([bdraco](https://github.com/bdraco))
- Make cli interface more consistent [\python-kasa#232](python-kasa#232) ([rytilahti](https://github.com/rytilahti))
- Add KL400, KL50 fixtures [\python-kasa#231](python-kasa#231) ([bdraco](https://github.com/bdraco))
- Add fixture for newer KP400 firmware [\python-kasa#227](python-kasa#227) ([bdraco](https://github.com/bdraco))
- Switch to poetry-core [\python-kasa#226](python-kasa#226) ([fabaff](https://github.com/fabaff))
- Add fixtures for LB110, KL110, EP40, KL430, KP115 [\python-kasa#224](python-kasa#224) ([bdraco](https://github.com/bdraco))

**Fixed bugs:**

- Discovery on WSL results in OSError: \[Errno 22\] Invalid argument [\python-kasa#246](python-kasa#246)
- New firmware for HS103 blocking local access? [\python-kasa#42](python-kasa#42)
- Pin mistune to \<2.0.0 to fix doc builds [\python-kasa#270](python-kasa#270) ([rytilahti](https://github.com/rytilahti))
- Catch exceptions raised on unknown devices during discovery [\python-kasa#240](python-kasa#240) ([rytilahti](https://github.com/rytilahti))

**Closed issues:**

- Control device with alias via python api? [\python-kasa#285](python-kasa#285)
- Can't install using pip install python-kasa [\python-kasa#255](python-kasa#255)
- Kasa Smart Bulb KL135 - Unknown color temperature range error [\python-kasa#252](python-kasa#252)
- KL400 Support [\python-kasa#247](python-kasa#247)
- Cloud support? [\python-kasa#245](python-kasa#245)
- Support for kp401 [\python-kasa#241](python-kasa#241)
- LB130 Bulb stopped working [\python-kasa#237](python-kasa#237)
- Unable to constantly query bulb in loop [\python-kasa#225](python-kasa#225)
- HS103: Unable to query the device: unpack requires a buffer of 4 bytes [\python-kasa#187](python-kasa#187)
- Help request - query value [\python-kasa#171](python-kasa#171)
- Can't Discover Devices [\python-kasa#164](python-kasa#164)
- Concurrency performance question [\python-kasa#110](python-kasa#110)
- Define the port by self? [\python-kasa#108](python-kasa#108)
- Convert homeassistant integration to use the library [\python-kasa#9](python-kasa#9)

**Merged pull requests:**

- Publish to pypi on github release published [\python-kasa#287](python-kasa#287) ([rytilahti](https://github.com/rytilahti))
- Relax asyncclick version requirement [\python-kasa#286](python-kasa#286) ([rytilahti](https://github.com/rytilahti))
- Do not crash on discovery on WSL [\python-kasa#283](python-kasa#283) ([rytilahti](https://github.com/rytilahti))
- Add python 3.10 to CI [\python-kasa#279](python-kasa#279) ([rytilahti](https://github.com/rytilahti))
- Use codecov-action@v2 for CI [\python-kasa#277](python-kasa#277) ([rytilahti](https://github.com/rytilahti))
- Add coverage\[toml\] dependency to fix coverage on CI [\python-kasa#271](python-kasa#271) ([rytilahti](https://github.com/rytilahti))
- Allow publish on test pypi workflow to fail [\python-kasa#248](python-kasa#248) ([rytilahti](https://github.com/rytilahti))
@rytilahti rytilahti mentioned this issue Jan 14, 2022
Merged
rytilahti added a commit that referenced this issue Jan 14, 2022
@rytilahti
Prepare 0.4.1 (#288)
b4036e5 
This minor release fixes issues that were found after homeassistant integration got converted over from pyhs100.

[Full Changelog](0.4.0...0.4.1)

**Implemented enhancements:**

- Add --type option to cli [\#269](#269) ([rytilahti](https://github.com/rytilahti))
- Minor improvements to onboarding doc [\#264](#264) ([rytilahti](https://github.com/rytilahti))
- Add fixture file for KL135 [\#263](#263) ([ErikSGross](https://github.com/ErikSGross))
- Add KL135 color temperature range [\#256](#256) ([rytilahti](https://github.com/rytilahti))
- Add py.typed to flag that the package is typed [\#251](#251) ([rytilahti](https://github.com/rytilahti))
- Add script to check supported devices, update README [\#242](#242) ([rytilahti](https://github.com/rytilahti))
- Add perftest to devtools [\#236](#236) ([rytilahti](https://github.com/rytilahti))
- Add KP401 US fixture [\#234](#234) ([bdraco](https://github.com/bdraco))
- Add KL60 US KP105 UK fixture [\#233](#233) ([bdraco](https://github.com/bdraco))
- Make cli interface more consistent [\#232](#232) ([rytilahti](https://github.com/rytilahti))
- Add KL400, KL50 fixtures [\#231](#231) ([bdraco](https://github.com/bdraco))
- Add fixture for newer KP400 firmware [\#227](#227) ([bdraco](https://github.com/bdraco))
- Switch to poetry-core [\#226](#226) ([fabaff](https://github.com/fabaff))
- Add fixtures for LB110, KL110, EP40, KL430, KP115 [\#224](#224) ([bdraco](https://github.com/bdraco))

**Fixed bugs:**

- Discovery on WSL results in OSError: \[Errno 22\] Invalid argument [\#246](#246)
- New firmware for HS103 blocking local access? [\#42](#42)
- Pin mistune to \<2.0.0 to fix doc builds [\#270](#270) ([rytilahti](https://github.com/rytilahti))
- Catch exceptions raised on unknown devices during discovery [\#240](#240) ([rytilahti](https://github.com/rytilahti))

**Closed issues:**

- Control device with alias via python api? [\#285](#285)
- Can't install using pip install python-kasa [\#255](#255)
- Kasa Smart Bulb KL135 - Unknown color temperature range error [\#252](#252)
- KL400 Support [\#247](#247)
- Cloud support? [\#245](#245)
- Support for kp401 [\#241](#241)
- LB130 Bulb stopped working [\#237](#237)
- Unable to constantly query bulb in loop [\#225](#225)
- HS103: Unable to query the device: unpack requires a buffer of 4 bytes [\#187](#187)
- Help request - query value [\#171](#171)
- Can't Discover Devices [\#164](#164)
- Concurrency performance question [\#110](#110)
- Define the port by self? [\#108](#108)
- Convert homeassistant integration to use the library [\#9](#9)

**Merged pull requests:**

- Publish to pypi on github release published [\#287](#287) ([rytilahti](https://github.com/rytilahti))
- Relax asyncclick version requirement [\#286](#286) ([rytilahti](https://github.com/rytilahti))
- Do not crash on discovery on WSL [\#283](#283) ([rytilahti](https://github.com/rytilahti))
- Add python 3.10 to CI [\#279](#279) ([rytilahti](https://github.com/rytilahti))
- Use codecov-action@v2 for CI [\#277](#277) ([rytilahti](https://github.com/rytilahti))
- Add coverage\[toml\] dependency to fix coverage on CI [\#271](#271) ([rytilahti](https://github.com/rytilahti))
- Allow publish on test pypi workflow to fail [\#248](#248) ([rytilahti](https://github.com/rytilahti))
@rytilahti rytilahti mentioned this issue Feb 14, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pcwalden @rytilahti
0