8000 Baseline benchmark · Issue #153 · pyscript/pyscript · GitHub
[go: up one dir, main page]
Skip to content

Baseline benchmark #153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
fpliger opened this issue May 3, 2022 · 5 comments
Closed

Baseline benchmark #153

fpliger opened this issue May 3, 2022 · 5 comments
Labels
backlog issue has been triaged but has not been earmarked for any upcoming release tag: example Related to adding or improving examples tag: performance

Comments

@fpliger
Copy link
Contributor
fpliger commented May 3, 2022

Follow task up from #152 issue. It'd be extremely helpful to have a baseline benchmark where we capture the following loading times on each example page:

  • pyscript.js loading
  • Pyodide loading
  • Environment loading
  • Total initialization (interpreter ready and all scripts ran)
  • Total page
@fpliger fpliger added tag: example Related to adding or improving examples tag: tests Related to tests tag: performance labels May 3, 2022
@marimeireles marimeireles added backlog issue has been triaged but has not been earmarked for any upcoming release and removed tag: tests Related to tests labels Sep 28, 2022
@marimeireles
Copy link
Contributor

I wonder what's @antocuni's update on this?
Should we close this because of the work he has done?
Or you mean you'd like to have scripts for it @fpliger ?
Sorry, I'm confused.

@fpliger
Copy link
Contributor Author
fpliger commented Feb 12, 2023

I think we are in a way better place for this but I think it still stands. The tldr; of it is that would be useful to capture these (or similar) metrics when we run our tests to have a good measure of deltas between releases or even just we merge PRs, so that we can understand how much we are making things faster overtime or early identify PRs that are significant performance regressions...

@WebReflection
Copy link
Contributor

I think we should close this for the following reasons:

  • pyodide bootstrap is heavily bandwidth and hardware dependent ... any metric would be "on my machine"
  • MicroPython is almost instantly bootstrapped even on Mobile
  • we can't offer metrics for both Desktop and Mobile in a meaningful way
  • we could, using the same hardware and Internet bandwidth connection, provide the difference between pyodide and MicroPython ... but:
    • pyodide can take much longer as soon as some package is required
    • MicroPython could also take some extra time as soon as some package is required, whenever that will be available
    • pyodide will always take way longer than MicroPython to bootstrap by design so it's hard to understand how valuable that benchmark is if not for re-stating "yeah, MicroPython is really fast at bootstrapping and it weight less"
  • we don't get to control the size of neither projects but npm already provide their bundled size and CDNs (Network Panel) already provide size and time to parse ... so I am not sure, once again, how useful any of these details is, when the real-world usage variety is huge and full of different cases

@JeffersGlass
Copy link
Contributor

In terms of load time, I suspect anything we (PolyScript or PyScript) are doing is going to get dwarfed by the load/init time of Pyodide, so probably the thing for us to do is just keep an eye on (and contribute to the improvement of!) Pyodide load time/size.

But agreed, I don't think trying to generate our own metrics here makes a ton of sense. If anything, something we can keep in the back of our minds, but not worth generating metrics on every PR for, if that makes sense?

So agreed, let's close.

@fpliger
Copy link
Contributor Author
fpliger commented Dec 7, 2023

Agreed. Let's close.

@fpliger fpliger closed this as completed Dec 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
backlog issue has been triaged but has not been earmarked for any upcoming release tag: example Related to adding or improving examples tag: performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@WebReflection @fpliger @JeffersGlass @marimeireles
0