Focus Group: OpenMS Visualization Tools and Web apps #7321
Replies: 14 comments 20 replies
-
|
@timosachsenberg, @jpfeuffer, @Arslan-Siraj, @axelwalter, @jcharkow, @JeeH-K |
Beta Was this translation helpful? Give feedback.
-
|
@matteopilz Include me too; I am also interested. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for starting this discussion, I think this is a great and also necessary discussion prior to the retreat. I think these are good point and I will add a few more: A. First, we should summarize the current state of our visualization capabilities (interactive vs. static) and data supported (spectra, chromatograms, feature maps, mobilograms, 2D peak maps). B. Discuss where interactive plotting is needed, e.g. using Plotly with Streamlit or JavaScript integrations with Vue as Streamlit component. How to handle big files? C. What are the implications to maintenance and automated testing. We all know: getting it to run once is easy but keeping it running and maintained is hard. D. what should be our priorities? How can we consolidate efforts? e.g. Can there be a joint library that can be published? E. come up with a list of tasks and tackle them in the team? |
Beta Was this translation helpful? Give feedback.
-
|
Good points. A. For the web apps interactive plots are certainly useful. Is there a downside to using an interactive library and for both interactive and static plots? I would avoid duplicating work here. B. For large datasets plotly WebGL could be an option up to a certain point. How far can we push the limitations e.g. with binning, rasterization using datashader? D. Extending the pyopenms plotting library for the more common OpenMS datastructures would be great. However, duplicating work that has already been done in e.g. spectrum utils should be avoided. @timosachsenberg do you know if they would be open to plotting functionalities besides spectra (such as feature maps etc.). If yes we could perhaps contribute there and fill the gaps. Otherwise rely on spectrum utils for actual spectra and extend pyopenms plotting for other data structures. |
Beta Was this translation helpful? Give feedback.
-
|
B. I think Datashader would be a great library to use, but I never succeeded in using it within Streamlit... |
Beta Was this translation helpful? Give feedback.
-
|
For large datafiles there is also the option of not keeping all datasets in memory. For instance within the docker container one could setup a Restful API (e.g. with Flask; I looked into it briefly and could not find any way to do it with Streamlit) exposing some type of lightweight database. This way data could be loaded on demand. |
Beta Was this translation helpful? Give feedback.
-
|
With regards to point A stated by Timo, it would be great, if everyone could write a few sentences about what they have implemented respectively. I am not sure, who has built tools themselves, but I think I remember @JeeH-K and @t0mdavid-m doing something with Streamlit and Javascript. |
Beta Was this translation helpful? Give feedback.
-
|
So far I have good experience with plotly, using plotly WebGL for larger plots such as 2D maps and forms, to avoid constant re-rendering. You can get pretty far with if data is prepared in advance in dataframes and saved in data formats such as feather, parquet or pickle for quick access. |
Beta Was this translation helpful? Give feedback.
-
|
I implemented TaggerViewer which is based on FLASHViewer and is a visualization tool for FLASHTagger. It introduces new/modified Vue.js components for sequence tag visualization. Of course I also adapted the dataflow but I did not conceptually change anything (so it is as discussed in the presentation by @JeeH-K). Since sequence tags can be detected for multiple proteins, I adapted the sequence view to update the sequence based on the selected protein in the table:
The shading of amino acids denotes the coverage and the tag selected in the appropriate table is also highlighted. I think we should think about how to combine both versions of this component (e.g. have one parent Furthermore, I modified the
Clicking on the mass labels of one of the highlighted peaks updates the plot to show the raw spectrum with peaks belonging to the the selected mass highlighted:
I think this component could be generalized to highlight arbitrary masses and could also be of use for similar visualization tasks. |
Beta Was this translation helpful? Give feedback.
-
|
So, adding to Tom, FLASHViewer has 2D Spectrum plots, heatmaps, tables, 3D signal/noise plot, and some protein sequence-related components (as in the image below). Plotting/figure options are not that challenging to implement on the TypeScript/JavaScript side; I believe the current obstacle is how we want to design/develop the data structure for this. FLASHViewer (and I believe FLASHTagger as well) has quite specific input types, and we need to update those for more "general" cases. |
Beta Was this translation helpful? Give feedback.
-
|
A starting point for discussions / focus group tasks for the web apps: Deployment
Testing
How to organize modules such as workflow manager / visualization?
Update and polish the template
Workflow manager framework
|
Beta Was this translation helpful? Give feedback.
-
|
Could the JavaScript components be wrapped in a streamlit streamlit custom component and published to PyPi? |
Beta Was this translation helpful? Give feedback.
-
|
I believe we can. Currently, FLASHViewer implements vis components via TypeScript to have a more concise structure, then translates codes to javascript for usage in FLASHViewer. In this case, more structural implementation is possible, and I believe this is better for maintaining purpose as well. |
Beta Was this translation helpful? Give feedback.
-
|
During the Developer retreat we can give an intro on the current state of web apps and visualization. Some ideas:
I could do the template, deployment, workflow framework and UmetaFlow. Any other suggestions welcome. |
Beta Was this translation helpful? Give feedback.
-
|
Yes I implemented TaggerViewer based on FlashViewer. Sure I can give an introduction about its current state. |
Beta Was this translation helpful? Give feedback.
-
|
I can prepare some slides for the plotting functions. |
Beta Was this translation helpful? Give feedback.
-
|
Several members of the roestlab have recently implemented MassDash. This is a streamlit based DIA/Targeted visualization so it is capable of chromatogram viewing, some 2D heatmap plots with ion mobility, and some 3D plots. These plots are all interactive using bokeh or plotly libraries. https://github.com/Roestlab/massdash . Unfortunately due to a family emergency, I won't be able to make it to the retreat this year but I can prepare some slides which describe the capabilities of the application. Depending on the time maybe I can zoom in for the discussion. |
Beta Was this translation helpful? Give feedback.
-
Sorry to hear that. Family emergencies can be overwhelming, so please don't feel obliged to zoom in for a discussion. @axelwalter @matteopilz @JeeH-K @t0mdavid-m, could you please take a look at MassDash? I'm particularly interested in whether the visualization components could be utilized by us. I'm curious about how they compare to the JS components in terms of functionalities and maintainability. I'm still hopeful for a single-language solution ;). Even if that's not feasible immediately or for the first version, I believe it's the right time to consider how to modularize. For example, we could encapsulate the JS Vue into components or explore emerging works in progress (WIP) like Streamlit components (Bokeh 3, etc.) that could serve as a slimmer, more maintainable replacement in the future. While I haven't delved into the details, it appears that there are ongoing developments that might provide more lightweight interfaces with JS components, allowing users to stay in the Python world (which imho is a huge win if we want to gain external users). |
Beta Was this translation helpful? Give feedback.
-
|
I just looked into everything in more detail and what I found is that Streamlit rerenders the entire page (i.e. all components) when their input data is updated: https://discuss.streamlit.io/t/avoiding-chart-redraws-on-irrelevant-changes/33171
@JeeH-K previously reported performance issues when creating the graphs with Plotly Python. I think this might explain it. Basically, the Vue.js component seems to be a workaround on what I figured is a fundamental limitation of Streamlit at the moment. Instead of having multiple components that are all redrawn every time new input data is selected, we have a single Vue.js component (containing all plots and tables) that does not change from a Streamlit point of few. Within the Vue.js component we are able to dynamically update the plots instead of redrawing the entire page. Unfortunately, this would mean that, if we were to modularize the Vue.js plots/tables into individual Streamlit compoents, we would loose the performance benefit experienced with Vue.js. My experience with MassDash was also consistent with this (although it is not as problematic as there are less plots and they do not need to be updated as frequently as with FlashViewer/TaggerViewer). |
Beta Was this translation helpful? Give feedback.
-
|
Would that also be the case with current experimental developments? e.g., https://discuss.streamlit.io/t/partial-reruns-preview-share-your-feedback-and-cool-implementations/60851/20 |
Beta Was this translation helpful? Give feedback.
-
|
The partial re-run feature would be a game changer. Until now the only (without JavaScript) solution was putting large data graphs in separate pages or limiting displayed information. |
Beta Was this translation helpful? Give feedback.
-
|
This feature does indeed look very promising! To my understanding it allows the creation of containers that act as "independent Streamlit pages". Thus, in situations where all plots/tables are interdependent on each other (as is the case in FlashViewer/TaggerViewer) it seems like the plots would need to be placed in a single container to facilitate updates. Hence, to me it does not seem like there would be much benefit for this specific use case. |
Beta Was this translation helpful? Give feedback.
-
|
Could you chain multiple partial functions so that if input happens somewhere in the middle, the earlier widgets don't update? |
Beta Was this translation helpful? Give feedback.
-
|
Hi, folks in this discussion! @axelwalter @t0mdavid-m @timosachsenberg and I will have a discussion about this focus group via Zoom tomorrow (Feb 19) at 11 AM CET We will use the "MS SIG" Zoom Meeting room for this, and everyone is welcome to join us. Let me know if you need the link for the meeting |
Beta Was this translation helpful? Give feedback.
-
|
Summary of the discussion:
Correct me if anything is missing or wrong here. Also, we will have a short Zoom meeting to discuss the initial design of the ToppView alternative this Thursday (Feb 22) at 11 am CET. Feel free to join if you have any good ideas! |
Beta Was this translation helpful? Give feedback.
-
|
something that might be relevant for the discussion (e.g., data exchange / storage) https://github.com/bigbio/quantms.io |
Beta Was this translation helpful? Give feedback.
-
|
As it does not cover spectra: https://github.com/compomics/ThermoRawFileParser/blob/master/Writer/ParquetSpectrumWriter.cs#L115-L133 could be something that could be built upon (e.g. supporting ion mobility) |
Beta Was this translation helpful? Give feedback.
-
Action points for web apps / visualizations:streamlit web app template:
visualization module:
|
Beta Was this translation helpful? Give feedback.
-
|
I think what could also be cool is a port to run locally. E.g. using https://github.com/whitphx/stlite We have some DIA plotting capabilities in massDash so it would be nice to incorporate some of that. Currently loading large MzMLFiles still takes a long time even with OnDIscImplementation() unless metadata is not loaded. Workaround where only loading parts of the metadata might be useful. Alternatively, we could support vendor specific formats. |
Beta Was this translation helpful? Give feedback.
-
|
The serverless streamlit sounds really cool! Would you be interested in checking that out? |
Beta Was this translation helpful? Give feedback.
-
|
Yes I would like to participate as well. Please make the meeting an afternoon time (e.g. 3 or 4) because of the time difference. I checked out stlite and it is a bit more complicated than originally thought. Because pyopenms is not a pure python package we would have to integrate it in pyodide. I've tried briefly but running into some issues. I'm not sure if all the extra effort is w B41A orth it at this time right now. |
Beta Was this translation helpful? Give feedback.
-
|
maybe interesting for the vue.js guys: https://discuss.streamlit.io/t/a-method-to-create-components-almost-entirely-in-python/68357 |
Beta Was this translation helpful? Give feedback.
-
We should discuss how to proceed with the different visualization tools and if we want to use a common package / language. I can give a quick overview of what I have been adding so far.
Both tools are exclusively for plotting mass spectra and depend on matplotlib. Spectrum utils also allows interactive plotting with altair, which should also be possible after converting our MSSpectrum with our conversion function.
It would be good to discuss the major applications we need for visualization in OpenMS like spectrum, chromatogram, interactivity, etc. It would also be good to discuss whether we want to build something included in OpenMS or something separate which OpenMS uses as dependency.
Beta Was this translation helpful? Give feedback.
All reactions