Add performance benchmarks for dataframe arithmetic operations #6827
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
9 tasks
Codecov Report
@@ Coverage Diff @@
## main #6827 +/- ##
==========================================
+ Coverage 68.99% 69.01% +0.01%
==========================================
Files 1237 1237
Lines 253558 253564 +6
Branches 26542 26542
==========================================
+ Hits 174944 174990 +46
+ Misses 71663 71616 -47
- Partials 6951 6958 +7
Flags with carried forward coverage won't be shown. Click here to find out more.
|
|
@JakeRadMSFT could you please take a look? |
|
On the surface these look fine to me - I'm not sure how we should integrate into our builds though. It would be great to have reports show up in PRs or somewhere ... so we can track how changes are impacting perf. |
JakeRadMSFT
reviewed
Sep 25, 2023
| </PropertyGroup> | ||
|
|
||
| <ItemGroup> | ||
| <PackageReference Include="BenchmarkDotNet" Version="0.13.5" /> |
There was a problem hiding this comment.
We should use the same version as the other performance projects - $(BenchmarkDotNetVersion)
There was a problem hiding this comment.
There was a problem hiding this comment.
@JakeRadMSFT there is an issue with the DotNetBenchmark version 0.12.0, that is currently used in ML .Net. It crashes with NotSupportedException("Unknown .NET Runtime") on .Net 6.0.
However. upgrading solution to the latest 0.13.5 leads to build error as Obsolete methods from this library are used in Microsoft.ML.CpuMath.PerformanceTests.
I am not able to build Native part of ML .Net on my laptop ( due to #6823 ), so I can't verify my changes in the CpuMath.PerformanceTests project (or check if it's working correctly on .Net 6.0). That's why I used the latest version only for the DataFrame project.
Could you please help me with building native dlls for the project?
There was a problem hiding this comment.
Yeah let me take a look!
…rnov82/machinelearning into dataframe_performance_tests
JakeRadMSFT
approved these changes
Sep 26, 2023
JakeRadMSFT
added a commit
that referenced
this pull request
Oct 5, 2023
* Update build templates to handle feature branches (#6744) * Update build templates * Update build templates to include all releases/* and feature/* * Update releases to release * Update triggers for PR Validation Build * Add triggers for Code Coverage * Update version to 4.0 for feature branch (#6743) * Add missing implementation for datetime relevant arrow type into dataframe (#6675) * Add missing implementation for datetime relevant arrow type * Return required usage * Fix the behavior or column SetName method (#6676) * Fix the behavior or column SetName method * Fix stack overflow exception * Fix merge issues --------- Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com> * Fix DataFrame to allow to store columns with size more than 2 Gb (#6710) * Fix error with allocating more than MaxCapacity of Byte Memory Buffer * Remove Unit test as it consumes too much memory * Fix issue with increasing buffer capacity over limit when double it size * avoid empty dataset (#6756) * Fix dataframe arithmetics for columns having several value buffers (column size is more than 2 Gb) (#6724) * Fix dataframe arithmetics * Fix * Run tests that requires more than 2 Gb of Memory only on 64-bit env (#6758) * Reduce coupling of Data.Analysis.Tests project (#6759) * Provide ability to filter dataframe column by null via ElementWise Methods (#6723) * Provide ability to filter by null value * Add comments * Fix code review findings * Fix incorrect DataFrame min max computation with NULL (#6734) * Step 1 * Step 2 * Fixed code review findings * Clean DataFrame meaningless code (#6761) * Add NameEntityRecognition and Q&A deep learning tasks. (#6760) * NER * QA almost done, runtime error * QA finished * fixes from PR comments * fixed build * build fixes * perf changes * made disposable * fixed not disposing model * added some disposables to TensorFlow for memory * build testing * fixing build * added missing dispose * build fixes * build fixes * testing macos fix * fix issue (#6768) * fixed mac build and minor torch sharp changes (#6776) * Improve DataFrame Arithmetics implementation (#6763) * Change methods signature generation * Change DataFrameColumn Arithmetics * Change DataFrameColumn Operations * Fix unit tests * Fix spaces * Fix code review findings * Add QA sweepable estimator in AutoML (#6781) * Add QA sweepable * clean * Modernized some argument checks that still used string literals for parameter names (#6766) Co-authored-by: John Doe <john@doe> * removed deprecated yosemite brew (#6805) * Add TargetType to Type_convert (#6785) * Add target Type in convert type * Add custom type "DataKind" * clean * Add DataKind name space * clean test * File-scoped namespaces in files under `Environment` (`Microsoft.ML.Core`) (#6791) Co-authored-by: Lehonti Ramos <john@doe> * File-scoped namespaces in files under `EntryPoints` (`Microsoft.ML.Core`) (#6790) Co-authored-by: Lehonti Ramos <john@doe> * Fix issue with addIndexColumn in DataFrame.LoadCsv (#6769) * Fix issue with addIndexColumn in DataFrame.LoadCsv * Fix tests * Fix DataFrame.LoadCsv can not load CSV with duplicate column names (#6772) * File-scoped namespaces in files under `ComponentModel` (`Microsoft.ML.Core`) (#6788) Co-authored-by: Lehonti Ramos <john@doe> * File-scoped namespaces in files under `Data` (`Microsoft.ML.Core`) (#6789) Co-authored-by: Lehonti Ramos <john@doe> * Fix inconsistent null handling in DataFrame Arithmetics (#6770) * Fix inconsistent null handling in DataFrame Arithmetics * Fix Null Count and division by zero issues * Minor changes to restart build and rerun flaky tests * File-scoped namespaces in files under `Prediction` (`Microsoft.ML.Core`) (#6792) Co-authored-by: Lehonti Ramos <john@doe> * Allow to define CultureInfo for parsing values on reading DataFrame from csv (#6782) * Use CultureInfo for parsing values in csv file * Fix merge issues * Append dataframe rows based on column names (#6808) * Append dataframe rows based on column names * Update DataFrame.cs --------- Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com> * removed codecov token (#6811) * Fix wrong type conversion on PrimitiveDataFrameColumn (#6834) * Fix wrong type conversion on PrimitiveDataFrameColumn * Added tests for #6829 * Fix test * Add file generated from tt template and fix unit tests --------- Co-authored-by: Aleksei Smirnov <tlalok@inbox.ru> * update interactive kernel version (#6836) * update interactive kernel version * update * Update Microsoft.Data.Analysis.Interactive.Tests.csproj * Add performance benchmarks for dataframe arithmetic operations (#6827) * Add performance tests * Add extra tests * Fix * Fix typo * Fix Divide_Int16 and Divide_Int32_Int16 benchmarks * Fix * Change csproj file * Update BenchmarkDotNetVersion to 0.13.5 * Fix * Change to 0.13.1 because that is what is latest version in our nuget feeds. --------- Co-authored-by: Jake Radzikowski <JakeRad@Microsoft.com> * Improve performance of column cloning inside DataFrame arithmetics (#6814) * Optimize PrimitiveColumnContainer.Clone method * Avoid unnecessary type conversion during binary operations * Remove using * Fix DataFrameBuffer constructor * remove uncorrectly added using * Make DataFrameBuffer Length field protected * Fix typo * Use RawSpan * Simplify tt files for PrimitiveDataFrameColumnAritmetics (#6830) * First step of tt refactoring * Step 2 * Step 3 * Addresses #6533 (#6838) * Initial structure and started fleshing out some sections * Some corrections and paragraph on DL usages * Starting fleshing out DL on ML.NET section * Addresses #6533 * Update dependencies (#6837) * Update dependencies * Add reference to NuGet.Packaging.Core * PrimitiveDataFrameColumn.Clone method crashes when is used with IEnumerable mapIndices argument (#6822) * Split Test for AppendMany into 4 different tests * Block init of null validity buffer instead of setting individual bits * Add unit tests for PrimitiveDataFrameColumn.Clone * Fixes #6821 * Fix * Fix bug with AppendMany values to not empty column * Restart unit tests * Add more unit tests * Fix failing unit test * Fix code review findings * 6847 incorrectly sets column value (#6849) * Fix DataFrame incorrectly sets column value for index higher than Buffer.MaxCapacity * Revert renaming * Increase performance of arithmetic operations by enhancing calculations on nullable values (#6846) * Optimize PrimitiveColumnContainer.Clone method * Avoid unnecessary type conversion during binary operations * Remove using * Fix DataFrameBuffer constructor * remove uncorrectly added using * Make DataFrameBuffer Length field protected * Add performance tests * Split Test for AppendMany into 4 different tests * Block init of null validity buffer instead of setting individual bits * Add unit tests for PrimitiveDataFrameColumn.Clone * Fixes #6821 * Fix * Add extra tests * Fix * Fix typo * Fix Divide_Int16 and Divide_Int32_Int16 benchmarks * Fix * Avoid using constructor, that copies memory * First step of tt refactoring * Step 2 * Step 3 * Move iteration over buffers outside of the PrimitiveDataFrameColumnArithmetic * Change PrimitiveDataFrameColumnArithmetic * Fix typo * Use RawSpan * Fix bug with AppendMany values to not empty column * Restart unit tests * Add more unit tests * Add GetBitCount method * Fix failing unit test * Implementation * Change unit tests * Update unit tests * Refactoring BinaryOperation * Intermediate changes * Intermediate results * Implement Binary Scalar Reverse Operarions * Add implementation for BinaryIntOperations * Implement Comparison Operations * Implement actual calculations for Comparison operations * Uncomment performance tests * Remove unintentional code changes * Add reference to Apache Arrow project license in THIRD-PARTY-NOTICES * Fix license issues * Fixes incorrect work of DataFrame with VBufferColumn when number of e… (#6851) * Fixes incorrect work of DataFrame with VBufferColumn when number of elements is greater than Int.MaxValue * Fix calculation of max capacity and amount of required buffers * Fix unit test * Run test allocating more than 2 Gb of memory on 64bit env only * Fix StringDataFrameColumn same way as VBufferDataFrameColumn * Fix wrong amount of buffers created in constructor of StringDataFrameColumn * Fix code review findings --------- Co-authored-by: Aleksei Smirnov <tlalok@inbox.ru> Co-authored-by: Michael Sharp <51342856+michaelgsharp@users.noreply.github.com> Co-authored-by: Xiaoyun Zhang <xiaoyuz@microsoft.com> Co-authored-by: zewditu Hailemariam <36615490+zewditu@users.noreply.github.com> Co-authored-by: Lehonti Ramos <17771375+Lehonti@users.noreply.github.com> Co-authored-by: John Doe <john@doe> Co-authored-by: Raffaello Fraboni <10281615+novelhawk@users.noreply.github.com> Co-authored-by: R. G. Esteves <rodolfo.g.esteves@intel.com> Co-authored-by: Eric StJohn <ericstj@microsoft.com>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
fixes #6826