10000 [ML] Report the "actual" memory usage of the autodetect process by edsavage · Pull Request #2846 · elastic/ml-cpp · GitHub
[go: up one dir, main page]

Skip to content

[ML] Report the "actual" memory usage of the autodetect process #2846

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 51 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
a7b1ce2
[ML] Report the "actual" memory usage of the autodetect process
edsavage Apr 4, 2025
cb957ca
Update changelog
edsavage Apr 4, 2025
8f73f02
Formatting
edsavage Apr 4, 2025
d3a39ae
Appease SonarQube
edsavage Apr 7, 2025
e2d1bf5
Tweak unit test for platform portability
edsavage Apr 7, 2025
1a9a99a
Attend to review comments
edsavage Apr 9, 2025
5ae22cb
Update bin/autodetect/Main.cc
edsavage Apr 10, 2025
fe6f1fa
Update include/model/CResourceMonitor.h
edsavage Apr 10, 2025
582430e
Update include/model/CResourceMonitor.h
edsavage Apr 10, 2025
9476ede
Attend to review comments
edsavage Apr 11, 2025
e29d8f7
Merge remote-tracking branch 'origin/ad_real_mem_usage' into ad_real_…
edsavage Apr 11, 2025
475fef1
Formatting
edsavage Apr 11, 2025
fc8888c
Merge branch 'main' of github.com:elastic/ml-cpp into ad_real_mem_usage
edsavage May 23, 2025
6945d3f
Attend to failing unit tests (hopefully)
edsavage May 27, 2025
3b69b72
Formatting.. grr
edsavage May 27, 2025
1bafa8c
On Linux only, use the value of the system memory usage (max resident…
edsavage Jun 4, 2025
efc311b
Fix copyright headers
edsavage Jun 4, 2025
6be6395
Nits in test code
edsavage Jun 4, 2025
134a494
Attend to code review comments
edsavage Jun 9, 2025
ed426ac
Formatting
edsavage Jun 9, 2025
4c8bf8b
Attend to code review comments
edsavage Jun 11, 2025
fa9c4fa
Small tidy up of CProcessMemoryUsage.cc
edsavage Jun 12, 2025
014be1c
Small tidy up
edsavage Jun 12, 2025
41c02de
On Linux, return system memory (max resident set size) from CResource…
edsavage Jun 17, 2025
50c535f
[ML] Add a script to run each unit test separately
edsavage Jun 27, 2025
1c5c46e
Slight tweak to reduce load on linux builds
edsavage Jun 29, 2025
0109cce
Replace use of python script with bash function
edsavage Jun 30, 2025
01c1e6b
Pass all JUNIT result files to test collector
edsavage Jun 30, 2025
e2d2d17
Better reporting of unit test results
edsavage Jul 2, 2025
7904e76
Slight tweak to parameters to test script
edsavage Jul 2, 2025
185a0e0
Tweaks and typos
edsavage Jul 2, 2025
0a6a086
Portability fixes
edsavage Jul 3, 2025
6c97572
Fix failing test cases
edsavage Jul 3, 2025
40f522a
Merge branch 'main' of github.com:elastic/ml-cpp into run_tests_indiv…
edsavage Jul 3, 2025
733b038
Formatting
edsavage Jul 3, 2025
62dce12
Tidy up of scripts
edsavage Jul 4, 2025
55875c0
Further tidy up of scripts
edsavage Jul 4, 2025
9c67353
Tweaks for linux aarch64
edsavage Jul 4, 2025
d265102
Tweak for linux aarch64 seccomp test
edsavage Jul 6, 2025
2cf5752
Typo
edsavage Jul 6, 2025
3ace6f4
Better isolation of tests
edsavage Jul 8, 2025
4014a72
Formatting
edsavage Jul 8, 2025
e3d2f8a
Formatting
edsavage Jul 8, 2025
7b39899
Reworking test case
edsavage Jul 8, 2025
58fafb4
Small tidy up
edsavage Jul 9, 2025
033adef
Minor tweaks to test runner script
edsavage Jul 11, 2025
fe71d2a
typo
edsavage Jul 13, 2025
e8c00e7
Merge branch 'main' of github.com:elastic/ml-cpp into ad_real_mem_usage
edsavage Jul 14, 2025
13fa5cf
Merge branch 'run_tests_individually' of github.com:edsavage/ml-cpp i…
edsavage Jul 14, 2025
bbe25b1
Set both adjusted usage and adjusted peak usage to system memory.
edsavage Jul 25, 2025
3ab09ba
Formatting
edsavage Jul 25, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Small tidy up
* uncomment test code
* formatting
  • Loading branch information
edsavage committed Jun 12, 2025
commit 014be1ccb22d59995e7054c54c3987f6331cdc95
60 changes: 30 additions & 30 deletions lib/core/unittest/CNamedPipeFactoryTest.cc
Original file line number Diff line number Diff line change
Expand Up @@ -158,36 +158,36 @@ BOOST_AUTO_TEST_CASE(testServerIsCppWriter) {
BOOST_REQUIRE_EQUAL(std::string(TEST_SIZE, TEST_CHAR), threadReader.data());
}

// BOOST_AUTO_TEST_CASE(testServerIsCWriter) {
// ml::test::CThreadDataReader threadReader{PAUSE_TIME_MS, MAX_ATTEMPTS, TEST_PIPE_NAME};
// BOOST_TEST_REQUIRE(threadReader.start());
//
// std::atomic_bool dummy{false};
// ml::core::CNamedPipeFactory::TFileP file{
// ml::core::CNamedPipeFactory::openPipeFileWrite(TEST_PIPE_NAME, dummy)};
// BOOST_TEST_REQUIRE(file);
//
// sleep(1);
// std::size_t charsLeft{TEST_SIZE};
// std::size_t blockSize{7};
// while (charsLeft > 0) {
// if (blockSize > charsLeft) {
// blockSize = charsLeft;
// }
// BOOST_TEST_REQUIRE(std::fputs(std::string(blockSize, TEST_CHAR).c_str(),
// file.get()) >= 0);
// charsLeft -= blockSize;
// }
//
// file.reset();
//
// BOOST_TEST_REQUIRE(threadReader.waitForFinish());
// BOOST_TEST_REQUIRE(threadReader.attemptsTaken() <= MAX_ATTEMPTS);
// BOOST_TEST_REQUIRE(threadReader.streamWentBad() == false);
//
// BOOST_REQUIRE_EQUAL(TEST_SIZE, threadReader.data().length());
// BOOST_REQUIRE_EQUAL(std::string(TEST_SIZE, TEST_CHAR), threadReader.data());
// }
BOOST_AUTO_TEST_CASE(testServerIsCWriter) {
ml::test::CThreadDataReader threadReader{PAUSE_TIME_MS, MAX_ATTEMPTS, TEST_PIPE_NAME};
BOOST_TEST_REQUIRE(threadReader.start());

std::atomic_bool dummy{false};
ml::core::CNamedPipeFactory::TFileP file{
ml::core::CNamedPipeFactory::openPipeFileWrite(TEST_PIPE_NAME, dummy)};
BOOST_TEST_REQUIRE(file);

sleep(1);
std::size_t charsLeft{TEST_SIZE};
std::size_t blockSize{7};
while (charsLeft > 0) {
if (blockSize > charsLeft) {
blockSize = charsLeft;
}
BOOST_TEST_REQUIRE(std::fputs(std::string(blockSize, TEST_CHAR).c_str(),
file.get()) >= 0);
charsLeft -= blockSize;
}

file.reset();

BOOST_TEST_REQUIRE(threadReader.waitForFinish());
BOOST_TEST_REQUIRE(threadReader.attemptsTaken() <= MAX_ATTEMPTS);
BOOST_TEST_REQUIRE(threadReader.streamWentBad() == false);

BOOST_REQUIRE_EQUAL(TEST_SIZE, threadReader.data().length());
BOOST_REQUIRE_EQUAL(std::string(TEST_SIZE, TEST_CHAR), threadReader.data());
}

BOOST_AUTO_TEST_CASE(testCancelBlock) {
CThreadBlockCanceller cancellerThread{ml::core::CThread::currentThreadId()};
Expand Down
3 changes: 2 additions & 1 deletion lib/model/CProcessMemoryUsage.cc
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
namespace ml {
namespace model {
// On platforms other than Linux the process memory usage is the estimated size of the models.
const CProcessMemoryUsage::EMemoryStrategy CProcessMemoryUsage::MEMORY_STRATEGY{EMemoryStrategy::E_Estimated};
const CProcessMemoryUsage::EMemoryStrategy CProcessMemoryUsage::MEMORY_STRATEGY{
EMemoryStrategy::E_Estimated};
}
}
3 changes: 2 additions & 1 deletion lib/model/CProcessMemoryUsage_Linux.cc
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ namespace ml {
namespace model {

// On Linux the process memory usage is determined by the OS.
const CProcessMemoryUsage::EMemoryStrategy CProcessMemoryUsage::MEMORY_STRATEGY{EMemoryStrategy::E_System};
const CProcessMemoryUsage::EMemoryStrategy CProcessMemoryUsage::MEMORY_STRATEGY{
EMemoryStrategy::E_System};
}
}
0