-
Notifications
You must be signed in to change notification settings - Fork 852
call munmap more frequently? #48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
I repeated the tests and it seems munmap is now being called. Below is the output of the test run with the mmap and munmap operations plus the number of active mappings at the time of each mmap/munmap call. Furthermore, it is logged when the compaction started and finished. << SHELL SERVER TESTS (BASICS) >>./avocado "/tmp/vocdir.3904" --startup.directory ./js --startup.modules-path ./js/server/modules:./js/common/modules --action.system-directory ./js/actions/system --unit-tests ./js/common/tests/shell-document.js --unit-tests ./js/common/tests/shell-edge.js --unit-tests ./js/common/tests/shell-collection.js --unit-tests ./js/common/tests/shell-simple-query.js --unit-tests ./js/common/tests/shell-index.js --unit-tests ./js/common/tests/shell-index-geo.js --unit-tests ./js/common/tests/shell-cap-constraint.js || test "0" == "1" |
Calling MUNMAP as soon as possible. |
Right, munmap is called when the collection is unloaded. |
I cannot run the AvocadoDB unittests on my 32 bit laptop.
At some point during the test execution I am encountering mmap errors. mmap fails when creating datafiles and the database reports "cannot allocate memory".
I have marked all mmap and munmap calls to report the cumulated size of mapped memory.
Results below for running "make unittests":
|| SHELL SERVER TESTS ||
./avocado "/tmp/vocdir.28161" --unit-tests ./js/server/tests/shell-document.js --unit-tests ./js/server/tests/shell-collection.js --unit-tests ./js/server/tests/aql-simple.js --unit-tests ./js/server/tests/aql-keywords.js --unit-tests ./js/server/tests/aql-bind.js --unit-tests ./js/server/tests/aql-joins.js --unit-tests ./js/server/tests/aql-operators.js --unit-tests ./js/server/tests/aql-indexes.js
2012-04-11T15:40:10Z [28222] INFO no user init file '/home/steemann/.avocado/avocado.conf' found
2012-04-11T15:40:10Z [28222] INFO no system init file '/usr/local/etc/avocado.conf' found
2012-04-11T15:40:10Z [28222] INFO using built-in JavaScript startup files
2012-04-11T15:40:10Z [28222] INFO using database action files at '/tmp/vocdir.28161/_ACTIONS'
2012-04-11T15:40:10Z [28222] INFO using system action files at '/usr/local/share/avocado/js/actions/system'
2012-04-11T15:40:10Z [28222] INFO using JavaScript modules path '/usr/local/share/avocado/js/server/modules;/usr/local/share/avocado/js/common/modules'
AvocadoDB shell [V8 version 3.9.4, DB version 0.3.7]
2012-04-11T15:40:10Z [28222] INFO Running collectionDocumentSuiteErrorHandling
2012-04-11T15:40:10Z [28222] INFO 3 tests found
MMAP: total mmaped size is now: 2465792
MMAP: total mmaped size is now: 36020224
MMAP: total mmaped size is now: 69574656
2012-04-11T15:40:10Z [28222] INFO [PASSED] testErrorHandlingBadHandle
MMAP: total mmaped size is now: 72040448
MMAP: total mmaped size is now: 105594880
MMAP: total mmaped size is now: 139149312
2012-04-11T15:40:10Z [28222] INFO [PASSED] testErrorHandlingUnknownDocument
MMAP: total mmaped size is now: 141615104
MMAP: total mmaped size is now: 175169536
MMAP: total mmaped size is now: 208723968
2012-04-11T15:40:10Z [28222] INFO [PASSED] testErrorHandlingCrossCollection
2012-04-11T15:40:10Z [28222] INFO 3 tests passed
2012-04-11T15:40:10Z [28222] INFO 0 tests failed
2012-04-11T15:40:10Z [28222] INFO 392 milliseconds elapsed
2012-04-11T15:40:10Z [28222] INFO Running collectionDocumentSuiteReadDocument
2012-04-11T15:40:10Z [28222] INFO 5 tests found
MMAP: total mmaped size is now: 211189760
MMAP: total mmaped size is now: 244744192
MMAP: total mmaped size is now: 278298624
2012-04-11T15:40:10Z [28222] INFO [PASSED] testSaveDocument
MMAP: total mmaped size is now: 280764416
MMAP: total mmaped size is now: 314318848
MMAP: total mmaped size is now: 347873280
2012-04-11T15:40:10Z [28222] INFO [PASSED] testReadDocument
MMAP: total mmaped size is now: 350339072
MMAP: total mmaped size is now: 383893504
MMAP: total mmaped size is now: 417447936
2012-04-11T15:40:11Z [28222] INFO [PASSED] testReadDocumentConflict
MMAP: total mmaped size is now: 419913728
MMAP: total mmaped size is now: 453468160
MMAP: total mmaped size is now: 487022592
2012-04-11T15:40:11Z [28222] INFO [PASSED] testUpdateDocument
MMAP: total mmaped size is now: 489488384
MMAP: total mmaped size is now: 523042816
MMAP: total mmaped size is now: 556597248
2012-04-11T15:40:11Z [28222] INFO [PASSED] testDeleteDocument
2012-04-11T15:40:11Z [28222] INFO 5 tests passed
2012-04-11T15:40:11Z [28222] INFO 0 tests failed
2012-04-11T15:40:11Z [28222] INFO 911 milliseconds elapsed
2012-04-11T15:40:11Z [28222] INFO Running documentSuiteErrorHandling
2012-04-11T15:40:11Z [28222] INFO 2 tests found
2012-04-11T15:40:11Z [28222] INFO [PASSED] testErrorHandlingBadHandle
MMAP: total mmaped size is now: 559063040
MMAP: total mmaped size is now: 592617472
MMAP: total mmaped size is now: 626171904
2012-04-11T15:40:11Z [28222] INFO [PASSED] testErrorHandlingUnknownDocument
2012-04-11T15:40:11Z [28222] INFO 2 tests passed
2012-04-11T15:40:11Z [28222] INFO 0 tests failed
2012-04-11T15:40:11Z [28222] INFO 77 milliseconds elapsed
2012-04-11T15:40:11Z [28222] INFO Running documentSuiteReadDocument
2012-04-11T15:40:11Z [28222] INFO 4 tests found
MMAP: total mmaped size is now: 628637696
MMAP: total mmaped size is now: 662192128
MMAP: total mmaped size is now: 695746560
2012-04-11T15:40:11Z [28222] INFO [PASSED] testReadDocument
MMAP: total mmaped size is now: 698212352
MMAP: total mmaped size is now: 731766784
MMAP: total mmaped size is now: 765321216
2012-04-11T15:40:11Z [28222] INFO [PASSED] testReadDocumentConflict
MMAP: total mmaped size is now: 767787008
MMAP: total mmaped size is now: 801341440
MMAP: total mmaped size is now: 834895872
2012-04-11T15:40:12Z [28222] INFO [PASSED] testUpdateDocument
MMAP: total mmaped size is now: 837361664
MMAP: total mmaped size is now: 870916096
MMAP: total mmaped size is now: 904470528
2012-04-11T15:40:12Z [28222] INFO [PASSED] testDeleteDocument
2012-04-11T15:40:12Z [28222] INFO 4 tests passed
2012-04-11T15:40:12Z [28222] INFO 0 tests failed
2012-04-11T15:40:12Z [28222] INFO 697 milliseconds elapsed
2012-04-11T15:40:12Z [28222] INFO 14 total, 14 passed, 0 failed, 2077 ms
2012-04-11T15:40:12Z [28222] INFO Running collectionSuiteErrorHandling
2012-04-11T15:40:12Z [28222] INFO 6 tests found
2012-04-11T15:40:12Z [28222] INFO [PASSED] testErrorHandlingBadNameUnderscore
2012-04-11T15:40:12Z [28222] INFO [PASSED] testErrorHandlingBadNameEmpty
2012-04-11T15:40:12Z [28222] INFO [PASSED] testErrorHandlingBadNameNumber
2012-04-11T15:40:12Z [28222] INFO [PASSED] testErrorHandlingBadNameUnderscoreShortCut
2012-04-11T15:40:12Z [28222] INFO [PASSED] testErrorHandlingBadNameEmptyShortCut
2012-04-11T15:40:12Z [28222] INFO [PASSED] testErrorHandlingBadNameNumberShortCut
2012-04-11T15:40:12Z [28222] INFO 6 tests passed
2012-04-11T15:40:12Z [28222] INFO 0 tests failed
2012-04-11T15:40:12Z [28222] INFO 1 millisecond elapsed
2012-04-11T15:40:12Z [28222] INFO Running collectionSuite
2012-04-11T15:40:12Z [28222] INFO 18 tests found
MMAP: total mmaped size is now: 906936320
MMAP: total mmaped size is now: 940490752
MMAP: total mmaped size is now: 974045184
2012-04-11T15:40:12Z [28222] INFO [PASSED] testReadingByName
MMAP: total mmaped size is now: 976510976
MMAP: total mmaped size is now: 1010065408
MMAP: total mmaped size is now: 1043619840
2012-04-11T15:40:12Z [28222] INFO [PASSED] testReadingByIdentifier
MMAP: total mmaped size is now: 1046085632
MMAP: total mmaped size is now: 1079640064
MMAP: total mmaped size is now: 1113194496
2012-04-11T15:40:12Z [28222] INFO [PASSED] testReadingByNameShortCut
MMAP: total mmaped size is now: 1115660288
MMAP: total mmaped size is now: 1149214720
MMAP: total mmaped size is now: 1182769152
2012-04-11T15:40:12Z [28222] INFO [PASSED] testReadingAll
MMAP: total mmaped size is now: 1185234944
MMAP: total mmaped size is now: 1218789376
MMAP: total mmaped size is now: 1252343808
2012-04-11T15:40:12Z [28222] INFO [PASSED] testCreatingDefaults
MMAP: total mmaped size is now: 1254809600
MMAP: total mmaped size is now: 1255858176
MMAP: total mmaped size is now: 1256906752
2012-04-11T15:40:12Z [28222] INFO [PASSED] testCreatingProperties
MMAP: total mmaped size is now: 1259372544
MMAP: total mmaped size is now: 1292926976
MMAP: total mmaped size is now: 1326481408
2012-04-11T15:40:12Z [28222] INFO [PASSED] testDroppingNewBornDB
MMAP: total mmaped size is now: 1328947200
MMAP: total mmaped size is now: 1362501632
MMAP: total mmaped size is now: 1396056064
2012-04-11T15:40:12Z [28222] INFO [PASSED] testDroppingLoadedDB
MMAP: total mmaped size is now: 1398521856
MMAP: total mmaped size is now: 1432076288
MMAP: total mmaped size is now: 1465630720
2012-04-11T15:40:13Z [28222] INFO [PASSED] testDroppingUnloadedDB
MMAP: total mmaped size is now: 1468096512
MMAP: total mmaped size is now: 1501650944
MMAP: total mmaped size is now: 1535205376
2012-04-11T15:40:13Z [28222] INFO [PASSED] testTruncatingNewBornDB
MMAP: total mmaped size is now: 1537671168
MMAP: total mmaped size is now: 1571225600
2012-04-11T15:40:13Z [28222] ERROR cannot memory map file '/tmp/vocdir.28161/collection-32699786/journal-33748362.db': 'Cannot allocate memory'
2012-04-11T15:40:13Z [28222] ERROR cannot create new journal in '/tmp/vocdir.28161/collection-32699786/journal-33748362.db'
(same error repeats here...)
As can be seen above, the first few tests (shell-document.js and shell-collection.js) already amount for 1.5 GB of memory mapped file data. This seems to be the maximum that my computer supports.
1.5 GB of free RAM should be sufficient to run a few tests I think.
I think the problem is that the tests create new collections in the setUp routine. When creating a new collection, mmap is called for it. This is fine, however, munmap is not called when the collections are dropped when tests execute their tearDown routine.
That means the mmap'ed value will only increase, at least while the tests are running. It be helpful if munmap could be called when a collection is explicitly dropped (or at least shortly after) to free resources.
The text was updated successfully, but these errors were encountered: