Linear RAM memory increase with Dataset and Estimator with epoch loops #32052
Description
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04.2 LTS
- Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
- TensorFlow installed from (source or binary): Binary
- TensorFlow version (use command below): tensorflow-gpu==2.0.0-rc0
- Python version: 3.7.1
- Bazel version (if compiling from source): NA
- GCC/Compiler version (if compiling from source): NA
- CUDA/cuDNN version: Cuda 10.1 / cuDNN 7.6
- GPU model and memory: GeForce GTX 1060 Mobile 6GB
You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
When using Dataset with Estimator, the memory foot print of RAM keeps raising when estimator's train and evaluate APIs are called in loop.
Describe the expected behavior
RAM usage should not increase with epochs.
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
Please find the source code @ https://gist.github.com/Mageswaran1989/facc3fc2a003807d029a914c721629db
[Update] My latest test case @ https://github.com/dhiraa/tf_issue_32052
StackOverflow Rereference : https://stackoverflow.com/questions/55211315/memory-leak-with-tf-data
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
[Updated the graph]
