Python memory monitor is very important for debugging application performance and fix bugs. This article will introduce two popular python modules, memory_profiler, and objgraph. memory_profiler can monitor your app code memory usage for each line of code, objgraph can display the python objects relationship and generate an image to visualize it.
1. memory_profiler.
This module can monitor each line of python source code used memory. It will display line number, memory usage, memory usage increment, and source code.
- Install memory_profiler.
192:~ $ pip install memory_profiler
- Decorate python source code with @profile annotation.
@profile def read_big_file(big_file_path): # Open file with read permission. file = open(big_file_path, 'rb') # Read to buffer. buf = file.read(100*1024*1024) # Read until EOF while(buf): buf = file.read(100*1024*1024) if __name__ == '__main__': read_big_file('/Users/zhaosong/Documents/WorkSpace/tool/mysql-8.0.11-macos10.13-x86_64.tar.gz') - Save the above code in the python source file ProfileMemoryExample.py and run it with -m memory_profiler argument. Then the memory usage data will be listed in the console.
$ python -m memory_profiler ProfileMemoryExample.py Filename: ProfileMemoryExample.py Line # Mem usage Increment Line Contents ================================================ 7 30.234 MiB 30.234 MiB @profile 8 def read_big_file(big_file_path): 9 30.238 MiB 0.004 MiB file = open(big_file_path, 'rb') 10 130.242 MiB 100.004 MiB buf = file.read(100*1024*1024) 11 12 158.586 MiB 0.000 MiB while(buf): 13 158.586 MiB 28.344 MiB buf = file.read(100*1024*1024)
2. objgraph.
The objgraph module can be used to describe the python objects’ information and their relationship in your python source code. It will generate a .dot file in a folder, and if you install graphviz tool, it can help you to convert the .dot file to a png image file. The below example will show you how to use it.
My OS is macOS, for other OS, the steps are similar.
- Install objgraph use pip install.
$ pip install objgraph
- After that, do not install xdot, if you install xdot may lead to the below error when execute.
$ python ProfileMemoryExample.py Graph written to /var/folders/hs/fg0h6drj18l6m3v56m3756fc0000gn/T/objgraph-8j7fyoxz.dot (2 nodes) Image renderer (dot) not found, not doing anything else
- The above error message tells you that, the Image renderer ( dot ) is not installed, and dot is included in graphviz, so you should install graphviz to install dot also. In the macOS terminal, run the below command to install graphviz.
$ brew install graphviz
- After successfully install graphviz, the brew console will tell you that graphviz is installed in a folder like /usr/local/Cellar/graphviz/2.40.1, and there is a folder bin in it. You can find the dot executable file in the bin folder.
- Now run
cd ~command to go to your home directory and edit your profile file to add graphviz bin folder in the system environment PATH variable so that module objgraph can find the dot command in the PATH value.# go to user home directory. 192:bin $ cd ~ # edit .bash_profile file, add graphviz bin folder ( /usr/local/Cellar/graphviz/2.40.1/bin ) in the PATH environment variable value. 192:~ $ vim .bash_profile # Make the PATH value change take effect. 192:~ $ source .bash_profile 192:~ $ env PATH=/usr/local/Cellar/graphviz/2.40.1/bin:/usr......
- Now save the below code in the file ProfileMemoryExample.py.
import objgraph def read_text_file_objgraph(text_file_path): # Save all text file data in a list. One item save one line text. line_list = [] # Open text file return file object. file = open(text_file_path, 'r') # Read one line. line = file.readline() # While not reach End Of File. while(len(line) > 0): # Append the line string to list. line_list.append(line) # Read next line. line = file.readline() print("********************* objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png') ************************") # Display line_list and file reference objects. Generate the Python objects relations in a png file with the file name. objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png') print("********************* objgraph.show_most_common_types() ************************") # Display most common types in console. objgraph.show_most_common_types() print("********************* objgraph.show_growth(limit=5) ************************") # Display common type growth in console. objgraph.show_growth(limit=5) if __name__ == '__main__': read_text_file_objgraph('./csv2json/json_user_info.json') - Open a terminal and run the above python file. Then you can find a .png image file ( contains the objects references graph ) has been generated in the current execution folder.
192:file $ python ProfileMemoryExample.py ********************* objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png') ************************ Graph written to /var/folders/hs/fg0h6drj18l6m3v56m3756fc0000gn/T/objgraph-cv84v7m_.dot (12 nodes) Image generated as read_text_file_objgraph.png ********************* objgraph.show_most_common_types() ************************ function 2119 dict 1181 wrapper_descriptor 1002 tuple 951 weakref 866 method_descriptor 724 builtin_function_or_method 662 set 454 getset_descriptor 383 list 359 ********************* objgraph.show_growth(limit=5) ************************ function 2119 +2119 dict 1181 +1181 wrapper_descriptor 1002 +1002 tuple 950 +950 weakref 866 +866
Reference
