Python Profile Memory Usage And Object Graph Example

Python memory monitor is very important for debug application performance and fix bug. This article will introduce two popular python modules, memory_profiler and objgraph. memory_profiler can monitor your app code memory usage for each line of code, objgraph can display the python objects relationship and generate an image to visualize it.

1. memory_profiler.

This module can monitor each line of python source code used memory. It will display line number, memory usage, memory usage increment and source code.

  1. Install memory_profiler.
    192:~ $ pip install memory_profiler
  2. Decorate python source code with @profile annotation.
    @profile
    def read_big_file(big_file_path):
        # Open file with read permission.
        file = open(big_file_path, 'rb')
        # Read to buffer.
        buf = file.read(100*1024*1024)
       
        # Read until EOF
        while(buf):
            buf = file.read(100*1024*1024)
    
    if __name__ == '__main__':
        read_big_file('/Users/zhaosong/Documents/WorkSpace/tool/mysql-8.0.11-macos10.13-x86_64.tar.gz')
  3. Save above code in ProfileMemoryExample.py file and run it with -m memory_profiler argument. Then the memory usage data will be listed in the console.
    192:file $ python -m memory_profiler ProfileMemoryExample.py
    Filename: ProfileMemoryExample.py
    Line #    Mem usage    Increment   Line Contents
    ================================================
         7   30.234 MiB   30.234 MiB   @profile
         8                             def read_big_file(big_file_path):
         9   30.238 MiB    0.004 MiB       file = open(big_file_path, 'rb')
        10  130.242 MiB  100.004 MiB       buf = file.read(100*1024*1024)
        11                                 
        12  158.586 MiB    0.000 MiB       while(buf):
        13  158.586 MiB   28.344 MiB           buf = file.read(100*1024*1024)

2. objgraph.

objgraph module can be used to describe the python objects information and their relationship in your python source code. It will generate a .dot file in a folder, and if you install graphviz tool, it can help you to convert the .dot file to a png image file. Below example will show you how to use it.

READ :   How To Make A Website With Python And Django

My OS is MacOS, for other OS, the steps are similar.

  1. Install objgraph use pip install.
    192:file $ pip install objgraph
  2. After that, do not install xdot, if you install xdot may lead to below error when execute.
    192:file zhaosong$ python ProfileMemoryExample.py 
    Graph written to /var/folders/hs/fg0h6drj18l6m3v56m3756fc0000gn/T/objgraph-8j7fyoxz.dot (2 nodes)
    Image renderer (dot) not found, not doing anything else
    
  3. The red error message tell you that, dot is not installed, and dot is included in graphviz, so you should install graphviz to install dot also. In MacOS terminal, run below command to install graphviz.
    192:file $ brew install graphviz
  4. After successfully install graphviz, the brew console will tell you that graphviz is installed in folder /usr/local/Cellar/graphviz/2.40.1. and there is a folder bin in it. You can find dot executable file in the bin folder.
    dot executable file in macos graphviz bin folder
  5. Now run cd ~ command to goto your home directory and edit your profile file to add graphviz bin folder in the system environment PATH variable so that objgraph can find the dot command in the PATH value.
    # go to user home directory.
    192:bin $ cd ~
    # edit .bash_profile file, add graphviz bin folder ( /usr/local/Cellar/graphviz/2.40.1/bin ) in the PATH environment variable value. 
    192:~  $ vim .bash_profile
    # Make the PATH value change take effect.
    192:~ $ source .bash_profile
    192:~ zhaosong$ env
    PATH=/usr/local/Cellar/graphviz/2.40.1/bin:/usr......
  6. Now save below code in file ProfileMemoryExample.py .
    import objgraph
    
    def read_text_file_objgraph(text_file_path):
        # Save all text file data in a list. One item save one line text.
        line_list = []
    
        # Open text file return file object.
        file = open(text_file_path, 'r')
    
        # Read one line.
        line = file.readline()
    
        # While not reach End Of File.
        while(len(line) > ):
            # Append the line string to list.
            line_list.append(line)
            # Read next line.
            line = file.readline()
    
        print("********************* objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png') ************************")    
        # Display line_list and file reference objects. Generate the Python objects relations in a png file with the file name.     
        objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png')
    
        print("********************* objgraph.show_most_common_types() ************************")
        # Display most common types in console.
        objgraph.show_most_common_types()
    
        print("********************* objgraph.show_growth(limit=5) ************************")
        # Display common type growth in console.
        objgraph.show_growth(limit=5)
    
    if __name__ == '__main__':
        read_text_file_objgraph('./csv2json/json_user_info.json')
  7. Open a terminal and run above python file. Then you can find the object references graph png file has been generated in current execution folder.
    192:file $ python ProfileMemoryExample.py 
    ********************* objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png') ************************
    Graph written to /var/folders/hs/fg0h6drj18l6m3v56m3756fc0000gn/T/objgraph-cv84v7m_.dot (12 nodes)
    Image generated as read_text_file_objgraph.png
    
    ********************* objgraph.show_most_common_types() ************************
    function                   2119
    dict                       1181
    wrapper_descriptor         1002
    tuple                      951
    weakref                    866
    method_descriptor          724
    builtin_function_or_method 662
    set                        454
    getset_descriptor          383
    list                       359
    
    ********************* objgraph.show_growth(limit=5) ************************
    function               2119     +2119
    dict                   1181     +1181
    wrapper_descriptor     1002     +1002
    tuple                   950      +950
    weakref                 866      +866

    read_text_file_objgraph

Reference

  1. Python Object Graphs
  2. Graphviz – Graph Visualization Software
(Visited 30 times, 1 visits today)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.