Python Profile Memory Usage And Object Graph Example

Python memory monitor is very important for debugging application performance and fix bugs. This article will introduce two popular python modules, memory_profiler, and objgraph. memory_profiler can monitor your app code memory usage for each line of code, objgraph can display the python objects relationship and generate an image to visualize it.

1. memory_profiler.

This module can monitor each line of python source code used memory. It will display line number, memory usage, memory usage increment, and source code.

  1. Install memory_profiler.
    192:~ $ pip install memory_profiler
  2. Decorate python source code with @profile annotation.
    def read_big_file(big_file_path):
        # Open file with read permission.
        file = open(big_file_path, 'rb')
        # Read to buffer.
        buf =*1024*1024)
        # Read until EOF
            buf =*1024*1024)
    if __name__ == '__main__':
  3. Save the above code in the python source file and run it with -m memory_profiler argument. Then the memory usage data will be listed in the console.
    $ python -m memory_profiler
    Line #    Mem usage    Increment   Line Contents
         7   30.234 MiB   30.234 MiB   @profile
         8                             def read_big_file(big_file_path):
         9   30.238 MiB    0.004 MiB       file = open(big_file_path, 'rb')
        10  130.242 MiB  100.004 MiB       buf =*1024*1024)
        12  158.586 MiB    0.000 MiB       while(buf):
        13  158.586 MiB   28.344 MiB           buf =*1024*1024)

2. objgraph.

The objgraph module can be used to describe the python objects’ information and their relationship in your python source code. It will generate a .dot file in a folder, and if you install graphviz tool, it can help you to convert the .dot file to a png image file. The below example will show you how to use it.

My OS is macOS, for other OS, the steps are similar.

  1. Install objgraph use pip install.
    $ pip install objgraph
  2. After that, do not install xdot, if you install xdot may lead to the below error when execute.
    $ python 
    Graph written to /var/folders/hs/fg0h6drj18l6m3v56m3756fc0000gn/T/ (2 nodes)
    Image renderer (dot) not found, not doing anything else
  3. The above error message tells you that, the Image renderer ( dot ) is not installed, and dot is included in graphviz, so you should install graphviz to install dot also. In the macOS terminal, run the below command to install graphviz.
    $ brew install graphviz
  4. After successfully install graphviz, the brew console will tell you that graphviz is installed in a folder like /usr/local/Cellar/graphviz/2.40.1, and there is a folder bin in it. You can find the dot executable file in the bin folder.
  5. Now run cd ~ command to go to your home directory and edit your profile file to add graphviz bin folder in the system environment PATH variable so that module objgraph can find the dot command in the PATH value.
    # go to user home directory.
    192:bin $ cd ~
    # edit .bash_profile file, add graphviz bin folder ( /usr/local/Cellar/graphviz/2.40.1/bin ) in the PATH environment variable value. 
    192:~  $ vim .bash_profile
    # Make the PATH value change take effect.
    192:~ $ source .bash_profile
    192:~ $ env
  6. Now save the below code in the file
    import objgraph
    def read_text_file_objgraph(text_file_path):
        # Save all text file data in a list. One item save one line text.
        line_list = []
        # Open text file return file object.
        file = open(text_file_path, 'r')
        # Read one line.
        line = file.readline()
        # While not reach End Of File.
        while(len(line) > 0):
            # Append the line string to list.
            # Read next line.
            line = file.readline()
        print("********************* objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png') ************************")    
        # Display line_list and file reference objects. Generate the Python objects relations in a png file with the file name.     
        objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png')
        print("********************* objgraph.show_most_common_types() ************************")
        # Display most common types in console.
        print("********************* objgraph.show_growth(limit=5) ************************")
        # Display common type growth in console.
    if __name__ == '__main__':
  7. Open a terminal and run the above python file. Then you can find a .png image file ( contains the objects references graph ) has been generated in the current execution folder.
    192:file $ python 
    ********************* objgraph.show_refs([line_list, file], filename='read_text_file_objgraph.png') ************************
    Graph written to /var/folders/hs/fg0h6drj18l6m3v56m3756fc0000gn/T/ (12 nodes)
    Image generated as read_text_file_objgraph.png
    ********************* objgraph.show_most_common_types() ************************
    function                   2119
    dict                       1181
    wrapper_descriptor         1002
    tuple                      951
    weakref                    866
    method_descriptor          724
    builtin_function_or_method 662
    set                        454
    getset_descriptor          383
    list                       359
    ********************* objgraph.show_growth(limit=5) ************************
    function               2119     +2119
    dict                   1181     +1181
    wrapper_descriptor     1002     +1002
    tuple                   950      +950
    weakref                 866      +866



  1. Python Object Graphs
  2. Graphviz – Graph Visualization Software

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.