How To Get / Set Http Headers, Cookies And Manage Sessions Use Python Requests Module

In the previous article How To Use Python Requests Module To Send Get Or Post Request Example, we have learned how to install and use the python requests module to send HTTP Get and Post request to a web server. It also tells you how to post form data or pass query string parameters use the python requests module. This article will tell you how to use the python requests module to manage HTTP headers, cookies, and sessions.

1. Get / Set HTTP Headers Use Python Requests Module.

1.1 Get Server Response HTTP Headers.

Python requests module’s headers property is used to get HTTP headers. The headers property is a dictionary-type object, you should provide the header name to get the header value.

>>> import requests

>>> response = requests.get("http://www.dev2qa.com")

>>> response.headers['content-type']
'text/html; charset=UTF-8'

If you want to get all headers, just call response.headers, it will list out all HTTP response headers in a JSON format string.

>>> response.headers

{'Date': 'Mon, ...... 'Content-Encoding': 'gzip'}

1.2 Add Custom Headers To HTTP Request.

import requests

# Create a python dictionary object to save custom http headers. 
custom_header = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'}

# Pass above custom_header dictionary object to requests module's get function's headers parameter.
response = requests.get('https://www.dev2qa.com', headers=custom_header)

2. Get / Set Cookies Use Python Requests Module.

2.1 Python Get HTTP Cookies.

  1. Get all HTTP response cookies by invoke response.cookies property. This property is an instance of requests.cookies.RequestsCookieJar class.
    >>> import requests
    
    >>> response = requests.get('http://www.dev2qa.com')
    
    >>> response.cookies
    <RequestsCookieJar[Cookie(versio...]>
  2. You can loop to print out each cookie’s data such as the domain, name, and cookie value like below.
    >>> for cookie in response.cookies:
    ...     print('cookie domain = ' + cookie.domain)
    ...     print('cookie name = ' + cookie.name)
    ...     print('cookie value = ' + cookie.value)
    ...     print('*************************************')
    ... 
    cookie domain = .dev2qa.com
    cookie name = __cfduid
    cookie value = d8dccd301af7c19f3286442254fe10aa51560824472
    *************************************
    cookie domain = .dev2qa.com
    cookie name = active_template::80464
    cookie value = pub_site.1560824472
    *************************************
    ......
  3. The RequestsCookieJar class also provide methods such as items(), iteritems(), iterkeys(), itervalues(), keys(), values() to loop all cookie’s key or values in response ( each cookie has a key and a value. ).
    >>> for item in rsponse.cookies.items():
    ...     print(item)
    ...
    ('__cfduid', 'd0c1cc559af0893eb31cd23bb9bc22a261560866202')
    ('active_template::80464', 'pub_site.1560866203')
    ('ezCMPCCS', 'true')
    ('ezepvv', '92')
    ('ezoab_80464', 'mod31')
    ('ezoadgid_80464', '-1')
    ('ezopvc_80464', '1')
    ('ezoref_80464', '')
    ('ezovid_80464', '1774096901')
    ('ezovuuid_80464', '2d40f1eb-b006-4540-75cb-09a37a3b8387')
    ('ezovuuidtime_80464', '1560866206')
    ('lp_80464', 'https://www.dev2qa.com/?user_name=jerry&password=jerry')
    
    
    >>> for item in cookies.iterkeys():
    ...     print(item)
    ...
    __cfduid
    active_template::80464
    ezCMPCCS
    ezepvv
    ezoab_80464
    ezoadgid_80464
    ezopvc_80464
    ezoref_80464
    ezovid_80464
    ezovuuid_80464
    ezovuuidtime_80464
    lp_80464
  4. Get cookie value by cookie name.
    # Get cookie value by provided cookie name.
    >>> response.cookies['ezCMPCCS']
    'true'
  5. Get all cookies by domain.
    # Use get_dict function to get related domain's cookies.
    >>> response.cookies.get_dict('.dev2qa.com')
    {'__cfduid': 'd8dccd301af7c19f3286442254fe10aa51560824472', 'active_template::80464': 'pub_site.1560824472', 'ezCMPCCS': 'true', 'ezepvv': '4265', 'ezoab_80464': 'mod40', 'ezoadgid_80464': '-2', 'ezopvc_80464': '1', 'ezoref_80464': '', 'ezovid_80464': '1493552757', 'ezovuuid_80464': 'd073b9a0-ff53-454e-7f7b-877b4d21718e', 'ezovuuidtime_80464': '1560824473', 'lp_80464': 'https://www.dev2qa.com/'}
  6. List response cookie domain.
    >>> response.cookies.list_domains()
    ['.dev2qa.com']

2.2 Python Send HTTP Cookies To Web Server.

  1. Send cookie use dictionary object.
    # Import python requests module.
    >>> import requests
    >>> 
    # Set url value.
    >>> url = 'https://www.dev2qa.com'
    >>> 
    # Create a dictionary object.
    >>> cookies = dict(name='jerry', password='888')
    >>> 
    # Use python requests module to get related url and send cookies to it with cookies parameter. 
    >>> response = requests.get(url, cookies=cookies)
  2. Send cookie use RequestsCookieJar object.
    >>> import requests
    >>> 
    >>> url = 'https://www.dev2qa.com'
    >>> 
    # Create a RequestsCookieJar object.
    >>> cookies_jar = requests.cookies.RequestsCookieJar()
    >>> 
    # Add first cookie, the parameters are cookie_key, cookie_value, cookie_domain, cookie_path.
    >>> cookies_jar.set('name', 'jerry', domain='dev2qa.com', path='/cookies')
    Cookie(version=0, name='name', value='jerry', port=None, port_specified=False, domain='dev2qa.com', domain_specified=True, domain_initial_dot=False, path='/cookies', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)
    >>>
    # Add second cookie.
    >>> cookies_jar.set('password', 'jerry888', domain='dev2qa.com', path='/cookies')
    Cookie(version=0, name='password', value='jerry888', port=None, port_specified=False, domain='dev2qa.com', domain_specified=True, domain_initial_dot=False, path='/cookies', path_specified=True, secure=False, expires=None, discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False)
    >>> 
    # Get url with cookie parameter.
    >>> response = requests.get(url, cookies=cookies_jar)

3. Use Session In Python Requests Module.

Python requests module’s session() method will return a request.sessions.Session object, then all the later operations ( such as to browse a related URL page ) on this session object will use one same session.

>>> import requests
>>>
# Call requests module's session() method to return a requests.sessions.Session object.
>>> session = requests.session()
>>> 
>>> print(session)
<requests.sessions.Session object at 0x1025f8e10>

The returned request.sessions.Session objects provide a lot of attributes and methods for you to get related headers, cookie value in the same session. The session object also provides a get method to request a web page by URL. Below is an example of how to use the python requests module session object.

# Show all headers and cookies in this session.
>>> session.headers
{'User-Agent': 'python-requests/2.22.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive'}
>>> 
>>> session.cookies
<RequestsCookieJar[]>
>>>
# Use this session object to get a web page by url.
>>> response = session.get('http://www.dev2qa.com')
>>>
# When above browse web page process complete, this session has cookies.
>>> session.cookies
<RequestsCookieJar[Cookie(......)]


# You can add custom headers for this session when get web page by url.
>>>
>>> custom_header = {'hello': 'hello_888'}

# Get web page url and send custom headers.
>>>
>>> response = session.get('http://www.dev2qa.com', headers=custom_header)

The below link will show you python requests module session class source code, you can read it if you need it.

0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

2 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
David
David

I use python requests module to access a web page ( a.jsp ) and this web page only allows login user to access. So I need to first request the web site login page (login.jsp) to login, and the page will set a user account cookie in the response, and I want to append the login cookies in my next request to the a.jsp web page. How can I implement this? Thanks.

2
0
Would love your thoughts, please comment.x
()
x