How To Use Python Requests Module To Send Get Or Post Request Example

Leave a Comment / Python Tutorial / Python Requests

Python requests module provide functions to write web request process code easy and simple. It can send both http get and post request to web server. When you send get request to web server, you can use python request module to pass parameters. When you send post request to web server, you can use python request module to post form data. It can also send or get http headers, cookies and send files. This article will focus on send get or post request, we will write other article to show headers, cookies and session management.

1. How To Install Python Requests Module.

1.1 Install Python Requests Module Use Pip Command.

  1. Run $ pip (pip3) show requests command to check whether python requests module has been installed on your operating system or not. If nothing print out, then it means python requests module has not been installed. 
    $ pip3 show requests
  2. If python requests module is not installed, then run $ pip3 install requests in terminal to install it like below. 
    $ pip3 install requests
Collecting requests
  Downloading https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl (57kB)
     |████████████████████████████████| 61kB 124kB/s 
Collecting idna<2.9,>=2.5 (from requests)
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
     |████████████████████████████████| 61kB 152kB/s 
Collecting certifi>=2017.4.17 (from requests)
  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)
     |████████████████████████████████| 163kB 127kB/s 
Collecting chardet<3.1.0,>=3.0.2 (from requests)
  Downloading https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl (133kB)
     |████████████████████████████████| 143kB 127kB/s 
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests)
  Downloading https://files.pythonhosted.org/packages/e6/60/247f23a7121ae632d62811ba7f273d0e58972d75e58a94d329d51550a47d/urllib3-1.25.3-py2.py3-none-any.whl (150kB)
     |████████████████████████████████| 153kB 104kB/s 
Installing collected packages: idna, certifi, chardet, urllib3, requests
Successfully installed certifi-2019.3.9 chardet-3.0.4 idna-2.8 requests-2.22.0 urllib3-1.25.3
  3. Now run $ pip (pip3) show requests again, you can see python requests module installation information like below, this means it has been installed successfully. 
    $ pip3 show requests
Name: requests
Version: 2.22.0
Summary: Python HTTP for Humans.
Home-page: http://python-requests.org
Author: Kenneth Reitz
Author-email: [email protected]
License: Apache 2.0
Location: /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages
Requires: urllib3, certifi, idna, chardet
Required-by:

1.2 Install Python Requests Module From Source Code.

Beside use pip command line, you can also install python request module from source code.

  1. Run command $ wget https://github.com/requests/requests/tarball/master  in a terminal to get python requests module source code to a local directory, the directory name is master. 
    $ wget https://github.com/requests/requests/tarball/master
--2019-06-17 10:11:48--  https://github.com/requests/requests/tarball/master
Resolving github.com (github.com)... 52.74.223.119
Connecting to github.com (github.com)|52.74.223.119|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/kennethreitz/requests/tarball/master [following]
--2019-06-17 10:11:49--  https://github.com/kennethreitz/requests/tarball/master
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://codeload.github.com/kennethreitz/requests/legacy.tar.gz/master [following]
--2019-06-17 10:11:49--  https://codeload.github.com/kennethreitz/requests/legacy.tar.gz/master
Resolving codeload.github.com (codeload.github.com)... 13.229.189.0
Connecting to codeload.github.com (codeload.github.com)|13.229.189.0|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/x-gzip]
Saving to: ‘master’

master                                                                [               <=>                                                                                                                                          ]   2.92M   760KB/s    in 4.0s    

2019-06-17 10:11:54 (741 KB/s) - ‘master’ saved [3066122]
  2. Run $ ls -l in command line to verify the master directory has been created. 
    $ ls -l
total 6208
......
-rw-r--r--   1 songzhao  staff  3066122 Jun 17 10:11 master
......
  3. Run $ mv master requests.tgz command to move the master directory to a zip file. Run $ ls -l command again to verify the zip file has been created. 
    $ mv master requests.tgz
$ ls -l
......
-rw-r--r--   1 songzhao  staff  3066122 Jun 17 10:11 requests.tgz
......
  4. Run $ tar xzf requests.tgz command to uncompress the tgz file to a local folder. Run ls -l to check the local folder. From below output, we can see the local folder name is kennethreitz-requests-4983a9b. 
    $ tar xzf requests.tgz

$ ls -l
total 6208
......
drwxr-xr-x  26 songzhao  staff      832 Jun  3 22:01 kennethreitz-requests-4983a9b
-rw-r--r--   1 songzhao  staff  3066122 Jun 17 10:11 requests.tgz
......
  5. Run cd kennethreitz-requests-4983a9b to go into the uncompressed directory, and you can list all the files in the directory. 
    $ cd kennethreitz-requests-4983a9b/

$ ls -l

total 280
-rw-r--r--   1 songzhao  staff   7688 Jun  3 22:01 AUTHORS.rst
-rw-r--r--   1 songzhao  staff     96 Jun  3 22:01 CODE_OF_CONDUCT.md
-rw-r--r--   1 songzhao  staff   3010 Jun  3 22:01 CONTRIBUTING.md
-rw-r--r--   1 songzhao  staff  48996 Jun  3 22:01 HISTORY.md
-rw-r--r--   1 songzhao  staff    582 Jun  3 22:01 LICENSE
-rw-r--r--   1 songzhao  staff    122 Jun  3 22:01 MANIFEST.in
-rw-r--r--   1 songzhao  staff    861 Jun  3 22:01 Makefile
-rw-r--r--   1 songzhao  staff    442 Jun  3 22:01 Pipfile
-rw-r--r--   1 songzhao  staff  36746 Jun  3 22:01 Pipfile.lock
-rw-r--r--   1 songzhao  staff   4037 Jun  3 22:01 README.md
drwxr-xr-x   3 songzhao  staff     96 Jun  3 22:01 _appveyor
-rw-r--r--   1 songzhao  staff   1659 Jun  3 22:01 appveyor.yml
drwxr-xr-x  13 songzhao  staff    416 Jun  3 22:01 docs
drwxr-xr-x   4 songzhao  staff    128 Jun  3 22:01 ext
-rw-r--r--   1 songzhao  staff     33 Jun  3 22:01 pytest.ini
drwxr-xr-x  20 songzhao  staff    640 Jun  3 22:01 requests
-rw-r--r--   1 songzhao  staff     63 Jun  3 22:01 setup.cfg
-rwxr-xr-x   1 songzhao  staff   3312 Jun  3 22:01 setup.py
drwxr-xr-x  15 songzhao  staff    480 Jun  3 22:01 tests
-rw-r--r--   1 songzhao  staff     84 Jun  3 22:01 tox.ini
  6. Now run $ pip install .  to install it. 
    $ pip install .
Processing /Users/songzhao/kennethreitz-requests-4983a9b
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (3.0.4)
Requirement already satisfied: idna<2.9,>=2.5 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (1.25.3)
Requirement already satisfied: certifi>=2017.4.17 in /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages (from requests==2.22.0) (2019.3.9)
Installing collected packages: requests
  Found existing installation: requests 2.22.0
    Uninstalling requests-2.22.0:
      Successfully uninstalled requests-2.22.0
  Running setup.py install for requests ... done
Successfully installed requests-2.22.0

2. How To Send Http Get Request Use Python Requests Module.

You can use python requests module’s get function to send http get request, the get function has a string value parameter which is a web page url. The page url must start with http or https protocol, if not it will throw a requests.exceptions.MissingSchema exception. Below is an example.

  1. Open a terminal and run command python3 to enter python interactive mode. 
    $ python3
Python 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 16:52:21) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
  2. First you should import python requests module as below. 
    >>> import requests
  3. Then call requests module’s get function with provided url string, the url string should start with http or https protocol. 
    >>> response = requests.get('http://www.dev2qa.com')
  4. If the url string do not start with http or https protocol, then it will throw requests.exceptions.MissingSchema exception. 
    >>> response = requests.get('www.dev2qa.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/sessions.py", line 519, in request
    prep = self.prepare_request(req)
  File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/sessions.py", line 462, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/models.py", line 313, in prepare
    self.prepare_url(url, params)
  File "/Users/songzhao/kennethreitz-requests-4983a9b/requests/models.py", line 387, in prepare_url
    raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'www.dev2qa.com': No schema supplied. Perhaps you meant http://www.dev2qa.com?
>>>
  5. When the requests module’s get function execute successfully, you can get a response object, you can call various methods or property of the response object, such as get http response status code with status_code property. 
    >>> response.status_code
200
  6. Get http response encoding with encoding property. 
    >>> response.encoding
'UTF-8'
  7. Get request url web page content use content or text property. Please note the content property return the binary format of the web page, you can see the content is prefixed with b. The text property return the text format of the web page content. 
    >>> response.content
b'<!DOCTYPE html><html lang="en-US">....</html>

>>> response.text
'<!DOCTYPE html><html lang="en-US">....</html>'
  8. Get response raw object by call response.raw property. We can see the raw property return a urllib3.response.HTTPResponse object in below code. 
    >>> response.raw
<urllib3.response.HTTPResponse object at 0x1101b5ef0>

3. How To Pass Parameters In Http Get Request URL.

Create a python dictionary object to save query string key value pair, then pass this dictionary object to requests module’s get method’s params argument.

>>> import requests
>>>
# Create dictionary object to save query string key value pair.
>>> query_string = {'user_name':'jerry', 'password':'jerry'}
>>>
# Pass above object to get function's params argument.
>>> response = requests.get('http://www.dev2qa.com', params=query_string)
>>>
# Display the request url. You can see the query string in the url.
>>> response.url
'https://www.dev2qa.com/?user_name=jerry&password=jerry'

4. How To Send Http Post Request Use Python Requests Module.

Invoke requests module’s post function to send post data to special form url. The post data should be saved in a python dictionary object. Then pass this dictionary object to the post function’s data parameter.

# First import python requests module.
>>> import requests
>>> 

# Create a python dictionary object to contain post data. The dictionary object item data format is form_element_name: form_element_value. 
>>> post_data={'user_name':'jerry', 'password':'password'}
>>> 

# Invoke post function to post above dictionary data to special form url.
>>> response = requests.post("http://www.dev2qa.com", data=post_data)
>>> 
>>> response.url
'https://www.dev2qa.com/'

Leave a Comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.