Using requests module, how to handle 'set-cookie' in request response?

Question

I'm attempting to open a login page (GET), fetch the cookies provided by the webserver, then submit a username and password pair to log into the site (POST).

Looking at this Stackoverflow question/answer, I would think that I would just do the following:

import requests
from http.cookiejar import CookieJar

url1 = 'login prompt page'
url2 = 'login submission URL'
jar = CookieJar()

r = requests.get(url1, cookies=jar)
r2 = requests.post(url2, cookies=jar, data="username and password data payload")

However, in r there is a Set-Cookie in the headers, but that isn't changing the jar object. In fact, nothing is being populated into jar as the linked question's response would indicate.

I'm getting around this in my code by having a headers dict and after doing the GET or POST, using this to handle the Set-Cookie header:

headers['Cookie'] = r.headers['set-cookie']

Then passing around the headers in the requests methods. Is this correct, or is there a better way to apply the Set-Cookie?

Answer 1

Ignore the cookie-jar, let requests handle cookies for you. Use a Session object instead, it'll persist cookies and send them back to the server:

with requests.Session() as s:
    r = s.get(url1)
    r = s.post(url2, data="username and password data payload")

Answer 2

There's an included class called a session which automatically handles this sort of thing for you. You can create an instance of it, and then call get and set right on that instance instead.

import requests

URL1 = 'login prompt page'
URL2 = 'login submission URL'

session = requests.Session()

r = session.get(URL1)
r2 = session.post(URL2, data="username and password data payload")

Answer 3

Another way that has worked for me (without using session objects) is the following (tested in v2.18.4).

jar = requests.cookies.RequestsCookieJar()
response1 = requests.get(some_url, cookies=jar) # or post ...
jar.update(response1.cookies)
response2 = requests.get(some_other_url, cookies=jar) # or post ...

Note that the above code will fail in the presence of redirects which are handled transparently by the Requests library. In such a case, you also have to update your jar with the cookies sent in the redirect responses. E.g. by doing something like the following:

if (response.history): # we have to add to the cookie jar, the cookies sent by the server in intermediate responses 
    for historicResponse in response.history:
        jar.update(historicResponse.cookies)