Menu Home

Beating the vendor by 3 weeks

So a client has a vendor supplied python script that routinely connects to said vendor’s servers and downloads files. I won’t go into too much detail, but it’s a pretty important file transfer with far reaching financial influence, and we’ll leave it exactly at that.

So, as you can guess by the existence of this post, the file transfer stopped working. A colleague looked into it, didn’t see anything immediately obvious and logged a call with the vendor. Our team leader shoulder tapped me for a second opinion, and here’s a sanitised version of the email I sent to him and my colleague, based on testing in a dev environment. This was promptly put into change control and pushed into DR and PROD environments.

Three weeks later, the vendor got back to us with fix recommendations which were essentially what I found out below.

Think I earned a payrise/holiday/bottle of scotch with this one? Brace yourselves, gents.

As far as I understand it, python typically uses openssl, so we can manually walk through the SSL communication like this:

$ openssl s_client -connect vendor.com:443 < /dev/null
CONNECTED(00000003)
depth=1 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
0 s:/C=US/ST=Texas/L=Allen/O=Ven Dor/CN=vendor.com
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
1 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2

The last line is of interest to us - that’s the root CA in this certificate chain, and because it’s missing, the intermediate certificate (L1K) is complaining. If we grab the root CA from Entrust and manually feed it into openssl, note how we get a complete chain now:

 
$ openssl s_client -connect vendor.com:443 -CAfile entrustG2.pem < /dev/null
CONNECTED(00000003)
depth=2 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2009 Entrust, Inc. - for authorized use only", CN = Entrust Root Certification Authority - G2
verify return:1
depth=1 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
verify return:1
depth=0 C = US, ST = Texas, L = Allen, O = "Ven Dor", CN = vendor.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=Texas/L=Allen/O=Ven Dor/CN=vendor.com
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
1 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2

So, we move the pem file to /etc/pki/ca-trust/source/anchors and run the following commands to enter it into the OS CA store:

update-ca-trust force-enable
update-ca-trust extract
update-ca-trust

The first one we need to force because it will otherwise complain about the existence of a 32 bit nss library that in this instance we don’t care about. Now we run openssl again without declaring the CAfile to check:

 
$ openssl s_client -connect vendor.com:443 < /dev/null | head -20
depth=2 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2009 Entrust, Inc. - for authorized use only", CN = Entrust Root Certification Authority - G2
verify return:1
depth=1 C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
verify return:1
depth=0 C = US, ST = Texas, L = Allen, O = "Ven Dor", CN = vendor.com
verify return:1
DONE
CONNECTED(00000003)
---
Certificate chain
0 s:/C=US/ST=Texas/L=Allen/O=Ven Dor/CN=vendor.com
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
1 s:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2012 Entrust, Inc. - for authorized use only/CN=Entrust Certification Authority - L1K
   i:/C=US/O=Entrust, Inc./OU=See www.entrust.net/legal-terms/OU=(c) 2009 Entrust, Inc. - for authorized use only/CN=Entrust Root Certification Authority - G2

Ok, so openssl is happy. BUT it turns out that this doesn’t fix the problem, it does get us a little bit closer though. The problem is we're calling python3 when, upon checking, this script is called via python2.7, and even after applying the above fix, running the faulty script shows it's still faulty. So...

strace /usr/local/bin/python2.7 /path/to/brokenscript.py

has this tucked away in there:

 
open("/usr/local/lib/python2.7/site-packages/requests/cacert.pem", O_RDONLY) = 4

Well then, python2.7 isn’t calling the OS CA store. Let’s confirm with not-python2.7:

 
strace python -c "import requests; r = requests.get('https://vendor.com', verify=True);"

This correctly uses the OS CA store (open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 4), but if we point to python2.7, we don’t even need strace:

 
$ /usr/local/bin/python2.7 -c "import requests; r = requests.get('https://vendor.com');"
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 68, in get
    return request('get', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)

So, process of elimination continues. Let’s point python2.7 at the OS CA store (no output = good news):

 
$ /usr/local/bin/python2.7 -c "import requests; r = requests.get('https://vendor.com', verify='/etc/pki/tls/certs/ca-bundle.crt');"
$

Okay, so I grabbed the newest cacert.pem file from here: https://pypi.python.org/pypi/certifi, checked it included the G2 cert, backed up /usr/local/lib/python2.7/site-packages/requests/cacert.pem, put the new cacert.pem in and tested again:

 
$ /usr/local/bin/python2.7 -c "import requests; r = requests.get('https://vendor.com');"
$

Okay, good. Good. Now let’s try the script just to prove that it’s all good now:

 
$ !1859
/usr/local/bin/python2.7 /path/to/brokenscript.py >&  /var/log/brokenscript.log
$ cat /var/log/brokenscript.log
Package: NZL   ; Vintage: 2015-09-30
Requesting download URI for file /path/to/blah.
Downloading [redacted]...
Downloaded file /path/to/blah.
[rinse and repeat for the backlog of files]
All data files up-to-date.

Do I make it look easy? Because this really wasn’t. Damned gratifying though, I love figuring out puzzles. It gives you that rarely experienced sense of achievement.

Categories: Geeking Out Lunix Lunacy

rawiri