Python help, specifically with urllib2

Feek · 28 Jul 2017 at 17:52

Hi,

I use a Raspberry Pi as an internet watchdog but I'm not entirely convinced I have the code right.

This is supposed to check for the presence of a web site, and wait for up to four minutes for a response.

However, I don't think the timeout bit is working correctly. I think that if it can't access the site it's trying to then it errors straight away.

Here's the full script I'm using. I've part pinched it from the internet and tweaked it a bit.

Code:

import urllib2
import RPi.GPIO as GPIO
import time

RELAY = 21 #RELAY PIN, physical pin 40
ALIVE = 20 #LED PIN to flash LED on live internet, physical pin 38
CHECK_EVERY = 10 #Seconds

GPIO.setmode(GPIO.BCM)
GPIO.setup(RELAY, GPIO.OUT)
GPIO.setup(ALIVE, GPIO.OUT)

# Force both outputs low
GPIO.output(RELAY, GPIO.LOW)
GPIO.output(ALIVE, GPIO.LOW)

def flash_ok():
    GPIO.output(ALIVE, GPIO.HIGH)
        time.sleep(0.2)
        GPIO.output(ALIVE, GPIO.LOW)


def internet_on():
    try:
        response=urllib2.urlopen('https://www.website.com', timeout=240)
          flash_ok()
        return True
    except urllib2.URLError as err: pass
    return False

def reboot_router():
    print "Rebooting Router"
    GPIO.output(RELAY, GPIO.HIGH)
    time.sleep(10) #Energise relay for ten seconds
    GPIO.output(RELAY, GPIO.LOW)
    time.sleep(240) #Wait for four minutes for router to reboot

while True:
    if not internet_on():
        reboot_router()
    time.sleep(CHECK_EVERY)

GPIO.cleanup()

This is the part of the code in question.

Code:

def internet_on():
   try:
        response=urllib2.urlopen('https://www.website.com', timeout=240)
          flash_ok()
        return True
    except urllib2.URLError as err: pass
    return False

Have I got something wrong here, specifically in the timeout bit?

I'm (clearly) not a Python programmer, just someone who finds code and attempts to tweak it!

Thanks for any advice/suggestions.

Rroff · 28 Jul 2017 at 17:56

response=urllib2.urlopen('https://www.website.com, timeout=240)

should that not have a ' at the end of the URL? as in:

response=urllib2.urlopen('https://www.website.com', timeout=240)

Feek · 28 Jul 2017 at 18:01

My fault on the copy/paste where I edited out the exact url I'm checking for. Yes it does, I've added it back in the code above, sorry.

Rroff · 28 Jul 2017 at 18:02

Don't really have my head in python at the moment but you may need a conditional check against socket.timeout as part of that function to get what you want.

D.P. · 28 Jul 2017 at 18:23

Code:
Code:

def internet_on(): try: response=urllib2.urlopen('https://www.website.com', timeout=240) flash_ok() return True except urllib2.URLError as err: pass return False

Have I got something wrong here, specifically in the timeout bit?

I'm (clearly) not a Python programmer, just someone who finds code and attempts to tweak it!

Thanks for any advice/suggestions.

Instead of pass-ing the exception why not print it out and then you can see if there is an actual exception being thrown.

You can easily test the behavior of your code. Just give it a broken URL and it will fail, I doubt it will wait 4 minutes before the timeout is raised and instead will raise some URLError exception to which you are oblivious.

Feek · 28 Jul 2017 at 18:48

Good suggestion, thanks.

I've added two lines just to give me an idea of what's happening.

Code:

def internet_on():
    try:
        response=urllib2.urlopen('https://www.unavailable.com', timeout=240)
        print "It's working"
        flash_ok()
        return True
    except urllib2.URLError as err: pass
    print "it's broken"
    return False

And yes, it errors after just a few seconds so the timeout isn't doing what it's supposed to.

The suggestion above about needing a conditional check against socket.timeout may be a good one but I really wouldn't know where to start when it comes to adding that.

D.P. · 28 Jul 2017 at 22:27

I was more thinking:

Code:

def internet_on():
    try:
        response=urllib2.urlopen('https://www.unavailable.com', timeout=240)
        print "It's working"
        flash_ok()
        return True
    except urllib2.URLError as err: 
        print "it's broken,url error, reason = ", err
   exception urllib.error.HTTPErroras err:
       print "it's broken, http error, reason = ", err
   return False

But I fully expect if there is some http error then it simply wont wait. For example:

Code:

response = urllib2.urlopen('http://a_bad_URL.org/',timeout=240)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 1228, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/Users/timstirling/anaconda/lib/python2.7/urllib2.py", line 1198, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

The timeout makes no difference because it isn't blocking on an HTTP connection

Feek · 29 Jul 2017 at 08:57

Right, I tried the first code you've added there, I had to make some changes to actually make it work (all part of the learning process, that's good). It looks like this now:

Code:

def internet_on():
    try:
        response=urllib2.urlopen('https://www.udfsdf5sdg.com', timeout=240)
        print "It's working"
        flash_ok()
        return True
    except urllib2.URLError as err:
        print "It's broken, url error, reason = ", err
    except urllib2.HTTPError as err:
       print "It's broken, http error, reason = ", err
    return False

And you're right, the timeout still does nothing, I get the following after just a few seconds:

Code:

It's broken, url error, reason =  <urlopen error [Errno -2] Name or service not known>

I'm really not sure what that last code segment you've posted is all about.

mrkev · 29 Jul 2017 at 15:47

Is that failing on the hostname which happens before the URL open? In other words, the hostname has to resolve first. Try putting a duff IP address in instead to test the timeout option.

MTA99 · 30 Jul 2017 at 14:19

Reading this I was sure I'd had a similar issue. Just searched my code and found this:

Code:

        # Set timeout to 30s to stop bad connections ruining my life
        socket.setdefaulttimeout(30)

This is using httplib2 so YMMV with urlib2.

If you can switch to requests you can use the timeout=x parameter.

I'll test it out now and report back...

Feek · 30 Jul 2017 at 14:27

Switching to requests wouldn't be a problem - I just want go get this working (and of course, understand what I'm doing at the same time).

MTA99 · 30 Jul 2017 at 15:24

Code:

import urllib2
import socket
# import RPi.GPIO as GPIO
from time import gmtime, strftime, sleep


RELAY = 21  # RELAY PIN, physical pin 40
ALIVE = 20  # LED PIN to flash LED on live internet, physical pin 38
CHECK_EVERY = 10  # Seconds

# GPIO.setmode(GPIO.BCM)
# GPIO.setup(RELAY, GPIO.OUT)
# GPIO.setup(ALIVE, GPIO.OUT)
#
# # Force both outputs low
# GPIO.output(RELAY, GPIO.LOW)
# GPIO.output(ALIVE, GPIO.LOW)


# def flash_ok():
#     GPIO.output(ALIVE, GPIO.HIGH)
#     time.sleep(0.2)
#     GPIO.output(ALIVE, GPIO.LOW)


def internet_on():
    # This is where the magic happens! ;)
    socket.setdefaulttimeout(3)
    print "{}: Internet On".format(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
    try:
        response = urllib2.urlopen('http://www.fakeresponse.com/api/?data={"hello":"World"}&sleep=5')
        print "{}: It's working".format(strftime("%Y-%m-%d %H:%M:%S", gmtime()))
        # flash_ok()
        return True
    except urllib2.URLError as err:
        print "It's broken, url error, reason = {}".format(err)
    except urllib2.HTTPError as err:
        print "It's broken, http error, reason = {}".format(err)
    except socket.error as err:
        print "It's broken, socket error, reason = {}".format(err)
    return False


def reboot_router():
    print "Rebooting Router"
    # GPIO.output(RELAY, GPIO.HIGH)
    # time.sleep(10)  # Energise relay for ten seconds
    # GPIO.output(RELAY, GPIO.LOW)
    # time.sleep(240)  # Wait for four minutes for router to reboot


while True:
    if not internet_on():
        reboot_router()
    sleep(CHECK_EVERY)

# GPIO.cleanup()

It's basically working by adding the "socket.setdefaulttimeout(3)" line.

If you run this code it will fail at socket timeout because the url waits 5 secs to respond but the socket timeout is 3 secs

Make the sockettimeout 10 at it will work as expected.

The rest of the changes are to add timestamps to the print statements and "fix" some coding style

Oh and since I don't have the rpi libs installed I had to comment out all the references to them.

Feek · 30 Jul 2017 at 16:27

That's good, yes I can see that and I can actually see how it's working, thanks.

The web page you're calling doesn't complete the load for five seconds - Providing I increase the setdefaulttimeout to higher than that then I should be OK. If I set it to 180 then it'll wait for three minutes before it errors.

Is that a legit URL to carry on using or is that likely to change? Would using something like google.co.uk be more reliable?

So simply replace

Code:

        response = urllib2.urlopen('http://www.fakeresponse.com/api/?data={"hello":"World"}&sleep=5')

With

Code:

        response = urllib2.urlopen('https://www.google.co.uk')

Looks like I'm nearly there

MTA99 · 30 Jul 2017 at 17:07

I would definitely swap fakeresponse.com for google.co.uk. The only reason it's there is to check the timeout condition.

Feek · 30 Jul 2017 at 19:16

Very good, thank you. It's all working now and hopefully I won't get any more false triggers (where it times out too quickly).