Python Requests to Check Link Status

As I’ve mentioned here before, I’m currently involved with some ongoing research regarding ‘hyperlocal’ sites (due to me running this one), and I spotted one of the people involved mentioning on twitter that they were having to check out ~650 RSS feeds to see which ones were alive and which ones weren’t.

Of course, being the geeky type, I decided it’d be interesting to give it a quick try.  Here’s the full code to start off with.

Before we get started, this is a fairly long post mainly aimed at people with very little experience in the world of coding and Python.

So, what are we doing here?  First off, we’re importing the libraries we need.  requests is the most important bit here, and a massively useful tool that I find myself using fairly frequently.  It’s an HTTP library that makes web services really easy to handle, often with only a few lines of code.  It’s well worth checking out.

Other than that, we’re importing sqlite3 so that we’ve got somewhere to store the data,  along with the sys library (for reasons I actually can’t remember!).

ol_uri = ''

This is just the URL where we’ll be grabbing the JSON data from, which is handily stored by OpenlyLocal.

Now, we need to set up somewhere to store our results. SQLite is perfect for our needs here, as there’s no requirement for multiple user access, and we just want something fast and lightweight.

First off, we need to make a connection to our database file. It doesn’t matter if the file doesn’t exist yet, as it’ll be created the first time we run our code.

con = lite.connect('ol.db')

Next, we need to create a cursor (a kind of temporary reference to our table) to run our database commands on.

cursor = con.cursor()

…and then we need to set up the database. Here I’m DROPping the table first, ensuring that we remove any past traces of the table in case we’re running this multiple times, then executing a CREATE command to recreate our hyperdb table.

cursor.execute("DROP TABLE IF EXISTS hyperdb")
cursor.execute("CREATE TABLE hyperdb (title TEXT, feed TEXT, RSSAlive TEXT )"

Now that our storage is setup, we’re ready to do some of the heavier work and get our hands on some live data. Fortunately, because we’ve got the grunt of the request library behind us, this is really easy to do.

Ready for this?

#grab the JSON data
r = requests.get(ol_uri)
objs = r.json()

That’s literally it. That’s all we needed to do to get our hands on some hot, live data. We tell requests where to get the info (remember our ol_uri variable earlier?), and then we ask it to format it as JSON, which we’re storing in the rather unimaginatively named objs for now.

From here, we can manipulate the data pretty much as we want, so let’s do exactly that.

for index, site in enumerate(objs):

We’ve got a FOR..NEXT loop here (although this being Python, we don’t need to tell it the scope of the loop, it knows due to the formatting), which goes through each item in objs and will run the code underneath.

It’s worth noting here that we’ve slightly made the FOR..NEXT loop slightly more complex by using the enumerate() function . This is purely so we can print the current index of the site in the list we’re iterating over. We could just as easily replace it with the perfectly acceptable:

for site in objs:

..only this wouldn’t allow us to print the index later, which for this example makes it harder to see that your code is running.

Getting back to our FOR loop, this is the code that will execute on each object within objs.

    current = site['hyperlocal_site']
    print index
    sitename = current['title']
    sitefeed = current['feed_url']
    RSSAlive = "NO FEED"

Each of our site objects within objs contains something called “hyperlocal_site”, which contains all the information we want for each site. We can see this in the raw JSON at

{"hyperlocal_site":{"created_at":"2011-03-11T14:42:36+00:00","title":"AbergelePost","country":"Wales","updated_at":"2011-03-12T17:55:27+00:00","party_affiliation":"","feed_url":"","distance_covered":4.0,"url":"","lng":-3.58704,"id":845,"metres":{"coordinates":[294301.803348089,377978.785978634],"type":"Point"},"area_covered":"Abergele","hyperlocal_group_id":null,"council_id":52,"description":"Abergele local news, history and community information.","lat":53.2882,"platform":"Wordpress"}}

To make things a little easier to read, I’m storing the title and feed_url properties from site into some variables (sitename and sitefeed), along with another variable which will store the status of the sites RSS feed(RSSAlive).

How can we check if the RSS feed is alive? Again, it’s nice and easy using the requests library.

    #is the RSS feed alive?
        if sitefeed:
            f = requests.get(sitefeed)
            if f.status_code != 200:
                RSSAlive = "dead"
                RSSAlive = "alive!"
        RSSAlive = "Errored"
    sitedata = sitename, sitefeed, RSSAlive

We can explain this fairly simply – where the site has an RSS feed (‘if sitefeed:’) then ask requests to try and open it. If the server returns a status code of 200, the RSS feed is presumably alive. If not, then we report back it’s dead.

The TRY / EXCEPT section is simply because the server won’t always respond in a way we’re expecting. If it gives us a return code of any form, then it’s fine, but sometimes something unexpected might happen which might result in our code breaking, and our script halting. We don’t want that to happen, so we tell Python to ignore any errors* and simply return ‘Errored’ if something untoward happened in the process.

In the last line of the excerpt above, we do something that you can’t do in many languages – we pack three separate variables into one variable, called a tuple. One variable that carries everything else is easier to carry around, right?

Finally, we need to store our data. Again, it’s nice and easy.

    cursor.execute('insert into hyperdb values (?,?,?)',  sitedata)

We tell the cursor what we want to do (store our variables into the three fields of our previously created database) and look – because sitedata is holding the site name, the RSS feed, and RSSAlive, we don’t need to individually unpack them. We can just tell Python to use sitedata, because it knows it’s really three things stored in one place.

Clever, huh?

*as a general rule, it’s bad practise to completely ignore any error – it might well be something urgent we actually want to stop on, or maybe it’s something we need to deal with differently. In the real world, best practise is to explicitly state how you want exception to be handled.

Monitor FOI

A couple of months ago, around the middle of January, I received a press release from the office of our local MP, Aidan Burley, entitled ”MP SUBMITS PLANS FOR TRANSFERRING CANNOCK HOSPITAL TO THE COMMUNITY” (yes, it’s all in caps. They always are for some reason – maybe it’s a PR thing I’ve not been made aware of yet).

The fundamentals of the release are contained quite nicely in the opening paragraph of the email, which reads:

Chase MP Aidan Burley has today submitted his first report to Health RegulatorMonitor setting out how Cannock Hospital could be transferred at a peppercorn rent to the local community to own and run.

This is interesting stuff.  The Mid Staffs NHS trust, which handles both Cannock and the now  notorious Stafford hospital, has been claiming poverty for quite some time, stating that their services are unsustainable while borrowing heavily from the Ministry of Health.  Cannock, viewed as a minor hospital as it has no Accident and Emergency ward and, somewhat bizarrely, only around a 40% occupancy rate, was starting to look like a potential scapegoat for the cash-strapped authority, especially as health service watchdog Monitor stepped in to oversee management of the trust.

Of course, there has never been a direct suggestion that Cannock is to be sold.  Mid Staffs NHS chiefs have repeatedly stated that they will be doing all they can to continue provisions at both Cannock and Stafford, and specialist treatment continues to happen at Cannock hospital.  However, this hasn’t stopped local people from being concerned about the potential future of the site, especially when cut-backs were announced in the local press.

Mr Burley, the MP for Cannock Chase, has seemingly been trying to put into place an alternate plan, presumably in case Cannock Hospital does get earmarked for sale.  Initially, he suggested that the council should purchase the hospital, retaining a majority stake to ensure that the services continued.  However, the local council couldn’t see how this was feasible, as it would involve a large sum of money that simply wouldn’t be in any budget.

This is where we return to where we started – the MPs plans to retain services by transferring ownership to local people, who would pay a ‘peppercorn rent‘ in order to maintain the hospital.  Now, before we go any further, local people don’t seem to be keen in this idea.  From visiting and sitting in on NHS ‘Question Time’ style meetings, quite a number of people angered quickly at the suggestion that they pay extra for a service they already pay for via their tax.

The PR article includes a few brief statements, but never really gives any detail as to how this would work, or what kind of payments constitutes a ‘peppercorn rent’ exactly.  So, being the good little local news prodder that I am, I asked the obvious question – where’s the report?

Hi Alex,

Thanks for the information.
Is the full report publicly available?

Seems simple enough.  It’s a report involving the future of a locally-provisioned hospital, which is set to suggest that people pay an extra fee in order to retain the services, so it’s of public interest to see the detail, yes?


Hi Ian,

It is not currently publically available as it is being considered by Monitor.

Oh.  Well, maybe if I ask Locality, the group identified in the PR piece as being involved in creating the report?

I believe that you have been working on an Outline Business Plan for Cannock Hospital, pertaining to how the hospital could be released to the local community to run.
As a local community organisation, I’d really be interested to read the full report but unfortunately I don’t appear to be able to find anywhere to access it.
Is it at all possible to obtain a copy of the report?
Many thanks,
A little bit more detail this time, but still the same answer:

Dear Ian,

Many thanks for your enquiry and your interest.

The outline business case for Cannock Hospital has been prepared at the request of Aiden Burley MP and is currently confidential as it is under consideration by Monitor. My understanding is that Mr Burley is planning to publish it at some stage in the future, although as yet I have no details. I will let you know once I have such information.

Many thanks

Neal Trup
Financial Analyst

This is a bit.. odd.  I’m sure there’s no real reason why the detail wouldn’t be released publicly, as it’s related to the community and potentially has an effect on what they’ll be paying to retain what’s essentially a fundamentally service.

I went to twitter and checked out a few things.  Apparently, you can’t perform an FOI request against an MP, and nor can one be applied to Locality.

However, you can FOI Monitor, which is where the report was headed – so guess what I did next? I put in my request, waited patiently for the FOI to be dealt with, and got a response.

Monitor holds the information which you have requested, however I have decided not to allow you access to the information on the basis of the exemption under the FOI Act as described below.

Section 41 of the FOI Act – Information provided in Confidence
Section 41 provides an exemption from the right to know if the information in question was
provided to Monitor in confidence. There are 2 components to the exemption:
1. the information must have been obtained by Monitor from another person, and
2. disclosure of the information would give rise to an actionable breach of confidence.

The first component of the exemption is satisfied because Monitor obtained the information from
Aidan Burly, MP for Cannock Chase.

You can read the full response here: Mellet FOI Decision Letter 13.02.2013 (PDF).

So, as a resident of Cannock, I’m still not actually allowed to read a report mentioned in a PR email from Cannocks MP regarding Cannock Hospital.


Creative Citizens Post – 10/01/2013

As a few people may know, I’m currently involved with some interesting research conducted by.. well, a bunch of people, but mainly (I believe) carried out between Birmingham and Cardiff University.  As part of the ongoing research for the Hyperlocal Publish segment of the Creative Citizens project, I’ve been asked to create a kind of diary, logging my thoughts on various aspects of my Hyperlocal site, Connect Cannock.

I’d been trying to decide about where to put this, as there’s a few things I’m (again) toying with, but in the end, I’ve decided to keep it on this site, and, because I do believe it’s best to be transparent, I’m trying to keep it open and public.  If there’s anything I’m uncomfortable with publishing publicly, then, as per my normal internal set of rules, I’ll mention why publicly, and keep the detail so that only the research team and I can read it.

These posts will be short – somewhere around 100 words – and might be completely useless, so don’t expect any startling insights… this post has the obligatory blurb just so people understand why the hell I’m putting almost stream of consciousness on here, with no links etc.

Todays Entry

As far as actual news, things are a little quiet as far as the blog front goes (without looking, I think there’s been a maximum of two posts since the New Year), but this seems to be fairly normal for early January – bearing in mind that much of the input for the site comes from press releases which I then try, dependent on time and interest, to delve into, many places aren’t really doing much and are just getting back up to speed from Christmas.

Sarah, my partner and the sites photographer has posted up an event for a ‘photo walk’ around the  area, which seems like a good idea.  I like the idea in general anyway, as there’s a lot around Cannock that’s fairly photogenic, and the fact that it will help promote the site ‘offline’ as well is never a bad thing, and something I’d really like to try and do more of.

As usual, we seem to be more active on twitter than on Facebook.  Not sure why, as it’s the reverse of most sites I’ve seen, but we have a lot more followers on Twitter than we do on Facebook.  Whether that’s because we have some extra posts automated on Twitter, or if it’s just coincidence I’m not sure.

Given time (and energy – and sometimes confidence) I’d really like to try and speak to people a little  more – as an example, a local dodgeball tournement has started, which I saw and thought would be interesting, just as an odd sport, and planned to go and take a look, maybe get some interviews / video.  Never really got chance and wasn’t quite sure it was relevant – but then it got picked up by local TV news, so presumably I should’ve gone with my instinct!  Hopefully will try and actually carry out some proper interviews soon.

On a scale of 1-10 for motivation, I think I’m about a 6 at the minute. I still enjoy doing the site, I’d just really like to do more, and often at the moment I either don’t get chance, or I don’t have the initial motivation to start.



Who Watches the Watch, Man?

I’m always wary of being assimilated into the nerd doctrine of dressing up and playing Warhammer tournament marathons, but I have to admit my love of the steampunk aesthetic.   Really, I can only assume it’s part of my childhood obsession over robots and robotics – things that clanked and clunked as they made their way around the kitchen floor were always endlessly entertaining for me.

I mean, just look at this thing:

Devon Limited Edition Steampunk Watch

From the Devon website:

Devon is a reinvention of the watch. The Tread 1, daring and inventive, represents the exploratory American spirit. The exposed movement is a mesmerizing display of the patented interwoven system of conveyor belts. This series of belts includes critical elements that allow the optical recognition system to know every belt position at all times.

So, anybody want to buy me $25k worth of Christmas present – free shipping included!


(h/t BoingBoing)

The Psychology of Haribo

This post was written as part of a series of blog posts I wrote while studying DSE212 – Exploring Pyschology with the Open University, as part of my efforts to study for the exam.

I’m not the most avid of TV viewers – I do what most people do, recording what I want to watch and viewing it later, when I’ve got a few minutes of downtime, fast-forwarding through the adverts.  Well, apart from one.

Okay, so admittedly part of the reason I stopped over this advert is because it mentions a certain gellotine-based sweet which I and my kids have a slight prediliction for.  But that wasn’t the main reason, oh no.  The main reason is because Haribo have basically recreated a well-known psychological experiment, edited it downm and used that as the basis for an advert.


The advert starts with a basic premise; if the kid can resist the delicious looking sweet in front of them for an unspecified length of time, they’ll be allowed to eat the sweet that’s taunting them as well as getting an extra sweet.  Pretty sweet, right?

Unsurprisingly, we end up with a series of vignettes showing cute kids desperately to avoid eating the sweet before failing miserably.  Mainly because this is an ad to show just how delicious said sweet is.

At this point, you’re probably wondering what relevance this has on psychology. It’s an advert for sweets, showing how much kids like sweets.  Hardly mind-blowing stuff, right?

Well, no, actually.  The original Stanford experiment that this was based on shows how kids deal with delayed gratification.

If you saw a wider range of clips, showing the different methods that each kid employed to ensure that they didn’t fall for the nefarious sweet-taunting trap, you’d see that some kids realised that they just couldn’t trust themselves, so they start devising ways of taking their mind off the task at hand – looking around, singing or whistling to themselves – anything to distract them from the sweet in front of them.

Initially, this experiment was purely to look at delayed (or deferred) gratification – when given the possibility of a better long term reward, would the kids be able to ignore the short-term result of a sneaked sweet?  For about 70% of them, the answer was no.


Read more on the archived New Yorker post, or check out the Wikipedia entry for the Stanford Marshmallow Experiment.

Over the Parapet

I’ve always loved writing, but – as with most things that I learn to do out of sheer pleasure- it’s always been a private thing. I’m not the sharing type in general, and I’ve no real desire to be validated. I just do stuff in general because it sticks in my head otherwise, and I’m not one much for being in the spotlight.

However, a few things have pushed me towards being a little more transparent about these things…

Continue reading


Instagram – Photography Debaser

A fair few years ago, a new type of game appeared on the market. Cleverly disguising what was essentually a marginally more complex variation of ‘Simple Simon’ as a way of pretending you were playing Guitar, it allowed non-musicians to pick up an ‘instrument’ and live our their rock n’ roll fantasies for a few minutes.

No longer did they have to spend months trying to build muscle memory in order to form finger contortioning shapes, nor did they have to go through the pain of building up calluses to succesfully press down cheesewire strings without peeling off layers of skin in an instant. They could just plug in their console, boot up the game and wait for the applause.

A large proportion of the people who had spent hours each day suddenly became infuriated. Who were these people, stealing their glory without any of the work?

Continue reading

Keeping It Simple

I have a horrible confession to make.  Each morning, when I wake up, the first thing I nearly always do is.. read Twitter.  And Facebook, to a lesser degree, but nearly always Twitter.
This morning, I got a link to a blog called, and the first thing I saw was this:
reproduced from
Show that image to pretty much any software developer, and you’ll generally be greeted by a slightly guilty-looking grin.  The fact is that almost every developer has created something like that, where there’s a billion options scatter around the page, vaguely grouped together in sections that we hope make sense.
The real problem here is that, while it goes against common sense, is that simple is actually hard.  There’s probably a bunch of engineers at Google going “right, now we’ve got that search bar, we can add in a ‘number of searches’ button.  And we can display your current location, and just how many coffee shops are around the user, and then…”.  But that takes away from purpose, and then we either have to assume that the user has some inherent knowledge that they can take from the real world (buttons and switches turn things on and off, sliders increase or decrease ‘how much’), or that they can figure it out for themselves… somehow.
The only problem with the ‘somehow’ approach is that your users probably just won’t bother.

Dealing With Negative Comments

This evening, while throwing myself around Arkham City dressed as a flying mammal (this is the rather excellent Batman game on the Xbox, I should clarify) I received this email to the Connect Cannock account:

To: Webmaster

Richard Truman
[email redacted]


I work with Biffa and have just been alerted to the bribery
allegations comment with this story:

The claim is not true so we ask that you remove the comment from the

Thank you,
Richard Truman
Pelican PR
On behalf of Biffa

After reviewing the original tweet, which you can read here, I could see his point.  While there had been something a little strange happening (if you read the article, essentially it can be boiled down to the county council accepting a planning decision when £80k was also offered, but turning it down when the money was taken out of the equation), the tweet itself specifically mentioned the ‘b’ word, which was perhaps a little heavy handed.

The hard part really is deciding what action to take from here; while the comment itself isn’t really adding anything of value, removing it entirely could come across as censorship of the ‘community’ simply because a company didn’t like what was said about them.

Fundamentally in these instances, the thought process with regards to removing the comment can be summarised to:

Is the comment adding value? In this instance, no, because it’s simply linking back to the article it’s displayed on.

Is the comment abusive? Not particularly in this instance, but comments that are abusive or could be seen as bullying can put off other people who want to comment on the site.

Is there a potential legal issue? The term ‘bribery’ is specifically mentioned here, so potentially, yes.

Simply removing the comment doesn’t explain the reasoning behind it, so I added a short paragraph explaining why the comment had been deleted, with a link to the original tweet should somebody still want to read it to understand what had happened.

The article now ends with:

We received the following email from Biffas PR team with regards to an automated comment appearing on the site, which was generated via a ‘scraping’ tool, which automatically logs tweets that link the article.

[original email]

While in general our policy is not to remove comments wherever possible, in this instance we would agree that the statements used in the original tweet could be misconstrued and, in recognition of this, we have chosen to remove the tweet.

The tweet that has been removed can be found on the originators twitter feed here.


Buffer – How To Pretend You’re Working When You’re Not


We all know people who just simply talk too much.  Initially, we’re interested in what they’re saying, then it quickly all turns into white noise which we greet with a vague nod and the odd assuring noise to convince them we’re listening.  The same works with social media – Initially, your message might be interesting, but overshare with your followers / fans and they’ll quickly switch off, either mentally skipping your tweets in their timeline, or simply unfollow you.

Buffer ( lets you solve this in a nice and simple manner – create an account, link it with your Twitter feed, and then start writing tweets.  Buffer.. well, buffers them, holding them back and gently drip feeding them to your followers via a preset schedule.

For hyperlocal accounts – and probably pretty much everybody else who sends out regular news to their followers – this resolves two issues:


As detailed above, sending too many tweets will generally cause people either to ignore your tweets, or to simply stop following you.  Sending tweets via Buffer allows you to schedule what goes out and when.

Staying Active

Because Buffer spreads out your tweets, it means that your account has some constant activity even when you don’t have time to tweet yourself.

Of course, this can’t replace real engagement – Buffer won’t reply to your followers, and on the odd occasion it still confuses me when I get a response to something I didn’t realise had posted…



Using Buffer


Once you’ve set up an account and linked it to your Twitter account (more than one account can be added, but requires upgrading to a paid account), using Buffer is fairly simple – type your tweet and select either ‘Tweet Now’ to send now, or ‘Add to Buffer’ to add the tweet to your schedule.

Adding a Tweet to Buffer


As soon as you click ‘Add to Buffer’ your tweet will drop into the schedule listed at the bottom of the page, like this (you can edit times / settings by clicking onto the time if necessary):


…And that’s pretty much all there is to it.  Once your tweet has been sent, you can check out the metrics from the ‘Analytics’ tab, which will show you the number of retweets and the amount of people your tweet reached.

Buffer Metrics