Category Archives: Hyperlocal

‘hyperlocal’ news stuff

Python Requests to Check Link Status

As I’ve mentioned here before, I’m currently involved with some ongoing research regarding ‘hyperlocal’ sites (due to me running this one), and I spotted one of the people involved mentioning on twitter that they were having to check out ~650 RSS feeds to see which ones were alive and which ones weren’t.

Of course, being the geeky type, I decided it’d be interesting to give it a quick try.  Here’s the full code to start off with.

Before we get started, this is a fairly long post mainly aimed at people with very little experience in the world of coding and Python.

So, what are we doing here?  First off, we’re importing the libraries we need.  requests is the most important bit here, and a massively useful tool that I find myself using fairly frequently.  It’s an HTTP library that makes web services really easy to handle, often with only a few lines of code.  It’s well worth checking out.

Other than that, we’re importing sqlite3 so that we’ve got somewhere to store the data,  along with the sys library (for reasons I actually can’t remember!).

ol_uri = 'http://openlylocal.com/hyperlocal_sites.json'

This is just the URL where we’ll be grabbing the JSON data from, which is handily stored by OpenlyLocal.

Now, we need to set up somewhere to store our results. SQLite is perfect for our needs here, as there’s no requirement for multiple user access, and we just want something fast and lightweight.

First off, we need to make a connection to our database file. It doesn’t matter if the file doesn’t exist yet, as it’ll be created the first time we run our code.

con = lite.connect('ol.db')

Next, we need to create a cursor (a kind of temporary reference to our table) to run our database commands on.

cursor = con.cursor()

…and then we need to set up the database. Here I’m DROPping the table first, ensuring that we remove any past traces of the table in case we’re running this multiple times, then executing a CREATE command to recreate our hyperdb table.

cursor.execute("DROP TABLE IF EXISTS hyperdb")
cursor.execute("CREATE TABLE hyperdb (title TEXT, feed TEXT, RSSAlive TEXT )"

Now that our storage is setup, we’re ready to do some of the heavier work and get our hands on some live data. Fortunately, because we’ve got the grunt of the request library behind us, this is really easy to do.

Ready for this?

#grab the JSON data
r = requests.get(ol_uri)
objs = r.json()

That’s literally it. That’s all we needed to do to get our hands on some hot, live data. We tell requests where to get the info (remember our ol_uri variable earlier?), and then we ask it to format it as JSON, which we’re storing in the rather unimaginatively named objs for now.

From here, we can manipulate the data pretty much as we want, so let’s do exactly that.

for index, site in enumerate(objs):
    ....

We’ve got a FOR..NEXT loop here (although this being Python, we don’t need to tell it the scope of the loop, it knows due to the formatting), which goes through each item in objs and will run the code underneath.

It’s worth noting here that we’ve slightly made the FOR..NEXT loop slightly more complex by using the enumerate() function . This is purely so we can print the current index of the site in the list we’re iterating over. We could just as easily replace it with the perfectly acceptable:

for site in objs:

..only this wouldn’t allow us to print the index later, which for this example makes it harder to see that your code is running.

Getting back to our FOR loop, this is the code that will execute on each object within objs.

    current = site['hyperlocal_site']
    print index
 
    sitename = current['title']
    sitefeed = current['feed_url']
    RSSAlive = "NO FEED"

Each of our site objects within objs contains something called “hyperlocal_site”, which contains all the information we want for each site. We can see this in the raw JSON at
openlylocal.com
.

{"hyperlocal_site":{"created_at":"2011-03-11T14:42:36+00:00","title":"AbergelePost","country":"Wales","updated_at":"2011-03-12T17:55:27+00:00","party_affiliation":"","feed_url":"http://www.abergelepost.com/feed","distance_covered":4.0,"url":"http://www.abergelepost.com","lng":-3.58704,"id":845,"metres":{"coordinates":[294301.803348089,377978.785978634],"type":"Point"},"area_covered":"Abergele","hyperlocal_group_id":null,"council_id":52,"description":"Abergele local news, history and community information.","lat":53.2882,"platform":"Wordpress"}}

To make things a little easier to read, I’m storing the title and feed_url properties from site into some variables (sitename and sitefeed), along with another variable which will store the status of the sites RSS feed(RSSAlive).

How can we check if the RSS feed is alive? Again, it’s nice and easy using the requests library.

    #is the RSS feed alive?
    try:
        if sitefeed:
            f = requests.get(sitefeed)
            if f.status_code != 200:
                RSSAlive = "dead"
            else:
                RSSAlive = "alive!"
    except:
        RSSAlive = "Errored"
 
    sitedata = sitename, sitefeed, RSSAlive

We can explain this fairly simply – where the site has an RSS feed (‘if sitefeed:’) then ask requests to try and open it. If the server returns a status code of 200, the RSS feed is presumably alive. If not, then we report back it’s dead.

The TRY / EXCEPT section is simply because the server won’t always respond in a way we’re expecting. If it gives us a return code of any form, then it’s fine, but sometimes something unexpected might happen which might result in our code breaking, and our script halting. We don’t want that to happen, so we tell Python to ignore any errors* and simply return ‘Errored’ if something untoward happened in the process.

In the last line of the excerpt above, we do something that you can’t do in many languages – we pack three separate variables into one variable, called a tuple. One variable that carries everything else is easier to carry around, right?

Finally, we need to store our data. Again, it’s nice and easy.

    cursor.execute('insert into hyperdb values (?,?,?)',  sitedata)
    con.commit()

We tell the cursor what we want to do (store our variables into the three fields of our previously created database) and look – because sitedata is holding the site name, the RSS feed, and RSSAlive, we don’t need to individually unpack them. We can just tell Python to use sitedata, because it knows it’s really three things stored in one place.

Clever, huh?

*as a general rule, it’s bad practise to completely ignore any error – it might well be something urgent we actually want to stop on, or maybe it’s something we need to deal with differently. In the real world, best practise is to explicitly state how you want exception to be handled.

Creative Citizens Post – 10/01/2013

As a few people may know, I’m currently involved with some interesting research conducted by.. well, a bunch of people, but mainly (I believe) carried out between Birmingham and Cardiff University.  As part of the ongoing research for the Hyperlocal Publish segment of the Creative Citizens project, I’ve been asked to create a kind of diary, logging my thoughts on various aspects of my Hyperlocal site, Connect Cannock.

I’d been trying to decide about where to put this, as there’s a few things I’m (again) toying with, but in the end, I’ve decided to keep it on this site, and, because I do believe it’s best to be transparent, I’m trying to keep it open and public.  If there’s anything I’m uncomfortable with publishing publicly, then, as per my normal internal set of rules, I’ll mention why publicly, and keep the detail so that only the research team and I can read it.

These posts will be short – somewhere around 100 words – and might be completely useless, so don’t expect any startling insights… this post has the obligatory blurb just so people understand why the hell I’m putting almost stream of consciousness on here, with no links etc.

Todays Entry

As far as actual news, things are a little quiet as far as the blog front goes (without looking, I think there’s been a maximum of two posts since the New Year), but this seems to be fairly normal for early January – bearing in mind that much of the input for the site comes from press releases which I then try, dependent on time and interest, to delve into, many places aren’t really doing much and are just getting back up to speed from Christmas.

Sarah, my partner and the sites photographer has posted up an event for a ‘photo walk’ around the  area, which seems like a good idea.  I like the idea in general anyway, as there’s a lot around Cannock that’s fairly photogenic, and the fact that it will help promote the site ‘offline’ as well is never a bad thing, and something I’d really like to try and do more of.

As usual, we seem to be more active on twitter than on Facebook.  Not sure why, as it’s the reverse of most sites I’ve seen, but we have a lot more followers on Twitter than we do on Facebook.  Whether that’s because we have some extra posts automated on Twitter, or if it’s just coincidence I’m not sure.

Given time (and energy – and sometimes confidence) I’d really like to try and speak to people a little  more – as an example, a local dodgeball tournement has started, which I saw and thought would be interesting, just as an odd sport, and planned to go and take a look, maybe get some interviews / video.  Never really got chance and wasn’t quite sure it was relevant – but then it got picked up by local TV news, so presumably I should’ve gone with my instinct!  Hopefully will try and actually carry out some proper interviews soon.

On a scale of 1-10 for motivation, I think I’m about a 6 at the minute. I still enjoy doing the site, I’d just really like to do more, and often at the moment I either don’t get chance, or I don’t have the initial motivation to start.

 

 

Dealing With Negative Comments

This evening, while throwing myself around Arkham City dressed as a flying mammal (this is the rather excellent Batman game on the Xbox, I should clarify) I received this email to the Connect Cannock account:

To: Webmaster

From:
Richard Truman
[email redacted]

Message:
Hi,

I work with Biffa and have just been alerted to the bribery
allegations comment with this story:
http://www.connectcannock.co.uk/2011/09/staffordshire-district-council-rejects-biffa-incinerator-proposals-for-cannock/

The claim is not true so we ask that you remove the comment from the
website.

Thank you,
Richard Truman
Pelican PR
On behalf of Biffa

After reviewing the original tweet, which you can read here, I could see his point.  While there had been something a little strange happening (if you read the article, essentially it can be boiled down to the county council accepting a planning decision when £80k was also offered, but turning it down when the money was taken out of the equation), the tweet itself specifically mentioned the ‘b’ word, which was perhaps a little heavy handed.

The hard part really is deciding what action to take from here; while the comment itself isn’t really adding anything of value, removing it entirely could come across as censorship of the ‘community’ simply because a company didn’t like what was said about them.

Fundamentally in these instances, the thought process with regards to removing the comment can be summarised to:

Is the comment adding value? In this instance, no, because it’s simply linking back to the article it’s displayed on.

Is the comment abusive? Not particularly in this instance, but comments that are abusive or could be seen as bullying can put off other people who want to comment on the site.

Is there a potential legal issue? The term ‘bribery’ is specifically mentioned here, so potentially, yes.

Simply removing the comment doesn’t explain the reasoning behind it, so I added a short paragraph explaining why the comment had been deleted, with a link to the original tweet should somebody still want to read it to understand what had happened.

The article now ends with:

We received the following email from Biffas PR team with regards to an automated comment appearing on the site, which was generated via a ‘scraping’ tool, which automatically logs tweets that link the article.

[original email]

While in general our policy is not to remove comments wherever possible, in this instance we would agree that the statements used in the original tweet could be misconstrued and, in recognition of this, we have chosen to remove the tweet.

The tweet that has been removed can be found on the originators twitter feed here.

 

Buffer – How To Pretend You’re Working When You’re Not

Overview

We all know people who just simply talk too much.  Initially, we’re interested in what they’re saying, then it quickly all turns into white noise which we greet with a vague nod and the odd assuring noise to convince them we’re listening.  The same works with social media – Initially, your message might be interesting, but overshare with your followers / fans and they’ll quickly switch off, either mentally skipping your tweets in their timeline, or simply unfollow you.

Buffer (www.bufferapp.com) lets you solve this in a nice and simple manner – create an account, link it with your Twitter feed, and then start writing tweets.  Buffer.. well, buffers them, holding them back and gently drip feeding them to your followers via a preset schedule.

For hyperlocal accounts – and probably pretty much everybody else who sends out regular news to their followers – this resolves two issues:

Oversharing

As detailed above, sending too many tweets will generally cause people either to ignore your tweets, or to simply stop following you.  Sending tweets via Buffer allows you to schedule what goes out and when.

Staying Active

Because Buffer spreads out your tweets, it means that your account has some constant activity even when you don’t have time to tweet yourself.

Of course, this can’t replace real engagement – Buffer won’t reply to your followers, and on the odd occasion it still confuses me when I get a response to something I didn’t realise had posted…

 

 

Using Buffer

 

Once you’ve set up an account and linked it to your Twitter account (more than one account can be added, but requires upgrading to a paid account), using Buffer is fairly simple – type your tweet and select either ‘Tweet Now’ to send now, or ‘Add to Buffer’ to add the tweet to your schedule.

Adding a Tweet to Buffer

 

As soon as you click ‘Add to Buffer’ your tweet will drop into the schedule listed at the bottom of the page, like this (you can edit times / settings by clicking onto the time if necessary):

 

…And that’s pretty much all there is to it.  Once your tweet has been sent, you can check out the metrics from the ‘Analytics’ tab, which will show you the number of retweets and the amount of people your tweet reached.

Buffer Metrics

 

Published in Print – Without Permission

About a week ago I published an article on this site with regards to a ‘story’ appearing in the national press regarding local councillor Jodie Jones having photos on Facebook of her (shock horror) actually drinking alcohol.

I’d originally planned for the article to go on the ConnectCannock site, but decided it wasn’t really ‘news’.  The original article had been a cheap excuse to print a 22 year old in a bikini and pretend it was newsworthy, so instead I wrote it as a comment, published it on this site, and fed a link through the ConnectCannock twitter account explaining my decision.

A few days later a friend of mine mentioned how he’d seen my comment on the issue in the local paper, the Cannock Chase Post.   Only I hadn’t made a comment to the paper.  In fact, I’d been disappointed by their lack of interest in dealing with me – I realised ConnectCannock isn’t necessarily a source of professional journalism, but I’d always thought there’d been areas where we could’ve helped each other out.

Bearing in mind I hadn’t spoken with the paper at all, compare my original blog peice and the text I apparently ‘quoted’ for this article, available online from the Chase Post website.

So I sent them an email:

Hi Charlotte,

I sent a tweet earlier to your @chasepost account, but it doesn’t seem to be very active so I thought I’d send a quick email as well.
With regards to the article entitled “Residents’ support for Jodie in Facebook row”  (http://www.chasepost.net/news-in-cannock/cannock-burntwood-news/cannock-news/2011/09/01/residents-support-for-jodie-in-facebook-row-93633-29346552/) , you’ve apparently quoted me, as per the paragraph below:

Cannock resident Ian Mellett said: “Now 23, Jodie Jones is still one of the youngest councillors in the country, and while her politics may not be in alignment with that of others, it’s hard to really take this as serious news.
“I have a deep suspicion that if this had been a 23-year-old male slouching drunkenly in Speedos, then there wouldn’t be quite so much interest.”However, this isn’t actually a quote, and is a section lifted directly from my blog, which was used to ‘report’ on this as I didn’t see it really as fitting with my local news site, ConnectCannock.

I have no real issue with being quoted, but fundamentally this was taken from a written source without any permission being granted, nor with any real attribution (a link back to the personal site, or ConnectCannock itself, which is what it really relates to), and it’s specifically covered under a creative commons license.

Just to be clear though, this isn’t specifically a complaint (my only complaint is that I don’t get the Chase Post delivered to my flat still!), more that I think it needs clarification.  There are instances when we publish elements in other newspapers and websites, and I’m fairly certain that you wouldn’t simply copy their text and claim that they’ve quoted directly to you, and the shortening of the original text does somewhat skew my intentions.

 

Fundamentally (as I hope is conveyed with my email) my intention isn’t to try and belittle the Chase Post.  As a local paper, it does a good job, and it’s one of the few free papers I do generally read.  My real intentions can be reduced to a handful of key points.

Attribution

The work is taken directly from my website, and while I’m fully aware I’m not going to get thousands of hits, if text is being used from the site then I’d expect that the reader would be informed of the original source, especially when the quote taken along is somewhat out of context.

Permission

I have no issue being quoted – but a simple request to allow what is essentially covered by copyright really would have been the minimum expected.  In that instance, I might have modified the quote slightly, as the current text is fairly blatantly from a written point, not a quote.

Copyright

Without trying to seem precious about what’s written, fundamentally the text of any blog is covered by copyright, and it seems a number of news sources at all levels have difficulty with digital copyright (see Andy Mabbetts example of the BBC’s misunderstanding).

I’m fairly certain that the Trinity Mirror group would not take kindly to me printing sections of their site without consent and proper attribution, so why should the work of a blog be any different?

At the time of writing, I’ve still yet to receive any form of response with regards to the matter.

 

Jodie_Jones[1]

In Defence of Cllr Jodie Jones

While this post may seem more relevant to ConnectCannock, it was decided that this was not a ‘real’ news item and would not be on the site as such.  Instead, it’s been written as on opinion piece here.  All views are personal and those of the author.

 

“One of Britain’s youngest Conservative councillors has been slammed for a series of Facebook photos that show her swigging from a champagne bottle and partying in a bikini” writes Lee Moran for the Daily Mail, next to an enlarged sequence of photos presumably taken directly from said Facebook account.  Which, in turn, sits alongside stories of celebrities getting pregnant, or trying to avoid cameras to hide their lack of makeup.

Now 23, Jodie Jones is still one of the youngest councillors in the country, and while her politics may not be in alignment with that of others, it’s hard to really take this as serious news.  If anything, there’s a definite link to the phenomena of attractive girls passing their exams.   I have a deep suspicion that if this had been a 23 year old male slouching drunkenly in Speedos, then there wouldn’t be quite so much interest from press, who were probably rubbing their hands together in glee at the potential to publish photos of 20-something girls in bikinis whilst still provisioning the pretence of reporting ‘real news’.

Of course, it’s hard not to suggest that in these instances that this is indeed the price of the press.  Jodie has appeared in local newspapers – such as this Express & Star article – seemingly purely on the merits of being a 22 year old (at the time of writing), attractive women in politics.   Even now, the alternate version of the running story are variations of “Is Jodie the Sexiest Woman in Politics?” (metro), which is just as detrimental to Jodies political character.

I’ve had direct experience of dealing with Jodie over my time of writing for ConnectCannock.  I don’t necessarily agree with her opinions nor views, and sometimes I perhaps think that there’s still a level of maturity thats still not quite there (and considering her age, I don’t feel that’s particularly condescending nor patronising).  I also have a mild issue with those who choose a political career without ever experiencing external employment.  However, she’s on of the few local councillors who will deal with me, and will actually discuss issues over channels such as twitter.

ConnectCannock has a large audience in Cannock on both Twitter and Facebook, so we often get to see the ‘chatter’ that goes on around – it’s simply part of trying to ensure we find out what’s of interest to local people – and even now I’ve yet to see anybody post even the vaguest interest (or even knowledge) in the apparent story.  ’23 year old girl parties / drinks alcohol direct from a bottle’ just isn’t really that interesting.

One of my big bugbears with modern politics is the level of accessibility to those representing us in parliament.  but I would hate for the duplicitous nature of the press in these instances to force those in governance to further recoil from direct public contact. In this instance, Cllr Jodie Jones should be judged on her merits as a councillor, not on the fact that in her personal life she acts much as ‘normal’ people do.

Starting Out In ‘Hyperlocal’ – A Rough Guide To ConnectCannock

ConnectCannock, the hyperlocal news site that I created for Cannock, Staffordshire, is well over a year old now (according to the whois info – although the twitter account has been going for a little longer).

A number of people have written their own reflections on their own sites, and how their experiences have shaped the processes they’re now going through.  So, in the spirit of imitation being the best form of flattery, I’ve taken their idea and blogged about it.

JFDI*

 

ConnectCannock started out as a twitter account, which then turned into a website (mainly because I already had hosting which was doing very little, and I had the technical knowledge for the configuration of WordPress not to be a big deal).

The twitter account was easy.  I saw other people in surrounding areas making hyperlocal sites, checked that nobody was already doing the same for Cannock, and just went ahead without really thinking about it or planning what I was actually going to do once I had an account (you’ll notice a running theme emerging here).

The website proved more of a challenge; not because I had any difficulties with the technical side, but more because I really didn’t know what should be happening with the site.  The site got installed, I played around with a million wordpress themes, not really happy with any of them.. and after weeks of indecisive theme switching, I realised all these things were malleable – I could change anything I wanted at a later date, but while I was pontificating about which shade of red I preferred, nobody was visiting the site.

So instead, I just launched it.  No fireworks, no opening ceremony – one day it wasn’t public, the next it was.   The moral of the story? Stop thinking about it, trying to rationalise it, trying to make it look exactly the way you want it.  JFDI*.

*For the sake of not getting blocked by filters, I’ll leave you to google the terminology.

 

Don’t Quit the Day Job

 

I enjoy the experience of writing for ConnectCannock massively.  I’ve discovered things about the small town I live in that I didn’ t know existed, I’ve been involved in things I wouldn’t normally have even considered (council meetings, reporting live from local election results until 4am!) and I’ve discovered some interesting people that I just wouldn’t have been aware of previously.

However, there’s no way in hell I can see this being a self-sustaining project.  If that’s what you’re after – and there’s a lot of talk about this in the US – then I’d personally say that this isn’t the project for you.  The days of reasonable revenue from online ads are long gone, and lets face it, how many people even click ads these days? I genuinely can’t remember the last time I did.  They get filtered out either by my browser, or by the fact that I’ve seen enough ads for my brain to turn them to white noise automatically.

In theory, I could’ve tried to turn the business directory side of ConnectCannock into a paying system – it has a fair few features that a lot of paid systems dont, such as QR codes per business, proper Twitter / Facebook connections, streetview and a few other bits – but then I’d need to spend time selling space, and it’d mean spending less time on the main site.

 

How To Make Friends and Influence People

 

One of the best parts of doing this is that a bit of googling will lead you onto a lot of other people who are doing similar things in nearby areas, pretty much all of whom in my experience are friendly, and have been willing to share advice from the outset.

For me, ‘borrowing’ ideas from sites like WV11.co.uk, LichfieldLive.co.uk and alittlebitofstone.co.uk has been an easier climb in working out how things should work, which then makes the path to further experimentation a little easier.

There’s also invaluable resources like TalkAboutLocal, who provide advice on setting up sites and actually putting content on them, hyperlocal directory OpenlyLocal, and the Social Media Surgery site, where groups create and attend social media surgeries (which I’m planning on doing locally – more on this at a later stage).

My local council  have been extremely helpful and open about various plans as well, and have always supplied a constant feed of PR and information I can use for the site.

Failure is Always An Option

 

The real delight in doing local news this way is that you can experiment without much fear of.. well, anything.  There’s nobody in charge to fire you if it all goes downhill, and you’re not making money from it – so who cares?  Try stuff.  If it doesn’t work, change it, try something else.  Most hyperlocal sites operate at a minimal cost, so it’s not really like it’s going to involve losing anything more than a couple of quid.

 

The End.  Honest.

 

One of the main reasons I deal with ConnectCannock is that I’ve discovered I actually enjoy this stuff, and it appeals to my general desire to not only learn new things, but to be able to share that knowledge.  Need help with technical information? Need to understand how to interrogate a set of data in a few seconds, or how to best use QR codes?  I can probably help, but I might end up trying to find out what you know about journalism. Or facebook groups.  Or something else that you’ve done that I haven’t.