IP Camera Data Privacy

I’ve had a few goes at setting up the Motion package on my Raspberry Pi, but I’ve finally abandoned it. In its stead, we recently bought an IP camera manufactured by a company called Annke which, at the time of writing, is among the best selling surveillance cameras on Amazon. It’s a nice piece of kit but given that it only cost £40 and has a lot of moving parts, it’s not one that I expect to survive down the years.

I thought it would be interesting to proxy the traffic on my phone to see what’s happening.

My iPhone isn’t jailbroken, which would have been a showstopper if the server that the app is talking to was using certificate pinning. It’s not. The first call is over plain HTTP. I’m not going to copy it here, because some of the payload is decodes to binary, and there’s a possibility that I might be broadcasting my own password. Doh!

So the first call is a GET to a server running on Amazon’s cloud service, listening on port 7080. I checked the IANA registry: while there is something assigned to that ‘officially’ [some identity management software called empowerid] I think it’s a coincidence, and it’s probably just a web server of some kind running on a non standard port. The response doesn’t report back the server software name. Included in the GET parameters, there are a series of comma and slash [url encoded] separated parameters which are base64 encoded. These decode into binary, and could be anything. Included among the readable parameters is my username.

The response back is a block of JSON, referencing different URLs on the same server, which a geolocation service reliably informs me is in a data centre operated by an outfit called OVH in Roubaix, northern France. The URLs have helpful prefixes: ‘signal’, ‘debug’, ‘ping’, and ‘ntp’ among them. Not all of the URLs are referring to web traffic: there’s one reference to telnet, which is a blast from the past, and another called ‘binnet://’ which is sufficiently non standard that Google keeps insisting on telling me about ‘bonnets’ :). That final ‘binnet’ URL refers back to the original AWS server.

The app then does a second GET to a server, this time in Tampa, Florida. I’m not going to break this one down in any sort of detail because the server refuses the connection, so it can’t be too important!

Next, the app opens a TLS connection to the French server, and does 10 separate GETs. The 3rd of these includes my password, which I registered on first run.

Here’s what I imagine is happening: the camera is going to be polling the server in France cyclically, asking the question, ‘do I need to transmit to you yet?’ When I connect to the same server via the phone app, the answer comes back as a ‘yes’. The camera starts to transmit, and the server then relays the stream back to my app. WireShark should be able to give me some pointers, but I’m running out of time to look at it today. If I find anything interesting or contradictory when I do get round to looking at it, I’ll do a separate post on it.

By the way, the app seems to be using JavaScript to instantiate the video stream in HTML5. Apple have a video from the WWDC in 2013 on exactly this topic.

So in summary, it looks like the video stream of our back garden / the cat / my wife and I occasionally waving at the camera ends up in France, with the server there ‘joining’ the connection from the camera to the app.

The end state with my Raspberry Pi was, well, while not necessary secure in and of its own right, certainly wasn’t nearly as ‘mediated’, shall we say. I set up our broadband router with a port forwarding rule and configuration for a DDNS service. That meant we could connect from our phones to the web server integrated into the Motion package. That was all over vanilla HTTP – hey, at least I set a 401 password!

While I can’t do anything about how the camera operates – a trade-off I’m willing to make based on pure utility – I’m probably going to take a look at partitioning off the network with a proper firewall behind the broadband router [which can be configured to operate as a modem only]. I’ll put all of the ‘less trusted’ devices on their own little segment. Pfsense seems to be the way to go.

I’m sure it’s nothing personal…

Like probably the vast majority of people who are running WordPress for more than a few months, my site is frequently being hit with automated attacks. I’ve only recently noticed this in my logs so I thought it would be interesting to have a closer look.

Around the turn of the year, for reasons I can’t recall, I happened to look at the raw access logs and noticed a lot of references to ‘xmlrpc.php’, which look like this: - - [31/Jan/2016:18:13:42 +0000] "POST /blog/xmlrpc.php HTTP/1.0" 200 58043 "-" "-"

This is a real log file entry, and is a classic example of an XMLRPC bruteforce amplification attack: someone has posted 58k at this page, to try and bruteforce the admin password. I disabled the mechanism – and just verified that it’s working this morning [two months later :)], as the 200 server response is a bit more polite than I would have expected.

At the same time I installed [yet another] plugin, which rate limits failed admin password authentication attempts. It started triggering last week with repeated admin authentication failures from a machine in Hanoi. In my latest access log file [31st January to about half an hour ago], I have 1500 POST attempts which look like this: - - [26/Feb/2016:13:37:47 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 3766 "-" "-"

I’ve not paid much attention to log formats in a long time so I had to google what those final two hyphens are: a blank referer [note to my wife on the spelling :)] and user agent field respectively. The blank user agent is indicative of some sort of automated attack and, by virtue of the fact that the person who’s running it hasn’t even bothered to make it look like a real browser, one that isn’t particularly sophisticated.

The logging pattern suggests what you’d expect: someone has harvested a set of servers that are running WordPress [how? by virtue of having the common pages that WordPress hosts. So a 200 in response to a GET for a ~/wp-login.php page, for instance], and is stepping through them.

This is another indicator of the lack of sophistication: - - [26/Feb/2016:16:41:35 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:41:37 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:41:38 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:41:44 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:41:46 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:41:58 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:41:59 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:42:05 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:42:06 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:42:13 +0000] "POST /blog/wp-login.php HTTP/1.0" 200 1643 "-" "-" - - [26/Feb/2016:16:42:14 +0000] "POST /blog/wp-login.php HTTP/1.0" 403 9 "-" "-" - - [26/Feb/2016:16:42:20 +0000] "POST /blog/wp-login.php HTTP/1.0" 403 9 "-" "-" - - [26/Feb/2016:16:42:21 +0000] "POST /blog/wp-login.php HTTP/1.0" 403 9 "-" "-" - - [26/Feb/2016:16:42:22 +0000] "POST /blog/wp-login.php HTTP/1.0" 403 9 "-" "-" - - [26/Feb/2016:16:42:23 +0000] "POST /blog/wp-login.php HTTP/1.0" 403 9 "-" "-"

What’s happening here is that some software I’m running is blocking the user’s IP address after 10 authentication failures, shown by the 403, which is the server returning a ‘Forbidden’. What I’ve deleted from the log extract above is  that there are a total of 25 Forbidden responses by the server in a row: the attack software isn’t checking the server response codes, which is a waste of resource on their part.

I’ve had a bit of a trawl through my logs and am seeing similar, albeit less determined attacks like this, coming from all sorts of far flung places: - - [13/Feb/2016:07:46:48 +0000] "POST /blog/xmlrpc.php HTTP/1.0" 200 58043 "-" "-"

That’s another XMLRPC bruteforce amplification attack, from Russia. A geolocation site reckons this one… - - [12/Feb/2016:07:12:33 +0000] "POST /blog/xmlrpc.php HTTP/1.0" 200 58043 "-" "-"

…is in San Antonio, Texas. Interesting that the byte sizes being posted through are identical: 58,043. Again, that’s indicative of the same off the shelf attack software running with a pre-canned payload. Let’s do one more of these: - - [11/Feb/2016:02:19:14 +0000] "POST /blog/xmlrpc.php HTTP/1.1" 200 45387 "-" "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"

I can honestly say that since I first started messing around on the internet in 1992, I’ve never seen an IP address that starts with 1. The geolocation service dutifully informs me that the machine that sent this parcel of good intention is located in Xi’an in China. At least they’ve spiced things up a bit with a different sized payload.

So here’s a thing: I have a couple of blog posts on this site about a holiday we had in Vietnam. I blogged about a holiday to China that included a trip to Xi’an. I’ve also got a posting about a work trip to Russia. So… Russia and China are massive, populous countries. But Xi’an, in China? That looks like a pattern to me. I wonder if the bundle of joy – malware, whatever it is – that would be deposited on my site if it were to be compromised is tailored or localised in some way or other, based on the occurrences of those locations.


As per the title, and the obvious lack of finesse, I know that my server is just one on what’s probably a very long list of candidates that these automated attacks are hitting. WordPress has had something of a chequered history from a security point of view: it’s a natural target. While I’ve done the easy stuff to shore it up – like blocking a blank user agent – the options are relatively limited. That’s fine, given the fairly low-rent nature of the stuff being thrown at it, but I’d really prefer not to be distributing malware to people. Migrating off WordPress looks like it would be a pain so if the ancillary approaches start to look like they’re too much trouble I’ll just delete the site.

Building an Electronic Programme Guide [part 3]

This is my third and final write-up on the development of an electronic programme guide app. As of part 2, the main scrolling view with the programme details, scaled to length, are displayed.

Next up I wanted to have the ability to display programme details. This seemed like it was going to be pretty straightforward: when building the per-programme view, I included a button, which is transparent, and has the same dimensions as the view itself. The first problem is associating the programme details with the button itself. There is plenty of discussion on StackOverflow about how to do this in the least offensive way. I went with a category and then object association. This allowed me to set the GUID as a property for the button.

I rebuilt the app, hit the button and… entered into a fortnight of debugging an EXC_BAD_ACCESS error. I knew what the problem was: the ARC memory management was dereferencing the button object once it was set. I tried lots of different options, such as adding the buttons to an array, set with various properties, and passing the array back to the main view controller. Nothing worked until I did more reading around the property attributes, and ended up redefining the Interface Builder defaults for the scrolling contentView to:

@property (nonatomic, strong) IBOutlet UIView *contentView;

That ‘strong’ means that everything in the view is held in memory. It has to be said that the app is very heavy on memory – as a direct consequence of that view object retention. It routinely occupies 63Mb in my testing.

Next up is the popup that is rendered. So finding the programme itself is pretty easy, using an NSPredicate based on the GUID. What proved a bit harder to deal with is if the main view [the ‘contentView’ for the scrollView] is zoomed. As you have to add the popup view to the zoomed parent, the former is going to inherit the zoom setting. I couldn’t think of an elegant way around this so I worked around it in stages. First off, the popup sits on a blurred view of the current background:

// This is quite neat: make a CGRect of the currently visible part of the scrollview:
CGRect visibleRect = [scrollView convertRect:scrollView.bounds toView:contentView];
visualEffectView = [[UIVisualEffectView alloc] initWithFrame:visibleRect];
visualEffectView.effect = blurEffect;
visualEffectView.frame = contentView.bounds;
[contentView addSubview:visualEffectView];

Next, I register the scrollView offset in a property:

scrollOffSet = scrollView.contentOffset;

…set the zoomScale to 1, and disable the ability to zoom:

[scrollView setZoomScale:1.0];
scrollView.scrollEnabled = NO;
scrollView.maximumZoomScale = 1.0;
scrollView.minimumZoomScale = 1.0;

Placing the programme details subview is then relative to the currently visible rectangle:

float xForLittleView = visibleRect.origin.x + 30 ;
float yForLittleView = visibleRect.origin.y + 100;

CGRect progViewRect = CGRectMake(xForLittleView, yForLittleView, 350, 500);

I then have to undo the various view settings when the button to dismiss the view is touched:

[visualEffectView removeFromSuperview];
[littleView removeFromSuperview];
scrollView.scrollEnabled = YES;
scrollView.maximumZoomScale = 2.0;
scrollView.minimumZoomScale = 0.8;
[scrollView setZoomScale:zoomScale];
[scrollView setContentOffset:scrollOffSet];

It’s all a bit clunky, but it works. I imagine that this sort of interface plumbing actually happens quite a lot behind the scenes. That said, I may have missed a trick to do it in an easier way.

I’ll call out two more details that I wrestled with. The first is a search facility on the programme title. I wanted the NSPredicate to support as many search terms as the user entered. My initial idea was to split the UITextField input on spaces, and then loop through the resulting array, appending to a stringWithFormat, where all but the first element would be in the form:

AND (title CONTAINS[c][/c] %@)

Having experimented with this, it appears that predicateWithFormat has to have the actual string passed to it, as opposed to a variable containing the string. Which I have to say strikes me as a little odd. The functional upshot of this is that I couldn’t support a variable number of search terms. I support up to three, and construct a separate predicateWithFormat for each possibility.

One final problem that I couldn’t find a fix for was implementing a UITableView’s delegates in a class that I pass the view into as a parameter. I couldn’t find a way of getting the cellForRowAtIndexPath delegate method to be called. The conclusion I came to with this was that it was setting the delegate to self, when ‘self’ was the custom object, rather than the view. It was largely a cosmetic thing [I’ve noticed that for complicated apps, I have a tendency to pile way to much code into the main viewController] so it was easily solved.

Here’s what may be the final version of the app looks like, showing a search result in the popup view, and the to/from dates for the EPG coverage:

Search Results

Search Results

The other buttons that I haven’t talked about explicitly are an ability to switch between days, and initiate a download of EPG data – but which are pretty straightforward. What’s still either ugly or hasn’t been fully implemented is the download progress indicator, and also the what’s-on-now quick look on the Apple Watch, as I want to have a mess around with something completely unrelated to this app: the motion detection capability.

I did add a quick fix to ‘justify’ the right hand side of the ‘table’ of programmes. Formerly they were falling off the contentView. I simply check if the rightmost width of the cell is going to be greater than the width of the contentView. If so, I set it to be the same as the width.

So that’s it. I have a pretty serviceable EPG app, which I use myself over the ad-funded variant I had before, which I guess is a fair indicator of utility. Main lesson learned: not knowing what those property attributes meant tripped me up really badly!

Building an Electronic Programme Guide [Part 2]

A little later than intended, here’s the follow-up to my first posting on building an EPG.

So, having marshalled the data into a structure that can be displayed, on to the guts of this app, which is the main UIView for guide itself. The pre-requisite of that main view is configuring the scrollView: I’m not going to dwell on this too long, as it’s a well documented feature.

For the formatting of the content in that classic programme guide UI, with rows of variable width cells, I initially started with looking at collectionViews, but couldn’t find a way of doing it – or certainly an amenable way. I also briefly looked at customising a tableView, which I could add multiple elements to, but rejected it for the same reason.

I finally settled on a custom implementation, with each programme being represented by its own UIView, with:

  • the width: the duration in minutes divided by the number of minutes in the day, then multiplied by the width of the scrollview.
  • the height: this is a constant for all the cells, just based on experimenting with the scrollView.

For what it’s worth, my scrollView is 4000, and the row height is 60. [Actually it’s the width of the contentView the scrollView contains, but I’ll refer to the scrollView from here for purposes of readability.]

Before I go on to calculating the [X,Y] coordinates of the top left corner of the subview, I need to recap on how the programme information is written to Core Data and then read back. At some point in the implementation as I envisage it, it will be possible for the user to both download more programme data, as only a couple of weeks worth come down at a time, and also to configure which channels to do the download for. For ease of simplicity of managing the stored data, I decided to write all of the programme data to the same Core Data entity [think database table], and when the user repeats the download, with or without changing the channel configuration, I delete the existing data – a fairly blunt but effective instrument.

When it comes to retrieving the data, which I do on a per-channel basis, I need to search using an NSPredicate:

NSPredicate *predicate = [NSPredicate predicateWithFormat:@"(channel = %@) AND (startDate >= %@) AND (endDate <= %@)", channelName, startDateForSearch, endDateForSearch];
NSFetchRequest *allChannelDataReq = [[NSFetchRequest alloc] init];
[allChannelDataReq setEntity:[NSEntityDescription entityForName:@"Programme" inManagedObjectContext:managedObjectContext]];
[allChannelDataReq setPredicate:predicate];
[allChannelDataReq setIncludesPropertyValues:NO];
NSError *error = nil;
tempArrayOfFilteredProgData = [managedObjectContext executeFetchRequest:allChannelDataReq error:&error];

So this block of code is searching for hits on a given channel, startDate and endDate [attributes of the Entity], and loading them into the array. I’ve already noted these are unsorted, so…

NSSortDescriptor *sortByRunningOrderInt = [[NSSortDescriptor alloc] initWithKey:@"progOrderNum" ascending:YES];
NSArray *descriptors = [NSArray arrayWithObject:sortByRunningOrderInt];
NSArray *sortedProgDataForChannel = [channelData sortedArrayUsingDescriptors:descriptors];

…where the progOrderNum is the corresponding attribute in the Entity.

The sortedProgDataForChannel can now be looped through, for each programme:

- (NSMutableDictionary *)drawRect:(float)width startXPosn:(float) topLeftX rowNum:(int)rowNum forColour:(UIColor *)cellColour
NSMutableDictionary *viewAndCoords = [[NSMutableDictionary alloc] init];
float topLeftY = rowNum * rowHeight;
float topLeftXForNextView = topLeftX + width;
CGRect rectangle = CGRectMake(topLeftX, topLeftY, width, rowHeight);
UIView *thisProgView = [[UIView alloc] initWithFrame:rectangle];
thisProgView.backgroundColor = cellColour;
thisProgView.layer.borderColor = [UIColor blackColor].CGColor;
thisProgView.layer.borderWidth = 1.0;
[viewAndCoords setObject:thisProgView forKey:@"view"];
[viewAndCoords setObject:[NSNumber numberWithFloat: topLeftXForNextView] forKey:@"newTopLeftCoord"];
return viewAndCoords;

The idea here is to calculate the width, as described above, and create the UIView [with a couple of frills like a border], and then return the X co-ordinate for the starting point for the next view. The Y coordinate is a simple calculation: I assign the channels a number, and then multiply that by the height of each cell.

That, in essence is the primary logic for building the ‘wireframe’ for the programmes. I extract the text for a given programme’s title, start and end times, etc, and add those as labels programmatically. Again, these are calculated as offsets from the starting X coordinate and the calculated Y based on channel number.

There are a couple of gotchas, and I had to take a couple of bites at this – which is a polite way of saying I made a couple of idiotic mistakes – and the visual results of which were so bad that I decided to record them for posterity. The first is an example of not reading the documentation. I misremembered the parameters for a UIView as being X+Y for the top left corner, and then X+Y for the bottom right corner. This resulted in the following:


It really is a thing of beauty :).

Next up was dealing with the simple matter of the start of the day. The NSPredicate I use extracts per-channel programme data for start times on a given date. I was then merrily plastering these onto the scrollView with the following results:


Obviously a refinement is required, so first port of call was to make 5am the equivalent of the left edge of the scrollView. I decided that, rather than hardwiring this into the search parameter for the NSPredicate – in case I changed my mind later – I’d do it when looping through the channel data.

So first of all, I see if the start time of the given programme is >= 5am. If not I disregard it. If it is, I then calculate the offset for the X coordinate. This is based on calculating the difference between the start time of the programme and 5am in minutes. I then divide this by the number of minutes in the day and multiply it by the width of the scrollView. There’s an additional offset [which actually applies to all of the programme UIViews] for a channel title list down the left hand side of the scrollView.

-(int) calcPosOfFirstProg:(NSDate *)progStartTime
int xPos = 0;
// This start x posn allows for the channel name label
int offSetForChanTitle = 100;
// So: the left edge of the view equates to 5am.
NSDateFormatter *dateFormatter = [[NSDateFormatter alloc] init];
// formatting from just HH:mm didn't work because the progStartTime has a date.
// need to extract today's date from progStartTime and bake it into the viewEdgeTime...

[dateFormatter setDateFormat:@"dd/MM/yyyy"];
NSString *dayDateForDisplayDay = [dateFormatter stringFromDate:progStartTime];
NSString *viewEdgeTimeString = [NSString stringWithFormat:@"%@ 05:00", dayDateForDisplayDay];
[dateFormatter setDateFormat:@"dd/MM/yyyy HH:mm"];
NSDate *viewEdgeTime = [dateFormatter dateFromString:viewEdgeTimeString];
// Time interval gives us the number of seconds:
NSTimeInterval timeDiff = [progStartTime timeIntervalSinceDate:viewEdgeTime];
// All other calculations on positioning have been in minutes so:
float minsBetweenTimes = timeDiff / 60;
NSLog(@"minsBetweenTimes is %f", minsBetweenTimes);
float minutesInDay = 1140;
xPos = ((minsBetweenTimes / minutesInDay) * scrollViewWidth) + offSetForChanTitle;
return xPos;

That channel listing is a bit ugly – it’s why I bake the channel name into the progamme summary. What I’d really like to do is have a “floating” listing that the scrollView passes under. I’ve used key value observers a lot in the past, so one approach would be to set a KVO on the delegate that returns the current scroll scale value, and to resize based on the changes in scale. One to come back to…

It would, of course, be necessary to do the same for the right hand of the scrollView, to make sure that for a given cell, there’s enough room to display it. I haven’t gotten round to this yet, so late finishing programmes dangle precariously into space.

Next up I’ll talk about creating the pop-up view for the programme details, including navigating an ARC related memory error, which took me a loooooooong time to figure out!

Adding SATA 3 Support to a Mac Pro

Caveat: be careful with your expensive, delicate computer. This has all the characteristics of a hack, but it was wildly successful for me. There may also be a much more sensible way of doing this out there. YMMV.

I have a mid 2010 spec Mac Pro, with the 6 core Westmere chip. I’ve been running out of disk space for about half of the elapsed time since I upgraded to a SATA 2 SSD about 3 years ago. Amazon are doing a pretty good deal on a Samsung SATA 3 SSD at the moment so I thought I’d take a punt on upgrading.

The long and short of it is that I’ve hacked together a pretty cheap option, which has given me a 13x speed up on sustained write speeds.

I went for this card [which appears to have risen in price by a couple of quid since last week], on the basis of seeing a reference to a particular chipset [ASM1061] on a forum somewhere along the line.

I have to admit straight off the bat that I’d never worked with PCIE before, and it took me a while to work out that the card was missing a power supply. This led me on a merry dance to figure out how to get power into my SSD. There are a couple of raised interfaces on the card which may be power outlets, but I’ve never seen anything like them before. Having looked at a tutorial on how to install a USB 3 device, I thought I was going to end up having to take the fan out, and get some sort of MOLEX to Mac 4 pin style power adapter cable. Having then hit on the fact that your stock Mac Pro [of my era at least] comes fitted with a single occupied bay of 4 designed for hard drives with SATA [2] interfaces, I bought one of these. Here’s the really hacky bit: the extender cable comes with a couple of clips either side to stabilise the connection. You have to clip one of them off, and plug it into the SATA interface of your choice.

And it works. And it’s bootable [a gotcha for some PCIE cards], and it handles resurrection from sleep. I verified that it was registered as a SATA 3 [look for the negotiated link speed of 6 Gb/sec] and used BlackMagic Disk Speed Test to verify a sustained write speed of 354 Mb/sec.

While there are integrated cards that you slot your SSD straight into, this turned out to be cheaper. That said, the new SSD is currently sitting loose inside the machine. There are integrated ‘sleds’ that you can screw the SSD into, and then slide into one of the four disk bays. I haven’t seen one that’s less than about £20. I also haven’t experimented with any of the 3 spare interfaces [2 of which are external] on the card: I’ve no idea if you can RAID up two SSDs on the same card, for instance. I’d be interested to find out!

Location Display – Arduino Based Clock

Once I finally got everything working with the weather station clock, predictably, it started to lose its appeal. So I’ve taken the bones of it and refactored to display my current location. The source of the location data is another little iOS app, this time using background fetch.

I may split the various moving parts of this into a couple of posts. There are 6 in all, most of which I’ve cannibalised from the weather app. I’ll only talk about the new [or what I think are more interesting pieces]. And as I’ve mentioned before, messing with this kind of data has personal privacy implications. While people may not be interested, it’s best to make it difficult. The components are:

  • The iOS app generating the data.
  • This posts to my hosting service via a rest interface called Arrest-MySQL.
  • There is a database behind the REST interface, which is on my hosting service.
  • In order to limit the number of annotations / pins on the iOS app, I trim them with a SQL script embedded in PHP [the latter is a restriction imposed by my hosting service]. I call this as a nightly cron job from the Pi.
  • I then display this via pretty much the same Arduino sketch that for the clock for the weather data.
  • This receives data via Serial USB from the Pi based on a ‘nohup’ Perl script. I start and check this is running every 10 minutes using cron.

There is a 7th: we are going to South East Asia in a while, so I’ve written another little iPad app which my mother in law can use to track us with. My idea; I thought she’d be interested :).

I’ll start with the iOS app. One of the most immediate points to note about the iOS 7 background fetch capability is that, in comparison to using the Significant Change mechanism I’ve implemented before, it’s a lot gentler on the battery. While we are comparing the two, the fetch has two frequency settings: ‘none’ and ‘a lot’. I guess this might be something that Apple might increase the granularity on at some point in the future.

I guess there is a shopping list of features that I’ve picked up along the line which raise this above ‘bare bones’. The first is that if there is no network, I allow a manual ‘pin drop’ which writes the current coordinates to Core Data. Each time the background fetch fires [only when the network is available], I try to post these via the REST interface.

I also attempt to use Google’s GeoCoding API, which converts the latitude and longitude into a string for the given address, which then subsequently appears on the map annotation, or on the clock’s LCD display. I have to admit that in both instances, this is a bit of a dog, due to character limits, and custom annotations with mapping – well, while I’ve done it before, it’s substantially more effort than what this app warrants . I’m also not too sure what Google is going to return if we happen to be dropping location data in a jungle location with fantastic WiFi [!].

One more confession on the Core Data approach: originally I thought I would store failed writers to the database based on Reachability telling me I was offline when the background event fired. Actually, iOS does this for you: no network, no attempt to do a background fetch. While this is obvious, it never occurred to me based on prior experience with the Significant Change functionality which percolates away based on cell tower activity.

One or two more points of interest for the mobile app. There are limits on what you can do with the network in terms of libraries used / delegates available, so no AFNetworking. Actually for what I want to do in the foreground, AFNetworking is much too rich: for instance, I had to Google to find a workaround to its rather sticky caching. Also, something that really stumped me until I thought about it: if you are dynamically returning results [such as from the REST interface] don’t bother trying to plumb in a network progress indicator, unless your server is also setting a Content Length header. That’s a couple of hours of my life I’m not getting back :).

Reliable Serial Read with Arduino

I’ve refactored my weather station project significantly in the last few days, having found the RF based communications between the the two Arduinos too unreliable for my purposes. So I’ve invested in a WiFi shield instead, specifically an Adafruit CC3300. It was quite expensive, but I have to say it’s working fantastically well. I am now posting data directly to my JSON interface from the externally located Arduino. This has freed up the second Arduino [and Raspberry Pi] for what I originally intended: some sort of permanent internal display. I’ve actually got this working pretty well in conjunction with an Adafruit 1.8″ colour TFT [lovely piece of kit]. A Perl script on the Pi is in a constant loop, pulling down various piece of data, such as an RSS feed from the BBC, and the latest weather data. I combine this with the time and, based on string prefixes, position it on the screen accordingly.

It’s brought a problem that I’ve struggled with in the past back into play: the reliability of Serial.Read(), and I’ve found a stable solution which I’ve not come across elsewhere:

char inByte = Serial.read();
if (inByte != '^')
   // serialText string is defined earlier, and overwritten each time through the main event loop:
   serialText = serialText + inByte;

That delay() is enough to allow the serial buffer to refill and for me to reliably read the next character, up to the point that I hit the termination symbol I’m using. This has worked consistently for me now with the sending Perl script / receiving sketch merrily interplaying for days on end now, on 9600 baud. The serial comms are extremely sensitive to timing issues: for instance, before I put this delay in I was seeing different results on the Arduino between sending the data from the serial monitor on my Mac versus sending the same data from the Pi, via the Perl script. Suffice to say that your mileage may vary…

One other approach I’ve tried out is with the displaying of text. The TFT library doesn’t appear to support a delete function without zapping the entire display, so when I update the time display, which I send down the serial port every 10 seconds, I hold the prior value in a string, then display that in the background colour [black], display the new value, and finally set that for the buffer for deletion next time round:

// This is the prefix identifying that the string is the time:
String msgCheck = serialText.substring(0,5);
if (msgCheck == “TIME:”)
// There is a space so shuffling the substring over 1. Could take the space out but…!
String timeString = serialText.substring(6);
// delete the displayed time:
// set the value to the current time for the next delete operation
lastTimeMessage = timeString;
timeString = “”;

A final observations on the TFT library: the very accurate placement of the background colour text for deletion only seems to work if all of the text being displayed is the same size: so a write / delete cycle will only work with the correct placement of the cursor if all other interim writes around the screen are the same size text.

Shed Weather: a Journey Into Questionable Purpose

I’ve been continuing to fiddle around with my Arduino over the last few weeks, trying out various sensors and screens. One breakout board on the online store I’ve been using caught my eye, the delightfully named BMP180 Barometric Pressure / Temperature / Altitude sensor. I’d very nearly taken the plunge a few months ago with one of these, so I thought it would be an interesting project to try and build something along the same lines: install an Arduino in the shed [which has a power supply] and somehow or other get the temperature back to the Raspberry Pi. Some quick googling led me to here. The author of that Instructable article has done a very nice job, which I’ve borrowed the principle of: using two Arduinos, and a 433MHz RF link between them to transmit the data. What I wanted to add into the mix was an iPhone app, which necessitates a server component. That factor is why I’m not going to list all the code here: having a hosting service comes with a monthly bill [and a lot of other capability], which I think will very much limit the broader applicability. But there are a few points that I encountered, which I think are worth going into some detail.

First, the shopping list:

  • 2 x Arduino Uno boards. [£22 each].
  • As well as the barometric pressure board [£8], also using an HTU21D humidity sensor.
  • Aforementioned RF Link kit: [£2].
  • 2 x wire coat hangers, more of which later. Not sure how much these cost, but they tend to come with free clothes.
  • 1 plastic takeaway food container for the shed based components to sit in.

There were some other incidentals in the mix such as breadboards and jumper wires; I also repurposed an old iPhone charger for the outdoor sensor.

The pattern I decided to take was pretty simple: I’d connect the sensors to the first Arduino which is outside [well, inside the shed] and send the results in a continuous loop [every second] over the RF link, to be picked up by the receiver. I’d then run some code on the Pi, picking the data off the serial port, which I’d then send to a database server.

Getting the RF Link to work, using the VirtualWire library, was pretty straightforward. The hard bit is getting it to work if the transmitters were more than a few centimetres apart – note the picture in the Instructables article’s final page, and how far apart the two components are [i.e., not very!].

I spent a lot of time on this. I really didn’t want to have to run a second power adaptor to crank up the DC input to 12V, which is the maximum that the RF transmitter can handle [obviously this would be a direct connection, not the toast-making approach of trying to run 12V through the Arduino]. I ended up making a Yagi antenna, which I have to admit was a term I’d never come across before yesterday. If you google for some combination of terms for home made antennas for 433Mhz transmitters, you will invariably come across the same image, which is half way down this bulletin board. I followed the instructions, using the wire coat hangers, and it works. I have one end of the ‘live’ parts of the antenna connecting to the transmitter, and the other connecting to ground. I also used a dirty little hack which I saw somewhere else: I connect both the 3.3V and the 5V outputs of the Arduino to the input of the transmitter – note that this on its own wasn’t enough to cover the distance I need with the V1 antenna [17cm of wire], something like around 25 feet.

Yagi Antenna [post feline damage]So I was chuffed with the result, as without it, the project would have been a right off. One unintended consequence: within an hour of putting the thing in the garden, the cat promptly sat on it and broke off one of the elements. It still works. I have the cable running under the door of the shed, and used a glue gun to try to make sure that all of the connections were waterproof.

One slight problem I encountered was that the signal wouldn’t penetrate through to the place I originally intended having the receiver and Raspberry Pi: the corner of the house is in the way, so I had to relocate the Pi to a spare bedroom. This meant that my lovely little TFT screen is now redundant.

I messed around with the code quite a bit, and it wasn’t without its complexity. I’ve written before about coding in Perl to read off the serial port, and the approach that I had been taking was to embed a call to the serialPy Python library. I’ve discarded this and now use the Device::SerialPort library which is actually pretty elegant. My original approach to this was to run the Perl as a process, and then check that the process was still running with a cron job. This was based on some prototyping when I was taking readings from the sensors on a directly connected Arduino. This is much more reliable, so I was just sending the data from the sensor at the intervals that I wanted – hourly. As I mentioned earlier, I am now sending from the external board every second. I’ve refactored the code to be called from cron, which runs once and then fires the data up to the database. Note that I currently assume that the script will complete within the hourly cycle. I might as well include the Perl actually, as it introduces the database component:


# post to Arrest-MySQL via curl looks something like:
# curl --data "tempC=77.4&time=2014-01-22 19:02:01" http://yoursite.com/Arrest-MySQL_installdir/tablename

use Device::SerialPort;
use DateTime;

my $port = Device::SerialPort->new("/dev/ttyACM0");

$gotAllThree = 0;

$gotTemp = 0;
$gotHumidity = 0;
$gotPress = 0;

while($gotAllThree == 0) 
   $lastEpoch = time();
   my $char = $port -> lookfor();
   $char =~ s/[^!-~\s]//g;
   # print $char;
   if ($char =~ /T:/)
      $dt2 = DateTime->now(time_zone=>'local');
      $ymd = $dt2->ymd;
      $timeString = $dt2->hms(':');
      $timestamp = $ymd . " " . $timeString ;
      $char =~ s/Received: T: //;
      print "In temp:\n";
      if ($gotTemp == 0)
         print "T: curling\n";
         my $server_endpoint = "http://yoursite.com/Arrest-MySQL_installdir/tablename";
         system("curl --data \"tempC=$char&time=$timestamp\" $server_endpoint");
         $gotTemp  = 1;
   if ($char =~ /H:/)
      # print "got humidity: $char\n";
      $dt2 = DateTime->now(time_zone=>'local');
      $ymd = $dt2->ymd;
      $timeString = $dt2->hms(':');
      $timestamp = $ymd . " " . $timeString ;
      $char =~ s/Received: H: //;
      print "In Humidity\n";  
      if ($gotHumidity == 0)
         print "H: curling\n";
         my $server_endpoint = "http://yoursite.com/Arrest-MySQL_installdir/tablename";
         system("curl --data \"humidity=$char&time=$timestamp\" $server_endpoint");
         $gotHumidity = 1;
   if ($char =~ /P:/)
      # print "got pressure: $char\n";
      $dt2 = DateTime->now(time_zone=>'local');
      $ymd = $dt2->ymd;
      $timeString = $dt2->hms(':');
      $timestamp = $ymd . " " . $timeString ;
      # print "pressure time stamp is $timestamp\n";
      $char =~ s/Received: P: //;
      # last char is a dot?
      $char =~ s/\..*//;
      # print "pressure is now $char\n";
      print "In Pressure\n";  
      if ($gotPress == 0)
         print "P: curling\n";
         my $server_endpoint = "http://yoursite.com/Arrest-MySQL_installdir/tablename";
         system("curl --data \"millBars=$char&time=$timestamp\" $server_endpoint"); 
         $gotPress = 1;
   if (($gotPress == 1) && ($gotHumidity == 1) && ($gotTemp == 1))
      print "we got at least one hit\n";
      $gotAllThree = 1;

My original intention was to use an Amazon RDS instance, as this is free for a year which, if I’m honest, will be long after I’ll probably be bored with this set up and have pulled it apart. Using the whole AWS interface is kind of what I imagine having a quick go at piloting an aircraft carrier would be like: you tend to wonder what all of the buttons do. To put it another way, it’s very much geared up for an enterprise approach. All that said, I got it working [once I’d figured out how to disable the default and admirably paranoid firewall settings that prevent any external connections whatsoever]. Three days later I got my first bill, which was for a couple of bucks, for storage usage I think. From what I can see, the first year is free provided that you don’t actually use it.

I had a fallback option, which was the hosting service I use for this blog, and which allows me up to 3 mySQL installs, including the one that is used for WordPress. The only downside, and the reason it was second choice, is that there is no shell access: all of the database setup is via a web interface [phpMyAdmin].

I’d already realised from some early research that I needed to transform the data into JSON somehow or other. Googling ‘iOS SQL client’ quickly pointed me away from a native interface. What I finally settled on was a thing called Arrest MySQL. Note to self: the link I use for this is buried, but I need to add a .htaccess file to at least have a fig leaf of security. By default, simply posting to the interface will insert data into the table, if you know where to look.

So it’s working, and I am reliably informed that the outside [well, ‘shed’] temperature was 7.43 degrees Celcius, pressure was 988 Mb and humidity 68% the last time the cron job fired off. The iOS client is well under way too. It’s back to the coalface that is Core Plot for the primary functionality.

I’ve learned a lot from this, but end to end it’s something of a plate spinning exercise, however. The RF link obviously isn’t that reliable. I’ve also not really used a database before, and am having to figure out how to trim the data as I go, using events. It also makes sense to do other transformations [say turning the 24 hourly values into a daily average] server side as well.

But I think the reliability issue is what may make me strip it down and move onto the next Arduino-driven bauble that catches my interest :).