Sky Broadband, YouTube Restricted by Network Administrator and Other Woes

TL;DR: if you’re a Sky customer and YouTube suddenly seems a bit broken, have a look at a service called ‘Broadband Shield’. Read about what it’s doing before you turn it off: you may actually want to live with it.

During the week, I started noticing something weird was happening with the DNS on my home network. I don’t have anything vastly complicated set up, but I use pfSense to apply rules to route traffic between three physically separate networks. I’ve had a mess around with DNS configuration before, so that was my first port of call. I tried changing over from Google to CloudFlare to see if that helped, in a just-home-from-work-let’s-poke-a-stick-at-it sort of way. It didn’t. I was able to get access to various sites in the .com domain – and we’re talking, so not exactly off the beaten track – for a while, and then all of a sudden, the names would stop resolving. Oddly enough, I seemed to be having the same problems with my VPN, which I thought offloaded DNS. I didn’t test this thoroughly though.

The real clue came with YouTube, where I was noticing that some videos and all comments were being blocked, and some rendering problems in the app view layout – stuff where there should have been content was blank. Initially I thought this was because Google was delivering content from different domains, and my DNS woes were continuing. Looking at the settings for the YouTube app, the switch for ‘restricted mode’ was greyed out with some text explaining that it had been enabled by the network administrator. Up until this point, I’d have assumed this was me :).

A little bit of googling later to see what sort of parental controls Sky were offering me landed me at the Broadband Shield site. I turned it off, and suddenly all of the name resolution weirdness disappeared, and the YouTube comments were back.

I appreciate the intent with the Shield service but I’d appreciate it a lot more if Sky had told me that they were turning it on, or changing it in a way that would apply a pretty broken policy to name resolution.

Parsing FIDO Raw Messages

I’ve spent the last few weeks working on iOS app to use a Feitian BLE Multipass token. When I get to the final version of that I’ll do a separate posting but parsing the raw messages to and from the device turned out to be pretty challenging. Unfortunately, if you send something that doesn’t conform to the required format, rather than getting an error back, you get no response at all. This makes debugging [particularly at the start, when I really didn’t know what I was doing] tricky.

While the standards for both the FIDO BlueTooth spec and the raw message format are both well written, there are a few bits that aren’t immediately clear, and that I couldn’t find documented anywhere.

The Feitian device enforces a maximum fragment size of 20 bytes. Any arithmetic I use in the rest of this write-up will assume this, but that will vary from one device to another. Also, I will tend to use the actual number of hex characters transmitted. As the payloads are double byte, this will mean that a decimal value will be derived by converting a hex value to decimal, and then doubling it.

Sending An Enrolment Command

The first fragment you send looks like this:   

The FIDO command prefix has the value 830049. The 83 indicates a message, the 00 is a blank high data length, and 49 is the length in hex, which equates to 73 in decimal, 146 characters. This is derived from adding together the APDU command prefix [14 characters], the concatenated challenge and application parameters [128], and the a final 4 characters, all zeroes, which are the APDU Le values. 

The APDU command – the example I’m using is enrolment – has the value 00010000000040. As per section 3.1 of the raw format spec, that’s a preceding 00, then the enrolment command of 01, then a bunch of blank parameters, and ultimately the 40, which equates to the 64 bytes for the combined length of the challenge and application parameters.

For each of the subsequent chunks, you have a payload [in the Feitian’s case] of 19 double bytes, plus one more byte for the order. The last message includes the 4 bytes for the Le values.

Parsing the Response

The response is a bit trickier. First, the device may respond with a FIDO keep alive command, which is 82000101. So before I start concatenating the full response, I check that the individual chunk value doesn’t equal this. Just a quick detour into the implementation: the BlueTooth spec says that the values you get back from the token will be Big Endian double bytes. I couldn’t find a nice way of processing these using an NSUTF value, so I just convert them to text and parse out the angle brackets and spaces.

On to the processing. The first six characters I get are 830259. Per the spec, 83 is playing back the original command. 259 in hex means that the total length of the response will be 1202 characters. That’s going to vary: both the attestation certificate and the signature are variable length. For my implementation, I calculate a running total, so that I know when the final message has been sent.

So, there is no APDU header at all in the response. After the 6 character FIDO header, you get straight into the standard payload as per the spec. You have a legacy 05, then 130 characters for the public key, then a key handle length. For my device, the value is 60, which translates [96 in decimal x 2] to 192 characters. 

The next part took quite a bit of digging, which is figuring out the length of the attestation certificate. The certificate starts with the following: 30820168. While my read of DER encoding of ASN.1 [ugh!] is that the length specification can vary, for me, that 0168 is the length in hex, equating to 360 bytes. Add back in 4 bytes to include the header [30820168], that gives me an attestation cert length of 728 characters. 

What remains is 144 characters. The last 4 are 9000, which per the raw message spec is the SW_NO_ERROR value. 

And that’s it. 6 weeks of trial and error in about 700 words.

Using Sunset to adjust Philips Hue Sensor On/Off Times

I’ve been meaning to do this for ages, and it turned out to be a bit easier than I thought. The Python I’m using is a bit rough and ready – rather than the 8 separate API calls to PUT the date changes back [4 rules for each sensor], I could do this with one API call, based on the single data structure for all of the rules. It doesn’t really matter that much.

I haven’t done anything for sunrise processing, which fits our usage patterns with the sensors.

Here’s the script, which I’m calling from cron, once a day, which is a bit of overkill. I could probably do it weekly:

#!/usr/bin/env python3

import requests
import json
import time
from datetime import datetime, timedelta

fudgeFactorMins = -30
morningOffTime = "T08:00:00"

The fudgeFactorMins variable is an offset for the actual sunset: I eventually subtract this from the sunset date. The rule numbers are not going to be nailed on, but they do appear to be contiguous for each sensor, in groups of 4:

# Format: "T20:00:00/T08:00:00"

# 8am / 8pm: day-on
bathroomDayOnRuleURL = "http://yourIP/api/yourKey/rules/1"
# 8am / 8pm: day-dark-on
bathroomDayDarkOnRuleURL = "http://yourIP/api/yourKey/rules/2"
# 8pm / 8am: night-on
bathroomNightOnRuleURL = "http://yourIP/api/yourKey/rules/3"
# 8pm / 8am: night-dark-on
bathroomNightDarkOnRuleURL = ""

# 8am / 8pdm: day-on
livingroomDayOnRuleURL = "http://yourIP/api/yourKey/rules/9"
# 8am / 8pm: day-dark-on
livingroomDayDarkOnRuleURL = "http://yourIP/api/yourKey/rules/10"
# 8pm / 8am: night-on
livingroomNightOnRuleURL = "http://yourIP/api/yourKey/rules/11"
# 8pm / 8am: night-dark-on
livingroomNightDarkOnRuleURL = "http://yourIP/api/yourKey/rules/12"

This is a bit messy but…

allUrls = [bathroomDayOnRuleURL,bathroomDayDarkOnRuleURL,bathroomNightOnRuleURL,bathroomNightDarkOnRuleURL,li
vingroomDayOnRuleURL, livingroomDayDarkOnRuleURL, livingroomNightOnRuleURL, livingroomNightDarkOnRuleURL]

The sunsetAPIurl refers to a free API. I had a look at one implementation on StackOverflow which calculates locally. This endpoint is returning results within a few mins of the app on my phone, which is certainly good enough.

One slight nuisance is that I need to adjust for British SummerTime. What I’m doing for now is using Daylight Saving as the Python datetime library gives you it for free. It’s going to ‘wrong’ for the week or so when DST and BST are out of synch at the start and end of the summer, but I can live with that for now:

sunsetAPIurl = ""

def getSunset(sunsetAPIurl):
   r = requests.get(sunsetAPIurl)
   jsonData = r.json()
   # date format is 8:08:18 PM
   dateFromJson = datetime.strptime(jsonData["results"]["sunset"], '%I:%M:%S %p')
   if is_dst():
     adjustedSunset = dateFromJson + timedelta(hours=1)
     return adjustedSunset
     return dateFromJson

def is_dst( ):
    return bool(time.localtime( ).tm_isdst)

So another quick adjustment to reformat to the requirements of the Hue endpoint date handling, after processing the fudge factor:

def adjustSunsetToString(sunset, fudgeFactorMins):
  fudgeAdjusted = sunset + timedelta(minutes=fudgeFactorMins)
  return fudgeAdjusted.strftime('T%H:%M:%S')

def hitEndpoint(allUrls, dayOnString, nightOnString):
  for oneUrl in allUrls:
    r = requests.get(oneUrl)
    jsonData = r.json()
    currentDate = jsonData["conditions"][0]["value"]
    #print("currentDate: " + currentDate)
    ruleName = jsonData["name"]
    #print("rulename: " + ruleName)

The dates that I push back into the JSON need to conditionally flick around [earliest / latest or vice versa], which I do based on the substring search for ‘day’ on the rule name:

    if "day" in ruleName:
      jsonData["conditions"][0]["value"] = dayOnString
      jsonData["conditions"][0]["value"] = nightOnString
    newJsonPayload = {"conditions": jsonData["conditions"], "actions": jsonData["actions"]}
    r = requests.put(oneUrl, json = newJsonPayload)
    #print("response: " + r.text)

sunset = getSunset(sunsetAPIurl)
adjStrDate = adjustSunsetToString(sunset, fudgeFactorMins)

dayOnString = morningOffTime + "/" + adjStrDate
nightOnString = adjStrDate + "/" +  morningOffTime

hitEndpoint(allUrls, dayOnString, nightOnString)


Philips’ Hue Sensor Automation

I wrote about my first pass at the Philips’ Hue API to customise the motion sensor functionality back in December. At the time, I tested out my approach to force a third time slot [use of a nightlight setting for part of the the night] using a simple mobile app in Objective C. The approach isn’t that fancy: it’s basically 3 API PUT calls: two to rules, and then a third to a resourcelink all basically doing the same thing: inverting references to scenes [one for full brightness, and the other for the nightlight setting].

At the time I thought it would be handy to use a Raspberry Pi using cron jobs to toggle it off and on, and the long and short of it, that’s what I’ve gone with. It’s a simple Python script [using the Requests module, which is great, to make the API calls], and it’s working really well so far.

A couple of observations. First, I found an article on the Hue discussion forum – which I only found when I got a weird error and which, unfortunately I can’t find any more – that has what’s probably a more purist way of using the API to create extra slots. While it’s smarter than what I’m doing, unfortunately [and the author of the post does point this out] it has the effect of stopping the mobile app from working to set the sensor.

Secondly, my approach is pretty brittle. Between writing the app and implementing the Python script the state had changed in the bridge: there was a new scene [not one we added – at least intentionally] which was causing the API calls to fail.

Currently, my script is doing exactly what my mobile app did, which is creating the PUT payload for the API calls from files, and ‘hoping’ that they work. What the Philips’ mobile app does is to GET the various API resources and then PUT them back. I’ll do this when the current implementation breaks as it’s more trouble than it’s worth for now.

I was telling someone at work [a proper developer] about my results with proxying via ZAP, which I had to revisit to figure out why my original implementation had stopped working, and he couldn’t believe the fact that the app is making around 55 API calls to effect a change in the sensor setup. Based on the diffs I looked at, It seems to pretty blindly do the GETting and PUTting where no changes have been made.

I managed to completely corrupt the setup on the bridge by playing the ‘I wonder what will happen if I do this?’ game: I deleted the scene that was breaking my API calls. I ended up having to do a factory reset – about a five minute job for me, but worth bearing in mind if you are playing with this stuff and have a more sophisticated configuration.

Automating Philips Hue Motion Sensor Functionality

OK, this turned out to be a *lot* more complicated than I thought. Here’s what I wanted to do: we have a Hue motion sensor in the bathroom which I’ve set up to do nothing during the day, and then come on at night. Nothing too controversial there. But I wanted to have a 3rd option: for a certain section of the night – say after midnight – to come on at a nightlight setting.

There’s a ‘3 times a charm formula‘ from the Hue Labs which sounded promising, but that only allows you to set the idle time after motion activation. So I started looking at the API. There were a few obvious options I tried which didn’t work [like simply setting the brightness – over-written by whatever scene you have set in the accessory setup].

This article set me on the right track for what I needed to do insofar as rule settings. By toggling changes on the smartphone app and then looking at the results in the Clip API debugger I narrowed down the changes to two rules called ‘MotionSensor 5.night-on’ and ‘MotionSensor 5.night-dark-on’. [Obviously the number is dependent on the identifier the sensor is registered under.]

This isn’t enough on its own. I gave up on this after my first pass, waiting for a rainy day to come back to it – today.

I ended up using OWASP ZAP to proxy the Hue app traffic, to see what it was doing when you submit a change to the sensor configuration. That’s quite interesting actually: I was initially a bit puzzled at what looked like a mountain of traffic on first pass, but what in part turns out to be a very zealous keep-alive – roughly twice a second.

Long and short of it is that resetting the scene config using the app overwrites a ‘resourcelink’ with new scene specific values, as well as updating the rules.

I’ve roughed this out with my own iOS app just to check that it works in a way that I can transplant to another setup [a cron job on a Raspberry Pi would be a reasonable candidate]. To toggle on the ‘dim at night after midnight’ setting I make 3 API calls. You’ll need to fiddle around with the clip tool to get the equivalents for your hub – specifically, your sensor number will be different. For me, these are:

  • Rule: MotionSensor 5.night-on
  • Rule: MotionSensor 5.night-dark-on
  • Resourcelink: MotionSensor 5

For the rules, I Post the same ‘actions’ payload to both of the corresponding endpoints. It looks like:

{"actions": [
                        "address": "/groups/YourActionNumber/action",
                        "method": "PUT",
                        "body": {
                                "scene": "YourDimSceneIDHere"
                        "address": "/sensors/YourStateNumber/state",
                        "method": "PUT",
                        "body": {
                                "status": 1

Then for the resourcelink, I post:

    "name": "MotionSensor 5",
    "description": "MotionSensor 5 behavior",
    "type": "Link",
    "classid": 10020,
    "owner": "OwnerIDHere",
    "recycle": false,
    "links": [

The sensor and rule set you’re linking to will be specific to your own setup.

I’ve no idea what that ‘recover scene’ is. I’m guessing it’s something that’s created when you configure a sensor in the first instance, and is some sort of default state.

There’s a corresponding 3 API calls which I need to make to toggle back to the scene for ‘bright pre-midnight’, replacing the scene ID accordingly.

At the point I gave up – where I was just making the rule changes – I could tell that it wasn’t right by loading up the config in the app. The partial configuration – without the resourcelink change – manifest itself simply as ‘do nothing’ for the nighttime slot. Resetting this restored the full config. I guess there’s always the danger that if you run wild and free with setting values incorrectly, you might invalidate the ruleset to the point where unpicking it could be a problem. All I can say is that the above approach works for me, but test carefully!

Making Your WordPress Site Harder to Break Into

If you are using an infrastructure-as-a-service offering to host your WordPress site, you might find the following suggestions useful. If you have web access to site administration, e.g., via CPanel, there’s not much more you can do beyond:

  • Install WordFence.
  • Pick a very long password for your admin account. Have a look at secondary authentication options [described below].
  • Keep your software up to date.
  • Have a long hard look at all of those plugins you’re using, and try to cut them to the bone.

Here are a few Linux specific things to consider:

  • Install fail2ban.
  • Install the fail2ban WordPress plugin.
  • Change the default port you are running sshd on from 22 to something above 1024. You might want to check the IANA port listing to avoid colliding with other daemons you may be running on your box. This is pretty painless to do. You could do the same for all of your other network services [obviously excluding your web site and mail transfer agent, if you’re using one].
  • Consider running TCP wrappers on your ssh daemon access. Be careful not to lock yourself out if you don’t have a fixed IP address from your service provider. I use BT: a bit of googling suggested a couple of likely address ranges [81. and 86.], but my get out of jail card is a static IP I can use with a VPN service, which I’ve also configured.
  • Consider restricting access to mysqld to localhost. You may have already done this during install. I have to admit I was pretty careless when I was installing the database in terms of notetaking so I just added a TCP wrapper to be on the safe side.
  • Have an extremely long password for your admin account. Mine is basically half a mile long in 8 point text.
  • Have a look at some secondary authentication [or at least Captcha style robot identification] mechanism on the login form. I’ve experimented with Duo in the past. Make sure you understand the implications of changing your mobile phone if you are using one of the app based mechanisms.
  • Consider chrooting your website. I don’t but I may.
  • Have a look at some of the security scanners, like ZAP. Again, be careful not to lock yourself out with fail2ban.

The rest of these are host specific. A good starting point is to run

nmap -n localhost

to confirm what network services you are running. For instance, I found that my Linux distro had an FTP server running out of the box which I didn’t know about. In my case it was as simple as

pgrep ftp

to get the process id, then

ps -ef | grep <process_number>

[substituting appropriately] to find what the process was, and then finally

apt-get remove <package_name>

For any other servers you need to run, there’s a pretty good chance that there is fail2ban config that you can use off the shelf.

Why I felt I needed to pay more attention…

A few months ago, based on the daily activity reports I get from WordFence, I started blocking ranges of addresses using iptables. So if I got repeated brute force attempt on my admin password from say [this is a real example], I’d block the the entire range by doing:

iptables -A INPUT -s -j DROP

This was pretty tedious, but I only needed to do it around once a week. For some reason [presumably it’s some sort of off the shelf attack script], the attack counts from specific IPs always seemed to top out at exactly 1460 hits. Last week, I got my daily email which showed someone had hit the site 49k times that day. By the time I got home and did some digging in the logs, it was totalling more than 80k for the week. I suddenly realised that my manual blocking was way too blunt a tool, so last weekend I wrote a log scanning  script launched by cron every 30 minutes. It pattern matches on the login form in the Apache log file and sends me an email if an individual IP tops out at over a certain threshold. The longer term plan was that I’d call another script that would automatically insert the block rule in iptables. There would be a bit of messing around with state management [seek to a point in the log file based on the timestamp to not go over the same ground twice or something] and it would have to be run as root, but it would be better than nothing. I happened to mention this to someone at work who told me about fail2ban, which is doing the same thing [but better than I could have implemented it].

I also had a dig around at ssh logs, and discovered that some kind soul was trying to brute force the root password, and was up to 8k attempts for the week. Subsequent digging showed that the same thing was happening with my Imap server, but on a smaller scale.

While what I’ve suggested here isn’t necessarily bullet proof, it’s a reasonable start. Bear in mind that the types of attacks I’m seeing are commensurate with the value of my site, which in commercial terms is zero. And oh, the irony, when my server gets brought to its knees next week by someone with serious intent :).

Pin Your Pics…

My year and a bit dalliance with Android concluded as of yesterday, when I took delivery of an iPhone 8. I’m going to start developing again [although I’m going to have to divide my attention with C#, which I’m learning at work].

I’ll need to dust down a couple of projects: I have a password manager which I wrote as an experiment to learn about the Touch ID back in the day. I also really need to update my apps on the App Store…

…All of which leads me in a round about way to Pin Your Pics. It’s had very modest downloads since I released it: very low hundreds. That was up until July of this year, when it appears to have been ‘discovered’ in some way. I’ve no idea why or how. Anyway, this is what the downloads look like for this month which, by comparison to before, is pretty interesting:

Downloads for Pin Your Pics

I’m sure there’s every chance that it will sink without trace again.

Regardless, at some point in the next few weeks I’ll have a look at screen resolution – when I wrote it, it was fixed to the 5S screen dimensions, I think. Also, there’s a perennial bug with the controller for launching the camera, which I’ll have another bash at.

Creating an Alexa Custom Skill on a Web Service

The short version: there is no short version. Implementing server side functionality inevitably involves a fair amount of legwork. Also, an advance warning: I’m a novice at the programming language I chose to implement my web service in – Java. For this reason, I’m only going to post code snippets for problems you need to solve. There may be better ways of doing this. Finally, I’m leaving my Skill permanently in development mode: I don’t intend to publish. While it’s something I find useful, this was primarily a learning exercise for me. If you’ve ended up here via Google, hopefully you’ll find something you can re-purpose.

I’ll come to the Amazon side of things in a bit. First up, you will need a server to host your web service. You have a couple of options here [as an alternative to Lambda]: first, you can host on your home network. If you have DHCP, the typical process is to configure your broadband router for DDNS, and have a port forwarding rule on to your server. I decided to take another approach, simply because I wanted to try it out. I’ve paid for a vm with a hosting outfit called OVH. They have a really nice setup, so I’ve spent less than £10 for a server for 3 months. I’ll probably keep it on permanently as I like the flexibility.

I decided to implement my web service as a servlet. There are plenty of tutorials on how to set up the stub methods [and all the various libraries and files structures you will need when you come to export a War file]. I used Eclipse as my IDE. My testing cycle – which I imagine is pretty common – was to run tomcat on my laptop where I was doing the development. I could then build and deploy to that locally, and then test the interface by POSTing data to it with the Chrome plugin Postman. When I had something I wanted to deploy to my vm, I exported the WAR file, scp’ed it to the OVH server’s web apps directory, removed the old War [and expanded directory] and restarted.

A couple of foundational steps on the remote server. The OVH setup is a vanilla install of your operating system of choice – I went with Debian. As well as installing your server software [tomcat] and its requirements [Java], you’ll need to create a user for it. As Amazon requires the server to run on a privileged port, you’ll need to take this into account. As you really won’t want to run the server as root, I googled around for a snippet of C code which I could adapt and compile, which could use setuid to start my server as another user. This is something to take care over if you intend to run your service in anger, and you have no choice: you have got to run your service on port 443.

The next step on the server side, related to that privileged port, is TLS. Because I tried a ‘real’ CA, which I thought was on Amazon’s supported list [Let’s Encrypt], I also went to the trouble of registering a name for my vm’s IP. If you’re going to stay in developer mode, you may not need to do this. When I realised I was getting TLS handshake errors, I changed the config and went with an openssl self-signed certificate.

Turning to the configuration set up on the Amazon side, it’s generally pretty well documented, but there are a couple of exceptions to this. First, in the Skill Information, you have to provide an Invocation Name. I know of one other developer who made the same mistake as me because of the way the debugging pop-up information is worded: I assumed that the invocation string needed to start with ‘ask’, or equivalent. It doesn’t. If you just want your skill to be invoked with ‘widget counter’, you just have those two words, and then kick it off on your Echo with ‘Alexa, ask widget counter’, followed by one of your sample utterances.

This seems incredibly obvious, but there is a warning on the testing page which states, ‘Please complete the Interaction Model tab to start testing this skill’, I went off on a wild goose chase thinking I’d done something else wrong. That warning is arguably one of the misleading features of the whole orchestration interface that Amazon provides. It stays there permanently, no matter what you do.

There are some good examples of the Intent Schema, and other parts of the interaction model already out there but one more can’t hurt. Before I get into it, a quick explanation of what my skill does. My server side code ‘screen scrapes’ a train operator’s web site every 10 minutes. I then parse the rendered HTML looking for specific information about service conditions. That information forms the basis of the response.

Intent Schema:


The custom slot looks like:

LIST_OF_SERVICES Peterborough | Luton | Cambridge | Brighton

and then some sample utterances:

TrainTimes how is {TrainServices}
TrainTimes how is the {TrainServices} service
TrainTimes how is the {TrainServices} line

[Brief aside: why ‘train times’, when it’s not timing related? This is a vestige of trying to navigate the mistake I made with the invocation name. I have a strong accent and so was trying all sorts, so at one point it was called ‘big dog’ :)].

The rest of the Amazon interface is pretty straightforward, such as the uploading of self signed certificate [if that’s the approach you choose to take]. One final step before moving away from the interface is the testing. The example question, ‘how is the peterborough service’ gets transformed into the following blob of JSON:

  "session": {
    "sessionId": "SessionId.[GUID here]",
    "application": {
      "applicationId": "amzn1.ask.skill.[another GUID here]"
    "attributes": {},
    "user": {
      "userId": "amzn1.ask.account.[long account string here]"
    "new": true
  "request": {
    "type": "IntentRequest",
    "requestId": "EdwRequestId.[one more GUID here]",
    "locale": "en-GB",
    "timestamp": "2017-02-06T13:43:37Z",
    "intent": {
      "name": "TrainTimes",
      "slots": {
        "TrainServices": {
          "name": "TrainServices",
          "value": "Peterborough"
  "version": "1.0"

You will obviously need to check that your web service is able to parse this. As per my comments on  my testing cycle, I copied this into Postman as the raw content of a POST.

I’ll briefly go into some more detail about how my implementation works, because when you get into the business end of parsing the input JSON, you need to map it back to some data to respond with.

The parsing of the train company website as a skill isn’t my first bite at the implementation. I started by trying to write a widget for Android. It’s not as simple as just grabbing the HTML, though: the site uses Ajax to render the results. I ended up using HtmlUnit. Because it’s pretty slow, I implemented a class outside the servlet which uses a combination of a ServletContextListener, and then a ScheduledExectuorService. I got the example code from here to work from. The one gotcha is that your web.xml needs to refer to the class.

Next we get on to the hacky part of passing the information on to the servlet. What I did – having the listener write to a flat file which the servlet reads – is a reflection of my novice levels with Java.

So, returning to the list of general problems to solve: having received the POST data, your service then needs to get hold of the data, and then parse the JSON. For the first part, I tried  a few different options and then settled on the getBody implementation I found here.

The parsing of the JSON itself: it’s the first time I’ve ever done this and it feels way more complicated than it should be. I tried a couple of different parsing libraries, but ended up using JSON.simple.

Going back to the JSON data from the Amazon test interface, you have two consecutive objects, the session and the request. For an industrial strength implementation, there are things you need to do with the session data [I’ve also skipped over the TLS client certificate validation, which Amazon stipulates. While perfectly sensible for security purposes, it’s not necessary to get your development implementation up and running.

So the request itself contains a series of embedded objects that you need to unpack, in turn, until I finally get to the slots data. Your implementation will vary, but mine looks like this:

        JSONParser parser = new JSONParser();
        try {
            JSONObject obj = (JSONObject) parser.parse(postData);
            JSONObject jrequest = (JSONObject) obj.get("request");
            log.warning("request: " + jrequest.toString());
            JSONObject intent = (JSONObject) jrequest.get("intent");
            log.warning("intent: " + intent.toString());
            JSONObject slots = (JSONObject) intent.get("slots");
            log.warning("slots: " + slots.toString());
            JSONObject trainSvces = (JSONObject) slots.get("TrainServices");
            log.warning("trainSvces: " + trainSvces.toString());
            String trainSvceVal = (String) trainSvces.get("value");
            log.warning("trainSvceVal: " + trainSvceVal);
            resultString = trainSvceVal;
        } catch (ParseException e) {
            log.warning("ParseException error: ");

Again, for a ‘proper’ implementation, you are going to have to test each of the sub-objects. My code is taking a leap of faith :).

Having lost the will to live with the parsing, I have to admit that, having figured out which piece of the train-related data the web service needs to respond with, I then print the JSON straight out in the required format.


        PrintWriter out = response.getWriter();
        out.println("  \"response\": {");
        out.println("      \"outputSpeech\": {");
        out.println("         \"type\": \"PlainText\",");
        out.println("         \"text\": \"" + resultsString + "\"");
        out.println("    },");
        out.println("    \"shouldEndSession\": true");
        out.println("    }");

So, not exactly a thing of beauty, but it works.

If I have any better ideas for future skills to implement, I’ll start with the certificate that Amazon needs. There’s an outfit called StartSSL which it looks like they accept, but which requires email based validation on an address corresponding to the domain. I started looking at setting up PostFix for this – one for another day.