The objective of this project is to add a generic text interface to our Home Control System (HCS), which is agnostic to the underlying transport protocol. The goal is to enable a 'natural language' interface to our home automation and entertainment, that the whole family (and guests) can use, without having to understand the technology or master any new interfaces. It enables a person to have a dialogue with our Home Control System (HCS) over SMS, IM, Apple iMessage or similar text-based transport protocols. The chosen name is a slight misnomer though, because many of these protocols enable transport of images, video and audio clips. The initial focus is on text dialogue but, we are planing multimedia capability in the future.
This interface will allow all family members to interact simply and easily with our home, using standard apps/protocols on Smartphones, iPods, etc. More importantly, it provides a 'natural language' interface that requires no learning or prior knowledge. Security is enabled by the transport protocols so, e.g. SMS messages are accepted from 'authorised' numbers only, IM conversations are restricted to 'friends' of our home, etc. All interactions are also captured in our HCS log, so that we can understand how these interfaces are being used or attempting to be used, so that we can adapt the 'intelligence' and responses accordingly.
We have been using an SMS gateway to our Home Control System (HCS) project for a while now but, we wanted to incorporate other transport methods. To do this it makes sense to provide a more generic capability that can work with any transport protocol. Apple iMessage is of particular interest because of the number of iOS devices we have in our home. We also like the fact that it provides read receipts, has multimedia capability and is free when connected to an IP network.
The new process is basically a simple 'Artificial Intelligence (AI)' style engine (it is not really intelligent!) between the users and the HCS. It 'translates' text messages into commands that can be sent to the Home Control System (HCS) and then maps the responses back to meaningful responses that can be rendered in text. Having said that the concept could equally be applied to a voice interface and the responses spoken back as text-to-speech (TTS). We have already used Java TTS in our home automation system.
We have focus on a core subset of important commands and then spent far too much time trying to be clever! All entries in the form below are logged and our system is then updated to answer insightful or useful questions posed. In this respect, it learns :-)
To keep things simple and make rapid progress, the initial process was quite simple and used keyword spotting. The text interface is case-insensitive to make interaction simpler. Some simple examples are:
- 'In' - We are now at home, so house status is set to 'In'.
- 'Out' - We are now out, so house status is set to 'Out'.
- 'Away' - We are now away (on holiday, etc.), so house status is set to 'Away'.
- 'Alarm' - Sets the alarm status and looks for 'On' or 'Off'.
- 'Status' - Report current house and alarm status.
- 'Tomorrow' - Tomorrow's local weather report from the BBC.
- 'Recent' - Report back on most recent activity in our home. This will include rooms occupied and doors opened, etc.
- 'Weather' - Local weather report from the BBC.
The main code is written in PHP and runs on our Home Control System (HCS) Apache web server. The code is passed user input and responds with an appropriate response. It was done this way as it is easy to integrate into a wide range applications and via many different languages. It also means the one 'AI engine' can be used across all transport methods.
The process will recognise references to known devices, so that they can be controlled directly, e.g. 'lounge lamp on'.
The plan is to add context in the coming weeks, so that you could have a dialogue, e.g. 'What's the weather today?', 'and tomorrow?'.
To test the process out during development, we also built a web interface using AJAX, to enable a dialogue with our Home Control System (HCS). Below is a demo of the web/AJAX interface we have developed to our process and it supports a subset of the final implementation on our Home Control System (HCS). Have a play and let us know what you think. It is obviously not linked to our live Home Control System (HCS) but, in terms of the responses to questions and statements, it works and responds as our live service does (some responses are mocked up!):
Note: The set of characters that can be entered above is restricted to a-z, A-Z, 0-9, .=? and space.
We are currently re-using our previous SMS gateway to interface directly to our house.
After SMS, a Skype interface is the next logical step because it is very widely and a Skype app is available on a wide range of devices. There are also good APIs and developer support available. We have started to investigate the Java API.
XMPP is a widely adopted protocol for IM services and we plan to implement this interface soon. We have settled on the Smack API though as it provides a full framework that enables presence. Our XMPP home automation client can thus see when we are connected and on-line. This then enables our HCS to make a decision to send notifications via XMPP, instead of SMS and in many respects this makes it behave like Apple iMessage (which is also XMPP based).
We have also noticed that we can use the free text status message to convey useful information, such as alarm status and house status.
The text engine processes natural language requests but you can use 'short cuts', knowing the keywords and phrases that are being looked for. Today's weather and tomorrow's weather are good examples of this.
We are using the native GTalk on our Nexus 7 tablet and the Vtok app on our iOS devices (iPhone 5). These are both much more reliable. They also support video chat, which opens up a few more opportunities :-)
Apple iMessage Interface
This is where we really want to get to but, Apple's iMessage protocol is currently closed to 3rd party services and applications. Everyone in our home uses an iOS device and we all love iMessage for it's simplicity and features (read receipts, free when abroad, etc.). This shows the worst side of Apple and it's eco-system, where it restricts access to otherwise useful capabilities.
The core process is basically a 'chat bot' and it is easily possible to link it to other services like Twitter. One other interface we could see being implemented soon is a bespoke one in our home control Smartphone app.
The other goal we have is to provide a natural language speech interface, using technology like Apple's Siri.
Progress & Updates
So far we have the 2-way SMS gateway and web interface up and running. We have also produced a mobile phone optimised web front end for further testing.
We now also have the XMPP interface working, with our Home Control System (HCS) logging into Google Talk and monitoring family members presence and responding to messages sent via text chat sessions.
Having recently re-worked the database behind our Home Control System (HCS), we have now integrated the database into our text engine. This means that the system can respond with much more real time data and information.
Using an iOS Google Talk app and the inherent speech input on our iPhone 5, we are now very close to the planned natural language speech interface that we were aiming for.
We have greatly increased the set of commands that the
Combined with other projects like Z-Wave monitoring of our hot water system, we can now answer questions like, 'Is there enough hot water for a shower?' (or bath).
Commands Currently Supported
Our currently supports a long list of commands.
We have re-written the core response engine to generate an XML response, to make it more efficient and to also accept more command options. This is to better support voice recognition, thus enabling voice control of all aspects of our home automation system. We have also extended the range and number of commands accepted and understood.
The main interface used to access to this natural language interface is currently our iPhone web app, which exposes a text/dialogue box and some of the core features of our Home Control System (HCS) via buttons and visual indicators. Through experimentation and usage we are updating this on a regular basis.
There will be no further updates to this project page. This project has now morphed into an artificial intelligence engine within our smart home.