The 'Self-Driving' Smart Home

Since 2004, I've been developing the next generation of smart homes, something I call the contextual smart home. The main purpose of the contextual smart home is to deliver "intelligent assisted living for all". When most people hear the term 'assisted living' they tend to think of the elderly and vulnerable, but my definition is much broader and I in this article I will show many examples of how we can all benefit from it. [Photo by Tesla]

Within the contextual smart home many things work autonomously. Your home intelligently adapts and reacts and in many respects it starts to look like the home equivalent of a self-driving car. It can look after itself and its occupants, whilst providing numerous benefits, including safety, security, luxury and a great user experience. The contextual smart home essentially becomes 'self-driving' and delivers a zero touch user experience, much like the self-driving cars that Tesla are planning to deliver.

Why Would You Want A 'Self-Driving' Home?

The driving force behind my contextual smart home work is the concept of 'intelligent assisted living for all' (which is also the strapline for my company, Smartisant Ltd) and this is purely about improving quality of life for everyone, regardless of their physical or mental abilities.

Whilst the contextual smart home delivers a great automated user experience, this doesn't mean losing the ability to manually control things. You can still control all the features of your smart home manually (where permissions allow), using any of the user interfaces available, if that's what you want to do. In many cases you simply don't need to though. The contextual smart home has enough information an context, to predict your needs and do many things for you, before you need to ask. Not only is this incredibly convenient but it's also a very affordable luxury. For those with physical and mental impairments it also becomes a great way to maintain their independence and improve their comfort and quality of life.

Some Background

I started building my own smart home in 2004. The main reason was to get a great user experience, one that worked for my whole family. Back then, there really wasn't much smart home technology available but, that actually drove me to use the concept of technology abstraction and use a hybrid technology and technology agnostic approach, knowing that advances would come. My smart home is experiemental and I add new features almost daily. It currently has about 400 sensors and devices and it is on Twitter (@smartest_home), to educate people and show real examples of things happening in real-time. A real, lived-in home, provides far greater insight than any lab environment. In this article, I will often refer to tweets my @smartest_home has made to show real examples.

Self-Driving Car Architecture

This diagram shows the basic architecture of a self-driving car [Diagram from Udacity]:

I'm not going to go explain it in detail here but, I want to highlight a few of the key elements:

• There are lots of sensors and many are hidden and embedded in the fabric of the car and have been very carefully chosen to meet the required criteria in terms of cost, performance, reliability, etc. Basically, you don't build a self-driving car by going on a spending spree at your local Halfords.
• This is also true of the contextual smart home to some extent too. It makes a lot of sense to embed sensors into the fabric of your home, whilst it is being built because retrospectively fitting them entails a lot more work. Cobbling together devices from Amazon, eBay, etc. is not going to give you good performance, reliability or a great user experience.
• The perception stage shown above is about modelling the world in which the car operates, this includes the roads, maps, environment, etc.
• This then allows the car to intelligently plan what action to take and to then control the hardware elements of the vehicle (throttle, steering, etc.) accordingly.

Contextual Smart Home Architecture

If we look at the contextual smart home, we can see it has a very similar architecture:

Sensing

The contextual smart home can only do powerful things because it has many sensors and many types of sensors. Whilst my @smartest_home has hundreds of sensors, the contextual smart home can still do really useful things with a lot less but, the more sensors you add, the more understanding and context it has to work with.

Whilst sensors can be quite cheap, accuracy, timeliness and latency are important and good sensor selection and networking are critical. Wired sensors are better.

The contextual smart home can also collate information from trusted 3^rd party sources, such as weather forecasts, air quality reports, etc. This information has to be processed to extract meaning and value from the data.

The contextual smart home also supports the concept of virtual sensors, which exist in its models only. These allow it to operate more intelligently, by having wider context to work with.

Perception

At the core of the Home Control System perception element is a real-time events engine, receiving updates from sensors and services. My Home Control System will typically see 100,000 to 200,000 events per day but can easily handle many millions. Each is processed in just a few milliseconds, to enable rapid decision making an action. Low latency is key to a great user experience.

Some sensors may send an update very infrequently (e.g. a smoke alarm may only report changes in state when they happen). Some may send thousands of events a day. A good example being a PIR motion sensor in a busy part of your house.

Models

The key to the contextual smart home making sense of all this data is by making sure that ALL of the important elements of the smart home are modelled. The models must also enable understanding how all these elements relate to each other. The model may contain simple things, like which object resides in which zone but, it may also be complex, such a nested occupancy, presence and location model. In my @smartest_home every sensor and device is modelled as an object with an object type. Each type will also have many attributes that can be configured, for example an upper limit for its value.

The most important thing modelled by the smart home is its users, their personal devices and permissions. Having this centrally means that a consistent user experience can be delivered across all the user interfaces and each will enforce permissions. This means guests, children, etc. can not control or change things without permission.

Monitoring & Performance

My contextual smart home does extensive monitoring of its performance and a 'dashboard' allows many views into its workings, to better understand what is happening. This is an essential tool in allowing me to continually improve and extend its capabilities.

Services

My contextual smart home models key services and their availability. This includes things like mains power, the Internet connection and weather services.

AI & NLP

My contextual smart home has local artificial intelligence and Natural Language Processing (NLP), along with local and cloud-based Text-To-Speech (TTS). This means it can hold a threaded conversation (via text or voice) with several people at once. With known individuals (using an authenticated user interface), this is personalised and can work across several user interfaces at once.

Action

In the contextual smart home anything can generate a voice announcement or a notification. These are 'common capabilities' that anything connected to it will inherit. The ability to speak is enabled by a 'Text-To-Speech (TTS)' function.

The controllers are another common capability. You can assign a controller to any connected device and it allows you to simply and easily define how it should behave. These are incredibly powerful and allow complex things to happen simply, by using a concept I call 'technology abstraction'.

Control

The control element is mainly about networks, protocols and standards and this is covered in more detail elsewhere on my websites. It is a massive topic in its own right.

Summary

• A contextual smart home is one logical Home Control System, tightly integrated and it "sees" everything.
• Everything within my contextual smart home talks the same language, my unified communications protocol. Any 3^rd party components are "wrapped" so that they use this protocol too.
• It models all of the important things and these models are shared with all parts of the smart home and every user interface.

• It's not a collection of 3rd party hardware & services loosely bolted together.
• Having all this information immediately to hand means it makes better decisions and actions happen instantly. This results in a great user experience.

Self Driving?

A "self-driving" car is one that can drive itself without supervision or intervention and that's also the primary objective of the contextual smart home.

All the user interfaces work as expected and you still retain control but, mostly we aim for a "zero touch user experience", where things just work intelligently around you. And the great thing about this is that it works for everyone, young or old, and regardless of mobility or dexterity impairments.

Context

Before I talk about some real world examples, I want to reiterate the importance of context. The contextual smart home holds a lot of context, which helps drive its behaviours

• It can automatically know things like if you we are in, out or away on holiday and his will change the way many things work, such as Heating, Ventilation & Cooling (HVAC), IP cameras or lighting.
• It uses a nested zone model to track occupancy and presence in each room or zone and it also understands the relationship between rooms/zones.
• Other useful context is knowing if the mains power has been cut, the Internet connection is down, environmental data or the weather.

Occupancy, Presence, Location & Proximity

• Occupancy is when it sees an unknown person in a zone. A door contact sensor and PIR sensor are examples of sensors that provides occupancy information.
• Presence is when it can identify a known individual, e.g. when my son's PC is on in his bedroom it know he is also upstairs and in the house.
• Personal networked devices like smartphones provide proximity. We can't be sure a person is inside the home with devices like this because Wi-Fi passes through walls.
• Location tracking using smartphones and other connected devices provides more context, like estimated time of arrival home.

Algorithms

We also use algorithms to provide context. I have developed an 'Up Up' algorithm, so that my @smartest_home knows if we have really got up in the morning and someone hasn't just popped downstairs to get a glass of water.

My 'wasp in a box' algorithm algorithm is incredibly simple but very powerful. It means that when you walk into a room and shut the door, if my smart home then detects movement, it knows the room is occupied until the door is opened again. In practice, this means you never get left in the dark, even if you stand perfectly still. Applied at the home level, it means it knows our home is still occupied whilst all exits are closed.

Context Summary

All of this context combined is something I call 'whole home context' and it is extremely powerful. It also has huge privacy implications.

Lighting Examples

Most modern cars have an 'auto' setting for the lights, driven by a light level sensor. Some more advanced cars like my wife's Mazda CX-30 have really smart lights that automatically dip, point around corners and also light up signs at the side of the road.

The contextual smart home can also do some incredibly smart lighting control and the primary aim is for zero touch user experience, where the lights just work intelligently around us. A zero touch user experience doesn't mean you have lost control! We also want the smart home to be intuitive, so light switches and other user interfaces must work as expected too.

Lights and light switches are a great example of why you need local control. A great user experience requires very low latency, or you may push the switch a second time because nothing appeared to happen the first time.

Advanced Lighting Examples

To deliver a great user experience, I have also developed my own light switches and lighting. Push switches and rotary switches that allow brightness and colour control of LED light strips and smart bulbs.

I have also developed dynamic smart lighting which enable visual notifications, useful if you are in a noisy environment or hearing impaired. They also support effects, e.g. the telephone is ringing or the door bell has been pressed.

Safety & Security Examples

With the contextual smart home you get an incredibly advanced, context aware, safety AND security alarm system which can use any of the connected sensors. And it is basically free! All you need to add is internal & external alarm sounders and an alarm strobe, which are modelled objects.

My @smartest_home already supports safety lighting and it can also support emergency lighting.

Another key element of intelligent assisted living is improved safety and security. The contextual smart home can use all of the sensors in your home to improve your safety and security:

• It includes an extremely powerful alarm with the ability to send notifications or make voice announcements, clearly explaining what is happening.
• Every door contact sensor and PIR sensor can be used to track occupancy or to detect intruders.
• Each temperature sensor can be used to intelligently control Heating, Ventilation & Cooling (HVAC) or for early fire detection.

Safety & Security Sensor Examples

When building my @smartest_home, I would often find that there was no sensor available to meet my needs. This is why I developed the Smartisant Flood & Leak Sensor FLD1, a wired, 'fit and forget' sensor with a 25-year+ lifetime, that requires no batteries.

HVAC Examples

With numerous temperature and humidity sensors installed in optimal locations, the contextual smart home can do a much better job of heating or cooling your home and it can support any number of zones. Because this is all done in software, you basically get this functionality for free, so expensive "smart thermostats" are no longer required. The contextual smart home can directly control manifolds and boilers and also control boilers more intelligently using algorithms such as TPI, load compensation, etc. and standards like OpenTherm.

The contextual smart home knows if you are in, out or away on holiday automatically and so it can save energy by changing its behaviour accordingly. It also knows which room/zones are not in use or unoccupied. Since it knows who is at home it can also personalise and tailor the environment to those still at home.

With whole home context your Heating, Ventilation & Cooling (HVAC) is adaptive, so it will save you energy when the house is empty and it will take account of external weather factors to keep you more comfortable. It will use all the other sensors in your home to act more intelligently, so opening the front door to accept a delivery, won't immediately turn on the heating just because the sensor in your entrance hall detects the cold air coming in.

It is also aware of other heat sources in your home such as underfloor heating, heated towel rails, or even your airing cupboard.

The contextual smart home also gives you virtual sensor for free, so things like 'average house temperature' and 'average upstairs temperature' exist in its models only and support more intelligent control.

Assisted Living Examples

The contextual smart home can support traditional assisted living scenarios. It can track people's movement around the home and report exceptions. Some examples:

• They are still in bed at 11am.
• They have not made it into the kitchen by 10am.
• They visited the bathroom numerous times in the night.

The contextual smart home can also report exceptions, for example an elderly person living on their own will not usually have both the front and back doors open. This might be an indication of a distraction burglary.

It can also support the usual sensors in terms of panic buttons, fall sensors, gas hob sensor left on, etc.

Intelligent Assisted Living Examples

Intelligent assisted living benefits all of us. It's about making life simpler, safer and more convenient. It covers a huge range of things and there are too many examples to mention but, what follows is a taster:

Arriving Home

A simple example is when my wife arrives back home. My smart home will detect both her iPhone and her car, which has a £4 Wi-Fi enabled micro-processor in the glove box. If it is dark, it then lights up the drive and porch for her. The front door could also be unlocked too.

This is an incredibly easy thing to configure as I use a controller to control both the drive light and porch light and they are simple sent an extend (the time the light is on) request.

Appliances

I have numerous examples showing how my contextual smart home monitors appliances in my contextual smart home in my home and intelligently controls them.

Things like my wife's hair straighteners will generate notification and voice announcement if they are left on. This informs family members that preventative action has been taken and that they have been switched off.

Curtains & Blinds

Smart curtain and blinds are very popular right now. A controller enables very powerful control and the simplest example is that they open at dawn and close at dusk. They can make your home look occupied when you are away on holiday, improving security. Venetian blinds can even be tilted based on the position of the sun, to passively heat or cool your home.

Heated Towel Rails

Heated towel rails are another good example, making sure you always have a nice warm towel when you step out of the bath or shower.

Smart Mirrors

The term 'smart mirror means many things but, in my home I currently have 'reactive smart mirrors' that sense you standing in front of them. They then light up and a heater attached to the mirror glass ensures that will never mist up. The brightness of the light is determined by my Home Control System based on whole home context, so you won't ever be dazzled at night. A more advanced 'interactive smart mirror' uses a one-way mirror with a touch-screen display behind it. (project page)