Occupancy, Presence, Proximity & Location
This is our smart home automation project to model occupancy and the presence of people in our home, so that we can accurately determine who is at home and ideally know which room (zone) they are in. This information is then be used by our Home Control System (HCS) to make informed decisions and ultimately be able to predict events and take pre-emptive action. This is an on-going project that we continue to refine and improve. Occupancy and presence are one of the key defining features of the smart home. Without this capability it is pretty much impossible to move from simple home automation to smart home automation.
A key objective for our home is to be able to automatically update the house status, which then alters things like target temperatures for rooms, etc. The biggest challenge is achieving the right balance between manual control (i.e. us telling the house we are out), scheduled control (e.g. a learning pattern that defines when we are likely to be in) and detected occupancy or presence of people in our home.
Devices like the Nest thermostat use a PIR to detect the presence of people and modify their behaviour but, a single PIR sensor is a very compromised approach, especially if mounted in a 'quiet' area. It is also not linked to other smart home sensors, devices and services. A 'whole house' approach is always going to provide a more useful and accurate solution, especially when it is linked to all other smart home capabilities and uses a hybrid technology approach to utilise occupancy and presence data from the all the technologies found in our smart home.
- We define occupancy as knowing if there is person (or several people) inside our home, without knowing who they are. We also track occupancy down to room or zone level.
- The definition of presence is very similar but, the identity of the person (or identities of the people) within our home or each zone is known.
Whilst presence is very desirable, it assumes people are carrying some kind of device or technology that allows you to identify them (short of using feature recognition, which is not really accurate) and this is very often not the case. Occupancy detection is much easier and it is just as important in the smart home. This is why we have adopted a model that combines both occupancy and presence. Our software models identified individuals and their known devices and we also support a person called 'Unknown' for occupancy. This unknown person(s) can occupy more than one zone/room at a time, whereas identified individuals cannot.
- Proximity is when a personal device (which can be associated with a specific person) is detected in the neighbourhood of a zone. We have developed the concept of proximity because wireless networked devices use radio waves and radio waves can pass through walls.
- Location is the ability to pin-point a device/person (with centimetre level accuracy) within a room or zone.
At its simplest level, occupancy detection in the home enables more advanced control of things like heating and cooling, hot water heating, etc. Occupancy applied at a zone level enables basic functionality such as automated lighting that doesn't switch off whilst someone is still in the room (e.g. in a toilet or bathroom).
The real goal of this project is to provide reliable and accurate presence detection, to enable intelligent personalisation in our home. This could be entertainment that is personalised and follows you through the house, personal environmental settings such as temperatures, lighting and even lighting colour. It can also include information and communications being delivered to our current location. Ultimately, we see our home making predictions and taking pre-emptive action to improve quality of life. We would also like it to deliver some personalised serendipity.
The bottom line though, is that a smart home with occupancy and presence is incredibly efficient and convenient. The added intelligence and behaviour also adds some subtle style.
Our approach is based upon a combination of many years work (in my day job), leading a team of researchers on future communications and online presence and many projects and experiments in our current home. For many of the same reasons that we have adopted a hybrid technology approach to our Home Control System (HCS), we are taking a similar approach with presence in our home. This ensures a 'whole home' approach using all the information available for all of the technologies (and suppliers equipment) in our home.
Our initial focus has been on passive detection of people and this allows us to have occupancy for 'Unknown' people. We can use door contact sensors, PIR sensors, etc. to detect people and their location. This approach allows us to do real things with occupancy immediately and to add and extend as new technologies come along.
Not everyone carries a Smartphone. We have guests and children in our home that have no technology to help identify them. It is essential that we therefore include passive approaches to occupancy detection as well as presence. Our approach is also extensible and future-proof. Apple's iBeacon technology is an example of a relatively new technology that we are currently testing out to extend our presence and location capability.
We also want the house to be able to answer the question, 'Who is at home?' and we expect this to typically be done via our natural language interface. So our solution has to be able to handle events that can identify people as well as those that can't.
Detection & Probability
Much like services with on-line presence, it is going to be very hard to be sure that someone is in the house (c.f. available on-line) with 100% certainty. That '100%' number also infers a guarantee that we can't really deliver on. Equally, we can't provide a guarantee that someone is not in the home because they may have just walked in before they could be detected (though door contact sensors make this extremely unlikely). Because of this we are modelling the probability of an individual person (or an unknown person) being in a particular zone as a probability between 1% and 99%.
We can know when a device associated with a person is connected but, even this is historical data (albeit very recent history) as it is based on when the check was last made. Because of this, we treat all occupancy data as being a historical. We model this as a significance that decays over time, so we store the time of each presence update using Java system time (in milliseconds). The decayed value can thus be easily calculated knowing the start value and the elapsed time since the event.
Presence also has different meaning depending on what sort of sensor or device was used to capture the event. We assign a different rate of decay to each source type and we are refining our model over time, to optimise these decay rates.
If we have to answer a yes or no question as to whether a zone is occupied we interpret a probability of 50% or more as meaning that it is. This is what we call the threshold value.
Ownership of a device enables us to associate its presence to an individual. As an example, we can see when our son's laptop is connected to the home network and we know that because only he uses this device, we can be sure it is him present in our home. There could be situations where he has left it on though but, we have assumed for now that this doesn't happen.
The biggest challenge with this ownership model is that when someone really does leave their 'owned' device on and at home, then they will incorrectly be shown as being at home and the house as being occupied. Because of this we have had be extremely cautious in assigning device ownership and we do not use this approach for example with our children's personal Smartphones.
Timeliness & Latency
Whilst occupancy can be achieved in near real-time, presence is subject to greater time delays and there is some latency in knowing whether a space is occupied or unoccupied by a specific person. We have written an article on smart home latency.
Accuracy is more important than timeliness. Our software models and algorithms are aim to allow us to say with almost 100% certainty whether our home is occupied or not, assuming you give us a few minutes to answer the question. In the real-world of smart home automation this is good enough.
Location is an area where we are currently focussing a lot of our effort. We have realised that mobile devices in the home provide occupancy, presence and proximity data but, we also model and store in-zone location. The objective is to not just know that someone is within a zone but, to be able to pin-point their location within it.
One of the biggest challenges with occupancy and presence is in providing an accurate view of how many people are in the house. Occupancy and presence down to room or zone level goes a long way to answering this question but, it is a lot more complicated than that. As we refine our algorithms through experimentation we have now reached a point where we can do this with a high degree of confidence but, we will continue to optimise them based on observation and learning.
Occupancy, Presence & Proximity Sources
Doors (or more specifically the door contact sensors) are the cheapest (if wired) and best source of occupancy events in the smart home but, they don't allow us to identify people. We have assumed that we can't link a specific person to a particular door with any degree of confidence. The more doors with contact sensors, the more useful this information is. In our current home, every single internal and external door has an invisible (hidden in the door frame or the door itself) contact sensor.
Doors become more important when you model relationships between doors and zones. Our 'wasp in a box' model only works if you do this. Some doors can also provide occupancy information within a zone, e.g. an ensuite bathroom door within a bedroom.
The use of door contact sensors like this assumes that they are actually being opened and closed by people, i.e. not being moved by drafts, etc.
PIR sensors are the second best source of occupancy data in the smart home. They are also the most frequent and we can see over 7000 such events in day (when our house is occupied all day). Because of the sheer volume of PIR events, we have implemented only wired PIR sensors, for best performance and reliability. This eliminates any potential battery issues too.
We have at least one PIR sensor in nearly every room in our house and plan to have all of the rooms covered very soon. Because these events occur so often and we have lots of PIR sensors in our home, we can assign them with a fast decay rate of presence probability.
Pressure mats are another simple source of occupancy data. They can be used on stairs (narrow versions are available for this specific application) but they are particularly effective when used close to external doors. They should not be too close to a door though, or whilst entering the house the door won't be closed before the pressure mat is activated. In this respect the ideal installation for occupancy data is different to the one typically used when these are installed purely for security purposes.
Chair & Bed Pressure Mats
In the telecare and telehealth industry the use of chair and bed sensors to track the movement of the elderly and vulnerable is also becoming more common. This technology provides both occupancy and potentially presence information too.
Bed Occupancy & Presence
We tried standard pressure mats for bed occupancy detection and they simply don't work very well. They require a fairly localised pressure to activate them, something that doesn't happen through a thick mattress.
We subsequently developed our own very low-cost bed occupancy sensor and have found it to be a highly accurate and useful source of both occupancy and presence data. It provides a very clear indication of a bed being occupied and works on both single and double beds. It is also not prone to oscillating output as people move around on the bed either.
We are experimenting with the above sensor on other pieces of furniture and our lounge sofa will soon be a connected device in our smart home.
Our IP cameras have movement detection capability and we also use this to trigger events around our home. We have no immediate plans to use this technology to identify individuals though (i.e. using face detection).
We have built IR remote activity detection as an optional feature in our smart home thermostats This means that when an infra-red remote control is operated, the signals are 'seen' and used to indicate occupancy in that room. This is particularly effective in rooms like our lounge where people sit still for long periods of time and PIR sensors will not see any movement.
Automated locks can often use technologies like RFID to identify people but, they can not be used for presence and occupancy as they are located outside the home. They can be used if events can be linked to provide added context though and they do provide reliable proximity information.
Mains Water Pressure
We have recently started a project to monitor mains water pressure, to see if this can provide presence information.
Our Home Control System (HCS) has audio input and output capability. We do have the ability to detect audio levels and could use this as a means to drive presence. The downside of this approach would be that external noise sources (e.g. planes overhead) might trigger it, so we would ideally need some kind of noise characterisation. This is going to be the focus of a future project.
When someone presses a light switch, this generates an event that indicates occupancy with a very accurate location associated with it. Where we have installed automated lighting in our current home, we are using this occupancy information.
In our next home, every light will be automated and every light switch will be a source of occupancy information.
We have developed our own automated curtains and the manual control switches for these also generate occupancy events.
In a similar manner to light switches, we can track our Z-Wave appliance modules being switched on manually. These events are currently used to provide occupancy information associated with with pin-point location information.
We can see that it could be possible to monitor mains power usage and be able to characterise and identify specific devices switched on in our home. There are some companies that already do this as part of their smart energy monitoring solutions. It would then be possible to associate some of these devices to specific people, e.g. the washing machine to my wife (only joking!).
This technology exists already and we have very powerful demonstrations of it in operation from a couple of suppliers. It is a technology that we are closely tracking.
We have already investigated using a Raspberry Pi to monitor Bluetooth and a Raspberry Pi to monitor Wi-Fi and Ethernet network presence to detect the proximity of individual's device and we are now actively using this information in our smart home. This has proved amazingly accurate in the case of the adults in our home (who always carry a phone with them) but is not a lot of use with our children (who do not).
We have other Wi-Fi network connected devices in our home that can signal the proximity of people such as iPods, tablets, etc. We also have Ethernet connected devices that have know locations and can provide occupancy information. In total, we now have over 90+ IP-networked devices in our home.
The most important events are when a devices has just appeared online (at home) but, in practice Smartphones appear to regularly connect and disconnect from Wi-Fi networks whilst stationary in the home. Our algorithms can handle these small breaks in connectivity but, this has the effect of adding latency to presence results.
We are currently using the presence of our Sony Playstation 4 and PS3 games consoles on our home network to signify occupancy of our Lounge and ensure that lights stay on, etc. when the PS4 is in use.
We also use an Amazon Echo device and each time a query is made, this is used to update house occupancy.
Our Smart Home also spots our Wi-Fi connected TomTom Go 6200 satnav as we pull up onto the drive and uses this to turn on the drive lights.
All rooms have a high precision temperature sensor within them. In theory, we could detect people in rooms through rising temperatures but, in practice there is too much 'noise' and many external factors. Simple things like the opening of doors and windows make this an unrealistic proposition in our view.
Our bathrooms have networked humidity sensors in them, which are used to efficiently control extractor fans. We have noticed good correlation between occupancy of bathrooms and levels of humidity required to activate extractor fans. This is another source of occupancy information that we are looking at.
One easy way to enable presence detection is if you actively tag people. Obviously, this needs their permission and even then it isn't fool-proof. Let's say you add a personal key fob which also features a Immobilise type label in case they get lost. This won't help if your children leave their keys at home, something they are prone to doing. Smart Things (a Kickstarter project) SmartTag uses this approach.
There are various tag technologies with RFID and NFC being common ones.
We are currently looking at Apple iBeacons, which use Bluetooth Low Energy. We have ordered some iBeacons to test within our home. The biggest challenge is going to be support by Smartphone. Only the latest iOS devices support this technology. On it's own, it is unlikely to solve the problem as it requires a bespoke app to be also running on the Smartphone or tablet. It is also not going to work with devices that are left on are also left at home. We do see it augmenting the presence information we are already collecting though.
At CES2015 the Wi-Fi Alliance Wi-Fi Aware which would enable indoor location based services:
We have already modelled zones in our home. This project required us to revisit some of our thinking on zones though, as occupancy and presence needed a more detailed understanding of the relationships between them:
- A 'Global' zone covers our house, garage and garden.
- A 'House' zone covers everything within the footprint of our home.
- A 'Network' zone sits over the top of the 'House' zone but has fuzzy edges. In this respect it is used to model proximity only. It sits within the 'Global' zone.
- Most zones have clearly defined boundaries and one or more entry/exit points (i.e. doors). The entry and exit points can only be modelled if they have a contact sensor on them.
- Some zones (e.g. 'Main Ensuite') can only be reached by passing through another zone (e.g. 'Main Bedroom'). This makes the zone model more complex.
- Some zones (e.g. 'Entrance Hall') have an open boundary with another zone (e.g. 'Landing'). We cannot track movement between these zones.
- Adjoining zones (e.g. archway between 'Lounge' and 'Dining Room') with an open boundary like this could be separated using a beam-break detector. This works much like a door that both opens and closes in one event. This also makes our model more complex but we have also modelled this in case we need it later.
- When one or more doors associated with a zone are opened, we model the concept of a zone being 'Open'. Similarly, when all doors associated with a zone are closed, we model the concept of a zone being 'Closed'. Occupancy and presence events mean different things in these two situations are an essential part of our wasp in a box model.
Zones have an attribute that defines whether they are 'Open' or 'Closed' and our Java class has a functions to both set and query this. There is also an attribute to model whether they are occupied (wasp in the box).
Our zone model also supports the concept of 'nested zones'. This is because we have rooms within rooms. If someone is in the 'Main ensuite Bathroom' then they are also in the 'Main Bedroom' zone and also within the 'Upstairs' zone and thus also within the 'House' zone.
Our design has to model the fact that doors are two-way devices. Opening a door or breaking a beam-break detector has implications on the zones on both sides of it. This is even more complex when the door leads into a zone that has an open boundary with another zone.
Occupancy and presence are also event driven and we have an event type of 'Presence' to support them. As part of this project we have added three new zones: 'Bluetooth', 'LAN' and 'WiFi'. The event object is the source or occupancy and presence events, which equates to the MAC address of a networked device.
We have processes running on a Raspberry Pi, which runs the Linux operating system. This processor also runs our Smart Front Door project. A process monitors our Local Area Network (LAN) and reports back the presence of any devices on it as events. One side effect of this approach is that we have a heartbeat for our Wi-Fi network and can see if it isn't working and thus raise an alert.
Another process monitors Bluetooth, looking for visible devices and reporting their presence (Bluetooth MAC address) as events.
One simple and clever technique we use in our home is the 'wasp in a box' algorithm. If we detect a specific person in the home or an unknown person, we can assume they are still inside our home until one of the external doors (front, back and conservatory doors in our house) is opened. We model this in our presence system and capture and store this information. Once a perimeter door is open, we can not know for sure who has left the house and have to assume that everyone has.
Our testing has shown that this simple algorithm massively improves our occupancy knowledge. It completely solves the main issue with occupancy detection, which is when people go to sleep at night, there is no obvious movement (and therefore no events) to update occupancy.
This algorithm only works if you have contact sensors on all of the perimeter doors. The great thing about it though is that it can be implemented very cheaply, with a single sensor in the home (PIR or door contact sensor) and a sensor on each perimeter door. The more sensors you have though (especially ones just inside the perimeter doors), the better it works. Up to a point obviously!
This same approach can also be applied to an individual room (e.g. a toilet) and used to ensure a convenience light doesn't turn off with someone still in it.
This approach also works with our 'nested zone' model, so that we can use occupancy of one room to infer occupancy of a zone or room that encompasses it.
The Wasp In A Box Algorithm
- Assume there is no wasp in the box.
- Have you heard the wasp buzzing in the box?
YES = There is a wasp in the box until you are told otherwise.
- Have you opened the box since you last heard buzzing?
YES = There is no wasp in the box.
NO = There is still a wasp in the box if you heard one earlier.
- Got to step 2.
Our presence model assumes that we have a number of known devices and that each device has a number of interfaces, each with a unique ID (MAC address), e.g. LAN, Wi-Fi, Bluetooth, etc.
In our home we have approximately 50 such devices and some of them can be considered unique to a person in our household and assigned an owner. Many of them are not significant and are not assigned an owner. Quite a few are significant as sources of occupancy information but are not owned or uniquely associated with a person. These devices are assigned an owner called 'Unknown'.
Zones & Location
Many of these devices are mobile devices such as Smartphones, tablets and laptops. Some have a fixed location that can be used to determine which zone (or room) they are in. Some can even be associated with a very specific location within a zone. We are still trying to work out the best way to model location within a room or zone and will be experimenting with different solutions in the coming weeks. The key thing we want to be able to infer is distance between devices and known objects, i.e. centimetre level proximity.
For now, we have assumed that if a device has a zone or location it is not specific to a given interface. In future we may add a zone and location associated with each device interface though.
One of the key things we want to be able to do is know is when someone has returned to an empty home and how long the house has been empty. This information will be used to make certain announcements.
The initial states of the elements modelled in our home are important. We assume the 'House' zone is a closed zone (i.e. all the external doors are initially shut). Because we have so many sensors around our home providing occupancy data, we don't need to make any initial assumptions about occupancy and presence. Networked devices are picked up almost instantly.
We have implemented a static Java class called 'Presence2' that is our occupancy and presence engine. It can be queried by any element of our Home Control System (HCS) using the public functions (our presence interface). In addition, all of the members of our household are now objects that can be referenced and queried by any other part of our Home Control System (HCS).
Why 'Presence2'? Because this is our second implementation of presence and we are running both versions in parallel, to compare the performance of different algorithms and ensure V2.0 is more efficient, more timely and more accurate than our first version.
Both versions get passed every events that comes into our Home Control System (HCS). In a typical day this means each class is processing approximately 500,000 real-time events. At the weekend, this number is much higher. We have implemented it in this way because we want our presence class to hold all the intelligence and make all the decisions with regards to presence. If we add new event types, devices, sensors, etc. in the future it can be updated to handle them.
This class models presence and occupancy scores and weightings for a given person and zone. It also stores the presence update time. In instance of this class exists (in an array) for every possible combination of person and zones and these are created as needed. This means our presence system learns and expands its knowledge over time.
With 20 zones and 5 people, there would eventually be 100 instances, assuming every person visits every zone and that we can actually track this. In reality the number is much lower. At any time we can request a presence report, which presents a list of all the scores held. This is amazing useful in understanding how well things are working is an essential test tool.
This is a function of the Presence2 class, which uses a complex algorithm to calculate the number of people in our home. We are not completely sure what we will do with this information yet but, so far it has proved very accurate.
Proximity To Presence Conversion
Our implementation and experiments so far have shown that we can use proximity to infer presence with a very high degree of accuracy and this hugely improves the high-level view of who is at home. This is based on the simple premise that if a known person is in proximity of the home for more than a few minutes, then they are probably within it.
The challenge is in defining what period of time must pass before we can do this. The shorter the time period, the more timely the presence information but, there is a higher chance of making the wrong assumption. We are currently capturing real-world data, to determine the optimum time period.
Who Is At Home
Our presence class has a function called WhoIsAtHome(), which returns a human readable string that can be presented via various interfaces, e.g. natural language interface or using text-to-speech voice announcements.
Interfaces & Testing
We have integrated the presence information into our natural language interface, so that it can be exposed via XMPP (and other protocols). Here we are having a Google Hangout with our smart home.
They are some interesting scenarios that we have had to address in implementing this project, which are worth a mention.
Someone Leaves Home
If someone leaves home, the House zone is opened and there will be a limited time period before existing presence scores decay, indicating no one is home. Typically, the front door is shut (closing the House zone) and one of the many sensors pick up movement and flag the House zone as occupied again. This happens most mornings when one of us leaves for work. If no movement was detected, then the house would think it was unoccupied until movement was detected.
The challenge is when the rest of the family is in bed because there are then no other events picked up. Our implementation solves this though, so long as bedroom doors are shut and the bedrooms themselves are closed zones and are flagged as occupied, thus flagging the house as occupied.
This project has shown that things like occupancy and presence need to be tried and tested in a real home to get a full understanding of how well they work and implications and issues. Whilst you can simulate elements of this in a lab, it is very hard to model numerous zones and the relationships between them, whilst collecting high volumes of occupancy and presence data with a real-world distribution.
It also shows the value of a 'whole home' approach to occupancy and presence. Our approach provides complete context and understanding, to get an accurate view of house occupancy and also detailed zone views. Whilst you can buy 'smart' devices that claim to know if you are home or away, we would question whether this is really the right way to get an accurate answer to this question.
Whilst many think presence is the most important aspect of a smart home, we have shown that occupancy, presence, location and proximity are all linked and deserve equal weighting in the smart home. It is the combination of all four concepts that really make a smart home smart.
As with most of our projects, this isn't about the technology and software. None of the stuff here is exposed to people living in our home and none of it needs to be. Accurate and timely presence detection and occupancy knowledge enables a smart home to be much more efficient and much more intelligent in its behaviour. It simply make our smart home more usable and much nicer place to be.
The Learning Smart Home
In undertaking this project, we have collected some really interesting statistics about our home and how we use it. The appearance of devices on networks and the detection of people in rooms has enabled us to build a detailed profile of usage (down to zone/room level) over time. We can see variations throughout each day, a typical working week and also expect to see variations across the seasons in each year. We can see things like who is at home most often and spot patterns in behaviour and usage. These are all valuable inputs that can be used to add learning capability to our smart home and this is now a related project that we have now started.
We have spent a lot of time looking at other sources of occupancy and presence information in the smart home. There are obviously a huge number of theoretical sources but these are some of the more likely ones that we are looking to implement and actually use in the future:
- Raspberry Pi to monitor Bluetooth
- Raspberry Pi to monitor Wi-Fi and Ethernet network presence
- XML configured home automation
In our home we currently use presence and track whether individuals are at home or not. This is done using a hybrid approach and works pretty well. We can use the presence of Smartphone for the adults in our home to do this, as the two of us always carry our iPhones around with us and out of the home. This approach does not work with our children though, as they often leave their Smartphones on and also leave them at home. They are not allowed to have phones at school. So for the children we have used other devices to detect their presence, such as games consoles.
Our home also tracks anonymous users within it using PIR sensors, switch activations, door contact sensors, etc. This means that we can know if someone is at home. We can also track anonymous people in our home down to zone (room) level using a combination of sensors.
We are currently updating our Java zone class to support the presence of multiple, identified individuals in each zone, so that we can start to offer a fully personalised experience throughout our home. This also means modelling the boundary points (doors, archways, etc.) between rooms and the relationships between rooms in our home.
We have re-written and extended our current occupancy and presence software many times, as our thinking has evolved and we have learnt through trying things out in the real-world. We now think we have a model that works reliably and accurately. This is a significant step towards our objective of creating the world's smartest home :-)
One thing we have realised in undertaking this project is that our approach has application in the telecare and telehealth markets. Not only can we accurately monitor occupancy and presence with very low-cost hardware but, we can also monitor lack of activity and detect abnormal patterns of activity and inactivity.
27th October 2013
This is a screen shot showing some of our HCS logs. It shows the study door being opened and the zone then be identified as opened and hence marked as unoccupied. This latter flag really means that we don't know that it is occupied. The next line shows our house now logging that an unknown person is in. The door opening was the trigger event for this and up to this point in time the house thought it was empty. The study door is then closed and the zone is then flagged as closed, since this is the only door into the study. The study PIR is then activated and the study zone is flagged as now being occupied. It will remain in this state until the door is opened. The next two entries show presence changes as my wife and daughter come home with their mobile phones and are detected as being in. The study door is then opened and the study zone is now open and flagged as no longer occupied. The final two events show the PIR being triggered again but because the door is open they update the presence score only.
Some occupancy and presence events from this project are tweeted by our smart home: