We could store all the data generated by the HCS in a database but, this would be a bit over the top for this purpose. It would also make the data harder to access and extract by both humans and our software. To keep things nice and simple, our Home Control System (HCS) logs debugging, sensor and event data to plain text files. We generate one file per day and store each one in a folder for the month and then this sits in a folder for each year and month. E.g.
Typically, each daily log file is can be between 30Kb and 80Kb. Writing so much data and so frequently to hard disk is not good from a longevity and power consumption perspective, so the data is stored on a USB flash drive and backed up to hard disk once a day.
At the end of each day the daily file is copied across onto the hard disk, within a Dropbox synchronisation folder, so that it is backed up into the cloud. This means that we can then remotely access these logs from mobile devices that run the Dropbox app. Our natural language text interface also accepts commands to immediately synchronise this file into the cloud and to also send the latest version via email. This is very useful to see what has recently happened and to diagnose things.
We now have other processors in our home, running various processes (such as our smart front door) and these generate logs in a similar format.
Log entries are in the form:
2011/12/31 18:45:56 E Description message text
After the time and data a single character is used to identify the type of log entry. The characters currently used are:
- Process or thread startup and shutdown.
- Initialisation and initial values on process startup.
- Alarm from HCS or a sensor.
- Error, e.g. Failed to control a device.
- Warning, e.g. home network speed is low.
- Battery level indication or change.
- Camera event, e.g. still image or video captured.
- User command received, e.g. alarm on or off.
- Debugging information.
- Network event or change.
- Changed output status requested.
- Presence event or change.
- Reported data (performance, monitoring, etc.)
- Scene request.
- Controller target or requested value.
- Sensor value change, e.g. room temperature, wind direction, wind speed, etc.
- Environmental change, e.g. sunset, sunrise, dusk, dawn, mains power failure, Internet connection failure, etc.
- Device confirming changed state.
- Internet activity by our smart home (XMPP, email, FTP, Twitter, etc.)
Start & End Of Day
The logs are processed to data visualisation. To simplify the plotting of daily data, the HCS automatically adds values into the log file at the start and end each day. This means that the data in each log file nicely spans the complete day when plotted out. This is then also true for logs files aggregated to provide a monthly views.
The Dallas DS1920 temperature sensors on our 1-Wire are set to 0.5°C resolution. This approach was used as we continually poll the sensors, to detect changes in temperature. Initially, we logged every change detected but this generated a lot of log data. The main culprit behind all this data was hysteresis, because the reported temperature was seen to oscillate between two values as it approached a transition point. We have resolved the hysteresis problem by using a software algorithm that maintains resolution, removes hysteresis and allows detection of rapid temperature changes.
The logs are essential for debugging and experimenting. Changes made to the Home Control System (HCS) code may have impact over long periods of time, such as room temperature controllers. The logging of the data enables us to form a view of how well the changes work.
Usage & Analysis
We log a lot of data in each day and it provides real insight into how things are working in our home and how and when things get used. There maybe more data than we know what to do with but, we are continually writing tools to process and analyse it and also have a project to visualise the log data.
The data logged is incredibly sensitive and we certainly would not entrust this data to a third party. It has detailed occupancy and presence data, which would allow someone to easily predict when out home is occupied and unoccupied. It contains data about people arriving and leaving our home. It contains device data that would allow devices to be tracked and associated with family members.
In 2010 we generated 3.7MB of log data. In 2011 we generated 10.5MB. In 2012 we generated 29.7MB. In 2013 we generated 35.6MB.
There is obviously a lot of sensitive data in our HCS logs and a huge amount that can be inferred from them. For this reason we don't allow any third party access to them. This edited sample log shows the format and highlights that fact that they are clear and readable by humans. They have to be because these logs provide the best insight into our smart home and how it is working.