analysis

Building a Spatial Interpolation Model for IoT sensors by Usamah Khan

Photo by  oxana v  on  Unsplash

Photo by oxana v on Unsplash

 

Over the past few years I’ve been passionate about working with open IoT sensors and building models to enrich whatever data I find. Like with most things that happen to grab my interest, I spent a lot of time reading about spatial modelling and prediction. This has come in handy for some projects I’ve been a part of like inferring missing data to build virtual sensors to working with wearables to measure air quality through subjective experience.

As the caliber and scope of the projects have grown, I’ve been spending more time looking for solutions that are scalable and offer a degree of interactivity. This is when I stumbled upon ‘Kriging’.

Before all this, just a bit of background. I started looking into this problem during our second phase of WearAQ that you can learn more about over on the blog. To recap, we hosted workshops and pollution exploration walks with students in London and asked them to measure air quality through their own perceptions using low tech wearable devices. We contrasted this data with predictions generated from regression experiments we conducted on air quality data available close by. The idea being, could we use the near by data as a ‘ground truth’ approximation of the air quality at certain time.

However as we grew the experiment, we wanted to make some better comparisons, so we looked at how we could go about predicting air quality at any point, any time between sensors and not just a nearby approximation.

Now enter Kriging, a spatial prediction method that takes a regression function and, in conjunction with the residual (the difference between the estimated and predicted value) and iterates in an effort to minimize the error to give a better fit to the data. In 1-dimension this can be shown below.

 
By Tomislav Hengl - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=19985745

By Tomislav Hengl - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=19985745

 

If you’re interested in the math I recommend Engineering Design via Surrogate Modelling by Forrester, Sóbester and Keane and Capaulson's great library for Kriging in Python.

We wanted to apply this to a 2-D problem. In this case, we were interested in air quality in the borough of Tower Hamlets.

 
Map of Sensors.png

There are 7 sensors in Tower Hamlets that we were able to pull data from. We were lucky to be able to find data for 5 pollutants ranging back 10 years from London Air Quality Network. That helped us get a desired coverage over every hour of every day in a year and wherever there was missing data, we used an algorithm to impute.

But that’s for another blog post. Just know that by doing this, we now had the ability to query a value of one of five pollutants related to air quality at any hour of any hour, day and month of the year.

 
 

With the ability to query a prediction at any time we wanted, we could now move on to building out the model in Python. As we wanted to build an interactive and responsive web app, we chose to develop in Python as we could implement our model in Flask as a framework. To help with that, pyKriging is a great library I found that takes care of most of the complexities of the algorithms and has useful functions that make it easy to handle and train your data.

 
 

The documentation is pretty straightforward and there are a couple of examples on the repo to take a look at, but quite simply all you need to do is make two arrays - one for your location (lon, lat) and one for your pollution values. Simply pass those through the model and voila.

X /*Initialize an array for lon, lat*/
y /*Initialize a vector of output values*/

optimizer = 'pso'

k = kriging(X, y)
k.train(optimizer = optimizer)
k.plot()
 

Prediction is only one aspect of the model however and is the nature of all prediction, it’s not always 100% accurate. This is where Kriging comes in handy. To determine how confident we are, we also took a look at the Mean Squared Error (MSE). Simply put the the MSE is a measure of the quality of the estimator. It is a non-negative number where the closer the values are to zero, the better. Plotting this side by side to our prediction we can get a good idea of the quality of the model

 
An example output over the experiment zone

An example output over the experiment zone

 

There are three parts to the chart. The graph on the top left shows the sensors with a heat map of the Air Quality - in this particular example, the NO levels displayed from 2-56 ug/m3. On the top right, it shows a similar graph, however, instead of the showing the value for AQ, it shows, at each point, the estimated error. As we can see, around the sensors, the error is 0 or close to. This makes sense since we can assume that close to the sensor, the AQ should be more or less similar. Lastly the bottom chart shows the NO levels, but modelled in 3 dimensions as a topographical map.

The yellow shaded areas on the graph show zones of high error where our confidence in our prediction is low. However, if we were to take some measurements in those areas, we might be able to improve the overall accuracy of the model.

This error is the model error. There may be a higher or lower error based on environmental factors in the area between the sensors. Like we mentioned, air quality can change from street to street, but by using this model we can at least provide a baseline for our analysis to a relative accuracy.

One last note to make is that if the data varies wildly between sensors or not enough, the model has a hard time making accurate predictions the further you get from each point. To deal with this after some experimentation, we settled on the idea of adding 2 to 3 ‘dummy’ locations a short distance away from our known points to force the data to converge at that point. Testing this on good datasets shows that it had a negligible impact on the prediction but made a big difference with data that the algorithm had difficulty in modelling.

This idea of looking at areas of low confidence was the basis of the idea behind building an engine that could recommend new locations to take measurements. Determining areas of high error, we would then be able to crowdsource a better understanding of air quality.

We wanted to use the participants to add a layer to our model and give their perception of the data certain points around the workshop area. To do this we use a process known as ‘infill’. Based on a choice of one of two criteria the infill model chooses where to take new measurements. The two criteria are “expected improvement’ to the model and ‘maximum mean squared error’ improvement. These are just two ways of looking at the area and asking whether to improve the whole model, or pick in areas with the highest error.

We use this method to pick locations for our participants to walk to. As we are trying to model Air Quality and have the workshop participants have a voice in their community the more measurements we have and the better the location of the measurement, the better the overall coverage would be.

So to infill for a workshop, we can “zoom” into a location and see where best to take measurements. To zoom in, we query 3-4 locations around the workshop area, extract the prediction and push it through the kriging model again to estimate the spread of air quality in that zone. From there, we can ask the model to infill new locations for us.

 
Workshop_1_init.png
Workshop_1_loc.png
 

Above shows an example of zooming and infilling. The chart on the right shows the infill locations and how they would affect the model if the prediction there was to be taken as truth and modelled. The locations may not look like they are where the highest error is in the chart on the left, however this is due to the fact that the model works by iterating - adding a location, re running the model and then picking a new point. Through this whole process the error and prediction is tightened to give a more accurate representation of the area.

This is how we were able to build this second phase of the project using spatial interpolation to power an interactive platform for crowd-sourcing air quality. For any more information, feel free to reach out or check out our full write up on the Umbrellium blog!

 

WearAQ - Measuring Air Quality Through Our Subjective Experience by Usamah Khan

Photo by    veeterzy

Photo by veeterzy

 

Last year, I worked on a project analyzing data from Thingful’s repositories in an effort to predict “missing data”. Over the last few months I got the chance to reconnect with Umbrellium to tackle a new challenge in IoT. This project combined some of the same thought processes and ideas that I explored with Thingful, however, this time we were specifically tackling Air Quality and incorporating the use of wearables and our own data sources.

Over the past few months, I worked on an experimental project, WearAQ alongside architect/designer/maker/coder Ling Tan. We aimed to answer certain questions surrounding air quality, namely, what can we do make us feel more empowered when it comes to dealing with pollution? Sometimes it can feel like we can't change anything and if thats the case how can we make good decisions to minimize the effect of pollution on us? We wanted to see how can we give people the tools to at least understand air quality so they can make these decisions.

We designed an experiment to look at how school children could make sense of these complex issues and considered how people might combine their subjective perception and intuition with wearable technologies and machine learning algorithms to investigate air quality issues. We hosted workshops and exploration walks with students at the Marner Primary School in Tower Hamlets, London UK. We supervised them as they went out into their surrounding neighbourhoods and measured air quality through their own perceptions and recorded their subjective experience using wearable devices that catalogued their gestures.

We used the perceptual data as the basis of two experiments:

  • Experiment 1: To contrast and correlate the data with measurements from mobile pollution monitoring equipment and photos taken by students on their environment during the walk

  • Experiment 2: Use the perceptual readings as ground truth data and compare them with predicted perception data created from conducting machine learning experiments using existing public data sources

In our findings, we identified that there was indeed a correlation between air pollution and the students' perception of air quality. Additionally, this provided an exciting experience for students to engage with wearable technology. They were able to contribute data about air quality and while going through the process, explore other ways of understanding air quality. Through the experiment, we were able to obtain an accurate 8/8 correct predictions on students perceptual data based on our machine learning model. We were also able to obtain a 6/8 accuracy when we compared recorded gesture data with ground truth data from the mobile pollution monitoring equipment. We recognised that there was a lack of data, however as a prototype we were confident in it's implication that there is a correlation between perceptual data to actual air quality measurements.

 

What can we do about pollution? http://umbrellium.co.uk/initiatives/wearaq/

 

There were many stages to the project. Ling and I worked in tandem for some, but for other stages, based on our skills, we worked largely independently of each other. We put together a blog series documenting the whole project over on the Umbrellium blog, but in this post I wanted to share a quick overview of the project and dive a little deeper into some of the work I was leading up.

 

Designing the technology

 

This step of the experiment was one that was entirely covered by Ling. She shared a detailed post on the creation and the technology behind the wearables. But I wanted to briefly describe it here because it was pretty incredible work. In her words:

10 wearable devices were designed and manufactured to record students’ perceptual experience of situated AQ through hand gestures. Each wearable device consisted of an arduino compatible microcontroller on an adjustable wristband and the sensor was attached onto a glove that read and detect hand gestures. Each device was connected via bluetooth to an android phone and each phone had a prepaid Vodafone SIM card to enable the phone to send data to the web during the walking tour. The device was designed to detect three types of gestures (arm up parallel to body, pinch nose, and arm perpendicular to body). The three gestures correlate to the quality of air that the students will record during the walking tour.

 
Close up of the final wearable device fitted on a hand and the 10 wearable devices, phones and airbeam device

Close up of the final wearable device fitted on a hand and the 10 wearable devices, phones and airbeam device

Structuring The Experiment

 

Once we had the tools, structuring the experiment was one a task that we had to work on. Ling had experience working with young students form another project, Cinder, and she knew some of the challenges we'd expect with students i.e short attention spans, peer influence veering them off track from their experience and grasping and understanding of the issue.

Outside of these challenges, we had to structure a set of defined tasks for the students to perform to at have control over some aspects of the experiment.

 

1 - Defining the route 

 

We wanted the students to walk around and record their perception of air quality. This we expected to range between "bad" and "good". However, we couldn't very well ask children to walk out to the most polluted areas around the school just to get that information. We needed to show the organizers at the school that we had at least considered their safety in some ways. 

Hence we used statistical modeling to identify 8 locations within 1km distance from Marner Primary School that were the least polluted. The students were not given any information beforehand about the locations as this was to ensure that their subjective perception of air quality at the locations remained unbiased.

 
Visualization of the data we pulled in determining a walking path for the students

Visualization of the data we pulled in determining a walking path for the students

 

Looking at a few sources, namely open data sensors in Tower Hamlets from the London Air Quality Network and open traffic data, and plotting a test walk to see how accurate our readings would be, we determined a walking path away from the highest sources of pollution and locations that were determined "least" polluted.

 

2 - Recording the students' subjective experience

 

Students were taught the various hand gestures that relates to different air quality levels:

  • Gesture 1 = arm all the way up parallel to body = if you think the air is good

  • Gesture 2 = arm mid air perpendicular to body = if you think the air is ok, or if you have no opinion

  • Gesture 3 = pinch nose = if you think the air is bad

 
Students recording their experience based on the gestures outlined

Students recording their experience based on the gestures outlined

3 - Recording the environment 

 

We also asked the students to record any visual cues related to their experience and so we asked them to take photos at each location so we could examine these qualitative factors.

Now that we planned everything out, it was time to perform the the workshops. We were lucky enough to have the Marner Primary School invite us for two sessions since this proved useful in understanding our equipment and iterating on the design to make sure we could achieve the desired results.

 

Results

 

There were two parts to analyzing the results of the experiment. The first was an analysis of the qualitative data such as the gestures and their relation to the photos. Ling covered this in the second part of the blog series. My job was to look at the data and see if we could build a model to validate our hypothesis.

So to reiterate the problem that Ling outlined in her posts, we were looking to find the answer to a simple question -  “Can we measure air quality through our subjective experience?”.

We had the students record their perception and we took measurements of PM 2.5 from a low resolution sensor that we carried while we walked. Then we had one sensor at the Marner primary school that was highly calibrated for a range of variables (NO, NOX, NO2, O3, PM2.5, PM10) that we used as our ground truth.

Our first challenge was how to incorporate the ground truth data. The AirSensa device’s data was specific to one location, while as the data collected by the students was over variable locations. When we consulted with air quality experts in the planning stage of the project, the first point they made was that measuring air quality reliably would be very hard as even being on different sides of the street has an effect. So even though students did not walk very far from the school and we knew that air quality is location dependent, we took a look at the variability of air quality over time to see what trend, if any, existed between the AQ readings from the school’s fixed sensor to the low calibrated hand sensor.

 
 

We observed a slightly downward trend being repeated. Both data sets were collected at large irregular intervals accounting for sharp dips and peaks. Not robust enough to warrant a direct comparison, for the sake of the experiment we went ahead to make our first assumption: AQ over time can indicate AQ over a short distance.

 

Building the Model

 

The first thing we needed to do was learn more about prediction techniques for air quality. We spoke with experts in the field and researched modelling and prediction techniques. Additionally, since we were taking a new approach incorporating data from IoT sensors of varying quality and calibration we needed to devise a method that could handle mixed types of data.

The AirSensa device reports data at a resolution of 10 minute measurement intervals. The data we retrieved from the wearable Airbeam device was a little more sparse in it’s recording. To deal with this, we had to fill in the data to match the resolution between the two set of readings. For the purpose of a prototype, we wanted to explore how well a model could predict these values despite the varying differences in data quality between the two.

 
Raw Dataset from the Marner Primary School AirSensa device

Raw Dataset from the Marner Primary School AirSensa device

 

We also had to incorporate the results of the gestures. Due the nature of subjective measurements, we found the students were not 100% aligned in their perception of air quality. So to account for this we had combined the data as an average to give us a continuous range, between 1-3 (each number referring to a specific perception) to get an idea of air quality at each location and between each location.

So to put this all together, initially, we considered simply linearly interpolating between the data as a prediction model. However, we also used the "random forests" machine learning technique to impute some of the missing data.

 
Table showing interpolated values and imputed values through machine learning model

Table showing interpolated values and imputed values through machine learning model

Now we had a “full” data set to use, mimicking the ideal scenario where perception and air quality measurements were being captured every minute. The experiment was designed to see if we could use the perception data as a substitute to measure AQ. So the next step was to build out our model to predict air quality and see if we could line up the model to predict, with a certain accuracy, the perception data. We used two models, a random forest and a neural network.

* Note - The values in bold display the raw values. Note that this table shows both imputed and interpolated gesture results but only interpolated pollutant values

 
 

Random forests are an extension of tree algorithms. Tree algorithms work by taking into account the likelihood of an output occurring based on a range of values of the predictors. A random forest simulates a number of these tree models and takes an average likelihood so as to eliminate the chances of an over or under estimating. Temperature is typically measured in whole steps i.e 16°, 17°, 18° and tree algorithms are more suitable to predict discrete outputs. Applying this method to the problem at hand, this meant that for air quality on a continuous scale there were too many possible outputs that rendered this approach relatively ineffective unless we isolated discrete values or buckets.

We also researched the possibility of using a neural network. Simply put, neural networks take a set of predictors, model them to see the relative effect of each on the output and then combine variables together to get another layer of predictors. These are also used more often for discrete value prediction, however can be adjusted to fit a continuous scale output. When we tested a model on a years worth of sample of data from an air quality sensor it responded with a high accuracy - between 90% and 100% accuracy dependent on size of the training sets. This results of the model fit the prediction of pollutants best and as such we used it for the experiment.

We applied the neural network over the data by splitting a dataset of 30 observations and 8 values into divisions of 70% and 30% to train and test our models respectively. We did this for both the interpolated and imputed datasets.

The results displayed below are in the form of a confusion matrix. This is a method of showing the effectiveness of a model by building a matrix by which the vertical axis shows what the expected output was to be and the horizontal shows the predicted in order. A value of 1 is given per column to indicate the intersection between the prediction and actual result and 0 otherwise. An ideal result is a matrix with a leading diagonal. This works best with factor or classes of variables, but since we were dealing with such a small set of outputs and rounding off our outputs, a leading line of 1s would be the ideal.

From our results we found the following.

 
Results of both models

Results of both models

Comparing the two models side by side, we saw that the model that was applied to the imputed values gave us a lower accuracy. With a test set of this size we can’t conclusively say that one is better than the other due to a number of variables. Firstly comparing imputed to the interpolated values, we had a different spread of data. This could be due to the fact that the imputation model was simple and not very well optimized.

 
 

Also, due to the complexity and small sample size of the of the data we may not have had enough data to go on to build an accurate model. Additionally, perhaps the imputed model returned a more accurate or “realistic” sense of perception data, and hence the lack of accuracy in the model is most likely what to expect if we were to pull from a complete actual dataset. 

We did, however, manage to build a perfect model, under these specific conditions, for the interpolated data. As a result, we took this to validate the students’ perception of air quality by correctly predicting an assumption of their subjective opinion. We were careful about being aware of overfitting the model - i.e making it very specific to the data at hand. However, for the purpose of building a prototype we concluded this would give a good idea of how we expect the model to run.

So as a check of our hypothesis, we were able to build a model (taking into account certain assumptions) that validates with some accuracy that a person’s perception of air quality is correct.

Moving on, we wanted to check the other way i.e can we use perception data to inform us of air quality. Due to time and the constraints of limited data, we went about this by comparing the gesture data to the variability of a pollutant to see if by using a simple line of best fit we could obtain a workable result. Plotting groundtruth gestures against three pollutants we obtained the following results:

 
 

These results were clear in that location plays an important role in perception. Since we arranged the values for the pollutants in ascending order we would expect the graph to show an upward trend. However, this was not the case indicating that in fact, there was something other than just the air quality at the Marner sensor affecting the students’ perception.

All three graphs shared the same trend and we were curious to see if that was to be expected. Analyzing a data set of approx 250,000 observations, we found with the exception of O3, pollutants are positively correlated with one another. This verified our result indicating that the data was indeed correct and location and other factors were having an effect on AQ perception.

 
Pairwise correlation results of different pollutants

Pairwise correlation results of different pollutants

 

So, ideally for the the experiment to work effectively we would need a steady stream of data to feed into the model at each location of interest.

 

Findings and Suggestions for Improvement

 

Using a machine learning model we were able to build an accurate model (8/8 correct predictions) to predict students perception data. This validated a person’s perception of air quality by correctly predicting our assumption of their subjective opinion. When using perception data to backtrack and predict air quality we verified that location plays an important role in air quality and perception data. Values that should have been increasing experienced non-linear trend indicating the fact that something other than just air quality was affecting the students’ perception.

To address these issues moving forward, we’d need to take into account other environmental factors such as traffic, wind and incorporate the visual cues that the students’ recorded. We initially wanted to use sensors such as those connected over IoT networks as part of the machine learning model. This however proved difficult due to location and time of last refresh or update of the sensor. As IoT repositories such as Thingful grow we would be able to add more data to the models.

 

Moving Forward

 

We were happy to build out a workable framework for a prototype to tackle this issue. However, we were dealing with a static dataset and accounting for a lot of the analysis after the fact. The real excitement will come from being able to build out something a little more dynamic that people can act upon. Learning how people will choose to react to this information will be the next point of interest. We're looking to continue this so follow this space and hopefully we'll have something more to share in a little bit.

 

Thanks and Acknowledgements 

 

There were a lot of people who helped me with this project and I wanted to give them a shout out here. Firstly thanks to LingUsman and everyone else at Umbrellium for all their help with the project and for reaching out to talk about making something awesome. Secondly, thanks to Organicity for funding the project and also giving us guidance throughout the whole process. Finally thanks to all the people who lent their advice and took the time to explain some pretty complex issues to me when I knew very little.

I did a lot of my research and found most of my information from a mix of open data sources such as the LAQN and HERE. Please, feel free to get in touch if you have questions regarding any aspect of the project! I'd love to take a moment and chat.

 

How well are we working out? by Usamah Khan

The Human Activity Recognition Project (HAR) put together a study placing accelerometers on 6 participants. They were asked to lift the weights correctly and incorrectly and the goal was to examine how well an exercise was being performed. This project creates and contrasts Machine Learning techniques in R to determine the answers.

Read More