Related Work

Related Work

General Research

D. Sachs (2010). Sensor Fusion on Android Devices: A Revolution in Motion Processing. In Google Tech Talk. Gyroscopes, accelerometers, and compasses are increasingly prevalent in mainstream consumer electronics. Applications of these sensors include user interface, augmented reality, gaming, image stabilization, and navigation. This talk will demonstrates how all three sensor types work separately and in conjunction on a modified Android handset running a modified sensor API, then explains how algorithms are used to enable a multitude of applications.

E. Miluzzo, N. Lane, K. Fodor, R. Peterson, H. Lu, M. Musolesi, S. Eisenman, X. Zheng and A. Campbell (2008). Sensing Meets Mobile Social Networks: The Design, Implementation and Evaluation of the CenceMe Application. In The 6th ACM Conference on Embedded Networked Sensor Systems, 337-350. CenceMe is an application that combines sensor enabled-mobile phones with social networks. The intention is to be able to automatically identify a user's actions using accelerometer, GPS, Bluetooth, and audio sensor data. The information is then broadcast through the user’s various social media channels.

M. Weiser (1991). The Computer for the 21st Century, Scientific American, 94-10. Weiser forecasts a future that has embraced ubiquitous computing. Ubiquitous computing refers to the complete integration of computers into everyday life. Weiser discusses the lifestyle changes that would occur in the workplace and at home. His predictions, made 20 years ago, have for the most part been realized, with only a few exceptions.


Mauro Conti, Irina Zachia-Zlatea, and Bruno Crispo, Mind How You Answer Me! (Transparently Authenticating the User of a Smartphone when Answering or Placing a Call). In Proceedings of the Sixth ACM Symposium on Information, Computer and Communications Security (ACM SIGSAC ASIACCS 2011), Hong Kong, March 22-24, 2011.

J. Frank, S. Mannor, and D. Precup, Activity and Gait Recognition with Time-Delay Embeddings. In Proceedings of the 24th AAAI Conference on Artificial Intelligence, Atlanta, GA, 2010. There is a related clip on the Discovery channel (the clip starts midway through the segment).

R. Srinivasan, C. Chen, and D. Cook (2010). Activity Recognition using Actigraph Sensor. In Proceedings of the 4th International workshop on Knowledge Discovery from Sensor Data, 2010.

D. Gafurov and E. Snekkenes (2009). Gait Recognition Using Wearable Motion Recording Sensors. In EURASIP Journal on Advances in Signal Processing, 2009: Article ID 415817. Researchers analyzed data collected from motion recording sensors with tri-axial accelerometers placed on the foot, hip, pocket, and arm (in separate experiments). They examined the best performances of recognition methods based on the motion of these different body parts, as well as how robust gait-based authentication is under three attack scenarios and what attributes contribute to the uniqueness of human gait.

S. Zahid, M. Shazhad, S.A. Khayem, M. Farooq (2009). Keystroke-based User Identification on Smart Phones. In Proceeding RAID '09 Proceedings of the 12th International Symposium on Recent Advances in Intrusion Detection, Springer-Verlag, 2009. Researchers analyze keystroke data of smart phone users and produce a form of user identification. Six distinguishing keystroke features are used as the basis to identify the 25 smart phone users who participated in the study. A fuzzy classifier was used to cluster and classify the keystroke features. Particle Swarm Optimizer (PSO) and Genetic Algorithm (GA) were used as front-end fuzzy classifier optimizer and as a back-end optimizer respectively. A PIN verification mode using keystroke dynamics was also provided in order to guarantee information security on the smart phones.

D. Gafurov (2007). A Survey of Biometric Gait Recognition: Approaches, Security, and Challenges. In Annual Norwegian Computer Science Conference, November 19-21, 2007. Gafurov presents an overview of recent biometric research focusing on gait recognition. He first describes basic evaluation metrics in biometrics, including false accept rates, false reject rates, equal error rates, DET curves, and CMC curves. He identifies three types of gait recognition--machine vision based, floor sensor based, and wearable sensor based--and describes various challenges that the wearable sensor based-approach encounters.

D. Gafurov, K. Helkala and T. Sondrol (2006). Biometric Gait Authentication Using Accelerometer Sensor. In Journal of Computers, 1(7):51-59. This paper examines how gait patterns can be used as an unobtrusive means to authenticate the identity of the user. Gait patterns were recorded by attaching an accelerometer to the right lower leg of the user. Histogram similarity and cycle length were used as part of the authentication procedure, which was tested on a population of 21 participants.

J. Mantyjarvi, M. Lindholdm,et. al. (2005). Identifying Users of Portable Devices from Gait Pattern with Accelerometers. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP '05). The goal of this work was to identify users using data from accelerometers in portable devices that users normally carry. Data was collected from 36 subjects who walked at fast, normal, and slow walking speeds on two separate days. The accelerometer device was placed on their belts, at the middle of their waistline in back. Correlation, frequency domain, and data distribution statistics were used to identify users.

Activity Monitoring

J. Hicks, N. Ramanathan, et. al. (2010). AndWellness: An Open Mobile System for Activity and Experience Sampling. In Poster Session 2010 mHealth Summit.

J. Yang (2009). Toward physical activity diary: motion recognition using simple acceleration features with mobile phones. In ICME '09 Proceedings of the 1st International Workshop on Interactive Multimedia for Consumer Electronics.

T. Brezmes, J.L. Gorricho and J. Cotrina (2009). Activity Recognition from Accelerometer Data on Mobile Phones. In IWANN '09: Proceedings of the 10th International Work-Conference on Artificial Neural Networks, 796-799. Researchers implemented a real time system for classifying six basic daily activities using a mobile phone containing an accelerometer. No server processing data was involved. The K-nearest neighbors algorithm was used, with the intent that users could train the device to detect their motions for whatever location the particular user normally carries his or her phone.

N. Gyorbiro, A. Fabian and G. Homanyi (2008). An Activity Recognition System for Mobile Phones. In Mobile Networks and Applications , 14:82-91. Researchers used feed-forward backpropogation neural networks to distinguish between six different motion patterns using data collected from three MotionBands attached to the dominant wrist, hip, and ankle of each subject. Each MotionBand contained a tri-axial accelerometer, magnetometer, and gyroscope. A smartphone collected data from the MotionBand sensors.

N. Krishnan, D. Colbry, et. al. (2008). Real Time Human Activity Recognition Using Tri-Axial Accelerometers. In Sensors Signals and Information Processing Workshop. Researchers designed a real time system for identifying five lower body activities with data from 3 subjects collected from tri-axial accelerometers. Accelerometers were placed on the right ankle and the left thigh. Researchers extracted statistical and spectral features from the data, including mean, variance, energy, spectral entropy, and correlation between the data of all axes, and the AdaBoost algorithm built on decision stump for classification was trained with three-fold cross validation, and, in addition, probability of classifications were calculated.

N. Krishnan and S. Panchanathan (2008). ?Analysis of Low Resolution Accelerometer Data for Continuous Human Activity Recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2008), 3337-3340. Researchers evaluated the performance of different classifiers on a subset of the accelerometer data used in the Bao and Intille experiment. Classifiers included AdaBoost, SVM, and Regularized Logistic Regression (RLogReg). Ten random subjects and seven lower body activities were examined. The continuous acceleration stream was divided into fixed length frames, and each frame was classified. Statistical features including mean, variance, correlation between all the axis of the accelerometers and spectral features including energy and entropy were calculated.

Y. Cho, Y. Nam, et. al. (2008). SmartBuckle: Human Activity Recognition using a 3-axis Accelerometer and a Wearable Camera. In HealthNet, ?08. Researchers used the SmartBuckle device for medical monitoring to recognize each of 9 different activities. The device contained a tri-axial accelerometer as well as an image sensor. Correlation between axes and the magnitude of the FFT were used as features for the accelerometer data.

E.M. Tapia, S.S. Intille, et al. (2007). Real-Time Recognition of Physical Activities and Their Intensities Using Wireless Accelerometers and a Heart Rate Monitor . In Proceedings of the 2007 11th IEEE International Symposium on Wearable Computers, 1-4. Researchers used five triaxial accelerometers to collect data in real-time from 21 users while the users performed thirty different gymnasium activities. Several of these "activities" involved performing the same activity at different levels of intensity. Using C4.5 DT and the Naive Bayes classifiers in WEKA, researchers found that they could achieve high accuracies for activity recognition for subject dependent analysis but much lower accuracies for subject independent analysis. Adding in a heart rate monitor only slightly improved the results, and combining activities that were the same activity being performed at different intensities also improved accuracies slightly more.

U. Maurer, A. Smailagic, et. al. (2006). Activity Recognition and Monitoring Using Multiple Sensors on Different Body Positions. In IEEE Proceedings of the International Workshop on Wearable and Implantable Body Sensor Networks,3(5):113-116. Researchers sought to identify users? activities in real time using ?eWatch? sensors placed on the belt, shirt pocket, trouser pocket, backpack, and necklace. Features from the accelerometer axes, the light sensor, and a combined value of the accelerometer signals were calculated. The Correlation-based Feature Selection (CFS) method from WEKA was used to find feature sets that were highly correlated with a particular class but uncorrelated with each other.

N. Ravi, N. Dandekar, et. al. (2005). Activity Recognition from Accelerometer Data. In Proceedings of the Seventeenth Conference on Innovative Applications of Artificial Intelligence. Researchers collected data from a triaxial accelerometer worn at subjects' waists. Mean, standard deviation, energy, and correlation features were extracted from the data. In addition to analyzing the performance of base-level classifiers like decision tables, decision trees, K-nearest neighbors, SVM, and Naive Bayes, meta-level classifiers such as boosting, bagging, plurality voting, stacking using ODTs, and stacking using MDTs were applied to classify windows as one of eight daily activities.

L. Bao and S. Intille (2004). Activity Recognition from User-Annotated Acceleration Data . In PERVASIVE, LNCS 3001, 1?17. Five biaxial accelerometers placed on the right hip, dominant wrist, non-dominant upper arm, dominant ankle, and non-dominant thigh were used to collect data from 20 subjects. Twenty daily activities were considered. Mean energy, frequency-domain entropy, and correlation of acceleration data were calculated, and C4.5, instance-based learning, decision tables, and Naive Bayes classifiers in WEKA were tested using these features.

M. Mathie, B. Celler et. al. (2004). Classification of Basic Daily Movements Using a Triaxial Accelerometer. In Medical & Biological Engineering & Computing, 42:679-687. Researchers developed their own binary tree for classification of basic daily activities and created algorithms to describe each of these activities. Using this model, they tested data collected from 26 subjects using a triaxial accelerometer attached at the subjects? waists. Of particular interest in this study was the detection of falls.

Gait Studies

J. Frank, S. Mannor, and D. Precup, Activity and Gait Recognition with Time-Delay Embeddings. In Proceedings of the 24th AAAI Conference on Artificial Intelligence, Atlanta, GA, 2010. There is a related clip on the Discovery channel (the clip starts midway through the segment).

Verghese J, Lipton RB, Katz MJ, Hall CB, Kuslansky G, Buschke H. Abnormality of Gait As A Predictor Of Non-Alzheimer Dementia. New England Journal of Medicine 2002; 347: 1760-1767.

Holtzer R, Verghese J, Xue X, Lipton R. Cognitive processes related to gait velocity: Results from the Einstein Aging Study. Neuropsychology 2006; 20(2):215-23.

Visitor Studies

Bonnie M. Perdue, Tara S. Stoinski, Terry L. Maple. Using Technology to Educate Zoo Visitors About Conservation. Visitor Studies Vol. 15, Iss. 1, 2012

Bonnie Pitman. Dallas Museum Announces Results of Groundbreaking Visitor Study.

Stephen Bitgood. An Attention-Value Model of Museum Visitors. Center for the Advancement of Informal Science Education. (2010).

Geo-spatial Data Mining (of GPS)

G. Agamennoni, J. Nieto, E. Nebot (2008). Mining GPS Data for Extracting Significant Place. In Proceedings of the 2008 IEEE International Conference on Robotics and Automation, 2008. Researchers present a fast and robust algorithm for extracting significant places from a set of raw GPS data points. The research took place in Australia at a mining field for the purpose of improving mine safety by exploiting the large amounts of raw GPS data points. Using the hauling vehicles, data was gathered and collected using a scoring algorithm to determine the places of significance.

D. Luper, D. Cameron, J.A. Miller, H. Arabnia (2007). Spatial and Temporal Target Association through Semantic Analysis and GPS Data Mining. In the 2007 World Congress in Computer Science, Computer Engineering, & Applied Computing (IKE '07), Las Vegas, USA, June 25-28 2007. Researchers analyzed temporal and spatial interactions in an Association Network that integrated GPS, RDF Metadata, and Data Mining. The experiment generated random data for 133 "targets," 33 of which were "terrorists" and the rest were control units. Data was recorded for 60 days at 10 second intervals and kept the generation within Athens, GA. They implemented Data Mining tools such as Binning and Association Rules to find interesting places and "targets" that were associated with other "targets." The probability of someone being at a meeting if the "target" was at a meeting and the likelihood of someone visiting a place if the "target" visited the place were found and using these statistics and multiplying them by their semantic scores, spatial being in the same area as a target and temporal being people who met with the target, two final results were produced.

S. Khetarpaul, R. Chauhan, SK Gupta, LV Subramaniam, U. Nambiar (2011). Mining GPS Data to Determine Interesting Locations. In Proceedings of the 8th International Workshop on Information Integration on the Web, New York, NY, USA 2011. The researchers analyzed aggregate GPS info of 62 users over 2 years to mine the list of interesting location rank, which was determined by visits per user. They used relational Algebra Operations and Statistical formulas to determine these "interesting locations." The main topic dealt with was "Stay Point Determination," wherein an individual spent a significant amount of time in a location, either indoors – where they would enter, lose signal, and exit – or a general outdoor location, both determined by a threshold.

Z. Yan (2010). Traj-ARIMA: A Spatial-Time Series Model for Network-Constrained Trajectory. In 3rd ACM SIGSPATIAL International Workshop on Computational Transportation Science, San Jose, CA.

Security & Privacy

Liang Cai and Hao Chen. TouchLogger: Inferring Keystrokes On Touch Screen From Smartphone Motion. In Proceedings of the sixth USENIX Workshop on Hot Topics in Security (HotSec '11), San Francisco, CA, August 9, 2011.

L. Cai, S. Machiraju, and H. Chen (2009). Defending against sensor-sniffing attacks on mobile phones. In Proceedings of the 1st ACM workshop on Networking, Systems, and Applications for Mobile Handhelds, 31–36, New York, NY, USA, 2009. ACM.

Burns J. 2009. Developing secure mobile applications for Android. Black Hat DC.

Alastair R. Beresford, et al. 2011. MockDroid: trading privacy for application functionality on smartphones. In Proceedings of HotMobile.

N. Xu, F. Zhang, Y. Luo, W. Jia, D. Xuan, and J. Teng. Stealthy video capturer: a new video-based spyware in 3g smartphones. In Proceedings of the second ACM conference on Wireless Network Security, 69–78, New York, NY, USA, 2009. ACM.

W. Enck, P. Gilbert, B. gon Chun, L. P. Cox, J. Jung, P. McDaniel, and A. N. Sheth. Taintdroid: An information-flow tracking system for realtime privacy monitoring on smartphones. In Proceedings of the 9th USENIX Symposium on Operating Systems Design and Implementation, 393-408, August 2010.

Firzpatrick, M. Mobile that allows bosses to snoop on staff developed. BBC News, March 2010.

Schlegel, R., Zhang, K., Zhou, X., Intwala, M., Kapadia, A., and Wang, X. (2011). Soundcomber: A Stealthy and Context-Aware Sound Trojan for Smartphones. In Proceedings of the Network and Distributed System Security Symposium 2011.

Related Research Groups/Stories

MIT Media Lab has a focus on reality mining which involves getting information from mobile phones to get insight into individual and group behavior. The project involved giving 100 students a cell phone and collecting all of the data. The focus here is on learning from mobile phone data, but the focus is not on sensor data and much of the focus is on group behavior and social networks. Thus, the focus is largely different than that of the WISDM lab. There is a short movie on reality mining (it may take a long time to load).

An NPR interview discusses the pros and cons of crowd sourcing personal data, including data from cell phones. The interview discusses privacy issues.