Fordham University

Robotics& Computer Vision Laboratory

Current Research Projects

o Robotic Team Navigation Using Terrain Spatiograms

o Cognitive Robotics: Synchronizing Real and Synthetic Video

o Agile, Legged Locomotion.

o Sensory Fusion for Multiple Target Tracking

 In the News

Yonkers MS Robotics Program: Fordham eNews 08/09

Robotics Lab: Alumni News V1N2

Inside Fordham 02/09

Previous Research Projects

o Performance Guarantees for Mobile Robots used in Disaster Recovery (with R. Arkin, Georgia Tech).

o Automated Management of Multiple Camera Resources.

o Combining Recognition with tracking: Discrete-Event Modeling of PTZ targets

o Controlling Graphical Animation using Computer-Vision

Current Research Projects

Robotic Team Navigation Using Terrain Spatiograms (TSG).

A team of robots working to explore and map a space may need to share information about landmarks so as register local maps and to plan effective exploration strategies. In this paper we investigate the use of spatial histograms (spatiograms) as a common representation for exchanging landmark information between robots.  Each robot can use sonar, stereo, laser and image information to identify potential landmarks. The sonar, laser and stereo information provide the spatial dimension of the spatiogram in a landmark-centered coordinate frame while video provides the image information. We call the result a terrain spatiogram (TSG) . This representation can be shared between robots in a team to recognize landmarks and to fuse observations from multiple sensors or multiple platforms.

We report experimental results sharing indoor and outdoor landmark information between a two different models of robot equipped with differently configured stereocameras and show that the terrain spatiogram (1) allows the robots to recognize landmarks seen only by the other with high confidence and, (2) allows multiple views of a landmark to be fused in a useful fashion.

We have developed a mixture of Gaussians (MOG) model to represent the spatial probabilities in this case that the color distribution is multimodal. Multiple views of landmarks can be combined into a single ‘360 degree’ TSG and the MOG representation is even more strongly supported by this case.  A video of the TSG for the robot landmark is attached the left..

Lyons, D.M., “Sharing Landmark Information using Mixture of Gaussian Terrain Spatiograms,” IEEE/RSJ International Conference on Intelligent RObots and Systems (IROS), St Lous, MO, October 2009. [doc]

Damian M. Lyons. Tracking and sharing landmarks in a team of autonomous robots, Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications at the SPIE Defense and Security Symposium, March 2009, Orlando (Kissimmee), FL. [doc]

Lyons, D.M., Isner, G.R., Evaluation of a Parallel Algorithm and Architecture for Mapping and Localization. 7th International Symposium on Computational Intelligence In Robotics and Automation, CIRA 2007, Jacksonville FL June 20-23, 2007. [doc]

Lyons, D.M., Hsu, D.F., Ma, Q., and Wang, L., Combinatorial Fusion Criteria for Robot Mapping. 21^st International Conference on Advanced Information Networking and Applications (AINA 2007), May 21-23 2007, Niagara Falls Canada. [doc]

Lyons, D.M., Hsu, D.F., Ma, Q., and Wang, L., Selection of fusion operations using rank-score diversity for robot mapping and localization. Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications at the SPIE Defense and Security Symposium, 9-13 April 2007, Orlando (Kissimmee), FL. [doc]

Cognitive Robotics: Synchronizing Real and Synthetic Video.

We are implementing ADAPT, a cognitive architecture for a Pioneer mobile robot, to give the robot the full range of cognitive abilities including perception, use of natural language, learning and the ability to solve complex problems. Our perspective is that an architecture based on a unified theory of robot cognition has the best chance of attaining human-level performance.

One of the objectives of Cognitive Robotics is to construct robot systems that can be directed to achieve real-world goals by high-level directions rather than a complex, low-level robot programming. Such a system must have the ability to represent, problem-solve and learn about its environment as well as communicate with other agents. In previous work, we have proposed ADAPT, a Cognitive Architecture that views perception as top-down and goal-oriented, so that perception becomes part of the problem solving process. This approach is linked to a SOAR-based problem-solving and learning framework. In this paper, we focus on the architecture for the perceptive and world modelling components of ADAPT and report on experimental results using this architecture to perceive events in a complex, real-world scenario.

Consider an object, such as a ball, moving in cluttered environment and consider a robot given the objective of intercepting the object. The robot must track the object as it moves and determine how to intercept it. However, in a cluttered environment, it is highly likely the object will collide and rebound from the environment. Tracking can at most handle this on a collision-by-collision basis. In our approach a world modelling system - a 3D game engine - is intimately linked into the visual perception process. Predictions of complex object behavior such as repeated rebounds from the environment are generated from the world modelling system at very high rates and are available as 'imagined' perceptions, directly comparable with actual perceptions.

Lyons, D.M., Benjamin, P., Locating and Tracking Objects by Efficient Comparison of Real and Predicted Synthetic Video Imagery. SPIE Conference on Intelligent Robots and Computer Vision XXVI: Algorithms and Techniques, San Jose, CA, January 2009. [doc]

Benjamin, D.P., Lonsdale, D., and Lyons, D.M., Using Cognitive Semantics to Integrate Perception and Motion in a Behavior-Based Robot. ECSIS Symposium on Learning and Adaptive Behaviors for Robotic Systems, LAB-RS '08. Aug. 2008, Edinburgh, United Kingdom. Pp.77-82.

Benjamin, D.P., Lonsdale, D., and Lyons, D.M., Developing a Cognitive Architecture to be Embedded in the Physical World, Proceedings of BRIMS206 (Behavior Representation in Modeling and Simulation), Baltimore, May, 2006.

Agile, Legged Locomotion.

We have developed a novel design for a mechanism for legged locomotion that is

o an extremely agile mobile platform,

o an efficient approach to legged locomotion and

o an inexpensive template mechanism for robot teams.

The Rotopod is a novel robot mechanism which combines the features of wheeled and legged locomotion in an unusual way.  This robot has the advantage of legged locomotion in stepping its 1-DOF legs over objects, but its drive mechanism is a rotating reaction mass that rotates the robot, in a controllable fashion, around each of its legs, similar to a rotating wheel. The mechanism has the potential to transfer the energy from the rotating reaction mass in an efficient manner to the legs, effecting a spinning forward motion

For more details, see the rotopod project description.

Lyons, D., A Novel Approach to Efficient Legged Locomotion. 10^th Int. Conference on Climbing and Walking Robots. 16-18 July 2007, Singapore.[doc]

Lyons, D., and Pamnany, K., Analysis of gaits for a rotating tripedal robot. SPIE Conference on Intelligent Robots and Computer Vision XXIV, 23-26 Oct. 2005, Boston, MA. [pdf]

Lyons, D., and Pamnany, K., Rotational Legged Locomotion. IEEE Int. Conf. on Advanced Robotics, July 2005, Seattle, WA. [pdf]

Sensory Fusion for Multiple Target Tracking.

Most existing visual tracking systems do not handle crowded scenes well. Our goal is to develop algorithms that take multiple sensory cues from the video (e.g., target locations, colors, shapes, etc) and fuse this information to robustly track in crowded scenes. We focus on the issue of occluding targets - since this is where a lot of the difficulty in visual tracking arises.

We use sensory fusion to disambiguate occluding targets. This is a difficult problem, since the process of occlusion gives rise to dramatic and non-linear changes in the feature values. We exploit an approach that determine which cues to use and how to best combine them by looking at the distribution of feature measurement values to candidate targets - the so-called rank-score behavior. Experimentally we have shown that this approach, which we call the Rank and Fuse approach improves on a weighted sum or mahalanobis-sum for fusion.

Lyons, D.M., and Hsu, D.F., Method of Combining Multiple Scoring Systems for Target Tracking using Rank-Score Characteristics. Information Fusion 10(2) 2009 [ doc]

Lyons, D.M. and Hsu, D.F., Comparing CFA and Discrimination for Selecting Tracking Features. Multisensor, Multisource Information Fusion: Architectures, Algorithms, and Applications at the SPIE Defense and Security Symposium, 18-20 March 2008, Orlando (Kissimmee), Fl. [doc]

Hsu, D.F., Lyons, D.M., and Ai, J., Selecting and Evaluating Combinatorial Fusion Criteria for to Improve Multitarget Tracking. Fusion 2006, July 10-13 2006, Florence, Italy. [pdf]

Back to the top

Previous Research Projects

Performance Guarantees for Mobile Robots used in Disaster Recovery (with R. Arkin, Georgia Tech).

Mobile robot bases equipped with cameras and sonar are being increasingly used in disaster recovery applications to search debris piles. They are safer than using rescue teams, require less care than rescue dogs, and can traverse toxic areas off-limits to both. However, the unpredictability of debris structure and the state-of-the-art in behavior-based robot control mean that there is little way to decide if a robot can navigate effectively and whether it can detect what needs to be located.  We are jointly developing a formal theory and software toolkit to address this problem. 

Automated Management of Multiple Camera Resources.

Our goal is to automate the process of switching between multiple cameras when (manually or automatically) tracking a target. A major question in this is to understand the connectivity between camera views. We have developed algorithms and set of software libraries to automatically learn (using a NN) the candidate handoff cameras for each camera in a building. The cameras do not need to have overlapping views, exist and entrances can be anywhere in the field of view, and no map is needed. Future work will include software to periodically update the handoff information to account for camera or building changes.

Combining Recognition with tracking: Discrete-Event Modeling of PTZ targets

Most PTZ tracking systems decide when to pan, tilt or zoom based only on providing the best operator view of the target. While the operator view is clearly an important end-goal for tracking, it is not the only constraint that needs to be acknowledged. A second constraint is that the tracker be able to robustly recognize the target. There is no reason that these two constraints should always agree, and ignoring the second constraint means the operator may get an excellent view of the wrong target! We have developed a discrete-event control approach to modelling the target shape and color in such a way that we can determine when we need to zoom to maintain recognition of the target as well as maintain the operator view. Future work involves extending the discrete-event model to a hybrid model to allow fine control of PTZ.

Controlling Graphical Animation using Computer-Vision

Within the next few decades society will move beyond using the keyboard and display for interacting with computers, phones, PDAs, etc.  Instead we will take advantage of inexpensive and commonplace sensor technology to allow these devices to directly sense the user and to allow the user to control the device using gesture and speech. Our goal is to explore this augmented reality style of interaction. We have developed a graphical model of human eyes (using  Open Inventor/VRML in C++) and control those eyes to follow the closest person using information from a simple visual motion tracker. Future work involves understanding to what extend this gives the person the cue that the computer is paying attention to them and ready for interaction.

Back to the top

Home| Research | Publications| Software

dml 2005