Projects

Uncertainty Aware Cross Modal Semantic Segmentation

Depth information from sensors may not be reliable so we propose a novel mechanism called uncertatinty-aware self-attention that explicitly controls the information flow from unrelaible depth pixels to confident depth pixels during feature extraction. In addition, we propose an effective and scalable fusion module based on Cross-Attention that can adaptively fuse and exchange information between the RGB encoder and depth encoder.

Efficient Single-stage Video Instance Segmentation

The task of instance segmentation in videos aims to consistently identify objects at pixel level throughout the entire video sequence. Existing state-of-the-art methods either follow the tracking-bydetection paradigm to employ multi-stage pipelines or directly train a complex deep model to process the entire video clips as 3D volumes. However, such methods are typically slow and resource consuming. We propose SRNet, a simple and efficient framework for joint segmentation and tracking of object instances in videos. The key to achieving both high efficiency and accuracy in our framework is to formulate the instance segmentation and tracking problem into a unified spatial-relation learning task where each pixel in the current frame relates to its object center, and each object center relates to its location in the previous frame.

Few Shot Image Segmentation

Semantic Segmentation is one of the fundamental tasks in Computer Vision. Given an input image, the algorithm will assign a class label to each pixel. In recent years, many researchers have proposed semantic segmentation schemes that produce great performance but such schemes rely on the availability of large annotated datasets. Since dtaa labeling is costly, researchers are now exploring few shot image sementation schemes that allow the model to be trained using fewer images. We have started to explore this type of research.

Robust Autonomous Driving

Autonomous vehicles (AVs), one of the most impressive technology developed in recent years, have the potential to impact the daily lives of many people. Level 4 vehicles will include automated system which handles all driving tasks under limited driving conditionsl even if the human driver does not respond to requests to intervene while the goal of Level 5 autonomous driving is to make a vehicle perceive its environment and safely navigate any traffic situation efficiently without human intervention. However, significant technical challenges still need to be overcome before Level 4 & 5 AVs can be deployed. In this project, we intend to investigate if we can improve the accuracy of pedestrian detection under all weather conditions. We also are interested in exploring the issue of pedestrian tracking and the prediction of their future intentions. See a related NSF funded project here

Silver LifeTogether

Silver LifeTogether is a project where we investigate how we can use technology and engage community partners to help seniors age in place. Percentage of seniors in US will be 21.4% by 2050. Gradual functional decline limit seniros' ability to conduct daily life activities. Seniors' mental, cognitive health decline put increasing burdens on US health systems. In this project, we will develop mobile applications that can help seniors improve their mental and physical health. In addition, Health, Medicine and Society related students help to engage community partners to provide useful services to seniors in a nearby community center, e.g., having choirs sing for the seniors, medical professional volunteer to provide health checks for the seniors,etc

Robust and Efficient Human Action Recognition

Designing a scheme that can achieve a good performance in predicting single person activities and group activities is a challenging task. In this project, we propsoe a novel robust and efficient human rcognition scheme (REHAR) which can be used to handle single person activities and group actitivites prediction. We utilize both video frames and their coorsponding optical flow images to derive represenations using a single frame representation model. Then, a LSTM is used to predict the final activities based on the generated representations. Our evaluation results using the NCAA basketball dataset and UCF Sports datasets show that REHAR achieves a higher activity recognition accuracy with an order of magnitude shorter computation time compared to the state of the art methods. This project is partially funded by Qualcomm Technologies, Inc. and FORD.

Drone Assisted Smart City Application

Smart parking solutions for big cities are critically important for reducing traffic congestion and vehicle energy consumption in big cities. Visual-based detection methods have been proposed since their maintenance costs are typically lower. However, visual based detection methods in existing systems are not robust due to varying light intensities, occlusions, and their deployment costs can still be expensive. In this paper, we present a collaborative UAV aided smart parking solution where a small team of low cost UAVs are used to collaboratively identify free parking spots of parking lots within campuses. We design a novel navigation control scheme to allow such UAVs to avoid obstacles and cover the parking lots sufficiently. In addition, we also present a visual detection scheme using the generative adversarial network (GAN) which allows us to predict parking availability without requiring pre-identification of parking spots. Simulation results indicate that our approach show promising results.

Optimizing Deep Learning Schemes for Mobile Vision Applications

In recent years, image recognition related applications has become very popular especially after Google popularized the technology with Google Googles in 2010. Image recognition is used in many new technologies such as self-driving cars, e-commerce e.g. searching and buying similar apparel based on a snapshot. However, several challenges remain if we want highly accurate image recognition anywhere anytime using mobile devices. Traditional object recognition algorithms operate robustly only in controlled environments but exhibit high variation in accuracy for more generic mobile environments. Some key reasons are (i) visual domain shift, e.g., changes in image resolution, lighting, background, viewpoint often causes performance degradation, (ii) hard to predict all objects to be recognized. We recently designed an adaptive mobile object recognition framework (joint research with IBM researchers) which allows deep learning techniques to be used successfully in mobile environments. Together with a Georgia Tech researcher, we also design MOCA, an energy-efficient and privacy-preserving deep processing system for mobile vision-based applications. Our solution decouples the hierarchical neural network layers and splits them across mobile and cloud for energy efficiency. MOCA provides adjustable privacy settings using a DNN-based privacy quantification framework to meet the privacy requirements of different mobile applications.

Robust Human Action Recognition

Human action recognition is a major component of many computer vision applications, e.g., video surveillance, patient monitoring, and smart homes. Existing algorithms designed to recognize single-person actions from sequences of 3D joint positions cannot be used to recognize multi-person interactions, and vice versa. This is because the two categories of actions are exclusive by definition. We present a category-blind human action recognition method (CHARM) which is more suitable for realworld scenarios. Compared to the existing action recognition approaches that are designed to cope with only one action category, and are difficult to be extended to scenarios where different action categories co-exist, CHARM achieves comparable or better performance without any prior action categorization. Our work has been published in ICCV 2015. Below is a video demo.

Smartphone Enabled Healthcare System (partially funded by NSF)

Mobile phones have become increasingly popular and gradually woven into our lives. Smartphones equipped with powerful embedded sensors (e.g., accelerometers, GPS, microphones, and etc.) can be used to monitor multiple dimensions of human behaviors including physical, mental and social behaviors of wellbeing. The collected sensing data can thus be mined not only for the understanding of daily life activities or human behaviors but also for supporting a broad range of mobile healthcare applications. We are designing a smartphone based healthcare monitoring system which monitor users for their mental, cognitive, and physical well-being and hence facilitate early diagnosis of potential illnesses and taking possible preventive measures. One project called SENSCOPS is funded by NSF. For more information about SENSCOPS , click here.

Secure Mobile Information Sharing System (SEMOIS)(funded by NSF)

Rapid advancement of the wireless technologies provide new opportunities for mobile users to have easy access to real-time data, derive useful social information, and stay connected with business partners, colleagues and friends. Towards this end, mobile social networking applications have recently emerged to meet these needs. This project aims to build a secure mobile information sharing system (SEMOIS) that supports secure and privacy-preserving real-time information sharing. SEMOIS has the ability to store secure data items with flexible access control at insecure storage nodes and enables users to send context-based messages with late-binding features. Additionally, a set of smart learning methods are developed to extract short-term and long-term geo-social patterns from multimodal sensing data collected by mobile devices for social networking purposes, e.g., geo-social patterns are used to derive hidden communities. For more information on SEMOIS , click here.

Robust Smartgrid Communication Infrastructure

A smartgrid contains components that enable userful information to be collected from a power grid such that control centers can estimate the current states of the power grid. Such information is sent via wireless or wired networks. Local-area communication subsystems in a smartgrid tend to employ wireless links for their last mile communication. Unfortunately, various attacks e.g. channel jamming attacks or DoS attacks can be launched in wireless networks. Together with collaborators from Steven Institute of Technology, we designed a communication subsystem with self-healing feature via intelligent local controller switching to combat against jamming attacks. Our design allows the smartgrid to continue to collect smart meter readings under jamming attacks. We also provide guidelines on optimal placement of local controllers to ensure effective smart meter switching. Our work has been published as IEEE CNS 2013 conference paper which won the best runner-up paper award and as a journal paper in IEEE Transaction on Dependable and Secure Computing (TDSC). Recently, we secure DOE funding to work on protecting smartgrid against cyber attacks. Specifically, we will design P2PBotnet detection scheme suitable for smartgrid.

VRGames for Parents with ADHD children

In this project, we work with three faculty in College of Education and their research team to develop web-based VR Games to let parents with ADHD children practise how to use the intervention skills they learn in PEAK training sessions. Our VR games target typical daily life scenarios where children often act out. A sample VR game demo is shown below

LILI - A Low Cost Interactive Assistive Robot for Special Need Children

In this project, we build a low-cost interactive assistive robot for special need children. Our initial prototype called LILI was built by several REU students and a MS degree student under the supervision of Prof Spletzer and Prof Chuah. LILI can be controlled by authorized users (via face recognition) using voice commands. LILI can navigate through environments with obstacles. We intend to add more human intelligence to LILI. More information about LILI can be found here.

Undergraduate Research

Several undergraduates work with Prof Chuah via senior design projects or NSF REU grants. One senior design project called "Kinect-for-Kids" is a game using Kinect sensor designed by Jesse Kurtz and Kyle Moore. This game was introduced to children with Autism Spectrum Disorder in an Autisim Respite event held in Dec 6, 2014 at a nearby church in Bethlehem. Chidlren who have played with this game loved it. Additionally, we released several Android applications for special need children. Click here for more details. In Fall 2015, undergraduates Noel Morris and Andrew McMullen designed MemCare (photo, poster) which is a memory game for evaluating cognitive functions, e.g., memory recall of sleep deprived college students or old people.

Secure Content-Centric Mobile Network (SECON)(funded by NSF)

New wireless technologies allow mobile users to have easy access to real-time data, and stay connected with friends and colleagues. However, emerging mobile applications are data-centric but existing IP oriented communication paradigms are not flexible enough to support this. Additionally, serious security and privacy concerns have also been raised. To fully support emerging mobile applications, we are developing a next generation mobile network called SECON that supports mobile content centric networking features and applications that use such features. For more information on SECON , click here.

NSF REU SmartSpace project

The Smart Spaces REU Site will provide motivated computer science and engineering students with a unique research experience centered around the redevelopment of the Mountaintop Campus at Lehigh University. The Smart Spaces REU Site will be an interdisciplinary research environment. Some potential research topics include: 3-D Augmented Reality, Ambient Intelligence, Computer Vision, Embedded Devices, Mobile Computing, Networking and Security, Robotics, and User Interface Design. For more information on SmartSpace, click here.