ESR3 – Classifying and Predicting Interactions Between AV and VRUs Using AI

Popular scientific abstract

Given that human error is involved in about 95% of all road traffic accidents, automated vehicles can greatly reduce these figures and improve road safety. More than half of all road traffic deaths and injuries involve vulnerable road users (VRUs), such as pedestrians, cyclists, and motorcyclists. Therefore, it’s essential for automated driving (AD) systems to predict the behavior and intention of VRUs to help AVs to make better decisions and to prevent hazardous situations. In ESR3-“Classifying and Predicting Interactions Between AV and VRUs Using AI”, we aim to better understand how VRUs behave when interacting with AVs in urban traffic, in order to improve human-AV interaction design.

In this project, we’ll use powerful AI tools, such as machine learning and deep learning methods for the prediction of VRU behavior. With the help of various sensors, AVs can better perceive the VRUs. We’ll collect real-world driving data in Europe to train and test our algorithms. The classification and prediction of VRU behavior will be used to better understand the interactions of external road user with AVs, and to improve road safety.

My affiliation

My host university is the University of Gothenburg (UGOT).

Contact of supervisors:

Associate Prof. Christian Berger (UGOT;

Prof. Marco Dozza (Chalmers;


The prediction of VRUs such as pedestrians can be very challenging. This is because pedestrians are very agile that can change both their direction and velocity without reducing the speed [1, 2], and are easily influenced by other road participants’ behavior and surroundings [3]. It is difficult to reliably predict the intentions of pedestrians by hand-crafted features; hence, we aim to design a deep learning method that can encode the feature and can deal with complicated traffic environments. Our goal is to propose the models that can predict the future behaviors of VRUs (e.g. pedestrians), as well as deal with the interactions between the VRUs and other road users.

Aims and objectives

This research aims at better understanding how VRUs behave when interacting with AVs in urban traffic in order to improve human-AV interaction design. AI methods will be applied to AV/VRU interaction data to classify the interactions and predict VRU behaviors. The classifications and predictions of VRU behavior will be used to improve understanding of AV-external road user behavior in interactions with AVs.

The objectives of this research are:

  • To develop AI tools to classify and predict the behaviors of VRUs in interactions with cars;
  • To use these tools to predict VRU behavior and intention in more complex interactions;
  • To compare the behavior classifications and predictions using AI with results from literature and others traditional computational models.

Research description

As a part of the “SHAPE-IT” project, my research topic is “Classifying and Predicting Interactions Between AVs and VRUs Using AI”, which aims at using AI to better understand human behaviors.

My research plan for this project is as below:

Study1: Literature review and dataset exploring. In this research, we explore the following research questions:(a) What are the state-of-the-art algorithm of pedestrian behavior prediction in urban scenarios? (b) How does the existing research model the Automated Vehicle (AV)-Vulnerable Road User (VRU) interaction? (c) What datasets are available to support the testing of pedestrian behavior prediction and AV-VRU interaction?

Study 2: Pedestrian trajectory prediction. In this research, we explore the following research questions: (a) How to precisely predict the trajectory of pedestrians? (b) How to model the influence of vehicles when predicting pedestrians’ trajectories?

Study 3: VRU (e.g., Pedestrian) intention prediction and classification. In this research, we explore the following research questions: (a) How to accurately predict the crossing intention of pedestrians? (b) What information should be considered for predicting and how to model it?

Study 4: The interaction between VRU (e.g., Pedestrian) and vehicles/AVs. In this research, we explore the following research questions: (a) How do the pedestrians interact with AV? (b) How to use this information to support human-machine interface design?


The expected results are:

  • Tools for classifying and predicting interaction behaviors between AVs and VRUs in urban environments, to support human factors researchers and automation designers;
  • Requirements and guidelines for the use of AI in classifying and predicting VRU interactions with AVs;
  • Assessment/validation of the AI tool to classify and predict AV/VRU interactions (compared to traditional methods)

My publications

Papers under submission:

  • Zhang, C. & Berger, C. (2021). Pedestrian Behavior Prediction Using Deep Learning Methods for Urban Scenarios: A Review. (Revising)
  • Zhang, C. & Berger, C. (2021). Learning the Vehicle-Pedestrian Interaction for Pedestrian Trajectory Prediction. (Revising)


[1] N. Schneider and D. M. Gavrila, “Pedestrian path prediction with recursive bayesian filters: A comparative study,” in German Conference on Pattern Recognition. Springer, pp. 174–183. 2013.

[2] B. Volz, K. Behrendt, H. Mielenz, I. Gilitschenski, R. Siegwart, and J. Nieto. A data-driven approach for pedestrian intention estimation. In 2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC), pages 2607–2612. IEEE, 2016.

[3] B. Volz, H. Mielenz, G. Agamennoni, and R. Siegwart. Feature relevance estimation for learning pedestrian behavior at crosswalks. In 2015 IEEE 18th International Conference on Intelligent Transportation Systems, pages 854–860. IEEE, 2015.