Automated scene graph generation from camera images of traffic scenes to validate AI-based object recognition

Context

To better understand weaknesses in AI-based object recognition, it is not enough to simply recognize which objects a system perceives in images. Research in AI explainability shows that the attributes of these objects and their environment also play a central role. The aim of the RepliCar project is to develop a new type of black box method in which an automated scene graph is created to take these influencing factors into account in development processes. This scene graph arranges objects and their relationships in a defined tree structure and thus enables more precise analyses of incorrect behavior during object recognition. A pipeline of classifiers is used to systematically decompose and semantically classify traffic images from the demonstrator vehicle. Detailed information and an introduction to the scene graph can be found in the publication "The Machine Vision Iceberg Explained".

Goals

The development of a pipeline for the decomposition of camera images from road traffic poses a particular challenge, as a large number of potential classification methods are available. On the one hand, AI methods with high-resolution models offer a wide range of classes, but are prone to false positives (hallucinations). On the other hand, "classical" classifiers often provide less false-positive results, but offer less variability in classes and require high manual effort.

The aim of this work is to select existing classifiers and evaluate them on different data sets. Promising combinations will then be applied to the camera images of the demonstrator vehicle in order to automatically generate the required scene graph. The scientific goal is to discuss the suitability of AI classification and classical classification methods for the validation intention and whether a systematic approach can be derived from this.

• Familiarization with the logic of the scene graph from the associated publication
• Research and prototypical implementation of suitable classifiers to create a decision matrix
• Combination and comparison of different classifiers
• Evaluation on suitable image data sets with ground truth
• Automated generation and formatting of a scene graph
• Testing the scene graph on images from the demonstrator vehicle

Requirements

• Degree course: Electrical Engineering, Mechatronics, Computer Science or similar (Master's degree)
• Previous knowledge of Python, image processing, machine learning, statistics in data science