Research

My research interests span computer vision, multimodal learning and, more recently, multimodal large language models, post-training with on-policy optimisation and conversational AI. In the past, I have been doing some research in image segmentation and optical flow estimation using deep learning, as well as the classification of dynamic textures (human crowds, herds of animals, etc.) which is closely related to my PhD subject. Currently, at Rizlum AI, I work on multimodal-model systems for document analysis, post-training of models with on-policy optimisation, and speech-to-speech conversational AI systems (realtime speech-to-text and text-to-speech) achieving sub-1 second latency. Here are some details of my recent researches.

Multimodal LLM, post-training and conversational AI research

During my postdoctoral research at Toyota Belgium and LISSI, I worked on multimodal dataset analysis (VQA, Conceptual Caption), the design of multimodal LLMs, and the design of an intelligent emotion agent. This line of work continues at Rizlum AI, where I focus on multimodal-model systems for document analysis, post-training of models with on-policy optimisation methods, and realtime speech-to-speech conversational AI.

PhD research

The thesis research work is about video analysis. More specifically, we are interested in identification and characterization of trajectories in videos. Contrary to common trajectories corresponding to the movement of identifiable objects in videos (tracking of a person, a vehicle, etc.), the goal of this work will be rather on the analysis of the movement of a set of objects forming an heterogeneous "mass", like some fluids, human crowds, or animal swarms, for example. Therefore, with respect to the above context, this thesis will focus on three highly interdependent aspects:

• Discovery and extraction of several spatio-temporal patterns that are specific from studied video sequences.
• Joint analysis of these patterns along the time (multi-trajectory/sequence analysis).
• Use of machine learning procedures (in particularly deep learning) to facilitate the discovery and analysis of such patterns and sequences.

The extraction of spatio-temporels patterns should allow to characterize locally the movements observable in the video. In order to do this, we should be able to build some spatio-temporal descriptors based on dictionaries for example. One way to extract these patterns could be to use deep learning techniques. Once obtained, these patterns can be mixed together to build sequences.

The next step is to analyze these sequences to find more complex behaviours such as reoccurrence of subsequences, abnormal subsequences, etc. For this, we will focus on the use of data mining methods (sequence mining), as well as sequence matching techniques and pattern recognition techniques, that will have to be adapted to the specificities of the videos. In particular, the goal will not be to work on a unique sequence but jointly with several ones that will characterize the behaviour of the heterogenous “mass” observed in the video.