PhD Research Fellow in Structured Vision-Language Models

About the position Integreat - the Norwegian Centre for Knowledge-driven Machine Learning is seeking a motivated PhD candidate to join our interdisciplinary research centre at the University of Oslo, Norway. You will be part of a dynamic group of early career researchers, supervised by senior experts, collaborating across disciplines to tackle fundamental challenges through innovative methods, theory and critical analysis. The fellowship period is 3 years. A fourth year may be considered, and it will involve 25 % of other career-promoting work. Other career-promoting work may consist of teaching, supervision, and/or research assistance. This is dependent upon the qualifications of the applicant and the current needs of the Department of Informatics Starting date from January 2027 and upon individual agreement thereafter. A candidate may be appointed to only one PhD Research Fellowship at the University of Oslo.

About Integreat See the video for more information about Integreat. Video: https://www.youtube.com/shorts/-_I52cJHiW4About Integreat Integreat - Norwegian Centre for Knowledge-driven Machine Learning is a Centre of Excellence, funded by the Research Council of Norway (2023-33). Integreat has two branches, one in Oslo (University of Oslo, UiO) and one in Tromsø (UiT The Arctic University of Norway). Machine learning is the mathematical and computational engine of Artificial Intelligence (AI), and therefore it is a fundamental force of technological progress in our increasingly digital, data- and algorithm-driven world. Integreat develops theories, methods, models, and algorithms that integrate general and domain-specific knowledge with data, laying the foundations of next generation machine learning. We do this by combining the mathematical and computational cultures, and the methodologies and theories, of statistics, logic, language technology, ethics and machine learning, in new and unique ways. Focus of Integreat is to develop groundbreaking methods and theories, and therefore solving fundamental problems in science, technology, health and society. Integreat draws on the research strengths of researchers and students from the departments of Mathematics, Informatics, Philosophy, and the Oslo Centre for Biostatistics and Epidemiology at UiO, the Norwegian Computing Centre (NR) and the ML group at UiT, with members from the departments of Physics and Technology, Mathematics and Statistics, and Computer Science.

Project description Vision-language models (VLMs) can name what is in a picture, but they often miss what is happening in it. Shown a park scene, a model may list "a dog", "a Frisbee" and "a park" and still fail to say that the dog is leaping to catch the Frisbee mid-air. The same gap shows up when a model misreads a symbolic painting or gives a vague description of a medical image. More training data does not close this gap, because the models have no explicit representation of how the objects in a scene relate to each other. This project gives VLMs that representation. You will develop panoptic scene graphs, in which every object and background region in an image becomes a node and the edges encode spatial, semantic and relational context. The graphs are built from superpixels rather than the fixed square patches used in standard vision transformers. Superpixels follow object boundaries, so the nodes of the graph are meaningful from the start. The project builds directly on our recent work on superpixel tokenization for vision transformers (ECCV Workshops 2024, NeurIPS 2025). The scene graph is then fused with a large language model (LLM), and the two parts inform each other. The LLM guides which objects and relations the vision model should focus on, and the graph grounds the LLM's answers in what is actually in the image, which reduces hallucination. Because a scene graph has the same form as a knowledge graph, it also opens the way to graph-based visual retrieval-augmented generation (graph visual RAG), where a model links what it sees to external knowledge before it answers. The aim is a new class of VLMs that reason over images instead of only describing them. Such models would be more accurate, more explainable (each answer can be traced to nodes and edges in the graph) and more useful where deep understanding matters. Examples include a robot that follows an instruction such as "place the red mug next to the laptop onto the top shelf", a medical system that reasons about the relations between structures in an X-ray, and a self-driving car that anticipates how pedestrians and vehicles will interact. As a PhD fellow on the project, you will contribute to:

• develop superpixel-based panoptic segmentation models built on modern transformer architectures • design methods that predict relations between segmented regions and generate panoptic scene graphs • fuse scene graphs with LLMs for visual question answering and grounded reasoning, including graph visual RAG • publish and present your results at leading machine learning and computer vision venues Project supervisors: Professor Adín Ramírez Rivera at the Department of Informatics, UiO, in collaboration with Benjamin Ricaud and Filippo Maria Bianchi at UiT The Arctic University of Norway. Working language: English.

What skills are important in this role? The Faculty of Mathematics and Natural Sciences has a strategic ambition to be among Europe’s leading communities for research, education and innovation. Candidates for these fellowships will be selected in accordance with this, and expected to be in the upper segment of their class with respect to academic credentials. Qualification requirements:

• Master’s degree or equivalent in computer science, machine learning, data science, mathematics, statistics, electrical engineering, or a related field. Foreign completed degree (M.Sc.-level) corresponding to a minimum of four years in the Norwegian educational system. • Experience in scientific programming. Desired qualifications:

• Proficiency in scientific programming (e.g., Python) • Proficiency in deep learning frameworks (e.g., PyTorch) • Background in machine learning, mathematics, linear algebra, and statistics • Experience in creating, adapting, or training modern, token-based transformer architectures • Experience working with image / vision models • Experience in creating or adapting large language models (LLMs) and interfacing them with other models • Familiarity with image segmentation • Familiarity with Scene Graph Generation • Familiarity with LLM fine-tuning and alignment • Knowledge of graph-based retrieval-augmented generation (Graph RAG) • Understanding of probabilistic modelling and information-based regularizers • Strong scientific writing skills Language requirement:

• Good oral and written communication skills in English • English requirements for applicants from outside of EU/ EEA countries and exemptions from the requirements: https://www.mn.uio.no/english/research/phd/regulations/regulations.html#toc8 Grade requirements: The norm is as follows:

• the average grade point for courses included in the Bachelor’s degree must be C or better in the Norwegian educational system • the average grade point for courses included in the Master’s degree must be B or better in the Norwegian educational system • the Master’s thesis must have the grade B or better in the Norwegian educational system The purpose of the fellowship is research training leading to the successful completion of a PhD degree. For more information see: Supplementary Regulations at the Faculty of Mathematics and Natural Sciences to the Regulations for the Degree of Philosophiae Doctor (PhD) at the University of Oslo - The Faculty of Mathematics and Natural Sciences

Personal skills We are looking for candidates who are curious, motivated to learn, and interested in tackling challenging scientific questions. You should be able to work independently while also contributing actively to collaborativ