Neuro-Symbolic Learning under Multi-Modal Data
Master's thesis: teaching models to be right for the right reasons
- SOTA across multiple NSAI frameworks
- OOD gains in and out of distribution
- A100 reproducible experiment pipeline
In neuro-symbolic AI, a neural network perceives the input and a symbolic layer reasons over the concepts it extracts. The reasoning step is interpretable, which is the whole appeal.
It also breaks in a specific way. The network learns concept labels that produce the right final answer while being wrong about what is in the image. The system is correct for the wrong reasons, and nothing in the final accuracy number tells you so. These are called reasoning shortcuts.
My thesis takes on the harder case: concepts that appear in many different visual forms. A stop signal is a red octagon in one frame, a hand gesture in another, a brake light in a third. Models that latch onto one surface form collapse when the modality shifts.
What I did
- Characterised where reasoning shortcuts emerge when concepts are multi-modal.
- Developed an approach that improves concept-level accuracy, not just task accuracy, using a fraction of the concept supervision these methods normally need.
- Evaluated across multiple established neuro-symbolic frameworks rather than one favourable baseline, reaching a new state of the art on each.
- Held the gains out of distribution, which is the test that separates learning a concept from learning a dataset.
- Validated on real autonomous-driving data, not only synthetic benchmarks.
Engineering
PyTorch, with a reproducible pipeline on A100 GPUs: seeded runs, versioned configs, automated sweeps. Comparing fairly across frameworks meant re-implementing the baselines under one harness rather than trusting published numbers.
The paper is under review at a leading AI conference.
Supervised by Luca Andolfi and Eleonora Giunchiglia.