Interview with Patrick Fernandes
Patrick Fernandes, a CMU Portugal Dual Degree Ph.D. student in Language Technologies (Técnico I CMU), is part of the Portuguese team that developed the European Large Language Model (EuroLLM), launched in September 2024. The initiative was being led in Portugal by their Ph.D. supervisor André Martins, a CMU Portugal Alumni, Researcher at Instituto de Telecomunicações from Instituto Superior Técnico and co-founder and VP of AI Research at Unbabel.
The “EuroLLM: Open Source European Large Language Model” is the most advanced language model of its size developed in Europe to date. Built using the cutting-edge EuroHPC infrastructure, the EuroLLM is a fully open LLM capable of understanding and generating text across all the 24 official European Union (EU) languages, as well as 11 commercially and strategically important international languages.
Let’s take a look at what it means for a 28-year-old Ph.D. student to be involved in this major initiative.
How did the opportunity to work on the development of EuroLLM come about?
I had just returned to Portugal after spending two years at CMU. André told me they were starting a new project for a European LLM and, since I had recently gained experience training and working with large models and distributed systems (from time doing research at Google) he asked me if I could join the project and help out.
- Could you tell us more about your specific role and contributions to this project?
I developed the main codebase and pipelines for the pre-training of EuroLLM, the phase where the model sees massive amounts of data across 30+ languages and learns its core capabilities.
It was actually developed for another LLM that we trained, CroissantLLM, where I led the training of the model. We then reused and adapted the codebase for EuroLLM, but Pedro (the lead author of EuroLLM) took charge of the training runs, and I mostly helped out discussing design decisions.
- How does your participation in EuroLLM align with your Ph.D. research?
My PhD is focused on understanding and aligning models for machine translation. Given the multilingual nature of EuroLLM (and its already exceptional capabilities at machine translation) it will probably serve as a base model for further research I will do in these last few months of my PhD. In particular, I’m currently interested in adding other modalities to EuroLLM (such as vision) and how these other modalities might help models learn new language and translation capabilities.
- How do you see the EuroLLM initiative impacting the accessibility and inclusivity of language technologies in Europe and beyond?
I think EuroLLM is a massively important step for the European AI landscape, in the currently English-&-American centered AI world.
Europe has mostly been a “follower” in recent developments with LLMs, and models like EuroLLM show that Europe can lead and innovate in this field.
It also gives us sovereignty, allowing us to prioritize not only English but all languages in Europe, and increase the inclusivity of language technologies in Europe.
- Looking ahead, how do you envision the future of this field?
I think we are finally getting over the idea that scale is all you need, so I think we will see more and more models that are smaller, but trained on very high-quality data.
But in all honesty, I think the future of this field will not be about new & better models (and whether they will replace us) but about how we can best apply current and future models to new problems and ease the interaction between humans and machines, as there is still *so much* potencial there.
- Has your experience with the CMU Portugal program met your initial expectations?
Overall, my experience in CMU Portugal has been great. I didn’t really know what to expect, but I was lucky to get a pair of advisors that work well together and provided me with a lot of support.
The experience, connections and friends I made in Pittsburgh were also invaluable to my path as a researcher (and I overall loved my time there), but being able to do world-class research while in Portugal (close to family and friends) is something I’m very grateful for.
- You are set to graduate in 2025, and the future looks promising. What are your plans after graduation?
Honestly, I wish I knew 😅. I’m definitely taking a big, multi-month break after my Ph.D. to recharge and think about what I want out of my career (and life in general).
But after that, I might pivot to working on the more applied side of AI, on applications close to my heart (such as AI for drug discovery, or better mental health treatments).
But who knows, let’s see what the future holds 🙂