Oct. 30, 2026
How OpenEuroLLM Selects and Prepares Training Data
Most large language models (LLMs) today are built on close variations of the same architecture. What increasingly separates one model from another is the data each one is trained on: what went in, how clean it was, and how well it covered the languages and domains the model needs to handle. A model has no knowledge beyond what its training data contains. Clean, well-selected data improves what it can do, noisy or narrow data lowers it.
We are OpenEuroLLM, a project building open, multilingual language models for Europe, and for us, this makes dataset selection one of the most significant decisions in the entire process. We span dozens of languages, many with far less text available online than English, and commit to transparency about what goes into our models and how they were built. That combination makes certain questions harder to avoid, such as where the data comes from and what makes it good enough to use.
Where Does the Data Come From?
The main source is usually the open web. Proprietary datasets are a possible alternative, but our commitment to full openness rules it out, a real constraint when text is scarce for a given language. Open datasets and public repositories, like Common Crawl and HPLT, let projects build on each other's work instead of starting from scratch. We maintain our own shared catalogue on top of this.
When real text runs short, two things help fill the gap. Synthetic translated data is one: MultiSynt, a EuroHPC-funded initiative running on CINECA's Leonardo supercomputer, supports both us and the related EuroLLM project to improve language representation and diversity where real text is scarce. The other is non-web material, text extracted or transcribed from permissively licensed PDFs, office documents, audio, and video pulled from public repositories. We make a dedicated effort to gather this material, aiming to eventually cover all our target languages.
What to Consider When Selecting
More data is not automatically better data.
Quality, representativeness, and compliance decide whether a dataset earns a place in training. We use existing metadata, and score documents with purpose-built classifiers, one rating educational value, another assessing a broader set of properties. We validate those against external quality benchmarks, aiming for a filter that removes weak text without cutting so deep that too little volume is left to train on.
Representativeness matters alongside quality too: a dataset built by volume alone would overrepresent the languages and domains already dominant online, so we also select to preserve linguistic diversity.
Compliance adds a non-negotiable layer: data must be usable under the regulatory framework that applies. In practice we check documents against mechanisms like robots.txt or blocklists like the Common Crawl Opt-Out Registry, and work through what obligations under copyright law, the AI Act, and the GDPR mean for how the data can be used. Documenting the provenance of the datasets selected to train our models or masking personal data are among the activities we perform to comply with EU rules. Scoring well on quality doesn't exempt a document from being excluded here.
Preparing and Decontaminating the Data
Once data clears selection, two more decisions set what the model learns from it. First, there's a hard ceiling on how much data a model can usefully see, set by its size and the compute available to train it. Within that ceiling, we decide the data mix: what fills it, the proportion of each natural language and category (math, code and reasoning text), and the source, in the final training set. These proportions are motivated by project goals and grounded in prior experimentation work where different distributions are tested and evaluated. In practice, this means we set a target token count for each language and source.
We close out the process with two checks. Decontamination removes evaluation texts leaked into the training data, since a model that has already seen the answers can look more capable than it is. Finally, before any data reaches the model, we break it into tokens, small chunks of language represented as numbers, with training happening in batches of these tokenized examples. Tokenization is a recurrent activity in every data experiment, and the last step in the data preparation for final training.
Why This Matters At Scale
All of this, quality scoring, representativeness, compliance, the data composition, decontamination, etc. feeds into what we plan to publish. Alongside each model, tokenizers and model weights, we will also share the training texts for every model in the family. With dozens of languages and sources behind every model, that kind of documentation is what will make the earlier steps verifiable and open.All of this, quality scoring, representativeness, compliance, the data composition, decontamination, etc. feeds into what we plan to publish. Alongside each model, tokenizers and model weights, we will also share the training texts for every model in the family. With dozens of languages and sources behind every model, that kind of documentation is what will make the earlier steps verifiable and open.
The result will be a model family whose training data can be traced back to a dataset, a language, and a specific filtering step, not just a benchmark score: an important detail for us as an open, multilingual project.
We are OpenEuroLLM, a project building open, multilingual language models for Europe, and for us, this makes dataset selection one of the most significant decisions in the entire process. We span dozens of languages, many with far less text available online than English, and commit to transparency about what goes into our models and how they were built. That combination makes certain questions harder to avoid, such as where the data comes from and what makes it good enough to use.
Where Does the Data Come From?
The main source is usually the open web. Proprietary datasets are a possible alternative, but our commitment to full openness rules it out, a real constraint when text is scarce for a given language. Open datasets and public repositories, like Common Crawl and HPLT, let projects build on each other's work instead of starting from scratch. We maintain our own shared catalogue on top of this.
When real text runs short, two things help fill the gap. Synthetic translated data is one: MultiSynt, a EuroHPC-funded initiative running on CINECA's Leonardo supercomputer, supports both us and the related EuroLLM project to improve language representation and diversity where real text is scarce. The other is non-web material, text extracted or transcribed from permissively licensed PDFs, office documents, audio, and video pulled from public repositories. We make a dedicated effort to gather this material, aiming to eventually cover all our target languages.
What to Consider When Selecting
More data is not automatically better data.
Quality, representativeness, and compliance decide whether a dataset earns a place in training. We use existing metadata, and score documents with purpose-built classifiers, one rating educational value, another assessing a broader set of properties. We validate those against external quality benchmarks, aiming for a filter that removes weak text without cutting so deep that too little volume is left to train on.
Representativeness matters alongside quality too: a dataset built by volume alone would overrepresent the languages and domains already dominant online, so we also select to preserve linguistic diversity.
Compliance adds a non-negotiable layer: data must be usable under the regulatory framework that applies. In practice we check documents against mechanisms like robots.txt or blocklists like the Common Crawl Opt-Out Registry, and work through what obligations under copyright law, the AI Act, and the GDPR mean for how the data can be used. Documenting the provenance of the datasets selected to train our models or masking personal data are among the activities we perform to comply with EU rules. Scoring well on quality doesn't exempt a document from being excluded here.
Preparing and Decontaminating the Data
Once data clears selection, two more decisions set what the model learns from it. First, there's a hard ceiling on how much data a model can usefully see, set by its size and the compute available to train it. Within that ceiling, we decide the data mix: what fills it, the proportion of each natural language and category (math, code and reasoning text), and the source, in the final training set. These proportions are motivated by project goals and grounded in prior experimentation work where different distributions are tested and evaluated. In practice, this means we set a target token count for each language and source.
We close out the process with two checks. Decontamination removes evaluation texts leaked into the training data, since a model that has already seen the answers can look more capable than it is. Finally, before any data reaches the model, we break it into tokens, small chunks of language represented as numbers, with training happening in batches of these tokenized examples. Tokenization is a recurrent activity in every data experiment, and the last step in the data preparation for final training.
Why This Matters At Scale
All of this, quality scoring, representativeness, compliance, the data composition, decontamination, etc. feeds into what we plan to publish. Alongside each model, tokenizers and model weights, we will also share the training texts for every model in the family. With dozens of languages and sources behind every model, that kind of documentation is what will make the earlier steps verifiable and open.All of this, quality scoring, representativeness, compliance, the data composition, decontamination, etc. feeds into what we plan to publish. Alongside each model, tokenizers and model weights, we will also share the training texts for every model in the family. With dozens of languages and sources behind every model, that kind of documentation is what will make the earlier steps verifiable and open.
The result will be a model family whose training data can be traced back to a dataset, a language, and a specific filtering step, not just a benchmark score: an important detail for us as an open, multilingual project.