DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain
In: Fourteenth Language Resources and Evaluation Conference (LREC-COLING 2024) ; https://hal.science/hal-04470938 ; Fourteenth Language Resources and Evaluation Conference (LREC-COLING 2024), Nicoletta Calzolari; Min-Yen Kan, May 2024, Torino, Italy, 2024
Online
Konferenz
Zugriff:
International audience ; The biomedical domain has sparked a significant interest in the field of Natural Language Processing (NLP), which has seen substantial advancements with pre-trained language models (PLMs). However, comparing these models has proven challenging due to variations in evaluation protocols across different models. A fair solution is to aggregate diverse downstream tasks into a benchmark, allowing for the assessment of intrinsic PLMs qualities from various perspectives. Although still limited to few languages, this initiative has been undertaken in the biomedical field, notably English and Chinese. This limitation hampers the evaluation of the latest French biomedical models, as they are either assessed on a minimal number of tasks with non-standardized protocols or evaluated using general downstream tasks. To bridge this research gap and account for the unique sensitivities of French, we present the first-ever publicly available French biomedical language understanding benchmark called DrBenchmark. It encompasses 20 diversified tasks, including named-entity recognition, part-of-speech tagging, question-answering, semantic textual similarity, and classification. We evaluate 8 state-of-the-art pre-trained masked language models (MLMs) on general and biomedical-specific data, as well as English specific MLMs to assess their cross-lingual capabilities. Our experiments reveal that no single model excels across all tasks, while generalist models are sometimes still competitive.
Titel: |
DrBenchmark: A Large Language Understanding Evaluation Benchmark for French Biomedical Domain
|
---|---|
Autor/in / Beteiligte Person: | Labrak, Yanis ; Bazoge, Adrien ; El Khettari, Oumaima ; Rouvier, Mickaël ; Constant Dit Beaufils, Pacôme ; Grabar, Natalia ; Daille, Béatrice ; Quiniou, Solen ; Morin, Emmanuel ; Gourraud, Pierre‐antoine ; Dufour, Richard ; Laboratoire Informatique d'Avignon (LIA) ; Avignon Université (AU)-Centre d'Enseignement et de Recherche en Informatique -, CERI ; Zenidoc ; Service de Santé publique - Clinique des données CHU Nantes (Pôle Hospitalo-Universitaire 11) ; Centre Hospitalier Universitaire de Nantes = Nantes University Hospital (CHU Nantes) ; Traitement Automatique du Langage Naturel (LS2N - équipe TALN ) ; Laboratoire des Sciences du Numérique de Nantes (LS2N) ; Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique) ; Institut Mines-Télécom Paris (IMT)-Institut Mines-Télécom Paris (IMT)-NANTES UNIVERSITÉ - École Centrale de Nantes (Nantes Univ - ECN) ; Nantes Université (Nantes Univ)-Nantes Université (Nantes Univ)-Nantes université - UFR des Sciences et des Techniques (Nantes univ - UFR ST) ; Nantes Université - pôle Sciences et technologie ; Nantes Université (Nantes Univ)-Nantes Université (Nantes Univ)-Nantes Université - pôle Sciences et technologie ; Nantes Université (Nantes Univ)-Institut National de Recherche en Informatique et en Automatique (Inria)-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique) ; Nantes Université (Nantes Univ) ; Savoirs, Textes, Langage (STL) - UMR 8163 (STL) ; Université de Lille-Centre National de la Recherche Scientifique (CNRS) ; Calzolari, Nicoletta ; Kan, Min-Yen ; ANR-23-IAS1-0005,MALADES,Grands modèles de langue adaptables et souverains pour le domaine médical français(2023) ; ANR-20-THIA-0011,AIby4,AI by / for Human, Health and Industry(2020) |
Link: | |
Zeitschrift: | Fourteenth Language Resources and Evaluation Conference (LREC-COLING 2024) ; https://hal.science/hal-04470938 ; Fourteenth Language Resources and Evaluation Conference (LREC-COLING 2024), Nicoletta Calzolari; Min-Yen Kan, May 2024, Torino, Italy, 2024 |
Veröffentlichung: | HAL CCSD, 2024 |
Medientyp: | Konferenz |
Schlagwort: |
|
Sonstiges: |
|