A comparison of two approaches to large scale biomedical literature classification: LLM based classification and a hybrid architecture combining named entity recognition, ontology embeddings, and disease normalization. Using 38.6 million PubMed records, we evaluated throughput, cost, ontology alignment, and traceability, finding that hybrid pipelines remain the more practical choice for production scale, ontology driven systems.
Key engineering takeaway
At 38.6M records, classification was not the hardest problem. Ontology alignment and consistent normalization were.
Overview
Large language models have expanded the range of tasks that can be automated in text analysis. However, their suitability for large-scale scientific data classification remains limited in practice. Many enterprise use cases require strict ontology alignment, predictable throughput, and traceable outputs.
This paper examines two approaches to disease classification in the biomedical literature: LLM-based classification and hybrid pipelines that combine named-entity recognition with ontology embedding. We use a dataset of 38.6 million PubMed records to evaluate these methods, focusing on throughput, cost, ontology alignment, and error characteristics.
The results indicate that hybrid NER–ontology pipelines remain the more practical architecture for production-scale classification systems.
Why is large-scale biomedical classification different
Automated classification of biomedical literature underpins a wide range of modern research systems, from disease-landscape mapping to clinical-intelligence platforms. These systems rely on the ability to convert unstructured text into structured entities linked to standardised ontologies.
At a small scale, this task can be approached using flexible text processing methods. At a large scale, the problem changes. The volume of data, the variability of terminology, and the need for consistent entity identity introduce constraints that require different architectural choices.
The PubMed corpus currently contains more than 38 million publications. However, processing this dataset is not only about model accuracy but also about throughput, cost, and system design.
Recent advances in large language models have made them a natural candidate for text classification tasks. At the same time, their use in large-scale, ontology-driven systems introduces tradeoffs that are often overlooked.
This article compares LLM-based classification with hybrid NER–ontology pipelines. It then focuses on how these approaches behave in production-scale environments.
The challenge isn't extraction. It's normalisation.
The real challenge does not lie in entity extraction, but in successfully normalising entities at scale. Imagine the impact when normalisation is achieved efficiently—it transforms data usability and drives meaningful outcomes.
Biomedical terminology is inherently inconsistent, posing daily challenges. The same disease may appear under multiple names, abbreviations, or contextual descriptions, disrupting analysis. Without normalisation to a shared ontology, these variations fragment the dataset and hinder deeper insights. Imagine trying to answer critical questions when key data is scattered under different terms—it's like chasing shadows.
For example:
- ALS
- Amyotrophic lateral sclerosis
- Lou Gehrig's disease
All these names point to the same disease. Classification systems must cut through ambiguity by mapping every mention to a single, definitive identifier. Each mapping is a step toward clarity.
This introduces three constraints:
Consistency
Each mention must resolve to the same identifier across the dataset.
Scalability
The system must process tens of millions of documents without degradation in performance.
Traceability
Each classification decision must be explainable and auditable.
These constraints define the requirements for a production-grade classification architecture.
How the classification pipeline works
The hybrid pipeline takes on the challenge of separating entity detection from entity normalisation, ensuring each step receives focused attention and precision.
The system consists of four main components: named entity recognition, ontology embedding, entity normalisation and distributed processing.
1 Named Entity Recognition
Discover how disease mentions are extracted at the sentence level using domain-trained models. Models such as BioBERT, designed for biomedical corpora, capture domain-specific terminology with greater accuracy than general-purpose models, highlighting the importance of specialised tools in this field.
2 Ontology Embedding
Extracted entities transform into vectors aligned with a fixed ontology. SapBERT maps words to concepts—turning raw mentions into precise coordinates in knowledge space.
3 Entity Normalisation
As soon as an entity is embedded, it is eagerly compared against a pre-indexed ontology. The system then swiftly retrieves the most similar concepts and enthusiastically assigns a canonical identifier to them, guaranteeing precise connections are made.
This step guarantees that all references to the same disease converge to a single ontology entry.
4 Distributed Processing
To handle large datasets, the pipeline runs distributed processing in chunks with checkpointing for resilience. Large-scale analysis moves forward seamlessly, no matter the hurdles.
This allows the system to process tens of millions of records without manual intervention.
Processing 38.6 million PubMed records
The classification pipeline was tested on the full PubMed dataset through June 2025. Millions of documents. Each was an opportunity—each process, a step toward understanding.
Input data consisted of publication titles and abstracts. Each document was processed independently.
The pipeline followed these steps:
- Sentence-level NER to detect disease mentions.
- Embedding of detected entities using SapBERT
- Similarity search against an indexed ontology
- The selection of the top candidate matches the similarity scores.
- Storage of normalised entities and metadata
Imagine a system that, instead of assigning a single label, keeps multiple candidate mappings. This empowers downstream applications to apply different confidence thresholds and discover greater insights through co-occurrence analysis.
The PubMed dataset and MONDO ontology
The dataset consists of 38.6 million PubMed records, including titles and abstracts.
Disease classification was performed using the MONDO ontology, which provides a unified representation of disease concepts across multiple biomedical vocabularies.
By combining broad PubMed coverage with MONDO, the system untangles language variation, mapping shifting expressions to stable disease identifiers. This is the foundation of meaningful biomedical analysis.
Results at scale
Our pipeline enabled data exploration by connecting publications to normalised disease concepts, creating a structured dataset.
For example, one analysis spotlighted nervous system disorders—revealing trends hidden in plain sight.
The system identified:
- 5,092,286 publications
- 3,343 distinct MONDO-classified diseases.
Further analysis revealed that:
45.5% of these diseases showed limited mechanistic research despite substantial clinical characterisation
Unlocking insights at the dataset level requires large-scale processing and consistent ontology alignment. Only with thorough normalisation can these patterns become visible and impactful.
LLMs vs hybrid pipelines
LLM-based classification systems offer flexibility; however, they behave differently at scale.
Throughput is specifically constrained by API latency and rate limits, which challenge LLMs when processing tens of millions of records. Imagine attempting to sift through a sea of data at high speed, only to be slowed by API barriers, which create a significant bottleneck for large-scale operations.
As token usage increases, so does cost, which can impact your project's budget at the dataset scale.
Entity normalisation prompts additional engagement through additional prompting or post-processing, which increases system complexity.
Outputs from LLMs are probabilistic, which can lead to inconsistent mapping to ontology identifiers. Consistency is always at risk, demanding vigilance.
On the other hand, hybrid pipelines unlock reliable results by pairing similarity scoring, native ontology integration, and consistent operational behaviour—empowering teams to navigate data landscapes with confidence.
The following summary highlights these differences.
What these results mean in practice
Hybrid pipelines excel where classification must be consistent, aligned, auditable, and scalable. Each requirement protects trust, progress, and discovery.
Although LLMs have limitations, they remain useful for complementary tasks such as summarisation, exploratory analysis, and interaction layers.
In practice, you can combine these two approaches. Hybrid pipelines offer a reliable backbone, while LLMs unleash creative, higher-level reasoning.
Key takeaways
Large-scale biomedical classification calls for more than just accurate text interpretation—it demands rigorous, consistent mapping to structured knowledge systems, underscoring its vital role in advancing the field.
While large language models provide flexibility, their operational characteristics can hinder performance, especially when navigating the demanding pace and intricate requirements of high-volume, ontology-driven environments.
Hybrid NER–ontology pipelines tackle these challenges directly, fusing domain-specific models with deterministic normalisation and scalable processing to deliver robust, efficient solutions.
For datasets as large as PubMed, hybrid pipelines remain the practical foundation for production-ready systems.
From research to production systems
As part of a transformative initiative, this system was designed and implemented to enable large-scale platforms for literature intelligence and ontology-driven analytics, facilitating users to discover insights and accelerate research.
If your work aligns with these challenges, adapt these architectural strategies to other domains. Each new application is a step in advancing structured data analysis.