A health database rarely captures the exact clinical state a researcher wants to study. It contains traces of that state: diagnoses, medication orders, laboratory results, procedures, changes in the level of care, and clinical notes. These records support care, communication, billing, and hospital operations. They were not created for future research questions.
Computable phenotyping turns these traces into an algorithm for identifying patients, clinical events, or periods of interest. The process may sound simple: define a condition, select the relevant codes, and run a query. In practice, researchers must still determine:
A defensible phenotype answers these questions explicitly. It is a measurement model for inferring a partially observed clinical state from data collected during routine care.
The Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) provides a shared framework for this work. It standardizes longitudinal health data, maps source codes to common vocabularies, and supports shareable cohort logic. It does not automate phenotype development or turn imperfect data into clinical truth.
Large language models (LLMs) can read clinical descriptions, organize evidence, propose criteria, review logic, and interpret structured patient profiles. They can also invent terminology, mishandle temporal relationships, overlook local data conventions, and produce plausible explanations for incorrect algorithms.
The central question is which parts of phenotype development can be safely delegated to an LLM, which require deterministic checks, and which remain the responsibility of accountable experts.
OHDSI defines a cohort as people who meet specified criteria for a defined period. A code set is only one component. A complete cohort definition specifies entry, additional eligibility criteria, and exit [1].
Timing is essential. A code recorded during an emergency encounter may represent the current episode, a suspected diagnosis, a complication, a copied historical condition, or follow-up after an earlier event. Its meaning depends on the encounter, related findings and treatments, and their timing.
A computable phenotype therefore includes:
Formal representations should support common data models, standard terminologies, temporal relationships, human-readable and machine-executable logic, reuse, and connections to external algorithms [2]. These features reduce the risk of silent reinterpretation during implementation or reuse.
OMOP gives phenotype development a common grammar. It provides defined structures for conditions, drugs, procedures, measurements, observations, visits, devices, specimens, and other clinical records. Standard concepts create a shared semantic layer across source terminologies, while cohort definitions can be exported from OHDSI ATLAS as Circe-compliant JSON or executable SQL, allowing institutions to reuse the same logic without independently implementing a narrative protocol.
In an eMERGE network study, researchers converted two phenotypes to OMOP and implemented them across participating sites. Once the data were transformed into the common model, sites could execute the shared phenotypes in less than one day instead of spending weeks adapting them to local schemas [3].
This demonstrated implementation portability, but agreement with previous local implementations varied substantially. Laboratory data were particularly difficult because of local encoding. Other discrepancies resulted from differences in phenotype definitions, databases, source data, and earlier implementations [3].
OMOP reduces repeated translation of cohort logic. It does not eliminate differences in recorded data, source mappings, or missing information. The research question becomes portable, but the available evidence may still differ across databases.
A source code may map to a valid standard concept that is too broad, narrow, historically qualified, or unsuitable for the research purpose. Mapping can also remove context. A specific bedside assessment may become a general Measurement or Observation concept; a procedure may be recorded without its indication; and a medication exposure may suggest treatment without proving the condition being studied.
A missing record does not mean that an event did not occur. Patients may receive care outside the captured network, laboratory interfaces may be incomplete, and medication data may represent orders rather than administration. Hospital data may contain detailed inpatient records but little reliable pre-admission history, while claims may omit clinically important details about reimbursed services.
Observation periods indicate when a patient is visible in a database but cannot reconstruct events that were never captured.
Acute and critical-care phenotypes may depend on hours or minutes. Date-level records may support chronic-condition research but fail to show whether treatment preceded deterioration, a signal appeared after admission, or two monitor events belonged to the same episode.
A clinically meaningful criterion cannot always be represented reliably in OMOP. Before implementation, classify each required signal as:
Otherwise, data limitations may become unstated assumptions about the patient’s clinical state.
Conventional OMOP clinical tables support many analyses of structured EHR and claims data but are less suited to continuous waveforms, monitor alarms, dense device telemetry, image-derived features, and other high-frequency data.
Recent critical-care research has explored OMOP extensions that preserve detailed ICU information rather than placing every signal in conventional event tables [4]. Some critical-care questions require representations beyond OMOP’s original transactional structure.
A concept set represents a hypothesis about how a clinical idea appears in data. Concepts may be included directly, through descendants, or through mappings from source vocabularies. Each choice affects which patients and events the phenotype captures.
Including all descendants may improve recall but can also introduce:
Excluding broad concepts may improve apparent precision while missing patients documented with less specific terminology. It may also shift the observed index date to a later stage of the disease.
PHOEBE supports concept-set development with empirical data on code use across 22 EHR and claims databases in six countries. Its recommendations combine semantic and lexical relationships with observed utilization. In the evaluated conditions, PHOEBE-derived sets identified more patients while maintaining a similar proxy positive predictive value, although improvement varied by disease and database [5].
Code frequency does not establish clinical validity. Review should consider clinical meaning, ontology structure, source-to-standard mappings, domain placement, observed utilization, effects on cohort entry, and the reasons for including or excluding each concept.
A complete concept set should have a clinically defensible scope, be empirically reviewed and versioned, and reflect the phenotype’s intended use.
A cohort definition may be technically portable while its performance remains data-source-specific. Hospitals differ in coding density, laboratory interfaces, treatment documentation, observation history, and record-generating workflows. EHR and claims databases capture different information, international databases may use different source terminologies, and even hospitals using the same EHR vendor may populate OMOP differently.
Natural language processing (NLP) creates further local dependencies. In a multisite eMERGE evaluation, phenotype components using structured data and NLP varied because of differences in document types, clinical text, and local implementation [6].
Transportability therefore requires several questions:
Execution is only the technical requirement. The remaining questions require further diagnostics, and the last requires a reference standard or another defensible evaluation design.
A plausible cohort count, successful SQL run, familiar diagnosis list, or developer agreement does not validate a phenotype. Systematic diagnostics should come before case-level evaluation.
CohortDiagnostics analyzes OMOP cohorts through inclusion-rule attrition, source-code and orphan-code inspection, incidence by age, sex, and calendar year, index-event breakdown, cohort overlap, temporal characterization, and patient-profile exploration [7]. It can reveal that:
Population-level diagnostics do not establish sensitivity or specificity. They show whether the phenotype behaves consistently with its clinical and analytical design.
The phrase “LLM-based phenotyping” covers several distinct tasks. Each task requires its own evaluation and controls.
An LLM can read publications, clinical criteria, and protocol text and organize the information into a structured draft containing:
This is useful because phenotype literature is often fragmented. A definition may describe some criteria precisely while leaving onset, exit, recurrence, or observation requirements unclear. The model can help extract and organize this information, but its output should not be treated as an authoritative interpretation of the literature.
An LLM can suggest synonyms, abbreviations, spelling variants, broader terms, and related clinical expressions to improve recall when searching OHDSI Standardized Vocabularies and source-code inventories.
The model should not generate concept identifiers from memory. Reviewers must resolve each candidate against the relevant vocabulary release, verify its standardness, domain, and hierarchical relationships, and document the decision to accept or reject it.
An LLM can also group local source-code labels and identify possible mapping gaps. Any interpretation requires verification against source-system documentation, source-to-concept maps, and observed utilization.
An LLM can compare a conceptual phenotype specification with its human-readable implementation and flag:
This resembles code review rather than clinical validation. The model can identify issues for investigation but cannot determine from text alone whether the database implements the intended logic correctly.
LLMs can draft concept-set rationales, phenotype summaries, change logs, review checklists, and diagnostic narratives. These are relatively low-risk uses when reviewers verify the text against source artifacts. They reduce repetitive writing while clinical authority remains with human reviewers.
Clinical notes may contain symptoms, clinical reasoning, severity, negation, uncertainty, and context that structured codes omit.
LLMs can extract predefined information or classify whether the notes support a phenotype. Performance depends on the task, document type, prompt, model, institution, and reference standard. Results for one symptom or disease should not be generalized to other phenotypes.
KEEPER presents phenotype-relevant structured evidence in a standardized format organized around clinical reasoning. In a four-condition crossover study, reviewing KEEPER profiles was approximately twice as fast as conventional chart review. Patient classifications agreed in 88.1% of cases, and clinician agreement was higher with KEEPER, although results varied by condition and missing data remained important [8].
A later study used LLMs to adjudicate these profiles across ten diseases. Sensitivity ranged from 78% to 98% and specificity from 48% to 98%, with performance varying by model, prompt, disease, and available evidence [9]. The findings support LLM-assisted adjudication but not universally reliable automated review.
Applying an LLM to every patient may be unnecessary and expensive. A selective workflow can use structured-data models first and send only uncertain or likely misclassified cases for note-based analysis.
In a 2026 study of two registry phenotypes, this approach improved sensitivity while limiting LLM use to selected patients. For one phenotype, only about 10% required LLM review [10]. However, the results remain specific to the evaluated diseases, institutions, classifiers, and note-processing pipeline.
LLMs produce contextually plausible language. A concept, threshold, or temporal rule may sound medically reasonable while being absent from the cited source or unsuitable for the intended use.
Phenotypes often depend on exact boundaries: before versus on the index date, within the same encounter, events separated by a minimum interval, no prior exposure during a washout period, or evidence appearing only after a specified duration.
Fluent descriptions can conceal off-by-one errors, inverted intervals, or incorrect time references. Temporal logic should therefore use a structured schema and undergo deterministic testing.
A model cannot reliably infer whether an institution stores a result in MEASUREMENT, OBSERVATION, a source-specific extension, or not at all. Unless verified information is provided, it also cannot know whether timestamps were preserved, medications represent orders or administrations, ICU stays were derived correctly, or laboratory units were normalized.
A fabricated concept ID may resemble a real one, while a valid ID may belong to the wrong vocabulary release, domain, or standardness class. All terminology outputs must be resolved against authoritative OHDSI Standardized Vocabularies tables or services.
An LLM cannot recover facts that were never captured. It may instead create a coherent explanation from incomplete evidence, making missing information less visible.
Model updates, prompt revisions, reordered context, or output-format changes may alter the classification of the same patient profile. Reproducibility therefore requires the prompt template, model version, decoding settings, retrieved context, structured inputs, and parsing logic — not only the model name.
Agreement between an LLM and a phenotype algorithm is not an independent reference standard when both use the same coded evidence or development materials. Their errors may be correlated because the model can reproduce assumptions embedded in the phenotype.
In such cases, agreement may reflect consistency with the narrative definition, concept rationale, or developer documentation rather than the patient’s actual clinical state.
An LLM-assisted workflow requires evaluation separate from that of the phenotype, with metrics based on the model’s role:
Operational evaluation should account for both expert time saved and time spent correcting outputs. It should also examine whether the LLM changes scientific conclusions, particularly for rare outcomes, differential misclassification, or estimates sensitive to phenotype specificity.
Security claims are incomplete without a clear data-flow description. A credible implementation should specify:
Local deployment can reduce disclosure risks, but it does not by itself ensure security, governance, or reproducibility [9].
Many development tasks, including literature synthesis, terminology exploration, specification drafting, documentation, and logic revie, can use public evidence, metadata, synthetic examples, or aggregate diagnostics. Patient-level processing should be introduced only when less sensitive inputs cannot provide the required information.
Phenotype development is iterative. Data assessment, population diagnostics, or case-level evaluation may require revisions to the clinical specification, concept sets, or executable logic before release. An LLM can support selected stages but should not direct the process.
Define whether the phenotype represents a target population, exposure, outcome, comparator, covariate, surveillance signal, model label, feasibility estimate, or another analytical construct.
Specify the unit of inference: a person, event, episode, encounter, or time-varying state. Document the target population, clinical setting, incident or prevalent interpretation, reuse scope, and consequences of false-positive and false-negative classifications.
Where possible, establish operating characteristics or other acceptance criteria in advance. Assign an owner, clinical and methodological reviewers, approval authority, and conditions for acceptance, revision, or rejection. The appropriate sensitivity–specificity balance depends on the intended use and decision context.
Define the phenotype conceptually before choosing its technical implementation. Describe its clinical meaning, target population, care setting, unit of inference, onset or index event, and whether it represents an incident, prevalent, recurrent, or persistent state.
Specify the required, supporting, alternative, exclusionary, and contradictory evidence, together with temporal relationships, observation requirements, episode construction, and cohort exit. Document relevant evidence unavailable in the data, assumptions about missing records, and known limitations.
Keep the underlying clinical state distinct from the EHR records used to infer it. Define whether baseline windows include the index date, whether evidence must occur during the same visit, and how simultaneous or closely spaced events are handled. Record unresolved ambiguities and the decisions addressing them instead of allowing an LLM or implementation code to fill the gaps silently.
Assess whether the required domains, values, timestamps, observation history, and care-setting information are available and reliable. For high-stakes real-world evidence, regulators similarly emphasize evaluating whether EHR or claims data are relevant and reliable for the specific research question [11].
Create a phenotype-specific data-requirement matrix that classifies each signal as mandatory, optional, alternative, or unavailable. Separately, evaluate the following areas for every data source:
Use ACHILLES or ARES for broad source characterization, existing DataQualityDashboard results for systematic quality assessment, and targeted queries for phenotype-specific requirements.
A global data-quality pass rate does not establish that a database is suitable for a particular phenotype. Data fitness depends on the phenotype’s required signals and intended use. If a required signal is unavailable or unreliable, revise the specification, define a justified fallback implementation, restrict the supported data sources, or stop development.
Build concept-set expressions using authoritative OMOP vocabularies, hierarchical relationships, source-to-standard mappings, lexical searches, source-code inventories, and empirical utilization.
For each concept set:
Preserve both the expression and resolved membership for a specified vocabulary release, including its version and release date.
Empirical frequency can reveal missing or rarely used representations but does not establish clinical relevance. An LLM may suggest search terms, synonyms, and candidate concepts, but reviewers must verify all identifiers and relationships against the authoritative vocabulary and approve the final set.
For measurement-based criteria, separately specify tests, value representations, permitted units, conversion rules, categorical values, and plausible ranges.
Represent cohort entry, event restrictions, inclusion rules, temporal relationships, observation requirements, recurrent-event behavior, cohort duration, exit strategy, and era construction in ATLAS JSON, programmatic cohort specifications, or another formally testable format.
Maintain one authoritative specification for generating executable SQL and human-readable logic. Avoid editing generated SQL independently unless any differences are explicitly managed and tested.
Separate reusable phenotype logic from analysis-specific elements such as time at risk, follow-up, competing-risk handling, and analytical censoring. Distinguish executable cohorts from downstream phenotypes, subphenotypes, scores, and probabilistic classifications.
Before database execution, perform static and synthetic tests covering:
Record the CDM and ATLAS/WebAPI versions, cohort-generation tools, SQL dialect, and other execution dependencies.
Before interpreting a generated cohort, verify that the underlying OMOP data support the phenotype. Use targeted DataQualityDashboard checks to assess the tables, fields, concepts, relationships, measurement values, units, dates, and mappings required by the phenotype.
If a failed or non-applicable check affects a required signal, treat it as a limitation of execution or interpretation. It does not show that the clinical signal itself is absent.
Use CohortDiagnostics as the primary tool for population-level phenotype evaluation. Inspect:
Compare alternative definitions where appropriate. Run the same diagnostic package across data sources to identify differences in concept utilization, attrition, population characteristics, data availability, timestamp precision, and ETL behavior.
Diagnostic findings should lead to a documented decision to accept, revise, restrict, or reject the phenotype. Population-level plausibility is necessary, but it does not establish sensitivity, specificity, or positive predictive value.
Design the validation protocol before reviewing any cases. Specify:
Direct reference-standard approaches may include chart review or registry linkage. KEEPER can support more standardized and efficient review by presenting structured patient profiles containing phenotype-relevant evidence. However, these profiles remain limited to the data available in the OMOP source. Reviewing only algorithm-positive cases generally supports estimation of positive predictive value, but not full sensitivity. Estimating sensitivity requires a broader high-sensitivity population, sampled non-cases, registry linkage, or another appropriate design. PheValuator provides model-based probabilistic evaluation rather than independent case-level adjudication.
If an LLM supports or performs adjudication, evaluate that component independently. Fix and record the model, prompt, examples, parameters, input representation, parsing logic, and abstention policy.
Report performance separately by phenotype, data source, and relevant subgroup instead of relying on one aggregate metric. At minimum, include sensitivity, specificity, positive and negative predictive values, sample sizes, sampling weights where applicable, and confidence intervals. Also report inter-reviewer agreement, the frequency of uncertain cases, and limitations of the reference standard.
A released phenotype package should contain: Released versions should be immutable. Material changes should create a new version instead of silently modifying an existing referenced artifact.
Phenotype libraries are most useful when they preserve executable definitions together with provenance, validation evidence, version history, and review records [12].
Define the events that should trigger phenotype reassessment:
After a relevant change, resolve the concept sets again and compare their membership with the previous version. Repeat static tests, targeted data-quality checks, and CohortDiagnostics. Repeat case-level evaluation when the change may affect the phenotype’s clinical meaning or expected operating characteristics.
Document whether the new version corrects an implementation issue, changes analytical behavior, or changes the phenotype’s clinical meaning. Supersede or deprecate versions that are no longer suitable for their documented use.
A reliable OMOP phenotype must do more than run without errors. A reviewer should be able to answer:
LLMs can help experts expose, organize, inspect, and document this complexity more efficiently. Responsibility for the phenotype remains distributed across the full development and evaluation process:
An LLM can assist at several points, but the accountable roles and evaluation steps must remain explicit.
Computable phenotyping converts a clinical idea into an explicit inference based on imperfect observational data.
OMOP makes this inference easier to represent, execute, compare, and reuse. It reduces repeated implementation work and provides a common vocabulary for distributed research. However, phenotype quality still depends on concept selection, data completeness, temporal precision, and consistent implementation across institutions.
LLMs can support evidence synthesis, specification drafting, terminology exploration, documentation, logic review, evidence extraction from clinical notes, and case adjudication. Their limitations remain significant: generated logic may be too broad or restrictive, adjudication performance varies by phenotype and prompt, and models cannot reliably compensate for missing data or undocumented ETL behavior [7,9].
A defensible workflow assigns each task to the appropriate method. LLMs support language-intensive work, deterministic tools enforce precise logic, empirical diagnostics reveal phenotype behavior, and accountable experts decide whether the result is clinically and methodologically fit for purpose.
