The vast majority of clinical trials fail to meet their patient recruitment goal. NIH has estimated that 80% of clinical trials fail to meet their patient recruitment timeline and, more critically, many (or most) fail to recruit the minimum number of patients to power the study as originally anticipated. Efficient patient trial recruitment is thus one of the major barriers to medical research, both delaying trials and forcing others to terminate entirely.
An important solution to this problem is to utilize the vast amounts of patient data that is already available in the form of the electronic health record (EHR). EHRs maintain medical records for routine medical care, but their secondary use for research such as clinical trial recruitment is well-known (Hersh, 2007). This was part of the inspiration for the TREC Medical Records track (2011-2012) (Voorhees and Hersh, 2012). However, that track was ultimately discontinued due to the difficulty in obtaining an EHR dataset of sufficient size (due to privacy issues) to merit a reasonable evaluation. The TREC Clinical Trials track flips the trial-to-patients paradigm to a patient-to-trials paradigm to enable the evaluation of patient matching systems and the building of a test collection for clinical trial search. That is, the query/topic will be (synthetic) patient descriptions and the corpus will be a large set of clinical trial descriptions.
Participants of the track will be challenged with retrieving clinical trials from ClinicalTrials.gov, a required registry for clinical trials in the United States. Clinical trial descriptions can be quite long, but the core aspect of the trial description are the inclusion/exclusion criteria. These are not all-inclusive statements about the trial to the point that other trial information can be ignored, but they are key aspects to defining trial eligibility. The topics present a lengthy (5-10 sentence) patient case description that simulates an admission statement in an EHR. The evaluation will further be broken down into eligible, excludes, and not relevant to allow retrieval methods to distinguish between patients that do not have sufficient information to qualify for the trial (not relevant) and those that are explicitly excluded (excludes). The topics are limited to just the free text description of a patient record, as the structured data in EHRs, while helpful, is more routinely used for clinical trial matching and therefore better-studied.
| Date | Note |
|---|---|
| April 28, 2021 | Document collection available for download |
| May 19, 2021 | Topics available for download |
| May 28, 2021 | Applications for participation in TREC 2021 due (contact organizers thereafter) |
| August 18, 2021 | Submission deadline |
| October 2021 | Relevance judgments and individual evaluation scores released |
| November 17–19, 2021 | TREC 2021 conference at NIST in Gaithersburg, MD, USA (maybe) |
Clinical Trials: An April 27, 2021 snapshot of ClinicalTrials.gov will be used as the corpus.
The files are formatted using the ClinicalTrials.gov XML schema.
The topics for the track consist of synthetic patient cases created by individuals with medical training. The topics consist of a synthetic case in the form of an admission note. Take, for example, these synthetic case descriptions from the TREC Clinical Decision Support track:
A 2-year-old boy is brought to the emergency department by his parents for 5 days of high fever and irritability. The physical exam reveals conjunctivitis, strawberry tongue, inflammation of the hands and feet, desquamation of the skin of the fingers and toes, and cervical lymphadenopathy with the smallest node at 1.5 cm. The abdominal exam demonstrates tenderness and enlarged liver. Laboratory tests report elevated alanine aminotransferase, white blood cell count of 17,580/mm, albumin 2.1 g/dL, C-reactive protein 4.5 mg, erythrocyte sedimentation rate 60 mm/h, mild normochromic, normocytic anemia, and leukocytes in urine of 20/mL with no bacteria identified. The echocardiogram shows moderate dilation of the coronary arteries with possible coronary artery aneurysm.
A 75F with a PMHx significant for severe PVD, CAD, DM, and CKD presented after being found down unresponsive at home. She was found to be hypoglycemic to 29 with hypotension and bradycardia. Her hypotension and confusion improved with hydration. She had a positive UA which eventually grew klebsiella. She had temp 96.3, respiratory rate 22, BP 102/26, a leukocytosis to 18 and a creatinine of 6 (baseline 2). Pt has blood cultures positive for group A streptococcus. On the day of transfer her blood pressure dropped to the 60s. She was anuric throughout the day. She received 80mg IV solumedrol this morning in the setting of low BPs and rare eos in urine. On arrival to the MICU pt was awake but drowsy. On ROS, pt denies pain, lightheadedness, headache, neck pain, sore throat, recent illness or sick contacts, cough, shortness of breath, chest discomfort, heartburn, abd pain, n/v, diarrhea, constipation, dysuria. Is a poor historian regarding how long she has had a rash on her legs.
We plan to release between 30 and 50 topics.
<topics task="2021 TREC Clinical Trials">
<topic number="-1">
A 2-year-old boy is brought to the emergency department by his parents for 5 days of high fever
and irritability. The physical exam reveals conjunctivitis, strawberry tongue, inflammation of
the hands and feet, desquamation of the skin of the fingers and toes, and cervical
lymphadenopathy with the smallest node at 1.5 cm. The abdominal exam demonstrates tenderness
and enlarged liver. Laboratory tests report elevated alanine aminotransferase, white blood cell
count of 17,580/mm, albumin 2.1 g/dL, C-reactive protein 4.5 mg, erythrocyte sedimentation rate
60 mm/h, mild normochromic, normocytic anemia, and leukocytes in urine of 20/mL with no bacteria
identified. The echocardiogram shows moderate dilation of the coronary arteries with possible
coronary artery aneurysm.
</topic>
</topics>
A similar setup was used by Koopman & Zuccon (SIGIR 2016). Their data has a limited number of judged results and may be of use to participants.
To see additional examples of clinical terminology in a case-like format, the TREC Clinical Decision Support 2014-2016 topics might be useful:
The evaluation will follow standard TREC evaluation procedures for ad hoc retrieval tasks. Participants may submit a maximum of five automatic or manual runs, each consisting of a ranked list of up to one thousand IDs (NCT IDs provided by ClinicalTrials.gov). The highest ranked results for each topic will be pooled and judged by physicians trained in medical informatics. Assessors will be instructed to judge trials as either eligible (patient meets inclusion criteria and exclusion criteria do not apply), excluded (patient meets inclusion criteria, but is excluded on the grounds of the trial's exclusion criteria), or not relevant. Because we plan to use a graded relevance scale, the performance of the retrieval submissions will be measured using normalized discounted cumulative gain (NDCG).
As in past evaluations of medically-oriented TREC tracks, we are fortunate to have the assessment conducted by the Department of Medical Informatics of the Oregon Health and Science University (OHSU). We are extremely grateful for their participation.
The tentative submission deadline will be August, 2021.
The format for run submissions follows the standard trec_eval format. Each line of the submission file should follow the form:
where TOPIC_NO is the topic number (1–30), 0 is a required but ignored constant, ID is the identifier of the retrieved document (PMID or NCT ID), RANK is the rank (1–1000) of the retrieved document, SCORE is a floating point value representing the confidence score of the document, and RUN_NAME is an identifier for the run. The RUN_NAME is limited to 12 alphanumeric characters (no punctuation).
The file is assumed to be sorted numerically by TOPIC_NO, and SCORE is assumed to be greater for documents that should be retrieved first. For example, the following would be a valid line of a run submission file:
The above line indicates that the run named "my-run" retrieves for topic number 1 document NCT00760162 at rank 1 with a score of 0.9999.