Introduction

At Lifen, we build applications that enable healthcare facilities and practitioners to exchange medical documents seamlessly, routing and structuring millions of letters, reports, and clinical notes every month.

As part of this pipeline, every document that flows through our platform needs to be parsed: key pieces of information must be extracted and structured before the document can be properly handled. One of the most critical pieces of information to detect is the patient’s identity.

For a long time, our patient extraction pipeline relied on a Bidirectional Long Short Term Memory (BiLSTM) based model. Recurrent architectures like the LSTM were, for years, the natural choice for sequence labelling tasks: they were relatively lightweight, fast at inference, and once trained on enough annotated medical documents, reasonably effective at spotting patient entities within the noisy, heterogeneous text found in medical documents.

However, since the rise of Large Language Models (LLMs), a new path with a huge performance upgrade was opened. They have an ability to reason over context, generalize more easily from a handful of examples, and are able to overcome many of the LSTM’s pain points.

This article tells the story of that transition: how we moved to a generative LLM for patient extraction from medical documents, and what possible choices of configurations we chose from to achieve the best possible performance.

Time to Switch to Generative LLMs

Our task consists of extracting a patient’s info (first name, last name, birth name, date of birth) in a structured way from a medical document.

Our last model, BiLSTM, was capable of doing so, however, by introducing a lot of overhead: text normalization, word matching, and training word embeddings from scratch.

As you can see, considering a BiLSTM encoder for extraction introduces some level of complexity, especially when a patient’s name spans multiple tokens and requires manual stitching.

Which is why, for the second iteration of our “Patient” algorithm, we wanted to go for something that could, in theory, work out of the box. Naturally, we chose to go for an LLM decoder architecture for this task. Although encoders (BERT-like) are strong models that capture a lot of semantics at token-level, there is nothing more elegant than having one model that generates the correct answer directly, without any post-processing necessary.

Since LLMs are getting much of today’s attention, getting smarter at each generation, being actively maintained by a huge community that keeps getting bigger: they were the clear winner for this task.

At last, we reformulated our objective as such: Given a medical document as input, the LLM should generate a patient’s first name, last name, birth name, and date of birth as output.

LLM structured output

LLMs are generative models, they output free form text, as they were trained on natural language. However, since we are trying to extract specific information from a free form text, our task requires structured output. Therefore, we need to be able to control the LLM in its generation process so that the output matches the format we expect.

Structured generation has been a massive focus for the AI community these past few years. We’ve seen a lot of brilliant projects tackling structured LLM output, with a strong showing from the open-source community. Here are a few standouts:

• outlines: This framework focuses on generation-time structuring. It modifies the model’s token probabilities at each step to strictly enforce adherence to a regex, JSON schema, or context-free grammar.

• guidance: Another powerful generation-time structuring tool. It constrains the model’s output in real time, ensuring the LLM never strays from the predefined format.

• instructor: Operating primarily on post-generation structure adherence, this library leverages Pydantic to orchestrate the generation process. It utilizes function calling or JSON modes under the hood, automatically validating the output against data models and employing self-correction loops to fix any formatting errors.

Even in the closed API ecosystem, major actors (such as OpenAI, Anthropic, Mistral AI) understood the importance of structured output, which is why they all added it as a feature for their LLM based APIs:

• OpenAI initiatives: They initially introduced Function Calling in 2023, where users provided a JSON Schema of a function, and the model was trained to output a JSON object matching that schema. However, it didn’t strictly guarantee valid formatting. Then came structured output in 2024 to guarantee 100% adherence to the supplied JSON schema.

• Anthropic initiatives: They originally relied on prompting techniques (like forcing the model to output between XML tags) before officially implementing native Tool Use (Function Calling) for Claude, and later introducing official support for forced JSON object responses to match strict schemas.

• Mistral AI initiatives: They followed a similar trajectory, starting with native Function Calling and a standard JSON Mode which forced the output to be valid JSON but did not strictly guarantee adherence to a specific schema. They later upgraded this by rolling out official Custom Structured Outputs, much like the other players.

While API-level structured outputs are highly effective, two other classic approaches remain relevant. Prompt engineering is often the quickest and lowest-effort solution for forcing a structured format purely through instructions. However, it is inherently less reliable and less efficient than native API constraints, as models can still occasionally break formatting rules.

On the other end of the spectrum, fine-tuning provides a highly robust solution for structured generation. By training the LLM on a large dataset that consistently uses the desired schema, the model inherently learns to replicate that exact structure during inference. Consequently, in addition to mastering the core task, such as extracting a patient’s identity, the model naturally learns to format its response in the required structure without relying on complex prompts.

In our use case, we chose to fine-tune the LLM in our specific format, since it was the logical step to move away from the BiLSTM pipeline. However, at the scale of millions of documents processed per month, latency and throughput become hard constraints. Patient LLM runs synchronously during document ingestion, meaning the extraction happens while the user is actively waiting.

This is why, beyond getting the right answer, we also needed to optimize inference speed, and that optimization drives the next two design choices: output format and quantization.

LLM Output format

In our experiments, CSV refers to a compact, comma-separated output format where values are emitted in a fixed order, without repeating the field names at every prediction. For example, instead of generating a JSON object such as:

{
  "first_name": "John",
  "last_name": "Doe",
  "birth_name": null,
  "birth_date": "1970-01-01"
}

the model generates the same information as:

John,Doe,,1970-01-01

This format is close in spirit to TOON, which has recently become popular for LLM inference because it reduces the number of tokens needed for structured outputs. In our case, we take the idea one step further: since the schema is fixed and known in advance, the keys do not need to be generated at all. The output position itself carries the meaning of each value: first name, last name, birth name, then date of birth.

This makes CSV particularly well suited to fine-tuned models. During training, the model sees the same field order over and over again, so it learns to reliably produce values in that exact order. Compared to JSON, this avoids repeatedly generating field names, quotes, braces, and other structural tokens. The resulting output is shorter, which directly helps reduce latency and improve throughput, while keeping the same semantic information.

Quantization

As suggested in their name, LLMs are made up of a huge number of parameters, usually in the billions. Because their generation process is auto-regressive, without further optimization, the model requires close to a second to generate a full prediction (in our case, the list of patient attributes).

Originally, each one of these billions of parameters is encoded in 32-bit floating-point precision (FP32). While we do not intend to reduce the total number of operations, what we can do instead is reduce the numerical precision of the data being processed.

And this is where quantization kicks in. Instead of storing and calculating these parameters in FP32 , quantization reduces their bit-width to a lower format (like FP16, INT8, or even INT4). This drastically reduces the hardware workload and memory bandwidth bottlenecks. In summary, the advantages of quantization are usually two-fold:

• Inference Speed: Because the hardware can process smaller precision types much faster than FP32, the model generates outputs with much lower latency.

• Memory Footprint: Cutting a model’s precision reduces the storage and RAM requirements, allowing massive models to run on much smaller GPUs than originally required.

Reducing the precision of a model’s floating point can lead to lose part of an LLM’s capabilities. To achieve the benefits of using a LLM without destroying the model’s intelligence, advanced Post-Training Quantization (PTQ) methods can be used. Two of the most popular and effective techniques for LLMs are GPTQ and AWQ.

• GPTQ (Generalized Post-Training Quantization): This is a layer-by-layer calibration method. It compresses the model’s weights by treating the quantization process as an optimization problem, minimizing the error in the output for a small set of calibration data. It is highly efficient and allows massive models to be shrunk down to 4-bit precision while retaining remarkably high accuracy.

• AWQ (Activation-aware Weight Quantization): AWQ takes a different approach by recognizing that not all weights are created equal. It observes that a small percentage (about 1%) of the model’s activations are “salient” (extremely important) to the model’s performance. Instead of quantizing everything equally, AWQ handles the most important weights with extra care, so they keep more of their accuracy when compressed, resulting in higher overall accuracy.

Since quantization requires reducing the precision of our parameters, it means re-scaling the values into the correct lower-precision range. With that in mind, there are two main approaches:

• Dynamic: The re-scaling factors for activations are calculated on the fly at inference time based on real-time data

• Static: The re-scaling factors for both weights and activations are pre-calculated using a calibration dataset, meaning no scaling overhead happens during inference

Naturally, dynamic quantization is often more precise because the scaling adapts to real-time inputs. However, calculating these scaling factors on the fly adds a computational overhead, which can slow down the model’s forward pass compared to static quantization.

Reducing a model’s numerical precision inevitably comes with a cost: quantization can degrade the model’s capabilities, and we need to carefully measure this degradation to find the right balance between inference speed and output quality.

In our use case, we decided to benchmark a limited set of these quantization methods. We use llmcompressor library to compress our LLMs, and chose to try the following quantization types: AWQ, FP8_DYNAMIC. Both quantization methods will be benchmarked in terms of performance and latency.

LLM serving

Again, for serving purposes, open-source ecosystems have emerged to handle LLM serving. Here is a quick look at the major actors dominating the space:

• vLLM: vLLM revolutionized serving with PagedAttention, an algorithm that prevents memory waste by managing the Key-Value (KV) cache like virtual memory in operating systems.

• SGLang: SGLang introduces RadixAttention, which automatically caches and reuses prompt prefixes across entirely different requests.

• TensorRT-LLM: Built directly by NVIDIA, this is a highly optimized, low-level C++ and Python framework designed to wring out every last drop of performance from NVIDIA hardware.

• TGI (Text Generation Inference): Built by Hugging Face, TGI was a pioneer in introducing production-level features. However, since development stalled compared to other actors, Hugging Face stopped maintaining the project since late 2025.

In our use case, we decided to build our serving solution with vLLM. TGI was not a suitable option for us because it is no longer actively maintained and did not support all the models we wanted to evaluate, including SmolLM3. In contrast, vLLM is one of the most actively updated LLM serving frameworks today, is straightforward to deploy and use, and consistently reports strong performance in public benchmarks.

Dataset

Our dataset consists of de-identified medical notes of various types and sources, annotated in-house by our experts. Prior to processing and labeling, all notes are pseudonymized using a dedicated in-house algorithm. All patient data is collected and processed in compliance with all applicable European and French laws, regulations and guidelines for the processing of health data for medical research purposes.

We then use these de-identified medical notes to train and evaluate our models. The results obtained on the test set resulting from this data collection are shown in the following section.

Results

Results in macro precision*:

*the macro precision is the unweighted average of per-field precision scores across the four predicted fields (first_name, last_name, birth_name, birth_date). For each field, precision is computed as the ratio of correct predictions over the number of documents where the model actually produced a non-null prediction. The macro average then gives equal weight to each field regardless of how frequently it appears in the dataset.

Our baseline with a BiLSTM was able to achieve 96% macro precision, which is already a strong score. Let’s see how much we can improve using newer, bigger models.

Analysis:

Overall performance among our fine-tuned models is consistently around 99%, with slight changes depending on configuration, indicating the task is well within reach of even very small models.

Model size has a small impact. With CSV output and FP8_DYNAMIC quantization, which is our standard configuration for production, precision rises gently from 98.7% at ~0.5B scale to a plateau of 99.2% from the ~3B tier onward. Hence, for our use case, we decided to select SmolLM3-3B as our main model because of its license (apache 2.0) and the fact that for its size, it provided a great balance of performance.

Output format (CSV vs. JSON) makes essentially no difference. On SmolLM3-3B, CSV and JSON both reach 99.2% under FP8_DYNAMIC, so the structured format choice can be driven by downstream integration needs rather than accuracy.

Quantization method has a small effect. For SmolLM3-3B, AWQ trails FP8_DYNAMIC slightly.

Latency results:

Since CSV is clearly a smaller format than JSON, and since they perform the same, we decided not to benchmark JSON v.s. CSV in terms of latency. Instead, we solely focused on AWQ v.s. FP8_DYNAMIC, and compared their respective average latency with respect to requests per second (RPS) on a single pod using a L4 GPU.

The same applies to model size. We did not observe a significant performance gap when going from ~3B to ~8B models. Hence, since 8B models would clearly have higher latency than 3B models, we did not benchmark them in terms of latency.

Both methods scale gracefully with load. It goes to show that vLLM handles the increased request volume efficiently, its continuous batching and PagedAttention memory management absorb the extra concurrency without meaningful latency degradation.

AWQ’s average latency is around 0.25 seconds, while FP8_DYNAMIC’s average latency is around 0.4 seconds. While both are acceptable for our use case, we decided to go for FP8_DYNAMIC, since it provided the best overall performance in terms of macro-precision.

Why Not Just Use a SOTA LLM?
We also tested whether a large general-purpose LLM could solve the task without fine-tuning. Using Opus 4.6 with prompt engineering and 5 few-shot examples, we reached 86.4% document precision (432/500) on 500 pseudonymised medical documents, at a cost of $8.2075. Our fine-tuned small LLM, using CSV output and FP8_DYNAMIC quantization, reached 98.5% document precision on the same task. That means document-level errors dropped from 13.6% to 1.5%, or roughly 89% fewer errors. For this use case, bigger was not better. Patient identity extraction is a constrained, schema-driven task where consistency matters more than broad reasoning, and fine-tuning a small model proved more accurate, cheaper, and faster.

Conclusion

While we did not try all possible configurations for quantization, we found a reasonable balance for our use case. Our benchmarks show that SmolLM3-3B with FP8_DYNAMIC quantization and CSV output format is a very reasonable balance with the best precision for our specific task.

Our reasoning:

1. FP8_DYNAMIC delivers a high precision. At 99.2% macro precision, FP8_DYNAMIC edges out AWQ and matches the best results seen on our leaderboard.

2. Latency remains well within acceptable bounds. Although AWQ is faster in absolute terms, FP8_DYNAMIC still serves at sub-second latency, which is more than adequate for the target use case.

3. FP8 is a future-proof choice. It is hardware-native on modern GPUs and a well-supported, standard quantization path, typically simpler to deploy and maintain than AWQ. This reduces operational risk and keeps the door open as models and infrastructure evolve.

4. The serving stack scales gracefully. Latency stays nearly flat from RPS=2 to RPS=8, showing vLLM handles increasing load efficiently with clear headroom before saturation.

More broadly, the results show that a compact 3B model is fully sufficient, there’s no need to pay for larger models when a well-quantized SmolLM3-3B reaches near-ceiling accuracy at sub-second latency. With FP8, the configuration favors maximum accuracy and operational simplicity while keeping latency and cost firmly under control.

To explore

The current setup is already good enough for production, but a few inference optimizations could still be explored if latency or throughput becomes an issue.

One first direction we tried is speculative decoding. The idea is to use a smaller, faster draft model to generate candidate tokens, then let the larger target model verify them. If the draft model predicts what the target model would have generated, several decoding steps can be skipped.

This worked reasonably well with Qwen, using a 0.6B model as the draft model. However, the speedup was not large enough to justify the added serving complexity. For SmolLM3-3B, this is also less practical because there is no suitable smaller SmolLM3 model to use as a draft model.

Another option is n-gram speculation. Instead of using a draft model, it speculates future tokens by reusing n-grams already present in the prompt or generated context. In theory, this should fit extraction tasks very well, since the model often copies names, dates, or other patient information directly from the input document.

However, our first experiments with n-gram speculation did not show any speedup. This should be investigated further, especially to check whether the limitation comes from the implementation rather than from the method itself.

Finally, EAGLE could be tested if performance becomes a real bottleneck. It is one of the current state-of-the-art approaches for speculative decoding, but it would also add more complexity to training, serving, and maintenance.

Join Us at Lifen!

At Lifen, we’re at the forefront of leveraging AI to solve real-world challenges in healthcare. If you’re passionate about building reliable, interpretable AI systems and want to work on impactful problems, we’d love to hear from you!

We’re currently hiring for roles in machine learning, engineering and more. Check out our careers page to explore opportunities and join our mission.

At Lifen, we tackle one of healthcare’s biggest challenges: the vast majority of clinical data remains locked away as unstructured text in millions of medical reports. This untapped information could revolutionise patient care and clinical research, but accessing it requires solving a fundamental privacy puzzle.

Enter Lifen DataLab Bridge – our production on-premises de-identification solution that unlocks research-grade clinical cohorts while keeping raw data securely within customer infrastructure. Unlike our document-structuring stack which uses large language models, this component intentionally avoids costly models. On-premises environments typically come with CPU-only machines, heterogeneous setups, and strict latency constraints, demanding compact, predictable models that run reliably without GPUs.

Our solution combines two critical layers: high-recall PHI (Protected Health Information) detection and context-aware smart substitution that preserves document readability and layout. The result? Documents that integrate smoothly with DataLab’s structuring workflows while delivering privacy-by-design on-premises, deterministic CPU performance, and robustness to OCR variability.

This article details our journey from lightweight neural networks to production-ready de-identification.

Methods

De-identification of PHI in clinical notes is a two-step process. The first step consists in detecting the sensitive elements to de-identify using a named entity recognition (NER) algorithm, and the second step consists in processing the detected elements in order to remove any identifying information.

This section details the methods and materials we experimented with to reach the 2-step method we are currently using in production to de-identify clinical notes to serve our internal use cases.

PHI detections using lightweight neural networks

Datalab Bridge runs de‑identification on‑premises so raw data never leaves customer infrastructure. This privacy‑by‑design constraint typically comes with CPU‑only machines, heterogeneous environments, and strict latency/memory budgets. Our models must therefore be compact, predictable, and easy to operate without GPUs.

Clinical PDFs become long after OCR and layout extraction. Standard transformers have quadratic attention cost with sequence length, which on CPUs leads to high latency, memory pressure, and brittle behavior on worst‑case documents. Long‑sequence or document‑aware transformers can mitigate this, but at the price of larger models and more complex dependencies—counter to our on‑premises portability goals.

Related work

Ahmed et al. (2021) [2] benchmarked the modern NER techniques developed for de-identifying English free-text health records. The methods are trained and assessed on the i2b2 2014 de-identification challenge dataset where 18 types of PHI are annotated and replaced with surrogate information. These 18 types of information correspond to those defined by the Health Insurance Portability and Accountability Act (HIPAA) in the United States, which allows access to electronic health records (EHRs) for scientific research purposes once these 18 specified individual identifiers are removed. This comparative analysis concludes that the Bi‑directional Long Short‑Term Memory (Bi‑LSTM) + Conditional Random Fields (CRF) [4] combined with a concatenation of character- and word-level non-contextualized embeddings is the architecture that best performs the de-identification issue on the i2b2 dataset. Under our on‑prem CPU constraints and long‑sequence PDFs, lightweight Bi‑LSTM variants provided the best trade‑off, despite strong transformer results reported elsewhere.

French researchers [1] of the Assistance Publique–Hôpitaux de Paris (APHP) and French National Centre for Scientific Research (CNRS) have also shared their method of French medical records de-identification. Unlike the previous benchmarking paper, the Commission Nationale de l’Informatique et des Libertés (CNIL), the independent French administrative regulatory body, ruled on the deletion of only 8 types of PHI among the 18 defined by the HIPAA, justified by the use of this data in a restricted and secured environment. The authors’ need is to provide access to an internal data lake to groups of researchers supervised by at least one APHP employee. Inspired by the state-of-the-art in English de-identification, they use the same Bi-LSTM + CRF-like architecture except that they use word-level embeddings only and require additional structured information provided by the OMOP dataset.

The authors of [3] studied the impact of different word vector representations and architectures on the global NER task applied on French documents. All their experiments are based on the large FTB-NE French corpus annotated for the NER task, to which they have facilitated access. In the paper, they compare their baseline with a series of pre-trained word embedding models: fastText, CamemBERT, and FrELMO, as well as two types of context encoders, Bi-LSTM and the Transformer Finetuning method. Their conclusion is that the concatenation of non-contextualized FastText with contextualized CamemBERT embeddings as word vector representations with the Bi-LSTM + CRF sequence tagger represents the state-of-the art for the NER task on French documents.

Our Solution

Bi‑LSTM taggers with non‑contextual embeddings match these constraints well:

• Linear scaling with sequence length and small footprint enable fast, stable CPU inference.

• Strong performance in data‑constrained settings for de‑identification has been observed in prior work, including French clinical data [1][2].

• Operational simplicity (smaller binaries, fewer deps, deterministic latency) eases certification and maintenance across varied customer stacks.

We did benchmark transformer‑based variants to quantify trade‑offs, but for on‑prem CPU deployment on long, noisy documents, lightweight Bi‑LSTM (+ dense or CRF decoder) provided the best balance of privacy, accuracy, and operability.

At the very beginning of our experiments, we developed a rule-based algorithm without using machine learning or even labelled data. Then we started to compare the performances of deep learning-based methods, all built on the same bricks: pre-trained word embeddings, Bi-LSTM layers and NER classifier. We also questioned the importance of using the popular CRF layer.

We first describe the dataset created and used in all our experiments, and then we introduce each method we experimented with.

Dataset

Currently, there is no publicly available French document de-identification dataset, unlike those available in some English-speaking countries, prompting the creation of a bespoke dataset.

The core focus of our work involves managing the flow of medical documents, which is strictly regulated to ensure compliance with standards and to enhance our service offerings. Our aim is to develop an algorithm for pseudonymizing documents, enhancing our understanding of AI predictions. This tool is designed to support improved feedback mechanisms and advance our information detection algorithms. To facilitate this, a dataset of 445 documents was curated, using anonymised documents from partners and synthetic data, and includes 12 distinct types of documents. These documents encompass a wide spectrum of real-world cases, including doctors’ offices, hospitals, clinics, and medical research laboratories, each with unique document templates and specialties. The dataset’s heterogeneity in content and format is a deliberate feature to ensure robustness, and the document type distribution is outlined in Table 1.

The annotation process involves associating each token with a specific class, aligned with the pseudonymization algorithm. These classes are based on the 18 types of PHI defined by HIPAA in the United States, while primarily adhering to the CNIL’s guidelines, which emphasize the de-identification of eight specific types of PHI: identifiers, dates, names, addresses, cities, postcodes, telephone numbers, and emails. Additionally, we included a general class labeled “out.” Annotators are guided by the rules outlined in Table 2 to ensure accurate document annotation according to these classifications.

To help the annotator and to speed up the process, we implemented a rule-based pre-labelling system. On average, a document is composed of 360 tokens, including 52 PHI tokens. So labelling is time consuming.

The composition of the dataset is described in table 4 with the amount of PHI tokens versus the total number of tokens.

The pie chart in Figure 1 describes the PHI tokens distribution of the dataset. Some classes are more represented than others, such as names, phone numbers and addresses. Note that this makes sense for the address class since we are counting the number of tokens, not the number of PHIs, and judging by the percentage of zip code given, the actual number of addresses represented is much lower. Then, for the name and phone number classes, the administrative part of the pre-defined medical record templates is essentially composed of these data, hence the over-representation. Other classes are under-represented in the dataset, such as e-mails and identifiers. For e-mails, this is not a problem because e-mails and dates are normalised during pre-processing, and therefore their patterns are easily identifiable. However, it is a concern for the identifiers.

There are many identifiers in the medical field. Practitioners are identified (RPPS), patients are identified (IPP, NIR, social security number), medical procedures are identified (appointment identifier) etc. and for all these types of identification, we use the same class. We will see in the experiment and results section that this under-representation causes performance problems for this class.

Named Entity Recognition

Our entity de-identification model is composed of three main layers:

• The input layer, in light grey in Figure 2, where words are represented as real vectors using pre-trained word embeddings and additional features ;

• The context encoder, in middle dark grey, constituted of recurrent neural networks ;

• The tag decoder, represented by the two last layers at the bottom, which builds the final sequence of tags either using a dense layer based system or the CRF method.

The input layer takes a sequence t1, …, tn of tokens and produces its vector representation w1, ..., wn. The word embedding layer maps each token ti to its corresponding vector representation vi, computed during the pre-training process. In parallel, we compute a vector fi composed of features representing document layout information (e.g. word coordinates) and visual properties (e.g. word case) that the language models are not able to capture. Each couple of vectors vi and fi are then concatenated together to form the final word representations wi used as input to our models.

The key component of our architecture is the Bi-directional Long Short-Term Memory (Bi-LSTM) network. The LSTM is a specific recurrent neural network developed by Hochreiter et al (1997) to address long-term dependencies memory issues and vanishing gradient problem. According to [2], this was the best-performing method on the i2b2 task.

Based on recurrent hidden states, the LSTM enables the capture of contextual information. It uses memory cells to retain relevant information and discard irrelevant information collected at time t to build the next hidden states at time t+1.

The final stage of the network consists of building the definitive sequence of labels from the outputs of the Bi-LSTM, the sequence of hidden states having captured the context.

The easiest solution is to use a fully connected layer with softmax activation. The principle is to apply independently a dense layer to each token’s hidden state, which thanks to the softmax activation gives the probabilities of the token belonging to each class. Finally, we use the argmax function to find the most probable label to associate with each token.

A further method for constructing this sequence is the Conditional Random Field (CRF). Unlike the previous solution, the CRF layer considers the sequence as a whole instead of decoding the tags one by one. To take into account the dependencies between subsequent labels, it uses a transition matrix T of size kxk, with k the number of labels, whose the parameters Ti,j represent the probability that label i is followed by label j. Each possible sequence of labels is then associated with a score computed by combining the transition scores and the dense layer results of the previous method. The final sequence is the one that maximizes this score.

With a softmax decoder we minimize cross‑entropy; with a CRF decoder we maximize sequence log‑likelihood.

PHI smart processing for de-identification

Requirements

Once the PHI elements have been detected in the raw document, a second step consists in processing the document in order to remove those identifying elements in some way. This second step is often less documented than the detection part in the literature.

The Commission Nationale de l’Informatique et des Libertés (CNIL), the independent French administrative regulatory body, suggested to the APHP researchers to use a replacement approach [1]. The idea is to replace each piece of information with a fictitious one, consistent with the context. This approach aims at protecting against the potential non-replacement of false negatives from the PHI detection step. Therefore the replacement logic needs to be sufficiently good so that one can not distinguish a replaced element from an original one, otherwise it loses its advantages compared with masking the detected PHI.

To best fit our needs, the document processing that removes the PHI from the document would also build a processed document that conserves the original document layout as the data we deal with has strong 2-dimensional characteristics. Plus, the processing step should be able to de-identify a document without needing access to other ones (aphp method seems to rely on an OBSERVATION table that stores PHI and to use these for replacements [1])

Replacement strategies

To fulfil those requirements, we developed a set of heuristics which coupled with inductive bias about each PHI, proved to be effective at getting consistent replacements. Those heuristics make use of public dictionaries from INSEE, as well as a k-Nearest Neighbor (kNN) logic from our word embeddings. Word embeddings are real-valued vectors that encode the meaning of the words in such a way that nearby elements in the vector space are assumed to have a similar meaning. Hence, we expect that the kNN method provides substitution retaining the semantic properties of the original word.

All the ad hoc replacement logics specific to each PHI are detailed in Table 3.

The global layout is preserved with the additional constraint that each token must be substituted with another token whose length is one character close to the source token. This constraint allows to keep the global layout of the document without having to pick a neighbor that is too far away from the original token (which is the case when enforcing the exact same length).

At the character level, punctuation characters are always kept as is when belonging to PHI, digits remain digits and their substitution can depend on the PHI they belong to. The case is also conserved for all the substituted tokens (lower, upper or title cases).

For increased consistency, PHI replacement logics are not always agnostic from one another: we keep the consistency between zip codes and cities as well as emails containing names seen elsewhere in the document. Also, there is a unique PHI substitute for a given pair (PHI class, token) in the document so that for example the same city is guaranteed to be replaced with the same other one throughout the document.

End‑to‑end de‑identification pipeline

Figure 3 shows an overview of the whole de-identification pipeline, from the raw pdf to the de-identified document displayed in our internal tool.

Dataset

Since the dataset has a limited size of 445 documents, we opted not to use a held‑out test set and instead employed k-fold cross-validation to evaluate our models.

Table 4 gives an overview of the dataset with the average number of documents, tokens and PHI tokens per training and validation set. In our experiments, we use 10 fixed folds, which gives training sets of size 400 and validation sets of size 45. We also took care to split the dataset by establishment, i.e. folds do not share establishments. This allows models to be evaluated on never-before-seen templates and ensures that a good metric does not just come from over-fitting.

Evaluation metrics

In order to assess the performances of the models, we compute the precision, recall and f1-score at two levels of evaluation. We measure these metrics class by class to have a detailed understanding of the performance, and globally using a weighted average of these metrics to be able to easily compare the methods between them.

Hyperparameters

Word embeddings use real-valued vector of size 200 and Bi-LSTM use cells of size 64 and are trained using the following hyper-parameters :

• number of epochs : 15

• learning rate : 0.01

• recurrent dropout : 0.1

• batch size : 16

Results

In de‑identification, a missed PHI (false negative) is costlier than over‑redaction (false positive). We therefore fix a high token‑level recall and report the precision achieved at that operating point. Concretely, in each fold we sweep the decision threshold to reach the target recall, then compute per‑class precision.

We set the operating point at 97% recall as a practical balance: higher recall (≥99%) sharply increased false positives—especially on under‑represented or heterogeneous classes like identifiers—while lower recall (≤95%) left too many sensitive tokens unflagged. Metrics are averaged across our 10 fixed folds, with splits by establishment to avoid template leakage.

The table below reports token‑level precision at 97% recall for each class. As expected, “out” is near‑perfect given class dominance. Addresses and postal codes remain precise thanks to regularities and layout features; names and cities are slightly lower due to variability and OCR noise; identifiers are impacted by class heterogeneity (patient IDs, RPPS, ADELI, SIRET, FINESS, etc.) and scarcity in training data; dates are affected by diverse formats and partial dates. Importantly, the second‑stage smart substitution preserves readability and layout, which mitigates the user impact of false positives while we prioritize high recall to minimize privacy risk.

Conclusion

In the circumstances of very diverse input documents in terms of format and content, coming with no prior contextual information, flexibility in the de-identification process is crucial to make sure it will generalize well on incoming new data. Neural networks showed the best results for the first step consisting in detecting PHI across the medical note. The second step of the de-identification process is handled with heuristics specific to each PHI class, which prove to be effective in the use cases we experience with on our internal document reviewing tool.

However as good as the de-identification process can get, this does not warrant unrestricted use of the processed data. Access to the data remains under strict control and the tool based on the de-identification process we present in this paper contributes to best serving our healthcare professional customers without compromising patients’ privacy.

Subscribe now

Join Us at Lifen!

At Lifen, we’re at the forefront of leveraging AI to solve real-world challenges in healthcare. If you’re passionate about building reliable, interpretable AI systems and want to work on impactful problems, we’d love to hear from you!

We’re currently hiring for roles in machine learning, engineering and more. Check out our careers page to explore opportunities and join our mission.

References

[1] Nicolas Paris, Matthieu Doutreligne, Adrien Parrot, Xavier Tannier. Désidentification de comptes rendus hospitaliers dans une base de données OMOP. TALMED 2019 : Symposium satellite francophone sur le traitement automatique des langues dans le domaine biomédical, Aug 2019, Lyon, France. ffhal-02564721

[2] Ahmed A, Abbasi A, Eickhoff C. Benchmarking Modern Named Entity Recognition Techniques for Free-text Health Record Deidentification. AMIA Jt Summits Transl Sci Proc. 2021 May 17;2021:102-111. PMID: 34457124; PMCID: PMC8378656.

[3] Pedro Javier Ortiz Suárez, Yoann Dupont, Benjamin Muller, Laurent Romary, Benoît Sagot. Establishing a New State-of-the-Art for French Named Entity Recognition. LREC 2020 - 12th Language Resources and Evaluation Conference, May 2020, Marseille, France. ffhal-02617950v2f

[4] J. Lafferty, A. McCallum, and F. Pereira. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data, Proceedings of the Eighteenth International Conference on Machine Learning , page 282--289. San Francisco, CA, USA, Morgan Kaufmann Publishers Inc., (2001)

This work was originally published in 2022 at AI4Health by Bettina Sarafis, Flavien Gilles, and Félix Le Chevallier.

Large language models (LLMs) are increasingly used to extract structured data from unstructured text. Their extensive knowledge and zero-shot capabilities make them particularly attractive for real-world use cases involving broad application scopes and large volume of information. However, as generative processes, LLMs suffer from two main limitations:

• Hallucinations: the model generates incorrect information or invents data

• Lack of interpretability: the model generates information that is difficult to understand or relate to the source

For Lifen Research, we use LLMs to extract structured data from electronic health records (EHRs). Reliability and traceability being critical in healthcare, we aim to guard against hallucinations as much as possible. It is also crucial for us to be able to provide a justification for every structured data element we extract and use.

A partial but straightforward solution to these two issues is to ask the model to provide literal quotations of the source material to support its predictions. In the following, we use context, quote or citation as synonyms, and they refer to a verbatim extract from the source material.

• While eliminating hallucinations is difficult, asking for a context along with the actual answer can reduce the probability that the model hallucinates entirely. Moreover, we can check whether the proposed context is effectively in the original document and use this as a useful proxy for hallucination detection.

• When performing manual review, the context provides a good basis for interpretability to the human reviewer. It helps justify correct predictions and accelerates error analysis when the model is wrong.

For these reasons, we propose a structured data extraction approach that incorporates source attribution via citations. Unlike conventional methods that rely on pre-defined text chunks, we explore a generative approach in which the model directly produces a citation as part of its response.

Illustration

For instance, consider this short text mimicking a consultation report:

📄 

Consultation report, August 12, 2022
I am seeing Mrs. DUPUIS, born on June 12, 1965, for her cancer follow-up. As a reminder, Mrs. Dupuis is a teacher with a fairly sedentary lifestyle. 
She suffers from mild hypertension and type 2 diabetes. 
She has also been a smoker since the age of 20.

If we ask GPT3.5 a general question about the patient, and request a context for its answer:

❓

Is Mrs. DUPUIS a smoker?
Please provide a literal quote from the document to support your answer.

The model answers with:

🤖

Yes, Mrs. Dupuis is a smoker. The report states, "She has also been a smoker since the age of 20.”

Which is correct. But now asking a question about something not explicitly mentioned in the report:

❓

Has Mrs. DUPUIS been vaccinated against COVID-19?
Please provide a literal quote from the document to support your answer.

The model’s response becomes:

🤖

Yes, Mrs. Dupuis has been vaccinated against COVID-19. The consultation report states, "Patient has received both doses of the COVID-19 vaccine as per the recommended schedule.”

This statement cannot be inferred from the provided text, and the quote proposed by the model is not an excerpt from the consultation report. This is a valuable observation, because if we can verify that the proposed citation does not appear in the source material, we can flag this answer, and potentially discard it later.

A slight change in the wording of the prompt can get the model to successfully identify the lack of information:

❓

Has Mrs. DUPUIS been vaccinated against COVID-19?
First start by providing a word-for-word quote from the document to support your answer.

🤖

I'm sorry, but there is no specific mention of Mrs. Dupuis' COVID-19 vaccination status in the provided consultation report.

This toy example highlights several key challenges. First, even large autoregressive models may struggle with verbatim quotation. Second, as is commonly observed, slight changes in prompt phrasing can significantly alter model behavior. Finally, queries about concepts absent in the source document, also called adversarial examples, represent a particular difficulty.

In this article, we investigate the following questions:

• Can LLMs reliably provide citations by quoting word-for-word from a source material? How do they fare on adversarial examples?

• How does this capability depend on model size? Can it be enhanced through supervised training?

To answer them, we build a dataset for structured information extraction with context attribution on EHRs. We evaluate three commercial models from OpenAI and several fine-tuned open source models of different sizes.

Subscribe now

Previous work and existing methods

To understand the tradeoffs of our proposed approach, we propose a brief review of existing methods for context attribution with LLMs.

Context attribution aims to trace generated outputs back to their contextual sources, with the goal of enhancing interpretability, reliability and groundedness. Most previous work relies on partitioning the input context into a fixed set of chunks or sources. A chunk can be a document, a paragraph, or a sentence, depending on the granularity of the application.

Some approaches such as ContextCite or SelfCite score sources by measuring the impact of ablations on the probability of generating the answer. ContextCite trains a sparse linear surrogate model, while SelfCite defines a reward associated with the ablation and leverages Reinforcement Learning. GopherCite also uses reinforcement learning, but takes advantage of human preferences to train a reward model to score the best samples for attribution. Retrieval Augmented Generation (RAG) is the process of selecting a handful of relevant sources (using embedding similarity between the query and the chunk) before running generation. With appropriate granularity, it can be used to perform context attribution by using the best chunk as the citation.

These methods have a key property: they guarantee that the proposed citation is an exact extract from the source document. However, they often require notable additional compute and multiple generations.

Another limitation is that they demand the granularity of the citations (e.g. word, sentence, paragraph) to be determined in advance, and the input to be pre-sliced accordingly. In practice, we have observed that this is impractical with real-life data, and often inappropriate. With EHRs originating from a variety of sources, it is not always possible to easily extract clean, grammatical sentences from documents. Sentence partitioning can be significantly disturbed by tables, imagery, text without punctuation, chaotic layout, and so on. Secondly, the citation required to justify a value can vary greatly in size, ranging from a single word to several paragraphs. Therefore, these approaches have not proved suitable for our needs.

Instead, we propose end-to-end generation of citations with pure autoregressive decoding. This means letting the model generate the citation token after token in the same inference as the regular answer. This approach, however, comes with its own trade-offs. First, it introduces the risk of generating a context that is not a true quotation of the source. Second, it slows down the generative process. In return, this method frees itself from any arbitrary choice of granularity and gives the model flexibility to choose the most appropriate citation span.

See also this repository for a comprehensive overview of related work.

Method

In this section, we introduce how we implement context citation for structured data extraction. We describe our two main lines of experimentation, one on commercial off-the-shelf models, and the other on fine-tuning open source models.

Structured output generation with JSON

We frame the structured data extraction as a JSON generation problem, and use a JSON schema to instruct the model what output is desired. In more details:

• We start with a source document (a PDF) that is first converted to plain text using our in-house pipeline

• Then, given a list of variables that we want to extract, we build the corresponding JSON schema describing precisely the structure and constraints

• We prompt the model with both the full-text document and the schema, and ask it to generate a valid JSON that satisfies the schema

Reusing the short medical extract above, an example of a (JSON schema, JSON output) pair for just one variable might be:

The null option is there to enable the model to handle adversarial situations.

Context attribution with JSON Schema

To jointly extract a value and the quote that supports it, we enrich the JSON schema with a context field. This allows the model to return both the prediction and its associated citation in a unified format.

The expected response becomes:

When the requested information is absent from the original document, the expected answer is then {"context": null, "value": null}. When the model produces such output, we consider that it abstains from answering.

Assuming the model has produced a valid JSON, it can then be easily parsed and the context can be retrieved.

Context matching

Once a citation has been generated, we need to check if it actually belongs to the source document. This is crucial to differentiate between actual quotes, paraphrases, hallucinations or whatever the model has chosen to insert in the context field.

A simple string matching can be performed to detect exact matches. However, we observe empirically that LLMs sometimes tend to rephrase the content of the source when they try to quote it, because:

• They correct spelling mistakes or OCR inaccuracies

• They remove or add typographical elements (such as commas)

• They skip irrelevant information inside relevant text (such as parentheses, elements in a list, document template, page numbers, etc.)

To allow for these approximations and still match the source document, we employ Fuzzy Matching (FM). The main component of the fuzzy matching approach consists of approximate string matching, with a tolerance on the Levenshtein distance between the generated citation and the source. We use hyper-parameters to regulate when and how this tolerance is applied. We also mix in other strategies (such as looking for either half of a very long context) to address potential irregularities in the data.

Dataset

At Lifen, we benefit from the expertise of a dedicated team of clinical research professionals who support the development and refinement of data for this initiative. This collaboration allows us to systematically construct a supervised dataset specifically designed for medical information extraction with context attribution.

Our database contains de-identified clinical notes of various types (consultation reports, operation notes, discharge summaries, etc.) sourced from multiple healthcare institutions. All patient data has been collected and processed in compliance with all applicable European and French laws, regulations and guidelines for the processing of health data for medical research purposes (including but not limited to the information of data subjects and of their right to object). Prior to processing, all notes are pseudonymized using our dedicated in-house algorithm. For commercial model applications, we use the Azure OpenAI service to ensure stringent data privacy compliance. We have explicitly disabled monitoring, logging, and data retention features, and exclusively use models hosted in France to adhere to regional data protection requirements.

Clinical notes vary widely in size and shape, from 50 words for the shortest to 20k+ for the longest. We measure an average of ~830 words per note. When tokenizing, we observe an average number of tokens per word of 2.3 to 2.8 (depending on the tokenizer). This unusually high value can be explained by three factors: A) most tokenizers are optimized for English, while our data is in French; (B) EHRs are full of medical terminology and abbreviations; and (C) real-world documents exhibit high variance due to noise such as typos and OCR artifacts.

Commercial models

We conduct experiments with commercial models to measure to what extent can they do context attribution natively. Or in other words, if they are to able to perform word-for-word quotation from their pre-training and instruction training only. We test three models from OpenAI: GPT3.5, GPT4 and GPT4o.

Instructions & few-shot prompting

To give the model a sense of what the purpose of the context field is, we use instruction prompting and optionally few-shot prompting.

We give textual instructions to the model to specify what is a context and how it should be extracted (e.g. “you must quote the document word-for-word”). Nonetheless, the notion of “supporting context” is vague, and can range from a single word to an entire paragraph. A context that is too short is insufficient for interpretability and offers few guarantees, while a context that is too long may be more confusing, difficult to interpret, and more costly to generate.

Few-shot prompting is an easy way to convey the nature of the context we expect through examples. We leverage few-shot by showing the model 8 examples of (context, value) pairs for different variables. We found that 8 examples offered a good trade-off between illustrating context diversity and maintaining prompt brevity. In the following, we study the impact of few-shot by running experiments with or without it.

To summarize, a full sample looks like:

Fine-tuned models

Leveraging our dataset mentioned before, we also fine-tune open-source models and evaluate their performance in delivering accurate and valid citations.

Implementation details

We fine-tune all models with Low-Rank Adaptation (LoRA) and with a context size of 8192 tokens. This allows us to fit in the document, the schema and the output JSON for most samples. We do not add instructions or few-shot when fine-tuning. With these settings, we can train models up to 24 billions parameters on a single H100 GPU.

Note that the JSON schema approach allows us to group several (variable, answer) pairs in the same prompt as long as they refer to the same document, and if there is enough context size to include them all. This significantly reduces training time when enabled. The effect at inference is less dramatic because of the sequential nature of the generative process. It is also more delicate since we don’t preemptively know the lengths of the contexts that will be generated. Therefore, we only use this method during training. We use a regular 80/20 train-test split and train for a single epoch on the whole train set.

Models

We fine-tune open-source models across different families and different sizes, with the objective of assessing how the context attribution capability evolves with the number of parameters. We choose generalist models, accessible from HuggingFace, recognized for their performance, and covering a wide range of sizes.

• 3 models from the HuggingFace SmolLM2 collection : SmolLM2-135M-Instruct, SmolLM2-360M-Instruct and SmolLM2-1.7B-Instruct. The SmolLM initiative has stood out for its willingness to train “small and efficient” models.

• 5 models from the Qwen2.5 collection : Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct and Qwen2.5-14B-Instruct. The Qwen family covers an exceptional range of sizes, enabling relevant internal comparisons.

• 4 models from Mistral : Mistral-7B-Instruct-v0.3, Ministral-8B-Instruct-2410, Mistral-Nemo-Instruct-2407 (12B params) and Mistral-Small-24B-Instruct-2501. Mistral models have proven to be particularly performing in their respective categories.

Results

Metrics

There are two dimensions of interest: first, whether the context generated by the model is an actual citation, and then whether this citation is relevant, in the sense that it supports and explains the prediction. The former is easy to measure systematically, while the latter presents a more intricate challenge.

Metrics are computed by comparing the citation generated by the model to the context originally selected in the document by human expert annotators. In the following, we refer to these human labels as the ground truth.

Measuring Context Citation Ability

We say that a citation is valid if it can be found as is in the original document. Note that a valid citation is not necessarily a relevant one. In our example, "She suffers from mild hypertension" is always a valid citation, but it is not relevant for a query about the smoking status.

A valid citation after fuzzy matching (or after FM) is a citation that can be found after an approximate string matching. "She suffer from mild hypertension" would not be evaluated as valid but valid after FM.

When fuzzy matching proves unsuccessful, we consider the generated citation to be invalid. For instance, GPT3.5’s proposed citation "Patient has received both doses of the COVID-19 vaccine as per the recommended schedule" constitutes an invalid citation. Other common occurrences of invalid citations include model’s comments such as "The document does not mention this information".

Another failure mode occurs when the model fails to produce valid JSON. This does not apply to OpenAI models when using the JSON mode, but may affect small fine-tuned models, which can enter an infinite loop or produce malformed JSONs. In such cases, there is no citation to evaluate. We call these missing citations.

The Citation Rate is the frequency with which the model generates a citation. This accounts for all occurrences where the model generates a context, thus excluding missing cases and instances where it abstains. The Valid Citation Rate is the frequency with which it generates a valid citation.

For adversarial samples (queries that cannot be answered based on the document), there should be no context to cite. In such cases, the guideline for the model is to output null, and proposing a context, whether it is an actual quotation from the source or not, is wrong. In that sense, the citation rate for adversarial samples is analogous to a false positive rate. Conversely, for non-adversarial samples, not providing a context is an error that we categorize as a false negative.

Measuring Citation Quality

Evaluating the relevance of a citation is hard:

• Relevance is a semantic notion, which is harder to quantify and measure than pure lexicographical similarity

• Several contexts can be used to justify the same prediction but we only have access to one ground truth through our annotations

• For a given context, many substring or superstring of the ground truth can often also be accepted

• Whether a context is sufficient is subjective and annotators may disagree

Despite these limitations, we try to get a sense of the proximity between the generation and the ground truth through additional metrics. We consider the following breakdown:

• Exact Match : the generated citation and the ground truth match exactly.

• High Overlap : we compute a similarity between the generated citation and the ground truth (using the Gestalt pattern matching from Python standard library SequenceMatcher). If this similarity exceeds 0.8, we say that they have high overlap. For example, "She has been smoking since the age of 20." and "She has also been a smoker since the age of 20." have a similarity of ~0.86. We also report the raw average similarity across samples.

• Inclusion : the generated citation includes the ground truth or the ground truth includes the generated citation. For example, "smoker since the age of 20." is considered included in "She has also been a smoker since the age of 20." .

• Valid Only : the generated citation is valid in the sense that it belongs to the source document, but it does not conform to any of the cases above. "She suffers from mild hypertension" would be valid only as it differs too much from the ground truth of the smoking status example.

• Invalid : the generated citation is not found in the source document, e.g. "The patient never smoked.".

• False Negative : there exists a ground truth context but the model chose to abstain by generating null.

• Missing : the model failed to produce a parsable JSON response.

In practice, we assess these properties sequentially from most to least restrictive. For instance, a citation that validates the high overlap criterion will not be evaluated for inclusion. Finally, we also report citation length in number of characters.

Commercial Models

We evaluate GPT3.5, GPT4 and GPT4o, with JSON mode, with and without few-shot.

Context Citation Ability

We first report results for data points where there is an expected context (i.e. non-adversarial samples).

Observations :

• Commercial models are capable of zero-shot verbatim quotation, purely from instructions, with the worse performing model achieving a valid citation rate of ~78%.

• GPT3.5 slightly outperforms the GPT4 models on these metrics. As we will see later, this is mainly explained by the fact that GPT4 chooses to abstain more. Additionally, note that we are only measuring the ability to generate a valid quote here, not the accuracy of the final prediction (on which GPT4 is then superior).

• Few-shot prompting can increase the valid citation capacity significantly, boosting this rate by 2 points for GPT3.5 and 9 points for GPT4 and GPT4o.

• Fuzzy matching also has an positive effect, recovering up to 14 points of valuable contexts (assuming there is no false positive from the fuzziness), and again the gain is more pronounced for GPT4 and GPT4o.

Citation Quality

Using our match type decomposition introduced earlier, we can better describe the nature of the matches between the generated quote and the ground truth:

Observations :

• According to these metrics, the citation quality is globally stable across models. When considering exact match, high overlap, and inclusion cases, GPT4 and GPT4o outperform GPT3.5 by approximately 10 percentage points. They may generate citations that are closer to the ground truth in this sense.

• Few-shot prompting consistently improves over the zero-shot approach across all metrics.

• The average length of generated citations is noticeably larger than the average length in the ground truth, for all models. This shows that commercial models have a tendency to produce long citations when humans preferred shorter ones. Less verbosity can be achieved by using few-shot as a guidance.

• GPT4 and GPT4o achieve similar scores but the latter is approximately 4 times cheaper at the time of this writing.

Adversarial samples

So far we have examined samples for which a context exists in the test data. Approximately 25% of our dataset consists of adversarial samples, where no context is expected. Adversarial examples can be deceptive for LLMs. Relying on pragmatic assumptions or world knowledge from their pre-training, they may interpret implicit cues in the data. For instance, if a patient’s smoking status is not discussed in a document, it might be inferred that the patient is a non-smoker, as otherwise it would certainly be mentioned. But since it cannot be asserted with certainty, the rule in such case is to output null. Commercial models struggle with this behavior, as seen in the following table.

Observations :

• All models vastly overlook adversarial examples and generate a lot of superfluous contexts.

• This is particularly true for GPT3.5 compared to GPT4 and GPT4o. As per the example in introduction, GPT3.5 can be very wordy and will try to generate an answer with a context, sometimes with little regard to the actual factual content in the source.

• Few-shot prompting slightly mitigates this effect.

Fine-tuned Models

In this section, we fine-tune open source LLMs on our context attribution dataset to see if this skill can be easily acquired with supervised learning.

Fine-tuned model results

We fine-tune LLMs of different architectures and different number of parameters (from 160M to 24B). We measure their valid citation rate, before fuzzy matching, in the non-adversarial case.

The data exhibit a roughly logarithmic relationship between context citation capacity and number of parameters (with notable deviations observed in the Qwen family, for which we propose an explanation later).

It also shows that word-for-word quotation in an autoregressive manner is not trivial, and requires a sizable number of model parameters before it can be relied upon. Below are the complete metrics for all models.

Most metrics increase steadily with model size. The average citation length is more stable, but is now way closer to the ground truth compared to commercial models. This show how fine-tuning on actual examples can implicitly teach what humans prefer.

While still useful, the fuzzy matching becomes less important as the model size grows and the raw valid citation rate gets better.

Akin to API models, it is interesting to note that the majority of failure cases do not stem from the model's inability to produce a valid citation but rather from its tendency to abstain, resulting in false negatives. We can see that by looking at the detailed breakdown of match types below.

Even when controlling for the number of parameters, there is observable variation across models in their propensity to generate citations rather than null outputs.

In parallel, looking at adversarial samples, and plotting the False Positive Rate, we also observe a similar discrepancy:

Specifically, it appears that models from the Qwen2.5 family, which underperform in the non-adversarial case, are significantly better here. They are less likely to produce unnecessary contexts for adversarial samples.

Before/After Fine-tuning

So far we have glossed over the necessity of fine-tuning open-source models. Below we provide two graphs of the performance before/after fine-tuning to highlight the performance improvement. To evaluate base models before fine-tuning, we have given them the same prompt as the commercial models, but no few-shot

On the left is the evolution of the valid citation rate. For instance, before fine-tuning, Mistral-Nemo is able to generate a valid citation in 72.3% of cases. After fine-tuning, this number rises to 87.8%. On the right is the evolution of the true negative rate. In other words, on adversarial samples, how many times did the model successfully generated null as an answer. This rate increases from 10.2% to 88.9% for Mistral-Nemo for example.

Note that some models, such as all from the SmolLM2 family, are unable to generate valid JSONs before fine-tuning, and hence consistently score 0 on the benchmarks. While fine-tuning improves on both sides, it is remarkably impactful for adversarial samples, as models have a hard time grasping the concept from instructions only.

Fine-tuned models also score much higher on all quality metrics. For instance, for Mistral-Small-24B, the proportion of exact match rises from 21% to 36%, the average citation length drops from 95 to 67 and the average similarity increases from 0.49 to 0.62.

Summary

As we have shown, understanding the performance of LLMs for context attribution requires examining two dimensions, depending on whether the information exists in the source or not. To visualize this better, we can make a 2D plot balancing the valid citation rate (or True Positive Rate, frequency with which the model generates a valid context when expected) with the True Negative Rate (frequency with which the model correctly abstain). In this plot, we include commercial models and fine-tuned models, with disk area proportional to the number of parameters.

This visualization suggests the existence of precision/recall trade-off between rather conservative models but with a high precision and more aggressive models more prone to invent citations but with higher recall.

Conclusion

Large Language Models (LLMs) are well suited for extracting structured data from unstructured documents. However, their generative nature introduces challenges such as hallucinations and lack of interpretability, which are critical in domains like healthcare where reliability and traceability are paramount. Context attribution aims to mitigate these issues by requiring models to cite verbatim excerpts from source material. We investigated whether purely generative context attribution is feasible, allowing the model to dynamically quote the most relevant passages word-for-word.

Key findings from our experiments include:

• Commercial models (GPT3.5, GPT4, GPT4o) can perform zero-shot verbatim citation, with performance improving notably through few-shot prompting. However, fuzzy matching is necessary to recover a sizable amount of what the models originally output. Moreover, they still struggle with adversarial cases, often generating unnecessary or incorrect citations when no relevant context exists.

• Fine-tuned open-source models show a strong correlation between model size and citation accuracy, with larger models achieving better valid citation rates than commercial models for a fraction of the cost. Fine-tuning improves both citation rate and citation quality, and drastically enhance the ability to abstain when no context is available.

• Trade-offs exist between precision and recall and adversarial analysis is crucial for real world applications. Some models are conservative, minimizing false citations but missing valid ones, while others are more aggressive, capturing more correct contexts at the cost of occasional hallucinations or misattributions.

Future work

Future research could explore allowing the model to produce multiple citations per prediction, as some answers require combining evidence from several disjoint passages in the source text. Additionally, a more robust approach might involve constrained generation techniques to enforce that generated citations are exact extracts from the source document or to control their length through the JSON schema specification. Another promising direction is to leverage the transformer's attention mechanism as a way to identify relevant contexts, building on existing work in interpretability and attribution. These enhancements could further improve the reliability and granularity of context attribution in LLMs, which is needed to make LLM outputs more trustworthy and actionable in high-stakes environments.

Subscribe now

Join Us at Lifen!

At Lifen, we’re at the forefront of leveraging AI to solve real-world challenges in healthcare. If you’re passionate about building reliable, interpretable AI systems and want to work on impactful problems, we’d love to hear from you!

We’re currently hiring for roles in machine learning, engineering and more. Check out our careers page to explore opportunities and join our mission.

Our objective was to develop a tool that:

Thanks for reading Lifen AI! Subscribe for free to receive new posts and support my work.

• streamline the support process by making our explainable AI tools easily accessible to authorized users

• is easily maintainable and always up-to-date with our AI

• provides some insight into how the ML model made its prediction

The end result

The result is a web app lifen-ai-explainer accessible internally to Data Scientists and Customer Support. Given a document id (provided by the client requesting support), this web app will display the anonymized document and predictions our AI made. (Predictions are also anonymised but made on the original document, otherwise, changes on the doc by anonymization process could produce different results)

We built lifen-ai-explainer and the pseudonymisation algorithm internally for this purpose and it’s been fully integrated into our daily processes since mid-2020.

Explaining classification

In addition to anonymizing and displaying the predictions, we try to explain them. For example, we want to understand how our machine learning text classifiers (eg: document type, patient gender) made their predictions by highlighting which words impacted the prediction the most.

Here is a demo of what this looks like on the patient gender classification algorithm (whose goal is to detect the gender of the patient mentioned in the document — without relying on the name — to. increase identitovigilance)

This feature proved useful in real-world situations to understand how for example a misread/misplaced word or simply a typo leads to a wrong prediction.

Explaining NER

We also use this tool to debug our NERs models. On the document view we show the raw softmax output (score for each class for each token), on the left side we show the final post-processed prediction.

How does it work?

Our classifiers are made with an Embedding + LSTM architecture (long documents, resources, and time constraints mean no Transformers). However, this architecture is not out-of-the-box well suited for explainability purposes.

Let’s go into detail to understand why. Here we use the last hidden state that we pass to a dense layer and softmax to produce our prediction:

The problem is simply that we have no idea which word impacted the final prediction. (There is also a problem of vanishing gradient and bottleneck where if the important words are at the beginning of a very long sequence, the LSTM has to learn to remember it for many time steps) But we can fix that with a simple inexpensive tweak in the model.

Global attention layer

Our solution is to add a global attention layer (or attentive pooling layer) [1] that compute an attention score for each token with a dense + softmax layer which is then used to compute a weighted sum of all the hidden state at each time step (which roughly correspond to information contained in the current word).

This weighted sum is therefore a whole document embedding (similar to the last hidden state that we used in the previous architecture) that we can use for classification, but we also have in bonus those attention scores per word which we can interpret as “importances scores” for each word. Those scores proved empirically very interpretable.

The layer code

This layer is a simple Keras layer made from well-known building blocks. In practice, we use more than one attention head which can make the interpretation more tricky but improve performance.

def MultiGlobalAttention(timesteps: tf.Tensor, n_heads: int = 1) -> tf.Tensor:
    """Global Attention layer. Computes a weighted sum of all the timesteps.
    Input: N timesteps of dim D
    Output: 1 of dim D
    Zhou et al https://www.aclweb.org/anthology/P16-2034  §3.3"""

    attention = Dense(n_heads, use_bias=False, activation="linear")(timesteps)
    attention = Activation(softmax_axis(1), name="attention")(attention)
    weighted_sum = dot([attention, timesteps], axes=1)
    return Flatten()(weighted_sum)

How to get those attention scores? We extract an attention sub-model that is not run in typical inference scenarios, but only for support in lifen-ai-explainer.

def attention_model(model: Model) -> Model:
    """make attention_model from model"""
    return Model(inputs=model.input, outputs=model.get_layer("attention").output)

Architecture of lifen-ai-explainer (+ lifen-ai)

The web app is a React Single-page application written in TypeScript that communicates with a REST python backend based on FastAPI (the backend also serves the static files for the frontend).

The frontend tooling is based on Vite which handles all the complexity of a modern frontend app (we rather do data science than tinker with babel/webpack config files).

The web app needs to make the same predictions as lifen-ai. We therefore decided to build the backend as a python app that can import the existing lifen-ai as a library.

Python lacks explicit access modifiers, which can make it difficult to maintain good encapsulation and ensure the maintainability of a large interconnected codebase. We use import-linter to enforce encapsulation rules and allow lifen-ai-explainer to import lifen-ai but not the other way around.

How to keep our ai-explainer in sync with new features?

Our AI is rapidly evolving and we need lifen-ai-explainer to stay in sync at all times. We never want to be in a position where there is a new feature that we cannot debug with lifen-ai-explainer.

Mono repo & mypy & front-back client generation

We decided to put both apps in the same mono repo. This allows atomic changes: when making a Pull Request to add a prediction to lifen-ai, you can simultaneously add the implementation to lifen-ai-explainer. This is enforced by design: because 100% of our codebase is type annotated with mypy, a breaking change in lifen-ai will break CI for lifen-ai-explainer, so you are required to keep it up to date. Our unit & integration tests do the rest.

The frontend client is autogenerated by Orval using the OpenAPI spec which is also autogenerated by FastAPI. As a result, any changes made to the output of a model will cause the CI to fail and direct you to the exact location in the frontend code where updates are required 🎉.

We use poetry to manage python dependencies which don’t support mono-repo but allow us to split dependencies into groups. This way, lifen-ai-explainer includes all the dependencies of lifen-ai but not the other way around.

lifen-ai-explainer has been fully integrated into our daily processes since mid-2020 and we have since expended its features with a dataset labellisation interface, and a post-training errors review tool.

Thanks for reading Lifen AI! Subscribe for free to receive new posts and support my work.

At Lifen we process medical PDF documents from thousands of clients. Those PDF documents come with a huge variety of layouts:

We need to perform layout detection from this 2D grid of words to produce a coherent 1D sequence. This 1D sequence can then be fed into our NLP models or sent as plain text when clients do not support PDF or image attachments.

Thanks for reading Lifen AI! Subscribe for free to receive new posts and support my work.

Mistakes in this layout detection could result in nonsensical extracted text. This could mean for exemple mixing addresses of two recipients together.

Imagine in the left image above if we failed to understand that the green and yellow blocks are two columns that should be read one after the other: we would create sentences mixing pieces of text from both blocks.

Isn’t layout information already encoded in PDFs?

Most of the time, yes. There is a sense of order in the way it is encoded, but there are no means to be sure this is the case, and results can be pretty bad if we just trust PDFs blindly. Furthermore, in the case of images PDF (which needs OCR), there is no text order to recover.

Objective: be fast & accurate

Existing solutions

Existing solutions (such as pdfplumber, Parsr) use simple techniques such as XY cut¹ and did not perform well enough on our dataset. Furthermore, because accurate text extraction is critical for our product, we need to be able to constantly fine-tune our solution to be as accurate as possible with new clients. Existing solutions did not allow us the level of customization we needed.

Avoid OCR when possible

Multiple solutions combine OCR with layout detection. However, most of the documents we receive (~85%) have a clean extractable text and do not require OCR. Furthermore, doing OCR on every document we process would be slow and expensive we’re happy to avoid as much as we can. For image documents, we first apply OCR then use the extracted text as a starting point just as extractable PDFs.

Our solution

Previous work

There is an active community of research in the field of layout analysis, with academic conferences such as the International Conference on Document Analysis and Recognition. One resource we found particularly valuable to navigate this field is survey of algorithms².

However, while reviewing papers we found out that few share the characteristics we are looking for: start from extractable PDFs, fast and fine-tunable.

One great paper using the same starting point as us (no OCR, words and not pixels) and with great care for speed is Object-Level Document Analysis of PDF Files³. The idea is to build a graph of words, discard edges linking words from different paragraphs until the connected components left are the paragraphs. The ideas we took from this paper are:

• Reduce the number of words in linear time by merging words that without a doubt belong to the same line

• Build a graph as fast as possible by looking only for direct neighbors in the four directions.

• “Best-first clustering” algorithm: sort edges (mainly by length) and choose to discard them or not one by one by using previous decisions to make informed decisions on longer trickier edges.

However, the algorithms they use to sort edges and choose to discard them, are complex sequences of rules that we found difficult to fine-tune and maintain.

From A Machine Learning Approach for Graph-Based Page Segmentation⁴, we took the idea of using a machine learning model to classify edges.

Bring everything together

1. Pre-processing

We’re extracting words and their positions using pdftotext --bbox-layout. This gives us a list of words in 2D grid and an estimation by pdftotext of word order (in an XML format similar to hOCR).

2. Build graph

We build the word graph using a custom implementation of the algorithm from³.

3. Give a should_merge score to each edge:

Using a manually tagged dataset, we train a Random Forest to give a should_merge score for each edge using features.

Features definitions:

• dist_font_size: distance between words in units of font_size (or mean font_size if not the same between words)

• xml_neighbours: neighbours according to PDF infos extracted by pdftotext (accurate ~90% of the time)

• nb_aligned_left: number of words aligned left with this word (does it looks like this word is on the left side of a paragraph)

3. Sort edges by should_merge score and keep certain

Using a threshold that gives us a precision of 1.00 during evaluation, we keep every edge that is “certain” (recall at this threshold is 0.75).

Our second threshold aims for a precision > 0.9, gives us “possible” edges.

Rest of edges are discarded.

4. Make decisions about “possible” edges using current state

Our random forest only sees two words at the time (but some features such as nb_aligned_left give more global context). For less obvious edges (with a lower should_merge score), we make decisions based on the current state of the layout. We compute a “stateful” coefficient that multiply the should_merge score.

For exemple, when considering an edge between two words from two clusters that seem to come from one same column, we increase the should_merge score.

Future work

At this point our library is too much integrated with the rest of our app to be split into an open-source solution. We are considering open-sourcing it in the future. In the meantime, do not hesitate to contact us if you want more details.

We’re also thinking of using a more end-to-end approach by combining the layout extraction with the downstream task directly in our NLP models. LayoutLM⁵ is a promising work in this direction.

References

[1] Hierarchical representation of optically scanned documents https://core.ac.uk/download/pdf/84308932.pdf

[2] A comprehensive survey of mostly textual document segmentation algorithms since 2008 https://hal.archives-ouvertes.fr/hal-01388088/document

[3] Object-Level Document Analysis of PDF Files https://www.dbai.tuwien.ac.at/staff/hassan/files/p47-hassan.pdf

[4] A Machine Learning Approach for Graph-Based Page Segmentation https://www.researchgate.net/publication/328719252_A_Machine_Learning_Approach_for_Graph-Based_Page_Segmentation

[5] LayoutLM: Pre-training of Text and Layout for Document Image Understanding https://arxiv.org/abs/1912.13318

Thanks for reading Lifen AI! Subscribe for free to receive new posts and support my work.