DEV Community: Parth Patil

Reducing hallucinations when extracting data from PDF using LLMs

Parth Patil — Thu, 27 Feb 2025 05:38:03 +0000

Strategies for Mitigating Hallucinations in Large Language Models During PDF Information Extraction

The extraction of structured data from PDF documents using large language models (LLMs) represents a significant challenge in natural language processing, particularly due to the risk of hallucination—the generation of plausible but factually incorrect or unsupported information. This report synthesizes insights from recent research and practical implementations to provide a comprehensive framework for designing prompts that minimize hallucinations while maximizing extraction accuracy. By analyzing technical approaches across retrieval-augmented generation, few-shot learning, and self-assessment mechanisms, we outline actionable strategies for practitioners building PDF-based information extraction systems.

Foundational Concepts in Hallucination Mitigation

Defining Hallucination in Document Processing Contexts

Hallucination in LLMs manifests when models generate information not present in source documents, often due to:

Overgeneralization from limited context[4][8]
Pattern completion biases in pretrained models[10]
Ambiguity tolerance in poorly structured documents[9]

Recent studies demonstrate that 68% of extraction errors in financial document processing stem from hallucinated numerical values, while 32% involve incorrect entity relationships[9]. These errors persist across both commercial and open-source models, with GPT-4 Turbo showing 12% lower hallucination rates compared to LLaMA-2 in controlled tests[6][9].

Core Components of Effective Extraction Prompts

Structural Elements for Hallucination Prevention

An optimized prompt should integrate:

1. Explicit Instruction Framework

You are an expert document analyst tasked with:
- Extracting ONLY information explicitly present in the provided PDF text
- NEVER inferring, assuming, or generating content beyond what is written
- Flagging sections where requested data appears missing or ambiguous

This declarative structure establishes clear role boundaries, reducing the model's tendency to "fill in gaps" through speculation[3][8].

2. Contextual Grounding Mechanisms

Available document content:
{PDF_TEXT}

Instructions:
1. Analyze ONLY the text between  tags
2. If asked about information not present, respond "Not explicitly stated"
3. For numerical values, verify against original formatting in section [X]

Grounding the model in specific document sections decreases hallucination likelihood by 41% compared to generic prompts[7][9].

Advanced Prompt Engineering Techniques

Multistage Verification Protocols

Implementing sequential processing steps enhances accuracy:

Primary Extraction

   Extract all company names mentioned in the document. 
   Present as: ["Name", "Context Paragraph Number"]

Cross-Validation

   For each extracted name:
   - Confirm presence in original text
   - Note any spelling variations
   - Flag potential OCR errors

Confidence Scoring

   Assign confidence levels (1-5) based on:
   - Text clarity
   - Contextual support
   - Document consistency

This approach reduced hallucination incidents by 63% in clinical document processing trials[8].

Few-Shot Learning Integration

Including annotated examples significantly improves pattern recognition:

Example 1:
Document Text: "The invoice (No. INV-2023-045) dated 2023-07-15 totals $4,500."
Extracted: {"invoice_number": "INV-2023-045", "date": "2023-07-15", "amount": 4500}

Example 2:
Document Text: "Payment due by end of month per terms 2/10 net 30."
Extracted: {"payment_terms": "2/10 net 30", "due_date": "Not explicitly stated"}

Systems using 3-5 examples per document type achieve 89% higher precision than zero-shot approaches[3][6].

Technical Implementation Considerations

OCR Quality Assurance

Implement preprocessing checks:

def validate_ocr_quality(text):
    alpha_ratio = sum(c.isalpha() for c in text)/len(text)
    if alpha_ratio 0.85), these systems achieve 94% error detection rates[4][7].

---

## Conclusion

The optimal prompt structure combines explicit constraints, contextual grounding, and validation mechanisms:

markdown
As a precision-focused document analyst, process the PDF text between tags:

Extract only explicitly stated information matching these fields: [FIELD_LIST]
For each field: a. Include verbatim text from source b. Note page number of occurrence c. If absent, mark "Not present"
Format output as JSON with confidence scores (0-1)
Prohibit any assumptions, inferences, or external knowledge

Document Content:
{PDF_TEXT}




This framework reduced hallucination rates to 4.2% in recent benchmark tests across 15,000 documents, compared to 23.8% in baseline models[8][9]. Future research directions include developing model-specific uncertainty estimators and hybrid symbolic-neural verification systems to further enhance reliability.

Data Processing with Elixir (Part 2)

Parth Patil — Sun, 13 Aug 2023 16:41:34 +0000

In the last post we saw how to process data in Elixir using the Task.async module. It's a great way do parallel processing in Elixir. Please read the previous post to see what the dataset schema looks like in order to understand the code below.

In this part we will see how we can use the Flow library to process Github events. Flow is great library for processing data in Elixir. It allows one to utilise all cores on the machine. Its a much nicer way to write data pipelines than using Task.async as it allows us to use the familiar map and reduce abstractions that we are so used to in the Elixir world.

Following is the code to use the Flow library to process the TSV file chunks in parallel

# BINARY
tab_char = :binary.compile_pattern("\t")

streams =
  for file <- File.ls!("/Users/parth/temp/github_events/clickhouse_tsv") do
    full_path = "/Users/parth/temp/github_events/clickhouse_tsv/#{file}"
    File.stream!(full_path, [{:read_ahead, 100_000}])
  end

streams
|> Flow.from_enumerables()
|> Flow.map(fn line ->
  line |> String.split(tab_char) |> Enum.at(1)
end)
|> Flow.partition()
|> Flow.reduce(fn -> %{} end, fn event_type, acc ->
  Map.update(acc, event_type, 1, &(&1 + 1))
end)
|> Enum.to_list()

# Result
# [{"CommitCommentEvent", 9954564}, {"CreateEvent", 295163106}]

In the above code I create a list of streams - one each for each tsv file chunk I have. Next I pipe it into Flow.from_enumerables that creates flow definition from it. Then Flow.map processes each line in parallel. Under the hood Flow.map will spin up as many mapper processes as there are CPU cores. You can read more about Flow in its excellent documentation.
The mapper is very simple - it splits the line on tab character and returns the 2nd item in the list.

Then the Flow.partition applies a hash function on the results of the Flow.map stage and sends the result to one of the reducers. The strings that have the same hash value will go to the same reducer.

Finally there is a Enum.to_list() at the end of the Flow that actually causes the Flow pipeline to start running. Remember Flow is lazy it won't start running until you call a Enum.to_list or a Flow.run on the pipeline.

When I ran the above pipeline it took around 630 seconds to run and the CPU usage was around consistent 4.5 cpu cores

I wanted to see if using a regex in the Flow.map stage will make things any faster. Below is updated code with regex parsing in the Flow.map stage.

regex = Regex.compile!("[^\t]+\t([^\t]+)\t.+")

streams =
  for file <- File.ls!("/Users/parth/temp/github_events/clickhouse_tsv") do
    full_path = "/Users/parth/temp/github_events/clickhouse_tsv/#{file}"
    File.stream!(full_path, [{:read_ahead, 100_000}])
  end

streams
|> Flow.from_enumerables()
|> Flow.map(fn line ->
  [[_, event_type]] = Regex.scan(regex, line)
  event_type
end)
|> Flow.partition()
|> Flow.reduce(fn -> %{} end, fn event_type, acc ->
  Map.update(acc, event_type, 1, &(&1 + 1))
end)
|> Enum.to_list()

The above code ran in 470 secs which is a ~25% improvement in speed compared to 630 seconds. I suspect the regex approach creates less garbage compared to the previous approach of splitting the string into a list and then picking the second item and hence might be faster.

In the next part we will explore if we can further optimise this code.

Data processing with Elixir (Part 1)

Parth Patil — Sat, 12 Aug 2023 07:28:28 +0000

I wanted to see how fast is the Elixir flow library when processing non trivial amount of data. I downloaded part of GH Events dataset from ClickHouse's website. This dataset has roughly 300 million rows of Github events. It's in TSV format. Uncompressed it comes out to be 78 GBs of data. I have split the file in 10 parts so we can benchmark processing this dataset in parallel.

The schema of this events in the ClickHouse table format looks as follows

CREATE TABLE github_events
(
    file_time DateTime,
    event_type Enum('CommitCommentEvent' = 1, 'CreateEvent' = 2, 'DeleteEvent' = 3, 'ForkEvent' = 4,
                    'GollumEvent' = 5, 'IssueCommentEvent' = 6, 'IssuesEvent' = 7, 'MemberEvent' = 8,
                    'PublicEvent' = 9, 'PullRequestEvent' = 10, 'PullRequestReviewCommentEvent' = 11,
                    'PushEvent' = 12, 'ReleaseEvent' = 13, 'SponsorshipEvent' = 14, 'WatchEvent' = 15,
                    'GistEvent' = 16, 'FollowEvent' = 17, 'DownloadEvent' = 18, 'PullRequestReviewEvent' = 19,
                    'ForkApplyEvent' = 20, 'Event' = 21, 'TeamAddEvent' = 22),
    actor_login LowCardinality(String),
    repo_name LowCardinality(String),
    created_at DateTime,
    updated_at DateTime,
    action Enum('none' = 0, 'created' = 1, 'added' = 2, 'edited' = 3, 'deleted' = 4, 'opened' = 5, 'closed' = 6, 'reopened' = 7, 'assigned' = 8, 'unassigned' = 9,
                'labeled' = 10, 'unlabeled' = 11, 'review_requested' = 12, 'review_request_removed' = 13, 'synchronize' = 14, 'started' = 15, 'published' = 16, 'update' = 17, 'create' = 18, 'fork' = 19, 'merged' = 20),
    comment_id UInt64,
    body String,
    path String,
    position Int32,
    line Int32,
    ref LowCardinality(String),
    ref_type Enum('none' = 0, 'branch' = 1, 'tag' = 2, 'repository' = 3, 'unknown' = 4),
    creator_user_login LowCardinality(String),
    number UInt32,
    title String,
    labels Array(LowCardinality(String)),
    state Enum('none' = 0, 'open' = 1, 'closed' = 2),
    locked UInt8,
    assignee LowCardinality(String),
    assignees Array(LowCardinality(String)),
    comments UInt32,
    author_association Enum('NONE' = 0, 'CONTRIBUTOR' = 1, 'OWNER' = 2, 'COLLABORATOR' = 3, 'MEMBER' = 4, 'MANNEQUIN' = 5),
    closed_at DateTime,
    merged_at DateTime,
    merge_commit_sha String,
    requested_reviewers Array(LowCardinality(String)),
    requested_teams Array(LowCardinality(String)),
    head_ref LowCardinality(String),
    head_sha String,
    base_ref LowCardinality(String),
    base_sha String,
    merged UInt8,
    mergeable UInt8,
    rebaseable UInt8,
    mergeable_state Enum('unknown' = 0, 'dirty' = 1, 'clean' = 2, 'unstable' = 3, 'draft' = 4),
    merged_by LowCardinality(String),
    review_comments UInt32,
    maintainer_can_modify UInt8,
    commits UInt32,
    additions UInt32,
    deletions UInt32,
    changed_files UInt32,
    diff_hunk String,
    original_position UInt32,
    commit_id String,
    original_commit_id String,
    push_size UInt32,
    push_distinct_size UInt32,
    member_login LowCardinality(String),
    release_tag_name String,
    release_name String,
    review_state Enum('none' = 0, 'approved' = 1, 'changes_requested' = 2, 'commented' = 3, 'dismissed' = 4, 'pending' = 5)
)
ENGINE = MergeTree
ORDER BY (event_type, repo_name, created_at)

I am doing this benchmark locally on my Macbook M1 Pro with 10 cores and 16 GB ram.

On this dataset I wanted to do a few queries as part of benchmarking. Following are the ones I picked

Frequency count of the event_type
Top repository
Top github user

Firstly I wanted to set a baseline on how fast can I process this data on my laptop using any modern stack and with minimal effort on my end ;) I chose ClickHouse Local Its an amazing tool to process files locally on your laptop without having to even start a server. It automatically uses all the available cores on the host machine.

I executed the 1st query via ClickHouse Local by running the following

~/tools/ch_may_23/clickhouse local -q "select c2, count(*) as cnt from file('github_events_a*', TabSeparated) group by c2"

The query was consistently using around 5.7 cpu cores out of 10 and finished in 85 secs.

As you see ClickHouse Local allows you to directly query a bunch of files by giving a glob pattern and specifying SQL query on this data. Each column in the input data gets automatically named as c1, c2, etc...

Now let's see how fast we can process this data in Elixir. If we were doing a single core processing we would use the Stream module's functions. But that would take too long to process 300m and 78GB of data so I am not going to bother benchmarking it. Let's jump to the next best thing we can do in Elixir. I am going to use Task.async_stream to process the 10 files in parallel. This is how the code looks

tab_char = :binary.compile_pattern("\t") # BINARY

streams =
  for file <- File.ls!("/Users/parth/temp/github_events/clickhouse_tsv") do
    full_path = "/Users/parth/temp/github_events/clickhouse_tsv/#{file}"
    File.stream!(full_path, [{:read_ahead, 100_000}])
  end

streams
|> Task.async_stream(fn stream ->
  stream
  |> Stream.map(fn line ->
    line |> String.split(tab_char) |> Enum.at(1)
  end)
  |> Enum.reduce(%{}, fn event_type, acc ->
    Map.update(acc, event_type, 1, &(&1 + 1))
  end)
end, max_concurrency: 10, timeout: :infinity)
|> Enum.to_list()
|> Enum.flat_map(fn {_, mp} -> Map.to_list(mp) end)
|> Enum.reduce(%{}, fn {etype, count}, acc ->
  Map.update(acc, etype, count, &(&1 + count))
end)

In the above code there are some optimisations like pre-compiling the pattern that will be used to split each tab separated line in the files. Task.async_stream processes all the 10 files in parallel. The above took about 625 secs to process and CPU utilisation was around 8 CPU cores out of 10

625 secs for processing 300 million events and 78 GB of data is not too shabby given that we have the full power of Elixir at our disposal to process this data and are not constrained by limitations of SQL.

In Part 2 we will explore if we can make the Elixir processing faster.

Beautiful pipelines in Elixir

Parth Patil — Fri, 11 Aug 2023 18:45:35 +0000

Even though I have used Elixir for several years now it still
amazes me how much you can achieve with a single pipeline in Elixir language. The other day I had to find the total size of the files in a given dir. I thought of just parsing the output of the ls -l command and then adding up all the file sizes to get the total. Here is what I ended up with.

"ls -l /Users/parth/temp/github_events/2015/"
|> to_charlist()
|> :os.cmd()
|> to_string()
|> String.split("\n")
|> Enum.reject(&String.starts_with?(&1, "total"))
|> Enum.map(&String.split(&1, ~r/\s+/))
|> Enum.map(&Enum.at(&1, 4))
|> Enum.reject(&(is_nil &1))
|> Enum.map(&(String.to_integer &1))
|> Enum.sum()
|> then(&(&1/:math.pow(10, 9)))
|> Float.round(2)
|> IO.inspect(label: "Size in GBs => ")

P.S In practise this code might be shortened by using the standard library functions associated with file handling. But it was fun coming up with this!

Using ECharts in Elixir Livebook

Parth Patil — Tue, 08 Aug 2023 04:37:01 +0000

I have been using a lot of Elixir Livebook lately and I am having a lot of fun with data exploration and analysis using Livebook. Livebook has great support for drawing charts using the excellent Vega-Lite library. Even though Vega-Lite is great and is sufficient for many usecases I wanted to try using Apache ECharts inside Livebook. We use ECharts at work and are happy with it.

I looked around and could not find any prior work/library that allowed me to use ECharts in Livebook so I decided to create a hex package for Livebook that used ECharts to draw charts. Its called lb_echarts check it out if you are interested.

After some research I found that Kino.JS could be used to integrate custom JS into Livebook.

To use Kino.JS I had to use Kino.JS in my module and then define the init JS function within the asset macro. This looks as follows. Note in the following code I have only shown how do define the asset section, later I will show how to write rest of the code.

defmodule LbEcharts do
  @moduledoc """
  Library to use Echarts in LiveBook
  """

  use Kino.JS

  asset "main.js" do
    """
    export async function init(ctx, chart_data) {
      await ctx.importJS("https://cdn.jsdelivr.net/npm/echarts@5.4.3/dist/echarts.min.js");

      ctx.root.innerHTML = `
        <div id="echart_window" style="width: 800px;height:500px;"></div>`

        var myChart = echarts.init(document.getElementById('echart_window'));

        // Specify the configuration items and data for the chart
        var options = JSON.parse(chart_data);

        // Display the chart using the configuration items and data just specified.
        myChart.setOption(options);
    }
    """
  end
end

The entire contents of the asset section are enclosed in triple quotes. Whatever you write inside these triple quotes are considered as JS code. In the above code you will note that I have used async and await in the JS code. This is because I am loading a JS file and I want to wait for the JS file to be loaded before executing rest of the code.

The init function in JS gets the context in ctx variable and second argument is what was passed to the Kino.JS.new() function. One can define the HTML markup for the output Livebook cell in the ctx.root.innerHTML attribute. Note how the markup is enclosed in the backticks.

Now lets look at how we implement a bar chart.

defmodule LbEcharts do
  @moduledoc """
  Library to use Echarts in LiveBook
  """

  use Kino.JS

  def new(chart_data) do
    Kino.JS.new(__MODULE__, chart_data)
  end

  def bar(opts) do
    options =
      %{
        "title" => %{"text" => opts.title},
        "tooltip" => %{},
        "legend" => %{"data" => opts.legend},
        "xAxis" => %{"data" => opts.x},
        "yAxis" => %{},
        "series" => [
          %{
            "name" => "sales",
            "type" => "bar",
            "data" => opts.y
          }
        ]
      }
      |> Jason.encode!()

    LbEcharts.new(options)
  end

  asset "main.js" do
    """
    export async function init(ctx, chart_data) {
      await ctx.importJS("https://cdn.jsdelivr.net/npm/echarts@5.4.3/dist/echarts.min.js");

      ctx.root.innerHTML = `
        <div id="echart_window" style="width: 800px;height:500px;"></div>`

        var myChart = echarts.init(document.getElementById('echart_window'));

        // Specify the configuration items and data for the chart
        var options = JSON.parse(chart_data);

        // Display the chart using the configuration items and data just specified.
        myChart.setOption(options);
    }
    """
  end
end

Following is how you would use the library to draw a bar chart

LbEcharts.bar(%{
  title: "Bar Chart",
  legend: ["sales"],
  x: ["shirt", "cardign", "chiffon shirt", "pants", "heels", 
  "socks"],
  y: [5, 20, 36, 10, 10, 20]
})

To learn more check out lb_echarts