AI Task Done, Output File Corrupted?! How Preserving 'Transcripts' Saved My Data

Hey everyone, it's me, your friendly neighborhood senior dev. I've been running some AI agents for my side hustle during weeknights, and recently had a heart-stopping moment I wanted to log and share.

My market analysis agent, which takes several hours to complete, sent a Slack notification that its task was done. "Great!" I thought, and opened the final report file it generated. To my horror, the content was truncated midway. The file size was also suspiciously small.

My blood ran cold for a second. This task involves a fair number of LLM API calls and a long execution time. Honestly, rerunning it was out of the question, both in terms of time and cost.

Cause Unknown, But "Writes Can Fail" Is a Reality

First, I tried to figure out why the file was corrupted. Looking at the agent's execution logs, the process exited with a normal 0 status code. No exceptions were thrown.

Possible culprits include:

• The process was killed by some external factor while writing a large amount of text data to the file.
• The process terminated before the I/O buffer was flushed to storage.
• ...or something along those lines.

Honestly, pinpointing the root cause is difficult. But the important thing is the fact that "final writes to a file are not always guaranteed to succeed." Especially when writing large texts of tens of thousands of characters at once with a single write() call, there's always a risk of it ending in an incomplete state for some reason.

In this case, data salvage was my top priority, more so than root cause analysis.

The Lifeline: 'Transcripts'

This is where I really had to pat myself on the back: I had implemented a "transcript" design that logs the agent's entire thought process to a file.

This transcript includes:

• The initial prompt given to the agent
• The Chain of Thought
• The Python code executed
• The standard output of the code execution
• The history of API calls with the LLM

...Essentially, a complete, chronological record of everything from task start to finish, appended to a log file.

And sure enough, when I rummaged through this log, there it was: the log of the step where the agent generated the final report. At the end of the transcript, along with a ## Final Report ## marker, the full text of the generated report was perfectly recorded.

The transcript file was written in append mode step-by-step during processing, so even if something failed at the very last moment, all logs up to that point remained intact. This truly became my lifeline.

I discarded the corrupted final output file and simply copied and pasted the report section from the transcript into a new file. This saved me hours of work and API costs. Seriously, I was so relieved.

Implementing Automated Validation and Recovery

While manually recovering the data was good, I really didn't want to be woken up in the middle of the night if the same thing happened again. So, I decided to automate this recovery process.

Specifically, I added a step at the end of the agent's task flow: "Output file validation and automatic recovery on failure."

Here's what the code looks like:

def validate_and_recover(output_path, transcript_path):
    """
    Validates the output file and attempts recovery from the transcript if corrupted.
    """
    try:
        with open(output_path, 'r', encoding='utf-8') as f:
            content = f.read()
        # For this case, a simple validation: > 40k characters and a specific marker at the end
        if len(content) > 40000 and 'END_OF_REPORT' in content:
            print(f'Output file {output_path} is valid.')
            return True # File is good
    except (IOError, UnicodeDecodeError):
        # If file doesn't exist or can't be read (corrupted), proceed to recovery
        pass

    # Recovery logic starts here
    print(f'Output file {output_path} is corrupted or missing. Attempting recovery from transcript...')
    try:
        with open(transcript_path, 'r', encoding='utf-8') as f_trans:
            full_transcript = f_trans.read()
            # Extract the final report section from the transcript
            report_start = full_transcript.rfind('## Final Report ##')
            if report_start != -1:
                recovered_content = full_transcript[report_start:]
                with open(output_path, 'w', encoding='utf-8') as f_out:
                    f_out.write(recovered_content)
                print(f'Successfully recovered the report to {output_path}.')
                return True
    except IOError:
        print(f'Error: Transcript file {transcript_path} not found.')

    print('Recovery failed.')
    return False

The process is simple:

1. First, try to open the final output file. If it can't be opened, or if its content doesn't match the expected format (in this case, character count and an end marker), it's considered abnormal.
2. If abnormal, open the transcript file.
3. Extract the report section from the transcript using regex or string search.
4. Overwrite the original output file with the extracted content.

By calling this function at the end of the task, if the file write ever fails, it will attempt to self-heal automatically before passing control to the next process. This increased the robustness of my system by a notch.

Lesson Learned: Proactive, Defensive Programming Saves Future You

The lesson I learned from this incident is subtle but crucial:

• For high-cost, non-idempotent processes, don't trust the final output alone.
• Keeping a "transcript" of all thought processes and intermediate generations serves not only for debugging but also as insurance for data recovery.
• It's essential to have a validation step (not just "generate and done") to ensure the generated output is complete. If possible, implementing a self-healing mechanism allows you to sleep soundly at night.

Especially in personal development, working with limited time and resources, unexpected troubles like this can be a real mental drain. It reinforced in me the importance of putting in a little extra effort now for defensive design, to make things easier for my future self.

I'll write again if I mess something up. Cheers. 👨‍💻