DEV Community: Roomal Seferaj

Transforming PDFs into Audio

Roomal Seferaj — Sat, 17 Aug 2024 09:13:51 +0000

In this guide, I will walk you through the process of converting PDF content into real-time audio playback using a combination of Python libraries. This approach is particularly useful for those who prefer to consume information audibly or for accessibility purposes. The code leverages text-to-speech technology and handles user interruptions gracefully.

Part 1 - Importing the Necessary Libraries

To begin, we need to import several Python libraries that will assist in loading PDFs, processing text, generating audio, and managing user interactions.

from gtts import gTTS
from io import BytesIO
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.chat_models import ChatOllama
from langchain_community.document_loaders import PyPDFLoader
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from pydub import AudioSegment
from pydub.playback import play
import signal
import sys
import threading

Overview of the Libraries:

gTTS: Google Text-to-Speech for converting text to audio.
BytesIO: In-memory binary stream for handling audio data.
LangChain: Tools for splitting text and processing it using language models.
PyPDFLoader: Specialized loader for extracting text from PDFs.
pydub: For audio manipulation and playback.
signal and threading: To manage user interruptions during playback.

Part 2: Handling User Interruption

We will allow the user to interrupt the audio playback gracefully. To achieve this, we set up a signal handler that listens for a Ctrl+C command.

# Flag to control the loop
stop_playback = False

def signal_handler(sig, frame):
    global stop_playback
    print("\nGracefully stopping playback...")
    stop_playback = True

# Assign the signal handler to SIGINT (Ctrl+C)
signal.signal(signal.SIGINT, signal_handler)

In addition to handling Ctrl+C, we also create a separate thread that listens for the Enter key press, providing an alternative way to stop playback.

# Function to listen for an Enter key press in a separate thread
def listen_for_stop():
    global stop_playback
    input("Press Enter to stop playback...\n")
    print("\nStopping playback...")
    stop_playback = True

# Start the listener thread
listener_thread = threading.Thread(target=listen_for_stop)
listener_thread.daemon = True
listener_thread.start()

Part 3: Loading and Splitting the PDF Document

Next, we load the PDF document using PyPDFLoader and split it into manageable chunks using RecursiveCharacterTextSplitter.

# Load and split the PDF document
loader = PyPDFLoader("/path/to/your/document.pdf")
pages = loader.load_and_split()

# Split text into chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=2048, chunk_overlap=100)
all_splits = text_splitter.split_documents(pages)

This approach allows us to process the document piece by piece, making it easier to generate and play audio incrementally.

Part 4: Generating Text Summaries

We use the ChatOllama model to generate summaries of the text chunks. The model is initialized with specific parameters, and a prompt template is created to guide the model's responses.

# Initialize the ChatOllama model
llm = ChatOllama(model="llama3:instruct", temperature=0.6)

# Create a prompt template
prompt = ChatPromptTemplate.from_template("Summarize the findings of: {page_content}")

# Define the chain
chain = prompt | llm | StrOutputParser()

Text Generation Function

We define a function to generate text summaries in chunks, which will be used later for audio playback.

def generate_text_chunks(page_content):
    try:
        text = chain.invoke({"page_content": page_content})
        sentences = text.split('. ')
        for sentence in sentences:
            yield sentence + '.'
    except Exception as e:
        print(f"Error generating text: {e}")

Part 5: Converting Text to Speech and Playing Audio

Once we have the text chunks, the next step is converting these chunks into speech and playing them.

# Function to play audio from a text chunk
def play_audio_chunk(text_chunk):
    if not text_chunk.strip():
        return
    try:
        tts = gTTS(text=text_chunk, lang='en')
        with BytesIO() as audio_fp:
            tts.write_to_fp(audio_fp)
            audio_fp.seek(0)
            audio_segment = AudioSegment.from_file(audio_fp, format="mp3")
            play(audio_segment)
    except Exception as e:
        print(f"Error generating or playing audio: {e}")

This function uses Google Text-to-Speech (gTTS) to generate audio from text and pydub to play the audio in real-time.

Part 6: Real-Time Text Generation and Audio Playback

Finally, we combine everything into a single function that handles real-time text generation and audio playback. This function will also respect user interruptions.

# Function to generate and play text in real-time
def generate_and_play():
    global stop_playback
    for split in all_splits:
        for chunk in generate_text_chunks(split.page_content):
            if stop_playback:
                print("Playback stopped by user.")
                return
            print(".", end="", flush=True)  # Visual feedback
            play_audio_chunk(chunk)
    print("\nPlayback finished.")

Starting the Process

To start the generation and playback process, simply call the generate_and_play() function.

generate_and_play()

Conclusion

With this approach, you can convert lengthy PDF documents into summarized audio files that are played back in real-time. This method is particularly useful for those who prefer auditory learning or need accessible formats for consuming information. The integration of text-to-speech with user-interruption handling makes this solution robust and user-friendly.

By following the steps outlined in this guide, you can develop a custom tool that turns text into audio, providing an alternative way to engage with content.

Final Code:

from gtts import gTTS
from io import BytesIO
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.chat_models import ChatOllama
from langchain_community.document_loaders import PyPDFLoader
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from pydub import AudioSegment
from pydub.playback import play
import signal
import sys
import threading

# Flag to control the loop
stop_playback = False

def signal_handler(sig, frame):
    global stop_playback
    print("\nGracefully stopping playback...")
    stop_playback = True

# Assign the signal handler to SIGINT (Ctrl+C)
signal.signal(signal.SIGINT, signal_handler)

# Function to listen for an Enter key press in a separate thread
def listen_for_stop():
    global stop_playback
    input("Press Enter to stop playback...\n")
    print("\nStopping playback...")
    stop_playback = True

# Start the listener thread
listener_thread = threading.Thread(target=listen_for_stop)
listener_thread.daemon = True
listener_thread.start()

# Load and split the PDF document
loader = PyPDFLoader("/home/roomal/Downloads/Stephen Mulhall/1 - Mulhall, Stephen - Heidegger and Being and Time - Scepticism, Cognition And Agency.pdf")
pages = loader.load_and_split()

# Split text into chunks
text_splitter = RecursiveCharacterTextSplitter(chunk_size=2048, chunk_overlap=100)
all_splits = text_splitter.split_documents(pages)

# Initialize the ChatOllama model
llm = ChatOllama(model="llama3:instruct", temperature=0.6)

# Create a prompt template
prompt = ChatPromptTemplate.from_template("Summarize the findings of: {page_content}")

# Define the chain
chain = prompt | llm | StrOutputParser()

# Function to generate text in chunks
def generate_text_chunks(page_content):
    try:
        text = chain.invoke({"page_content": page_content})
        sentences = text.split('. ')
        for sentence in sentences:
            yield sentence + '.'
    except Exception as e:
        print(f"Error generating text: {e}")

# Function to play audio from a text chunk
def play_audio_chunk(text_chunk):
    if not text_chunk.strip():
        return
    try:
        tts = gTTS(text=text_chunk, lang='en')
        with BytesIO() as audio_fp:
            tts.write_to_fp(audio_fp)
            audio_fp.seek(0)
            audio_segment = AudioSegment.from_file(audio_fp, format="mp3")
            play(audio_segment)
    except Exception as e:
        print(f"Error generating or playing audio: {e}")

# Function to generate and play text in real-time
def generate_and_play():
    global stop_playback
    for split in all_splits:
        for chunk in generate_text_chunks(split.page_content):
            if stop_playback:
                print("Playback stopped by user.")
                return
            print(".", end="", flush=True)  # Visual feedback
            play_audio_chunk(chunk)
    print("\nPlayback finished.")

# Start the generation and playback process
generate_and_play()

Until next time.

Best,

Roomal

Topic Modeling with Top2Vec: Dreyfus, AI, and Wordclouds

Roomal Seferaj — Mon, 08 Jul 2024 11:18:31 +0000

Extracting Insights from PDFs with Python: A Comprehensive Guide

This script demonstrates a powerful workflow for processing PDFs, extracting text, tokenizing sentences, and performing topic modeling with visualization, tailored for efficient and insightful analysis.

Libraries Overview

os: Provides functions to interact with the operating system.
matplotlib.pyplot: Used for creating static, animated, and interactive visualizations in Python.
nltk: Natural Language Toolkit, a suite of libraries and programs for natural language processing.
pandas: Data manipulation and analysis library.
pdftotext: Library for converting PDF documents to plain text.
re: Provides regular expression matching operations.
seaborn: Statistical data visualization library based on matplotlib.
nltk.tokenize.sent_tokenize: NLTK function to tokenize a string into sentences.
top2vec: Library for topic modeling and semantic search.
wordcloud: Library for creating word clouds from text data.

Initial Setup

Import Modules

import os
import matplotlib.pyplot as plt
import nltk
import pandas as pd
import pdftotext
import re
import seaborn as sns
from nltk.tokenize import sent_tokenize
from top2vec import Top2Vec
from wordcloud import WordCloud
from cleantext import clean

Next, ensure the punkt tokenizer is downloaded:

nltk.download('punkt')

Text Normalization

def normalize_text(text):
    """Normalize text by removing special characters and extra spaces,
    and applying various other cleaning options."""

    # Apply the clean function with specified parameters
    cleaned_text = clean(
        text,
        fix_unicode=True,  # fix various unicode errors
        to_ascii=True,  # transliterate to closest ASCII representation
        lower=True,  # lowercase text
        no_line_breaks=False,  # fully strip line breaks as opposed to only normalizing them
        no_urls=True,  # replace all URLs with a special token
        no_emails=True,  # replace all email addresses with a special token
        no_phone_numbers=True,  # replace all phone numbers with a special token
        no_numbers=True,  # replace all numbers with a special token
        no_digits=True,  # replace all digits with a special token
        no_currency_symbols=True,  # replace all currency symbols with a special token
        no_punct=False,  # remove punctuations
        lang="en",  # set to 'de' for German special handling
    )

    # Further clean the text by removing any remaining special characters except word characters, whitespace, and periods/commas
    cleaned_text = re.sub(r"[^\w\s.,]", "", cleaned_text)
    # Replace multiple whitespace characters with a single space and strip leading/trailing spaces
    cleaned_text = re.sub(r"\s+", " ", cleaned_text).strip()

    return cleaned_text

PDF Text Extraction

def extract_text_from_pdf(pdf_path):
    with open(pdf_path, "rb") as f:
        pdf = pdftotext.PDF(f)
    all_text = "\n\n".join(pdf)
    return normalize_text(all_text)

Sentence Tokenization

def split_into_sentences(text):
    return sent_tokenize(text)

Processing Multiple Files

def process_files(file_paths):
    authors, titles, all_sentences = [], [], []
    for file_path in file_paths:
        file_name = os.path.basename(file_path)
        parts = file_name.split(" - ", 2)
        if len(parts) != 3 or not file_name.endswith(".pdf"):
            print(f"Skipping file with incorrect format: {file_name}")
            continue

        year, author, title = parts
        author, title = author.strip(), title.replace(".pdf", "").strip()

        try:
            text = extract_text_from_pdf(file_path)
        except Exception as e:
            print(f"Error extracting text from {file_name}: {e}")
            continue

        sentences = split_into_sentences(text)
        authors.append(author)
        titles.append(title)
        all_sentences.extend(sentences)
        print(f"Number of sentences for {file_name}: {len(sentences)}")

    return authors, titles, all_sentences

Saving Data to CSV

def save_data_to_csv(authors, titles, file_paths, output_file):
    texts = []
    for fp in file_paths:
        try:
            text = extract_text_from_pdf(fp)
            sentences = split_into_sentences(text)
            texts.append(" ".join(sentences))
        except Exception as e:
            print(f"Error processing file {fp}: {e}")
            texts.append("")

    data = pd.DataFrame({
        "Author": authors,
        "Title": titles,
        "Text": texts
    })
    data.to_csv(output_file, index=False, quoting=1, encoding='utf-8')
    print(f"Data has been written to {output_file}")

Loading Stopwords

def load_stopwords(filepath):
    with open(filepath, "r") as f:
        stopwords = f.read().splitlines()
    additional_stopwords = ["able", "according", "act", "actually", "after", "again", "age", "agree", "al", "all", "already", "also", "am", "among", "an", "and", "another", "any", "appropriate", "are", "argue", "as", "at", "avoid", "based", "basic", "basis", "be", "been", "begin", "best", "book", "both", "build", "but", "by", "call", "can", "cant", "case", "cases", "claim", "claims", "class", "clear", "clearly", "cope", "could", "course", "data", "de", "deal", "dec", "did", "do", "doesnt", "done", "dont", "each", "early", "ed", "either", "end", "etc", "even", "ever", "every", "far", "feel", "few", "field", "find", "first", "follow", "follows", "for", "found", "free", "fri", "fully", "get", "had", "hand", "has", "have", "he", "help", "her", "here", "him", "his", "how", "however", "httpsabout", "ibid", "if", "im", "in", "is", "it", "its", "jstor", "june", "large", "lead", "least", "less", "like", "long", "look", "man", "many", "may", "me", "money", "more", "most", "move", "moves", "my", "neither", "net", "never", "new", "no", "nor", "not", "notes", "notion", "now", "of", "on", "once", "one", "ones", "only", "open", "or", "order", "orgterms", "other", "our", "out", "own", "paper", "past", "place", "plan", "play", "point", "pp", "precisely", "press", "put", "rather", "real", "require", "right", "risk", "role", "said", "same", "says", "search", "second", "see", "seem", "seems", "seen", "sees", "set", "shall", "she", "should", "show", "shows", "since", "so", "step", "strange", "style", "such", "suggests", "talk", "tell", "tells", "term", "terms", "than", "that", "the", "their", "them", "then", "there", "therefore", "these", "they", "this", "those", "three", "thus", "to", "todes", "together", "too", "tradition", "trans", "true", "try", "trying", "turn", "turns", "two", "up", "us", "use", "used", "uses", "using", "very", "view", "vol", "was", "way", "ways", "we", "web", "well", "were", "what", "when", "whether", "which", "who", "why", "with", "within", "works", "would", "years", "york", "you", "your", "suggests", "without"]
    stopwords.extend(additional_stopwords)
    return set(stopwords)

Filtering Stopwords from Topics

def filter_stopwords_from_topics(topic_words, stopwords):
    filtered_topics = []
    for words in topic_words:
        filtered_topics.append([word for word in words if word.lower() not in stopwords])
    return filtered_topics

Word Cloud Generation

def generate_wordcloud(topic_words, topic_num, palette='inferno'):
    colors = sns.color_palette(palette, n_colors=256).as_hex()
    def color_func(word, font_size, position, orientation, random_state=None, **kwargs):
        return colors[random_state.randint(0, len(colors) - 1)]

    wordcloud = WordCloud(width=800, height=400, background_color='black', color_func=color_func).generate(' '.join(topic_words))
    plt.figure(figsize=(10, 5))
    plt.imshow(wordcloud, interpolation='bilinear')
    plt.axis('off')
    plt.title(f'Topic {topic_num} Word Cloud')
    plt.show()

Main Execution

file_paths = [f"/home/roomal/Desktop/Dreyfus-Project/Dreyfus/{fname}" for fname in os.listdir("/home/roomal/Desktop/Dreyfus-Project/Dreyfus/") if fname.endswith(".pdf")]

authors, titles, all_sentences = process_files(file_paths)

output_file = "/home/roomal/Desktop/Dreyfus-Project/Dreyfus_Papers.csv"
save_data_to_csv(authors, titles, file_paths, output_file)

stopwords_filepath = "/home/roomal/Documents/Lists/stopwords.txt"
stopwords = load_stopwords(stopwords_filepath)

try:
    topic_model = Top2Vec(
        all_sentences,
        embedding_model="distiluse-base-multilingual-cased",
        speed="deep-learn",
        workers=6
    )
    print("Top2Vec model created successfully.")
except ValueError as e:
    print(f"Error initializing Top2Vec: {e}")
except Exception as e:
    print(f"Unexpected error: {e}")

num_topics = topic_model.get_num_topics()
topic_words, word_scores, topic_nums = topic_model.get_topics(num_topics)
filtered_topic_words = filter_stopwords_from_topics(topic_words, stopwords)

for i, words in enumerate(filtered_topic_words):
    print(f"Topic {i}: {', '.join(words)}")

keywords = ["heidegger"]
topic_words, word_scores, topic_scores, topic_nums = topic_model.search_topics(keywords=keywords, num_topics=num_topics)
filtered

_search_topic_words = filter_stopwords_from_topics(topic_words, stopwords)

for i, words in enumerate(filtered_search_topic_words):
    generate_wordcloud(words, topic_nums[i])

for i in range(reduced_num_topics):
    topic_words = topic_model.topic_words_reduced[i]
    filtered_words = [word for word in topic_words if word.lower() not in stopwords]
    print(f"Reduced Topic {i}: {', '.join(filtered_words)}")
    generate_wordcloud(filtered_words, i)

Reduce the number of topics

reduced_num_topics = 5
topic_mapping = topic_model.hierarchical_topic_reduction(num_topics=reduced_num_topics)

# Print reduced topics and generate word clouds
for i in range(reduced_num_topics):
    topic_words = topic_model.topic_words_reduced[i]
    filtered_words = [word for word in topic_words if word.lower() not in stopwords]
    print(f"Reduced Topic {i}: {', '.join(filtered_words)}")
    generate_wordcloud(filtered_words, i)

Transform Your Video Transcripts: From Raw to Readable Text

Roomal Seferaj — Fri, 07 Jun 2024 01:34:35 +0000

I cannot help but think of all the YouTube videos I have had to watch, simply because I didn't know how to save their transcripts. When I finally learned how, the resulting text was a mess. For instance, here’s a snippet from a required video for my biological psychology course:

Input: I've come here to California on the trail of one of the most infamous doctors of the with century Or Alter Freemen the last Alter Freemen began practicing as a doctor in the 1920s going on to work in one of the last institutions set up to house growing numbers of mentally ill people the Shell Shocked victims of the first world war and inmates with Dreadful psychiatric problems lived out their lives in what were known as snake pits psychiatric hospitals in the 1930s were terrible places to be as a patient and they were terrible places because they were they were places of hopelessness there were really no effective treatments for most mental disorders for the most part these hospitals warehouse patients for long periods of times decades even entire lives Freemen was horrified at the sheer waste of human potential now he started out with with good intentions and here was a terribly serious problem and it wasn't getting any better it was getting worse it was a public health problem Freemen was convinced that the root cause of many of the patients problems lay in the physical structure of their brains so he decided to change them a pulled by what he was seeing in these snake pits Freemen now spent increasing amounts of time in the laboratory coat brains dissecting brains examining brains looking for differences this is the brain received through the courtesy of Washington sanitary Freemen thought that surgery could help patients far more than the current treatments are cut off at the level of the middle of the pond spurred on by the growing understanding of what different regions of the brain do he finally decided that the problem lay in a set of connections between the thalamus and the frontal love the pointer demonstrates the thalamus and the anterior thalamic radiation going to all parts of the frontal L Freemen believed if he could never the connections between the thalamus and the frontal love and this would damper down all those awful emotions and it would if you like cure the patients he saw this as surgery of the Soul a way of bringing the Damned back to life but there was a problem Freemen was not himself a surgeon so he got together with a man who was James wants and together they started performing the operation action they called labotomy
[ Music ]

Not bad...
However, the lack of punctuation makes the experience less-than-desirable. So, to remedy this situation, I weaved together the original script that relied on the YouTubeTranscriptApi, TextBlob, and by importing the functionalities of the punctuators module.

from punctuators.models import PunctCapSegModelONNX
from textblob import TextBlob
from tqdm import tqdm
from typing import List
from youtube_transcript_api import YouTubeTranscriptApi
import nltk
import spacy

# Initialize models and download necessary resources
m = PunctCapSegModelONNX.from_pretrained("1-800-BAD-CODE/xlm-roberta_punctuation_fullstop_truecase")
nltk.download('punkt')
nlp = spacy.load("en_core_web_sm")

def nest_sentences(document: str, max_length: int = 1024) -> List[str]:
    """
    Nest sentences into groups ensuring each group does not exceed max_length.
    """
    nested, sent, length = [], [], 0
    for sentence in nltk.sent_tokenize(document):
        length += len(sentence)
        if length < max_length:
            sent.append(sentence)
        else:
            nested.append(" ".join(sent))
            sent = [sentence]
            length = len(sentence)
    if sent:
        nested.append(" ".join(sent))
    return nested

def process_transcript(video_id: str) -> str:
    """
    Retrieve and concatenate transcripts from YouTube video.
    """
    transcript_list = YouTubeTranscriptApi.list_transcripts(video_id)
    all_transcript_text = []

    for transcript in transcript_list:
        transcript_data = transcript.fetch()
        transcript_text = " ".join(entry["text"] for entry in transcript_data)
        all_transcript_text.append(transcript_text)

    return " ".join(all_transcript_text)

def filter_tokens(text: str) -> str:
    """
    Remove spaces from tokens in the text using spaCy.
    """
    doc = nlp(text)
    return " ".join(token.text for token in doc if not token.is_space)

def correct_text(text: str) -> str:
    """
    Correct the text using TextBlob.
    """
    blob = TextBlob(text)
    return str(blob.correct())

def punctuate_text(texts: List[str]) -> List[str]:
    """
    Punctuate and segment the texts using the pre-trained model.
    """
    return m.infer(texts=texts, apply_sbd=True)

def main(video_id: str):
    """
    Main processing function.
    """
    transcript_text = process_transcript(video_id)
    filtered_text = filter_tokens(transcript_text)
    corrected_text_str = correct_text(filtered_text)
    nested_sentences = nest_sentences(corrected_text_str)

    results = punctuate_text(nested_sentences)

    for input_text, output_texts in tqdm(zip(nested_sentences, results), desc="Processing", total=len(nested_sentences)):
        print(f"Input: {input_text}")
        print("Outputs:")
        for text in output_texts:
            print(f"\t{text}")
        print()

if __name__ == "__main__":
    video_id = "CUgtGjA6VvA"
    main(video_id)

Script Summary and Utility

This script is designed to process transcripts from YouTube videos and enhance their readability by applying punctuation, correcting errors, and ensuring proper segmentation. Here's a breakdown of its components and why it's useful:

Key Components:

Import Libraries:
- PunctCapSegModelONNX from punctuators.models: Adds punctuation and capitalization to the text.
- TextBlob: Corrects grammatical and spelling errors in the text.
- YouTubeTranscriptApi: Fetches transcripts from YouTube videos.
- nltk and spacy: Tokenizes and processes text to ensure proper segmentation.
Initialization:
- Load the pre-trained punctuation and capitalization model.
- Download necessary NLTK resources and load spaCy's English language model.
Functions:
- nest_sentences(document: str, max_length: int = 1024) -> List[str]: Groups sentences into segments not exceeding a specified length to maintain context and readability.
- process_transcript(video_id: str) -> str: Retrieves and concatenates transcripts from a given YouTube video ID.
- filter_tokens(text: str) -> str: Removes spaces and ensures proper tokenization using spaCy.
- correct_text(text: str) -> str: Uses TextBlob to correct grammatical and spelling errors in the text.
- punctuate_text(texts: List[str]) -> List[str]: Applies punctuation and segmentation to the text using the pre-trained model.
Main Function (main(video_id: str)):
- Retrieves the YouTube video transcript.
- Processes the transcript by filtering tokens, correcting text, nesting sentences, and applying punctuation.
- Prints the input and processed text for each nested segment.

Why This Script is Useful:

Improves Readability:
- Adds punctuation and capitalization, transforming raw transcripts into more readable text.
Corrects Errors:
- Uses TextBlob to automatically correct grammatical and spelling errors, enhancing the accuracy of the text.
Ensures Proper Segmentation:
- Splits text into manageable segments to maintain context and readability, especially useful for long transcripts.
Automates Transcript Processing:
- Simplifies the process of retrieving and enhancing YouTube video transcripts, saving time and effort for users.
Educational Tool:
- Can be included in a student package toolset to aid in processing and analyzing video transcripts for study purposes, making lecture notes or online video content more accessible and easier to study from.

Below is the output of the original CC from the video after it was processed:

Outputs:
I've come here to California on the trail of one of the most infamous doctors of the with century.
Or Alter Freemen, the last Alter Freemen began practicing as a doctor in the 1920s, going on to work in one of the last institutions set up to house growing numbers of mentally ill people.
The Shell Shocked victims of the First World War and inmates with Dreadful psychiatric problems lived out their lives in what were known as snake pits.
Psychiatric hospitals in the 1930s were terrible places to be as a patient, and they were terrible places because they were, they were places of hopelessness.
There were really no effective treatments for most mental disorders.
For the most part, these hospitals warehouse patients for long periods of times, decades, even entire lives.
Freemen was horrified at the sheer waste of human potential.
Now he started out with with good intentions.
And here was a terribly serious problem, and it wasn't getting any better, it was getting worse.
It was a public health problem.
Freemen was convinced that the root cause of many of the patients problems lay in the physical structure of their brains, so he decided to change them, a pulled by what he was seeing in these snake pits.
Freemen now spent increasing amounts of time in the laboratory, coat brains, dissecting brains, examining brains, looking for differences.
This is the brain received through the courtesy of Washington Sanitary.
Freemen thought that surgery could help patients far more than the current treatments are cut off at the level of the middle of the pond.
Spurred on by the growing understanding of what different regions of the brain do, he finally decided that the problem lay in a set of connections.
Between the thalamus and the frontal love The pointer demonstrates the thalamus and the anterior thalamic radiation going to all parts of the frontal.
Freemen believed if he could, never the connections between the thalamus and the frontal love.
And this would damper down all those awful emotions, and it would, if you like, cure the patients.
He saw this as Surgery of the Soul, a way of bringing the Damned back to life.
But there was a problem.
Freemen was not himself a surgeon, so he got together with a man who was James Wants, and together, they started performing the operation action they called lobotomy
[ Music ].

Overall, the quality is significantly better! All things considered, this script is a valuable tool for anyone looking to enhance the quality and readability of YouTube video transcripts, making it especially useful for students, researchers, and content creators. The script can also be modified with a transformer and used for translation processes, too!

Till next time,

Roomal

Ace Your Exams: Automated Question Generation for the Diligent Student

Roomal Seferaj — Wed, 05 Jun 2024 02:59:59 +0000

This one goes out to all my students who are always anxious about the upcoming exam.

I see you guys, asking questions such as "What will professor put on the exam?!" It's simple, who cares?

With this script, Ollama will generate line per line multiple choice questions for you to prepare on!

Now, a few words of caution to those that this is an easy way to get an A--it is not.

This is for those who have done the heavy lifting and want to simply take that extra step to be MORE prepared.

Well, this first script is for you guys.

This will generate a multiple choice question with 4 options per document. I would recommend breaking your book down into manageable sections by chapter, that way you don't have to deal with the overhead of parsing a whole book AND generating questions on it! For your consideration:

########################
# LOAD LIBRARIES
########################
import spacy
import warnings
import time
import logging

from langchain.llms import Ollama
from langchain.prompts import PromptTemplate
from pdfminer.high_level import extract_text
from tqdm import tqdm

###################################################################################
# SUPPRESS WARNINGS THAT CAN CLUTTER THE OUTPUT, SUCH AS DEPRECATION WARNINGS, ETC.
###################################################################################
warnings.filterwarnings("ignore")

#####################################
# LOAD THE SPACY LANGUAGE MODEL.
#####################################
nlp = spacy.load("en_core_web_sm")

def nest_sentences(document, max_length=4096):
    """
    Break down a document into manageable chunks of sentences where each chunk is under a specified length.

    Parameters:
    - document (str): The input text document to be processed.
    - max_length (int): The maximum character length for each chunk.

    Returns:
    - list: A list where each element is a group of sentences that together are less than max_length characters.
    """
    nested = []  # List to hold all chunks of sentences
    sent = []    # Temporary list to hold sentences for a current chunk
    length = 0   # Counter to keep track of the character length of the current chunk
    doc = nlp(document)  # Process the document using Spacy to tokenize into sentences

    for sentence in doc.sents:
        length += len(sentence.text)
        if length < max_length:
            sent.append(sentence.text)
        else:
            nested.append(' '.join(sent))  # Join sentences in the chunk and add to the nested list
            sent = [sentence.text]  # Start a new chunk with the current sentence
            length = len(sentence.text)  # Reset the length counter to the length of the current sentence

    if sent:  # Don't forget to add the last chunk if it's not empty
        nested.append(' '.join(sent))

    return nested

def generate_summary(text, llm, max_length=4096):
    """
    Generate a summary for provided text using the specified large language model (LLM).

    Parameters:
    - text (str): Text to summarize.
    - llm (LLMChain): The large language model to use for generating summaries.
    - max_length (int): The maximum character length for each summary chunk.

    Returns:
    - str: A single string that is the concatenated summary of all processed chunks.
    """
    sentences = nest_sentences(text, max_length)
    summaries = []  # List to hold summaries of each chunk
    seen_questions = set()  # Set to track unique questions

    prompt_template = PromptTemplate(
        input_variables=["text"],
        template="Generate diverse multiple-choice questions/answer are one sentence long based on the context here: {text}. "
                 "Ensure each question is unique and not repetitive. "
                 "Format:\nQuestion: Question?\n- A) Option A.\n- B) Option B.\n- C) Option C.\n- D) Option D.\nAnswer: Answer\n***\n"
    )

    for chunk in tqdm(sentences, desc="Generating summaries"):
        # Use the LLM to generate the summary based on the prompt.
        prompt = prompt_template.format(text=chunk)
        result = llm.invoke(prompt)
        result_lines = result.strip().split("\n")

        for line in result_lines:
            if line.startswith("Question:"):
                question = line.strip()
                if question not in seen_questions:
                    summaries.append(question)
                    seen_questions.add(question)
                    answer = next((l.strip() for l in result_lines if l.startswith("Answer:")), "")
                    summaries.append(answer)

        # Optionally print each generated summary.
        print(result.strip())

    # Join all summaries into a single string with spaces in between.
    return "\n".join(summaries)

def main_loop(delay):
    """
    Run the main loop, which generates summaries periodically, for 30 minutes.

    Parameters:
    - delay (int): The delay in seconds between each iteration of the loop.
    """
    end_time = time.time() + 30 * 60  # 30 minutes from now
    while time.time() < end_time:
        try:
            # Extract text from a PDF file.
            text = extract_text("/home/roomal/Desktop/PSY-3180/pdfs/Book 1/13 - Reproductive Behavior.pdf")

            # Generate and print the summary for the extracted text.
            summary = generate_summary(text, llm)
            print(summary)
        except Exception as e:
            logging.error(f"An error occurred: {e}")

        # Pause for the specified delay before the next iteration.
        time.sleep(delay)

#####################################
# CONFIGURATION FOR THE LANGUAGE MODEL.
#####################################
llm = Ollama(model="llama3:latest", temperature=0.9)

#######################################
# RUN THE MAIN LOOP FOR 30 MINUTES
#######################################
if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    logging.info("Starting the main loop for 30 minutes... Or whatever.")
    delay = int(input("Enter the delay time in seconds between each iteration: "))
    main_loop(delay)
    logging.info("Main loop completed.")

This code automates the process of scraping Wikipedia for a specified topic, processing the text, identifying topics using BERTopic, and generating questions based on these topics. It begins by installing necessary packages and setting up logging. The scrape_wikipedia function collects content from Wikipedia pages related to a given topic. The text is then cleaned, tokenized, and stopwords are removed. Using UMAP and CountVectorizer, the text is vectorized, and BERTopic is employed to identify key topics, with further fine-tuning using the Ollama model. Finally, questions are generated for each topic using a language model and saved for future use. This workflow is ideal for creating educational content, facilitating research, and enhancing study materials.

# pip install wikipedia-api bertopic umap-learn pandas nltk pdfminer.six tqdm rich langchain

import wikipediaapi
import pandas as pd
import concurrent.futures
from tqdm import tqdm
import json
import nltk
import re
import time
import umap
import logging
from bertopic import BERTopic
from bertopic.representation import KeyBERTInspired
from langchain import PromptTemplate
from langchain.llms import Ollama
from nltk.corpus import stopwords
from nltk.tokenize import sent_tokenize, word_tokenize
from rich.progress import Progress
from sklearn.feature_extraction.text import CountVectorizer

nltk.download('punkt')
nltk.download('stopwords')

logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def scrape_wikipedia(name_topic, verbose=True, max_workers=5):
    def link_to_wikipedia(link):
        try:
            page = api_wikipedia.page(link)
            if page.exists():
                return {
                    "page": link,
                    "text": page.text,
                    "link": page.fullurl,
                    "categories": list(page.categories.keys()),
                }
        except Exception as e:
            print(f"Error processing {link}: {e}")
            return None

    api_wikipedia = wikipediaapi.Wikipedia(
        language="en",
        user_agent="YourUserAgentHere",
        extract_format=wikipediaapi.ExtractFormat.WIKI,
    )

    name_of_page = api_wikipedia.page(name_topic)
    if not name_of_page.exists():
        print(f"Page {name_topic} is not present")
        return

    links_to_page = list(name_of_page.links.keys())
    procceed = tqdm(desc="Scraped links", unit="", total=len(links_to_page)) if verbose else None

    origin = [{
        "page": name_topic,
        "text": name_of_page.text,
        "link": name_of_page.fullurl,
        "categories": list(name_of_page.categories.keys()),
    }]

    with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
        links_future = {executor.submit(link_to_wikipedia, link): link for link in links_to_page}
        for future in concurrent.futures.as_completed(links_future):
            info = future.result()
            if info:
                origin.append(info)
            if verbose:
                procceed.update(1)
    if verbose:
        procceed.close()

    # Define namespaces to exclude
    namespaces = (
        "Wikipedia", "Special", "Talk", "LyricWiki", "File", "MediaWiki",
        "Template", "Help", "User", "Category talk", "Portal talk"
    )

    # Create DataFrame and filter based on text length and namespaces
    origin_df = pd.DataFrame(origin)
    origin_df = origin_df[
        (origin_df["text"].str.len() > 20) & 
        (~origin_df["page"].str.startswith(namespaces, na=True))
    ]

    # Process categories to remove 'Category:' prefix
    origin_df["categories"] = origin_df["categories"].apply(lambda cats: [cat[9:] for cat in cats])

    origin_df["topic"] = name_topic
    print("Scraped pages:", len(origin_df))

    return origin_df

def clean_text(text):
    text = text.lower()
    text = re.sub(r'\s+', ' ', text)  # Remove extra whitespace
    text = re.sub(r'[^a-z0-9\s.]', '', text)  # Remove non-alphanumeric characters except periods
    return text

def tokenize_text(text):
    tokens = word_tokenize(text)
    return tokens

def remove_stopwords(tokens):
    filtered_tokens = [token for token in tokens if token not in stopwords.words('english')]
    return filtered_tokens

def preprocess_text(text):
    cleaned_text = clean_text(text)
    tokens = tokenize_text(cleaned_text)
    filtered_tokens = remove_stopwords(tokens)
    return ' '.join(filtered_tokens)

def generate_questions(context, llm, prompt_template):
    formatted_prompt = prompt_template.format(context=context)
    response = llm.invoke(formatted_prompt)
    return response.strip()

# Start the process
logging.info("Starting the process...")

# 1. Scrape Wikipedia for the specified topic
topic = input("Enter the Wikipedia topic to scrape: ")
data = scrape_wikipedia(topic)
if data is None or data.empty:
    logging.error(f"No data found for topic: {topic}")
    exit()

# Save the scraped data to a CSV file
data.to_csv("/home/roomal/Desktop/scraped_data.csv", index=False)
logging.info("Scraped data saved to CSV.")

# Convert the scraped text data into a single text format
text_data = " ".join(data["text"].tolist())

# 2. Preprocess the text
processed_text = preprocess_text(text_data)
logging.info("Text preprocessing complete.")

# 3. Tokenize the text into sentences
sentences = sent_tokenize(processed_text)
logging.info(f"Number of sentences extracted: {len(sentences)}")

# 4. Configure UMAP and vectorizer parameters
umap_model = umap.UMAP(n_neighbors=10, n_components=5, min_dist=0.1, metric='cosine')
vectorizer_model = CountVectorizer(min_df=1, max_df=0.95)

# 5. Fit BERTopic model
logging.info("Fitting BERTopic model...")
topic_model = BERTopic(
    umap_model=umap_model,
    vectorizer_model=vectorizer_model,
    representation_model=KeyBERTInspired()
)
with Progress() as progress:
    task = progress.add_task("Fitting BERTopic model...", total=len(sentences))
    topics, probs = topic_model.fit_transform(sentences)
    progress.update(task, advance=len(sentences))

logging.info("BERTopic model fitted successfully.")
topic_info = topic_model.get_topic_info()

# Save the topic information to a CSV file
topic_info.to_csv("/home/roomal/Desktop/topic_info.csv", index=False)
logging.info("Topic information saved to CSV.")

# 6. Fine-tune topic representations with Ollama
logging.info("Fine-tuning topic representations with Ollama...")
topic_representation_model = Ollama(model="mistral:latest", temperature=0.8)
topic_model = BERTopic(
    representation_model=topic_representation_model,
    umap_model=umap_model,
    vectorizer_model=vectorizer_model
)
with Progress() as progress:
    task = progress.add_task("Fitting BERTopic with Ollama...", total=len(sentences))
    topic_model.fit(sentences)
    progress.update(task, advance=len(sentences))

logging.info("Topic representations fine-tuned successfully.")

# 7. Get representative documents for each topic
docs_by_topic = topic_model.get_representative_docs()

# 8. Generate questions using LangChain
llm_for_questions = Ollama(model="llama3:latest", temperature=0.5)

# Define a more varied prompt template for generating questions
prompt_template = PromptTemplate(
    input_variables=['context'],
    template='''Generate several insightful and varied questions based on the context below:

Context: {context}

Questions:\n
1. '''
)

# Generate questions for each topic
questions = []
with Progress() as progress:
    task = progress.add_task("Generating questions...", total=len(docs_by_topic))
    for topic, docs in docs_by_topic.items():
        context = " ".join(docs)
        question = generate_questions(context, llm_for_questions, prompt_template)
        questions.append(question)
        progress.update(task, advance=1)

logging.info("Questions generated successfully.")

# Print the generated questions
for i, question in enumerate(questions):
    print(f"Topic {i+1}: {question}")

# Save the questions to a text file
with open("/home/roomal/Desktop/generated_questions.txt", "w") as f:
    for i, question in enumerate(questions):
        f.write(f"Topic {i+1}:\n{question}\n\n")

logging.info("Generated questions saved to text file.")

This next one is a variation of the script above, except it will generate comprehensive questions, essay questions.

This code automates the extraction and processing of text from a specified PDF, identifies topics within the text using BERTopic, and generates insightful questions based on these topics using a language model. It begins by loading necessary libraries and configuring logging for progress tracking. The text is extracted from the PDF and cleaned, tokenized, and filtered to remove stopwords. The processed text is then analyzed using UMAP for dimensionality reduction and BERTopic for topic modeling. Fine-tuning of topic representations is performed using the Ollama model. Representative documents for each topic are identified, and questions are generated using LangChain with a specified prompt template. The process is repeated in a loop for a defined duration, continuously generating and printing questions, which can be useful for educational and research purposes.

####################
# LOAD LIBRARIES
####################
import json
import nltk
import re
import time
import umap
import logging

from bertopic import BERTopic
from bertopic.representation import KeyBERTInspired
from langchain import PromptTemplate
from langchain.llms import Ollama
from nltk.corpus import stopwords
from nltk.tokenize import sent_tokenize, word_tokenize
from pdfminer.high_level import extract_text
from rich.progress import Progress
from sklearn.feature_extraction.text import CountVectorizer

####################
# CONFIGURE LOGGING
####################
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

################################################
# ENSURE NECESSARY NLTK RESOURCES ARE DOWNLOADED
################################################
nltk.download('punkt')
nltk.download('stopwords')

######################
# PATH TO THE PDF FILE
######################
file_path = "C:\\Users\\sefer\\OneDrive\\Desktop\\PSY-3180\\pdfs\\Book 1\\13 - Reproductive Behavior.pdf"

############################
# EXTRACT TEXT FROM THE PDF
############################
logging.info("Extracting text from PDF...")
text = extract_text(file_path)

def clean_text(text):
    """Clean the input text using various preprocessing steps."""
    text = text.lower()
    text = re.sub(r'\s+', ' ', text)  # Remove extra whitespace
    text = re.sub(r'[^a-z0-9\s.]', '', text)  # Remove non-alphanumeric characters except periods
    return text

def tokenize_text(text):
    """Tokenize text into individual words."""
    tokens = word_tokenize(text)
    return tokens

def remove_stopwords(tokens):
    """Remove stopwords from the list of tokenized words."""
    filtered_tokens = [token for token in tokens if token not in stopwords.words('english')]
    return filtered_tokens

def preprocess_text(text):
    """Full preprocessing pipeline integrating all the steps."""
    cleaned_text = clean_text(text)
    tokens = tokenize_text(cleaned_text)
    filtered_tokens = remove_stopwords(tokens)
    return ' '.join(filtered_tokens)

def generate_questions(context, llm, prompt_template):
    formatted_prompt = prompt_template.format(context=context)
    response = llm.invoke(formatted_prompt)
    return response.strip()

def main_loop(duration_minutes):
    end_time = time.time() + duration_minutes * 60

    while time.time() < end_time:
        logging.info("Starting new iteration of text processing and question generation...")
        # Preprocess the text
        processed_text = preprocess_text(text)
        logging.info("Text preprocessing complete.")

        # Tokenize the text into sentences
        sentences = sent_tokenize(processed_text)
        logging.info(f"Number of sentences extracted: {len(sentences)}")

        # Configure UMAP parameters
        umap_model = umap.UMAP(n_neighbors=10, n_components=5, min_dist=0.1, metric='cosine')
        vectorizer_model = CountVectorizer(min_df=1, max_df=0.95)

        # BERTopic model
        logging.info("Fitting BERTopic model...")
        topic_model = BERTopic(
            umap_model=umap_model,
            vectorizer_model=vectorizer_model,
            representation_model=KeyBERTInspired()
        )
        with Progress() as progress:
            task = progress.add_task("Fitting BERTopic model...", total=len(sentences))
            topics, probs = topic_model.fit_transform(sentences)
            progress.update(task, advance=len(sentences))

        logging.info("BERTopic model fitted successfully.")
        topic_model.get_topic_info()

        # Fine-tune topic representations with Ollama
        logging.info("Fine-tuning topic representations with Ollama...")
        topic_representation_model = Ollama(model="llama3:latest", temperature=0.8)
        topic_model = BERTopic(
            representation_model=topic_representation_model,
            umap_model=umap_model,
            vectorizer_model=vectorizer_model
        )
        with Progress() as progress:
            task = progress.add_task("Fitting BERTopic with Ollama...", total=len(sentences))
            topic_model.fit(sentences)
            progress.update(task, advance=len(sentences))

        logging.info("Topic representations fine-tuned successfully.")
        # Get representative documents for each topic
        docs_by_topic = topic_model.get_representative_docs()

        # Generate questions using LangChain
        llm_for_questions = Ollama(model="llama3:latest", temperature=0.5)

        # Define a more varied prompt template for generating questions
        prompt_template = PromptTemplate(
            input_variables=['context'],
            template='''Generate several insightful and varied questions based on the context below:

Context: {context}

Questions:\n
1. '''
        )

        # Generate questions for each topic
        questions = []
        with Progress() as progress:
            task = progress.add_task("Generating questions...", total=len(docs_by_topic))
            for topic, docs in docs_by_topic.items():
                context = " ".join(docs)
                question = generate_questions(context, llm_for_questions, prompt_template)
                questions.append(question)
                progress.update(task, advance=1)

        logging.info("Questions generated successfully.")
        # Print the generated questions
        for i, question in enumerate(questions):
            print(f"Topic {i+1}: {question}")

        # Pause for a moment before the next iteration
        time.sleep(1)

#######################################
# RUN THE MAIN LOOP FOR 30 MINUTES
#######################################
if __name__ == '__main__':
    logging.info("Starting the main loop for 30 minutes...Or whatever.")
    main_loop(30)
    logging.info("Main loop completed.")

I have many more scripts to help my fellow students, just let me know what you wanna see, what ideas are rattling in that brilliant mind, and I will try to to make it happen.

For my next post, I'm going to show some hacking scripts I have developed to make things easier during an engagement.

Best,

Roomal

Frege, Machine Learning, and Logic: Fun with Python

Roomal Seferaj — Mon, 03 Jun 2024 04:05:55 +0000

As a philosophy graduate and as someone who has a newfound interest in machine learning and AI, Frege's work on logic and language has crossed my mind on more than one occasion. Indeed, Frege's contributions are seen as foundational in the field of philosophy. Translating some of it into modern English and demonstrating its principles using Python can make these concepts more accessible and practical for contemporary audiences. By constructing Frege’s logic from the ground up using basic axioms and principles, we can create a valuable resource for learning and applying logic. For the most part, we will be relying on my new translation of Frege's Über Sinn und Bedeutung. I must confess, however, that the technicalities of a philosophically rich article such as Über Sinn und Bedeutung required more than what the transformer used for the translation could offer (heat-tip to GPT4o for assisting).

Well, without further ado, this is how I spent my free time this Sunday. I apologize if my musings are fragmented at times, I am a terse writer.

Using Python to Explore Frege's Logic

The gist of my train of thought went something like this-before I realized that I was out of my depth--haha!

Translation and Explanation:
- Translate key articles of Frege's work into modern English.
- Provide clear explanations and context for each translated section.
Implementing Logic in Python:
- Translate Frege's logical principles into Python code.
- Use libraries such as sympy for symbolic mathematics and logic operations.
- Demonstrate the implementation of basic logical operations, axioms, and theorems.
Constructing Logical Structures:
- Build up from basic logical operations to more complex structures.
- Show how to derive conclusions from premises using Frege's methods.
- Create examples and exercises to illustrate each concept.
Practical Applications:
- Apply these logical structures to solve real-world problems.
- Show how logic can be used in programming, artificial intelligence, and data science.

Example Outline:

Part 1: Introduction to Frege's Logic

Overview of Frege's Contributions
Key Concepts: Sense, Reference, Function, Concept, Relation

Part 2: Translating and Explaining Frege's Texts

Detailed Translations of Key Sections
Explanations and Context

Part 3: Implementing Basic Logic in Python

Logical Operations: AND, OR, NOT
Truth Tables and Logical Equivalences

Part 4: Constructing Logical Proofs

Using Axioms and Inference Rules
Proof Strategies and Techniques

Part 5: Practical Applications of Logic

Programming with Logic
Logic in AI and Data Science

Example Code Snippet

Here’s a simple example to get started with implementing Huggigface transformers and OpenAI to help translate Frege's article into more contemporary prose:

from transformers import MarianMTModel, MarianTokenizer
from pdfminer.high_level import extract_text
import torch
import nltk

# Download NLTK data
nltk.download('punkt')

# Path to the PDF file
file_path = "C:/Users/sefer/OneDrive/Desktop/Frege - Sinn & Bedeutung.pdf"

# Extract text from the PDF
text = extract_text(file_path)

# Function to nest sentences within a limit
def nest_sentences(document):
    nested = []
    sent = []
    length = 0
    for sentence in nltk.sent_tokenize(document):
        length += len(sentence)
        if length < 1024:  # Ensure the length stays within the limit
            sent.append(sentence)
        else:
            nested.append(" ".join(sent))
            sent = [sentence]
            length = len(sentence)
    if sent:
        nested.append(" ".join(sent))
    return nested

# Load the model and tokenizer
model_name = 'Helsinki-NLP/opus-mt-de-en'  # Example: German to English translation model
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)

# Move model to GPU if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

# Function to translate text
def translate(text, model, tokenizer):
    # Tokenize the input text
    inputs = tokenizer.encode(text, return_tensors="pt", truncation=True).to(device)
    # Generate translation
    translated = model.generate(inputs, max_length=512)
    # Decode the translation
    translated_text = tokenizer.decode(translated[0], skip_special_tokens=True)
    return translated_text

# Nest the sentences
nested_sentences = nest_sentences(text)

# Translate nested sentences and combine them
translated_texts = [translate(sentence, model, tokenizer) for sentence in nested_sentences]
full_translated_text = " ".join(translated_texts)

# Print the translated text
print(full_translated_text)

And next here’s a simple example to get started with implementing basic logic in Python using sympy:

import sympy as sp

# Define logical variables
A, B = sp.symbols('A B')

# Logical operations
AND = sp.And(A, B)
OR = sp.Or(A, B)
NOT_A = sp.Not(A)

# Print truth tables
print("Truth Table for AND:")
print(sp.simplify_logic(AND, form='dnf'))

print("Truth Table for OR:")
print(sp.simplify_logic(OR, form='dnf'))

print("Truth Table for NOT A:")
print(sp.simplify_logic(NOT_A, form='dnf'))

# Define an implication
implication = sp.Implies(A, B)
print("Implication A => B:")
print(sp.simplify_logic(implication, form='dnf'))

Translating and Explaining a Key Concept

Original Text:

Die Bedeutung eines Eigennamens ist der Gegenstand selbst, den wir damit bezeichnen; die Vorstellung, welche wir dabei haben, ist ganz subjektiv; dazwischen liegt der Sinn, der zwar nicht mehr subjektiv wie die Vorstellung, aber doch auch nicht der Gegenstand selbst ist.

Translated Text:

The meaning of a proper name is the object itself that we designate with it; the idea we have in mind is entirely subjective; between them lies the sense, which is not as subjective as the idea, but is also not the object itself.

Explanation:
Frege distinguishes between three elements:

Meaning (Bedeutung): The actual object referred to by the name.
Idea (Vorstellung): The subjective mental image or concept of the object.
Sense (Sinn): The way the object is presented, which is shared among speakers of the language but is not the object itself.

Combining the translation of Frege's philosophical texts with practical implementations in Python can create an educational and insightful project. It bridges historical philosophical concepts with modern computational applications, making the abstract principles of logic tangible and useful in various fields.

Next Steps:

Identify Key Logical Concepts:
- Start with basic logical operations and concepts such as propositions, logical connectives (AND, OR, NOT), and truth values.
Implement Basic Logic in Python:
- Create functions to represent and manipulate these concepts.
- Use sympy for symbolic mathematics and logical operations.
Expand to More Complex Logical Structures:
- Implement more advanced logical operations and proof techniques.
- Demonstrate how to derive conclusions from premises.

Example Code to Implement Basic Logic:

Here’s an example of how to start implementing basic logical operations in Python:

import sympy as sp

# Define logical variables
A, B = sp.symbols('A B')

# Logical operations
AND = sp.And(A, B)
OR = sp.Or(A, B)
NOT_A = sp.Not(A)

# Print truth tables
print("Truth Table for AND:")
print(sp.simplify_logic(AND, form='dnf'))

print("Truth Table for OR:")
print(sp.simplify_logic(OR, form='dnf'))

print("Truth Table for NOT A:")
print(sp.simplify_logic(NOT_A, form='dnf'))

# Define an implication
implication = sp.Implies(A, B)
print("Implication A => B:")
print(sp.simplify_logic(implication, form='dnf'))

This code demonstrates the basic logical operations and prints out their truth tables. We can build upon this foundation by adding more complex logical structures and exploring how to construct proofs and logical arguments in the spirit of Frege's work.

Frege’s Views on Logic and Underlying Assumptions

Concept and Object:
- Concept (Begriff): A function that returns a truth value when applied to an object. In modern terms, this is akin to a predicate.
- Object (Gegenstand): An entity that a concept can be applied to.
Function and Argument:
- Function (Funktion): Frege viewed functions as mappings from arguments to values. For Frege, functions in logic are similar to mathematical functions.
- Argument (Argument): The input to a function, which when applied to the function yields a value.
Quantifiers:
- Universal Quantifier (∀): Indicates that a property holds for all elements in a domain.
- Existential Quantifier (∃): Indicates that there is at least one element in the domain for which a property holds.
Logical Form and Syntax:
- Frege introduced a formal language with a strict syntax to represent logical relations clearly.
- He used a two-dimensional notation for functions and quantifiers, which is quite different from our linear text-based programming languages.

Mapping Frege’s Logic to Python Functions

1. Concepts as Predicates

In Python, we can represent concepts (predicates) as functions that return boolean values.

def is_even(n):
    return n % 2 == 0

2. Functions and Arguments

Functions in Python can directly represent Frege’s idea of mapping arguments to values.

def add(x, y):
    return x + y

3. Universal and Existential Quantifiers

Quantifiers can be represented using functions that iterate over a domain.

Universal Quantifier (∀):

def for_all(domain, predicate):
    return all(predicate(x) for x in domain)

# Example usage:
numbers = [2, 4, 6, 8]
print(for_all(numbers, is_even))  # True, because all numbers are even

Existential Quantifier (∃):

def there_exists(domain, predicate):
    return any(predicate(x) for x in domain)

# Example usage:
numbers = [1, 3, 4, 7]
print(there_exists(numbers, is_even))  # True, because there exists at least one even number

Example: Translating a Logical Expression

Frege might express a logical statement like "For all x, if x is a human, then x is mortal" as:

for all x (Human(x) > Mortal(x))

In Python, assuming we have predicates is_human and is_mortal, we can translate this as:

def is_human(x):
    # Placeholder implementation
    return x in ["Socrates", "Plato", "Aristotle"]

def is_mortal(x):
    # Placeholder implementation
    return x in ["Socrates", "Plato", "Aristotle"]

def implies(p, q):
    return not p or q

def for_all_humans_implies_mortal(domain):
    return for_all(domain, lambda x: implies(is_human(x), is_mortal(x)))

# Example usage:
entities = ["Socrates", "Plato", "Aristotle", "Zeus"]
print(for_all_humans_implies_mortal(entities))  # True, assuming Zeus is not human and others are humans and mortal

Eventually after a few hours, OpenAI's finest and I had a project folder filled with the basic logical laws spelled out, including some more idiosyncratic contributions by the likes of Abalard, Ockham, Frege, and etc., etc. Long story short, things are harder up close:

This is what the philosophica_logic module looks like so far:

# philosophical_logic/__init__.py

from .aristotelian_logic import *
from .frege import *
from .logic import *
from .logical_laws import *
from .modal import *
from .propositional import *

# philosophical_logic/logic.py

def implies(p, q):
    return not p or q

def and_op(p, q):
    return p and q

def or_op(p, q):
    return p or q

def not_op(p):
    return not p

def for_all(domain, predicate):
    return all(predicate(x) for x in domain)

def there_exists(domain, predicate):
    return any(predicate(x) for x in domain)

def possibly(p):
    return p

def necessarily(p):
    return p

# philosophical_logic/logical_laws.py

def involution(p):
    return p

def de_morgan_conjunction(p, q):
    return not (p and q) == (not p or not q)

def de_morgan_disjunction(p, q):
    return not (p or q) == (not p and not q)

def commutativity_and(p, q):
    return p and q == q and p

def commutativity_or(p, q):
    return p or q == q or p

def associativity_and(p, q, r):
    return (p and (q and r)) == ((p and q) and r)

def associativity_or(p, q, r):
    return (p or (q or r)) == ((p or q) or r)

def distributivity_and_or(p, q, r):
    return (p and (q or r)) == ((p and q) or (p and r))

def distributivity_or_and(p, q, r):
    return (p or (q and r)) == ((p or q) and (p or r))

def law_of_excluded_middle(p):
    return p or not p

def law_of_non_contradiction(p):
    return not (p and not p)

def idempotence_and(p):
    return p and p == p

def idempotence_or(p):
    return p or p == p

def identity_laws(p):
    return p == p

def transposition_laws(p, q):
    return (p implies q) == (not q implies not p)

def definition_of_conditional(p, q):
    return (p implies q) == (not p or q)

def definition_of_biconditional(p, q):
    return (p == q) == ((p implies q) and (q implies p))

def modus_ponens(p, q):
    return (p implies q) and p

def modus_tollens(p, q):
    return (p implies q) and not q

def syllogism(p, q, r):
    return (p implies q) and (q implies r) and (p implies r)

def disjunctive_syllogism(p, q):
    return (p or q) and not p

def transitivity(p, q, r):
    return (p implies q) and (q implies r) and (p implies r)

def simplification(p, q):
    return p and q

def addition(p, q):
    return p or q

def constructive_dilemma(p, q, r):
    return (p or q) and (p implies r) and (q implies r)

def second_law_constructive_dilemma(p, q, r, s):
    return (p implies q) and (r implies s) and (p or r) and (q or s)

def destructive_dilemma(p, q, r, s):
    return (p implies q) and (r implies s) and (not q or not s) and (not p or not r)

def law_of_contrapositive(p, q):
    return (p implies q) and (not p implies not q)

def exportation(p, q, r):
    return (p and q implies r) == (p implies (q and r))

def negation_of_conditional(p, q):
    return not (p implies q) == (p and not q)

def absorption_laws_and(p, q):
    return p and (p or q) == p

def absorption_laws_or(p, q):
    return p or (p and q) == p

def permutation_law(p, q, r):
    return (p implies (q implies r)) == (q implies (p implies r))

def expansion_laws_or(p, q):
    return (p implies q) == ((p or q) == q)

def expansion_laws_and(p, q):
    return (p implies q) == ((p and q) == p)

def known_true(p):
    return True or p == True

def known_false(p):
    return False or p == p

def true_and(p):
    return True and p == p

def false_and(p):
    return False and p == False

def contradiction(p):
    return p and not p == False

def tautology(p):
    return p or not p == True

# philosophical_logic/modal.py

def possibly(p):
    return p

def necessarily(p):
    return p

# Placeholder for more modal logic functions

# philosophical_logic/propositional.py

def implies(p, q):
    return not p or q

def and_op(p, q):
    return p and q

def or_op(p, q):
    return p or q

def not_op(p):
    return not p

def for_all(domain, predicate):
    return all(predicate(x) for x in domain)

def there_exists(domain, predicate):
    return any(predicate(x) for x in domain)

# Placeholder for more propositional logic functions

# stoic_logic.py

# Define logical connectives
def implies(p, q):
    return not p or q

def and_op(p, q):
    return p and q

def or_op(p, q):
    return p or q

def not_op(p):
    return not p

# Indemonstrable Arguments
def modus_ponens(p, q):
    """
    If p, then q. p. Therefore, q.
    """
    return implies(p, q) and p

def modus_tollens(p, q):
    """
    If p, then q. Not q. Therefore, not p.
    """
    return implies(p, q) and not_op(q)

def modus_ponendo_tollens(p, q):
    """
    Not both p and q. p. Therefore, not q.
    """
    return not and_op(p, q) and p

def strong_modus_tollendo_ponens(p, q):
    """
    Either p or q. Not p. Therefore, q.
    """
    return or_op(p, q) and not_op(p)

def strong_modus_ponendo_tollens(p, q):
    """
    Either p or q. p. Therefore, not q.
    """
    return or_op(p, q) and p

# Example usage
if __name__ == "__main__":
    p = True
    q = False

    print("Modus Ponens:", modus_ponens(p, q))
    print("Modus Tollens:", modus_tollens(p, q))
    print("Modus Ponendo Tollens:", modus_ponendo_tollens(p, q))
    print("Strong Modus Tollendo Ponens:", strong_modus_tollendo_ponens(p, q))
    print("Strong Modus Ponendo Tollens:", strong_modus_ponendo_tollens(p, q))

# philosophical_logic/frege.py

class Term:
    def __init__(self, sense, reference):
        self.sense = sense  # Sinn
        self.reference = reference  # Bedeutung

    def __str__(self):
        return f"Term(sense={self.sense}, reference={self.reference})"

# Example terms
morning_star = Term("The star seen in the morning", "Venus")
evening_star = Term("The star seen in the evening", "Venus")

def compare_terms(term1, term2):
    same_reference = term1.reference == term2.reference
    same_sense = term1.sense == term2.sense
    return same_reference, same_sense

# Logical functions applied to terms
def implies(term1, term2):
    return not term1.reference or term2.reference

def and_op(term1, term2):
    return term1.reference and term2.reference

def not_op(term):
    return not term.reference

# abelard_logic.py

# Define logical connectives in a truth-functional manner
def negation(p):
    """
    Abelard's definition of negation: not-p is false/true if and only if p is true/false.
    """
    return not p

def conjunction(p, q):
    """
    Conjunction: p and q.
    """
    return p and q

def disjunction(p, q):
    """
    Disjunction: p or q.
    """
    return p or q

def implication(p, q):
    """
    Implication: if p then q.
    """
    return not p or q

def biconditional(p, q):
    """
    Biconditional: p if and only if q.
    """
    return p == q

# Define entailment (inferentia)
def entailment(p, q):
    """
    Abelard's entailment: The conclusion (q) is required by the sense of the preceding statement (p).
    """
    return implication(p, q)

# Examples and testing
if __name__ == "__main__":
    p = True
    q = False

    print("Negation of p:", negation(p))
    print("Conjunction of p and q:", conjunction(p, q))
    print("Disjunction of p and q:", disjunction(p, q))
    print("Implication (p implies q):", implication(p, q))
    print("Biconditional (p if and only if q):", biconditional(p, q))
    print("Entailment (p entails q):", entailment(p, q))

# philosophical_logic/aristotelian_logic.py

def categorical_syllogism(major_premise, minor_premise, conclusion):
    return (major_premise == 'All M are P' and minor_premise == 'All S are M' and conclusion == 'All S are P')

def hypothetical_syllogism(major_premise, minor_premise, conclusion):
    return (major_premise == 'If P then Q' and minor_premise == 'If Q then R' and conclusion == 'If P then R')

def disjunctive_syllogism(major_premise, minor_premise, conclusion):
    return (major_premise == 'P or Q' and minor_premise == 'Not P' and conclusion == 'Therefore Q') or \
           (major_premise == 'P or Q' and minor_premise == 'Not Q' and conclusion == 'Therefore P')

def modus_ponens(major_premise, minor_premise, conclusion):
    return (major_premise == 'If P then Q' and minor_premise == 'P' and conclusion == 'Therefore Q')

def modus_tollens(major_premise, minor_premise, conclusion):
    return (major_premise == 'If P then Q' and minor_premise == 'Not Q' and conclusion == 'Therefore Not P')

def validate_syllogism(syllogism_type, major_premise, minor_premise, conclusion):
    if syllogism_type == 'categorical':
        return categorical_syllogism(major_premise, minor_premise, conclusion)
    elif syllogism_type == 'hypothetical':
        return hypothetical_syllogism(major_premise, minor_premise, conclusion)
    elif syllogism_type == 'disjunctive':
        return disjunctive_syllogism(major_premise, minor_premise, conclusion)
    elif syllogism_type == 'modus_ponens':
        return modus_ponens(major_premise, minor_premise, conclusion)
    elif syllogism_type == 'modus_tollens':
        return modus_tollens(major_premise, minor_premise, conclusion)
    else:
        raise ValueError("Invalid syllogism type")

Conclusion

Frege’s approach to logic involves precise definitions of concepts, functions, and quantifiers. Translating his ideas into Python involves creating functions that represent predicates, mappings, and quantifiers. By understanding these fundamental concepts, we can design Python functions that reflect Frege’s logical structure accurately. We can continue by developing more complex logical constructs and ensuring that our Python implementations align closely with Frege’s logical framework. This will allow us to leverage the power of Python to explore and demonstrate Fregean logic in a practical and computational context. Indeed, the first 1800 years of philosophy and its reliance on Aristotelian logic were farily easy to code for, thanks to the help of easily accessible data and my trusty AI sidekicks. The more recent forms of logic, however, are a bit more...technical.

Anyhow, I hope to share more of my musings with the DEV family.

Best,

Roomal