Why I stopped trading and started tendering with AI

#ai #automation #opensource

Why I Stopped Trading and Started Tendering with AI

For years, I was hooked. The flashing charts, the quick wins (and devastating losses), the constant hunt for the ‘edge’… I was a trader. Primarily focused on crypto, but dabbled in Forex and equities too. I built complex trading bots, backtested strategies until my eyes blurred, and poured hours into technical analysis. Yet, despite all the effort, consistent profitability remained elusive. It wasn’t a skill problem, I realised. It was a game problem. The game of trading is a zero-sum (or even negative-sum with fees) competition against other people, increasingly sophisticated algorithms, and ultimately, information asymmetry.

Then, I stumbled into a different world: government and private sector tendering. And, surprisingly, I found a way to apply my programming skills – specifically, AI – to create value, rather than just redistribute it. This isn't about high-frequency trading levels of complexity, but a different beast entirely, focused on information processing, document analysis, and strategic bidding.

Here’s why I hung up my trading hat and started tendering with AI, and how I'm building a system to automate the process.

The Allure of Tendering: A Different Kind of Opportunity

Tendering, for those unfamiliar, is the process of publicly announcing projects and inviting bids from suppliers. Think government contracts for roadworks, software development, marketing services, even supplying stationery. It’s a huge market – trillions of dollars globally.

The key difference from trading is that you aren't competing for a slice of existing value, you're competing to deliver new value. Success isn’t about predicting market movements, but about accurately assessing a project, understanding its requirements, and building a compelling, cost-effective bid.

However, the tendering process is painful. Manually sifting through hundreds of pages of RFPs (Requests for Proposals), specifications, and addendums is incredibly time-consuming. The information is often poorly structured, filled with legal jargon, and scattered across multiple documents. Identifying relevant opportunities, let alone crafting a winning response, requires a dedicated team… or a very good AI.

From Trading Bots to Tender Bots: The Tech Stack

My trading background gave me a solid foundation in Python, data processing, and API integration. I leveraged that, but shifted focus. Instead of time-series analysis, I needed Natural Language Processing (NLP) and document parsing.

Here's the core tech stack I'm using:

Python: The glue that holds everything together.
Beautiful Soup & PDFMiner: For scraping tender websites and extracting text from PDFs. While OCR (Optical Character Recognition) is sometimes needed (Tesseract via pytesseract), well-formatted PDFs are surprisingly common.
LangChain: This is the engine. LangChain allows me to build applications powered by Large Language Models (LLMs) like OpenAI's GPT-3.5 or, increasingly, open-source alternatives.
OpenAI API (or alternatives like Hugging Face models): For NLP tasks like summarisation, keyword extraction, and requirements analysis.
Pinecone (or other vector database): To store embeddings of tender documents for semantic search. This allows me to find opportunities even if they don't explicitly contain my target keywords.
Flask/FastAPI: To build a simple web interface for managing the system and reviewing results.

The Workflow: Automating the Tender Hunt & Analysis

The system I'm building can be broken down into a few key stages:

1. Data Acquisition:

This involves regularly scraping tender websites (e.g., government procurement portals, industry-specific websites). The structure varies wildly, so robust scraping logic is crucial.

import requests
from bs4 import BeautifulSoup

def scrape_tender_website(url):
  """Scrapes a tender website and returns a list of tender URLs."""
  try:
    response = requests.get(url)
    response.raise_for_status()  # Raise HTTPError for bad responses (4xx or 5xx)
    soup = BeautifulSoup(response.content, 'html.parser')
    #  -- Specific parsing logic for the website --
    tender_links = [a['href'] for a in soup.find_all('a', href=True) if "tender" in a['href'].lower()]
    return tender_links
  except requests.exceptions.RequestException as e:
    print(f"Error scraping {url}: {e}")
    return []

2. Document Processing & Embedding:

Once we have the URLs, we download the tender documents (often PDFs) and extract the text using PDFMiner. Then, using LangChain, we generate embeddings (vector representations) of the documents.

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.document_loaders import PyPDFLoader
from langchain.text_splitter import CharacterTextSplitter

# Load the document
loader = PyPDFLoader("path/to/tender_document.pdf")
documents = loader.load()

# Split the document into chunks
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

# Create embeddings
embeddings = OpenAIEmbeddings()
vectorstore = Pinecone.from_documents(texts, embeddings, index_name="tender-index")

3. Semantic Search & Filtering:

This is where the power of vector databases comes in. I can query the database with a description of my capabilities (e.g., "software development for healthcare") and find tenders that are semantically similar, even if they don't contain those exact keywords.

query = "Develop a mobile application for patient monitoring."
results = vectorstore.similarity_search(query, k=5)  # Find top 5 similar documents

for doc in results:
  print(f"Document Title: {doc.metadata['source']}")
  print(f"Relevance Score: {doc.metadata['score']}")
  print(f"Snippet: {doc.page_content[:200]}...\n")

4. Requirements Analysis & Risk Assessment:

Once a relevant tender is identified, the AI can analyze the RFP to identify key requirements, potential risks, and areas where our company can add value. This involves using LangChain to perform summarization, keyword extraction, and sentiment analysis. I’ve started experimenting with chain-of-thought prompting to get the LLM to reason through complex requirements.