The latest articles on DEV Community by Priyansh Kumar (@pkumar18). https://dev.to/pkumar18 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4027956%2Fb370baac-c5f3-4464-9f03-487728aa6eca.jpg https://dev.to/pkumar18 en Priyansh Kumar Tue, 14 Jul 2026 02:15:50 +0000 https://dev.to/pkumar18/i-got-tired-of-bad-fanfiction-recommendations-so-im-building-my-own-taste-engine-1ag1 https://dev.to/pkumar18/i-got-tired-of-bad-fanfiction-recommendations-so-im-building-my-own-taste-engine-1ag1 <p>I read fanfiction. A lot of it, on AO3 and FanFiction.net. Finding something worth reading takes longer than it should.</p> <p>AO3 has over 10 million works. The discovery tool is tag search. You type a fandom, a ship, maybe a few tags, and scroll through whatever comes back. Kudos and bookmarks sort the list, but those measure popularity, not whether you'll enjoy it. A fic with 10k kudos might be beloved by thousands and still not your thing.</p> <p>I tried describing what I wanted to Claude once. It couldn't help. No domain knowledge, no idea what makes a fic actually good, no model of my taste. It guessed from a description. That's not useful.</p> <p>Every existing recommendation tool in this space matches surface features. Same fandom, same tags, same ship. None of them look at pacing, character voice, emotional tone, the things that make you stay up until 3am finishing something.</p> <p>I'm building Siagnos to fix that.</p> <h2> What Siagnos actually does </h2> <p>Siagnos is a personal taste engine. Track what I actually read, build a model of my taste from that behavior, score unseen fics against it.</p> <p>A preference model trained on my own reading data. The pipeline has six parts:</p> <ol> <li>Scrape fic metadata from AO3: title, fandom, tags, word count, kudos, completion status, summary</li> <li>Track real reading sessions: what I open, how far I get, whether I come back</li> <li>Generate embeddings from summaries using HuggingFace sentence transformers</li> <li>Run a local LLM (Mistral 7B via Ollama) to extract richer features from summaries (tone, pacing, character dynamics)</li> <li>Train a preference model on reading history using XGBoost</li> <li>Given a fic URL, return a taste score and a plain-language reason</li> </ol> <p>The output I want: "This matches your taste because it has the same slow-burn structure and morally grey protagonist you keep coming back to." Actual signal, not a popularity rank.</p> <h2> Where it stands right now </h2> <p>I'm three stages in.</p> <p><strong>Stage 0</strong>: project structure, Git, virtual environment, mental model of embeddings and cosine similarity.</p> <p><strong>Stage 1</strong>: sentence transformers installed, embeddings generated for a handful of fic summaries, cosine similarity working. Proof of concept done.</p> <p><strong>Stage 2</strong>: scraper complete. Built with cloudscraper and BeautifulSoup, retry logic, failed URL logging, adult content bypass, resume from last point. Scraped around 7,000 My Hero Academia fics to CSV. Also collected behavior data for about 100 fics I've personally read.</p> <p><strong>Stage 3</strong>: PostgreSQL schema designed and implemented. Tables for fics, reading sessions, six value tables, six join tables, ENUMs for status and rating types. Schema committed to the repo.</p> <p>Stage 4 is next: a reading tracker, either a browser extension or a bookmarklet, that logs real sessions to the database.</p> <h2> The tech stack </h2> <p>Python throughout. FastAPI for the backend when I get there. PostgreSQL replacing SQLite, which I'd outgrown. HuggingFace sentence transformers for embeddings. Ollama running Mistral 7B locally for feature extraction. XGBoost for the preference model. Docker for containerization. Azure for deployment.</p> <p>One constraint I set myself: no LangChain or similar for the core pipeline. Build it first, understand what's actually happening, add abstractions later if they help.</p> <h2> Why this is worth building </h2> <p>Nobody with real ML experience is building for fanfiction readers. The domain is large (AO3 hit 879 million weekly page views earlier this year) and the recommendation problem is unsolved. I know the domain as a reader, which most ML engineers building recommendation systems don't.</p> <p>Siagnos builds a preference model trained on individual reading behavior. I'm a BCA student, this is my second serious project, and I'm learning ML from first principles by building it. Future posts will cover the pipeline, the modeling approach, and what failed.</p> <p><em>Siagnos is on GitHub: <a href="https://github.com/P-Kumar-18/Siagnos" rel="noopener noreferrer">github.com/P-Kumar-18</a></em></p> python machinelearning nlp beginners