Welcome to the POT 👻
⚠️ A small note before trying the live demo
I'm hosting the application on Render's free tier, so there's one little thing to keep in mind.
If the demo hasn't received any traffic for a while, the backend service may have gone to sleep. The first request can take a little longer while Render wakes it up.
So, if the application appears to be loading or the first API request takes time, please give it a moment before refreshing the page.
It's a free-tier cold start, not necessarily a broken demo. :)
Building People of the Past for Hacktoberfest Open Source AI Challenge — Week 4: Ghost in the Machine
Every time I thought I had figured out the AI part of this project, the problem moved one layer deeper.
First, I needed a model that could hold a conversation. Then I realised a conversation without memory was just another chatbot. Once I started building memory, I had to think about retrieval, context, identity, privacy, and what happens when the AI confidently says something that was never actually told to it.
And somewhere in between, a modal decided it wanted to live in the top-left corner of the screen. :)
This is the story of building People of the Past (POT): the decisions that shaped it, the things that broke, the experiments that didn't go as planned, and the architecture I ended up with.
01. What did I build?
People of the Past started with a simple idea:
What if we could have meaningful conversations with people from the past, while preserving the memories and stories that make those people special?
I eventually split that idea into two experiences.
Links
- Live Demo: https://youtu.be/zwuG8uBSCRQ
- Source Code: https://github.com/utkarsh-g19/PeopleOfdPast
- Hosted on Render: https://people-of-the-past-web.onrender.com/
- Hacktoberfest Challenge: Open Source AI Challenge — Week 4: Ghost in the Machine
👻 Ghost Gallery — conversations with people from history
Ghost Gallery lets users explore conversations with notable historical figures.
The challenge here isn't simply making an AI imitate a historical personality. A convincing response that invents historical facts is not a good response.
I wanted the experience to be informed by historical profiles and source details, rather than relying entirely on whatever the model happens to know.
The historical profiles are stored separately from private TYLO companions. That distinction matters: historical information and personal family memories have very different privacy and grounding requirements.
❤️ TYLO — Talk to Your Loved Ones
TYLO is the more personal side of the project.
It allows someone to create an AI memory companion using information they provide about a loved one who has passed away.
Instead of asking users to write one enormous prompt, I built a structured profile that captures different dimensions of a person's life:
| Section | Information collected |
|---|---|
| Their life | Life stages, relationship, family and friends, important dates |
| Their personality | Nature, nicknames, speaking style, favourite phrases |
| Your shared world | Special moments, stories, routines, interests and hobbies |
| What they believed | Values, likes and dislikes, lessons and advice |
A profile also includes a photograph and an introduction.
The goal is to give the system more than a name and a generic personality description. It needs the little details: the nickname, the familiar phrase, the Christmas tradition, the way someone approached a difficult day.
Those details are what make a memory personal.
One distinction I want to make very clear: TYLO does not bring someone back, and it does not recreate their actual consciousness. It generates conversations using information and memories supplied by the user. It is a space for remembrance, not a replacement for the person.
02. The first big realisation: a prompt is not a memory system
My earliest instinct was to think about the model first.
Which model should I use? How should I write the system prompt? How do I make the responses sound natural?
But I eventually ran into a much more important problem.
Imagine someone creates a companion and tells it:
- Their grandmother was always positive.
- She loved being around people.
- She had a particular nickname for them.
- Every Christmas, the family had a special tradition.
- She would often say, "Oh, come on, we will get it done."
Now imagine the user returns after several conversations and asks about that Christmas tradition.
A model that only sees the latest message has no reliable way of recalling it.
Sending every piece of profile information and the entire conversation on every request is not a sensible long-term solution either. Context needs to be selected.
That is where the TYLO memory architecture became the centre of the project.
I stopped thinking of TYLO as simply a chatbot with a personality prompt. I started treating it as a system with several sources of context:
- The companion's profile.
- Recent conversation history.
- Saved memories and the companion summary.
- Relevant older conversation exchanges retrieved through vector search.
The model generates the response. The memory system decides what information the model gets to work with.
That separation changed how I approached the backend.
03. How TYLO remembers: the actual memory architecture
This is the current architecture behind a TYLO message.
The database structure
I'm using MongoDB Atlas for both the application data and the vector retrieval layer.
The main collections are:
| Collection | Responsibility |
|---|---|
personas |
Published historical profiles and their source details |
companions |
TYLO profiles, saved memories and summaries |
tylomessages |
Persisted TYLO conversation messages |
tylomemories |
Conversation exchanges and their 768-dimensional embeddings |
The distinction between tylomessages and tylomemories is intentional.
The message collection stores the conversation history. The memory collection provides a retrieval-oriented representation of conversation exchanges, including their embeddings.
This allows the application to maintain recent conversational context while also looking for older, semantically relevant exchanges.
Embeddings and Atlas Vector Search
When a user sends a message, the API creates an embedding for the question.
The expected embedding size in this implementation is 768 dimensions.
That vector is used to query the MongoDB Atlas Vector Search index:
tylo_memory_vector_index
The current retrieval configuration is:
| Setting | Value |
|---|---|
| Vector index | tylo_memory_vector_index |
| Expected vector dimensions | 768 |
| Candidate documents | 50 |
| Relevant exchanges returned | Up to 4 |
| Ownership filters |
ownerSub and companionId
|
| Model compatibility filter | embeddingModel |
Why retrieve 50 candidates if we only need up to four?
The idea is to search a wider candidate set and then provide a smaller, relevant selection to the generation step. The model doesn't need fifty old exchanges in its prompt. It needs the few that are useful for the current message.
There is also an important reason for filtering on embeddingModel: vectors generated by different embedding models should not be casually treated as interchangeable. Retrieval needs to respect the embedding model used to create the stored vectors.
What actually enters the model's context?
This is the part I wanted to make explicit because it is easy to say "we use vector search" without explaining what happens after the search.
For a typical TYLO message, the context is assembled from:
| Context source | Maximum included |
|---|---|
| Companion profile | Relevant profile information |
| Recent conversation history | Latest 8 message records |
| Saved memories | Up to 3 |
| Companion summary | Current summary |
| Retrieved conversation exchanges | Up to 4 |
The result is a bounded context assembled from several sources, rather than blindly sending the entire conversation.
The memory retrieval step also has a fallback.
If vector retrieval fails, the API logs the failure and continues with the companion profile and recent conversation history.
That doesn't magically make missing memories available. It does, however, prevent a retrieval failure from automatically turning into a completely failed conversation.
The important part is that memory creation and memory retrieval are separate stages.
A conversation exchange can be persisted first and processed for embedding in the background. On a later message, that exchange can become a candidate for semantic retrieval.
This is the foundation of TYLO's longer-term memory behaviour.
04. Why Auth0? Because TYLO is not an ordinary chatbot.
This was one partner decision I didn't want to treat as just another item in the technology stack.
Ghost Gallery deals with published historical information.
TYLO is different.
A user could upload a photograph of a loved one, describe their family relationships, share private experiences, explain personal traditions, and write down memories they may never have shared publicly.
That information deserves a proper identity and access-control layer.
I chose Auth0 because I needed to establish who was making a request before the API accessed their private companion data.
What the integration does
On the frontend, I'm using:
@auth0/auth0-react
On the Express API, token validation is handled using:
express-oauth2-jwt-bearer
The basic flow is:
- The user authenticates through Auth0.
- The frontend sends an Auth0 access token with a protected TYLO API request.
- The Express API validates the token's issuer and audience.
- The API reads the authenticated user's subject.
- Companion and memory access is scoped to that identity.
This is especially important for vector search.
Consider two users who have both created a companion profile for their grandmother. Their memories might contain similar words, names, family relationships or traditions.
A semantic search that ignores ownership could return a memory belonging to the wrong user.
That is unacceptable for TYLO.
The vector retrieval query is therefore scoped using:
ownerSubcompanionIdembeddingModel
The ownership identity must come from the validated token, not from an arbitrary user identifier supplied by the browser.
What Auth0 does not automatically do
I also want to be precise about the security claims.
Auth0 establishes and verifies identity and gives the API a trusted identity context. It does not automatically encrypt every MongoDB record, enforce every database filter, or make the application end-to-end encrypted.
Those responsibilities still belong to the application's backend, database configuration and data-handling design.
I have also not claimed to implement Auth0's more advanced AI-agent capabilities such as Token Vault, asynchronous authorization or fine-grained authorization unless those features are actually integrated.
For this project, the implemented value is the identity and protected API layer around private AI conversations.
That is why Auth0 mattered to the architecture rather than simply being another login button.
05. Choosing Gemma: my laptop had an opinion too.
Now for the model side.
I wanted to experiment with open-weight models and run local inference during development. Ollama made it straightforward to pull a model and connect it to the application.
Then I met the less exciting part of local AI development:
Hardware limitations.
I experimented with different local model options, but my laptop's available RAM made larger models uncomfortable to work with. I needed something practical enough to test repeatedly, not a model that turned every experiment into a hardware-management exercise.
I settled on experimenting with Gemma through Ollama, including the gemma4:e2b model.
It wasn't about declaring one model universally better than everything else. It was about finding a practical model that allowed me to keep building and testing.
The model was only half the work
I did not fine-tune Gemma or retrain its weights.
The improvements I worked on were at the application level:
- Refining the instructions given to the model.
- Supplying relevant companion profile information.
- Selecting recent conversation context.
- Retrieving older exchanges through vector search.
- Including saved memories and the companion summary.
- Adjusting how the application assembles the final context.
In other words, I was improving the information available to the model and the way the task was framed.
This distinction is important.
A model can produce a poor answer because its underlying capabilities are limited. But it can also produce a poor answer because the application supplied incomplete, irrelevant or misleading context.
Those are different problems, and they need different solutions.
The model provider is configurable
The backend is not tightly coupled to a single local Ollama process.
The inference architecture supports Cloudflare Workers AI when the required API credentials are configured. Otherwise, the local inference path uses Ollama through MODEL_BASE_URL, which defaults to:
http://localhost:11434
This separation helped keep the API architecture flexible.
It also created an important deployment lesson: Render does not automatically have access to the Ollama instance running on my laptop.
The API service needs a reachable inference endpoint. Cloudflare credentials or another correctly configured model endpoint must be supplied to the deployed API.
I don't want to claim that Render hosts Gemma inference. It hosts the application services; inference is handled separately.
06. The architecture: Render, Express and MongoDB Atlas
The application is split into separate services.
| Layer | Technology | Responsibility |
|---|---|---|
| Frontend | React + TypeScript + Vite | User interface and application experience |
| Styling | Tailwind CSS | Layout and visual design |
| Routing | React Router | Navigation between experiences |
| Identity | Auth0 | Authentication and access-token identity |
| Backend | Node.js + Express + TypeScript | API, validation and application logic |
| Database | MongoDB Atlas | Profiles, messages and memories |
| Retrieval | Atlas Vector Search | Semantic memory retrieval |
| Inference | Cloudflare Workers AI / Ollama | Model inference through configured provider |
| Deployment | Render | Frontend and API hosting |
Why separate the frontend and API?
The React application is served as a Render static site.
The Express backend runs as a separate Render web service.
This keeps the responsibilities clear:
- The frontend handles the user experience.
- The API handles identity validation, database access and context assembly.
- MongoDB Atlas stores application data and supports memory retrieval.
- The configured inference provider generates the response.
Cloudflare credentials are not declared in render.yaml in the architecture snapshot I'm documenting. They need to be supplied through the API service's environment configuration to use the Cloudflare path in production.
Otherwise, the API falls back to the configured Ollama endpoint. On Render, localhost refers to the API service environment, not my personal laptop.
It sounds obvious once you draw the diagram. It is considerably less obvious when you're staring at a deployment that works locally and doesn't behave the same way remotely.
07. The testing: where the architecture had to justify itself
I don't think "I sent a message and the model replied" is a meaningful test for a project like this.
A memory companion can generate a warm, natural response and still get the important details wrong.
For TYLO, I wanted to look at whether the conversation reflected the profile, whether the memory architecture could supply relevant context, and whether the system respected the boundaries around private information.
Here are some of the checks and observations that shaped the implementation.
Test 1 — Does the profile influence the response?
I created a sample companion profile with information across the four sections: personality, speaking style, favourite phrases, shared experiences, beliefs and routines.
Then I tried a simple greeting.
The resulting conversation reflected the positive, encouraging tone described in the profile.
That was encouraging. It showed that the companion profile could influence the generated response rather than the model replying with a completely generic greeting.
But the response also included a present-day detail that had not been supplied in the profile.
That was an important observation.
The response sounded natural, but sounding natural is not the same as being grounded in memory.
It exposed a distinction I need to preserve throughout the project:
The system can use a person's personality and memories as conversational context, but it should not invent personal facts and present them as remembered experiences.
That is an ongoing quality and prompt-design challenge, not something I will pretend has been completely solved.
Test 2 — Can the application use older context?
A conversation has two different memory requirements.
The first is immediate continuity: understanding what was said a few messages ago.
The second is long-term recall: finding something meaningful from an older exchange that is no longer in the recent message window.
The current context assembly addresses both:
- The latest 8 message records provide recent conversational continuity.
- The companion summary provides compressed context.
- Up to 3 saved memories provide additional profile-level information.
- Atlas Vector Search retrieves up to 4 relevant older exchanges from a candidate set of 50.
The consequence is architectural: an older relevant exchange does not have to be present in the latest eight messages to be considered for the next response.
I haven't produced a formal retrieval-accuracy benchmark, so I won't attach a made-up percentage to this improvement. The retrieval configuration is implemented; its quality still needs systematic evaluation across a larger set of real test cases.
Test 3 — Can one user's memories leak into another user's conversation?
This is one of the most important tests for TYLO.
A normal search might find semantically similar memories across multiple users. But TYLO cannot treat all stored memories as one shared pool.
The retrieval query includes the verified owner identity and companion identifier.
That gives the memory search a defined ownership boundary, rather than relying on the model to decide which memories are appropriate.
The implementation uses ownerSub and companionId as filters.
The next level of verification here is a dedicated multi-user integration test: create separate users with similar companion profiles, seed similar memories, and verify that each user can retrieve only their own data. The filters are part of the architecture, but I won't describe that as a completed security audit.
Test 4 — What happens when retrieval fails?
Vector search is an additional source of context, not the only source.
The failure path logs a retrieval error and continues with the companion profile and recent history.
That gives the application a graceful-degradation path.
It is better to have a conversation with limited context than to silently pretend that missing memories were successfully retrieved.
Test 5 — Does the profile creation flow enforce the required information?
The TYLO creation flow was also tested beyond just its appearance.
The companion creation process requires a photograph and the structured profile details. Validation exists on the server as well as in the interface.
That matters because frontend validation alone can be bypassed.
The image flow also needed attention: image selection, JPEG compression, previewing the selected image and saving the resulting image data with the companion document.
The current implementation stores the photo as image data in MongoDB. That is straightforward for an initial version, but it increases document payload size. Moving images to dedicated object storage is a sensible future improvement.
What the tests actually establish
I want to be honest about the level of verification.
| Area | What has been established | What is still needed |
|---|---|---|
| Profile creation | Required photo and profile validation, including server-side validation | Broader browser and edge-case testing |
| Image handling | Selection, compression, preview and persistence flow implemented | Evaluate object storage as usage grows |
| Conversation | A sample profile-driven response was generated | Larger grounded-response evaluation |
| Memory retrieval | Vector index, candidate retrieval, limits and filters are part of the architecture | Formal recall and relevance evaluation |
| Isolation | Owner and companion filters are used for retrieval | Dedicated multi-user security integration tests |
| API | Health endpoint returned successfully outside the restricted development environment | Full deployed request-path verification |
| Build |
npm run build passed at the documented development snapshot |
Re-run after subsequent changes |
| Full AI conversation | The application flow and a sample conversation are present | A complete end-to-end test across deployed Auth0, API, MongoDB and the configured inference provider |
This is not a claim that the system is production-audited or that retrieval accuracy has been scientifically measured.
It is a record of what works, what the design does, and what still deserves testing.
08. The bugs that had nothing to do with AI
Some of the most frustrating problems had absolutely nothing to do with the model.
While building the frontend, I ran into several regressions that reminded me how much work exists between a feature working in isolation and the application feeling coherent.
A few examples:
The Explore Story page. It initially displayed a large chat composer where I wanted a focused story experience. The selected historical figure also needed to load directly from its slug. The navigation and route handling had to be corrected so a figure could be opened reliably.
The Talk Now modal. The modal had an issue with its lifecycle and could close unexpectedly. React Strict Mode exposed behaviour that wasn't as stable as I wanted. I had to fix the modal's interaction handling, including Escape and backdrop behaviour.
The modal position. After a CSS reset, the modal appeared pinned to the top-left instead of sitting in the centre. It was one of those bugs where the code looked reasonable until the browser rendered it. Explicit positioning fixed it.
The TYLO layout. The companion creation form was appearing before the existing memory spaces and conversations. That made the page feel like a setup screen even for someone who had already created companions. Reordering the experience made the saved spaces and conversations easier to reach.
Historical portraits. Finding images for historical figures was another problem. A URL returning an image does not prove that the image depicts the right person. Some candidate images needed stricter filtering and manual review. An unrelated film still or an incorrectly attributed portrait can undermine the whole experience.
I also used Codex to help inspect and fix targeted issues. It was useful for identifying regressions and making focused changes, but I still had to review the results and build the application again.
At one point, the changes had passed the build check, but the full live AI conversation had not been verified end to end. That distinction matters.
A successful build tells me the code compiles. It doesn't tell me that Auth0, the database, vector retrieval and the deployed inference provider all cooperate correctly in a real conversation.
09. The DigitalOcean plan that never happened
I initially considered provisioning a DigitalOcean Droplet to run model inference.
The idea was straightforward: host the model separately and have the API communicate with it.
But I got stuck at the billing setup and, honestly, I was too lazy to finish the provisioning process at that stage.
So I moved forward with the architecture I could actually work with: Render for the web application and API, MongoDB Atlas for persistence and retrieval, and a configurable inference provider.
It wasn't some grand technical decision after an extensive infrastructure benchmark.
Sometimes the engineering story is simply that the infrastructure you planned is not the infrastructure you end up using.
I am not claiming a DigitalOcean integration, because I did not provision one.
10. Why each partner technology belongs in this project
I didn't want to select partner categories just because a technology appeared somewhere in the dependency list.
Each one needs to have a meaningful responsibility in the application.
Best Use of MongoDB Atlas
MongoDB Atlas is the persistence and memory-retrieval layer.
It stores the historical profiles, TYLO companions, conversation messages and memory exchanges. Atlas Vector Search supports semantic retrieval over the 768-dimensional memory embeddings.
The tylo_memory_vector_index index, candidate limit, returned exchanges and owner/companion filters are part of the actual memory architecture.
Without this layer, TYLO would have a much weaker approach to long-term conversational recall.
Best Use of Render
Render hosts two distinct application services:
- The React frontend as a static site.
- The Express API as a separate web service.
The API handles protected requests, database operations, context assembly and communication with the configured model provider.
Render is responsible for hosting the application services. It is not the machine running my laptop's local Gemma model.
Best Use of Auth0 for AI Agents
TYLO works with private, user-provided information.
Auth0 establishes the authenticated identity used by protected API requests. The Express backend validates the access token, reads the user's subject and uses that identity when accessing companion and memory data.
The relationship between identity and memory retrieval is the important part here.
A user should not be able to access another person's companion simply because a vector search finds similar text.
I'm describing the Auth0 functionality actually integrated: authentication, access-token validation and protected API access. I'm not claiming Token Vault, asynchronous authorization or fine-grained authorization features that are not part of the current implementation.
Best Use of Gemma
Gemma was part of my local model experimentation through Ollama.
The hardware limitations on my laptop influenced the model I could practically use during development. I then iterated on the application-side instructions and context assembly to improve the quality of the conversations.
I did not fine-tune Gemma.
One important submission detail: the deployed inference provider is configurable. I will make sure the submitted demo's actual model configuration supports the Gemma category claim. A local development experiment alone should not be presented as proof that the deployed demo is currently running Gemma.
Categories I am not claiming
- DigitalOcean: I considered it but never provisioned the Droplet.
- ElevenLabs: It belongs to a different project direction, not the current POT implementation.
- GitHub Copilot: I used Codex for coding assistance; I am not claiming GitHub Copilot integration.
I would rather enter fewer categories with a defensible technical explanation than list every partner name without meaningful usage.
11. Open innovation: why open models mattered to me
One of the most interesting parts of this project was being able to experiment with open-weight models rather than treating the AI layer as an invisible external service.
Using Ollama gave me a practical local development path. I could experiment with a model, modify the application prompt, change the context being supplied and inspect the response.
That made the model layer feel like something I could investigate and improve around.
But there is a distinction I want to preserve:
Open-weight model experimentation does not automatically make an application fully local or private.
POT also uses hosted infrastructure. MongoDB Atlas stores application data, Render hosts the application services, and the configured inference provider may be cloud-based.
The system's privacy and data-handling properties depend on the actual deployment and configuration, not simply on whether the model's weights are open.
That is especially important for TYLO.
12. What I would improve next
There are several things I would like to take further.
A proper memory evaluation suite. I want a repeatable collection of questions where the correct answer depends on an older profile detail or a previous conversation. I would measure whether the right exchange is retrieved, whether irrelevant memories are excluded and whether the generated answer stays within the available evidence.
Stronger grounding behaviour. The sample conversation showed that a warm response can still introduce unsupported details. I want the companion to distinguish between something explicitly remembered, a reasonable conversational response and something it simply does not know.
A dedicated privacy test suite. Multi-user tests should verify that similar companion profiles and overlapping memories never cross ownership boundaries. I also want broader tests around protected routes and companion access.
Object storage for photographs. Storing compressed image data directly with companion documents is convenient at this stage, but dedicated object storage will be a better direction as the number of users and images grows.
Historical source quality. Ghost Gallery needs a stronger process for reviewing profile information and portrait attribution. A historical conversation should not sound confident about information that is uncertain or unsupported.
A more complete deployed integration test. I want to verify the full path—from Auth0 authentication to the Express API, MongoDB retrieval, configured model inference and the streamed response—in the actual deployed environment.
13. The part I enjoyed the most
The most satisfying moment wasn't getting an API endpoint to return a response.
It was seeing a structured companion profile turn into an actual conversation.
A photograph, a relationship, a collection of everyday details, favourite phrases and stories—all of that information started to become useful conversational context.
It made me think differently about what a memory system needs to do.
It also made me more cautious about what it should say.
A convincing imitation is not necessarily a respectful one. The system needs to be thoughtful about the difference between what a person actually shared and what a language model is capable of inventing.
That is probably the biggest lesson I am taking away from this build.
The AI part is exciting. The engineering around it is what makes the experience meaningful.
And there is still a lot I want to get right.
Prize Categories
- Overall Winner 👀
- Best Use of MongoDB Atlas
- Best Use of Render
- Best Use of Auth0 for AI Agents
- Best Use of Gemma
Top comments (0)