What to expect from this article?
We finished building the account management app, and integrated the diary AI model on previous articles,this article will cover Gemini API integration for memories logic in details
I'll try to cover as many details as possible without boring you, but I still expect you to be familiar with some aspects of Python and Django.
the final version of the source code can be found at https://github.com/saad4software/alive-diary-backend
Series order
Check previous articles if interested!
- AI Project from Scratch, The Idea, Alive Diary
- Prove it is feasible with Google AI Studio
- Django API Project Setup
- Django accounts management (1), registration and activation
- Django accounts management (2), login and change password
- Django Rest framework with Swagger
- Django accounts management (3), forgot password and account details
- Diary App, diaries API
- Diary App, diary AI integration
- Diary App, memories AI integration (You are here 📍)
The Idea 💡
In the last article, we integrated Gemini API for diary logic, including creating and talking to diaries. we agreed on having one diary for each user, and each diary data is saved as a list of conversations that contain a list of messages.
Well, memories are kinda similar to diaries, we create a memory by asking the right questions (refer to this Prove it is feasible with Google AI Studio for more details) and talking to memories is similar to talking to diaries. the only differences are in the prompts 🤖
okay, so... let's create an API to create and remove memories, another one for creation conversation (asking the right questions), and a final one for talking to it. This should do, let's start
The memories API
Since we are using the same Diary model for memories, we will use the same diary serializer (refer to Diary App, diaries API for more details), let's move to views
class MemoriesViewSet(ModelViewSet):
permission_classes = (IsAuthenticated, )
lookup_field = 'id'
serializer_class = DiarySerializer
pagination_class = StandardResultsSetPagination
renderer_classes = [CustomRenderer, BrowsableAPIRenderer]
filter_backends = [filters.DjangoFilterBackend, rest_framework.filters.SearchFilter]
search_fields = ['title']
def get_queryset(self):
return Diary.objects.filter(
user=self.request.user,
is_memory=True,
).order_by("-created")
def perform_create(self, serializer):
serializer.save(
user=self.request.user,
is_memory=True,
)
def destroy(self, request, *args, **kwargs):
instance = self.get_object()
self.perform_destroy(instance)
return Response("success")
app_main/views.py
We are using DiarySerializer, getting only the memories is_memory=True, ordering them from the newest to oldest, and making sure memories belong to this user and are set as memory when saved.
now for the URLs
memories_list = MemoriesViewSet.as_view({
'get': 'list',
'post': 'create'
})
memories_details = MemoriesViewSet.as_view({
'delete': 'destroy'
})
urlpatterns = [
path('diaries/', diaries_list),
path('diaries/<int:id>/', diaries_details),
path('conversation/', ConversationsView.as_view()),
path('diaries/conversation/', DiaryConversationsView.as_view()),
path('memories/', memories_list),
path('memories/<int:id>/', memories_details),
]
app_main/urls.py
simply connecting the model view-set to list, create, and delete URLs. we can test the APIs directly with Swagger
We need to log in (using the login API) and authenticate using Bearer authentication (refer to Django Rest framework with Swagger for more details), and we should be able to list and create memory instances.
Create memory AI integration
Similar to creating diary AI integration. so, how should we integrate it? mmmm....
I believe one conversation per memory should be enough, so it would look somewhat like
from .gemini_script import create_diary_session, talk_to_diary, create_memory_session
class MemoryCreateView(APIView):
permission_classes = (IsAuthenticated, )
renderer_classes = [CustomRenderer, BrowsableAPIRenderer]
def get_conversation(self, memory_id):
memory = Diary.objects.filter(
id=memory_id,
user=self.request.user,
is_memory=True,
).first()
if not memory:
raise APIException("invalid_id")
conversation = Conversation.objects.filter(
diary=memory,
).first()
if not conversation:
conversation = Conversation(
diary=memory,
)
conversation.save()
return conversation
def get(self, request, **kwargs):
conversation = self.get_conversation(kwargs['id'])
messages = conversation.messages.order_by("created")
history = list(map(lambda msg: {
"role": "user" if msg.is_user else "model",
"parts":[msg.text]
}, messages))
chat_session = create_memory_session(history)
ai_response = chat_session.send_message("hi")
ai_message = Message(
text=ai_response.text,
conversation=conversation,
is_user=False
)
ai_message.save()
return Response(MessageSerializer(ai_message).data)
@swagger_auto_schema(request_body=MessageSerializer)
def post(self, request, **kwargs):
serializer = MessageSerializer(data=request.data)
if not serializer.is_valid():
raise APIException(serializer.errors)
conversation = self.get_conversation(kwargs['id'])
message = serializer.save(
conversation=conversation,
is_user=True,
) # save users message
# ai_response = "hello world again" # get ai response
messages = conversation.messages.order_by("created")
history = list(map(lambda msg: {
"role": "user" if msg.is_user else "model",
"parts":[msg.text]
}, messages))
chat_session = create_memory_session(history)
ai_response = chat_session.send_message(message.text)
ai_message = Message(
text=ai_response.text,
conversation=conversation,
is_user=False
) # save ai response message
ai_message.save()
return Response(MessageSerializer(ai_message).data)
app_main/views.py
This API is designed to start the memory-capturing conversation, similar to the diary conversation, it starts with a GET request.
It requires selecting what memory to work with, which is derived from the memory ID from the request URL kwargs['id'] (we should add it in the URLs file later)
the get_conversation function get this memory conversation instance or create it if it doesn't exist. then we build the conversation history similar to the diary conversation history.
Since capturing memory requires another prompt, we created a new function create_memory_session in gemini_script. it should look somewhat like
def create_memory_session(history):
model = genai.GenerativeModel(
model_name="gemini-1.5-flash",
generation_config=generation_config,
system_instruction="You are trying to capture the moment, get as much details as you can about the event, date, time, place, weather, what you can see, hear, taste, and feel. ask one question at the time and don't use emojis",
)
chat_session = model.start_chat(
history=history
)
return chat_session
app_main/gemini_script.py
it is similar to create_diary_session function but with a different prompt (refer to Prove it is feasible with Google AI Studio for more information about the prompt)
now for the URLs file, we have the id as part of the URL path
urlpatterns = [
path('diaries/', diaries_list),
path('diaries/<int:id>/', diaries_details),
path('conversation/', ConversationsView.as_view()),
path('diaries/conversation/', DiaryConversationsView.as_view()),
path('memories/', memories_list),
path('memories/<int:id>/', memories_details),
path('memories/<int:id>/create/', MemoryCreateView.as_view()), # new
]
app_main/urls.py
Nice, testing this on Swagger should start with the GET request, after getting the conversation ID from it, we can continue the conversation using the POST request. Testing the API resulted in this conversation
+ What is the event you are trying to describe?
- graduation day
+ What is the date of the graduation?
- it was on the 15th of November
+ What year is the graduation?
- it was in 2008
+ What time did the graduation ceremony begin?
- it started at 11 am
+ Where did the graduation ceremony take place?
- at Damascus university, faculty of electronics and communication engineering
+ What was the weather like on that day?
- it was sunny but cold
...
great conversation! it is trying to get all useful details, even though the user didn't give it deliberately. what next?
We should also be able to talk to the memory, let's work on it next
Talk to memory AI integration
Similar to talking to diary, we will fine-tune the memory AI model using structured input. the views file should look like
class MemoryConversationsView(APIView):
permission_classes = (IsAuthenticated, )
renderer_classes = [CustomRenderer, BrowsableAPIRenderer]
def get(self, request, **kwargs):
memory = Diary.objects.filter(
id=kwargs['id'],
is_memory=True,
).first()
conversations = memory.conversations.order_by("-created")
prompt = [
"You are a memory, be as clear as possible, use short clear description from the third person point of view, don't use emojis"
]
for conv in conversations:
for msg in conv.messages.order_by("created"):
segment_1 = "output" if msg.is_user else "input"
prompt += [f"{segment_1}: {msg.text}"]
prompt += ["input: describe the memory for me", "output: "]
ai_response = talk_to_diary(prompt)
# conversation = Conversation(diary=diary)
# conversation.save()
ai_message = Message(
text=ai_response,
conversation=conversations.last(),
is_user=False
)
# ai_message.save()
return Response(MessageSerializer(ai_message).data)
@swagger_auto_schema(request_body=MessageSerializer)
def post(self, request, **kwargs):
serializer = MessageSerializer(data=request.data)
if not serializer.is_valid():
raise APIException(serializer.errors)
memory = Diary.objects.filter(
id=kwargs['id'],
is_memory=True,
).first()
conversations = memory.conversations.order_by("-created")
prompt = [
"You are a memory, be as clear as possible, use short clear description from the third person point of view, don't use emojis"
]
for conv in conversations:
for msg in conv.messages.order_by("created"):
segment_1 = "output" if msg.is_user else "input"
prompt += [f"{segment_1}: {msg.text}"]
prompt += [f"input: {serializer.validated_data.get('text')}", "output: "]
print(prompt)
ai_response = talk_to_diary(prompt)
# conversation = Conversation(diary=diary)
# conversation.save()
ai_message = Message(
text=ai_response,
conversation=conversations.last(),
is_user=False
)
# ai_message.save()
return Response(MessageSerializer(ai_message).data)
app_main/views.py
Similar to other conversations API, we start by calling the GET request. the memory is selected by the ID (should be included in the URL), then we build the structured data and fed it to the model.
To continue the conversation, we use the POST request.
the URLs file should look like
urlpatterns = [
path('diaries/', diaries_list),
path('diaries/<int:id>/', diaries_details),
path('conversation/', ConversationsView.as_view()),
path('diaries/conversation/', DiaryConversationsView.as_view()),
path('memories/', memories_list),
path('memories/<int:id>/', memories_details),
path('memories/<int:id>/create/', MemoryCreateView.as_view()),
path('memories/<int:id>/talk/', MemoryConversationsView.as_view()), # new
]
app_main/urls.py
That is it! we can test it on Swagger
We still need to create an API for sharing diaries and memories with other users I think 🤔, but it can wait. We will start working on the Flutter Alive Memory app development next so
Stay tuned 😎
Top comments (0)