DEV Community: Sebastien Blanc

Java & GenAI : The Ultimate Developer's joy with Quarkus, Langchain4j and Ollama

Sebastien Blanc — Tue, 05 Mar 2024 19:05:21 +0000

This is the companion blog for this video
.

In this short post I will show you how to get started with Quarkus, Langchain4j and Ollama to set up your local dev environment to developer GenAI applications. Ready ? Let's go !

Installing Ollama

Ollama is one of those magic those that makes developers happy, basically you install it and then you can run LLMs locally, it has also a embedded web server so that you make requests to it, just like you would do with OpenAI.

Head over to Ollamaand follow the installation instructions.

Open a terminal and type :

ollama list

It will list you the LLMs that are installed locally on your computer. You can easily add a new LLM by doing for instance :

ollama pull mistral:latest

Check the model page to see what is available, it's that easy !

Creating the Quarkus project

Your Quarkus app will need for now only 2 extensions : quarkus-resteasy-reactive and quarkus-langchain4j-ollama , you can also use this link and click on the big blue button "Generate your application"

Once unzipped, open your project in your favorite IDE.

First steps with Langchain4j and Quarkus

We need to define in the properties which LLM we will be using, open your application.properties and add this line :

quarkus.langchain4j.ollama.chat-model.model-id=llama2:latest

Now let's create our first AI Service, create a new interface called MyAIService.java :


package org.sebi;

import dev.langchain4j.service.UserMessage;
import io.quarki
verse.langchain4j.RegisterAiService;

@RegisterAiService
public interface MyAIService {
     public String chat(@UserMessage String message);
}

The important thing to notice here is the @RegisterAiService annotation, this indicates to Quarkus that when calling the method chat it will be using the LLM that we defined in the properties.

Run your app : mvn quarkus:dev , this will start your application in dev mode.

Once it's running, press d in the terminal to open the devUI :

You should see the langchain4j widget containing a link called chat, you can click on it and it will bring you to an interface where can prompt your LLM !

At any point, you can change the model in the properties, you can also enable logging for the outgoing requests to see what is exactly sent to the LLM :

quarkus.langchain4j.ollama.chat-model.model-id=llama2:latest
quarkus.langchain4j.ollama.log-requests=true

That's pretty cool, isn't it ?

Exposing our AI Service as a rest endpoint

The DevUI is useful but now we want to make it possible to access our LLM from a rest endpoint, we already have the resteasy-reactive extension and you should also have already a resource class called GreetingResource , change the content with :

package org.sebi;

import jakarta.inject.Inject;
import jakarta.ws.rs.Consumes;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.Produces;
import jakarta.ws.rs.core.MediaType;

@Path("/chat")
public class GreetingResource {

    @Inject
    MyAIService myAIService;

    @POST
    @Produces(MediaType.TEXT_PLAIN)
    @Consumes(MediaType.TEXT_PLAIN)
    public String hello(String body) {
        return myAIService.chat(body);
    }
}

Here we simply inject our AI Service and call it in our POST method.

Try it out with this curl command :

curl -X POST  -H 'Content-Type: text/plain' -d 'Tell me a joke' http://localhost:8080/chat

I got this one :

Sure, here's one:

Why don't scientists trust atoms?
Because they make up everything!

Adding a system prompt

Let's fine tune our prompt by providing a system message. Usually you can use this to define how the chat bot must behave. Go to your AI Service and add this annotation :

     @SystemMessage("You are a funny bot and always reply with a poem!")   
     public String chat(@UserMessage String message);

Try your curl command again and it should reply with a joke in the form of a poem !

You can change as you want the system prompt to experiment, change the model, all of this without having to stop&start your app because you are running in dev mode !

That all for today, next time we will explore how to add memory to our bot !

It's rough out there ...

Sebastien Blanc — Sun, 11 Dec 2022 19:55:22 +0000

It’s rough out there, this is how Matty started his closing keynote of DevRelCon 2022 in Prague. This was also Ben’s starting point in his talk Measuring DevRel during a downtown. We all know how hard it is to measure the impact of a DevRel team inside an organisation, and especially in those moments we might be an easy target when it comes to cutting down costs …

It’s rough out there but after those 2 days of conferences I feel energised, confident and motivated like there is no tomorrow. It was my first time attending a DevRelCon and to be honest I was a bit skeptical before joining, a bunch of DevRel people gathering ? To do what ? Is it worth it ? Shouldn’t we all be there on the front now, doing our job ? But I was wrong, completely wrong and sometimes I love being wrong. During those two days I saw so many inspiring talks, had so many great conversations, made some many new connections that it felt really like one of the best personal investment for me in 2022 and by transitivity a good investment for my company.

Ben says “The long term viability of a DevRel team is found in its ability to show value across the organization” but how do you do this in an effective manner ? There are some really pragmatic actionable things that you can easily implement that were presented during those 2 days and I will come back to this later in this post but before let me focus on Ben’s crucial statement “What is your team’s origin story?”. Why does your DevRel team exist in your company ? As a team, try to remember this story and while you may have started a lot of different initiatives since then it’s maybe time to return to the origin path and focus on that because this will have the most impact. That origin story also reminds me a bit of what Simon Sinek explains in his book Start with Why.
Choose metrics that are specific, achievable, realistic and the most important part : Tell your organization about your achievements, share it, make it visible, even sometimes brag a bit about when you think it deserves it !

So let’s go back to some fundamentals, and we as passionate DevRel people, there is one thing we love to do and which is central in our job : Building demos ! In his talk Beyond the kitchen sink: making the case for small demos, Kevin reminds us of all the different types of demos that we can build, going from the kitchen sink demo that shows everything to the really specific use-case focused demo. He also points out all the potential risks that these demos can have, one that particularly resonated to me was the risk of “Use-case Lock-in” where ”Demos intertwined with an overly-specific use case might make it difficult to abstract value for other use cases.”.
Also how far does your demo need to go ? All that complicated code that you had to add and which is not related to your product/API might be an obstacle, because you need to explain these lines of code while “THE goal is to teach devs how to use your product”. This is also a challenge we are facing right now in our team. Kevin’s proposition is to take as an example the “Atomic Design” methodology, primarily focused on building interfaces but which can be applied for building demos. He explains it way better than me in his blogs post and I really recommend you to read it.

“Blog posts and stickers are not enough” said Don in his really inspiring talk (more on this later) but it’s also true for demos, it’s pretty obvious but we need more than those to make an impact, to create, grow a community … Community is your audience, and why not grow your audience with your non-IT skills ? This is what Marc showed us during his talk Draw an audience with your non-IT skills to build your personal brand, again this really resonated to me since I’m doing that a lot, you might wonder what is my non-IT skill ? Well, it’s doing crazy/funny/absurd short videos where I cover songs replacing the lyrics with IT related stuff, like Highway to Helm , Killing in the namespace or Come as a jar …

And since we talk about community, there were several talks specific to this topic. It was really interesting to hear Ully or Alena speaking about how to build a community champions program, a lot of great tips, like starting small, the importance of the type of swag, the dashboards etc …
If done well, you will turn each of your community members into dev advocates and this is on you can scale your activity, because as we all know, scaling DevrRel is hard. Karin had an awesome talk on this where she basically told us that one way of doing this is by asking each community member/user to do a little bit more. She showed us the different types of users that you have, like a lurker or for instance an active contributor, on their level they all contribute in one way or the other, put a “like” on a tweet is already a contribution, if this person can next time comment on this tweet, well, then you are already scaling …

Be kind to your community, be respectful, this sounds obvious but actually doing this in the long term will have a concrete effect on human beings : it will release a really powerful neurotransmitter called oxytocin. “Oxytocin is literally Love”, “Oxytocin is literally Community”, these were some of the powerful quotes that Don delivered to us during the first closing talk. It was really inspiring and also hard to explain here in a blog post, so I can’t wait to have the replay and share it with you.

Community is not just the outside world, your company is also a community, and we as DevRel must do exactly the same thing internally than we do it externally. This can take the really concrete form of setting up a dedicated internal advocacy team, as Amy and Afzaal from Adyen explained to us, offering a first class onboarding experience, moving knowledge silos into open and standardized documentation …
Your internal community is not just the developers : marketing, sales, product … Matty, in his closing keynote, encouraged us to build relationships with those departments as well. It's crucial, and you will learn a lot about your own company while creating new opportunities. For instance, do you have an important message to deliver ? Ask your marketing team for help, they know better than no one how to do this.

This is it for now, maybe I will do a second post for some of the other talks that I saw but this was what was fresh in my mind and I wanted to write it down as quickly as possible. As I said in the intro, this conference was a blast for me and gave me so much food for thought and as always it was the opportunity to meet so many nice people (and this includes meeting for the first time part of my awesome team at Aiven !).