DEV Community: Jonathan U

Running DeepSeek-R1 Locally - Use with Open WebUI, Chatbox, CodeGPT

Jonathan U — Fri, 31 Jan 2025 00:56:43 +0000

What is DeepSeek-R1?

DeepSeek-R1 is an open-source LLM developed by the Chinese AI startup DeepSeek. It achieves performance comparable to OpenAI's o1 model but was created at a fraction of the cost. The model uses a "chain-of-thought" reasoning approach, enhancing the quality of its responses by systematically breaking down problems into logical steps. This methodology allows DeepSeek-R1 to excel in tasks requiring detailed reasoning and analysis.

Install and run Ollama

Ollama is a platform that allows users to interact with AI models in a way that is optimized for local, privacy-respecting, and customizable experiences.

There are a couple ways to install Ollama:

Download from https://ollama.com/

Alternatively, if you're on Mac:

brew install ollama

Once Ollama is installed, if it's not running already:

ollama serve

DeepSeek-R1 Models

If you have a powerful computer, you can try to run DeepSeek-R1:

ollama run deepseek-r1:671b

I do not have a computer capable of running this model. The DeepSeek team has demonstrated that the reasoning patterns of larger models can be distilled into smaller models, resulting in better performance than the reasoning patterns discovered through RL on small models.

There are several distilled models to run with Ollama. The options for DeepSeek-R1 are:
DeepSeek-R1-Distill-Qwen-1.5B

ollama run deepseek-r1:1.5b

DeepSeek-R1-Distill-Qwen-7B

ollama run deepseek-r1:7b

DeepSeek-R1-Distill-Llama-8B

ollama run deepseek-r1:8b

DeepSeek-R1-Distill-Qwen-14B

ollama run deepseek-r1:14b

DeepSeek-R1-Distill-Qwen-32B

ollama run deepseek-r1:32b

DeepSeek-R1-Distill-Llama-70B

ollama run deepseek-r1:70b

I'm on a Macbook Pro with the M1 Pro chip and 16 GB of RAM. I tried running the deepseek-r1:14b model, but it was not able to perform well. I've had the best luck with the deepseek-r1:7b model.

The ollama run command is successful if you see something like this:

Using DeepSeek-R1 Locally

If you want to use something other than the terminal to interact with DeepSeek-R1, other options I've used are:

Chatbox AI

Chatbox AI is a simple download with a decent UI to interact with your local LLMs out of the box. It has several features that I haven't tried yet, but I've found this way to be the simplest to get it up and going.
To configure it to use your local DeepSeek-R1:

Settings -> Under the Model tab -> Select OLLAMA as the model provider -> API Host should be correct by default -> Select the Model you've installed in the Model dropdown (deepseek-r1:7b).
Then you should be all set!

Open WebUI

Open WebUI is an extensible, feature-rich, and user-friendly self-hosted AI platform designed to operate entirely offline. It supports various LLM runners like Ollama and OpenAI-compatible APIs, with a built-in inference engine for RAG, making it a powerful AI deployment solution.

There are a couple ways to run Open WebUI:

Using Python 3.11

pip install open-webui

open-webui serve

This will start the Open WebUI server at http://localhost:8080

Docker The Open-WebUI has documentation on different ways to run it in docker, which you can find here: https://docs.openwebui.com/getting-started/quick-start

For my case, I just ran this command:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

If you want to bypass the login page. You'll need to set the WEBUI_AUTH environment variable to False:

docker run -d -p 3000:8080 -e WEBUI_AUTH=False -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

I use Open WebUI the most when interacting with my local LLMs.

CodeGPT

CodeGPT is an extension for VSCode/Cursor, as well as a plugin for JetBrains IDEs. It's an alternative to Github Copilot and allows you to use your local LLMs, such as the DeepSeek-R1 model, for prompts, code completion, unit testing, auto-complete, and more.

Conclusion

It's simple and accessible to run open-source LLMs on your local machine thanks to Ollama, DeepSeek, and other open-source LLMs.
Tools such as Chatbox AI, Open WebUI, and CodeGPT make it easy and intuitive when using these LLMs.
These tools also provide the option to use the official hosted APIs if you have an API key.

How to make Apache's CloseableHttpAsyncClient explicitly use HTTP/1 or HTTP/2

Jonathan U — Wed, 12 Jun 2024 22:33:06 +0000

In this tutorial, we'll demonstrate how to explicitly use HTTP/1 or HTTP/2 in Apache's CloseableHttpAsyncClient, a base implementation of HttpAsyncClient that also implements ModalClosable.

You'll need to have this dependency to your pom.xml:

<dependency>
    <groupId>org.apache.httpcomponents.client5</groupId>
    <artifactId>httpclient5</artifactId>
    <version>5.3.1</version>
</dependency>

Deprecated Way

In earlier versions of httpclient5, the HttpAsyncClientBuilder allowed you to use setVersionPolicy(org.apache.hc.core5.http2.HttpVersionPolicy versionPolicy), where you could pass in HttpVersionPolicy.FORCE_HTTP_1 or HttpVersionPolicy.FORCE_HTTP_2. For example:

final CloseableHttpAsyncClient client = HttpAsyncClients
        .custom()
        .useSystemProperties()
        .setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_1) //Deprecated
        .build();

However, in recent versions of httpclient5, setVersionPolicy is deprecated.

Best Practice Way

Now, their documentation says that we should use TlsConfig and connection manager methods instead.

Here's an example:

TlsConfig tlsHttp1Config = TlsConfig.copy(TlsConfig.DEFAULT).setVersionPolicy(HttpVersionPolicy.FORCE_HTTP_1).build();
//Create a default connection manager
PoolingAsyncClientConnectionManager connectionManager = new PoolingAsyncClientConnectionManager();
connectionManager.setDefaultTlsConfig(tlsHttp1Config);

final CloseableHttpAsyncClient client = HttpAsyncClients
        .custom()
        .useSystemProperties()
        .setConnectionManager(connectionManager)
        .build();

I hope these examples help if you ever need to use an explicit version of HTTP for your AsyncHttpClients.

Next.js: Best Way to Organize Your Project Structure

Jonathan U — Tue, 04 Jun 2024 03:16:04 +0000

Whenever I start a new Next.js project, I search for the best way to organize my file structure, especially when I moved to App Router from Pages Router. There are app routing folder and file conventions every Next.js project would need to follow, but apart from that, there are several ways to organize your project files.

Before showing how I typically structure my projects, I would like to briefly discuss the features Next.js provides to help organize your project and a few common strategies.

Safe colocation by default
Private Folders
Router Groups
src Directory
Module Path Aliases

Safe colocation by default

In the app directory, nested folder hierarchy defines the route structure. However, a route is not publicly accessible until a page.js or route.js file is added to a route segment. Even when a route is publicly accessible, the content returned by the page.js or route.js file is sent to the client. This means your project files can be safely colocated inside route segments in the app directory without accidentally being routable.

/app/settings/page.tsx -> Routable
/app/settings/nav.tsx -> Not Routable
/app/settings/constants.ts -> Not Routable
/app/api/monkeys/route.ts -> Routable
/app/api/monkeys/db.ts -> Not Routable

Private Folders

Private folders can be created by prefixing the folder name with an underscore: _folderName.
This hides the folder and its contents from the routing system.

/app/_settings/page.tsx -> Not Routable

Private folders can be useful for separating UI logic from routing logic and avoiding potential naming conflicts with Next.js file conventions.

Router Groups

Route groups can be created by wrapping a folder in parenthesis: (folderName). This indicates the folder is for organizational purposes and should not be included in the route's URL path.
/app/(admin)/dashboard/page.tsx routes to /dashboard

src Directory

Next.js allows storing application code in an optional src directory. This separates application code from project configuration files as they mostly live at the root of the project.
I used to ignore the src directory since it wasn't the default when creating a Next.js project, but I've been using it recently as I like the separation of root configuration files from the application code.

Module Path Aliases

Next.js makes it easy to read and maintain imports across deeply nested project files with Module Path Aliases.

// Before
import { Button } from '../../../../../../components/button'

// After
import { Button } from '@/components/button'

Project Organization Strategies

Next.js is nice enough to provide a few strategies for organizing your project.

Store project files outside of the app
Store project files in top-level folders inside the app
Split project files by feature or route

Store project files outside of the app

Store all application code in shared folders at the root of your project and keep the app directory purely for routing purposes.

my-app/
├─ app/
│  ├─ settings/
│  │  ├─ page.tsx
├─ components/
│  ├─ settings.tsx
├─ utils/
│  ├─ server_actions.ts
├─ package.json
├─ README.md

I like this approach as this gives you more flexibility on how you want to structure your application code as well as decoupling Next.js file conventions.

Store project files in top-level folders inside the app

my-app/
├─ app/
│  ├─ components/
│  │  ├─ settings.tsx
│  ├─ utils/
│  │  ├─ server_actions.ts
│  ├─ settings/
│  │  ├─ page.tsx
├─ package.json
├─ README.md

You'll need to be careful not to add an accidental page.tsx or route.ts to these top-level folders if you don't want to expose these routes publicly.

Split project files by feature or route

Store globally shared application code in the root app directory and split more specific application code into the route segments that use them.

my-app/
├─ app/
│  ├─ components/
│  ├─ utils/
│  ├─ settings/
|  |  ├─ components/
│  |  ├─ utils/
│  │  ├─ page.tsx
├─ package.json
├─ README.md

I've been trying a structure similar to this recently. I find it easy to navigate my code this way. This tightly couples your application code with the routing system. One worry I have is depending on the routing system changes in future updates, there could be potential conflicts when upgrading.

Conclusion

There is no "right" or "wrong" way when it comes to organizing your files and folders in a Next.js project. As long as the structure is consistent across your project, choose a strategy that works for you and your team.

I'm curious to know how you organize your Next.js projects.

Next.js: Simple Example using revalidateTag

Jonathan U — Fri, 31 May 2024 02:52:02 +0000

In Next.js, revalidating data is the process of clearing the Data Cache and retrieving the latest data. This allows you to display the latest information to your users as your data changes.

There are two types of revalidation:

Time-Base
On-demand - revalidatePath and revalidateTag

In this post, we'll focus on On-demand revalidation revalidateTag, which manually revalidates data based on an event such as a form submission. This can only called in a Server Action or Router Handler.

Next.js has a cache tagging system for invalidating fetch requests across routes. In order to use revalidateTag, you'll need to add a tag in the fetch request:

const res = await fetch('https://baseurl.com', { next: { tags: ['mytag'] } });

This adds the cache tag mytag to the fetch request.

Then you can revalidate the fetch call by calling revalidateTag in a Server Action:

'use server'

import { revalidateTag } from 'next/cache'

export default async function action() {
  revalidateTag('mytag')
}

Basic Next.js 14 example with Typescript can be found here: https://github.com/juhlmann75/Next.js-Examples/tree/main/src/app/examples/revalidateTag

More on how On-demand revalidation works.

Next.js: Three Ways to Call Server Actions from Client Components

Jonathan U — Tue, 28 May 2024 02:00:40 +0000

In Next.js, Server Actions are asynchronous functions that execute on the server. They can be used in Client Components to handle form submissions and perform data mutations. More on Server Actions can be read here.

There are a few ways to use these Server Actions in the Client Component.

Form Action
Event Handlers
useEffect

Form Action

React allows Server Actions to be invoked with the action prop. When a user submits a form, the action is invoked, which will receive the FormData object.

Let's say you need a way to track how many bananas each monkey in the zoo has. There's a PostgreSQL table called monkeys. The form may look like this:

<form action={addMonkey}>
  <input type="text" id="name" name="name" required/>
  <input type="number" id="numBananas" name="numBananas" defaultValue={0} required/>
  <input type="submit" defaultValue="Submit"/>
</form>

Notice how the action will invoke a Server Action called addMonkey. This method is defined with the React "use server" directive.

"use server";

export async function addMonkey(formData: FormData): Promise<boolean> {
  const schema = z.object({
    name: z.string().min(1),
    numBananas: z.number().int()
  });
  const monkeyParse = schema.safeParse({
    name: formData.get("name"),
    numBananas: z.coerce.number().parse(formData.get("numBananas"))
  });

  if (!monkeyParse.success) {
    return false;
  }

  const data = monkeyParse.data;

  try {
    await sql`
        INSERT INTO monkeys (name, numBananas)
        VALUES (${data.name}, ${data.numBananas})
    `;

    revalidatePath("examples/server-actions")
    return true;
  } catch (e) {
    return false;
  }

}

Event Handlers

You can call Server Actions from Event Handlers, such as onClick. For example, to increment the number of bananas a monkey has:

"use client";

import {Monkey, updateNumBananas} from "@/app/examples/server-actions/server_actions";

export default function MonkeyList(props: { monkeys: Monkey[] }) {
  return (
    <ul>
      {props.monkeys.map((monkey) => {
        return <li key={monkey.id}>
          <span>{monkey.name} has {monkey.numbananas} banana(s).</span>
          <button onClick={async () => {
            await updateNumBananas(monkey.id, monkey.numbananas + 1)
          }}>
            Increment Bananas
          </button>
        </li>
      })}
    </ul>
  )
}

This adds a button next to each monkey to increment the bananas count by 1. The Server Action looks like this:

"use server";

export async function updateNumBananas(id: number, numBananas: number) {
  try {
    await sql`
        UPDATE monkeys 
        SET numBananas = ${numBananas}
        WHERE id = ${id}
    `;

    revalidatePath("examples/server-actions")
    return true;
  } catch (e) {
    return false;
  }
}

useEffect

Lastly, Server Actions can be called in the React useEffect hook when the component mounts or a dependency changes. For example, to display the number of monkeys from a Server Action:

"use client";

import {getMonkeyCount, Monkey, updateNumBananas} from "@/app/examples/server-actions/server_actions";
import {useEffect, useState} from "react";

export default function MonkeyList(props: { monkeys: Monkey[] }) {
  const [monkeyCount, setMonkeyCount] = useState(0);

  useEffect(() => {
    const updateMonkeyCount = async () => {
      const updatedNumMonkeys = await getMonkeyCount();
      setMonkeyCount(updatedNumMonkeys);
    }
    updateMonkeyCount();
  })

  return (
    <>
      <p>Monkeys: {monkeyCount}</p>
      <ul>
        {props.monkeys.map((monkey) => {
          return <li key={monkey.id}>
            <span>{monkey.name} has {monkey.numbananas} banana(s).</span>
            <button onClick={async () => {
              await updateNumBananas(monkey.id, monkey.numbananas + 1)
            }}>
              Increment Bananas
            </button>
          </li>
        })}
      </ul>
    </>
  )
}

In this example, useEffect is triggered automatically and will update the number of monkeys when a new one is added.
The server action looks like this:

"use server";

export async function getMonkeyCount() {
  const numMonkeys = await sql`
    SELECT *
    FROM monkeys;
  `;

  return numMonkeys.count;
}

There are better and more optimized ways to show the number of monkeys but this is just an example to show how we can use Server Actions in Client Components.

Source code of the complete example: https://github.com/juhlmann75/Next.js-Examples/tree/main/src/app/examples/server-actions

Setup Azure Cosmos DB for Gremlin in Spring Boot Java

Jonathan U — Wed, 01 May 2024 04:45:40 +0000

Azure Cosmos DB is a fully managed NoSQL, relational, and vector database for modern app development. It handles all the database administration with automatic management, updates, and patching. It also handles capacity management with serverless and automatic scaling options to match capacity and demand of your application.

Azure Cosmos DB has multiple database APIs, including NoSQL, MongoDB, PostgreSQL, Cassandra, Gremlin, and Table. This guide will focus on setting up Azure Cosmos DB with the Gremlin API to store and access data from a Spring Boot application in Java.

The API for Gremlin is built based on Apache TinkerPop, a graph computing framework that uses the Gremlin query language.

Prerequisites

An Azure Cosmos DB account
- Gremlin Database and Collection Created
JDK 1.7+, but we'll use JDK 21
Maven

In this example, I have a Cosmos DB Gremlin account with a database called graphdb and a collection called Persons.

Spring Boot Starter

Next step is to create a spring boot project using the Spring Initializr tool with the following options:

Java 21
Maven
Jar Packaging

Add Spring Web as a dependency, then create.

Once the spring boot project has been created, add the gremlin-driver dependency to the pom.xml:

<dependency>
    <groupId>org.apache.tinkerpop</groupId>
    <artifactId>gremlin-driver</artifactId>
    <version>3.7.2</version>
</dependency>

Make sure everything is installed and the build succeeds:

mvn clean install

Navigate to src/main/resources/application.yml. If application.yml doesn't exist, create it.
Add the following:

cosmosdb:
  host: ${GREMLIN_HOST} #cosmos db host (ex: <account-name>.gremlin.cosmos.azure.com)
  key: ${GREMLIN_KEY} #cosmos db key
  port: 443
  database: graphdb #database name
  collection: Persons #collection name

In the directory where your SpringBootApplication class is, create a class called GremlinConfig with the following code:

@Configuration
public class GremlinConfig {

    @Value("${cosmosdb.host}")
    private String gremlinHost;
    @Value("${cosmosdb.key}")
    private String gremlinKey;
    @Value("${cosmosdb.port}")
    private int port;
    @Value("${cosmosdb.database}")
    private String database;
    @Value("${cosmosdb.collection}")
    private String collection;

    @Bean
    public Client gremlinClient() {
        String username = "/dbs/%s/colls/%s".formatted(database, collection);
        Cluster cluster = Cluster.build()
                .addContactPoint(gremlinHost)
                .port(port)
                .credentials(username, gremlinKey)
                .enableSsl(true)
                .serializer(Serializers.GRAPHSON_V2)
                .create();
        return cluster.connect();
    }
}

This creates a gremlin client bean, using the values from application.yml. The client communicates with your gremlin database and can be used throughout your application.

Next, create a class called PersonsController. This will be the REST Controller with a GET endpoint that will retrieve a Vertex given an id.

@RestController
public class PersonsController {

    private final Client client;

    public PersonsController(Client client) {
        this.client = client;
    }

    @GetMapping("/{name}")
    public String getPerson(@PathVariable("name") String name) {
        ResultSet results = client.submit("g.V('%s')".formatted(name));
        Result result = results.one();
        return result.getString();
    }
}

Once you've added this, you can test to see if everything works. After you define the GREMLIN_HOST and GREMLIN_KEY environment variables, run the spring boot application.

Assuming you have a vertex in your collection with an id of "Test", you can request the url: `localhost:8080/Test', and it should return the vertex, along with it's properties if any.

To learn more about Tinkerpop and the Gremlin query language: https://tinkerpop.apache.org/docs/current/tutorials/getting-started/

This Github repository has all code needed to run this example: https://github.com/juhlmann75/cosmosdb-gremlin-spring-boot-starter