You Don't Need Python to Ship AI: Adding an LLM Feature to a Spring Boot API

AI owns DEV's front page this week: AI features running in the browser on one side, AI fatigue on the other. The Java corner of that conversation is still quiet. A few posts cover agent memory with Spring AI or RAG with LangChain4j, but most teams with a Spring Boot backend still assume that "adding AI" means standing up a Python service.

For a lot of features, it doesn't. If what you need is "call a model, get structured data back, use it in my existing API", you can do that in the service you already deploy, monitor and secure.

This post builds one such feature end to end: support ticket triage. A ticket comes in and the API returns a category, a priority, a short summary and tags. The goal isn't a demo. It's the boring parts that make it safe to ship: typed output, a deadline, a budget and a fallback.

What I actually ran: Spring Boot 4.1.1, Spring AI 2.0.1 (spring-ai-starter-model-openai), Java 21, against a local mock OpenAI-compatible server. The mock let me force each failure mode on demand: valid JSON, prose instead of JSON, a hang, a 500. I didn't call a real paid model for this post. The numbers below measure the code's behavior, not model quality.

1. Spring AI or LangChain4j?

Both are mature options for the JVM. A quick way to choose:

• Spring AI if you're already on Spring Boot and want auto-configuration, application.properties settings, Micrometer observations and a fluent ChatClient. It feels like RestClient for models.
• LangChain4j if you're not on Spring (Quarkus, Micronaut, plain Java), or if you want its higher-level "AI services" (interface-based agents) and its integrations catalogue.

Both support OpenAI-compatible APIs, which covers many hosted and local models (including servers like Ollama or vLLM that expose that API). This post uses Spring AI, but sections 3 to 5 apply to either.

2. Setup

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>4.1.1</version>
</parent>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.ai</groupId>
      <artifactId>spring-ai-bom</artifactId>
      <version>2.0.1</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
  </dependency>
</dependencies>

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.base-url=${OPENAI_BASE_URL}
spring.ai.openai.chat.options.model=${LLM_MODEL}
spring.ai.openai.chat.options.temperature=0.0
spring.ai.openai.chat.options.max-tokens=300
spring.ai.openai.chat.max-retries=1

triage.timeout=6s
triage.max-concurrent=8

The API key comes from the environment, never from the repo. max-tokens is your first cost control: a triage answer is small, so cap it. I'll come back to max-retries, because the default surprised me.

3. Ask for a record, not for text

Define exactly what you want back, and keep it small. Every field is a promise the model has to keep.

/** What we ask the model for. */
record LlmTriage(Category category, Priority priority, String summary, List<String> tags) {
    enum Category { BILLING, BUG, ACCOUNT, FEATURE_REQUEST, OTHER }
    enum Priority { LOW, MEDIUM, HIGH }
}

/** What our API returns. "source" is ours, not the model's. */
public record TicketTriage(LlmTriage.Category category, LlmTriage.Priority priority,
                           String summary, List<String> tags, String source) {
    static TicketTriage fromLlm(LlmTriage t) {
        return new TicketTriage(t.category(), t.priority(), t.summary(),
                t.tags() == null ? List.of() : t.tags().stream().limit(5).toList(), "llm");
    }
    static TicketTriage fallback() {
        return new TicketTriage(LlmTriage.Category.OTHER, LlmTriage.Priority.MEDIUM,
                null, List.of(), "fallback");
    }
}

Why two records? In my first version I passed the API record (with source) straight to .entity(...). When I logged the outgoing request, Spring AI had generated a JSON Schema from it and listed source as a required field for the model to fill. The model doesn't know whether it's a fallback; your code does. Keep the model-facing type separate from your API type.

Using enums is the cheapest guardrail available. The generated schema includes "enum": ["BILLING", "BUG", ...], and if the model invents "PAYMENTS", parsing fails instead of letting an unknown category leak into your database.

4. The service: deadline, budget, fallback

@Service
class TriageService {
    private static final Logger log = LoggerFactory.getLogger(TriageService.class);
    private static final String SYSTEM = """
        You triage customer support tickets for a web and mobile app.
        Classify the ticket. Keep the summary under 25 words.
        Never include personal data (names, phone numbers, emails) in the summary or tags.
        """;

    private final ChatClient chat;
    private final Duration timeout;
    private final ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();
    private final Semaphore inFlight;

    TriageService(ChatClient.Builder builder,
                  @Value("${triage.timeout:5s}") Duration timeout,
                  @Value("${triage.max-concurrent:8}") int maxConcurrent) {
        this.chat = builder.defaultSystem(SYSTEM).build();
        this.timeout = timeout;
        this.inFlight = new Semaphore(maxConcurrent);
    }

    TicketTriage triage(String ticketText) {
        if (ticketText == null || ticketText.isBlank() || ticketText.length() > 4_000) {
            throw new IllegalArgumentException("ticket text must be 1..4000 characters");
        }
        if (!inFlight.tryAcquire()) {        // budget: never queue unbounded LLM calls
            log.warn("triage: too many in-flight calls, using fallback");
            return TicketTriage.fallback();
        }
        Future<LlmTriage> call = executor.submit(() -> chat.prompt()
            .user(u -> u.text("Ticket:\n<ticket>\n{ticket}\n</ticket>").param("ticket", ticketText))
            .call()
            .entity(LlmTriage.class));
        try {
            LlmTriage t = call.get(timeout.toMillis(), TimeUnit.MILLISECONDS);
            if (t == null || t.category() == null || t.priority() == null) return TicketTriage.fallback();
            return TicketTriage.fromLlm(t);
        } catch (TimeoutException e) {
            call.cancel(true);
            log.warn("triage: LLM call exceeded {}", timeout);
            return TicketTriage.fallback();
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
            return TicketTriage.fallback();
        } catch (ExecutionException e) {
            log.warn("triage: LLM call failed: {}", e.getCause().toString());
            return TicketTriage.fallback();
        } finally {
            inFlight.release();
        }
    }
}

And a plain controller:

@RestController
@RequestMapping("/api/tickets")
class TriageController {
    private final TriageService service;
    TriageController(TriageService service) { this.service = service; }
    record TriageRequest(String text) {}

    @PostMapping("/triage")
    TicketTriage triage(@RequestBody TriageRequest req) { return service.triage(req.text()); }
}

The design choices:

• Input limits first. A 4,000-character cap protects both your token bill and your latency.
• The ticket is data, not instructions. It goes inside <ticket> tags via a template parameter, and the system prompt states the rules. That doesn't make prompt injection impossible, but combined with enum-only output it limits what an injected instruction can achieve: the worst case is a wrong category, not arbitrary text in your UI.
• A deadline the caller can rely on. The endpoint answers within triage.timeout, whatever the provider does. Virtual threads make it cheap to wrap a blocking call in a Future.
• A concurrency budget. The Semaphore caps simultaneous model calls. When it's full, you degrade instead of queueing, which protects both your rate limit and your threads.
• A fallback that keeps the product working. source: "fallback" tells the UI (and your metrics) that a human should triage this one. The feature degrades gracefully; it doesn't cause an outage.

5. What happened when things went wrong

I pointed the app at the mock server and forced each failure. These are the actual results:

valid JSON         → {"category":"BILLING","priority":"HIGH","summary":"Customer charged twice ...","source":"llm"}   ~0.7 s
prose, not JSON    → source=fallback   ~0.01 s   (StreamReadException: Unrecognized token 'Sure')
model hangs 8 s    → source=fallback   6.0 s     (our deadline fired)
HTTP 500           → source=fallback   ~0.5 s    (2 upstream calls with max-retries=1)

Two findings are worth calling out.

Retries stack. With the defaults, a single failing user request produced four upstream calls to the model endpoint. In Spring AI 2.0, the OpenAI integration is built on the official openai-java SDK, which has its own retry setting (spring.ai.openai.chat.max-retries, default 3 in the configuration metadata). For a user-facing endpoint with a fallback, that's three extra billed attempts the user is waiting for. Setting spring.ai.openai.chat.max-retries=1 brought it down to two. Also note that spring.ai.openai.chat.options.max-retries also exists in the metadata, but it had no effect in my test. The non-options property did. Count upstream calls in your provider's logs before you trust any retry setting.

Keep your own deadline. The metadata also lists spring.ai.openai.chat.timeout (default 60s). In my test, setting it to 4s didn't cut the hanging call before my 6-second application deadline did. I didn't dig further, so treat this as "verify in your setup". Either way, an application-level deadline is something you control and can unit-test.

6. When to stay in Java, and when to add a Python service

Stay in your Spring Boot API when:

• the feature is "call a model, parse structured output, store or return it" (classification, extraction, summarization, drafting replies);
• the model is hosted (a provider API or an OpenAI-compatible server you run separately);
• you want it behind the same auth, rate limits, logging and deployment you already have.

Reach for a separate Python service when:

• you train or fine-tune models, or run data science pipelines (that ecosystem is still Python-first);
• you need in-process model inference with libraries that only exist in Python;
• a team that owns the ML work wants its own deploy cycle.

Even then, the Spring Boot side looks the same as above: a typed call with a deadline and a fallback. Only the URL changes.

Before you ship: a checklist

• [ ] Output is a small record with enums wherever possible; the model-facing type is separate from the API type.
• [ ] Input is length-capped and passed as a template parameter, not concatenated into the instructions.
• [ ] max-tokens is set; temperature is low for classification.
• [ ] An application deadline is shorter than your HTTP client's and your users' patience.
• [ ] Retries are counted end to end (SDK + yours + the client's).
• [ ] Concurrency is capped; overflow degrades to a fallback.
• [ ] The response says whether it came from the model or the fallback, and you graph that ratio.
• [ ] You know what personal data leaves your system in the prompt, and your provider's data-retention terms allow it.

Notes

• Spring AI 2.0 targets Spring Boot 4.x. If you're still on Boot 3.x, use Spring AI 1.x; the ChatClient and .entity(...) API shown here exists there too, but property names may differ.
• .entity(...) also accepts reliability switches (spec -> spec.validateSchema() to retry with error feedback, useProviderStructuredOutput() to send the schema as a provider-level constraint). I didn't test them against the mock, so they aren't in the code above. Read the Structured Output reference before enabling them, and remember that each validation retry is another billed call.
• If you add springdoc to the same project, check for a swagger-annotations version conflict (I had to pin 2.2.55).

Are you adding LLM features in Java, or did your team go the Python-microservice route? I'd like to hear why in the comments.