DEV Community: marouane moutih

Semantic search with OpenAi, SpringBoot, Vaadin, ElasticSearch

marouane moutih — Tue, 04 Jul 2023 21:21:36 +0000

Semantic search

Semantic search is an information search approach that seeks to understand the meaning and intent behind a query rather than relying solely on the specific keywords used. Unlike traditional search based on exact textual matches, semantic search considers the context, meaning and relationship between words to provide more relevant and accurate results.

Semantic research uses language analysis, natural language processing (NLP) and artificial intelligence techniques to understand the meaning of words in a given context. It can take into account elements such as synonym, antonymy, co-occurrence, semantic relationships and grammatical structures to improve the understanding of queries and indexed documents.

One of the key aspects of semantic research is the ability to interpret the intent of the user behind a query. For example, if you enter the query "Best sailing catamarans in Corsica", a semantic search engine will be able to understand that you are looking for sailing catamarans in Corsica, rather than just looking for the words "best", "catamaran", "sailing" and "Corsica" separately.

In short, semantic research goes beyond keyword matching to understand the overall meaning of a query and provide more relevant results based on the semantic understanding of the language.

Give a sense to a request (embedding)

Word embedding is a technique used in the field of automatic natural language processing (TALN) to represent words in the form of numerical vectors in a multidimensional space. These word dips are usually learned from large amounts of text through machine learning models, such as neural networks, which are able to capture the semantic and syntactic relationships between words.

These vector representations of words are used in many TALN tasks, such as machine translation, information retrieval, sentiment analysis and text classification. Word embedding allows machine learning algorithms to better understand and work with textual data, transforming words into a numerical form more suitable for analysis and modeling.

Cosine similarity

In data analysis, cosine similarity is a measure of similarity between two non-zero vectors defined in an inner product space. Cosine similarity is the cosine of the angle between the vectors; that is, it is the dot product of the vectors divided by the product of their lengths. It follows that the cosine similarity does not depend on the magnitudes of the vectors, but only on their angle.

Elasticsearch

Elasticsearch is a distributed, open-source and highly scalable data search and analysis engine. It is designed to process and analyze large amounts of data in real time. Elasticsearch uses a data structure called "inverted index" to enable quick search and retrieval of information.

Elasticsearch’s main goal is to provide fast, flexible and easy to implement data search and analysis. It is particularly suitable for indexing and searching text documents, but can also be used for log analysis, real-time monitoring, content recommendation, report generation, and many other use cases.

In addition to full-text search, Elasticsearch offers advanced features such as data aggregation, geospatial search, similarity search, term suggestion, automatic correction, etc. It also provides RESTful APIs to interact with the system and manage indexes and data.

how does it work

Our example combines the three concepts (Semantic search, embedding, Cosine similarity) to create a more powerful search system based on an elasticsearch engine, using java as a programming language, spring-boot and vaadin as stuck and OpenAi for the generation of embedding.

The Model

Our model will be a simple post with a content title and our embedding vector.
Using spring-boot-starter-data-elasticsearch we must specify to elasticsearch that our embedding field is a dense vector with a dimension of 1536 bytes to be OpenAi compliant on this subject.

@Document(
    indexName="posts"
)
@ToString
public class Post {

    @Id
    private String id;

    @Field(type = FieldType.Text)
    private String title;

    @Field(type = FieldType.Text)
    private String content;

    @Field(type = FieldType.Dense_Vector, dims = 1536, index = true)
    private Vector<Double> embedding;

    public Post(String title, String content) {
        this.title = title;
        this.content = content;
    }
}

The Repository

Our repository will simply extend the ElasticsearchRepository. We will just define a function, which will make a match_all request with a scripted score based on cosine similarity between the embedding vector and the search text vector.

@Repository
public interface PostRepository extends ElasticsearchRepository<Post, String>{

    @Query("{" +
            "    \"script_score\": {" +
            "       \"query\": {\"match_all\": {}}," +
            "       \"script\": {"+
            "           \"source\": \"cosineSimilarity(params.queryVector, 'embedding') +0.1\","+
            "           \"params\": {\"queryVector\": ?0}"+
            "       }"+
            "   }"+
            "}")
    List<Post> findBySimilar( String content);
}

The service

Our service must ensure the generation of embedding for each post during creation and for the text of each search request.

public void save(Post post) {
        Vector<Double> vector = new Vector<Double>(getEmbedding(post.title() + "" + post.content()));
        post.embedding(vector);
        repository.save(post);
}

public List<Post> find(String text) {
        Vector<Double> vector = new Vector<Double>(getEmbedding(text));
        return repository.findBySimilar(JsonData.of(vector).toJson(new JacksonJsonpMapper()).toString());
}

The full exemple

https://github.com/mmoutih/sementic-search

flow me on twitter

Laravel multiPatternRoute

marouane moutih — Wed, 05 May 2021 22:36:45 +0000

when we develop a multilingual web application, sometimes we face a case where the pattern of the route is different between two languages.

how we can solve such a problem?

First, we need to create our standard for route names, in our case, we opt for this scheme: "{name}.{locale}" when {name} is the route name and {locale} is a language or a group of languages.

Route::get('/', App\Http\Controllers\Front\HomeController::class)->name("home.fr");
Route::get('/{locale}', App\Http\Controllers\Front\HomeController::class)->where(['locale' => '[a-z]{2,3}'])
    ->name("home.locale");

Second, we need our own Helper to generate URLs. we will need a class that will parse the list of routes and get the routes linked to the name we provide, After that the class will take the list of parameters that we pass and return just the parameters needed for the current pattern, also it will inject constant parameters if they exist.

class MultiPatternRoute{

    protected string $locale;
    protected $routes;

    public function __construct()
    {
        $this->routes =  Route::getRoutes()->getRoutesByName();
        $this->locale = app()->getLocale();
    }

    public function getParameters (string $routeName,array $parameters) : array
    {
        if(!isset($this->routes[$routeName.".fr"]) || !isset($this->routes[$routeName.".locale"]))
            throw new \Exception("unknown route : ". $routeName.".fr");

        $route_fr = $this->getCompiledRoute($this->routes[$routeName.".fr"]);
        $route_locale = $this->getCompiledRoute($this->routes[$routeName.".locale"]);
        if($this->locale === "fr"){
            $route = $routeName.".fr";
            return [$route,$this->getStaticParameters($route ,$this->cleanParameters($parameters, $route_fr->getVariables(), $route_locale->getVariables()))];
        }
        $route = $routeName.".locale";
        $parameters = $this->getStaticParameters($route, $this->cleanParameters($parameters, $route_locale->getVariables(), $route_fr->getVariables()));
        $parameters["locale"] = app()->getLocale();
        return [$route,$parameters];

    }

    private function getCompiledRoute(RoutingRoute $route): CompiledRoute
    {
        return $route->getCompiled()!==null?$route->getCompiled():$route->toSymfonyRoute()->compile();
    }

    private  function cleanParameters(array $parameters, array $neededParametersA, array $neededParametersB) : array
    {
        $neededParameters = array_intersect(array_keys($parameters), $neededParametersA);
        $queryStringParameters = array_diff(array_keys($parameters), $neededParametersB);
        $parameters_keys = array_unique (array_merge ($neededParameters,$queryStringParameters));
        return array_filter($parameters, fn($k)=>in_array($k,$parameters_keys),ARRAY_FILTER_USE_KEY);
    }

    private function getStaticParameters(string $routeName, array $parameters) : array
    {

        $controller = $this->routes[$routeName]->getController();
        if(!method_exists($controller,'getConstantRouteParameters'))
            return $parameters;
        return array_merge($this->routes[$routeName]->getController()::getConstantRouteParameters($this->locale),$parameters);
    }
}

Third, we provide this class as Singleton via the container in our AppServiceProvider.

$this->app->singleton(
            'MultiPatternRoute',
           function(){
              return new  MultiPatternRoute();
           }
        );

Fourth, we create a helper function that will consume our class and generate our URL.

if ( !function_exists('multiPatternRoute') )
{
    function multiPatternRoute(string $routeName, array $parameters =[], bool $isAbsolute = true){
        list($route, $parameters) = app("MultiPatternRoute")->getParameters($routeName, $parameters);
        return route($route, $parameters, $isAbsolute);
    }
}

Fifth, we prepare our controllers to communicate with our class by providing constant parameters if they exist. we do that thanks to a trait as a result.

trait MultiPatternRoutable
{

    public static function getConstantRouteParameters($locale): array
    {
        return  isset(static::$constantRouteParameters)?static::$constantRouteParameters[$locale]:[];
    }

    public function loadConstantRouteParameters($locale): array
    {
        return array_keys(static::getConstantRouteParameters($locale));
    }

}

then in the controller

class DestinationController extends Controller
{
    use MultiPatternRoutable;

    protected static array $constantRouteParameters =[
        "fr" => [],
        "en" => ["slugSeo"=>"destinations"],
        "de" => ["slugSeo"=>"unsere-reiseziele"],
        "it" => ["slugSeo"=>"destinazioni"],
    ];

    public function __invoke(Request $request, string $slug){
        return view("front.pages.destination", compact("slug"));
    }

}

finally, we add a middleware to clean the request object from the constant or unnecessary parameters for the current route.

class Locale
{
    /**
     * Handle an incoming request.
     *
     * @param  \Illuminate\Http\Request  $request
     * @param  \Closure  $next
     * @return mixed
     */
    public function handle(Request $request, Closure $next)
    {
        $locale = isset($request->route()->parameters['locale'])?$request->route()->parameters['locale']:null;
        if($locale){
            app()->setLocale($locale);
            $request->route()->forgetParameter('locale');
        }
        $otherParameterToForgot =$request->route()->getController()->loadConstantRouteParameters(app()->getLocale());
        foreach($otherParameterToForgot as $parameter){
            $request->route()->forgetParameter($parameter);
        }
        return $next($request);
    }
}

then in our route file


Route::group(["middleware" => ["locale"]], function () {

    Route::get('/', App\Http\Controllers\Front\HomeController::class)->name("home.fr");
    Route::get('/{locale}', App\Http\Controllers\Front\HomeController::class)->where(['locale' => '[a-z]{2,3}'])
    ->name("home.locale");

    Route::get("/{locale}/{slugSeo}", App\Http\Controllers\Front\DestinationsController::class)
        ->where(['locale' => '[a-z]{2,3}', 'slugSeo' => 'tutte-destinazioni|unsere-reiseziele|destinations-all'])
        ->name("destinations.locale");
    Route::get("/destinations/notre-list-destination", App\Http\Controllers\Front\DestinationsController::class)
        ->name("destinations.fr");

    Route::get("/{locale}/{slugSeo}/{slug}", App\Http\Controllers\Front\DestinationController::class)
        ->where(['locale' => '[a-z]{2,3}', 'slugSeo' => 'destinazioni|reiseziele|destinations'])
        ->name("destination.locale");
    Route::get("/destinations-{slug}-produit-{slugAlt}.html", App\Http\Controllers\Front\DestinationController::class)
        ->name("destination.fr");
});

then in blade file

 <a href="{{ multiPatternRoute('destination',["slug"=>"france","slugAlt"=>"france"]) }}">France</a>