DEV Community: Bedram Tamang

Duplicate Rows Detection in MySQL

Bedram Tamang — Sun, 13 Apr 2025 06:20:20 +0000

Duplicate Rows Detection in MySQL

We've been working on a Candidate Management system for JoBins, a platform that allows recruitment agents to Manage their Candidates. Sometimes, the same candidate is submitted by multiple agents, which causes redundancy and affects the overall data quality.

To tackle this, we wanted to build a system that automatically flags duplicate candidates.

For the sake of simplicity in this blog post, we’ll consider a candidate as a duplicate if their first name and last name match with another candidate in the system.

Requirements:

Our database may already contain thousands of candidates, so we need a way to compare each candidate against all others to detect duplicates.
Every time a new candidate is added, the system should check for duplicates among all existing records.
When a candidate's information is updated, we should recheck for duplicates, as the update might introduce or resolve a duplication.

Let's start solving

A good starting point for solving this problem is by using the GROUP BY clause. This allows us to group rows that share the same values in specified columns—in our case, first_name and last_name.

select first_name, last_name 
from candidates
group by first_name, last_name

Now, if you include JSON_ARRAYAGG(id) in your SELECT statement, you'll get a list of all candidate IDs that share the same name.

select first_name, last_name, json_arragg(id) as duplicates
from candidates
group by first_name, last_name

The output of this query would look like:

This data is enough to flag the row as a duplicate, with IDs [1, 2, 3] representing all matching candidates—including itself. While the current row's ID is part of the list, that's not a problem. We can easily filter it out on the frontend when displaying duplicate matches.

If you're curious about how to retrieve and display all related duplicates in the UI, feel free to drop a comment—I'm happy to help!

To support this, we simply add a new JSON column to store all duplicate IDs (including the current one).

ALTER TABLE candidates
    ADD COLUMN duplicates JSON NULL;

Populate duplicates

At the very beginning, we need to populate the duplicates column for each row. To do this, we group candidates by their first_name and last_name, aggregate their IDs using JSON_ARRAYAGG, and then join the result back to the original candidates table to update the duplicates field.

Here’s how the query looks:

update candidates
join (
  select json_arrayagg(id) as ids
  from candidates
  group by first_name, last_name
) as duplicates
on json_contains(duplicates.ids, cast(id as json))
set candidates.duplicates = duplicates.ids

The slightly better query would be:

update candidates
join (
  select json_arrayagg(id) as ids, first_name, last_name
  from candidates
  group by first_name, last_name
) as duplicates
on candidates.first_name = duplicates.first_name,
and candidates.last_name = duplicates.last_name
set candidates.duplicates = duplicates.ids

The second query is better as because it doesn't include expensive json_constains search in each row.

Congratulations!, we populated duplicates for each rows.

Add a row

In addition to a row, we should check if the row is duplicated with existing row or not, if it's duplicated we can simple update those rows which are duplicated with the current row, It can be implement as:

Example: a new row is added with first_name=Bedram and last_name=Tamang

update candidates
join (
  select json_arrayagg(id) as ids, first_name, last_name
  from candidates
+ where first_name = 'Bedram' and last_name = 'Tamang'
  group by first_name, last_name
) as duplicates
on candidates.first_name = duplicates.first_name,
and candidates.last_name = duplicates.last_name
set candidates.duplicates = duplicates.ids

Update a Row

Updating a row is a bit more complex than inserting one, because a row may already be marked as a duplicate of other records. When a candidate is updated, we not only need to re-evaluate the new data for duplicates, but also clean up the old duplicate group it previously belonged to.

For example, suppose two rows were marked as duplicates with first_name = 'Bedram' and last_name = 'Tamang'. Now, if one of them is updated to first_name = 'Adam' and last_name = 'Wathan', we need to:

Recalculate duplicates for the new name.
Remove the updated row from the old duplicate group and update that group accordingly.

So the idea is to search both the old and new groups, and update all affected rows in both.

update candidates
join (
  select json_arrayagg(id) as ids, first_name, last_name
  from candidates
+ where ((first_name = 'Bedram' and last_name = 'Tamang') # Searching for previous group
+ or (first_name = 'Adam' and last_name = 'Wathan')) # Searching for update
  group by first_name, last_name
) as duplicates
on candidates.first_name = duplicates.first_name,
and candidates.last_name = duplicates.last_name
set candidates.duplicates = duplicates.ids

Conclusion

Handling duplicate records is a common challenge in systems that rely on user-submitted data—especially in recruitment platforms like JoBins where multiple agents might submit the same candidate. In this post, we tackled this issue using MySQL by grouping candidates based on their first_name and last_name, and flagging duplicates using a JSON column.

We covered:

How to identify and group duplicate candidates.
How to initially populate duplicates for existing records.
How to handle duplicates when adding a new candidate.
How to manage updates and maintain accurate duplicate groupings.

With this setup, you can easily flag and manage duplicates directly in your database without needing additional backend logic. And since the duplicates field stores all related IDs, it becomes straightforward to visualize relationships and resolve conflicts in the UI.

If you want to extend this further, consider adding fuzzy matching for minor typos (e.g., using SOUNDEX() or Levenshtein distance), or applying more robust deduplication strategies using a scoring system.

Thanks for reading! Feel free to drop questions or suggestions in the comments—always happy to chat tech ✌️

Solving Problems the Right Way: Leveraging Frameworks Over Quick Fixes

Bedram Tamang — Wed, 12 Feb 2025 21:20:21 +0000

We have a feature that allows importing CSV files into a database. The CSV rows can contain date fields as well, so we implemented a date parser before inserting the data into the database. The date parser looked like this:

class DateParser {
  ....

    public function parse(mixed $value)
    {
        try {
            return Carbon::parse($value)->format('Y-m-d');
        } catch (Exception $exception) {
            return $value;
        }
    }
}

I would say that this is very standard date parsing code any Laravel developer would write, as we are very familiar with the Carbon library, which Laravel uses internally for all date and time manipulations. However, we encountered an issue. When a user imported a date field with just the month and day (e.g., 11/11), it was being parsed as 2025-11-11. While this makes sense—since the current year is added when the year is not provided—it was not the expected behavior for our case. Instead, we wanted it to throw an exception and return the original value.

I started looking for solutions on StackOverflow and GitHub issues to see if anyone had already discussed this case, but I couldn’t find any solutions. I considered writing a regex to check if the given date matched certain formats, such as YYYY-MM-DD or YYYY/MM/DD. However, there are so many possible combinations that it felt cumbersome to support all formats. I asked ChatGPT for help with the regex (I admit, I was too lazy to figure it out myself), and it provided this regex:

$pattern = '/\b(?:(?:19|20)\d{2}[-/.](?:0?[1-9]|1[0-2])[-/.](?:0?[1-9]|[12][0-9]|3[01])  # YYYY-MM-DD, YYYY/MM/DD, YYYY.MM.DD
| (?:0?[1-9]|1[0-2])[-/.](?:0?[1-9]|[12][0-9]|3[01])[-/.](?:19|20)\d{2}      # MM-DD-YYYY, MM/DD/YYYY, MM.DD.YYYY
| (?:0?[1-9]|[12][0-9]|3[01])[-/.](?:0?[1-9]|1[0-2])[-/.](?:19|20)\d{2})\b/x';

But I wasn’t sure if it would work, and I didn’t feel it was the right approach. Additionally, the Carbon::parse method also supports date strings like first day of December 2020, which returns 2020-12-01. The new regex solution completely missed this case.

I then decided to check if 11/11 was a valid date according to Laravel's validation. I tested it in a small tinker script:

Validator::make([
  'date' => '11/11'
], [
  'date' => [Rule::date()]
])

It was throwing validation errors, which indicated that Laravel is handling the validation correctly. So I dug deeper into how Laravel validates dates. I discovered that Laravel uses two methods for date parsing and validation: date_parse() and checkdate(). These methods are built-in PHP functions. The date_parse() function returns an array like this:

[
  "year" => false,
  "month" => 11,
  "day" => 11,
  "hour" => false,
  "minute" => false,
  "second" => false,
  "fraction" => false,
  "warning_count" => 0,
  "warnings" => [],
  "error_count" => 0,
  "errors" => [],
  "is_localtime" => false
]

We pass the year, month, and day to the checkdate() method. It returns true if the date is valid, and we create the date and return the valid result. This method also supports string-based dates, as shown above. The final solution we came up with was:

...
public function parse(mixed $value){
    $date = date_parse($value);

    if (checkdate($date['month'], $date['day'], $date['year'])) {
        return Carbon::create($date['year'], $date['month'], $date['day'])->format('Y-m-d');
    }

    return $value;
}

Conclusion

When we encounter problems and are unable to solve them using native Carbon library, many of us instinctively jump to writing regex-based solutions or complex custom logic to fix the issue immediately. However, such solutions often come at a cost—they can be hard to maintain, difficult to extend, and prone to breaking when requirements change.

Instead of taking that approach, we decided to look into the framework itself to understand how Laravel handles date validation. By doing so, we found that Laravel relies on PHP’s built-in date_parse() and checkdate() functions, which provided a more reliable and framework-aligned solution.

This reinforces an essential lesson: before writing custom fixes, it’s always worth exploring how the framework or language solves similar problems. This approach not only saves time but also ensures that our solution is maintainable, scalable, and in line with best practices.

Test Driven Development in Laravel - Folder Structure & Setup (1)

Bedram Tamang — Wed, 22 Jan 2025 21:02:03 +0000

Folder structure

In this article, we will explore how to organize tests effectively, using the example of a feature that allows users to write and publish a blog post. Consider an API endpoint that allows authenticated users to add a new post. There are various way to sturcture your tests, for instance you could create a Feature/PostTest class and write all the test realted to a feature about creating a post in this class.

Benefits

Each feature has its own dedicated test class.
All tests related to a specific feature are grouped within a single class.
Maintains a flat folder structure without nested levels, simplifying organization.
Ideal for small codebases and teams with fewer members.

Drawbacks of this structure:

In larger codebases, it becomes challenging to locate specific test cases related to the code.
All the test cases related to one feature will be in single file
Difiiculties to mange in larger team or larger code base.

I personally prefer this approach when working alone or in a small team. However, in larger codebases, it can be challenging to group and identify tests for each feature effectively, making this structure harder to maintain.

Another approach of organizaing folder strucure is to create a Test class that mirrors the directory of the corrosponding code files but resides within the tests folder, for example, if a controller is located at src/application/controllers/BlogController.php the test for this controller will be written in tests/Feature/application/controllers/BlogControllerTest.php. In this setup, the Feature prefix refers to complete end-to-end tests.

Benefits

Simplifies locating tests for a specific piece of code and vice versa.
Enhances organization, making tests easily discoverable, especially in larger teams.

Test Setup

To create a new Laravel application, use the command laravel new laravel-tdd. During the setup, you'll be prompted to select a testing framework. For simplicity, I chose PHPUnit as the testing framework and sqlite as the database. Laravel includes a basic test setup by default, and you can run your tests using php artisan test. By default, it runs and passes two tests.

Database Setup

Consider a feature, where user should able to write a blog post and publish it. Consider a blog contains only title and body for now for simplicity. To implement this feature we need database table, Model and Controller, so Let's quickly create a database model, migrations, factory and controller with this command php artisan make:model Blog -mfcr. The following files are created below:

app/Http/Controllers/BlogController.php
app/Models/Blog.php
database/migrations/2025_01_22_195944_create_blogs_table.php
database/factories/BlogFactory.php

Let's add title and body into our migration.

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

return new class extends Migration {
    public function up(): void
    {
        Schema::create('blogs', function (Blueprint $table) {
            $table->id();
            $table->string('title');
            $table->longText('body')->nullable();
            $table->timestamps();
        });
    }

    public function down(): void
    {
        Schema::dropIfExists('blogs');
    }
};

we need to add these fields are fillable in our model:

namespace App\Models;

use Illuminate\Database\Eloquent\Factories\HasFactory;
use Illuminate\Database\Eloquent\Model;

class Blog extends Model
{
    /** @use HasFactory<\Database\Factories\BlogFactory> */
    use HasFactory;

    protected $fillable = [
        'title',
        'body'
    ];
}

We added the following route in web.php

use App\Http\Controllers\BlogController;

Route::post('blogs', [BlogController::class, 'store'])->name('blogs.store');

and we also implemented the store method in BlogController as:

class BlogController extends Controller
{
    ...

    public function store(Request $request)
    {
        Blog::query()->create($request->all());

        return redirect()->back()->with('success', 'Blog created successfully');
    }

    ...
}

The code above processes data from a POST request and saves it to the blogs database table. Now, let’s write an integration test to ensure the feature works as expected by verifying:

A user can successfully submit a request.
The submitted data is correctly stored in the database.

Create a test class Tests\Feature\Http\Controllers\BlogControllerTest. A basic test for this feature would look like this:

<?php

namespace Tests\Feature\Http\Controllers;

use Tests\TestCase;

class BlogControllerTest extends TestCase
{
    public function testUserCanCreateABlogPost(): void
    {
        $this->post(route('blog.store'), [
            "title"       => "Blog title",
            "body" => "Blog description",
        ]);

        $this->assertDatabaseHas("blogs", [
            "title"       => "Blog title",
            "body" => "Blog description",
        ]);
    }
}

Illuminate\Database\QueryException: SQLSTATE[HY000]: General error: 1 no such table: blogs (Connection: sqlite, SQL: select exists(select * from "blogs" where ("title" = Blog title and "body" = Blog description)) as "exists")

We got exception as we haven't setup our database yet. The easiest things to do is uncomment these two lines in phpunit.xml file

<env name="DB_CONNECTION" value="sqlite"/>
<env name="DB_DATABASE" value=":memory:"/>

We are telling our PHPUnit to use in memory sqlite as our database. and add use \Illuminate\Foundation\Testing\RefreshDatabase; as trait in our Tests\Feature\Http\Controllers\BlogControllerTest class. Our complete class would look like:

<?php

namespace Tests\Feature\Http\Controllers;

use Tests\TestCase;

class BlogControllerTest extends TestCase
{
    use \Illuminate\Foundation\Testing\RefreshDatabase;

    public function testUserCanCreateABlogPost(): void
    {
        $this->post(route('blogs.store'), [
            "title"       => "Blog title",
            "body"              => "Blog description",
        ]);

        $this->assertDatabaseHas("blogs", [
            "title"       => "Blog title",
            "body"              => "Blog description",
        ]);
    }
}

Congratulations our first test passed.

Conclusion

In this guide, we implemented a feature that allows users to write and publish blog posts. We structured the project step by step, from creating the necessary database table, model, and controller to setting up routes and implementing the logic to handle POST requests.

We also demonstrated how to write an integration test to verify the functionality of the feature, ensuring that:

A user can successfully submit a request.
The submitted data is correctly stored in the database.

Through this process, we encountered and resolved common challenges, such as setting up the database for testing. By using an in-memory SQLite database and the RefreshDatabase trait, we streamlined the test environment to ensure reliable and efficient test execution.

This approach highlights the importance of structured development and testing in Laravel, ensuring that features are both functional and maintainable. By passing our first test, we’ve laid the foundation for building robust, well-tested features in our application.

Fine-Tuning T5-Small Model for a Completely New Language: Limbu

Bedram Tamang — Sat, 21 Dec 2024 00:59:11 +0000

Introduction

Natural Language Processing (NLP) is expanding its reach into underserved languages. In this blog, we’ll explore how to fine-tune the T5-Small model to translate between English and Limbu, a Tibeto-Burman language spoken in Nepal and neighboring regions.

Preparing the Data

We created an English-Limbu translation dataset in JSON format, containing over 1,500 pairs. Below is a sample of the data:

[
   {
        "id": 1,
        "translation": {
            "en": "hi",
            "lim": "ᤜᤠᤤ ॥"
        }
    },
    {
        "id": 2,
        "translation": {
           "en": "Let's eat.",
           "lim": "ᤀᤠᤏᤡ᤹ ᤆᤠᤶ ॥ "
        }
    },
    {
        "id": 3,
        "translation": {
            "en": "We saw it.",
            "lim": "ᤀᤏᤡᤃᤧ ᤁᤴ ᤏᤡᤔᤠᤏᤠ ॥ "
        }
    },
    ...
]

The dataset was saved as limbu-english.json.

Setting Up the Environment

Install the required libraries in Google Colab:

!pip install transformers datasets evaluate sacrebleu
!pip install transformers[sentencepiece]
!pip install sentencepiece

Load the dataset:

from datasets import load_dataset

path = 'limbu-english.json'
translations = load_dataset('json', data_files=path)
translations = translations["train"].train_test_split(test_size=0.2)

Loading the Pretrained Model

We initialized the T5-Small model:

from transformers import AutoTokenizer, TFAutoModelForSeq2SeqLM

checkpoint = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = TFAutoModelForSeq2SeqLM.from_pretrained(checkpoint)

Tokenizing the Dataset

We generated a custom tokenizer and tokenized the dataset:

def get_training_corpus():
    dataset = translations["train"]
    for start_idx in range(0, len(dataset), 1000):
        yield [item['lim'] for item in dataset[start_idx:start_idx + 1000]['translation']]

lim_tokenizer = tokenizer.train_new_from_iterator(get_training_corpus(), 52000)

source_lang = "en"
target_lang = "lim"
prefix = "translate English to Limbu: "

def preprocess_function(examples):
    inputs = [prefix + example[source_lang] for example in examples["translation"]]
    targets = [example[target_lang] for example in examples["translation"]]
    return lim_tokenizer(inputs, text_target=targets, max_length=128, truncation=True)

tokenized_translations = translations.map(preprocess_function, batched=True)

Preparing for Training

The tokenized data was prepared for the TensorFlow model:

from transformers import DataCollatorForSeq2Seq

data_collator = DataCollatorForSeq2Seq(tokenizer=lim_tokenizer, model=checkpoint)

tf_train_set = model.prepare_tf_dataset(
    tokenized_translations["train"],
    shuffle=True,
    batch_size=16,
    collate_fn=data_collator,
)

tf_test_set = model.prepare_tf_dataset(
    tokenized_translations["test"],
    shuffle=False,
    batch_size=16,
    collate_fn=data_collator,
)

Training the Model

We used AdamWeightDecay for optimization:

from transformers import AdamWeightDecay
import tensorflow as tf

optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)
model.compile(optimizer=optimizer)

Let's define the metrics to observe while training

from transformers.keras_callbacks import KerasMetricCallback

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = lim_tokenizer.batch_decode(preds, skip_special_tokens=True)

    labels = np.where(labels != -100, labels, lim_tokenizer.pad_token_id)
    decoded_labels = lim_tokenizer.batch_decode(labels, skip_special_tokens=True)

    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels)
    result = {"bleu": result["score"]}

    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    result = {k: round(v, 4) for k, v in result.items()}
    return result

metric_callback = KerasMetricCallback(metric_fn=compute_metrics, eval_dataset=tf_test_set)

These metrics can be seen in log file as well, but instead we will store it into the huggingface

from huggingface_hub import notebook_login

notebook_login()

and push the logs into huggingface as

from transformers.keras_callbacks import PushToHubCallback
push_to_hub_callback = PushToHubCallback(output_dir="eng-limbu-t5-001", tokenizer=lim_tokenizer)

callbacks = [push_to_hub_callback, tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=10)]
history = model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=500, callbacks=callbacks)

Visualizing Training Progress

We visualized the training loss:

import matplotlib.pyplot as plt

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()

Testing the Model

We tested the model using the pipeline module:

from transformers import pipeline

translator = pipeline("text2text-generation", model="bedus-creation/eng-limbu-t5-001")
result = translator("translate English to Limbu: Hello")
print(result)

Evaluating with BLEU Score

Finally, we calculated the BLEU score for translation accuracy:

bleu = evaluate.load("bleu")

predictions = [
    "Hi",
    ]
references = [
    ["ᤜᤠᤤ ॥"],
]

results = bleu.compute(predictions=predictions, references=references)

print(results)

Conclusion

Fine-tuning the T5-Small model for Limbu demonstrates the potential of NLP in preserving and advancing underrepresented languages. With more training data and optimization, such models can become invaluable tools for language preservation and cross-cultural communication.

Fine-Tuning ChatGPT for a Completely New Language: Limbu

Bedram Tamang — Tue, 08 Oct 2024 20:49:42 +0000

Limbu is a language spoken by the Limbu ethnic group native to Eastern Nepal and some parts of India. It has its own script, called the Limbu script or Kirat Sirijunga, which is used to write the language. The language has a rich cultural and historical significance for the Limbu community and is still spoken by a significant number of people in the region. Despite being spoken by a significant number of individuals, there is a notable scarcity of digital traces for these languages. They are predominantly transmitted orally, although there has been a recent emergence of primary schools in eastern Nepal and certain regions of India incorporating them into their curriculum.

In the evolving world of machine learning, fine-tuning pre-trained models can significantly enhance their performance on specific tasks. In this blog, I’ll walk you through the process of fine-tuning OpenAI’s GPT-4 model to build an English-to-Limbu language translator. By leveraging a dataset and OpenAI’s API, we create a custom translator that can bridge the language gap.

Why Fine-Tune Models?

Fine-tuning involves taking a general-purpose pre-trained model and refining it on a specific task, such as translating between languages. In this case, we used OpenAI’s GPT-4o-mini model and trained it using a dataset consisting of English and Limbu sentences. The primary advantage is that fine-tuned models can significantly outperform generic models when working with niche datasets.

Steps Involved

1. Preparing the Dataset

The first step was preparing a dataset consisting of more than 1000 pairs of English-Limbu translations, created with the help of a linguist. This dataset contains English phrases and Limbu script translations, which provides a comprehensive resource for fine-tuning.

The dataset was structured in a CSV file with the following format:

English	Limbu
Hi.	ᤜᤠᤤ ॥
Run!	ᤗᤠ᤺ᤣᤰᤋᤧ᤹ ᥄
Who?	ᤜᤠ᤺᤺ᤳ ᥅
Wow!	ᤀᤠᤠᤶᤔᤤ ᥄
Fire!	ᤔᤡ ᥄

The first step was loading the dataset from a CSV file containing English phrases and their corresponding translations in Limbu.

import pandas as pd
df = pd.read_csv('001-english-limbu-manual.csv')

We used Pandas to load and handle the CSV file, making it easier to manipulate and filter the data as required.

2. Structuring the Training Data

To make the data compatible with OpenAI's fine-tuning API, we structured it into conversations. Each conversation contains a system message, user input, and the corresponding assistant response.

system = "You are a english to limbu or limbu to english language translator."

def prepare_example_conversation(row):
    return {
        "messages": [
            {"role": "system", "content": system},
            {"role": "user", "content": row['English']},
            {"role": "assistant", "content": row["Limbu"]},
        ]
    }

training_df = df.loc[0:1000]
training_data = training_df.apply(prepare_example_conversation, axis=1).tolist()

The dataset was divided into training and validation sets to ensure the model’s generalization ability.

3. Creating JSONL Files

The OpenAI fine-tuning API requires the data to be in JSONL format (JSON Lines), where each line represents a single data entry. We wrote two files: one for training and another for validation.

def write_jsonl(data_list: list, filename: str) -> None:
    with open(filename, "w") as out:
        for ddict in data_list:
            jout = json.dumps(ddict) + "\n"
            out.write(jout)

write_jsonl(training_data, "english_limbu_train.jsonl")
write_jsonl(validation_data, "english_limbu_valite.jsonl")

4. Uploading Data for Fine-Tuning

Once the files were ready, we uploaded them to OpenAI using their client.files.create method. The purpose parameter is set to "fine-tune" to specify the use case.

training_file_id = upload_file("english_limbu_train.jsonl", "fine-tune")
validation_file_id = upload_file("english_limbu_valite.jsonl", "fine-tune")

5. Fine-Tuning the Model

With the files uploaded, we kicked off the fine-tuning process by specifying the model to fine-tune (in our case, "gpt-4o-mini-2024-07-18"), the training and validation files, and a custom suffix.


response = client.fine_tuning.jobs.create(
    training_file=training_file_id,
    validation_file=validation_file_id,
    model=MODEL,
    suffix="limbu-trans",
)

Once the fine-tuning job starts, we periodically check its status and monitor the events using:

response = client.fine_tuning.jobs.retrieve(job_id)
print("Status:", response.status)

The job completes when the status is "succeeded".

6. Testing the Fine-Tuned Model

After successful fine-tuning, we retrieve the fine-tuned model ID. This model can now be used to translate between English and Limbu. For instance:

test_messages = [{"role": "system", "content": system},
                 {"role": "user", "content": "translate: hello my friend"}]

response = client.chat.completions.create(
    model=fine_tuned_model_id, messages=test_messages, temperature=0, max_tokens=500
)
print(response.choices[0].message.content)

In this example, the model generates the Limbu translation for "hello my friend."

Conclusion

By fine-tuning a GPT-4 model using a small, custom dataset, we created a functional English-Limbu language translator. This process is applicable not just for language translation, but for any task-specific model training. The fine-tuning API simplifies the workflow, making it accessible even to those without deep expertise in machine learning.

This translator is just one example of how AI can help preserve and promote lesser-known languages.

Sharing Authentication cookies across Laravel Apps

Bedram Tamang — Thu, 19 Sep 2024 20:58:28 +0000

We have been working on a Laravel application for more than 7, 8 years, with increasing numbers of features, the application has become very large with more than 500k lines of PHP code, which has around 300k lines of Vue code as well. With increasing number of complexity and difficulties in upgrading the project, we decided to write all new APIs on the new Laravel (11x) project. Our main application already serving the vue frontend with Laravel's old style, where you define routes in Laravel and inject Vue components in blade templates. So our challenge was to authenticate our new api project with the same login credentials.

There are various ways to architect the communication between frontend to the backend, Out of many, API gateway and Direct client to microservice communication are two of them. API gateway provides a single entry point for a group of microservices, and enables us to write common features such as authentication, rate-limiting by acting as reverse proxy. In the direct client-to microservice communication approach, each microservice exposes an public endpoint, allows clients to connect with microservices directly. In this approach, each microservice is responsible for their authentication, rate-limiting, and all other services on their own.

Cookie Sharing

Consider you are building an e-commerce platform that has multiple microservices providing APIs for the front-end, and each of these microservices is independent of each others and offers direct client-to-service communications. There could be a service to register and login a user, let's say user service, so whenever a user logs in with this service this service will return some credentials (token or cookie), and that credential should be valid across all other services.

Important Consideration

There are some important things to consider before implementing cookie-sharing

Cookie can be shared only in subdomains, for example, if the core application is running on jobins.jp, the cookie can be shared across the subdomains, let's say api.jobins.jp, but can't shared in jobins.com
The session driver shouldn't be file. Here we are using the same database so both the application has access to same users table.
The cookie name should be the same across all the services.
The APP_KEY should be the same across all the services

Setup

To demonstrate the example of cookie sharing, Let's start by creating two different Laravel (api & core) applications inside a folder say 'cookie-sharing'.

mkdir cookie_sharing
cd cookie_sharing
laravel new core
laravel new api

Further, I used laravel valet to configure the api and core sites to served from .test domain in my local machine. So, api project can be accessed from api.laravel.test and core project can be accessed from laravel.test.

I have installed Laravel breeze in laravel.test project to quicky setup registration and login feature.

Share Encryption Key

The first thing we require here is make the encryption key APP_KEY same in both of our project.

Sanctum Setup

We are using Laravel's Sanctum to authenticate our frontend served from our core application. So Sanctum can be easily installed on API project as:

php artisan install:api

To authenticate the SPA, we need to further our sanctum middleware, cors and Cookie, at first the middleware can be configured as:

->withMiddleware(function (Middleware $middleware) {
    $middleware->statefulApi();
})

and then the cors config can be published as

php artisan config:publish cors

Cookie Domain

As you can see in the application tab of browser, the cookie name is laravel_session and domain is api.laravel.test for api site. We want this cookie to shared with the laravel.test site as well. So to share the cookie, we change the domain in config/sessions.php or simply change SESSION_DOMAIN in .env file to .laravel.test

Cookie Conflict

Since we are sharing same cookie into two Laravel sites, and Laravel sends the cookie headers in it's each response, and the browsers picks up the cookie that comes first into the response headers, Our main idea here would be to use cookie from the laravel.test site, as the cookie from api.laravel.test could be invalid and logout the users from application

class StartSession extends \Illuminate\Session\Middleware\StartSession
{
    protected function addCookieToResponse(Response $response, Session $session):void
    {
        parent::addCookieToResponse($response, $session);

        if ($this->sessionIsPersistent($config = $this->manager->getSessionConfig()){
            $response->headers->setCookie(new Cookie(
                $config['api']['session_cookie'],
                $session->getId(),
                $this->getCookieExpirationDate(),
                $config['path'],
                $config['api']['session_domain'],
                $config['secure'] ?? false,
                $config['http_only'] ?? true,
                false, $config['same_site'] ?? null
            ));
        }
    }
}

Using Different Cookie in `api.laravel.test`

In the API site, we'll use the api_laravel_session cookie from the API routes. By default, Laravel uses the cookie name defined in the config/sessions.php file. To ensure the correct cookie is used for authentication, we'll swap the api_laravel_session cookie with laravel_session, allowing Laravel to retrieve the correct session value. Which we can retrieve using the middleware below.

class UseWebCookie
{
    public function handle(Request $request, Closure $next)
    {
        $cookie = $request->cookies->get(config('session.cookie_web'));
        $request->cookies->set(config('session.cookie'), $cookie);

        return $next($request);
    }
}

And also, we are not returning the cookie in respond headers unless it's explicitly asked by users in request.

$response->headers->removeCookie(config('session.cookie'));

Conclusion:

In conclusion, sharing authentication cookies across multiple Laravel applications can be effectively achieved by ensuring the apps share a common encryption key, domain, and session configuration. By using Laravel Sanctum for SPA authentication and properly managing cookie conflicts with separate session cookies for web and API routes, you can maintain consistent user sessions across different services. This approach provides a scalable solution for managing authentication in a microservices architecture while preventing potential cookie overwrites and session issues.

References:

Implementing boolean OR Search in meilisearch

Bedram Tamang — Wed, 04 Sep 2024 07:31:57 +0000

Meilisearch is a lightning-fast search engine implemented in rust, It's starightforward setup & easy APIs makes it very attractive among developers. Additionally, first-party support by Laravel with it's package scout helps it to span across Laravel community. In this article, I will show you how OR search is implemented in meilisearch.

Problem Statement:

We would like to perform a full-text search among selected attributes with OR conditions, for example: a search of "PHP" or "Engineer" should return all the documents that contain either "PHP" or "Engineer" or both "PHP" & "Engineer". This is a fundamental idea of the OR condition we studied in Boolean algebra. However, there is no direct solution for this problem in [Meilisearch][1]. The closest possible you can reach is to use "last" or "frequency" for MatchingStrategy option, but it's not exactly the OR, instead, these algorithms remove the one word (last or more frequent) at a time and perform search if there are not enough results.

Solution

The basic idea to solve this problem is to use the A U B formula of Math's SET formula, The union formula can be written as:

n(A∪B)=n(A)+n(B)−n(A∩B)

Where,

n(A U B) represents A OR B, that means it should return all the documents containing terms either A or either B, or both A and B
n(A) is the number of documents that contains the term A,
n(B) is the number of documents that contains the term B
n(A∩B) is the number of documents that contains both term A and B

By using above formula, the search for terms "PHP" OR "Engineer" would look like,

n(PHP ∪ Engineer) = n(PHP) + n(Engineer) - n(PHP ∩ Engineer)

Similarly, the formula for the three search terms would look like,

n(A∪B∪C)=n(A)+n(B)+n(C)−n(A∩B)−n(B∩C)−n(A∩C)+n(A∩B∩C)

Since n(A) doesn't only represent the cardinality of documents containing terms A, but also the documents that contain both A and other sets. For the simplicity of calculation, we would like to use the formula that deals with explicitly only A, or only that part. For example,

n(A∩B) = n0(A∩B) + n(A∩B∩C)

Here, n0(A∩B) is the set that only contains the intersection between A and B.

Which can be further written as,

n(A∪B∪C)=n0(A)+n0(B)+n0(C)+n0(A∩B)+n0(B∩C)+n0(A∩C)+n(A∩B∩C)

Where,

n0(A) represents containing only A, excluding B and C entirely. In search, n0(PHP) refers to documents that include PHP but exclude any mention of Engineer or Manager.
In Meilisearch, exclusion can be indicated using the - sign, such as 'PHP' -'Engineer' -'Manager'.

So, our search becomes:

n0(A) = PHP -Engineer -Manager
n0(B) = Engineer -PHP -Manager
n0(C) = Manager -Engineer -PHP
n0(A∩B) = PHP Engineer -Manager
n0(B∩C) = Engineer Manager -PHP
n0(A∩C) = PHP Manager -Engineer
n(A∩B∩C) = PHP Engineer Manager

This means we need to make 7 API calls for 3 search terms. As the number of search terms increases, the number of API calls required also increases, following the formula 2^n−1. So, the number of API calls needed is as follows:

Number of Search Terms (n)	Number of API Calls
1	1
2	3
3	7
4	15
5	31

To search using OR conditions with 5 terms, we would need to make 31 API calls. Since there’s no alternative to implement this feature directly, we could limit the number of search terms to 4 or 5. However, there’s a feature called multi-search that allows us to perform multiple searches simultaneously, enhancing performance by executing all the API calls at once.

We can generate these terms using a recursive approach:

function generateTerm(array $ans, $data, array $current)
{
    $last = end($current);

    if ($last == count($data) - 1) {
        return $ans;
    }

    $nextCurrent   = $current;
    $nextCurrent[] = $last + 1;

    array_pop($current);
    $current[] = $last + 1;

    $ans[formatTerm($data, $nextCurrent)] = count($nextCurrent);
    $ans[formatTerm($data, $current)]     = count($current);

    $ans = generateTerm($ans, $data, $nextCurrent);
    $ans = generateTerm($ans, $data, $current);

    return $ans;
}

The algorithm above can be a bit tricky to understand. If you’d like a detailed explanation, let me know. This method can be called as follows:

$data = ['php', 'engineer', 'manager'];
$ans[formatTerm($data, [0])] = 1;

$ans = generateTerm($ans, $data, [0]);
arsort($ans);

The formatTerm function is used to format the search terms specifically for Meilisearch. The output of the above algorithm would look something like this, depending on the input data:

We have prepared all the search components necessary to perform OR operations. In this structure, the key represents the search terms, while the value represents the score. The score indicates the number of matching terms in each document, with the idea being that documents with a higher number of matching terms will appear first in the search results.

To count the total documents for each search terms

$searches = [];
foreach ($terms as $term => $index) {
    $searches[] = (new SearchQuery())
        ->setQuery($term)
        ->setAttributesToRetrieve(['id'])
        ->setIndexUid('jobins_data')
        ->setPage(1)
        ->setHitsPerPage(1)
        ->setMatchingStrategy('all');
}

$client   = new \Meilisearch\Client('http://localhost:7700', 'masterKey');
$response = $client->multiSearch($searches);

To find the what are terms we should query for a particular page

$page     = 1;
$per_page = 10;
$offset   = ($page - 1) * $per_page;

$possibleSearchTerms = [];
$count               = 0;
$termKeys            = array_keys($terms);
$next                = $offset + $per_page;
$hasOffset           = false;
while (count($termKeys) > 0) {
    $term            = array_pop($termKeys);
    $neededForOffset = $offset - $count;
    $neededForLimit  = $offset + $per_page - $count;
    $count           = $count + $metadata[$term];

    if ($count > $offset && $count < $offset + $per_page) {
        $possibleSearchTerms[] = [
            'q'      => $term,
            'offset' => $hasOffset ? 0 : $neededForOffset,
        ];
        $hasOffset             = true;
    }

    if ($count >= $offset + $per_page) {
        $possibleSearchTerms[] = [
            'offset' => $hasOffset ? 0 : $neededForOffset,
            'q'      => $term,
            'limit'  => min($neededForLimit, $per_page),
        ];
        break;
    }
}

The $possibleSearchTerms gives the possible search terms to query for the results. Finally performing search to the possible search terms gives the necessary results.

$searches = [];
foreach ($possibleSearchTerms as $term) {
    $searches[] = (new SearchQuery())
        ->setQuery($term['q'])
        ->setAttributesToRetrieve(['id'])
        ->setIndexUid('jobins_data')
        ->setOffset($term['offset'] ?? 0)
        ->setLimit($term['limit'] ?? $per_page)
        ->setSort(['id:desc'])
        ->setMatchingStrategy('all');
}

$response        = $client->multiSearch($searches);
$results['hits'] = collect($response['results'])->pluck('hits')->flatten(1);

The complete example in Tinker is given below,

const BASE_PATH = '/Users/ellite/code/office/api.jobins.test';

require_once BASE_PATH.'/vendor/autoload.php';
use Illuminate\Foundation\Console\Kernel;
use Meilisearch\Contracts\SearchQuery;

$app = require BASE_PATH.'/bootstrap/app.php';
$app->make(Kernel::class)->bootstrap();

config()->set('scout.queue', false);
config()->set('scout.after_commit', false);

class Post extends \Illuminate\Database\Eloquent\Model
{
    use \Laravel\Scout\Searchable;

    protected $table = 'posts';

    protected $fillable = [
        'title',
        'body',
    ];

    public function searchableAs(): string
    {
        return 'posts';
    }

    public function toSearchableArray()
    {
        return $this->toArray();
    }
}

config()->set('scout.meilisearch.index-settings', [
    Post::class => [

    ],
]);

\Illuminate\Support\Facades\Artisan::call('scout:sync-index-settings');

\Illuminate\Support\Facades\Schema::dropIfExists('posts');
\Illuminate\Support\Facades\Schema::create('posts', function (\Illuminate\Database\Schema\Blueprint $table) {
    $table->id();
    $table->string('title');
    $table->text('body')->nullable();
    $table->timestamps();
});

$data = [
    ['title' => 'PHP'],
    ['title' => 'Engineer'],
    ['title' => 'Manager'],
    ['title' => 'PHP Engineer'],
    ['title' => 'PHP Manager'],
    ['title' => 'Engineer Manager'],
    ['title' => 'PHP Engineer Manager'],

    ['title' => 'Engineering'],
];

foreach ($data as $row) {
    $post = Post::query()->create($row);
}

function formatTerm($data, array $current): string
{
    $data    = array_map(fn($item) => "'$item'", $data);
    $current = array_map(fn($item) => $data[$item], $current);

    $exclude = array_diff($data, $current);

    $current = implode(' ', $current);
    $exclude = implode(' -', $exclude);

    return trim(implode(' -', [$current, $exclude]), ' -');
}

function generateTerm(array $ans, $data, array $current)
{
    $last = end($current);

    if ($last == count($data) - 1) {
        return $ans;
    }

    $nextCurrent   = $current;
    $nextCurrent[] = $last + 1;

    array_pop($current);
    $current[] = $last + 1;

    $ans[formatTerm($data, $nextCurrent)] = count($nextCurrent);
    $ans[formatTerm($data, $current)]     = count($current);

    $ans = generateTerm($ans, $data, $nextCurrent);
    $ans = generateTerm($ans, $data, $current);

    return $ans;
}

$data = ['PHP', 'Engineer', 'manager'];

$ans[formatTerm($data, [0])] = 1;

$terms = generateTerm($ans, $data, [0]);
arsort($terms);


$searches = [];
foreach ($terms as $term => $index) {
    $searches[] = (new SearchQuery())
        ->setQuery($term)
        ->setAttributesToRetrieve(['id'])
        ->setIndexUid('posts')
        ->setPage(1)
        ->setHitsPerPage(1)
        ->setMatchingStrategy('all');
}

$client   = new \Meilisearch\Client('http://localhost:7700', 'masterKey');
$response = $client->multiSearch($searches);
$metadata = collect($response['results'])->pluck('totalHits', 'query');


$page     = 1;
$per_page = 10;
$offset   = ($page - 1) * $per_page;

$count               = 0;
$possibleSearchTerms = [];
$termKeys            = array_keys($terms);
$next                = $offset + $per_page;
$hasOffset           = false;
while (count($termKeys) > 0) {
    $term            = array_pop($termKeys);
    $neededForOffset = $offset - $count;
    $neededForLimit  = $offset + $per_page - $count;
    $count           = $count + $metadata[$term];

    if ($count > $offset && $count < $offset + $per_page) {
        $possibleSearchTerms[] = [
            'q'      => $term,
            'offset' => $hasOffset ? 0 : $neededForOffset,
        ];
        $hasOffset             = true;
    }

    if ($count >= $offset + $per_page) {
        $possibleSearchTerms[] = [
            'offset' => $hasOffset ? 0 : $neededForOffset,
            'q'      => $term,
            'limit'  => min($neededForLimit, $per_page),
        ];
        break;
    }
}

$searches = [];
foreach ($possibleSearchTerms as $term) {
    $searches[] = (new SearchQuery())
        ->setQuery($term['q'])
        ->setAttributesToRetrieve(['id'])
        ->setIndexUid('jobins_data')
        ->setOffset($term['offset'] ?? 0)
        ->setLimit($term['limit'] ?? $per_page)
        ->setSort(['id:desc'])
        ->setMatchingStrategy('all');
}

$response        = $client->multiSearch($searches);
$results['hits'] = collect($response['results'])->pluck('hits')->flatten(1);

dd($results['hits']);

Conclusion

In conclusion, implementing OR search in Meilisearch requires a strategic workaround due to the absence of a native OR operation. By leveraging the principles of set theory, we can craft a solution that involves performing multiple search queries and combining the results. This approach, while computationally intensive with the number of API calls increasing exponentially with the number of search terms, ensures that we accurately retrieve documents that match any combination of the specified terms. The use of multi-search functionality in Meilisearch helps mitigate some of the performance concerns, allowing for simultaneous execution of queries. This method provides a practical solution for complex search scenarios where OR conditions are essential.

References

[1]: https://github.com/orgs/meilisearch/discussions/626 "Boolean search? AND, OR"
[2]: https://laravel.com/docs/11.x/scout
[3]: https://www.meilisearch.com/docs/reference/api/search#matching-strategy

Eloquent Trick: Laravel Model from Subquery

Bedram Tamang — Fri, 16 Aug 2024 20:35:34 +0000

In Laravel, it's common to define a model's corresponding table using the table property, such as:

class User extends Model {
  protected $table = 'users';
}

However, instead of mapping directly to a table, we can use a subquery. This technique allows us to encapsulate complex queries within the application layer, much like creating a database view, but with the added flexibility of Eloquent operations.

For instance, consider a users table that stores information about both admins and regular users. We can create an AdminUser model using a subquery:

class AdminUser
{
    public function getTable(): string|\Illuminate\Contracts\Database\Query\Expression
    {
        $sql = User::query()
            ->select('id', 'name')
            ->where('admin_user', true)
            ->toRawSql();

        return DB::raw(sprintf('(%s) as admin_users', $sql));
    } 
}

This model pulls data from a subquery (select id, name from users where admin_user = 1), allowing you to query it just like a standard model:

AdminUser::query()->get();

AdminUser::query()->first();

However, be aware that certain queries, like find(1), won't work directly:

AdminUser::query()->find(1);

To work around this, you can use a where condition combined with first():

AdminUser::query()->where('id', 1)->first();

Conclusion

The example provided is a straightforward illustration of how we can structure logic within a model. However, this approach can be scaled and adapted to accommodate much more complex scenarios. By leveraging such structures, we can efficiently manage and extract intricate logic, ensuring that our models remain organized, maintainable, and scalable as they grow in complexity.

Simplifying Form Handling in Vue Applications with Form JS - Inspired by Inertia JS

Bedram Tamang — Tue, 16 Jul 2024 19:35:34 +0000

Recently I have been working on FormJs, which is a form helper and wrapper around Axios, inspired by inertiaJs. The purpose behind writing this new library was to streamline the process of how we handle the form on the front-end side. Validating forms is a crucial aspect of the form-handling process, and a lot of us who work in front-end development, particularly with Vue, have relied on vee-validate for this purpose. However, vee-validate has undergone significant changes and evolved in a different direction.

For example:

To validate a text field and display an error message using vee-validate version 2, we would typically write the following code:

<input type="text" name="field" v-validate="'required'">
<span>{{ errors.first('field') }}</span>

But in version 3, the whole API has been changed, and the same logic should be written as,

<ValidationProvider name="field" rules="required" v-slot="{ errors }">
  <input type="text" v-model="value">
  <span>{{ errors[0] }}</span>
</ValidationProvider>

As, the composition-API has been backported to vue-2, but the vee-validation version 2 has very poor support over it.

The suggestion might be to use the latest version of vee-validate, but many of us might have been stuck in version 2, as they started using it early in the project.

Please see the below link to the migration guide from version 2 to version 3. Migrate vee-validate from 2x to 3.x

Formjs

Let's delve into this further by looking at the installation process.

yarn add formjs-vue2

npm install formjs-vue2

After installing it, Let's create a user registration form, containing the user's name, email, and password. For simplicity, I have created a simple layout using TailwindCSS,

<template>
    <div class="container mx-auto">
        <input type="text" class="block border">
        <input type="text" class="block border">
        <input type="text" class="block border">
        <input type="text" class="block border">
        <button type="submit" class="block border">Submit</button>
    </div>
</template>

Now let's import the useForm composable from formjs-vue2 and define our fields in the script section

<script setup>
import {useForm} from "formjs-vue2";

const form = useForm({
    name: '',
    email: '',
    password: '',
    password_confirm: ''
})
</script>

and let's bind the form for the inputs as

<template>
    <div class="container mx-auto">
        <input type="text" v-model='form.name' class="block border">
        <input type="text" v-model='form.email' class="block border">
        <input type="text" v-model='form.password' class="block border">
        <input type="text" v-model='form.password_confirm' class="block border">
        <button type="submit" class="block border">Submit</button>
    </div>
</template>

Front end Validation

Formjs comes with built-in support for yup, a schema-based validation library. Let's proceed with the installation of yup by running the following command:

yarn add yup

and define a form schema as,

<script setup>
import { object, string } from 'yup'

const formSchema = object({
    name: string().required(),
    email: string().email(),
    password: string().required(),
    password_confirm: string().required()
})
</script>

Here is the basic validation rule that has been defined for form, which we will further assign the schema in,

<script setup>
...
const form = useForm({
    name: '',
    email: '',
    password: '',
    password_confirm: ''
}, { schema: formSchema })
</script>

Validate Form

Validation of the form can be accomplished using the validate() method, which validates all fields and attaches the respective errors to each field. For instance, to access the error for the email field, we would use form.errors.email, and so on. Additionally, individual fields can be validated by passing them into the validate('email') method. In the example below, we are validating the name input as it changes:

<input type="text" v-model='form.name' @input="validate('email')" class="block border">
<span v-if="form.errors.name" class='text-red-500' v-text='form.errors.name'/>

The complete example would look like this,

<template>
    <div class="container mx-auto">
        <form @submit.prevent.stop="submit">
            <input
                type="text"
                class="block border border-black"
                v-model="form.name"
                @input="form.validate('name')">
            <span 
                v-if="form.errors.name" 
                class='text-red-500' 
                v-text='form.errors.name'/>

            <input
                type="text"
                class="block border border-black"
                v-model="form.email"
                @input="form.validate('email')">
            <span 
                v-if="form.errors.email" 
                class='text-red-500' 
                v-text='form.errors.email'/>

            <input
                type="password"
                class="block border border-black"
                v-model="form.password"
                @input="form.validate('password')">
            <span 
                v-if="form.errors.password" 
                class='text-red-500' 
                v-text='form.errors.password'/>

            <input
                type="password"
                class="block border border-black"
                v-model="form.password_confirm"
                @input="form.validate('password_confirm')">
            <span 
                v-if="form.errors.password_confirm" 
                class='text-red-500' 
                v-text='form.errors.password_confirm'/>

            <button type="submit" class="block border">Submit</button>
        </form>
    </div>
</template>
<script setup>
import {useForm} from "formjs-vue2";
import {object, string} from 'yup'

const formSchema = object({
    name: string().required(),
    email: string().required().email(),
    password: string().required(),
    password_confirm: string()
        .required()
        .test('passwords-match', 'Passwords must match', function (value) {
            return this.parent.password === value
        })
})

const form = useForm({
    name: '',
    email: '',
    password: '',
    password_confirm: ''
}, {schema: formSchema})

const submit = async () => {
    await form.validate()

    if (!form.hasErrors) {
        form.post('/api/users', {
            onSuccess: (response) => {
                // handle success
            }
        })
    }
}

</script>

Backend validation

Formjs includes built-in support for backend validation, with Laravel's validation error response being considered the standard response format. Any validation errors will be automatically accessible through form.errors[field].

Demo

Link for demo: https://stackblitz.com/edit/vitejs-vite-4vj3mb?file=src/App.vue

Conclusion

👏👏In conclusion, Form JS is a powerful form-handling library that simplifies the process of form validation and submission in Vue applications. Inspired by Inertia JS, it features built-in support for Axios and schema-based validation using Yup. Form JS also supports backend validation errors through Laravel's error response format. Its easy-to-use API allows for effortless integration into any Vue project, making form handling a breeze for front-end developers.

Overall, Form JS is a valuable addition to any Vue developer's toolkit and can help streamline the development process, enabling developers to focus on building better user experiences.

Feel free to share your thoughts and opinions and leave me a comment if you have any problems or questions.

Easiest Way to Tinker with Your Laravel Application

Bedram Tamang — Sun, 02 Jun 2024 07:24:17 +0000

Tinkering with your Laravel application is an excellent way to interact with your application, explore your database, and debug your code. Laravel provides a console command, php artisan tinker, which allows you to play with every part of your application interactively. However, using a PHP file for tinkering can be even more convenient, as it allows you to write, modify, and save your code easily.

In this blog post, we will demonstrate how to tinker with your Laravel application using a PHP file. This method is advantageous because it is easier to work with a PHP file than with a console command.

Getting Started

First, create a PHP file named tinker.php, If you are using an IDE like PhpStorm, you can create a scratch file so that you don't have to delete it each time you commit your code. This approach ensures a cleaner and more organized workflow.

Next, set up the Laravel application in tinker.php as follows:

<?php

const BASE_PATH = '/Users/ellite/code/jobins';

require_once BASE_PATH.'/vendor/autoload.php';

use Illuminate\Foundation\Console\Kernel;

$app = require BASE_PATH.'/bootstrap/app.php';
$app->make(Kernel::class)->bootstrap();

dd(\App\Model\AdminModel::query()->first());

Explanation

Autoload Dependencies:

const BASE_PATH = '/Users/Bedram/code/example';
require_once BASE_PATH.'/vendor/autoload.php';
# This line sets the base path to your Laravel project directory.

Bootstrap the Laravel Application:

use Illuminate\Foundation\Console\Kernel;

$app = require BASE_PATH.'/bootstrap/app.php';
$app->make(Kernel::class)->bootstrap();

These lines initialize the Laravel application by requiring the app.php file and bootstrapping the Kernel. That's it, That all you need to setup your tinkering.

Additional Examples

Fetching All Records:

dd(\App\Model\AdminModel::all());
dd(\App\Model\AdminModel::where('status', 'active')->get());

Handling Relationships:

$admin = \App\Model\AdminModel::with('roles')->first();
dd($admin);

Creating a New Record:

$newAdmin = \App\Model\AdminModel::create([
    'name' => 'New Admin',
    'email' => 'newadmin@example.com',
    'password' => bcrypt('password123'),
    'status' => 'active',
]);
dd($newAdmin);

Updating a Record:

$admin = \App\Model\AdminModel::first();
$admin->update(['status' => 'inactive']);
dd($admin);
Deleting a Record:

Deleting a Record

$admin = \App\Model\AdminModel::first();
$admin->delete();
dd('Record deleted');

Conclusion

Laravel's Eloquent ORM and the php artisan tinker command make it incredibly easy to interact with your application and database. However, using a PHP file for tinkering can offer even more flexibility and ease of use. By following the steps outlined in this blog post, you can quickly set up a PHP file to explore and debug your Laravel application efficiently.

Happy tinkering!

Vue 2 reaches EOL, upgrade to Vue 3 now

Bedram Tamang — Wed, 24 Jan 2024 20:11:41 +0000

Vue 2 has reached its end of life on December 31, 2023, It's been more than 3 years since Vue 3 was available, but still, there are a lot of Vue Js applications that are running in Vue 2. In this article, I will demonstrate two different approaches to upgrading the Vue Js application.

Approach 1: Follow the Vue Js upgrade guide

Vue Js Team has provided the upgrade guide to migrate from Vue 2 to Vue 3. The documentation has also listed all the breaking changes, which you may need to consider while upgrading your application. Besides that, you also need to consider all the dependencies and its upgrade guide. But even after you consider all the factors, there will be a lot of work to do. But there is a solution Migration Build.

Migration Build

Migration build or @vue/compat is a package that provides compatibility of your current Vue 2 application to run with Vue 3. This package provides the same APIs as Vue 2 and runs in Vue 2 mode by default so that the same application runs in Vue 3 but in compatibility Vue 2 mode. This feature helps to write new code in Vue 3 and change a component one by one in the long run. Another feature of this package is that you can turn on or turn off a single feature, which means you start with full compatibility mode and migrate one by one turning the migrated features off and finally removing the compatibility mode. The basic example with Vite would look like:

// vite.config.js
export default {
  resolve: {
    alias: {
      vue: '@vue/compat'
    }
  },
  plugins: [
    vue({
      template: {
        compilerOptions: {
          compatConfig: {
            MODE: 3,
          }
        }
      }
    })
  ]
}

Example: v-model compat mode

In 2.x, v-model passes the value as a prop and emits an input event. This concept can be presented as:

<BaseInput v-model="model"/>
<!-- is equivalent to -->
<BaseInput :value="model" @input="model=$event"/>

But in 3.x version, the value props of v-model is changed to modelValue, and input is changed to update:modelValue. This concept can be presented as:

<BaseInput v-model="model"/>
<!-- is equivalent to -->
<BaseInput :modelValue="model" @update:modelValue="model=$event"/>

So to work with the Vue 2 Version of code in Vue 3 as well, we can simply enable the COMPONENT_V_MODEL migration build flag.

compilerOptions: {
    compatConfig: {
        MODE: 3,
        COMPONENT_V_MODEL: true,    
    }
}

Compat mode Limitation

NOT all the features and APIs of Vue 2 are supported by @vue/compat, there are some features that are not supported even by Migration build mode, they have also listed the Limitations in details. One of my experiences was with the vee-validate library. We were solely depending on the vee-validate version 2.x library to validate input on the frontend side, this library was using some internal APIs (VDOM and VNode API) to make things work, However, those internal APIs are removed in Vue 3 and not even supported by compact mode as well, which means this library never gonna upgrade to 3.x version, resulted that we never can do upgrade our Vue 2 application to Vue 3.

Approach 2: Separate a new Vue project

In this approach, instead of migrating the old code-base into Vue 3, a new Vue project will be created in a new directory, let say assets. Basically, it would be completely new Vue project, inside assets directory, that would be it's own package.json files, it's own index.html, it's own node_modules, everything new. The benefits of this approach is that you don't need to consider any breaking change, it doesn't interfere the existing system, however, you can use the components from the root projects. The folder structure would look like, Please go through a source code for this demo App:

root
│   package.json    
│   node_modules
│   vite.config.js
└───src <!-- Vue 2 Application -->
│   │   App.vue 
│   │   main.js
│   │   index.html
└───src-vue-3 <!-- Vue 3 Application -->
│   │   package.json
│   │   node_modules
│   │   index.html
│   │   main.js
│   │   vite.config.js

Routing

Routing is little bit tricky here, your Vue 2 application is serving for the root route http::127.0.0.1:8080, and Vue 3 application will be served on some path let's say http::127.0.0.1:8080/vue-3, which means vue-3 path will be root for this Vue 3 application. The one limitation of this is that you can't define multiple immediate routes path from root domain, i.e, you can't define http::127.0.0.1:8080/vue-3 and http::127.0.0.1:8080/vue-3-another for the Vue 3 application unless you explicitly define a nginx rule for it.

Conclusion

In this article, I have discussed two different approach to migrate your Vue application from version 2 to Version 3, For many of us, the 1st approach doesn't work because of various reasons, that's where the second approach gives us a way to write all new code in Vue 3, where as leaving the existing code in Vue 2. Another solution would be to get help from paid Nerver Ending Support, in which they are offering never ending support for Vue 2 (Non Promotional Link). If none of these solution works for you, you can stay at Vue 2 as long as you want, End of life doesn't mean It will stop working from tomorrow, It just means there will be no active development, and no any more new features, and no security fix, otherwise, it would work forever, like it's working now.

Upgrading Laravel: An easy Step-by-Step Guide from 5.7 to 10.x

Bedram Tamang — Mon, 11 Dec 2023 19:37:18 +0000

In this article, I'll detail my method for upgrading any Laravel application, regardless of its current version, to the most recent Laravel release. Throughout this blog, I'll be showcasing my approach by upgrading an older Laravel 5.7 application directly to Laravel 10.x. Instead of a step-by-step version-by-version upgrade, we'll be transitioning directly to the latest version from any starting point.

Step 1: Environment Setup

The first step of the upgrade is to set up your environment to run your application into 10.x (intended version). The requirement for the Laravel version 10 is PHP 8.1 and extensions specified in Laravel Server Requirement.

Step 2: Laravel ONLY Upgrade

I understand that your application currently has numerous packages installed. However, for the initial phase, we will focus solely on upgrading the default packages that are specified within the framework. To begin this process, please make a backup of the existing composer.json file by renaming it to composer.json.back. Additionally, remember to remove the composer.lock file.

Next, create a new composer.json file, and copy the content from Laravel 10.x branch from the GitHub repository into it. This composer.json file from Laravel's repository will serve as the base for our upgrade.

Add scripts and custom config to new composer.json file from the backup.

Step 3: Install Dependencies

composer install

Running composer install at this stage is expected to encounter errors. This is because we haven't yet defined any previous dependencies or followed any Laravel upgrade guides. However, this deliberate approach allows us to identify and address issues in an iterative manner.

At this stage, you will get errors mostly related to the framework itself. In my example above, the error is because of that the signature of the method report of the class App\Exceptions\Handler has been changed. The best way to deal with this is to visit Laravel's repository and Compare what has been changed between your current version to the Latest Laravel version. The changes can be accessed from the git differences in GitHub at https://github.com/laravel/laravel/compare/5.7...10.x

The total number of modified files is 93. But most of these files such as .editorconfig, .env.example, .gitattributes, .gitignore, etc. don't require any attention. Your version of these files should be replaced by the new version of itself. Nevertheless, certain files cannot be directly replaced with the new version, as your version may have undergone modifications. In such cases, merging is required by determining the differences between the Laravel-side changes and your own. An example of this merging scenario is seen in the App\Exceptions\Handler file, where custom logic for handling 403 error is added from my side and Laravel changed the signature of the report method.

Don't affraid with the number of files count. This is not complicated as it seems.

Step 4: Installing Remaining Dependencies

At this stage, we've successfully addressed the framework-related issues. However, it's possible that when running the composer install command, errors may still arise due to pending dependencies. Therefore, the next task is to install these remaining dependencies. To begin, let's compare the composer.json.back and the new composer.json files, as illustrated in the image below.

Install any additional dependencies that you have installed into your application. Make sure not to install any dependencies that have been removed by Laravel. Utilize the Git comparison above to identify which dependencies have been removed by Laravel.

In the required section, the fideloper/proxy package has been removed in between 5.x to 10.x. Therefore, there is no need to install it. All of your remaining dependencies can be installed with composer require command. In my example,

composer require aammui/role-permission ausi/slug-generator barryvdh/laravel-

Follow the same procedure for dev dependencies with composer requrie --dev flag.

Step 5: Final suggestion

There could be numerous dependencies, each demanding a careful review of their specific upgrade guides. Robust test cases are pivotal in verifying the successful implementation of upgrades and ensuring the system's proper functionality. Consulting Laravel's GitHub repository for the most recent updates and adjusting your code accordingly is crucial. I recommend referring to the Laravel upgrade guides for further insights.

Conclusion

This method provides the advantage of skipping incremental version updates, enabling a direct transition from any version to your intended target version. prioritizes identifying errors initially and then addressing them accordingly. Robust test coverage is strongly recommended to validate the upgraded system's functionality. Additionally, it's highly encouraged to thoroughly review both Laravel's upgrade guides and those of various dependencies for a smoother upgrade process.

DEV Community: Bedram Tamang

Duplicate Rows Detection in MySQL

Duplicate Rows Detection in MySQL

Requirements:

Let's start solving

Populate duplicates

Add a row

Update a Row

Conclusion

Solving Problems the Right Way: Leveraging Frameworks Over Quick Fixes

Conclusion

Test Driven Development in Laravel - Folder Structure & Setup (1)

Folder structure

Benefits

Drawbacks of this structure:

Benefits

Test Setup

Database Setup

Conclusion

Fine-Tuning T5-Small Model for a Completely New Language: Limbu

Introduction

Preparing the Data

Setting Up the Environment

Loading the Pretrained Model

Tokenizing the Dataset

Preparing for Training

Training the Model

Visualizing Training Progress

Testing the Model

Evaluating with BLEU Score

Conclusion

Fine-Tuning ChatGPT for a Completely New Language: Limbu

Why Fine-Tune Models?

Steps Involved

Conclusion

Sharing Authentication cookies across Laravel Apps

Cookie Sharing

Important Consideration

Setup

Share Encryption Key

Sanctum Setup

Cookie Domain

Cookie Conflict

Using Different Cookie in api.laravel.test

Conclusion:

References:

Implementing boolean OR Search in meilisearch

Problem Statement:

Solution

Conclusion

References

Eloquent Trick: Laravel Model from Subquery

Conclusion

Simplifying Form Handling in Vue Applications with Form JS - Inspired by Inertia JS

For example:

Formjs

Front end Validation

Validate Form

Backend validation

Demo

Conclusion

Easiest Way to Tinker with Your Laravel Application

Getting Started

Explanation

Additional Examples

Conclusion

Vue 2 reaches EOL, upgrade to Vue 3 now

Approach 1: Follow the Vue Js upgrade guide

Migration Build

Example: v-model compat mode

Compat mode Limitation

Approach 2: Separate a new Vue project

Routing

Conclusion

Upgrading Laravel: An easy Step-by-Step Guide from 5.7 to 10.x

Step 1: Environment Setup

Step 2: Laravel ONLY Upgrade

Step 3: Install Dependencies

Step 4: Installing Remaining Dependencies

Step 5: Final suggestion

Using Different Cookie in `api.laravel.test`