Unprocessed data monitor in laravel.

Sometimes you need to select multiple values and process them in the background upon a button click. For example, like this screen:

As you can see, I can select some items with specific ids and by clicking the Process Them button, I submit them for processing.

Basic Algorithm

NOTE this section contains stack - agnostic table structures and describes the general idea ignoring any table implementation.

Once I submit the data, I use the following table for the process:

Name: batch_process
-----------
batch_id: PK INTEGER AUTO INCREMENT
selected_ids: INTEGER ARRAY
batch_started: CURRENT TIMESTAMP (Optional Field)
batch_ended: TIMESTAMP (Optional Field)

Each record contains the submited ids in selected_ids field and a record is populated once I submit the data via post (Either via ajax or not).

Then I can dispatch a job-backround process where is processed. For each Item I process, I populate a table that has the following structure (table structure can be modified depending the case)

Name: processed_items
-----
id: PK INTEGER AUTO INCREMENT
batch_id: FK references batch_process table
selected_id: INTEGER (Containing a single id from batch_process.selected_ids)
status: ENUM(FAILED, SUCCESS, PROCESSING)
error_msg: VARCHAR,
timestamp_started: TIMESTAMP
timestamp_ended: TIMESTAMP

Each time I process an id from a batch_process table record before processing it I create a record with populated these fields:

• batch_id, With the batch Id provided
• selected_id, With an item from batch_process.selected_ids
• status=PROCESSING
• timestamp_started: with the timestamp before record is created.

Once I finish processing the selected item I just need to update the status either with SUCCESS If the successfully has been processed or with FAILED if the item had fails to be processed, the failure reason is recorded in error_msg column.

In case that an item either succeeds or fails to be processed, the column timestamp_ended is populated with the timestamp the processing has failed.

Implementation

Once a form is submitted the following controller will be used:

class BatchProcessController extends Controller
{
  public function initiateBulkProcess(Request $request)
  {
    $ids = $request->post('ids');
    $now = Carbon::now()->format('Y-m-d H:i:s');

    $batch_id = DB::table('batch_process')->insertGetId([
'selected_ids' => "{".trim(implode(',',$ids),",")."}"
'batch_started' => date('Y-m-d H:i:s')
]);
    // We can also use a dedicated class for this but for me to explain it easily I use closure as a sample showcase.
    dispatch(function() use ($batch_id,$ids) {
      foreach($ids as $id){
        $log_id = DB::table('processed_items')->insertGetId([
            'batch_id' => $batch_id,
            'selected_id' => $id,
            'status' => 'PROCESSING',
            'timestamp_started' => date('Y-m-d H:i:s')
        ]);

        // Do some process on the item having $id
        $sucess = false;
        $error_msg = NULL'
        try {
          // processItem contains the logic for bulk processes 
          $sucess = prossessItem($id);
          $error_msg = $sucess ? NULL: "Uknown";
        } catch (Exception $e) {
          $sucess = false;
          $error_msg = $d->setMessage();
        }
      }

      $status = $sucess? "SUCESS":"FAILED";
  DB::table('processed_items')->update(['status'=>$status,'error_msg'=>$error_msg,'timestamp_ended'=>date('Y-m-d H:i:s')])->where('id',$log_id);

    });
  }

}

What did we achieve

First, we managed to implement a mechanism that allows us to easily check what is processed and if the processed item is successful or not from user-initiated batch processes.

Secondly, we are able to extract more user info and statistics regarding the batch process frequency from our users. Furthermore, we can easily make watchdogs that reprocess for delayed processed items.

So we have a better logging and monitoring regarding of who processes and what is processed.