DEV Community: AneeqMalik

Create RESTAPI of YOLOv5 on Flask deployed on Azure App Service and use in Flutter App

AneeqMalik — Wed, 08 Mar 2023 17:48:14 +0000

STEPS:

Creating YOLOV5 Flask App
Creating Docker Image
Running Docker Container and Pushing to DockerHub
Deploying Rest API to Azure for Free.
Using API in flutter project to perform detections.

Creating YOLOV5 Flask App

Create a flask app with your ML model with requests defined.

"""
Run a rest API exposing the yolov5s object detection model
"""
import argparse
import io
from PIL import Image

import torch
from flask import Flask, request

app = Flask(__name__)

DETECTION_URL = "/v1/object-detection/yolov5"


@app.route(DETECTION_URL, methods=["POST"])
def predict():
    if not request.method == "POST":
        return

    if request.files.get("image"):
        image_file = request.files["image"]
        image_bytes = image_file.read()
        img = Image.open(io.BytesIO(image_bytes))
        results = model(img, size=640) # reduce size=320 for faster inference
        return results.pandas().xyxy[0].to_json(orient="records")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Flask api exposing yolov5 model")
    parser.add_argument("--port", default=5000, type=int, help="port number")
    parser.add_argument('--model', default='yolov5s', help='model to run, i.e. --model yolov5s')
    args = parser.parse_args()

    model = torch.hub.load('ultralytics/yolov5', args.model)
    app.run(host="0.0.0.0", port=args.port)  # debug=True causes Restarting with stat

you can use a your own model as well with the help of following place the model on the same path as the app.py or flask app file.

model = torch.hub.load('.','custom', path='best.pt',force_reload=True,source='local', pretrained =Flase)

Creating Docker Image:

First create a DockerFile with the following:

# Use an official Python runtime as a parent image
FROM python:3.8-slim-buster

RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6  -y

# Set the working directory to /app
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install -r requirements.txt

# Make port 5000 available to the world outside this container
EXPOSE 5000

# Run app.py when the container launches
CMD ["python", "app.py", "--port=5000"]

change the app.py if you have different app name

Run the Following Commands to create an Image and push to the docker hub:

Running Docker Container and Pushing to DockerHub

Run to Ensure it is Working:

Login to Docker Hub:

Create a tag to push to Docker Hub image:

Deploying Rest API to Azure for Free.

Create a new Web App Service in Azure:

Login to your Azure account.
Select "Create a resource" and search for "Web App".
Select "Web App" from the search results and click "Create".
Choose a unique name, subscription, resource group, and app service plan.

Choose "Docker Container" as the Publish option and "Linux" as the Operating System.
Choose "Docker Hub" as the Registry and enter the name and version of the image you created in step 1.
Click "Create" to create the Web App Service.
Wait for Azure to deploy your container to the Web App Service. This may take a few minutes.

Using API in flutter project to perform detections.

main.dart

import 'package:flutter/material.dart';
import 'package:flutter_restapi/ObjectDetectionScreen.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        // This is the theme of your application.
        //
        // Try running your application with "flutter run". You'll see the
        // application has a blue toolbar. Then, without quitting the app, try
        // changing the primarySwatch below to Colors.green and then invoke
        // "hot reload" (press "r" in the console where you ran "flutter run",
        // or simply save your changes to "hot reload" in a Flutter IDE).
        // Notice that the counter didn't reset back to zero; the application
        // is not restarted.
        primarySwatch: Colors.blue,
      ),
      home: ObjectDetectionScreen(),
    );
  }
}

ObjectScreen.dart

import 'dart:io';
import 'dart:convert';

import 'package:flutter/material.dart';
import 'package:image_picker/image_picker.dart';
import 'package:http/http.dart' as http;

class BoundingBox {
  final double xMin;
  final double yMin;
  final double xMax;
  final double yMax;
  final double confidence;
  final String name;

  BoundingBox({
    required this.xMin,
    required this.yMin,
    required this.xMax,
    required this.yMax,
    required this.confidence,
    required this.name,
  });
}

class ObjectDetectionScreen extends StatefulWidget {
  const ObjectDetectionScreen({super.key});

  @override
  State<ObjectDetectionScreen> createState() => _ObjectDetectionScreenState();
}

class _ObjectDetectionScreenState extends State<ObjectDetectionScreen> {
  late List<BoundingBox> _boundingBoxes = [];
  File? _image;
  bool _loading = false;
  final picker = ImagePicker();

  Future<List<BoundingBox>> detectObjects(File image) async {
    final url =
        "https://flask-restapi-yolov5s.azurewebsites.net/v1/object-detection/yolov5";
    final request = await http.MultipartRequest("POST", Uri.parse(url));
    request.files.add(await http.MultipartFile.fromPath("image", image.path));
    final response = await request.send();

    if (response.statusCode == 200) {
      final jsonStr = await response.stream.bytesToString();
      final jsonResponse = json.decode(jsonStr);
      print(jsonResponse);
      return List<BoundingBox>.from(jsonResponse.map((bbox) => BoundingBox(
            xMin: bbox["xmin"],
            yMin: bbox["ymin"],
            xMax: bbox["xmax"],
            yMax: bbox["ymax"],
            confidence: bbox["confidence"],
            name: bbox["name"],
          )));
    } else {
      throw Exception('Failed to detect objects');
    }
  }

  Future<void> getImage(ImageSource source) async {
    setState(() {
      _loading = true;
    });

    final pickedFile =
        await picker.pickImage(source: source, maxWidth: 340, maxHeight: 340);
    if (pickedFile != null) {
      setState(() {
        _image = File(pickedFile.path);
      });
      final bboxes = await detectObjects(_image!);
      setState(() {
        _boundingBoxes = bboxes;
        _loading = false;
      });
    } else {
      setState(() {
        _loading = false;
      });
    }
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
        appBar: AppBar(
          title: const Text('Object Detection Using Flask Rest Api'),
        ),
        body: Center(
          child: SingleChildScrollView(
            child: Column(
              children: [
                _image == null
                    ? const Text('No image selected.')
                    : Stack(
                        children: [
                          Image.file(_image!),
                          ..._boundingBoxes.asMap().entries.map((entry) {
                            final index = entry.key;
                            final bbox = entry.value;
                            final xMin = bbox.xMin;
                            final yMin = bbox.yMin;
                            final xMax = bbox.xMax;
                            final yMax = bbox.yMax;
                            final confidence = bbox.confidence;
                            final name = bbox.name;

                            final left = xMin;
                            final top = yMin;
                            final width = xMax - xMin;
                            final height = yMax - yMin;

                            Color color;
                            if (index % 3 == 0) {
                              color = Colors.green;
                            } else if (index % 3 == 1) {
                              color = Colors.yellow;
                            } else {
                              color = Colors.red;
                            }

                            return Positioned(
                              left: left,
                              top: top,
                              width: width,
                              height: height,
                              child: Container(
                                decoration: BoxDecoration(
                                  border: Border.all(
                                    color: color,
                                    width: 2.0,
                                  ),
                                  borderRadius: BorderRadius.circular(20),
                                ),
                                child: Text(
                                  "$name ${(confidence * 100).toStringAsFixed(0)}%",
                                  textAlign: TextAlign.center,
                                  style: TextStyle(
                                    color: color,
                                    fontSize: 12.0,
                                    fontWeight: FontWeight.bold,
                                    shadows: const [
                                      Shadow(
                                        color: Colors.black,
                                        offset: Offset(1, 1),
                                        blurRadius: 2,
                                      )
                                    ],
                                  ),
                                ),
                              ),
                            );
                          }).toList(),
                        ],
                      ),
                ElevatedButton(
                  onPressed: () => getImage(ImageSource.camera),
                  child: const Text("Take a Picture"),
                ),
                ElevatedButton(
                  onPressed: () => getImage(ImageSource.gallery),
                  child: const Text("Choose from Gallery"),
                ),
                const SizedBox(height: 10),
                _loading
                    ? const CircularProgressIndicator(
                        color: Colors.blue,
                      )
                    : const SizedBox(),
              ],
            ),
          ),
        ));
  }
}

BoundingBox.dart

import 'dart:ui';

import 'package:flutter/material.dart';

class BoxPainter extends CustomPainter {
  final dynamic predictions;

  BoxPainter(this.predictions);

  @override
  void paint(Canvas canvas, Size size) {
    final width = size.width;
    final height = size.height;
    final colors = [
      Colors.red,
      Colors.green,
      Colors.blue,
      Colors.yellow,
      Colors.orange,
      Colors.purple,
      Colors.pink,
    ];

    if (predictions != null) {
      for (var i = 0; i < predictions.length; i++) {
        final prediction = predictions[i];
        final bbox = prediction['bbox'];

        final left = bbox['xmin'].toDouble();
        final top = bbox['ymin'].toDouble();
        final right = bbox['xmax'].toDouble();
        final bottom = bbox['ymax'].toDouble();

        final rect = Rect.fromLTWH(
          left / 640 * width,
          top / 640 * height,
          (right - left) / 640 * width,
          (bottom - top) / 640 * height,
        );

        final paint = Paint()
          ..color = colors[i % colors.length]
          ..style = PaintingStyle.stroke
          ..strokeWidth = 2.0;

        final labelPaint = Paint()
          ..color = colors[i % colors.length]
          ..style = PaintingStyle.fill
          ..strokeWidth = 2.0;

        canvas.drawRect(rect, paint);

        final label = '${prediction['name']} (${prediction['confidence']})';
        final labelOffset = Offset(
          left / 640 * width,
          top / 480 * height - 20,
        );
        canvas.drawRect(
          Rect.fromPoints(
            labelOffset,
            Offset(
              labelOffset.dx + label.length * 8,
              labelOffset.dy + 20,
            ),
          ),
          labelPaint,
        );

        final textStyle = TextStyle(
          color: Colors.white,
          fontSize: 14.0,
        );
        final textSpan = TextSpan(
          text: label,
          style: textStyle,
        );
        final textPainter = TextPainter(
          text: textSpan,
          textDirection: TextDirection.ltr,
        );
        textPainter.layout(
          minWidth: 0,
          maxWidth: size.width,
        );
        textPainter.paint(
          canvas,
          Offset(
            labelOffset.dx + 4,
            labelOffset.dy + 2,
          ),
        );
      }
    }
  }

  @override
  bool shouldRepaint(BoxPainter oldDelegate) => false;
}

Compile and debug the project.

20230308205236.mp4 - Google Drive

drive.google.com

App is Running 🎉🎉🎉

Resource and Links:

https://github.com/AneeqMalik/YoloV5-Rest-Api-Flask

Flutter Object Detection App + YOLOV5 Model.

AneeqMalik — Wed, 15 Feb 2023 15:43:12 +0000

Flutter is a popular open-source mobile application development framework that allows developers to create high-performance, cross-platform applications for both Android and iOS devices. With its ease of use and flexibility, Flutter has become a popular choice for developers looking to build mobile applications.

One of the most popular use cases for mobile applications is object detection, where an application can identify and classify objects in images or videos. YOLOv5 is an advanced object detection algorithm that has gained popularity in recent years for its high accuracy and speed.

In this post, we will explore how to integrate YOLOv5 with Flutter to create an object detection application.

1. Setting Up the Environment:

To get started, you'll need to set up your development environment. You'll need to have Flutter and Python installed on your computer. After that open the Vscode and initialize a new flutter project.

flutter create object_detection

Wait for the project to be created.
Now, open the https://pub.dev/packages/flutter_pytorch and navigate to Installing tab there you can find the latest package version for the time being the package version is 1.0.1
Run this command:

With Flutter:
$ flutter pub add flutter_pytorch
This will add a line like this to your package's pubspec.yaml (and run an implicit flutter pub get):

dependencies:
  flutter_pytorch: ^1.0.1

Alternatively, your editor might support flutter pub get. Check the docs for your editor to learn more.

Import it
Now in your Dart code, you can use:

import 'package:flutter_pytorch/flutter_pytorch.dart';

2. Preparing the Model.

Before you can use YOLOv5 in your Flutter application, you'll need to train the model on your specific dataset. You can use an existing dataset or create your own dataset to train the model.
For this post I am using the pretrained model of yolov5 available on https://github.com/ultralytics/yolov5 as we are performing object detection we need to converts the pretrained model weights to torchscript format.

classification

import torch
from torch.utils.mobile_optimizer import optimize_for_mobile
model = torch.load('model_scripted.pt',map_location="cpu")
model.eval()
example = torch.rand(1, 3, 224, 224)
traced_script_module = torch.jit.trace(model, example)
optimized_traced_model = optimize_for_mobile(traced_script_module)
optimized_traced_model._save_for_lite_interpreter("model.pt")

object detection (yolov5)

!python export.py --weights "the weights of your model" --include torchscript --img 640 --optimize

example

!python export.py --weights yolov5s.pt --include torchscript --img 640 --optimize

3. Creating a basic UI for the App.

Open the lib folder of the app there you will see main.dart file this is the main compile file of dart.
Run the file in debug mode on an emulator or phone connected.

emulator -avd "Your emulator name"

Now run your App through Vscode in debug as it offers quick reload and hot restart.

Your initial app should like the following

Creating the UI design
To design the App we will follow a WBS (Work Breakdown Structure) for designing app and integrating the ml model.
Go to the main.dart file and replace all the code with the following:

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  // This widget is the root of your application.
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'OBJECT DETECTOR',
      debugShowCheckedModeBanner: false,
      theme: ThemeData(
        // This is the theme of your application.
        //
        // Try running your application with "flutter run". You'll see the
        // application has a blue toolbar. Then, without quitting the app, try
        // changing the primarySwatch below to Colors.green and then invoke
        // "hot reload" (press "r" in the console where you ran "flutter run",
        // or simply save your changes to "hot reload" in a Flutter IDE).
        // Notice that the counter didn't reset back to zero; the application
        // is not restarted.
        primarySwatch: Colors.blue,
      ),
    );
  }
}

The above code is nothing just a basic class structure of App with all the unnecessory code removed. After reload you will see a black screen like the following:

If you are a beginner don't panic, be patient we are just starting 😂.

Create a new dart file with the name HomeScreen.dart inside the lib folder.
Now if you type stf and press enter this will auto-magically create the relevant structure of code here we are using statefull widget.

class HomeScreen extends StatefulWidget {
  const HomeScreen({super.key});

  @override
  State<HomeScreen> createState() => _HomeScreenState();
}

class _HomeScreenState extends State<HomeScreen> {
  @override
  Widget build(BuildContext context) {
    return Container();
  }
}

Now create a text inside the Scaffold Class as:

 return Scaffold(
      backgroundColor: Colors.white,
      body: Text("Home Screen"),
    );

and call this HomeScreen state to your App State as:

home: HomeScreen(),

Now you can see the App changes to:

Now add the following code on HomeScreen.dart to design the base UI of the App.

import 'package:flutter/material.dart';
import 'package:flutter/src/widgets/container.dart';
import 'package:flutter/src/widgets/framework.dart';

class HomeScreen extends StatefulWidget {
  const HomeScreen({super.key});

  @override
  State<HomeScreen> createState() => _HomeScreenState();
}

class _HomeScreenState extends State<HomeScreen> {
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: const Text("OBJECT DETECTOR APP")),
      backgroundColor: Colors.white,
      body: Center(
          child: Column(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          //Image with Detections....
          //Button to click pic
          ElevatedButton(
            onPressed: () {},
            child: const Icon(Icons.camera),
          )
        ],
      )),
    );
  }
}

Now the app is Ready to be integerated with the model.

4. Integrating the Model with Flutter

Once you've trained the model, you can integrate it with your Flutter application. First create a folder of assets with sub folder of model and labels also declare it on your pubspec.yaml file. As shown in the images bellow:

Folder Structure after creating the additional model and labels folder.

Pubspec.yaml file after adding the assets path:

Now place the models converted to the models folder and create labels.txt file place your labels there.

Or you can download the model and labels from the following link:
https://github.com/AneeqMalik/flutter_pytorch/tree/main/example/assets
Plz star the repository 😊.
After placing the labels and model to their respective folders it is the time we are waiting for integrating the model to our app.

Integrating the YOLOv5 Model

Create the following Variables:

File? _imageFile;
  late ModelObjectDetection _objectModel;
  String? _imagePrediction;
  List? _prediction;
  File? _image;
  ImagePicker _picker = ImagePicker();
  bool objectDetection = false;
  List<ResultObjectDetection?> objDetect = [];

Create a Function to load the model into the App:

Future loadModel() async {
    String pathObjectDetectionModel = "assets/models/yolov5s.torchscript";
    try {
      _objectModel = await FlutterPytorch.loadObjectDetectionModel(
          //Remeber here 80 value represents number of classes for custom model it will be different don't forget to change this.
          pathObjectDetectionModel, 80, 640, 640,
          labelPath: "assets/labels/labels.txt");
    } catch (e) {
      if (e is PlatformException) {
        print("only supported for android, Error is $e");
      } else {
        print("Error is $e");
      }
    }
  }

Call the load function inside the initState() to load the model as the App Opens:

@override
  void initState() {
    // TODO: implement initState
    super.initState();
    loadModel();
  }

Add the following Widgets inside the Column:

 body: Center(
          child: Column(
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          //Image with Detections....
          Expanded(
            child: Container(
              height: 150,
              width: 300,
              child: objDetect.isNotEmpty
                  ? _image == null
                      ? Text('No image selected.')
                      : _objectModel!.renderBoxesOnImage(_image!, objDetect)
                  : _image == null
                      ? Text('No image selected.')
                      : Image.file(_image!),
            ),
          ),
          Center(
            child: Visibility(
              visible: _imagePrediction != null,
              child: Text("$_imagePrediction"),
            ),
          ),
          //Button to click pic
          ElevatedButton(
            onPressed: () {
              runObjectDetection();
            },
            child: const Icon(Icons.camera),
          )
        ],
      )),

Finally create a object detection function to get the inferences of the image detected:

Future runObjectDetection() async {
    //pick an image

    final XFile? image = await _picker.pickImage(
        source: ImageSource.gallery, maxWidth: 200, maxHeight: 200);
    objDetect = await _objectModel.getImagePrediction(
        await File(image!.path).readAsBytes(),
        minimumScore: 0.1,
        IOUThershold: 0.3);
    objDetect.forEach((element) {
      print({
        "score": element?.score,
        "className": element?.className,
        "class": element?.classIndex,
        "rect": {
          "left": element?.rect.left,
          "top": element?.rect.top,
          "width": element?.rect.width,
          "height": element?.rect.height,
          "right": element?.rect.right,
          "bottom": element?.rect.bottom,
        },
      });
    });
    setState(() {
      _image = File(image!.path);
    });
  }

Changing the default sdk version
When you run the App after the above aditions the app will not compile and will give an issue similiar to the following:

You need to change the default sdk version from the following file
ProjectName\object_detection\android\app\build.gradle:

Save the changes and re-run the app build.
Wait for the build to complete it may give some warning but ignore them for the time being.

5. Testing the Application

To test your application, you can provide an image or video to the application and see how the model detects objects in the media.

Screenshots

🥳🥳🥳🥳The App is Running Fine and giving detections.

Link to Source Code

https://github.com/AneeqMalik/Flutter-Object-Detector-App-YOLOv5-

Additional Resources/Info

https://pub.dev/packages/flutter_pytorch