{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "QMO87DRgiRTL"
},
"source": [
"# Lab 8 -- Object Detection Model Training & Deployment using YOLOv5\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "t6MPjfT5NrKQ"
},
"source": [
" About
\n",
"\n",
"Being able to identify objects is a very important skill for robots to have in order to perform tasks in a human environment. Hence, this notebook seeks to cover and demonstrate a rough workflow for training and deploying a deep learning model for object detection. While this notebook covers most of what you would need in a basic deep learning object detection project, the concepts covered here are non-exhaustive. PyTorch & TensorFlow are open source libraries that provide commmon infrastructures for machine learning applications and to accelerate the use of machine perception in commercial products. \n",
"\n",
"For more information and extra readings, do read through the following:\n",
"\n",
"Free online data labelling tools
\n",
"\n",
"- [CVAT](https://github.com/openvinotoolkit/cvat) \n",
"- [Roboflow](https://roboflow.com/)\n",
"- [LabelImg](https://github.com/tzutalin/labelImg)\n",
"\n",
"Using TensorFlow for Model Training & Deployment
\n",
"\n",
"- [Model Training & Deployment for TensorFlow](https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/) \n",
"- [TensorFlow Docs](https://www.tensorflow.org/) \n",
"\n",
"Using PyTorch for Model Training & Deployment
\n",
"\n",
"- [Model Training & Deployment for PyTorch](https://pytorch.org/tutorials/beginner/introyt/trainingyt.html)\n",
"\n",
"- [YOLOV5 Training & Deployment](https://github.com/ultralytics/yolov5)
\n",
"\n",
"\n",
"
\n",
"Note: This notebook has been purposely written with some brevity, such that students will have to learn how to look for relevant information and approach a problem without step-by-step instructions (which is an important skill for any engineer to have). Nevertheless, most of the concepts in this notebook had been discussed and most problems that you may encounter can be solved with an understanding of the concepts discussed."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "TVBrSwS7IrpK"
},
"source": [
"# What is Object Detection?\n",
"Object Detection is a computer technology related to computer vision and image processing that deals with detecting instances of semantic objects of a certain class (such as humans, buildings, or cars) in digital images and videos. \n",
"\n",
"Specifically, object detection draws bounding boxes around these detected objects, which allow us to locate where the objects are in (or how they move through) a given scene, as well as track their precise locations. \n",
"\n",
"Object detection has applications in many computer vision tasks such as face recognition, vehicle counting and tracking objects. \n",
"\n",
"![](\"https://www.mathworks.com/discovery/object-detection/_jcr_content/mainParsys3/discoverysubsection/mainParsys3/image.adapt.full.medium.jpg/1655457511638.jpg\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "A8n1kUQIlLLh"
},
"source": [
"# How is Object Detection done?\n",
"\n",
"As you have learnt in the previous lab, digital images are represented as arrays of integer values that represent the different intensities of colours for each pixel within the image.\n",
"\n",
"To identify objects within the image, we would first need to pass our image (defined as a array of values) through a neural network. From the lectures, you would know that a neural network functions just like our brain, where it takes in multiple inputs in order to calculate an output value. \n",
"\n",
"In the case of object detection, our neural network first takes in multiple inputs in the form of image arrays. The neural network will then make predictions on the image and draw bounding boxes to locate and identify objects with a certain precision.\n",
"\n",
"![](\"https://www.mathworks.com/help/vision/ug/yolo_model70e45989d8550f0aba95189d6401bdac.png\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "uNc5oK7GNz4W"
},
"source": [
"# Object Detection Workflow\n",
"\n",
"In order to train and deploy our own Object Detection Model, we will be using the same workflow that was discussed in the lectures:\n",
"\n",
"![](\"https://www.researchgate.net/profile/Jonathan-Heras-2/publication/327570700/figure/fig1/AS:669520230375427@1536637470393/Workflow-of-object-detection-projects.ppm\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "7mGmQbAO5pQb"
},
"source": [
"# Setup\n",
"\n",
"Before starting work on developing our own Object Detection Model, we first have to download the following files and dependencies:\n",
"\n",
"1. Clone the [YOLOV5](https://github.com/ultralytics/yolov5) repository. We will be using this as our base neural network model to develop and train our new model on.\n",
"2. Install dependencies that are required to run the YOLOV5 base model\n",
"3. Importing the following libraries:
\n",
" - PyTorch for training & developing neural network based deep learning models\n",
" - utils for various helper functions"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "wbvMlHd_QwMG",
"outputId": "5d18cffc-56a4-48d4-ed18-a04dbe6729e6",
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"#Cloning the repository\n",
"!git clone https://github.com/ultralytics/yolov5 \n",
"\n",
"#Installing dependencies\n",
"%cd yolov5\n",
"%pip install -qr requirements.txt # install\n",
"\n",
"#Importing required libraries\n",
"import torch\n",
"import utils\n",
"display = utils.notebook_init() # checks"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "4JnkELT0cIJg"
},
"source": [
"# 0.5..? -- YOLOV5 Model Inference\n",
"\n",
"Once we have installed and imported the relevant libraries, we can first check that everything is installed correctly by running inferences on a pre-trained model.\n",
"\n",
"Pre-trained models can come in many different architectures and implementations of the neural network models. There are many libraries available for model training and deployment, where each library often requires the model to be in a specific format for compatibility. The list of libraries and the corresponding compatible model formats for YOLOv5 are shown in the [latest YOLOv5 release](https://github.com/ultralytics/yolov5/releases).\n",
"\n",
"Not all model formats are compatible with every library. Hence, it is important to first determine the library and model format that you are going to deploy your model with, and checking the compatability with your model training before training your model.\n",
"\n",
"In this notebook, we will be using the YOLOv5 model and the PyTorch library for model training & inference. Hence, the pre-trained model that is compatible with the PyTorch library is stored in the form of weights (ie. `yolov5s.pt`) in a .pt file.\n",
"\n",
"In order to use our pre-trained YOLOv5 model to run inferences, we can make use of `detect.py` to run inferences on various sources and saving the results to `runs/detect`. Example inference sources are:\n",
"\n",
"```shell\n",
"python detect.py --source 0 # webcam\n",
" img.jpg # image \n",
" vid.mp4 # video\n",
" path/ # directory\n",
" path/*.jpg # glob\n",
" 'https://youtu.be/Zgi9g1ksQHc' # YouTube\n",
" 'rtsp://example.com/media.mp4' # RTSP, RTMP, HTTP stream\n",
"```\n",
"Before running the inference, we can specify various parameters to fine-tune and customize our inferences. Some of the parameters include:
\n",
"- weights -- specifying the path of the pre-trained model file\n",
"- img_size -- the height & width of the image for inference\n",
"- conf_threshold -- The accuracy threshold required for a bounding box to be drawn on the object. Specified within a range of 0 to 1\n",
"- source_path -- The path for the files (ie. images/videos) to perform inferences on.\n",
"\n",
"You should be able to see a test image with bounding boxes drawn after running this cell. Otherwise, do take a look at the various parameters and change accordingly to your required use case."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 589
},
"id": "zR9ZbuQCH7FX",
"outputId": "b2218956-a354-4ce8-9ab1-156d8f3c8f54",
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# Weights\n",
"weights_path = \"yolov5s.pt\"\n",
"\n",
"# Img_size\n",
"img_size = 640\n",
"\n",
"# conf_threshold\n",
"conf_threshold = 0.25\n",
"\n",
"# Source_path\n",
"source_path = \"data/images\"\n",
"\n",
"# Executing script with specified parameters\n",
"!python3 detect.py --weights $weights_path --img $img_size --conf $conf_threshold --source $source_path\n",
"\n",
"# Showing the image\n",
"display.Image(filename='runs/detect/exp/zidane.jpg', width=600)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "r7dzW9r5WBjj",
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"#Optional - You can run inferences on your model using a live video feed (not available on Google Collab)\n",
"\n",
"!python3 detect.py --weights yolov5s.pt --img 640 --source 0"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "i0fXwGzgiA5S"
},
"source": [
"# 1. Preparing and cleaning the dataset\n",
"\n",
"At this point, you should have already gathered your data and exported your dataset to a suitable format (YOLOv5 PyTorch Format). If you have not, do go to [Roboflow](https://roboflow.com/) and follow the instructions in the Lab Notes on how you can create your dataset.\n",
"\n",
"If you have used Roboflow to export your dataset, you can import it into this notebook by replacing the code template below and running the cell.\n",
"\n",
"The export contains the following files and folders:\n",
"- Test & Train folders that store the images and annotations for the test and train dataset respectively \n",
"- Labels folder within the Test & Train folders with files that store the coordinate information for the bounding boxes that exist within each image annotated\n",
"- Images folder within the Test & Train folders with images from the Test and Train dataset respectively \n",
"- A YOLO v5 .yaml file `data.yaml`, which functions as a dataset directory by specifying the location of the YOLO v5 images folder, YOLO v5 labels folder, and information on our custom classes."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"id": "A4PZR6jXWBjo",
"outputId": "c7b5f587-85ce-449f-d161-60a6dec8e072",
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"!pip install roboflow\n",
"\n",
"from roboflow import Roboflow\n",
"rf = Roboflow(api_key=\"QoJUUCcZQ4f1Bqh8Z1me\") #QoJUUCcZQ4f1Bqh8Z1me\n",
"project = rf.workspace(\"convert-coco-73sdy\").project(\"lab-8\")\n",
"dataset = project.version(3).download(\"yolov5\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "YK0aJjBBmKGo"
},
"source": [
"# 2. Model Training\n",
"\n",
"With our dataset and our base model imported into the notebook, it is time to train our own custom model! If you are following the task given in the lab notes, you will be training a model to identify bottle caps.\n",
"\n",
"In order to train on the pre-trained YOLOv5 model using our imported custom dataset, we can make use of `train.py` to train our model and save the results to `runs/train/` with incrementing run directories, i.e. `runs/train/exp2`, `runs/train/exp3` etc.\n",
"\n",
"Before running the model training, we can specify various parameters to fine-tune, optimize and customize our model training. Some of the parameters include:
\n",
"- no_of_epochs -- defines the number of **complete passes** that the learning algorithm will work through the entire training dataset\n",
"- batch_size -- defines the number of samples to work through before updating the internal model parameters\n",
"- img_size -- the height & width of the images to be resized to for model training\n",
"- path_to_data_yaml -- specifying the path to your dataset's data.yaml file\n",
"- weights_path -- specifying the path of the pre-trained model file that you want to train on\n",
"\n",
"\n",
"Note: The `data.yaml` file generated sets a wrong path to the test dataset folder. Hence, you will need to change the path to the following for line 5:
\n",
"`val: Lab-8--3/test/images`\n",
"\n",
"Otherwise, your model training would not be able to find the test dataset, and the training process will fail.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "1NcFxRcFdJ_O",
"outputId": "493223d7-3ae6-44f9-8e22-9a7c2e72e440",
"vscode": {
"languageId": "python"
}
},
"outputs": [],
"source": [
"# You can change your parameters here\n",
"num_of_epochs = 80\n",
"batch_size = 32\n",
"img_size = 640\n",
"path_to_data_yaml = \"/content/yolov5/Lab-8--3/data.yaml\"\n",
"weights_path = \"yolov5s.pt\"\n",
"\n",
"# Executing script with specified parameters\n",
"!python3 train.py --img $img_size --batch $batch_size --epochs $num_of_epochs --cfg yolov5s.yaml --data $path_to_data_yaml --weights $weights_path --cache"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "hkGK5awj7NyG"
},
"source": [
"# 3. Evaluating Model\n",
"\n",
"Instead of jumping right into running inferences on images with our newly trained model, we can use certain training metrics in order to evaluate the accuracy and efficiency of our model. This can give us better insights on how we can improve and optimize our model training.\n",
"\n",
"While the training output above shows some training metric values during runtime, it is usually better to be able to visualize your training results for further evaluation & analysis.\n",
"\n",
"There are many programmes/softwares currently available for visualizing training results. In this notebook, we will be using [Tensorboard](https://www.tensorflow.org/tensorboard/get_started), which is free to use and already built into the Tensorflow Library.\n",
"\n",
"![](\"https://github.com/tensorflow/tensorboard/blob/master/docs/images/quickstart_model_fit.png?raw=1)