OpenCV Java Object Detection

In this article, OpenCV Java Object Detection demonstrates how to identify specific objects in a real image using OpenCV DNN(Deep Neural Network) module, powered by the OpenCV inference engine.

Moreover, object detection technology has become a widely used method for identifying specific objects in images from pictures or videos across various fields.

Additionally, OpenCV provides the OpenCV DNN module as an inference engine. Therefore, the goal of OpenCV Java Object Detection is to detect real-world objects from different image sources using both the Mask R-CNN model, trained with TensorFlow, and the YOLO model, implemented in Darknet.

1. OpenCV Java Object Detection Overview

The following figure represents my intended approach to implementing OpenCV Java Object Detection from a programmer’s perspective.

The OpenCV Java Object Detection process works as follows:

• Takes user input for the CNN models and image information.
• Processes object detection based on the provided input.
• Displays the identified objects on the screen.

2. DNN Models For Object Detection

The Mask R-CNN and YOLO models are essential components for implementing OpenCV Java Object Detection. Therefore, see the References section for more details.

Specifically, for the YOLO model, YOLOv4 remains a powerful choice in terms of reusability with YOLOv3, offering an improved model that is still very effective, even though the latest release is YOLOv7.

On the other hand, Mask R-CNN, it is an extension of Faster R-CNN, designed as a deep learning model for both object detection and object segmentation. Mask R-CNN performs object segmentation by predicting a pixel-level mask for each object, allowing you to obtain not only the bounding box but also the exact shape of the object.

2.1 YOLO Model

YOLO (You Only Look Once) is a single Convolutional Neural Network (CNN) structure:

• Optimizes detection speed by dividing the image into grids.
• Predicts the class (type) and bounding box for each grid cell simultaneously.
• Allows real-time, high-speed detection.
• Achieves approximately 43-50% accuracy (Coco dataset) in object detection.

2.2 Mask R-CNN Model

Based on Faster R-CNN, Mask R-CNN offers higher accuracy than YOLO, but at a lower speed.

• Predicts the bounding box, class (object type), and mask (at the pixel level) for each object.
• Provides more detailed predictions by adopting ROI Align.
• Achieves approximately 60-70% accuracy (Coco dataset) in object detection.

3. Download Pre-trained Model

The OpenCV DNN module is an inference engine in OpenCV that supports various frameworks, including:

• TensorFlow (.pb, .pbtxt)
• Caffe (.prototxt, .caffemodel)
• ONNX (.onnx)
• Darknet (YOLO, .cfg, .weights)
• Torch (.t7)
• OpenVINO Model (.xml, .bin)

To use both YOLO and Mask R-CNN models, the OpenCV DNN module requires downloading pre-trained models for both Darknet and TensorFlow.

3.1 Download Pre-trained Model for YOLO

In OpenCV Java Object Detection, the following two files for YOLOv4 must be downloaded:

• YOLOv4 pre-trained weights
• YOLOv4 config

The yolov4.weights file contains the learned parameters of the Darknet model, while the yolov4.config file defines the model’s architecture and training method. Therefore, these two files are required as input parameters for OpenCV’s inference engine.

3.2 Download Pre-trained Model for Mask R-CNN

You can use a model in the Frozen Graph (.pb) format, which is compatible with TensorFlow 1.x and can be loaded into the OpenCV DNN module. Additionally, the latest version of the Mask R-CNN model uses the Inception V2 architecture, released on January 28, 2018. You can download the model from the following site:

• Mask R-CNN frozen model (TensorFlow).
• Mask R-CNN Label.

4.Setting Up OpenCV Java Object Detection

Before diving into OpenCV Java Object Detection, we need to set up the necessary tools and identify the main class to execute. Therefore, it’s important to familiarize yourself with the tools required for development and launch the main program.

4.1 Tools for Downloading

The following tools must be downloaded or installed for the development of OpenCV Java Object Detection:

• JDK 17 or higher: The base SDK for OpenCV Java Object Detection.
• IDE: Apache NetBeans or Eclipse can be used. NetBeans includes the Matisse design tool, which makes building GUI applications easier.
• OpenCV: The Computer Vision library for object detection from various image sources.

You can refer to the previous post for instructions on how to set up Apache NetBeans and OpenCV.

4.2 How to Launch the Program

There are several ways to run the OpenCV Java Object Detection program. For example, you can use development tools like JDK with the command line, Eclipse IDE, or any other Java IDE. However, I will explain how to run it using Apache NetBeans IDE, where I wrote and tested the source code. First, you can download and extract the zip file from the download section to any location on your computer.

a. Open Properties

Right-click on the project named ‘JavaOpenCVObjDetector’ and select Properties.

a. Add OpenCV Java library

You need to add the OpenCV Java library under the Libraries category.

b. Add Native OpenCV library

In the Run category, the ObjectDetectorMain class is our main class to launch. In addition, you need to set the VM options as follows:

-Djava.library.path=where_to_opecv\opencv\build\java\x64

5. Implement OpenCV Java Object Detection

OpenCV’s inference engine, the DNN module, supports various deep learning models. Therefore, the implementation of our OpenCV Java Object Detection is ready using this module, as we have downloaded the model files for both YOLO and Mask R-CNN, as described in Section 3, ‘Download Pre-trained Models.’

5.1 Process of OpenCV Java Object Detection How-to

We will review the steps and methods involved in the process before implementing OpenCV Java Object Detection.

a. User Input:

Set up the model, configuration file, and image file from user input.

b. Load CNN Model:

Use different methods depending on the model type:

• readNetFromTensorflow → Load Mask R-CNN model
• readNetFromDarknet → Load YOLO model

c. Prepare the Blob for CNN Input:

• blobFromImage: Convert the image into a blob, suitable for CNN input.
• VideoCapture: Extract a frame from an image source (e.g., video).

d. Perform Inference with the Blob:

• forward: Run the actual object detection.
• setInput: Pass the blob into the CNN model — the data is now in a format the model can understand.

e. Retrieve the Detection Results:

• Mask R-CNN: Extract Bounding Box and Mask (Segmentation).
• YOLO: Extract Bounding Box Coordinates, Class ID, and Confidence Score.

f. Draw the Detected Objects on the Original Image:

• rectangle: Draw bounding boxes.
• putText: Display Class ID and Confidence Score.
• For Mask R-CNN, render the mask on the image.

g. Save or Display the Output Result Base on User Option:

• imshow: Display the result on the screen.
• VideoWriter: Save the final detected image (optional).

5.2 Configuring Output Layers for Models

First, we must define or specify which layer will serve as the output layer before processing object detection. The output layer refers to the layer that we need to identify.

In contrast, the YOLO model can automatically detect its output layers, whereas Mask R-CNN cannot. Therefore, you need to manually add an output layer named detection_out_final if you choose the Mask R-CNN model.

5.2.1 YOLO Model Output Layers

The YOLO model can detect predicted values from its connected layers. As a result, we can obtain detection outputs even if we do not explicitly define an output layer, since the predicted values are derived from these connected layers.

• YOLO predicts objects across multiple output layers, requiring us to select the predictions with the highest confidence scores. To produce the final detection results, we apply the Non-Maximum Suppression (NMS) technique, which eliminates redundant and overlapping bounding boxes.

• The output results can be extracted directly without explicitly defining output layers, even though necessary information can be found across disconnected layers.

5.2.2 Output Layers of the Mask R-CNN Model

The Mask R-CNN model performs object detection along with object segmentation.

This model processes two layers simultaneously:

• detection_out_final: Bounding boxes and confidence scores of detected objects.
• detection_masks: Masks for each detected object.

The detection_out_final output layer contains bounding boxes and class information, which identify the locations and types of objects. However, this layer is not considered the output layer for mask predictions. On the other hand, the detection_masks layer is responsible for predicting the masks. This means that mask prediction is crucial for the Mask R-CNN model.

5.2.3 Structure of the detection_out_final Layer

This layer is composed of a 4D array, and the description of each axis is as follows:

a. Meaning of each dimension:

• First axis (Batch size): Represents the batch size. Mask R-CNN typically processes one image at a time, so the batch size is usually 1.
• Second axis (Number of Channels): Represents the number of channels, which is not used in this context.
• Third axis (Number of Objects): The maximum number of detected objects. Mask R-CNN supports a maximum of 100 objects for detection.
• Fourth axis (Attributes of Objects): Contains information about each detected object, including 7 values:
  • detection_out_final[0, 0, i, 0]: Confidence score
  • detection_out_final[0, 0, i, 1]: Class ID
  • detection_out_final[0, 0, i, 2]: Confidence score (same as the one in index 0)
  • detection_out_final[0, 0, i, 3]: Bounding box (x1)
  • detection_out_final[0, 0, i, 4]: Bounding box (y1)
  • detection_out_final[0, 0, i, 5]: Bounding box (x2)

5.2.3 Structure of the detection_masks Layer

The structure of the detection_masks layer is a 4D array, similar to the detection_out_final layer. This layer provides object masks, which are an important output. To better understand its function, let’s take a closer look at the structure of the detection_masks layer and its dimensions.

a. Background

This layer contains binary masks of the detected objects in the Mask R-CNN model. The binary mask array represents the object within the bounding box at the pixel level. A value of 1 indicates the presence of the object, while 0 represents the absence of the object.

b. Structure of the detection_masks Layer

The structure of the detection_masks layer is a 4D array with a general shape of (1, 1, 100, H, W):

• First axis (Batch size):
  • Typically, one image is processed at a time, so this is generally set to 1.
• Second axis (Number of channels):
  • This represents the number of channels, which is usually set to 1.
• Third axis (Number of detected objects):
  • This axis represents the maximum number of detected objects. The Mask R-CNN model supports a maximum of 100 objects for detection.
• Fourth and fifth axes (Size of Mask: H x W):
  • These axes represent the size of the mask for each object. The mask indicates the exact bounding box of the object, and the size (H, W) is smaller than the original image size.

5.3 Input Parameters of blobFromImage

Typically, the readNetFromDarknet method is used for the YOLO model, while the readNetFromTensorflow method is used for the Mask R-CNN model, both of which require a CNN configuration.

Additionally, a blob is a type of parameter for the CNN and is one of the key parameters of the blobFromImage function.

Method signature:

Let’s find out the general input size of image fit used by YOLO or Mask R-CNN model .

5.3.1 Arguments of blobFromImage for YOLO Model

• scalefactor=1/255.0: Normalizes pixel values to a range between 0 and 1.
• size=(416, 416): Adjusts the image size for the YOLO model.
• swapRB=true: Converts the image to RGB, as OpenCV uses BGR order by default.
• crop=false: Resizes the image while keeping the original aspect ratio (no cropping).

5.3.2 Arguments of blobFromImage for Mask R-CNN Model

• scalefactor=1.0: No normalization of pixel values (keeps original scale).
• size=(1024, 1024): Adjusts the image size. Common sizes are 800×800 or 1024×1024.
• swapRB=true: Converts the image to RGB, as OpenCV uses BGR order by default.
• crop=false: Resizes the image while maintaining the original aspect ratio.

5.4 YOLO Model Size Parameters Explained

The size parameter of the YOLO model defines the input dimensions of the Deep Neural Network (DNN), typically represented as (width, height). As a result, you must resize your images to match the model’s expected input size, which was defined during the training of the Convolutional Neural Network (CNN).

For example:

• YOLOv4 → (416, 416)
• SSD MobileNet → (300, 300)
• ResNet → (224, 224)

Using the correct input size is crucial for achieving accurate results. Furthermore, in YOLO models, the input size is typically fixed and determined by the model’s architecture.

• The default size for YOLOv3 and YOLOv4 is 416×416, while a size of 608×608 is also available for YOLOv4.
• Starting from YOLOv5, the default size is 640×640.

5.5 Mask R-CNN Model Size Parameters Explained

• Mask R-CNN with the COCO Dataset:
• Mask R-CNN supports various input sizes, but sizes of (800, 800) or (1024, 1024) are commonly used. The exact input size is specified by the Mask R-CNN model type.
• For certain models, 1024×1024 is used as the input size.
• Research has shown that using a 1024×1024 image can lead to higher accuracy.

6. Class Design for OpenCV Java Object Detection

Now that we have an understanding of both the YOLO and Mask R-CNN models for the OpenCV DNN module, we can move on to the classes involved in our OpenCV Java Object Detection. Therefore, I believe six classes are necessary for the implementation of our OpenCV Java Object Detection.

6.1 Class Diagram and Associations

Here is my concept of the classes for OpenCV Java Object Detection.

The responsibilities of each class are described as follows:

• YoloDetector: The implementation class for object detection using the YOLO model. It is inherited from ObjectDetector.
• DNNOption: The user input class that accepts files and options for detection.
• ObjectDetectorMain: The main class that contains the entry point the user must run.
• IObjectDetector: The interface class that defines the detectObject method, which subclasses must implement.
• ObjectDetector: The parent class of both YOLO and Mask R-CNN implementations, which has an abstract detectObject method.
• MaskRCNNDetector: The implementation class for object detection using the Mask R-CNN model. It is inherited from ObjectDetector.

7. User Input UI Layout

Using the OpenCV DNN module requires several parameters, which vary depending on the selected model for the OpenCV Java Object Detection implementation. Initially, I considered using a command-line parser to gather input from the user, but this would make the program more complicated. Therefore, I decided to create a simple window for user input. Afterward, we will focus on the methods provided by OpenCV.

Next, we will discuss the components and their associated variables, along with a description table.

7.1 Implementation of User Input Layout

You can save the parameters from the user input GUI to the DNNOption class for later use. In this case, the role of this class is to store the user input.

This is the GUI implementation:

setUpOption – The main method is based on the showConfirmDialog method of JOptionPane and uses JPanel for user input.

The JPanel uses GridBagLayout for user input. All information entered by the user can be saved when the “OK” button is clicked, as shown below:

Next, here is the full source code of the setUpOption method.

7.2 OpenCV Java Object Detection Main Method

The ObjectDetectorMain class contains the main method for OpenCV Java Object Detection. Specifically, this class uses the DNNOption class to store the user input parameters.

Here is the source code:

It calls the setUpOptions method for user input and performs a validation check on the options.

The MaskRCNNDetector class is instantiated if the model selected by the user is Mask R-CNN, while the YoloDetector class is instantiated if the user selects the YOLO model. In either case, each class must implement the detectObject method inherited from the IObjectDetector interface.

Based on the modelName selected by the user, the appropriate class is instantiated.

Once the class instantiation is complete, the detectObject method of the implementation class is called.

8. ObjectDetector The Parent Class

The ObjectDetector class is an abstract class that serves as the parent class for both the YOLO and Mask R-CNN models. It implements the IObjectDetector interface, which contains the detectObject method as an abstract method

Below is the full source code of the ObjectDetector class:

8.1 Setting Output Layers for Models

The getMapOutputLayer method implements the section in “5.2 Configuring Output Layers for Models” The method selects the appropriate output layer based on the model, as follows:

8.1.1 Setting Output Layers for YOLO

This method saves an unconnected output layer in the model’s CNN network.

The signature of the getYoloOutputLayer method is as follows:

The method searches for an unconnected layer among all layers in the CNN network.

The implementation code is as follows:

8.1.2 Setting Output Layers for Mask R-CNN

The getMaskRCnnOutputLayer method implements the section “5.2.2 Output Layers of the Mask R-CNN Model.” Specifically, this method saves the unconnected output layers and adds the detection_out_final layer to the model’s CNN network. The key point of this method is adding the detection_out_final layer, as both the detection_out_final and detection_masks layers are used in the Mask R-CNN model.

The method signature is:

The method uses a Net object as a parameter and returns a map containing layer names and Mat objects. In this process, the current status of the Mat object is set to null because the blobFromImage method is called, which applies object detection.

This is source code of the method :

8.2 CNN Net Initialization

The readDNNetByModelName method creates a DNN from the model and configuration files based on the user input.

The method signature is:

Implementation code :

The readNetFromDarknet method is used for YOLO, and the readNetFromTensorflow method is used for Mask R-CNN during DNN generation.

The rest of the code handles the selection of CPU or GPU for processing.

9. Setting up Labels and Colors

The CocoLabel class stores the appropriate label data based on the selected model and assigns a randomly generated color to each label. Therefore, the class is instantiated when the setupLabelSet() method is called in the constructor of the ObjectDetector class, as shown in the following code:

9.1 Label Initialization

“The setupLabelSet method selects the YOLO_LABEL_FILE option for the YOLO model; otherwise, it selects the MASK_RCNN_LABEL_FILE option for the Mask R-CNN. Additionally, both YOLO_LABEL_FILE and MASK_RCNN_LABEL_FILE are parameters of the setupCocoLabelSet method in the CocoLabel class.

9.2 Save Label and Color

The setupCocoLabelSet method saves a List object containing both COCO_LABELS and the labels read from the label data file.

9.3 Label File

Each detected object has a class ID, which is used to identify the object during the training process. Additionally, there are two different label files for each model: coco.names for YOLO and mscoco_labels.names for Mask R-CNN. These are the files used as labels for the respective models.

Make sure to use the appropriate file for each model to ensure proper functionality.

Below is a sample label file that saves the labels and their corresponding class IDs.

9.3 Colors and Labels for Bounding Box

The following code describes how the labels are matched with colors for the bounding box. Specifically, these colors are randomly generated when the label data is first read from the file.

You can use the method as follows:

10. Object detection in Mask R-CNN Model

The mediaType can be either ‘Image’ or ‘Video’. The ‘Image’ type calls the detectObjectOnImage method, while the ‘Video’ type calls the detectObjectOnVideo method.

10.1 User Input Values

The user must fill in the following information: Select ‘MRCNN’, choose either ‘Image’ or ‘Video’, and select either ‘GPU’ or ‘CPU’ in the respective combo boxes. Based on the selected file type (‘Image’ or ‘Video’), choose the corresponding file for the ‘InFile’.

Here is the description of each component:

• Model: The file path for frozen_inference_graph.pb. You can select this file by clicking the Find button.
• Config: The file path for mask_rcnn_inception_v2_coco_2018_01_28.pbtxt. You can select this file by clicking the Find button.
• InFile: The media file path, which can either be an image or a video file depending on what you select from the first combo box.
• Save Output: A flag that determines whether to save the result or not. The output will be saved in the ‘out’ folder within the image or video file path selected by the user.
• Shading Mask: A flag that determines whether to shade masks on the original image. If unchecked, the masks will not be shaded on the original image.

You can get some test images from the Reference section.

a. Rendering masks, labels, and scores will be displayed when the checkbox is checked.

b. Labels will be shaded on the image only when the checkbox is checked.

10.2 detectObject Method in Object Detection

This method implements the detectObject method from the IObjectDetector interface.

In the case of ‘Image’, the method calls detectObjectOnImage, while in the case of ‘Video’, it calls detectObjectOnVideo, based on the mediaType variable selected by the user.

10.3 detectObjectOnImage for Image Detection

This is more streamlined while still effectively communicating the core idea. Therefore, the method performs object detection on image inputs.

The method signature is as follows:

Description of parameters:

• winName: The title of the window.
• dnnNet: The CNN network loaded from the readDNNetByModelName method.
• outputLayers: The list of output layers returned by the getMaskRCnnOutputLayer method.
• mediaFile: The path to the media file (image or video).
• isSaveOutput: A flag indicating whether to save the results.
• isMaskShading: A flag indicating whether to draw masks on the result.

The full source code is as follows:

10.4 Resizing The Image Source

The optimal image size for the Mask R-CNN model is 800; therefore, the input image may be resized if its original size is out of range.

10.4.1 Resize Mat Object

The resizeMatImage method resizes the source image to fit the Mask R-CNN model’s optimized input size of 800. Additionally, the method adjusts the image size accordingly, filling any extra space with padding to ensure the image meets the required dimensions if its original size is either too large or too small.

The method returns a HashMap object containing the following items:

• Adjustment Scale: The scale factor used to resize the original image to the new size.
• Resized Mat Object: The resized image that has been adjusted to the target size.
• Padding Information: Information about the padding applied to the image to maintain the aspect ratio.

10.5 Find Blob for Mask R-CNN Model

Now, you can use the blobFromImage method to create a blob as the input for the Mask R-CNN model using the resized Mat object, adjusted to an 800-pixel size.

The information about the detected objects is now saved after the forward pass of the OpenCV DNN module.

We have declared the saved information, namely detection_out_final and detection_masks, as output layers, as described in section 5.2.2 ‘Output Layers of the Mask R-CNN Model.’

Here are sample representations of both the detection_out_final and detection_masks layers:

a The detection_out_final layer

b. The detection_masks layer

The dimensional information for both layers, as shown above, has already been described in sections 5.2.3 (‘Structure of detection_out_final layer’) and 5.2.4 (‘Structure of detection_masks layer’).

10.5.1 Obtain Information of detection_out_final

We can obtain bounding boxes and confidence scores from the detection_out_final layer, and this information is used to generate bounding boxes and labels on the original images.

The following outlines the steps and code involved in this process.

a. Reshape the Mat object by 7 groups.

b. Assign the confidence value, convert the Mat data to 1D array.

c. Draw the detected objects.

The following code selects bounding boxes with a default threshold value of 0.5 and draws the boxes on the image. You can adjust the threshold value if necessary.

10.5.2 Drawing Bounding Box

In this final step, we draw bounding boxes using either the drawMaskedBBox or drawBoxSimple method. The choice of method is determined by the boolean value isMaskShading, which is set based on the user’s input. For example, Specifically, if isMaskShading is true, the drawMaskedBBox method is used; otherwise, drawBoxSimple is applied.

a. drawMaskedBBox method
This method draws bounding boxes, masks, and labels on the detected objects.

The output is shown in the following figure:

b. Simple Bounding Box Drawing

This method draws bounding boxes around the objects without applying masks or labels.

The output is slightly different from the previous one:

10.6 Object Detection in Video

The detectObjectOnVideo method detects objects in a video in which The same detection technique is applied. However, the key difference is that the video is processed frame by frame. Specifically, each frame is converted to a Mat object, and then objects are detected individually in each frame.

Next, you can refer to the following skeleton code to extract frames from a video file, one by one.

The result is :

a. Video rendering with OpenCV, displaying labels, bounding boxes, and masks.

b. Video rendering with OpenCV, displaying labels and bounding boxes without masks.

10.6.1 Threshold Adjustment

In the figure below, we can see a bird identified as a false detection because the confidence score is approximately 0.65 or lower. Therefore, to improve the result, we can adjust the threshold from 0.5 to 0.655, as shown in the following figure.

Now, we have a clean result that no longer contains the falsely detected object, the bird.

10.6.2 Check Out of Range ROI

We also need to check whether the ROI(Region of Interest) is out of range when drawing masked bounding boxes. For example, if the ROI is out of range and not checked, you may encounter an error like this :

The following getMrcnnROI method can eliminate this error:

11. Object detection by YOLO Model

In this section, we explore the object detection technique using the YOLO model in our OpenCV Java Object Detection.

Similar to the Mask R-CNN approach, detection can be performed using either the detectObjectOnImage or detectObjectOnVideo method, depending on the mediaType value provided by the user, which must be either ‘Image’ or ‘Video’.