Deep Learning in UAV images — A Brief Review
Introduction
The application of deep learning algorithms on data obtained by UAVs is increasingly generating products and research that seek to automate processes and extract increasingly useful insights to users. In this article we will see some of the main areas that have benefited from this combination, such as precision agriculture, renewable energy, smart cities, etc.
First, let’s see the main tasks that involve deep learning algorithms in order to extract information from images:
Deep Learning Tasks
Some computer vision models can detect objects, determine their shape and predict the direction in which they will travel. For example, such models are at work in self-driving cars. Three important tasks undertaken by computer vision are image classification, object detection and image segmentation.
Image Classification
Classification is a machine learning task for determining which objects are in an image or video. It refers to training machine learning models with the intent of finding out which classes (objects) are present. Classification is useful at the yes-no level of deciding whether an image contains an object/anomaly or not.
Object detection
Object detection combines classification and localization to determine what objects are in the image or video and specify where they are in the image. It applies classification to distinct objects and uses bounding boxes, as shown below.
Object detection is useful in identifying objects in an image or video. Below, the image on the left illustrates classification, in which the classes Donut and Coffee are identified. The image on the right illustrates object detection by surrounding the members of each class — donut and coffee — with a bounding box.
Use cases for object detection include facial detection with any post-detection analysis; for example, expression detection, age estimation or drowsiness detection. Many real-time object detection applications exist for traffic management, such as vehicle detection systems based on traffic scenes.
As described above, the most popular approaches to computer vision are classification and object detection to identify objects present in an image and specify their position. But many use cases call for analyzing images at a lower level than that. That is where image segmentation comes in.
Image segmentation
Any image consists of both useful and useless information, depending on the user’s interest. Image segmentation separates an image into regions, each with its particular shape and border, delineating potentially meaningful areas for further processing, like classification and object detection. The regions may not take up the entire image, but the goal of image segmentation is to highlight foreground elements and make it easier to evaluate them. Image segmentation provides pixel-by-pixel details of an object, making it different from classification and object detection.
Below, the image on the left illustrates object detection, highlighting only the location of the objects. The image on the right illustrates image segmentation, showing pixel-by-pixel outlines of the objects.
Agriculture and Forestry
The use of drones to obtain high-resolution images of agricultural crops has made a breakthrough in precision agriculture possible. The analysis went from the field level, to the plant level, and even to the leaf level. We now have Millions of pixels in a Drone image, making it possible to feed Deep Learning algorithms to analyze the image and extract a huge amount of information from the image. Let’s list some of the main applications:
Crop segmentation
One of the most interesting applications is crop segmentation in Drone images. Traditional techniques even manage to separate the vegetation and soil pixels, but as the separation is per pixel, the crop class is often linked to invasive vegetation. Using convolutional networks, it is possible to train the algorithm to separate this type of vegetation by other characteristics such as shape, texture, spectral curve, etc.
In the example below we have a segmentation between Sugarcane and Soil:
Plant Detection and counting
Another task that has become very important in precision agriculture is plant detection and counting. Object detection architectures, such as FasterRCNN, YOLO, RetinaNet applied to Drone images, make it possible to extract this information with great precision.
It is also useful to separate detected plants into similar classes, by some interesting characteristic such as leaf area, volume, NDVI mean, or some other spectral index.
With the geographic coordinates of each plant obtained, it is possible to create an infinity of analyses, identifying problems at the plant level and no longer at the field level. In addition, the amount of information that can be extracted and stored from a field is immense, allowing correlations with productivity, use of water, fertilizers, etc. From these correlations, the farmer or agronomist can perform localized actions, reducing costs.
Weed detection
Just as the separation between soil and vegetation is important, the detection and segmentation of weed or invasive plants is very useful to avoid competition for nutrients and water with the main crop.
Crop row detection
Some crops are planted densely in rows. Be more interesting to identify as crop lines. Whether to facilitate the use of harvesting machinery or to analyze dense vegetation.
Urban
Using drone mapping in cities still presents many challenges, just like in large agricultural fields, large cities are difficult to map by drones, but some tasks can be simplified when we combine Deep Learning algorithms to extract important information.
Building segmentation
The segmentation of built-up areas can be performed both in high resolution satellite images and by drones. The main parameters that define the means by which the images will be collected are the analysis area, the cost of the images, among other topics that must be well analyzed at the beginning of the project.
Road segmentation
Highways, roads and streets can also be mapped in order to analyze traffic, find pavement defects, etc.
Cars in Parking
Large parking lots such as malls, stadiums and arenas can also be mapped for vehicle count and flow during a day or event.
Pool detection
The detection of swimming pools is also an example of how the use of artificial intelligence can be useful in tax collection.
Renewable energy
One area that together with AI can bring us huge gains for environmental preservation is the area of renewable energies. Energy generated by the sun, wind, or some other renewable method, allows us to reduce the consumption of energy generated by conventional methods.
Solar panel detection
With the dissemination of solar panels on roofs, it is interesting to create both commercial and public analyzes of locations, such as neighborhoods and cities that have a greater number of panels installed.
Solar panel thermal inspection
It is also important to maintain and detect defects and failures in solar panels, which limit energy production by these equipment. The use of drones with thermal cameras on solar farms allows for a faster way to identify and mitigate these problems.
Soil, Water and Environment Conservation
Satellite mapping is very important in the environmental and conservation analysis of an area or region. Even though we do not have the capacity to replace this analysis with an analysis with Drones, however, using these drones located in a certain area can give us a broader view of some environmental problem that is occurring.
Wild Life Detection
Counting animals of a species in a certain region can give us a total population estimate. This information over time is essential in the preservation of endangered species.
Water Quality
The analysis of water in rivers, lakes, mangroves and coastal areas can be optimized using Deep Learning algorithms applied to drone images. Pollution identification, or water color change can be extracted from images, making possible actions to solve these serious problems.
Conclusion
We have seen that the use of Deep Learning in Drone images allows us to extract a huge amount of information in the most diverse areas. The era of Big Data has also arrived in the geospatial area, enabling the union of satellite data, field data and Drone data in order to improve, automate and predict behaviors and tasks.
If you have a project in mind but still don’t know how you can apply these algorithms that are revolutionizing the world in your images, contact me so we can analyze a way of cooperation.
Follow me or Connect me on LinkedIn:
https://www.linkedin.com/in/jo%C3%A3o-otavio-firigato-4876b3aa/
Thanks!
References:
https://www.sciencedirect.com/science/article/pii/S030324342100163X
