Background
In the past few decades, researchers have used canopy surface temperature combined with microclimate measurements for detecting plant biotic and abiotic stress and yield estimation (Ihuoma and Madramootoo, 2017). Fruit surface temperature is an important parameter in fruit loss management due to heat damage (i.e. sunburn) during pre- and post-harvest produce handling and management (Khanal et al., 2017). Other applications include, but are not limited to, frost monitoring for effective actuation of management methods, crop and fruit pest and disease monitoring and prediction, monitoring in fruit storage rooms, and high throughput phenotyping. Surface temperature measurements can be carried out using either thermal imagers or infrared thermal (IRT) radiometers. Currently, the prevalent method for the measurement of plant canopy surface temperature is infrared thermometry using IRT sensors. However, the limitations of IRTs and recent advancements leading to less expensive, high-resolution thermal cameras (Osroosh and Peters, 2018) have encouraged researchers to find a wide range of applications for thermal imagery in agriculture.
Plant breeding programs are among those in the need of affordable thermal-spectral imaging systems for high throughput phenotyping. Normally, very expensive yet delicate thermal and spectral cameras worth tens of thousands of dollars are taken to the field to take pictures. Sometimes, the expensive hardware is returned damaged due to the nature of such trips. MATLAB is the main software in use across disciplines for computer vision and image processing needs. MATLAB is, however, very expensive for non-students. Very often, these experiments need to be repeated because the images are not good enough. Even when everything goes well, it might take weeks to process all captured images.
Therefore, it is crucial to find solutions to obstacles in widespread adoption of thermal imaging in agriculture such as cost, complicated image processing and unavailability of commercial imagers suitable for continuous field measurements. There is a need for cameras that are capable of capturing images, and can automatically convert them to information in real-time. This would help scientists or growers with no or little experience in image processing to use them in research and field scouting. In the following paragraphs, I have discussed some of the most challenging aspects of conventional thermal imaging in agriculture, possible solutions, and my research and development efforts to effectively address these issues.
Challenges of Thermal Imaging in Agriculture and Solutions
Unavailability of affordable, reliable hardware for continuous monitoring
A simple literature review will reveal that thermal imagers have been rarely used for continuous unattended canopy measurements, and almost all related studies have reported occasional field measurements. Almost all commercially available thermal cameras are expensive and designed for non-agricultural applications such as security business and industrial machine vision applications. Despite their high price, these cameras are often unable to meet the expectations of the agricultural community. For example, they are not appropriate for unattended field applications in agriculture due to high power consumption and associated risks (e.g. getting stolen or damaged). A thermal imaging hardware developed and optimized for agricultural applications can significantly reduce the expenses and save the hassle (Solution).
Difficulty of identifying plants in thermal images
In many agricultural applications, we need to identify a specific target (e.g. leaves) in thermal images. In order to do that, we need to remove the unwanted background (e.g. soil, water, artificial objects). In the conventional thermal imaging; however, there is no standard way for background removal and researchers often need to improvise. One effective way to cope with this problem is to combine thermal imagery with another type of imaging such as visible or multispectral (Solution).
Unavailability of affordable, convenient software for image processing
At their best and at any price range, commercial thermal cameras can only provide unprocessed thermal images. Sometimes thermal images are accompanied with RGB images as well. To extract useful data, thermal images need to be processed using a software in the lab, which might take weeks. However, software packages such as MATLAB can be very expensive. In addition, majority of people in the agricultural research community are unfamiliar with image processing basics and do not know how to extract useful data from thermal images. This is why we often need even more expensive software to process captured images or give them to a computer vision expert for analysis. A software package that is optimized for agricultural applications can minimize the required level of expertise, and save time and money (Solution).
Big data problem
One of the biggest challenges in today’s digital agriculture is the large amount of unprocessed data that is produced. Big data is always perceived as something that keeps data scientists busy and customers happy. However, the reality is that most growers do not appreciate unprocessed data, especially in the from of high-resolution images. Companies often provide unprocessed images along with a software package and leave it to the user (customer or consultant) to make sense of images. The images are usually taken by drones or satellites, which means they are not real-time, and it might take weeks to extract useful information from them. Automation and image processing on-board thermal imaging system can provide the user only with processed, on-time information that they can use in decision-making (Solution).
Economical Thermal-RGB Imaging System for Continuous Monitoring of Crops
Osroosh et al. (2018) developed a robust and low-cost thermal-RGB imaging system (Fig. 1)optimized for in-field agricultural applications. The system can be used for continuous spatial and temporal monitoring of crops at a fixed position or aboard moving systems (e.g. center pivot machine). The imager has a Raspberry Pi SBC at its core, takes and overlays thermal and RGB images in real-time, has a weatherproof 3D printed enclosure,and can pull data from a microclimate unit wirelessly. A blackbody calibrator (BB701, Omega Engineering, Norwalk, CT) was used to calibrate individual thermal modules.
The design of the imager allows for creating a star network of imaging units in the field to monitor plant canopies in real-time. A power management panel was specifically designed for the imager to turn them on/off at specified times of day to save power and allow for unattended continuous monitoring in the field for a long period of time. The cost of assembling one unit (only hardware) was ~$400 in 2017.
Figure 1. Thermal-RGB imager and its components (Osroosh et al., 2018). The electronics were protected by a weatherproof 3D printed enclosure. A blackbody calibrator (BB701, Omega Engineering, Norwalk, CT) was used to calibrate the thermal module.
​They also developed a computer vision algorithm (Fig. 2) to extract the surface temperature of the target (e.g. leaves, fruits, animals) from captured images. The algorithm uses RGB images to identify the desired target in a thermal image and separate it from the background. It then calculates the average temperature of all the remaining image pixels that delineate the target.
Figure 2. Image processing algorithm. The algorithm extracts the surface temperature of specific target (e.g. leaves, fruits, animals) from captured images.
Mounting imagers on a center pivot irrigation system
Osroosh et al. (2018) mounted two units of the aforementioned thermal-RGB imager on a center pivot irrigation system in a mint field near Toppenish, WA (Fig. 3). The center pivot was retrofitted with Medium Elevation Spray Application (MESA) and Low Elevation Spray Application (LESA) systems. The computer vision algorithm was used to extract sunlit leaf temperatures and canopy cover percentage from images (Fig. 4). The system survived the harsh weather of central Washington during an entire growing season.
Figure 3. Thermal-RGB imager mounted on center pivot irrigation system
Figure 4. Sample of captured RGB image in a mint field under center pivot irrigation. RGB images were used to detect the target (e.g. plant leaves) and remove soil background from thermal images (Osroosh et al., 2018).
Figure 5. Thermal-RGB imagers were used to monitor cherry fruits surface temperature and wetness.
Apple and cherry fruit skin temperature and wetness monitoring
​Osroosh and Peters (2019) determined the feasibility of thermal-RGB imaging for detecting cherry fruit surface wetness onset and duration as a decision aid tool for efficient rainwater removing to prevent fruit cracking. They carried out an experiment in plots of a cherry orchard (Fig. 5). They used a rain simulator to artificially wet cherry fruits. The in-field sensing setup included two custom-built thermal-RGB imagers, a microclimate-measuring unit and two leaf wetness sensors (as reference). They developed and used a computer vision algorithm to identify cherries in the images, extract fruit surface temperature, and related the temperature change to fruit surface wetness (Fig. 6).
Figure 6. Sample of captured RGB image in a cherry orchard. RGB images were used to detect cherry fruits and remove the background (leaves and objects) from thermal images (Osroosh et al., 2019).
In another experiment, our team investigated the potential of using the custom-built thermal-RGB imaging system for monitoring apple fruit skin temperature as a decision aid tool to prevent apple sunburn. The performance of the thermal-RGB imaging system was compared with an extensive (~$40,000) high-resolution thermal camera. The results of the research showed that the inexpensive imager could detect the critical skin temperatures with almost the same accuracy.
Multimodal Thermal-Multispectral Imaging System (BINA Pro)
Our previous research and development on affordable thermal-RGB imaging systems for agricultural applications laid foundation for the development of a portable multi-sensor, multi-band thermal-spectral imaging system (Model: BINA Pro, DurUntash Lab, San Diego, CA) (Fig. 7).
Figure 7. BINA Pro multi-sensor, multi-band crop analysis system combines thermal- spectral imaging with microclimate and range data to generate real-time image layers (maps), and to calculate useful plant parameters such as NDVI, transpiration and stomatal conductance.
The BINA Pro is specifically designed and optimized for agricultural applications. The hardware of this computer vision-based system combines thermal and multispectral imagery and pulls data from a microclimate unit to calculate a variety of useful plant parameters. The BINA Pro can overlay and display up to three image layers (e.g. raw multispectral image, raw thermal image, ΔT, NDVI, CWSI, actual transpiration and stomatal conductance) in real-time. Like its predecessors, the BINA Pro relies on the Raspberry Pi single-board computer. The BINA Pro comes with an onboard, powerful software (graphical user interface) that allows for both manual and automatic image processing. Embedded algorithms combine biophysical and empirical crop models, computer vision and soft computing to meet the needs of different applications. It stores raw and processed images, and exports resulting data in ‘csv’ format with the click of a button. The target can be as small as part of a leaf, a number of leaves or entire canopy.
Other key features and specifications of the system are as the following:
100% on-board, real-time and automatic image processing
Calculates surface temperature of the target by automatically detecting and removing background from thermal image
Automatic and real-time overlaying and alignment of images by measuring distance from the target
Pulls data from a microclimate unit (SDI-12 port); calculates canopy cover percentage, canopy height, reference ET, crop coefficient (Kc), and crop ET (ETc)
Automatic image adjustments for change in light source
Thermal pixel resolution of 160 x 120 (19,200 pixels); thermal measurement accuracy of ±0.5 °C (after calibration)
Multispectral image resolution of up to 8 MP
Combined thermal and multispectral imaging
​Unlike the previous model of the imaging system where thermal and visible bands were combined, the BINA Pro relies on multispectral images to detect the target and remove the background (Fig. 8). The added NIR band improves the ability of the onboard image-processing algorithm in distinguishing dead plants and non-organic objects from healthy leaves (Fig. 9). The user can also benefit from raw multispectral or NDVI images that are automatically generated and stored.
Figure 8. The BINA Pro combines thermal and multispectral bands, and can automatically generate NDVI and single multispectral band (e.g. Red or NIR) images.
Figure 9. Unlike the previous model of the imaging system, the BINA Pro uses multispectral images to detect the target and remove the background.
Automatic overlaying of three image layers
​The BINA Pro application supports automatic overlaying and display of up to three image layers in real-time (Fig. 10). The image layers can be one of the following: surface temperature (thermal image), canopy and air temperature difference (ΔT), raw visible bands, NIR band, raw multispectral image, NDVI (different indexes), crop water stress index (CWSI), stomatal conductance (SC), and actual transpiration (Ea). For information about these models please refer to articles by Osroosh et al. (2014, 2015a, 2015b, 2016a, 2016b). Any other image layer (e.g. photosynthesis) that can be generated based on the available raw input data (e.g. thermal-spectral, range, microclimate) can be easily added to the software.
Figure 10. The BINA Pro application supports automatic overlay and display of up to three image layers in real-time.
Automatic and manual image alignment
​Because of the distance between the center of camera modules and their different field of views, captured thermal and spectral images are not aligned by default. This is why a mathematical model (calibrated) is used to automatically align thermal and multispectral images (Fig. 11). The model requires distance (from the target; maximum ~10 m) data, which is measured by an embedded ultrasonic range finder. The application also allows for manual image alignment.
Figure 11. A calibrated mathematical model is used to automatically align thermal and multispectral images.
Automatic background removal
​There are different ways to detect a target of interest in an image ranging from simple color-based to more accurate machine/deep learning algorithms (e.g. convolutional neural network - CNN). The BINA Pro software relies on an automatic image segmentation algorithm based on NDVI values calculated for image pixels to separate plant leaves and canopies from their background (e.g. soil, water). The user can set upper and lower NDVI thresholds that are used to remove the background from multispectral images. We are currently working on improving the process by developing and trying different machine/deep learning-based methods for semantic and instance image segmentation. This will allow the system to cover a wider range of application with a much higher accuracy.
Figure 12. The BINA Pro software relies on an automatic image segmentation algorithm based on NDVI values calculated for image pixels to separate plant leaves and canopies from their background.
Automatic detection of hottest and coldest points
An algorithm is developed to measure and display maximum and minimum values of target (moving or stationary surface temperature in real-time (Fig. 13). This feature allows the user to quickly detect the points of stress in the image with no effort.
Figure 13. The system automatically detects the hottest and coldest points in the segmented thermal image.
Communication with a microclimate unit
​The BINA Pro is programmed to pull microclimate data (air temperature, relative humidity, solar radiation and wind speed) from a commercially available unit (Model: ClimaVUE50, Campbell Scientific, Logan, UT). Currently, only wired communications (SDI-12) with the unit is supported (Fig. 14). Depending on the availability, an API can potentially replace the in-field microclimate unit and allow for pulling data from any other commercial weather station.
Figure 14. The BINA Pro has a SDI-12 port and can pull data (air temperature, relative humidity, solar radiation and wind speed) from a commercially available microclimate measuring unit (Model: ClimaVUE50, Campbell Scientific, Logan, UT).
Calculation of crop coefficient
The BINA Pro automatically reports the canopy cover percentage (Cc, %) in the spectral image by segmenting the image into soil and vegetation pixels and calculating the vegetative fraction value. It also calculates crop coefficient (Kc) and crop ET (ETc) by using polynomial equations that define Kc as a function of Cc (Kc = f(Cc)). These equations (Fig. 15) are available in the literature (Bryla et al., 2010) for a variety of crops, and can be developed for others.
Figure 15. To calculate crop coefficient (Kc) and crop ET (ETc), the system uses polynomial equations that define Kc as a function of Cc (Kc = f(Cc)). These equations are available for a variety of crops, and can be easily developed for others.
System output and meta data
​The BINA Pro stores raw and processed images. It also automatically calculates a wide range of useful plant parameters and exports the results of on-board processing along with meta data to a ‘csv’ file (Fig. 16). The unit is configured as FTP server allowing the user to connect using its IP address and download raw images and available csv files. As long as there is WiFi connection, the user can also access the desktop and application remotely from PC using the VNC Viewer app (Cambridge, UK). This feature is inherited from the Raspberry Pi.
Figure 16. The BINA Pro automatically calculates a wide range of useful plant parameters and writes the results of on-board processing along with meta data to a ‘csv’ file.
References
BINA Pro Multi-Sensor Thermal-Multispectral Crop Analysis System, 2019. https://www.envitronicslab.com/bina-pro
Bryla, D.R., Trout, T.J., Ayars, J.E., 2010. Weighing lysimeters for developing crop coefficients and efficient irrigation practices for vegetable crops. HortScience, 45(11):1597-1604. https://doi.org/10.21273/HORTSCI.45.11.1597
Ihuoma, S.O., Madramootoo, C.A., 2017. Recent advances in crop water stress detection. Comput. Electron. Agr., 141: 267–275.
Khanal, S., Fulton, J., Shearer, S., 2017. An overview of current and potential applications of thermal remote sensing in precision agriculture. Comput. Electron. Agr., 139: 22–32.
Osroosh, Y. et al., 2019. Detecting fruit surface wetness using a custom-built low-resolution thermal-RGB imager. Computers and Electronics in Agriculture, 157: 509-517.
Osroosh, Y. et al., 2018. Economical thermal-RGB imaging system for monitoring agricultural crops. Computers and Electronics in Agriculture, 147: 34–43.
​
Osroosh, Y., Peters, R.T., Campbell, C., Zhang, Q., 2016b. Comparison of irrigation automation algorithms for drip-irrigated apple trees. Computers and Electronics in Agriculture, 128: 87–99.
​
Osroosh, Y., Peters, R.T., Campbell, C., 2016a. Daylight crop water stress index for continuous monitoring of water status in apple trees. Irrigation Science, 34(3): 209–219.
​
Osroosh, Y., Peters, R.T., Campbell, C., Zhang, Q., 2015b. Automatic irrigation scheduling of apple trees using theoretical crop water stress index with an innovative dynamic threshold. Computers and Electronics in Agriculture, 118: 193–203.
Osroosh, Y., Peters, R.T., Campbell, C., 2015a. Estimating potential transpiration of apple trees using theoretical non-water-stressed baselines. Journal of Irrigation and Drainage Engineering, 141(9): 04015009.
Osroosh, Y., Peters, R.T., Campbell, C., 2014. Estimating actual transpiration of apple trees based on infrared thermometry. Journal of Irrigation and Drainage Engineering, 141(8): 04014084.
Osroosh, Y., 2020. Do plant-based irrigation scheduling methods work? EnviTronics Lab, Jun 24.
Osroosh, Y., 2020. Challenges of conventional thermal imaging in agricultural applications. EnviTronics Lab, Sep 15.
Osroosh, Y., 2020. Can thermal imaging be used for crop loss and water management? EnviTronics Lab, Oc 17.
Osroosh, Y., 2020. Economical thermal-RGB imaging system for in-field continuous monitoring of crops. EnviTronics Lab, Oct 25.