All-in-One Multimodal Computer Vision System

BINA Pro

BINA Pro (2019 - present) is a low-cost portable multi-sensor, multimodal imaging system with the Raspberry Pi single-board computer at its heart. The BINA Pro is designed mainly for agricultural research in high throughput phenotyping and precision agriculture. The unit combines crop models and algorithms, Internet of Things (IoT), and onboard computer vision. It combines thermal and multispectral bands (thermal-VIS-NIR) and pulls data from a microclimate unit to calculate a variety of useful plant parameters including crop canopy cover and crop coefficient. The unit has the ability to carry out automatic distance-based image registration, target detection (e.g. plant leaf) and background removal, and displays up to three image layers such as plant surface temperature, NDVI, CWSI, actual transpiration, etc. in real-time. It stores raw and processed geo-tagged, timestamped images, and also exports meta data and model outputs in ‘csv’ format with the click of a button. The target can be as small as part of a leaf, a number of leaves or entire canopy.

Key Features & Specifications

  • System specifically designed and optimized for agricultural applications

  • Relies on Raspberry Pi single-board computer 

  • Combines thermal and spectral bands (red and NIR)

  • 100% on-board, real-time and automatic image processing (max 5 f/s)
  • On-board Graphical User Interface

  • Pulls data from a microclimate unit

  • Multiple image layers: surface temperature, multispectral, NDVI, ΔT, CWSI, stomatal conductance, transpiration, etc

  • Automatic and real-time alignment of images based on distance from the target

  • Calculates canopy cover percentage, canopy height & reference ET

  • Calculates crop coefficient (Kc) & crop ET (ETc)

  • Automatic white balance (adjustment for change in light source)

  • Calculates surface temperature of the target by automatically detecting & removing background from thermal image

  • Stores raw & processed images

  • Exports meta data and a wide range of calculated plant parameters to a ‘csv’ file

  • Thermal pixel resolution of 160 x 120 (19,200 pixels)

  • Thermal measurement accuracy of ±0.5 °C (after calibration)

  • Multispectral image resolution of up to 8 MP

Thermal camera
Thermal camera

Designed for Agricultural Applications

Please watch the YouTube video and read our research and blog articles to learn more about the history of the BINA Pro (2016 - 2019).

Multimodal: Thermal and Multispectral Imaging

In many agricultural applications, we need to identify specific targets (e.g. leaves) in the thermal image. In order to do that we need to remove the unwanted background (e.g. soil, water, artificial objects). The solution is to combine thermal imaging with another imaging technique (e.g. RGB). Instead of conventional visible RGB images, the BINA Pro unit uses  NIR multispectral images to detect the target/background. The added NIR band improves the ability of the onboard computer vision algorithm in distinguishing dead plants and non-organic objects from healthy leaves. The user can also benefit from saved raw multispectral or NDVI images in nutrient or disease management.

BINAPro1_edited.jpg
BINAPro7_edited.png

Three Image Layers & Automatic Image Fusion

The GUI supports automatic overlay and display of image layers in real-time. The image layers include surface temperature, canopy and air temperature difference (ΔT), raw visible bands, NIR band, multispectral image, NDVI (different indexes), crop water stress index (CWSI), stomatal conductance (SC), and actual transpiration (Ea). Additional image layers (e.g. photosynthesis) can be easily added to the software.

Image_Layers3.JPG
BINAPro5_edited.jpg
BINAPro16_edited.jpg
Distance_Correction_4.JPG

Automatic & Manual Image Alignment

A calibrated mathematical equation and distance (from the target) data measured by the embedded ultrasonic range finder are used to automatically align thermal and multispectral images. The GUI also allows for a very simple-to-do manual image alignment. 

Automatic Background Removal

There are different ways to detect a target of interest in an image ranging from simple color-based to more accurate machine and deep learning algorithms (e.g. convolutional neural network - CNN). The BINA Pro software relies on an automatic image segmentation algorithm based on NDVI values calculated for image pixels to separate plant leaves and canopies from their background (e.g. soil, water). The user can set upper and lower NDVI thresholds that are used to remove the background from multispectral images. We are currently working on improving the process by developing and trying different machine/deep learning-based models and algorithms for semantic and instance image segmentation. This will allow the system to cover a wider range of applications with a much higher accuracy. 

Thermal camera

Communication with Microclimate Unit

The BINA Pro is programmed to pull microclimate data (air temperature, relative humidity, solar radiation and wind speed) from a commercially available unit (Campbell Scientific ClimaVUE50). Currently, only wired communications (SDI-12) with the unit is supported. Depending on the availability, an API can potentially replace the in-field microclimate unit and allow for pulling data from any other commercial weather station.

BINA_Pro_Field_1.PNG

Calculation of Crop Coefficient

 

The GUI automatically calculates the canopy cover percentage (Cc, %) in the multispectral image. To calculate crop coefficient (Kc) and crop ET (ETc), it uses polynomial equations that define Kc as a function of Cc (Kc = f(Cc)). These equations are available for a variety of crops, and can be easily developed for others. Please read this article for more information on using the imaging to determine crop coefficients. 

BINAPro13_edited.png
BINAPro12_edited.jpg

Sensor Output & Data Retrieval

The GUI stores raw and processed images. It also exports the results of on-board processing in ‘csv’ format. The unit is configured as FTP server allowing the user to connect to it using its FTP address. Raw images and output data can be accessed and downloaded easily.

 

As long as there is WiFi access, the user can also access the desktop and GUI remotely from any computer using a "Remote Desktop" app.

SensorOutput_Excel_August 30, 2019.JPG

Compatibility with Third-Party Hardware

 

The BINA Pro thermal imaging module puts a pixel resolution of 160x120 at user's disposal. The multispectral sensor also supports images up to 8 MP. However, some applications might require higher resolutions. With some modifications, the BINA Pro software can work with other commercially available camera hardware setups. 

BINA Pro Demo

 

Most features of the BINA Pro are explained in the following videos. The first four videos are parts of the same demo session, in which an actual ''plant' 🌿 is used to demonstrate how the system works. To simplify things, in the last video we used a 'hot' cup of tea ☕ instead of a plant, put it in front of the camera and ran the software. Let's watch videos below to learn how the system 1) uses multispectral images to separate the target (plant or cup of tea) from its background, and 2) uses thermal images to calculate the hottest and coldest points on the surface of the target in real-time.

Applications

Monitoring of crop transpiration, and growth

High throughput phenotyping

Monitoring  temperature in fruit storage facilities

Precision livestock management

Detection and prediction of pests

Fruit and tuber vegetables size measuremet

Fruit loss management (e.g. apple sunburn prevention)

Bud, blossom and fruit count

Irrigation scheduling

Handheld imaging-based leaf porometer

Non-invasive detection of disease before visual symptoms are apparent

Monitoring of greenhouse environment (root zone, heating system, irrigation system, fans, …)

Ground-truthing data collected by  drone or satellite

Detection of crop water stress

Produce quality assessment and sorting

Automation and robotics

Detection of apple bitter bit disorder

Detection of disorders (e.g. leaf tip burn due to calcium deficiency at high light)

Detection of nutrient deficiency in controlled environment agriculture