top of page

Low-cost Machine Vision: Lepton Versus Heimann Thermal Modules

Updated: Mar 28

A few years ago, I had a project on using low-cost thermal camera modules for machine vision and automation in the baked food industry. It was followed by an interesting visit of a “dog treat” factory. We wanted to develop a basic setup to show to our clients and see their reaction. So, the question I asked myself was that what is the best approach to this problem, and if our low-cost system developed for agricultural applications (Osroosh et al, 2018; Osroosh and Peters, 2019) could be used for this purpose?


My first thought was that, while I’m working on developing a new imaging system and figuring out how to make it work, perhaps commercially available cameras built for machine vision applications, can be used for the show. I also needed to investigate if the imaging system I had previously developed for agricultural applications could be used for the show or actual industrial applications.


Machine vision-based industrial automation has its differences with machine-vision in agriculture, but there are also a lot of similarities between the two. For example, in high throughput phenotyping, we combine a variety of sensors and imaging systems to collect as much information as we can on plants. This data is usually post processed. Here we are dealing with a food factory, which is considered an industrial setting. There are companies whose area of expertise is development of machine vision systems (hardware and software) for monitoring and control of industrial processes. These systems are specifically designed to work in harsh environments and/or deal with fast moving targets on conveyors. In an industrial setting, expense (relatively) and electrical power consumptions are not that much of a concern. Instead, it is crucial to be fast, survive mechanical vibrations and high relative humidity and temperature, stay clean, and meet safety standards and regulations.


I was not the first to combine thermal and RGB images to separate target from background. This has been out there and companies like FLIR have been doing it for so long. It was my introduction to the world of tiny thermal modules that allowed me to develop my first imaging system for agricultural applications. So, what is new about my system and what makes it interesting? The answer is that the area of application is perhaps the most important aspect. I was the first person to develop such a system to be used for continuous monitoring of plant canopies at a potentially affordable price. If you take my system to an industrial setting it would look like a toy compared to its commercially available industrial counterparts, which interestingly enough, cannot be used in agriculture for continuous monitoring despite their sophisticated design and high price. There are basically no competitors in agriculture when it comes to low-cost imaging systems for continuous monitoring, but there is a long list of fearsome competitors in the food industry who cover both small and large scales!


Development Challenges

It may seem pretty straightforward to take the thermal-RGB camera that I developed in my research (Osroosh et al, 2018) and use it in a food processing facility, but it is really challenging. Provided we can develop such system, it would lead to a completely different product that we may not be able to use in agriculture. Alternatively, we may be able to OEM what we need or buy it from industrial equipment manufacturers, at a lower cost (time, labor, capital investment) and with higher reliability. I will try to explain some of these challenges (in fitting my imaging system into food industry) that I have identified:


1) Ambient temperature range. The sensors and electronics have specified operating temperatures. There is no guarantee that the current design can survive the harsh factory environment. This becomes more crucial if the system is installed right above the hot conveyor close to an oven.


2) Target temperature range. Because of the high temperature measurement requirements, thermal module options are limited. The Lepton FLIR modules I have tested in the past (V2.5) cover only 0-50 deg C (blackbody test results), which is not enough for monitoring baked food. The latest version of Lepton FLIR can cover a wider temperature range of up to 400 deg C when configured for its low gain mode but it would compromise the accuracy to ±10 deg C. Heimann, on the other hand, has good sensors. Their sensors cover -20 to >1000 deg C and the accuracy is expected to be better than ±2 deg C. Heimann sensors, however, have other limitations.


3) Speed (frame per second) and response time. There is a limitation to the number of frames per seconds we can get from the thermal and RGB camera modules used in my current design, and this may not be enough when scanning a fast-moving target on a conveyor. If not fast enough, the camera will record the same object at different points in the same frame. This is similar to what happens with a handheld camera when your hand moves. The speed of the camera modules at their highest resolution (required) does not exceed 10-20 fps. One way to deal with the resolution issue is to install the camera system very close to the target. This will, however, require the modification of the processing line and a heat/vibration resistant enclosure/platform design. Again, the camera modules may not tolerate the heat stress. Response time is an associated concept. The RGB and thermal modules in my imaging system use different technologies. The Heimann thermal sensor is not a real camera. It is a thermopile array sensor meaning that it uses hundreds of sensors in one package. These sensor outputs (pseudo-pixels) are not measured at the same time. If it takes the module 100 ms to complete a frame, then a moving target will be distributed over more pixels and look like it has been cut in pieces or is longer. As an example, in the dog food factory I recorded speeds as high as 0.9 m/s in the oven area. At this speed the target moves about 9 cm in 100 ms. This is several times the size of dog biscuits (target).


4) Processing power. The Raspberry Pi 3 is the SBC use in my design. Higher resolution images and higher frame rates may exceed the processing power of the Raspberry Pi. Longer processing time means that a smaller number of frames per seconds can be processed in real-time and slows down the whole process (the new Raspberry Pi 4 might change the equation here).


5) Resolution. If measurements are carried out at a safe distance in terms of heat and radiation reflected from surrounding objects, the resolution has to increase significantly. Unfortunately, with the thermal modules I’m using we don’t have that luxury. In addition, I have to decrease the resolution from 80x60 to 80x16 pixels (possible in Heimann sensor) to be able to increase the thermal module frame rate to a higher level (> 20Hz).


6) Cleaning. The camera lenses need to be kept clean. This has not been envisioned in the original application in agriculture, but it is necessary in a food processing facility. My understanding is that a continuous air flow can be helpful here.


7) Lighting. The added RGB light sensor to the design and white balancing can be helpful in outdoors and account for temporary changes (passing clouds etc). In indoor applications, such as in the food processing facility, a consistent uniform illumination is necessary. Unwanted shadows and inappropriate or low-level lighting can complicate image processing and increase the error. The higher the speed the brighter/better lighting source is required because the exposure time is shorter.


8) Background. The conveyer can have a variety of colors, which affects the image processing. It may be necessary to modify part of the line to provide a consistent background with a color of our choice.


9) Mechanical shock. Continuous vibrations of the production line (and possibly resonance) can damage the system over time. I imagine, a vibration-resistant design is required for such application.


10) Enclosure and mounting platform. This project is more than just developing a camera and we may need to modify the production line and built a mounting platform. The least expected from such platform is to be heat stress, humidity and [mechanical] shock tolerant. It also needs to provide a controlled source of light over the target. In addition, thermal measurements are easily affected by reflected longwave radiation (heat) from surrounding objects. For accurate measurements, the mounting platform should be able to isolate the samples and keep the camera unaffected by reflected radiation. Longwave radiation can have negative effect on thermal image processing.


11) Standards and regulations. Following safety/quality standards and regulations in the food industry is extremely important. This related article (“Machine Vision Challenges and Applications in the Food and Beverage Industry”) mentions how important these are:



Lepton vs Heimann thermal modules

Based on my research, commercially available “radiometric” thermal modules on the higher end of resolution include the FLIR, Heimann and Melexis brands. Here’s a comparison of modules from these companies (focused on FLIR and Heimann sensors) around the most important specs in the intended application:



Resolution. The highest resolution provided by Melexis is 32x24 (MLX90640) and was not available for individual sale the last time I checked. Heimann’s highest resolution module is the HTPA80x64d (5,120 pixels). FLIR has more options. FLIR high resolution [radiometric] modules are the Lepton 2.5 and 3.5. The Lepton 3.5 has a resolution of 160x120 pixels (19,200 pixels).


Speed (frame rate). The effective frame rate for both the HTPA80x64d and Lepton 3.5 is less than 9 Hz. According to the brochures, this is an export compliant frame rate because thermal modules have military applications. The frame rate of the Lepton cannot be changed (hackable?). The effective frame rate of the HTPA80x64d can be changes, but it comes at a cost. The pixel resolution and ADC resolutions need to be decreased to 80x16 to make it faster.


Target temperature range. The HTPA80x64d has an impressive range of -20 to 1000 °C and higher. The Lepton appears to allow for low/high gain settings to determine the target temperature range it can measure. However, this comes at a cost. The range for the high gain is -10 to 140 °C and for the low gain is up to 450°C. The compromise is that the high gain setting would result in a radiometric accuracy of greater of +/- 5°C or 5% (typical) and the low gain setting in the greater of +/- 10°C or 10%. The accuracy can be increased by region-specific calibration but it is costly and time-consuming. Just as an example, [Company Name] was able to decrease the cost of their IRTs to half, just by removing a few calibration points.


Noise. The Lepton looks more like a camera module and has integrated digital thermal image processing functions including automatic thermal environment compensation, noise filters, non-uniformity correction, and gain control. The Heimann is more like a sensor and all the calibration and improvements need to be carried out by an external processor (e.g. single board computer). The end result especially in terms of noise looks much better for the Lepton.


Shutter (exposure time). The Lepton has a mechanical shutter allowing it to do auto-calibration. The Heimann sensor is shutterless. During my experiments with Lepton modules, I experienced failures of the mechanical shutter. The shutter speed of the Lepton is unknown to me but it takes about half a second for the module to complete FCC. Shutter speed has to be high enough to avoid blurry images. The module does not allow for stopping this automatic feature. Considering the low frame rate of the Lepton I expect low shutter speeds as well. Here is a note from FLIR on why shutter is needed:


For the majority of applications, a shuttered version of OEM camera is the best option. All uncooled microbolometer-based cameras drift with temperature changes, and for optimum image quality the pixels occasionally need to be re-normalized. This process is called a Flat Field Correction, or FFC. The parameters for FFC are time and temperature change.The shuttered versions of FLIR's OEM cameras perform this operation automatically, however the user has some control over the frequency of FFC by way of setting the amount of time between shuttering, and/or the amount that the ambient temperature can change before shuttering. In certain situations where the scene or the system is always moving, a non-shuttered version can be used, as FLIR's OEM cameras employ sophisticated software that will compensate for this drift. However, an initial FFC should be performed using a uniform object or source in front of the lens (even a hand in front of the lens will work). Without details of the intended end-use of the camera or system, FLIR's guidance is to specify a shuttered version of OEM camera.


ESD. The Lepton module is extremely sensitive to electrostatic discharge (ESD). The Heimann sensor, on the other hand, has an ESD protection of at least 2.0 kV.


Industrial alternatives to low-cost thermal camera modules

There appears to be a variety of technologies already developed for food quality control, as well as process monitoring and control. Some of them don't involve imaging or need to be combined with imaging. Some are to be used online and some are for offline quality control, and cover large and small scale processing lines. What I’m sharing is examples of what I have found on the Internet and there may be more options to explore:


1) Industrial machine vision systems (different technologies).


2) Industrial thermal and RGB camera systems (to build MV system). There is this option to buy commercial industrial cameras and build your own machine vision system. Hardware and software are available from different vendors and can be combined.

FLIR A65 (thermal camera); FLIR A315 (thermal camera); FLIR A615 (thermal camera)

3) Line scan cameras. Regular cameras, called area scan cameras, have rows and columns of pixels whereas a line scan camera has only one row of pixels allowing it to easily scan a very fast-moving target. Line scan cameras are more common in industrial monitoring lines. Line scan camera needs the speed of the conveyor as an input so the captured “slices” can be used correctly to create an image of the object.


4) Laser-based industrial machine vision systems. Laser triangulation appears to be the dominant way in the food industry for determining 3D profile of the target. Perhaps, it is because of its lower cost and robustness compared with area or line scan cameras.


5) Machine vision software . Usually, the same software and libraries can be used for processing thermal and RGB images. Here’s a list of robust industrial packages available in the machine vision market:

References

Khanal, S., Fulton, J., Shearer, S., 2017. An overview of current and potential applications of thermal remote sensing in precision agriculture. Comput. Electron. Agr., 139: 22–32.


Osroosh, Y. et al., 2019. Detecting fruit surface wetness using a custom-built low-resolution thermal-RGB imager. Computers and Electronics in Agriculture, 157: 509-517.


Osroosh, Y. et al., 2018. Economical thermal-RGB imaging system for monitoring agricultural crops. Computers and Electronics in Agriculture, 147: 34–43.


0 comments

Yorumlar


bottom of page