A High-Performance Falling Objects Sensing System Based on Machine Vision

Author(s):
Kevin Lu - V I Engineering

Industry:
Manufacturing

Products:
LabVIEW, Vision, PXI/CompactPCI

The Challenge:
Designing and developing a system that measures time spacing and XY position of fast-falling objects. The system must distinguish objects in a clump. The object counting accuracy must be higher than 99.0 percent at a rate of up to 200 per second.

The Solution:
Designing and developing a system based on machine vision that comprises IEEE 1394 line scan cameras and backlighting units. Special image acquisition algorithms were developed to identify and count objects, measure object time spacing, and object XY coordinates. Challenged by John Deere, V I Engineering designed and developed a machine vision system that counts and measures falling objects in high speed and high density. The system measures time and XY position of each object when it is falling through a sensing plane. Line scan cameras and back illumination units are used in the system. Special algorithms were developed to distinguish objects in a clump. The required minimum detectable object size is <1 mm. The rate of the falling object can be more than 450 per second. The accuracy of object counting is 99 percent at high falling rate and more than 99.5 percent at low and medium falling rates. We used National Instruments LabVIEW, the NI Vision Development Module, and NI-IMAQ for IEEE 1394 in system development.

There is a strong demand for systems that accurately measure counts, times, and positions of objects falling in high speed and high rate in industries that make ball bearings, chemical pellets, seeds, pharmaceuticals, and other products. Such systems can serve as tools to improve manufacturing processes, as well as quality control in these industries. Previously developed techniques such as grease belt systems and LED/photodetector grids have been used to measure the distribution of falling objects. The limitation of grease belt techniques is that they do not take real-time measurements and they require extensive post-measurement processing. Addressing the limitations of the grease belt technique, the LED/photodetector grid provides real-time, high-speed measurement, but suffers some critical shortcomings, such as poor spatial resolution limiting the capability to measure small objects (<4 mm), and the inability to resolve multiple objects when they appear too closely to form a clump. A machine vision-based system that uses one line scan camera demonstrated better results than the grease belt and LED/photodetector grid methods, but the one-camera design did not solve the problem of distinguishing object clumps, or multiple objects that are too close and appear to be one object.

The present paper describes a novel machine vision solution based on double line scan cameras and backlighting units. Special image processing algorithms were developed to identify, match, count, and measure objects. The system demonstrated superior performance for objects with irregular shapes, sizes ranging from 1 mm to more than 25 mm, and dispense rates up to 450 objects per second. The system is able to distinguish objects in a clump and deliver counting accuracy of more than 99 percent. We selected the camera and computer that meet the customer’s current requirement for both performance and budget. The system specifications can be improved by using higher-performance components, such as cameras with higher line rate, more pixels, and frame grabbers with PCI Express technology. The machine vision system we developed exceeds the original customer requirements. The customer has been using it to improve the design and manufacturing processes of their products.

Vision System Design

V I Engineering designed a machine vision system based on two line scan cameras and two linear backlighting units. One camera and one back illumination light are centered to the region of interest, or the area the objects fall through. Another pair of cameras and backlights is oriented perpendicular to the first pair. It is aligned that the scan lines of both cameras and the center lines of backlights are in a same plane.

With the backlighting, each falling object appears as a black particle in a white background regardless of surface condition, brightness, and color of the falling object. This means the vision algorithm does not need to be adjusted according to the appearances of the different objects. Two identical pairs of cameras/backlights are orthogonally placed, so that with the special designed algorithm, they can measure the XY coordinates of the objects when they fall through the image plane. The image plane is a virtual plane that is constructed by the two cameras’ sensor lines and the linear backlights when they are properly aligned. A special designed alignment fixture is used to align the camera sensor line’s position and angle so that the line scan sensors and the two linear backlights are aligned in one image plane. The orthogonally oriented cameras provide another advantage over the one-camera configuration in that it allows the software to distinguish clump objects. When multiple objects are very close spatially, they may appear as one single object in one camera image. However, from a perpendicular angle, the other camera most likely sees these objects as separate particles in the image. With a specially designed algorithm, we are able to match and identify all the objects in two images and separate the “clumped” objects.  

We chose two IEEE 1394 line scan cameras with 1024 pixels from Imaging Solution Group for this application. The cameras provide the right performance for the customer’s system requirements. Along with two linear lights manufactured by Advanced Illumination, the cameras provide object resolution of better than 1 mm in a 6 in by 6 in image area where all the falling objects are measured. The bright backlighting does provide a challenge for smaller objects – it tends to make small objects not dark enough in the image due to an edge diffraction effect that reduces the contrast of the small object. To overcome this, we adjusted the threshold value for smaller objects and were able to measure objects 1 mm or less in diameter.

The synchronization of two cameras is critical to accurately counting and measuring falling objects. The cameras are externally triggered using a pulse train signal from an NI PCI-6601 counter/timer card. With this single-source triggering of the scanning lines and the precise physical alignment of the two cameras, an object appears in both camera images at the same vertical position.  

We used one Dell PC to acquire images, process images, and run an object classification algorithm, as well as display, generate, and report results. A NI PCI-8252 IEEE 1394 interface card plugs in to the PC and connects the two line scan cameras.

Calibration

The budget constraint of the project prevented us from purchasing telecentric lenses for the cameras. Instead, we used low-cost machine vision lenses in the system. Due to the system’s space limitation, we also had to put the camera lens close to the inspection region, causing the images from the cameras to appear severely distorted. This could easily affect the measurement accuracy if not corrected. We came up with a unique method to compensate for the image distortion in the software that provides excellent results. The main sources of the image distortion is lens distortion at the edge of the field of view, and the perspective error of the lens due to the lens’ proximity to the objects. The lens distortion causes an object to change size and shape when it is closer to the edge. The perspective error causes an object to change size when it is at a different distance from the lens. Both distortions can cause time and position measurement error and a miscount of falling objects.

We designed a novel calibration method using a calibration target to mimic a calibration grid target that is often used in lens field calibration. A thin cylindrical target is placed and moved in a 15 by 15 uniformly distanced grid map. Images are acquired and the positions of the target in both cameras are measured when the target is in each position. After the whole grid is finished, we have a field calibration map that is basically a distorted grid image. We applied the simple calibration function in the NI vision library to convert all images to a uniform, undistorted image. All pixels are converted to real-world coordinates that are in millimeters. The size of the object in the image is also calibrated against its distance from the lens. After the calibration process, all objects’ coordinates and sizes are corrected in the measurement result.

Alignment Fixture

We designed a special alignment fixture to meet the critical optical alignment requirement. In order to synchronize the two cameras in both time and space so they can accurately measure the objects, the two cameras have to look at the same vertical position and angle. In other words, the line of sight and the scan line of the two cameras must lie in a single plane. With this alignment fixture, the user can set up the optical system so that the two cameras synchronize their scans in both time and position. This alignment fixture provides an easy tool to the user both for first-time setup and for maintaining the system.

Clump Objects

When a large quantity of objects fall down at a high rate, some objects may appear as one clump, thus posing a challenge to previous measurement technologies like the LED/photodetector grid or the one-camera vision system. These technologies cannot distinguish clump objects, thus they cannot accurately count in high-rate situations that often lead to clumps. V I Engineering’s two-camera design and special algorithm can distinguish objects in a clump and provide accurate measurement of object count and position at higher rates.

The basic idea is rather simple: Two or more objects that appear connected in one camera image most likely show up as separate individual objects in the image from another camera that looks from a perpendicular angle. By counting and crosschecking the objects between the two camera images at the same vertical locations in the synchronized images, the algorithm can identify and distinguish objects in clumps. In rare situations that multiple objects appear in both cameras as a single object, the clump has a larger size, and we can tell the approximate number of objects by the size. This situation has a very low probability of happening, so the approximation has proven to have little effect on counting accuracy.

System Performance

Spatial Resolution and Accuracy

The spatial resolution of the system is determined by the camera pixel number, lens quality, lighting condition, scanning rate, and the physical dimension of the field of view. The camera we selected for this application has 1024 pixels. It covers more than a 150 mm field of view. Each pixel covers about 150 micrometers, which equates to a spatial resolution of 150 micrometers. We used 1 mm ball bearing to test the minimum detectable object size of the system. The system can easily count and measure these ball bearings. Although we have not done actual tests using objects smaller than 1 mm, we can safely assume it can resolve objects as small as 0.5 mm based on the current hardware configuration.

The accuracy of the spatial measurement is determined by the following factors:

• image resolution
• camera sensor signal-to-noise ratio
• lens quality
• lighting quality
• optics calibration (as mentioned above)

The spatial resolution, the minimum detectable object size, and the accuracy of spatial measurement can be improved further when all or some of these factors improve.

Time Resolution and Accuracy

The time resolution of the system is determined by the line scan rate of the camera and the speed of the image processing. In our system, the camera has a maximum line rate of 10 KHz, which translates to 100 microseconds of time spacing between scan lines. By using different line scan cameras available in the market that have faster line scan rates, the time resolution of the system can be easily improved.

Because the cameras’ line scan is triggered by an external precision pulse signal, the accuracy of the object timing measurement is mainly determined by the time resolution. It is estimated to be 200 microseconds.

Counting Accuracy

As stated above, with the current low-end system configuration, we are able to count objects with variable sizes at up to 450 objects per second with 99 percent accuracy. At the rate lower than 200 objects per second, the system has 99.5 percent counting accuracy.

Conclusion

V I Engineering has developed a machine vision-based measurement system that counts and measures time and XY coordinates of falling objects. Several novel techniques were designed and implemented during system design and development. The system delivers superior performance compared to other types of systems for measuring falling objects. We selected system components that deliver best performance-to-cost ratio. We found that a machine vision system based on IEEE 1394 line scan camera delivers good performance at a lower cost.  

One special advantage of our system is that it can distinguish clump objects that were miscounted by previous-generation systems. We designed and built hardware and software that calibrates and compensates lens image distortion and perspective error.

Using NI LabVIEW, we were able to develop robust, flexible, and user-friendly software in a short amount of time. The rich library of the NI Vision Development Module provided us with many readily available functions and tools for our complex algorithm development.  

The system is cost-effective and meets or exceeds all the customer requirements, and it can be easily upgraded using higher-performance cameras, lenses, and lighting without making many changes to the software.

For more information, contact:

Kevin Lu

V I Engineering, Inc.

27300 Haggerty Rd.

Suite F-10

Farmington Hills, MI

Tel: (248) 489-1200

E-mail: ylu@viengineering.com

 

 

 

Browse All Case Studies »