Developing a High-Performance Falling Objects Sensing System Based on Machine Vision

Author(s):
Kevin Lu - V I Engineering

Industry:
Manufacturing, Machine Vision/Imaging

Products:
NI-IMAQ, Machine Vision, LabVIEW, Vision Development Module, IEEE 1394 Cameras

The Challenge:
Designing and developing a system that measures time spacing and the X-Y position of fast-falling objects to distinguish objects in a clump and has object counting accuracy higher than 99 percent at a rate of up to 200 per second.

The Solution:
Creating a system based on machine vision that comprises IEEE 1394 line-scan cameras and backlighting units, and special image acquisition algorithms to identify, count, and measure objects, object time spacing, and object X-Y coordinates.

We designed and developed a machine vision system that counts and measures high-speed and high-density falling objects. The system measures time and the X-Y position of each object when falling through a sensing plane. We used line-scan cameras and back illumination units in the system and developed special algorithms to distinguish objects in a clump. The required minimum detectable object size is less than 1 mm and the rate of the falling object can be more than 450 per second. The accuracy of object counting is 99 percent at the high falling rate and more than 99.5 percent at low and medium falling rates. We used NI LabVIEW, the NI Vision Development Module, and NI-IMAQ for IEEE 1394 to develop our system.

Industries that produce ball bearings, chemical pellets, seeds, and pharmaceuticals demand systems that accurately measure counts, time, and the position of objects falling at high speed to improve manufacturing processes and quality control. Previously developed techniques such as grease belt systems and LED/photodetector grids have been used to measure the distribution of falling objects, but the grease belt systems do not take real-time measurements and require extensive postmeasurement processing, and the LED/photodetector grid provides real-time, high-speed measurement but has poor spatial resolution that limits the capability to measure small objects (less than 4 mm) and cannot resolve multiple objects when they appear too close to form a clump. A machine-vision-based system using one line-scan camera demonstrated better results than the grease belt and LED/photodetector grid methods, but the one-camera design could not distinguish object clumps or multiple objects that are too close and appear to be one object.

We based our novel machine vision solution on two line-scan cameras and backlighting units. We developed special image processing algorithms to identify, match, count, and measure objects. The system demonstrated superior performance for objects with irregular shapes, sizes ranging from 1 mm to more than 25 mm, and dispense rates up to 450 objects per second. The system can distinguish objects in a clump and deliver counting accuracy of more than 99 percent.

We selected the camera and computer that met the customer’s current performance and budget requirements. The system specifications can be improved using higher-performance components, such as cameras with higher line rate, more pixels, and frame grabbers with PCI Express technology. The machine vision system exceeds the original requirements, and the customer has been using it to improve the design and manufacturing processes of their products.

Vision System Design

We designed a machine vision system based on two line-scan cameras and two linear backlighting units. One camera and one back illumination light are centered to the region of interest, or the area the objects fall through. Another pair of cameras and backlights is oriented perpendicular to the first pair and aligned so that the scan lines of both cameras and the center lines of backlights are in the same plane.

With the backlighting, each falling object appears as a black particle on a white background regardless of the surface condition, brightness, and color of the falling object. This means the vision algorithm does not need to be adjusted according to the appearance of the different objects. We orthogonally place two identical pairs of cameras/backlights so we can use the specially designed algorithm to measure the X-Y coordinates of the objects when they fall through the image plane, which is a virtual plane constructed by the two cameras’ sensor lines and the linear backlights when properly aligned. We use a specially designed alignment fixture to align the camera sensor line position and angle so that the line-scan sensors and the two linear backlights are aligned in one image plane. The orthogonally oriented cameras provide another advantage over the one-camera configuration because they allow the software to distinguish clump objects. When multiple objects are very close spatially, they may appear as one single object in one camera image. However, from a perpendicular angle, the other camera most likely sees these objects as separate particles in the image. With a specially designed algorithm, we can match and identify all the objects in two images and separate the “clumped” objects.  

We chose two IEEE 1394 line-scan cameras with 1,024 pixels from Imaging Solution Group for this application. Along with two linear lights manufactured by Advanced Illumination, the cameras provide the right performance for the customer’s system requirements and object resolution better than 1 mm in a 6 by 6 in. image area where the falling objects are measured. The bright backlighting presents a challenge for smaller objects because it doesn’t make small objects dark enough in the image due to an edge diffraction effect that reduces the contrast of the small object. To overcome this, we adjusted the threshold value for smaller objects and measured objects 1 mm or less in diameter.

Synchronizing the two cameras is critical to accurately count and measure falling objects. The cameras are externally triggered using a pulse-train signal from an NI PCI-6601 counter/timer card. With this single-source triggering of the scanning lines and the precise physical alignment of the two cameras, an object appears in both camera images at the same vertical position.  

We used a Dell PC to acquire and process images; run an object classification algorithm; and display, generate, and report results. An NI PCI-8252 IEEE 1394 interface card plugs into the PC and connects the two line-scan cameras.

Calibration

The project budget constraints prevented us from purchasing telecentric lenses for the cameras. Instead, we used low-cost machine vision lenses. Due to the system’s space limitation, we also had to put the camera lens close to the inspection region, which caused the images from the cameras to appear severely distorted and could easily affect the measurement accuracy if not corrected. We came up with a unique method to compensate for the image distortion in the software that provides excellent results. The main sources of the image distortion are lens distortion at the edge of the field of view and the perspective error of the lens due to the lens’ proximity to the objects. The lens distortion causes an object to change size and shape when closer to the edge. The perspective error causes an object to change size when it is at a different distance from the lens. Both distortions can cause time and position measurement error and a miscount of falling objects.

We designed a novel calibration method using a calibration target to mimic a calibration grid target used in lens field calibration. A thin cylindrical target is placed and moved in a 15 by 15 uniformly distanced grid map. Images are acquired and the positions of the target in both cameras are measured when the target is in each position. After the whole grid is finished, we have a field calibration map that is basically a distorted grid image. We applied the simple calibration function in the NI vision library to convert all images to a uniform, undistorted image. All pixels are converted to real-world coordinates in millimeters. The size of the object in the image is also calibrated against its distance from the lens. After the calibration process, all objects’ coordinates and sizes are corrected in the measurement results.

Alignment Fixture

We designed a special alignment fixture to meet the critical optical alignment requirement. To synchronize the two cameras in time and space so they can accurately measure the objects, the two cameras have to look at the same vertical position and angle. In other words, the line of sight and the scan line of the two cameras must lie in a single plane. With this alignment fixture, the user can set up the optical system so that the two cameras synchronize their scans in time and position. This alignment fixture provides an easy user tool for first-time setup and maintaining the system.

Clump Objects

When several objects fall down at a high rate, some may appear as one clump, which presents a challenge to previous measurement technologies like the LED/photodetector grid or the one-camera vision system. These technologies cannot distinguish clump objects nor accurately count in high-rate situations that often lead to clumps. Our two-camera design and special algorithm can distinguish objects in a clump and accurately measure object count and position at higher rates.

Two or more objects that appear connected in one camera image may show up as separate individual objects in the image from another camera that looks from a perpendicular angle. By counting and crosschecking the objects between the two camera images at the same vertical locations in the synchronized images, the algorithm can identify and distinguish objects in clumps. In rare situations in which multiple objects appear in both cameras as a single object, the clump has a larger size and we can tell the approximate number of objects by the size. The probability that this will happen is very low, so the approximation has proven to have little effect on counting accuracy.

System Performance

Spatial Resolution and Accuracy

The system spatial resolution is determined by the camera pixel count, lens quality, lighting condition, scanning rate, and the physical dimension of the field of view. The camera we selected for this application has 1,024 pixels and covers more than a 150 mm field of view. Each pixel covers about 150 µm, which equates to a spatial resolution of 150 µm. We used 1 mm ball bearing to test the minimum detectable object size of the system, which can easily count and measure these ball bearings. Even though we have not conducted actual tests using objects smaller than 1 mm, we can safely assume it can resolve objects as small as 0.5 mm based on the current hardware configuration.

The accuracy of the spatial measurement is determined by image resolution, camera sensor signal-to-noise ratio, lens quality, lighting quality, and optics calibration.

We can further improve the spatial resolution, minimum detectable object size, and accuracy of spatial measurement when all or some of these factors improve.

Time Resolution and Accuracy

The system time resolution is determined by the camera line-scan rate and the image processing speed. In our system, the camera has a maximum line rate of 10 KHz, which translates to 100 µs of time spacing between scan lines. By using different line-scan cameras available in the market that have faster line-scan rates, we can easily improve the time resolution of the system.

Because the cameras’ line scan is triggered by an external precision pulse signal, the accuracy of the object timing measurement is mainly determined by the time resolution estimated to be 200 µs.

Counting Accuracy

With the current low-end system configuration, we can count objects with variable sizes up to 450 objects per second with 99 percent accuracy. At the rate lower than 200 objects per second, the system has 99.5 percent counting accuracy.

We successfully developed a machine-vision-based measurement system that counts and measures time and X-Y coordinates of falling objects. We designed and implemented several novel techniques during system design and development. The system delivers superior performance compared to other types of systems for measuring falling objects. We selected system components that deliver best performance-to-cost ratio and learned that a machine vision system based on IEEE 1394 line-scan camera delivers good performance at a lower cost.  

Another advantage to our system is that it can distinguish clump objects that were miscounted by previous-generation systems. We designed and built hardware and software that calibrates and compensates lens image distortion and perspective error.

Using LabVIEW, we developed powerful, flexible, and user-friendly software in a short amount of time. The rich library of the NI Vision Development Module provided many readily available functions and tools for our complex algorithm development.  

Furthermore, the system is cost-effective and meets or exceeds all the customer requirements. We can easily upgrade the system using higher-performance cameras, lenses, and lighting without making many changes to the software.

Author Information:
Kevin Lu
V I Engineering
27300 Haggerty Rd.
Farmington Hills, MI
United States
Tel: 248-489-1200
ylu@viengineering.com

Browse All Case Studies »