|
Stereovision and 3D Modeling for
Remote Operation of Mining Equipment
Project Director: Dr. John P. H. Steele
Engineering Division
Colorado School of Mines
e-mail: jsteele@mines.edu
tel: +1 (303) 273 3663
Project Team:
Chris Debrunner, Tyrone Vincent, Mark Whitehorn
This project aims at improving miner health and safety
related to the operation of haulage equipment. The ultimate goal of the project
is to move the equipment operator away from the immediate vicinity of the
equipment by automating the operation of the machine. To do this, new advanced
sensing and control techniques must be developed.
The current focus is on developing stereovision
and 3D modeling of underground mining environments as a basis for remote supervisory
control of LHDs. This will allow migration from on-board manual operation
to a mode of remote supervision. Applications of this technology span both
underground and surface mining operations, and the opportunities for improved
health and safety are significant.
Abstract
The goal of this project is to increase the health and safety of underground
miners. The approach taken is to move the operators of LHD’s to remote
locations, away from the vehicle where they can tele-manage the operation
of the LHD. This has the benefit of:
- Reducing risk of accidents
- Reducing exposure to hydrocarbon particulate
- Reducing exposure to repetitive shock loading
Of the 53 metal and nonmetal fatalities that occurred in 1999, 13 were
in underground mines. Of those that occurred underground, powered haulage
was the leading cause. In addition, it has been reported that there is
a correlation between mine safety and productivity.
Top
Progress
Miners safety can be improved by the level of automation present, i.e.,
by removing the miners from direct exposure to hazards and the less healthy
environment, e.g., being onboard the LHD. In order to do this, the loading
operation of the LHD must be automated. This requires the development of
new sensor data algorithms using vision, to interpret the scene, and the
development of system models to be used in the control and monitoring of
the machine.
To date we have been successful in obtaining stereo images in real time
and we have developed algorithms to estimate distance to the muck pile
using stereovision. In addition, we have developed a simulation of the
loading operations that includes sensor data taken from the vehicle (load
cylinder pressures, angular position of the boom and bucket). This simulation
is based upon work with Dr. Paul Lever at University of Arizona (UA), who
is collaborating with us on this project. Dr. Lever has contributed data
from his autodig algorithm and provided the opportunity to collect data
using the Cat Loader at UA.
In addition, we have obtained an experimental LHD machine from the Spokane
office of NIOSH. It is our intent to rejuvenate this system and to use it as
the test bed for our automation work.
Top
Stereo Vision and 3D Modeling for Remote operation of Mining Equipment (RP-1)
|
Year
|
|
1st Quarter
|
2nd Quarter
|
3rd Quarter
|
4th Quarter
|
|
-2
|
Planned deliverable
|
Review of MSHA database to establish parameters of safety and
health.
|
Review of relevant literature to establish context of this project
within current work
|
Initial System configuration and equipment acquisition
|
Development of stereo algorithms for feature identification and
disparity measurement
|
|
Achieved
|
Power haulage was identified as a major contributor to injuries
and fatalities. Fatality statistics for underground haulage for
the period 1995-2000 were collected showing a total of 16 deaths related
to LHD operations for that period. Proximity to operating equipment
also increases risk.
|
An extensive review of the literature revealed that while there
has been significant work in stereo vision, very little has been applied
to mining, and that has been applied to non-mobile equipment.
Competing mobile systems had used other sensors, e.g., laser-based delineation.
|
Sony CCD cameras; Dell Laptop and hardware for wide baseline
stereo were selected. Calibration routines were developed.
|
Software was written to collect synchronized images from both
cameras. Stereo data sets were collected at the Edgar Mine. Disparity
based 3D measurement algorithms were developed and tested using correlation-based
stereo feature matching.
|
|
-1
|
Planned deliverable
|
Initial 3D modeling.
|
Development of temporal tracking of image features
|
Evaluation of the feature tracking approach
|
Assessment of the uncertainty of the image analysis and model
building Error analysis
|
|
Achieved
|
Images of underground muck piles were analyzed and 3D information
was developed using disparity measurement of grayscale image features
identified using correspondence. Images were acquired using normal
lighting levels as well as scenes augmented with additional lighting.
|
Correlation-based features are tracked over multiple image pairs.
|
We have demonstrated the ability to track features for both linear
motion and turning motion of the LHD vehicle.
|
Real image-pair sequences obtained from operation of an LHD at
the Edgar mine has been analyzed for position error. Over a sequence
of 14 image pairs, the upper bound on the mean squared error of the
positional distance grew to 50 mm2.
|
|
0
|
Planned deliverable
|
Development of self-calibration algorithms for stereo vision
imaging.
|
Investigation and assessment of approaches for representation
within 3D models from temporal data.
|
Assessment of pose accuracy derived from correlation based stereo
and integrated over time.
|
Development of integrated 3D model
Feasibility study of application of this approach to Surface
Mining and Collision Detection/Avoidance
|
|
Achieved
|
|
|
|
|
|
1
|
Planned deliverable
|
System characterization Investigation of application of this
approach to UG coal mining
|
Selection of hardware for real-time system
|
Development of real-time hardware solution
Work on integration of this approach in to surface mining collision
detection and avoidance (in collaboration with NIOSH-Spokane), assuming
assessment in year 0 is positive.
|
System integration and initial testing
|
|
Achieved
|
|
|
|
|
|
2
|
Planned deliverable
|
System testing
|
Testing of real-time hardware on longwall shearer
|
Testing of system on surface haul trucks
|
Final report preparation
|
|
Achieved
|
|
|
|
|
Top
Project Results
To date we have successfully assembled the components necessary to do stereovision
underground. We have used this equipment to take video sequences, and we have
developed 3D information about the scene by using a correlation-based stereo
matching algorithm. We have used the 3D information to build maps of the environment.
Using the feature extraction algorithm and a clustering algorithm to match
between succeeding frames, we have been successful at tracking these feature
over time (i.e., through a sequence of image pairs). We refer to this as "integration
over time", since the image information is collected over a number of images
and the features are correlated between the images.
We are currently developing algorithms that will allow us to build 3D models
of the world (e.g., muckpile and drift). We are using data structure referred
to as an octree representation because of its compact representation.
This is done so we can deal with the large volume of data we expect to encounter.
The movies shown demonstrate that we can
track the features; now we need to demonstrate that we can register those features
into a world model. This model will then become the basis for decision making
by the operator as he observes the operation of the machine from his remote
location.
This summer we will take this system to Spokane to test whether we can develop
3D models of objects around surface haul trucks. This would provide the ability
to detect possible collisions with various objects (boulders, trucks, people)
before they occur.
Top
Publications
Top
Final Report
Top
Images and Movies
- Here is an image of the CSM JCI-125 with the two
Sony cameras mounted on the top of the cage
Here are some movies made with the stereovision system. You can select individual
frames by using your left/right arrow keys.
- This sequence shows the LHD approaching
the muck pile. The square boxes are the boundaries of features extracted from
the image pairs. The cross hair shows the location of the feature as it is tracked
in sequential images.
- This sequence show the LHD as it turns.
Turning makes tracking the features more challenging, but you can see that the
system does well.
- fullview.mov:
This is a visualization of the 3D muckpile data collected from a 15 frame stereo
video sequence shot at 7.5 frames per second. The 3D point clouds from each
frame are registered to the model coordinate frame, then binned into voxels
using an octree-based adaptive subdivision technique. A plane is fitted to the
3D points within each voxel and displayed in a color corresponding to the voxel
population. Planes representing the lowest populations are drawn in red and
planes representing the highest populations are drawn in blue, with intermediate
colors for population values in between. The box outline (white) represents
the boundaries of the model, and is 2 meters on each side. Each voxel is a cube
approximately 16 millimeters across. It is clear from this display that very
few outliers remain after thresholding to the selected minimum voxel population
(20% of the maximum population).
-
closerview.mov:
This is a closer look at the same data. The density of the data (with the 20%
population threshold) is high across most of the muckpile face.
- detail.mov:
This is a detail view of a small region of the face spanning the width of about
6 voxels, or 10 centimeters. The blue box outlines are individual voxel boundaries,
and the green dots are individual 3D points measured from individual stereo
pairs. The multicolored crosses represent the tracking/registration of each
3D point cloud to the model frame. The model frame is chosen to be the camera
frame for the first image in the sequence, and tracked regions in this view
are represented by red crosses. The 3D regions matched to the first view in
subsequent views are represented by separate colors, again ranging from red
through blue. The orientation of each cross indicates the orientation of the
plane fit to the voxel within which it lies.
- trackedpoints.mov:
This is a larger view of the previous detail showing the crosses representing
tracked regions over most of the muckpile face.
-
denseRegion.mov:
This is another detail view of one of the denser regions of the registered 3D
point clouds. Again, the blue box outlines represent voxel boundaries, and the
green dots represent individual 3D points. The transparent yellow box represents
the variance of the of a least-squares planar fit to all of the 3D points within
the drawn voxels. Note that the variance in the direction normal to this plane
is smaller than the voxel dimension, and is less than 1 centimeter.
Top
Presentations
The following is a list of the presentations associated with this project. Links are yet to be added.
-
Application of Computer in Mining (APCOM) 2001, Tampere Finland.
-
2001 International Symposium on Mine Mechanication and Automation, (ISMMA),
Johannesburg, South Africa<
-
Society of Mining Engineers Annual Conference (SME) 2002, Phoenix, Arizona
-
Denver Mining Club, February, 2002.
-
Omnitech Robotics, Littleton, Colorado, October, 2001.
Top
|
