3d/4d Medical Imaging

IntroductIon
Image registration, segmentation, and visualization are
three major components of medical image processing.
Three-dimensional (3D) digital medical images are
three dimensionally reconstructed, often with minor
artifacts, and with limited spatial resolution and gray
scale, unlike common digital pictures. Because of
these limitations, image filtering is often performed
before the images are viewed and further processed
(Behrenbruch, Petroudi, Bond, et al., 2004). Different
3D imaging modalities usually provide complementary
medical information about patient anatomy or physiology.
Four-dimensional (4D) medical imaging is an
emerging technology that aims to represent patient
motions over time. Image registration has become
increasingly important in combining these 3D/4D images
and providing comprehensive patient information
for radiological diagnosis and treatment.
3D images have been utilized clinically since computed
tomography (CT) was invented (Hounsfield,
1973). Later on, magnetic resonance imaging (MRI),
positron emission tomography (PET), and single photon
emission computed tomography (SPECT) have been
developed, providing 3D imaging modalities that
complement CT. Among the most recent advances in
clinical imaging, helical multislice CT provides improved
image resolution and capacity of 4D imaging
(Pan, Lee, Rietzel, & Chen, 2004; Ueda, Mori, Minami
et al., 2006). Other advances include mega-voltage CT
(MVCT), cone-beam CT (CBCT), functional MRI,
open field MRI, time-of-flight PET, motion-corrected
PET, various angiography, and combined modality
imaging, such as PET/CT (Beyer, Townsend, Brun
et al., 2000), and SPECT/CT (O’Connor & Kemp,
2006). Some preclinical imaging techniques have
also been developed, including parallel multichannel
MRI (Bohurka, 2004), Overhauser enhanced MRI
(Krishna, English, Yamada et al., 2002), and electron
paramagnetic resonance imaging (EPRI) (Matsunoto,
Subramanian, Devasahayam et al., 2006).
Postimaging analysis (image processing) is required
in many clinical applications. Image processing
includes image filtering, segmentation, registration,
and visualization, which play a crucial role in medical
diagnosis/treatment, especially in the presence of patient
motion and/or physical changes. In this article, we
will provide a state-of-the-art review on 3D/4D image
registration, combined with image segmentation and
visualization, and its role in image-guided radiotherapy
(Xing, Thorndyke, Schreibmann et al., 2006).

Background
3d/4d Medical Imaging
A 3D medical image contains a sequence of parallel
two-dimensional (2D) images representing anatomic
or physiologic information in 3D space. The smallest
element of a 3D image is a cubic volume called voxel.
A 4D medical image contains a temporal series of 3D
images. With a subsecond time resolution, it can be
used for monitoring respiratory/cardiac motion (Keall,
Mageras, Malter et al., 2006).
Patient motion is always expected: faster motion
relative to imaging speed causes a blurring artifact;
whereas slower motion may not affect image quality.
A multislice CT scanner provides improved spatial
and temporal resolution (Ueda et al., 2006), which
can be employed for 4D imaging (Pan et al., 2004).
Progresses in MRI imaging have also been reported,
including parallel multichannel MRI (Bodurka, Ledden,
van Gelderen et al., 2004).
Because PET resolution and speed are limited by the
physics and biology behind the imaging technique, some
motion suppression techniques have been developed
clinically, including patient immobilization (Beyer,
Tellmann, Nickel, & Pietrzyk, 2005), respiratory gating
(Hehmeh, Erdi, Pan et al., 2004), and motion tracking
(Montgomery, Thielemans, Mehta et al., 2006). Motion
tracking data can be used to filter the imaging signals
prior to PET image reconstruction for reliable motion
correction. Motion blurring, if uncorrected, can reduce
registration accuracy. Visual-based volumetric registration
technique provides blurring correction (filtering)
before registration, by defining the PET volume with
reference to the CT volume, causing blurred PET
surface voxels to be rendered invisible (Li, Xie, Ning
et al., 2007).

Image Segmentation and Visualization
Medical image segmentation defines regions of interest
used to adapt image changes, study image deformation,
and assist image registration. Many methods for
segmentation have been developed including thresholding,
region growing, clustering, as well as atlas-guided
and level sets (Pham, Xu, & Prince, 2000; Suri, Liu, &
Singh et al., 2002). Atlas-guided methods are based on
a standard anatomical atlas, which serves as an initial
point for adapting to any specific image. Level sets, also
called active contours, are geometrically deformable
models, used for fast shape recovery. Atlas-based level
sets have been applied clinically for treatment planning
(Lu, Olivera, Chen et al., 2006a; Ragan, Starkschall,
McNutt et al., 2005; ) and are closely related to image
registration (Vemuri, Ye, & Chen, et al., 2003). Figure
1 shows automatic contours. Depending on how the 3D
image is segmented, it can be either 2D-based or 3Dbased
(Suri, Liu, Reden, & Laxminarayan, 2002).
3D medical image visualization has been increasingly
applied in diagnosis and treatment (Salgado,
Mulkens, Bellinck, & Termote, 2003), whereas 2Dbased
visualization is predominantly applied clinically.
Because of the demand on computing power, real-time
3D image visualization is supported by specialized
graphics hardware (Terarecon, Inc.) (Xie, Li, Ning et
al., 2004) or high-end consumer graphics processors
(Levin, Aladl, Germanos, & Slomak, 2005). 3D image
visualization has been applied to registration of four
imaging modalities with improved spatial accuracy (Li,
Xie, Ning et al., 2005; Li et al., 2007). Figure 2 shows
3D volumetric image registration using external and
internal anatomical landmarks.

Figure 1. Orthogonal 2D-views of CT images and comparison of automatic recontours (solid-lines) and manual
contours (dash-lines) in different phases (A&B) of a radiotherapeutic treatment (courtesy of Dr. Weiguo Lu

Figure 2. 3D-views of CT (A, B, and C: head) and MR (D, E, and F: segmented brain) phantom images. The
homogeneity of color distributed on an anatomic landmark is used as the registration criterion. From (A) to (C),
three images (red, green, and blue) are approaching registration with shifts from 5.0mm to 0.5mm and 0.0mm, respectively. From (D) to (E and F), four images (grey, blue, green, and red) are 5.0mm apart from each other
(D) and registered in front (E) and side (F) views.

rigid Image registration
Rigid image registration assumes a motionless patient
such that the underlying anatomy is identical in different
imaging modalities for alignment. Three approaches
to rigid registration are: coordinate-based, extrinsicbased,
and intrinsic-based (Maintz & Viergever, 1998).
Coordinate-based registration is performed by calibrating
the coordinate system to produce “co-registered”
images. Multimodality scanners, such as PET/CT and
SPECT/CT, are typical examples.
Extrinsic-based image registration relies on the
alignment of extrinsic objects placed in/on a patient
invasively/noninvasively. Such objects can be fiducials
or frames that are visible in all imaging modalities and
serve as local coordinate markers (sets of points) for
rigid registration. Examples are gold seeds for prostate
localization in radiotherapy and head frame for stereotactic
radiosurgery.
Intrinsic-based image registration uses a patient’s
anatomy (anatomic landmarks, segmented geometries,
or intact voxels) as the registration reference. Alignment
of visual landmarks or segmented geometries
requires user interaction, so the registration is manual
or semi-automatic. The statistical similarity of the intact
voxels (grayscale) of two images, such as mutual
information (Viola & Wells, 1995), has been widely
used for fully automated registration (Pluim, Maintz,
& Viergever, 2003).
Automatic image registration requires three key
elements: a metric function, a transformation, and an
optimization process. One common voxel-based image
registration uses mutual information as the metric, a
rigid or nonrigid transformation and a maximization algorithm.
Recently, the homogeneity of color distributed
on a volumetric landmark has been used as quantitative
metric, assisted by the ray-casting algorithm in 3D
visualization (Li et al., 2007). Figures 3 and 4 show
clinical examples using the 3D visualization-based
registration technique.

Figure 3. 3D-views of before (A, B, and C) and after (D, E, and F) 3D volumetric image registration, using
homogeneity color distribution as registration criterion. Voluntary patient head movement is corrected in three
MR images, T1 (green), FLAIR (red), and T2 (blue), which are acquired in the same scanner with time interval
of 3 and 20 minutes.

Figure 4. 3D-views of before (A, B, and C) and after (D, E, and F) rigid volumetric image registration of PET/CT
images, correcting patient movement.

deformable Image

registration
Deformable image registration contains a nonrigid
transformation model that specifies the way to deform
one image to match another. A rigid image
registration is almost always performed to determine
an initial position using rigid transformation with six
variables (3 translations and 3 rotations). For nonrigid
transformation, the number of variables will increase
dramatically, up to three times the number of voxels.
Common deformable transformations are spline-based
with control points, the elastic model driven by image
similarity, the viscous fluid model with region growth,
the finite element model using rigidity classification,
the optical flow with motion estimation, and free-form
deformation (Chi, Liang, & Yan, 2006; Crum, Hartkens,
& Hill, 2004; Lu, Olivera, Chen et al., 2006b).
The image similarity measures are ultimately the
most important criteria for determining the quality of
registration. They can be feature-based and voxel-based
(Maintz & Viergever, 1998). The former is usually
segmentation/classification based, adapting changes
in shape of anatomical landmarks, while the latter is
based on statistical criteria for intensity pattern match-
ing, including mutual information. Most deformable
registrations are automated. Combining the two methods
can improve registration accuracy, reliability, and/or
performance (Hellier & Barillot, 2003; Liu, Shen, &
Davatizikos, 2004; Wyatt & Noble, 2003). Figure 5
shows one example of deformable registration.

Figure 5. Orthogonal 2D-views of before (A) and after (B) deformable image registration of two 3D images (red
and green) in a 4D CT image (courtesy of Dr. Weiguo Lu)

a challenge from 3d/4d conformal
radiotherapy: deformable Image
registration
Broadened Concept of D Medical
Imaging
The 4D imaging concept has been broadened to cover
various time resolutions. The common 4D image has
subsecond temporal resolution (Pan et al., 2004), while
a series of 3D images, reflecting patient changes over a
longer time span, should be also qualified as a 4D image
with sufficient resolution to assess slower changes,
including tumor growth/shrinkage and weight gain/loss
during a course of treatment.
Registration of a 3D image to a 4D image involves
a series of deformable registration. Because patient motion/
change is inevitable, deformable image registration
is the key to combining these images for clinical use.
Clinically, MVCT images can be acquired daily and
used for patient daily setup via rigid registration to the
reference planning image, assuming minimal patient
changes. Within a treatment, 4D CT imaging has shown
dramatic anatomical changes during respiration (Keall
et al., 2006). Image-guided frameless cranial and extracranial
stereotactic radiosurgery has been performed
clinically (Gibbs, 2006). Rigid image registration
provides the best current solution to these clinical applications.
Ultimately, deformable image registration
will improve the registration accuracy substantially
(Lu et al., 2006b), permitting highly conformal 3D/4D
radiotherapy (Barbiere, Hanley, Song et al., 2007;
Mackie, Kapatoes, Ruchala et al., 2003).

challenges in deformable Image
registration
For deformable image registration, the underlying
anatomy changes, therefore voxel mapping among images
is a challenge. First, the deformable transformation
handles a large number of positioning variables that must
be determined for every voxels within the anatomy. This
can be abstracted as a multiple variable optimization
problem in mathematics, limiting the performance of
deformable image registration for many years (Crum,
2004). Second, the deformable registration is extremely
difficult to be validated as there is lack of absolutes
with respect to the location of corresponding voxels.
Therefore, the accuracy and reliability of deformable
registration should be evaluated on a case specific basis
(Sanchez Castro, Pollo, Meuli et al., 2006; Wang,
Dong, O’Daniel et al., 2005).
Regardless the limitations above, progress has been
made by combining image registration and segmentation/
classification to provide intrinsic simplification
and cross verification. It remains a challenge, however,
to develop a fully automated deformable registration
algorithm because image segmentation often requires
human interaction.
Deformable image registration is generally a “passive”
mapping process. It does not anticipate how
patient anatomy might deform. An example is whether
superficial 3D contour information detected by a realtime
infrared camera can be used to predict the motion
of internal organs (Rietzel, Rosenthal, Gierga et al.,
2004). Anatomically, the correlation between superficial
and internal organ motion should exist, although as a
complex relationship. Therefore, an anatomic model
based image registration with motion estimation can
provide an “active” mapping process, but is beyond thecurrent scope of deformable image registration.

gaps between Frontier research and
clinical Practice
Despite of the advances in 3D and 4D imaging and image
registration, 2D-based rigid registration techniques are
predominantly used in the clinic; although automatic
rigid registration methods exist in most commercial
treatment planning software. Two reasons are primarily
responsible for this disconnect: First, the user must
visually verify the final registration using the 2D-based
visualization tools available for image fusion in most
commercial software. Second, most clinical images
have some degree of pre-existing deformation so that
automatic rigid registration can prove unreliable but
manual methods allow the user to perform local organ
registration. Some recent commercial software has
recognized this problem and provides the option of
selecting the region-of-interest to ease the deformation
problem. This method, however, is only a partial
solution to cope with image changes.
The gap between the clinical research and routine
practice can be reduced by translational research and
development. Recently, open-source medical image
processing and visualization tool kits have become
available for public use. Many recently published algorithms
in medical image processing are implemented
in a generic, object-oriented programming style, whichpermits reusability of such toolkits.

Future trendS
3D rigid image registration will dominate clinical
practice and will remain essential as more specialized
complementary 3D imaging modalities become clinically
relevant. Although the simplicity of automatic
image registration is more attractive, manual image
registration with 2D/3D visualization is irreplaceable
because it permits incorporation of medical knowledge
for verification and adjustment of the automatic
registration results.
As awareness of the problems of the patient motion
and anatomic changes increases, further research on 4D
imaging and deformable registration will be stimulated
to meet the clinical demands. Motion correction in the
PET/CT and SPECT/CT will continue to improve the
“coregistration” of these images. Interdisciplinary approaches
are expected to offer further improvements
for the difficult registration problem. With advances in
hybrid registration algorithms and parallel computing,
more progresses are expected, resulting in improved
accuracy and performance.

concluSIon
Higher dimensional deformable image registration
has become a focus of clinical research. The accuracy,reliability, and performance of 3D/4D image registration

key terMS
3D Medical Imaging: A process of obtaining a 3D
volumetric image composed of multiple 2D images,
which are computer reconstructed using a mathematical
“back-projection” operation to retrieve pixel data from
projected image signals through a patient, detected via
multichannel detector arrays around the patient.
4D Medical Imaging: A process of acquiring
multiple 3D images over time prospectively or retrospectively,
so that patient motions and changes can be
monitored and studied.
Imaging Modality: A type of medical imaging
technique that utilizes a certain physical mechanism
to detect patient internal signals that reflect either anatomical
structures or physiological events.
Image Processing: A computing technique in which
various mathematical operations are applied to images
for image enhancement, recognition, or interpretation,
facilitating human efforts.
Image Registration: A process of transforming a
set of patient images acquired at different times and/or
with different modality into the same coordinate system,
mapping corresponding voxels of these images in 3D
space, based on the underlying anatomy or fiducial
markers.
Image Segmentation: A process in which an image
is partitioned into multiple regions (sets of pixels/voxels
in 2D/3D) based on a given criterion. These regions
are nonoverlapping, homogeneous with respect to
some characteristics such as intensity or texture. If
the boundary constraint of the region is removed, the
process is defined as classification.
Image Visualization: A process of converting
(rendering) image pixel/voxel into 2D/3D graphical
representation. Most computers support 8-bit (256)
grayscale display, sufficient to human vision that can
only resolve 32-64 grayscale. A common 12/16-bit
(4096/65536 grayscales) medical image can be selectively
displayed based on grayscale classification.
Window width (display range in grayscale) and linear
level function (center of the window width) are frequently
used in adjusting display content.