two data use examples that discuss recommended
data quality screening flags and thresholds. A brief list of available data quality products are:
- Provides a numerical confidence
level for the classification of layers by the CALIOP cloud-aerosol discrimination algorithm.
- Extinction_QC_532, Extinction_QC_1064
- Provides a bit-mapped set of flags that report the status of
the extinction retrieval algorithm for each feature where a retieval was attempted.
- Provides a qualitative assessment of the confidence that users
should assign to each layer reported.
- Feature_Classificaiton_Flags or
Atmospheric Volume Description
- Reports (a) feature type (e.g., cloud vs. aerosol vs. stratospheric layer);
(b) feature subtype; (c) layer ice-water phase (clouds only); and (d) the amount of horizontal
averaging required for layer detection. Quality assessments are also provided for all layer classifications,
such as the confidence of the ice-water phase discrimination and aerosol-typing.
We've also provided uncertainty estimates for most data sets, they can further help you
assess data quality.
How can I use the CAD score to identify possibly unreliable layers?
Assuming that you are interested in the optical properties of the aerosol layer and not just the
layer boundaries then there are several other data products you should consider utilizing in your
The CALIOP CAD_Score quantifies our confidence in the classification
of any given layer as either a cloud or an aerosol. CAD_Score values
range from -100 to 100: positive values indicate a cloud, negative an
aerosol. Values close to zero indicate less confidence in the type
classification. In general, layers for which the magnitude of the CAD
score is less than 30 should be regarded with suspicion. We recommend
excluding these features from analyses of layer optical properties.
There are also several CAD values that fall outside the nominal range
of -100 to 100. See the description here
for information and usage advice about these "special" CAD_Score
If you are interested in using the CAD_Score to evaluate layer optical
properties, there are several other data products you should consider
including in your analyses. Ideally, one would check the total optical
depth above any layer to determine the quality of the it's measured
properties. This is because the backscatter signal in layers with
larger overlying optical depths will have relatively poorer signal-to-
noise (SNR), which in turn reduces the confidence one should place in
any derived layer optical properties. However, since elastic
backscatter lidars such as CALIOP cannot measure optical depth
directly, we have instead provided the
Layer_IAB_QA_Factor as a qualitative
proxy to the optical depth above the layer. The Layer_IAB_QA_Factor
ranges between zero and one, with low values indicating that there is
likely a high amount of attenuation above the feature and the quality
of the extinction data and other layer descriptors are greatly reduced.
The layer Extinction_QC_Flag_532 should also be examined, and if any
of bit field indices greater than 5 (value > 31) are set, then the
aerosol extinction data should probably not be used.
If you are just interested in the layer geometric extent, then
screening the CAD_Score and Layer_IAB_QA_Factor would be sufficient.
Note that for completely attenuating features (i.e. opaque) the lidar
is only measuring the apparent base not the true base of the feature.
A layer Opacity_Flag of 1 flags a given feature as being opaque.
Is CALIOP data quality homogeneous over the whole Earth?
In theory, yes, as the same suite of algorithms is applied to the
data uniformly around the entire globe. In practice, however, the
data quality can and sometimes does vary from one point on the
globe to another.
For any global assessment data quality variability, the very first
thing to realize is this: there is a substantial difference
between data acquired when the satellite is in full darkness (i.e.,
nighttime data), and data acquired during the sunlit portions of
the orbit (i.e., daytime data). The high solar background signal
present during daylight measurements makes the daytime data much
noisier than the nighttime data. As a consequence, the quality of
data acquired in the lat/lon region will vary according to the
local diurnal cycle.
The additional noise in the daytime data especially prominent when
CALIOP passes over bright clouds, where the noise magnitude above the stratus clouds is
significantly larger than over the ocean surface. Retrievals of
spatial and optical properties of layers lying above these clouds
are typically much more uncertain than retrievals for layers lying
above a dark ocean.
The additional noise during daylight measurements also introduces
additional uncertainty in the calibration of the lidar, and this is
reflected in the greater uncertainty ascribed to optical parameters
retrieved from daytime measurements relative to their nighttime
Another fundamental limitation on the quality of the CALIOP
retrievals is the signal-to-noise ratio (SNR) of the measurement.
For the uppermost layer in any profile, the SNR for nighttime data
is largely a function of the lidar instrument state, and generally
does not vary as a function of orbit location. However, for both
nighttime and daytime measurements, the backscatter signal is
attenuated and the SNR degrades as the lidar penetrates increasing
amounts of optical depth. Thus measurements of an otherwise
identical aerosol layer will have higher SNR for cloud-free
conditions than (for example) on those occasions when there are
cirrus clouds overhead.
Some other conditions that may contribute to "inhomogeneous data
quality" include the following:
What are 'constrained cirrus cloud retrievals', how do I
find them, and why should I bother to look for them?
- In the South Atlantic Anomaly, excess
radiation in the Van Allen Belts introduces
into the backscatter measurement, thus reducing the quality
of the nighttime calibration, which in turn can affect the
accuracy of all subsequent parameters derived from this data.
- Clouds and aerosols are distributed inhomogeneously around the globe,
and therefore altitude and latitude dependent probability distribution
functions (5D PDFs) are used in the cloud-aerosol discrimination (CAD).
The CAD algorithm may perform differently from one point on the globe to another.
In general, the discrimination between clouds and aerosols is worse at higher
altitudes and latitudes where cirrus clouds and moderate dense nonspherical
aerosol tend to be present simultaneously. In addition, the latitude resolution
of the five-dimensional probability distribution functions (5D PDFs) used
in the cloud-aerosol discrimination (CAD) algorithm is fairly coarse (10 degrees),
and thus there are occasional CAD discontinuities at the boundaries.
- There is a known flaw in the aerosol subtyping scheme that
sometimes causes a misidentification of lofted smoke layers as
marine layers. there is a geographical bias to the occurrence
of this error, as it happens most frequently in those regions
where dense smoke is transported out over the ocean.
Constrained cirrus cloud retrievals are those cirrus cloud cases where it was possible to measure
the backscatter molecular return below the cirrus cloud in a 'clear air' region and obtain a direct measurement
of the cirrus cloud two-way transmittance or optical depth. This measured transmittance is then used to
constrain the retrieval of the extinction profile of the cloud, thus reducing the uncertainty from using a default lidar ratio.
These so-called constrained cirrus cloud retrievals are generally the highest quality retrievals of
extinction provided by the lidar level 2 processing software.
The Extinction_QC_Flag_532, and Extinction_QC_Flag_1064 data sets will have bit field 1 set for
constrained retrievals. These data sets can be found the 5 km layer products and bin-by-bin in the
5 km profile products.
LIDAR Level 1B Product
Why are there negative attenuated backscatter values in the L1 lidar data?
(Aren't these values completely non-physical???) Are these negative values removed before
calculating the Level 2 Column Integrated Backscatter?
The negative attenuated backscatter values arise from statistical variations in the CALIOP
receiver photodectors, as explained in section 2.3 (page 11)
CALIPSO lidar level ATBD.
"The solar background signal can be significant - as large as the clear-sky atmospheric signal.
The instrument measures the DC background of each profile from the signal acquired between
112 km and 97 km, where the laser backscatter signals are negligible. This DC signal is
electronically subtracted from the analog profile before digitization to allow the dynamic range
of the digitizer to be used most effectively. This subtraction will result in negative-going noise
excursions if the laser backscatter signal is small."
The negative values are not removed when computing column or layer integrated attenuated
backscatters. In fact, doing so would introduce a bias into the calculation, as the lowest
components of the random noise in the signal would be systematically removed, while the highest
components would be retained.
When I plot the lidar level 1B profile data, why are there discontinuities at ~8 km and again at ~20 km?
Data are collected on board the satellite at a nominal vertical resolution of 15 m with a profile spacing of 333 m in the horizontal (along track)
dimension. However do to bandwidth limitations the data is averaged on board before downlinking.
The discontinuities that you may be observing at ~8 km and ~20 km correspond to the altitude where the horizontal averaging
switches from 333 m to 1 km and then to 5/3 km ~(1.67). See the discussion in “Essential Reading”
about the horizontal and vertical averaging.
LIDAR Level 2 Layer Products
In the layer products the data set Number_Layers_Found indicates that there are N (e.g. 3)
layers in a column, does that mean there are N (e.g. 3) unique atmospheric layers in the column?
Interpretation of the number of layers found parameter is straightforward for the 1-km and 1/3-km layer products:
individual layers are always separated by regions of "clear air", and layer boundaries never overlap in the
vertical dimension. However, this simplicity of interpretation does not always carry over into the 5-km cloud
and aerosol layer products.
CALIPSO uses a nested
multi-grid feature finding algorithm,
(also see the layer detection ATBD) ,
and thus the search for layer boundaries is conducted at multiple horizontal averaging resolutions.
While the 1-km and 1/3-km layer products report only those features detected at, respectively,
averaging resolutions of 1-km and 1/3-km, the 5-km products report layers detected at multiple averaging
resolutions (5-km, 20-km, and 80-km in the version 3 products). Because the reporting resolution (5-km)
is not always identical to the detection resolution, layers may appear to overlap in the vertical dimension.
I have discovered, at least in the 5 km cloud layer files, that
the top of the second cloud layer is often higher than the bottom
of the first (top) layer. Can you explain to me what this means?
The 5 km products report layers detected at multiple averaging resolutions (5 km, 20 km, and 80 km
in the version 3 products). Because the reporting resolution (5-km) is not always identical to the
detection resolution, layers may appear to overlap in the vertical dimension.
Please read through the answer to question 6 (above), and read through the User Guide section in
“Essential Reading” about the
multi-grided averaging scheme.
The file name of the 5 km layer products would seem to indicate that features are reported at 5 km,
but in those files there are features reported at 20 and 80 km, what's up with that?
We agree that there is a bit of a disconnect between the resolution of reported features and
the file name, and that it may be confusing at first for those users familiar with remote sensing data from
imaging sensors. The '5 km' is intended to indicate the highest resolution that the data is reported
at, not necessarily the average resolution that was required to detect the feature.
Do to the large variability in the measured attenuated backscatter that occurs in the atmosphere
it was necessary to develop an algorithm that was capable of finding both weak and relatively
homogenous layers (i.e. aerosols) and strong and highly variable features (i.e. water clouds).
Please see our primer on the
multi-grided averaging scheme
for further information. Rather than report each horizontal averaging resolution in a separate file we decided
to do you all a favor and report everything at the nominal 5 km horizontal resolution. The caveat is
that 20 and 80 km features are only reported in the columns where it was found. Since high resolution
data is replace by zeros before where ever features were found there are often 5 km columns and range
bins where the higher resolution completely over-writes the 20 or 80 km feature.
LIDAR Level 2 Profile Products
In the profile products what is the difference between 'attenuated backscatter' and 'backscatter'?
'Attenuated backscatter' refers to the data products available in the lidar
L1B profile product
These products are the geolocated, calibrated, and range-squared corrected raw lidar data.
Note that these data are reported at a horizontal resolution of 333 m, however due to the satellite
on-board averaging only data in the altitude range -0.5 to 8.2 km is available at that resolution.
Data from higher alitudes are reported at the 333 m horizontal resolution but are over-sampled
according to the figure shown here.
'backscatter' refers to the Total_Backscatter_Coefficient_532
available in the lidar level 2 5 km cloud or aerosol
profile product. These products are the output from the
lidar level 2 extinction retrieval
algorithm (pdf). The cloud and aerosol backscatter and derivative products are reported at a
uniform spatial resolution of 60-m vertically and 5-km horizontally, over a nominal altitude range
from 20-km to -0.5-km for clouds, and over a broader altitude range of 30-km to -0.5-km for aerosols.
Also note the following with regard to differences between the 'attenuated backscatter' reported in
the Level 1 and Level 2 data products: At any altitude, the magnitudes of the measured backscatter coefficients
(i.e., the level 1 attenuated backscatter coefficients) are lower than the corresponding magnitudes
of the 'true' backscatter coefficients (i.e., the backscatter coefficients reported in the level 2
data products). This difference in magnitudes is due to the attenuation of the laser beam over the
two-way path between the instrument and the altitude at which the measurement is made. One of the
primary goals of the CALIOP level 2 processing is to determine how much attenuation has occurred,
and then provide estimates of the true backscatter coefficients.
Please consider reading the data quality summaries before using extinction data products, and don't forget
to use QC flag information in your analyses.
What's the difference between volume depolarization ratios and particulate depolarization ratios?
The volume depolarization ratio refers to the depolarization ratio calculated from the attenuated backscatter data.
Volume depolarization data sets are available in the lidar 5 km layer products as
The particulate depolarization ratio refers to the depolarization ratio calculated from the backscatter data, which is post-extinction data product.
Particulate depolarization data sets are available in the lidar 5 km layer products as
and in the 5 km profile products as:
Are layer boundaries and layer type information provided in the CALIOP profile products?
Layer boundary information is NOT provided in the profile products, however layer top and base are provided
the lidar level 2 layer products. While layer boundaries are not explicitly provided in the profile products
it is possible to determine where boundaries are because valid data is only provide in the bins where layers
have been found. All the other data bins are set to the FILL_VALUE for that particular data set.
Users are also cautioned that the layer base of near-attenuating or fully attenuating features is only the
apparent base of the feature, not the true base. The lidar is capable of penetrating to a level in the
atmosphere with a one-way optical depth ~3.
LIDAR Level 2 VFM Product
Why does the surface occupy multiple range bins in the Vertical Feature Mask (VFM)?
The CALIOP 532 nm channels do not recover instantaneously from very strong returns
such as those from ocean and land surfaces. Instead, there is a decay which is approximately exponential.
Due to this effect, there is often apparent lidar return from beneath the surface.
This effect is particularly noticeable in Antarctica where frequent cloud-free atmospheres
and a highly reflective surface result in surface signals which apparently extend
significantly below the true surface.
The lidar level 2 feature finding algorithm interprets and records the entire 'surface-spike' as
a single feature, thus extending many range bins below the surface level.
Users of Lidar Level 2 layer products can expect that the bases of strong scattering targets
(i.e. optically dense water clouds) will be lower than expected.
In most of these cases however, the observed layer is opaque to the lidar and the measurement
of the true cloud base is not possible. The impact from the transient response to measurements of
cirrus clouds and aerosol layers are considered negligible,
please see the following document for further information.
What information is provided by the Feature_Classification_Flag(s)?
The Feature_Classification_Flag data set provides information on the type (e.g. aerosol,
cloud, stratospheric), cloud phase, cloud or aerosol type, horizontal averaging, and qa flags,
for all features detected.
In the profile product, the Atmospheric_Volume_Description data set contains layer type information, that
is found in the Feature_Classification_Flag data set. The Feature_Classification_Flag data set contains
a bit field that indicates the feature type (e.g. cloud, aerosol, stratospheric, etc.).
How do I say CALIOP?
CALIOP stands for Cloud-Aerosol LIdar with Orthogonal
Polarization. It is the name of the lidar onboard the CALIPSO satellite, we like
to pronounce it as the same as calliope, like the musical instrument (phonetically it is cal-eye-o-pee, not as cal-ee-op).
See Merriam-Webster Dictionary for audio
(use the left-hand speaker button for the correct pronunciation).