Detailed Data Quality Summary: CALIPSO Lidar Level 3 Tropospheric Aerosol Profile Data Monthly Products Versions 4.20 and 4.21
Last Updated: October 13, 2020
Data Version:
4.20
Data Version:
4.21
Data Release Date:
September 11, 2019
Data Release Date:
October 09, 2020
Data Date Range:
June 13, 2006 to June 30, 2020
Data Date Range:
July 01, 2020 to present
10-09-2020:
A minor version bump (+0.01) has been applied to all CALIPSO data
products due to a required upgrade to the operating system on the CALIPSO production
cluster. All algorithms were re-compiled to process in this new environment with
no change to the underlying science algorithms or inputs.
Summary statement for the release of the CALIPSO Lidar Level 3 Tropospheric
Aerosol Profile Product Version 4.20, September 2019
The version 4.20 release of the CALIPSO lidar level 3 tropospheric aerosol profile
product marks several improvements over the prior version 3 release. Most importantly,
the new level 3 product is constructed from version 4 level 2 input data which are
the highest quality and most sophisticated of all CALIOP level 2 data products. New
level 3 quality screening procedures have been implemented to improve the quality
of statistics reported by the product. Minor changes to science data set names and
bug fixes are also included in this release. These improvements are detailed below.
Product Name Change
With the version 4.20 release, this product has been renamed to the CALIPSO lidar level 3
tropospheric aerosol profile product. The “tropospheric” descriptor was
added to differentiate this product from the recently released CALIPSO lidar level 3
stratospheric aerosol profile product which only reports data above the tropopause.
This is not to say however that the level 3 tropospheric aerosol product covers the
entire troposphere or is bound solely within the troposphere. Because it has a fixed
top altitude of 12 km, (a) some tropospheric aerosols (e.g., in tropical latitudes)
will not be reported in the tropospheric aerosol profile product, and (b) some level 3
range bins will reside above the tropopause (e.g., in polar regions). However,
because the focus of this product is on tropospheric aerosol, only aerosol extinction
for layers detected below the tropopause contributes to the statistics. Aerosol layers
detected above the tropopause are ignored (i.e., all level 2 stratospheric aerosol layers).
Input Data Change
Version 4 level 2 data is used as the input data for V4.20 release of the level 3 tropospheric
aerosol product. This takes advantage of the improved calibration, cloud-aerosol discrimination,
aerosol subtyping, and extinction retrievals embodied in the version 4 data products
(see the
Atmospheric Measurement Techniques special issue: CALIPSO version 4 algorithms and data products).
Algorithm Changes
Several changes in data quality screening have been made in version 4 level 3 in
order to further avoid extinction retrieval errors, reduce the occurrence of cloud
contamination, and mitigate the influence of low-energy laser shots in the South
Atlantic Anomaly region. These changes are described below. Otherwise, the gridding
and averaging methods are the same as in the previous release.
Extinction QC flag filter now evaluates all detected layers and removes aerosol extinction below
In previous versions, the extinction QC flag was only evaluated for layers classified
as aerosol, but not for layers classified as cloud or for stratospheric features. In
version 4, the extinction QC flag is evaluated for all detected layers, regardless
of feature type, and aerosol extinction samples beneath layers having suspicious
extinction QC flags are now rejected. This is important because errors in the top-down
level 2 extinction retrieval are propagated to layers at lower altitudes. These changes
will reduce the impact of suspicious extinction retrievals on level 3 statistics.
In version 4 level 2, the generic stratospheric feature type has been eliminated.
Tropospheric and stratospheric aerosol layers are required to have extinction QC = 0, 1, 16, or 18.
Cloud layers are required to have extinction QC = 0, 1, or 2. The extinction QC value of 2
indicates that the initial lidar ratio is reduced to achieve a successful solution. This
value is allowed for clouds but excluded for aerosols for two reasons. First, for layers
with relatively large optical depths, even small overestimates of the lidar ratio can
cause the extinction solutions to fail before the full vertical extent of a layer has
been retrieved. Obtaining a successful solution thus requires reducing the layer lidar
ratio, and assigning a QC flag of 2. Within the global CALIPSO data set, the median
cloud optical depth is substantially higher than the median aerosol optical depth, and
hence the occurrence frequency of QC = 2 is much higher for clouds than for aerosols
(e.g., only ~0.7% of all unique aerosol layers detected during 2012 had extinction QC
flags of 2, whereas for clouds the fraction rises to ~5.2%). Second, for the uppermost
layer in a column, QC = 2 almost certainly indicates an overestimate of the layer lidar
ratio. However, for layers lying beneath other layers, the interpretation of QC = 2
is ambiguous, as it is much more likely to result from overestimates of the optical
depths in overlying layers. Once again, within the global CALIPSO data set, clouds are
the uppermost layer in a column far more often than aerosols. Readers looking for more
detail on this topic are referred to section 5.3.1 in
Tackett et al, 2018
and section 3.5 in
Young et al., 2018.
Modifications to the extinction QC flag filter increases the rejection frequency from
about 5% to 20% of version 4 aerosol extinction samples. Global mean AOD is reduced by approximately 15%.
New cloud layer fraction filter
A new filter has been implemented to reduce the possibility of cloud contamination
in the level 3 tropospheric aerosol product. It evaluates the fraction of each aerosol
extinction sample containing clouds (i.e., the cloud layer fraction) and rejects those
samples where the fraction exceeds a threshold. In the lower troposphere, level 2 extinction
samples are comprised of at most 30 single shot attenuated backscatter measurements.
Only those measurements classified as aerosol are averaged together and ultimately
contribute to the retrieved aerosol extinction within the 60 m vertical x 5 km horizontal
resolution sample. The level 2 Cloud Layer Fraction science data set (SDS), ranging from 0
to 30, is used to determine the number of single shot values classified as cloud within
an aerosol extinction sample.
Figure 1. Mean level 2 aerosol extinction as a function of cloud layer fraction
(30 = 100% cloudy) for June 2013 at night.
As illustrated in Figure 1, the retrieved aerosol extinction can be spuriously large
when most of the level 2 range bin is dominated by cloud attenuated backscatter. Questionably
large aerosol extinction is reported when the cloud layer fraction is 29 or 30. In fact,
the cloud layer fraction = 30 case is especially dubious because this indicates that
the sample is 100% cloudy, yet an aerosol extinction is reported. This rare special case,
affecting < 0.01% of all aerosol extinction samples in level 2 version 4.20, is being
investigated and will be remedied in a forthcoming level 2 data release.
In accordance with the new level 3 tropospheric aerosol product filter, all aerosol
samples having Cloud Layer Fraction > 28 (corresponding to 97% cloudy) are rejected,
along with all aerosol extinction samples at lower altitudes within the same profile.
For two test months in 2011, this filter rejected less than 0.06% of aerosol extinction
samples and decreased global all-sky AOD by less than 0.04 during daytime and 0.75% at night.
New misclassified opaque cloud filter
Another new cloud contamination filter has been added to level 3 to reject opaque clouds
that have been misclassified as opaque aerosol. This occurs mostly beneath optically
thick cloud overlying cover, which degrades the fidelity of the cloud-aerosol discrimination
algorithm for lower layers. Though opaque aerosol layers are rare, and those that are
misclassified clouds are even more rare, their retrieved extinction is substantial. In
order to avoid overestimating the level 3 mean aerosol extinction, opaque aerosol layers
that are horizontally adjacent to opaque cloud layers are rejected as misclassified clouds.
For two test months in 2011, this filter rejected less than 0.6% of aerosol extinction samples
and decreased global mean AOD by roughly 5%, demonstrating the importance of rejecting these
misclassified features.
Low laser energy mitigation
Since September 2016, CALIOP has been experiencing low energy laser shots,
primarily over the
South Atlantic Anomaly (SAA)
region. This behavior has significantly degraded the quality of affected profiles. In order
to avoid impacting level 3 data quality, all level 2 profiles within the SAA region are
excluded for data months September 2016 onward. Figure 2 below illustrates the excluded
region. More information on low energy laser shots is given on the
Data Advisory Page.
Figure 2. Samples searched by level 3 algorithm for June 2018 at night. No samples are
searched in white regions.
Changes in aerosol optical depth, extinction, and sampling
The primary changes in level 3 mean aerosol optical depth and extinction as compared
to the previous version are caused by differences in the input level 2 data. Specifically,
changes to the aerosol subtyping algorithm and to default aerosol lidar ratios in version 4 level 2
(Kim et al., 2018)
are most influential. The impacts of these changes are readily apparent in Figure 3 below.
AOD decreases occur over the western Atlantic and over the Arabian Sea due to the reclassification
of polluted dust to dusty marine. Decreases occur below 60° S because aerosols over snow, ice, and
tundra are no longer constrained to the clean continental or polluted continental types. AOD
increases occur over northern Africa and over the ocean due to lidar ratio increases for
dust and clean marine. These AOD changes are consistent with version 4 level 2 changes documented by
Kim et al., 2018.
Figure 3. Difference in level 3 mean AOD (V4 - V3) for June-August, 2007-2018, all-sky at night.
Globally, the version 4 level 2 changes alone cause the level 3 cloud-free mean AOD to
increase by 25% at night and increase by 15% at day relative to the previous version. The
new level 3 quality screening procedures (excluding the SAA mitigation filter) decrease the
global mean level 3 AOD, yielding a net AOD increase of 16% at night and 6% at day between the
previous and the current version of the level 3 product. This is because the new quality
screening procedures double the rejection frequency of aerosol extinction samples. Globally,
the all-sky rejection frequency has increased to 30% (similar for night and day), and the
cloud-free rejection frequency has increased to 16% at night and 22% at day. We believe the
new quality screening procedures are effective at only retaining the highest quality aerosol
extinction retrievals and the consequent decrease in samples contributing to the level 3 product
is justified.
Science Data Set (SDS) Changes
Histogram SDS sizes have been increased to accommodate the level 2 version 4 dusty marine aerosol type.
All SDS names containing “Extinction_532” are now “Extinction_Coefficient_532” to conform with level 2 nomenclature.
All SDS names containing “Smoke” are now “Elevated_Smoke” to conform with version 4 level 2 aerosol subtyping nomenclature (Kim et al., 2018).
The list of input level 2 aerosol profile granules contributing to the monthly level 3 file are now included in the file metadata as “List_of_Input_Files”.
Bug Fixes
Relative humidity now reported as ratio rather than percent.
Excluded profiles no longer contribute to meteorological or surface SDS calculations.
Corrected an error where some aerosol samples within layers that should have been rejected by the misclassified cirrus cloud filter were occasionally not rejected.
Sample statistics now contain fill values (-9999) rather than zeros in grid cells where no observations are made during the month.
References
Kim, M.-H., A. H. Omar, J. L. Tackett, M. A. Vaughan, D. M. Winker, C. R. Trepte, Y. Hu, Z. Liu,
L. R. Poole, M. C. Pitts, J. Kar and B. E. Magill, 2018: “The CALIPSO Version 4 Automated Aerosol
Classification and Lidar Ratio Selection Algorithm”, Atmos. Meas. Tech., 11, 6107-6135,
doi:10.5194/amt-11-6107-2018.
Tackett, J. L., D. M. Winker, B. J. Getzewich, M. A. Vaughan, S. A. Young and J. Kar, 2018:
“CALIPSO lidar level 3 aerosol profile product: version 3 algorithm design”,
Atmos. Meas. Tech., 11, 4129-4152,
doi:10.5194/amt-11-4129-2018.
Young, S. A., M. A. Vaughan, J. L. Tackett, A, Garnier, J. B. Lambeth and K. A. Powell, 2018:
“Extinction and Optical Depth Retrievals for CALIPSO’s Version 4 Data Release”,
Atmos. Meas. Tech., 11, 5701-5727,
doi:10.5194/amt-11-5701-2018.
Last Updated: May 09, 2022
Curator: Charles R. Trepte
NASA Official: Charles R. Trepte