Detailed Data Quality Summary: CALIPSO Lidar Level 3 Tropospheric Aerosol Profile Data Monthly Products Versions 4.20 and 4.21

Last Updated: October 13, 2020

10-09-2020: A minor version bump (+0.01) has been applied to all CALIPSO data products due to a required upgrade to the operating system on the CALIPSO production cluster. All algorithms were re-compiled to process in this new environment with no change to the underlying science algorithms or inputs.

Summary statement for the release of the CALIPSO Lidar Level 3 Tropospheric Aerosol Profile Product Version 4.20, September 2019

The version 4.20 release of the CALIPSO lidar level 3 tropospheric aerosol profile product marks several improvements over the prior version 3 release. Most importantly, the new level 3 product is constructed from version 4 level 2 input data which are the highest quality and most sophisticated of all CALIOP level 2 data products. New level 3 quality screening procedures have been implemented to improve the quality of statistics reported by the product. Minor changes to science data set names and bug fixes are also included in this release. These improvements are detailed below.

Product Name Change

With the version 4.20 release, this product has been renamed to the CALIPSO lidar level 3 tropospheric aerosol profile product. The “tropospheric” descriptor was added to differentiate this product from the recently released CALIPSO lidar level 3 stratospheric aerosol profile product which only reports data above the tropopause. This is not to say however that the level 3 tropospheric aerosol product covers the entire troposphere or is bound solely within the troposphere. Because it has a fixed top altitude of 12 km, (a) some tropospheric aerosols (e.g., in tropical latitudes) will not be reported in the tropospheric aerosol profile product, and (b) some level 3 range bins will reside above the tropopause (e.g., in polar regions). However, because the focus of this product is on tropospheric aerosol, only aerosol extinction for layers detected below the tropopause contributes to the statistics. Aerosol layers detected above the tropopause are ignored (i.e., all level 2 stratospheric aerosol layers).

Input Data Change

Version 4 level 2 data is used as the input data for V4.20 release of the level 3 tropospheric aerosol product. This takes advantage of the improved calibration, cloud-aerosol discrimination, aerosol subtyping, and extinction retrievals embodied in the version 4 data products (see the Atmospheric Measurement Techniques special issue: CALIPSO version 4 algorithms and data products).

Algorithm Changes

Several changes in data quality screening have been made in version 4 level 3 in order to further avoid extinction retrieval errors, reduce the occurrence of cloud contamination, and mitigate the influence of low-energy laser shots in the South Atlantic Anomaly region. These changes are described below. Otherwise, the gridding and averaging methods are the same as in the previous release.

In previous versions, the extinction QC flag was only evaluated for layers classified as aerosol, but not for layers classified as cloud or for stratospheric features. In version 4, the extinction QC flag is evaluated for all detected layers, regardless of feature type, and aerosol extinction samples beneath layers having suspicious extinction QC flags are now rejected. This is important because errors in the top-down level 2 extinction retrieval are propagated to layers at lower altitudes. These changes will reduce the impact of suspicious extinction retrievals on level 3 statistics.

In version 4 level 2, the generic stratospheric feature type has been eliminated. Tropospheric and stratospheric aerosol layers are required to have extinction QC = 0, 1, 16, or 18. Cloud layers are required to have extinction QC = 0, 1, or 2. The extinction QC value of 2 indicates that the initial lidar ratio is reduced to achieve a successful solution. This value is allowed for clouds but excluded for aerosols for two reasons. First, for layers with relatively large optical depths, even small overestimates of the lidar ratio can cause the extinction solutions to fail before the full vertical extent of a layer has been retrieved. Obtaining a successful solution thus requires reducing the layer lidar ratio, and assigning a QC flag of 2. Within the global CALIPSO data set, the median cloud optical depth is substantially higher than the median aerosol optical depth, and hence the occurrence frequency of QC = 2 is much higher for clouds than for aerosols (e.g., only ~0.7% of all unique aerosol layers detected during 2012 had extinction QC flags of 2, whereas for clouds the fraction rises to ~5.2%). Second, for the uppermost layer in a column, QC = 2 almost certainly indicates an overestimate of the layer lidar ratio. However, for layers lying beneath other layers, the interpretation of QC = 2 is ambiguous, as it is much more likely to result from overestimates of the optical depths in overlying layers. Once again, within the global CALIPSO data set, clouds are the uppermost layer in a column far more often than aerosols. Readers looking for more detail on this topic are referred to section 5.3.1 in Tackett et al, 2018 and section 3.5 in Young et al., 2018. Modifications to the extinction QC flag filter increases the rejection frequency from about 5% to 20% of version 4 aerosol extinction samples. Global mean AOD is reduced by approximately 15%.

A new filter has been implemented to reduce the possibility of cloud contamination in the level 3 tropospheric aerosol product. It evaluates the fraction of each aerosol extinction sample containing clouds (i.e., the cloud layer fraction) and rejects those samples where the fraction exceeds a threshold. In the lower troposphere, level 2 extinction samples are comprised of at most 30 single shot attenuated backscatter measurements. Only those measurements classified as aerosol are averaged together and ultimately contribute to the retrieved aerosol extinction within the 60 m vertical x 5 km horizontal resolution sample. The level 2 Cloud Layer Fraction science data set (SDS), ranging from 0 to 30, is used to determine the number of single shot values classified as cloud within an aerosol extinction sample.

Figure 1. Mean level 2 aerosol extinction as a function of cloud layer fraction (30 = 100% cloudy) for June 2013 at night.

As illustrated in Figure 1, the retrieved aerosol extinction can be spuriously large when most of the level 2 range bin is dominated by cloud attenuated backscatter. Questionably large aerosol extinction is reported when the cloud layer fraction is 29 or 30. In fact, the cloud layer fraction = 30 case is especially dubious because this indicates that the sample is 100% cloudy, yet an aerosol extinction is reported. This rare special case, affecting < 0.01% of all aerosol extinction samples in level 2 version 4.20, is being investigated and will be remedied in a forthcoming level 2 data release.

In accordance with the new level 3 tropospheric aerosol product filter, all aerosol samples having Cloud Layer Fraction > 28 (corresponding to 97% cloudy) are rejected, along with all aerosol extinction samples at lower altitudes within the same profile. For two test months in 2011, this filter rejected less than 0.06% of aerosol extinction samples and decreased global all-sky AOD by less than 0.04 during daytime and 0.75% at night.

Another new cloud contamination filter has been added to level 3 to reject opaque clouds that have been misclassified as opaque aerosol. This occurs mostly beneath optically thick cloud overlying cover, which degrades the fidelity of the cloud-aerosol discrimination algorithm for lower layers. Though opaque aerosol layers are rare, and those that are misclassified clouds are even more rare, their retrieved extinction is substantial. In order to avoid overestimating the level 3 mean aerosol extinction, opaque aerosol layers that are horizontally adjacent to opaque cloud layers are rejected as misclassified clouds. For two test months in 2011, this filter rejected less than 0.6% of aerosol extinction samples and decreased global mean AOD by roughly 5%, demonstrating the importance of rejecting these misclassified features.

Since September 2016, CALIOP has been experiencing low energy laser shots, primarily over the South Atlantic Anomaly (SAA) region. This behavior has significantly degraded the quality of affected profiles. In order to avoid impacting level 3 data quality, all level 2 profiles within the SAA region are excluded for data months September 2016 onward. Figure 2 below illustrates the excluded region. More information on low energy laser shots is given on the Data Advisory Page.

Figure 2. Samples searched by level 3 algorithm for June 2018 at night. No samples are searched in white regions.

Changes in aerosol optical depth, extinction, and sampling

The primary changes in level 3 mean aerosol optical depth and extinction as compared to the previous version are caused by differences in the input level 2 data. Specifically, changes to the aerosol subtyping algorithm and to default aerosol lidar ratios in version 4 level 2 (Kim et al., 2018) are most influential. The impacts of these changes are readily apparent in Figure 3 below. AOD decreases occur over the western Atlantic and over the Arabian Sea due to the reclassification of polluted dust to dusty marine. Decreases occur below 60° S because aerosols over snow, ice, and tundra are no longer constrained to the clean continental or polluted continental types. AOD increases occur over northern Africa and over the ocean due to lidar ratio increases for dust and clean marine. These AOD changes are consistent with version 4 level 2 changes documented by Kim et al., 2018.

Figure 3. Difference in level 3 mean AOD (V4 - V3) for June-August, 2007-2018, all-sky at night.

Globally, the version 4 level 2 changes alone cause the level 3 cloud-free mean AOD to increase by 25% at night and increase by 15% at day relative to the previous version. The new level 3 quality screening procedures (excluding the SAA mitigation filter) decrease the global mean level 3 AOD, yielding a net AOD increase of 16% at night and 6% at day between the previous and the current version of the level 3 product. This is because the new quality screening procedures double the rejection frequency of aerosol extinction samples. Globally, the all-sky rejection frequency has increased to 30% (similar for night and day), and the cloud-free rejection frequency has increased to 16% at night and 22% at day. We believe the new quality screening procedures are effective at only retaining the highest quality aerosol extinction retrievals and the consequent decrease in samples contributing to the level 3 product is justified.

Science Data Set (SDS) Changes

• Histogram SDS sizes have been increased to accommodate the level 2 version 4 dusty marine aerosol type.
• All SDS names containing “Extinction_532” are now “Extinction_Coefficient_532” to conform with level 2 nomenclature.
• All SDS names containing “Smoke” are now “Elevated_Smoke” to conform with version 4 level 2 aerosol subtyping nomenclature (Kim et al., 2018).
• The list of input level 2 aerosol profile granules contributing to the monthly level 3 file are now included in the file metadata as “List_of_Input_Files”.

Bug Fixes

• Relative humidity now reported as ratio rather than percent.
• Excluded profiles no longer contribute to meteorological or surface SDS calculations.
• Corrected an error where some aerosol samples within layers that should have been rejected by the misclassified cirrus cloud filter were occasionally not rejected.
• Sample statistics now contain fill values (-9999) rather than zeros in grid cells where no observations are made during the month.

References

Kim, M.-H., A. H. Omar, J. L. Tackett, M. A. Vaughan, D. M. Winker, C. R. Trepte, Y. Hu, Z. Liu, L. R. Poole, M. C. Pitts, J. Kar and B. E. Magill, 2018: “The CALIPSO Version 4 Automated Aerosol Classification and Lidar Ratio Selection Algorithm”, Atmos. Meas. Tech., 11, 6107-6135, doi:10.5194/amt-11-6107-2018.

Tackett, J. L., D. M. Winker, B. J. Getzewich, M. A. Vaughan, S. A. Young and J. Kar, 2018: “CALIPSO lidar level 3 aerosol profile product: version 3 algorithm design”, Atmos. Meas. Tech., 11, 4129-4152, doi:10.5194/amt-11-4129-2018.

Young, S. A., M. A. Vaughan, J. L. Tackett, A, Garnier, J. B. Lambeth and K. A. Powell, 2018: “Extinction and Optical Depth Retrievals for CALIPSO’s Version 4 Data Release”, Atmos. Meas. Tech., 11, 5701-5727, doi:10.5194/amt-11-5701-2018.