Data User’s Guide - Data Product Descriptions - Lidar Level 2 5 km Cloud and Aerosol Profile Products Version 4.20

Optical depth of all clouds, aerosol, or stratospheric layers within a 5 km column. The optical depths are obtained by integrating the 532 nm cloud/aerosol/stratospheric extinction profile, reported in these profile products. For aerosols and stratospheric layers the optical depth is provided at both 532 and 1064 nm wavelengths.

The column optical depth uncertainties provide the only unique QA indicator for each column optical depth product; no other QA or data quality flags are provided. Therefore, users are strongly encouraged to make use of the Extinction_QC ,CAD_Score, cloud water phase, cloud water phase QA values (both found in the Feature_Classification_Flags in the layer products or the Atmospheric_Volume_Description in the profile products), Column_IAB_Cumulative_Probability, and feature opacity information to make their own quality assessment of the column depths.

The column optical depths are a column integral product. Any large uncertainties or poor extinction retrievals from layers within the column (i.e. clouds, aerosols, or stratospheric features) will propogate downward and may impact the quality of all (i.e. cloud, aerosol, and stratospheric) the column optical depths in the column. The following paragraphs outline notes with regard to specific data products that users should be aware of when using column optical depth data.

Opacity: We remind data users that the CALIPSO lidar is only capable of penetrating to the surface if the total column optical depth is less than ~5. (Note that this value takes into account the contribution of multiple scattering.) If the column is opaque to the lidar, then the reported column optical depths are a measure to the apparent base of the lowest feature observed. Feature opacity can be determined by inspecting the extinction QC flag for the lowest extinction coefficient in any 5-km column; if bit 5 (value = 16) is set then the feature is totally attenuating.

Extinction QC: The extinction QC values in the column should be examined to determine if any of the extinction retrievals were bad. Users are reminded that any poor extinction retrievals in the column may impact the quality of all column optical depths. In general, solutions where the final lidar ratio is unchanged (extinction QC = 0) or the extinction solution is constrained (extinction QC = 1) yield physically plausible solutions more often. Conversely, retrievals tend to be more uncertain in those cases where the lidar ratio for either wavelength must be reduced.

CAD_Score and feature sub-type: Features with low absolute CAD_Scores, "special" CAD_Scores, or uncertain aerosol type classifications may impact the quality of the column optical depths. For example, if the top-most feature in the column has a low absolute CAD_Score it is possible that the assigned lidar ratio may be incorrect; this would impact the extinction retrieval for that feature which would lead to an incorrect rescaling of all the data below that feature.

Cloud phase: If there are clouds in the column that are found to have horizontal oriented ice (HOI) crystals it is likely that the quality of the column optical depths are low. The anomalously high backscatter from HOI clouds generally makes the extinction retrieval more difficult. Because all the data below the HOI cloud is rescaled by the retrieved optical depth, the extinction data below could be suspect.

Estimated uncertainty in the column optical depth at each wavelength, computed according to the formulas give in the CALIPSO Version 3 Extinction Uncertainty Document (PDF). Ignoring multiple scattering concerns for the moment, errors in column optical depth calculations typically arise from three main sources: signal-to-noise ratio (SNR) within a layer, calibration accuracy, and the accuracy of the lidar ratio specified for use in the solution. Except for constrained solutions, where a lidar ratio estimate can be obtained directly from the attenuated backscatter data, lidar ratio uncertainties are almost always the dominant contributor to optical depth uncertainties, and the relative error in the layer optical depth will always be at least as large as the relative error in the layer lidar ratio.

Calculation of the layer optical depth uncertainty is an iterative process. On some occasions when the SNR is poor, or an inappropriate lidar ratio is being used, the iteration will attempt to converge asymptotically to positive infinity. Whenever this situation is detected, the iteration is terminated, and the layer optical depth uncertainty is assigned a fixed value of 99.99. Any time an uncertainty of 99.99 is reported, the extinction calculation should be considered to have failed. The associated optical depths cannot be considered reliable, and should therefore be excluded from all science studies.

Note: optical depth uncertainties are reported as absolute errors, not relative errors.

Reports the fraction (by area) within each 5 km horizontal * 60 m vertical range bin containing aerosols or clouds in the profile products. The Aerosol and Cloud Layer Fractions are conceptually identical and the same procedure is used to calculate both quantities.

Since the array elements of the profile products can be larger than the native resolution of the extinction retrieval (5 km * 30 m vs. 5 km * 60 m in the lower troposphere), and because atmospheric features can be identified at horizontal resolutions of 1 km and 1/3 km, the atmospheric composition within each profile range bin is not guaranteed to be homogeneous. Thus, the Aerosol and Cloud Layer Fractions report the fraction (by area) of each 5 km * 60 m profile range bin identified as containing aerosols or clouds by the Scene Classification Algorithms. By referencing Cloud Layer Fraction, the fractional amount of cloud clearing performed within each profile range bin of aerosol backscatter and extinction can be determined.

Figure 4: Cloud clearing scenarios for strongly scattering clouds detected at single shot resolution. Red indicates clouds detected at 1/3 km resolution, blue indicates clouds found at 1 km or coarser resolution, yellow indicates an aerosol layer found at 5 km resolution, and white indicates clear air. Scenarios: Clouds embedded in aerosol (upper panel), clouds embedded in clear air (middle panel), and dense clouds embedded in within a weakly scattering cloud layer (lower panel). Each row extends 5 km horizontally and 30 m vertically. Each column extends 1/3 km horizontally.

Shown in Figure 4 are 3 possible scenarios illustrating how the Cloud Layer Fraction would be reported for each 5 km x 60 m cloud profile range bin. There are at most 30 single shot (1/3 km x 30 m) cloud layers in each 5 km x 60 m cloud profile range bin - fifteen 1/3 km horizontally and two 30 m vertically. In the top panel, red indicates clouds found at 1/3 km resolution and the yellow indicates an aerosol layer found at a 5 km horizontal resolution after the 1/3 km clouds had been removed. In this case, the cloud fraction for the top row would be 11/30 = 0.36. In the middle panel, no features were detected at any coarser spatial resolution after the 1/3 km features were removed. The Atmospheric Volume Description for the 5 km average would report the cell as being "clear air", but the cloud fraction for the top row would still be 11/30 = 0.36. In the lower panel, a cloud was detected in the data remaining after all 1/3 km features had been removed. In this case the cloud fraction would be 1 for all rows shown.

The Aerosol and Cloud Layer Fractions must be values between zero and one, yet both layer fractions are reported as integers between 0 and 30. For example, a value for the Aerosol_Layer_Fraction reported as 11 would indicate a fraction of 11/30 = 0.367.

Atmospheric Volume Description is a profile descriptive flag containing the Feature Classification Flags associated with each 5 km x 60 m (or 5 km x 180 m) range bin in the Profile Products. The Feature Classification Flags provide assessments of (a) feature type (e.g., cloud vs. aerosol vs. stratospheric layer); (b) feature subtype; (c) layer ice-water phase (clouds only); and (d) the amount of horizontal averaging required for layer detection. Note that the interpretation of final three bits in the atmospheric volume description (i.e., the averaging required for detection) is slightly different from the interpretation that would be used for the feature classification flags. These differences are summarized in the table below.

Atmospheric Volume Description, CAD Score and Extinction QC [532|1064] are all profile descriptive flags that are stored in the Level 2 Profile Products in the same manner explained here.

Ideally, each profile descriptive flag would be an array of the size [# altitude bins, # profiles] with each array element providing a complete description of the range-resolved atmospheric state. However, because the range resolution of the Level 1 profile data below ~8.3-km is 30 m, and because the feature-finder, scene classification, and extinction algorithms all operate at this finer spatial resolution, providing a genuinely complete description of the atmospheric state for each 60 m Level 2 range bin requires that the profile descriptive flags be stored as 3-D arrays of the size [#profiles, # altitude bins, 2]. The first dimension, [ : , : , 1], corresponds to the standard altitude array of the Profile Products. Thus, below 8.3 km, the first dimension contains the descriptive flags of the higher of the two full resolution (30 m) bins that comprise the single 60 m bin reported in the Profile Products. Meanwhile, below 8.3 km, the second dimension [: , : , 2] contains the descriptive flags for the lower of the two 30 m range bins. Above 8.3 km, where the range resolution of the Level 1 data is 60 m or greater, the descriptive flags for each single 60 m (or 180 m) range bin are replicated in both array elements.

Figure 5: Wholly fictitious but heuristically useful schematic of layer detection results for a data segment extending 80-km horizontally and 480-m vertically. Yellow/orange/brown indicates an aerosol layer detected at horizontal averaging resolutions of, respectively, 80, 20 or 5 km. Shades of blue likewise represent clouds at 80, 20, and 5 km. Red represents a surface detected layer at 5 km, and the white regions are (presumably) clear air, where no features were found. The right-hand side of the figure shows the atmospheric volume descriptor for columns 1 and 16.

Note that this field is stored in a similar format as the Atmospheric Volume Descriptor.

The extinction QC flags are bit-mapped 16-bit integers, reported for each layer and for each wavelength for which an extinction retrieval was attempted. Aerosol extinction is computed for both wavelengths; cloud extinction is only reported at 532 nm. The information content of each bit is as follows

The bit assignments are additive, so that (for example) an extinction QC value of 18 represents an unconstrained retrieval (bit 1 is NOT set) for which the lidar ratio was reduced to prevent divergence (+2; bit 2 is set), and for which the feature finder has indicated that the layer is opaque (+16; bit 5 is set). For the version 2.01 release, bits 10-15 are not used. Complete information about the conditions under which each extinction QC bit is toggled can be found in the CALIPSO Extinction Retrieval ATBD (PDF) and in Young et al, 2018.

Note that this field is stored in a similar format as the Atmospheric Volume Descriptor.

The cloud-aerosol discrimination (CAD) score provides a numerical confidence level for the classification of layers by the CALIOP cloud-aerosol discrimination algorithm. The CAD algorithm separates clouds and aerosols based on multi-dimensional histograms of scattering properties (e.g., intensity and spectral dependence) as a function of geophysical location. In areas where there is no overlap or intersection between these histograms, features can be classified with complete confidence (i.e., |CAD score| = 100). In the current release (version 4) the CAD algorithm uses five-dimensional (5D) probability density functions (PDFs),i.e. layer mean attenuated backscatter at 532 nm, layer-integrated attenuated backscatter color ratio, altitude, latitude and layer-integrated volume depolarization ratio. These PDFs were newly developed to take into account the significantly improved calibration of CALIOP version 4 level 1 data and using a latitude resolution of 5° (as opposed to 10° in earlier versions) leading to an overall improvement in CAD reliability. Detailed descriptions of the CAD algorithm can be found in Sections 4 and 5 of the CALIPSO Scene Classification ATBD (PDF). Further information on the CAD algorithm architecture may be found in Liu et al., 2010, 2009, 2004.

For the profile product, the CAD score for a layer is replicated over the samples spanning the vertical extent of the layer. For a more complete description of CAD score for layers in version 4, the reader is referred to the layer data description.

The standard CAD scores reported in the CALIPSO layer products range between -100 and 100. The sign of the CAD score indicates the feature type: positive values signify clouds, whereas negative values signify aerosols. The absolute value of the CAD score provides a confidence level for the classification. The larger the magnitude of the CAD score, the higher our confidence that the classification is correct. An absolute value of 100 therefore indicates complete confidence. Absolute values less that 100 indicate some ambiguity in the classification; that is, the scattering properties of the feature are represented to some degree in both the cloud PDF and in the aerosol PDF. In this case, a definitive classification cannot be made; that is, although we can provide a "best guess" classification, this guess could be wrong, with a probability of error related to the absolute value of the CAD score. A value of 0 indicates that a feature has an equal likelihood of being a cloud and an aerosol. Users are encouraged to refer to the CAD score when the cloud and aerosol classification results are used and interpreted.

Beginning with the version 2.01 release, several "special" CAD score values have been added. These are listed in the table below. Each of these new values represents a classification result that is based on additional information beyond that normally considered in the standard CAD algorithm.

Ice clouds: In Version 3 and earlier, ice clouds were assigned a range-independent multiple scattering factor of η₅₃₂ = 0.6. In Version 4, the multiple scattering factor is implemented as a sigmoid approximation function of the layer attenuated backscatter centroid temperature, with η₅₃₂ increasing from 0.46 at 270 K to 0.76 at 190 K. This approximation function was derived from extensive analysis of collocated measurements acquired by the CALIPSO lidar and the CALIPSO IIR, which reconciled observed and theoretical ratios of 532 nm optical depths derived from Version 3 CALIOP measured two-way transmittances to the absorption optical depth retrieved from IIR measurements at 12.05 μm (Garnier et al., 2015). The theoretical ratios are computed assuming severely roughened aggregated columns.

Water clouds: In Version 3 and later versions, ice clouds were assigned a range-independent multiple scattering factor of η₅₃₂ = 0.6 is used. Based on Monte Carlo simulations of multiple scattering, this value appears to be appropriate for semitransparent water clouds (τ < 1). (It is purely coincidental this is the same value used for ice clouds.) For denser water clouds (τ > 1) the multiply-scattered component of the signal becomes much larger than the single-scattered component, η₅₃₂ becomes dependent on both cloud extinction and range into the cloud, and the retrieval becomes very sensitive to errors in the multiple scattering factor used. In these cases the multiple scattering cannot be properly accounted for in the current retrieval algorithm and retrieval results are unreliable.

Aerosols: simulations of multiple scattering effects on retrievals of aerosol layer optical depth indicate the effects are small in most cases. There is uncertainty in these estimates, however, due to poor knowledge of aerosol scattering phase functions. Validation comparisons conducted to date do not indicate significant multiple scattering effects on aerosol extinction profile retrievals. Multiple scattering effects may become significant in dense aerosol layers (σ > 1 /km), but in these cases retrieval errors are usually dominated by uncertainties in the lidar ratio or failure to fully penetrate the layer. In version 2 and later, multiple scattering factors for both wavelengths are set to unity.

Ice water content (IWC) is reported for all ice clouds with valid extinction retrievals detectedby CALIOP. IWC values of 0.00001.54 gm,^-3 account for 99.5% of the values measured by CALIOP, although values up to 10 g/m3 are considered to be physically possible.

Cloud ice water content is a provisional data product that is calculated as a parameterized function of the CALIOP retrieved and interpolated MERRA-2 temperature. This parametrization comes from an emipical relationship derived from aircraft in situ observations, as described in Heymsfield et al., 2014:

Here, σ is the 532 nm volume extinction coefficient in km^-1, and c₀ = 119 gm^-3 and c₁ = 1.22 are coefficients derived from an observed empirical relationship between lidar extinction and an extensive set of in situ measurements of cloud particle properties from numerous field campaigns [1]. The relationship between 532 nm extinction and IWC was developed using IWC data between 0-1.0 gm^-3 with temperatures between -70 and 0 °C. Cloud ice amount has been shown to vary with temperature, cloud particle size distribution, and by location inside a cloud. A temperature-dependent parameterization is being considered and tested for the next CALIOP data release. The effect of particle size distribution on IWC as seen by CALIOP is also currently being evaluated by comparison with in situ cloud data. Preliminary results show that CALIOP IWC has sufficient spatial resolution and precision to realistically resolve cloud morphology. A more detailed preliminary evaluation of the CALIOP version 3 IWC is available as an ILRC extended abstract [2], which includes CALIOP IWC probability distributions and example browse images. For a brief discussion containing critical information needed to intelligently use CALIOP IWC, please see the "data screening" section, below. Range bins where particulates were detected but the extinction retrieval failed are assigned a fill value of -333.

Resolution
IWC is reported at 60 m vertically, with a horizontal spatial resolution of 5 km along-track, and effectively the width of the laser beam across-track.

Precision
The precision of IWC is directly linked to the precision of the associated extinction retrieval. The precision of the extinction retrieval is ultimately limited by signal-to-noise ratio, and this varies between night and day and according to the overhead two-way 532 nm transmission. Therefore, the precision of CALIOP IWC has to be evaluated for each individual case. The team is currently developing a best-case precision estimate for nighttime high altitude Cirrus clouds.

Accuracy
Because this is a provisional data product, assessment of IWC accuracy is ongoing. This assessment can be approached in two different ways; (1) by establishing the accuracy of the 532 nm extinction and Ice particle area to mass conversion empirical relationship on which it is based, or (2) by assessing the IWC product directly. Direct comparison of CALIOP IWC with other measured IWC values includes evaluates a combination of both the extinction retrieval and the IWC parameterization. The uncertainty reported in the data products is related to the uncertainty in the CALIOP extinction retrieval. If the total IWC uncertainty is needed the user is referred to Heymsfield et al., 2014 for the aircraft data-based uncertainty in the empirical extinction to IWC relationship, and to Molod et al., 2015, for uncertainty in the MERRA-2 interpolated temperature.

Data screening
Users that do not wish to dig more deeply are recommended to use data of type=2 (cloud) and phase=1 (randomly-oriented ice) or 3 (horizontally-oriented ice) with a valid extinction retrieval with extinction QCFlag of 0, 1, 2, 16 or 18, as described above. Valid IWC values are considered to be 0.00001 - 10 g/m³. There is a more detailed discussion of the various factors impacting IWC accuracy, below.

CALIOP IWC is a highly derived data product. Besides the parameterization, it relies on these activities:

Cloud determination (CAD)
Bits 6 and 7 in the atmospheric volume descriptor indicates a feature of type 2=cloud, determined using 5-dimensional probability distribution functions as described above. A CAD "score" higher than 70 indicates "high confidence" in the cloud/aerosol discrimination, although the relationship between "CAD score" and phase confidence is currently under evaluation.

Cloud phase determination
The atmospheric volume descriptor indicates that the cloud phase is 1=randomly oriented ice (ROI) or 3=horizontally oriented ice (HOI) as determined for cloud particles (type=2) primarily using the integrated layer attenuated backscatter coefficient and depolarization. Although extinction is available, IWC is not calculated for cloud particle phase 2=water. Users should use caution with HOI data because the preferred horizontal orientation of ice particles in the 0.3 degree, nadir-viewing data (before November 28, 2007) causes anomalously large backscatter that makes the extinction retrievals more difficult, which in turn may affect the accuracy of the IWC. This problem was much alleviated by tilting the viewing angle to 3 degrees after November, 2007, and also by improvement in the Version 4 extinciton retrievals, so it is not considered to be a problem for the Version 4 data after November 28, 2007.

Extinction retrievals
Users that wish to understand the nuances and details of CALIOP extinction retrievals are referred to the V4 lidar ratio, multiple scattering and extinction discussions, above. The extinction quality flag provides information about the type and reliability of extinction retrievals. Because IWC parameterization relies on accurate extinction retrieval, it is recommended to use IWC only where the extinction retrieval is valid. For this reason, data screening for accurate IWC should follow that for extinction coefficients.