Detailed Data Quality Summary for the CALIPSO Version 1.00 Lidar Level 3 GEWEX Cloud Data Monthly Product

The lidar level 3 (L3) cloud product for the Global Energy and Water Cycle Experiment (GEWEX) Cloud Assessment (hereafter the L3 CALIPSO format GEWEX cloud product) reports global distributions of cloud occurrence and cloud top in terms of temperature, pressure and height on a 2-dimentional (2D) longitudinal and latitudinal grid. Each month, there are three files describing statistics during day (with day granules only), night (with night granule only) and all (with both day and night granules).

Due to an elevated frequency of low energy shots within and near the Southern Atlantic Anomaly region since late May 2017, all L3 cloud products including this L3 CALIPSO format GEWEX cloud product have been stopped processing after December 2016. The lidar science working group is currently working on understanding how this issue might affect all levels of CALIPSO product.

Level 2 Input Data

The version 1.00 of level 3 (L3) CALIPSO format GEWEX cloud product is processed from V4.20 level 2 (L2) 5-km merged cloud layer product as shown in the following table.

What is the Global Energy and Water Cycle Experiment (GEWEX) Cloud Assessment?

The goal of the GEWEX Cloud Assessment is to evaluate the overall quality of available, global, long-term cloud products. It focuses on intercomparision of global level 3 (L3) cloud products retrieved from satellites with different measuring techniques, including multi-spectral imagers (e.g., MODIS), multi-angle multi-spectral imagers (e.g., MISR), infrared sounders (e.g., HIRS-NOAA) and active lidar (e.g., CALIPSO).

Following the general guidance, each cloud team used the same approach to create its L3 dataset, providing one file per cloud property, per individual year, and per observation time of day on a 1 degree latitude by 1 degree longitude grid. Table 2 lists the spatial coverage and resolution of the dataset.

Cloud properties include cloud amount (fractional cloud cover), cloud height (in terms of pressure, temperature or altitude), cloud radiative properties (visible optical depth or infrared emissivity), cloud thermodynamic phase (liquid or ice) and bulk microphysical properties (effective particle size and water path).

Cloud properties are reported at three altitude levels, i.e., high, middle and low levels, based on cloud top pressure (CP). Table 3 lists the pressure ranges in hectopascal (hPa) of the three levels.

CALIPSO Contribution to the GEWEX Cloud Assessment

The CALIPSO science team at the NASA Langley Research Center have contributed to the GEWEX cloud assessment since its early stage.

In 2012, the team provided two years 2007-2008 data (hereafter 2012 version) as instructed. These data files were processed from the version 2 level 2 333-m, 5-km cloud and 5-km aerosol layer products. The cloud parameters included cloud amount and cloud top in terms of temperature and altitude.

Recently the team expanded the dataset till December 2016 due to a strong desire from the cloud research community. This new dataset (hereafter 2019 version) was processed from the version 4.20 level 2 5-km merged layer product. The cloud parameters included cloud amount and cloud top in terms of temperature, pressure and altitude.

Compared to 2012 version data, there are several improvements in the 2019 version data. First, the 2019 version data were processed with version 4.20 level 2 data, which included numbers of improvements in retrieval algorithms. For example, a better cloud aerosol discrimination, an improved cloud ice-water phase discrimination and a new lidar surface detection. Second, it included cloud top pressure which was not reported in the version 2 level 2 data product back then. Third, the new version data described a cloud top using the temperature and pressure at the cloud top instead of those at the centroid. In fact, starting from version 3 level 2 product, cloud temperature and pressure are reported at cloud top, centroid and cloud bottom. In the version 2 level 2 data product, only cloud temperature at centroid was reported. Consequently, if a user compares the cloud top temperature patterns between 2012 and 2019 versions data, s/he would notice some differences. The CALIOP science team evaluated the differences and recommended using cloud top temperature and pressure since using the centroid temperature led to a biased-high ice cloud top temperature. Last, the 2012 version data were available in years 2007 and 2008 only, while the 2019 version data expanded from June 2006 to December 2016.

The two versions of data were produced using slightly different quality filters, partly due to improvements of version 4 level 2 data product which made several quality filters obsolete. For example, improved aerosol cloud scene classifications algorithm suggested no correction needed for misclassified aerosols at the Greenland and Antarctic, and misclassified clouds at the mid-latitude Northern hemisphere, which were found in the version 2 level 2 data product. More on quality filters would be explained later.

The Level 3 Standard and CALIPSO formats GEWEX Cloud Products

The level 3 standard GEWEX product refers to the cloud product that the CALIPSO science team provided to the GEWEX cloud assessment. The data are available on the cloud assessment website with all L3 cloud products from other satellites. The level 3 CALIPSO format GEWEX product means a cloud product with the GEWEX cloud contents but wrapped in the traditional CALIPSO data format. It is specially designed to keep consistency with other CALIOP level 3 cloud and aerosol products which are reported monthly. The product is hosted both on the NASA Atmospheric Science Data Center and Cloud-Aerosol-Water-Radiation Interactions (ICARE) Data and Service Center websites.

Both standard and CALIPSO formats GEWEX cloud products were produced from the same intermediate cloud mask files with merged cloud layers retrieved at different horizontal averaging from the L2 cloud product after applying a numbers of quality filters. They have the exact same contents but with different formats and parameter names.

The standard format GEWEX cloud product describes each cloud property as yearly files, while the CALIPSO format GEWEX cloud product reports all cloud properties as monthly files. The file formats are different, with the standard format GEWEX product data files in Network Common Data Form (netCDF) 4 format and the CALIPSO format GEWEX product data files in Hierarchical Data Format (HDF) 4 format. Other than different file names, they have different science data set names too. The standard format GEWEX cloud product followed the GEWEX cloud assessment general guidance and used the same parameter names as other L3 cloud products participated the GEWEX cloud assessment. The CALIPSO format GEWEX cloud product followed the traditional CALIPSO naming system. It also utilized vgroup data structure for a neat organization of hundreds of parameters. A user could find one-to-one correspondence of science data set names by reading the attributes of the CALIPSO format GEWEX data files or from the Data Product Description page.

Flavors and Parameters of the CALIPSO Format GEWEX Cloud Product

Four flavors of the CALIPSO format GEWEX cloud products were developed following four different philosophies: top layer, passive, column and opaque. The top layer flavor chooses the top layer cloud only in each profile. Similar to top layer flavor, passive flavor chooses the top layer cloud but which would be detected by a passive sensor only. This typically refers to an optically thick cloud layer with the layer optical depth (OD) greater than or equal to 0.3, which could be either a water cloud layer assuming all water clouds detected by the CALIOP were optically thick or an ice cloud layer with OD ≥ 0.3. As explained in the Extinction Retrieval Algorithm of the CALIOP Algorithm Theoretical Basis Document, the current extinction retrieval algorithm and multiple scattering parameterization are not well suited to water clouds thus the water cloud extinction coefficients and layer OD should not be used. The column flavor considers all detected cloud layers in each profile, i.e., considering a multiple cloud layer scenario. The opaque flavor includes profiles with opaque cloud layers only. Note on the GEWEX cloud assessment website, the 2012 version level 3 standard format GEWEX cloud product is the top layer flavor.

Table 4: Parameters included for four flavors of the level 3 CALIPSO format GEWEX cloud product. Letters “a” and “h” mean parameters are reported as average and histogram, “/” means this parameter is not reported for this flavor.

As mentioned before, cloud properties are further described based on cloud top altitude and cloud phase. Table 4 listed parameters reported for each flavor. The cloud amount and cloud top of the four flavors can be slightly different. Figure 1 presents the average cloud amount in July, 2008. The patterns are the same for the top layer and column flavors. The cloud coverage and cloud amount magnitude are slightly larger than those of passive flavor, since profiles with optically thin ice cloud as the top layer would be considered cloud free in the passive flavor scenario. Those of opaque flavor are the smallest as all transparent profiles with detected lidar surface regardless cloudy or not are considered as “cloud free”.

Figure 1: Cloud amount patterns of four flavors in July 2008. The patterns are plotted as functions of longitude as X axis and latitude as Y axis. The cloud amount is shown as a fraction from 0.0 to 1.0. The estimated global mean is given at the top right in each panel.

Grid Structure

Different from other CALIPSO level 3 cloud and aerosol products describing cloud and aerosol properties with a 3-dimensional (3D) grid, this cloud product describes cloud property in a 2-dimensional (2D) grid at three altitude regions: high, middle and low. Comparisons of ranges and grid resolutions of all current L3 CALIPSO products are listed in Table 5. The grid resolutions of L3 cloud products are specially designed to be compatible with L3 aerosol products so a user could easily merge two or more products if needed.

Quality Filters Information

The current version of level 3 CALIPSO format GEWEX cloud product is produced applying similar quality filters as level 3 cloud occurrence product, to ensure only high confidence clouds retrieved from the level 2 algorithm going to level 3 aggregation algorithm. Other than cloud feature quality filter, cloud phase quality filter and low water cloud filter, one extra range filter has been applied to accommodate the cloud top temperature/pressure/altitude ranges defined by the GEWEX cloud assessment general guidance.

1. Cloud Feature Quality Filter: this filter chooses confident cloud features only. The quality assurance FeatureQA is set to low, medium or high, i.e., FeatureQA = 1, 2, 3. The FeatureQA is decoded from Feature_Classification_Flags. This is equivalent to choosing clouds with CAD score between 20 and 100. Compared to version 3 feature retrieval algorithm, the version 4 algorithm was less confident resulting a smaller percentage of high confident clouds with CAD between 70 and 100. To maximize clouds sampling count, low and medium confidence clouds are allowed to enter the next stage of filtering.
    
2. Cloud Phase Quality Filter: this filter chooses high confident ice and water clouds only. If cloud phase was water (cloud phase = 2) or ice (cloud phase = 1 or 3) the cloud phase quality assurance should be high (PhaseQA = 3). Note ice clouds include both randomly orientated ice (cloud phase = 1) and horizontally orientated ice (cloud phase = 3). A special rule is made for unknown/not determined phase clouds (cloud phase = 0): choose unknown/not determined phase clouds with PhaseQA = 0, as all these clouds have none confidence in PhaseQA (PhaseQA = 0).
    
3. Low water cloud filter: this filter excludes water clouds with cloud top less than 8.2 km identified from coarse horizontal averaging (HA), i.e., HA = 5, 20 or 80 km. An assumption is that optically thick broken water clouds such as marine boundary clouds would be detected from HA = 333 m attenuated backscattering. If the same water cloud were reported from HA = 5, 20, or 80 km, including this water clouds would overestimate the water cloud coverage. There is also an argument of possibly underestimation of optically thin water cloud coverage over the Southern Ocean with this filter. More analyses would be conducted in future to weigh the pros and cons of this filter.
    
4. Temperature/pressure/altitude range filters: following the GEWEX cloud assessment general guidance, only clouds with cloud top temperature between 150 K and 320 K, and cloud top pressure between 100 hPa and 1100 hPa and cloud top altitude between 0.0 km and 20.0 km are included in the monthly statistics.
   

   Figure 2: High cloud occurrence of top layer flavor, without (a) and with (b) range filters at night in July 2008. The format of each panel is the same as that in Figure 1.

   
    
   These range filters excluded high thin cirrus clouds around tropics and polar stratospheric clouds at polar regions, which can be observed by the CALIOP only due to its high sensitivity but probably would be missed by a passive sensor. As a result, it leads to slightly smaller cloud occurrence as shown in Figure 2, hotter cloud top temperature, higher cloud top pressure and lower cloud top in those regions. However, if a user is interested in comparing cloud top from CALIOP and passive sensors, the exclusion seems a valid approximation since those high clouds appear “clear” or “invisible” for passive sensors.
    

How cloud amount mean and histogram are estimated?

Figure 3: An illustration of two CALIPSO tracks with clear profiles as blue dots and cloudy profiles as white dots at a 1 degree longitude by 1 degree latitude grid (left). The average or mean CA in this grid is estimated using the formula shown on right.

Figure 3 explained how the cloud amount (CA) mean or average at each 2-dimensional (2D) grid was estimated, adopting a similar methodology of a passive sensor. On the left, two CALIPSO orbits were illustrated with gray lines and each dot represents one profile, while a white dot means a cloudy profile and a blue one a clear profile. A track CA is estimated from the cloudy and clear profiles along the track within each grid assuming the track CA would be able to represent the grid CA. For these two tracks, the track CA would be 4/5 and 1/3, respectively. The monthly average CA would be the means of all track CA and each track CA would be binned into the CA histogram. If these two tracks were the only tracks passing this grid during the whole month, the monthly average CA would be 0.57 as shown on the right. One track CA 4/5 (0.8) would go to bin 8 holding 0.8 ≤ CA < 0.9 and other CA 1/3 (0.33) would go to bin 3 holding 0.3 ≤ CA < 0.4

No Clouds Observed Above 17 km With The Pressure Range Filter

As mentioned before, three range filters were applied to clouds to ensure the cloud top are within the histogram ranges as defined in GEWEX cloud assessment general guidance: cloud top temperature is from 150 K to 320 K, cloud top pressure is from 100 hPa to 1100 hPa and cloud top altitude from 0.0 km to 20.0 km.

These filters removed thin cirrus clouds around tropical region and polar stratospheric clouds at polar regions detected by the CALIOP, resulting a slightly smaller cloud occurrence, hotter cloud top temperature, higher cloud top pressure and lower cloud top at those regions.

Figure 4: Zonal patterns of atmosphere pressure in 2008-01 and 2008-07. The patterns are filled in black if pressure less than 100 hPa.

Other than those, they also led to no clouds reported above 17 km as the pressure is less than 100 hPa above 17 km globally in the Modern Era Retrospective-Analysis for Research (MERRA-2) reanalysis product provided by the NASA GMAO Data Assimilation System. As shown in Figure 4, the pressure is less than 100 hPa at 17 km at all latitudes in both winter and summer, though the exact altitude threshold varies with latitude and there is a seasonal asymmetry. Note the contours were filled in black if pressure less than 100 hPa.

When applying the pressure range filter, it automatically chose clouds with cloud top less about 17 km. Therefore, there would be no clouds with cloud top higher than 17.0 km in the cloud top (CZ) histogram as shown in Figure 5.

Figure 5: Histogram of cloud top height of top layer flavor from 2006/06 to 2016/12. The number of each cloud top height bin in each month are represented as logarithm.