Detailed Data Quality Summary for the CALIPSO Version 1.00 Lidar Level 3 GEWEX Cloud Data Monthly Product
Data Version:
1.00
Data Release Date:
December 2019
Data Date Range:
June 13, 2006 to December 31, 2016
The lidar level 3 (L3) cloud product for the Global Energy and Water Cycle Experiment
(GEWEX) Cloud Assessment (hereafter the L3 CALIPSO format GEWEX cloud product) reports
global distributions of cloud occurrence and cloud top in terms of temperature, pressure
and height on a 2-dimentional (2D) longitudinal and latitudinal grid. Each month, there
are three files describing statistics during day (with day granules only), night
(with night granule only) and all (with both day and night granules).
Due to an elevated frequency of low energy shots within and near the Southern
Atlantic Anomaly region since late May 2017, all L3 cloud products including this
L3 CALIPSO format GEWEX cloud product have been stopped processing after December
2016. The lidar science working group is currently working on understanding how
this issue might affect all levels of CALIPSO product.
Level 2 Input Data
The version 1.00 of level 3 (L3) CALIPSO format GEWEX cloud product is processed
from V4.20 level 2 (L2) 5-km merged cloud layer product as shown in the following
table.
Table 1: Input data for the L3 CALIPSO format GEWEX cloud product.
Lidar Level 2 Input Data for Level 3 CALIPSO format GEWEX cloud product, V1.00
Level 2 Version
Data Date Range
V4.20
June 2006 - December 2016
What is the Global Energy and Water Cycle Experiment (GEWEX) Cloud Assessment?
The goal of
the GEWEX Cloud Assessment
is to evaluate the overall quality of available, global, long-term cloud products.
It focuses on intercomparision of global level 3 (L3) cloud products retrieved from
satellites with different measuring techniques, including multi-spectral imagers
(e.g., MODIS), multi-angle multi-spectral imagers (e.g., MISR), infrared sounders
(e.g., HIRS-NOAA) and active lidar (e.g., CALIPSO).
Following
the general guidance,
each cloud team used the same approach to create its L3 dataset, providing one file
per cloud property, per individual year, and per observation time of day on a 1 degree
latitude by 1 degree longitude grid. Table 2 lists the spatial coverage and resolution
of the dataset.
Table 2: The spatial coverage and resolution of L3 GEWEX cloud product.
Spatial Coverage
Spatial Resolution
360° longitude (180° W to 180° E)
1.0° longitude
180° latitude (90° S to 90° N)
1.0° latitude
Cloud properties include cloud amount (fractional cloud cover), cloud height
(in terms of pressure, temperature or altitude), cloud radiative properties
(visible optical depth or infrared emissivity), cloud thermodynamic phase (liquid or ice)
and bulk microphysical properties (effective particle size and water path).
Cloud properties are reported at three altitude levels, i.e., high, middle and
low levels, based on cloud top pressure (CP). Table 3 lists the pressure ranges in
hectopascal (hPa) of the three levels.
Table 3: Pressure ranges of three altitude regions.
Altitude Region
Cloud Top Pressure Range
High-level
CP < 400 hPa
Mid-level
440 hPa ≤ CP < 680 hPa
Low-level
680 hPa ≤ CP
CALIPSO Contribution to the GEWEX Cloud Assessment
The CALIPSO science team at the NASA Langley Research Center have contributed
to the GEWEX cloud assessment since its early stage.
In 2012, the team provided two years
2007-2008 data
(hereafter 2012 version) as
instructed. These data files were processed from the version 2 level 2 333-m, 5-km
cloud and 5-km aerosol layer products. The cloud parameters included cloud amount
and cloud top in terms of temperature and altitude.
Recently the team expanded the dataset till December 2016 due to a strong desire
from the cloud research community. This new dataset (hereafter 2019 version) was
processed from the version 4.20 level 2 5-km merged layer product. The cloud parameters
included cloud amount and cloud top in terms of temperature, pressure and altitude.
Compared to 2012 version data, there are several improvements in the 2019 version data.
First, the 2019 version data were processed with version 4.20 level 2 data, which included numbers of
improvements in retrieval algorithms.
For example, a better cloud aerosol discrimination, an improved cloud ice-water phase
discrimination and a new lidar surface detection. Second, it included cloud top
pressure which was not reported in the version 2 level 2 data product back then.
Third, the new version data described a cloud top using the temperature and pressure
at the cloud top instead of those at the centroid. In fact, starting from version 3
level 2 product, cloud temperature and pressure are reported at cloud top, centroid
and cloud bottom. In the version 2 level 2 data product, only cloud temperature at
centroid was reported. Consequently, if a user compares the cloud top temperature
patterns between 2012 and 2019 versions data, s/he would notice some differences.
The CALIOP science team evaluated the differences and recommended using cloud top
temperature and pressure since using the centroid temperature led to a biased-high
ice cloud top temperature. Last, the 2012 version data were available in years 2007
and 2008 only, while the 2019 version data expanded from June 2006 to December 2016.
The two versions of data were produced using slightly different quality filters,
partly due to improvements of version 4 level 2 data product which made several
quality filters obsolete. For example, improved aerosol cloud scene classifications
algorithm suggested no correction needed for misclassified aerosols at the Greenland
and Antarctic, and misclassified clouds at the mid-latitude Northern hemisphere, which
were found in the version 2 level 2 data product. More on quality filters would be
explained later.
The Level 3 Standard and CALIPSO formats GEWEX Cloud Products
The level 3 standard GEWEX product refers to the cloud product that the CALIPSO
science team provided to the GEWEX cloud assessment. The data are available on
the cloud assessment website
with all L3 cloud products from other satellites. The level 3 CALIPSO format GEWEX
product means a cloud product with the GEWEX cloud contents but wrapped in the
traditional CALIPSO data format. It is specially designed to keep consistency with
other CALIOP level 3 cloud and aerosol products which are reported monthly. The
product is hosted both on the
NASA Atmospheric Science Data Center
and Cloud-Aerosol-Water-Radiation Interactions (ICARE) Data and Service Center
websites.
Both standard and CALIPSO formats GEWEX cloud products were produced from the
same intermediate cloud mask files with merged cloud layers retrieved at different
horizontal averaging from the L2 cloud product after applying a numbers of quality
filters. They have the exact same contents but with different formats and parameter
names.
The standard format GEWEX cloud product describes each cloud property as yearly
files, while the CALIPSO format GEWEX cloud product reports all cloud properties
as monthly files. The file formats are different, with the standard format GEWEX
product data files in Network Common Data Form (netCDF) 4 format and the CALIPSO
format GEWEX product data files in Hierarchical Data Format (HDF) 4 format. Other
than different file names, they have different science data set names too. The standard
format GEWEX cloud product followed the
GEWEX cloud assessment general guidance
and used the same parameter names as other L3 cloud products participated the GEWEX
cloud assessment. The CALIPSO format GEWEX cloud product followed the traditional
CALIPSO naming system. It also utilized vgroup data structure for a neat organization
of hundreds of parameters. A user could find one-to-one correspondence of science data
set names by reading the attributes of the CALIPSO format GEWEX data files or from the
Data Product Description page.
Flavors and Parameters of the CALIPSO Format GEWEX Cloud Product
Four flavors of the CALIPSO format GEWEX cloud products were developed following
four different philosophies: top layer, passive, column and opaque. The top layer
flavor chooses the top layer cloud only in each profile. Similar to top layer
flavor, passive flavor chooses the top layer cloud but which would be detected
by a passive sensor only. This typically refers to an optically thick cloud layer
with the layer optical depth (OD) greater than or equal to 0.3, which could be either
a water cloud layer assuming all water clouds detected by the CALIOP were optically
thick or an ice cloud layer with OD ≥ 0.3. As explained in the
Extinction Retrieval Algorithm
of the CALIOP Algorithm Theoretical Basis Document, the current extinction retrieval
algorithm and multiple scattering parameterization are not well suited to water
clouds thus the water cloud extinction coefficients and layer OD should not be used.
The column flavor considers all detected cloud layers in each profile, i.e.,
considering a multiple cloud layer scenario. The opaque flavor includes profiles
with opaque cloud layers only. Note on the GEWEX cloud assessment website, the 2012
version level 3 standard format GEWEX cloud product is the top layer flavor.
Table 4: Parameters included for four flavors of the level 3 CALIPSO format GEWEX cloud product. Letters “a” and “h” mean parameters are reported as average and histogram, “/” means this parameter is not reported for this flavor.
As mentioned before, cloud properties are further described based on cloud top
altitude and cloud phase. Table 4 listed parameters reported for each flavor. The
cloud amount and cloud top of the four flavors can be slightly different. Figure 1
presents the average cloud amount in July, 2008. The patterns are the same for the
top layer and column flavors. The cloud coverage and cloud amount magnitude are
slightly larger than those of passive flavor, since profiles with optically thin
ice cloud as the top layer would be considered cloud free in the passive flavor
scenario. Those of opaque flavor are the smallest as all transparent profiles with
detected lidar surface regardless cloudy or not are considered as “cloud free”.
Figure 1: Cloud amount patterns of four flavors in July 2008. The patterns are plotted as functions of longitude as X axis and latitude as Y axis. The cloud amount is shown as a fraction from 0.0 to 1.0. The estimated global mean is given at the top right in each panel.
Grid Structure
Different from other CALIPSO level 3 cloud and aerosol products describing cloud
and aerosol properties with a 3-dimensional (3D) grid, this cloud product describes
cloud property in a 2-dimensional (2D) grid at three altitude regions: high, middle
and low. Comparisons of ranges and grid resolutions of all current L3 CALIPSO products
are listed in Table 5. The grid resolutions of L3 cloud products are specially designed
to be compatible with L3 aerosol products so a user could easily merge two or more
products if needed.
Table 5: Grid resolution and ranges of all current level 3 CALIPSO cloud and aerosol products.
L3 GEWEX Cloud Product
L3 Cloud Occurrence Product
L3 Cloud Product
L3 Tropospheric Aerosol Product
L3 Stratospheric Aerosol Product
Grid Resolution
lon. 1.0° x lat. 1.0°
lon. 2.5° x lat 2.0° x alt. 60 m
lon. 2.5° x lat 2.0° x alt. 120 m
lon. 5° x lat 2.0° x alt. 60 m
lon. 20.0° x lat 5.0° x alt. 900 m
Range
lon. 180.0° ˜ 180.0°, lat. -90.0° ˜ 90.0°
lon. 180.0° ˜ 180.0°, lat. -85.0° ˜ 85.0°, alt. -0.5 km ˜ 20.2 km
lon. 180.0° ˜ 180.0°, lat. -85.0° ˜ 85.0°, alt. -0.5 km ˜ 20.2 km
lon. 180.0° ˜ 180.0°, lat. -85.0° ˜ 85.0°, alt. -0.5 km ˜ 12.0 km
lon. 180.0° ˜ 180.0°, lat. -85.0° ˜ 85.0°, alt. 8.2 km ˜ 36.2 km
Quality Filters Information
The current version of level 3 CALIPSO format GEWEX cloud product is produced
applying similar quality filters as level 3 cloud occurrence product, to ensure
only high confidence clouds retrieved from the level 2 algorithm going to level 3
aggregation algorithm. Other than cloud feature quality filter, cloud phase quality
filter and low water cloud filter, one extra range filter has been applied to
accommodate the cloud top temperature/pressure/altitude ranges defined by the
GEWEX cloud assessment general guidance.
Cloud Feature Quality Filter: this filter chooses confident cloud features
only. The quality assurance FeatureQA is set to low, medium or high, i.e.,
FeatureQA = 1, 2, 3. The FeatureQA is decoded from
Feature_Classification_Flags.
This is equivalent to choosing clouds with
CAD score
between 20 and 100. Compared to version 3
feature retrieval algorithm, the version 4 algorithm was less confident resulting a smaller
percentage of high confident clouds with CAD between 70 and 100. To maximize clouds sampling
count, low and medium confidence clouds are allowed to enter the next stage of filtering.
Cloud Phase Quality Filter: this filter chooses high confident ice and
water clouds only. If cloud phase was water (cloud phase = 2) or ice (cloud phase = 1 or 3)
the cloud phase quality assurance should be high (PhaseQA = 3). Note ice clouds
include both randomly orientated ice (cloud phase = 1) and horizontally orientated
ice (cloud phase = 3). A special rule is made for unknown/not determined phase
clouds (cloud phase = 0): choose unknown/not determined phase clouds with PhaseQA = 0,
as all these clouds have none confidence in PhaseQA (PhaseQA = 0).
Low water cloud filter: this filter excludes water clouds with cloud
top less than 8.2 km identified from coarse horizontal averaging (HA), i.e., HA = 5, 20 or 80 km.
An assumption is that optically thick broken water clouds such as marine boundary
clouds would be detected from HA = 333 m attenuated backscattering. If the same water
cloud were reported from HA = 5, 20, or 80 km, including this water clouds would
overestimate the water cloud coverage. There is also an argument of possibly underestimation
of optically thin water cloud coverage over the Southern Ocean with this filter. More
analyses would be conducted in future to weigh the pros and cons of this filter.
Temperature/pressure/altitude range filters: following the
GEWEX cloud assessment general guidance,
only clouds with cloud top temperature between 150 K
and 320 K, and cloud top pressure between 100 hPa and 1100 hPa and cloud top altitude
between 0.0 km and 20.0 km are included in the monthly statistics.
Figure 2: High cloud occurrence of top layer flavor, without (a) and with (b) range filters at night in July 2008. The format of each panel is the same as that in Figure 1.
These range filters excluded high thin cirrus clouds around tropics and polar stratospheric
clouds at polar regions, which can be observed by the CALIOP only due to its high sensitivity
but probably would be missed by a passive sensor. As a result, it leads to slightly smaller
cloud occurrence as shown in Figure 2, hotter cloud top temperature, higher cloud top
pressure and lower cloud top in those regions. However, if a user is interested in comparing
cloud top from CALIOP and passive sensors, the exclusion seems a valid approximation
since those high clouds appear “clear” or “invisible” for
passive sensors.
How cloud amount mean and histogram are estimated?
Figure 3: An illustration of two CALIPSO tracks with clear profiles as blue dots and cloudy profiles as white dots at a 1 degree longitude by 1 degree latitude grid (left). The average or mean CA in this grid is estimated using the formula shown on right.
Figure 3 explained how the cloud amount (CA) mean or average at each 2-dimensional
(2D) grid was estimated, adopting a similar methodology of a passive sensor. On the
left, two CALIPSO orbits were illustrated with gray lines and each dot represents one
profile, while a white dot means a cloudy profile and a blue one a clear profile. A
track CA is estimated from the cloudy and clear profiles along the track within each
grid assuming the track CA would be able to represent the grid CA. For these two
tracks, the track CA would be 4/5 and 1/3, respectively. The monthly average CA would
be the means of all track CA and each track CA would be binned into the CA histogram.
If these two tracks were the only tracks passing this grid during the whole month, the
monthly average CA would be 0.57 as shown on the right. One track CA 4/5 (0.8) would
go to bin 8 holding 0.8 ≤ CA < 0.9 and other CA 1/3 (0.33) would go to bin 3
holding 0.3 ≤ CA < 0.4
No Clouds Observed Above 17 km With The Pressure Range Filter
As mentioned before, three range filters were applied to clouds to ensure the
cloud top are within the histogram ranges as defined in
GEWEX cloud assessment general guidance:
cloud top temperature is from 150 K to 320 K, cloud top pressure is from 100 hPa to 1100 hPa and cloud top altitude from 0.0 km to 20.0 km.
These filters removed thin cirrus clouds around tropical region and polar
stratospheric clouds at polar regions detected by the CALIOP, resulting a slightly
smaller cloud occurrence, hotter cloud top temperature, higher cloud top pressure
and lower cloud top at those regions.
Figure 4: Zonal patterns of atmosphere pressure in 2008-01 and 2008-07. The patterns are filled in black if pressure less than 100 hPa.
Other than those, they also led to no clouds reported above 17 km as the pressure
is less than 100 hPa above 17 km globally in the
Modern Era Retrospective-Analysis for Research (MERRA-2)
reanalysis product provided by the NASA
GMAO Data Assimilation System.
As shown in Figure 4, the pressure is less than 100 hPa at 17 km at all latitudes
in both winter and summer, though the exact altitude threshold varies with latitude and
there is a seasonal asymmetry. Note the contours were filled in black if pressure less
than 100 hPa.
When applying the pressure range filter, it automatically chose clouds with cloud
top less about 17 km. Therefore, there would be no clouds with cloud top higher than
17.0 km in the cloud top (CZ) histogram as shown in Figure 5.
Figure 5: Histogram of cloud top height of top layer flavor from 2006/06 to 2016/12. The number of each cloud top height bin in each month are represented as logarithm.
Last Updated: January 22, 2021
Curator: Charles R. Trepte
NASA Official: Charles R. Trepte