> CALIPSO - Data File Ordering Web Tool - By Data Month
NASA - National Aeronautics and Space Administration
ABOUT CALIPSO PRODUCTS OUTREACH DOCUMENTS RESOURCES TOOLS CONTACTS        

CALIPSO Data User's Guide - Essential Reading


CALIPSO HOMECALIPSO’s User Guide HOME → Essential Reading

NOTE: This page provides information supporting the version 3.01 data products. Please refer to the latest version of the Data Products Catalog that supports the current versions of the data products - CALIPSO Data Products Catalog (Release 4.80) (PDF).


We strongly encourage prospective and current CALIPSO data users read through all of the following page. This will help guide you to use the data in the most appropriate manner and hopefully prevent mistakes in your analyses. The following topics will be covered:

 

Available Resources

There is a plethora of information available about CALIPSO data and how it is generated. The resources listed in this section should help you find it.

Data Quality Summaries

These documents provide a high-level quality assessment of all level 1 and level 2 products derived from the CALIPSO lidar measurements, as described in section 2 of the CALIPSO Data Products Catalog (Version 3.6) (PDF). As such, they represent the minimum information needed by scientists and researchers for appropriate and successful use of these data products. We strongly suggest that all authors, researchers, and reviewers of research papers review this document periodically, and familiarize themselves with the latest status before publishing any scientific papers using these data products.

Two examples of data quality flag usage and screening methods are provided in the Examples and Tools section section of this user guide.


Algorithm Theoretical Basis Documents (ATBDs)

The information found in the set of CALIPSO algorithm papers published in a special issue of the Journal of Atmospheric and Oceanic Technology supersedes some of the algorithm descriptions in the following theoretical basis documents.


General References

CALIPSO Data Products Catalog
The CALIPSO Data Products Catalog (DPC) provides a complete enumeration of the contents of the data products that are used or produced by the CALIPSO project Data Management System.
This is a very terse document that lists all of the scientific data sets (SDSs) available in each data product. For each SDS, data type, and data size are given, as are the geophysical units and a nominal range over which the data values can be expected to extend. The DPC also provides some information about the source and format of ancillary data, such as GMAO met data, IGBP surface data types, and DEM data sources (GTOPO30). However, the DPC is not the document to go to for usage information about individual SDSs. To obtain this kind of information please consult the data summaries section of this user guide.

Other information



 

Available Data Products

A tabular format of the following information may be found on the Data Product Descriptions page.

CALIPSO's payload includes a polarization-sensitive, two-wavelength lidar (CALIOP), a three-channel Infrared Imaging Radiometer (IIR), and a visible channel Wide Field Camera. The data from CALIOP is distributed as range-resolved vertical profiles (level 1 and level 2); as tabulated sets of spatial and optical parameters describing all layers detected (level 2); and as a high resolution bit-mapped vertical feature mask (level 2). Level 1 and level 2 swath data from the IIR and WFC are available in formats similar to those used by other passive remote sensors (i.e. MODIS, AVHRR, etc.).

All CALIPSO data products are available in the 'self-describing' HDF4 data format.

CALIOP Data

A description of the EOS data product levels can be found below.

Level 1B Profile Data (see DPC section 2.4)
This data product contains a half orbit (day or night) of calibrated and geolocated single-shot (highest resolution*) lidar profiles, including 532 nm and 1064 nm attenuated backscatter and depolarization ratio at 532 nm. The product released contains data from nominal science mode measurement.

The CALIOP Level 1B data product also contains additional parameters such as post processed ephemeris data, celestial data, and converted payload status data.

*Because of the satellite on-board averaging, the highest horizontal resolution data (333 m) is only available from -0.5 to 8.2 km. While contiguous profiles are provided from -2.0 to 30.1 km, the data in other altitude regions is over-sampled, a full description of this is available below.

Each granule of the level 1B profile products is further processed to generate the suite of level 2 lidar data products described below.

Level 2 Layer Data (see DPC section 2.4)
The level 2 analyses of the CALIOP lidar backscatter data begins with an attempt to locate all coherent "features" - i.e., clouds and aerosol layers - in each granule of the level 1 data. The results of this search are reported in four different lidar level 2 layer products. Cloud layers are reported at three different horizontal averaging resolutions: 1/3-km, 1-km, and 5-km. The 1/3-km data is reported only in those regions where single shot information is available in the downlinked data; that is, between ~8.2-km and -0.5-km. The 1-km and 5-km cloud layers are reported between ~20.2 and -0.5-km. Aerosol layers are reported between at a 5-km horizontal resolution between ~30.1-km and -0.5km. Layers detected in the stratosphere are recorded in the aerosol layer products, and thus users seeking measurements of polar stratospheric clouds should order the 5-km aerosol layer products. Note that the layers reported in the 5 km horizontal resolution data products may have been required averaging to as much as 80 horizontally before being detected by the feature finder algorithm. However, these fainter layers are still reported on the same uniform 5-km horizontal grid. To effectively use the 5-km layer products, users should first thoroughly familiarize themselves with the multi-gridded averaging scheme described here and in the feature finder ATBD.

Level 2 Profile Data (see DPC sections 2.5 and 2.6)
Each layer identified in the 5 km cloud and aerosol layer products is further analyzed to determine the profiles of particulate extinction and backscatter within the layer. This profile data, along with ancillary information (e.g., meteorological data from the GMAO) is reported in the 5 km cloud and aerosol layer products. As in the layer products, clouds and aerosols are reported in separate data products, and the profile data for stratospheric features is included in the aerosol profile products.

Level 2 Vertical Feature Mask (VFM; see DPC section 2.7)
The feature mask data set describes the vertical and horizontal distribution of cloud and aerosol layers observed by the CALIPSO lidar. Each range bin in the Lidar Level 0 data is characterized by a single 16-bit integer, with the various bits in the integer representing flags that describe some aspect of the data measured within the bin. Instructions on how to decode these integer data are given in the vertical feature mask summary page. The data are recorded in nominal increments of 15 consecutive laser pulses, which is nominally equivalent to a distance of 5-km along the laser ground-track.

IIR Data

The IIR instrument is a 3 channel imaging radiometer in the thermal infrared at 8.65, 10.6 and 12.05 microns. More information may be obtained from the CNES website: http://smsc.cnes.fr/CALIPSO/GP_iir.htm The following data products are distributed:

Level 1B radiance data (see DPC section 2.2)
Geolocated, calibrated radiances.

Level 2 track data (see DPC section 2.9)
Emissivity and cloud particle data related to pixels that have been co-located to the Lidar track.

Level 2 swath data (see DPC section 2.10)
Emissivity and cloud particle data assigned to IIR pixels on a 1 km grid centered on the Lidar track.

WFC Data

The primary Wide Field Camera Level 1B data products are calibrated radiances and bidirectional reflectances registered to an Earth-based grid centered on the lidar ground track. During normal operations, the WFC acquires science data only during the daylight portions of the CALIPSO orbits. For each daytime orbit segment, three different data products are produced: 1 km Native Science grid, 125 m Native Science grid, and 1 km Registered Science grid.

In addition to radiance and reflectance grids, the WFC Level 1 data products include two parameters that quantify the homogeneity of the cross track image frames: swath homogeneity and track homogeneity.

Level 1B 125m Native science data (see DPC section 2.3)
Contains only the central 5 km-wide high resolution (125 m) portion of the WFC swath.

Level 1B 1 km Native science data (see DPC section 2.3)
Covers the full 61 km swath centered on the Lidar track.

Level 1B 1 km Registered science data (see DPC section 2.3)
On the identical grid as the CALIPSO IIR data and is produced to facilitate the use of the WFC data in the IIR retrievals.


 

Vertical and horizontal resolution of the lidar data

The CALIPSO lidar collects profile data at a nominal vertical resolution of 15 m (10 MHz sampling rate), and at a laser pulse repetition rate of 20.16 Hz, which when accounting the the satellites orbital velocity, translates into a horizontal profile spacing of ˜333 m. Within the lower atmosphere the footprint of each profile is ˜90 m in diameter. Due to bandwidth limitations for downlinking the data from the satellite, the data is averaged on-board the satellite in both the horizontal and vertical dimensions.

The purpose of the on-board averaging scheme is to maximize spatial resolution and dynamic range of the signal while minimizing the telemetry data volume. The resulting algorithm reduced the required telemetry bandwidth by more than an order of magnitude relative to the raw data, with minimum impact on the information content of the data. Unfortunately this comes at a price in terms of complexity in the telemetry data.

The on-board averaging scheme provides the highest resolution in the lower troposphere where the spatial variability of clouds and aerosols is the greatest and coarser resolutions higher in the atmosphere. The degree of averaging varies with altitude, as detailed in the figure below.

Altitude Array and Regions.

The left panel above is a schematic of 15 consecutive full-resolution lidar profiles (i.e., a 5-km horizontal distance), annotated to illustrate how CALIOP's on-board averaging scheme varies vertically and horizontally as a function of altitude. The five different averaging regimes are shown using five different shades of blue. The vertical lines within each blue band delineate the individual profiles created by horizontally averaging the full resolution data. For example, between 8.2-km and 20.2-km the data is averaged horizontally to a nominal spatial resolution of 1-km; i.e., the data from 3 full resolution (333 m) profiles are averaged to create each 1 km averaged profile segment. Similarly, between 20.2 km and 30.1 km, 5 full resolution (333 m) profiles are averaged to create 3 profiles that each spans a nominal horizontal extent of 5/3 km. Processing in the vertical is done in a similar fashion, as indicated by the thin horizontal black lines shown in the profile on the far right within each altitude regime.

The table in the middle gives the numerical values of the horizontal and vertical averaging for all the altitude regions. The highest resolution data available is in the region between -0.5 and 8.5 km which is provided at 333 m horizontally and 30 m in the vertical.

During level 1 processing the data is regridded to a uniform horizontal grid of 333 m (see the right hand image above); however, the vertical (i.e. altitude) resolution remains identical to that of the downlinked data. Thus even though the level 1B profile data is reported at a uniform horizontal resolution of 333 m, in regions above 8.5 km it is over sampled. For example, to create 15 "pseudo single shot" profiles in the altitude region between 8.5 km to 20.1 km, the downlinked 1 km profiles in this region are replicated and grafted onto the top of three consecutive full resolution (single shot) profiles.

The preceding commentary refers specifically to the 532 nm parallel and perpendicular channels only. The 1064 nm channel is recorded at a vertical resolution of 60 m in the range be -0.5 and 8.2 km, also no 1064 nm data is available above 20.1 km.

Detailed information about the altitude array and vertical and horizontal averaging of the data users are encouraged to read section 2.3 in the Lidar Level 1 ATBD or section 3.3 in the mission overview document.

In all of the profile data sets the altitude array can be found in the metadata field (VDATA) called Lidar_Data_Altitudes.



 

Multi-Gridded Averaging Scheme

The peak backscatter intensities of the features measured by space-borne lidar range over several orders of magnitude. Strongly scattering features such as stratus and fair weather cumulus are easily detected using a single laser pulse. For more tenuous features - e.g., thin cirrus clouds - the average of several laser pulses may be required to obtain the signal-to-noise ratio necessary to differentiate feature boundaries from the ambient scattering environment. The unambiguous detection of the very weakest features - faint aerosol layers and subvisible cirrus - may require averaging over a substantial number of pulses.

To identify all of the features within a given scene at the maximum possible spatial resolution we employ a Selective Iterated Boundary Location (SIBYL) scheme. The SIBYL algorithm makes multiple passes through a specified scene, constructing profiles of attenuated scattering ratios at a series of increasingly coarse spatial resolutions, nominally at 5, 20 and 80 km.

Immediately after construction, each profile is scanned for the presence of clouds, aerosol layers, and/or surface returns using a profile scanner, (the 'Layer detection' box at right).

The backscatter data from those regions identified as containing a feature are removed from subsequent processing. As a consequence, features found at high spatial resolutions (i.e., with less averaging) will not be included in the profiles of attenuated scattering ratios scanned at coarser resolutions (i.e., more averaging). (This encompasses the 'Remove layers...' and 'Average data to 20 or 80 km' boxes.)

Note that the layer detection at 1 km and 333 m horizontal resolution are only performed for those regions where 5 km layers were detected. The boundary layer homogenization algorithm (BLaH) will not be further discussed in this document. Users are encouraged to read section 4 (section 4.4 is about BLaH) in the feature detection ATBD.

Region overview.
Figure 1.

Real world example of the multi-gridded averaging scheme

The following series of slides serve as a guide to explain more details of the averaging scheme.

Browse image of the total 532 nm attenuated backscatter signal (the sum of the 532 nm parallel and perpendicular return signals).

The signal strength has been color coded such that blues correspond to molecular scattering and weak aerosol scattering, aerosols generally show up as yellow/red/orange. Stronger cloud signals are plotted in gray scales, while weaker cloud returns are similar in strength to strong aerosol returns and coded in yellows and reds.

Region overview.
Figure 2.

Zoomed in version of above image from the area between the black lines.

There is a transparent cirrus cloud located at ~17 km and an aerosol layer (most likely dust) located from the surface or in some places ~ 1km up to an altitude of 4 km. The surface return is seen as the white/red/yellow band near 0 km.

The horizontal line at 8 km is from the changing noise characteristics in the data due to the change in horizontal and vertical averaging of the data. Data is plotted a 333 m horizontal resolution.

Zoomed in level 1 browse image.
Figure 3.

The following image shows the vertical and horizontal locations of all features that were detected at the 5 km horizontal resolution. Only the data in the regions as identified by the magenta color are used to compute the layer optical properties as would be found in the 5 km layer products.

The layer data in those regions will then be zeroed-out before the 5 km is averaged to 20 km.

Region overview.
Figure 4.

After the removal of features that were detected at the 5 km resolution, a search is made for features at the 20 km resolution. The image at right shows features that are found at the 20 km resolution in blue. The image (below-right) shows both the 5 km features and the 20 km features. Only the data in the regions shown in blue are used to calculate the layer optical properties for those same regions. The regions in magenta are not included in the averages for the features detected on the 20 km resolution. Layer optical properties are repeated (i.e. reported in the data products) for each 5 km column in which they appear.

Region overview.
Figure 5.

Corrected version of above image to show the true spatial contribution of data to the detected layer, 20 km features are blue, 5 km features are magenta.

Region overview.
Figure 6.

The following image shows the vertical and horizontal locations of all features that were detected at the 80 km horizontal resolution. Only the data in the regions as identified by the white color are used to compute the layer optical properties as would be found in the 5 km layer products. Layer data are reported in every 5 km column the feature appeared in.

As in the 20 km averaging image, this one has plotted the vertical extent of each 80 km Feature as it appears in the 5 km layer product file. However that area contribution of the underlying data is actually a lot less, see the corrected image below.

Region overview.
Figure 7.

Corrected version of above image to show the true spatial contribution of data to the detected layer.

Region overview.
Figure 8.

Final composite 'feature mask' image showing 5, 20 and 80 km horizontal resolutions.

Region overview.
Figure 9.

Another version of the 'feature mask' showing the same horizontal resolution as above-right but including the 1 km and 333 m layer extents. The 1 km and 333 m profile data are only scanned for layers in regions where a 5 km layer was found, therefore 1 km layers will only be found in regions where a 5 km layers were found and subsequently 333 m are only found where 1 km layers were found.

VFM with averaging.
Figure 10.

This 'vertical feature mask' image shows the vertical locations of all layers detected by the level 2 processing code, colors represent the type of layer as determined by the scene classification algorithm. Cloud layers are identified by the cyan color, aerosols by green, dark blue is 'clear air' where no layers were detected, etc.

Vertical feature mask image.
Figure 11.



 

Improvements and highlights for Version 3.01

The third release of the CALIPSO data products features a comprehensive restructuring and expansion of the Lidar Level 2 cloud and aerosol profile products; significant enhancements to the Lidar Level 2 cloud and aerosol profile products; and the implementation of an improved calibration technique for the Lidar Level 1 532 nm daytime calibration.

The primary contents of the CALIOP Level 1 product are calibrated profiles of 532 nm perpendicular attenuated backscatter and total attenuated backscatter at 532 nm and 1064 nm. The version 3 CALIOP Level 1 data are released with a product maturity classification of Validated Stage 1, indicating that initial validation of the CALIOP attenuated backscatter products has been successful.

The CALIOP Level 2 products consist of the full resolution vertical feature mask, cloud and aerosol layer products reported at several different spatial resolutions, and cloud and aerosol profile products reported at a uniform 5-km horizontal resolution. Validation of the Level 2 products is an on-going process, and some products are better characterized than others. Preliminary validation has been accomplished for the layer detection, cloud-aerosol discrimination, and layer sub-typing algorithms. Data products derived solely from these algorithms are designated as Validated Stage 1. Validation of the optical properties algorithms that generate extinction profiles and layer optical depth estimates is a considerably more challenging task. As a consequence, while all studies to date indicate acceptable performance, the status of those data products derived from this second class of algorithms remains Provisional. All obvious artifacts have been identified and corrected in these data, but only limited comparisons with independent data sets are currently available.

Detailed data quality statements for each data release may be found under the data quality section of this user guide.


 

Miscellaneous

 

EOS Data Product Levels

The following is copied from the EOS Data Products Handbook Volume 2.
Level 0
Reconstructed unprocessed instrument/payload data at full resolution; any and all communications artifacts (e.g., synchronization frames, communications headers) removed.

Level 1A
Reconstructed unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and geo-referencing parameters (i.e., platform ephemeris) computed and appended, but not applied, to the Level 0 data.

Level 1B
Level 1A data that have been processed to sensor units (not all instruments have a Level 1B equivalent).

Level 2
Derived geophysical variables at the same resolution and location as the Level 1 source data.

Level 3
Variables mapped on uniform space-time grid scales, usually with some completeness and consistency.

Level 4
Model output or results from analyses of lower level data (e.g., variables derived from multiple measurements).



NASA
Last Updated: October 20, 2020
Curator: Charles R. Trepte
NASA Official: Charles R. Trepte

+ No Fear Acts
+ FOIA
+ Privacy

+ Office of Inspector General
+ Office of Special Councel
+ Agency Financial Reports

+ Contact NASA