IAGOS evaluation dataset

The IAGOS evaluation dataset included in ContrailBench provides a gridded dataset of PCR observations based on in-situ measurements from the IAGOS program. This dataset is used to calculate forecast hit rates in ContrailBench benchmarks. See preprocess_iagos.py for details on how this data is prepared from IAGOS measurements.

The IAGOS evaluation dataset is available alongside other evaluation datasets in a public cloud bucket (gs://contrailbench-public-data). This notebook shows

1. How to load IAGOS evaluation data using Pandas

2. How to interpret the contents of the IAGOS evaluation dataset

3. How to use the IAGOS evaluation dataset to compute ContrailBench hit rate metrics

import datetime
import os

import aiohttp
import cartopy.crs as ccrs
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import xarray as xr

Loading IAGOS evaluation data

IAGOS evaluation data is sharded by time (one file per hour) and altitude (one file per flight level). Coverage for each ContrailBench release cycle is as follows:

Release	Times	Altitudes	GCS path
v1	Jan-Dec 2024	FL270-440	gs://contrail-bench-public-data/v1/iagos

Because gridded IAGOS data is relatively sparse, the dataset is stored as parquet files with one row for each grid cell with non-zero flight distance. These files can be read directly from GCS with Pandas.

Warning: be very careful about timezone-naive datetimes when using ContrailBench evaluation datasets. Python’s datetime module assumes that naive datetimes represent local time, which affects the POSIX timestamp returned by datetime.timestamp().

gcs_path = "gs://contrailbench-public-data/v1/iagos"
time = datetime.datetime(2024, 6, 1, 1, tzinfo=datetime.UTC)
flight_level = 350

df = pd.read_parquet(f"{gcs_path}/{int(time.timestamp())}_{flight_level}.pq")
df

	longitude	latitude	pcr_distance	total_distance
0	-54.50	-32.50	8446.716965	8446.716965
1	-54.25	-32.50	7402.788859	7402.788859
2	-54.25	-32.25	22223.452812	22223.452812
3	-54.00	-32.25	23174.986600	23174.986600
4	-54.00	-32.00	7395.536809	7395.536809
5	-53.75	-32.00	32273.638929	32273.638929
6	-53.75	-31.75	2062.059471	2062.059471
7	-53.50	-31.75	32962.845584	32962.845584
8	-53.50	-31.50	4116.337698	4116.337698
9	-53.25	-31.50	31893.240717	31893.240717
10	-53.25	-31.25	6142.800760	6142.800760
11	-53.00	-31.25	29757.554541	29757.554541
12	-53.00	-31.00	8182.882624	8182.882624
13	-52.75	-31.00	25642.348014	26649.458313
14	-52.75	-30.75	12335.472745	12335.472745
15	-52.50	-30.75	22753.039993	23798.840797
16	-52.50	-30.50	14501.361623	14501.361623
17	-52.25	-30.50	20736.577725	20736.577725
18	-52.25	-30.25	17630.198242	17630.198242
19	-52.00	-30.25	17644.237722	17644.237722
20	-52.00	-30.00	20785.444459	20785.444459
21	-51.75	-30.00	13499.483442	13499.483442
22	-51.75	-29.75	2069.563041	16627.123659

Interpretation

Rows of the dataset provide IAGOS flight distances (in meters) aggregated on a spatiotemporal grid with 0.25 degree horizontal resolution. Horizontal bounds are from -180 degrees to 179.75 degrees longitude and -80 degrees to 80 degrees latitude. Each grid cell includes IAGOS flight segments

1. within 0.125 degrees latitude and longitude of the provided latitude and longitude coordinates,

2. within 250 vertical ft of the target flight level, and

3. within 30 minutes of the target time

The pcr_distance column provides the IAGOS flight distance inside PCRs, as determined by measured temperature and humidity. The total_distance column provides the total IAGOS flight distance, both within and outside PCRs.

Grid cells without any IAGOS measurements are omitted from the dataset to reduce data volume. Because IAGOS measurements are sparse, many of the parquet files in the evaluation dataset contain zero rows. If necessary, omitted grid cells can be restored using xarray:

all_lons = np.arange(-180.0, 180.0, 0.25)
all_lats = np.arange(-80, 80.25, 0.25)
ds = df.set_index(["longitude", "latitude"]).to_xarray().fillna(0.0)
ds = ds.reindex(longitude=all_lons, latitude=all_lats, fill_value=0.0)
ds

<xarray.Dataset> Size: 15MB
Dimensions:         (longitude: 1440, latitude: 641)
Coordinates:
  * longitude       (longitude) float64 12kB -180.0 -179.8 ... 179.5 179.8
  * latitude        (latitude) float64 5kB -80.0 -79.75 -79.5 ... 79.75 80.0
Data variables:
    pcr_distance    (longitude, latitude) float64 7MB 0.0 0.0 0.0 ... 0.0 0.0
    total_distance  (longitude, latitude) float64 7MB 0.0 0.0 0.0 ... 0.0 0.0

xarray.Dataset

• Dimensions:
  • longitude: 1440
  • latitude: 641
• Coordinates: (2)
  • longitude
    (longitude)
    float64
    -180.0 -179.8 ... 179.5 179.8

    array([-180.  , -179.75, -179.5 , ...,  179.25,  179.5 ,  179.75],
          shape=(1440,))

  • latitude
    (latitude)
    float64
    -80.0 -79.75 -79.5 ... 79.75 80.0

    array([-80.  , -79.75, -79.5 , ...,  79.5 ,  79.75,  80.  ], shape=(641,))

• Data variables: (2)
  • pcr_distance
    (longitude, latitude)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([[0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           ...,
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.]], shape=(1440, 641))

  • total_distance
    (longitude, latitude)
    float64
    0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0

    array([[0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           ...,
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.],
           [0., 0., 0., ..., 0., 0., 0.]], shape=(1440, 641))

plt.figure(figsize=(12, 4))
ax = plt.subplot(111, projection=ccrs.PlateCarree())
im = ax.pcolormesh(
    ds["longitude"],
    ds["latitude"],
    ds["pcr_distance"].T / 1e3,
    shading="nearest",
    cmap="gist_heat_r",
    transform=ccrs.PlateCarree(),
)
plt.colorbar(im, ax=ax, label="IAGOS PCR distance (km)")
ax.set_extent([-90, -10, -50, -20])
ax.coastlines(color="gray");

Use in ContrailBench metrics

The GRUAN evaluation dataset is used in PCR benchmarks to calculate forecast hit rates, a proxy for the effectiveness of avoidance.

This notebook uses the Contrails.org forecast for an example calculation. See the Contrails.org example notebook for details about accessing and preprocessing the Contrails.org forecast.

url = "https://api.contrails.org/v1/grids"
params = {
    "aircraft_class": "default",
    "flight_level": str(flight_level),
    "time": time.strftime("%Y-%m-%dT%H"),
    "units": "ef_per_m",
}
headers = {"x-api-key": os.environ["CONTRAILS_API_KEY"]}

async with (
    aiohttp.ClientSession(raise_for_status=True) as session,
    session.get(url, params=params, headers=headers) as resp,
):
    content = await resp.read()

with open("forecast.nc", "wb") as f:
    f.write(content)
ds = xr.open_dataset("forecast.nc")

ds["pcr"] = ds["ef_per_m"] != 0
processed = ds[["pcr"]]

Forecast hit rates can be calculated using some xarray indexing tricks. This notebook computes hit rates treating all grid cells with non-zero pcr_distance as observed PCRs and weighting grid cells by area, consistent with ContrailBench benchmarks.

target_lon = xr.DataArray(df["longitude"].loc[df["pcr_distance"] > 0], dims="row")
target_lat = xr.DataArray(df["latitude"].loc[df["pcr_distance"] > 0], dims="row")
area = xr.DataArray(np.cos(np.deg2rad(df["latitude"].loc[df["pcr_distance"] > 0])), dims="row")

pcr = processed["pcr"].sel(longitude=target_lon, latitude=target_lat)
fcst_area = area.where(pcr).sum().item()
tot_area = area.sum().item()

print(f"Forecast hit rate: {fcst_area / tot_area:.3f}")

Forecast hit rate: 0.697