Tutorial 7: Other Computational Tools in Xarray

Week 1, Day 1, Climate System Overview

Content creators: Sloane Garelick, Julia Kent

Content reviewers: Katrina Dobson, Younkap Nina Duplex, Danika Gupta, Maria Gonzalez, Will Gregory, Nahid Hasan, Paul Heubel, Sherry Mi, Beatriz Cosenza Muralles, Jenna Pearson, Agustina Pesce, Chi Zhang, Ohad Zivan

Content editors: Paul Heubel, Jenna Pearson, Chi Zhang, Ohad Zivan

Production editors: Wesley Banfield, Paul Heubel, Jenna Pearson, Konstantine Tsafatinos, Chi Zhang, Ohad Zivan

Our 2024 Sponsors: CMIP, NFDI4Earth

Pythia credit: Rose, B. E. J., Kent, J., Tyle, K., Clyne, J., Banihirwe, A., Camron, D., May, R., Grover, M., Ford, R. R., Paul, K., Morley, J., Eroglu, O., Kailyn, L., & Zacharias, A. (2023). Pythia Foundations (Version v2023.05.01) https://zenodo.org/record/8065851

Tutorial Objectives

Estimated timing of tutorial: 15 minutes

Thus far, we’ve learned about various climate processes in the videos, and we’ve explored tools in Xarray that are useful for analyzing and interpreting climate data in the tutorials.

In this tutorial, you’ll continue using the SST data from CESM2 and practice using some additional computational tools in Xarray to resample your data, which can help with data comparison and analysis. The functions you will use are:

• .resample(): Groupby-like functionality specifically for time dimensions. Can be used for temporal upsampling and downsampling. Additional information about resampling in Xarray can be found here.

• .rolling(): Useful for computing aggregations on moving windows of your dataset e.g. computing moving averages. Additional information about resampling in Xarray can be found here.

• .coarsen(): Generic functionality for downsampling data. Additional information about resampling in Xarray can be found here.

Setup

# installations ( uncomment and run this cell ONLY when using google colab or kaggle )
#!pip install pythia_datasets cftime nc-time-axis

# imports
import matplotlib.pyplot as plt
import xarray as xr
from pythia_datasets import DATASETS

Install and import feedback gadget

Show code cell source Hide code cell source

# @title Install and import feedback gadget

!pip3 install vibecheck datatops --quiet

from vibecheck import DatatopsContentReviewContainer
def content_review(notebook_section: str):
    return DatatopsContentReviewContainer(
        "",  # No text prompt
        notebook_section,
        {
            "url": "https://pmyvdlilci.execute-api.us-east-1.amazonaws.com/klab",
            "name": "comptools_4clim",
            "user_key": "l5jpxuee",
        },
    ).render()


feedback_prefix = "W1D1_T7"

Figure Settings

Show code cell source Hide code cell source

# @title Figure Settings
import ipywidgets as widgets  # interactive display

%config InlineBackend.figure_format = 'retina'
plt.style.use(
    "https://raw.githubusercontent.com/neuromatch/climate-course-content/main/cma.mplstyle"
)

Video 1: Carbon Cycle and the Greenhouse Effect

Submit your feedback

Show code cell source Hide code cell source

# @title Submit your feedback
content_review(f"{feedback_prefix}_Carbon_Cycle_Video")

If you want to download the slides: https://osf.io/download/sb3n5/

Submit your feedback

Show code cell source Hide code cell source

# @title Submit your feedback
content_review(f"{feedback_prefix}_Carbon_Cycle_Slides")

Section 1: High-level Computation Functionality

In this tutorial, you will learn about several methods for dealing with the resolution of data. Here are some links for quick reference, and we will go into detail in each of them in the sections below.

• .resample(): Groupby-like functionality especially for time dimensions. Can be used for temporal upsampling and downsampling

• .rolling(): Useful for computing aggregations on moving windows of your dataset e.g. computing moving averages

• .coarsen(): Generic functionality for downsampling data

First, let’s load the same data that we used in the previous tutorials (monthly SST data from CESM2):

filepath = DATASETS.fetch("CESM2_sst_data.nc")
ds = xr.open_dataset(filepath)
ds

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
Cell In[9], line 1
----> 1 filepath = DATASETS.fetch("CESM2_sst_data.nc")
      2 ds = xr.open_dataset(filepath)
      3 ds

File ~/micromamba/envs/climatematch/lib/python3.11/site-packages/pooch/core.py:598, in Pooch.fetch(self, fname, processor, downloader, progressbar)
    595     if downloader is None:
    596         downloader = choose_downloader(url, progressbar=progressbar)
--> 598     stream_download(
    599         url,
    600         full_path,
    601         known_hash,
    602         downloader,
    603         pooch=self,
    604         retry_if_failed=self.retry_if_failed,
    605     )
    607 if processor is not None:
    608     return processor(str(full_path), action, self)

File ~/micromamba/envs/climatematch/lib/python3.11/site-packages/pooch/core.py:823, in stream_download(url, fname, known_hash, downloader, pooch, retry_if_failed)
    819 try:
    820     # Stream the file to a temporary so that we can safely check its
    821     # hash before overwriting the original.
    822     with temporary_file(path=str(fname.parent)) as tmp:
--> 823         downloader(url, tmp, pooch)
    824         hash_matches(tmp, known_hash, strict=True, source=str(fname.name))
    825         shutil.move(tmp, str(fname))

File ~/micromamba/envs/climatematch/lib/python3.11/site-packages/pooch/downloaders.py:231, in HTTPDownloader.__call__(self, url, output_file, pooch, check_only)
    229 try:
    230     response = requests.get(url, timeout=timeout, **kwargs)
--> 231     response.raise_for_status()
    232     content = response.iter_content(chunk_size=self.chunk_size)
    233     total = int(response.headers.get("content-length", 0))

File ~/micromamba/envs/climatematch/lib/python3.11/site-packages/requests/models.py:1167, in Response.raise_for_status(self)
   1162     http_error_msg = (
   1163         f"{self.status_code} Server Error: {reason} for url: {self.url}"
   1164     )
   1166 if http_error_msg:
-> 1167     raise HTTPError(http_error_msg, response=self)

HTTPError: 429 Client Error: Too Many Requests for url: https://raw.githubusercontent.com/ProjectPythia/pythia-datasets/main/data/CESM2_sst_data.nc

Section 1.1: Resampling Data

For upsampling or downsampling temporal resolutions, we can use the .resample() method in Xarray. For example, you can use this function to downsample a dataset from hourly to 6-hourly resolution.

Our original SST data is monthly resolution. Let’s use .resample() to downsample to annual frequency:

# resample from a monthly to an annual frequency
tos_yearly = ds.tos.resample(time="AS")
tos_yearly

# calculate the global mean of the resampled data
annual_mean = tos_yearly.mean()
annual_mean_global = annual_mean.mean(dim=["lat", "lon"])
annual_mean_global.plot()

Section 1.2: Moving Average

The .rolling() method allows for a rolling window aggregation and is applied along one dimension using the name of the dimension as a key (e.g. time) and the window size as the value (e.g. 6). We will use these values in the demonstration below.

Let’s use the .rolling() function to compute a 6-month moving average of our SST data:

# calculate the running mean
tos_m_avg = ds.tos.rolling(time=6, center=True).mean()
tos_m_avg

# calculate the global average of the running mean
tos_m_avg_global = tos_m_avg.mean(dim=["lat", "lon"])
tos_m_avg_global.plot()

Section 1.3: Coarsening the Data

The .coarsen() function allows for block aggregation along multiple dimensions.

Let’s use the .coarsen() function to take a block mean for every 4 months and globally (i.e., 180 points along the latitude dimension and 360 points along the longitude dimension). Although we know the dimensions of our data quite well, we will include code that finds the length of the latitude and longitude variables so that it could work for other datasets that have a different format.

# coarsen the data
coarse_data = ds.coarsen(time=4, lat=len(ds.lat), lon=len(ds.lon)).mean()
coarse_data

coarse_data.tos.plot()

Section 1.4: Compare the Resampling Methods

Now that we’ve tried multiple resampling methods on different temporal resolutions, we can compare the resampled datasets to the original.

original_global = ds.mean(dim=["lat", "lon"])

original_global.tos.plot(size=6)
coarse_data.tos.plot()
tos_m_avg_global.plot()
annual_mean_global.plot()


plt.legend(
    [
        "original data (monthly)",
        "coarsened (4 months)",
        "moving average (6 months)",
        "annually resampled (12 months)",
    ]
)

Questions 1.4: Climate Connection

1. What type of information can you obtain from each time series?

2. In what scenarios would you use different temporal resolutions?

# to_remove explanation

"""
1. In general, by examining the data at these different time scales, you can get a more comprehensive understanding of the SST variations and their potential causes.
2. The original monthly data gives you the most granular view of the data, allowing you to see monthly variations in SST. Coarsening the data over 4-month periods reduces the temporal resolution but provides a slightly smoothed series that could help identify patterns or trends over this larger time period. A 6-month moving average could be useful for identifying semi-annual trends and reducing the impact of short-term noise in the data. The annually resampled (12 months) data provides a high-level view of the SST data, emphasizing the annual pattern. This can be useful for identifying long-term trends or changes in the data over the span of years.
"""

'\n1. In general, by examining the data at these different time scales, you can get a more comprehensive understanding of the SST variations and their potential causes.\n2. The original monthly data gives you the most granular view of the data, allowing you to see monthly variations in SST. Coarsening the data over 4-month periods reduces the temporal resolution but provides a slightly smoothed series that could help identify patterns or trends over this larger time period. A 6-month moving average could be useful for identifying semi-annual trends and reducing the impact of short-term noise in the data. The annually resampled (12 months) data provides a high-level view of the SST data, emphasizing the annual pattern. This can be useful for identifying long-term trends or changes in the data over the span of years.\n'

Submit your feedback

Show code cell source Hide code cell source

# @title Submit your feedback
content_review(f"{feedback_prefix}_Questions_1_4")

Summary

In this tutorial, we’ve explored Xarray tools to simplify and understand climate data better. Given the complexity and variability of climate data, tools like .resample(), .rolling(), and .coarsen() come in handy to make the data easier to compare and find long-term trends. You’ve also looked at valuable techniques like calculating moving averages.

Resources

Code and data for this tutorial is based on existing content from Project Pythia.