XArray
In [ ]:
Copied!
# to run this cell uncomment the line below by removing the #
!pip install xarray==0.20.2 dask netCDF4 bottleneck pooch
# to run this cell uncomment the line below by removing the #
!pip install xarray==0.20.2 dask netCDF4 bottleneck pooch
Xarray installation¶
Install Xarray and some of its dependencies if not already installed using conda:
conda install -c conda-forge xarray==0.20.2 dask netCDF4 bottleneck pooch
It may take a while resolving installation environments. If it is successful, will install other package dependecies.
Xarray comes with a collection of datasets to explore: xarray.tutorial.open_dataset
Available datasets:
"air_temperature": NCEP reanalysis subset
"air_temperature_gradient": NCEP reanalysis subset with approximate x,y gradients
"basin_mask": Dataset with ocean basins marked using integers
"ASE_ice_velocity": MEaSUREs InSAR-Based Ice Velocity of the Amundsen Sea Embayment, Antarctica, Version 1
"rasm": Output of the Regional Arctic System Model (RASM)
"ROMS_example": Regional Ocean Model System (ROMS) output
"tiny": small synthetic dataset with a 1D data variable
"era5-2mt-2019-03-uk.grib": ERA5 temperature data over the UK
"eraint_uvz": data from ERA-Interim reanalysis, monthly averages of upper level data
"ersstv5": NOAA’s Extended Reconstructed Sea Surface Temperature monthly averages
In [3]:
Copied!
# Load required libraries
%matplotlib inline
import numpy as np
import pandas as pd
import dask.array as da
import dask.dataframe as dd
import pooch
import xarray as xr
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['figure.figsize'] = (8,5)
# Load required libraries
%matplotlib inline
import numpy as np
import pandas as pd
import dask.array as da
import dask.dataframe as dd
import pooch
import xarray as xr
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams['figure.figsize'] = (8,5)
In [4]:
Copied!
# Load the air_temperature dataset and define a xarray datastructure
# 4 x Daily Air temperature in degrees K at sigma level 995
# (2013-01-01 to 2014-12-31)
# Spatial Coverage
# 2.5 degree x 2.5 degree global grids (144x73) [2.5 degree = 172.5 miles]
# 0.0E to 357.5E, 90.0N to 90.0S
ds = xr.tutorial.open_dataset('air_temperature')
#ds.info()
# Load the air_temperature dataset and define a xarray datastructure
# 4 x Daily Air temperature in degrees K at sigma level 995
# (2013-01-01 to 2014-12-31)
# Spatial Coverage
# 2.5 degree x 2.5 degree global grids (144x73) [2.5 degree = 172.5 miles]
# 0.0E to 357.5E, 90.0N to 90.0S
ds = xr.tutorial.open_dataset('air_temperature')
#ds.info()
In [5]:
Copied!
# Show the components of the xarray.Dataset
ds
# Show the components of the xarray.Dataset
ds
Out[5]:
<xarray.Dataset>
Dimensions: (lat: 25, time: 2920, lon: 53)
Coordinates:
* lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
* lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
* time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Data variables:
air (time, lat, lon) float32 ...
Attributes:
Conventions: COARDS
title: 4x daily NMC reanalysis (1948)
description: Data is from NMC initialized reanalysis\n(4x/day). These a...
platform: Model
references: http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...In [6]:
Copied!
# Show the values of data variable: air
ds.air
# Show the values of data variable: air
ds.air
Out[6]:
<xarray.DataArray 'air' (time: 2920, lat: 25, lon: 53)>
[3869000 values with dtype=float32]
Coordinates:
* lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
* lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
* time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Attributes:
long_name: 4xDaily Air temperature at sigma level 995
units: degK
precision: 2
GRIB_id: 11
GRIB_name: TMP
var_desc: Air temperature
dataset: NMC Reanalysis
level_desc: Surface
statistic: Individual Obs
parent_stat: Other
actual_range: [185.16 322.1 ]In [7]:
Copied!
# Direct access to values
#ds.air.values
# Direct access to values
#ds.air.values
In [8]:
Copied!
# Underneath is a Numpy N dimensional Array
type(ds.air.values)
# Underneath is a Numpy N dimensional Array
type(ds.air.values)
Out[8]:
numpy.ndarray
DataArray Properties¶
In [9]:
Copied!
# Show dimensions
ds.dims
# Show dimensions
ds.dims
Out[9]:
Frozen({'lat': 25, 'time': 2920, 'lon': 53})
In [10]:
Copied!
# Show coordinates
ds.coords
# Show coordinates
ds.coords
Out[10]:
Coordinates: * lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0 * lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0 * time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
In [11]:
Copied!
# Show attributes
ds.attrs
# Show attributes
ds.attrs
Out[11]:
{'Conventions': 'COARDS',
'title': '4x daily NMC reanalysis (1948)',
'description': 'Data is from NMC initialized reanalysis\n(4x/day). These are the 0.9950 sigma level values.',
'platform': 'Model',
'references': 'http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html'}
We can use xarray.DataArray.groupby to caculate average monthly temperatures and anomalies.
In [12]:
Copied!
# calculate monthly climatology
climatology = ds.groupby('time.month').mean('time')
# calculate anomalies
anomalies = ds.groupby('time.month') - climatology
# calculate monthly climatology
climatology = ds.groupby('time.month').mean('time')
# calculate anomalies
anomalies = ds.groupby('time.month') - climatology
In [13]:
Copied!
# Show the resulting Dataset
climatology
# Show the resulting Dataset
climatology
Out[13]:
<xarray.Dataset>
Dimensions: (lat: 25, month: 12, lon: 53)
Coordinates:
* lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
* lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
* month (month) int64 1 2 3 4 5 6 7 8 9 10 11 12
Data variables:
air (month, lat, lon) float32 246.3 246.4 246.2 ... 297.6 297.6 297.5In [14]:
Copied!
# Show the sesulting Dataset
anomalies
# Show the sesulting Dataset
anomalies
Out[14]:
<xarray.Dataset>
Dimensions: (lat: 25, lon: 53, time: 2920)
Coordinates:
* lat (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
* lon (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
* time (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
month (time) int64 1 1 1 1 1 1 1 1 1 1 ... 12 12 12 12 12 12 12 12 12 12
Data variables:
air (time, lat, lon) float32 -5.15 -3.886 -2.715 ... -1.375 -1.848Much like in Pandas, Xarray includes an interface to Matplotlib that we can access through the .plot() method of every DataArray.
Plotting the air temperature for the first time value, using longitude as the x variable.
In [15]:
Copied!
ds.air.isel(time=1).plot(x="lon");
ds.air.isel(time=1).plot(x="lon");
In [16]:
Copied!
# We can take time average of air temperature over all coordinates
ds.air.mean("time").plot(x="lon");
# We can take time average of air temperature over all coordinates
ds.air.mean("time").plot(x="lon");
Indexing and selecting data¶
Positional indexing¶
In [8]:
Copied!
# Create the following Dataset
da = xr.DataArray(
np.random.rand(4, 3),
[
("time", pd.date_range("2000-01-01", periods=4)),
("space", ["IA", "IL", "IN"]),
],
)
# Show the Dataset
da
# Create the following Dataset
da = xr.DataArray(
np.random.rand(4, 3),
[
("time", pd.date_range("2000-01-01", periods=4)),
("space", ["IA", "IL", "IN"]),
],
)
# Show the Dataset
da
Out[8]:
<xarray.DataArray (time: 4, space: 3)>
array([[0.17633122, 0.59377299, 0.30699365],
[0.4243673 , 0.90601059, 0.2009795 ],
[0.37157228, 0.28094102, 0.56942557],
[0.60069197, 0.88338457, 0.07061534]])
Coordinates:
* time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
* space (space) <U2 'IA' 'IL' 'IN'In [10]:
Copied!
# Show Dataset dimensions
da.dims
# Show Dataset dimensions
da.dims
Out[10]:
('time', 'space')
In [9]:
Copied!
# Select the first 2 values of first variable (time)
da[:2]
# Select the first 2 values of first variable (time)
da[:2]
Out[9]:
<xarray.DataArray (time: 2, space: 3)>
array([[0.17633122, 0.59377299, 0.30699365],
[0.4243673 , 0.90601059, 0.2009795 ]])
Coordinates:
* time (time) datetime64[ns] 2000-01-01 2000-01-02
* space (space) <U2 'IA' 'IL' 'IN'In [11]:
Copied!
# Select the first values of (time, space)
da[0,0]
# Select the first values of (time, space)
da[0,0]
Out[11]:
<xarray.DataArray ()>
array(0.17633122)
Coordinates:
time datetime64[ns] 2000-01-01
space <U2 'IA'In [12]:
Copied!
# get all the values of the `time` variable and
# select the third and second values of the `space` variable in that order.
da[:, [2, 1]]
# get all the values of the `time` variable and
# select the third and second values of the `space` variable in that order.
da[:, [2, 1]]
Out[12]:
<xarray.DataArray (time: 4, space: 2)>
array([[0.30699365, 0.59377299],
[0.2009795 , 0.90601059],
[0.56942557, 0.28094102],
[0.07061534, 0.88338457]])
Coordinates:
* time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
* space (space) <U2 'IN' 'IL'Indexing with dimension names¶
In [13]:
Copied!
# index by integer array indices
# Select by index the first space variable and first 2 values of time
da.isel(space=0, time=slice(None, 2))
# index by integer array indices
# Select by index the first space variable and first 2 values of time
da.isel(space=0, time=slice(None, 2))
Out[13]:
<xarray.DataArray (time: 2)>
array([0.17633122, 0.4243673 ])
Coordinates:
* time (time) datetime64[ns] 2000-01-01 2000-01-02
space <U2 'IA'In [14]:
Copied!
# index by dimension coordinate labels
da.sel(time=slice("2000-01-01", "2000-01-02"))
# index by dimension coordinate labels
da.sel(time=slice("2000-01-01", "2000-01-02"))
Out[14]:
<xarray.DataArray (time: 2, space: 3)>
array([[0.17633122, 0.59377299, 0.30699365],
[0.4243673 , 0.90601059, 0.2009795 ]])
Coordinates:
* time (time) datetime64[ns] 2000-01-01 2000-01-02
* space (space) <U2 'IA' 'IL' 'IN'In [15]:
Copied!
# Same as: da[0,0], but using indexes
da.isel(space=[0], time=[0])
# Same as: da[0,0], but using indexes
da.isel(space=[0], time=[0])
Out[15]:
<xarray.DataArray (time: 1, space: 1)> array([[0.17633122]]) Coordinates: * time (time) datetime64[ns] 2000-01-01 * space (space) <U2 'IA'
In [16]:
Copied!
# Select a specific time slice
da.sel(time="2000-01-01")
# Select a specific time slice
da.sel(time="2000-01-01")
Out[16]:
<xarray.DataArray (space: 3)>
array([0.17633122, 0.59377299, 0.30699365])
Coordinates:
time datetime64[ns] 2000-01-01
* space (space) <U2 'IA' 'IL' 'IN'Droping labels and dimensions¶
The drop_sel() method returns a new object with the listed index labels along a dimension dropped:
In [17]:
Copied!
da
da
Out[17]:
<xarray.DataArray (time: 4, space: 3)>
array([[0.17633122, 0.59377299, 0.30699365],
[0.4243673 , 0.90601059, 0.2009795 ],
[0.37157228, 0.28094102, 0.56942557],
[0.60069197, 0.88338457, 0.07061534]])
Coordinates:
* time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
* space (space) <U2 'IA' 'IL' 'IN'In [18]:
Copied!
# Drop 2 space coordinates using labels
da.drop_sel(space=["IN", "IL"])
# Drop 2 space coordinates using labels
da.drop_sel(space=["IN", "IL"])
Out[18]:
<xarray.DataArray (time: 4, space: 1)>
array([[0.17633122],
[0.4243673 ],
[0.37157228],
[0.60069197]])
Coordinates:
* time (time) datetime64[ns] 2000-01-01 2000-01-02 2000-01-03 2000-01-04
* space (space) <U2 'IA'In [19]:
Copied!
# Drop by index the first values of space and time variables
da.drop_isel(space=[0], time=[0])
# Drop by index the first values of space and time variables
da.drop_isel(space=[0], time=[0])
Out[19]:
<xarray.DataArray (time: 3, space: 2)>
array([[0.90601059, 0.2009795 ],
[0.28094102, 0.56942557],
[0.88338457, 0.07061534]])
Coordinates:
* time (time) datetime64[ns] 2000-01-02 2000-01-03 2000-01-04
* space (space) <U2 'IL' 'IN'Use drop_vars() to drop a full variable from a Dataset. Any variables depending on it are also dropped:
In [20]:
Copied!
# Drop the time coordinate variable reference
da.drop_vars("time")
# Drop the time coordinate variable reference
da.drop_vars("time")
Out[20]:
<xarray.DataArray (time: 4, space: 3)>
array([[0.17633122, 0.59377299, 0.30699365],
[0.4243673 , 0.90601059, 0.2009795 ],
[0.37157228, 0.28094102, 0.56942557],
[0.60069197, 0.88338457, 0.07061534]])
Coordinates:
* space (space) <U2 'IA' 'IL' 'IN'
Dimensions without coordinates: timeExample of reading a netCDF file using Xarray¶
We will read an Argo data file that describes the temperature and salinity of the water and some of the floats measure other properties that describe the biology/chemistry of the ocean.
The Argo robot instruments drift along the ocean and collect data which are stored in netCDF format and can be acccessed via HTTP and FTP.
Fot this example, we download the file: nodc_4901112_prof.nc
In [ ]:
Copied!
import numpy as np
import xarray as xr
from matplotlib import pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (8,5)
import numpy as np
import xarray as xr
from matplotlib import pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (8,5)
In [21]:
Copied!
# Reading an Argo dataset using Xarray
argo_data = xr.open_dataset('data/nodc_4901112_prof.nc')
# Show file keys
argo_data.keys()
# Reading an Argo dataset using Xarray
argo_data = xr.open_dataset('data/nodc_4901112_prof.nc')
# Show file keys
argo_data.keys()
Out[21]:
KeysView(<xarray.Dataset>
Dimensions: (n_prof: 163, n_param: 3, n_levels: 72, n_calib: 1, n_history: 0)
Dimensions without coordinates: n_prof, n_param, n_levels, n_calib, n_history
Data variables: (12/65)
data_type object ...
format_version object ...
handbook_version object ...
reference_date_time object ...
date_creation object ...
date_update object ...
... ...
history_parameter (n_history, n_prof) object ...
history_start_pres (n_history, n_prof) float32 ...
history_stop_pres (n_history, n_prof) float32 ...
history_previous_value (n_history, n_prof) float32 ...
history_qctest (n_history, n_prof) object ...
crs int32 ...
Attributes: (12/49)
title: Argo float vertical profile
institution: FR GDAC
source: Argo float
history: 2019-10-20T10:17:46Z boyer convAGDAC.f90...
references: https://www.nodc.noaa.gov/argo/
user_manual_version: 3.1
... ...
time_coverage_end: 2015-01-11T22:56:00Z
time_coverage_duration: point
time_coverage_resolution: point
gadr_ConventionVersion: GADR-3.0
gadr_program: convAGDAC.f90
gadr_programVersion: 1.2)
In [22]:
Copied!
# Show dataset structure
argo_data
# Show dataset structure
argo_data
Out[22]:
<xarray.Dataset>
Dimensions: (n_prof: 163, n_param: 3, n_levels: 72, n_calib: 1, n_history: 0)
Dimensions without coordinates: n_prof, n_param, n_levels, n_calib, n_history
Data variables: (12/65)
data_type object b'Argo profile '
format_version object b'3.1 '
handbook_version object b'1.2 '
reference_date_time object b'19500101000000'
date_creation object b'20100919031633'
date_update object b'20190406013003'
... ...
history_parameter (n_history, n_prof) object
history_start_pres (n_history, n_prof) float32
history_stop_pres (n_history, n_prof) float32
history_previous_value (n_history, n_prof) float32
history_qctest (n_history, n_prof) object
crs int32 -2147483647
Attributes: (12/49)
title: Argo float vertical profile
institution: FR GDAC
source: Argo float
history: 2019-10-20T10:17:46Z boyer convAGDAC.f90...
references: https://www.nodc.noaa.gov/argo/
user_manual_version: 3.1
... ...
time_coverage_end: 2015-01-11T22:56:00Z
time_coverage_duration: point
time_coverage_resolution: point
gadr_ConventionVersion: GADR-3.0
gadr_program: convAGDAC.f90
gadr_programVersion: 1.2