Lecture 9 Dask for Parellel Computing and Big Data¶

In past lectures, we learned how to use numpy, pandas, and xarray to analyze various types of geoscience data. In this lecture, we address an incresingly common problem: what happens if the data we wish to analyze is "big data"

What is Dask?¶

Dask is a tool that helps us easily extend our familiar python data analysis tools to medium and big data, i.e. dataset that can't fit in our computer's RAM. In many cases, dask also allows us to speed up our analysis by using mutiple CPU cores. Dask can help us work more efficiently on our laptop, and it can also help us scale up our analysis on HPC and cloud platforms. Most importantly, dask is almost invisible to the user, meaning that you can focus on your science, rather than the details of parallel computing.

Dask was created by the brilliant Matt Rocklin. You can learn more about it on

• The Dask Documentation
• The Dask Github Site

Dask provides collections for big data and a scheduler for parallel computing.

Dask Arrays¶

A dask array looks and feels a lot like a numpy array. However, a dask array doesn't directly hold any data. Instead, it symbolically represents the computations needed to generate the data. Nothing is actually computed until the actual numerical values are needed. This mode of operation is called "lazy"; it allows one to build up complex, large calculations symbolically before turning them over the scheduler for execution.

If we want to create a numpy array of all ones, we do it like this:

In [14]:

import numpy as np
shape = (1000, 4000)
ones_np = np.ones(shape)
ones_np

import numpy as np shape = (1000, 4000) ones_np = np.ones(shape) ones_np

Out[14]:

array([[1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       ...,
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.]])

This array contains exactly 32 MB of data:

In [2]:

ones_np.nbytes / 1e6

ones_np.nbytes / 1e6

Out[2]:

32.0

Now let's create the same array using dask's array interface.

In [3]:

import dask
import dask.array as da
ones = da.ones(shape)

import dask import dask.array as da ones = da.ones(shape)

In [4]:

ones

ones

Out[4]:

Array Chunk Bytes 30.52 MiB 30.52 MiB Shape (1000, 4000) (1000, 4000) Dask graph 1 chunks in 1 graph layer Data type float64 numpy.ndarray

In [5]:

ones = da.ones((100,1000,1000))

ones = da.ones((100,1000,1000))

In [6]:

# Dask will determine an optimal chunk size. 
ones

# Dask will determine an optimal chunk size. ones

Out[6]:

Array Chunk Bytes 762.94 MiB 127.63 MiB Shape (100, 1000, 1000) (100, 409, 409) Dask graph 9 chunks in 1 graph layer Data type float64 numpy.ndarray

By default, dask reads the entire array as a single chuck. If the data size is too big, dask will split the dataset to different chunks. "Chunks" describes how the array is split up over many sub-arrays.

There are several ways to specify chunks. In this lecture, we will use a block shape.

In [7]:

chunk_shape = (1000, 1000)
shape = (1000, 4000)
ones = da.ones(shape, chunks=chunk_shape)
ones

chunk_shape = (1000, 1000) shape = (1000, 4000) ones = da.ones(shape, chunks=chunk_shape) ones

Out[7]:

Array Chunk Bytes 30.52 MiB 7.63 MiB Shape (1000, 4000) (1000, 1000) Dask graph 4 chunks in 1 graph layer Data type float64 numpy.ndarray

In [8]:

ones.chunksize

ones.chunksize

Out[8]:

(1000, 1000)

Notice that we just see a symbolic represetnation of the array, including its shape, dtype, and chunksize. No data has been generated yet. When we call .compute() on a dask array, the computation is trigger and the dask array becomes a numpy array.

In [9]:

ones.compute()

ones.compute()

Out[9]:

array([[1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       ...,
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.]])

In order to understand what happened when we called .compute(), we can visualize the dask graph, the symbolic operations that make up the array

In [19]:

# Install graphviz: mamba install -c conda-forge python-graphviz graphviz

# Install graphviz: mamba install -c conda-forge python-graphviz graphviz

In [18]:

ones.visualize()

ones.visualize()

Out[18]:

Our array has four chunks. To generate it, dask calls np.ones four times and then concatenates this together into one array.

Rather than immediately loading a dask array (which puts all the data into RAM), it is more common to want to reduce the data somehow. For example

In [20]:

sum_of_ones = ones.sum()
sum_of_ones.visualize()

sum_of_ones = ones.sum() sum_of_ones.visualize()

Out[20]:

Here we see dask's strategy for finding the sum. This simple example illustrates the beauty of dask: it automatically designs an algorithm appropriate for custom operations with big data.

If we make our operation more complex, the graph gets more complex.

In [21]:

fancy_calculation = (ones * ones[::-1, ::-1]).mean()
fancy_calculation.visualize()

fancy_calculation = (ones * ones[::-1, ::-1]).mean() fancy_calculation.visualize()

Out[21]:

A Bigger Calculation¶

The examples above were toy examples; the data (32 MB) is nowhere nearly big enough to warrant the use of dask.

We can make it a lot bigger!

In [22]:

bigshape = (200000, 4000)
big_ones = da.ones(bigshape, chunks=chunk_shape)
big_ones

bigshape = (200000, 4000) big_ones = da.ones(bigshape, chunks=chunk_shape) big_ones

Out[22]:

Array Chunk Bytes 5.96 GiB 7.63 MiB Shape (200000, 4000) (1000, 1000) Dask graph 800 chunks in 1 graph layer Data type float64 numpy.ndarray

In [23]:

big_ones.nbytes / 1e6

big_ones.nbytes / 1e6

Out[23]:

6400.0

This dataset is 6.4 GB, rather MB! This is probably close to or greater than the amount of available RAM than you have in your computer. Nevertheless, dask has no problem working on it.

Do not try to .visualize() this array!

When doing a big calculation, dask also has some tools to help us understand what is happening under the hood

In [24]:

from dask.diagnostics import ProgressBar

big_calc = (big_ones * big_ones[::-1, ::-1]).mean()

with ProgressBar():
    result = big_calc.compute()
result

from dask.diagnostics import ProgressBar big_calc = (big_ones * big_ones[::-1, ::-1]).mean() with ProgressBar(): result = big_calc.compute() result

[########################################] | 100% Completed | 2.38 sms

Out[24]:

1.0

Reduction¶

All the usual numpy methods work on dask arrays. You can also apply numpy function directly to a dask array, and it will stay lazy.

In [25]:

big_ones_reduce = (np.cos(big_ones)**2).mean(axis=0)
big_ones_reduce

big_ones_reduce = (np.cos(big_ones)**2).mean(axis=0) big_ones_reduce

Out[25]:

Array Chunk Bytes 31.25 kiB 7.81 kiB Shape (4000,) (1000,) Dask graph 4 chunks in 8 graph layers Data type float64 numpy.ndarray

Plotting also triggers computation, since we need the actual values

In [26]:

from matplotlib import pyplot as plt

from matplotlib import pyplot as plt

In [27]:

plt.plot(big_ones_reduce)

plt.plot(big_ones_reduce)

Out[27]:

[<matplotlib.lines.Line2D at 0x7f50b7bfa820>]

Dask + Xarray¶

In [1]:

import numpy as np
import xarray as xr

import numpy as np import xarray as xr

In [30]:

ds = xr.tutorial.open_dataset("air_temperature")

ds = xr.tutorial.open_dataset("air_temperature")

In [31]:

ds = xr.tutorial.open_dataset(
    "air_temperature",
    chunks={  # this tells xarray to open the dataset as a dask array
        "lat": "auto",
        "lon": "auto",
        "time": 1000,
    },
)
ds

ds = xr.tutorial.open_dataset( "air_temperature", chunks={ # this tells xarray to open the dataset as a dask array "lat": "auto", "lon": "auto", "time": 1000, }, ) ds

Out[31]:

<xarray.Dataset>
Dimensions:  (lat: 25, time: 2920, lon: 53)
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Data variables:
    air      (time, lat, lon) float32 dask.array<chunksize=(1000, 25, 53), meta=np.ndarray>
Attributes:
    Conventions:  COARDS
    title:        4x daily NMC reanalysis (1948)
    description:  Data is from NMC initialized reanalysis\n(4x/day).  These a...
    platform:     Model
    references:   http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...

xarray.Dataset

• Dimensions:
  • lat: 25
  • time: 2920
  • lon: 53
• Coordinates: (3)
  • lat
    (lat)
    float32
    75.0 72.5 70.0 ... 20.0 17.5 15.0

    standard_name :

    latitude

    long_name :

    Latitude

    units :

    degrees_north

    axis :

    Y

    array([75. , 72.5, 70. , 67.5, 65. , 62.5, 60. , 57.5, 55. , 52.5, 50. , 47.5,
           45. , 42.5, 40. , 37.5, 35. , 32.5, 30. , 27.5, 25. , 22.5, 20. , 17.5,
           15. ], dtype=float32)

  • lon
    (lon)
    float32
    200.0 202.5 205.0 ... 327.5 330.0

    standard_name :

    longitude

    long_name :

    Longitude

    units :

    degrees_east

    axis :

    X

    array([200. , 202.5, 205. , 207.5, 210. , 212.5, 215. , 217.5, 220. , 222.5,
           225. , 227.5, 230. , 232.5, 235. , 237.5, 240. , 242.5, 245. , 247.5,
           250. , 252.5, 255. , 257.5, 260. , 262.5, 265. , 267.5, 270. , 272.5,
           275. , 277.5, 280. , 282.5, 285. , 287.5, 290. , 292.5, 295. , 297.5,
           300. , 302.5, 305. , 307.5, 310. , 312.5, 315. , 317.5, 320. , 322.5,
           325. , 327.5, 330. ], dtype=float32)

  • time
    (time)
    datetime64[ns]
    2013-01-01 ... 2014-12-31T18:00:00

    standard_name :

    time

    long_name :

    Time

    array(['2013-01-01T00:00:00.000000000', '2013-01-01T06:00:00.000000000',
           '2013-01-01T12:00:00.000000000', ..., '2014-12-31T06:00:00.000000000',
           '2014-12-31T12:00:00.000000000', '2014-12-31T18:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • air
    (time, lat, lon)
    float32
    dask.array<chunksize=(1000, 25, 53), meta=np.ndarray>

    long_name :

    4xDaily Air temperature at sigma level 995

    units :

    degK

    precision :

    2

    GRIB_id :

    11

    GRIB_name :

    TMP

    var_desc :

    Air temperature

    dataset :

    NMC Reanalysis

    level_desc :

    Surface

    statistic :

    Individual Obs

    parent_stat :

    Other

    actual_range :

    [185.16 322.1 ]

    Array Chunk Bytes 14.76 MiB 5.05 MiB Shape (2920, 25, 53) (1000, 25, 53) Dask graph 3 chunks in 2 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([75.0, 72.5, 70.0, 67.5, 65.0, 62.5, 60.0, 57.5, 55.0, 52.5, 50.0, 47.5,
           45.0, 42.5, 40.0, 37.5, 35.0, 32.5, 30.0, 27.5, 25.0, 22.5, 20.0, 17.5,
           15.0],
          dtype='float32', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([200.0, 202.5, 205.0, 207.5, 210.0, 212.5, 215.0, 217.5, 220.0, 222.5,
           225.0, 227.5, 230.0, 232.5, 235.0, 237.5, 240.0, 242.5, 245.0, 247.5,
           250.0, 252.5, 255.0, 257.5, 260.0, 262.5, 265.0, 267.5, 270.0, 272.5,
           275.0, 277.5, 280.0, 282.5, 285.0, 287.5, 290.0, 292.5, 295.0, 297.5,
           300.0, 302.5, 305.0, 307.5, 310.0, 312.5, 315.0, 317.5, 320.0, 322.5,
           325.0, 327.5, 330.0],
          dtype='float32', name='lon'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2013-01-01 00:00:00', '2013-01-01 06:00:00',
                   '2013-01-01 12:00:00', '2013-01-01 18:00:00',
                   '2013-01-02 00:00:00', '2013-01-02 06:00:00',
                   '2013-01-02 12:00:00', '2013-01-02 18:00:00',
                   '2013-01-03 00:00:00', '2013-01-03 06:00:00',
                   ...
                   '2014-12-29 12:00:00', '2014-12-29 18:00:00',
                   '2014-12-30 00:00:00', '2014-12-30 06:00:00',
                   '2014-12-30 12:00:00', '2014-12-30 18:00:00',
                   '2014-12-31 00:00:00', '2014-12-31 06:00:00',
                   '2014-12-31 12:00:00', '2014-12-31 18:00:00'],
                  dtype='datetime64[ns]', name='time', length=2920, freq=None))

• Attributes: (5)

  Conventions :

  COARDS

  title :

  4x daily NMC reanalysis (1948)

  description :

  Data is from NMC initialized reanalysis (4x/day). These are the 0.9950 sigma level values.

  platform :

  Model

  references :

  http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html

Extracting underlying data¶

There are two ways to pull out the underlying array object in an xarray object.

1. .to_numpy or .values will always return a NumPy array. For dask-backed xarray objects, this means that compute will always be called
2. .data will return a Dask array

{tip}
Use `to_numpy` or `as_numpy` instead of `.values` so that your code generalizes to other array types (like CuPy arrays, sparse arrays)

In [32]:

ds.air.data  # dask array, not numpy

ds.air.data # dask array, not numpy

Out[32]:

Array Chunk Bytes 14.76 MiB 5.05 MiB Shape (2920, 25, 53) (1000, 25, 53) Dask graph 3 chunks in 2 graph layers Data type float32 numpy.ndarray

In [33]:

ds.air.as_numpy().data  ## numpy array

ds.air.as_numpy().data ## numpy array

Out[33]:

array([[[241.2    , 242.5    , 243.5    , ..., 232.79999, 235.5    ,
         238.59999],
        [243.79999, 244.5    , 244.7    , ..., 232.79999, 235.29999,
         239.29999],
        [250.     , 249.79999, 248.89   , ..., 233.2    , 236.39   ,
         241.7    ],
        ...,
        [296.6    , 296.19998, 296.4    , ..., 295.4    , 295.1    ,
         294.69998],
        [295.9    , 296.19998, 296.79   , ..., 295.9    , 295.9    ,
         295.19998],
        [296.29   , 296.79   , 297.1    , ..., 296.9    , 296.79   ,
         296.6    ]],

       [[242.09999, 242.7    , 243.09999, ..., 232.     , 233.59999,
         235.79999],
        [243.59999, 244.09999, 244.2    , ..., 231.     , 232.5    ,
         235.7    ],
        [253.2    , 252.89   , 252.09999, ..., 230.79999, 233.39   ,
         238.5    ],
        ...,
        [296.4    , 295.9    , 296.19998, ..., 295.4    , 295.1    ,
         294.79   ],
        [296.19998, 296.69998, 296.79   , ..., 295.6    , 295.5    ,
         295.1    ],
        [296.29   , 297.19998, 297.4    , ..., 296.4    , 296.4    ,
         296.6    ]],

       [[242.29999, 242.2    , 242.29999, ..., 234.29999, 236.09999,
         238.7    ],
        [244.59999, 244.39   , 244.     , ..., 230.29999, 232.     ,
         235.7    ],
        [256.19998, 255.5    , 254.2    , ..., 231.2    , 233.2    ,
         238.2    ],
        ...,
        [295.6    , 295.4    , 295.4    , ..., 296.29   , 295.29   ,
         295.     ],
        [296.19998, 296.5    , 296.29   , ..., 296.4    , 296.     ,
         295.6    ],
        [296.4    , 296.29   , 296.4    , ..., 297.     , 297.     ,
         296.79   ]],

       ...,

       [[243.48999, 242.98999, 242.09   , ..., 244.18999, 244.48999,
         244.89   ],
        [249.09   , 248.98999, 248.59   , ..., 240.59   , 241.29   ,
         242.68999],
        [262.69   , 262.19   , 261.69   , ..., 239.39   , 241.68999,
         245.18999],
        ...,
        [294.79   , 295.29   , 297.49   , ..., 295.49   , 295.38998,
         294.69   ],
        [296.79   , 297.88998, 298.29   , ..., 295.49   , 295.49   ,
         294.79   ],
        [298.19   , 299.19   , 298.79   , ..., 296.09   , 295.79   ,
         295.79   ]],

       [[245.79   , 244.79   , 243.48999, ..., 243.29   , 243.98999,
         244.79   ],
        [249.89   , 249.29   , 248.48999, ..., 241.29   , 242.48999,
         244.29   ],
        [262.38998, 261.79   , 261.29   , ..., 240.48999, 243.09   ,
         246.89   ],
        ...,
        [293.69   , 293.88998, 295.38998, ..., 295.09   , 294.69   ,
         294.29   ],
        [296.29   , 297.19   , 297.59   , ..., 295.29   , 295.09   ,
         294.38998],
        [297.79   , 298.38998, 298.49   , ..., 295.69   , 295.49   ,
         295.19   ]],

       [[245.09   , 244.29   , 243.29   , ..., 241.68999, 241.48999,
         241.79   ],
        [249.89   , 249.29   , 248.39   , ..., 239.59   , 240.29   ,
         241.68999],
        [262.99   , 262.19   , 261.38998, ..., 239.89   , 242.59   ,
         246.29   ],
        ...,
        [293.79   , 293.69   , 295.09   , ..., 295.29   , 295.09   ,
         294.69   ],
        [296.09   , 296.88998, 297.19   , ..., 295.69   , 295.69   ,
         295.19   ],
        [297.69   , 298.09   , 298.09   , ..., 296.49   , 296.19   ,
         295.69   ]]], dtype=float32)

{exercise}
:label: data-values
Try calling `ds.air.values` and `ds.air.data`. Do you understand the difference?

In [34]:

ds.air.to_numpy()

ds.air.to_numpy()

Out[34]:

array([[[241.2    , 242.5    , 243.5    , ..., 232.79999, 235.5    ,
         238.59999],
        [243.79999, 244.5    , 244.7    , ..., 232.79999, 235.29999,
         239.29999],
        [250.     , 249.79999, 248.89   , ..., 233.2    , 236.39   ,
         241.7    ],
        ...,
        [296.6    , 296.19998, 296.4    , ..., 295.4    , 295.1    ,
         294.69998],
        [295.9    , 296.19998, 296.79   , ..., 295.9    , 295.9    ,
         295.19998],
        [296.29   , 296.79   , 297.1    , ..., 296.9    , 296.79   ,
         296.6    ]],

       [[242.09999, 242.7    , 243.09999, ..., 232.     , 233.59999,
         235.79999],
        [243.59999, 244.09999, 244.2    , ..., 231.     , 232.5    ,
         235.7    ],
        [253.2    , 252.89   , 252.09999, ..., 230.79999, 233.39   ,
         238.5    ],
        ...,
        [296.4    , 295.9    , 296.19998, ..., 295.4    , 295.1    ,
         294.79   ],
        [296.19998, 296.69998, 296.79   , ..., 295.6    , 295.5    ,
         295.1    ],
        [296.29   , 297.19998, 297.4    , ..., 296.4    , 296.4    ,
         296.6    ]],

       [[242.29999, 242.2    , 242.29999, ..., 234.29999, 236.09999,
         238.7    ],
        [244.59999, 244.39   , 244.     , ..., 230.29999, 232.     ,
         235.7    ],
        [256.19998, 255.5    , 254.2    , ..., 231.2    , 233.2    ,
         238.2    ],
        ...,
        [295.6    , 295.4    , 295.4    , ..., 296.29   , 295.29   ,
         295.     ],
        [296.19998, 296.5    , 296.29   , ..., 296.4    , 296.     ,
         295.6    ],
        [296.4    , 296.29   , 296.4    , ..., 297.     , 297.     ,
         296.79   ]],

       ...,

       [[243.48999, 242.98999, 242.09   , ..., 244.18999, 244.48999,
         244.89   ],
        [249.09   , 248.98999, 248.59   , ..., 240.59   , 241.29   ,
         242.68999],
        [262.69   , 262.19   , 261.69   , ..., 239.39   , 241.68999,
         245.18999],
        ...,
        [294.79   , 295.29   , 297.49   , ..., 295.49   , 295.38998,
         294.69   ],
        [296.79   , 297.88998, 298.29   , ..., 295.49   , 295.49   ,
         294.79   ],
        [298.19   , 299.19   , 298.79   , ..., 296.09   , 295.79   ,
         295.79   ]],

       [[245.79   , 244.79   , 243.48999, ..., 243.29   , 243.98999,
         244.79   ],
        [249.89   , 249.29   , 248.48999, ..., 241.29   , 242.48999,
         244.29   ],
        [262.38998, 261.79   , 261.29   , ..., 240.48999, 243.09   ,
         246.89   ],
        ...,
        [293.69   , 293.88998, 295.38998, ..., 295.09   , 294.69   ,
         294.29   ],
        [296.29   , 297.19   , 297.59   , ..., 295.29   , 295.09   ,
         294.38998],
        [297.79   , 298.38998, 298.49   , ..., 295.69   , 295.49   ,
         295.19   ]],

       [[245.09   , 244.29   , 243.29   , ..., 241.68999, 241.48999,
         241.79   ],
        [249.89   , 249.29   , 248.39   , ..., 239.59   , 240.29   ,
         241.68999],
        [262.99   , 262.19   , 261.38998, ..., 239.89   , 242.59   ,
         246.29   ],
        ...,
        [293.79   , 293.69   , 295.09   , ..., 295.29   , 295.09   ,
         294.69   ],
        [296.09   , 296.88998, 297.19   , ..., 295.69   , 295.69   ,
         295.19   ],
        [297.69   , 298.09   , 298.09   , ..., 296.49   , 296.19   ,
         295.69   ]]], dtype=float32)

Lazy computation¶

Xarray seamlessly wraps dask so all computation is deferred until explicitly requested.

In [35]:

mean = ds.air.mean("time")
mean

mean = ds.air.mean("time") mean

Out[35]:

<xarray.DataArray 'air' (lat: 25, lon: 53)>
dask.array<mean_agg-aggregate, shape=(25, 53), dtype=float32, chunksize=(25, 53), chunktype=numpy.ndarray>
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0

xarray.DataArray

'air'

• lat: 25
• lon: 53

• dask.array<chunksize=(25, 53), meta=np.ndarray>

  Array Chunk Bytes 5.18 kiB 5.18 kiB Shape (25, 53) (25, 53) Dask graph 1 chunks in 4 graph layers Data type float32 numpy.ndarray

• Coordinates: (2)
  • lat
    (lat)
    float32
    75.0 72.5 70.0 ... 20.0 17.5 15.0

    standard_name :

    latitude

    long_name :

    Latitude

    units :

    degrees_north

    axis :

    Y

    array([75. , 72.5, 70. , 67.5, 65. , 62.5, 60. , 57.5, 55. , 52.5, 50. , 47.5,
           45. , 42.5, 40. , 37.5, 35. , 32.5, 30. , 27.5, 25. , 22.5, 20. , 17.5,
           15. ], dtype=float32)

  • lon
    (lon)
    float32
    200.0 202.5 205.0 ... 327.5 330.0

    standard_name :

    longitude

    long_name :

    Longitude

    units :

    degrees_east

    axis :

    X

    array([200. , 202.5, 205. , 207.5, 210. , 212.5, 215. , 217.5, 220. , 222.5,
           225. , 227.5, 230. , 232.5, 235. , 237.5, 240. , 242.5, 245. , 247.5,
           250. , 252.5, 255. , 257.5, 260. , 262.5, 265. , 267.5, 270. , 272.5,
           275. , 277.5, 280. , 282.5, 285. , 287.5, 290. , 292.5, 295. , 297.5,
           300. , 302.5, 305. , 307.5, 310. , 312.5, 315. , 317.5, 320. , 322.5,
           325. , 327.5, 330. ], dtype=float32)

• Indexes: (2)
  • lat
    PandasIndex

    PandasIndex(Index([75.0, 72.5, 70.0, 67.5, 65.0, 62.5, 60.0, 57.5, 55.0, 52.5, 50.0, 47.5,
           45.0, 42.5, 40.0, 37.5, 35.0, 32.5, 30.0, 27.5, 25.0, 22.5, 20.0, 17.5,
           15.0],
          dtype='float32', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([200.0, 202.5, 205.0, 207.5, 210.0, 212.5, 215.0, 217.5, 220.0, 222.5,
           225.0, 227.5, 230.0, 232.5, 235.0, 237.5, 240.0, 242.5, 245.0, 247.5,
           250.0, 252.5, 255.0, 257.5, 260.0, 262.5, 265.0, 267.5, 270.0, 272.5,
           275.0, 277.5, 280.0, 282.5, 285.0, 287.5, 290.0, 292.5, 295.0, 297.5,
           300.0, 302.5, 305.0, 307.5, 310.0, 312.5, 315.0, 317.5, 320.0, 322.5,
           325.0, 327.5, 330.0],
          dtype='float32', name='lon'))

• Attributes: (0)

Dask actually constructs a graph of the required computation. Here it's pretty simple: The full array is subdivided into 3 arrays. Dask will load each of these subarrays in a separate thread using the default single-machine scheduling. You can visualize dask 'task graphs' which represent the requested computation:

In [36]:

mean.data  # dask array

mean.data # dask array

Out[36]:

Array Chunk Bytes 5.18 kiB 5.18 kiB Shape (25, 53) (25, 53) Dask graph 1 chunks in 4 graph layers Data type float32 numpy.ndarray

In [37]:

# visualize the graph for the underlying dask array
# we ask it to visualize the graph from left to right because it looks nicer
dask.visualize(mean.data, rankdir="LR", optimize_graph =True)

# visualize the graph for the underlying dask array # we ask it to visualize the graph from left to right because it looks nicer dask.visualize(mean.data, rankdir="LR", optimize_graph =True)

Out[37]:

Getting concrete values¶

At some point, you will want to actually get concrete values (usually a numpy array) from dask.

There are two ways to compute values on dask arrays.

1. .compute() returns an xarray object just like a dask array
2. .load() replaces the dask array in the xarray object with a numpy array. This is equivalent to ds = ds.compute()

In [38]:

ds.load()

ds.load()

Out[38]:

<xarray.Dataset>
Dimensions:  (lat: 25, time: 2920, lon: 53)
Coordinates:
  * lat      (lat) float32 75.0 72.5 70.0 67.5 65.0 ... 25.0 22.5 20.0 17.5 15.0
  * lon      (lon) float32 200.0 202.5 205.0 207.5 ... 322.5 325.0 327.5 330.0
  * time     (time) datetime64[ns] 2013-01-01 ... 2014-12-31T18:00:00
Data variables:
    air      (time, lat, lon) float32 241.2 242.5 243.5 ... 296.5 296.2 295.7
Attributes:
    Conventions:  COARDS
    title:        4x daily NMC reanalysis (1948)
    description:  Data is from NMC initialized reanalysis\n(4x/day).  These a...
    platform:     Model
    references:   http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanaly...

xarray.Dataset

• Dimensions:
  • lat: 25
  • time: 2920
  • lon: 53
• Coordinates: (3)
  • lat
    (lat)
    float32
    75.0 72.5 70.0 ... 20.0 17.5 15.0

    standard_name :

    latitude

    long_name :

    Latitude

    units :

    degrees_north

    axis :

    Y

    array([75. , 72.5, 70. , 67.5, 65. , 62.5, 60. , 57.5, 55. , 52.5, 50. , 47.5,
           45. , 42.5, 40. , 37.5, 35. , 32.5, 30. , 27.5, 25. , 22.5, 20. , 17.5,
           15. ], dtype=float32)

  • lon
    (lon)
    float32
    200.0 202.5 205.0 ... 327.5 330.0

    standard_name :

    longitude

    long_name :

    Longitude

    units :

    degrees_east

    axis :

    X

    array([200. , 202.5, 205. , 207.5, 210. , 212.5, 215. , 217.5, 220. , 222.5,
           225. , 227.5, 230. , 232.5, 235. , 237.5, 240. , 242.5, 245. , 247.5,
           250. , 252.5, 255. , 257.5, 260. , 262.5, 265. , 267.5, 270. , 272.5,
           275. , 277.5, 280. , 282.5, 285. , 287.5, 290. , 292.5, 295. , 297.5,
           300. , 302.5, 305. , 307.5, 310. , 312.5, 315. , 317.5, 320. , 322.5,
           325. , 327.5, 330. ], dtype=float32)

  • time
    (time)
    datetime64[ns]
    2013-01-01 ... 2014-12-31T18:00:00

    standard_name :

    time

    long_name :

    Time

    array(['2013-01-01T00:00:00.000000000', '2013-01-01T06:00:00.000000000',
           '2013-01-01T12:00:00.000000000', ..., '2014-12-31T06:00:00.000000000',
           '2014-12-31T12:00:00.000000000', '2014-12-31T18:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • air
    (time, lat, lon)
    float32
    241.2 242.5 243.5 ... 296.2 295.7

    long_name :

    4xDaily Air temperature at sigma level 995

    units :

    degK

    precision :

    2

    GRIB_id :

    11

    GRIB_name :

    TMP

    var_desc :

    Air temperature

    dataset :

    NMC Reanalysis

    level_desc :

    Surface

    statistic :

    Individual Obs

    parent_stat :

    Other

    actual_range :

    [185.16 322.1 ]

    array([[[241.2    , 242.5    , 243.5    , ..., 232.79999, 235.5    ,
             238.59999],
            [243.79999, 244.5    , 244.7    , ..., 232.79999, 235.29999,
             239.29999],
            [250.     , 249.79999, 248.89   , ..., 233.2    , 236.39   ,
             241.7    ],
            ...,
            [296.6    , 296.19998, 296.4    , ..., 295.4    , 295.1    ,
             294.69998],
            [295.9    , 296.19998, 296.79   , ..., 295.9    , 295.9    ,
             295.19998],
            [296.29   , 296.79   , 297.1    , ..., 296.9    , 296.79   ,
             296.6    ]],
    
           [[242.09999, 242.7    , 243.09999, ..., 232.     , 233.59999,
             235.79999],
            [243.59999, 244.09999, 244.2    , ..., 231.     , 232.5    ,
             235.7    ],
            [253.2    , 252.89   , 252.09999, ..., 230.79999, 233.39   ,
             238.5    ],
    ...
            [293.69   , 293.88998, 295.38998, ..., 295.09   , 294.69   ,
             294.29   ],
            [296.29   , 297.19   , 297.59   , ..., 295.29   , 295.09   ,
             294.38998],
            [297.79   , 298.38998, 298.49   , ..., 295.69   , 295.49   ,
             295.19   ]],
    
           [[245.09   , 244.29   , 243.29   , ..., 241.68999, 241.48999,
             241.79   ],
            [249.89   , 249.29   , 248.39   , ..., 239.59   , 240.29   ,
             241.68999],
            [262.99   , 262.19   , 261.38998, ..., 239.89   , 242.59   ,
             246.29   ],
            ...,
            [293.79   , 293.69   , 295.09   , ..., 295.29   , 295.09   ,
             294.69   ],
            [296.09   , 296.88998, 297.19   , ..., 295.69   , 295.69   ,
             295.19   ],
            [297.69   , 298.09   , 298.09   , ..., 296.49   , 296.19   ,
             295.69   ]]], dtype=float32)

• Indexes: (3)
  • lat
    PandasIndex

    PandasIndex(Index([75.0, 72.5, 70.0, 67.5, 65.0, 62.5, 60.0, 57.5, 55.0, 52.5, 50.0, 47.5,
           45.0, 42.5, 40.0, 37.5, 35.0, 32.5, 30.0, 27.5, 25.0, 22.5, 20.0, 17.5,
           15.0],
          dtype='float32', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([200.0, 202.5, 205.0, 207.5, 210.0, 212.5, 215.0, 217.5, 220.0, 222.5,
           225.0, 227.5, 230.0, 232.5, 235.0, 237.5, 240.0, 242.5, 245.0, 247.5,
           250.0, 252.5, 255.0, 257.5, 260.0, 262.5, 265.0, 267.5, 270.0, 272.5,
           275.0, 277.5, 280.0, 282.5, 285.0, 287.5, 290.0, 292.5, 295.0, 297.5,
           300.0, 302.5, 305.0, 307.5, 310.0, 312.5, 315.0, 317.5, 320.0, 322.5,
           325.0, 327.5, 330.0],
          dtype='float32', name='lon'))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2013-01-01 00:00:00', '2013-01-01 06:00:00',
                   '2013-01-01 12:00:00', '2013-01-01 18:00:00',
                   '2013-01-02 00:00:00', '2013-01-02 06:00:00',
                   '2013-01-02 12:00:00', '2013-01-02 18:00:00',
                   '2013-01-03 00:00:00', '2013-01-03 06:00:00',
                   ...
                   '2014-12-29 12:00:00', '2014-12-29 18:00:00',
                   '2014-12-30 00:00:00', '2014-12-30 06:00:00',
                   '2014-12-30 12:00:00', '2014-12-30 18:00:00',
                   '2014-12-31 00:00:00', '2014-12-31 06:00:00',
                   '2014-12-31 12:00:00', '2014-12-31 18:00:00'],
                  dtype='datetime64[ns]', name='time', length=2920, freq=None))

• Attributes: (5)

  Conventions :

  COARDS

  title :

  4x daily NMC reanalysis (1948)

  description :

  Data is from NMC initialized reanalysis (4x/day). These are the 0.9950 sigma level values.

  platform :

  Model

  references :

  http://www.esrl.noaa.gov/psd/data/gridded/data.ncep.reanalysis.html

Reading multi-file dataset with dask¶

In [2]:

ds_multiple = xr.open_mfdataset('/scratch/xj103/rcaes/GEOS_Chem_Sims/GEOSChem.SpeciesConcHourlyNO2.2020*_2100z.nc4')

ds_multiple = xr.open_mfdataset('/scratch/xj103/rcaes/GEOS_Chem_Sims/GEOSChem.SpeciesConcHourlyNO2.2020*_2100z.nc4')

In [3]:

ds_multiple

ds_multiple

Out[3]:

<xarray.Dataset>
Dimensions:          (time: 30, lev: 47, lat: 81, lon: 65)
Coordinates:
  * time             (time) datetime64[ns] 2020-09-01T21:30:00 ... 2020-09-30...
  * lev              (lev) float64 0.9925 0.9775 0.9625 ... 0.0001387 3.8e-05
  * lat              (lat) float64 27.0 27.25 27.5 27.75 ... 46.5 46.75 47.0
  * lon              (lon) float64 -130.0 -129.7 -129.4 ... -110.6 -110.3 -110.0
Data variables:
    SpeciesConc_NO2  (time, lev, lat, lon) float32 dask.array<chunksize=(1, 47, 81, 65), meta=np.ndarray>

xarray.Dataset

• Dimensions:
  • time: 30
  • lev: 47
  • lat: 81
  • lon: 65
• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2020-09-01T21:30:00 ... 2020-09-...

    long_name :

    Time

    axis :

    T

    array(['2020-09-01T21:30:00.000000000', '2020-09-02T21:30:00.000000000',
           '2020-09-03T21:30:00.000000000', '2020-09-04T21:30:00.000000000',
           '2020-09-05T21:30:00.000000000', '2020-09-06T21:30:00.000000000',
           '2020-09-07T21:30:00.000000000', '2020-09-08T21:30:00.000000000',
           '2020-09-09T21:30:00.000000000', '2020-09-10T21:30:00.000000000',
           '2020-09-11T21:30:00.000000000', '2020-09-12T21:30:00.000000000',
           '2020-09-13T21:30:00.000000000', '2020-09-14T21:30:00.000000000',
           '2020-09-15T21:30:00.000000000', '2020-09-16T21:30:00.000000000',
           '2020-09-17T21:30:00.000000000', '2020-09-18T21:30:00.000000000',
           '2020-09-19T21:30:00.000000000', '2020-09-20T21:30:00.000000000',
           '2020-09-21T21:30:00.000000000', '2020-09-22T21:30:00.000000000',
           '2020-09-23T21:30:00.000000000', '2020-09-24T21:30:00.000000000',
           '2020-09-25T21:30:00.000000000', '2020-09-26T21:30:00.000000000',
           '2020-09-27T21:30:00.000000000', '2020-09-28T21:30:00.000000000',
           '2020-09-29T21:30:00.000000000', '2020-09-30T21:30:00.000000000'],
          dtype='datetime64[ns]')

  • lev
    (lev)
    float64
    0.9925 0.9775 ... 0.0001387 3.8e-05

    long_name :

    hybrid level at midpoints ((A/P0)+B)

    units :

    level

    axis :

    Z

    positive :

    up

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula_terms :

    a: hyam b: hybm p0: P0 ps: PS

    array([9.925000e-01, 9.774999e-01, 9.624998e-01, 9.475000e-01, 9.325001e-01,
           9.174999e-01, 9.024999e-01, 8.875000e-01, 8.725000e-01, 8.575001e-01,
           8.425001e-01, 8.275002e-01, 8.100002e-01, 7.875000e-01, 7.625000e-01,
           7.375001e-01, 7.125001e-01, 6.875001e-01, 6.562501e-01, 6.187502e-01,
           5.812502e-01, 5.437501e-01, 5.062501e-01, 4.687501e-01, 4.312501e-01,
           3.937501e-01, 3.562501e-01, 3.127916e-01, 2.664790e-01, 2.265135e-01,
           1.925410e-01, 1.636615e-01, 1.391150e-01, 1.182500e-01, 1.005144e-01,
           8.543901e-02, 6.745011e-02, 4.828166e-02, 3.427161e-02, 2.407970e-02,
           1.454227e-02, 6.684757e-03, 2.863681e-03, 1.133785e-03, 4.140641e-04,
           1.386745e-04, 3.800000e-05])

  • lat
    (lat)
    float64
    27.0 27.25 27.5 ... 46.5 46.75 47.0

    long_name :

    Latitude

    units :

    degrees_north

    axis :

    Y

    array([27.  , 27.25, 27.5 , 27.75, 28.  , 28.25, 28.5 , 28.75, 29.  , 29.25,
           29.5 , 29.75, 30.  , 30.25, 30.5 , 30.75, 31.  , 31.25, 31.5 , 31.75,
           32.  , 32.25, 32.5 , 32.75, 33.  , 33.25, 33.5 , 33.75, 34.  , 34.25,
           34.5 , 34.75, 35.  , 35.25, 35.5 , 35.75, 36.  , 36.25, 36.5 , 36.75,
           37.  , 37.25, 37.5 , 37.75, 38.  , 38.25, 38.5 , 38.75, 39.  , 39.25,
           39.5 , 39.75, 40.  , 40.25, 40.5 , 40.75, 41.  , 41.25, 41.5 , 41.75,
           42.  , 42.25, 42.5 , 42.75, 43.  , 43.25, 43.5 , 43.75, 44.  , 44.25,
           44.5 , 44.75, 45.  , 45.25, 45.5 , 45.75, 46.  , 46.25, 46.5 , 46.75,
           47.  ])

  • lon
    (lon)
    float64
    -130.0 -129.7 ... -110.3 -110.0

    long_name :

    Longitude

    units :

    degrees_east

    axis :

    X

    array([-130.    , -129.6875, -129.375 , -129.0625, -128.75  , -128.4375,
           -128.125 , -127.8125, -127.5   , -127.1875, -126.875 , -126.5625,
           -126.25  , -125.9375, -125.625 , -125.3125, -125.    , -124.6875,
           -124.375 , -124.0625, -123.75  , -123.4375, -123.125 , -122.8125,
           -122.5   , -122.1875, -121.875 , -121.5625, -121.25  , -120.9375,
           -120.625 , -120.3125, -120.    , -119.6875, -119.375 , -119.0625,
           -118.75  , -118.4375, -118.125 , -117.8125, -117.5   , -117.1875,
           -116.875 , -116.5625, -116.25  , -115.9375, -115.625 , -115.3125,
           -115.    , -114.6875, -114.375 , -114.0625, -113.75  , -113.4375,
           -113.125 , -112.8125, -112.5   , -112.1875, -111.875 , -111.5625,
           -111.25  , -110.9375, -110.625 , -110.3125, -110.    ])

• Data variables: (1)
  • SpeciesConc_NO2
    (time, lev, lat, lon)
    float32
    dask.array<chunksize=(1, 47, 81, 65), meta=np.ndarray>

    long_name :

    Dry mixing ratio of species NO2

    units :

    mol mol-1 dry

    averaging_method :

    time-averaged

    Array Chunk Bytes 28.32 MiB 0.94 MiB Shape (30, 47, 81, 65) (1, 47, 81, 65) Dask graph 30 chunks in 61 graph layers Data type float32 numpy.ndarray

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-09-01 21:30:00', '2020-09-02 21:30:00',
                   '2020-09-03 21:30:00', '2020-09-04 21:30:00',
                   '2020-09-05 21:30:00', '2020-09-06 21:30:00',
                   '2020-09-07 21:30:00', '2020-09-08 21:30:00',
                   '2020-09-09 21:30:00', '2020-09-10 21:30:00',
                   '2020-09-11 21:30:00', '2020-09-12 21:30:00',
                   '2020-09-13 21:30:00', '2020-09-14 21:30:00',
                   '2020-09-15 21:30:00', '2020-09-16 21:30:00',
                   '2020-09-17 21:30:00', '2020-09-18 21:30:00',
                   '2020-09-19 21:30:00', '2020-09-20 21:30:00',
                   '2020-09-21 21:30:00', '2020-09-22 21:30:00',
                   '2020-09-23 21:30:00', '2020-09-24 21:30:00',
                   '2020-09-25 21:30:00', '2020-09-26 21:30:00',
                   '2020-09-27 21:30:00', '2020-09-28 21:30:00',
                   '2020-09-29 21:30:00', '2020-09-30 21:30:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

  • lev
    PandasIndex

    PandasIndex(Index([         0.99250002413,          0.97749990013,     0.9624997759999999,
                      0.947499955,             0.93250006,             0.91749991,
                       0.90249991,             0.88749996,             0.87249996,
               0.8575000599999999,            0.842500125,             0.82750016,
                        0.8100002,     0.7875000200000001,            0.762499965,
               0.7375001050000001,              0.7125001,              0.6875001,
                       0.65625015,     0.6187501999999999,             0.58125015,
                        0.5437501,              0.5062501,    0.46875009999999995,
              0.43125009999999997,              0.3937501,              0.3562501,
              0.31279157999999996,             0.26647905,           0.2265135325,
               0.1925410165877065,     0.1636615040877065,    0.13911500000000002,
                          0.11825,             0.10051436,            0.085439015,
                       0.06745011,   0.048281660000000004,   0.034271609999999994,
             0.024079704999999996,   0.014542270999999999,   0.006684756500000001,
                     0.0028636805,          0.00113378455, 0.00041406405000000003,
                   0.000138674505, 3.8000004999999996e-05],
          dtype='float64', name='lev'))

  • lat
    PandasIndex

    PandasIndex(Index([ 27.0, 27.25,  27.5, 27.75,  28.0, 28.25,  28.5, 28.75,  29.0, 29.25,
            29.5, 29.75,  30.0, 30.25,  30.5, 30.75,  31.0, 31.25,  31.5, 31.75,
            32.0, 32.25,  32.5, 32.75,  33.0, 33.25,  33.5, 33.75,  34.0, 34.25,
            34.5, 34.75,  35.0, 35.25,  35.5, 35.75,  36.0, 36.25,  36.5, 36.75,
            37.0, 37.25,  37.5, 37.75,  38.0, 38.25,  38.5, 38.75,  39.0, 39.25,
            39.5, 39.75,  40.0, 40.25,  40.5, 40.75,  41.0, 41.25,  41.5, 41.75,
            42.0, 42.25,  42.5, 42.75,  43.0, 43.25,  43.5, 43.75,  44.0, 44.25,
            44.5, 44.75,  45.0, 45.25,  45.5, 45.75,  46.0, 46.25,  46.5, 46.75,
            47.0],
          dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([   -130.0, -129.6875,  -129.375, -129.0625,   -128.75, -128.4375,
            -128.125, -127.8125,    -127.5, -127.1875,  -126.875, -126.5625,
             -126.25, -125.9375,  -125.625, -125.3125,    -125.0, -124.6875,
            -124.375, -124.0625,   -123.75, -123.4375,  -123.125, -122.8125,
              -122.5, -122.1875,  -121.875, -121.5625,   -121.25, -120.9375,
            -120.625, -120.3125,    -120.0, -119.6875,  -119.375, -119.0625,
             -118.75, -118.4375,  -118.125, -117.8125,    -117.5, -117.1875,
            -116.875, -116.5625,   -116.25, -115.9375,  -115.625, -115.3125,
              -115.0, -114.6875,  -114.375, -114.0625,   -113.75, -113.4375,
            -113.125, -112.8125,    -112.5, -112.1875,  -111.875, -111.5625,
             -111.25, -110.9375,  -110.625, -110.3125,    -110.0],
          dtype='float64', name='lon'))

• Attributes: (0)

In [5]:

%time ds_multiple.SpeciesConc_NO2.mean(dim = 'time').compute()

%time ds_multiple.SpeciesConc_NO2.mean(dim = 'time').compute()

CPU times: user 462 ms, sys: 89.5 ms, total: 552 ms
Wall time: 2.52 s

Out[5]:

<xarray.DataArray 'SpeciesConc_NO2' (lev: 47, lat: 81, lon: 65)>
array([[[4.9062303e-11, 4.9062081e-11, 4.9061859e-11, ...,
         4.9071434e-11, 4.9071955e-11, 4.9072468e-11],
        [1.6923058e-11, 1.6950108e-11, 1.6952899e-11, ...,
         9.9948189e-11, 9.9900643e-11, 9.9872791e-11],
        [1.6929434e-11, 1.6926667e-11, 1.6934249e-11, ...,
         9.9928288e-11, 9.9878307e-11, 9.9870709e-11],
        ...,
        [8.5456746e-11, 8.5386170e-11, 8.5320598e-11, ...,
         4.0959905e-10, 4.0894488e-10, 4.1077211e-10],
        [8.5364313e-11, 8.5232314e-11, 8.5183277e-11, ...,
         4.0959866e-10, 4.0940437e-10, 4.1138837e-10],
        [8.0792273e-10, 8.0792262e-10, 8.0792262e-10, ...,
         8.0796042e-10, 8.0796181e-10, 8.0796331e-10]],

       [[4.8373788e-11, 4.8373711e-11, 4.8373645e-11, ...,
         4.8377212e-11, 4.8377392e-11, 4.8377573e-11],
        [1.6216743e-11, 1.6300355e-11, 1.6300242e-11, ...,
         9.9929100e-11, 9.9903898e-11, 9.9897722e-11],
        [1.6222215e-11, 1.6223784e-11, 1.6226818e-11, ...,
         9.9919122e-11, 9.9897909e-11, 9.9892851e-11],
...
        [1.2932498e-09, 1.2932503e-09, 1.2932567e-09, ...,
         1.2929462e-09, 1.2929313e-09, 1.2928540e-09],
        [1.2932498e-09, 1.2936645e-09, 1.2936429e-09, ...,
         1.2944885e-09, 1.2945151e-09, 1.2928540e-09],
        [1.2952455e-09, 1.2952455e-09, 1.2952455e-09, ...,
         1.2952455e-09, 1.2952455e-09, 1.2952455e-09]],

       [[1.3088649e-09, 1.3088649e-09, 1.3088649e-09, ...,
         1.3088649e-09, 1.3088649e-09, 1.3088649e-09],
        [1.3110013e-09, 1.3099452e-09, 1.3099236e-09, ...,
         1.3121275e-09, 1.3121271e-09, 1.3131806e-09],
        [1.3110013e-09, 1.3109217e-09, 1.3109236e-09, ...,
         1.3131171e-09, 1.3131758e-09, 1.3131806e-09],
        ...,
        [1.3325602e-09, 1.3325523e-09, 1.3325528e-09, ...,
         1.3367689e-09, 1.3354362e-09, 1.3367826e-09],
        [1.3325604e-09, 1.3321834e-09, 1.3321892e-09, ...,
         1.3367548e-09, 1.3367537e-09, 1.3367827e-09],
        [1.3349080e-09, 1.3349080e-09, 1.3349080e-09, ...,
         1.3349080e-09, 1.3349080e-09, 1.3349080e-09]]], dtype=float32)
Coordinates:
  * lev      (lev) float64 0.9925 0.9775 0.9625 ... 0.0004141 0.0001387 3.8e-05
  * lat      (lat) float64 27.0 27.25 27.5 27.75 28.0 ... 46.25 46.5 46.75 47.0
  * lon      (lon) float64 -130.0 -129.7 -129.4 -129.1 ... -110.6 -110.3 -110.0

xarray.DataArray

'SpeciesConc_NO2'

• lev: 47
• lat: 81
• lon: 65

• 4.906e-11 4.906e-11 4.906e-11 ... 1.335e-09 1.335e-09 1.335e-09

  array([[[4.9062303e-11, 4.9062081e-11, 4.9061859e-11, ...,
           4.9071434e-11, 4.9071955e-11, 4.9072468e-11],
          [1.6923058e-11, 1.6950108e-11, 1.6952899e-11, ...,
           9.9948189e-11, 9.9900643e-11, 9.9872791e-11],
          [1.6929434e-11, 1.6926667e-11, 1.6934249e-11, ...,
           9.9928288e-11, 9.9878307e-11, 9.9870709e-11],
          ...,
          [8.5456746e-11, 8.5386170e-11, 8.5320598e-11, ...,
           4.0959905e-10, 4.0894488e-10, 4.1077211e-10],
          [8.5364313e-11, 8.5232314e-11, 8.5183277e-11, ...,
           4.0959866e-10, 4.0940437e-10, 4.1138837e-10],
          [8.0792273e-10, 8.0792262e-10, 8.0792262e-10, ...,
           8.0796042e-10, 8.0796181e-10, 8.0796331e-10]],
  
         [[4.8373788e-11, 4.8373711e-11, 4.8373645e-11, ...,
           4.8377212e-11, 4.8377392e-11, 4.8377573e-11],
          [1.6216743e-11, 1.6300355e-11, 1.6300242e-11, ...,
           9.9929100e-11, 9.9903898e-11, 9.9897722e-11],
          [1.6222215e-11, 1.6223784e-11, 1.6226818e-11, ...,
           9.9919122e-11, 9.9897909e-11, 9.9892851e-11],
  ...
          [1.2932498e-09, 1.2932503e-09, 1.2932567e-09, ...,
           1.2929462e-09, 1.2929313e-09, 1.2928540e-09],
          [1.2932498e-09, 1.2936645e-09, 1.2936429e-09, ...,
           1.2944885e-09, 1.2945151e-09, 1.2928540e-09],
          [1.2952455e-09, 1.2952455e-09, 1.2952455e-09, ...,
           1.2952455e-09, 1.2952455e-09, 1.2952455e-09]],
  
         [[1.3088649e-09, 1.3088649e-09, 1.3088649e-09, ...,
           1.3088649e-09, 1.3088649e-09, 1.3088649e-09],
          [1.3110013e-09, 1.3099452e-09, 1.3099236e-09, ...,
           1.3121275e-09, 1.3121271e-09, 1.3131806e-09],
          [1.3110013e-09, 1.3109217e-09, 1.3109236e-09, ...,
           1.3131171e-09, 1.3131758e-09, 1.3131806e-09],
          ...,
          [1.3325602e-09, 1.3325523e-09, 1.3325528e-09, ...,
           1.3367689e-09, 1.3354362e-09, 1.3367826e-09],
          [1.3325604e-09, 1.3321834e-09, 1.3321892e-09, ...,
           1.3367548e-09, 1.3367537e-09, 1.3367827e-09],
          [1.3349080e-09, 1.3349080e-09, 1.3349080e-09, ...,
           1.3349080e-09, 1.3349080e-09, 1.3349080e-09]]], dtype=float32)

• Coordinates: (3)
  • lev
    (lev)
    float64
    0.9925 0.9775 ... 0.0001387 3.8e-05

    long_name :

    hybrid level at midpoints ((A/P0)+B)

    units :

    level

    axis :

    Z

    positive :

    up

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula_terms :

    a: hyam b: hybm p0: P0 ps: PS

    array([9.925000e-01, 9.774999e-01, 9.624998e-01, 9.475000e-01, 9.325001e-01,
           9.174999e-01, 9.024999e-01, 8.875000e-01, 8.725000e-01, 8.575001e-01,
           8.425001e-01, 8.275002e-01, 8.100002e-01, 7.875000e-01, 7.625000e-01,
           7.375001e-01, 7.125001e-01, 6.875001e-01, 6.562501e-01, 6.187502e-01,
           5.812502e-01, 5.437501e-01, 5.062501e-01, 4.687501e-01, 4.312501e-01,
           3.937501e-01, 3.562501e-01, 3.127916e-01, 2.664790e-01, 2.265135e-01,
           1.925410e-01, 1.636615e-01, 1.391150e-01, 1.182500e-01, 1.005144e-01,
           8.543901e-02, 6.745011e-02, 4.828166e-02, 3.427161e-02, 2.407970e-02,
           1.454227e-02, 6.684757e-03, 2.863681e-03, 1.133785e-03, 4.140641e-04,
           1.386745e-04, 3.800000e-05])

  • lat
    (lat)
    float64
    27.0 27.25 27.5 ... 46.5 46.75 47.0

    long_name :

    Latitude

    units :

    degrees_north

    axis :

    Y

    array([27.  , 27.25, 27.5 , 27.75, 28.  , 28.25, 28.5 , 28.75, 29.  , 29.25,
           29.5 , 29.75, 30.  , 30.25, 30.5 , 30.75, 31.  , 31.25, 31.5 , 31.75,
           32.  , 32.25, 32.5 , 32.75, 33.  , 33.25, 33.5 , 33.75, 34.  , 34.25,
           34.5 , 34.75, 35.  , 35.25, 35.5 , 35.75, 36.  , 36.25, 36.5 , 36.75,
           37.  , 37.25, 37.5 , 37.75, 38.  , 38.25, 38.5 , 38.75, 39.  , 39.25,
           39.5 , 39.75, 40.  , 40.25, 40.5 , 40.75, 41.  , 41.25, 41.5 , 41.75,
           42.  , 42.25, 42.5 , 42.75, 43.  , 43.25, 43.5 , 43.75, 44.  , 44.25,
           44.5 , 44.75, 45.  , 45.25, 45.5 , 45.75, 46.  , 46.25, 46.5 , 46.75,
           47.  ])

  • lon
    (lon)
    float64
    -130.0 -129.7 ... -110.3 -110.0

    long_name :

    Longitude

    units :

    degrees_east

    axis :

    X

    array([-130.    , -129.6875, -129.375 , -129.0625, -128.75  , -128.4375,
           -128.125 , -127.8125, -127.5   , -127.1875, -126.875 , -126.5625,
           -126.25  , -125.9375, -125.625 , -125.3125, -125.    , -124.6875,
           -124.375 , -124.0625, -123.75  , -123.4375, -123.125 , -122.8125,
           -122.5   , -122.1875, -121.875 , -121.5625, -121.25  , -120.9375,
           -120.625 , -120.3125, -120.    , -119.6875, -119.375 , -119.0625,
           -118.75  , -118.4375, -118.125 , -117.8125, -117.5   , -117.1875,
           -116.875 , -116.5625, -116.25  , -115.9375, -115.625 , -115.3125,
           -115.    , -114.6875, -114.375 , -114.0625, -113.75  , -113.4375,
           -113.125 , -112.8125, -112.5   , -112.1875, -111.875 , -111.5625,
           -111.25  , -110.9375, -110.625 , -110.3125, -110.    ])

• Indexes: (3)
  • lev
    PandasIndex

    PandasIndex(Index([         0.99250002413,          0.97749990013,     0.9624997759999999,
                      0.947499955,             0.93250006,             0.91749991,
                       0.90249991,             0.88749996,             0.87249996,
               0.8575000599999999,            0.842500125,             0.82750016,
                        0.8100002,     0.7875000200000001,            0.762499965,
               0.7375001050000001,              0.7125001,              0.6875001,
                       0.65625015,     0.6187501999999999,             0.58125015,
                        0.5437501,              0.5062501,    0.46875009999999995,
              0.43125009999999997,              0.3937501,              0.3562501,
              0.31279157999999996,             0.26647905,           0.2265135325,
               0.1925410165877065,     0.1636615040877065,    0.13911500000000002,
                          0.11825,             0.10051436,            0.085439015,
                       0.06745011,   0.048281660000000004,   0.034271609999999994,
             0.024079704999999996,   0.014542270999999999,   0.006684756500000001,
                     0.0028636805,          0.00113378455, 0.00041406405000000003,
                   0.000138674505, 3.8000004999999996e-05],
          dtype='float64', name='lev'))

  • lat
    PandasIndex

    PandasIndex(Index([ 27.0, 27.25,  27.5, 27.75,  28.0, 28.25,  28.5, 28.75,  29.0, 29.25,
            29.5, 29.75,  30.0, 30.25,  30.5, 30.75,  31.0, 31.25,  31.5, 31.75,
            32.0, 32.25,  32.5, 32.75,  33.0, 33.25,  33.5, 33.75,  34.0, 34.25,
            34.5, 34.75,  35.0, 35.25,  35.5, 35.75,  36.0, 36.25,  36.5, 36.75,
            37.0, 37.25,  37.5, 37.75,  38.0, 38.25,  38.5, 38.75,  39.0, 39.25,
            39.5, 39.75,  40.0, 40.25,  40.5, 40.75,  41.0, 41.25,  41.5, 41.75,
            42.0, 42.25,  42.5, 42.75,  43.0, 43.25,  43.5, 43.75,  44.0, 44.25,
            44.5, 44.75,  45.0, 45.25,  45.5, 45.75,  46.0, 46.25,  46.5, 46.75,
            47.0],
          dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([   -130.0, -129.6875,  -129.375, -129.0625,   -128.75, -128.4375,
            -128.125, -127.8125,    -127.5, -127.1875,  -126.875, -126.5625,
             -126.25, -125.9375,  -125.625, -125.3125,    -125.0, -124.6875,
            -124.375, -124.0625,   -123.75, -123.4375,  -123.125, -122.8125,
              -122.5, -122.1875,  -121.875, -121.5625,   -121.25, -120.9375,
            -120.625, -120.3125,    -120.0, -119.6875,  -119.375, -119.0625,
             -118.75, -118.4375,  -118.125, -117.8125,    -117.5, -117.1875,
            -116.875, -116.5625,   -116.25, -115.9375,  -115.625, -115.3125,
              -115.0, -114.6875,  -114.375, -114.0625,   -113.75, -113.4375,
            -113.125, -112.8125,    -112.5, -112.1875,  -111.875, -111.5625,
             -111.25, -110.9375,  -110.625, -110.3125,    -110.0],
          dtype='float64', name='lon'))

• Attributes: (0)

In [8]:

rolling_mean = ds_multiple.SpeciesConc_NO2.rolling(time=5).mean()  
rolling_mean  # contains dask array

rolling_mean = ds_multiple.SpeciesConc_NO2.rolling(time=5).mean() rolling_mean # contains dask array

Out[8]:

<xarray.DataArray 'SpeciesConc_NO2' (time: 30, lev: 47, lat: 81, lon: 65)>
dask.array<truediv, shape=(30, 47, 81, 65), dtype=float32, chunksize=(5, 47, 81, 65), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) datetime64[ns] 2020-09-01T21:30:00 ... 2020-09-30T21:30:00
  * lev      (lev) float64 0.9925 0.9775 0.9625 ... 0.0004141 0.0001387 3.8e-05
  * lat      (lat) float64 27.0 27.25 27.5 27.75 28.0 ... 46.25 46.5 46.75 47.0
  * lon      (lon) float64 -130.0 -129.7 -129.4 -129.1 ... -110.6 -110.3 -110.0
Attributes:
    long_name:         Dry mixing ratio of species NO2
    units:             mol mol-1 dry
    averaging_method:  time-averaged

xarray.DataArray

'SpeciesConc_NO2'

• time: 30
• lev: 47
• lat: 81
• lon: 65

• dask.array<chunksize=(5, 47, 81, 65), meta=np.ndarray>

  Array Chunk Bytes 28.32 MiB 4.72 MiB Shape (30, 47, 81, 65) (5, 47, 81, 65) Dask graph 6 chunks in 86 graph layers Data type float32 numpy.ndarray

• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2020-09-01T21:30:00 ... 2020-09-...

    long_name :

    Time

    axis :

    T

    array(['2020-09-01T21:30:00.000000000', '2020-09-02T21:30:00.000000000',
           '2020-09-03T21:30:00.000000000', '2020-09-04T21:30:00.000000000',
           '2020-09-05T21:30:00.000000000', '2020-09-06T21:30:00.000000000',
           '2020-09-07T21:30:00.000000000', '2020-09-08T21:30:00.000000000',
           '2020-09-09T21:30:00.000000000', '2020-09-10T21:30:00.000000000',
           '2020-09-11T21:30:00.000000000', '2020-09-12T21:30:00.000000000',
           '2020-09-13T21:30:00.000000000', '2020-09-14T21:30:00.000000000',
           '2020-09-15T21:30:00.000000000', '2020-09-16T21:30:00.000000000',
           '2020-09-17T21:30:00.000000000', '2020-09-18T21:30:00.000000000',
           '2020-09-19T21:30:00.000000000', '2020-09-20T21:30:00.000000000',
           '2020-09-21T21:30:00.000000000', '2020-09-22T21:30:00.000000000',
           '2020-09-23T21:30:00.000000000', '2020-09-24T21:30:00.000000000',
           '2020-09-25T21:30:00.000000000', '2020-09-26T21:30:00.000000000',
           '2020-09-27T21:30:00.000000000', '2020-09-28T21:30:00.000000000',
           '2020-09-29T21:30:00.000000000', '2020-09-30T21:30:00.000000000'],
          dtype='datetime64[ns]')

  • lev
    (lev)
    float64
    0.9925 0.9775 ... 0.0001387 3.8e-05

    long_name :

    hybrid level at midpoints ((A/P0)+B)

    units :

    level

    axis :

    Z

    positive :

    up

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula_terms :

    a: hyam b: hybm p0: P0 ps: PS

    array([9.925000e-01, 9.774999e-01, 9.624998e-01, 9.475000e-01, 9.325001e-01,
           9.174999e-01, 9.024999e-01, 8.875000e-01, 8.725000e-01, 8.575001e-01,
           8.425001e-01, 8.275002e-01, 8.100002e-01, 7.875000e-01, 7.625000e-01,
           7.375001e-01, 7.125001e-01, 6.875001e-01, 6.562501e-01, 6.187502e-01,
           5.812502e-01, 5.437501e-01, 5.062501e-01, 4.687501e-01, 4.312501e-01,
           3.937501e-01, 3.562501e-01, 3.127916e-01, 2.664790e-01, 2.265135e-01,
           1.925410e-01, 1.636615e-01, 1.391150e-01, 1.182500e-01, 1.005144e-01,
           8.543901e-02, 6.745011e-02, 4.828166e-02, 3.427161e-02, 2.407970e-02,
           1.454227e-02, 6.684757e-03, 2.863681e-03, 1.133785e-03, 4.140641e-04,
           1.386745e-04, 3.800000e-05])

  • lat
    (lat)
    float64
    27.0 27.25 27.5 ... 46.5 46.75 47.0

    long_name :

    Latitude

    units :

    degrees_north

    axis :

    Y

    array([27.  , 27.25, 27.5 , 27.75, 28.  , 28.25, 28.5 , 28.75, 29.  , 29.25,
           29.5 , 29.75, 30.  , 30.25, 30.5 , 30.75, 31.  , 31.25, 31.5 , 31.75,
           32.  , 32.25, 32.5 , 32.75, 33.  , 33.25, 33.5 , 33.75, 34.  , 34.25,
           34.5 , 34.75, 35.  , 35.25, 35.5 , 35.75, 36.  , 36.25, 36.5 , 36.75,
           37.  , 37.25, 37.5 , 37.75, 38.  , 38.25, 38.5 , 38.75, 39.  , 39.25,
           39.5 , 39.75, 40.  , 40.25, 40.5 , 40.75, 41.  , 41.25, 41.5 , 41.75,
           42.  , 42.25, 42.5 , 42.75, 43.  , 43.25, 43.5 , 43.75, 44.  , 44.25,
           44.5 , 44.75, 45.  , 45.25, 45.5 , 45.75, 46.  , 46.25, 46.5 , 46.75,
           47.  ])

  • lon
    (lon)
    float64
    -130.0 -129.7 ... -110.3 -110.0

    long_name :

    Longitude

    units :

    degrees_east

    axis :

    X

    array([-130.    , -129.6875, -129.375 , -129.0625, -128.75  , -128.4375,
           -128.125 , -127.8125, -127.5   , -127.1875, -126.875 , -126.5625,
           -126.25  , -125.9375, -125.625 , -125.3125, -125.    , -124.6875,
           -124.375 , -124.0625, -123.75  , -123.4375, -123.125 , -122.8125,
           -122.5   , -122.1875, -121.875 , -121.5625, -121.25  , -120.9375,
           -120.625 , -120.3125, -120.    , -119.6875, -119.375 , -119.0625,
           -118.75  , -118.4375, -118.125 , -117.8125, -117.5   , -117.1875,
           -116.875 , -116.5625, -116.25  , -115.9375, -115.625 , -115.3125,
           -115.    , -114.6875, -114.375 , -114.0625, -113.75  , -113.4375,
           -113.125 , -112.8125, -112.5   , -112.1875, -111.875 , -111.5625,
           -111.25  , -110.9375, -110.625 , -110.3125, -110.    ])

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-09-01 21:30:00', '2020-09-02 21:30:00',
                   '2020-09-03 21:30:00', '2020-09-04 21:30:00',
                   '2020-09-05 21:30:00', '2020-09-06 21:30:00',
                   '2020-09-07 21:30:00', '2020-09-08 21:30:00',
                   '2020-09-09 21:30:00', '2020-09-10 21:30:00',
                   '2020-09-11 21:30:00', '2020-09-12 21:30:00',
                   '2020-09-13 21:30:00', '2020-09-14 21:30:00',
                   '2020-09-15 21:30:00', '2020-09-16 21:30:00',
                   '2020-09-17 21:30:00', '2020-09-18 21:30:00',
                   '2020-09-19 21:30:00', '2020-09-20 21:30:00',
                   '2020-09-21 21:30:00', '2020-09-22 21:30:00',
                   '2020-09-23 21:30:00', '2020-09-24 21:30:00',
                   '2020-09-25 21:30:00', '2020-09-26 21:30:00',
                   '2020-09-27 21:30:00', '2020-09-28 21:30:00',
                   '2020-09-29 21:30:00', '2020-09-30 21:30:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

  • lev
    PandasIndex

    PandasIndex(Index([         0.99250002413,          0.97749990013,     0.9624997759999999,
                      0.947499955,             0.93250006,             0.91749991,
                       0.90249991,             0.88749996,             0.87249996,
               0.8575000599999999,            0.842500125,             0.82750016,
                        0.8100002,     0.7875000200000001,            0.762499965,
               0.7375001050000001,              0.7125001,              0.6875001,
                       0.65625015,     0.6187501999999999,             0.58125015,
                        0.5437501,              0.5062501,    0.46875009999999995,
              0.43125009999999997,              0.3937501,              0.3562501,
              0.31279157999999996,             0.26647905,           0.2265135325,
               0.1925410165877065,     0.1636615040877065,    0.13911500000000002,
                          0.11825,             0.10051436,            0.085439015,
                       0.06745011,   0.048281660000000004,   0.034271609999999994,
             0.024079704999999996,   0.014542270999999999,   0.006684756500000001,
                     0.0028636805,          0.00113378455, 0.00041406405000000003,
                   0.000138674505, 3.8000004999999996e-05],
          dtype='float64', name='lev'))

  • lat
    PandasIndex

    PandasIndex(Index([ 27.0, 27.25,  27.5, 27.75,  28.0, 28.25,  28.5, 28.75,  29.0, 29.25,
            29.5, 29.75,  30.0, 30.25,  30.5, 30.75,  31.0, 31.25,  31.5, 31.75,
            32.0, 32.25,  32.5, 32.75,  33.0, 33.25,  33.5, 33.75,  34.0, 34.25,
            34.5, 34.75,  35.0, 35.25,  35.5, 35.75,  36.0, 36.25,  36.5, 36.75,
            37.0, 37.25,  37.5, 37.75,  38.0, 38.25,  38.5, 38.75,  39.0, 39.25,
            39.5, 39.75,  40.0, 40.25,  40.5, 40.75,  41.0, 41.25,  41.5, 41.75,
            42.0, 42.25,  42.5, 42.75,  43.0, 43.25,  43.5, 43.75,  44.0, 44.25,
            44.5, 44.75,  45.0, 45.25,  45.5, 45.75,  46.0, 46.25,  46.5, 46.75,
            47.0],
          dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([   -130.0, -129.6875,  -129.375, -129.0625,   -128.75, -128.4375,
            -128.125, -127.8125,    -127.5, -127.1875,  -126.875, -126.5625,
             -126.25, -125.9375,  -125.625, -125.3125,    -125.0, -124.6875,
            -124.375, -124.0625,   -123.75, -123.4375,  -123.125, -122.8125,
              -122.5, -122.1875,  -121.875, -121.5625,   -121.25, -120.9375,
            -120.625, -120.3125,    -120.0, -119.6875,  -119.375, -119.0625,
             -118.75, -118.4375,  -118.125, -117.8125,    -117.5, -117.1875,
            -116.875, -116.5625,   -116.25, -115.9375,  -115.625, -115.3125,
              -115.0, -114.6875,  -114.375, -114.0625,   -113.75, -113.4375,
            -113.125, -112.8125,    -112.5, -112.1875,  -111.875, -111.5625,
             -111.25, -110.9375,  -110.625, -110.3125,    -110.0],
          dtype='float64', name='lon'))

• Attributes: (3)

  long_name :

  Dry mixing ratio of species NO2

  units :

  mol mol-1 dry

  averaging_method :

  time-averaged

In [9]:

computed = rolling_mean.compute()  
computed  # has real numpy values

computed = rolling_mean.compute() computed # has real numpy values

Out[9]:

<xarray.DataArray 'SpeciesConc_NO2' (time: 30, lev: 47, lat: 81, lon: 65)>
array([[[[           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
         [           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
         [           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
         ...,
         [           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
         [           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
         [           nan,            nan,            nan, ...,
                     nan,            nan,            nan]],

        [[           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
         [           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
         [           nan,            nan,            nan, ...,
                     nan,            nan,            nan],
...
          1.27299549e-09, 1.27348032e-09, 1.27298894e-09],
         [1.27392474e-09, 1.27276323e-09, 1.27275179e-09, ...,
          1.27361832e-09, 1.27358102e-09, 1.27298905e-09],
         [1.27646427e-09, 1.27646427e-09, 1.27646427e-09, ...,
          1.27646427e-09, 1.27646427e-09, 1.27646427e-09]],

        [[1.25867883e-09, 1.25867883e-09, 1.25867883e-09, ...,
          1.25867883e-09, 1.25867883e-09, 1.25867883e-09],
         [1.25231525e-09, 1.25114186e-09, 1.25108790e-09, ...,
          1.25833632e-09, 1.25830835e-09, 1.25872757e-09],
         [1.25231525e-09, 1.25225497e-09, 1.25224831e-09, ...,
          1.25869781e-09, 1.25869914e-09, 1.25872757e-09],
         ...,
         [1.28924105e-09, 1.28924049e-09, 1.28924071e-09, ...,
          1.29703770e-09, 1.29703892e-09, 1.29704647e-09],
         [1.28924105e-09, 1.28915789e-09, 1.28917210e-09, ...,
          1.29697664e-09, 1.29696454e-09, 1.29704658e-09],
         [1.29171085e-09, 1.29171085e-09, 1.29171085e-09, ...,
          1.29171085e-09, 1.29171085e-09, 1.29171085e-09]]]],
      dtype=float32)
Coordinates:
  * time     (time) datetime64[ns] 2020-09-01T21:30:00 ... 2020-09-30T21:30:00
  * lev      (lev) float64 0.9925 0.9775 0.9625 ... 0.0004141 0.0001387 3.8e-05
  * lat      (lat) float64 27.0 27.25 27.5 27.75 28.0 ... 46.25 46.5 46.75 47.0
  * lon      (lon) float64 -130.0 -129.7 -129.4 -129.1 ... -110.6 -110.3 -110.0
Attributes:
    long_name:         Dry mixing ratio of species NO2
    units:             mol mol-1 dry
    averaging_method:  time-averaged

xarray.DataArray

'SpeciesConc_NO2'

• time: 30
• lev: 47
• lat: 81
• lon: 65

• nan nan nan nan nan ... 1.292e-09 1.292e-09 1.292e-09 1.292e-09

  array([[[[           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
           [           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
           [           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
           ...,
           [           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
           [           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
           [           nan,            nan,            nan, ...,
                       nan,            nan,            nan]],
  
          [[           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
           [           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
           [           nan,            nan,            nan, ...,
                       nan,            nan,            nan],
  ...
            1.27299549e-09, 1.27348032e-09, 1.27298894e-09],
           [1.27392474e-09, 1.27276323e-09, 1.27275179e-09, ...,
            1.27361832e-09, 1.27358102e-09, 1.27298905e-09],
           [1.27646427e-09, 1.27646427e-09, 1.27646427e-09, ...,
            1.27646427e-09, 1.27646427e-09, 1.27646427e-09]],
  
          [[1.25867883e-09, 1.25867883e-09, 1.25867883e-09, ...,
            1.25867883e-09, 1.25867883e-09, 1.25867883e-09],
           [1.25231525e-09, 1.25114186e-09, 1.25108790e-09, ...,
            1.25833632e-09, 1.25830835e-09, 1.25872757e-09],
           [1.25231525e-09, 1.25225497e-09, 1.25224831e-09, ...,
            1.25869781e-09, 1.25869914e-09, 1.25872757e-09],
           ...,
           [1.28924105e-09, 1.28924049e-09, 1.28924071e-09, ...,
            1.29703770e-09, 1.29703892e-09, 1.29704647e-09],
           [1.28924105e-09, 1.28915789e-09, 1.28917210e-09, ...,
            1.29697664e-09, 1.29696454e-09, 1.29704658e-09],
           [1.29171085e-09, 1.29171085e-09, 1.29171085e-09, ...,
            1.29171085e-09, 1.29171085e-09, 1.29171085e-09]]]],
        dtype=float32)

• Coordinates: (4)
  • time
    (time)
    datetime64[ns]
    2020-09-01T21:30:00 ... 2020-09-...

    long_name :

    Time

    axis :

    T

    array(['2020-09-01T21:30:00.000000000', '2020-09-02T21:30:00.000000000',
           '2020-09-03T21:30:00.000000000', '2020-09-04T21:30:00.000000000',
           '2020-09-05T21:30:00.000000000', '2020-09-06T21:30:00.000000000',
           '2020-09-07T21:30:00.000000000', '2020-09-08T21:30:00.000000000',
           '2020-09-09T21:30:00.000000000', '2020-09-10T21:30:00.000000000',
           '2020-09-11T21:30:00.000000000', '2020-09-12T21:30:00.000000000',
           '2020-09-13T21:30:00.000000000', '2020-09-14T21:30:00.000000000',
           '2020-09-15T21:30:00.000000000', '2020-09-16T21:30:00.000000000',
           '2020-09-17T21:30:00.000000000', '2020-09-18T21:30:00.000000000',
           '2020-09-19T21:30:00.000000000', '2020-09-20T21:30:00.000000000',
           '2020-09-21T21:30:00.000000000', '2020-09-22T21:30:00.000000000',
           '2020-09-23T21:30:00.000000000', '2020-09-24T21:30:00.000000000',
           '2020-09-25T21:30:00.000000000', '2020-09-26T21:30:00.000000000',
           '2020-09-27T21:30:00.000000000', '2020-09-28T21:30:00.000000000',
           '2020-09-29T21:30:00.000000000', '2020-09-30T21:30:00.000000000'],
          dtype='datetime64[ns]')

  • lev
    (lev)
    float64
    0.9925 0.9775 ... 0.0001387 3.8e-05

    long_name :

    hybrid level at midpoints ((A/P0)+B)

    units :

    level

    axis :

    Z

    positive :

    up

    standard_name :

    atmosphere_hybrid_sigma_pressure_coordinate

    formula_terms :

    a: hyam b: hybm p0: P0 ps: PS

    array([9.925000e-01, 9.774999e-01, 9.624998e-01, 9.475000e-01, 9.325001e-01,
           9.174999e-01, 9.024999e-01, 8.875000e-01, 8.725000e-01, 8.575001e-01,
           8.425001e-01, 8.275002e-01, 8.100002e-01, 7.875000e-01, 7.625000e-01,
           7.375001e-01, 7.125001e-01, 6.875001e-01, 6.562501e-01, 6.187502e-01,
           5.812502e-01, 5.437501e-01, 5.062501e-01, 4.687501e-01, 4.312501e-01,
           3.937501e-01, 3.562501e-01, 3.127916e-01, 2.664790e-01, 2.265135e-01,
           1.925410e-01, 1.636615e-01, 1.391150e-01, 1.182500e-01, 1.005144e-01,
           8.543901e-02, 6.745011e-02, 4.828166e-02, 3.427161e-02, 2.407970e-02,
           1.454227e-02, 6.684757e-03, 2.863681e-03, 1.133785e-03, 4.140641e-04,
           1.386745e-04, 3.800000e-05])

  • lat
    (lat)
    float64
    27.0 27.25 27.5 ... 46.5 46.75 47.0

    long_name :

    Latitude

    units :

    degrees_north

    axis :

    Y

    array([27.  , 27.25, 27.5 , 27.75, 28.  , 28.25, 28.5 , 28.75, 29.  , 29.25,
           29.5 , 29.75, 30.  , 30.25, 30.5 , 30.75, 31.  , 31.25, 31.5 , 31.75,
           32.  , 32.25, 32.5 , 32.75, 33.  , 33.25, 33.5 , 33.75, 34.  , 34.25,
           34.5 , 34.75, 35.  , 35.25, 35.5 , 35.75, 36.  , 36.25, 36.5 , 36.75,
           37.  , 37.25, 37.5 , 37.75, 38.  , 38.25, 38.5 , 38.75, 39.  , 39.25,
           39.5 , 39.75, 40.  , 40.25, 40.5 , 40.75, 41.  , 41.25, 41.5 , 41.75,
           42.  , 42.25, 42.5 , 42.75, 43.  , 43.25, 43.5 , 43.75, 44.  , 44.25,
           44.5 , 44.75, 45.  , 45.25, 45.5 , 45.75, 46.  , 46.25, 46.5 , 46.75,
           47.  ])

  • lon
    (lon)
    float64
    -130.0 -129.7 ... -110.3 -110.0

    long_name :

    Longitude

    units :

    degrees_east

    axis :

    X

    array([-130.    , -129.6875, -129.375 , -129.0625, -128.75  , -128.4375,
           -128.125 , -127.8125, -127.5   , -127.1875, -126.875 , -126.5625,
           -126.25  , -125.9375, -125.625 , -125.3125, -125.    , -124.6875,
           -124.375 , -124.0625, -123.75  , -123.4375, -123.125 , -122.8125,
           -122.5   , -122.1875, -121.875 , -121.5625, -121.25  , -120.9375,
           -120.625 , -120.3125, -120.    , -119.6875, -119.375 , -119.0625,
           -118.75  , -118.4375, -118.125 , -117.8125, -117.5   , -117.1875,
           -116.875 , -116.5625, -116.25  , -115.9375, -115.625 , -115.3125,
           -115.    , -114.6875, -114.375 , -114.0625, -113.75  , -113.4375,
           -113.125 , -112.8125, -112.5   , -112.1875, -111.875 , -111.5625,
           -111.25  , -110.9375, -110.625 , -110.3125, -110.    ])

• Indexes: (4)
  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['2020-09-01 21:30:00', '2020-09-02 21:30:00',
                   '2020-09-03 21:30:00', '2020-09-04 21:30:00',
                   '2020-09-05 21:30:00', '2020-09-06 21:30:00',
                   '2020-09-07 21:30:00', '2020-09-08 21:30:00',
                   '2020-09-09 21:30:00', '2020-09-10 21:30:00',
                   '2020-09-11 21:30:00', '2020-09-12 21:30:00',
                   '2020-09-13 21:30:00', '2020-09-14 21:30:00',
                   '2020-09-15 21:30:00', '2020-09-16 21:30:00',
                   '2020-09-17 21:30:00', '2020-09-18 21:30:00',
                   '2020-09-19 21:30:00', '2020-09-20 21:30:00',
                   '2020-09-21 21:30:00', '2020-09-22 21:30:00',
                   '2020-09-23 21:30:00', '2020-09-24 21:30:00',
                   '2020-09-25 21:30:00', '2020-09-26 21:30:00',
                   '2020-09-27 21:30:00', '2020-09-28 21:30:00',
                   '2020-09-29 21:30:00', '2020-09-30 21:30:00'],
                  dtype='datetime64[ns]', name='time', freq=None))

  • lev
    PandasIndex

    PandasIndex(Index([         0.99250002413,          0.97749990013,     0.9624997759999999,
                      0.947499955,             0.93250006,             0.91749991,
                       0.90249991,             0.88749996,             0.87249996,
               0.8575000599999999,            0.842500125,             0.82750016,
                        0.8100002,     0.7875000200000001,            0.762499965,
               0.7375001050000001,              0.7125001,              0.6875001,
                       0.65625015,     0.6187501999999999,             0.58125015,
                        0.5437501,              0.5062501,    0.46875009999999995,
              0.43125009999999997,              0.3937501,              0.3562501,
              0.31279157999999996,             0.26647905,           0.2265135325,
               0.1925410165877065,     0.1636615040877065,    0.13911500000000002,
                          0.11825,             0.10051436,            0.085439015,
                       0.06745011,   0.048281660000000004,   0.034271609999999994,
             0.024079704999999996,   0.014542270999999999,   0.006684756500000001,
                     0.0028636805,          0.00113378455, 0.00041406405000000003,
                   0.000138674505, 3.8000004999999996e-05],
          dtype='float64', name='lev'))

  • lat
    PandasIndex

    PandasIndex(Index([ 27.0, 27.25,  27.5, 27.75,  28.0, 28.25,  28.5, 28.75,  29.0, 29.25,
            29.5, 29.75,  30.0, 30.25,  30.5, 30.75,  31.0, 31.25,  31.5, 31.75,
            32.0, 32.25,  32.5, 32.75,  33.0, 33.25,  33.5, 33.75,  34.0, 34.25,
            34.5, 34.75,  35.0, 35.25,  35.5, 35.75,  36.0, 36.25,  36.5, 36.75,
            37.0, 37.25,  37.5, 37.75,  38.0, 38.25,  38.5, 38.75,  39.0, 39.25,
            39.5, 39.75,  40.0, 40.25,  40.5, 40.75,  41.0, 41.25,  41.5, 41.75,
            42.0, 42.25,  42.5, 42.75,  43.0, 43.25,  43.5, 43.75,  44.0, 44.25,
            44.5, 44.75,  45.0, 45.25,  45.5, 45.75,  46.0, 46.25,  46.5, 46.75,
            47.0],
          dtype='float64', name='lat'))

  • lon
    PandasIndex

    PandasIndex(Index([   -130.0, -129.6875,  -129.375, -129.0625,   -128.75, -128.4375,
            -128.125, -127.8125,    -127.5, -127.1875,  -126.875, -126.5625,
             -126.25, -125.9375,  -125.625, -125.3125,    -125.0, -124.6875,
            -124.375, -124.0625,   -123.75, -123.4375,  -123.125, -122.8125,
              -122.5, -122.1875,  -121.875, -121.5625,   -121.25, -120.9375,
            -120.625, -120.3125,    -120.0, -119.6875,  -119.375, -119.0625,
             -118.75, -118.4375,  -118.125, -117.8125,    -117.5, -117.1875,
            -116.875, -116.5625,   -116.25, -115.9375,  -115.625, -115.3125,
              -115.0, -114.6875,  -114.375, -114.0625,   -113.75, -113.4375,
            -113.125, -112.8125,    -112.5, -112.1875,  -111.875, -111.5625,
             -111.25, -110.9375,  -110.625, -110.3125,    -110.0],
          dtype='float64', name='lon'))

• Attributes: (3)

  long_name :

  Dry mixing ratio of species NO2

  units :

  mol mol-1 dry

  averaging_method :

  time-averaged

In [ ]: