---
jupytext:
  text_representation:
    extension: .md
    format_name: myst
    format_version: 0.13
    jupytext_version: 1.16.4
---
# Quickstart

First install qubed with `pip install qubed`. Now, let's dive in with a real world dataset from the [Climate DT](https://destine.ecmwf.int/climate-change-adaptation-digital-twin-climate-dt/). We'll pull a prebuild qube from github and render it in it's default HTML representation.

```{code-cell} python3
import requests
from qubed import Qube
climate_dt = Qube.from_json(requests.get("https://github.com/ecmwf/qubed/raw/refs/heads/main/tests/example_qubes/climate_dt.json").json())
climate_dt.html(depth=1)
```

Click the arrows to expand and drill down deeper into the data.

Fundamentally a Qube represents a set identifiers which are a set of key value pairs, here's the one leaf in the Climate DT dataset:

```{code-cell} python3
next(climate_dt.leaves())
```

We can look at the set of values each key can take:
```{code-cell} python3
axes = climate_dt.axes()
for key, values in axes.items():
    print(f"{key} : {list(sorted(values))[:10]}")
```

This dataset isn't dense, you can't choose any combination of the above key values pairs, but it does contain many dense datacubes. Hence it makes sense to store and process the set as a tree of dense datacubes, which is what a Qube. For a sense of scale, this dataset contains about 8 million distinct datasets but only contains a few hundred unique nodes.

```{code-cell} python3
import objsize
print(f"""
Distinct datasets: {climate_dt.n_leaves}
Number of nodes in the tree: {climate_dt.n_nodes}
Number of dense datacubes within this qube: {len(list(climate_dt.datacubes()))}
In memory size according to objsize: {objsize.get_deep_size(climate_dt) / 2**20:.0f} MB
""")
```

## Building your own Qubes

You can do it from nested dictionaries with keys in the form "{key=value}":

```{code-cell} python3
from qubed import Qube

q1 = Qube.from_dict({
    "class=od" : {
        "expver=0001": {"param=1":{}, "param=2":{}},
        "expver=0002": {"param=1":{}, "param=2":{}},
    },
    "class=rd" : {
        "expver=0001": {"param=1":{}, "param=2":{}, "param=3":{}},
        "expver=0002": {"param=1":{}, "param=2":{}},
    },
})
print(f"{q1.n_leaves = }, {q1.n_nodes = }")
q1
```

If someone sends you a printed qube you can convert that back to a Qube too:

```{code-cell} python3
q2 = Qube.from_tree("""
    root, frequency=6:00:00
    ├── levtype=pl, param=t, levelist=850, threshold=-2/-4/-8/2/4/8
    └── levtype=sfc
        ├── param=10u/10v, threshold=10/15
        ├── param=2t, threshold=273.15
        └── param=tp, threshold=0.1/1/10/100/20/25/5/50
""")
q2
```
We would not recommend trying to write this representation by hand though.

Finally, quite a flexible approach is to take the union of a series of dense datacubes:

```{code-cell} python3
q3 = Qube.from_datacube(
    dict(
        param="10u/10v/2d/2t/cp/msl/skt/sp/tcw/tp".split("/"),
        threshold="*",
        levtype="sfc",
        frequency="6:00:00",
    )
) | Qube.from_datacube(
    dict(
        param="q/t/u/v/w/z".split("/"),
        threshold="*",
        levtype="pl",
        level="50/100/150/200/250/300/400/500/600/700/850".split("/"),
        frequency="6:00:00",
    )
)
q3
```

## Operations on Qubes

Going back to that first qube:
```{code-cell} python3
q1
```

We can compress it:

```{code-cell} python3
cq = q1.compress()
assert cq.n_leaves == q1.n_leaves
print(f"{cq.n_leaves = }, {cq.n_nodes = }")
cq
```

With the HTML representation you can click on the leaves to expand them. You can copy a path representation of a node to the clipboard by alt/option/⌥ clicking on it. You can then extract that node in code using `[]`:

```{code-cell} python3
cq["class=rd,expver=0001"]
```

Select a subtree:

```{code-cell} python3
cq["class", "od"]["expver", "0001"]
```

Intersect with a dense datacube:

```{code-cell} python3
dq = Qube.from_datacube({
    "class": ["od", "rd", "cd"],
    "expver": ["0001", "0002", "0003"],
    "param": "2",
})

(cq & dq).print()
```


## Iteration

Iterate over the leaves:

```{code-cell} python3
for i, identifier in enumerate(cq.leaves()):
    print(identifier)
    if i > 10:
        print("...")
        break
```

Or if you can it's more efficient to iterate over the datacubes:

```{code-cell} python3
list(cq.datacubes())
```

## Selection
Select a subset of the tree:

```{code-cell} python3
climate_dt.select({
    "activity": "scenariomip"
}).html(depth=1)
```

Use `.span("key")` to get the set of possibles values for a key, note this includes anywhere this key appears in the tree.

```{code-cell} python3
climate_dt.span("activity")
```

Use `.axes()` to get the span of every key in one go.

```{code-cell} python3
axes = climate_dt.axes()
for key, values in axes.items():
    print(f"{key} : {list(values)[:10]}")
```


## Set Operations

The union/intersection/difference of two dense datacubes is not itself dense.

```{code-cell} python3
A = Qube.from_dict({"a=1/2/3" : {"b=i/j/k" : {}},})
B = Qube.from_dict({"a=2/3/4" : {"b=j/k/l" : {}},})

A.print(), B.print();
```

Union:

```{code-cell} python3
(A | B).print();
```

Intersection:

```{code-cell} python3
(A & B).print();
```

Difference:

```{code-cell} python3
(A - B).print();
```

Symmetric Difference:

```{code-cell} python3
(A ^ B).print();
```

## Transformations

`q.transform` takes a python function from one node to one or more nodes and uses this to build a new tree. This can be used for simple operations on the key or values but also to split or remove nodes. Note that you can't use it to merge nodes beause it's only allowed to see one node at a time.

```{code-cell} python3
def capitalize(node): return node.replace(key = node.key.capitalize())
climate_dt.transform(capitalize).html(depth=1)
```

## Save to disk

There is currently a very simple JSON serialisation format. More compact binary serialisations are planned.
```{code-cell} python3
json = climate_dt.to_json()
Qube.from_json(json) == climate_dt
```

## Advanced Selection

There is currently partial support for different datatypes in addition to strings. Here we can convert datatypes by key to ints and timedeltas and then use functions as filters in select.

```{code-cell} python3
from datetime import timedelta, datetime
def to_timedelta(t):
    dt = datetime.strptime(t, "%H:%M:%S")
    return timedelta(hours=dt.hour, minutes=dt.minute, seconds=dt.second)

q = Qube.from_tree("""
root, frequency=6:00:00
├── levtype=pl, levelist=850, threshold=-2/-4/-8/2/4/8
└── levtype=sfc
    ├── param=10u/10v, threshold=10/15
    ├── param=2t, threshold=273.15
    └── param=tp, threshold=0.1/1/10/100/20/25/5/50
""").convert_dtypes({
    "threshold": float,
    "levelist": int,
    "frequency": to_timedelta,
})

r = q.select({
        "threshold": lambda t: t > 5,
        "frequency": lambda dt: dt > timedelta(hours = 2),
})

r
```