Create README.md
Signed-off-by: David Rotermund <54365609+davrot@users.noreply.github.com>
This commit is contained in:
David Rotermund
GitHub
parent
f73ebeae72
commit
c6d2666db5
1 changed files with 650 additions and 0 deletions
650
python_basics/dataclass/README.md
Normal file
650
python_basics/dataclass/README.md
Normal file
|
|
@ -0,0 +1,650 @@
|
|||
# Data structures: [dataclass](https://docs.python.org/3/library/dataclasses.html)
|
||||
{:.no_toc}
|
||||
|
||||
<nav markdown="1" class="toc-class">
|
||||
* TOC
|
||||
{:toc}
|
||||
</nav>
|
||||
|
||||
## The goal
|
||||
|
||||
There is a new build-in [dataclass](https://docs.python.org/3/library/dataclasses.html) class which is highly interesting for data scientists. Obviously it is a class for storing your data. Who would have guessed...
|
||||
|
||||
Questions to [David Rotermund](mailto:davrot@uni-bremen.de)
|
||||
|
||||
**Type annotations required!!!**
|
||||
|
||||
This is the first construct in Python that requires type annotation.
|
||||
|
||||
If we do this:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClass:
|
||||
a
|
||||
b
|
||||
```
|
||||
|
||||
We get this nice error:
|
||||
|
||||
|
||||
|
||||
|
||||
With type annotations:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClass:
|
||||
a: int
|
||||
b: str
|
||||
```
|
||||
|
||||
No error:
|
||||
|
||||
|
||||
|
||||
|
||||
## What is a dataclass?
|
||||
|
||||
[@dataclass](https://docs.python.org/3/library/dataclasses.html) is a decorator that tells Python that this class is a dataclass. A dataclass is a class with different properties compared to a normal class.
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
sample_rate_in_hz: float
|
||||
dt: float
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 1000, 1 / 1000)
|
||||
print(data_1)
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```python
|
||||
TestClassA(name='Dataset A', number_of_electrodes=100, sample_rate_in_hz=1000, dt=0.001)
|
||||
```
|
||||
|
||||
## Default values
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
dt: float
|
||||
sample_rate_in_hz: float = 1000.0
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 1 / 1000)
|
||||
print(data_1)
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```python
|
||||
TestClassA(name='Dataset A', number_of_electrodes=100, dt=0.001, sample_rate_in_hz=1000.0)
|
||||
```
|
||||
|
||||
An alternative is to use field with the default argument:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
dt: float
|
||||
sample_rate_in_hz: float = field(default=1000.0)
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 1 / 1000)
|
||||
print(data_1)
|
||||
```
|
||||
|
||||
|
||||
## Default factory
|
||||
|
||||
We can use the field's default_factory to put suitable generic default into attributes. default and default_factory can not used together.
|
||||
Why should we use a default_factory? Well, please see the problem with [mutable](https://docs.python.org/3/glossary.html#term-mutable) objects in the [official Python documentation](https://docs.python.org/3/tutorial/classes.html#class-and-instance-variables).
|
||||
Or in other words: Using = [ ] as default will cause you pain.
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str = field(default_factory=str)
|
||||
number_of_electrodes: int = field(default_factory=int)
|
||||
dt: float = field(default_factory=float)
|
||||
sample_rate_in_hz: float = field(default_factory=float)
|
||||
|
||||
|
||||
data_1 = TestClassA()
|
||||
print(data_1)
|
||||
```
|
||||
|
||||
## Keyword only attributes (Python >= 3.10)
|
||||
|
||||
We can mark attributes as key word only (kw_only=true). Normally we would need them to put at the end of the definition of attributes. However, with this allows us to mix it in between:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int = field(kw_only=True, default=42)
|
||||
dt: float = field(init=False)
|
||||
sample_rate_in_hz: float = 1000.0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.dt = 1.0 / self.sample_rate_in_hz
|
||||
|
||||
def __str__(self) -> str:
|
||||
output: str = (
|
||||
f"Name: {self.name}"
|
||||
"\n"
|
||||
f"Number of electrodes: {self.number_of_electrodes}"
|
||||
"\n"
|
||||
f"dt: {self.dt:.4f}s"
|
||||
"\n"
|
||||
f"Sample Rate: {self.sample_rate_in_hz:.2f}Hz"
|
||||
)
|
||||
return output
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 500)
|
||||
print(data_1)
|
||||
print("")
|
||||
|
||||
data_2 = TestClassA("Dataset B", 500, number_of_electrodes=33)
|
||||
print(data_2)
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```python
|
||||
Name: Dataset A
|
||||
Number of electrodes: 42
|
||||
dt: 0.0020s
|
||||
Sample Rate: 500.00Hz
|
||||
|
||||
Name: Dataset B
|
||||
Number of electrodes: 33
|
||||
dt: 0.0020s
|
||||
Sample Rate: 500.00Hz
|
||||
```
|
||||
|
||||
## def \_\_post_init\_\_(self):
|
||||
|
||||
We can do operations after the init:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
dt: float = field(init=False)
|
||||
sample_rate_in_hz: float = 1000.0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.dt = 1.0 / self.sample_rate_in_hz
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 500)
|
||||
print(data_1)
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```python
|
||||
TestClassA(name='Dataset A', number_of_electrodes=100, dt=0.002, sample_rate_in_hz=500)
|
||||
```
|
||||
|
||||
## def \_\_str\_\_(self):
|
||||
|
||||
Make the print output nicer:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
dt: float = field(init=False)
|
||||
sample_rate_in_hz: float = 1000.0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.dt = 1.0 / self.sample_rate_in_hz
|
||||
|
||||
def __str__(self) -> str:
|
||||
output: str = (
|
||||
f"Name: {self.name}"
|
||||
"\n"
|
||||
f"Number of electrodes: {self.number_of_electrodes}"
|
||||
"\n"
|
||||
f"dt: {self.dt:.4f}s"
|
||||
"\n"
|
||||
f"Sample Rate: {self.sample_rate_in_hz:.2f}Hz"
|
||||
)
|
||||
return output
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 500)
|
||||
print(data_1)
|
||||
```
|
||||
|
||||
|
||||
|
||||
Output
|
||||
```python
|
||||
Name: Dataset A
|
||||
Number of electrodes: 100
|
||||
dt: 0.0020s
|
||||
Sample Rate: 500.00Hz
|
||||
```
|
||||
|
||||
## Read Only data
|
||||
|
||||
We can protect the data from being modified later. Note: If we need to modify data in e.g. the \_\_post\_init\_\_ function then we need to use object.\_\_setattr\_\_.
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
dt: float = field(init=False)
|
||||
sample_rate_in_hz: float = 1000.0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "dt", 1.0 / self.sample_rate_in_hz)
|
||||
|
||||
def __str__(self) -> str:
|
||||
output: str = (
|
||||
f"Name: {self.name}"
|
||||
"\n"
|
||||
f"Number of electrodes: {self.number_of_electrodes}"
|
||||
"\n"
|
||||
f"dt: {self.dt:.4f}s"
|
||||
"\n"
|
||||
f"Sample Rate: {self.sample_rate_in_hz:.2f}Hz"
|
||||
)
|
||||
return output
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 500)
|
||||
data_1.name = "New Name" # -> FrozenInstanceError: cannot assign to field 'name'
|
||||
```
|
||||
|
||||
## Inheritance
|
||||
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class BasicDataset:
|
||||
x: int = 1
|
||||
y: int = 2
|
||||
|
||||
|
||||
@dataclass
|
||||
class NewDataSet(BasicDataset):
|
||||
a: int = 3
|
||||
x: int = 4
|
||||
|
||||
|
||||
data_1 = BasicDataset()
|
||||
print(data_1)
|
||||
data_2 = NewDataSet()
|
||||
print(data_2)
|
||||
```
|
||||
|
||||
Output:
|
||||
```python
|
||||
BasicDataset(x=1, y=2)
|
||||
NewDataSet(x=4, y=2, a=3)
|
||||
```
|
||||
|
||||
## Why should want we to use a data class?
|
||||
|
||||
### Comparing datasets
|
||||
|
||||
We can compare datasets now
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class MyDataset:
|
||||
x: int
|
||||
y: int
|
||||
|
||||
|
||||
data_1a = MyDataset(x=1, y=1)
|
||||
data_1b = MyDataset(x=1, y=1)
|
||||
print(data_1a == data_1b)
|
||||
data_2 = MyDataset(x=1, y=2)
|
||||
print(data_1a == data_2)
|
||||
```
|
||||
|
||||
Output:
|
||||
```python
|
||||
True
|
||||
False
|
||||
```
|
||||
|
||||
We can remove attributes from the comparison
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class MyDataset:
|
||||
x: int
|
||||
y: int = field(compare=False)
|
||||
|
||||
|
||||
data_1a = MyDataset(x=1, y=1)
|
||||
data_1b = MyDataset(x=1, y=1)
|
||||
print(data_1a == data_1b)
|
||||
data_2 = MyDataset(x=1, y=2)
|
||||
print(data_1a == data_2)
|
||||
```
|
||||
|
||||
Output:
|
||||
```python
|
||||
True
|
||||
True
|
||||
```
|
||||
|
||||
### Sorting datasets
|
||||
|
||||
We can add a custom sort_index attribute. Which we can also hide with [repr=False](https://docs.python.org/3/library/dataclasses.html#dataclasses.field):
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(order=True)
|
||||
class MyDataset:
|
||||
sort_index: int = field(init=False, repr=False)
|
||||
x: int
|
||||
y: int
|
||||
|
||||
def __post_init__(self):
|
||||
self.sort_index = self.x + self.y * 10
|
||||
|
||||
|
||||
data_0 = MyDataset(x=2, y=2)
|
||||
data_1 = MyDataset(x=1, y=1)
|
||||
data_2 = MyDataset(x=1, y=2)
|
||||
data_3 = MyDataset(x=1, y=2)
|
||||
|
||||
print([data_0, data_1, data_2, data_3])
|
||||
print("")
|
||||
print(sorted([data_0, data_1, data_2, data_3]))
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```python
|
||||
[MyDataset(x=2, y=2), MyDataset(x=1, y=1), MyDataset(x=1, y=2), MyDataset(x=1, y=2)]
|
||||
|
||||
[MyDataset(x=1, y=1), MyDataset(x=1, y=2), MyDataset(x=1, y=2), MyDataset(x=2, y=2)]
|
||||
```
|
||||
|
||||
## Slots (Python >= 3.10)
|
||||
|
||||
{: .topic-optional}
|
||||
This is an optional topic!
|
||||
|
||||
|
||||
What? Slots?
|
||||
|
||||
[3.3.2.4. \_\_slots\_\_](https://docs.python.org/3/reference/datamodel.html#slots)
|
||||
> The space saved over using \_\_dict\_\_ can be significant. Attribute lookup speed can be significantly improved as well.
|
||||
|
||||
Normally:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass
|
||||
class MyDataset:
|
||||
x: int
|
||||
y: int
|
||||
|
||||
|
||||
data_0 = MyDataset(x=2, y=2)
|
||||
print(data_0.__dict__)
|
||||
data_0.a = 1
|
||||
print(data_0.__dict__)
|
||||
```
|
||||
|
||||
Output:
|
||||
```python
|
||||
{'x': 2, 'y': 2}
|
||||
{'x': 2, 'y': 2, 'a': 1}
|
||||
```
|
||||
|
||||
With slots=True:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class MyDataset:
|
||||
x: int
|
||||
y: int
|
||||
|
||||
|
||||
data_0 = MyDataset(x=2, y=2)
|
||||
print(data_0.__dict__) # -> AttributeError: 'MyDataset' object has no attribute '__dict__'
|
||||
data_0.a = 1 # -> AttributeError: 'MyDataset' object has no attribute 'a'
|
||||
```
|
||||
|
||||
## Convert to tuple / dictionary
|
||||
We can easily convert a dataclass object and convert it into a tuple or dictionary:
|
||||
|
||||
```python:
|
||||
from dataclasses import dataclass, astuple, asdict
|
||||
|
||||
|
||||
@dataclass
|
||||
class MyDataset:
|
||||
x: int
|
||||
y: int
|
||||
|
||||
|
||||
data_0 = MyDataset(x=2, y=2)
|
||||
data_tuple = astuple(data_0)
|
||||
data_dict = asdict(data_0)
|
||||
print(data_tuple)
|
||||
print(data_dict)
|
||||
```
|
||||
|
||||
Output:
|
||||
```python
|
||||
(2, 2)
|
||||
{'x': 2, 'y': 2}
|
||||
```
|
||||
|
||||
## dataclasses_json
|
||||
|
||||
If this third party package is missing:
|
||||
|
||||
```shell
|
||||
pip install dataclasses-json
|
||||
```
|
||||
|
||||
### JSON
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
import dataclasses_json
|
||||
|
||||
|
||||
@dataclasses_json.dataclass_json
|
||||
@dataclass(frozen=True)
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
dt: float = field(init=False)
|
||||
sample_rate_in_hz: float = 1000.0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "dt", 1.0 / self.sample_rate_in_hz)
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 500)
|
||||
print(data_1)
|
||||
print("")
|
||||
|
||||
print("as JSON:")
|
||||
string_json = data_1.to_json()
|
||||
print(string_json)
|
||||
|
||||
print("")
|
||||
|
||||
data_2 = TestClassA.from_json(string_json)
|
||||
print(data_2)
|
||||
```
|
||||
|
||||
Output:
|
||||
|
||||
```python
|
||||
TestClassA(name='Dataset A', number_of_electrodes=100, dt=0.002, sample_rate_in_hz=500)
|
||||
|
||||
as JSON:
|
||||
{"name": "Dataset A", "number_of_electrodes": 100, "dt": 0.002, "sample_rate_in_hz": 500}
|
||||
|
||||
TestClassA(name='Dataset A', number_of_electrodes=100, dt=0.002, sample_rate_in_hz=500)
|
||||
```
|
||||
|
||||
|
||||
### Dict
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass, field
|
||||
import dataclasses_json
|
||||
|
||||
|
||||
@dataclasses_json.dataclass_json
|
||||
@dataclass(frozen=True)
|
||||
class TestClassA:
|
||||
name: str
|
||||
number_of_electrodes: int
|
||||
dt: float = field(init=False)
|
||||
sample_rate_in_hz: float = 1000.0
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
object.__setattr__(self, "dt", 1.0 / self.sample_rate_in_hz)
|
||||
|
||||
|
||||
data_1 = TestClassA("Dataset A", 100, 500)
|
||||
print(data_1)
|
||||
print("")
|
||||
|
||||
print("as dict:")
|
||||
string_dict = data_1.to_dict()
|
||||
print(string_dict)
|
||||
|
||||
print("")
|
||||
|
||||
data_2 = TestClassA.from_dict(string_dict)
|
||||
print(data_2)
|
||||
```
|
||||
|
||||
Output:
|
||||
```python
|
||||
TestClassA(name='Dataset A', number_of_electrodes=100, dt=0.002, sample_rate_in_hz=500)
|
||||
|
||||
as dict:
|
||||
{'name': 'Dataset A', 'number_of_electrodes': 100, 'dt': 0.002, 'sample_rate_in_hz': 500}
|
||||
|
||||
TestClassA(name='Dataset A', number_of_electrodes=100, dt=0.002, sample_rate_in_hz=500)
|
||||
```
|
||||
|
||||
### Pickle (deals with numpy ndarray)
|
||||
|
||||
While JSON doesn't like numpy ndarrays, pickle has no problem with it. But we loose the human readability. An example:
|
||||
|
||||
```python
|
||||
from dataclasses import dataclass
|
||||
import numpy as np
|
||||
import pickle
|
||||
|
||||
|
||||
@dataclass
|
||||
class TestClassA:
|
||||
name: str
|
||||
measured_data: np.ndarray
|
||||
|
||||
|
||||
rng = np.random.default_rng()
|
||||
data_1 = TestClassA(name="Recording X", measured_data=rng.random((3, 6)))
|
||||
print(data_1)
|
||||
print("")
|
||||
|
||||
|
||||
print("as pickle string:")
|
||||
string_pickle = pickle.dumps(data_1)
|
||||
print(string_pickle)
|
||||
|
||||
print("")
|
||||
|
||||
data_2 = pickle.loads(string_pickle)
|
||||
print(data_2)
|
||||
```
|
||||
|
||||
Output
|
||||
```python
|
||||
TestClassA(name='Recording X', measured_data=array([[0.03041124, 0.90241323, 0.06146134, 0.0207697 , 0.03924572,
|
||||
0.62343829],
|
||||
[0.03930966, 0.34830424, 0.53869473, 0.76964259, 0.64897337,
|
||||
0.76441662],
|
||||
[0.40438748, 0.95079476, 0.44350839, 0.17806159, 0.31114876,
|
||||
0.59675174]]))
|
||||
|
||||
as pickle string:
|
||||
b'\x80\x04\x95a\x01\x00\x00\x00\x00\x00\x00\x8c\x08__main__\x94\x8c\nTestClassA\x94\x93\x94)\x81\x94}\x94(\x8c\x04name\x94\x8c\x0bRecording X\x94\x8c\rmeasured_data\x94\x8c\x15numpy.core.multiarray\x94\x8c\x0c_reconstruct\x94\x93\x94\x8c\x05numpy\x94\x8c\x07ndarray\x94\x93\x94K\x00\x85\x94C\x01b\x94\x87\x94R\x94(K\x01K\x03K\x06\x86\x94h\x0b\x8c\x05dtype\x94\x93\x94\x8c\x02f8\x94\x89\x88\x87\x94R\x94(K\x03\x8c\x01<\x94NNNJ\xff\xff\xff\xffJ\xff\xff\xff\xffK\x00t\x94b\x89C\x90\xe0\xd1r\xb1\x1f$\x9f?\xbe\xa8\xb8\xb5\x91\xe0\xec?\xc0i\xf35\xdcw\xaf?\x00NT)\xa7D\x95?\xe0\xca\x9e\xdc\x03\x18\xa4?\xc0\x12&\xdb4\xf3\xe3?\xe0.\xd2Xe \xa4?\x16\xb4\xbc\xde\x9dJ\xd6?\x95\x00\x03\xbc\xfc<\xe1?5\xb3U\x80\xe9\xa0\xe8?-\r\x8d\xccc\xc4\xe4?4\x1e8\xd7\x19v\xe8?\\c\xb3\t|\xe1\xd9?\xcf\xaf\xbb#\xe9l\xee?R/5\x07qb\xdc?\xecr\xdd\xe3\xb8\xca\xc6?\x10\x12R~\xdc\xe9\xd3?\xcfz;\x1d\x97\x18\xe3?\x94t\x94bub.'
|
||||
|
||||
TestClassA(name='Recording X', measured_data=array([[0.03041124, 0.90241323, 0.06146134, 0.0207697 , 0.03924572,
|
||||
0.62343829],
|
||||
[0.03930966, 0.34830424, 0.53869473, 0.76964259, 0.64897337,
|
||||
0.76441662],
|
||||
[0.40438748, 0.95079476, 0.44350839, 0.17806159, 0.31114876,
|
||||
0.59675174]]))
|
||||
```
|
||||
Loading…
Reference in a new issue