HyperPyYAML

KO | EN

gitlite — search

#react #typescript #golang #go #android #docker #nodejs #macos #java #windows #kubernetes #awesome-list #pytorch #open-source

(★ 80)

Extensions to YAML syntax for better python interaction

파일 탐색기

최종 버전 다운로드 (.zip)

- - pythonapp.yml
- __init__.py
- core.py
- test_hyperyaml.py
.gitignore
LICENSE
README.md
setup.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Introduction
------------

A crucial element of systems for data-analysis is laying out all the
hyperparameters of that system so they can be easily examined and modified.
We add a few useful extensions to a popular human-readable data-serialization
language known as YAML (YAML Ain't Markup Language). This provides support
for a rather expansive idea of what constitutes a hyperparameter, and cleans
up python files for data analysis to just the bare algorithm.

### Table of Contents
* [YAML basics](#yaml-basics)
* [HyperPyYAML](#hyperpyyaml)
    * [Objects](#objects)
    * [Aliases](#aliases)
    * [Tuples](#tuples)
* [How to use HyperPyYAML](#how-to-use-hyperpyyaml)
* [Conclusion](#conclusion)

### Security note
Loading HyperPyYAML allows arbitrary code execution. 
This is a feature: HyperPyYAML allows you to construct *anything* and *everything*
you need in your experiment.
However, take care to verify any untrusted recipes' YAML files just as you would verify the Python code.

YAML basics
-----------

YAML is a data-serialization language, similar to JSON, and it supports
three basic types of nodes: scalar, sequential, and mapping. PyYAML naturally
converts sequential nodes to python lists and mapping nodes to python dicts.

Scalar nodes can take one of the following forms:

```yaml
string: abcd  # No quotes needed
integer: 1
float: 1.3
bool: True
none: null
```

Note that we've used a simple mapping to demonstrate the scalar nodes. A mapping
is a set of `key: value` pairs, defined so that the key can be used to easily
retrieve the corresponding value. In addition to the format above, mappings
can also be specified in a similar manner to JSON:

```yaml
{foo: 1, bar: 2.5, baz: "abc"}
```

Sequences, or lists of items, can also be specified in two ways:

```yaml
- foo
- bar
- baz
```

or

```yaml
[foo, bar, baz]
```

Note that when not using the inline version, YAML uses whitespace to denote
nested items:

```yaml
foo:
    a: 1
    b: 2
bar:
    - c
    - d
```

YAML has a few more advanced features (such as
[aliases](https://pyyaml.org/wiki/PyYAMLDocumentation#aliases) and
[merge keys](https://yaml.org/type/merge.html)) that you may want to explore
on your own. We will briefly discuss one here since it is relevant for our
extensions: [YAML tags](https://pyyaml.org/wiki/PyYAMLDocumentation#tags).

Tags are added with a `!` prefix, and they specify the type of the node. This
allows types beyond the simple types listed above to be used. PyYAML supports a
few additional types, such as:

```yaml
!!set                           # set
!!timestamp                     # datetime.datetime
!!python/tuple                  # tuple
!!python/complex                # complex
!!python/name:module.name       # A class or function
!!python/module:package.module  # A module
!!python/object/new:module.cls  # An instance of a class
```

These can all be quite useful, however we found that this system was a bit
cumbersome, especially with the frequency with which we were using them. So
we decided to implement some shortcuts for these features, which we are
calling "HyperPyYAML".

HyperPyYAML
-----------

We make several extensions to yaml including easier object creation, nicer
aliases, and tuples.

### Objects

Our first extension is to simplify the structure for specifying an instance,
module, class, or function. As an example:

```yaml
model: !new:collections.Counter
```

This tag, prefixed with `!new:`, constructs an instance of the specified class.
If the node is a mapping node, all the items are passed as keyword arguments
to the class when the instance is created. A list can similarly be used to
pass positional arguments. See the following examples:

```yaml
foo: !new:collections.Counter
  - abracadabra
bar: !new: collections.Counter
  a: 2
  b: 1
  c: 5
```

We also simplify the interface for specifying a function or class or other
static Python entity:

```yaml
add: !name:operator.add
```

This code stores the `add` function. It can later be used in the usual way:

```python
>>> loaded_yaml = load_hyperpyyaml("add: !name:operator.add")
>>> loaded_yaml["add"](2, 4)
6
```

### Aliases

Another extension is a nicer alias system that supports things like
string interpolation. We've added a tag written `!ref` that
takes keys in angle brackets, and searches for them inside the yaml
file itself. As an example:

```yaml
folder1: abc/def
folder2: ghi/jkl
folder3: !ref <folder1>/<folder2>

foo: 1024
bar: 512
baz: !ref <foo> // <bar> + 1
```

This allows us to change some values and automatically change the
dependent values accordingly.
You can also refer to other references, and to sub-nodes using brackets.

```yaml
block_index: 1
cnn1:
    out_channels: !ref <block_index> * 64
    kernel_size: (3, 3)
cnn2: 
    out_channels: !ref <cnn1[out_channels]>
    kernel_size: (3, 3)
```

Finally, you can make references to nodes that are objects, not just scalars.

```python
yaml_string = """
foo: !new:collections.Counter
  a: 4
bar: !ref <foo>
baz: !copy <foo>
"""
loaded_yaml = load_hyperpyyaml(yaml_string)
loaded_yaml["foo"].update({"b": 10})
print(loaded_yaml["bar"])
print(loaded_yaml["baz"])
```

This provides the output:
```
Counter({'b': 10, 'a': 4})
Counter({'a': 4})
```

Note that `!ref` makes only a shallow copy, so updating `foo`
also updates `bar`. If you want a deep copy, use the `!copy` tag.

There are some issues (#7 #11) mentioning that `!ref` cannot refer to the return value of `!apply` function. 
Thus we provide another `!applyref` tag to work with `!ref`, which can be used in four ways:

```yaml
# 1. Pass the positional and keyword arguments at the same time. Like `!!python/object/apply:module.function` in pyyaml
c: !applyref:sorted
    _args: 
        - [3, 4, 1, 2]
    _kwargs:
        reverse: False
d: !ref <c>-<c>

# 2. Only pass the keyword arguments
e: !applyref:random.randint
    a: 1
    b: 3
f: !ref <e><e>

# 3. Only pass the positional arguments
g: !applyref:random.randint
    - 1
    - 3
h: !ref <g><g>

# 4. No arguments
i: !applyref:random.random
j: !ref <i><i>
```

Note that `!applyref` cannot return an object, otherwise the `RepresenterError` will be raised.

### Tuples

One last minor extension to the yaml syntax we've made is to implicitly
resolve any string starting with `(` and ending with `)` to a tuple.
This makes the use of YAML more intuitive for Python users.

How to use HyperPyYAML
---------------------

All of the listed extensions are available by loading yaml using the
`load_hyperpyyaml` function. This function returns an object in a similar
manner to pyyaml and other yaml libraries.
Also, `load_hyperpyyaml` takes an optional argument, `overrides`
which allows changes to any of the parameters listed in the YAML.
The following example demonstrates changing the `out_channels`
of the CNN layer:

```python
>>> yaml_string = """
... block_index: 1
... cnn1:
...   out_channels: !ref <block_index> * 64
...   kernel_size: (3, 3)
... cnn2: 
...   out_channels: !ref <cnn1[out_channels]>
...   kernel_size: (3, 3)
... """
>>> overrides = {"block_index": 2}
>>> with open("hyperparameters.yaml") as f:
...    hyperparameters = load_hyperpyyaml(f, overrides)
>>> hyperparameters["block_index"]
2
>>> hyperparameters["cnn2"]["out_channels"]
128
```

Conclusion
----------

We've defined a number of extensions to the YAML syntax, designed to
make it easier to use for hyperparameter specification. Feedback is welcome!

HyperPyYAML

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

HyperPyYAML

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README