Contents¶

ODIN documentation¶

Object Data Mapper for Python

First Steps¶

Quick overview to get up to speed with Odin.

Working with resources¶

Resources are the basic building block of Odin.

Resources: Field reference | Resource Meta options
Loading and Saving: JSON Codec | TOML Codec | YAML Codec | CSV Codec | MessagePack Codec
Mapping: Mapping Classes
Adapters: Adapters
Documenting: Sphinx Extension

See the Odin Examples section for examples on how to use features of Odin.

Extras¶

Integration¶

Project Links¶

Indices and tables¶

Getting started¶

Quick overview to get up to speed with Odin.

Creating resources¶

Odin was designed to solve several common development problems:

Loading data from file or data stream and converting into an object graph
Validating data to ensure it meets the parameters of the program
Mapping/transforming data into a different format or structure
Do all of the above in a way that is easy to maintain, read and test

The goal of this document is to give you an overview of how to use the tools provided by Odin to accomplish the goals set out in the previous paragraph.

Design your resources¶

The resource syntax offers many rich ways of representing your resources. Here’s a quick example:

import odin

class Author(odin.Resource):
    name = odin.StringField()

class Book(odin.Resource):
    title = odin.StringField()
    author = odin.DictAs(Author)
    genre = odin.StringField(max_length=255)
    num_pages = odin.IntegerField(min_value=1)

From this we can see that a resource is a collection of fields, each field maps to a specific data type and accepts options that describe validation rules and how data is handled.

The above example also demonstrates relationships between the two resources. This simple example allows for an other object to be attached to a book. With these simple primitives complex data-structures can be built.

Working with resources¶

With a resource defined the API can be used to work with resources:

# Import the resources we created from our "library" app
>>> from library.resources import Author, Book

# Create an instance of an Author
>>> a = Author(name="Iain M. Banks")

# Create an instance of a Book
>>> b = Book(title="Consider Phlebas", author=a, genre="Space Opera", num_pages=471)
>>> b
<Book: library.resources.Book resource>

# Fields are represented as attributes on the Python object.
>>> b.title
'Consider Phlebas'

# DictAs fields are references to other resources.
>>> b.author.name
'Iain M. Banks'

# Get all the data as a dict
>>> a.to_dict()
{'name': 'Iain M. Banks'}

# Validate that the information entered is valid.
# Create an instance of a Book
>>> b = Book(title="Consider Phlebas", genre="Space Opera", num_pages=471)
>>> b.full_clean()
ValidationError: {'author': [{'name': ['This field cannot be null.']}]}

Annotated resources¶

As of Odin 2.0, resources can now be defined using Python type annotations. This greatly simplifies definition of resource structures while maintaining compatibility with all existing features. Types defined using the existing odin.Resource base class can be used in conjunction with new annotation style resources.

Defining annotated resources¶

The annotated resource syntax offers many rich ways of representing your resources. Here’s a quick example:

import odin

class Author(odin.AnnotatedResource):
    name: str

class Book(odin.AnnotatedResource):
    title: str
    author: Author
    genre: Optional[str] = odin.Options(max_length=255)
    num_pages: int = odin.Options(min_value=1)

Tip

If you prefer the shorthand odin.AResource can be used in place of odin.AnnotatedResource

From this we can see that a resource is a collection of fields, each field maps to a specific data type and accepts options that describe validation rules and how data is handled.

The above example also demonstrates relationships between the two resources. This simple example allows for another object to be attached to a book. With these simple primitives complex data-structures can be built.

Working with resources¶

Just like any other resource the API behaves the same:

# Import the resources we created from our "library" app
>>> from library.resources import Author, Book

# Create an instance of an Author
>>> a = Author(name="Iain M. Banks")

# Create an instance of a Book
>>> b = Book(title="Consider Phlebas", author=a, genre="Space Opera", num_pages=471)
>>> b
<Book: library.resources.Book resource>

# Fields are represented as attributes on the Python object.
>>> b.title
'Consider Phlebas'

# DictAs fields are references to other resources.
>>> b.author.name
'Iain M. Banks'

# Get all the data as a dict
>>> a.to_dict()
{'name': 'Iain M. Banks'}

# Validate that the information entered is valid.
# Create an instance of a Book
>>> b = Book(title="Consider Phlebas", genre="Space Opera", num_pages=471)
>>> b.full_clean()
ValidationError: {'author': [{'name': ['This field cannot be null.']}]}

Abstract Resources¶

Abstract resources no longer require the use of a Meta block then are identified through the use of a keyword argument eg:

import odin

class LibraryBase(odin.AnnotatedResource, abstract=True):
    """
    Common base class for library resources
    """
    name: str

class Library(LibraryBase):
    address: str

The Library resource inherits the _name_ field from LibraryBase but cannot be instantiated directly itself.

Loading and saving data¶

Saving and loaded of resources is handled by codecs for each format.

JSON data¶

JSON data is loaded and saved using the odin.codecs.json_codec module. This module exposes an API that is very similar to Pythons built in json module.

Using the Book and Author resources presented in the Creating resources section:

# Import the resources we created from our "library" app
>>> from library.resources import Author, Book

# Create an instance of an Author
>>> a = Author(name="Iain M. Banks")

# Create an instance of a Book
>>> b = Book(title="Consider Phlebas", author=a, genre="Space Opera", num_pages=471)

# Dump as a JSON encoded string
from odin.codecs import json_codec
>>> json_codec.dumps(b)
'{"genre": "Space Opera", "title": "Consider Phlebas", "num_pages": 471, "$": "library.resources.Book", "author": {
"name": "Iain M. Banks", "$": "library.resources.Author"}}'

Note

Note the $ entries. The $ symbol is used to keep track of an objects type in the serialised state and to aid deserialization of data.

Similarly data can be deserialized back into an object graph using the odin.codecs.json_codec.loads() method.

Other file formats¶

Odin includes codecs for many different file formats including:

Or using each resource as a row:

CSV Codec

[1]	XML is write only

Mapping between resources¶

Defining a mapping¶

Given the following resources:

import odin

class CalendarEvent(odin.Resource):
    name = odin.StringField()
    start_date: odin.DateTimeField()

class CalendarEventFrom(odin.AnnotatedResource):
    name: str
    event_date: datetime.date
    event_hour: int
    event_minute: int

A mapping can be defined to map from a basic Event to the EventFrom:

class CalendarEventToEventFrom(odin.Mapping):
    from_resource = CalendarEvent
    to_resource = CalendarEventFrom

    # Simple mappings (From Field, Transformation, To Field)
    mappings = (
        odin.define(from_field='start_date', to_field='event_date'),
    )

    # Mapping to multiple fields
    @odin.map_field(to_field=('event_hour', 'event_minute'))
    def start_date(self, v):
        # Return a tuple that is mapped to fields defined as to_fields
        return v.hour, v.minute

When a field name is matched on both resources it will be automatically mapped, for other fields mappings need to be specified along with a transformation method or alternatively a method with a map_field() decorator can be used to handle more complex mappings.

Hint

Both simple mappings and map_field() can accept multiple fields as input and output, although care must be taken to ensure that the transformation method accepts and returns the same number of parameters as is defined in the mapping.

Converting between resources¶

Once a mapping has been defined the Resource.convert_to() or Mapping.apply() are used to convert between object, in addition Mapping.update() can be used to update a existing object:

# Create and instance of a CalendarEvent
>>> event = CalendarEvent(
    name='Launch Party',
    start_date=datetime.datetime(2014, 01, 11, 22, 30))

# Convert to CalendarEventFrom
>>> event_from = event.convert_to(CalendarEventFrom)
>>> event_from
<CalendarEventFrom: example.resources.CalendarEventFrom resource>

>>> event.to_dict()
{'event_date': datetime.datetime(2014, 01, 11, 22, 30),
 'event_hour': 22,
 'event_minute': 30,
 'name': 'Launch Party'}

# Or use the mapping definition CalendarEventToEventFrom
>>> event_from = CalendarEventToEventFrom.apply(event)

# Or update an an existing resource
event.name = 'Grand Launch Party'
event.update_existing(event_from)

>>> event.to_dict()
{'event_date': datetime.datetime(2014, 01, 11, 22, 30),
 'event_hour': 22,
 'event_minute': 30,
 'name': 'Grand Launch Party'}

# Similarly the mapping definition can also be used
>>> CalendarEventToEventFrom(event).update(event_from)

Polymorphism and abstract resources¶

Just like Python itself through the use of ABCs Odin supports the concept of Polymorphic data structures.

For this example we are going to use animals as an example, first we define a basic base class and provide the abstract option to indicate this is not an instantiatable object

import odin

class Animal(odin.AnnotatedResource, abstract=True)
    name: str

Next we can defined specific types of animal (very loose types!)

class Dog(Animal):
    pass

class Cat(Animal):
    colour: str

class Bird(Animal):
    flightless: bool
    beak_colour: str

Finally we can define a resource that contains a resource that supports multiple types of Animal:

class Family(odin.AnnotateResource):
    pets: List[Animal]

With this definition we an define a families pets, each of them has a name but also includes other characteristics.

When this data is exported using a codec sub-types of an abstract resource can be specified, eg with JSON:

{
  "$": "Family",
  "pets": [
    {
      "$": "Cat",
      "name": "Gypsy Sun & Rainbows",
      "colour": "Tortoise shell"
    },
    {
      "$": "Bird",
      "name": "Squeek",
      "flightless": false,
      "beak_colour": "Orange"
    }
  ]
}

Through the use of the type_field (“$” by default) the codecs can resolve the correct type and validate that the type is supported.

Tip

The type_field value can be changed via the Meta block.

API Reference¶

Codecs¶

To load and save data from external data files Odin provides a number of codecs to facilitate the process.

Often the API used by the codec mirrors or extends the API provided by the library the Odin extends on. For example for JSON data Odin provides the same load, loads style interface.

CSV Codec¶

Codec for serialising and de-serialising sequences of resources into CSV format.

The CSV CODEC differs from many other CODECS in that data can be streamed, or read using an iterator.

Dict Codec¶

Codec for serialising and de-serialising Dictionary and List objects.

Methods¶

Example usage¶

Loading a resource from a file:

from odin.codecs import dict_codec

my_dict = {}

resource = dict_codec.load(my_dict, MyResource)

JSON Codec¶

Codec for serialising and de-serialising JSON data. Supports both array and objects for mapping into resources or collections of resources.

The JSON codec uses the simplejson module if available and falls back to the json module included in the Python standard library.

Methods¶

Customising Encoding¶

Serialisation of Odin resources is handled by a customised json.Encoder. Additional data types can be appended to the odin.codecs.json_codec.JSON_TYPES dictionary.

Example usage¶

Loading a resource from a file:

from odin.codecs import json_codec

with open('my_resource.json') as f:
    resource = json_codec.load(f)

MessagePack Codec¶

Codec for serialising and de-serialising data in MessagePack format. Supports both array and objects for mapping into resources or collections of resources.

The MessagePack codec uses the msgpack-python module.

Methods¶

Customising Encoding¶

Serialisation of Odin resources is handled by a customised msgpack.Packer. Additional data types can be appended to the odin.codecs.msgpack_codec.TYPE_SERIALIZERS dictionary.

Example usage¶

Loading a resource from a file:

from odin.codecs import msgpack_codec

with open('my_resource.msgp') as f:
    resource = msgpack_codec.load(f)

TOML Codec¶

Codec for serialising and de-serialising TOML data. Supports both array and objects for mapping into resources or collections of resources.

The TOML codec uses the toml module.

Methods¶

Customising Encoding¶

Serialisation of Odin resources is handled by a customised toml.Encoder. Additional data types can be appended to the odin.codecs.toml_codec.TOML_TYPES dictionary.

Example usage¶

Loading a resource from a file:

from odin.codecs import toml_codec

with open('my_resource.toml') as f:
    resource = toml_codec.load(f)

YAML Codec¶

Codec for serialising and de-serialising YAML data. Supports both array and objects for mapping into resources or collections of resources.

The YAML codec uses the yaml module and uses the compiled C versions of the library if available.

Methods¶

Customising Encoding¶

Serialisation of Odin resources is handled by a customised yaml.Dumper. Additional data types can be appended to the odin.codecs.yaml_codec.YAML_TYPES dictionary.

Example usage¶

Loading a resource from a file:

from odin.codecs import yaml_codec

with open('my_resource.yaml') as f:
    resource = yaml_codec.load(f)

XML Codec¶

Codec for serialising and de-serialising XML data. Supports both array and objects for mapping into resources or collections of resources.

Methods¶

Unsupported Fields¶

There is no direct representation for a odin.fields.DictField.

Example usage¶

Loading a resource from a file:

from odin.codecs import xml_codec

with open('my_resource.xml') as f:
    resource = xml_codec.load(f)

Saving a resource to a file:

from odin.codecs import xml_codec

with open('my_resource.xml', 'w') as f:
    xml_codec.dump(f, resource)

contrib packages¶

Odin provides integration with other Python libraries to simplify common development situations. These packages are kept separate from the rest of Odin to only require dependencies on other libraries if you so chose to use this additional functionality.

This code lives in odin/contrib in the Odin distribution.

Generating resource documentation¶

Note

When using document generation Jinja2 is required.

Odin has built in support for generating documentation of resources that have been registered. This is where the various verbose_name, doc_text and doc strings are used generate documentation.

Quick example¶

The default documentation format is reStructuredText, to enable easy integration with Sphinx for producing project documentation.

A basic example:

from odin import doc_gen
import my_project.resources  # Import required resources so they get registered

with file("resources.rst", "w") as fp:
    doc_ren.dump(fp)

The resources.rst file can now be registered into your Sphinx documentation.

Doc-gen API¶

The documentation generation API consists of two methods in the vain of the main Odin API:

odin.doc_gen.dump - Output documentation to a file, requires a file pointer.
odin.doc_gen.dumps - Return the documentation as a string

Both methods take the same optional parameters.

fmt

Format of the output, by default this is RESTRUCTURED_TEXT.

There are not currently any other options available.

exclude

List of resources to exclude when generating documentation.

template_path

Template path to include in template search path, this is used to customise the look of outputted templates.

See Customising output for more information.

Customising output¶

Geographic Values¶

Fields and data types that handle latitude and longitude values.

todo:	This section is in progress

Datatypes¶

latitude¶

A latitude value. A latitude is a value between -90.0 and 90.0.

longitude¶

A longitude value. A longitude is a value between -180.0 and 180.0.

latlng¶

Combination latitude and longitude value.

point¶

A point in cartesian space. This type can be either 2D (on a plain) or 3D (includes a z-axis).

Fields¶

LatitudeField¶

class LatitudeField([min_value=None, max_value=None, **options])

A latitude.

LatitudeField has two extra arguments:

LatitudeField.min_value: The minimum latitude that can be accepted (within the range of a latitude).
LatitudeField.max_value: The maximum latitude that can be accepted (within the range of a latitude).

LongitudeField¶

class LongitudeField([min_value=None, max_value=None, **options])

A longitude.

LongitudeField has two extra arguments:

LongitudeField.min_value: The minimum longitude that can be accepted (within the range of a longitude).
LongitudeField.max_value: The maximum longitude that can be accepted (within the range of a longitude).

LatLngField¶

class LatLngField([**options])

A latlng.

PointField¶

class PointField([**options])

A point.

Currency and Money Values¶

Fields and data types that handle money values.

todo:	This section is in progress

Datatypes¶

Amount¶

Combines an value and a Currency to represent a monetary amount.

Currency¶

Defines a currency and maintains metadata about the currency.

Fields¶

AmountField¶

class AmountField([allowed_currencies=None, min_value=None, max_value=None, **options])

An amount.

AmountField has three extra arguments:

AmountField.allowed_currencies: The currencies that can be accepted by this field, value is enforced Odin’s validation. If None is supplied any currency is acceptable.
AmountField.min_value: The minimum amount that can be accepted.
AmountField.max_value: The maximum amount that can be accepted.

Physical Quantities with Pint¶

Additional resources fields that include a unit.

Note

This contrib module depends on the Pint units library this can be installed with:

pip install pint

Fields¶

FloatField¶

class FloatField([max_length=None, **options])

A float.

FloatField has one extra argument:

Sphinx Extension¶

Odin provides a Sphinx extension for documenting resources. It behaves similarly to the builtin Autodoc tools.

Setup¶

Add the Odin extension into your Sphinx conf.py:

extensions = [
    ...
    'odin.contrib.sphinx',
    ...
]

Usage¶

Extension to sphinx.ext.autodoc to support documenting Odin resources.

This extension supports output in two forms, class doc form (for documenting the python objects) and API form for documenting information about fields for use in an API document.

Usage:

.. autoodin-resource:: namespace.path.to.your.Resource
    :include_virtual:
    :hide_choices:

To select API form use the include_virtual option.

Options¶

The following options are provided by the documenter:

include_virtual virtual fields should be included in docs
include_validators validators should be listed for fields
hide_choices don’t include the list of valid choices for fields

Exceptions¶

Exceptions defined by Odin.

Mapping¶

Resource Mapping API reference. For introductory material, see Mapping between resources.

Mapping Classes¶

A mapping is a utility class that defines how data is mapped between objects, these objects are typically odin.Resource but other Python objects can be supported.

The basics:

Each mapping is a Python class that subclasses odin.Mapping.
Each mapping defines a from_obj and a to_obj
Parameters and decorated methods define rules for mapping between objects.

Defining a Mapping¶

`from_obj` and `to_obj`¶

These attributes specify the source and destination of the mapping operation and must be defined to for a mapping to be considered valid.

Note

For an object to be successfully mapped it’s fields need to be known. This is handled by a FieldResolver instance.

Example:

class AuthorToNewAuthor(odin.Mapping):
    from_obj = Author
    to_obj = NewAuthor

Auto generated mappings¶

When a field of the same name exists on both the from and to objects an automatic mapping is created. If no data transformation or data type conversion is needed no further work is required. The mappings will be automatically created.

You can exclude fields from automatic generation by; including the field in the exclude_fields list() (or tuple()). Fields are also excluded from automatic generation when they are specified as a target by any other mapping rule.

Fields that defined relationships (eg DictAs and ArrayOf) are also handled by the auto generated mapping process, provided that a mapping has been defined previously for the Resource types that these relations refer to. There is one special case however, this is where Resource types match; in this situation the odin.mapping.helpers.NoOpMapper is used to transparently copy the items.

Applying transformations to data¶

Complex data manipulation can be achieved by passing a field value through a mapping function.

The simplest way to apply a transformation action is to use the map_field() decorator.

Example:

class AuthorToNewAuthor(odin.Mapping):
    from_obj = Author
    to_obj = NewAuthor

    @odin.map_field
    def title(self, value):
        return value.title()

In this simple example we are specifying a mapping of the title field that ensures that the title value is title case.

But what about if we want to split a field into multiple outputs or combine multiple values into a single field? Not a problem, transformation actions can accept and return multiple values. These values just need to specified:

class AuthorToNewAuthor(odin.Mapping):
    from_obj = Author
    to_obj = NewAuthor

    @odin.map_field(from_field='name', to_field=('first_name', 'last_name'))
    def split_author_name(self, value):
        first_name, last_name = value.split(' ', 1)
        return first_name, last_name

Conversely we could combine these fields:

class AuthorToNewAuthor(odin.Mapping):
    from_obj = Author
    to_obj = NewAuthor

    @odin.map_field(from_field=('first_name', 'last_name'))
    def name(self, first_name, last_name):
        return "%s %s" % (first_name, last_name)

While this example is extremely simplistic it does demonstrate the flexibility of mapping rules. Not also that a value for the to_field is not specified, the mapping decorators will default to using the method name as the to or from field if it is not specified.

Odin includes several decorators that preform handle different mapping scenarios.

map_field¶

Decorate a mapping class method to mark it as a mapping rule.

from_field: The string name or a tuple of names of the field(s) to map from. The function that is being decorated must accept the same number of parameters as fields specified.
to_field: The string name or tuple of names of the field(s) to map to. The function that is being decorated must return a tuple with the same number of parameters as fields specified.

map_list_field¶

Decorate a mapping class method to mark it as a mapping rule. This decorator works in much the same way as the basic map_field except rather than treating the response as a set of fields it treats it as a list result. This allows you to map list of objects.

from_field: The string name or a tuple of names of the field(s) to map from. The function that is being decorated must accept the same number of parameters as fields specified.
to_field: The string name of tuple of names of the field(s) to map to. The function that is being decorated must return a tuple with the same number of parameters as fields specified.

assign_field¶

This is a special decorator that allows you to generate a value that is assigned to the resulting field.

to_field: The string name or tuple of names of the field(s) to map to. The function that is being decorated must return a tuple with the same number of parameters as fields specified.

Low level mapping¶

The final way to specify a mapping is by generating the actual mapping rules directly. A basic mapping rule is a three part tuple that contains the name of the from field (or tuple of multiple source fields) a transform action or None if there is no transform required and finally the name of the two field (or tuple of multiple destination fields).

Note

The number of input parameters and number of parameters returned by the action methods must much the number defined in the mapping. A odin.exceptions.MappingExecutionError will be raised if an incorrect number of parameters is specified.

A list of mapping rules:

class AuthorToNewAuthor(odin.Mapping):
    from_obj = Author
    to_obj = NewAuthor

    mappings = (
        ('dob', None, 'date_of_birth'),
    )

Tip

Use the define() and assign() methods to simplify the definition of mappings. They provides many sensible defaults.

While the basic mapping only includes and source, action and destination definitions the mappings structure actually supports three additional boolean parameters. These are to_list, bind and skip_if_none.

to_list¶

The two list option is what the map_list_field decorator uses to indicate the the returned object is a list value.

bind¶

For use with action methods defined outside the mapping class, if bind is set to True the mapping instance is passed to the action method as the first parameter.

skip_if_none¶

This flag changes the way that values that are None are handled. If set to True if the from value is None the value will not be supplied to the destination object allowing the destination objects defaulting process to handle the value.

Mapping Instances¶

Mapping instances provide various methods to aid in the mapping process.

Initialisation¶

The init method accepts a source object and an optional context. The context is a defaults to a dict() and allows any value to be stored or supplied to the mapping.

When using the apply class method to map a list of objects the context is used to track the index count.

`convert`¶

Method that starts the mapping process and returns a populated to_obj.

`update`¶

Update an existing object with fields from the provided by the source_obj.

`diff`¶

Compare the field values from the source_obj with a supplied destination and return all the fields that differ.

`loop_idx`¶

A convenience property gives access to the current loop index when converting a list of objects.

`loop_depth`¶

A convenience property that provides the nested loop depth of the current mapping operation.

`in_loop`¶

A convenience property that indicates if the current mapping operation is in a loop.

Mapping Factories¶

When mapping between two objects that are similar eg between a Django model and a resource, or between versions of resources.

There is also the simpler method when only a forward mapping is desired.

Mapping Other Types¶

Mappers can also be used to map other to and from other object types that are not odin Resources.

Tip

There is worked example for Django Models in the integration section.

For mapping rules to be determined the mapper must resolve which fields/attributes are available on a type that can be mapped. This is done using a field resolver that is associated with the type (or sub type) of the object to be mapped.

The field resolver is a class inherited from FieldResolverBase that can resolve what fields are available. This class is them registered with Odins type registry.

Mapping Helpers¶

Odin includes a few helpers to help defining mappings.

Defining Mappings¶

When defining mappings using the shorthand mappings property these methods simplify the definition of mapping rules. They also provide sensible defaults.

Action Helpers¶

Predefined actions for use in mapping definitions.

Special Mappers¶

Resources¶

Resource API reference. For introductory material, see Creating Resources.

Field reference¶

This section contains all the details of the resource fields built into Odin.

Field options¶

The following arguments are available to all field types. All are optional.

verbose_name¶

Field.verbose_name

A human-readable name for the field. If the verbose name isn’t given, Odin will automatically create it using the field’s attribute name, converting underscores to spaces.

verbose_name_plural¶

Field.verbose_name_plural

A human-readable name for the field. If the verbose name isn’t given, Odin will automatically create it using the field’s attribute name, converting underscores to spaces.

name¶

Field.name

Name of the field as it appears in the exported document. If the name isn’t given, Odin will use the field’s attribute name.

null¶

Field.null

If True Odin will raise a validation error if a value is null. Default is False.

choices¶

Field.choices

An iterable (e.g., a list or tuple) of 2-tuples to use as choices for this field. If this is given, the choices are used to validate entries.

Note

Choices are also used by the odin.contrib.doc_gen to generate documentation.

A choices list looks like this:

GENRE_CHOICES = (
    ('sci-fi', 'Science Fiction'),
    ('fantasy', 'Fantasy'),
    ('others', 'Others'),
)

The first element in each tuple is the value that will be used to validate, the second element is used for documentation. For example:

import Odin

class Book(Odin.Resource):
    GENRE_CHOICES = (
        ('sci-fi', 'Science Fiction'),
        ('fantasy', 'Fantasy'),
        ('others', 'Others'),
    )
    title = Odin.StringField()
    genre = Odin.StringField(choices=GENRE_CHOICES)
>>> b = Book(title="Consider Phlebas", genre="sci-fi")
>>> b.genre
'sci-fi'

default¶

Field.default

The default value for the field. This can be a value or a callable object. If callable it will be called every time a new object is created.

doc_text (help_text)¶

Field.doc_text

Doc text is used by the odin.contrib.doc_gen to generate documentation.

Note

help_text will be deprecated in a future release in favor of doc_text.

Also useful for inline documentation even if documentation is not generated.

validators¶

Field.validators

error_messages¶

Field.error_messages

is_attribute¶

Field.is_attribute

use_default_if_not_provided¶

Field.use_default_if_not_provided

Standard fields¶

Simple field types.

StringField¶

class StringField([max_length=None, **options])

A string.

StringField has one extra argument:

StringField.max_length: The maximum length (in characters) of the field. The max_length value is enforced Odin’s validation.

IntegerField¶

class IntegerField([min_value=None, max_value=None, **options])

An integer.

IntegerField has two extra arguments:

IntegerField.min_value: The minimum value of the field. The min_value value is enforced Odin’s validation.
IntegerField.max_value: The maximum value of the field. The max_value value is enforced Odin’s validation.

FloatField¶

class FloatField([**options])

A floating-point number represented in Python by a float instance.

FloatField has two extra arguments:

FloatField.min_value: The minimum value of the field. The min_value value is enforced Odin’s validation.
FloatField.max_value: The maximum value of the field. The max_value value is enforced Odin’s validation.

BooleanField¶

class BooleanField([**options])

A true/false field.

DateField¶

class DateField([**options])

A date field or date encoded in ISO-8601 date string format.

TimeField¶

class TimeField([assume_local=True, **options])

A datetime.time field or time encoded in ISO-8601 time string format.

TimeField has an extra argument:

TimeField.assume_local: This adjusts the behaviour of how a naive time (time objects with no timezone) or time strings with no timezone specified. By default assume_local is True, in this state naive time objects are assumed to be in the current system timezone. Similarly on decoding a time string the output time will be converted to the current system timezone.

NaiveTimeField¶

class NaiveTimeField([ignore_timezone=False, **options])

A datetime.time field or time encoded in ISO-8601 time string format. The naive time field differs from TimeField in the handling of the timezone, a timezone will not be applied if one is not specified.

NaiveTimeField has an extra argument:

NaiveTimeField.ignore_timezone: Ignore any timezone information provided to the field during parsing. Will also actively strip out any timezone information when processing an existing time value.

DateTimeField¶

class DateTimeField([**options])

A datetime.datetime field or date encoded in ISO-8601 datetime string format.

DateTimeField has an extra argument:

DateTimeField.assume_local: This adjusts the behaviour of how a naive time (date time objects with no timezone) or date time strings with no timezone specified. By default assume_local is True, in this state naive datetime objects are assumed to be in the current system timezone. Similarly on decoding a date time string the output datetime will be converted to the current system timezone.

NaiveDateTimeField¶

class NaiveDateTimeField([ignore_timezone=False, **options])

A datetime.datetime field or time encoded in ISO-8601 datetime string format. The naive date time field differs from DateTimeField in the handling of the timezone, a timezone will not be applied if one is not specified.

NaiveDateTimeField has an extra argument:

NaiveDateTimeField.ignore_timezone: Ignore any timezone information provided to the field during parsing. Will also actively strip out any timezone information when processing an existing datetime value.

HttpDateTimeField¶

class HttpDateTimeField([**options])

A datetime field or date encoded in ISO-1123 or HTTP datetime string format.

UUIDField¶

class UUIDField([**options])

A UUID field.

This field supports most accepted values for initializing a UUID except bytes_le.

EnumField¶

class EnumField(enum, [**options])

A enum.Enum field that will convert to and from an enum and it’s native type.

Ensure that the enum value is compatible with the codec being used.

The enum.IntEnum variant is also supported.

Changed in version 1.5.0: Choices can be used with EnumField to specify a subset of options. A sequence of enum values should be used that will be converted to choice tuples by Odin.

PathField¶

class PathField([**options])

A pathlib.Path field that will convert to and from a Path value and a string type.

New in version 2.0.

UrlField¶

class UrlField([**options])

Based on a string field, validates that the supplied value is a valid URL.

EmailField¶

class EmailField([**options])

Based on a string field, validates that the supplied value is a valid email address.

IP Fields¶

class IPv4Field([**options]) class IPv6Field([**options]) class IPv46Field([**options])

Based on a string field, validates that the supplied value is a valid IP address.

Use the IPv46Field to support either a v4 or v6 address.

ArrayField¶

class ArrayField([**options])

An array structure represented in Python by a list instance.

TypedArrayField¶

class TypedArrayField(field, [**options])

An array structure represented in Python by a list instance accepts an additional parameter of another field type that each entry in the array is validated against.

TypedArrayField.field: An instance of an odin field that is used to validate each entry in the array.

TypedDictField¶

class TypedDictField(key_field, value_field, [**options])

A object structure represented in Python by a dict instance accepts additional parameters of both a key and value field type that each item in the dict is validated against.

TypedDictField.key_field: An instance of an odin field that is used to validate each key in the dict; default is StringField.
TypedDictField.value_field: An instance of an odin field that is used to validate each value in the dict; default is StringField.

DictField¶

class DictField([**options])

A dictionary.

Note

The object values in the object are not defined.

Composite fields¶

Odin also defines a set of fields that allow for composition.

DictAs field¶

class DictAs(of[, **options])

A child object. Requires a positional argument: the class that represents the child resource.

Note

A default dict is automatically assigned.

ArrayOf field¶

class ArrayOf(of[, **options])

A child list. Requires a positional argument: the class that represents a list of resources.

Note

A default list is automatically assigned.

DictOf field¶

class DictOf(of[, **options])

A child dict. Requires a positional argument: the class that represents a dict (or hash map) of resources.

Note

A default dict is automatically assigned.

Virtual fields¶

Virtual fields are special fields that can be used to calculate a value or provide a value lookup. Unlike using a property a virtual field is also a treating like field in that it can be mapped or exported.

Note

You can use the

Virtual fields share many of the options of regular fields:

verbose_name
verbose_name_plural
name
doc_text (help_text)

CalculatedField¶

class CalculatedField(expr[, **options])

ConstantField¶

class ConstantField(value[, **options])

A fixed value that remains unchanged.

MultiPartField¶

class MultiPartField(field_names, separator=""[, **options])

A field that combines strings from other resource fields with a joining value.

Resources Instances¶

A resource is a collection of fields and associated meta data that is used to validate data within the fields.

The basics:

Each resource is a Python class that subclasses odin.Resource.
Each field attribute of the resource is defined by a odin.Field subclass.

Quick example¶

This example resource defines a Book, which has a title, genre and num_pages:

import odin

class Book(odin.Resource):
    title = odin.StringField()
    genre = odin.StringField()
    num_pages = odin.IntegerField()

title, genre and num_pages are fields. Each field is specified as a class attribute.

The above Book resource would create a JSON object like this:

{
    "$": "resources.Book",
    "title": "Consider Phlebas",
    "genre": "Space Opera",
    "num_pages": 471
}

Some technical notes:

The $ field is a special field that defines the type of Resource.
The name of the resource, resources.Book, is automatically derived from some resource metadata but can be overridden.

Fields¶

The most important part of a resource – and the only required part of a resource – is the list of fields it defines. Fields are specified by class attributes. Be careful not to choose field names that conflict with the resources API like clean.

Example:

class Author(odin.Resource):
    name = odin.StringField()

class Book(odin.Resource):
    title = odin.StringField()
    authors = odin.ArrayOf(Author)
    genre = odin.StringField()
    num_pages = odin.IntegerField()

Field types¶

Each field in your resource should be an instance of the appropriate Field class. ODIN uses the field class types to determine a few things:

The data type (e.g. Integer, String).
Validation requirements.

Field options¶

Each field takes a certain set of field-specific arguments (documented in the resource field reference).

There’s also a set of common arguments available to all field types. All are optional. They’re fully explained in the reference, but here’s a quick summary of the most often-used ones:

null

If True, the field is allowed to be null. Default is False.

default

The default value for the field. This can be a value or a callable object. If callable it will be called every time a new object is created.

choices

An iterable (e.g., a list or tuple) of 2-tuples to use as choices for this field.

A choices list looks like this:

GENRE_CHOICES = (
    ('sci-fi', 'Science Fiction'),
    ('fantasy', 'Fantasy'),
    ('others', 'Others'),
)

The first element in each tuple is the value that will be stored field or serialised document, the second element is a display value.

Again, these are just short descriptions of the most common field options.

Verbose field names¶

Each field type, except for DictAs and ArrayOf, takes an optional first positional argument – a verbose name. If the verbose name isn’t given, Odin will automatically create it using the field’s attribute name, converting underscores to spaces.

In this example, the verbose name is “person’s first name”:

first_name = odin.StringField("person's first name")

In this example, the verbose name is “first name”:

first_name = odin.StringField()

DictAs and ArrayOf require the first argument to be a resource class, so use the verbose_name keyword argument:

publisher = odin.DictAs(Publisher, verbose_name="the publisher")
authors = odin.ArrayOf(Author, verbose_name="list of authors")

Resource level validation¶

Field validation can be customised at the resource level. This is useful as it allows validation of data using a value from another field for example ensuring a maximum value is greater than a minimum value, or checking that a password and a check password matches.

This is achieved using by defining a method called clean_FIELDNAME which accepts a single value argument, Odin will then use this method during the cleaning process to validate the field. Odin will then use the value that is returned from the clean method, allowing you to apply any customised formatting. If an issue is found with a value then raise a odin.exceptions.ValidationError and the error returned will be applied to validation results.

Example:

class Timing(odin.Resource):
    minimum_delay = odin.IntegerField(min_value=0)
    maximum_delay = odin.IntegerField()

    def clean_maximum_delay(self, value):
        if value < self.minimum_delay:
            raise ValidationError('Maximum delay must be greater than the minimum delay value')
        return value

Important

Ensure that a return value is provided, if no return value is specified the Python default is None and this is the value that Odin will use.

Relationships¶

To really model more complex documents objects and lists need to be able to be combined, Odin offers ways to define these structures, DictAs and ArrayOf fields handle these structures.

DictAs relationships¶

To define a object-as relationship, use odin.DictAs. You use it just like any other Field type by including it as a class attribute of your resource.

DictAs requires a positional argument: the class to which the resource is related.

For example, if a Book resource has a Publisher – that is, a single Publisher publishes a book:

class Publisher(odin.Resource):
    # ...

class Book(odin.Resource):
    publisher = odin.DictAs(Publisher)
    # ...

This would produce a JSON document of:

{
    "$": "resources.Book",
    "title": "Consider Phlebas",
    "publisher": {
        "$": "resources.Publisher",
        "name": "Macmillan"
    }
}

ArrayOf relationships¶

To define a array-of relationship, use odin.ArrayOf. You use it just like any other Field type by including it as a class attribute of your resource.

ArrayOf requires a positional argument: the class to which the resource is related.

For example, if a Book resource has a several Authors – that is, a multiple authors can publish a book:

class Author(odin.Resource):
    # ...

class Book(odin.Resource):
    authors = odin.ArrayOf(Author)
    # ...

This would produce a JSON document of:

{
    "$": "resources.Book",
    "title": "Consider Phlebas",
    "authors": [
        {
            "$": "resources.Author",
            "name": "Iain M. Banks"
        }
    ]
}

Resource inheritance¶

Resource inheritance in Odin works almost identically to the way normal class inheritance works in Python. The only decision you have to make is whether you want the parent resources to be resources in their own right, or if the parents are just holders of common information that will only be visible through the child resources.

Abstract base classes¶

Abstract base classes are useful when you want to put some common information into a number of other resources. You write your base class and put abstract=True in the Meta class. This resource will then not be able to created from a JSON document. Instead, when it is used as a base class for other resources, its fields will be added to those of the child class.

An example:

class CommonBook(odin.Resources):
    title = odin.StringField()

    class Meta:
        abstract = True

class PictureBook(CommonBook):
    photographer = odin.StringField()

The PictureBook resource will have two fields: title and photographer. The CommonBook resource cannot be used as a normal resource, since it is an abstract base class.

todo:	Add details of how to support multiple object types in a list using Abstract resources

Annotated Resources¶

A resource is a collection of fields and associated meta data that is used to validate data within the fields.

The basics:

Each annotated resource is a Python class that subclasses odin.AnnotatedResource.
Each field attribute of the resource is defined type attributes.
Additional field options can be provided using odin.Options.

Quick example¶

This example resource defines a Book, which has a title, genre and num_pages:

import odin

class Book(odin.AnnotatedResource):
    title: str
    genre: str
    num_pages: int
    rrp: float = 24.5

title, genre, num_pages and rrp are fields. Each field is specified as a class attribute, the rrp field has a default value of 24.5.

The above Book resource would create a JSON object like this:

{
    "$": "resources.Book",
    "title": "Consider Phlebas",
    "genre": "Space Opera",
    "num_pages": 471,
    "rrp": 24.5
}

Some technical notes:

The $ field is a special field that defines the type of AnnotatedResource.
The name of the resource, resources.Book, is automatically derived from some resource metadata but can be overridden.

Fields¶

The most important part of a resource – and the only required part of a resource – is the list of fields it defines. Fields are specified by attributes. Be careful not to choose field names that conflict with the resources API like clean.

Example:

class Author(odin.AnnotatedResource):
    name: str

class Book(odin.AnnotatedResource):
    title: str
    authors: List[Author]
    genre: str
    num_pages: int
    rrp: float = 24.5

Field types¶

Each field in your annotated resource must include a type annotation that odin uses to determine the appropriate field type.

Supported Annotations¶

All basic types of field are supported by Odin, str, bool, int, float, dict, list, datetime.datetime, datatime.date, datetime.time, uuid.UUID and pathlib.Path. These all map to the appropriate odin field along with any options.

Odin also includes a number of builtin type aliases for commonly used field types via the odin.types alias odin.types.Email, odin.types.IPv4, odin.types.IPv6, odin.types.IPv46 and odin.types.Url.

Enum types are supported by simply specifying an enum type.

Composite fields are supported using generic type hints, the List/Sequence and Mapping/Dict type specifiers will map to the appropriate fields based on the arguments provided.

Finally use of the Optional typing alias is used to set the null field option.

Field options¶

Options are provided to the appropriate field using the odin.Options class. This will be identified by Odin with the options passed to the resolved field type.

An example:

class Book(odin.AnnotatedResource):
    title: str = odin.Options(min_length=1)

The first value of the Options class is the default value, the Options object cna be left out if only a default value needs to be provided.

To defined a completely custom field type use the field_type option, to pass a field instance that will be used for the field.

Resource inheritance¶

Resource inheritance in Odin works almost identically to the way normal class inheritance works in Python. The only decision you have to make is whether you want the parent resources to be resources in their own right, or if the parents are just holders of common information that will only be visible through the child resources.

Abstract base classes¶

Abstract base classes are useful when you want to put some common information into a number of other resources. You write your base class and put abstract=True in the Meta class. This resource will then not be able to created from a JSON document. Instead, when it is used as a base class for other resources, its fields will be added to those of the child class.

An example:

class CommonBook(odin.AnnotatedResource, abstract=True):
    title: str


class PictureBook(CommonBook):
    photographer: str

The PictureBook resource will have two fields: title and photographer. The CommonBook resource cannot be used as a normal resource, since it is an abstract base class.

Resource Meta options¶

Meta options are flags that can be applied to a resource to affect it’s behaviour or how it is dealt with via Odin tools.

Give your resource metadata by using an inner class Meta, eg:

class Book(odin.Resource):
    class Meta:
        name_space = "library"
        verbose_name_plural = "Books"

    title = odin.StringField()

Resource metadata is “anything that’s not a field”, module_name and human-readable plural names (verbose_name and verbose_name_plural). None are required, and adding class Meta to a resource is completely optional.

Meta Options¶

name

Override the name of a resource. This is the codecs when serialising/de-serialising as a name to represent the resource. The default name is the name of the class used to define the resource.

name_space

The name space is an optional string value that is used to group a set of common resources. Typically a namespace should be in the form of dot-atoms eg: university.library or org.poweredbypenguins. The default is no namespace.

verbose_name

A long version of the name for used when displaying a resource or in generated documentation. The default verbose_name is a name attribute that has been converted to lower case and spaces put before each upper case character eg: LibraryBook -> “library book”

verbose_name_plural

A pluralised version of the verbose_name. The default is to use the verbose name and append an ‘s’ character. In the case of many words this does not work correctly so this attribute allows for the default behaviour to be overridden.

abstract

Marks the current resource as an abstract resource. See the section Abstract base classes for more detail of the abstract attribute. The default value for abstract is False.

doc_group

A grouping for documentation purposes. This is purely optional but is useful for grouping common elements together. The default value for doc_group is None.

type_field

The field used to identify the object type during serialisation/de-serialisation. This defaults to the $ character.

key_field_name

Used by external libraries like baldr for identifying what field is used as the key field to uniquely identify a resource instance (a good example would be an ID field).

key_field_names

Similar to the key_field_name but for defining multi-part keys.

field_name_format

Provide a function that can be used to format field names. Field names are used to identify values when a resource serialised/deserialised.

For example to use camelCase names specify the following option:

from odin.utils import snake_to_camel

class MyResource(Resource):
    class Meta:
        field_name_format = snake_to_camel

field_sorting

Used to customise how fields are sorted (primarily affects the order fields will be exported during serialisation) during inheritance. The default behaviour is to sort fields in the child resource before appending the fields from the parent resource(s).

Settings this option to True will cause field sorting to happen after all of the fields have been attached using the default sort method. The default method sorts the fields by the order they are defined.

Supplying a callable allows for customisation of the field sorting eg sort by name:

def sort_by_name(fields):
    return sorted(fields, key=lambda f: f.name)

class MyResource(Resource):
    class Meta:
        field_sorting = sort_by_name

user_data

Additional data that can be added to metadata. This can be used to provide additional parameters beyond those supported by odin for a custom application use-case.

For example:

class MyResource(Resource):
    class Meta:
        user_data = {
            "custom": "my-custom-value",
        }

Adapters¶

Adapters are wrappers around resources that allow for an alternate view of the data contained in a resource or provide additional functionality that is specific to part of your code base. An adapter can be used just like the plain resource with most methods within Odin, including codecs.

See Adapter Examples

Validators¶

Built in validators¶

These validators are provided with odin.

Traversal¶

Traversal package provides tools for iterating and navigating a resource tree.

TraversalPath¶

A method of defining a location within a data structure, which can then be applied to the datastructure to extract the value.

A TraversalPath can be expressed as a string using . as a separator:

field1.field2

Both lists and dicts can be included using [] and {} syntax:

field[1].field2

or:

field{key=value}.field2

ResourceTraversalIterator¶

Iterator for traversing (walking) a resource structure, including traversing composite fields to fully navigate a resource tree.

This class has hooks that can be used by subclasses to customise the behaviour of the class:

on_enter - Called after entering a new resource.

on_exit - Called after exiting a resource.

Utils¶

Collection of utilities for working with Odin as well as generic data manipulation.

Resources¶

Name Manipulation¶

Choice Generation¶

Iterables¶

Integration¶

Integrations of Odin with other libraries/services.

Integration with Amazon Web Services¶

Integration with Amazon Web Services is via odincontrib.aws <https://github.com/python-odin/odincontrib.aws/

Requires boto3.

Odin Contrib AWS includes:

Integration with DynamoDB. Dynamo versions of fields, resources (Table) and querying

Dynamo DB¶

Tables can be defined using extended Resources, this provides an interface for table creation, save (put) items, query, scan and batch operations. Querying is performed either using raw parameters (as used by Boto3) or using a SQLAlchemy style fluent interface.

Example:

from odincontrib_aws import dynamodb


class Book(dynamodb.Table):
    class Meta:
        namespace = 'library'

    title = dynamodb.StringField()
    isbn = dynamodb.StringField(key=True)
    num_pages = dynamodb.IntegerField()
    genre = dynamodb.StringField(choices=(
        ('sci-fi', 'Science Fiction'),
        ('fantasy', 'Fantasy'),
        ('biography', 'Biography'),
        ('others', 'Others'),
        ('computers-and-tech', 'Computers & technology'),
    ))

session = dynamodb.Session()

# Save a new book into a Dynamo DB table
book = Book(
    title="The Hitchhiker's Guide to the Galaxy",
    isbn="0-345-39180-2",
    num_pages=224,
    genre='sci-fi',
)
session.put_item(book)

# Scan through all books in the table (this method is transparently paged)
for book in session.scan(Book):
    print(book.title)

SQS¶

This is development in progress to use Odin for defining and verifying SQS messages.

Integration with Django¶

Mapping Django models to resources¶

To utilise a Django database model in a Mapping a FieldResolver is required to identify fields available on the model.

The following is an example of a resolver for Django models.

from odin import registration
from odin.utils import getmeta
from odin.mapping import FieldResolverBase


class ModelFieldResolver(FieldResolverBase):
    """
    Field resolver for Django Models
    """

    def get_field_dict(self):
        meta = getmeta(self.obj)
        return {f.attname: f for f in meta.fields}


registration.register_field_resolver(ModelFieldResolver, models.Model)

Odin Examples¶

Examples of odin usage to get you started.

Adapter Examples¶

Filtering out fields¶

There are many times where certain fields need to be filtered from a resource before the resource is serialised (either to file or into an HTTP response). This could be to remove any sensitive information, or internal debug fields that should not be returned to an API. The odin.adapters.ResourceAdapter class makes this trivial.

Which fields should be filtered can be defined in one of two ways:

# At instantiation to allowing for dynamic filtering
my_filtered_resource = odin.ResourceAdapter(my_resource, exclude=('password',))

# Of predefined if this adapter will be used multiple times (not an adapter is not tied to any particular resource
# type so a single adapter could be used to filter the password field from any resource)
class PasswordFilter(odin.ResourceAdapter):
    exclude = ['password',]

my_filtered_resource = PasswordFilter(my_resource)

Added additional functionality to a resource¶

I have a resource that define a set of simulation results. I want to be able to render the results of this simulation into several different formats eg an HTML table, a text file, a chart.

One approach would be to add to a method to the resource for each of the formats I wich to render like to_html or to_text, however this has a number of drawbacks:

The resource is not more complex with an additional method for each of the target formats
Rendering code is now mixed in with your data structure definition
In a larger team there is more chances of multiple people working on the same file making merges more complex
If I define another resource for a different set of results, sharing common code is harder
The resource has a different interface for each method

The odin.adapters.ResourceAdapter class simplifies this.

By defining a rendering adapter for each of the different targets, the rendering code for each of these targets is encapsulated in a single class:

class HtmlRenderingAdapter(odin.ResourceAdapter):
    def render(self):
        ...


class TextRenderingAdapter(odin.ResourceAdapter):
    def render(self):
        ...

The benefits of this approach:

All features required for each render adapter are encapsulated
Each of these adapters can exist in their own file
Both adapters include the same render interface they can be passed to code that understands that interface
Both classes can both inherit off a common base class that provides common code that is related to each rendering operation.

CSV Codec Examples¶

The CSV codec is different to other codecs in that it produces an iterable of odin.Resource objects rather than a single structure.

Typical Usage¶

When using the odin.codecs.csv_codec my recommendation is to define your CSV file format using a CSV csv.Dialect and a sub-class of the odin.codecs.csv_codec.Reader to configure how files are handled.

A simple example of a customised csv.Dialect and odin.codecs.csv_codec.Reader:

import csv

from odin.codecs import csv_codec


class MyFileDialect(csv.Dialect):
    """
    Custom CSV dialect
    """
    delimiter = '\t'       # Tab delimited
    quotechar = '"'        # Double quotes for quoting
    lineterminator = '\n'  # UNIX style line termination


class MyFileReader(csv_codec.Reader):
    """
    Custom file reader
    """
    # Specify our custom dialect
    csv_dialect = MyFileDialect
    # Header case is not important
    ignore_header_case = True
    # Treat empty values as `None`
    default_empty_value = None

This can then be used with:

# To read a file
with open("my_file.csv") as f:
    for r in MyFileReader(f, MyResource):
        pass

# To write a file
with open("my_file.csv", "w") as f:
    csv_codec.dump(f, my_resources, dialect=MyFileDialect)

Handling errors¶

As resources are generated as each CSV row is read any validation errors raised also need to be handled. The default behavior is for a validation error to be raised when a invalid row is encountered effectively stopping processing. However, it may be valid to skip bad rows or, a report of bad rows can be generated and processing continued.

There are two approaches that can be used:

Provide an error_callback value to the odin.codecs.csv_codec.Reader
Sub-class the odin.codecs.csv_codec.Reader class and implement a

custom handle_validation_error method to process errors.

Note

Providing an error_callback will overwrite any custom handle_validation_error method.

The error_callback option is the simplest:

def error_callback(validation_error, idx):
    """
    Handle errors reading rows from CSV file.

    :param validation_error: The validation error exception
    :param idx: The row index the error occurred on.
    :returns: ``None`` or ``False`` to explicitly case the exception to
        be raised.

    """
    print("Error in row {}: {}".format(idx, validation_error), file=sys.stderr)

with open("my_file.csv") as f:
    for r in MyFileReader(f, MyResource, error_callback=error_callback):
        ...

The sub-class method is more involved upfront but does allow for more customisation:

class MyReader(csv_codec.Reader):
    """
    Custom file reader that reports errors to a file.
    """
    def __init__(self, f, error_file, *args, **kwargs):
        super().__init__(self, *args, **kwargs)

        self.error_file = error_file

    def handle_validation_error(self, validation_error, idx):
        self.error_file.write("{}\t{}\n".format(idx, validation_error))


with open("my_file.csv") as f_in, open("my_file.error.csv", "w") as f_err:
    for r in MyReader(f_in, f_err, MyResource):
        ...

The second option allows for a lot of customisation and reuses. For example the error report could itself output a CSV file.

Change History¶

2.3.1¶

Bug fix¶

ResourceOptions.composite_fields filtered composite field by Resource instead of ResourceBase.

2.3¶

Add meta option to specify how to format field names for serialisation. For example support being able to specify camelCase.
Updates to documentation - Add field_name_format meta option to docs - Document odin.utils

2.2¶

Fix a bug where annotated resource fields failed to get resolved.
Fix issues where functools.cached_property does not work on annotated resources as __set_name__ was not being called.
Updates to documentation - Introduce integration example for mapping Django models - Flesh out mapping documentation - Fix missing link to dict_codec reference.

2.1¶

Annotated Resources¶

Final fields are now mapped to ConstantField and require a default value.
Fix issue where the Toml codec did not recognise annotated resources in dump and dumps functions.

Breaking changes¶

Virtual fields ConstantField and CalculatedField now require keyword arguments for options.

2.0¶

Annotated Resources! - Use annotations to define your resources including Aliases for Email, URL’s and UUID’s
Drop support for Python <3.8, officially tested on Python 3.8+
Removed a number of dependencies that where only required for Python <3.8 eg (enum, six etc)
Removed deprecated features: - odin.csv_codec.ResourceReader - odin.serializers.DatetimeEcmaFormat
Improvements to typing and utilising of modern Python features
Fix bug in toml_codec that prevented the use of the include_type_field option.
Invalid choices now generate a ValueError exception
New fields for builtin Python types - odin.PathField - odin.RegexField
Use builtin datatime.timezone.utc instead of custom UTC instance

1.8.1¶

Change composite field to pass the resource the field is defined with rather than the name of the field, this allows namespace and typing to work correctly.

1.8.0¶

Allow types to be resolved by assuming a types names space (simplifies specification of types where the full name space can be assumed)
Inherit a custom type_field (this could cause unexpected side effects, although unlikely as this is a required change)

1.7.3¶

Fix bug where __classcell__ was not passed into super_new during metaclass construction. This causes a RuntimeError with Python 3.8+ if the metaclass is subclassed.

1.7.2¶

Fix an edge case bug where validators are not executed against empty list/dict fields.
Ensure that all validate and run_validators are executed on all entries in a typed list field

1.7.1¶

Tweak to generation of sphinx choices so TypedLists use choices from type.
Fix issue in Sphinx plugin where choices_doc_text was not used to format choice values.
Add TypedDictField to docs

1.7.0¶

Migrate from collections.Iterable to typing.Iterable to remove a warning
Add user defined metadata to Resource.Meta object
Add a reserved word guard, starting with fields (using fields causes clean_fields to go into a infinite recursion loop)

1.6.2¶

Migrate string formatting from % to “”.format
Apply black code style on all code
Improve some type hints
Add a Text block to XML Codec

1.6.1¶

Fix poetry config error that restricted Python versions > 3!

1.6.0¶

Remove encoding option for msgpack (this has been removed from msgpack itself)
Correct some typing inconsistencies (EnumField was the most obvious fix)
Package is now managed by poetry
Migrate to GitHub Actions
Static analysis and Coverage reporting now handled by SonarCloud.
Toml Codec

1.5.2¶

Relax overly tight dependency specification. Removed pyup.io it is not suitable for packages.

1.5¶

Improvements to sphinx autodoc generation
Improvements to enum support

1.0b2¶

Added a redesigned CSV codec Reader to address issues with handling row-by-row processing and handling of row-level errors.
Removed filter-query (moved to feature branch)

0.10b9¶

Added support for MultiPartField field type. A virtual field that allows for the easy generation of key values composed of several other fields.
Documentation updates and some initial type hints added (using :type… style)
Allow a virtual field to be identified as a key
Introduce a common base class for ALL field types, this allows them to be identified separately via hash (for sorting and keys)
Tweaked the new string.empty option so by default it behaves in the same way as it was previously defined (eg null assumes empty)

0.10b¶

Possible breaking change with date and time serializers, they no longer apply a timezone as this is expected to be managed by date and time fields.
Added getmeta tool for getting meta data.
Changes to Resource/Meta options to allow for them to be overridden.
Fixes regarding ResourceBase/Resource changes

0.9¶

Fixed bug preparing naive date and time values.

0.9a0¶

Support for multipart keys.
Support for naive dates and times.
Boolean fields now accept y/n as truthy
Change of Resource iterable base class

0.8.2¶

Migrated all tests to py.test
Support for Python 2.6 for _most_ features (some contrib features are not supported see docs for detail)

0.8.1¶

Bug fix for compatibility.deprecated decorator. Was breaking baldr when handling classes.
Some documentation fixes.

0.8¶

Added key_fields to resource meta options. This allows definition of which field(s) are used to uniquely identify the resource.
Added support to TraversalPath (and ResourceTraversalIterator) for using key_fields, both generating and traversing a Resource tree.
CodecDecodeError and CodecEncodeError now inherit off CodecError.
TraversalPath now raises multiple error types InvalidPathError, NoMatchError, and MultipleMatchesError
Some additional tweaks to inspect tool, as of now is still only available to Python 3.
Additional test cases.

0.7¶

Python 3.5 added for Travis CI test runner
Added empty flag to ListOf and DictOf files to validate if fields is allowed to be empty
Added key_choices to DictOf field to allow allow keys to be validated
Added TypedDictField this is a analog of the TypedListField but for dict objects
Project is now owned by the python-odin group
Initial work on combined string serialisation framework. This is to unify text serialisation support between the various codecs so additional serialises only need to be registered in a central location.
Added DictCodec, this is codec for converting between a dict structure and resource structure. As Resource.to_dict is explicitly designed to not act in a recursively this codec fills in a gap where a developer my require a nested dict to be generated.
Initial implementation of resource filtering. Filtering allows for a filter expression to be defined and used to filter a list of resources or ensure a particular resource matches the expression. Expressions are easy to define and read and allow for filtering against sub-resources etc.
Added caching to the results of mapping operations. Previously iterating through a mapping multiple times caused the mapping to be re-run. This has been made the default behaviour, if the previous behaviour is desired the MappingResult class can be provided to Mapping.apply instead of CachingMappingResult.
ResourceAdapter.apply_to now caches Meta object of matching resources types, previously a new meta object was created for each resource.
ResourceAdapterOptions.virtual_fields is now filtered using include/exclude options.
More tests, more docs
Various bug fixes.

0.6¶

Odin¶

Changed the generated type name for mapping factory to ensure a more unique name.
Pass through excluded_fields when generating types in mapping factory.
Added ignored fields parameter to be provided to Resource.convert_to method.
Added included fields parameter to the Mapping.update method
Updated the ResourceAdapter interface to match that of a normal Resource, can now be used in place of a Resource with any codec.
Added ResourceAdapter.apply_to method that simplifies applying a resource adapter to a list of Resources.
Updated create_resource_from_dict to accept a list and return a list of resources.
Fixed bug in auto-mapping generation of MapListOf fields to set the to_list flag correctly
Added forward_mapping_factory a shortcut version of mapping_factory when only a forward mapping is required.
Added the assign (similar to the define shortcut) shortcut for defining an assignment mapping
Fixed a bug where performing a full clean on a Resource that was created via a mapping would fail when a MappingResult object was encountered.
Added ResourceOptions.element_field_map for use in XML codecs
Added container flag and ResourceOptions.container_fields for XML style codecs
Large number of documentation updates
Fixed bug where Mapping.loop_idx was not being updated correctly, after 2 levels of nesting the wrong index was being updated.

Sphinx Integration¶

Add max length into resource documentation
Added description to validators that can be used in sphinx documentation