Working with Metadata

Metadata is information associated with entities that tskit doesn’t use or interpret, but which is useful to pass on to downstream analysis such as sample ids, dates etc. (see Metadata for a full discussion). Each table has a MetadataSchema which details the contents and encoding of the metadata for each row. A metadata schema is a JSON document that conforms to JSON Schema (The full schema for tskit is at Full metaschema). Here we use an existing tree sequence which contains some demonstration metadata:

import tskit
import json

ts = tskit.load("data/metadata.trees")

Reading metadata and schemas

Metadata is automatically decoded using the schema when accessed via a TreeSequence or TableCollection Python API. For example:

print("Metadata for individual 0:", ts.individual(0).metadata)  # Tree sequence access
print("Metadata for individual 0:", ts.tables.individuals[0].metadata)  # Table access
Metadata for individual 0: {'accession': 'ERS0001', 'pcr': True}
Metadata for individual 0: {'accession': 'ERS0001', 'pcr': True}

Viewing the MetadataSchema for a table can help with understanding its metadata, as it can contain descriptions and constraints:

ts.table_metadata_schemas.individual
{"additionalProperties":false,"codec":"json","properties":{"accession":{"description":"ENA accession number","type":"string"},"pcr":{"description":"Was PCR used on this sample","name":"PCR Used","type":"boolean"}},"required":["accession","pcr"],"type":"object"}

The same schema can be accessed via a metadata_schema attribute on each table (printed prettily here using json.dumps)

schema = ts.tables.individuals.metadata_schema
print(json.dumps(schema.asdict(), indent=4))  # Print with indentations
{
    "additionalProperties": false,
    "codec": "json",
    "properties": {
        "accession": {
            "description": "ENA accession number",
            "type": "string"
        },
        "pcr": {
            "description": "Was PCR used on this sample",
            "name": "PCR Used",
            "type": "boolean"
        }
    },
    "required": [
        "accession",
        "pcr"
    ],
    "type": "object"
}

The top-level metadata and schemas for the entire tree sequence are similarly accessed with TreeSequence.metadata() and TreeSequence.metadata_schema().

Note

If there is no schema (i.e. it is equal to MetadataSchema(None)) for a table or top-level metadata, then no decoding is performed and bytes will be returned.

Modifying metadata and schemas

If you are creating or modifying a tree sequence by changing the underlying tables, you may want to record or add to the metadata. When doing so the schema must be modified first, as it is then used to validate and encode the metadata.

Todo

link to a tutorial section on editing the tree sequence via the tables interface

Schemas in tskit are held in a MetadataSchema. A Python dict representation of the schema is passed to its constructor, which will validate the schema. Here are a few examples: the first one allows arbitrary fields to be added, the second one (which will construct the schema we printed above) does not:

basic_schema = tskit.MetadataSchema({'codec': 'json'})

complex_schema = tskit.MetadataSchema({
    'codec': 'json',
    'additionalProperties': False,
    'properties': {'accession': {'description': 'ENA accession number',
                                 'type': 'string'},
                   'pcr': {'description': 'Was PCR used on this sample',
                           'name': 'PCR Used',
                           'type': 'boolean'}},
    'required': ['accession', 'pcr'],
    'type': 'object',
})

This MetadataSchema can then be assigned to a table or the top-level tree sequence e.g. metadata_schema:

tables = tskit.TableCollection(sequence_length=1)  # make a new, empty set of tables
tables.individuals.metadata_schema = complex_schema

This will overwrite any existing schema. Note that this will not validate any existing metadata against the new schema. Now that the table has a schema, calls to add_row() will validate and encode the metadata:

row_id = tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": True})
print(f"Row {row_id} added to the individuals table")
Row 0 added to the individuals table

If we try to add metadata that doesn’t fit the schema, such as accidentally using a string instead of a proper Python boolean, we’ll get an error:

tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": "false"})
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/tskit/metadata.py in validate_and_encode_row(self, row)
    657         try:
--> 658             self._validate_row(row)
    659         except jsonschema.exceptions.ValidationError as ve:

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jsonschema/validators.py in validate(self, *args, **kwargs)
    352             for error in self.iter_errors(*args, **kwargs):
--> 353                 raise error
    354 

ValidationError: 'false' is not of type 'boolean'

Failed validating 'type' in schema['properties']['pcr']:
    OrderedDict([('description', 'Was PCR used on this sample'),
                 ('name', 'PCR Used'),
                 ('type', 'boolean')])

On instance['pcr']:
    'false'

The above exception was the direct cause of the following exception:

MetadataValidationError                   Traceback (most recent call last)
/tmp/ipykernel_6501/4092786825.py in <module>
----> 1 tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": "false"})

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/tskit/tables.py in add_row(self, flags, location, parents, metadata)
    652         if metadata is None:
    653             metadata = self.metadata_schema.empty_value
--> 654         metadata = self.metadata_schema.validate_and_encode_row(metadata)
    655         return self.ll_table.add_row(
    656             flags=flags, location=location, parents=parents, metadata=metadata

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/tskit/metadata.py in validate_and_encode_row(self, row)
    658             self._validate_row(row)
    659         except jsonschema.exceptions.ValidationError as ve:
--> 660             raise exceptions.MetadataValidationError(str(ve)) from ve
    661         return self.encode_row(row)
    662 

MetadataValidationError: 'false' is not of type 'boolean'

Failed validating 'type' in schema['properties']['pcr']:
    OrderedDict([('description', 'Was PCR used on this sample'),
                 ('name', 'PCR Used'),
                 ('type', 'boolean')])

On instance['pcr']:
    'false'

and because we set additionalProperties to False in the schema, an error is also raised if we attempt to add new fields:

tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": True, "newKey": 25})
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/tskit/metadata.py in validate_and_encode_row(self, row)
    657         try:
--> 658             self._validate_row(row)
    659         except jsonschema.exceptions.ValidationError as ve:

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/jsonschema/validators.py in validate(self, *args, **kwargs)
    352             for error in self.iter_errors(*args, **kwargs):
--> 353                 raise error
    354 

ValidationError: Additional properties are not allowed ('newKey' was unexpected)

Failed validating 'additionalProperties' in schema:
    OrderedDict([('additionalProperties', False),
                 ('codec', 'json'),
                 ('properties',
                  OrderedDict([('accession',
                                OrderedDict([('description',
                                              'ENA accession number'),
                                             ('type', 'string')])),
                               ('pcr',
                                OrderedDict([('description',
                                              'Was PCR used on this '
                                              'sample'),
                                             ('name', 'PCR Used'),
                                             ('type', 'boolean')]))])),
                 ('required', ['accession', 'pcr']),
                 ('type', 'object')])

On instance:
    {'accession': 'Bob1234', 'newKey': 25, 'pcr': True}

The above exception was the direct cause of the following exception:

MetadataValidationError                   Traceback (most recent call last)
/tmp/ipykernel_6501/2608489950.py in <module>
----> 1 tables.individuals.add_row(0, metadata={"accession": "Bob1234", "pcr": True, "newKey": 25})

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/tskit/tables.py in add_row(self, flags, location, parents, metadata)
    652         if metadata is None:
    653             metadata = self.metadata_schema.empty_value
--> 654         metadata = self.metadata_schema.validate_and_encode_row(metadata)
    655         return self.ll_table.add_row(
    656             flags=flags, location=location, parents=parents, metadata=metadata

/opt/hostedtoolcache/Python/3.8.12/x64/lib/python3.8/site-packages/tskit/metadata.py in validate_and_encode_row(self, row)
    658             self._validate_row(row)
    659         except jsonschema.exceptions.ValidationError as ve:
--> 660             raise exceptions.MetadataValidationError(str(ve)) from ve
    661         return self.encode_row(row)
    662 

MetadataValidationError: Additional properties are not allowed ('newKey' was unexpected)

Failed validating 'additionalProperties' in schema:
    OrderedDict([('additionalProperties', False),
                 ('codec', 'json'),
                 ('properties',
                  OrderedDict([('accession',
                                OrderedDict([('description',
                                              'ENA accession number'),
                                             ('type', 'string')])),
                               ('pcr',
                                OrderedDict([('description',
                                              'Was PCR used on this '
                                              'sample'),
                                             ('name', 'PCR Used'),
                                             ('type', 'boolean')]))])),
                 ('required', ['accession', 'pcr']),
                 ('type', 'object')])

On instance:
    {'accession': 'Bob1234', 'newKey': 25, 'pcr': True}

To set the top-level metadata, just assign it. Validation and encoding happen as specified by the top-level metadata schema

tables.metadata_schema = basic_schema  # Allows new fields to be added that are not validated
tables.metadata = {"mean_coverage": 200.5}
print(tables.metadata)
{'mean_coverage': 200.5}

Note

Provenance information, detailing the origin of the data, modification timestamps, and (ideally) how the tree sequence can be reconstructed, should go in Provenance, not metadata.

To modify a schema — for example to add a key — first get the dict representation, modify, then write back:

schema_dict = tables.individuals.metadata_schema.schema
schema_dict["properties"]["newKey"] = {"type": "integer"}
tables.individuals.metadata_schema = tskit.MetadataSchema(schema_dict)
# Now this will work:
new_id = tables.individuals.add_row(metadata={'accession': 'abc123', 'pcr': False, 'newKey': 25})
print(tables.individuals[new_id].metadata)
{'accession': 'abc123', 'newKey': 25, 'pcr': False}

To modify the metadata of rows in tables use the Metadata for bulk table methods.

Viewing raw metadata

If you need to see the raw (i.e. bytes) metadata, you just need to remove the schema, for instance:

individual_table = tables.individuals.copy()  # don't change the original tables.individual

print("Metadata:\n", individual_table[0].metadata)

individual_table.metadata_schema = tskit.MetadataSchema(None)
print("\nRaw metadata:\n", individual_table[0].metadata)
Metadata:
 {'accession': 'Bob1234', 'pcr': True}

Raw metadata:
 b'{"accession":"Bob1234","pcr":true}'

Metadata for bulk table methods

In the interests of efficiency each table’s packset_metadata() method, as well as the more general set_columns() and append_columns() methods, do not attempt to validate or encode metadata. You can call MetadataSchema.validate_and_encode_row() directly to prepare metadata for these methods:

metadata_column = [
    {"accession": "etho1234", "pcr": True},
    {"accession": "richard1235", "pcr": False},
    {"accession": "albert1236", "pcr": True},
]
encoded_metadata_column = [
    tables.individuals.metadata_schema.validate_and_encode_row(r) for r in metadata_column
]
md, md_offset = tskit.pack_bytes(encoded_metadata_column)
tables.individuals.set_columns(flags=[0, 0, 0], metadata=md, metadata_offset=md_offset)
tables.individuals
idflagslocationparentsmetadata
00{'accession': 'etho1234', 'pcr': True}
10{'accession': 'richard1235', 'pcr': F...
20{'accession': 'albert1236', 'pcr': True}

Or if all columns do not need to be set:

tables.individuals.packset_metadata(
    [tables.individuals.metadata_schema.validate_and_encode_row(r) for r in metadata_column]
)

Binary metadata

To disable the validation and encoding of metadata and store raw bytes pass None to MetadataSchema

tables.populations.metadata_schema = tskit.MetadataSchema(None)
tables.populations.add_row(metadata=b"SOME CUSTOM BYTES #!@")
print(tables.populations[0].metadata)
b'SOME CUSTOM BYTES #!@'