Metadata

Metadata#

Overview#

SLiM puts SLiM-specific information into the metadata for the tree sequence, as well as for each population, individual, node and mutation. Here is a quick reference to what information is available: see the SLiM manual for the more technical writeup. A good way to get a generic metadata example is with default_slim_metadata().

Top-level: If ts is your tree sequence, then ts.metadata is a dict, and ts.metadata["SLiM"] contains information about the simulation:

  • file_version: the version of the SLiM tree sequence file format

  • tick: the value of community.tick within SLiM when the file was written out

  • cycle: the value of sim.cycle within SLiM when the file was written out

  • model_type: either "WF" or "nonWF"

  • nucleotide_based: whether this is a nucleotide-based simulation

  • separate_sexes: whether the simulation has separate sexes or not

  • spatial_dimensionality: for instance, "" or "x" or "xy" (etcetera)

  • spatial_periodicity: whether space wraps around in some directions (same format as dimensionality)

  • stage: the stage of the life cycle at which the file was written out (either "first", "early", or "late")

  • name: the name of this species in SLiM

  • this_chromosome: contains, for the chromosome in SLiM recorded in this tree sequence

    • id: SLiM’s ID

    • index: the index of this chromosome in the list of chromosomes

    • symbol: the user-assigned symbol

    • type: specifies inheritance type, e.g., "A" for autosome

  • chromosomes: (optional) a list of all chromosomes in the simulation, for each containing the same information as for this_chromosome

  • traits: a list of information for each of the traits:

    • index: the index of the trait in SLiM

    • name: the name in SLiM for the trait

    • type: "additive", "multiplicative", or "logistic"

    • baselineOffsetFromUser: a value added to all traits

    • baselineOffsetFromSubstitutions: the total effect of all substitutions on the traits at the time the tree sequence was saved

    • baselineAccumulation: whether the effect of substitutions accumulate in that value

    • directFitnessEffect: whether the trait has a direct effect on fitness

    • individualOffsetMean, individualOffsetSD: parameters governing the distribution of individual-level offsets (i.e., “environment” effects)

Populations: Information about each SLiM-produced population is written to metadata. The format uses JSON and is extensible, so other keys may be present and some keys may be missing (for instance, there are no spatial bounds in a nonspatial simulation). The metadata may be None for populations that SLiM did not use. The keys that SLiM uses are:

  • slim_id: the ID of this population in SLiM

  • name: the name of the population (by default, "p0", "p1", etcetera)

  • description: a string describing the population

  • selfing_fraction, female_cloning_fraction, male_cloning_fraction, and sex_ratio: only present when applicable (e.g., in WF simulations)

  • bounds_x0, bounds_x1, bounds_y0, bounds_y1, bounds_z0, and bounds_z1: the spatial bounds, when applicable

  • migration_records: A list of entries describing migration between populations in a WF model.

Individuals: Each individual produced by SLiM contains the following metadata:

  • pedigree_id: the “pedigree ID”, unique within the SLiM simulation

  • pedigree_p1, pedigree_p2: the pedigree IDs of the individuals’ two parents (they may be equal in the case of selfing, or -1 to indicate no parent, in the case of the initial generation)

  • age: the .age property within SLiM at the time the file was written out

  • subpopulation: the subpopulation within SLiM the individual was in at the time the file was written out

  • sex: the sex of the individual (either INDIVIDUAL_TYPE_FEMALE, INDIVIDUAL_TYPE_MALE, or INDIVIDUAL_TYPE_HERMAPHRODITE)

  • flags: additional information; currently only recording whether the individual was a “migrant” or not (see the SLiM manual)

  • tag, tagF: the corresponding properties in SLiM: default values returned by pyslim are the special values that SLiM uses to mean that the values are unset

  • tagL0, tagL0_set, etcetera: again, the corresponding properties in SLiM; the purpose of tagLX_set is to record whether the tag has been set in the simulation

  • per_trait: a list of information about the trait values for this individual; these are in the same order as the traits listed in top-level metadata;

    • phenotype: the trait value (which is NAN if the phenotype has not been calculated, or was invalidated)

    • offset: the individual’s offset (i.e., the “environmental effect”)

Nodes: Each “node” produced by SLiM (i.e., “haplosome” within SLiM) has:

  • slim_id: the unique ID associated with the haplosome by SLiM

  • is_vacant: records that the node is a “vacant” node (in which case it isn’t really there, so shouldn’t have any mutations or relationships in the tree sequence!) - see Vacant nodes: sex chromosomes and haploidy for more explanation

Mutations: Prior to SLiM 6.0, mutation metadata was associated with the tskit mutation objects. Now, this is stored in top-level metadata, under ts.metadata["SLiM_mutation_list"]. Each entry contains:

  • mutation_id: the numeric ID of mutation in SLiM

  • mutation_type: the numeric ID of the MutationType within SLiM

  • subpopulation: the numeric ID of the subpopulation the mutation occurred in

  • slim_time: the value of community.tick when the mutation occurred

  • nucleotide: either -1 if there is no associated nucleotide, or the numeric code for the nucleotide (see NUCLEOTIDES)

  • per_trait: a list of information in the same order as the traits in top-level metadata, recording for each:

    • effect_size: the effect on the trait of this mutation

    • dominance: its dominance coefficient

    • hemizygous_dominance: its hemizygous dominance coefficient (see the SLiM manual)

  • padding: this is simply empty bytes, here for byte-alignment reasons, and is always None

Using top-level metadata#

If you are going to be using information from top-level metadata, it is good practice to extract the metadata as a separate python object once and refer to that object, since otherwise you can incur runtime penalties for decoding and copying the metadata every time you call ts.metadata. This can be substantial, given the amount of mutation information in top-level metadata. For instance, to subtract baseline offsets from individual’s trait values, we might do:

md = ts.metadata
traits = md["SLiM"]["traits"]
values = [
    [x['phenotype'] - y["baselineOffsetFromUser"] for x, y in zip(ind.metadata['per_trait'], traits)]
    for ind in ts.individuals()
]

If we instead inserted ts.metadata["SLiM"]["traits"] directly into the loop, this would become infeasibly slow.

In some more detail: each time python evaluates ts.metadata (e.g., using ts.metadata["SLiM"]) a new copy of the metadata dict is decoded and returned. Furthermore, a number of pyslim functions need to look up information from metadata under the hood. For instance, previously it was acceptable to run [pyslim.slim_time(ts, mut.time) for mut in ts.mutations()]. However, this could now easily take hours even for moderately-sized simulations. There are several recommendations for how to mitigate this:

  • If you use information from top-level metadata, make a copy of it and refer to that copy instead: so, ts_metadata = ts.metadata after ts = tskit.load(...) and then use ts_metadata. However, be careful that you use the correct metadata object corresponding to the correct tree sequence object!

  • Use a single pyslim function call rather than many calls to the same one within a loop. For instance, run: slim_times = pyslim.slim_time(ts, ts.mutations_time) and extract slim times from this vector. Similarly, use nodes_vacant() instead of node_is_vacant().

  • Some pyslim methods will accept a pre-extracted metadata dictionary as an optional argument. If this is not provided, those methods will extract the metadata again. The methods that now take a ts_metadata argument are: individual_ages(), individual_ages_at(), individuals_alive_at(), and slim_time().

Metadata tools#

The dictionaries describing the schema for these metadata entries are available in pyslim.slim_metadata_schemas. Furthermore, these methods may be useful in working with metadata:

pyslim.default_slim_metadata(name, num_chromosomes=1, num_traits=1, **kwargs)[source]#

Returns default metadata of type name, where name is one of “tree_sequence”, “edge”, “site”, “mutation”, “mutation_list_entry”, “node”, “individual”, or “population”.

Additional keyword arguments may be passed in to modify the returned metadata dictionary: for each such key=value pair the returned dictionary has value (re)assigned to key.

Parameters:

  • name (str) – The type of metadata requested.

  • num_chromosomes (int) – The number of chromosomes (only relevant for “node”).

  • num_traits (int) – The number of traits (only relevant for “individual” and “mutation_list_entry”).

Rtype dict:

pyslim.slim_tree_sequence_metadata_schema(num_traits=1)[source]#

The top-level metadata schema depends on the number of traits, and {data}`.slim_metadata_schemas` returns the schema for a single-trait simulation. This function returns the correct schema for a simulation with arbitrary number of traits. (The resulting schemas only differ in the “length” of the “per_trait” property of schema["properties"]["SLiM_mutation_list"]["items"]).

Parameters:

num_traits (int) – The number of traits in the model.

Return tskit.MetadataSchema:

The metadata schema to be used in the node table.

pyslim.slim_individual_metadata_schema(num_traits=1)[source]#

The individual metadata schema depends on the number of traits, and {data}`.slim_metadata_schemas` returns the schema for a single-trait simulation. This function returns the correct schema for a simulation with arbitrary number of traits. (The resulting schemas only differ in the “length” of the “per_trait” property.)

Parameters:

num_traits (int) – The number of traits in the model.

Return tskit.MetadataSchema:

The metadata schema to be used in the node table.

pyslim.slim_node_metadata_schema(num_chromosomes=1)[source]#

The node metadata schema depends on the number of chromosomes in a multichromosome simulation, and {data}`.slim_metadata_schemas` returns the schema for a single-chromosome simulation. This function returns the correct schema for a simulation with arbitrary number of chromosomes. (The resulting schemas only differ in the schema["properties"]["is_vacant"]["length"] property, which is set to floor(num_chromosomes+7)/8, as described in the SLiM manual.)

Parameters:

num_chromosomes (int) – The number of chromosomes in the model.

Return tskit.MetadataSchema:

The metadata schema to be used in the node table.

pyslim.set_tree_sequence_metadata(tables, model_type, tick, *, cycle=None, spatial_dimensionality='', spatial_periodicity='', separate_sexes=False, nucleotide_based=False, stage='late', name='', description='', this_chromosome=None, chromosomes=None, file_version=None, set_table_schemas=True, traits=None, SLiM_mutation_list=None)[source]#
pyslim.set_metadata_schemas(tables, num_chromosomes=1, num_traits=1)[source]#

Modifying SLiM metadata#

For more on working with metadata, see tskit’s metadata documentation.

Top-level metadata#

The entries of the top-level metadata dict are read-only. So, although you might think that tables.metadata["SLiM"]["model_type"] = "nonWF" would switch the model type, this in fact (silently) does nothing. To modify the top-level metadata, we must (a) work with tables (as tree sequences are immutable), and (b) extract the metadata dict, modify the dict, and copy it back in. So, the code for modifying top-level metadata is:

md = tables.metadata
md["SLiM"]["model_type"] = "nonWF"
tables.metadata = md

Modifying the top-level metadata could be used to declare that an annotated msprime simulation is to be a spatial simulation in SLiM, by changing the “dimensionality” property in top-level metadata (and perhaps also the bounds in population metadata as well).

Modifying SLiM metadata in tables#

To modify the metadata that pyslim has introduced into the tree sequence produced by a coalescent simulation, or the metadata in a SLiM-produced tree sequence, we need to edit the TableCollection that forms the editable data behind the tree sequence. For instance, to set the ages of the individuals in the tree sequence to random numbers between 1 and 4, we will extract a copy of the underlying tables, clear it, and then iterate over the individuals in the tree sequence, as we go re-inserting them into the tables after replacing their metadata with a modified version:

tables = ts.dump_tables()
tables.individuals.clear()
for ind in ts.individuals():
    md = ind.metadata
    md["age"] = random.choice([1,2,3,4])
    tables.individuals.append(
        ind.replace(metadata=md)
    )

mod_ts = tables.tree_sequence()

# check that it worked:
print("First ten ages:", [mod_ts.individual(i).metadata["age"] for i in range(10)])
for ind in mod_ts.individuals():
    assert ind.metadata['age'] in [1, 2, 3, 4]

# save out the tree sequence
mod_ts.dump("modified_ts.trees")

First ten ages: [3, 1, 1, 3, 4, 4, 3, 2, 2, 3]

Technical details#

Metadata entries#

SLiM records additional information in the metadata columns of Individual and Node, and Mutation tables, and about mutations in top-level metadata, in a binary format using the python struct module. However, this is transparently taken care of by tskit’s metadata module, so the user does not have to work directly with binary information or the struct module itself. See tskit’s metadata documentation for details on how this works. Nothing besides this binary information can be stored in the metadata of these tables if the tree sequence is to be used by SLiM, and so when pyslim annotates an existing tree sequence, anything in those columns is overwritten. Population metadata is stored as JSON, however, which is more flexible. For more detailed documentation on the contents and format of the metadata, see the SLiM manual.

Of particular note is that nodes and populations may have empty metadata. SLiM will not use the metadata of nodes that are not associated with alive individuals, so this can safely be omitted (and makes recapitation easier). And, populations not used by SLiM will have empty metadata. All remaining metadata are required (besides edges and sites, whose metadata is not used at all).

Contents