tokenisers¶
tokenisers provide a collection of pre-defined tokenizers.
A tokenizer is a class that converts a sequence of nucleotides or amino acids into a sequence of indices. It is used to pre-process the input sequence before feeding it into a model.
Please refer to Tokenizer for more details.