跳转至

DotBracketTokenizer

DotBracketTokenizer provides a simple way to tokenize secondary structure in dot-bracket notation. It also supports tokenization into nmers and codons, so you don’t have to write complex code to preprocess your data.

By default, DotBracketTokenizer uses the standard alphabet. If nmers is greater than 1, or codon is set to True, it will instead use the streamline alphabet.

multimolecule.tokenisers.DotBracketTokenizer

Bases: Tokenizer

Tokenizer for Secondary Structure sequences.

Parameters:

Name Type Description Default

alphabet

Alphabet | str | List[str] | None

alphabet to use for tokenization.

  • If is None, the standard Secondary Structure alphabet will be used.
  • If is a string, it should correspond to the name of a predefined alphabet. The options include
    • standard
    • iupac
    • streamline
    • nucleobase
  • If is an alphabet or a list of characters, that specific alphabet will be used.
 None

nmers

int

Size of kmer to tokenize.

 1

codon

bool

Whether to tokenize into codons.

 False

Examples:

Python Console Session
>>> from multimolecule import DotBracketTokenizer
>>> tokenizer = DotBracketTokenizer()
>>> tokenizer('<pad><cls><eos><unk><mask><null>.()+,[]{}|<>-_:~$@^%*')["input_ids"]
[1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 2]
>>> tokenizer('(.)')["input_ids"]
[1, 7, 6, 8, 2]
>>> tokenizer('+(.)')["input_ids"]
[1, 9, 7, 6, 8, 2]
>>> tokenizer = DotBracketTokenizer(nmers=3)
>>> tokenizer('(((((+..........)))))')["input_ids"]
[1, 27, 27, 27, 29, 34, 54, 6, 6, 6, 6, 6, 6, 6, 6, 8, 16, 48, 48, 48, 2]
>>> tokenizer = DotBracketTokenizer(codon=True)
>>> tokenizer('(((((+..........)))))')["input_ids"]
[1, 27, 29, 6, 6, 6, 16, 48, 2]
>>> tokenizer('(((((+...........)))))')["input_ids"]
Traceback (most recent call last):
ValueError: length of input sequence must be a multiple of 3 for codon tokenization, but got 22
Source code in multimolecule/tokenisers/dot_bracket/tokenization_db.py
Python
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
class DotBracketTokenizer(Tokenizer):
    """
    Tokenizer for Secondary Structure sequences.

    Args:
        alphabet: alphabet to use for tokenization.

            - If is `None`, the standard Secondary Structure alphabet will be used.
            - If is a `string`, it should correspond to the name of a predefined alphabet. The options include
                + `standard`
                + `iupac`
                + `streamline`
                + `nucleobase`
            - If is an alphabet or a list of characters, that specific alphabet will be used.
        nmers: Size of kmer to tokenize.
        codon: Whether to tokenize into codons.

    Examples:
        >>> from multimolecule import DotBracketTokenizer
        >>> tokenizer = DotBracketTokenizer()
        >>> tokenizer('<pad><cls><eos><unk><mask><null>.()+,[]{}|<>-_:~$@^%*')["input_ids"]
        [1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 2]
        >>> tokenizer('(.)')["input_ids"]
        [1, 7, 6, 8, 2]
        >>> tokenizer('+(.)')["input_ids"]
        [1, 9, 7, 6, 8, 2]
        >>> tokenizer = DotBracketTokenizer(nmers=3)
        >>> tokenizer('(((((+..........)))))')["input_ids"]
        [1, 27, 27, 27, 29, 34, 54, 6, 6, 6, 6, 6, 6, 6, 6, 8, 16, 48, 48, 48, 2]
        >>> tokenizer = DotBracketTokenizer(codon=True)
        >>> tokenizer('(((((+..........)))))')["input_ids"]
        [1, 27, 29, 6, 6, 6, 16, 48, 2]
        >>> tokenizer('(((((+...........)))))')["input_ids"]
        Traceback (most recent call last):
        ValueError: length of input sequence must be a multiple of 3 for codon tokenization, but got 22
    """

    model_input_names = ["input_ids", "attention_mask"]

    def __init__(
        self,
        alphabet: Alphabet | str | List[str] | None = None,
        nmers: int = 1,
        codon: bool = False,
        additional_special_tokens: List | Tuple | None = None,
        **kwargs,
    ):
        if codon and (nmers > 1 and nmers != 3):
            raise ValueError("Codon and nmers cannot be used together.")
        if codon:
            nmers = 3  # set to 3 to get correct vocab
        if not isinstance(alphabet, Alphabet):
            alphabet = get_alphabet(alphabet, nmers=nmers)
        super().__init__(
            alphabet=alphabet,
            nmers=nmers,
            codon=codon,
            additional_special_tokens=additional_special_tokens,
            **kwargs,
        )
        self.nmers = nmers
        self.condon = codon

    def _tokenize(self, text: str, **kwargs):
        if self.condon:
            if len(text) % 3 != 0:
                raise ValueError(
                    f"length of input sequence must be a multiple of 3 for codon tokenization, but got {len(text)}"
                )
            return [text[i : i + 3] for i in range(0, len(text), 3)]
        if self.nmers > 1:
            return [text[i : i + self.nmers] for i in range(len(text) - self.nmers + 1)]  # noqa: E203
        return list(text)

MultiMolecule provides a set of predefined alphabets for tokenization.

Standard Alphabet

The standard alphabet is an extended version of the Extended Dot-Bracket Notation. This extension includes most symbols from the WUSS notation for better compatibility with existing tools.

Code Represents
. unpaired
( internal helices of all terminal stems
) internal helices of all terminal stems
+ nick between strand
, unpaired in multibranch loops
[ internal helices that includes at least one annotated () stem
] internal helices that includes at least one annotated () stem
{ all internal helices of deeper multifurcations
} all internal helices of deeper multifurcations
| mostly paired
< simple terminal stems
> simple terminal stems
- bulges and interior loops
_ unpaired
: single stranded in the exterior loop
~ local structural alignment left regions of target and query unaligned
$ Not Used
@ Not Used
^ Not Used
% Not Used
* Not Used

Extended Alphabet

Extended Dot-Bracket Notation is a more generalized version of the original Dot-Bracket notation may use additional pairs of brackets for annotating pseudo-knots, since different pairs of brackets are not required to be nested.

Code Represents
. unpaired
( internal helices of all terminal stems
) internal helices of all terminal stems
+ nick between strand
, unpaired in multibranch loops
[ internal helices that includes at least one annotated () stem
] internal helices that includes at least one annotated () stem
{ all internal helices of deeper multifurcations
} all internal helices of deeper multifurcations
| mostly paired
< simple terminal stems
> simple terminal stems

Note that we use . to represent a gap in the sequence.

Streamline Alphabet

The streamline alphabet includes one additional symbol to the nucleobase alphabet, N to represent unknown nucleobase.

Code Represents
. unpaired
( internal helices of all terminal stems
) internal helices of all terminal stems
+ nick between strand