3
� �XO � @ sz d Z ddlZddlmZ ddlmZmZmZ ddlm Z m
Z
m Z ddgZ G dd� de�Z
i Zejd �ZG d
d� de�ZdS ) z�
pygments.lexers.special
~~~~~~~~~~~~~~~~~~~~~~~
Special lexers.
:copyright: Copyright 2006-2017 by the Pygments team, see AUTHORS.
:license: BSD, see LICENSE for details.
� N)�Lexer)�Token�Error�Text)�get_choice_opt� text_type�BytesIO� TextLexer�
RawTokenLexerc @ s: e Zd ZdZdZdgZdgZdgZdZdd� Z d d
� Z
d S ) r z3
"Null" lexer, doesn't highlight anything.
z Text only�textz*.txtz
text/plaing{�G�z�?c c s dt |fV d S )Nr )r )�selfr � r
� /usr/lib/python3.6/special.py�get_tokens_unprocessed s z TextLexer.get_tokens_unprocessedc C s t jS )N)r �priority)r r
r
r � analyse_text# s zTextLexer.analyse_textN) �__name__�
__module__� __qualname__�__doc__�name�aliases� filenames� mimetypesr r r r
r
r
r r s s .*?
c @ s< e Zd ZdZdZdgZg ZdgZdd� Zdd� Z d d
� Z
d S ) r
aq
Recreate a token stream formatted with the `RawTokenFormatter`. This
lexer raises exceptions during parsing if the token stream in the
file is malformed.
Additional options accepted:
`compress`
If set to ``"gz"`` or ``"bz2"``, decompress the token stream with
the given compression algorithm before lexing (default: ``""``).
zRaw token data�rawz application/x-pygments-tokensc K s* t |dddddgd�| _tj| f|� d S )N�compress� Znone�gz�bz2)r r r �__init__)r Zoptionsr
r
r r < s zRawTokenLexer.__init__c c s� t |t�r|jd�}| jdkrDdd l}|jdddt|��}|j� }n | jdkr`dd l}|j |�}|j
d�d }x"| j |�D ]\}}}||fV qzW d S ) N�asciir r r �rb� r �
) �
isinstancer �encoder �gzipZGzipFiler �readr Z
decompress�stripr )r r r& Zgzipfiler �i�t�vr
r
r �
get_tokensA s
zRawTokenLexer.get_tokensc
c s� d}x�t j|�D ]�}y|j� jdd�\}}W n( tk
rT |j� jdd�}t}Y ntX tj|�}|s�t }|jd�dd � }x2|D ]*}| s�|d j
� r�td��t ||�}q�W |t|< |dd
� jd �}|||fV |t |�7 }qW d S ) Nr � � r �replace�.zmalformed token name� zunicode-escape���)
�line_re�finditer�group�split�
ValueError�decoder � _ttype_cache�getr �isupper�getattr�len) r r Zlength�matchZttypestr�valZttypeZttypesZttype_r
r
r r S s&
z$RawTokenLexer.get_tokens_unprocessedN) r r r r r r r r r r, r r
r
r
r r
+ s )r �reZpygments.lexerr Zpygments.tokenr r r Z
pygments.utilr r r �__all__r r9 �compiler3 r
r
r
r
r �<module>
s