3
� �XNy � @ s d Z ddlmZ ddlZddlZddlZddlmZmZ ddl m
Z
ddl m Z m
Z
mZmZ ddlmZmZmZmZmZmZmZmZmZ ddlmZ d d
d d d
dddddddg Z d=d>d?d@dAgZ e d d � �Z G d!d"� d"e �Z!ee!�G d#d � d e"��Z#G d$d � d e#�Z$G d%d� de%�Z&G d&d'� d'e"�Z'e'� Z(G d(d)� d)e)�Z*G d*d+� d+e"�Z+d,d� Z,G d-d.� d.e"�Z-e-� Z.d/d� Z/G d0d� d�Z0G d1d� de�Z1G d2d3� d3e!�Z2ee2�G d4d
� d
e#��Z3G d5d
� d
e"�Z4G d6d � d e3�Z5d7d8� Z6G d9d:� d:e2�Z7ee7�G d;d<� d<e3��Z8dS )Bz�
pygments.lexer
~~~~~~~~~~~~~~
Base lexer classes.
:copyright: Copyright 2006-2017 by the Pygments team, see AUTHORS.
:license: BSD, see LICENSE for details.
� )�print_functionN)�
apply_filters�Filter)�get_filter_by_name)�Error�Text�Other�
_TokenType) � get_bool_opt� get_int_opt� get_list_opt�make_analysator� text_type�
add_metaclass� iteritems�Future� guess_decode)� regex_opt�Lexer�
RegexLexer�ExtendedRegexLexer�DelegatingLexer� LexerContext�include�inherit�bygroups�using�this�default�words� �utf-8� �� �utf-32� ���utf-32be� ���utf-16� ���utf-16bec C s dS )Ng � )�xr* r* �/usr/lib/python3.6/lexer.py�<lambda>$ s r- c @ s e Zd ZdZdd� ZdS )� LexerMetaz�
This metaclass automagically converts ``analyse_text`` methods into
static methods which always return float values.
c C s( d|krt |d �|d< tj| |||�S )N� analyse_text)r
�type�__new__)Zmcs�name�bases�dr* r* r, r1 - s zLexerMeta.__new__N)�__name__�
__module__� __qualname__�__doc__r1 r* r* r* r, r. ' s r. c @ sZ e Zd ZdZdZg Zg Zg Zg ZdZ dd� Z
dd� Z dd � Z d
d � Z
dd
d�Zdd� ZdS )r a�
Lexer for a specific language.
Basic options recognized:
``stripnl``
Strip leading and trailing newlines from the input (default: True).
``stripall``
Strip all leading and trailing whitespace from the input
(default: False).
``ensurenl``
Make sure that the input ends with a newline (default: True). This
is required for some lexers that consume input linewise.
.. versionadded:: 1.3
``tabsize``
If given and greater than 0, expand tabs in the input (default: 0).
``encoding``
If given, must be an encoding name. This encoding will be used to
convert the input string to Unicode, if it is not already a Unicode
string (default: ``'guess'``, which uses a simple UTF-8 / Locale /
Latin1 detection. Can also be ``'chardet'`` to use the chardet
library, if it is installed.
``inencoding``
Overrides the ``encoding`` if given.
Nr c K s� || _ t|dd�| _t|dd�| _t|dd�| _t|dd�| _|jdd �| _|jd
�pZ| j| _g | _ x t
|d f �D ]}| j |� qrW d S ) N�stripnlT�stripallF�ensurenl�tabsizer �encoding�guessZ
inencoding�filters) �optionsr
r9 r: r; r r< �getr= r? r �
add_filter)�selfr@ �filter_r* r* r, �__init__b s zLexer.__init__c C s( | j rd| jj| j f S d| jj S d S )Nz <pygments.lexers.%s with %r>z<pygments.lexers.%s>)r@ � __class__r5 )rC r* r* r, �__repr__n s
zLexer.__repr__c K s& t |t�st|f|�}| jj|� dS )z8
Add a new stream filter to this lexer.
N)�
isinstancer r r? �append)rC rD r@ r* r* r, rB u s
zLexer.add_filterc C s dS )a~
Has to return a float between ``0`` and ``1`` that indicates
if a lexer wants to highlight this text. Used by ``guess_lexer``.
If this method returns ``0`` it won't highlight it in any case, if
it returns ``1`` highlighting with this lexer is guaranteed.
The `LexerMeta` metaclass automatically wraps this function so
that it works like a static method (no ``self`` or ``cls``
parameter) and the return value is automatically converted to
`float`. If the return value is an object that is boolean `False`
it's the same as if the return values was ``0.0``.
Nr* )�textr* r* r, r/ } s zLexer.analyse_textFc s� t �t�s� jdkr"t��\�}nʈ jdkr�y ddl}W n tk
rT td��Y nX d}x4tD ],\}}�j|�r`�t|�d� j |d�}P q`W |dkr�|j
�dd� �}�j |j d�p�d d�}|�n&�j � j���jd
�r�td
�d� �n �jd
��r
�td
�d� ��j d d ���j d
d ��� j
�r4�j� �n� j�rF�jd ��� jdk�r^�j� j��� j�r|�jd � �r|�d 7 �� �fdd�} | � }
|�s�t|
� j� �}
|
S )a=
Return an iterable of (tokentype, value) pairs generated from
`text`. If `unfiltered` is set to `True`, the filtering mechanism
is bypassed even if filters are defined.
Also preprocess the text, i.e. expand tabs and strip it if
wanted and applies registered filters.
r> �chardetr NzkTo enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/�replacei r= zutf-8u z
�
�
c 3 s( x"� j ��D ]\} }}||fV q W d S )N)�get_tokens_unprocessed)�_�t�v)rC rJ r* r, �streamer� s z"Lexer.get_tokens.<locals>.streamer)rH r r= r rK � ImportError�
_encoding_map�
startswith�len�decodeZdetectrA rL r: �stripr9 r< �
expandtabsr; �endswithr r? ) rC rJ Z
unfilteredrP rK ZdecodedZbomr= �encrS �streamr* )rC rJ r, �
get_tokens� sL
zLexer.get_tokensc C s t �dS )z�
Return an iterable of (index, tokentype, value) pairs where "index"
is the starting position of the token within the input text.
In subclasses, implement this method as a generator to
maximize effectiveness.
N)�NotImplementedError)rC rJ r* r* r, rO � s z Lexer.get_tokens_unprocessed)F)r5 r6 r7 r8 r2 �aliases� filenamesZalias_filenamesZ mimetypesZpriorityrE rG rB r/ r^ rO r* r* r* r, r 3 s
;c @ s$ e Zd ZdZefdd�Zdd� ZdS )r a
This lexer takes two lexer as arguments. A root lexer and
a language lexer. First everything is scanned using the language
lexer, afterwards all ``Other`` tokens are lexed using the root
lexer.
The lexers from the ``template`` lexer package use this base lexer.
c K s0 |f |�| _ |f |�| _|| _tj| f|� d S )N)�
root_lexer�language_lexer�needler rE )rC Z _root_lexerZ_language_lexerZ_needler@ r* r* r, rE � s zDelegatingLexer.__init__c C s� d}g }g }xX| j j|�D ]H\}}}|| jkrR|rH|jt|�|f� g }||7 }q|j|||f� qW |r||jt|�|f� t|| jj|��S )N� )rc rO rd rI rW �
do_insertionsrb )rC rJ Zbuffered�
insertionsZ
lng_buffer�irQ rR r* r* r, rO � s
z&DelegatingLexer.get_tokens_unprocessedN)r5 r6 r7 r8 r rE rO r* r* r* r, r � s c @ s e Zd ZdZdS )r zI
Indicates that a state should include rules from another state.
N)r5 r6 r7 r8 r* r* r* r, r � s c @ s e Zd ZdZdd� ZdS )�_inheritzC
Indicates the a state should inherit from its superclass.
c C s dS )Nr r* )rC r* r* r, rG s z_inherit.__repr__N)r5 r6 r7 r8 rG r* r* r* r, ri � s ri c @ s e Zd ZdZdd� Zdd� ZdS )�combinedz:
Indicates a state combined from multiple states.
c G s t j| |�S )N)�tupler1 )�cls�argsr* r* r, r1 s zcombined.__new__c G s d S )Nr* )rC rm r* r* r, rE s zcombined.__init__N)r5 r6 r7 r8 r1 rE r* r* r* r, rj s rj c @ sF e Zd ZdZdd� Zddd�Zddd�Zdd d
�Zd d � Zd
d� Z dS )� _PseudoMatchz:
A pseudo match object constructed from a string.
c C s || _ || _d S )N)�_text�_start)rC �startrJ r* r* r, rE s z_PseudoMatch.__init__Nc C s | j S )N)rp )rC �argr* r* r, rq s z_PseudoMatch.startc C s | j t| j� S )N)rp rW ro )rC rr r* r* r, �end" s z_PseudoMatch.endc C s |r t d��| jS )Nz
No such group)�
IndexErrorro )rC rr r* r* r, �group% s z_PseudoMatch.groupc C s | j fS )N)ro )rC r* r* r, �groups* s z_PseudoMatch.groupsc C s i S )Nr* )rC r* r* r, � groupdict- s z_PseudoMatch.groupdict)N)N)N)
r5 r6 r7 r8 rE rq rs ru rv rw r* r* r* r, rn s
rn c s d� fdd� }|S )zL
Callback that yields multiple actions for each group in the match.
Nc 3 s� x�t � �D ]�\}}|d kr q
q
t|�tkrT|j|d �}|r�|j|d �||fV q
|j|d �}|d k r
|r~|j|d �|_x.|| t|j|d �|�|�D ]}|r�|V q�W q
W |r�|j� |_d S )N� )� enumerater0 r ru rq �posrn rs )�lexer�match�ctxrh �action�data�item)rm r* r, �callback5 s" zbygroups.<locals>.callback)Nr* )rm r� r* )rm r, r 1 s c @ s e Zd ZdZdS )�_ThiszX
Special singleton used for indicating the caller class.
Used by ``using``.
N)r5 r6 r7 r8 r* r* r* r, r� K s r� c sj i �d�kr:�j d�}t|ttf�r.|�d< n d|f�d< � tkrTd��fdd� }nd � ��fdd� }|S )
a�
Callback that processes the match with a different lexer.
The keyword arguments are forwarded to the lexer, except `state` which
is handled separately.
`state` specifies the state that the new lexer will start in, and can
be an enumerable such as ('root', 'inline', 'string') or a simple
string which is assumed to be on top of the root state.
Note: For that to work, `_other` must not be an `ExtendedRegexLexer`.
�state�stack�rootNc 3 sn �r �j | j� | jf ��}n| }|j� }x0|j|j� f� �D ]\}}}|| ||fV q>W |rj|j� |_d S )N)�updater@ rF rq rO ru rs rz )r{ r| r} �lx�srh rQ rR )� gt_kwargs�kwargsr* r, r� i s zusing.<locals>.callbackc 3 sb �j | j� � f ��}|j� }x0|j|j� f��D ]\}}}|| ||fV q2W |r^|j� |_d S )N)r� r@ rq rO ru rs rz )r{ r| r} r� r� rh rQ rR )�_otherr� r� r* r, r� x s
)N)N)�poprH �listrk r )r� r� r� r� r* )r� r� r� r, r S s
c @ s e Zd ZdZdd� ZdS )r z�
Indicates a state or state action (e.g. #pop) to apply.
For example default('#pop') is equivalent to ('', Token, '#pop')
Note that state tuples may be used as well.
.. versionadded:: 2.0
c C s
|| _ d S )N)r� )rC r� r* r* r, rE � s zdefault.__init__N)r5 r6 r7 r8 rE r* r* r* r, r � s c @ s" e Zd ZdZddd�Zdd� ZdS ) r z�
Indicates a list of literal words that is transformed into an optimized
regex that matches any of the words.
.. versionadded:: 2.0
re c C s || _ || _|| _d S )N)r �prefix�suffix)rC r r� r� r* r* r, rE � s zwords.__init__c C s t | j| j| jd�S )N)r� r� )r r r� r� )rC r* r* r, rA � s z words.getN)re re )r5 r6 r7 r8 rE rA r* r* r* r, r � s
c @ sJ e Zd ZdZdd� Zdd� Zdd� Zdd � Zdd d �Zd
d� Z dd� Z
d
S )�RegexLexerMetazw
Metaclass for RegexLexer, creates the self._tokens attribute from
self.tokens on the first instantiation.
c C s t |t�r|j� }tj||�jS )zBPreprocess the regular expression component of a token definition.)rH r rA �re�compiler| )rl �regex�rflagsr� r* r* r, �_process_regex� s
z RegexLexerMeta._process_regexc C s& t |�tks"t|�s"td|f ��|S )z5Preprocess the token component of a token definition.z2token type must be simple type or callable, not %r)r0 r �callable�AssertionError)rl �tokenr* r* r, �_process_token� s z RegexLexerMeta._process_tokenc C s t |t�rd|dkrd
S ||kr$|fS |dkr0|S |dd� dkrRt|dd� � S dsbtd| ��n�t |t�r�d | j }| jd7 _g }x4|D ],}||ks�td
| ��|j| j|||�� q�W |||< |fS t |t��rx(|D ] }||ks�|dks�td | ��q�W |S d�std | ��dS )z=Preprocess the state transition action of a token definition.�#poprx �#pushN� z#pop:Fzunknown new state %rz_tmp_%dzcircular state ref %rzunknown new state zunknown new state def %r���)r� r� ) rH �str�intr� rj �_tmpname�extend�_process_staterk )rl � new_state� unprocessed� processedZ tmp_state�itokensZistater* r* r, �_process_new_state� s6
z!RegexLexerMeta._process_new_statec C s� t |�tkstd| ��|d dks0td| ��||kr@|| S g }||< | j}�x:|| D �],}t|t�r�||ks�td| ��|j| j||t|��� q^t|t�r�q^t|t �r�| j
|j ||�}|j t
jd�jd|f� q^t |�tks�td| ��y| j|d ||�}W n< tk
�rF } z td |d || | f ��W Y dd} ~ X nX | j|d
�}
t|�d k�rjd}n| j
|d ||�}|j ||
|f� q^W |S ) z%Preprocess a single state definition.zwrong state name %rr �#zinvalid state name %rzcircular state reference %rre Nzwrong rule def %rz+uncompilable regex %r in state %r of %r: %srx � )r0 r� r� �flagsrH r r� r� ri r r� r� rI r� r� r| rk r� � Exception�
ValueErrorr� rW ) rl r� r� r� �tokensr� Ztdefr� �rex�errr� r* r* r, r� � s>
&
z RegexLexerMeta._process_stateNc C s@ i }| j |< |p| j| }x t|�D ]}| j|||� q&W |S )z-Preprocess a dictionary of token definitions.)� _all_tokensr� r� r� )rl r2 � tokendefsr� r� r* r* r, �process_tokendef s
z RegexLexerMeta.process_tokendefc
C s� i }i }x�| j D ]�}|jjdi �}x�t|�D ]�\}}|j|�}|dkr~|||< y|jt�}W n tk
rr w,Y nX |||< q,|j|d�}|dkr�q,||||d �< y|jt�} W n tk
r� Y q,X || ||< q,W qW |S )a
Merge tokens from superclasses in MRO order, returning a single tokendef
dictionary.
Any state that is not defined by a subclass will be inherited
automatically. States that *are* defined by subclasses will, by
default, override that state in the superclass. If a subclass wishes to
inherit definitions from a superclass, it can use the special value
"inherit", which will cause the superclass' state definition to be
included at that point in the state.
r� Nrx )�__mro__�__dict__rA r �indexr r� r� )
rl r� Z inheritable�cZtoksr� �itemsZcuritemsZ inherit_ndxZ new_inh_ndxr* r* r, �
get_tokendefs
s0
z RegexLexerMeta.get_tokendefsc O sL d| j kr:i | _d| _t| d�r(| jr(n| jd| j� �| _tj | f|�|�S )z:Instantiate cls after preprocessing its token definitions.�_tokensr �token_variantsre )
r� r� r� �hasattrr� r� r� r� r0 �__call__)rl rm �kwdsr* r* r, r� ; s
zRegexLexerMeta.__call__)N) r5 r6 r7 r8 r� r� r� r� r� r� r� r* r* r* r, r� � s #,
1r� c @ s$ e Zd ZdZejZi Zddd�ZdS )r z�
Base for simple stateful regular expression-based lexers.
Simplifies the lexing process so that you need only
provide a list of states and regular expressions.
r� c
c s� d}| j }t|�}||d
}�x��x||D �]
\}}} |||�}
|
r*|dk r�t|�tkrj|||
j� fV nx|| |
�D ]
} | V qvW |
j� }| dk �r4t| t�r�x�| D ]8} | dkr�|j� q�| dkr�|j |d � q�|j | � q�W nDt| t
�� r�|| d�= n,| dk�r|j |d � nd�s(t d| ��||d
}P q*W yN|| dk�rnd g}|d }|t dfV |d7 }w"|t
|| fV |d7 }W q" tk
�r� P Y q"X q"W dS )z}
Split ``text`` into (tokentype, text) pairs.
``stack`` is the inital stack (default: ``['root']``)
r rx Nz#popz#pushFzwrong state def: %rrM r� r� r� r� r� )r� r� r0 r ru rs rH rk r� rI r� r� r r rt )
rC rJ r� rz r� Z
statestack� statetokens�rexmatchr~ r� �mr� r� r* r* r, rO h sP
z!RegexLexer.get_tokens_unprocessedN�r� )r� ) r5 r6 r7 r8 r� � MULTILINEr� r� rO r* r* r* r, r I s c @ s" e Zd ZdZddd�Zdd� ZdS )r z9
A helper object that holds lexer position data.
Nc C s* || _ || _|pt|�| _|p"dg| _d S )Nr� )rJ rz rW rs r� )rC rJ rz r� rs r* r* r, rE � s zLexerContext.__init__c C s d| j | j| jf S )NzLexerContext(%r, %r, %r))rJ rz r� )rC r* r* r, rG � s zLexerContext.__repr__)NN)r5 r6 r7 r8 rE rG r* r* r* r, r � s
c @ s e Zd ZdZddd�ZdS )r zE
A RegexLexer that uses a context object to store its state.
Nc c s | j }|s t|d�}|d }n|}||jd
}|j}�xސx�|D �]>\}}}|||j|j�} | rB|dk r�t|�tkr�|j|| j� fV | j� |_n.x|| | |�D ]
}
|
V q�W |s�||jd }|dk �r�t |t
��r(x�|D ]D} | dk� r�|jj � q�| dk�r|jj |jd � q�|jj | � q�W nJt |t
��rB|j|d�= n0|dk�r`|jj |jd
� nd�srtd| ��||jd }P qBW yt|j|jk�r�P ||j d k�r�dg|_|d }|jtd fV | jd7 _w:|jt||j fV | jd7 _W q: tk
�r P Y q:X q:W dS )z
Split ``text`` into (tokentype, text) pairs.
If ``context`` is given, use this lexer context instead.
r r� rx Nz#popz#pushFzwrong state def: %rrM r� r� r� r� r� )r� r r� rJ rz rs r0 r ru rH rk r� rI r� r� r r rt ) rC rJ �contextr� r} r� r� r~ r� r� r� r� r* r* r, rO � s^
z)ExtendedRegexLexer.get_tokens_unprocessed)NN)r5 r6 r7 r8 rO r* r* r* r, r � s c c s� t | �} yt| �\}}W n( tk
r@ x|D ]
}|V q.W dS X d}d}x�|D ]�\}}} |dkrf|}d}
x�|o~|t| � |k�r| |
|| � } ||| fV |t| �7 }x*|D ]"\} }
}||
|fV |t|�7 }q�W || }
yt| �\}}W ql tk
�r
d}P Y qlX qlW ||| |
d� fV |t| �|
7 }qPW xr|�r�|�pHd}x,|D ]$\}}} ||| fV |t| �7 }�qPW yt| �\}}W n tk
�r� d}P Y nX �q:W dS )ag
Helper for lexers which must combine the results of several
sublexers.
``insertions`` is a list of ``(index, itokens)`` pairs.
Each ``itokens`` iterable should be inserted at position
``index`` into the token stream given by the ``tokens``
argument.
The result is a combined token stream.
TODO: clean up the code here.
NTr F)�iter�next�
StopIterationrW )rg r� r� r� r� ZrealposZinsleftrh rQ rR ZoldiZtmpvalZit_indexZit_tokenZit_value�pr* r* r, rf � sL
rf c @ s e Zd ZdZdd� ZdS )�ProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c sL t |t�r t|j|j|jd��n|�tj�|��tjf� ���fdd� }|S )N)r� r� c s` � j d j��fddg�}tj� }�j| ||�}tj� }|d d7 < |d || 7 < |S )Nrx r g r� )�
_prof_data�
setdefault�timer| )rJ rz �endpos�infoZt0�resZt1)rl �compiledr� r� r* r, �
match_func@ s z:ProfilingRegexLexerMeta._process_regex.<locals>.match_func) rH r r r� r� r� r� �sys�maxsize)rl r� r� r� r� r* )rl r� r� r� r, r� 8 s
z&ProfilingRegexLexerMeta._process_regexN)r5 r6 r7 r8 r� r* r* r* r, r� 5 s r� c @ s" e Zd ZdZg ZdZddd�ZdS ) �ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes.� r� c # s� � j jji � x tj� ||�D ]
}|V q W � j jj� }tdd� |j� D �� fdd�dd�}tdd� |D ��}t � t d� j j
t |�|f � t d d
� t d d � t dd
� x|D ]}t d| � q�W t d d
� d S )Nc s sN | ]F\\}}\}}|t |�jd �jdd�dd� |d| d| | fV qdS )zu'z\\�\N�A i� )�reprrY rL )�.0r� �r�nrQ r* r* r, � <genexpr>X s z=ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<genexpr>c s
| � j S )N)�_prof_sort_index)r+ )rC r* r, r- [ s z<ProfilingRegexLexer.get_tokens_unprocessed.<locals>.<lambda>T)�key�reversec s s | ]}|d V qdS )� Nr* )r� r+ r* r* r, r� ] s z2Profiling result for %s lexing %d chars in %.3f ms�=�n z$%-20s %-64s ncalls tottime percallr� r� �-z%-20s %-65s %5d %8.4f %8.4f)r� r� ) rF r� rI r rO r� �sortedr� �sum�printr5 rW )rC rJ r� �tokZrawdatar Z sum_totalr4 r* )rC r, rO R s$
z*ProfilingRegexLexer.get_tokens_unprocessedN�r� )r� )r5 r6 r7 r8 r� r� rO r* r* r* r, r� K s r� )r r! )r" r# )r$ r% )r& r' )r( r) )9r8 Z
__future__r r� r� r� Zpygments.filterr r Zpygments.filtersr Zpygments.tokenr r r r Z
pygments.utilr
r r r
r r r r r Zpygments.regexoptr �__all__rU � staticmethodZ_default_analyser0 r. �objectr r r� r ri r rk rj rn r r� r r r r r� r r r rf r� r� r* r* r* r, �<module>
sX , '
2 )WE?