3
�G�\�o � # @ s� d Z ddlZddlmZ ddlmZ ddlmZmZm Z m
Z
ddl m Z ddl
mZ ed�Zejd ej�Zejd
ej�Zejd �Zyed d
d� W n" ek
r� ejd�ZdZY nBX ddlmZ ejdjej ��ZdZddl Z e j d= ddlZe`[ejd�Z ejd�Z ed�Z!ed�Z"ed�Z#ed�Z$ed�Z%ed �Z&ed �Z'ed �Z(ed �Z)ed �Z*ed!�Z+ed"�Z,ed#�Z-ed$�Z.ed%�Z/ed&�Z0ed'�Z1ed(�Z2ed)�Z3ed*�Z4ed+�Z5ed,�Z6ed-�Z7ed.�Z8ed/�Z9ed0�Z:ed1�Z;ed2�Z<ed3�Z=ed4�Z>ed5�Z?ed6�Z@ed7�ZAed8�ZBed9�ZCed:�ZDed;�ZEed<�ZFed=�ZGed>�ZHed?�ZIed@�ZJedA�ZKedB�ZLedC�ZMedD�ZNedE�ZOedF�ZPedG�ZQe!e9e%e(e1e0e4e:e,e6e-e7e+e5e'e2e)e*e.e/e"e&e#e3e$e8dH�ZReSdIdJ� e eR�D ��ZTejdKdLjUdMdN� eVeRdOdP� dQ�D �� �ZWeXeGeIeHe;eLeMeNg�ZYeXe;eOeIeNg�ZZdRdS� Z[dTdU� Z\dVdW� Z]dXdY� Z^dZd[� Z_G d\d]� d]e`�ZaG d^d_� d_eb�ZceG d`da� dae`��ZdeG dbdc� dce`��Zeddde� ZfG dfdg� dge`�ZgdS )ha�
jinja2.lexer
~~~~~~~~~~~~
This module implements a Jinja / Python combination lexer. The
`Lexer` class provided by this module is used to do some preprocessing
for Jinja.
On the one hand it filters out invalid operators like the bitshift
operators we don't allow in templates. On the other hand it separates
template code and python code in expressions.
:copyright: (c) 2017 by the Jinja Team.
:license: BSD, see LICENSE for more details.
� N)�deque)�
itemgetter)�implements_iterator�intern� iteritems� text_type)�TemplateSyntaxError)�LRUCache�2 z\s+z7('([^'\\]*(?:\\.[^'\\]*)*)'|"([^"\\]*(?:\\.[^"\\]*)*)")z\d+u fööz <unknown>�evalz[a-zA-Z_][a-zA-Z0-9_]*F)� _identifierz[\w{0}]+Tzjinja2._identifierz(?<!\.)\d+\.\d+z (\r\n|\r|\n)�addZassignZcolonZcommaZdiv�dot�eq�floordiv�gtZgteqZlbraceZlbracketZlparen�ltZlteq�mod�mul�ne�pipe�powZrbraceZrbracketZrparenZ semicolon�sub�tildeZ
whitespace�float�integer�name�string�operator� block_begin� block_endZvariable_begin� variable_end� raw_begin�raw_endZ
comment_beginZ comment_end�comment�linestatement_begin�linestatement_endZlinecomment_beginZlinecomment_end� linecomment�data�initial�eof)�+�-�/z//�*�%z**�~�[�]�(�)�{�}z==z!=�>z>=�<z<=�=�.�:�|�,�;c C s g | ]\}}||f�qS � r? )�.0�k�vr? r? �/usr/lib/python3.6/lexer.py�
<listcomp>� s rD z(%s)r< c c s | ]}t j|�V qd S )N)�re�escape)r@ �xr? r? rC � <genexpr>� s rH c C s
t | � S )N)�len)rG r? r? rC �<lambda>� s rJ )�keyc C sL | t krt | S tdtdtdtdtdtdtdtdt dt
d t d
t d i j
| | �S ) Nzbegin of commentzend of commentr$ zbegin of statement blockzend of statement blockzbegin of print statementzend of print statementzbegin of line statementzend of line statementztemplate data / textzend of template)�reverse_operators�TOKEN_COMMENT_BEGIN�TOKEN_COMMENT_END�
TOKEN_COMMENT�TOKEN_LINECOMMENT�TOKEN_BLOCK_BEGIN�TOKEN_BLOCK_END�TOKEN_VARIABLE_BEGIN�TOKEN_VARIABLE_END�TOKEN_LINESTATEMENT_BEGIN�TOKEN_LINESTATEMENT_END�
TOKEN_DATA� TOKEN_EOF�get)�
token_typer? r? rC �_describe_token_type� s r[ c C s | j dkr| jS t| j �S )z#Returns a description of the token.r )�type�valuer[ )�tokenr? r? rC �describe_token� s
r_ c C s2 d| kr&| j dd�\}}|dkr*|S n| }t|�S )z0Like `describe_token` but for token expressions.r; � r )�splitr[ )�exprr\ r] r? r? rC �describe_token_expr� s rc c C s t tj| ��S )zsCount the number of newline characters in the string. This is
useful for extensions that filter a stream.
)rI �
newline_re�findall)r] r? r? rC �count_newlines� s rf c C s� t j}t| j�d|| j�ft| j�d|| j�ft| j�d|| j�fg}| jdk rp|jt| j�dd|| j� f� | jdk r�|jt| j�dd|| j� f� d d
� t |d d �D �S )
zACompiles all the rules from the environment into a list of rules.r$ �blockZvariableNZ
linestatementz ^[ \t\v]*r' z(?:^|(?<=\S))[^\S\r\n]*c S s g | ]}|d d� �qS )r` Nr? )r@ rG r? r? rC rD � s z!compile_rules.<locals>.<listcomp>T)�reverse)
rE rF rI �comment_start_string�block_start_string�variable_start_string�line_statement_prefix�append�line_comment_prefix�sorted)� environment�e�rulesr? r? rC �
compile_rules� s
rs c @ s$ e Zd ZdZefdd�Zdd� ZdS )�FailurezjClass that raises a `TemplateSyntaxError` if called.
Used by the `Lexer` to specify known errors.
c C s || _ || _d S )N)�message� error_class)�selfru �clsr? r? rC �__init__� s zFailure.__init__c C s | j | j||��d S )N)rv ru )rw �lineno�filenamer? r? rC �__call__� s zFailure.__call__N)�__name__�
__module__� __qualname__�__doc__r ry r| r? r? r? rC rt � s rt c @ sT e Zd ZdZf Zdd� ed�D �\ZZZdd� Z dd� Z
d d
� Z d d � Z d
d� Z
dS )�Tokenz Token class.c c s | ]}t t|��V qd S )N)�propertyr )r@ rG r? r? rC rH � s zToken.<genexpr>� c C s t j| |tt|��|f�S )N)�tuple�__new__r �str)rx rz r\ r] r? r? rC r� � s z
Token.__new__c C s* | j tkrt| j S | j dkr$| jS | j S )Nr )r\ rL r] )rw r? r? rC �__str__� s
z
Token.__str__c C s2 | j |krdS d|kr.|jdd�| j | jgkS dS )z�Test a token against a token expression. This can either be a
token type or ``'token_type:token_value'``. This can only test
against string values and types.
Tr; r` F)r\ ra r] )rw rb r? r? rC �test� s
z
Token.testc G s x|D ]}| j |�rdS qW dS )z(Test against multiple token expressions.TF)r� )rw �iterablerb r? r? rC �test_any s
zToken.test_anyc C s d| j | j| jf S )NzToken(%r, %r, %r))rz r\ r] )rw r? r? rC �__repr__ s zToken.__repr__N)r} r~ r r� � __slots__�rangerz r\ r] r� r� r� r� r� r? r? r? rC r� � s
r� c @ s( e Zd ZdZdd� Zdd� Zdd� ZdS ) �TokenStreamIteratorz`The iterator for tokenstreams. Iterate over the stream
until the eof token is reached.
c C s
|| _ d S )N)�stream)rw r� r? r? rC ry s z TokenStreamIterator.__init__c C s | S )Nr? )rw r? r? rC �__iter__ s z TokenStreamIterator.__iter__c C s0 | j j}|jtkr"| j j� t� �t| j � |S )N)r� �currentr\ rX �close�
StopIteration�next)rw r^ r? r? rC �__next__ s
z TokenStreamIterator.__next__N)r} r~ r r� ry r� r� r? r? r? rC r� s r� c @ s~ e Zd ZdZdd� Zdd� Zdd� ZeZedd � d
d �Z d d
� Z
dd� Z d dd�Z dd� Z
dd� Zdd� Zdd� Zdd � Zd S ) � TokenStreamz�A token stream is an iterable that yields :class:`Token`\s. The
parser however does not iterate over it but calls :meth:`next` to go
one token ahead. The current active token is stored as :attr:`current`.
c C s> t |�| _t� | _|| _|| _d| _tdtd�| _ t
| � d S )NFr` � ) �iter�_iterr �_pushedr r{ �closedr� �
TOKEN_INITIALr� r� )rw � generatorr r{ r? r? rC ry / s
zTokenStream.__init__c C s t | �S )N)r� )rw r? r? rC r� 8 s zTokenStream.__iter__c C s t | j�p| jjtk S )N)�boolr� r� r\ rX )rw r? r? rC �__bool__; s zTokenStream.__bool__c C s | S )Nr? )rG r? r? rC rJ ? s zTokenStream.<lambda>z Are we at the end of the stream?)�docc C s | j j|� dS )z Push a token back to the stream.N)r� rm )rw r^ r? r? rC �pushA s zTokenStream.pushc C s" t | �}| j}| j|� || _|S )zLook at the next token.)r� r� r� )rw Z old_token�resultr? r? rC �lookE s
zTokenStream.lookr` c C s xt |�D ] }t| � q
W dS )zGot n tokens ahead.N)r� r� )rw �nrG r? r? rC �skipM s zTokenStream.skipc C s | j j|�rt| �S dS )zqPerform the token test and return the token if it matched.
Otherwise the return value is `None`.
N)r� r� r� )rw rb r? r? rC �next_ifR s zTokenStream.next_ifc C s | j |�dk S )z8Like :meth:`next_if` but only returns `True` or `False`.N)r� )rw rb r? r? rC �skip_ifY s zTokenStream.skip_ifc C sX | j }| jr| jj� | _ n:| j jtk rTyt| j�| _ W n tk
rR | j� Y nX |S )z|Go one token ahead and return the old one.
Use the built-in :func:`next` instead of calling this directly.
) r� r� �popleftr\ rX r� r� r� r� )rw �rvr? r? rC r� ] s zTokenStream.__next__c C s" t | jjtd�| _d| _d| _dS )zClose the stream.r� NT)r� r� rz rX r� r� )rw r? r? rC r� l s zTokenStream.closec C st | j j|�s^t|�}| j jtkr:td| | j j| j| j��td|t | j �f | j j| j| j��z| j S t
| � X dS )z}Expect a given token type and return it. This accepts the same
argument as :meth:`jinja2.lexer.Token.test`.
z(unexpected end of template, expected %r.zexpected token %r, got %rN) r� r� rc r\ rX r rz r r{ r_ r� )rw rb r? r? rC �expectr s zTokenStream.expectN)r` )r} r~ r r� ry r� r� Z __nonzero__r� Zeosr� r� r� r� r� r� r� r� r? r? r? rC r� ( s
r� c C sZ | j | j| j| j| j| j| j| j| j| j | j
| j f }t j
|�}|dkrVt| �}|t |< |S )z(Return a lexer which is probably cached.N)rj �block_end_stringrk �variable_end_stringri �comment_end_stringrl rn � trim_blocks�
lstrip_blocks�newline_sequence�keep_trailing_newline� _lexer_cacherY �Lexer)rp rK Zlexerr? r? rC � get_lexer� s"
r� c @ s> e Zd ZdZdd� Zdd� Zd
dd�Zdd d
�Zdd d �ZdS )r� a
Class that implements a lexer for a given environment. Automatically
created by the environment class, usually you don't have to do that.
Note that the lexer is not automatically bound to an environment.
Multiple environments can share the same lexer.
c s� dd� }t j}ttd fttd fttd ftt d ft
t d ft t
d fg}t|�}|jrTdpVd}i � |j�r\|d�}|d||j� �}|j|j�} || r�d|| jd�� p�d7 }|j|j�} || r�d|| jd�� p�d7 }|d||j� �}
|
j|j�} | �r
d || jd�� �p d} d
} d | ||j�|||j�f }
d | ||j�| ||j�f }|
� d
< |� d< nd||j� }
|j| _|j| _d|ddjd||j�|
||j�||j�f g� fdd�|D � � �tdfdf|d�td fgt|d||j �||j �|f �t t fdf|d�t d�fd fgt |d ||j�||j�|f �t!dfg| t"|d ||j#�||j#�f �t$dfg| t%|d ||j�|
||j�||j�|f �tt&fdf|d�t d �fd fgt'|d �t(dfg| t)|d!�t*t+fdfgi| _,d S )"Nc S s t j| t jt jB �S )N)rE �compile�M�S)rG r? r? rC rJ � s z Lexer.__init__.<locals>.<lambda>z\n?r� r+ z^%s(.*)z|%sr` z(?!%s)z^[ \t]*z%s%s(?!%s)|%s\+?z %s%s%s|%s\+?rg r$ z%s�rootz (.*?)(?:%s)r< z4(?P<raw_begin>(?:\s*%s\-|%s)\s*raw\s*(?:\-%s\s*|%s))c s& g | ] \}}d ||� j ||�f �qS )z(?P<%s_begin>\s*%s\-|%s))rY )r@ r� �r)� prefix_rer? rC rD � s z"Lexer.__init__.<locals>.<listcomp>z#bygroupz.+z(.*?)((?:\-%s\s*|%s)%s)z#popz(.)zMissing end of comment tagz(?:\-%s\s*|%s)%sz
\-%s\s*|%sz1(.*?)((?:\s*%s\-|%s)\s*endraw\s*(?:\-%s\s*|%s%s))z Missing end of raw directivez \s*(\n|$)z(.*?)()(?=\n|$))-rE rF �
whitespace_re�TOKEN_WHITESPACE�float_re� TOKEN_FLOAT�
integer_re�
TOKEN_INTEGER�name_re�
TOKEN_NAME� string_re� TOKEN_STRING� operator_re�TOKEN_OPERATORrs r� r� rj �matchri �grouprk r� r� �joinr� rW rM r� rO rN rt rQ rR rS r� rT �TOKEN_RAW_BEGIN�
TOKEN_RAW_ENDrU rV �TOKEN_LINECOMMENT_BEGINrP �TOKEN_LINECOMMENT_ENDrr )rw rp �crq Z tag_rulesZroot_tag_rulesZblock_suffix_reZ no_lstrip_reZ
block_diff�mZ comment_diffZno_variable_reZ lstrip_reZblock_prefix_reZcomment_prefix_rer? )r� rC ry � s�
zLexer.__init__c C s t j| j|�S )z@Called for strings and template data to normalize it to unicode.)rd r r� )rw r] r? r? rC �_normalize_newlines$ s zLexer._normalize_newlinesNc C s& | j ||||�}t| j|||�||�S )zCCalls tokeniter + tokenize and wraps it in a token stream.
)� tokeniterr� �wrap)rw �sourcer r{ �stater� r? r? rC �tokenize( s zLexer.tokenizec c sj �xb|D �]X\}}}|t kr"q�n2|dkr2d}�n"|dkrBd}�n|dkrPq�n|dkrd| j|�}n�|dkrr|}n�|d kr�t|�}tr�|j� r�td
|||��n�|d k�ry$| j|d d� �jd
d�jd�}W nH tk
�r } z*t|�j d�d j
� }t||||��W Y dd}~X nX n:|dk�r.t |�}n&|dk�rBt |�}n|dk�rTt
| }t|||�V qW dS )z�This is called with the stream as returned by `tokenize` and wraps
every token in a :class:`Token` and converts the value.
r% r r& r r" r# r( �keywordr z Invalid character in identifierr r` �ascii�backslashreplacezunicode-escaper; Nr r r )r" r# ���r� )�ignored_tokensr� r� � check_ident� isidentifierr �encode�decode� Exceptionra �strip�intr � operatorsr� ) rw r� r r{ rz r^ r] rq �msgr? r? rC r� . sD
"
z
Lexer.wrapc c s@ t |�}|j� }| jr>|r>x"d D ]}|j|�r |jd� P q W dj|�}d}d}dg} |dk rv|dkrv| j|d � nd}| j| d }
t|�} g } �x��x�|
D �]j\}
}}|
j||�}|dkr�q�| r�|d kr�q�t |t
��r�x�t |�D ]�\}}|j t
k�r|||��q�|d
k�r`x�t|j� �D ]0\}}|dk �r |||fV ||jd�7 }P �q W td|
��q�|j|d �}|�s~|tk�r�|||fV ||jd�7 }q�W n�|j� }|dk�r<|dk�r�| jd� nv|dk�r�| jd� n`|dk�r�| jd� nJ|d k�r<| �std| |||��| j� }||k�r<td||f |||��|�sL|tk�rX|||fV ||jd�7 }|j� }|dk �r�|dk�r�| j� nT|d
k�r�xHt|j� �D ] \}}|dk �r�| j|� P �q�W td|
��n
| j|� | j| d }
n||k�rtd|
��|}P q�W || k�r dS td|| |f |||��q�W dS )!z�This method tokenizes the text and returns the tokens in a
generator. Use this method if you just want to tokenize a template.
�
�
�
r� r r` r� NZ_beginr! r r& z#bygroupz?%r wanted to resolve the token dynamically but no group matchedr r5 r6 r3 r4 r1 r2 zunexpected '%s'z unexpected '%s', expected '%s'z#popzC%r wanted to resolve the new state dynamically but no group matchedz,%r yielded empty string without stack changezunexpected char %r at %d)r� r� r� r� )r! r r& )r6 r4 r2 r� )r �
splitlinesr� �endswithrm r� rr rI r� �
isinstancer� � enumerate� __class__rt r � groupdict�count� RuntimeErrorr� �ignore_if_emptyr �pop�end)rw r� r r{ r� �lines�newline�posrz �stackZ statetokensZ
source_lengthZbalancing_stackZregex�tokensZ new_stater� �idxr^ rK r] r( Z expected_opZpos2r? r? rC r� W s�
zLexer.tokeniter)NNN)NN)NN) r} r~ r r� ry r� r� r� r� r? r? r? rC r� � s
)r� )hr� rE � collectionsr r r Zjinja2._compatr r r r Zjinja2.exceptionsr Z jinja2.utilsr r� r� �Ur� r� r� r� � SyntaxErrorr� r� Zjinja2r �format�pattern�sys�modulesr� rd Z TOKEN_ADDZ TOKEN_ASSIGNZ TOKEN_COLONZ TOKEN_COMMAZ TOKEN_DIVZ TOKEN_DOTZTOKEN_EQZTOKEN_FLOORDIVZTOKEN_GTZ
TOKEN_GTEQZ TOKEN_LBRACEZTOKEN_LBRACKETZ TOKEN_LPARENZTOKEN_LTZ
TOKEN_LTEQZ TOKEN_MODZ TOKEN_MULZTOKEN_NEZ
TOKEN_PIPEZ TOKEN_POWZ TOKEN_RBRACEZTOKEN_RBRACKETZ TOKEN_RPARENZTOKEN_SEMICOLONZ TOKEN_SUBZ TOKEN_TILDEr� r� r� r� r� r� rQ rR rS rT r� r� rM rN rO rU rV r� r� rP rW r� rX r� �dictrL r� ro r� � frozensetr� r� r[ r_ rc rf rs �objectrt r� r� r� r� r� r� r? r? r? rC �<module> s�
+ ^