3
m�daB# �
@ s y d dl ZW n ek
r( d dlZY nX d dlZd dlmZ d dlmZ d dlm Z d dl
m Z d dl m
Z
mZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ e ed
�ee d �d d
��Z e ed
�eed �dd��Z e ed
�eee d �dd��Z e ed
�ee d �dd��Z ee d �dd�Z!e ed
�ee d �dd��Z"e ed
�ee d �dd��Z#e ed
�ee d �dd��Z$e ed
�ee d �d d ��Z%e ed
�ee d �d d ��Z&ee d �d d!�Z'e ed
�ee d �d"d#��Z(e ed
�ee d �d$d%��Z)e ed
�ee d �d&d'��Z*e ed
�ee d �d(d)��Z+e ed
�ee d �d*d+��Z,e e-e�d
�ee d,�d-d.��Z.dIe/e0ee d0�d1d2�Z1e d3d
�ee d4�d5d6��Z2e/eee e/f d7�d8d9�Z3ee d:�d;d<�Z4dJee ed>�d?d@�Z5ee
e dA�dBdC�Z6eee7dD�dEdF�Z8eee dD�dGdH�Z9dS )K� N)�IncrementalDecoder)�aliases)� lru_cache)�findall)�List�Optional�Set�Tuple�Union)�MultibyteIncrementalDecoder� )�ENCODING_MARKS�IANA_SUPPORTED_SIMILAR� RE_POSSIBLE_ENCODING_INDICATION�UNICODE_RANGES_COMBINED� UNICODE_SECONDARY_RANGE_KEYWORD�UTF8_MAXIMAL_ALLOCATION)�maxsize)� character�returnc C sT yt j| �}W n tk
r" dS X d|kpRd|kpRd|kpRd|kpRd|kpRd|kS )NFz
WITH GRAVEz
WITH ACUTEz WITH CEDILLAzWITH DIAERESISzWITH CIRCUMFLEXz
WITH TILDE)� unicodedata�name�
ValueError)r � description� r �/usr/lib/python3.6/utils.py�is_accentuated s r c C s. t j| �}|s| S |jd�}tt|d d��S )N� r � )r Z
decomposition�split�chr�int)r Z
decomposedZcodesr r r �
remove_accent) s
r" c C s. t | �}x tj� D ]\}}||kr|S qW dS )zK
Retrieve the Unicode range official name from a single character.
N)�ordr �items)r Z
character_ord�
range_nameZ ord_ranger r r �
unicode_range4 s
r& c C s, yt j| �}W n tk
r" dS X d|kS )NFZLATIN)r r r )r r r r r �is_latinB s
r' c C s( y| j d� W n tk
r" dS X dS )N�asciiFT)�encode�UnicodeEncodeError)r r r r �is_asciiK s
r+ c C s2 t j| �}d|krdS t| �}|d kr*dS d|kS )N�PTFZ Punctuation)r �categoryr& )r �character_category�character_ranger r r �is_punctuationS s
r0 c C s: t j| �}d|ksd|kr dS t| �}|d kr2dS d|kS )N�S�NTFZForms)r r- r& )r r. r/ r r r � is_symbolb s
r3 c C s t | �}|d krdS d|kS )NFZ Emoticons)r& )r r/ r r r � is_emoticonq s r4 c C s& | j � s| d krdS tj| �}d|kS )
N� ｜�+�,�;�<�>T�Z)r5 r6 r7 r8 r9 r: )�isspacer r- )r r. r r r � is_separator{ s
r= c C s | j � | j� kS )N)�islower�isupper)r r r r �is_case_variable� s r@ c C s t j| �}d|kS )NZCo)r r- )r r. r r r �is_private_use_only� s
rA c C s, yt j| �}W n tk
r" dS X d|kS )NFZCJK)r r r )r �character_namer r r �is_cjk� s
rC c C s, yt j| �}W n tk
r" dS X d|kS )NFZHIRAGANA)r r r )r rB r r r � is_hiragana� s
rD c C s, yt j| �}W n tk
r" dS X d|kS )NFZKATAKANA)r r r )r rB r r r � is_katakana� s
rE c C s, yt j| �}W n tk
r" dS X d|kS )NFZHANGUL)r r r )r rB r r r � is_hangul� s
rF c C s, yt j| �}W n tk
r" dS X d|kS )NFZTHAI)r r r )r rB r r r �is_thai� s
rG )r% r c C s xt D ]}|| krdS qW dS )NTF)r )r% �keywordr r r �is_unicode_range_secondary� s
rI � )�sequence� search_zoner c C s� t | t�st�t| �}tt| d||kr*|n|� jddd��}t|�dkrNdS xJ|D ]B}|j� jdd�}x,t j
� D ] \}}||kr�|S ||krr|S qrW qTW dS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
Nr( �ignore)�errorsr �-�_) �
isinstance�bytes� TypeError�lenr r �decode�lower�replacer r$ )rK rL Zseq_len�resultsZspecified_encoding�encoding_alias�
encoding_ianar r r �any_specified_encoding� s"
r[ � )r r c
C s | d kp t tjd
j| ��jt�S ) zQ
Verify is a specific encoding is a multi byte one based on it IANA name
�utf_8� utf_8_sig�utf_16� utf_16_be� utf_16_le�utf_32� utf_32_le� utf_32_be�utf_7z encodings.{}> r_ rd rc ra rb r] r^ r` re )�
issubclass� importlib�
import_module�formatr r )r r r r �is_multi_byte_encoding� s rj )rK r c C sJ xDt D ]<}t | }t|t�r"|g}x |D ]}| j|�r(||fS q(W qW dS )z9
Identify and extract SIG/BOM in given sequence.
N� )Nrk )r
rQ rR �
startswith)rK �
iana_encodingZmarksZmarkr r r �identify_sig_or_bom� s
rn )rm r c C s | dkS )Nr_ rb > rb r_ r )rm r r r �should_strip_sig_or_bom s ro T)�cp_name�strictr c C sP | j � jdd�} x(tj� D ] \}}| |ks2| |kr|S qW |rLtdj| ���| S )NrO rP z Unable to retrieve IANA for '{}')rV rW r r$ r ri )rp rq rY rZ r r r � iana_name s rr )�decoded_sequencer c C s8 t � }x(| D ] }t|�}|d kr"q |j|� q W t|�S )N)�setr& �add�list)rs Zrangesr r/ r r r �
range_scan" s
rw )� iana_name_a� iana_name_br c C s� t | �st |�rdS tjdj| ��j}tjdj|��j}|dd�}|dd�}d}x8tdd�D ]*}t|g�}|j|�|j|�kr\|d7 }q\W |d S ) Ng z encodings.{}rM )rN r � r � )rj rg rh ri r �rangerR rU ) rx ry Z decoder_aZ decoder_bZid_aZid_bZcharacter_match_count�iZ
to_be_decodedr r r �
cp_similarity0 s
r~ c C s | t ko|t | kS )z�
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )rx ry r r r �
is_cp_similarE s r )rJ )T):Z unicodedata2r � ImportErrorrg �codecsr Zencodings.aliasesr � functoolsr �rer Ztypingr r r r r
Z_multibytecodecr Zconstantr
r r r r r �str�boolr r" r& r' r+ r0 r3 r4 r= r@ rA rC rD rE rF rG rT rI rR r! r[ rj rn ro rr rw �floatr~ r r r r r �<module> sh