3
m�da C � @ s< d dl mZ d dlmZmZ ddlmZmZ ddlm Z m
Z
m Z m Z m
Z
mZmZmZmZmZmZmZmZmZmZ G dd� d�ZG dd � d e�ZG d
d � d e�ZG d d
� d
e�ZG dd� de�Z G dd� de�Z G dd� de�Z G dd� de�Z G dd� de�Z ee! ee! e"d�dd�Z#edd �d#e!e$e"e$d �d d!��Z%d"S )$� )� lru_cache)�List�Optional� )� COMMON_SAFE_ASCII_CHARACTERS� UNICODE_SECONDARY_RANGE_KEYWORD)�is_accentuated�is_ascii�is_case_variable�is_cjk� is_emoticon� is_hangul� is_hiragana� is_katakana�is_latin�is_punctuation� is_separator� is_symbol�is_thai�
remove_accent�
unicode_rangec @ sP e Zd ZdZeed�dd�Zedd�dd�Zdd�d d
�Ze e
d�d d ��Z dS )
�MessDetectorPluginzy
Base abstract class used for mess detection plugins.
All detectors MUST extend and implement given methods.
)� character�returnc C s t �dS )z@
Determine if given character should be fed in.
N)�NotImplementedError)�selfr � r �/usr/lib/python3.6/md.py�eligible s zMessDetectorPlugin.eligibleNc C s t �dS )z�
The main routine to be executed upon character.
Insert the logic in witch the text would be considered chaotic.
N)r )r r r r r �feed$ s zMessDetectorPlugin.feed)r c C s t �dS )zB
Permit to reset the plugin to the initial state.
N)r )r r r r �reset+ s zMessDetectorPlugin.resetc C s t �dS )z�
Compute the chaos ratio based on what your feed() has seen.
Must NOT be lower than 0.; No restriction gt 0.
N)r )r r r r �ratio1 s zMessDetectorPlugin.ratio) �__name__�
__module__� __qualname__�__doc__�str�boolr r r �property�floatr! r r r r r s r c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d
d �Ze e
d�d d
��Z dS )� TooManySymbolOrPunctuationPluginN)r c C s" d| _ d| _d| _d | _d| _d S )Nr F)�_punctuation_count�
_symbol_count�_character_count�_last_printable_charZ_frenzy_symbol_in_word)r r r r �__init__; s
z)TooManySymbolOrPunctuationPlugin.__init__)r r c C s |j � S )N)� isprintable)r r r r r r C s z)TooManySymbolOrPunctuationPlugin.eligiblec C sp | j d7 _ || jkrf|tkrft|�r8| jd7 _n.|j� dkrft|�rft|�dkrf| jd7 _|| _d S )Nr F� ) r- r. r r r+ �isdigitr r r, )r r r r r r F s
z%TooManySymbolOrPunctuationPlugin.feedc C s d| _ d| _d| _d S )Nr )r+ r- r, )r r r r r X s z&TooManySymbolOrPunctuationPlugin.resetc C s0 | j dkrdS | j| j | j }|dkr,|S dS )Nr g g333333�?)r- r+ r, )r Zratio_of_punctuationr r r r! ] s
z&TooManySymbolOrPunctuationPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r* : s r* c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d
d �Ze e
d�d d
��Z dS )�TooManyAccentuatedPluginN)r c C s d| _ d| _d S )Nr )r- �_accentuated_count)r r r r r/ j s z!TooManyAccentuatedPlugin.__init__)r r c C s |j � S )N)�isalpha)r r r r r r n s z!TooManyAccentuatedPlugin.eligiblec C s( | j d7 _ t|�r$| jd7 _d S )Nr )r- r r4 )r r r r r r q s z TooManyAccentuatedPlugin.feedc C s d| _ d| _d S )Nr )r- r4 )r r r r r w s z TooManyAccentuatedPlugin.resetc C s* | j dkrdS | j| j }|dkr&|S dS )Nr g gffffff�?)r- r4 )r Zratio_of_accentuationr r r r! { s
z TooManyAccentuatedPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r3 i s r3 c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d
d �Ze e
d�d d
��Z dS )�UnprintablePluginN)r c C s d| _ d| _d S )Nr )�_unprintable_countr- )r r r r r/ � s zUnprintablePlugin.__init__)r r c C s dS )NTr )r r r r r r � s zUnprintablePlugin.eligiblec C s@ |j � dkr.|j� dkr.|dkr.| jd7 _| jd7 _d S )NF�r )�isspacer0 r7 r- )r r r r r r � s
zUnprintablePlugin.feedc C s
d| _ d S )Nr )r7 )r r r r r � s zUnprintablePlugin.resetc C s | j dkrdS | jd | j S )Nr g � )r- r7 )r r r r r! � s
zUnprintablePlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r6 � s r6 c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d
d �Ze e
d�d d
��Z dS )� SuspiciousDuplicateAccentPluginN)r c C s d| _ d| _d | _d S )Nr )�_successive_countr- �_last_latin_character)r r r r r/ � s z(SuspiciousDuplicateAccentPlugin.__init__)r r c C s |j � ot|�S )N)r5 r )r r r r r r � s z(SuspiciousDuplicateAccentPlugin.eligiblec C st | j d7 _ | jd k rjt|�rjt| j�rj|j� rJ| jj� rJ| jd7 _t|�t| j�krj| jd7 _|| _d S )Nr )r- r= r �isupperr< r )r r r r r r � s
z$SuspiciousDuplicateAccentPlugin.feedc C s d| _ d| _d | _d S )Nr )r< r- r= )r r r r r � s z%SuspiciousDuplicateAccentPlugin.resetc C s | j dkrdS | jd | j S )Nr g r1 )r- r< )r r r r r! � s
z%SuspiciousDuplicateAccentPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r; � s
r; c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d
d �Ze e
d�d d
��Z dS )�SuspiciousRangeN)r c C s d| _ d| _d | _d S )Nr )�"_suspicious_successive_range_countr- �_last_printable_seen)r r r r r/ � s zSuspiciousRange.__init__)r r c C s |j � S )N)r0 )r r r r r r � s zSuspiciousRange.eligiblec C sx | j d7 _ |j� s&t|�s&|tkr0d | _d S | jd krD|| _d S t| j�}t|�}t||�rn| jd7 _|| _d S )Nr )r- r9 r r rA r � is_suspiciously_successive_ranger@ )r r �unicode_range_a�unicode_range_br r r r � s
zSuspiciousRange.feedc C s d| _ d| _d | _d S )Nr )r- r@ rA )r r r r r � s zSuspiciousRange.resetc C s. | j dkrdS | jd | j }|dk r*dS |S )Nr g r1 g�������?)r- r@ )r Z ratio_of_suspicious_range_usager r r r! � s
zSuspiciousRange.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r? � s r? c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d
d �Ze e
d�d d
��Z dS )�SuperWeirdWordPluginN)r c C s4 d| _ d| _d| _d| _d| _d| _d| _d| _d S )Nr F� )� _word_count�_bad_word_count�_is_current_word_bad�_foreign_long_watchr- �_bad_character_count�_buffer�_buffer_accent_count)r r r r r/ � s z SuperWeirdWordPlugin.__init__)r r c C s dS )NTr )r r r r r r s z SuperWeirdWordPlugin.eligiblec C s� |j � r�dj| j|g�| _t|�r0| jd7 _| jdkr�t|�dkr�t|�dkr�t|�dkr�t |�dkr�t
|�dkr�t |�dkr�d| _d S | js�d S |j � s�t
|�s�t|�o�| j�rT| jd7 _t| j�}| j|7 _|dkr�| j| dkr�d| _|dk�r| j�rd| _| j�r@| jd7 _| jt| j�7 _d| _d| _d| _d| _n6|d
k�r�|j� dk�r�t|��r�d| _| j|7 _d S )NrF r FT� g�(\���?� r �<�>�-�=> rR rS rQ rP )r5 �joinrL r rM rJ r r r
r r r r9 r r rG �lenr- rI rH rK r2 r )r r Z
buffer_lengthr r r r sH
zSuperWeirdWordPlugin.feedc C s. d| _ d| _d| _d| _d| _d| _d| _d S )NrF Fr )rL rI rJ rH rG r- rK )r r r r r : s zSuperWeirdWordPlugin.resetc C s | j dkrdS | j| j S )N�
g )rG rK r- )r r r r r! C s
zSuperWeirdWordPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r rE � s / rE c @ s^ e Zd ZdZdd�dd�Zeed�dd�Zedd�d d
�Zdd�d d �Z e
e d�d
d��Z dS )�CjkInvalidStopPluginu�
GB(Chinese) based encoding often render the stop incorrectly when the content does not fit and
can be easily detected. Searching for the overuse of '丅' and '丄'.
N)r c C s d| _ d| _d S )Nr )�_wrong_stop_count�_cjk_character_count)r r r r r/ Q s z CjkInvalidStopPlugin.__init__)r r c C s dS )NTr )r r r r r r U s z CjkInvalidStopPlugin.eligiblec C s4 |dkr| j d7 _ d S t|�r0| jd7 _d S )N� 丅� 丄r )rZ r[ )rX r rY )r r r r r r X s
zCjkInvalidStopPlugin.feedc C s d| _ d| _d S )Nr )rX rY )r r r r r _ s zCjkInvalidStopPlugin.resetc C s | j dk rdS | j| j S )N� g )rY rX )r r r r r! c s
zCjkInvalidStopPlugin.ratio)
r" r# r$ r% r/ r& r' r r r r( r) r! r r r r rW K s rW c @ sZ e Zd Zdd�dd�Zeed�dd�Zedd�dd �Zdd�d
d �Ze e
d�d d
��Z dS )�ArchaicUpperLowerPluginN)r c C s. d| _ d| _d| _d| _d| _d | _d| _d S )NFr T)�_buf� _character_count_since_last_sep� _successive_upper_lower_count�#_successive_upper_lower_count_finalr- �_last_alpha_seen�_current_ascii_only)r r r r r/ k s z ArchaicUpperLowerPlugin.__init__)r r c C s dS )NTr )r r r r r r x s z ArchaicUpperLowerPlugin.eligiblec C s$ |j � ot|�}|dk}|r�| jdkr�| jdkrV|j� dkrV| jdkrV| j| j7 _d| _d| _d | _d| _| j d7 _ d| _d S | jdkr�t
|�dkr�d| _| jd k r�|j � r�| jj � s�|j � r�| jj � r�| jdkr�| jd7 _d| _q�d| _nd| _| j d7 _ | jd7 _|| _d S )NFr �@ r Tr1 )
r5 r
r_ r2 rc ra r` rb r^ r- r r> �islower)r r Z is_concernedZ chunk_sepr r r r { s8
z ArchaicUpperLowerPlugin.feedc C s. d| _ d| _d| _d| _d | _d| _d| _d S )Nr FT)r- r_ r` ra rb r^ rc )r r r r r � s z ArchaicUpperLowerPlugin.resetc C s | j dkrdS | j| j S )Nr g )r- ra )r r r r r! � s
z ArchaicUpperLowerPlugin.ratio) r" r# r$ r/ r& r' r r r r( r) r! r r r r r] j s
* r] )rC rD r c C sT | dks|dkrdS | |kr dS d| kr4d|kr4dS d| ksDd|krHdS | j d�|j d� }}x"|D ]}|tkrrqd||krddS qdW | dk|dk }}|s�|r�d | ks�d |kr�dS |r�|r�dS d
| ks�d
|k� r�d | ks�d |kr�dS | d kp�|d k� r�dS d | k�s d |k�s | dk�rP|dk�rPd | k�s4d |k�r8dS d
| k�sLd
|k�rPdS dS )za
Determine if two Unicode range seen next to each other can be considered as suspicious.
NTFZLatinZ Emoticons� �Hiragana�KatakanaZCJKZHangulz Basic LatinZ PunctuationZForms)rg rh )rg rh )rh rg )rh rg )�splitr )rC rD Zkeywords_range_aZkeywords_range_bZelZrange_a_jp_charsZrange_b_jp_charsr r r rB � sL
rB i )�maxsize皙�����?F)�decoded_sequence�maximum_threshold�debugr c C s� dd� t j� D �}t| �}d}|dk r,d}n|dkr:d}nd}xzt| td |��D ]f\}}x |D ]} | j|�r^| j|� q^W |d kr�|| d ks�||d
krPtd d� |D ��}||krPP qPW |r�x|D ]}
t|
j |
j
� q�W t |d �S )
zw
Compute a mess ratio given a decoded bytes sequence. The maximum threshold does stop the computation earlier.
c S s g | ]
}|� �qS r r )�.0Zmd_classr r r �
<listcomp>� s z mess_ratio.<locals>.<listcomp>g i � i rd � r r c S s g | ]
}|j �qS r )r! )ro �dtr r r rp s � ) r �__subclasses__rU �zip�ranger r �sum�print� __class__r! �round) rl rm rn Z detectorsZlengthZmean_mess_ratioZ!intermediary_mean_mess_ratio_calcr �indexZdetectorrs r r r �
mess_ratio� s*
r} N)rk F)&� functoolsr Ztypingr r Zconstantr r Zutilsr r r
r r r
r r r r r r r r r r r* r3 r6 r; r? rE rW r] r& r' rB r) r} r r r r �<module> s" D"/ $6P M <| Name | Type | Size | Permission | Actions |
|---|---|---|---|---|
| __init__.cpython-36.opt-1.pyc | File | 1.54 KB | 0644 |
|
| __init__.cpython-36.pyc | File | 1.54 KB | 0644 |
|
| api.cpython-36.opt-1.pyc | File | 9.35 KB | 0644 |
|
| api.cpython-36.pyc | File | 9.35 KB | 0644 |
|
| cd.cpython-36.opt-1.pyc | File | 7.96 KB | 0644 |
|
| cd.cpython-36.pyc | File | 7.96 KB | 0644 |
|
| constant.cpython-36.opt-1.pyc | File | 16.51 KB | 0644 |
|
| constant.cpython-36.pyc | File | 16.51 KB | 0644 |
|
| legacy.cpython-36.opt-1.pyc | File | 3.18 KB | 0644 |
|
| legacy.cpython-36.pyc | File | 3.18 KB | 0644 |
|
| md.cpython-36.opt-1.pyc | File | 14.4 KB | 0644 |
|
| md.cpython-36.pyc | File | 14.4 KB | 0644 |
|
| models.cpython-36.opt-1.pyc | File | 12.84 KB | 0644 |
|
| models.cpython-36.pyc | File | 12.84 KB | 0644 |
|
| utils.cpython-36.opt-1.pyc | File | 7.32 KB | 0644 |
|
| utils.cpython-36.pyc | File | 7.32 KB | 0644 |
|
| version.cpython-36.opt-1.pyc | File | 197 B | 0644 |
|
| version.cpython-36.pyc | File | 197 B | 0644 |
|