3
m�da�* � @ sz d dl Z d dlmZ d dlmZmZ d dlmZ d dlm Z m
Z
m Z m Z ddl
mZ ddlmZmZmZ dd lmZ dd
lmZ dd lmZmZmZmZm Z e e
e d �d
d�Z e e
e d�dd�Z e� e e
e d �dd��Z e� e e
e d �dd��Z!d)e
e e"e
e d�dd�Z#e e
e e$d�dd �Z%e e
e d �d d �Z&e
e ed �d!d"�Z'ed#d$�d*e e$e e ed&�d'd(��Z(dS )+� N)�IncrementalDecoder)�Counter� OrderedDict)� lru_cache)�Dict�List�Optional�Tuple� )� FREQUENCIES)�KO_NAMES�TOO_SMALL_SEQUENCE�ZH_NAMES)� is_suspiciously_successive_range)�CoherenceMatches)�is_accentuated�is_latin�is_multi_byte_encoding�is_unicode_range_secondary�
unicode_range)� iana_name�returnc s� t | �rtd��tjdj| ��j}|dd�}i �d� xltdd�D ]^}|jt|g��}|r@t |�}|dkrjq@t
|�d kr@|�kr�d�|< �| d
7 < � d
7 � q@W t � �fd d ��D ��S )
zF
Return associated unicode ranges in a single byte code page.
z.Function not supported on multi-byte code pagez encodings.{}�ignore)�errorsr �@ � NFr
c s g | ]}�| � d kr|�qS )g333333�?� )�.0�character_range)�character_count� seen_rangesr �/usr/lib/python3.6/cd.py�
<listcomp>2 s z*encoding_unicode_range.<locals>.<listcomp>) r �IOError� importlib�
import_module�formatr �range�decode�bytesr r �sorted)r �decoder�p�i�chunkr r )r r r! �encoding_unicode_range s(
r/ )�
primary_ranger c C sD g }x:t j� D ].\}}x$|D ] }t|�| kr |j|� P q W qW |S )z>
Return inferred languages used with a unicode range.
)r �itemsr �append)r0 � languages�language�
characters� characterr r r! �unicode_range_languages9 s
r7 c C s> t | �}d}x|D ]}d|kr|}P qW |dkr6dgS t|�S )z�
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
NZLatinz Latin Based)r/ r7 )r Zunicode_rangesr0 Zspecified_ranger r r! �encoding_languagesH s
r8 c C sb | j d�s&| j d�s&| j d�s&| dkr,dgS | j d�s>| tkrFddgS | j d �sX| tkr^d
gS g S ) z�
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
Zshift_�
iso2022_jpZeuc_j�cp932ZJapaneseZgbZChinesezClassical Chinese�
iso2022_krZKorean)�
startswithr r )r r r r! �mb_encoding_languages\ s
r= F)r5 �ignore_non_latinr c
s� g }d}x� D ]}t |�rd}P qW x�tj� D ]�\}}d}d}x8|D ]0} |dkr\t | �r\d}|dkrDt| �dkrDd}qDW |r�|dkr�q.|dkr�|r�q.t|�}
t� fdd�|D ��} | |
} | dkr.|j|| f� q.W t|dd� dd�}d d� |D �S )
zE
Return associated languages associated to given characters.
FTc s g | ]}|� kr|�qS r r )r �c)r5 r r! r" � s z&alphabet_languages.<locals>.<listcomp>g�������?c S s | d S )Nr
r )�xr r r! �<lambda>� s z$alphabet_languages.<locals>.<lambda>)�key�reversec S s g | ] }|d �qS )r r )r Zcompatible_languager r r! r" � s )r r r1 r �lenr2 r* )
r5 r> r3 Zsource_have_accentsr6 r4 Zlanguage_charactersZtarget_have_accentsZtarget_pure_latinZlanguage_characterr Zcharacter_match_count�ratior )r5 r! �alphabet_languagesq s4
rF )r4 �ordered_charactersr c s6 | t krtdj| ���d}�x |D �]}|t | kr6q"t | dt | j|�� }t | t | j|�d� }|d|j|�� �||j|�d� � �fdd�|D �jd�}� fdd�|D �jd�}t|�dkr�|dkr�|d 7 }q"t|�dkr�|dkr�|d 7 }q"|t|� d
k�s |t|� d
kr"|d 7 }q"q"W |t|� S ) aN
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
z{} not availabler Nc s g | ] }|� k�qS r r )r �e)�characters_beforer r! r" � s z1characters_popularity_compare.<locals>.<listcomp>Tc s g | ] }|� k�qS r r )r rH )�characters_afterr r! r" � s � r
g�������?)r �
ValueErrorr& �index�countrD )r4 rG Zcharacter_approved_countr6 Zcharacters_before_sourceZcharacters_after_sourceZbefore_match_countZafter_match_countr )rJ rI r! � characters_popularity_compare� s: rO )�decoded_sequencer c C s� t � }x�| D ]�}|j� dkr q t|�}|dkr0q d}x |D ]}t||�dkr:|}P q:W |dkrb|}||krx|j� ||< q || |j� 7 < q W t|j� �S )a
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
One containing the latin letters and the other hebrew.
FN)r �isalphar r �lower�list�values)rP Zlayersr6 r Zlayer_target_rangeZdiscovered_ranger r r! �alpha_unicode_split� s(
rU )�resultsr c C s� t � }g }xD| D ]<}x6|D ].}|\}}||kr:|g||< q|| j|� qW qW x4|D ],}|j|tt|| �t|| � d�f� qVW t|dd� dd�S )z�
This function merge results previously given by the function coherence_ratio.
The return type is the same as coherence_ratio.
rK c S s | d S )Nr
r )r@ r r r! rA s z(merge_coherence_ratios.<locals>.<lambda>T)rB rC )r r2 �round�sumrD r* )rV Zper_language_ratios�merge�resultZ
sub_resultr4 rE r r r! �merge_coherence_ratios s"
r[ i )�maxsize皙�����?)rP � threshold� lg_inclusionr c C s� g }g }d}d}|dk r"|j d�}d|kr8d}|jd� x�t| �D ]�}t|�}|j� } tdd� | D ��}
|
tkrrqBd d� | D �} xZ|p�t| |�D ]H} t| | �}
|
|k r�q�n|
d
kr�|d 7 }|j | t
|
d �f� |d
kr�P q�W qBW t |dd� dd�S )z�
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
A layer = Character extraction by alphabets/ranges.
Fr N�,z Latin BasedTc S s g | ] \}}|�qS r r )r r? �or r r! r" = s z#coherence_ratio.<locals>.<listcomp>c S s g | ] \}}|�qS r r )r r? ra r r r! r" B s g�������?r
rK � c S s | d S )Nr
r )r@ r r r! rA U s z!coherence_ratio.<locals>.<lambda>)rB rC ) �split�removerU r � most_commonrX r
rF rO r2 rW r* )rP r^ r_ rV Zlg_inclusion_listr> Zsufficient_match_countZlayerZsequence_frequenciesre r Zpopular_character_orderedr4 rE r r r! �coherence_ratio# s8
rf )F)r] N))r$ �codecsr � collectionsr r � functoolsr Ztypingr r r r Zassetsr Zconstantr r
r Zmdr Zmodelsr Zutilsr r r r r �strr/ r7 r8 r= �boolrF �floatrO rU r[ rf r r r r! �<module> s0 %
/:' | Name | Type | Size | Permission | Actions |
|---|---|---|---|---|
| __init__.cpython-36.opt-1.pyc | File | 1.54 KB | 0644 |
|
| __init__.cpython-36.pyc | File | 1.54 KB | 0644 |
|
| api.cpython-36.opt-1.pyc | File | 9.35 KB | 0644 |
|
| api.cpython-36.pyc | File | 9.35 KB | 0644 |
|
| cd.cpython-36.opt-1.pyc | File | 7.96 KB | 0644 |
|
| cd.cpython-36.pyc | File | 7.96 KB | 0644 |
|
| constant.cpython-36.opt-1.pyc | File | 16.51 KB | 0644 |
|
| constant.cpython-36.pyc | File | 16.51 KB | 0644 |
|
| legacy.cpython-36.opt-1.pyc | File | 3.18 KB | 0644 |
|
| legacy.cpython-36.pyc | File | 3.18 KB | 0644 |
|
| md.cpython-36.opt-1.pyc | File | 14.4 KB | 0644 |
|
| md.cpython-36.pyc | File | 14.4 KB | 0644 |
|
| models.cpython-36.opt-1.pyc | File | 12.84 KB | 0644 |
|
| models.cpython-36.pyc | File | 12.84 KB | 0644 |
|
| utils.cpython-36.opt-1.pyc | File | 7.32 KB | 0644 |
|
| utils.cpython-36.pyc | File | 7.32 KB | 0644 |
|
| version.cpython-36.opt-1.pyc | File | 197 B | 0644 |
|
| version.cpython-36.pyc | File | 197 B | 0644 |
|