3
m�da�B � @ s� d dl mZmZ d dlmZmZmZmZ yd dlm Z W n e
k
rP e Z Y nX d dl Z ddl
mZmZmZmZ ddlmZmZmZ ddlmZ dd lmZmZ dd
lm Z m Z m Z m Z m Z m!Z! e j"d �Z#e#j$e j%� e j&� Z'e'j(e j)d �� e#j*e'� d e+e,e,e-ee ee e.e.ed� dd�Z/d ee,e,e-ee ee e.e.ed� dd�Z0d e e,e,e-ee ee e.e.ed� dd�Z1d!e e,e,e-ee ee e.ed�d d �Z2dS )"� )�basename�splitext)�BinaryIO�List�Optional�Set)�PathLikeN� )�coherence_ratio�encoding_languages�mb_encoding_languages�merge_coherence_ratios)�IANA_SUPPORTED�TOO_BIG_SEQUENCE�TOO_SMALL_SEQUENCE)�
mess_ratio)� CharsetMatch�CharsetMatches)�any_specified_encoding� iana_name�identify_sig_or_bom�
is_cp_similar�is_multi_byte_encoding�should_strip_sig_or_bomZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s� � 皙�����?TF) � sequences�steps�
chunk_size� threshold� cp_isolation� cp_exclusion�preemptive_behaviour�explain�returnc / C s t | ttf�s tdjt| ����|s2tjtj � n tjtj
� t | �}|dkrptj d� t
t| dddg d�g�S |dk r�tj d d
j|�� d d � |D �}ng }|dk r�tj d
d
j|�� dd � |D �}ng }||| kr�tj d|||� d}|}|dk�r|| |k �rt|| �}t | �tk } t | �tk}
| �rDtj dj|�� n|
�rZtjdj|�� g } |dk�rpt| �nd} | dk �r�| j| � tjd| � t� }
g }g }d}d}d}t
� }t| �\}}|dk �r�| j|� tjdt |�|� | jd� d| k�r| jd� �xJ| t D �]<}|�r*||k�r*�q|�r>||k�r>�q||
k�rL�q|
j|� d}||k}|�ont|�}|d5k�r�|dk�r�tjd|� �qy t|�}W n* t t fk
�r� tj d|� �wY nX yr|
�r|dk�rt |dk�r�| dtd�� n| t |�td�� |d � n&t |dk�r&| n| t |�d� |d �}W nT t t!fk
�r� } z2t |t!��sttj d |t |�� |j|� �wW Y dd}~X nX d} x |D ]} t"|| ��r�d} P �q�W | �r�tj d || � �qt#|dk�r�dnt |�|t|| ��} |�o|dk �ot |�|k } | �r&tjd |� tt | �d �} | d!k �rDd!} d}!g }"g }#�x@| D �]6}$| |$|$| � }%|�r�|dk�r�||% }%|%j$|d"d#�}&|�rB|$dk�rB| |$ d$k�rB|d%k�r�d%n|}'|�rB|&d|'� |k�rBxdt#|$|$d d6�D ]P}(| |(|$| � }%|�r|dk�r||% }%|%j$|d"d#�}&|&d|'� |k�r�P �q�W |"j|&� |#jt%|&|�� |#d7 |k�rr|!d7 }!|!| k�s�|�rX|dk�rXP �qXW |#�r�t&|#�t |#� })nd})|)|k�s�|!| k�r4|j|� tj d&||!t'|)d' d(d)�� |dd| gk�rt| ||dg |�}*|| k�r |*}n|dk�r,|*}n|*}�qtjd*|t'|)d' d(d)�� |�s^t(|�}+nt)|�}+|+�r�tjd+j|t |+��� g },x4|"D ],}&t*|&d,|+�r�d-j|+�nd�}-|,j|-� �q�W t+|,�}.|.�r�tjd.j|.|�� |jt| ||)||.|�� || ddgk�r(|)d,k �r(tjd/|� t
|| g�S ||k�rtjd0|� t
|| g�S �qW t |�dk�r |�sr|�sr|�r|tj d1� |�r�tj d2|j,� |j|� nd|�r�|dk�s�|�r�|�r�|j-|j-k�s�|dk �r�tj d3� |j|� n|�r tj d4� |j|� |S )8aD
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
If there is no results, it is a strong indicator that the source is binary/not text.
By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence.
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
but never take it for granted. Can improve the performance.
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
purpose.
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
z4Expected object of type bytes or bytearray, got: {0}r zXGiven content is empty, stopping the process very early, returning empty utf_8 str match�utf_8g F� Nz`cp_isolation is set. use this flag for debugging purpose. limited list of encoding allowed : %s.z, c S s g | ]}t |d ��qS )F)r )�.0�cp� r* �/usr/lib/python3.6/api.py�
<listcomp>X s z from_bytes.<locals>.<listcomp>zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c S s g | ]}t |d ��qS )F)r )r( r) r* r* r+ r, b s z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).Tz@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.�ascii�utf_16�utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecoderg �� A)�encodingz9Code page %s does not fit given bytes sequence at ALL. %szW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.� � �ignore)�errors� � zc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.�d � )Zndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g�������?�,z We detected language {} using {}z0%s is most likely the one. Stopping the process.z[%s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z#%s will be used as a fallback matchz&utf_8 will be used as a fallback matchz&ascii will be used as a fallback match> r/ r. ���r: ).�
isinstance� bytearray�bytes� TypeError�format�type�logger�setLevel�loggingZCRITICAL�INFO�lenZwarningr r �join�intr r �infor �append�setr r �addr r �ModuleNotFoundError� ImportError�debug�str�UnicodeDecodeError� LookupErrorr �range�decoder �sum�roundr r r
r
r0 Z fingerprint)/r r r r r! r" r# r$ ZlengthZis_too_small_sequenceZis_too_large_sequenceZprioritized_encodingsZspecified_encodingZtestedZtested_but_hard_failureZtested_but_soft_failureZfallback_asciiZ fallback_u8Zfallback_specified�resultsZ sig_encodingZ sig_payloadZ
encoding_ianaZdecoded_payloadZbom_or_sig_availableZstrip_sig_or_bomZis_multi_byte_decoder�eZsimilar_soft_failure_testZencoding_soft_failedZr_Zmulti_byte_bonusZmax_chunk_gave_upZearly_stop_countZ md_chunksZ md_ratios�iZ cut_sequence�chunkZchunk_partial_size_chk�jZmean_mess_ratioZfallback_entryZtarget_languagesZ cd_ratiosZchunk_languagesZcd_ratios_mergedr* r* r+ �
from_bytes% s�
r[ ) �fpr r r r! r" r# r$ r% c C s t | j� |||||||�S )z�
Same thing than the function from_bytes but using a file pointer that is already ready.
Will not close the file pointer.
)r[ �read)r\ r r r r! r" r# r$ r* r* r+ �from_fp� s r^ ) �pathr r r r! r" r# r$ r% c C s, t | d��}t||||||||�S Q R X dS )z�
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
Can raise IOError.
�rbN)�openr^ ) r_ r r r r! r" r# r$ r\ r* r* r+ � from_path� s rb )r_ r r r r! r" r# r% c C s� t | ||||||�}t| �}tt|��} t|�dkrBtdj|���|j� }
| d d|
j 7 < t djt
| �j |dj | ���d��} | j
|
j� � W dQ R X |
S )zi
Take a (text-based) file path and try to create another file next to it, this time using UTF-8.
r z;Unable to normalize "{}", no encoding charset seems to fit.�-z{}r' �wbN)rb r �listr rE �IOErrorr? Zbestr0 ra rO �replacerF �write�output) r_ r r r r! r" r# rV �filenameZtarget_extensions�resultr\ r* r* r+ � normalize� s* rl )r r r NNTF)r r r NNTF)r r r NNTF)r r r NNT)3Zos.pathr r Ztypingr r r r �osr rM rO rC Zcdr
r r r
Zconstantr r r Zmdr Zmodelsr r Zutilsr r r r r r Z getLoggerrA rB �DEBUGZ
StreamHandlerZhandlerZ setFormatterZ FormatterZ
addHandlerr= rG �float�boolr[ r^ rb rl r* r* r* r+ �<module> sd