3
m�da4 � @ s� d dl Z d dlmZ d dlmZ d dlmZ d dlmZ d dl m
Z
d dl m Z m
Z
mZmZmZmZmZ dd lmZmZ dd
lmZ dd lmZmZmZ G d d
� d
�Z G dd� d�Z ee e f Z ee Z!G dd� d�Z"dS )� N)�Counter)�aliases)�sha256)�dumps)�sub)�Any�Dict�Iterator�List�Optional�Tuple�Union� )�NOT_PRINTABLE_PATTERN�TOO_BIG_SEQUENCE)�
mess_ratio)� iana_name�is_multi_byte_encoding�
unicode_rangec @ s e Zd ZdBeeeedee d�dd�Ze ed�dd�Z
e ed�d d
�Z e ed �d d
��Z
e ed �dd��Ze ed �dd��Ze ed �dd��Zed �dd�Zed �dd�Zd dd�dd�Ze ed �dd��Ze ee d �d d ��Ze ed �d d ��Ze ed �d d!��Ze ee d �d"d#��Ze ed �d$d%��Ze ed �d&d'��Z e ed �d(d)��Z e ed �d*d+��Z e ed �d,d-��Z e ed �d.d/��Z e ed d �d0d1��Z!e ed �d2d3��Z"e ee d �d4d5��Z#e ee d �d6d7��Z$d d �d8d9�Z%d d �d:d;�Z&dCeed=�d>d?�Z'e ed �d@dA��Z(dS )D� CharsetMatchN�CoherenceMatches)�payload�guessed_encoding�mean_mess_ratio�has_sig_or_bom� languages�decoded_payloadc C sF || _ || _|| _|| _|| _d | _g | _d| _d | _d | _ || _
d S )Ng ) �_payload� _encoding�_mean_mess_ratio�
_languages�_has_sig_or_bom�_unicode_ranges�_leavesZ_mean_coherence_ratio�_output_payload�_output_encoding�_string)�selfr r r r r r � r( � /usr/lib/python3.6/models.py�__init__ s zCharsetMatch.__init__)�other�returnc C s> t |t�s&tdjt|j�t| j����| j|jko<| j|jkS )Nz&__eq__ cannot be invoked on {} and {}.)�
isinstancer � TypeError�format�str� __class__�encoding� fingerprint)r' r+ r( r( r) �__eq__( s
zCharsetMatch.__eq__c C sv t |t�st�t| j|j �}t| j|j �}|dk rj|dkrj|dkr^| j|jkr^| j|jkS | j|jkS | j|jk S )zQ
Implemented to make sorted available upon CharsetMatches items.
g{�G�z�?g{�G�z�?g )r- r �
ValueError�abs�chaos� coherence�multi_byte_usage)r' r+ Zchaos_differenceZcoherence_differencer( r( r) �__lt__1 s
zCharsetMatch.__lt__)r, c C s dt t| ��t | j� S )Ng �?)�lenr0 �raw)r' r( r( r) r9 D s z CharsetMatch.multi_byte_usagec C s t jdt� tt| �d�S )z�
Check once again chaos in decoded text, except this time, with full content.
Use with caution, this can be very slow.
Notice: Will be removed in 3.0
z=chaos_secondary_pass is deprecated and will be removed in 3.0g �?)�warnings�warn�DeprecationWarningr r0 )r' r( r( r) �chaos_secondary_passH s z!CharsetMatch.chaos_secondary_passc C s t jdt� dS )zy
Coherence ratio on the first non-latin language detected if ANY.
Notice: Will be removed in 3.0
z<coherence_non_latin is deprecated and will be removed in 3.0g )r= r> r? )r' r( r( r) �coherence_non_latinU s z CharsetMatch.coherence_non_latinc C s, t jdt� ttdt| �j� �}t|j� �S )z_
Word counter instance on decoded text.
Notice: Will be removed in 3.0
z2w_counter is deprecated and will be removed in 3.0� ) r= r> r? r r r0 �lowerr �split)r' Zstring_printable_onlyr( r( r) � w_countera s zCharsetMatch.w_counterc C s" | j d kr t| j| jd�| _ | j S )N�strict)r& r0 r r )r' r( r( r) �__str__o s
zCharsetMatch.__str__c C s dj | j| j�S )Nz <CharsetMatch '{}' bytes({})>)r/ r2 r3 )r' r( r( r) �__repr__u s zCharsetMatch.__repr__c C s: t |t� s|| kr$tdj|j���d |_| jj|� d S )Nz;Unable to add instance <{}> as a submatch of a CharsetMatch)r- r r5 r/ r1 r& r# �append)r' r+ r( r( r) � add_submatchx s
zCharsetMatch.add_submatchc C s | j S )N)r )r' r( r( r) r2 � s zCharsetMatch.encodingc C sH g }x>t j� D ]2\}}| j|kr,|j|� q| j|kr|j|� qW |S )z�
Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855.
)r �itemsr2 rI )r' Z
also_known_as�u�pr( r( r) �encoding_aliases� s
z CharsetMatch.encoding_aliasesc C s | j S )N)r! )r' r( r( r) �bom� s zCharsetMatch.bomc C s | j S )N)r! )r' r( r( r) �byte_order_mark� s z CharsetMatch.byte_order_markc C s dd� | j D �S )z�
Return the complete list of possible languages found in decoded sequence.
Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'.
c S s g | ] }|d �qS )r r( )�.0�er( r( r) �
<listcomp>� s z*CharsetMatch.languages.<locals>.<listcomp>)r )r' r( r( r) r � s zCharsetMatch.languagesc C sp | j sbd| jkrdS ddlm}m} t| j�r8|| j�n|| j�}t|�dksVd|krZdS |d S | j d d S )z�
Most probable language found in decoded sequence. If none were detected or inferred, the property will return
"Unknown".
�asciiZEnglishr )�encoding_languages�mb_encoding_languagesz Latin BasedZUnknown)r �could_be_from_charsetZcharset_normalizer.cdrU rV r r2 r; )r' rU rV r r( r( r) �language� s
zCharsetMatch.languagec C s | j S )N)r )r' r( r( r) r7 � s zCharsetMatch.chaosc C s | j s
dS | j d d S )Ng r r )r )r' r( r( r) r8 � s zCharsetMatch.coherencec C s t | jd dd�S )N�d � )�ndigits)�roundr7 )r' r( r( r) �
percent_chaos� s zCharsetMatch.percent_chaosc C s t | jd dd�S )NrY rZ )r[ )r\ r8 )r' r( r( r) �percent_coherence� s z CharsetMatch.percent_coherencec C s | j S )z+
Original untouched bytes.
)r )r' r( r( r) r< � s zCharsetMatch.rawc C s | j S )N)r# )r' r( r( r) �submatch� s zCharsetMatch.submatchc C s t | j�dkS )Nr )r; r# )r' r( r( r) � has_submatch� s zCharsetMatch.has_submatchc C s@ | j d k r| j S dd� t| �D �}ttdd� |D ���| _ | j S )Nc S s g | ] }t |��qS r( )r )rQ �charr( r( r) rS � s z*CharsetMatch.alphabets.<locals>.<listcomp>c S s h | ] }|r|�qS r( r( )rQ �rr( r( r) � <setcomp>� s z)CharsetMatch.alphabets.<locals>.<setcomp>)r" r0 �sorted�list)r' Zdetected_rangesr( r( r) � alphabets� s
zCharsetMatch.alphabetsc C s | j gdd� | jD � S )z�
The complete list of encoding that output the exact SAME str result and therefore could be the originating
encoding.
This list does include the encoding available in property 'encoding'.
c S s g | ]
}|j �qS r( )r2 )rQ �mr( r( r) rS � s z6CharsetMatch.could_be_from_charset.<locals>.<listcomp>)r r# )r' r( r( r) rW � s z"CharsetMatch.could_be_from_charsetc C s | S )z>
Kept for BC reasons. Will be removed in 3.0.
r( )r' r( r( r) �first� s zCharsetMatch.firstc C s | S )z>
Kept for BC reasons. Will be removed in 3.0.
r( )r' r( r( r) �best� s zCharsetMatch.best�utf_8)r2 r, c C s2 | j dks| j |kr,|| _ t| �j|d�| _| jS )z�
Method to get re-encoded bytes payload using given target encoding. Default to UTF-8.
Any errors will be simply ignored by the encoder NOT replaced.
N�replace)r% r0 �encoder$ )r' r2 r( r( r) �output s zCharsetMatch.outputc C s t | j� �j� S )zw
Retrieve the unique SHA256 computed using the transformed (re-encoded) payload. Not the original one.
)r rm Z hexdigest)r' r( r( r) r3
s zCharsetMatch.fingerprint)N)rj ))�__name__�
__module__� __qualname__�bytesr0 �float�boolr r* �objectr4 r: �propertyr9 r@ rA r rE rG rH rJ r2 r
rN rO rP r rX r7 r8 r] r^ r< r_ r` rf rW rh ri rm r3 r( r( r( r) r sb
r c @ s� e Zd ZdZdee d�dd�Zee d�dd�Ze e
e f ed �d
d �Z e
d�d d
�Z
ed�dd�Zedd �dd�Zed d�dd�Zed d�dd�ZdS )�CharsetMatchesz�
Container with every CharsetMatch items ordered by default from most probable to the less one.
Act like a list(iterable) but does not implements all related methods.
N)�resultsc C s |r t |�ng | _d S )N)rd �_results)r' rw r( r( r) r* s zCharsetMatches.__init__)r, c c s x| j D ]
}|V qW d S )N)rx )r' �resultr( r( r) �__iter__ s zCharsetMatches.__iter__)�itemr, c C sN t |t�r| j| S t |t�rFt|d�}x | jD ]}||jkr0|S q0W t�dS )z�
Retrieve a single item either by its position or encoding name (alias may be used here).
Raise KeyError upon invalid index or encoding not present in results.
FN)r- �intrx r0 r rW �KeyError)r' r{ ry r( r( r) � __getitem__" s
zCharsetMatches.__getitem__c C s
t | j�S )N)r; rx )r' r( r( r) �__len__0 s zCharsetMatches.__len__c C s t | j�dkS )Nr )r; rx )r' r( r( r) �__bool__3 s zCharsetMatches.__bool__c C s~ t |t�s tdjt|j����t|j�tkrbx4| j D ]*}|j
|j
kr4|j |j kr4|j |� dS q4W | j j
|� t| j �| _ dS )z~
Insert a single match. Will be inserted accordingly to preserve sort.
Can be inserted as a submatch.
z-Cannot append instance '{}' to CharsetMatchesN)r- r r5 r/ r0 r1 r; r< r rx r3 r7 rJ rI rd )r' r{ �matchr( r( r) rI 6 s
zCharsetMatches.appendr c C s | j s
dS | j d S )zQ
Simply return the first match. Strict equivalent to matches[0].
Nr )rx )r' r( r( r) ri J s zCharsetMatches.bestc C s | j � S )zP
Redundant method, call the method best(). Kept for BC reasons.
)ri )r' r( r( r) rh R s zCharsetMatches.first)N)rn ro rp �__doc__r
r r* r rz r
r| r0 r~ r rs r� rI r ri rh r( r( r( r) rv s rv c @ sj e Zd Zeee ee ee eee eeeee ed� dd�Ze e
ee f d�dd��Z ed�dd�Z
d S )
�CliDetectionResult) �pathr2 rN �alternative_encodingsrX rf r r7 r8 � unicode_path� is_preferredc C sF || _ |
| _|| _|| _|| _|| _|| _|| _|| _| | _ | | _
d S )N) r� r� r2 rN r� rX rf r r7 r8 r� ) r' r� r2 rN r� rX rf r r7 r8 r� r� r( r( r) r* ^ s zCliDetectionResult.__init__)r, c C s2 | j | j| j| j| j| j| j| j| j| j | j
d� S )N) r� r2 rN r� rX rf r r7 r8 r� r� ) r� r2 rN r� rX rf r r7 r8 r� r� )r' r( r( r) �__dict__x s zCliDetectionResult.__dict__c C s t | jddd�S )NT� )Z ensure_ascii�indent)r r� )r' r( r( r) �to_json� s zCliDetectionResult.to_jsonN)rn ro rp r0 r r
rs rr r* ru r r r� r� r( r( r( r) r� ] s r� )#r= � collectionsr Zencodings.aliasesr Zhashlibr Zjsonr �rer Ztypingr r r r
r r r
Zconstantr r Zmdr Zutilsr r r r rv r0 rr ZCoherenceMatchr r� r( r( r( r) �<module> s $ D