3
�gae7 � @ sv d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z m
Z
ddl m Z m
Z
mZ ddlmZ ddlZddlmZ dd lmZ dd
lmZ d Zd Zd
ZdZdZ dZ G dd� de �Z e e!e!d�dd�Z"e e
e d�dd�Z#e e
e e d�dd�Z$e e!e d �d d �Z%e e!d �d d!�Z&d.e e d#�d$d%�Z'ej(j)fe e d#�d&d'�Z*G d(d)� d)�Z+G d*d+� d+�Z,G d,d-� d-ej-�Z.dS )/z
Download mode implementation.
� N)�Message)�sleep� monotonic)�IO�Optional�Tuple)�urlsplit� )� HTTPResponse)� RawStream)�humanize_bytes�� z
[KzC{percentage: 6.2f} % {downloaded: >10} {speed: >10}/s {eta: >8} ETAz {downloaded: >10} {speed: >10}/sz/Done. {downloaded} in {time:0.5f}s ({speed}/s)
z|/-\c @ s e Zd ZdS )�ContentRangeErrorN)�__name__�
__module__� __qualname__� r r � /usr/lib/python3.6/downloads.pyr $ s r )�
content_range� resumed_from�returnc C s� | dkrt d��d}tj|| �}|s2t d| ����|j� }t|d �}t|d �}|d rft|d �nd}||ks�|dk r�||kr�t d| ����||ks�|dk r�|d |kr�t d
| �d |� d ���|d S )
a}
Parse and validate Content-Range header.
<https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html>
:param content_range: the value of a Content-Range response header
eg. "bytes 21010-47021/47022"
:param resumed_from: first byte pos. from the Range request header
:return: total size of the response body when fully downloaded.
NzMissing Content-RangezT^bytes (?P<first_byte_pos>\d+)-(?P<last_byte_pos>\d+)/(\*|(?P<instance_length>\d+))$z Invalid Content-Range format �first_byte_pos�
last_byte_pos�instance_lengthz Invalid Content-Range returned: r z#Unexpected Content-Range returned (z") for the requested Range ("bytes=z-"))r �re�match� groupdict�int)r r �patternr Zcontent_range_dictr r r r r r �parse_content_range( s. r )�content_dispositionr c C s< t d| � ��}|j� }|r8tjj|�jd�j� }|r8|S dS )z�
Extract and validate filename from a Content-Disposition header.
:param content_disposition: Content-Disposition value
:return: the filename if present and valid, otherwise `None`
zContent-Disposition: �.N)r � get_filename�os�path�basename�lstrip�strip)r �msg�filenamer r r �!filename_from_content_dispositionb s r* )�url� content_typer c C sr t | �jjd�}|r tjj|�nd}d|krn|rn|jd�d }|dkrLd}n
tj|�}|dkrbd }|rn||7 }|S )
N�/�indexr! �;r z
text/plainz.txtz.htmz.html)r r$ �rstripr# r% �split� mimetypesZguess_extension)r+ r, �fn�extr r r �filename_from_urlw s
r5 )r) �max_lenr c C sZ t | �|krVt | �| }tjj| �\}}|t |�krD| d | � } n|d | � | } | S )N)�lenr# r$ �splitext)r) r6 Ztrim_by�namer4 r r r �
trim_filename� s r: )� directoryr c C s( d}t td�r$dtjkr$tj| d�}|S )N� �pathconf� PC_NAME_MAX)�hasattrr# �pathconf_namesr= )r; r6 r r r �get_filename_max_length� s rA r! )r) r c C s& t |�| }t| �|kr"t| |�} | S )N)rA r7 r: )r) r; �extrar6 r r r �trim_filename_if_needed� s
rC c C sP d}xF|dkrd|� �nd}t | t|�d�}||7 }||�s@|S |d7 }qW d S )Nr �-� )rB r )rC r7 )r) �existsZattempt�suffixZ try_filenamer r r �get_unique_filename� s rH c @ s� e Zd Zddejfeeed�dd�Zed�dd�Z e
e j e
eef d �d
d �Zd d
� Zdd� Zeed�dd��Zed�dd�Zee
e j ed �dd��ZdS )�
DownloaderNF)� output_file�resume�
progress_filec C s4 d| _ t� | _|| _|| _d| _t| j|d�| _dS )aM
:param resume: Should the download resume if partial download
already exists.
:param output_file: The file to store response body in. If not
provided, it will be guessed from the response.
:param progress_file: Where to report download progress.
Fr )�status�outputN)�finished�DownloadStatusrM � _output_file�_resume�
_resumed_from�ProgressReporterThread�_progress_reporter)�selfrJ rK rL r r r �__init__� s zDownloader.__init__)�request_headersc C s< d|d< | j r8tjj| jj�}|r8d|� d�|d< || _dS )z^Called just before the HTTP request is sent.
Might alter `request_headers`.
ZidentityzAccept-Encodingzbytes=rD ZRangeN)rR r# r$ �getsizerQ r9 rS )rV rX Z
bytes_haver r r � pre_request� s zDownloader.pre_request)� initial_url�final_responser c C s( | j j st�yt|jd �}W n tttfk
r> d}Y nX | jsX| j ||d�| _n\| j
r~|j t kr~t
|jjd�| j�}n6d| _y| jjd� | jj� W n tk
r� Y nX | j j| j|d� tt|�dd| jdd �}| jjjd |dk � r�t|�d
nd� d| jj� d�� | jj � || jfS )a=
Initiate and return a stream for `response` body with progress
callback attached. Can be called only once.
:param initial_url: The original requested URL
:param final_response: Initiated response object with headers already fetched
:return: RawStream, output_file
zContent-LengthN)r[ r\ z
Content-Ranger )r �
total_sizeFTi � )r( Z with_headersZ with_bodyZon_body_chunk_downloadedZ
chunk_sizez Downloading � rE zto "z"
i ) rM � time_started�AssertionErrorr �headers�KeyError�
ValueError� TypeErrorrQ � _get_output_file_from_responserR Z status_code�PARTIAL_CONTENTr �getrS �seek�truncate�OSError�startedr r
�chunk_downloadedrU rN �writer r9 �start)rV r[ r\ r] �streamr r r ro � s@
.
zDownloader.startc C s | j s t�d| _ | jj � d S )NT)rO ra rM )rV r r r �finish s zDownloader.finishc C s | j j� d S )N)rU �stop)rV r r r �failed" s zDownloader.failed)r c C s | j o | jjo | jj| jjkS )N)rO rM r] �
downloaded)rV r r r � interrupted% s zDownloader.interrupted)�chunkc C s | j jt|�� dS )z�
A download progress callback.
:param chunk: A chunk of response body data that has just
been downloaded and written to the output.
N)rM rm r7 )rV rv r r r rm - s zDownloader.chunk_downloadedc C sH d }d|j kr t|j d �}|s4t| |j jd�d�}t|�}t|dd�S )NzContent-Dispositionz Content-Type)r+ r, za+b)�mode)rb r* r5 rh rH �open)r[ r\ r) Zunique_filenamer r r rf 7 s
z)Downloader._get_output_file_from_response)r r r �sys�stderrr �boolrW �dictrZ �str�requestsZResponser r ro rq rs �propertyru �bytesrm � staticmethodrf r r r r rI � s >
rI c @ s> e Zd ZdZdd� Zddd�Zdd � Zed
d � �Zd d
� Z dS )rP z(Holds details about the download status.c C s" d| _ d | _d| _d | _d | _d S )Nr )rt r] r r` �
time_finished)rV r r r rW M s
zDownloadStatus.__init__r Nc C s, | j d kst�|| _| | _| _t� | _ d S )N)r` ra r] rt r r )rV r r] r r r rl T s zDownloadStatus.startedc C s | j d kst�| j|7 _d S )N)r� ra rt )rV �sizer r r rm Z s z DownloadStatus.chunk_downloadedc C s
| j d k S )N)r� )rV r r r � has_finished^ s zDownloadStatus.has_finishedc C s( | j d k st�| jd ks t�t� | _d S )N)r` ra r� r )rV r r r rO b s zDownloadStatus.finished)r N)
r r r �__doc__rW rl rm r r� rO r r r r rP J s
rP c sJ e Zd ZdZdeed�� fdd�
Zdd� Zd d
� Zd d � Z d
d� Z
� Z S )rT z�
Reports download progress based on its status.
Uses threading to periodically update the status (speed, ETA, etc.).
皙�����?r )rM rN c sJ t � j� || _|| _|| _|| _d| _d| _d| _t � | _
t j � | _
d S )Nr rE )�superrW rM rN �_tick�_update_interval� _spinner_pos� _status_line� _prev_bytesr �
_prev_time� threadingZEvent� _should_stop)rV rM rN ZtickZupdate_interval)� __class__r r rW p s
z ProgressReporterThread.__init__c C s | j j� dS )z Stop reporting on next tick.N)r� �set)rV r r r rr � s zProgressReporterThread.stopc C s8 x2| j j� s2| jjr | j� P | j� t| j� qW d S )N)r� Zis_setrM r� �sum_up� report_speedr r� )rV r r r �run� s zProgressReporterThread.runc C s0 t � }|| j | jkr�| jj}|| j || j }| jjsTtjt |�t |�d�| _
n�| jjrl|| jj d nd}|szd}nHt | jj| | �}t |d �\}}t |d�\}}|� d|d�d|d��}t
j|t |�t |�|d�| _
|| _|| _| jjt� d t| j � d | j
� �� | jj� | jd
tt� | _d S ) N)rt �speed�d r z-:--:--�< �:z0>2)�
percentagert r� �etar_ r i )r r� r� rM rt r� r] �PROGRESS_NO_CONTENT_LENGTH�formatr r� r �divmod�PROGRESSrN rn �
CLEAR_LINE�SPINNERr� �flushr7 ) rV Znowrt r� r� r� �s�h�mr r r r� � s8
z#ProgressReporterThread.report_speedc C s| | j j| j j }| j j| j j }|r,|| n|}| jjt� | jjtj t
|�| j j o^t
| j j �t
|�|d�� | jj � d S )N)rt Ztotalr� �time)
rM rt r r� r` rN rn r� �SUMMARYr� r r] r� )rV Zactually_downloadedZ
time_takenr� r r r r� � s
z ProgressReporterThread.sum_up)r� r ) r r r r� rP r rW rr r� r� r� �
__classcell__r r )r� r rT h s
*rT )r! r )/r� r2 r# r ry r� Zmailboxr r� r r Ztypingr r r Z urllib.parser r~ Zmodelsr
Zoutput.streamsr Zutilsr rg r� r� r� r� r� rd r r} r r r* r5 r: rA rC r$ rF rH rI rP ZThreadrT r r r r �<module> s@ ;