3
\�me � � @ s� d dl Z d dlZd dlZd dlZd dlmZ d dlmZmZm Z d dlm
Z d dlm Z m
Z
d dlmZmZ d dlmZ d dlmZ d d lmZ eje�Zee jg�Z d5Z d
Z G dd� d�Z dd� Z e j!e j"gZ#G dd� de j$�Z%G dd� de%�Z&dd� Z'dd� Z(dd� Z)d d � Z*d d � Z+d d!� Z,d"d#� Z-d$d%� Z.d&d'� Z/d(d)� Z0d*d+� Z1d6d-d.�Z2d/d0� Z3d1d2� Z4e&e j5ffe%e j5e j6ffgZ7d3d4� Z8dS )7� N)�List)�dmi�net�sources)�
url_helper)�util�warnings)�
EventScope� EventType)�NoDHCPLeaseError)�EphemeralIPNetwork)�ec2�
datasource�Ec2� strict_id�warnc @ s, e Zd ZdZdZdZdZdZdZdZ dZ
d S )
�
CloudNamesZaliyunZawsZ brightboxZzstack�e24cloudZoutscale�unknownzno-ec2-metadataN) �__name__�
__module__� __qualname__�ALIYUN�AWS� BRIGHTBOX�ZSTACK�E24CLOUD�OUTSCALE�UNKNOWN�NO_EC2_METADATA� r r �#/usr/lib/python3.6/DataSourceEc2.pyr ! s r c C s | j dkod| jkS )Ni� zmeta-data/tags/)�code�url)� exceptionr r r! �skip_404_tag_errors1 s r% c sx e Zd ZU dZdddgZdZdddgZee d Z d
Z
d Z e j
Zd ZejejejejejhiZ� fd
d�Zdd� Zdd� Zdd� Ze dd� �Z e dd� �Z e dd� �Z e dd � �Z e d d � �Z!e d d � �Z"e d!d"� �Z#d#d$� Z$d%d&� Z%d'd(� Z&d)d*� Z'd+d,� Z(e d-d.� �Z)e d/d0� �Z*d1d2� Z+e d3d4� �Z,e � fd5d6��Z-d7d8� Z.dDd9d:�Z/d;d<� Z0d=d>� Z1dEd?d@�Z2dFdBdC�Z3� Z4S )G�
DataSourceEc2r zhttp://169.254.169.254zhttp://[fd00:ec2::254]zhttp://instance-data.:8773z
2009-04-04z
2021-03-23z
2018-09-24z
2016-09-02�x �2 NFc s t t| �j|||� d | _d S )N)�superr& �__init__�metadata_address)�self�sys_cfg�distro�paths)� __class__r r! r* d s zDataSourceEc2.__init__c C s t � S )z5Return the cloud name as identified during _get_data.)�identify_platform)r, r r r! �_get_cloud_nameh s z DataSourceEc2._get_cloud_namec C sF t tj| jtt�d�\}}tjd|| j| j � |dkrF| jt
j krFdS | jt
j krVdS | j
r�tj� rrtjd� dS yVt| j| jddd��8}|jr�d|j� �nd }tjtjd
|� �| jd �| _W d Q R X W n tk
r� dS X ntjtjd
| jd �| _| j�sdS | jjd d �| _| jjd
d �| _| jjdi �jdi �jdi �| _dS )Nr z0strict_mode: %s, cloud_name=%s cloud_platform=%s�trueFz1FreeBSD doesn't support running dhclient with -sfT)�ipv4�ipv6� � zCrawl of metadata service)�logfunc�msg�funcz meta-dataz user-data�dynamiczinstance-identity�document)r N)�read_strict_moder �get_cfg_by_pathr- �STRICT_ID_PATH�STRICT_ID_DEFAULT�LOG�debug�
cloud_name�platformr r r �perform_dhcp_setupZ
is_FreeBSDr r. �fallback_interface� state_msg�log_time�crawl_metadataZ_crawled_metadatar �get�metadataZ userdata_raw�identity)r, Z strict_modeZ_sleepZnetwrG r r r! � _get_datal sV
zDataSourceEc2._get_datac C sN | j s
dS | j jdi �jdi �}x(|jdi �j� D ]\}}d|kr2dS q2W dS )z6Report if this instance type is Ec2 Classic (non-vpc).F�network�
interfaces�macszvpc-idT)rK rJ �items)r, Z ifaces_mdZ_macZmac_datar r r! �is_classic_instance� s z!DataSourceEc2.is_classic_instancec C s | j s
d S | j jd�S )Nzami-launch-index)rK rJ )r, r r r! � launch_index� s zDataSourceEc2.launch_indexc C s. t | d�stjj� | _| js(tjj� | _| jS )N�_platform_type)�hasattrr& �dsname�lowerrT )r, r r r! rD � s
zDataSourceEc2.platformc C s dS )Nzlatest/api/tokenr )r, r r r! �api_token_route� s z DataSourceEc2.api_token_routec C s dS )NZ21600r )r, r r r! �imdsv2_token_ttl_seconds� s z&DataSourceEc2.imdsv2_token_ttl_secondsc C s dS )NzX-aws-ec2-metadata-tokenr )r, r r r! �imdsv2_token_put_header� s z%DataSourceEc2.imdsv2_token_put_headerc C s
| j d S )Nz -ttl-seconds)rZ )r, r r r! �imdsv2_token_req_header� s z%DataSourceEc2.imdsv2_token_req_headerc C s | j | jgS )N)rZ r[ )r, r r r! �imdsv2_token_redact� s z!DataSourceEc2.imdsv2_token_redactc C s� d}| j � }x�| jD ]�}|j| j|�}ytj||| jd�}W n2 tjk
rn } ztj d||� W Y dd}~X qX |j
dkr�tj d|� |S |j
dkrd}tj |||j � qW | j S ) aE Get the best supported api version from the metadata service.
Loop through all extended support metadata versions in order and
return the most-fully featured metadata api version discovered.
If extended_metadata_versions aren't present, return the datasource's
min_metadata_version.
z {0}/{1}/meta-data/instance-id)r# �headers�headers_redactzurl %s raised exception %sN�� z#Found preferred metadata version %si� z0Metadata api version %s not present. Headers: %s)
� _get_headers�extended_metadata_versions�formatr+ �uhelp�readurlr\ �UrlErrorrA rB r" r] �min_metadata_version)r, Zurl_tmplr] Zapi_verr# Zresp�er9 r r r! �get_metadata_api_version� s$
z&DataSourceEc2.get_metadata_api_versionc C sh | j tjkrZt| dd �sF| j� }tj|| j| j| j | j
d�j di �| _ | j j d| j
d �S | j
d S d S )NrL )�
headers_cbr^ � exception_cbr< Z
instanceIdz instance-id)rC r r �getattrrh r
�get_instance_identityr+ r` r\ �_refresh_stale_aws_token_cbrJ rL rK )r, � api_versionr r r! �get_instance_id� s
z DataSourceEc2.get_instance_idc
C s� | j tkrdS g }i }| j}d}x*|D ]"}dj||�}|j|� |||< q&W tjd� d}d}| j� } y0tj || j
| j tj | j
| j|| jdd� \}}W n tjk
r� Y nX |r�|r�|| _|| S dS )a Get an API token for EC2 Instance Metadata Service.
On EC2. IMDS will always answer an API token, unless
the instance owner has disabled the IMDS HTTP endpoint or
the network topology conflicts with the configured hop-limit.
N�PUTz{0}/{1}z Fetching Ec2 IMDSv2 API TokenF) �urls�max_wait�timeout� status_cbri rj �request_methodr^ Zconnect_synchronously)rC � IDMSV2_SUPPORTED_CLOUD_PLATFORMSrX rb �appendrA rB �get_url_paramsrc � wait_for_url�max_wait_seconds�timeout_seconds�warningr` �_imds_exception_cbr\ re �
_api_token)
r, �mdurlsrq �url2base�url_pathru r# �cur�response�
url_paramsr r r! �_maybe_fetch_api_token s>
z$DataSourceEc2._maybe_fetch_api_tokenc C sv | j }| j� }|jdkr dS |jd| j�}dd� |D �}t|�t|�krdtjdtt|�t|� �� t |�rr|}ntj
d� | j}| j |�}| o�| j t
k�r g }i }dj| jd �}d
} x*|D ]"}
d j|
|�} |j| � |
|| < q�W tj� } tj||j|jtj
| j| j| d �\}
}
|
�r ||
}|�r<|| _tjd
| j� n2| j t
k�rTtj
d� ntjd|ttj� | �� t|�S )Nr F�
metadata_urlsc S s g | ]}t j|�r|�qS r )r Zis_resolvable_url)�.0�xr r r! �
<listcomp>C s z;DataSourceEc2.wait_for_metadata_service.<locals>.<listcomp>z,Removed the following from metadata urls: %sz+Empty metadata url list! using default listz{ver}/meta-data/instance-id)ZverZGETz{0}/{1})rq rr rs rt r^ ri ru zUsing metadata source: '%s'z)IMDS's HTTP endpoint is probably disabledz(Giving up on md from %s after %s seconds)�ds_cfgrx rz rJ r� �setrA rB �list�lenr| r� rC rv rb rf rw �timerc ry r{ r\ r` r+ Zcritical�int�bool)r, Zmcfgr� r Zfilteredr+ rq r� r� ru r# r� Z
start_time�_r r r! �wait_for_metadata_service: s^
z'DataSourceEc2.wait_for_metadata_servicec C s� d| j krd S d }| j d }t|t�s6tjd|� d S x6|j� D ]*\}}||krV|}P |dkr@|dkr@|}q@W |d kr�tjd|� d S |}|jd�s�d| }tjj |�r�|S | j
tjj |��}|r�tjd||� |S |dkr�d S |S ) Nzblock-device-mappingz+block-device-mapping not a dictionary: '%s'Z ephemeralZ
ephemeral0z Unable to convert %s to a device�/z/dev/%sz Remapped device name %s => %s) rK �
isinstance�dictrA rB rQ �
startswith�os�path�existsZ
_remap_device�basename)r, �name�foundZbdmZentnameZdeviceZofoundZremappedr r r! �device_name_to_device� s8
z#DataSourceEc2.device_name_to_devicec C sP y6| j tjkr&| jjd| jd d �S | jd d S W n tk
rJ d S X d S )NZavailabilityZoneZ placementzavailability-zone)rC r r rL rJ rK �KeyError)r, r r r! �availability_zone� s z DataSourceEc2.availability_zonec C sT | j tjkr6| jjd�}| jr2| r2| jd d� }|S | j}|d k rP|d d� S d S )N�region� ���r� )rC r r rL rJ r� )r, r� Zazr r r! r� � s zDataSourceEc2.regionc C s, |sd S | j tjkr(ttj|tt�|� d S )N)rC r r �warn_if_necessaryr r> r? r@ )r, �cfgZis_new_instancer r r! �activate� s zDataSourceEc2.activatec C s� | j tjkr| j S | jdkr*tjd� dS d}td| jkoD| jtj k�}|rjtj
d� t j tj
d| j
d� | j}| jjd�}t|t�r�t||t j| jdd�d �}| j� r�tj| j�| _| jtj jt j � | jtj jt j � n tjd
|� || _ | j S ) z@Return a network config dict for rendering ENI or netplan files.Nz8Unexpected call to network_config when metadata is None.rN zPMetadata 'network' not present: Refreshing stale metadata from prior to upgrade.z Re-crawl of metadata service)r8 r9 r: Z apply_full_imds_network_configT)� fallback_nic�full_network_configz%Metadata 'network' key not valid: %s.) �_network_configr �UNSETrK rA r| r� rC r r rB r rH �get_datarF rJ r� r� �#convert_ec2_metadata_network_configZget_cfg_option_boolr� rR �copy�deepcopyZdefault_update_eventsr �NETWORK�addr
�BOOT� BOOT_LEGACY)r, �resultZno_network_metadata_on_awsZifaceZnet_mdr r r! �network_config� sF
z DataSourceEc2.network_configc s: | j d kr4t| dd �}|r(|| _ d | _n tt| �jS | j S )Nr� )Z_fallback_interfacerk r� r) r& rF )r, Z
_legacy_fbnic)r0 r r! rF s
z DataSourceEc2.fallback_interfacec C s | j � s i S | j� }| j}i }| jtkr:| j}| j}d}n&| jtjkrTd }}t }n d } }}yxt
j || j | j
||d�}tj|�|d< t
j|| j | j
|||d�|d< | jtjkr�t
j|| j | j
||d�}d|i|d< W n& tk
� r� tjtd| j � i S X ||d < |S )
z�Crawl metadata service when available.
@returns: Dictionary of crawled metadata content containing the keys:
meta-data, user-data and dynamic.
N)ri r^ rj z user-data)ri r^ rj Z retrieval_exception_ignore_cbz meta-datazinstance-identityr; z'Failed reading from metadata address %sZ_metadata_api_version)r� rh r\ rC rv rm �#_skip_or_refresh_stale_aws_token_cbr r r% r
Zget_instance_userdatar+ r` r Zmaybe_b64decodeZget_instance_metadatar rl � ExceptionZlogexcrA ) r, rn ZredactZcrawled_metadataZexc_cbZ exc_cb_udZskip_cbZ raw_userdatarL r r r! rI $ sX
z DataSourceEc2.crawl_metadatac C s� | j tkrdS |dkr | j}tjd� | j|i}dj| j| j�}yt j
||| j dd�}W n0 t j k
r� } ztj
d||� dS d}~X nX |jS )z�Request new metadata API token.
@param seconds: The lifetime of the token in seconds
@return: The API token or None if unavailable.
Nz!Refreshing Ec2 metadata API tokenz{}/{}rp )r] r^ ru z/Unable to get API token: %s raised exception %s)rC rv rY rA rB r[ rb r+ rX rc rd r\ re r| �contents)r, ZsecondsZrequest_headerZ token_urlr� rg r r r! �_refresh_api_token\ s$
z DataSourceEc2._refresh_api_tokenc C s$ t jt j||�}|sdS | j||�S )zSCallback will not retry on SKIP_USERDATA_CODES or if no token
is available.F)r
Zskip_retry_on_codesZSKIP_USERDATA_CODESrm )r, r9 r$ Zretryr r r! r� y s
z1DataSourceEc2._skip_or_refresh_stale_aws_token_cbc C s* t |tj�r&|jdkr&tjd� d| _dS )z=Exception handler for Ec2 to refresh token if token is stale.i� z+Clearing cached Ec2 API token due to expiryNT)r� rc re r" rA rB r~ )r, r9 r$ r r r! rm � s
z)DataSourceEc2._refresh_stale_aws_token_cbc C sD t |tj�r@|jr@|jdkr@|jdkr2tjd� n
tjd� |�dS )a2 Fail quickly on proper AWS if IMDSv2 rejects API token request
Guidance from Amazon is that if IMDSv2 had disabled token requests
by returning a 403, or cloud-init malformed requests resulting in
other 40X errors, we want the datasource detection to fail quickly
without retries as those symptoms will likely not be resolved by
retries.
Exceptions such as requests.ConnectionError due to IMDS being
temporarily unroutable or unavailable will still retry due to the
callsite wait_for_url.
i� i� zLEc2 IMDS endpoint returned a 403 error. HTTP endpoint is disabled. Aborting.z2Fatal error while requesting Ec2 IMDSv2 API tokensN)r� rc re r" rA r| )r, r9 r$ r r r! r} � s
z DataSourceEc2._imds_exception_cbr7 c C sN | j tkri S | j| ji}| j|kr(|S | jsB| j� | _| jsBi S | j| jiS )z�Return a dict of headers for accessing a url.
If _api_token is unset on AWS, attempt to refresh the token via a PUT
and then return the updated token header.
)rC rv r[ rY rX r~ r� rZ )r, r# Zrequest_token_headerr r r! r` � s
zDataSourceEc2._get_headers)N)N)r7 )5r r r rV r� rf ra r �strZ url_max_waitZ url_timeoutr~ r r� r� rE r r� r
ZBOOT_NEW_INSTANCEr� r� ZHOTPLUGZsupported_update_eventsr* r2 rM rR �propertyrS rD rX rY rZ r[ r\ rh ro r� r� r� r� r� r� r� rF rI r� r� rm r} r` �
__classcell__r r )r0 r! r& 9 sZ
8
2I3
=
8
r& c s$ e Zd ZdZdZ� fdd�Z� ZS )�DataSourceEc2Localay Datasource run at init-local which sets up network to query metadata.
In init-local, no network is available. This subclass sets up minimal
networking with dhclient on a viable nic so that it can talk to the
metadata service. If the metadata service provides network configuration
then render the network configuration for that instance based on metadata.
Tc s8 t jt jf}| j|kr*tjd|| j� dS tt| �j� S )Nz+Local Ec2 mode only supported on %s, not %sF) r r r rC rA rB r) r� r� )r, Zsupported_platforms)r0 r r! r� � s
zDataSourceEc2Local.get_data)r r r �__doc__rE r� r� r r )r0 r! r� � s r� c C s8 yt | �S tk
r2 } ztj|� |S d }~X nX d S )N)�parse_strict_mode�
ValueErrorrA r| )�cfgval�defaultrg r r r! r= � s
r= c C s� | dkr d
S | dkrd S | s d S | j d�\}}}|d
krHtd|| f ��|r�y t|�}W q� tk
r� } ztd || f �|�W Y d d }~X q�X nd }||fS )NTr3 F�falser r �,zUInvalid mode '%s' in strict_id setting '%s': Expected one of 'true', 'false', 'warn'.z<Invalid sleep '%s' in strict_id setting '%s': not an integer)r3 N)r� N)r r )r3 r� r )� partitionr� r� )r� �moder� �sleeprg r r r! r� � s*
r� c C s^ yt | �\}}W n* tk
r: } ztj|� d S d }~X nX |dkrHd S tjd|d|d� d S )Nr� Z
non_ec2_mdT)r� r� )r� r� rA r| r Z show_warning)r� r� r� r� rg r r r! r� � s
r� c C s | d dkrt jS d S )N� product_namezAlibaba Cloud ECS)r r )�datar r r! �identify_aliyun s r� c C s4 | d j d�r0| d dks*| d | d kr0tjS d S )N�uuidr
� uuid_source�
hypervisor�serial)r� r r )r� r r r! � identify_aws s r� c C s | d j d�rtjS d S )Nr� z.brightbox.com)�endswithr r )r� r r r! �identify_brightbox s r� c C s | d j d�rtjS d S )N� asset_tagz
.zstack.io)r� r r )r� r r r! �identify_zstack s r� c C s | d dkrt jS d S )N�vendorr )r r )r� r r r! �identify_e24cloud$ s r� c C s* | d dj � kr&| d dj � kr&tjS d S )Nr� z3DS Outscale VMr� z 3DS Outscale)rW r r )r� r r r! �identify_outscale) s r� c C sv t � } ttttttdd� f}xT|D ]L}y|| �}|r8|S W q" tk
rl } ztj d|| |� W Y d d }~X q"X q"W d S )Nc S s t jS )N)r r )r� r r r! �<lambda>; s z#identify_platform.<locals>.<lambda>z'calling %s with %s raised exception: %s)
�_collect_platform_datar� r� r� r� r� r� r� rA r| )r� ZchecksZcheckerr� rg r r r! r1 1 s
r1 c C s� i } yt jd�j� }d| d< W n& tk
rD tjd�}d| d< Y nX |dkrRd}|j� | d< tjd �}|dkrtd}|j� | d
< tjd �}|dkr�d}|j� | d < tjd
�}|r�|ndj� | d< tjd�}|r�|ndj� | d< | S )a� Returns a dictionary of platform info from dmi or /sys/hypervisor.
Keys in the dictionary are as follows:
uuid: system-uuid from dmi or /sys/hypervisor
uuid_source: 'hypervisor' (/sys/hypervisor/uuid) or 'dmi'
serial: dmi 'system-serial-number' (/sys/.../product_serial)
asset_tag: 'dmidecode -s chassis-asset-tag'
vendor: dmi 'system-manufacturer' (/sys/.../sys_vendor)
product_name: dmi 'system-product-name' (/sys/.../system-manufacturer)
On Ec2 instances experimentation is that product_serial is upper case,
and product_uuid is lower case. This returns lower case values for both.
z/sys/hypervisor/uuidr� r� z system-uuidr Nr7 r� zsystem-serial-numberr� zchassis-asset-tagr� zsystem-manufacturerr� zsystem-product-namer� )r Z load_file�stripr� r Z
read_dmi_datarW )r� r� r� r� r� r� r r r! r� H s.
r� Tc C s� di d�}|st j� }| d d }|s�x |j� D ]\}}||kr0P q0W ddd|j� i|d�}|j|�} | jd �rxd|d
< ||d |< |S d }
x�t|j� �D ]�\}}|j|�} | s�q�t| jd
|
��d }
d|
d i} d| dd|j� i|d�}| jd ��rd|d
< | |d< t| |�|d< |d �s(|jd� ||d |< q�W t |d �dk�r�x<|d j
� D ],}|d | jd� |d | jdd� �qXW |S )a Convert ec2 metadata to network config version 2 data dict.
@param: network_md: 'network' portion of EC2 metadata.
generally formed as {"interfaces": {"macs": {}} where
'macs' is a dictionary with mac address as key and contents like:
{"device-number": "0", "interface-id": "...", "local-ipv4s": ...}
@param: macs_to_nics: Optional dict of mac addresses and nic names. If
not provided, get_interfaces_by_mac is called to get it from the OS.
@param: fallback_nic: Optionally provide the primary nic interface name.
This nic will be guaranteed to minimally have a dhcp4 configuration.
@param: full_network_config: Boolean set True to configure all networking
presented by IMDS. This includes rendering secondary IPv4 and IPv6
addresses on all NICs and rendering network config on secondary NICs.
If False, only the primary nic will be configured and only with dhcp
(IPv4/IPv6).
@return A dict of network config version 2 based on the metadata and macs.
� )�version� ethernetsrO rP TFZ
macaddress)�dhcp4�dhcp6�matchzset-name�ipv6sr� r� r z
device-numberr� z route-metric�d )r� zdhcp4-overridesr� r� zset-namezdhcp6-overrides� addresseszdhcp4-overridesN) r Zget_interfaces_by_macrQ rW rJ �sortedr� �get_secondary_addresses�popr� �keys) Z
network_mdZ macs_to_nicsr� r� ZnetcfgZ
macs_metadata�macZnic_nameZ
dev_config� nic_metadataZnic_idxZ
dhcp_overrider r r! r� w sR
r� c C s� | j d�}| j d�}g }tt|t�o.t|�dk�rH|jt| d||d�� tt|t�o^t|�dk�rx|jt| d||d�� t|�S )z�Parse interface-specific nic metadata and return any secondary IPs
:return: List of secondary IPv4 or IPv6 addresses to configure on the
interface
z local-ipv4sr� r� zsubnet-ipv4-cidr-blockZ24zsubnet-ipv6-cidr-blockZ128)rJ r� r� r� r� �extend�_get_secondary_addressesr� )r� r� Zipv4sr� r� r r r! r� � s
r� c
C s� g }| j |�}|}| s*t|jd��dkrPd|kr6dnd}tjd|||||� n|jd�d }x(|dd� D ]} |jdj| |d �� qlW |S )
z�Return list of IP addresses as CIDRs for secondary IPs
The CIDR prefix will be default_prefix if cidr_key is absent or not
parseable in nic_metadata.
r� r� r4 r5 zJCould not parse %s %s for mac %s. %s network config prefix defaults to /%sr� Nz
{ip}/{prefix})�ip�prefix)rJ r� �splitrA r| rw rb )
r� Zcidr_keyr� ZipsZdefault_prefixr� Zcidrr� Zip_typer� r r r! r� � s
r� c C s t j| t�S )N)r Zlist_from_depends� datasources)Zdependsr r r! �get_datasource_list� s r� )r r r )NNT)9r� Zloggingr� r� Ztypingr Z cloudinitr r r r rc r r Zcloudinit.eventr r
Zcloudinit.net.dhcpr Zcloudinit.net.ephemeralr Zcloudinit.sources.helpersr
Z getLoggerr rA � frozensetZ NOT_FOUNDZSKIP_METADATA_URL_CODESr? r@ r r% r r rv Z
DataSourcer& r� r= r� r� r� r� r� r� r� r� r1 r� r� r� r� ZDEP_FILESYSTEMZ DEP_NETWORKr� r� r r r r! �<module> sT
!
0
H