src.core.downloader package

Submodules

src.core.downloader.AmazonBucketManager module

class src.core.downloader.AmazonBucketManager.AmazonBucketManager(configurations)

Bases: object

This class is an Adapter of Amazon AWS services. Main goal of this class is Retrieve the list of products from aws via tile list, and time interval

extract_date(item)

This method extracts the date from the tile name string (complete with the date).

Parameters:item – The tile name complete string. e.g.

tiles/32/T/MK/YYYY/MM/DD/

where

• YYYY is the year
• MM is the month
• DD is the day

Returns:The extracted date form the tile name string Return type:Datetime

The method apply the regex:

tiles/[0-9]2/[A-Z]/[A-Z]{2}/([0-9]{4})/([0-9]*)/([0-9]*)/

to the item parameter

generate_url(tile, year)

This method generates the aws url from the tile name string and the start year of filter interval. AWS service has the prefix parameter, to filter the list of available files in the bucket. This method generates the proper prefix parameter value, to give the list of files available in the year of year argument passed in the method.

The generated parameter is like:

prefix=tiles/{tile}/{year}/

Parameters:tile – The tile name string with a pattern like

32/T/MK

Parameters:year – The year Returns:request url Return type:String

get_products_list(searchFilter)

This method manages the retrieval of aws products. This method contains the AWS APIs semantic it is the core of Adapter. It manages also pagination.

Parameters:searchFilter (SearchFilter) – Search parameters to filter the available files in the bucket Returns:The pending products available in the Amazon bucket. Return type:None

Todo

Optimize list retrieval using set end date year instead NOW year.

load_products(tile, year, paginated=False)

load_products load list products form the Amazon AWS Bucket of Sinergise Sentinel-2 on AWS. This method contains the AWS APIs semantic it is the core of Adapter. It manages also pagination.

Parameters:

• tile (String) – Tile name
• year (Boolean) – Year of interest, the method extract the whole list of products for this year
• paginated – if this parameter is True that means that the list provided from AWS is paginated and the next page starts after the last_item

Returns:

None

Return type:

None

This method stores the list of products in the classes property product_list.

src.core.downloader.Configuration module

class src.core.downloader.Configuration.Configuration

Bases: object

This class is the container of Downloader configurations.

src.core.downloader.ConfigurationManager module

class src.core.downloader.ConfigurationManager.ConfigurationManager

Bases: object

datetime_from_string(date_string)

get_configuration()

load_aws_domain()

load_aws_products_regex()

load_aws_xmlns()

load_configuration()

load_end_date()

load_files_to_download()

load_inbox_path()

load_parallel_downloads()

load_start_date()

load_tiles()

src.core.downloader.Datasource module

class src.core.downloader.Datasource.Datasource(configurations)

Bases: object

This class is the datasource of all data used by the Downloader.

download_product(product)

This method donwloads the product passed as method paramenter.

Parameters:product (Product) – The product to download Returns:None Return type:None

get_products_list(searchFilter)

This method generates the product list from query filters defined in the config file by the user.

Parameters:searchFilter (SearchFilter) – Search parameters to filter the available files in the bucket Returns:The list of available products. Return type:List

src.core.downloader.Downloader module

class src.core.downloader.Downloader.Downloader

Bases: object

The main class of Downloader module, it is the entry point that receives the trigger of download events.

create_search_filter(tile)

This method generates the SearchFilter object from the tile name and configurations set by user.

Parameters:tile (String) – Tile name Returns:Search filter object to retrieve data from datasource Return type:SearchFilter

start()

Download event handler. The entry point of the Downloader module.

src.core.downloader.DownloaderJob module

class src.core.downloader.DownloaderJob.DownloaderJob(queue)

Bases: threading.Thread

This class manages the ProductDownloader classes. This is an async method, launched by the Downloader.

refresh_configurations()

This method reload the configurations from the Downloader config file.

run()

The run method of the thread, this method calls the datasource method to download the products available to download from the database (the products with the status pending.

src.core.downloader.ProductDownloader module

class src.core.downloader.ProductDownloader.ProductDownloader(inbox_path, files_to_download, domain)

Bases: object

This class is the interface with the internet to retrieve data from remote datasource.

download_product(product_name)

This method launches the http request to download the product via product name.

Parameters:product_name (String) – The name of product to download Returns:None Return type:None

src.core.downloader.SearchFilter module

class src.core.downloader.SearchFilter.SearchFilter(tile, start_date, end_date)

Bases: object

This class is the container of filtering parameters to retrieve data from Downloader datasource.

Module contents