provenaclient.modules.submodules

Submodules

• provenaclient.modules.submodules.datastore_io_submodule

Classes

IOSubModule

This class interface just captures that the client has an instantiated auth

Package Contents

class provenaclient.modules.submodules.IOSubModule(auth: provenaclient.modules.module_helpers.AuthManager, config: provenaclient.modules.module_helpers.Config, datastore_client: provenaclient.clients.DatastoreClient)

Bases: provenaclient.modules.module_helpers.ModuleService

This class interface just captures that the client has an instantiated auth manager which allows for helper functions abstracted for L3 clients.

_datastore_client: provenaclient.clients.DatastoreClient

_auth

_config

async _create_s3_path(dataset_id: str, access_type: AccessEnum) → cloudpathlib.S3Path

This helper function creates an S3 URI in PATH format by ingesting the dataset id and access type (read, write).

Parameters:

• dataset_id (str) – The ID of the dataset to download files for - ensure you have the right access.

• access_type (AccessEnum) – The access type required (Read or Write)

Returns:

S3Path instance that represent a path in S3 with filesystem path semantics.

Return type:

S3Path

async download_all_files(destination_directory: str, dataset_id: str) → None

Downloads all files to the destination path for a given dataset id.

• Fetches info

• Fetches creds

• Uses s3 cloud path lib to download all files to specified location

Parameters:

• destination_directory (str) – The destination path to save files to - use a directory

• dataset_id (str) – The ID of the dataset to download files for - ensure you have read access

async list_all_files(dataset_id: str, print_list: bool = False) → ProvenaInterfaces.DataStoreAPI.List[cloudpathlib.s3.S3Path]

Lists all files stored in the given dataset by ID.

• Fetches info

• Fetches creds

• Uses s3 cloud path lib to list all files to specified location

Parameters:

dataset_id (str) – The ID of the dataset to download files for - ensure you have read access

async upload_all_files(source_directory: str, dataset_id: str) → None

Uploads all files in the source path to the specified dataset id’s storage location.

• Fetches info

• Fetches creds

• Uses s3 cloud path lib to upload all files to specified location

Parameters:

• source_directory (str) – The source path to upload files from - use a directory

• dataset_id (str) – The ID of the dataset to upload files for - ensure you have write access

async download_specific_file(dataset_id: str, s3_path: str, destination_directory: str) → None

Downloads a specific file or folder from an S3 bucket to a provided destination path.

This method handles various cases: - If s3_path is a specific file, it downloads that file directly to destination_directory. - If s3_path is a folder (without a trailing slash), it downloads the entire folder and its contents, preserving the folder structure in destination_directory. - If s3_path is a folder (with a trailing slash), it downloads all contents (including subfolders) within that folder but not the folder itself to destination_directory.

Parameters:

• dataset_id (str) – The ID of the dataset that contains the files or folders to download from S3.

• s3_path (str) – The S3 path of the file or folder to download. - If this is a specific file, it will download just that file. - If this is a folder without a trailing slash (e.g., ‘nested’), it will download the entire folder and all its contents, preserving the structure. - If this is a folder with a trailing slash (e.g., ‘nested/’), it will download all contents within that folder but not the folder itself unless subfolders are present.

• destination_directory (str) – The destination path to save files to - use a directory.