Implementation of submit_cgap

submit_cgap package

submit_cgap.base module

class submit_cgap.base.GenericKeyManager

Bases: object

get_keydict_for_env(env)
get_keydict_for_server(server)
keydict_to_keypair(auth_dict)
keys_file
locally_selected_app(app: typing_extensions.Literal['cgap', 'fourfront', 'smaht'][cgap, fourfront, smaht])
select_app(app: typing_extensions.Literal['cgap', 'fourfront', 'smaht'][cgap, fourfront, smaht])
selected_app

submit_cgap.exceptions module

exception submit_cgap.exceptions.CGAPPermissionError(server)

Bases: PermissionError

submit_cgap.submission module

class submit_cgap.submission.SubmissionProtocol

Bases: object

S3 = 's3'
UPLOAD = 'upload'
class submit_cgap.submission.UploadMessageWrapper(uuid, no_query=False)

Bases: object

Class to provide consistent queries/messages to user when uploading file(s) to given File UUID.

wrap_upload_function(function, file_name)

Wrap upload given function with messages conerning upload.

Parameters:
  • function – Upload function to wrap
  • file_name – File to upload
Returns:

Wrapped function
submit_cgap.submission.check_submit_ingestion(uuid: str, server: str, env: str, app: Optional[typing_extensions.Literal['cgap', 'fourfront', 'smaht'][cgap, fourfront, smaht]] = None) → Tuple[bool, str, dict]
submit_cgap.submission.compute_file_post_data(filename, context_attributes)
submit_cgap.submission.compute_s3_submission_post_data(ingestion_filename, ingestion_post_result, **other_args)
submit_cgap.submission.do_any_uploads(res, keydict, upload_folder=None, ingestion_filename=None, no_query=False, subfolders=False)
submit_cgap.submission.do_app_arg_defaulting(app_args, user_record)
submit_cgap.submission.do_uploads(upload_spec_list, auth, folder=None, no_query=False, subfolders=False)

Uploads the files mentioned in the give upload_spec_list.

If any files have associated extra files, upload those as well.

Parameters:
  • upload_spec_list – a list of upload_spec dictionaries, each of the form {‘filename’: …, ‘uuid’: …}, representing uploads to be formed.
  • auth – a dictionary-form auth spec, of the form {‘key’: …, ‘secret’: …, ‘server’: …} representing destination and credentials.
  • folder – a string naming a folder in which to find the filenames to be uploaded.
  • no_query – bool to suppress requests for user input
  • subfolders – bool to search subdirectories within upload_folder for files
Returns:

None
submit_cgap.submission.execute_prearranged_upload(path, upload_credentials, auth=None)

This performs a file upload using special credentials received from ff_utils.patch_metadata.

Parameters:
  • path – the name of a local file to upload
  • upload_credentials – a dictionary of credentials to be used for the upload, containing the keys ‘AccessKeyId’, ‘SecretAccessKey’, ‘SessionToken’, and ‘upload_url’.
  • auth – auth info in the form of a dictionary containing ‘key’, ‘secret’, and ‘server’, and possibly other useful information such as an encryption key id.
submit_cgap.submission.extract_metadata_and_upload_credentials(response, filename, method, payload_data, uuid=None, schema_name=None)
submit_cgap.submission.get_defaulted_award(award, user_record, error_if_none=False)

Returns the given award or else if none is specified, it tries to infer an award.

Parameters:
  • award – the @id of an award, or None
  • user_record – the user record for the authorized user
  • error_if_none – boolean true if failure to infer an award should raise an error, and false otherwise.
Returns:

the @id of an award to use
submit_cgap.submission.get_defaulted_consortia(consortia, user_record, error_if_none=False)

Returns the given consortia or else if none is specified, it tries to infer any consortia.

Parameters:
  • consortia – a list of @id’s of consortia, or None
  • user_record – the user record for the authorized user
  • error_if_none – boolean true if failure to infer any consortia should raise an error, and false otherwise.
Returns:

the @id of a consortium to use (or a comma-separated list)
submit_cgap.submission.get_defaulted_institution(institution, user_record)

Returns the given institution or else if none is specified, it tries to infer an institution.

Parameters:
  • institution – the @id of an institution, or None
  • user_record – the user record for the authorized user
Returns:

the @id of an institution to use
submit_cgap.submission.get_defaulted_lab(lab, user_record, error_if_none=False)

Returns the given lab or else if none is specified, it tries to infer a lab.

Parameters:
  • lab – the @id of a lab, or None
  • user_record – the user record for the authorized user
  • error_if_none – boolean true if failure to infer a lab should raise an error, and false otherwise.
Returns:

the @id of a lab to use
submit_cgap.submission.get_defaulted_project(project, user_record)

Returns the given project or else if none is specified, it tries to infer a project.

Parameters:
  • project – the @id of a project, or None
  • user_record – the user record for the authorized user
Returns:

the @id of a project to use
submit_cgap.submission.get_defaulted_submission_centers(submission_centers, user_record, error_if_none=False)

Returns the given submission center or else if none is specified, it tries to infer a submission center.

Parameters:
  • submission_centers – the @id of a submission center, or None
  • user_record – the user record for the authorized user
  • error_if_none – boolean true if failure to infer a submission center should raise an error, and false otherwise.
Returns:

the @id of a submission center to use
submit_cgap.submission.get_metadata_bundles_bucket_from_health_path(key: dict) → str
submit_cgap.submission.get_s3_encrypt_key_id(*, upload_credentials, auth)
submit_cgap.submission.get_s3_encrypt_key_id_from_health_page(auth)
submit_cgap.submission.get_section(res, section)

Given a description of an ingestion submission, returns a section name within that ingestion.

Parameters:
  • res – the description of an ingestion submission as a python dictionary that represents JSON data
  • section – the name of a section to find either in the toplevel or in additional_data.
Returns:

the section’s content
submit_cgap.submission.get_user_record(server, auth)

Given a server and some auth info, gets the user record for the authorized user.

This works by using the /me endpoint.

Parameters:
  • server – a server spec
  • auth – auth info to be used when contacting the server
Returns:

the /me page in JSON format
submit_cgap.submission.ingestion_submission_item_url(server, uuid)
submit_cgap.submission.resolve_server(server, env)

Given a server spec or a beanstalk environment (or neither, but not both), returns a server spec.

Parameters:
Returns:

a server spec
submit_cgap.submission.resume_uploads(uuid, server=None, env=None, bundle_filename=None, keydict=None, upload_folder=None, no_query=False, subfolders=False)

Uploads the files associated with a given ingestion submission. This is useful if you answered “no” to the query about uploading your data and then later are ready to do that upload.

Parameters:
  • uuid – a string guid that identifies the ingestion submission
  • server – the server to upload to
  • env – the beanstalk environment to upload to
  • bundle_filename – the bundle file to be uploaded
  • keydict – keydict-style auth, a dictionary of ‘key’, ‘secret’, and ‘server’
  • upload_folder – folder in which to find files to upload (default: same as ingestion_filename)
  • no_query – bool to suppress requests for user input
  • subfolders – bool to search subdirectories within upload_folder for files
submit_cgap.submission.running_on_windows_native()
submit_cgap.submission.search_for_file(directory, file_name, recursive=False)

Search for file within directory.

Parameters:
  • directory – Directory path
  • file_name – Name of file to find
  • recursive – Whether to search subdirectories of given directory
Returns:

(Path to file or None, Error message or None)
submit_cgap.submission.show_section(res, section, caveat_outcome=None)

Shows a given named section from a description of an ingestion submission.

The caveat is used when there has been an error and should be a phrase that describes the fact that output shown is only up to the point of the caveat situation. Instead of a “My Heading” header the output will be “My Heading (prior to <caveat>).”

Parameters:
  • res – the description of an ingestion submission as a python dictionary that represents JSON data
  • section – the name of a section to find either in the toplevel or in additional_data.
  • caveat_outcome – a phrase describing some caveat on the output
submit_cgap.submission.show_upload_info(uuid, server=None, env=None, keydict=None, app: str = None, show_primary_result=True, show_validation_output=True, show_processing_status=True, show_datafile_url=True)

Uploads the files associated with a given ingestion submission. This is useful if you answered “no” to the query about uploading your data and then later are ready to do that upload.

Parameters:
  • uuid – a string guid that identifies the ingestion submission
  • server – the server to upload to
  • env – the beanstalk environment to upload to
  • keydict – keydict-style auth, a dictionary of ‘key’, ‘secret’, and ‘server’
  • app – the name of the app to use e.g., affects whether to expect –lab, –award, –institution, –project, –consortium or –submission_center and whether to use .fourfront-keys.json, .cgap-keys.json, or .smaht-keys.json
  • show_primary_result – bool controls whether the primary result is shown
  • show_validation_output – bool controls whether to show output resulting from validation checks
  • show_processing_status – bool controls whether to show the current processing status
  • show_datafile_url – bool controls whether to show the datafile_url parameter from the parameters.
submit_cgap.submission.show_upload_result(result, show_primary_result=True, show_validation_output=True, show_processing_status=True, show_datafile_url=True)
submit_cgap.submission.submit_any_ingestion(ingestion_filename, *, ingestion_type, server, env, validate_only, institution=None, project=None, lab=None, award=None, consortium=None, submission_center=None, app: typing_extensions.Literal['cgap', 'fourfront', 'smaht'][cgap, fourfront, smaht] = None, upload_folder=None, no_query=False, subfolders=False, submission_protocol='upload')

Does the core action of submitting a metadata bundle.

Parameters:
  • ingestion_filename – the name of the main data file to be ingested
  • ingestion_type – the type of ingestion to be performed (an ingestion_type in the IngestionSubmission schema)
  • server – the server to upload to
  • env – the beanstalk environment to upload to
  • validate_only – whether to do stop after validation instead of proceeding to post metadata
  • app – either ‘cgap’ (the default) or ‘fourfront’
  • institution – the @id of the institution for which the submission is being done (when app=’cgap’ or None)
  • project – the @id of the project for which the submission is being done (when app=’cgap’ or None)
  • lab – the @id of the lab for which the submission is being done (when app=’fourfront’)
  • award – the @id of the award for which the submission is being done (when app=’fourfront’)
  • consortium – the @id of the consortium for which the submission is being done (when app=’smaht’)
  • submission_center – the @id of the submission_center for which the submission is being done (when app=’smaht’)
  • upload_folder – folder in which to find files to upload (default: same as bundle_filename)
  • no_query – bool to suppress requests for user input
  • subfolders – bool to search subdirectories within upload_folder for files
  • submission_protocol – which submission protocol to use (default: ‘s3’)
submit_cgap.submission.summarize_submission(uuid: str, app: str, server: Optional[str] = None, env: Optional[str] = None)
submit_cgap.submission.upload_extra_files(credentials, uploader_wrapper, folder, auth, recursive=False)

Attempt upload of all extra files.

Similar to “do_uploads”, search for each file and then call a wrapped upload function. Here, since extra files do not correspond to Items on the portal, no need to PATCH an Item to retrieve AWS credentials; they are directly passed in from the parent File’s metadata.

Parameters:
  • credentials – AWS credentials dictionary
  • uploader_wrapper – UploadMessageWrapper instance
  • folder – Directory to search for files
  • auth – CGAP authorization tuple
  • recursive – Whether to search subdirectories for file
submit_cgap.submission.upload_file_to_new_uuid(filename, schema_name, auth, **context_attributes)

Upload file to a target environment.

Parameters:
  • filename – the name of a file to upload.
  • schema_name – the schema_name to use when creating a new file item whose content is to be uploaded.
  • auth – auth info in the form of a dictionary containing ‘key’, ‘secret’, and ‘server’.
Returns:

item metadata dict or None
submit_cgap.submission.upload_file_to_uuid(filename, uuid, auth)

Upload file to a target environment.

Parameters:
  • filename – the name of a file to upload.
  • uuid – the item into which the filename is to be uploaded.
  • auth – auth info in the form of a dictionary containing ‘key’, ‘secret’, and ‘server’.
Returns:

item metadata dict or None
submit_cgap.submission.upload_item_data(item_filename, uuid, server, env, no_query=False)

Given a part_filename, uploads that filename to the Item specified by uuid on the given server.

Only one of server or env may be specified.

Parameters:
  • item_filename – the name of a file to be uploaded
  • uuid – the UUID of the Item with which the uploaded data is to be associated
  • server – the server to upload to (where the Item is defined)
  • env – the beanstalk environment to upload to (where the Item is defined)
  • no_query – bool to suppress requests for user input
Returns:

submit_cgap.utils module

class submit_cgap.utils.FakeResponse(status_code, json=None, content=None)

Bases: object

json()
raise_for_status()
submit_cgap.utils.check_repeatedly(check_function: Callable, wait_seconds: int = 10, repeat_count: int = -1, check_message: str = None, wait_message: str = None, done_message: str = None, stop_message: str = None, response_message: bool = True, messages: bool = True, verbose: bool = True) → Union[Tuple[bool, str, Any], Any]

Calls the given function (check_function) repeatedly, until it returns either a tuple whose first element is truthy, or just a non-tuple truthy value, waiting between calls for the given number (wait_seconds) of seconds, and trying for a maximum of the given number (repeat_count) of times; if repeat_count is non-positive (default), then never stop calling the function. If the function returns either a tuple whose first element is truthy, or just a non-tuple truthy value, then returns that value. If the function never returns a truthy value, and repeat_count is postiive, then after that maxmimum number of tries (repeat_count), return False.

If the messages argument is True (default) then a message to the stdout will be printed indicating each time the function is called, how long (in seconds) till the next call, and how many times in total it has been called. Additionally if the response_message argument is True (default) then if the function finally returns a truthy value, then that value will be printed to the stdout.

submit_cgap.utils.keyword_as_title(keyword)

Given a dictionary key or other token-like keyword, return a prettier form of it use as a display title.

Example:
keyword_as_title(‘foo’) => ‘Foo’ keyword_as_title(‘some_text’) => ‘Some Text’
Parameters:keyword –
Returns:a string which is the keyword in title case with underscores replaced by spaces.
submit_cgap.utils.script_catch_errors()
submit_cgap.utils.show(*args, with_time: bool = False, same_line: bool = False)

Prints its args space-separated, as ‘print’ would, possibly with an hh:mm:ss timestamp prepended.

Parameters:
  • args – an object to be printed
  • with_time – a boolean specifying whether to prepend a timestamp
  • same_line – a boolean saying whether to do this output in a way that erases the current line and returns to the start of the line without advancing vertically so that subsequent same_line=True requests will erase (and so replace) the current line.