submit_cgap package

Subpackages

• submit_cgap.scripts package
  • Submodules
  • submit_cgap.scripts.make_sample_fastq_file module
  • submit_cgap.scripts.resume_uploads module
  • submit_cgap.scripts.submit_metadata_bundle module
  • submit_cgap.scripts.upload_item_data module
  • Module contents
• submit_cgap.tests package
  • Submodules
  • submit_cgap.tests.test_auth module
  • submit_cgap.tests.test_base module
  • submit_cgap.tests.test_exceptions module
  • submit_cgap.tests.test_make_sample_fastq_file module
  • submit_cgap.tests.test_resume_uploads module
  • submit_cgap.tests.test_submission module
  • submit_cgap.tests.test_submit_metadata_bundle module
  • submit_cgap.tests.test_upload_item_data module
  • submit_cgap.tests.test_utils module
  • Module contents

Submodules

submit_cgap.auth module

submit_cgap.auth.get_cgap_keydicts()

submit_cgap.auth.get_keydict_for_env(env=None)

Gets the appropriate auth info for talking to a given beanstalk environment.

Args:

env: the name of a beanstalk environment

Returns:

Auth information as a dict with keys ‘key’, ‘secret’, and ‘server’.

submit_cgap.auth.get_keydict_for_server(server=None)

Gets the appropriate auth info for talking to a given beanstalk environment.

Args:

server: the name of a server

Returns:

Auth information. The auth is a keypair, though we might change this to include a JWT token in the the future.

submit_cgap.auth.get_keypair_for_env(env=None)

Gets the appropriate auth info for talking to a given beanstalk environment.

Args:

env: the name of a beanstalk environment

Returns:

Auth information as a (key, secret) tuple.

submit_cgap.auth.get_keypair_for_server(server=None)

submit_cgap.auth.keydict_to_keypair(auth_dict)

submit_cgap.auth.keypair_to_keydict(auth_tuple, server)

submit_cgap.base module

class submit_cgap.base.KeyManager

Bases: object

DEFAULT_KEYDICTS_FILENAME = '/home/docs/.cgap-keys.json'

classmethod alternate_keydicts_filename(filename)

classmethod alternate_keydicts_filename_from_environ()

classmethod keydicts_filename()

submit_cgap.base.UsingCGAPKeysFile(fn)

submit_cgap.exceptions module

exception submit_cgap.exceptions.CGAPEnvKeyMissing(env, keyfile='/home/docs/.cgap-keys.json')

Bases: submit_cgap.exceptions.CGAPKeyMissing

exception submit_cgap.exceptions.CGAPKeyMissing(context, keyfile='/home/docs/.cgap-keys.json')

Bases: RuntimeError

exception submit_cgap.exceptions.CGAPPermissionError(server)

Bases: PermissionError

exception submit_cgap.exceptions.CGAPServerKeyMissing(server, keyfile='/home/docs/.cgap-keys.json')

Bases: submit_cgap.exceptions.CGAPKeyMissing

submit_cgap.submission module

class submit_cgap.submission.UploadMessageWrapper(uuid, no_query=False)

Bases: object

Class to provide consistent queries/messages to user when uploading file(s) to given File UUID.

wrap_upload_function(function, file_name)

Wrap upload given function with messages conerning upload.

Parameters:

• function – Upload function to wrap
• file_name – File to upload

Returns:

Wrapped function

submit_cgap.submission.do_any_uploads(res, keydict, upload_folder=None, ingestion_filename=None, no_query=False, subfolders=False)

submit_cgap.submission.do_uploads(upload_spec_list, auth, folder=None, no_query=False, subfolders=False)

Uploads the files mentioned in the give upload_spec_list.

If any files have associated extra files, upload those as well.

Parameters:

• upload_spec_list – a list of upload_spec dictionaries, each of the form {‘filename’: …, ‘uuid’: …}, representing uploads to be formed.
• auth – a dictionary-form auth spec, of the form {‘key’: …, ‘secret’: …, ‘server’: …} representing destination and credentials.
• folder – a string naming a folder in which to find the filenames to be uploaded.
• no_query – bool to suppress requests for user input
• subfolders – bool to search subdirectories within upload_folder for files

Returns:

None

submit_cgap.submission.execute_prearranged_upload(path, upload_credentials, auth=None)

This performs a file upload using special credentials received from ff_utils.patch_metadata.

Parameters:

• path – the name of a local file to upload
• upload_credentials – a dictionary of credentials to be used for the upload, containing the keys ‘AccessKeyId’, ‘SecretAccessKey’, ‘SessionToken’, and ‘upload_url’.
• auth – auth info in the form of a dictionary containing ‘key’, ‘secret’, and ‘server’, and possibly other useful information such as an encryption key id.

submit_cgap.submission.get_defaulted_institution(institution, user_record)

Returns the given institution or else if none is specified, it tries to infer an institution.

Parameters:

• institution – the @id of an institution
• user_record – the user record for the authorized user

Returns:

the @id of an institution to use

submit_cgap.submission.get_defaulted_project(project, user_record)

Returns the given project or else if none is specified, it tries to infer a project.

Parameters:

• project – the @id of a project, or None
• user_record – the user record for the authorized user

Returns:

the @id of a project to use

submit_cgap.submission.get_s3_encrypt_key_id(*, upload_credentials, auth)

submit_cgap.submission.get_s3_encrypt_key_id_from_health_page(auth)

submit_cgap.submission.get_section(res, section)

Given a description of an ingestion submission, returns a section name within that ingestion.

Parameters:

• res – the description of an ingestion submission as a python dictionary that represents JSON data
• section – the name of a section to find either in the toplevel or in additional_data.

Returns:

the section’s content

submit_cgap.submission.get_user_record(server, auth)

Given a server and some auth info, gets the user record for the authorized user.

This works by using the /me endpoint.

Parameters:

• server – a server spec
• auth – auth info to be used when contacting the server

Returns:

the /me page in JSON format

submit_cgap.submission.ingestion_submission_item_url(server, uuid)

submit_cgap.submission.resolve_server(server, env)

Given a server spec or a beanstalk environment (or neither, but not both), returns a server spec.

Parameters:

• server – a server spec or None A server is the first part of a url (containing the schema, host and, optionally, port). e.g., http://cgap.hms.harvard.edu or http://localhost:8000
• env – a cgap beanstalk environment

Returns:

a server spec

submit_cgap.submission.resume_uploads(uuid, server=None, env=None, bundle_filename=None, keydict=None, upload_folder=None, no_query=False, subfolders=False)

Uploads the files associated with a given ingestion submission. This is useful if you answered “no” to the query about uploading your data and then later are ready to do that upload.

Parameters:

• uuid – a string guid that identifies the ingestion submission
• server – the server to upload to
• env – the beanstalk environment to upload to
• bundle_filename – the bundle file to be uploaded
• keydict – keydict-style auth, a dictionary of ‘key’, ‘secret’, and ‘server’
• upload_folder – folder in which to find files to upload (default: same as ingestion_filename)
• no_query – bool to suppress requests for user input
• subfolders – bool to search subdirectories within upload_folder for files

submit_cgap.submission.running_on_windows_native()

submit_cgap.submission.search_for_file(directory, file_name, recursive=False)

Search for file within directory.

Parameters:

• directory – Directory path
• file_name – Name of file to find
• recursive – Whether to search sub-directories of given directory

Returns:

(Path to file or None, Error message or None)

submit_cgap.submission.show_section(res, section, caveat_outcome=None)

Shows a given named section from a description of an ingestion submission.

The caveat is used when there has been an error and should be a phrase that describes the fact that output shown is only up to the point of the caveat situation. Instead of a “My Heading” header the output will be “My Heading (prior to <caveat>).”

Parameters:

• res – the description of an ingestion submission as a python dictionary that represents JSON data
• section – the name of a section to find either in the toplevel or in additional_data.
• caveat_outcome – a phrase describing some caveat on the output

submit_cgap.submission.show_upload_info(uuid, server=None, env=None, keydict=None)

Uploads the files associated with a given ingestion submission. This is useful if you answered “no” to the query about uploading your data and then later are ready to do that upload.

Parameters:

• uuid – a string guid that identifies the ingestion submission
• server – the server to upload to
• env – the beanstalk environment to upload to
• keydict – keydict-style auth, a dictionary of ‘key’, ‘secret’, and ‘server’

submit_cgap.submission.submit_any_ingestion(ingestion_filename, ingestion_type, institution, project, server, env, validate_only, upload_folder=None, no_query=False, subfolders=False)

Does the core action of submitting a metadata bundle.

Parameters:

• ingestion_filename – the name of the main data file to be ingested
• ingestion_type – the type of ingestion to be performed (an ingestion_type in the IngestionSubmission schema)
• institution – the @id of the institution for which the submission is being done
• project – the @id of the project for which the submission is being done
• server – the server to upload to
• env – the beanstalk environment to upload to
• validate_only – whether to do stop after validation instead of proceeding to post metadata
• upload_folder – folder in which to find files to upload (default: same as bundle_filename)
• no_query – bool to suppress requests for user input
• subfolders – bool to search subdirectories within upload_folder for files

submit_cgap.submission.upload_extra_files(credentials, uploader_wrapper, folder, auth, recursive=False)

Attempt upload of all extra files.

Similar to “do_uploads”, search for each file and then call a wrapped upload function. Here, since extra files do not correspond to Items on the portal, no need to PATCH an Item to retrieve AWS credentials; they are directly passed in from the parent File’s metadata.

Parameters:

• credentials – AWS credentials dictionary
• uploader_wrapper – UploadMessageWrapper instance
• folder – Directory to search for files
• auth – CGAP authorization tuple
• recursive – Whether to search sub-directories for file

submit_cgap.submission.upload_file_to_uuid(filename, uuid, auth)

Upload file to a target environment.

Parameters:

• filename – the name of a file to upload.
• uuid – the item into which the filename is to be uploaded.
• auth – auth info in the form of a dictionary containing ‘key’, ‘secret’, and ‘server’.

Returns:

item metadata dict or None

submit_cgap.submission.upload_item_data(item_filename, uuid, server, env, no_query=False)

Given a part_filename, uploads that filename to the Item specified by uuid on the given server.

Only one of server or env may be specified.

Parameters:

• item_filename – the name of a file to be uploaded
• uuid – the UUID of the Item with which the uploaded data is to be associated
• server – the server to upload to (where the Item is defined)
• env – the beanstalk environment to upload to (where the Item is defined)
• no_query – bool to suppress requests for user input

Returns:

submit_cgap.utils module

class submit_cgap.utils.FakeResponse(status_code, json=None, content=None)

Bases: object

json()

raise_for_status()

submit_cgap.utils.keyword_as_title(keyword)

Given a dictionary key or other token-like keyword, return a prettier form of it use as a display title.

Example:

keyword_as_title(‘foo’) => ‘Foo’ keyword_as_title(‘some_text’) => ‘Some Text’

Parameters:keyword – Returns:a string which is the keyword in title case with underscores replaced by spaces.

submit_cgap.utils.script_catch_errors()

submit_cgap.utils.show(*args, with_time=False)

Prints its args space-separated, as ‘print’ would, possibly with an hh:mm:ss timestamp prepended.

Parameters:

• args – an object to be printed
• with_time – a boolean specifying whether to prepend a timestamp

Module contents