Skip to content

FileOperations

Access via client.files. Operates on individual files by MFID.

For dataset-scoped file operations (upload, bulk download) use client.datasets. For ingestion management use client.ingestions.

crucible.resources.files.FileOperations

Operations on individual files by MFID.

Access via: client.files.get(), client.files.list(), etc.

get(file_mfid)

Get metadata for a single file by its MFID.

Parameters:

Name Type Description Default
file_mfid str

File MFID

required

Returns:

Name Type Description
Dict Dict

File record (mfid, filename, storage_path, storage_backend, access_note, size, sha256_hash, dataset_mfid)

list(limit=DEFAULT_LIMIT, sha256_hash=None, offset=0, cursor=None)

List files across all accessible datasets.

Parameters:

Name Type Description Default
limit int

Maximum number of results

DEFAULT_LIMIT
sha256_hash Optional[str]

Filter by SHA-256 hex digest

None
offset int

Starting position in the full result set (default: 0).

0
cursor Optional[str]

Continuation token from a previous result's next_cursor; the listing resumes there.

None

Returns:

Type Description
List[Dict]

List[Dict]: File records (mfid, filename, storage_path, storage_backend, access_note, size, sha256_hash, dataset_mfid)

download(file_mfid, output_dir='.')

Download a single file by MFID to a local directory.

Parameters:

Name Type Description Default
file_mfid str

File MFID

required
output_dir str

Directory to save the file (default: current directory)

'.'

Returns:

Name Type Description
str str

Path of the downloaded file

Raises:

Type Description
RuntimeError

If the file has not been ingested yet, or if it's not stored on GCS (Crucible cannot fetch it directly in that case).

Get a signed download URL for a single file.

Parameters:

Name Type Description Default
file_mfid str

File MFID

required
disposition Optional[str]

"attachment" (the server default) to download, or "inline" to let a browser display the file. HTML, SVG, and XML always download.

None

Returns:

Name Type Description
str str

Signed URL valid for 1 hour, no auth required.

Raises:

Type Description
HTTPError 404

File has not been ingested yet.

request_ingestion(file_mfid, ingestion_class=None, wait_for_response=False)

Request ingestion of an uploaded file.

Parameters:

Name Type Description Default
file_mfid str

File MFID

required
ingestion_class Optional[str]

Ingestion class for the worker (e.g. 'lammps', 'nexus'). Defaults to the server-side default if omitted.

None
wait_for_response bool

Block until ingestion completes.

False

Returns:

Name Type Description
Dict Dict

IngestionRequest record (id, status, ...); with

Dict

wait_for_response, the record in its final state.

replace(file_mfid, file_path, etag=None, ingestion_class=None, wait_for_ingestion_response=False, skip_ingestion=False, max_workers=None, progress=None)

Replace a GCS file's content, keeping its MFID, filename, and metadata.

The new content uploads through the signed multipart flow and the record is repointed once it is verified; the file stays downloadable meanwhile. Needs EDIT on the file's dataset.

Parameters:

Name Type Description Default
file_mfid str

MFID of the file to replace.

required
file_path str

Local path to the new content.

required
etag Optional[str]

The file's etag as you last read it. The replace is then refused if the file changed since; None replaces whatever is current.

None
ingestion_class Optional[str]

Ingestion class for the worker; defaults to server auto-detection. With skip_ingestion it is recorded on the not-requested ingestion request.

None
wait_for_ingestion_response bool

Block until ingestion completes.

False
skip_ingestion bool

Replace without requesting ingestion. Also implied when client.request_ingestion is False.

False
max_workers Optional[int]

Parts uploaded in parallel (default 8).

None
progress Optional[Callable[[str, int, int], None]]

Called as progress(filename, bytes_done, total); defaults to client.upload_progress.

None

Returns:

Name Type Description
Dict Dict

{'associated_file': the updated record, 'ingestion_request':

Dict

IngestionRequest or None}. No ingestion is requested when the file

Dict

already held this content.

update(file_mfid, **updates)

Update fields on a file record.

Parameters:

Name Type Description Default
file_mfid str

File MFID

required
**updates Any

Fields to update, e.g. storage_path='...', access_note='...'

{}

Returns:

Name Type Description
Dict Dict

Updated file record.

Raises:

Type Description
HTTPError 409

filename_immutable when renaming a non-GCS file (its filename is its identity), storage_backend_class_change when moving a record between GCS and non-GCS storage, or filename_exists when renaming onto a filename already in the dataset.

delete(file_mfid)

Delete a file record by its MFID.

Parameters:

Name Type Description Default
file_mfid str

File MFID (AssociatedFile mfid)

required