FileOperations¶
Access via client.files. Operates on individual files by MFID.
For dataset-scoped file operations (upload, bulk download) use client.datasets.
For ingestion management use client.ingestions.
crucible.resources.files.FileOperations
¶
Operations on individual files by MFID.
Access via: client.files.get(), client.files.list(), etc.
get(file_mfid)
¶
Get metadata for a single file by its MFID.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_mfid
|
str
|
File MFID |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
File record (mfid, filename, storage_path, storage_backend, access_note, size, sha256_hash, dataset_mfid) |
list(limit=DEFAULT_LIMIT, sha256_hash=None, offset=0, cursor=None)
¶
List files across all accessible datasets.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
limit
|
int
|
Maximum number of results |
DEFAULT_LIMIT
|
sha256_hash
|
Optional[str]
|
Filter by SHA-256 hex digest |
None
|
offset
|
int
|
Starting position in the full result set (default: 0). |
0
|
cursor
|
Optional[str]
|
Continuation token from a previous result's next_cursor; the listing resumes there. |
None
|
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: File records (mfid, filename, storage_path, storage_backend, access_note, size, sha256_hash, dataset_mfid) |
download(file_mfid, output_dir='.')
¶
Download a single file by MFID to a local directory.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_mfid
|
str
|
File MFID |
required |
output_dir
|
str
|
Directory to save the file (default: current directory) |
'.'
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Path of the downloaded file |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If the file has not been ingested yet, or if it's not stored on GCS (Crucible cannot fetch it directly in that case). |
get_download_link(file_mfid, disposition=None)
¶
Get a signed download URL for a single file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_mfid
|
str
|
File MFID |
required |
disposition
|
Optional[str]
|
"attachment" (the server default) to download, or "inline" to let a browser display the file. HTML, SVG, and XML always download. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
str |
str
|
Signed URL valid for 1 hour, no auth required. |
Raises:
| Type | Description |
|---|---|
HTTPError 404
|
File has not been ingested yet. |
request_ingestion(file_mfid, ingestion_class=None, wait_for_response=False)
¶
Request ingestion of an uploaded file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_mfid
|
str
|
File MFID |
required |
ingestion_class
|
Optional[str]
|
Ingestion class for the worker (e.g. 'lammps', 'nexus'). Defaults to the server-side default if omitted. |
None
|
wait_for_response
|
bool
|
Block until ingestion completes. |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
IngestionRequest record (id, status, ...); with |
Dict
|
wait_for_response, the record in its final state. |
replace(file_mfid, file_path, etag=None, ingestion_class=None, wait_for_ingestion_response=False, skip_ingestion=False, max_workers=None, progress=None)
¶
Replace a GCS file's content, keeping its MFID, filename, and metadata.
The new content uploads through the signed multipart flow and the record is repointed once it is verified; the file stays downloadable meanwhile. Needs EDIT on the file's dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_mfid
|
str
|
MFID of the file to replace. |
required |
file_path
|
str
|
Local path to the new content. |
required |
etag
|
Optional[str]
|
The file's |
None
|
ingestion_class
|
Optional[str]
|
Ingestion class for the worker; defaults to server auto-detection. With skip_ingestion it is recorded on the not-requested ingestion request. |
None
|
wait_for_ingestion_response
|
bool
|
Block until ingestion completes. |
False
|
skip_ingestion
|
bool
|
Replace without requesting ingestion. Also implied when client.request_ingestion is False. |
False
|
max_workers
|
Optional[int]
|
Parts uploaded in parallel (default 8). |
None
|
progress
|
Optional[Callable[[str, int, int], None]]
|
Called as progress(filename, bytes_done, total); defaults to client.upload_progress. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
{'associated_file': the updated record, 'ingestion_request': |
Dict
|
IngestionRequest or None}. No ingestion is requested when the file |
|
Dict
|
already held this content. |
update(file_mfid, **updates)
¶
Update fields on a file record.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_mfid
|
str
|
File MFID |
required |
**updates
|
Any
|
Fields to update, e.g. storage_path='...', access_note='...' |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Updated file record. |
Raises:
| Type | Description |
|---|---|
HTTPError 409
|
|
delete(file_mfid)
¶
Delete a file record by its MFID.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_mfid
|
str
|
File MFID (AssociatedFile mfid) |
required |