DatasetOperations¶
Access via client.datasets.
File operations are also available via client.files (by MFID) and ingestion via client.ingestions.
crucible.resources.datasets.DatasetOperations
¶
Dataset-related API operations.
Access via: client.datasets.get(), client.datasets.list(), etc.
get(dataset_mfid, include_metadata=False, include_links=False, include_owner=True)
¶
Get a dataset by its canonical MFID.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
include_metadata
|
bool
|
Whether to include scientific metadata |
False
|
include_links
|
bool
|
Whether to include immediate parent/child/associated links |
False
|
include_owner
|
bool
|
Resolve the owner into a public-safe user object (default: True) |
True
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Optional[Dict]
|
Dataset object with optional metadata and links |
list(sample_mfid=None, include_metadata=False, include_links=False, include_owner=False, limit=DEFAULT_LIMIT, offset=0, accessible_to_user=None, accessible_to_project=None, instrument_mfid=None, instrument_id=None, project_id=None, project_mfid=None, project_scope=None, sort=None, direction=None, visibility=None, affiliation=None, include_total=None, creation_time_gte=None, creation_time_lte=None, modification_time_gte=None, modification_time_lte=None, cursor=None, mfid=None, **kwargs)
¶
List datasets with optional filtering and automatic pagination.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
sample_mfid
|
str
|
If provided, returns datasets for this sample |
None
|
mfid
|
str or list
|
Only these dataset MFIDs (up to 100). Unreadable, missing, or pending-deletion MFIDs are left out. |
None
|
limit
|
int
|
Maximum total results to return (default: 100). Larger requests are handled transparently by following the server's keyset cursor. Pass None to fetch all matches. |
DEFAULT_LIMIT
|
offset
|
int
|
Removed in 4.0.0; a nonzero value raises RemovedError. Pass the previous result's next_cursor as cursor. |
0
|
cursor
|
str
|
Continuation token from a previous result's next_cursor; the listing resumes there. |
None
|
include_metadata
|
bool
|
Include scientific metadata in results |
False
|
include_links
|
bool
|
Include linked resources (parents, children, associated) per dataset |
False
|
include_owner
|
bool
|
Resolve the owner into a public-safe user object per dataset |
False
|
accessible_to_user
|
Optional[Union[str, Sequence[str]]]
|
User reference or references whose effective access must include every result |
None
|
accessible_to_project
|
Optional[Union[str, Sequence[str]]]
|
Project reference or references whose direct access must include every result |
None
|
instrument_mfid
|
Optional[str]
|
Canonical instrument MFID assigned to every result |
None
|
instrument_id
|
Optional[str]
|
Instrument slug (any casing), resolved to its MFID. May accompany a matching instrument_mfid. |
None
|
project_id
|
Optional[str]
|
Project slug used to scope results by project relationship |
None
|
project_mfid
|
Optional[str]
|
Canonical project MFID used to scope results by project relationship |
None
|
project_scope
|
Optional[str]
|
Project relationship to include: assigned, shared, or all. Requires project_id or project_mfid and defaults to assigned. |
None
|
sort
|
Optional[str]
|
Explicit ordering, one of crucible.constants.RESOURCE_SORTS. Omit to keep the legacy unique_id descending order. |
None
|
direction
|
Optional[str]
|
Sort direction, asc or desc. Requires sort. |
None
|
visibility
|
Optional[str]
|
Restrict results by public visibility: all, public, or private. |
None
|
affiliation
|
Optional[Union[str, Sequence[str]]]
|
Caller relationship each result must have: 'owner', 'member' (reached through the caller's own grants), or 'instrument' (read as a member of the dataset's instrument). Accepts a single value or a sequence (OR). |
None
|
include_total
|
Optional[bool]
|
Request the exact authorization-filtered total. Omit to keep the endpoint's default behavior. |
None
|
creation_time_gte
|
Optional[Union[str, date, datetime]]
|
Inclusive lower creation-time bound. Accepts a datetime, date, or ISO 8601 string. |
None
|
creation_time_lte
|
Optional[Union[str, date, datetime]]
|
Inclusive upper creation-time bound, same types. |
None
|
modification_time_gte
|
Optional[Union[str, date, datetime]]
|
Inclusive lower modification-time bound, same types. |
None
|
modification_time_lte
|
Optional[Union[str, date, datetime]]
|
Inclusive upper modification-time bound, same types. |
None
|
**kwargs
|
Any
|
Query parameters for filtering. Supported fields include: keyword, public, dataset_name, owner_ref (up to 10 owners), owner_is_null, timestamp, size, data_format, data_type, measurement, and session_name. Filters expect exact matches (case sensitive) except for keywords, which are case insensitive and match substrings. |
{}
|
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: Dataset objects matching filter criteria |
count(project_id=None, project_mfid=None, project_scope=None, **kwargs)
¶
Return the total number of datasets matching the given filters without fetching items.
facets(field, limit=100, sort=None, direction=None, cursor=None, **filters)
¶
Group datasets into value buckets with counts, without fetching records.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
field
|
str
|
Field to group by, one of crucible.constants.DATASET_FACET_FIELDS. |
required |
limit
|
int
|
Maximum number of buckets to return (default: 100, max 1000). |
100
|
sort
|
Optional[str]
|
Bucket ordering, one of crucible.constants.FACET_SORTS (default: value). |
None
|
direction
|
Optional[str]
|
Bucket ordering direction, asc or desc (default: asc). |
None
|
cursor
|
Optional[str]
|
Continuation cursor from a previous response's next_cursor. |
None
|
**filters
|
Any
|
Any dataset list filter, restricting which datasets contribute to the counts, including instrument_id. |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
field, items (buckets with value, label, count, and a typed owner/project/instrument reference where applicable), limit, and next_cursor. |
search(q, project_id=None, limit=20, owner_ref=None)
¶
Fuzzy name search across datasets. Available to all authenticated users.
Matches against dataset_name. Returns datasets the caller can read. For scientific metadata search use search_metadata().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
q
|
str
|
Search term (min 3 chars). Typo-tolerant — "pero" finds "perovskite". |
required |
project_id
|
Optional[str]
|
Optional project to scope results to. |
None
|
limit
|
int
|
Max results (default 20, max 50). |
20
|
owner_ref
|
Optional[Union[str, Sequence[str]]]
|
Owner or owners (up to 10, any matches): unique_id, linked ORCID, username, or email. |
None
|
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: Matching DatasetResponse records, ranked by relevance. |
create(dataset, scientific_metadata=None, keywords=None, files=None, upload_files=True, files_to_upload=None, ingestor=None, verbose=False, wait_for_ingestion_response=False, skip_ingestion=False)
¶
Create a new dataset record with scientific metadata and keywords.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset
|
Dataset or dict
|
The dataset's fields, as a Dataset model or a
dict validated through it. Use |
required |
scientific_metadata
|
dict
|
Scientific metadata |
None
|
keywords
|
list
|
Keywords to associate with dataset |
None
|
files
|
list
|
Files to attach. Each item is either a local path (str) or an AssociatedFile describing a file that lives elsewhere (Globus, NERSC, a shared filesystem, etc.). - A str path is uploaded to GCS when upload_files=True (default), or cataloged without upload when upload_files=False: it is hashed, named by its path relative to the deepest folder holding all such paths, and recorded with its absolute path and storage_backend='local' (see AssociatedFile.from_local_files()). - An AssociatedFile is always cataloged via add_remote_file() - it must set storage_backend to something other than 'gcs', since there's no local file to upload from a model description. |
None
|
upload_files
|
bool
|
Whether str paths in |
True
|
files_to_upload
|
Optional[List[str]]
|
Removed in 4.0.0; passing it raises RemovedError. Use |
None
|
Returns:
Dict: created_record, scientific_metadata_record, dataset_mfid, files
(files is the per-item result of adding each entry in files,
in the same order). dsid is retained for compatibility.
update(dataset_mfid, **updates)
¶
Update an existing dataset with new field values.
'owner_orcid' and 'project_id' are no longer accepted here (422) - use transfer_ownership() / reassign_project() instead.
Instrument assignment is not available through generic PATCH - use assign_instrument().
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
**updates
|
Any
|
Fields to update (e.g., dataset_name="New Name") |
{}
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Updated dataset object |
Example
client.datasets.update("my-dataset-id", dataset_name="Updated Name")
add_file(dataset_mfid, file_path, ingestion_class=None, wait_for_ingestion_response=False, multipart=None, chunk_size_mb=None, max_workers=None, skip_ingestion=False, upload_protocol=None, progress=None, replace=False)
¶
Upload a file to a dataset and request ingestion.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
file_path
|
str
|
Local path to the file |
required |
ingestion_class
|
Optional[str]
|
Ingestion class for the worker (e.g. 'lammps', 'nexus'). Defaults to the server-side default if omitted. With skip_ingestion it is recorded on the not-requested ingestion request. |
None
|
wait_for_ingestion_response
|
bool
|
Block until ingestion completes. Ignored if skip_ingestion is True. |
False
|
multipart
|
Optional[bool]
|
Removed in 4.0.0; passing it raises RemovedError. |
None
|
chunk_size_mb
|
Optional[int]
|
Removed in 4.0.0; passing it raises RemovedError. |
None
|
max_workers
|
Optional[int]
|
Parts uploaded in parallel (default: the upload_max_workers setting). |
None
|
skip_ingestion
|
bool
|
Upload the file without requesting ingestion. Request it later with client.files.request_ingestion(mfid). Also implied when client.request_ingestion is False. |
False
|
upload_protocol
|
Optional[str]
|
Removed in 4.0.0; passing it raises RemovedError. |
None
|
progress
|
Optional[Callable[[str, int, int], None]]
|
Called as progress(filename, bytes_done, total) while the parts upload. |
None
|
replace
|
bool
|
If the dataset already has this filename with different content, replace that file's content (same MFID and filename) instead of raising FileExistsWithDifferentContent. Also implied by client.replace_existing. |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
{'associated_file': AssociatedFileRead, 'ingestion_request': IngestionRequest} |
add_remote_file(dataset_mfid, file, replace=False)
¶
Register a file that lives outside GCS (Globus, NERSC, a shared filesystem path, etc.) without uploading it.
Crucible only catalogs the pointer here; it never verifies the file
exists or fetches bytes on your behalf. The filename is the file's
identity in the dataset: repeating it with the same sha256_hash, size,
and storage_backend returns the existing record, and any other
difference is a 409 conflict unless replace is set.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
file
|
AssociatedFile
|
Must have |
required |
replace
|
bool
|
If the dataset already has this filename with different content, update that record's hash, size, location, backend, and access note instead of failing. Defaults to client.replace_existing. |
False
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
The created, existing, or replaced file record. |
Raises:
| Type | Description |
|---|---|
HTTPError 409
|
|
HTTPError 422
|
The filename is not a canonical relative path. |
list_files(dataset_mfid)
¶
List files attached to a dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: File records (mfid, filename, storage_path, storage_backend, access_note, size, sha256_hash, dataset_mfid). For a 'gcs' file, storage_path is null until it has been ingested. |
get_download_links(dataset_mfid, disposition=None)
¶
Get signed download URLs for all ingested files in a dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
disposition
|
Optional[str]
|
"attachment" (the server default) to download, or "inline" to let a browser display the files. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Mapping of file MFID → signed URL, or None for a file whose stored location the server will not serve. Empty dict if no ingested files. |
download(dataset_mfid, file_name=None, output_dir='crucible-downloads', no_files=False, no_record=False, overwrite_existing=True, include=None, exclude=None)
¶
Download a dataset's files and optionally save its record as JSON.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
file_name
|
Optional[str]
|
Removed in 4.0.0; raises. Use include=['pattern'] with glob syntax. |
None
|
output_dir
|
str
|
Directory to save files (default: 'crucible-downloads/') |
'crucible-downloads'
|
no_files
|
bool
|
Skip file download, save record.json only. |
False
|
no_record
|
bool
|
Skip saving record.json, just download files. |
False
|
overwrite_existing
|
bool
|
Overwrite existing files (default: True) |
True
|
include
|
Optional[List[str]]
|
Glob patterns - only download matching files |
None
|
exclude
|
Optional[List[str]]
|
Glob patterns - skip matching files |
None
|
Returns:
| Type | Description |
|---|---|
List[str]
|
List[str]: Paths of all downloaded items (record.json + data files) |
get_scientific_metadata(resource_mfid)
¶
Get scientific metadata for a resource.
update_scientific_metadata(resource_mfid, metadata, overwrite=False)
¶
Add or Update scientific metadata for a resource.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
overwrite
|
bool
|
If True, replace all metadata (POST); if False, merge with existing (PATCH) |
False
|
replace_scientific_metadata(resource_mfid, metadata)
¶
Create new scientific metadata entry for a resource.
search_metadata(q, limit=20, owner_ref=None)
¶
Full-text search across scientific metadata of all accessible resources.
Results are ranked by relevance (rank field) and include resource_type, name, owner_orcid, creation_time, modification_time, and scientific_metadata. Resources with pending or approved deletion requests are excluded.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
q
|
str
|
Plain-text search query (English-language stemmed). |
required |
limit
|
int
|
Max results to return (default 20, max 50). |
20
|
owner_ref
|
Optional[Union[str, Sequence[str]]]
|
Owner or owners (up to 10, any matches): unique_id, linked ORCID, username, or email. |
None
|
Returns:
| Type | Description |
|---|---|
list
|
List of result dicts with unique_id, resource_type, name, owner_orcid, |
list
|
creation_time, modification_time, rank, and scientific_metadata. |
add_thumbnail(dataset_mfid, image, thumbnail_name=None)
¶
Encode and add a thumbnail, returning the API thumbnail record.
get_thumbnails(dataset_mfid, limit=DEFAULT_LIMIT)
¶
Get all thumbnails for a dataset.
The limit parameter is retained for compatibility. The API route
is not paginated and returns every thumbnail.
update_thumbnail(dataset_mfid, thumbnail_id, image=None, thumbnail_name=None)
¶
Rename or replace a thumbnail and return the updated API record.
delete_thumbnail(dataset_mfid, thumbnail_id)
¶
Delete a thumbnail from a dataset.
add_keyword(dataset_mfid, keyword)
¶
Add a keyword to a dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
keyword
|
str
|
Keyword/tag to associate with dataset |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Keyword object with updated usage count |
list_keywords(dataset_mfid, limit=DEFAULT_LIMIT)
¶
List keywords, optionally filtered by dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
limit
|
int
|
Maximum number of results to return |
DEFAULT_LIMIT
|
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: Keyword objects with keyword text and num_datasets counts |
link_sample(dataset_mfid, sample_mfid)
¶
Link a sample to a dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
sample_mfid
|
str
|
Sample MFID |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Information about the created link |
unlink_sample(dataset_mfid, sample_mfid)
¶
Remove the link between a dataset and a sample.
Requires admin permissions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID |
required |
sample_mfid
|
str
|
Sample MFID |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Deletion confirmation |
link(parent_mfid, child_mfid, relationship_type=None)
¶
Link two datasets with a parent-child relationship.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parent_mfid
|
str
|
Parent dataset MFID |
required |
child_mfid
|
str
|
Child dataset MFID |
required |
relationship_type
|
str
|
Kind of link, one of crucible.constants.RELATIONSHIP_TYPES. Describes the child relative to the parent. |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Information about the created link |
unlink(parent_mfid, child_mfid)
¶
Remove the parent-child link between two datasets.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parent_mfid
|
str
|
Parent dataset MFID |
required |
child_mfid
|
str
|
Child dataset MFID |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
Deletion confirmation |
list_parents(child_mfid, limit=DEFAULT_LIMIT, offset=0, relationship_type=None, cursor=None, **kwargs)
¶
List the parents of a given dataset with optional filtering.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
child_mfid
|
str
|
Child dataset MFID |
required |
limit
|
int
|
Maximum number of results to return |
DEFAULT_LIMIT
|
offset
|
int
|
Starting position in the full result set (default: 0) |
0
|
cursor
|
str
|
Continuation token from a previous result's next_cursor; the listing resumes there. |
None
|
relationship_type
|
str
|
Only return parents linked with this kind of link, one of crucible.constants.RELATIONSHIP_TYPES. |
None
|
**kwargs
|
Any
|
Query parameters for filtering datasets |
{}
|
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: Parent datasets |
list_children(parent_mfid, limit=DEFAULT_LIMIT, offset=0, relationship_type=None, cursor=None, **kwargs)
¶
List the children of a given dataset with optional filtering.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
parent_mfid
|
str
|
Parent dataset MFID |
required |
limit
|
int
|
Maximum number of results to return |
DEFAULT_LIMIT
|
offset
|
int
|
Starting position in the full result set (default: 0) |
0
|
cursor
|
str
|
Continuation token from a previous result's next_cursor; the listing resumes there. |
None
|
relationship_type
|
str
|
Only return children linked with this kind of link, one of crucible.constants.RELATIONSHIP_TYPES. |
None
|
**kwargs
|
Any
|
Query parameters for filtering datasets |
{}
|
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: Children datasets |
list_access(mfid)
¶
List every principal with access to a resource.
Readable by anyone with a role on the resource (not through public
access alone); the resource's capabilities.can_view_access says
whether you can. Each grant's capabilities says whether you may
remove it or change its role, and to which roles.
set_access(mfid, kind, principal, permission)
¶
Grant or change a principal's non-owner access to a resource.
revoke_access(mfid, kind, principal)
¶
Revoke a principal's access to a resource.
set_public(mfid)
¶
Grant public viewer access to a resource.
set_private(mfid)
¶
Revoke public access to a resource.
transfer_ownership(mfid, new_owner, confirm=False)
¶
Preview or apply a resource ownership transfer.
reassign_project(mfid, project_id=None, confirm=False, *, project_mfid=None)
¶
Preview or apply reassignment to another project.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mfid
|
str
|
Resource MFID. |
required |
project_id
|
Optional[str]
|
Target project slug. |
None
|
confirm
|
bool
|
False previews the reassignment, True applies it. |
False
|
project_mfid
|
Optional[str]
|
Target project MFID; preferred over project_id. |
None
|
assign_instrument(dataset_mfid, instrument_id=None, instrument_mfid=None)
¶
Assign or reassign the dataset's instrument, by slug or MFID.
Sets instrument_id, instrument_mfid, and instrument_name together from the resolved instrument, and grants the instrument's access group contributor on the dataset. Reassignment drops the outgoing instrument's grant only. Requires manage_acl on the dataset.
instrument_name is not settable: it is a label derived from the instrument, so the server always overwrites it.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID. |
required |
instrument_id
|
Optional[str]
|
Registered instrument slug. |
None
|
instrument_mfid
|
Optional[str]
|
Canonical instrument MFID. If both are given they must identify the same instrument (422 otherwise). |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
InstrumentAssignment |
InstrumentAssignment
|
the new instrument plus what it replaced. |
Example
client.datasets.assign_instrument("
", instrument_id="xrd-1")
postprocess(dataset_mfid, pipeline, parameters=None, idempotency_key=None)
¶
Queue a postprocessing pipeline on a dataset. Requires contributor access.
Each call starts a new run unless it repeats an idempotency key: the same key, caller, and dataset return the original run, so a retried request cannot start a second one. Follow the run with client.postprocessing.get(run_id).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
str
|
Dataset MFID. |
required |
pipeline
|
str
|
Pipeline name from client.postprocessing.pipelines(). |
required |
parameters
|
dict
|
Pipeline parameters, validated by the server against the pipeline's parameters_schema. |
None
|
idempotency_key
|
str
|
Up to 128 characters identifying this request, e.g. f"{flow_run_id}:{pipeline}:{dataset_mfid}". |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
The PostprocessingRun record (status "requested" for a new run). |
Raises:
| Type | Description |
|---|---|
HTTPError 422
|
Unknown pipeline or invalid parameters. |