Skip to content

DatasetOperations

Access via client.datasets.

File operations are also available via client.files (by MFID) and ingestion via client.ingestions.

crucible.resources.datasets.DatasetOperations

Dataset-related API operations.

Access via: client.datasets.get(), client.datasets.list(), etc.

get(dataset_mfid, include_metadata=False, include_links=False, include_owner=True)

Get a dataset by its canonical MFID.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
include_metadata bool

Whether to include scientific metadata

False
include_links bool

Whether to include immediate parent/child/associated links

False
include_owner bool

Resolve the owner into a public-safe user object (default: True)

True

Returns:

Name Type Description
Dict Optional[Dict]

Dataset object with optional metadata and links

list(sample_mfid=None, include_metadata=False, include_links=False, include_owner=False, limit=DEFAULT_LIMIT, offset=0, accessible_to_user=None, accessible_to_project=None, instrument_mfid=None, instrument_id=None, project_id=None, project_mfid=None, project_scope=None, sort=None, direction=None, visibility=None, affiliation=None, include_total=None, creation_time_gte=None, creation_time_lte=None, modification_time_gte=None, modification_time_lte=None, cursor=None, mfid=None, **kwargs)

List datasets with optional filtering and automatic pagination.

Parameters:

Name Type Description Default
sample_mfid str

If provided, returns datasets for this sample

None
mfid str or list

Only these dataset MFIDs (up to 100). Unreadable, missing, or pending-deletion MFIDs are left out.

None
limit int

Maximum total results to return (default: 100). Larger requests are handled transparently by following the server's keyset cursor. Pass None to fetch all matches.

DEFAULT_LIMIT
offset int

Removed in 4.0.0; a nonzero value raises RemovedError. Pass the previous result's next_cursor as cursor.

0
cursor str

Continuation token from a previous result's next_cursor; the listing resumes there.

None
include_metadata bool

Include scientific metadata in results

False
include_links bool

Include linked resources (parents, children, associated) per dataset

False
include_owner bool

Resolve the owner into a public-safe user object per dataset

False
accessible_to_user Optional[Union[str, Sequence[str]]]

User reference or references whose effective access must include every result

None
accessible_to_project Optional[Union[str, Sequence[str]]]

Project reference or references whose direct access must include every result

None
instrument_mfid Optional[str]

Canonical instrument MFID assigned to every result

None
instrument_id Optional[str]

Instrument slug (any casing), resolved to its MFID. May accompany a matching instrument_mfid.

None
project_id Optional[str]

Project slug used to scope results by project relationship

None
project_mfid Optional[str]

Canonical project MFID used to scope results by project relationship

None
project_scope Optional[str]

Project relationship to include: assigned, shared, or all. Requires project_id or project_mfid and defaults to assigned.

None
sort Optional[str]

Explicit ordering, one of crucible.constants.RESOURCE_SORTS. Omit to keep the legacy unique_id descending order.

None
direction Optional[str]

Sort direction, asc or desc. Requires sort.

None
visibility Optional[str]

Restrict results by public visibility: all, public, or private.

None
affiliation Optional[Union[str, Sequence[str]]]

Caller relationship each result must have: 'owner', 'member' (reached through the caller's own grants), or 'instrument' (read as a member of the dataset's instrument). Accepts a single value or a sequence (OR).

None
include_total Optional[bool]

Request the exact authorization-filtered total. Omit to keep the endpoint's default behavior.

None
creation_time_gte Optional[Union[str, date, datetime]]

Inclusive lower creation-time bound. Accepts a datetime, date, or ISO 8601 string.

None
creation_time_lte Optional[Union[str, date, datetime]]

Inclusive upper creation-time bound, same types.

None
modification_time_gte Optional[Union[str, date, datetime]]

Inclusive lower modification-time bound, same types.

None
modification_time_lte Optional[Union[str, date, datetime]]

Inclusive upper modification-time bound, same types.

None
**kwargs Any

Query parameters for filtering. Supported fields include: keyword, public, dataset_name, owner_ref (up to 10 owners), owner_is_null, timestamp, size, data_format, data_type, measurement, and session_name. Filters expect exact matches (case sensitive) except for keywords, which are case insensitive and match substrings.

{}

Returns:

Type Description
List[Dict]

List[Dict]: Dataset objects matching filter criteria

count(project_id=None, project_mfid=None, project_scope=None, **kwargs)

Return the total number of datasets matching the given filters without fetching items.

facets(field, limit=100, sort=None, direction=None, cursor=None, **filters)

Group datasets into value buckets with counts, without fetching records.

Parameters:

Name Type Description Default
field str

Field to group by, one of crucible.constants.DATASET_FACET_FIELDS.

required
limit int

Maximum number of buckets to return (default: 100, max 1000).

100
sort Optional[str]

Bucket ordering, one of crucible.constants.FACET_SORTS (default: value).

None
direction Optional[str]

Bucket ordering direction, asc or desc (default: asc).

None
cursor Optional[str]

Continuation cursor from a previous response's next_cursor.

None
**filters Any

Any dataset list filter, restricting which datasets contribute to the counts, including instrument_id.

{}

Returns:

Name Type Description
Dict Dict

field, items (buckets with value, label, count, and a typed owner/project/instrument reference where applicable), limit, and next_cursor.

search(q, project_id=None, limit=20, owner_ref=None)

Fuzzy name search across datasets. Available to all authenticated users.

Matches against dataset_name. Returns datasets the caller can read. For scientific metadata search use search_metadata().

Parameters:

Name Type Description Default
q str

Search term (min 3 chars). Typo-tolerant — "pero" finds "perovskite".

required
project_id Optional[str]

Optional project to scope results to.

None
limit int

Max results (default 20, max 50).

20
owner_ref Optional[Union[str, Sequence[str]]]

Owner or owners (up to 10, any matches): unique_id, linked ORCID, username, or email.

None

Returns:

Type Description
List[Dict]

List[Dict]: Matching DatasetResponse records, ranked by relevance.

create(dataset, scientific_metadata=None, keywords=None, files=None, upload_files=True, files_to_upload=None, ingestor=None, verbose=False, wait_for_ingestion_response=False, skip_ingestion=False)

Create a new dataset record with scientific metadata and keywords.

Parameters:

Name Type Description Default
dataset Dataset or dict

The dataset's fields, as a Dataset model or a dict validated through it. Use owner with an ORCID, MFID, username, or email to create for a specific owner; owner_orcid raises RemovedError. project_id and instrument_id accept human-readable IDs; project_mfid and instrument_mfid accept canonical MFIDs. Matching ID and MFID selectors may be supplied together.

required
scientific_metadata dict

Scientific metadata

None
keywords list

Keywords to associate with dataset

None
files list

Files to attach. Each item is either a local path (str) or an AssociatedFile describing a file that lives elsewhere (Globus, NERSC, a shared filesystem, etc.). - A str path is uploaded to GCS when upload_files=True (default), or cataloged without upload when upload_files=False: it is hashed, named by its path relative to the deepest folder holding all such paths, and recorded with its absolute path and storage_backend='local' (see AssociatedFile.from_local_files()). - An AssociatedFile is always cataloged via add_remote_file() - it must set storage_backend to something other than 'gcs', since there's no local file to upload from a model description.

None
upload_files bool

Whether str paths in files are uploaded to GCS (default: True) or just cataloged by their local path (False). Only affects str items - AssociatedFile items are routed by their own storage_backend regardless of this flag.

True
files_to_upload Optional[List[str]]

Removed in 4.0.0; passing it raises RemovedError. Use files.

None

Returns: Dict: created_record, scientific_metadata_record, dataset_mfid, files (files is the per-item result of adding each entry in files, in the same order). dsid is retained for compatibility.

update(dataset_mfid, **updates)

Update an existing dataset with new field values.

'owner_orcid' and 'project_id' are no longer accepted here (422) - use transfer_ownership() / reassign_project() instead.

Instrument assignment is not available through generic PATCH - use assign_instrument().

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
**updates Any

Fields to update (e.g., dataset_name="New Name")

{}

Returns:

Name Type Description
Dict Dict

Updated dataset object

Example

client.datasets.update("my-dataset-id", dataset_name="Updated Name")

add_file(dataset_mfid, file_path, ingestion_class=None, wait_for_ingestion_response=False, multipart=None, chunk_size_mb=None, max_workers=None, skip_ingestion=False, upload_protocol=None, progress=None, replace=False)

Upload a file to a dataset and request ingestion.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
file_path str

Local path to the file

required
ingestion_class Optional[str]

Ingestion class for the worker (e.g. 'lammps', 'nexus'). Defaults to the server-side default if omitted. With skip_ingestion it is recorded on the not-requested ingestion request.

None
wait_for_ingestion_response bool

Block until ingestion completes. Ignored if skip_ingestion is True.

False
multipart Optional[bool]

Removed in 4.0.0; passing it raises RemovedError.

None
chunk_size_mb Optional[int]

Removed in 4.0.0; passing it raises RemovedError.

None
max_workers Optional[int]

Parts uploaded in parallel (default: the upload_max_workers setting).

None
skip_ingestion bool

Upload the file without requesting ingestion. Request it later with client.files.request_ingestion(mfid). Also implied when client.request_ingestion is False.

False
upload_protocol Optional[str]

Removed in 4.0.0; passing it raises RemovedError.

None
progress Optional[Callable[[str, int, int], None]]

Called as progress(filename, bytes_done, total) while the parts upload.

None
replace bool

If the dataset already has this filename with different content, replace that file's content (same MFID and filename) instead of raising FileExistsWithDifferentContent. Also implied by client.replace_existing.

False

Returns:

Name Type Description
Dict Dict

{'associated_file': AssociatedFileRead, 'ingestion_request': IngestionRequest}

add_remote_file(dataset_mfid, file, replace=False)

Register a file that lives outside GCS (Globus, NERSC, a shared filesystem path, etc.) without uploading it.

Crucible only catalogs the pointer here; it never verifies the file exists or fetches bytes on your behalf. The filename is the file's identity in the dataset: repeating it with the same sha256_hash, size, and storage_backend returns the existing record, and any other difference is a 409 conflict unless replace is set.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
file AssociatedFile

Must have storage_backend set to something other than 'gcs' (e.g. 'globus', 'local'), size, and sha256_hash. filename must be a canonical relative POSIX path such as "freq_001/kpoints.json" (no leading, trailing, or repeated "/", no "." or ".." components); the server does not normalize it. storage_path is optional; omit it to catalog the file now and set its location later via client.files.update(mfid, storage_path=...). AssociatedFile.from_local_files() builds these for local files.

required
replace bool

If the dataset already has this filename with different content, update that record's hash, size, location, backend, and access note instead of failing. Defaults to client.replace_existing.

False

Returns:

Name Type Description
Dict Dict

The created, existing, or replaced file record.

Raises:

Type Description
HTTPError 409

filename_exists without replace, or storage_backend_class_change when replacing a GCS file.

HTTPError 422

The filename is not a canonical relative path.

list_files(dataset_mfid)

List files attached to a dataset.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required

Returns:

Type Description
List[Dict]

List[Dict]: File records (mfid, filename, storage_path, storage_backend, access_note, size, sha256_hash, dataset_mfid). For a 'gcs' file, storage_path is null until it has been ingested.

Get signed download URLs for all ingested files in a dataset.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
disposition Optional[str]

"attachment" (the server default) to download, or "inline" to let a browser display the files.

None

Returns:

Name Type Description
Dict Dict

Mapping of file MFID → signed URL, or None for a file whose stored location the server will not serve. Empty dict if no ingested files.

download(dataset_mfid, file_name=None, output_dir='crucible-downloads', no_files=False, no_record=False, overwrite_existing=True, include=None, exclude=None)

Download a dataset's files and optionally save its record as JSON.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
file_name Optional[str]

Removed in 4.0.0; raises. Use include=['pattern'] with glob syntax.

None
output_dir str

Directory to save files (default: 'crucible-downloads/')

'crucible-downloads'
no_files bool

Skip file download, save record.json only.

False
no_record bool

Skip saving record.json, just download files.

False
overwrite_existing bool

Overwrite existing files (default: True)

True
include Optional[List[str]]

Glob patterns - only download matching files

None
exclude Optional[List[str]]

Glob patterns - skip matching files

None

Returns:

Type Description
List[str]

List[str]: Paths of all downloaded items (record.json + data files)

get_scientific_metadata(resource_mfid)

Get scientific metadata for a resource.

update_scientific_metadata(resource_mfid, metadata, overwrite=False)

Add or Update scientific metadata for a resource.

Parameters:

Name Type Description Default
overwrite bool

If True, replace all metadata (POST); if False, merge with existing (PATCH)

False

replace_scientific_metadata(resource_mfid, metadata)

Create new scientific metadata entry for a resource.

search_metadata(q, limit=20, owner_ref=None)

Full-text search across scientific metadata of all accessible resources.

Results are ranked by relevance (rank field) and include resource_type, name, owner_orcid, creation_time, modification_time, and scientific_metadata. Resources with pending or approved deletion requests are excluded.

Parameters:

Name Type Description Default
q str

Plain-text search query (English-language stemmed).

required
limit int

Max results to return (default 20, max 50).

20
owner_ref Optional[Union[str, Sequence[str]]]

Owner or owners (up to 10, any matches): unique_id, linked ORCID, username, or email.

None

Returns:

Type Description
list

List of result dicts with unique_id, resource_type, name, owner_orcid,

list

creation_time, modification_time, rank, and scientific_metadata.

add_thumbnail(dataset_mfid, image, thumbnail_name=None)

Encode and add a thumbnail, returning the API thumbnail record.

get_thumbnails(dataset_mfid, limit=DEFAULT_LIMIT)

Get all thumbnails for a dataset.

The limit parameter is retained for compatibility. The API route is not paginated and returns every thumbnail.

update_thumbnail(dataset_mfid, thumbnail_id, image=None, thumbnail_name=None)

Rename or replace a thumbnail and return the updated API record.

delete_thumbnail(dataset_mfid, thumbnail_id)

Delete a thumbnail from a dataset.

add_keyword(dataset_mfid, keyword)

Add a keyword to a dataset.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
keyword str

Keyword/tag to associate with dataset

required

Returns:

Name Type Description
Dict Dict

Keyword object with updated usage count

list_keywords(dataset_mfid, limit=DEFAULT_LIMIT)

List keywords, optionally filtered by dataset.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
limit int

Maximum number of results to return

DEFAULT_LIMIT

Returns:

Type Description
List[Dict]

List[Dict]: Keyword objects with keyword text and num_datasets counts

Link a sample to a dataset.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
sample_mfid str

Sample MFID

required

Returns:

Name Type Description
Dict Dict

Information about the created link

Remove the link between a dataset and a sample.

Requires admin permissions.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID

required
sample_mfid str

Sample MFID

required

Returns:

Name Type Description
Dict Dict

Deletion confirmation

Link two datasets with a parent-child relationship.

Parameters:

Name Type Description Default
parent_mfid str

Parent dataset MFID

required
child_mfid str

Child dataset MFID

required
relationship_type str

Kind of link, one of crucible.constants.RELATIONSHIP_TYPES. Describes the child relative to the parent.

None

Returns:

Name Type Description
Dict Dict

Information about the created link

Remove the parent-child link between two datasets.

Parameters:

Name Type Description Default
parent_mfid str

Parent dataset MFID

required
child_mfid str

Child dataset MFID

required

Returns:

Name Type Description
Dict Dict

Deletion confirmation

list_parents(child_mfid, limit=DEFAULT_LIMIT, offset=0, relationship_type=None, cursor=None, **kwargs)

List the parents of a given dataset with optional filtering.

Parameters:

Name Type Description Default
child_mfid str

Child dataset MFID

required
limit int

Maximum number of results to return

DEFAULT_LIMIT
offset int

Starting position in the full result set (default: 0)

0
cursor str

Continuation token from a previous result's next_cursor; the listing resumes there.

None
relationship_type str

Only return parents linked with this kind of link, one of crucible.constants.RELATIONSHIP_TYPES.

None
**kwargs Any

Query parameters for filtering datasets

{}

Returns:

Type Description
List[Dict]

List[Dict]: Parent datasets

list_children(parent_mfid, limit=DEFAULT_LIMIT, offset=0, relationship_type=None, cursor=None, **kwargs)

List the children of a given dataset with optional filtering.

Parameters:

Name Type Description Default
parent_mfid str

Parent dataset MFID

required
limit int

Maximum number of results to return

DEFAULT_LIMIT
offset int

Starting position in the full result set (default: 0)

0
cursor str

Continuation token from a previous result's next_cursor; the listing resumes there.

None
relationship_type str

Only return children linked with this kind of link, one of crucible.constants.RELATIONSHIP_TYPES.

None
**kwargs Any

Query parameters for filtering datasets

{}

Returns:

Type Description
List[Dict]

List[Dict]: Children datasets

list_access(mfid)

List every principal with access to a resource.

Readable by anyone with a role on the resource (not through public access alone); the resource's capabilities.can_view_access says whether you can. Each grant's capabilities says whether you may remove it or change its role, and to which roles.

set_access(mfid, kind, principal, permission)

Grant or change a principal's non-owner access to a resource.

revoke_access(mfid, kind, principal)

Revoke a principal's access to a resource.

set_public(mfid)

Grant public viewer access to a resource.

set_private(mfid)

Revoke public access to a resource.

transfer_ownership(mfid, new_owner, confirm=False)

Preview or apply a resource ownership transfer.

reassign_project(mfid, project_id=None, confirm=False, *, project_mfid=None)

Preview or apply reassignment to another project.

Parameters:

Name Type Description Default
mfid str

Resource MFID.

required
project_id Optional[str]

Target project slug.

None
confirm bool

False previews the reassignment, True applies it.

False
project_mfid Optional[str]

Target project MFID; preferred over project_id.

None

assign_instrument(dataset_mfid, instrument_id=None, instrument_mfid=None)

Assign or reassign the dataset's instrument, by slug or MFID.

Sets instrument_id, instrument_mfid, and instrument_name together from the resolved instrument, and grants the instrument's access group contributor on the dataset. Reassignment drops the outgoing instrument's grant only. Requires manage_acl on the dataset.

instrument_name is not settable: it is a label derived from the instrument, so the server always overwrites it.

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID.

required
instrument_id Optional[str]

Registered instrument slug.

None
instrument_mfid Optional[str]

Canonical instrument MFID. If both are given they must identify the same instrument (422 otherwise).

None

Returns:

Name Type Description
InstrumentAssignment InstrumentAssignment

the new instrument plus what it replaced.

Example

client.datasets.assign_instrument("", instrument_id="xrd-1")

postprocess(dataset_mfid, pipeline, parameters=None, idempotency_key=None)

Queue a postprocessing pipeline on a dataset. Requires contributor access.

Each call starts a new run unless it repeats an idempotency key: the same key, caller, and dataset return the original run, so a retried request cannot start a second one. Follow the run with client.postprocessing.get(run_id).

Parameters:

Name Type Description Default
dataset_mfid str

Dataset MFID.

required
pipeline str

Pipeline name from client.postprocessing.pipelines().

required
parameters dict

Pipeline parameters, validated by the server against the pipeline's parameters_schema.

None
idempotency_key str

Up to 128 characters identifying this request, e.g. f"{flow_run_id}:{pipeline}:{dataset_mfid}".

None

Returns:

Name Type Description
Dict Dict

The PostprocessingRun record (status "requested" for a new run).

Raises:

Type Description
HTTPError 422

Unknown pipeline or invalid parameters.