DatasetOperations

Access via client.datasets.

File operations are also available via client.files (by MFID) and ingestion via client.ingestions.

crucible.resources.datasets.DatasetOperations

Dataset-related API operations.

Access via: client.datasets.get(), client.datasets.list(), etc.

get(dsid, include_metadata=False, include_links=False, include_owner=False)

Get dataset details, optionally including scientific metadata and links.

Parameters:

Name	Type	Description	Default
dsid	str	Dataset unique identifier	required
include_metadata	bool	Whether to include scientific metadata	False
include_links	bool	Whether to include immediate parent/child/associated links	False
include_owner	bool	Whether to resolve owner_orcid into a full user object	False

Returns:

Name	Type	Description
Dict	Dict	Dataset object with optional metadata and links

list(sample_id=None, include_metadata=False, include_links=False, include_owner=False, limit=DEFAULT_LIMIT, offset=0, **kwargs)

List datasets with optional filtering and automatic pagination.

Parameters:

Name	Type	Description	Default
sample_id	str	If provided, returns datasets for this sample	None
limit	int	Maximum total results to return (default: 100). Larger requests are handled transparently by following the server's keyset cursor. Pass None to fetch all matches.	DEFAULT_LIMIT
offset	int	Deprecated for the top-level /datasets endpoint, which now uses keyset pagination and ignores offset. Still honored for the sample_id sub-listing.	0
include_metadata	bool	Include scientific metadata in results	False
include_links	bool	Include linked resources (parents, children, associated) per dataset	False
include_owner	bool	Resolve owner_orcid into a full user object per dataset	False
**kwargs	Any	Query parameters for filtering. Supported fields include: keyword, unique_id, public, dataset_name, owner_orcid, project_id, instrument_name, timestamp, size, data_format, data_type, measurement, session_name. Filters expect exact matches (case sensitive) except for keywords, which are case insensitive and match substrings.	{}

Returns:

Type	Description
List[Dict]	List[Dict]: Dataset objects matching filter criteria

count(**kwargs)

Return the total number of datasets matching the given filters without fetching items.

search(q, project_id=None, limit=20)

Fuzzy name search across datasets. Available to all authenticated users.

Matches against dataset_name. Returns datasets the caller can read. For scientific metadata search use search_metadata().

Parameters:

Name	Type	Description	Default
q	str	Search term (min 3 chars). Typo-tolerant — "pero" finds "perovskite".	required
project_id	Optional[str]	Optional project to scope results to.	None
limit	int	Max results (default 20, max 50).	20

Returns:

Type	Description
List[Dict]	List[Dict]: Matching DatasetResponse records, ranked by relevance.

create(dataset, scientific_metadata=None, keywords=None, files=None, upload_files=True, files_to_upload=None, ingestor=None, verbose=False, wait_for_ingestion_response=False)

Create a new dataset record with scientific metadata and keywords.

Parameters:

Name	Type	Description	Default
dataset	Dataset	Dataset object with dataset details	required
scientific_metadata	dict	Scientific metadata	None
keywords	list	Keywords to associate with dataset	None
files	list	Files to attach. Each item is either a local path (str) or an AssociatedFile describing a file that lives elsewhere (Globus, NERSC, a shared filesystem, etc.). - A str path is uploaded to GCS when upload_files=True (default), or cataloged by its resolved absolute path (storage_backend='local', no upload) when upload_files=False. - An AssociatedFile is always cataloged via add_remote_file() - it must set storage_backend to something other than 'gcs', since there's no local file to upload from a model description.	None
upload_files	bool	Whether str paths in files are uploaded to GCS (default: True) or just cataloged by their local path (False). Only affects str items - AssociatedFile items are routed by their own storage_backend regardless of this flag.	True
files_to_upload	list	Deprecated alias for files (str paths only, always uploaded). Use files instead.	None

Returns: Dict: created_record, scientific_metadata_record, dsid, files (files is the per-item result of adding each entry in files, in the same order)

update(dsid, **updates)

Update an existing dataset with new field values.

Parameters:

Name	Type	Description	Default
dsid	str	Dataset unique identifier	required
**updates	Any	Fields to update (e.g., dataset_name="New Name", public=True)	{}

Returns:

Name	Type	Description
Dict	Dict	Updated dataset object

Example

client.datasets.update("my-dataset-id", dataset_name="Updated Name", public=True)

add_file(dsid, file_path, ingestion_class=None, wait_for_ingestion_response=False, multipart=True, chunk_size_mb=None, max_workers=None)

Upload a file to a dataset and request ingestion.

Parameters:

Name	Type	Description	Default
dsid	str	Dataset unique identifier	required
file_path	str	Local path to the file	required
ingestion_class	Optional[str]	Ingestion class for the worker (e.g. 'lammps', 'nexus'). Defaults to the server-side default if omitted.	None
wait_for_ingestion_response	bool	Block until ingestion completes.	False
multipart	bool	Use parallel multipart upload (default: True). Set to False to use the sequential resumable upload (slower but simpler).	True
chunk_size_mb	Optional[int]	Override chunk size in MiB (uses config/default if None).	None
max_workers	Optional[int]	Override number of upload threads (uses config/default if None).	None

Returns:

Name	Type	Description
Dict	Dict	{'associated_file': AssociatedFileRead, 'ingestion_request': IngestionRequest}

list_files(dsid)

List files attached to a dataset.

Parameters:

Name	Type	Description	Default
dsid	str	Dataset unique identifier	required

Returns:

Type	Description
List[Dict]	List[Dict]: File records (mfid, filename, storage_path, storage_backend, access_note, size, sha256_hash, dataset_mfid). For a 'gcs' file, storage_path is null until it has been ingested.

get_download_links(dsid)

Get signed download URLs for all ingested files in a dataset.

Returns:

Name	Type	Description
Dict	Dict	Mapping of file MFID → signed URL. Empty dict if no ingested files.

download(dsid, file_name=None, output_dir='crucible-downloads', no_files=False, no_record=False, overwrite_existing=True, include=None, exclude=None)

Download a dataset's files and optionally save its record as JSON.

Parameters:

Name	Type	Description	Default
dsid	str	Dataset unique identifier	required
file_name	Optional[str]	Deprecated. Use include=['pattern'] with glob syntax.	None
output_dir	str	Directory to save files (default: 'crucible-downloads/')	'crucible-downloads'
no_files	bool	Skip file download, save record.json only.	False
no_record	bool	Skip saving record.json, just download files.	False
overwrite_existing	bool	Overwrite existing files (default: True)	True
include	Optional[List[str]]	Glob patterns - only download matching files	None
exclude	Optional[List[str]]	Glob patterns - skip matching files	None

Returns:

Type	Description
List[str]	List[str]: Paths of all downloaded items (record.json + data files)

add_thumbnail(dsid, image, thumbnail_name=None)

Add a thumbnail to a dataset.

get_thumbnails(dsid, limit=DEFAULT_LIMIT)

Get thumbnails for a dataset.

delete_thumbnail(dsid, thumbnail_id)

Delete a thumbnail from a dataset.

add_keyword(dsid, keyword)

Add a keyword to a dataset.

Parameters:

Name	Type	Description	Default
dsid	str	Dataset unique identifier	required
keyword	str	Keyword/tag to associate with dataset	required

Returns:

Name	Type	Description
Dict	Dict	Keyword object with updated usage count

get_keywords(dsid=None, limit=DEFAULT_LIMIT)

List keywords, optionally filtered by dataset.

Parameters:

Name	Type	Description	Default
dsid	str	Dataset unique identifier to filter keywords	None
limit	int	Maximum number of results to return	DEFAULT_LIMIT

Returns:

Type	Description
List[Dict]	List[Dict]: Keyword objects with keyword text and num_datasets counts

add_sample(dataset_id, sample_id)

Link a sample to a dataset.

Parameters:

Name	Type	Description	Default
dataset_id	str	Dataset unique identifier	required
sample_id	str	Sample unique identifier	required

Returns:

Name	Type	Description
Dict	Dict	Information about the created link

remove_sample(dataset_id, sample_id)

Remove the link between a dataset and a sample.

Requires admin permissions.

Parameters:

Name	Type	Description	Default
dataset_id	str	Dataset unique identifier	required
sample_id	str	Sample unique identifier	required

Returns:

Name	Type	Description
Dict	Dict	Deletion confirmation

link_parent_child(parent_dataset_id, child_dataset_id)

Link a derived dataset to a parent dataset.

Parameters:

Name	Type	Description	Default
parent_dataset_id	str	The unique ID for the parent dataset	required
child_dataset_id	str	The unique ID for the derived dataset	required

Returns:

Name	Type	Description
Dict	Dict	Information about the created link

remove_child(parent_dataset_id, child_dataset_id)

Remove the parent-child link between two datasets.

Parameters:

Name	Type	Description	Default
parent_dataset_id	str	The unique ID of the parent dataset	required
child_dataset_id	str	The unique ID of the child dataset	required

Returns:

Name	Type	Description
Dict	Dict	Deletion confirmation

list_parents(child_dataset_id, limit=DEFAULT_LIMIT, offset=0, **kwargs)

List the parents of a given dataset with optional filtering.

Parameters:

Name	Type	Description	Default
child_dataset_id	str	The unique ID of the dataset for which you want to find the parents	required
limit	int	Maximum number of results to return	DEFAULT_LIMIT
offset	int	Starting position in the full result set (default: 0)	0
**kwargs	Any	Query parameters for filtering datasets	{}

Returns:

Type	Description
List[Dict]	List[Dict]: Parent datasets

list_children(parent_dataset_id, limit=DEFAULT_LIMIT, offset=0, **kwargs)

List the children of a given dataset with optional filtering.

Parameters:

Name	Type	Description	Default
parent_dataset_id	str	The unique ID of the dataset for which you want to find the children	required
limit	int	Maximum number of results to return	DEFAULT_LIMIT
offset	int	Starting position in the full result set (default: 0)	0
**kwargs	Any	Query parameters for filtering datasets	{}

Returns:

Type	Description
List[Dict]	List[Dict]: Children datasets

graph(dataset_id, recursive=False, as_networkx=False)