Dataset Model¶
| Field | Description | Settable |
|---|---|---|
dataset_name |
Human-readable name for the dataset | create, update |
project_id |
Project this dataset belongs to | create; later changes use reassign_project() |
project_mfid |
Canonical project selector for creation | create |
measurement |
Industry-standard experiment type (e.g. "Raman Spectroscopy") |
create, update |
data_type |
Institution-specific data organization descriptor (e.g. "ScopeFoundry H5 file") |
create, update |
instrument_name |
Display name derived from the assigned instrument; may repeat, so it does not identify an instrument; deprecated in responses, read instrument |
server-assigned |
instrument_id |
Registered instrument slug | create; later changes use assign_instrument() |
instrument_mfid |
Canonical registered instrument selector | create; later changes use assign_instrument() |
instrument |
Current instrument identity and display fields, including its canonical MFID | server-assigned |
project |
Current project identity and display fields, including its canonical MFID | server-assigned |
data_format |
File type or extension (e.g. "h5", "dat") |
create, update |
session_name |
Optional tag grouping datasets collected in the same session | create, update |
timestamp |
When the data was collected (ISO 8601 format) | create, update |
public |
Whether the dataset is publicly accessible (default: False) |
create; later changes use set_public() / set_private() |
owner_orcid |
Deprecated by the API; read owner instead |
read |
owner |
Flexible owner identifier on create; public-safe user record on singleton reads | create with an ORCID, MFID, username, or email; expanded by default on get() |
unique_id |
System-assigned MFID identifier | server-assigned |
size |
Total file size in bytes | server-assigned |
creation_time |
When the record was created | server-assigned |
modification_time |
When the record was last modified | server-assigned |
capabilities |
Optional caller-specific actions calculated for an exact response | server-assigned |
Use client.datasets.assign_instrument(dataset_mfid, instrument_id=...) (or instrument_mfid=...) to assign or change a dataset's instrument; it sets instrument_id, instrument_mfid, and instrument_name together. update() raises RemovedError for instrument_id and instrument_name.
List datasets assigned to a canonical instrument across every project visible to the caller with:
The equivalent CLI command is crucible dataset list --instrument-mfid 0tkn2knjast3h0008nyq9zps2c. Supplying this explicit filter does not apply the saved current project, while combining it with --project-id intentionally narrows the results to that project. Legacy datasets without a canonical instrument MFID are not included.
Relationships¶
| Relationship | Key(s) | Description |
|---|---|---|
| Files | files in create(); add_file(dataset_mfid, file_path) to add later |
Zero or more files can be attached to a dataset. Each file is uploaded to cloud storage and triggers an ingestion process to parse metadata and generate thumbnails. |
| Scientific metadata | scientific_metadata in create(); metadata in update_scientific_metadata() / replace_scientific_metadata() |
A free-form JSON object for experiment-specific parameters. Stored separately from structured fields and searchable across datasets. |
| Thumbnails | add_thumbnail(dataset_mfid, image) |
Small preview images representing the data or results. Generated automatically by ingestors where supported, or uploaded manually. |
| Samples | sample_mfid in link_sample(dataset_mfid, sample_mfid) |
A dataset can be linked to one or more samples, and a sample to one or more datasets, capturing which material was measured. |
| Parent/child datasets | parent_mfid, child_mfid in link() |
Datasets can be linked in a directed hierarchy to represent processing pipelines, such as raw to calibrated to analyzed. |
Working with Datasets¶
Creating a dataset¶
Dataset records and associated files are separate API resources. Create the dataset record first, then add files using its returned MFID:
from crucible.models import Dataset
result = client.datasets.create(
dataset=Dataset(
dataset_name="XRD run 5",
measurement="X-ray diffraction",
instrument_id="beamline-123",
project_id="my-project",
),
scientific_metadata={"wavelength_angstrom": 0.7749, "temperature_K": 300},
keywords=["XRD", "powder"],
)
dataset_mfid = result["dataset_mfid"]
client.datasets.add_file(dataset_mfid, "xrd_run5.xy")
A dict with the same fields works in place of Dataset(...).
Applications may use canonical MFIDs instead of human-readable IDs. The API also accepts both forms when they resolve to the same resource:
dataset = Dataset(
dataset_name="XRD run 5",
project_id="my-project",
project_mfid="0tkn2knjast3h0008nyq9zps2c",
instrument_id="beamline-123",
instrument_mfid="0tk8pf1me0h3h0003fp91vr037",
)
Conflicting ID and MFID selectors produce an API validation error. Nano forwards the supplied selectors without resolving one into the other.
Files are optional. The CLI can create a record that receives its files later:
crucible dataset create --project-id my-project --name "Planned experiment"
crucible dataset add-file DATASET_MFID -i results.dat
The CLI defaults to human-readable --project-id and --instrument-id selectors. Integrations can use --project-mfid and --instrument-mfid; matching ID and MFID flags may be combined.
For convenience, create() can perform these operations in sequence when files is supplied. Files are not part of the API dataset-creation request, and a failure while adding one does not roll back the created dataset record:
result = client.datasets.create(
dataset=Dataset(dataset_name="Multi-file dataset", project_id="my-project"),
files=["file1.dat", "file2.dat", "thumbnail.png"],
)
What happens when you call create()
create() is a client-side convenience method that chains several API calls:
- POST
/datasetscreates the dataset record and returns itsunique_idMFID. - POST
/resources/{dataset_mfid}/metadataadds scientific metadata when provided. - POST
/datasets/{dataset_mfid}/keywordsadds each keyword individually when provided. - Adds each file through the associated-file and upload routes and triggers an ingestion request per file when
filesis provided
Retrieving a dataset¶
ds = client.datasets.get("0tkn2knjast3h0008nyq9zps2c")
ds_with_details = client.datasets.get(
"0tkn2knjast3h0008nyq9zps2c",
include_metadata=True,
include_links=True,
)
Datasets are retrieved only by their canonical 26-character MFID. Dataset names are display values, not identifiers.
Singleton retrieval expands owner by default as a public-safe user record containing unique_id, username, first_name, and last_name. Pass include_owner=False to suppress expansion. List operations remain opt-in with include_owner=True. The canonical owner identifier remains available as owner_orcid.
Canonical detail responses include caller-specific capabilities when the server has calculated them. Dataset capabilities always report can_change_status=False because datasets have no lifecycle-status operation. Collections and search results normally return capabilities=None, which means the guidance was not calculated rather than that every action is denied. The API remains authoritative for each mutation.
Dataset responses include lightweight instrument and project references when those canonical relationships resolve. Use dataset["instrument"]["unique_id"] and dataset["project"]["unique_id"] for stable navigation. The flat instrument_id, instrument_name, and project_id fields are deprecated by the API but remain available as display fallbacks for legacy records whose canonical relationship is unresolved. A reference exposes identity and display information only and does not imply access to the complete instrument or project.
Listing datasets¶
# All datasets in a project
datasets = client.datasets.list(project_id="my-project", limit=50)
# Datasets shared with a project but assigned elsewhere or unassigned
shared = client.datasets.list(project_id="my-project", project_scope="shared")
# Assigned and shared datasets using the project's canonical MFID
visible = client.datasets.list(
project_mfid="0tkn2knjast3h0008nyq9zps2c",
project_scope="all",
)
# Filter by measurement type
datasets = client.datasets.list(project_id="my-project", measurement="SEM imaging")
# Filter by keyword
datasets = client.datasets.list(project_id="my-project", keyword="gold")
# Datasets linked to a specific sample
datasets = client.datasets.list(sample_mfid="0td7evvtg5wb90005k1j97ak94")
# Datasets readable by both users and directly accessible to a project
datasets = client.datasets.list(
accessible_to_user=["alice", "bob"],
accessible_to_project="my-project",
)
Pass the project slug as project_id or the canonical project MFID as project_mfid. Both may be supplied when they identify the same project; conflicting identifiers produce an API validation error. project_scope accepts assigned, shared, or all and defaults to assigned. Scoped collection results expose project_relation as assigned or shared; project may be None when a shared resource has no primary project.
The sample_mfid relationship filter uses the normal paginated dataset collection and can be combined with compatible dataset filters such as project, instrument, and access selectors. Results follow cursor pagination and include only datasets the caller may read.
Owner and affiliation filters narrow any listing. owner_ref takes one owner or a list of up to 10 (unique_id, linked ORCID, username, or email) and matches records owned by any of them; owner_is_null=True selects records with no owner. affiliation relates each record to you: owner (you own it), member (you reach it through your own grants), or instrument (you read it as a member of its instrument).
mine = client.datasets.list(affiliation=["owner", "member"])
theirs = client.datasets.list(owner_ref=["alice", "bob"], project_id="my-project")
Access selectors accept user MFIDs, ORCIDs, usernames, or emails and project MFIDs or project IDs. Multiple selectors use intersection semantics and only narrow resources the authenticated caller may read. Inspecting another user requires platform-administrator access, while inspecting a project requires membership in that project or platform-administrator access.
Updating a dataset¶
client.datasets.update(
"0tkn2knjast3h0008nyq9zps2c",
dataset_name="XRD run 5 (corrected)",
measurement="Powder X-ray diffraction",
)
Project and ownership changes use preview-first workflows. Pass confirm=True only after reviewing the preview:
project_preview = client.datasets.reassign_project(dataset_mfid, "new-project")
client.datasets.reassign_project(dataset_mfid, "new-project", confirm=True)
owner_preview = client.datasets.transfer_ownership(dataset_mfid, "new-owner@example.org")
client.datasets.transfer_ownership(dataset_mfid, "new-owner@example.org", confirm=True)
Managing access¶
grants = client.datasets.list_access(dataset_mfid)
client.datasets.set_access(dataset_mfid, "users", "0000-0002-1825-0097", "editor")
client.datasets.revoke_access(dataset_mfid, "users", "0000-0002-1825-0097")
client.datasets.set_public(dataset_mfid)
client.datasets.set_private(dataset_mfid)
Normal access grants accept viewer, contributor, editor, or admin. Use transfer_ownership() for ownership.
Adding files to a dataset¶
Add files to an existing dataset:
To discover files recursively, build the list locally and add each file to the existing dataset. Crucible stores the filename, not its relative directory hierarchy, so verify that basenames are unique before uploading:
from collections import Counter
from pathlib import Path
source_root = Path("experiment-output")
source_files = sorted(path for path in source_root.rglob("*") if path.is_file())
duplicate_names = [name for name, count in Counter(path.name for path in source_files).items() if count > 1]
if duplicate_names:
raise ValueError(f"Duplicate filenames in source tree: {duplicate_names}")
for path in source_files:
client.datasets.add_file(dataset_mfid, str(path))
Replacing a file¶
client.files.replace() uploads new content into an existing file. The file keeps its MFID, filename, and metadata, stays downloadable during the upload, and is repointed only once the new content is verified. Pass the etag you last read to refuse the replace if someone changed the file in the meantime:
old = next(f for f in client.datasets.list_files(dataset_mfid) if f["filename"] == "results.dat")
client.files.replace(old["mfid"], "results.dat", etag=old["etag"])
add_file(..., replace=True) does the same when the filename already exists with different content; without it, add_file() raises FileExistsWithDifferentContent, so nothing is ever replaced silently. Re-adding identical content returns the existing record. Replacing needs signed multipart uploads, so it fails if the server has them switched off. Ingestion is requested again after a replace unless skip_ingestion=True.
In the CLI, crucible file replace FILE_MFID -i results.dat asks for confirmation (skip it with --yes), and dataset add-file asks before replacing a file that exists with different content, or replaces without asking with --replace.
Re-uploading to an existing dataset¶
Keep the dataset MFID and add the local files to that record again. Do not call datasets.create() again, because that creates another dataset record:
An unchanged file is not uploaded again. If a filename now holds different content, replace it as described above.
How the data ingestion process works¶
When a file is added to a dataset, three things happen:
- The file is uploaded to cloud storage
- A file record is created in the database linked to the dataset
- An ingestion request is sent to the backend
Data type-specific ingestion classes parse scientific metadata, structured metadata, and thumbnails from the file. If an ingestion class is specified via the ingestor parameter, that class is used. Otherwise available classes are scanned from most to least specific, if there are no ingestion classes that support the data type, then metadata and thumbnails will not be extracted from the file.
Ingestors will not overwrite the dataset attributes provided at dataset creation. Structured metadata for the primary dataset record (eg. timestamp, dataset_name, data_type) can be updated using the client.datasets.update() method.
For updates to the scientific metadata, the ingestion process uses the client.datasets.update_scientific_metadata(overwrite = False) method. As a result, new key-value pairs parsed during the ingestion process will be appended to the existing scientific_metadata and newly parsed values for existing keys will be updated. If you would like to replace the entire scientific_metadata dictionary, it can be done manally with update_scientific_metadata(overwrite=True).
Files are deduplicated by SHA-256 hash. If you add the same file twice, the client returns the existing associated-file record without uploading or requesting ingestion again. Adding a different file under an existing filename is refused unless you ask to replace it.
How uploads reach storage¶
add_file() and create() upload through the API's signed multipart flow: the client checksums the file, the API signs one URL per part (64 MiB), parts upload in parallel straight to storage, and the API verifies and assembles the object. Parts that fail are retried, expired URLs are renewed, and a failed upload is cancelled so it leaves nothing behind. Empty files skip the part step. No storage credentials or SDK are needed, so the same client works with any storage the API is configured for.
upload_max_workers (default 8) sets how many parts upload at once; add_file(max_workers=...) overrides it for one call. --no-ingest (or client.request_ingestion = False) uploads without requesting ingestion. Pass progress=callback to add_file(), or set client.upload_progress, to receive callback(filename, bytes_done, total) while parts upload; the CLI uses it to show a progress line.
Every upload uses signed multipart. The upload_protocol and upload_chunk_size_mb settings, --upload-protocol, and add_file()'s multipart, chunk_size_mb, and upload_protocol arguments were removed in 4.0.0.
If no ingestion class exists for your data type, reach out on Discord or contribute to the crucible-ingestion repository.
Remote (non-GCS) files¶
Sometimes a file isn't worth (or isn't possible to) upload to GCS - it lives on Globus, at an HPC center, or on a shared filesystem. Crucible can still catalog it: it records where the file lives, but never verifies it exists, uploads it, or fetches it on your behalf.
from crucible.models import AssociatedFile
client.datasets.add_remote_file(dataset_mfid, AssociatedFile(
filename="raw/run_01.tar",
size=52428800,
sha256_hash="9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08",
storage_backend="globus",
storage_path="https://app.globus.org/file-manager?origin_id=...&origin_path=...",
access_note="request access via NERSC allocation X",
))
size and sha256_hash are required. storage_path is optional - catalog the file now and set its location later with client.files.update(mfid, storage_path=...).
The filename is the file's identity in its dataset, and clients rebuild datasets on disk from it, so it must be a canonical relative POSIX path such as freq_001/kpoints.json: no leading, trailing, or repeated /, no . or .. components. The server rejects other names and does not normalize them. Adding the same filename again with the same hash, size, and backend returns the existing record; different content is a conflict unless you pass replace=True, which updates the existing record in place. A cataloged file's filename and its GCS or non-GCS kind cannot change later.
For files on a filesystem you can read, AssociatedFile.from_local_files() hashes them and names each by its path relative to their common folder:
files = AssociatedFile.from_local_files(
["/scratch/phonons/freq_001/kpoints.json", "/scratch/phonons/freq_002/kpoints.json"],
storage_backend="nersc-perlmutter",
)
# filenames: freq_001/kpoints.json, freq_002/kpoints.json
You can also mix remote and uploaded files in one create() call:
result = client.datasets.create(
dataset=Dataset(dataset_name="Mixed dataset", project_id="my-project"),
files=["local_results.csv", *files], # the str path is uploaded, the rest cataloged
)
Plain local paths with upload_files=False are cataloged the same way, with storage_backend="local":
result = client.datasets.create(
dataset=Dataset(dataset_name="Cataloged only", project_id="my-project"),
files=["/mnt/lustre/big_simulation_output.h5"],
upload_files=False,
)
On the command line, crucible dataset create -i FILE ... --no-upload --backend NAME does the same, and --replace updates files the dataset already has under the same name.
Downloading and ingestion are GCS-only: client.files.download()/client.datasets.download() skip non-GCS files (logging why), and the server rejects ingestion requests for them. Check storage_backend before assuming a file is fetchable - crucible file get MFID and crucible dataset list-files both show the backend name for non-GCS files instead of an "ingested" status.
Scientific metadata¶
Scientific metadata stores experiment-specific parameters as a free-form JSON object.
# Merge new keys into existing metadata (PATCH - appends/updates individual keys)
client.datasets.update_scientific_metadata(
dataset_mfid,
metadata={"temperature_K": 300, "pressure_bar": 1.0, "scan_rate_mV_s": 50},
)
# Retrieve it
meta = client.datasets.get_scientific_metadata(dataset_mfid)
# Search across all datasets
results = client.datasets.search_metadata("temperature", limit=20)
update_scientific_metadata() merges new keys into the existing metadata (PATCH). To replace all existing metadata entirely, use replace_scientific_metadata():
# Replace all metadata entirely (POST)
client.datasets.replace_scientific_metadata(dataset_mfid, {"new_key": "value"})
Keywords¶
client.datasets.add_keyword(dataset_mfid, "annealed")
keywords = client.datasets.list_keywords(dataset_mfid=dataset_mfid)
Thumbnails¶
thumbnail = client.datasets.add_thumbnail(dataset_mfid, "preview.png")
thumbnails = client.datasets.get_thumbnails(dataset_mfid)
print(thumbnail["id"])
print(thumbnail["mime_type"])
client.datasets.update_thumbnail(
dataset_mfid,
thumbnail["id"],
thumbnail_name="overview.png",
)
client.datasets.update_thumbnail(
dataset_mfid,
thumbnail["id"],
image="replacement.png",
)
Thumbnail create, list, and update responses contain id, dataset_id, thumbnail_name, thumbnail_b64str, and the authoritative mime_type. The API currently normalizes stored thumbnails to JPEG, regardless of the extension in the user-facing thumbnail name.
From the CLI, add a local image with:
Use --name NAME to override the thumbnail name stored by the API.
Downloading¶
# Download all files for a dataset
client.datasets.download(dataset_mfid, output_dir="./downloads")
# Download only matching files
client.datasets.download(dataset_mfid, output_dir="./downloads", include=["*.dat"])
# Get temporary signed download URLs, keyed by file MFID
links = client.datasets.get_download_links(dataset_mfid)
Parent-child relationships between datasets¶
Link datasets to represent a processing pipeline:
# raw to processed
client.datasets.link(
parent_mfid=raw_dataset_mfid,
child_mfid=processed_dataset_mfid,
)
# List relationships
parents = client.datasets.list_parents(processed_dataset_mfid)
children = client.datasets.list_children(raw_dataset_mfid)
Requesting dataset deletion¶
Note
A deletion request does not immediately remove the resource. An admin must approve it before permanent deletion.