Core Concepts¶

Crucible organizes scientific data around four types of objects. Understanding how they relate makes it easier to structure your data.

Users¶

Users can be people or service accounts with access to Crucible. Users can be members of access groups and projects. Human users may use an ORCID as their canonical identifier or receive a generated MFID when created without one. Crucible API keys are required for working with the Crucible API locally.

If your integration requires a service account with elevated privileges, please join Discord and submit a request for a service account.

Projects¶

A project is the top-level organizational unit. Projects commonly map to research projects or user facility proposals.

Projects serve two primary purposes:

Organization — resources belong to a project, making them easier to find and filter.
Access control — all members of a project can read the datasets and samples within it. Adding a user to a project grants them access to all associated data.

Every dataset and sample must belong to a project. The project_id is a unique, human-readable identifier between 3 and 25 characters chosen at creation time, for example MFP12345.

Project management guide →

Datasets¶

A dataset is the core data object in Crucible. It can combine:

Files — the actual measurement data (optional).
Structured metadata — general attributes collected about each dataset regardless of data type, such as measurement, data_type, instrument_name, session_name, timestamp, and data_format. Other information such as creation_time, modification_time, and download path will be generated and recorded on the server side.
Scientific metadata — free-form key-value pairs for experiment-specific parameters, notes, and comments that are considered necessary for reproducibility and provenance. The structure is intentionally flexible to accommodate a variety of use cases and reduce input burden for experimentalists as well as allow adaptibility over time. However, it is recommended to standardize the structure of the scientific data for specific data types within a project or organization to promote higher quality data curation and enable downstream analytics.
Keywords — searchable tags associated with each dataset
Thumbnails - Small, low resolution images to represent the results or underlying data in the dataset.

Datasets can be linked to each other in parent-child relationships to represent processing pipelines (e.g., raw data → calibrated → analyzed) or collections of related data.

Working with datasets →

Files¶

Files are the raw data objects attached to a dataset. Zero or more files may be added; each is stored in cloud storage and triggers an ingestion process to parse metadata and generate thumbnails. Files cannot exist independently of a dataset record.

Working with files and ingestion →

Samples¶

A sample represents a physical or computational material that data was collected from. Samples are useful for tracking provenance: which datasets were measured from which material, and how that material was prepared.

Like datasets, samples support parent-child hierarchies. A bulk crystal, a cleaved wafer cut from it, and a thin film deposited on the wafer can all be modeled as a parent-child sample chain.

A dataset can be linked to one or more samples, and a sample can be linked to one or more datasets.

Working with samples →

Instruments¶

An instrument represents the physical equipment from which a dataset originated. Linking a dataset to an instrument through its instrument_id provides a consistent way to filter and search data by the equipment that produced it.

Instruments are shared across projects and exist independently of any single project. However, each distinct physical instrument should have its own entry.

If an instrument already exists in Crucible, reference it by its unique, human-readable instrument_id. If not, create it first with client.instruments.create(). The method returns the existing record if one with that instrument ID already exists.

Working with instruments →

Identifiers¶

Every resource has a system-assigned, 26-character MFID generated by the mfid package. API responses expose it in the unique_id field. This is the stable identifier to persist and exchange for projects, datasets, samples, instruments, files, and service accounts.

Projects and instruments also have unique, human-readable identifiers between 3 and 25 characters. The API calls these project_id and instrument_id. They can be used for convenient lookup, but the returned unique_id remains canonical. Dataset and sample names are display values and cannot identify a single resource.

Client parameters ending in _mfid require a canonical MFID. Parameters ending in _ref accept more than one documented identifier format and dispatch accordingly. Canonical user identifiers can be ORCIDs or MFIDs, and an MFID does not by itself identify a service account, so user APIs use user_unique_id when either shape is accepted.