Changelog¶
Unreleased¶
4.0.1¶
Sign-in by pasted code needs Crucible API 4.0.1.
Added¶
crucible login --pasteandclient.login(method="paste"): sign in in any browser and paste the code its page shows.- Press Enter while a browser sign-in waits to paste a code instead, when the browser cannot reach the terminal.
- When the API answers from a new address for good (301 or 308), the client warns once and names the
crucible config set api_urlcommand, which also moves a stored sign-in to the new address.
Changed¶
- Signing in and renewing a session follow an API that moved to a new path on the same host; a move to another host or from https to http is still refused.
samples.create()anddatasets.create()accept the record's fields as a dict as well as a model, like the othercreate()methods.
Fixed¶
datasets.create()given something other than aDatasetor a dict raises aTypeErrorinstead of anAttributeError.
4.0.0¶
Requires Crucible API 4.0 and Python 3.11 or later. Upgrading to 4.0 lists what to change in existing scripts.
Added¶
- Sign in with a browser:
crucible login(--devicefor SSH and headless machines),crucible logout, andclient.login()/logout(); tokens go in the OS keyring, or a private file where none is available (auth_store). - Actions that need a recent sign-in, such as creating an API key, offer a browser sign-in in the CLI and retry.
whoamiandstatusshow whether the client uses an API key or a sign-in, and from where.- Named API keys:
account.create_api_key(),list_api_keys(),rename_api_key(), andrevoke_api_key(), withaccount keys list|create|rename|revoke; administrators list and revoke single keys withusers.list_api_keys(),users.revoke_api_key(),user list-keys, anduser revoke-keys --key. account verifyshows when the key was last used.- Administrators review accounts waiting for approval with
users.list(pending=True),users.approve(), andusers.reject(), anduser list --pending,user approve, anduser reject;user getshows a waiting account's status and sign-in methods. hub_urlsetting (CRUCIBLE_HUB_URL, defaulthttps://crucible.lbl.gov/hub);config initcreates an API key after a browser sign-in, or points to the Hub.- Join requests:
access_groups.withdraw_join_request(),join_request_facets(),roleonapprove_join_request(),ag withdraw, andag approve --role; lists filter by scope, several statuses, project, reviewer, time, and requester text, and sort (ag list --scope/--project/--search/--sort); requests carry the project, requester, reviewer, granted role, and capabilities. - Postprocessing:
datasets.postprocess()with an optional idempotency key,client.postprocessingfor pipelines and runs, anddataset postprocess,pipelines,list-runs, andget-run. datasets.add_remote_file(replace=True)(andclient.replace_existing,dataset create --no-upload --replace) updates a cataloged file whose filename already exists with different content.AssociatedFile.from_local_files()builds hashed catalog records named by their path relative to the files' common folder.- Dataset, sample, project, and instrument lists filter by up to 100 MFIDs with
mfid; unreadable or missing ones are left out. users.list()takes severalunique_idorusernamevalues at once.- Capabilities: access grants from
list_access(), instrument service accounts, and project members carryEntryCapabilities(can_remove,can_change_role,assignable_roles), and resource capabilities declarecan_view_access,can_delete, andcan_request_deletion. Userand project members carryorcidandorcid_verified, andwhoami,account showanduser getshow the linked ORCID separately from the user ID.- Shell completion reaches nested commands such as
account keys listand fixed-choice arguments such asconfig get KEY, and leaves out removed and renamed commands, administrator-onlyusercommands you cannot use, anduser createanddeletion deletefor callers who cannot elevate. --jsonon thelist-parents,list-children,list-samples,list-datasets,list-files,list-keywords,list-users,list-projectsandlist-access-groupscommands,file,ingestion,aganddeletionlistandget,ag mine,deletion list-deletedandget-deleted,file ingestion,ingestion list-ingestors,whoami, andstatus.- The package ships type information (
py.typed) and passes mypy;get()methods are typed as returningNonewhen the API sends no record. - The shell says at startup when PyPI has a newer release, with the command to upgrade;
update_check(CRUCIBLE_UPDATE_CHECK) turns it off. - The shell switches elevation off after 15 idle minutes at the prompt;
elevation_timeout(CRUCIBLE_ELEVATION_TIMEOUT) changes it,0never. owner_reffilter for up to 10 owners on lists, facets, and search (project and metadata search included), andowner_is_null;--owneris repeatable.affiliationvaluesmember(datasets, samples, instruments) andinstrument(datasets).- The shell opens signed out when no credential works, offering
loginandconfig init; a client without a credential raisescrucible.NotSignedInError(aValueError).
Removed¶
- Everything deprecated before 4.0.0. Calling a removed method, keyword, or setting raises
crucible.RemovedError(aTypeError) naming the replacement, and removed CLI commands and options stop with the one to use; the CLI reference lists them under "Removed in 4.0.0". - Removed settings (
graph_explorer_url,current_session,upload_protocol,upload_chunk_size_mb) are ignored in old config files;config showlists them andconfig unsetremoves them.get_graph_explorer_url()is replaced byget_hub_url(). CRUCIBLE_CURRENT_PROJECTstops the CLI; use--project-id.owner_orcidondatasets.list()andsamples.list()(useowner_ref),include_datasetsonsamples.list()(useinclude_links), andinstrument_id/instrument_nameindatasets.update()(useassign_instrument()).- Parser
owner_orcidandinstrument_namearguments and the_instrument_nameclass default; useowner,instrument_id/instrument_mfid, and_instrument_id. dataset update --set public=...; usedataset set-publicorset-private.users.get(orcid=)(pass the user reference),projects.list(orcid=)(usemember_reforusers.list_projects()), theAccessGrantpropertiesprincipal,kindandeffective_permission(useprincipal_id,principal_type,permission), therelationshipkey inget_links()results (usedirection), and thedsidkey indatasets.create()results (usedataset_mfid).- The legacy GCS upload routes, the fallback to them, and the
google-cloud-storagedependency. - The basic shell used without prompt_toolkit; a broken install now says to reinstall.
- Instrument
set_access(),revoke_access(), andaccess grant|revoke; use the member methods and commands. datasets.request_carrier_segmentation(),request_insitu_aggregation(),request_rga_analysis(), andrequest_mosaic_stitch(); usedatasets.postprocess().users.verify_api_key(); readapi_key_statusfromusers.get().
Changed¶
- Instrument members work like project members: every member reads the instrument's datasets, member methods return
ProjectMemberlists, andInstrument.membersholdsProjectMemberrecords. - Instrument
add_user()andadd-useracceptviewerand default to it instead ofeditor. instrument list --mineincludes every instrument you are a member of.- Creates send
owner_refand transfers sendnew_owner_ref; theowner_idandowner_id_is_nullfilters are deprecated. - Projects take
owneron create (project create --owner);project_lead*and--leadare deprecated. - Project lists and
project getshow the owner instead of the lead. --group-by instrument,--group-by project, and the download cache read the nested references.- Signed-in administrators run normal by default, while API keys still run elevated;
is_elevated,whoami, and the shell badge follow the mode the API reports. - Python 3.11 or later is required;
keyringis a new dependency. - API errors raise
crucible.CrucibleAPIError, arequests.HTTPErrorwithstatus,code,detail, andrequest_id; CLI errors show the request ID. - POST and PATCH requests without an
Idempotency-Keyare no longer retried after a 502, 503, 504, or lost response, which could duplicate records; 429 and failed connections are still retried. - Requests send a
nano-crucible/<version>User-Agent. - Calls to routes the API marks deprecated raise a
DeprecationWarningonce per route, pointing at the calling code, with the date, sunset, and link the API gives. - Uploads always use signed multipart, including empty files;
upload_max_workerssets how many parts upload at once, and a server without uploads enabled says so. - Resource, project, instrument, and user links,
crucible open, and Alt+O go to the Crucible Hub (hub_url) instead of the Graph Explorer. - The shell starts faster, says why when it cannot reach the API, and its summary shows the API's name, version, and how you are signed in. The status bar names the API (production, staging, or its host), shows the API version only for pre-release or older servers, shows whether you use a key or a sign-in, adapts to narrow terminals, and no longer shows a clock.
- In the shell,
helplists commands compactly andhelp COMMAND [SUBCOMMAND]shows a command's options, slow commands print how long they took, completion searches wait for a pause in typing, and creating or updating a project refreshes project completion. - Renamed, with the old names still working and warning:
datasets.list_keywords(),projects.list_users(),instruments.list_users(),users.list_projects(), and the parametersfile_mfid(files),resource_mfid(deletions),service_account_ref,instrument_ref, anduser_ref(users and project members).ingestions.list()takesdataset_mfidandfile_mfid;dsidandfile_idstill work. - Renamed CLI commands, with the old names still working and warning:
dataset link-sample,unlink-sample,unlink -p -c;sample link-dataset,unlink-dataset,unlink -p -c;dataset ingestorsis nowingestion list-ingestors, which takes--filter. datasets.graph()andsamples.graph()still work but are deprecated in favour ofgraphs.get().samples.get()andclient.get()no longer embed a sample's datasets;include_datasets=Truestill works with a warning (useinclude_linksordatasets.list(sample_mfid=...)), andsample get --jsonincludes links instead.deletion deleteandsa rotate-keyask for confirmation (-y/--yesskips it), and-yworks wherever--yesdoes.- Join requests use the project routes:
request_join()posts to/projects/{ref}/join_requests,access_groups.get()reads one request directly and works for requesters and reviewers, andaccount.join_requests()lists thesubmittedscope;ag listshows the requests you may review, including project editors' queues, and the shell's pending count follows it. - User references are no longer shape-checked:
users.get()and the CLI pass any value they cannot classify to the API. users.resolve()uses the exact filters ofGET /usersinstead of the deprecatedPOST /users/resolve; its result is unchanged.users.search()reads the paginateditemsresponse.datasets.create()leaves outinstrument_namewheninstrument_idorinstrument_mfidis set.- A cast recipe's
instrumentis an instrument ID or MFID, not a name. - CI tests Python 3.13 and macOS, runs mypy, checks the client's routes against staging, and gates PyPI releases on the tests; the live-API integration tests moved to the
crucible-testsrepository.
Fixed¶
skip_ingestion=Trueuploads recordingestion_classinstead of dropping it.- Datasets whose files have an absolute legacy filename, such as
/mnt/gcs/api-uploads/scan.h5, download instead of being skipped. datasets.create(upload_files=False)anddataset create --no-uploadsend each file's size and SHA-256, which the server requires, and name files by relative path so files sharing a basename in different folders stay separate.add_remote_file()setsstorage_pathin the create request, and updates it when re-adding an existing file at a new location.- Creating a dataset, sample, project, or instrument with scientific metadata on someone else's behalf no longer fails with 403 for platform contributors.
- The CLI shows UTC timestamps ending in
Zin local time, and table dates use the local day. project add-useraccepts user and service-account MFIDs and any other user ID instead of failing with an invalid ORCID error.- Downloads no longer send the API key to the storage host serving a signed URL.
- A refusal that needs a recent Hub sign-in (
hub_session_required) points to the Hub instead of suggesting--elevate. dataset ingestionandfile ingestionshow the file MFID and creation time.deletionandagviews show usernames again.sa create --unique-idkeeps the given MFID when it prompts for the username.- Upload progress no longer crashes when a part of a finished file is retried.
crucible.castimports on Windows.
3.3.0¶
Added¶
- User administration:
users.set_platform_role(),users.revoke_api_keys(),user set-role, anduser revoke-keys. users.list()anduser listfilter by text, name, role, and account type, and sort.- Project and instrument lists sort with
sort/direction(--sort/--direction). - Project lists filter by member with
member_ref/member_role(--member/--member-role). - Project and instrument lists and details show the caller's role.
user list-projectsshows the listed user's role and filters by--role.- Dataset and sample lists filter by visibility, owner, creation or modification time, missing fields, and linked records, and sort.
datasets.facets(),samples.facets(),dataset facets, andsample facetscount records per field value.- Dataset lists, counts, and facets filter by instrument slug (
instrument_id,--instrument-id). datasets.assign_instrument()sets a dataset's instrument.- Instrument maintainers:
instruments.get_users(),add_user(),remove_user(),update_user_role(), and the matchinginstrumentcommands. - Instrument lists filter by your role (
affiliation,--mine) and by instrument fields. instrument update --instrument-idrenames an instrument.- Service-account administration:
service_accounts.get_admin(),list_admin(),set_platform_role(), andservice-account set-role. - Deletion request and audit lists filter by scope and project, and sort.
deletion listshows reviewers their review queue by default.- Privilege mode: the
privilege_modekeyword and config key, and the--elevate/-Eflag. Administrators still run elevated by default; this becomesnormalin a future release. - Shell:
elevate on|off,elevate COMMAND, and anELEVATEDtoolbar badge. client.authorization,client.capabilities,client.can_elevate, andclient.refresh_profile().crucible whoamishows your platform role, privilege mode, and capabilities.print.barcode()andcrucible print barcodeprint sample labels.- Uploads use the API's signed multipart flow, chosen with
upload_protocol(auto,signed,legacy) or--upload-protocol. - Upload progress: a
progresscallback,client.upload_progress, and a progress line in the CLI. --no-ingestondataset createanddataset add-fileuploads without requesting ingestion.files.replace()andcrucible file replacereplace a file's content, keeping its MFID.add_file(replace=True)and--replacereplace a same-named file with different content.sample createanddataset createaccept--owner USER.- Download links accept
disposition="inline". - List methods return a
ListResultwithhas_more,total, and a continuation, and acceptcursor=. - Shell:
moreshows the next page of the last listing. - An optional pager for long listings (
config set pager auto,--no-pager). - "Did you mean ...?" suggestions for mistyped commands and flags.
- Next-step hints on common API errors.
- Shell completion of facet values, facet fields,
--instrument-id, and MFID flags. - Fuzzy shell completion of project IDs and facet values.
--ingestorcompletes from the API's ingestion classes.- The shell lists pending deletion reviews and join requests at startup.
- Creating projects, instruments, and service accounts fails early without the capability.
Changed¶
- List commands show 50 records by default (was 100), accept
--all, and report how many were left out. dataset createprompts for missing fields on a terminal instead of creating an unnamed dataset.dataset listandsample listprint oldest to newest and follow--sort.dataset listandsample listwithout a project list every accessible project.- The shell opens with a session summary beside the Crucible mark.
- Shell colors keep the brand blue on 256- and 16-color terminals.
- The
supportplatform role is removed. service-account listandgetshow platform roles and key status to administrators;showis an alias forget.- Identifying instruments by display name is deprecated; use
--instrument-idor--instrument-mfid. - Instrument operations accept a slug as well as an MFID.
reassign_project()acceptsproject_mfidand reports MFIDs.- The shell hides service-account commands from callers who cannot use them.
- Unknown dataset and sample list filters raise a local
ValueError. - Python 3.10 or newer is required.
- The configuration file is readable only by its owner.
ipywidgetsandpython-dotenvare no longer dependencies.
Fixed¶
- Shell
reloadworks when started withpython -m crucible.cliand on Windows. editcommands work on Windows without configuration.- Project ownership, access, and metadata commands accept a project ID.
- Renaming a project updates the saved current project.
- Waiting for ingestion returns the request's final state.
- Offset-paged listings no longer stop at 1000 records.
- Files from multipart uploads show and download under their filename.
- Downloads refuse paths outside the target folder.
dataset downloadwarns about files it cannot fetch.- Storage errors explain what happened.
- The API reference documents shared ownership, access, and project methods.
- The shell history no longer stores API keys.
3.2.1¶
Added¶
- Dataset and sample parent-child links support optional
is_derived_fromandis_part_ofrelationship types in Python and the CLI, including relationship-type filtering. - The interactive shell displays a centered 16x16 Crucible mark with square-proportioned half-block pixels on supported terminals.
- Dataset and sample creation accept canonical project MFIDs alongside matching project IDs, and dataset creation accepts canonical instrument MFIDs alongside matching instrument IDs.
- Dataset and sample lists accept a project ID or canonical project MFID with assigned, shared, or combined project scope in the Python client and CLI.
- Dataset thumbnails can be added from the CLI and renamed or replaced with
client.datasets.update_thumbnail(). - Dataset and sample responses expose typed project references with canonical MFIDs, and dataset responses also expose typed instrument references, while retaining legacy flat fields.
- The CLI supports
--no-colorand the standardNO_COLORenvironment variable. - Instrument workflows support lifecycle status changes, bound service-account listing, and dataset filtering by canonical instrument MFID.
- Exact dataset, sample, project, and instrument responses expose typed caller-specific resource capabilities when supplied by the API.
Changed¶
- Entries returned by
client.get_links()expose the API'sdirectionfield withsource,target, orundirectedvalues. The legacyrelationshipfield remains as a compatibility alias with its previousparent,child, orassociatedvalue. - The minimum supported Python version is now 3.9.
- Dataset thumbnail create, list, and update responses expose the API-provided authoritative MIME type.
- Instrument search results now show the user-facing instrument ID instead of manufacturer metadata.
crucible dataset createcan create a dataset record without--input; file-dependent options still require at least one input file.- Dataset and sample relationship methods now consistently use
link,unlink,link_sample,unlink_sample,link_dataset, andunlink_dataset; previous method names remain as deprecated wrappers. - Public visibility now uses
set_public()andset_private()in Python andset-publicandset-privatein the CLI; previous method names, command names, and update visibility options remain deprecated compatibility aliases that delegate to the dedicated access routes. - The interactive shell now remembers the project selected by
use PROJECT_ID,unuseclears it, and project context sources are visible;CRUCIBLE_CURRENT_PROJECTis deprecated because it can silently redirect operations. - The interactive shell status bar now renders the Crucible dark blue, light blue, and orange brand palette in true color on capable terminals while keeping autocomplete menus visually plain.
- Sample detail methods can suppress the deprecated embedded dataset expansion; existing Python and JSON behavior remains compatible, while human CLI workflows avoid loading it.
- CLI detail views group related fields, use consistent semantic colors, link resources and users to Crucible Explorer, distinguish slugs from MFIDs, and abbreviate given names.
- Generic
crucible get MFIDnow renders projects in addition to datasets, samples, and instruments. - CLI project and sample-type options use consistent canonical flags, and interactive completion now discovers flags and searches resource endpoints for identifier values.
- CLI failures now preserve HTTP status codes while formatting API validation details and warnings for readable terminal and JSON output.
- Human and service-account creation normalize usernames and validate the API's complete username rules, with immediate retry prompts during interactive creation.
- Project membership methods and commands validate named roles before requests, document strict lower-role management and owner-or-self removal, and
project add-userreports API errors through the shared status-preserving formatter. - Interactive creation and configuration prompts now share required, optional, default, secret, and choice handling; invalid values prompt again, API keys remain hidden, and missing required input fails clearly outside a terminal.
- Destructive CLI confirmations now use consistent yes-or-no handling, default to cancellation, and require
--yeswhen no interactive terminal is available. - CLI tables now adapt to terminal width while preserving full usernames, resource slugs, and MFIDs whenever space permits.
- Project listing, resource searches, user discovery, and service-account retrieval now support raw JSON output with structured JSON validation errors.
- Empty CLI listings identify their resource type, and mutations use consistent success messages with text fallbacks for redirected output.
crucible statusnow shows the configured endpoint, readiness state, client and API versions, deployed branch and commit, database latency and schema revisions, and authenticated identity.- Instrument search accepts an optional lifecycle-status filter in the Python client and CLI.
- Project member tables use role-priority ordering and semantic role colors, display the owner as the project lead, and the interactive shell status bar again identifies project, user, and API context with symbols.
- Dataset and sample relationship listing now uses canonical collection filters with cursor pagination instead of deprecated nested read routes.
Fixed¶
crucible file deletenow requires interactive confirmation or an explicit--yesbefore permanently deleting a file.- Dataset file guidance now documents the API v3 record-first workflow, recursive uploads, additions, deletion-based replacement, and re-uploading.
- Documentation navigation tabs now remain visible while scrolling using MkDocs Material's native sticky-tabs behavior.
- Instrument creation no longer sends inherited response-only fields rejected by API v3.
3.2.0¶
Added¶
add_file(skip_ingestion=True)uploads a file without requesting ingestion. Updates ingestion request table with provenance details for local parsing.- Dataset, sample, and project lists accept repeated user and project access selectors.
- Development skills for API, CLI, parser, and cast changes.
- Agent-agnostic contributor guidance and a skill for safe client workflows.
- New
file request-ingestionCLI command to (re)request ingestion for a cataloged file. - New
client.datasets.add_remote_file()andclient.files.update()to catalog files that live outside GCS (Globus, NERSC, a shared filesystem) without uploading them. dataset create --no-upload/--backend/--access-noteto catalog files by path instead of uploading them.- New
ag getcommand andclient.access_groups.get(). - Shell autocomplete for
agrequest IDs. - Shell autocomplete extended to more
user/project/sa/agcommands. - Shell autocomplete for
instrument getand--project/-pidflags. - Shell autocomplete for dataset/sample IDs across most subcommands.
- Shell autocomplete for
user searchshows live matches while typing. - New
dataset/sample/project reassign-projectandtransfer-ownershipcommands. - New
dataset/sample/project access list/grant/revokeandpublish/unpublishcommands. - New
project update-user-rolecommand andadd-user --roleflag. project list-usersnow shows each member's role.instrument createrequires a new--instrument-idflag (a unique slug, separate from its ID and name).project get --include-membersshows the project's member list and roles;--membersremains as a deprecated alias.project create/client.projects.create()accept a flexibleproject_leadfield as an alternative to the explicit ORCID/email/username fields.
Changed¶
- The built-in production endpoint now targets API v3; explicit v1/v2 overrides display a migration warning, and
config unset api_urlrestores the package default. - Instrument retrieval expands typed public owner records by default; instrument lists support owner expansion and lifecycle-status filtering.
- Instrument creation defaults ownership to the authenticated identity, while ownership changes use
transfer_ownership()andinstrument transfer-ownershipinstead of update fields. - Human users may be created without an ORCID, receiving a server-assigned MFID that user and owner displays treat as a canonical user ID rather than an ORCID.
- User operations that require a canonical identity use
user_unique_id; the previousorcidkeyword remains temporarily supported with a deprecation warning. - Health checks and
crucible statusaccept deployment provenance from the nested API readiness response while remaining compatible with the legacy flat response during rollout. - Singleton dataset and sample retrieval expands typed public owner records by default;
owner_orcidcreation inputs are deprecated in favor of flexibleowneridentifiers. - Project membership mutations resolve usernames and emails before using canonical user identifiers; the old
orcidkeyword remains temporarily supported. - Generic access-group mutation helpers and CLI commands are deprecated in favor of typed resource, project, and instrument operations.
- Exact user lookups show email to self and platform administrators and omit the email row when it is not disclosed; other user, project-lead, member, and operator views remain public-safe.
user list-datasetsnow uses the canonical paginated dataset collection and supports--limit.client.users.check_dataset_access()anduser check-accessnow report the canonical effective access role.- MFID-only parameters now use role-specific
_mfidnames, includingparent_mfidandchild_mfid; project and instrument slugs retain_id, and previous keywords remain temporarily supported. - Resource lookups now dispatch canonical MFIDs to single-resource routes and resolve project slugs, instrument slugs, usernames, and emails through exact collection filters without a second request. Returned records retain
unique_idas their canonical identifier. - README now focuses on installation, navigation, and project essentials.
- CLI reference now documents all command families from one canonical location.
dataset get/dataset list-filesnow show each file's MFID alongside its name.dataset getshows a dataset's files by default; no longer requires--verbose.datasets.create()'sfiles_to_uploadrenamed tofiles(accepts a mix of local paths andAssociatedFileobjects); old name still works with a deprecation warning.ag list/ag minenow default to pending requests, matchingdeletion list.deletion listandag list/ag mineshare one--statusflag.ag approve/ag rejectaccept multiple request IDs.- Shell username autocomplete now works for non-admins and completes to username.
- Broken third-party parsers now log a warning instead of silently disappearing.
project updateno longer accepts--lead-email/--lead-orcid; useproject transfer-ownershipinstead.project update --project-idnow renames the project.sample updateno longer accepts--project/--owner; usesample reassign-project/transfer-ownershipinstead.dataset update/sample update --setno longer acceptproject_id/owner_orcid; use the new reassign/transfer commands instead.- Generic access grants accept roles up to
admin; ownership changes usetransfer-ownership. - Dataset update fields now match the supported mutation contract:
data_formatis editable, while unsupporteddescriptionand frozen instrument-assignment fields are excluded. - Dataset, sample, and project operations expose only the access-control, ownership, and project-assignment capabilities supported by their API contracts.
- Project member add, role-update, and removal methods consistently return
list[ProjectMember].
Fixed¶
- CLI tables display usernames up to their full 24-character limit and project or instrument slugs up to their full 25-character limit.
- Instrument CLI get/list output formats expanded owners correctly and
instrument list --include-metadata --jsonexposes requested metadata. - Project and instrument lookup remains compatible with legacy slugs outside the current creation limits.
- Access-control operations now use the API's canonical principal and permission fields.
- Paginated list operations now request only the number of records needed to satisfy
limitinstead of over-fetching full server pages. - Quick-start examples now use current dataset creation return values.
- Download guides now use current namespaced methods and overwrite options.
files.download()crashed with a raw exception on a non-GCS file instead of a clear error.AssociatedFilesilently droppedstorage_backend/access_notefields returned by the API.cast: uploading files in a recipe always crashed.cast: resuming a recipe after a partial failure always crashed.dataset create/parser-based uploads forced generic ingestion instead of letting the server auto-detect.- Parser-based uploads sent a stale, unused field to the server.
- Users without a username disappeared from shell autocomplete; they now show by ORCID.
- Some
editcommands silently ignored--debug. - Docs and the tutorial notebook referenced removed/renamed methods and wrong parameter names in several places.
3.1.0¶
projects.get()/search()no longer require membership;lead/scientific_metadataare membership-gated.- New
client.access_groupsresource:request_join(),list_join_requests(),approve_join_request(),reject_join_request()(requires the pendingfeat/access-group-join-requestsAPI branch). client.projects.request_join()/list_join_requests()delegate toaccess_groups.client.account.join_requests()lists the caller's own join-request history.- New
crucible access-group/agCLI:request,list,mine,approve,reject. - New
crucible project request-joinandcrucible project list-join-requests. - Fix
access_groups.request_join(): always send a JSON body (API requires it even with no reason). - Join request and deletion request lists now show usernames instead of raw ORCIDs, drop the reason column, and show dates as
YYYY-MM-DD. - New shared CLI helpers:
helpers.resolve_usernames(),term.fmt_date(),term.status_label(),term.fmt_name()(de-duplication pass, no behavior change). - CLI: unified user identification. Most
usersubcommands andproject add-user/remove-usernow accept ORCID, username, or email directly. Old identifier flags are deprecated but still work. - CLI: unified service account identification.
sasubcommands accept MFID or username directly. Old flags deprecated but still work. - New
client.service_accounts.list_access_groups()/add_to_access_group()/remove_from_access_group()and matchingsaCLI commands. - Fix
deletion approve/deletion reject: batch commands now exit 1 if any request fails. - Fix
dataset add-access-group: removed the decorative--readflag (read access is always granted;--writeadds write). search_metadata(): response unwrapped from the paginated envelope; default limit now 20; results includeresource_type,name,owner_orcid, timestamps,rank.include_owner=Trueondatasets/samplesget()/list()andclient.get()resolves the owner into a public-safe user object.client.files.delete()andcrucible file deleteto delete a file by MFID.samples.create()now takes aSamplemodel as its first argument, consistent withdatasets.create().crucible project list-usersshows usernames and correctly readsunique_idas the ORCID.- Default graph explorer URL updated to
https://crucible.lbl.gov/explore.
3.0.1¶
- Fix
datasets.download(): data files now saved insideoutput_dir/{dsid}/alongsiderecord.json, not directly inoutput_dir/.
3.0.0¶
This is a major release. The flat client.* API that was deprecated in 2.x has been
removed. All operations now live under typed resource namespaces.
Breaking changes
Before upgrading, update any code that uses the old flat API:
| Old (removed) | New |
|---|---|
client.get_dataset(dsid) |
client.datasets.get(dsid) |
client.create_new_dataset(...) |
client.datasets.create(...) |
client.list_datasets(...) |
client.datasets.list(...) |
client.get_sample(sid) |
client.samples.get(sid) |
client.add_sample(...) |
client.samples.create(...) |
client.list_samples(...) |
client.samples.list(...) |
client.get_project(pid) |
client.projects.get(pid) |
client.get_user(orcid=...) |
client.users.get(orcid=...) |
client.download(resource_id) |
client.datasets.download(dsid) or client.samples.download(sid) |
pip install nano-crucible[shell] |
pip install nano-crucible (now core) |
pip install nano-crucible[gcs] |
pip install nano-crucible (now core) |
pip install nano-crucible[all] |
pip install nano-crucible |
New resource namespaces¶
client.account— self-service profile management (profile(),update_profile(),api_key(),verify(),whoami()). No admin required.client.ingestions— ingestion request management (list(),get(),wait(),update()).client.filesis now a proper peer resource (previously a base class forDatasetOperations). Scoped to file MFIDs:get(),list(),download(),get_download_link(),request_ingestion().
Architecture changes¶
DatasetOperationsno longer inherits fromFileOperations. All dataset-scoped file operations (add_file,list_files,get_download_links,download, thumbnails, ingestion) now live directly onDatasetOperations.- Upload logic extracted to
crucible/resources/gcs/upload.py— standalone functions testable in isolation. - Download logic extracted to
crucible/resources/gcs/download.py— parallel file downloads viaThreadPoolExecutor. - Scientific metadata methods (
search_metadata,get_scientific_metadata,update_scientific_metadata,get_access_groups,add_access_group) unified onBaseResource— available on all resource types. search_scientific_metadata()renamed tosearch_metadata()(deprecated alias kept).
New features¶
- Parallel GCS multipart upload —
add_file()usestransfer_manager.upload_chunks_concurrently()by default (8 workers, 64 MiB chunks, benchmarked). Falls back to sequential resumable upload withmultipart=False. Upload settings configurable viacrucible config set upload_chunk_size_mbandupload_max_workers. - Sequential per-dataset downloads — files within a dataset download sequentially; to
parallelise across datasets, wrap
datasets.download()in aThreadPoolExecutorat the caller level. - Fuzzy name search —
datasets.search(q),samples.search(q),projects.search(q),instruments.search(q)via new API endpoints. Typo-tolerant, returns top-N by relevance. - Scientific metadata search renamed —
search_metadata(q)on all resources. - Username support —
users.get(username=...),users.search(q),user set-username,project add-user -u USERNAME. samples.download(sid)— saves sample record asrecord.json.files.download(file_id)— single-file download by MFID.ingestions.wait(request_id)— public polling method.
Packaging¶
google-cloud-storage>=2.7.0andprompt_toolkit>=3.0moved to core dependencies.[shell],[gcs], and[all]extras removed —pip install nano-crucibleinstalls everything.[parsers]remains optional (ASE/LAMMPS support).
CLI additions¶
crucible account—show,edit,update,api-key,verifycrucible ingestion—list,get,waitcrucible user search TERM— non-admin user lookupcrucible {dataset,sample,project,instrument} search TERM— fuzzy name searchcrucible {dataset,sample,project,instrument} search-metadata TERM(alsosearch-md)crucible dataset list-access-groups,add-access-groupcrucible user add-access-group,remove-access-groupcrucible deletion list-deleted,get-deleted,delete --force--jsonflag on allgetandlistcommands (replaces--output json)sample update— named flags (-n,-t,--description,--project,--public)
Deprecations (still work, emit warnings)¶
client.download()→client.datasets.download()orclient.samples.download()datasets.get_associated_files()→datasets.list_files()datasets.search_scientific_metadata()→datasets.search_metadata()datasets.add_file_to_dataset()→datasets.add_file()datasets.graph()→client.graphs.get()datasets.get_ingestion_requests()→client.ingestions.list()datasets.get_request_status()→client.ingestions.get()datasets.update_ingestion_status()→client.ingestions.update()users.me()→client.account.profile()users.get_api_key()→client.account.api_key()
2.1.0 → 2.1.2¶
Although versioned as a patch series, this span (late March to mid-May) is effectively a feature release. The notes below focus on what affects code you write against the client.
Likely breaking changes
Most users only need to address three things:
- User responses now key identity as
unique_idinstead oforcid(e.g.user["orcid"]→user["unique_id"]). - File methods now live under
client.files(backward-compat shims onclient.*still work). add_metadatawas renamed toreplace_scientific_metadata.
New resource namespaces¶
client.files— file operations moved here fromdatasets: chunked GCS upload, plus listing, inspecting, and downloading files bydataset_mfidor SHA-256 hash.client.deletions— deletion-request workflow (request / approve / reject).client.graphs— graph queries.
New top-level client methods¶
client.live()/client.health()— unauthenticated health checks.client.search_scientific_metadata(q)— full-text search across scientific metadata.client.get_links(resource_id)— immediate links for any resource.
Moved or renamed methods¶
- File methods moved
datasets→files:add_associated_file,get_associated_files,upload_file, and download helpers. Backward-compat shims onclient.*still work. - Scientific-metadata methods moved to the shared base resource — now available on
datasets,samples,projects, andinstruments. add_metadata→replace_scientific_metadata(rename). Useupdate_scientific_metadata(..., overwrite=False)to merge (PATCH) oroverwrite=Trueto replace (POST).users: removedget_or_createandget_user_from_api; addedupdate(),get_api_key(), andremove_from_access_group().projects: addedupdate(),remove_user(),add_scientific_metadata(), andupdate_scientific_metadata().instruments: addedupdate(),add_scientific_metadata(), andupdate_scientific_metadata().samples/datasets: addedcount()andgraph().
Model field changes¶
User:orcidfield renamed tounique_id(with a.orcidproperty alias for backward compatibility); removedlbl_emailandemployee_number; addedis_service_account.Sample: addedpublic,scientific_metadata,datasets,links,deletion_request,resource_type; removedowner_user_id.Dataset: addeddata_type,scientific_metadata,links,deletion_request,resource_type; removedowner_user_idandinstrument_id; droppedfile_to_uploadandsha256_hash_file_to_upload.Project:project_lead_emailis now optional (previously required); addedproject_lead_orcid,lead,scientific_metadata, and timestamp fields.- New
DeletionRequestmodel.
Behavior changes¶
projects.add_user()/remove_user()accept an email directly — no more client-side ORCID resolution.- Chunked GCS uploads: 32 MiB chunks with per-chunk CRC32C and incremental SHA-256, verified
at
/complete. - Fixed a hashing bug affecting larger files (the actual 2.1.1 → 2.1.2 change).
- Public-sample support (
sample create --public).
Packaging¶
- New required dependencies:
configupdater,pyyaml,google-crc32c. - New optional extras:
[shell](prompt_toolkit) and[docs](mkdocs).
CLI¶
New subcommands: crucible file, crucible deletion, crucible status, crucible qr,
crucible tree, and crucible cast, plus an interactive shell when you run crucible with
no arguments.