PostprocessingOperations¶
Access via client.postprocessing. Start a run with client.datasets.postprocess(), then follow it here. A retried request that repeats an idempotency_key returns the original run instead of starting another, so automated flows should pass a stable key per task.
for pipeline in client.postprocessing.pipelines():
print(pipeline["name"], pipeline["description"])
run = client.datasets.postprocess(
dataset_mfid, "mosaic_stitch",
idempotency_key=f"{flow_run_id}:mosaic_stitch:{dataset_mfid}",
)
run = client.postprocessing.get(run["run_id"])
crucible.resources.postprocessing.PostprocessingOperations
¶
Operations for postprocessing pipelines and runs.
Access via: client.postprocessing.*
pipelines()
¶
List the pipelines that can postprocess a dataset.
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: Pipeline records with name, description, and the JSON schema of the parameters each accepts. |
list(dataset_mfid=None, pipeline=None, status=None, limit=DEFAULT_LIMIT, offset=0, cursor=None)
¶
List postprocessing runs on datasets the caller can read, newest first.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dataset_mfid
|
Optional[str]
|
Only runs on this dataset. |
None
|
pipeline
|
Optional[str]
|
Only runs of this pipeline. |
None
|
status
|
Optional[str]
|
One of crucible.constants.POSTPROCESSING_RUN_STATUSES. |
None
|
limit
|
int
|
Maximum number of results (default 100). |
DEFAULT_LIMIT
|
offset
|
int
|
Starting position in the full result set. |
0
|
cursor
|
str
|
Continuation token from a previous result's next_cursor; the listing resumes there. |
None
|
Returns:
| Type | Description |
|---|---|
List[Dict]
|
List[Dict]: PostprocessingRun records. |
get(run_id)
¶
Get one postprocessing run. Requires read access to its dataset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run_id
|
int
|
Integer id of the run. |
required |
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
The PostprocessingRun record. |
update(run_id, status, error=None)
¶
Report a run's progress. Requires edit access to its dataset.
Meant for the workers that execute pipelines.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
run_id
|
int
|
Integer id of the run. |
required |
status
|
str
|
"running", "completed", or "failed". |
required |
error
|
Optional[str]
|
Error message for a failed run (at most 4000 characters). |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
Dict |
Dict
|
The updated PostprocessingRun record. |
Raises:
| Type | Description |
|---|---|
HTTPError 409
|
The run already finished. |