Skip to content

PostprocessingOperations

Access via client.postprocessing. Start a run with client.datasets.postprocess(), then follow it here. A retried request that repeats an idempotency_key returns the original run instead of starting another, so automated flows should pass a stable key per task.

for pipeline in client.postprocessing.pipelines():
    print(pipeline["name"], pipeline["description"])

run = client.datasets.postprocess(
    dataset_mfid, "mosaic_stitch",
    idempotency_key=f"{flow_run_id}:mosaic_stitch:{dataset_mfid}",
)
run = client.postprocessing.get(run["run_id"])

crucible.resources.postprocessing.PostprocessingOperations

Operations for postprocessing pipelines and runs.

Access via: client.postprocessing.*

pipelines()

List the pipelines that can postprocess a dataset.

Returns:

Type Description
List[Dict]

List[Dict]: Pipeline records with name, description, and the JSON schema of the parameters each accepts.

list(dataset_mfid=None, pipeline=None, status=None, limit=DEFAULT_LIMIT, offset=0, cursor=None)

List postprocessing runs on datasets the caller can read, newest first.

Parameters:

Name Type Description Default
dataset_mfid Optional[str]

Only runs on this dataset.

None
pipeline Optional[str]

Only runs of this pipeline.

None
status Optional[str]

One of crucible.constants.POSTPROCESSING_RUN_STATUSES.

None
limit int

Maximum number of results (default 100).

DEFAULT_LIMIT
offset int

Starting position in the full result set.

0
cursor str

Continuation token from a previous result's next_cursor; the listing resumes there.

None

Returns:

Type Description
List[Dict]

List[Dict]: PostprocessingRun records.

get(run_id)

Get one postprocessing run. Requires read access to its dataset.

Parameters:

Name Type Description Default
run_id int

Integer id of the run.

required

Returns:

Name Type Description
Dict Dict

The PostprocessingRun record.

update(run_id, status, error=None)

Report a run's progress. Requires edit access to its dataset.

Meant for the workers that execute pipelines.

Parameters:

Name Type Description Default
run_id int

Integer id of the run.

required
status str

"running", "completed", or "failed".

required
error Optional[str]

Error message for a failed run (at most 4000 characters).

None

Returns:

Name Type Description
Dict Dict

The updated PostprocessingRun record.

Raises:

Type Description
HTTPError 409

The run already finished.