Datum Processing States

Learn about the concept of datum processing states in Pachyderm.

March 22, 2023

When a pipeline runs, it processes your datums. Some of them get processed successfully and some might be skipped or even fail. Generally, processed datums fall into either successful or failure state category.

The following table describes the processing states of datums that can occur in Pachyderm:

Successful States

State Description
Success The datum has been successfully processed in this job.
Skipped The datum has been successfully processed in a previous job, has not changed since then, and therefore, it was skipped in the current job.

Failure States

State Description
Failed The datum failed to be processed. Any failed datum in a job fails the whole job.
Recovered The datum failed, but was recovered by the user’s error handling code. Although the datum is marked as recovered, Pachyderm does not process it in the downstream pipelines. A recovered datum does not fail the whole job. Just like failed datums, recovered datums are retried on the next run of the pipeline.

You can view the information about datum processing states in the output of the pachctl list job <jobID> command:

datums in progress


Datums that failed are still included in the total, but not shown in the progress indicator.