Pachyderm Worker
About #
Pachyderm workers are kubernetes pods that run the docker image (your user code) specified in the pipeline specification. When you create a pipeline, Pachyderm spins up workers that continuously run in the cluster, waiting for new data to process.
Each datum goes through the following processing phases inside a Pachyderm worker pod:
| Phase | Description |
|---|---|
| Downloading | The Pachyderm worker pod downloads the datum contents into Pachyderm. |
| Processing | The Pachyderm worker pod runs the contents of the datum against your code. |
| Uploading | The Pachyderm worker pod uploads the results of processing into an output repository. |
