Run Commands
Read the PPS series >

Autoscaling PPS

Enable autoscaling of the worker pool based on datums in queue.

Spec #

This is a top-level attribute of the pipeline spec.

{
    "pipeline": {...},
    "transform": {...},
    "autoscaling": false,
    ...
}

Behavior #

The autoscaling attribute in a Pachyderm Pipeline Spec is used to specify whether the pipeline should automatically scale up or down based on the processing load.

If the autoscaling attribute is set to true, Pachyderm will monitor the processing load of the pipeline, and automatically scale up or down the number of worker nodes as needed to keep up with the demand. This can help to ensure that the pipeline is always running at optimal efficiency, without wasting resources when the load is low.

When to Use #

You should consider using the autoscaling attribute in a Pachyderm Pipeline Spec when you have a workload that has variable processing requirements or when the processing load of your pipeline is difficult to predict.

Example scenarios: