Run Commands
Read the PPS series >

Sidecar Resource Limits PPS

Set the upper threshold of resources allocated to the storage container.

Spec #

This is a top-level attribute of the pipeline spec.

{
  "pipeline": {...},
  "transform": {...},
  "sidecar_resource_limits": {
    "cpu": number,
    "memory": string,
    "gpu": {
      "type": string,
      "number": int
      }
    "disk": string,
  },
  ...
}

Attributes #

AttributeDescription
cpuThe maximum number of CPU cores that the sidecar container can use.
memoryThe maximum amount of memory that the sidecar container can use. This can be specified in bytes, or with a unit such as “Mi” or “Gi”.
gpuAn optional field that specifies the number and type of GPUs that the sidecar container can use.
typeThe type of GPU to use, such as “nvidia” or “amd”.
numberThe number of GPUs that the sidecar container can use.
diskThe maximum amount of disk space that the sidecar container can use. This can be specified in bytes, or with a unit such as “Mi” or “Gi”.

Behavior #

The sidecar_resource_limits field in a Pachyderm Pipeline Spec is used to specify the resource limits for any sidecar containers that are run alongside the main pipeline container.

In a Pachyderm Pipeline, sidecar containers can be used to perform additional tasks alongside the main pipeline container, such as logging, monitoring, or handling external dependencies. By specifying resource limits for these sidecar containers, you can ensure that they don’t consume too many resources and impact the performance of the main pipeline container.

This field can also be useful in deployments where Kubernetes automatically applies resource limits to containers, which might conflict with Pachyderm pipelines’ resource requests. Such a deployment might fail if Pachyderm requests more than the default Kubernetes limit. The sidecar_resource_limits enables you to explicitly specify these resources to fix the issue.

When to Use #

You should use the sidecar_resource_limits field in a Pachyderm Pipeline Spec when you have sidecar containers that perform additional tasks alongside the main pipeline container, and you want to set resource limits for those sidecar containers.

Example scenarios: