Reference
PachCTL

Pipeline Scaling Limits (CE)

Learn about the built-in scaling limitations of Pachyderm's Community Edition.

March 24, 2023

Our free Pachyderm Community Edition contains built-in scaling limitations and parallelism thresholds. To scale beyond these limits, request a Pachyderm Enterprise trial token and enjoy unlimited scaling, and more.

📖

You might qualify for a free Enterprise license.

Pachyderm offers activation keys for proofs-of-concept, startups, academic, nonprofit, or open-source projects. Tell us about your project!.

Scaling Limits #

Number of concurrent pipelines deployed Number of workers for each pipeline
Community Users can deploy up to 16 pipelines. Community Users can run up to 8 workers in parallel on each pipeline.

What happens when you exceed those limits? #

As a general rule, Pachyderm provides an error message in the STDERR whenever a limit is encountered that prevents you from successfully running a command. In that case, the alert message links to a free trial request form.

Limit on the number of pipelines #

When exceeding the number of pipelines:

ℹī¸

If update pipeline fails for any other reason, it does not log any message related to pipeline limits.

All of the commands listed above create a distinct message to STDERR and to the pachd logs. This message includes information such as the limit on the number of pipelines in the Community Edition, the total number of pipelines deployed, and provides a link to request an Enterprise key to lift those limitations.

Limit on the number of workers per pipeline #

When constant parallelism > 8:

What happens when your license expires? #

If your Enterprise License has expired and you have more than 16 pipelines, all existing pipelines continue to work. However, you will not be able to create additional pipelines. Same behavior if you upgrade your cluster.

⚠ī¸

Restoring or installing Pachyderm with an expired license will fail.

ℹī¸

Pipelines automatically generated by the system (for example cron…) are not considered when assessing the total number of pipelines deployed. The limit applies to user-created pipelines only.