Release Notes Highlights for Pachyderm

• Feature: You can now create pipelines in Console using jsonnet-compatible pipeline templates.
• Enhancement: Console now displays more information in the side panel for repos and pipelines, including the pipeline’s active workers and parallelism spec.
• Enhancement: Several web-responsive improvements have been made to Console: the side-panel now supports responsive sizing, and the DAG view side panel tab headers become scrollable on small screen sizes.
• Fix: Action buttons in the sidebar now become dropdown ellipsis menus when resizing the sidebar to a narrower size.
• Fix: When trying to download a single file, the file browser will download the file instead of opening the file in a new tab
• Fix: Correctly configured custom TLS certificates in Console.
• Fix: Improved our consistent hashing algorithm’s use of locking.

• Enhancement: Editing Project and Cluster Metadata is now available in Console.
• Enhancement: pachctl is now available in Determined notebooks.
• Fix: Resolve an unexpected file loading error when opening multiple files in the Jupyter extension explore view.
• Fix: Pachyderm Clients are now correctly loaded into Determined notebooks.

• Enhancement: Editing Commit and Repo Metadata is now available in Console.

• Feature: For enterprise customers using MLDM (Pachyderm) & MLDE (Determined) in a combined cluster environment, the MLDE Notebooks now include support for the Jupyter Pachyderm extension by default.
• Feature: You can now set up and maintain metadata on your Pachyderm artifacts. This includes clusters, projects, repos, branches, and commits.
• Enhancement: The blog storage configuration attribute GOCDK_ENABLED is now set to true by default in your Helm chart values; in 2.11.0 the option to disable it will be removed.
• Enhancement: The Console UI has undergone several improvements, including:
  • Improved file browsing experience
  • Improved DAG visualizations
    • Interactive DAG edge highlighting
    • Distinguishing colors and/or patterns based on pipeline types
    • Enhancing the ability to understand more about connections by their edges (like joins)
    • Visual indications of parallelism for pipelines when spec calls for it and when running
  • Pipeline and repo table paging

• Enhancement: Colors in the Console’s file browser have been improved for accessibility.
• Enhancement: Console’s callback page now properly redirects to the original page after login instead of redirecting to the home page (/).
• Enhancement: The visualization of the DAG in Console have been improved for better sizing and scaling with your browser window.
• Fix Resolved a bug where pipeline and job statuses were not correctly displaying.

• Enhancement: Updated to Go 1.22.2 and latest x/net.
• Enhancement: Upgraded to Envoy 1.27.4

• Enhancement: Updated to Go 1.22.2 and latest x/net.
• Enhancement: pinned Pachyderm-SDK versino to >=2.8.0<2.9.0 for the Jupyter Mount Extension.
• Enhancement: Upgraded to Envoy 1.27.4

• Enhancement: Updated to Go 1.22.2 and latest x/net.
• Enhancement: Upgraded to Envoy 1.27.4
• Fix: Resolved issue in Console where proxy port caused issues with downloading files.
• Feature: Added new args to debug dumps. You can review the full debug template by running pachctl debug template in the CLI.

Full Default Debug template #

$ pachctl debug template
defaults:
    clusterDefaults: true
inputRepos: true
starlarkScripts:
    - builtin: starlark/basic.star
      timeout: 60s
    - builtin: starlark/list_rcs.star
      timeout: 60s
system:
    database: true
    describes:
        - name: console                                                                                                                                                                                                                                                     - name: etcd
        - name: pachd
        - name: pachyderm-kube-event-tail
        - name: pachyderm-proxy
        - name: pg-bouncer
    helm: true
    logs:
        - name: console
        - name: etcd
        - name: pachd
        - name: pachyderm-kube-event-tail
        - name: pachyderm-proxy
        - name: pg-bouncer
    lokiLogs:
        - lokiArgs:
            maxLogs: "30000"
          name: console
        - lokiArgs:
            maxLogs: "30000"
          name: etcd
        - lokiArgs:
            maxLogs: "30000"
          name: pachd
        - lokiArgs:
            maxLogs: "30000"
          name: pachyderm-kube-event-tail
        - lokiArgs:
            maxLogs: "30000"
          name: pachyderm-proxy
        - lokiArgs:
            maxLogs: "30000"
          name: pg-bouncer
    profiles:
        - name: pachd
          profileArgs:
            profiles:
                - name: heap
                - name: goroutine
    version: true

• Enhancement: Refactored the Console’s Explore View.
• Enhancement: You can now adjust the Refecth Interval rate for Console in the Helm Chart.
• New Feature: You can now set roles on the Cluster level from within Console.