Usage

Fundamentals

A package manager is a collection of software tools that automates the process of installing, upgrading, configuring, and removing computer programs for a computer in a consistent manner. [1]

However you must still specify to the package manager which software you want to process so you can start using them. This list of software is called an “environment”.

kloch allow to store on disk (serialize) this request of software so you can create reproducible environment for any user.

We abstract the package manager concept even more by using “launchers”, which is an arbitrary set of instruction to start an environment and execute software.

Basics

For those explanations we will assume the following context:

  • we are a VFX production working on an animated movie project.

  • we have developers working on the software pipeline, and artists which are consumers of this pipeline.

  • we are using rez as package manager to manage our software infrastructure.

Installation

  • First, ensure you have completed the Install page.

We then need to ensure our package-manager is available as launcher with kloch. In our case we need to install the rez env plugin.

Goal

Now for our goal we would like for our animated movie project, all artists use the same software versions without having to remember them. So they just say they want to load the profile “animated movie” and they have access to all the software in its expected version.

With our package-manager rez, we have already installed a package for all the desired software version, like maya-2023, houdini-20, …

Instructions

With kloch we will be able to create a profile that will “save” this request of packages:

basic-profile.yml
__magic__: kloch_profile:4
identifier: myMovie
version: 0.1.0
launchers:
  rezenv:
    requires:
      houdini: 20.2
      maya: 2023
      nuke: 15

Let say we store this profile in a shared location /d/pipeline/profiles/.

The first step is to make our profile discoverable. To do so you can use the environment variable:

# (shell syntax, to adapt for you context)
export KLOCH_CONFIG_PROFILE_ROOTS=/d/pipeline/profiles/

Tip

You can also set locations on the fly by using the --profile_roots CLI argument.

Then you can validate your manipulation by using the list command. This imply we will be using kloch as a Command Line Interface tool [2].

Note

Calling kloch will depends on how you installed it, here we will assume kloch have been installed in the currently activated python virtual environment and we will use python -m kloch to launch it.

python -m kloch list

Searching 1 locations: ['/home/runner/work/kloch/kloch/doc/source/_injected/demo-usage-basics'] ...
Found 1 valid profiles:
- myMovie

All good, which mean we can use our profile using the run command and by giving it the identifier of the desire profile to launch:

Tip

The command line interface tool is documented in CLI.

python -m kloch run myMovie

Instead of providing the profile identifier, you can also provide a path to an existing profile file, that may not even be registred in a profile root !

python -m kloch run /d/pipeline/profiles/basic-profile.yml

Which in our case should start a rez interactive shell.

You can also execute a command as multiple argument specified after a --:

python -m kloch run myMovie -- echo "hello world"

Note

Because profiles can also store a command, the CLI command is always appened to whatever is already defined.

And that are really the fundamentals of kloch design. We have in a way “aliased” the build of a complex software environment to a profile that can be launched in one line by artists.

Of course nothing force your artist to use the command-line and you could create a GUI that wraps kloch and yoru available profile to offer a more intuitive user-experience for them !

Standard workflow

Usually the creation of environment for VFX pipeline is much more granular, meaning we might want to have a profile for each department, e.g. one for modeling, one for lighting, etc.

It usually lead to a tree-like pipeline hierarchy:

studio > project > departement > context > ....

To reduce maintenance we would like not to rewrite all our requests in each profile but instead inherit from each hierarchy level requests.

Profile inheritance

This is possible with kloch where a profile can inherit from another profile.

/d/pipeline/profiles/studio.yml
__magic__: kloch_profile:4
identifier: myStudio
version: 0.1.0
launchers:
  rezenv:
    config:
      default_shell: powershell
    requires:
      studio_util: 1+
    environ:
      TOOLS_PATH:
        - /d/pipeline/studio/tools
/d/pipeline/profiles/myMovie.yml
__magic__: kloch_profile:4
identifier: myMovie
inherit: myStudio
version: 0.1.0
launchers:
  rezenv:
    requires:
      maya: 2023
      houdini: 20.2
    environ:
      PROD_NAME: myMovie
      TOOLS_PATH:
        - /d/pipeline/myMovie/tools

In the above example the myMovie profile now inherits from the myStudio profile by specifying it in the inherit key (note that we can only inherit one profile at once).

If we resolve the myMovie profile we can see exactly what the final request to rez will be :

python -m kloch resolve myMovie

__magic__: kloch_profile:4
identifier: myMovie
version: 0.1.0
launchers:
  rezenv:
    config:
      default_shell: powershell
    requires:
      studio_util: 1+
      maya: 2023
      houdini: 20.2
    environ:
      PROD_NAME: myMovie
      TOOLS_PATH:
      - /d/pipeline/studio/tools
      - /d/pipeline/myMovie/tools

As you can see the requires key contain the studio_util package defined in myStudio profile, same for the environ.TOOLS_path key that have been extended. By default when merging two profiles, dict and list are deep merged.

merging rules with tokens

It might be possible that when you merge 2 profile you want a bit more control over how each content section are merged together. This done via the usage of merge-rule tokens.

Tokens are predefined characters that are prefixed to the key of the key/value mapping and specify how to merge the base profile value with the “over” profile value. The following are available:

token

description

+=

indicate the key’s value should be appended to the base’s value

-=

indicate the base’s key should be removed if it exist

==

indicate the base’s value should be overriden

!=

add the key and value only if the key doesn’t exist in base

unset

no token is similar to += (append)

Important

The token logic can be tricky to understand at first. Make sure to read the full documentation in Profile File page.

A good use-case to change over the default behavior would be to have some debugging content in a “beta” profile, that you may want removed in the “production-ready” profile:

/d/pipeline/profiles/myMovie-beta.yml
__magic__: kloch_profile:4
identifier: myMovie:beta
version: 0.1.0
launchers:
  rezenv:
    requires:
      maya: 2023
      houdini: 20.2
      debugTools: 2.1+
/d/pipeline/profiles/myMovie.yml
__magic__: kloch_profile:4
identifier: myMovie
inherit: myMovie:beta
version: 0.1.0
launchers:
  config:
    package_filter:
      - excludes:
          - after(1714574770)
  rezenv:
    requires:
      -=debugTools: x

__magic__: kloch_profile:4
identifier: myMovie
version: 0.1.0
launchers:
  config:
    package_filter:
    - excludes:
      - after(1714574770)
  rezenv:
    requires:
      maya: 2023
      houdini: 20.2

You will notice that most of the time you won’t need any other token than the default one. The other tokens are provided for special occasion where you need a different behavior.

.base inheritance

The above example works fine if you only have one launcher. But profile can define multiple of them and it is highly probable that you will not use rez to start all your software.

With this use-case comes the need of sharing information between launcher. And this where the .base launcher comes into play:

/d/pipeline/profiles/studio.yml
__magic__: kloch_profile:4
identifier: myStudio
version: 0.1.0
launchers:
  .base:
    environ:
      STUDIO_PIPELINE_PATH: /d/pipeline
      STUDIO_LOCAL_PREFS_PATH: ~/studio
      STUDIO_COMMON_TOOLS_PATH: $STUDIO_PIPELINE_PATH/tools/bin
      PATH:
        - $PATH
        - $STUDIO_COMMON_TOOLS_PATH
/d/pipeline/profiles/diagnose.yml
__magic__: kloch_profile:4
identifier: diagnose
inherit: myStudio
version: 0.1.0
launchers:
  .base:
    environ:
      LOG_LEVEL: DEBUG
  .system:
    command:
      - diagnose
  .python:
    cwd: $STUDIO_COMMON_TOOLS_PATH
    python_file: ./diagnose.py

Which once merged internally by kloch will produce a launchers structure like:

.system:
  environ:
    STUDIO_PIPELINE_PATH: /d/pipeline
    STUDIO_LOCAL_PREFS_PATH: ~/studio
    STUDIO_COMMON_TOOLS_PATH: $STUDIO_PIPELINE_PATH/tools/bin
    PATH:
    - $PATH
    - $STUDIO_COMMON_TOOLS_PATH
    LOG_LEVEL: DEBUG
  command:
  - diagnose
.python:
  environ:
    STUDIO_PIPELINE_PATH: /d/pipeline
    STUDIO_LOCAL_PREFS_PATH: ~/studio
    STUDIO_COMMON_TOOLS_PATH: $STUDIO_PIPELINE_PATH/tools/bin
    PATH:
    - $PATH
    - $STUDIO_COMMON_TOOLS_PATH
    LOG_LEVEL: DEBUG
  cwd: $STUDIO_COMMON_TOOLS_PATH
  python_file: ./diagnose.py

As you can view, we inherit the environ keys that were defined in the .base launcher (that is deleted) for both .system and .python launchers.

It is also good to notice that we define environment variable that re-use previously defined environment variable, at profile level or system level ($PATH).

supporting different os

You can create conditional behavior in your profile by using Context tokens (explained in Profile File).

This token allow you to restrict the “visibility” of a launcher only when the system reading the profile reach some conditions. The most common one being which operating system it is currently running on.

__magic__: kloch_profile:4
identifier: greetings
version: 0.1.0
launchers:
  .base:
    environ:
      BASE_GREETING: "hello"
  .system@os=windows:
    command: powershell -c "echo $Env:BASE_GREETING there 👋"
  .system@os=linux:
    command: sh -c "echo $BASE_GREETING there 👋"

In the above example we use the builtin .system launcher to run a different command depending on the operating system.

Storing profiles

Profile can be stored at arbitrary location, then inventored in the environment variable (in most case).

The profile file name is arbitrary and have no specific usage.

We recommend that each of this location is version controlled [3] (e.g. with Git) to ensure that you can track the history of changes made to profile during the lifetime of your production/studio.

This is especially important if change made to a profile actually break the environment and you need to quickly rollback to its previous state.

Note that the version key specified at the root of the profile content help tracking and communicating changes but despite being enforced by the API to exists, is actually not used by the API anywhere (yet).

Advices

  • When defining paths, always avoid adding trailling slash (/ or \). That way you know that when joining paths you can always starts by one.

References