Custom package import (Python)

Promote now supports the installation of model package dependencies from private, hosted git repositories via personal access token (PAT) authentication over HTTPS. SSH is currently not supported.

To import private packages, you will need Promote version 2018.4.0 or higher.

Creating an access token

Personal access token creation depends on which remote hosting service you are using, but they are all pretty similar. Generally, creating a token proceeds from your account settings:

Gitlab

Go to Profile Settings > Access tokens
Additional info:
https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html

Github

Go to Settings > Developer Settings > Personal access tokens

Bitbucket

Go to Manage account > Account settings > Personal access tokens

Once you are at the token creation stage, you are asked what kind of access scope you want to give the token, and whether you want to give it an expiration date. We recommend assigning the most limited access scope possible. This varies depending on hosting provider, but you would ideally choose a read-only repository scope.

Once a model is successfully deployed, the token should not be necessary for restarts and redeploys from the UI. However, if you are redeploying your deploy script from the Python or R client, you will need to make sure the token is valid, or has a new token in the link.

If you are not satisfied with the token scoping options presented by your remote hosting provider, keep in mind that the token will reflect (at most) the permissions of the user that creates it. If you want more granular control of your access token, we recommend creating a user that only has the permissions you want your Promote users to have (i.e., read (pull)-only rights to the specific package repositories you are looking to install and no others), and creating the token from that account.

Including an access token in your model

Once you have created a token, you can create an HTTPS link that allows you to pull the packages into your Promote model during the build phase. Refer to the documentation for your hosting provider for best practices in structuring this link. Generally, a link with the following format will work:

git+https://x-access-token:<yourPersonalAccessToken>@git.yourHostingProvider.com/username/packageName.git

Python Dependencies

For Python, dependencies are managed using the requirements.txt file. You can add the PAT link linewise to the requirements.txt file just like any other dependency, and it will be pip installed when the promote model is built.

A normal HTTPS link to a public repository, without the additional token authentication information, will also install correctly.

Example:

git+https://x-access-token:<yourPersonalAccessToken>@git.yourHostingProvider.com/username/packageName.git

git+http://git.yourHostingProvider.com/username/packageName.git@myBranch

git+http://git.yourHostingProvider.com/username/packageName.git@myTag

git+http://git.yourHostingProvider.com/username/packageName.git@myFullCommitHash

Specifying a "version" of your private repository

The Python client and Promote application use a requirements.txt file to install packages. In order to target a specific branch or version, use the correctly formatted link as defined by pip. Refer to the section on Git links here: https://pip.readthedocs.io/en/1.1/requirements.html.

In order to keep build times and redeploys fast, Promote caches a number of build steps when first building your model, including dependency installations. If one of your custom packages has been updated upstream, we recommend tagging all release branches in your package’s repository and targeting the tag, or ideally, SHA in your links to the repository in requirements.txt.

Promote currently only supports links using the HTTPS protocol.

Deactivating your access token after it has been deployed

Usually wherever you created the token in the respective remote host (Github, Gitlab, or Bitbucket), will have a list of currently active tokens. You can revoke and delete the token from the list. For a model that has been successfully deployed, the token should not be necessary for restarts and redeploys from the UI.

Be aware that when using importing private libraries, an expired PAT may make it impossible to roll back to past versions, requiring a new deploy with the older model code and a new PAT.

If you are redeploying your deploy script from the Python or R client (for instance, to a new model version) you will need to make sure there is a valid token in the link.