Dataprep In-VPC Execution

By default, Trifacta Photon and connectivity jobs execute in the Alteryx VPC. As needed, you can configure these jobs to run in your VPC.

For these job types, there are two types of configuration:

Details on these configuration methods are provided below.

The following limitations apply to this release. These limitations may change in the future:

Before you begin, please verify that your VPC environment has the following:

Acquire from Alteryx:

In-VPC execution must be enabled by an administrator. In the Dataprep Settings page, you can enable the following settings.

For more information, see Dataprep Project Settings Page.

This Service Account is assigned to the nodes in the GKE node pool and is configured to have minimal privileges.

Following are variables listed in the configuration steps. They can be modified based on your requirements and supported values:

myproject

myregion

Please execute the following commands from the gcloud CLI:

gcloud iam service-accounts create trifacta-service-account \
--display-name="Service Account for running Trifacta Remote jobs"

gcloud projects add-iam-policy-binding myproject \
--member "serviceAccount:trifacta-service-account@myproject.iam.gserviceaccount.com" \
--role roles/logging.logWriter


gcloud projects add-iam-policy-binding myproject \
--member "serviceAccount:trifacta-service-account@myproject.iam.gserviceaccount.com" \
--role roles/monitoring.metricWriter


gcloud projects add-iam-policy-binding myproject \
--member "serviceAccount:trifacta-service-account@myproject.iam.gserviceaccount.com" \
--role roles/monitoring.viewer


gcloud projects add-iam-policy-binding myproject \
--member "serviceAccount:trifacta-service-account@myproject.iam.gserviceaccount.com" \
--role roles/stackdriver.resourceMetadata.writer

gcloud projects add-iam-policy-binding myproject \
--member "serviceAccount:trifacta-service-account@myproject.iam.gserviceaccount.com" \
--role roles/artifactregistry.reader

Verification steps:

Command:

gcloud projects get-iam-policy myproject --flatten="bindings[].members" --format="table(bindings.role)" --filter="bindings.members:serviceAccount:trifacta-service-account@myproject.iam.gserviceaccount.com"

The output should look like the following:

ROLE
roles/artifactregistry.reader
roles/logging.logWriter
roles/monitoring.metricWriter
roles/monitoring.viewer
roles/stackdriver.resourceMetadata.writer

The following configuration is required for Internet access to acquire assets from Dataprep by Trifacta, if the GKE cluster has private nodes.

gcloud compute routers create myproject-myregion \
--network myproject-network \
--region=myregion

gcloud compute routers nats create myproject-myregion \
--router=myproject-myregion \
--auto-allocate-nat-external-ips \
--nat-all-subnet-ip-ranges \
--enable-logging

Verification Steps:

You can verify that the router NAT was created in the Google Cloud Platform Console: https://console.cloud.google.com/net-services/nat/list.

This configuration creates the GKE cluster for use in executing jobs. This cluster must be created in the VPC/sub-network that has access to your datasources, such as your databases and Cloud Storage.

In the following, please replace w.x.y.z with the IP address provided to you by Alteryx for authorized control plane access.

gcloud container clusters create "trifacta-cluster" \
--project "myproject" \
--region "myregion" \
--no-enable-basic-auth \
--cluster-version "1.20.8-gke.900" \
--release-channel "None" \
--machine-type "n1-standard-16" \
--image-type "COS_CONTAINERD" \
--disk-type "pd-standard" \
--disk-size "100" \
--metadata disable-legacy-endpoints=true \
--service-account "trifacta-service-account@myproject.iam.gserviceaccount.com" \
--max-pods-per-node "110" \
--num-nodes "1" \
--logging=SYSTEM,WORKLOAD \
--monitoring=SYSTEM \
--enable-ip-alias \
--network "projects/myproject/global/networks/myproject-network" \
--subnetwork "projects/myproject/regions/myregion/subnetworks/myproject-subnet-myregion" \
--no-enable-intra-node-visibility \
--default-max-pods-per-node "110" \
--enable-autoscaling \
--min-nodes "0" \
--max-nodes "3" \
--enable-master-authorized-networks \
--master-authorized-networks w.x.y.z/32 \
--addons HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver \
--no-enable-autoupgrade \
--enable-autorepair \
--max-surge-upgrade 1 \
--max-unavailable-upgrade 0 \
--workload-pool "myproject.svc.id.goog" \
--enable-private-nodes \
--enable-shielded-nodes \
--shielded-secure-boot \
--node-locations "myregion-a","myregion-b","myregion-c" \
--master-ipv4-cidr=10.1.0.0/28 \
--enable-binauthz 

Verification Steps:

You can verify that the cluster was created through the Google Cloud Platform Console: https://console.cloud.google.com/kubernetes/list/overview.

Use the following command to set up configuration to connect to the new cluster:

gcloud container clusters get-credentials trifacta-cluster --region myregion --project myproject

The following commands whitelist the Cloud shell for use on the cluster:

For basic configuration, Trifacta Photon uses the default node pool. No additional configuration is required.

For basic configuration, Trifacta Photon uses the default namespace. No additional configuration is required.

Please execute the following commands:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: false
metadata:
  namespace: default
  name: trifacta-job-runner
EOF

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
 name: trifacta-job-runner-secret
 annotations:
  kubernetes.io/service-account.name: trifacta-job-runner
type: kubernetes.io/service-account-token
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: trifacta-job-runner-role
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create", "delete"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "create", "delete", "watch"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["list", "get"]
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: trifacta-job-runner-rb
subjects:
- kind: ServiceAccount
  name: trifacta-job-runner
  namespace: default
roleRef:
  kind: Role
  name: trifacta-job-runner-role
  apiGroup: rbac.authorization.k8s.io
EOF

cat <<EOF | kubectl apply -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-list-role
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
EOF

cat <<EOF | kubectl apply -f -
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: node-list-rb
subjects:
- kind: ServiceAccount
  name: trifacta-job-runner
  namespace: default
roleRef:
  kind: ClusterRole
  name: node-list-role
  apiGroup: rbac.authorization.k8s.io
EOF

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: false
metadata:
  name: trifacta-pod-sa
EOF

The following commands create the encryption keys for credentials:

openssl genrsa -out private_key.pem 2048


openssl pkcs8 -topk8 -inform PEM -outform DER -in private_key.pem -out private_key.der -nocrypt


openssl rsa -in private_key.pem -pubout -outform DER -out public_key.der

base64 -i public_key.der > public_key.der.base64

base64 -i private_key.der > private_key.der.base64


kubectl create secret generic trifacta-credential-encryption -n default \
--from-file=privateKey=private_key.der.base64

// Create a new Kubernetes ServiceAccount on the GKE cluster with an annotation to bind it to the allAccess@myproject.iam.gserviceaccount.com Companion ServiceAccount.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
automountServiceAccountToken: false
metadata:

  annotations:
    iam.gke.io/gcp-service-account: allAccess@myproject.iam.gserviceaccount.com

  name: trifacta-pod-sa-allaccess
EOF    


// Allow the Kubernetes ServiceAccount to impersonate the Google IAM ServiceAccount by adding an IAM policy binding between the two service accounts. This binding allows the Kubernetes ServiceAccount to act as the IAM ServiceAccount.
gcloud iam service-accounts add-iam-policy-binding \
  --project <project_name>
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:<project_name>.svc.id.goog[default/trifacta-pod-sa-allaccess]" \

allAccess@myproject.iam.gserviceaccount.com

Wait a couple of minutes for the binding to take effect.