Nowadays migrating on-premise existing BigData workloads on the cloud is one of the main customer's challenges.
Based on our experience, it is really complicated for several reasons. For example, Data Scientists does not know what is needed to allow their models running in the cloud. Or IT and Business is struggling to define a organization process in respect of compliance and governance.
With this project, we want to show how SAS can support migrating analytics on Google Cloud Platform providing business process and governance by SAS Model Manager and SAS Workflow Manager
Below the on-premise application architecture of the solution:
In as-is:
- Data Scientist submits pyspark job to Hadoop-Spark cluster on premise.
Then migration starts. Then:
-
Data Scientist creates several pyspark model packages. And he registers different versions in SAS® Model Manager using SAS® sasctl.
-
Big Data Engineer maps on-prem cluster configuration and create a templates. Then DevOps engineer may create a cloud function to deploy the cluster (and submit jobs) on Google Cloud Platform
Then, the workflow starts and
- Each actor involved in the process claims and completes user tasks. In this case, a Validator approve the Champion model. And an IT provides Cloud bucket name for migration.
-
Clone the project.
-
Create virtual environment
cd mlops_sas_gcp_spark/ python3 -m venv env source env/bin/activate pip install --file requirements.txt source deactivate
-
Create a folder for Viya logs
mkdir logs
-
Change group ownership based on SAS Workflow Administration group. For example, if you have SAS Workflow Administration group called CASHostAccountRequired, run
chown -R sas:CASHostAccountRequired mlops_sas_gcp_spark/
-
Import ModelOps for GCP in SAS Workflow Manager
-
Import run_build.sas and run_migrate.sas jobs in SAS Job Execution
Notice: Don't forget to double checks RESTAPIs endpoints in the workflow definition service tasks and paths in the jobs
From SAS Model Manager project, start ModelOps for GCP workflow and user tasks one by one.
Below some frames of the demo.
Feedbacks and Pull requests are welcome.
For major changes, please reach out both me or Artem Glazkov, SAS Russia