Create a data pipeline

Create a Cloud Data Fusion instance

1. Click Create an instance.

   Go to Instances

2. Enter an Instance name.
3. Enter a Description for your instance.
4. Enter the Region in which to create the instance.
5. Choose the Cloud Data Fusion Version to use.
6. Choose the Cloud Data Fusion Edition.
7. For Cloud Data Fusion versions 6.2.3 and later, in the Authorization field, choose the Dataproc service account to use for running your Cloud Data Fusion pipeline in Dataproc. The default value, Compute Engine account, is pre-selected.
8. Click Create. It takes up to 30 minutes for the instance creation process to complete. While Cloud Data Fusion creates your instance, a progress wheel displays next to the instance name on the Instances page. After completion, it turns into a green check mark and indicates that you can start using the instance.

Navigate the Cloud Data Fusion web interface

When using Cloud Data Fusion, you use both the Google Cloud console and the separate Cloud Data Fusion web interface.

• In the Google Cloud console, you can do the following:

  • Create a Google Cloud console project
  • Create and delete Cloud Data Fusion instances
  • View the Cloud Data Fusion instance details

• In the Cloud Data Fusion web interface, you can use various pages, such as Studio or Wrangler, to use Cloud Data Fusion functionality.

To navigate the Cloud Data Fusion interface, follow these steps:

1. In the Google Cloud console, open the Instances page.

   Go to Instances

2. In the instance Actions column, click the View Instance link.
3. In the Cloud Data Fusion web interface, use the left navigation panel to navigate to the page you need.

Deploy a sample pipeline

Sample pipelines are available through the Cloud Data Fusion Hub, which lets you share reusable Cloud Data Fusion pipelines, plugins, and solutions.

1. In the Cloud Data Fusion web interface, click Hub.
2. In the left panel, click Pipelines.
3. Click the Cloud Data Fusion Quickstart pipeline.
4. Click Create.
5. In the Cloud Data Fusion Quickstart configuration panel, click Finish.

6. Click Customize Pipeline.

   A visual representation of your pipeline appears on the Studio page, which is a graphical interface for developing data integration pipelines. Available pipeline plugins are listed on the left, and your pipeline is displayed on the main canvas area. You can explore your pipeline by holding the pointer over each pipeline node and clicking Properties. The properties menu for each node lets you view the objects and operations associated with the node.

7. In the top-right menu, click Deploy. This step submits the pipeline to Cloud Data Fusion. You will execute the pipeline in the next section of this quickstart.

View your pipeline

The deployed pipeline appears in the pipeline details view, where you can do the following:

• View the structure and configuration of the pipeline.
• Run the pipeline manually or set up a schedule or a trigger.
• View a summary of historical runs of the pipeline, including execution times, logs, and metrics.

Execute your pipeline

In the pipeline details view, click Run to execute your pipeline.

When executing a pipeline, Cloud Data Fusion does the following:

1. Provisions an ephemeral Dataproc cluster
2. Executes the pipeline on the cluster using Apache Spark
3. Deletes the cluster

View the results

After a few minutes, the pipeline finishes. The pipeline status changes to Succeeded and the number of records processed by each node is displayed.

1. Go to the BigQuery web interface.

2. To view a sample of the results, go to the DataFusionQuickstart dataset in your project, click the top_rated_inexpensive table, then run a simple query. For example:

   SELECT * FROM PROJECT_ID.GCPQuickStart.top_rated_inexpensive LIMIT 10
   

   Replace PROJECT_ID with your project ID.