Let us begin with understanding of Google Cloud dataflow what it is all about before moving to our GCP Cloud Dataflow plugin and how it benefits to our workload automation users.
Data is generated in real-time from websites, mobile apps, IoT devices and other workloads. Capturing, processing, and analyzing this data is a priority for all businesses. However, data from these systems is not often in the format that is conducive for analysis or for effective use by downstream systems. That’s where Dataflow comes in! dataflow is used for processing and enriching batch or stream data for use cases such as analysis, machine learning or data warehousing.
Dataflow templates offer a collection of pre-built templates with an option to create your own custom ones! Here in workload automation, you can implement Text files on Cloud Storage to BigQuery template.
Let us now understand the plugin part with job definition parameters:
Log in to the Dynamic Workload Console and open the Workload Designer. Choose to create a new job and select “GCP Cloud Dataflow” job type in the Cloud section.
Figure 1: Job Definition Page
Establishing connection to the Google Cloud server:
Use this section to connect to the Google Cloud in two ways.
- GCP Default Credentials — If Customer VM’s already resides inside GCP environment then no need to provide credential explicitly.
- GCP Server Credentials – Manually provide below details.
Service Account – The service account associated to your GCS account. Click the Select button to choose the service account in the cloud console.
Project ID – The project ID is a unique name associated with each project. It is mandatory and unique for each service account.
Test Connection – Click to verify if the connection to the Google Cloud works correctly.
Figure 2: Connection Page
In Action tab specify the bucket name and operation which you want to perform.
- Job Name – Specify the name of the job, must be unique among running jobs.
- Region – Choose a dataflow regional endpoint to deploy worker instances and store job metadata.
- Template Path – Text files on Cloud Storage to BigQuery.
- JSON Path – The Cloud Storage path to the JSON file that defines your BigQuery schema. Ex: gs://your-bucket/your-schema.json
- BigQuery Output Table – The location of the BigQuery table in which to store your processed data. If you reuse an existing table, it will be overwritten. Ex: your-project:your-dataset.your-table.
- Cloud Storage Input Path – The path to the Cloud Storage text to read. Ex: gs://your-bucket/your-file.txt
- Temporary BigQuery Directory – Temporary directory for the BigQuery loading process. Ex: gs://your-bucket/your-files/temp-dir.
- Temporary Location – Path and filename prefix for writing temporary files. Ex: gs://your-bucket/temp.
- Network – Network to which workers will be assigned. If empty or unspecified, the service will use the network “default”.
- Subnetwork – Subnetwork to which workers will be assigned, if desired. Value can be either a complete URL or an abbreviated path. If the subnetwork is located in a Shared VPC network, you must use the complete URL.
Figure 3: Action Page
Submitting your job
It is time to Submit your job into the current plan. You can add your job to the job stream that automates your business process flow. Select the action menu in the top-left corner of the job definition panel and click on Submit Job into Current Plan. A confirmation message is displayed, and you can switch to the Monitoring view to see what is going on.
Figure 4: Submit Job Into Current Plan Page
Figure 5: Monitor Page
Users can cancel the running the job by clicking kill option.
Job Log Details
Figure 6: Job Log Page
Figure 7: Workflow Details Page
Are you curious to try out the GCP Cloud Dataflow plugin? Download the integrations from the Automation Hub and get started.
Suhas H N, Senior Developer at HCL Technologies
Rabic Meeran K, Senior Engineer at HCL Technologies
Responsible for design and develop integration plug-ins for Workload Automation. Having two decades of experience in creating software products to enterprise customers.
Saket Saurav, Tester (Senior Engineer) at HCL Technologies
Responsible for performing Automation and Manual Testing for different plugins in Workload Automation using Java Unified Test Automation Framework. Hands-on experience on Java programming language, Web Services with databases like Oracle and SQL Server.