How to trigger Cloud Composer-2 DAG Pipelines from Cloud Functions-2

Rajathithan Rajasekar
7 min readJul 22, 2023
Photo by Ricardo Gomez Angel on Unsplash

In this post , we are going to see how to trigger cloud composer-2 DAG from Cloud Functions-2. To start with , first enable the cloud composer API in the Google cloud console.

Cloud Composer API

Next, we will select cloud composer-2 to provision the environment .

Note: CloudComposer-2 has the advantage of Autoscaling (GKE Auto-Pilot) over cloud composer-1. Detailed comparison is available in the Google cloud documentation site below.

Before provisioning , it is recommended to create a custom network for cloud composer. I have created a node subnet, services subnet and pod subnet. Post creation , these subnet ranges can be selected in the Network configuration of the cloud composer.

Cloud Composer Network
Grant required roles and permissions

It is also to be noted that instead of using the default compute service account, it is recommended to create a custom service account for security purposes. Since this is only a demo, I have used the default compute service account and granted the role to that service account.( If you still get any permission issues, follow the error instructions and provide the required privileges for the listed service account)

I have chosen a small environment size for demo purposes.

--

--

Rajathithan Rajasekar

I like to write code in Python . Interested in cloud , dataAnalysis, computerVision, ML and deepLearning. https://rajathithanrajasekar.medium.com/membership