Mar 7, 2023
Yes, you are right, we can use the storage transfer service to transfer b/w GCS buckets.
We can also filter the filenames using regex patterns and execute the transfer at a particular time window or make it event driven.
But the scenario we are trying to achieve here is event driven batch load , I think you cannot make STS intelligent enough to "WAIT" until the arrival of final file and then transfer the data as a batch to the destination GCS bucket.