How to upload data to Google Cloud Firestore using python async client

Rajathithan Rajasekar
2 min readMar 16, 2023
Photo by Cullan Smith on Unsplash

In this post, we will see how to upload data to Google cloud firestore using asyncio and aiofiles packages for concurrent execution of code which speeds up the upload process.

GCP firestore asynclient code snippets are available in the below link.

There is also nice blog post from twilio on how to use asyncio and aiofiles packages in python to work with files asynchronously.

For testing, I have chosen the below newline json dataset from kaggle.

Enable the Firestore api and provision Firestore in native mode in the region of your choice.

gcloud app create
gcloud firestore databases create --region us-central

In the below code snippet , we download the data from a GCS bucket and use the downloaded newline json file for upload into Firestore.

The writes and transaction limits of firestore are given below,

since I encoutered “google.api_core.exceptions.deadlineexceeded: 504 deadline exceeded “ exceptions during upload, I have added a sleep interval of 1 second for every 500 records.

After introducing the sleep interval , I didn’t encounter RPC deadline exceeded error.

Writes and Transactions — Firestore

Code Snippet:

I was able to upload ~90K records in 5 mins using the asyncio package. (If any one can tweak it further to achieve better performance and time , let me know).

Thanks for reading this post, I hope it was useful to you.

--

--

Rajathithan Rajasekar

I like to write code in Python . Interested in cloud , dataAnalysis, computerVision, ML and deepLearning. https://rajathithanrajasekar.medium.com/membership