Upload data from CSV file to GCP BigQuery using Python

Photo by Markus Winkler on Unsplash

The easiest way to upload CSV file to a BQ table is using the Google cloud storage bucket, upload the data to a GCS bucket and refer that GCS uri location as the source when creating a new table, you can also provide the schema definition , perform cluster & partition settings and other advanced option as required. The details are provided in the link below.

Loading CSV data from Cloud Storage | BigQuery | Google Cloud

But in this post, we are going to see how we can upload data to BQ using python. Use the below script when it satisfies the below conditions.

• When your data upload is an one time activity and no future data uploads are required.
• You would like to have a progress-bar to keep a check on the data upload progress.
• You would like to define the schema or get the schema of an existing table.
• You would like to keep a tab on the upload errors.
• Completely replace of the data of an existing table or create a new table.
• Perform data transformations to the CSV data before the BQ upload.

Thanks for reading this post, I hope it was useful to you.