Webpage
Helping Google Cloud Customers Ramp-up their Adoption of BigQuery
- By Sanjay Kalra, VP, Digital Transformation and Head, Alliances
- 04 July, 2023
-
-
The first challenge customers face while adopting BigQuery is getting their data into BigQuery. Google cloud makes it easy to migrate your data pipelines, with options ranging from repurposing existing data pipelines (redirecting them to load to BigQuery) to rewriting the data pipelines in order to take advantage of Google Cloud-managed services. You might have batch data pipelines or streaming data pipelines. Batch data pipelines run on data collected over a period of time (for example, once a day). Streaming data pipelines handle real-time events being generated by your operational systems—for example, in Change Data Capture (CDC) row changes being generated by your Online Transaction Processing (OLTP) databases.
Choosing a Migration Approach
Redirect data pipelines to write to BigQuery
Some popular Independent software vendors (ISVs) offer data processing tools (ELT/ETL) with BigQuery connectors. If your legacy data warehouse is fed by data pipelines executing an ETL procedure; or if the transformation logic is executed before the data is stored in the data warehouse, you can leverage third party tools like Informatica, Talend, etc.
Details of their BigQuery connectors are provided below:
- Informatica: BigQuery connector guide
- Talend: Writing data in BigQuery
Rewrite data pipelines to use Google Cloud-managed services
In many cases, you might choose to rewrite your existing data pipelines to use new frameworks and services that are fully managed on Google Cloud. This option works well if your existing pipelines were originally implemented with technologies that are now deprecated, or if you anticipate that porting and continuing to maintain those pipelines unmodified in the cloud would be too impractical or cost prohibitive.
Cloud Data Fusion and Dataflow are excellent offerings from Google Cloud for this purpose.
Intelliswift’s Offerings to Speed up Migration to BigQuery
As a Google Cloud partner having significant experience with Data Cloud, Intelliswift’s Data Management and Analytics practitioners can help execute any of the approaches above for our customers. Our Data Cloud CoE has also built some interesting frameworks and accelerators to reduce time to market, improve data quality, and manage the entire process end to end if the customer needs. Some of these include:
- Intelliswift Data Conversion Factory: This is our Data Migration as a Service offering – our team manages the end to end, large scale data migration needs triggered by M&A, cloud adoption, etc. The offering includes our professional services, project management, and accelerators.
- iDeeQ: Intelliswift’s open source-based data quality platform that can help accelerate your decision-making journey using accurate data. It addresses BigQuery and a variety of data formats, including databases, data warehouses and data lakes. Data quality score card & analytics enables to perform successful data quality management.
- AI-based Data Lineage: Our AI-based approach extracts metadata from your entire data environment. With a quick, secure and simple process, we are instantly able to analyze the metadata, and allow you to access data lineage, data discovery and a data catalog in one centralized platform.
- Data Observability: Our AI-based data observability approach helps analyze and monitor data, detect and resolve data quality issues without writing code, SQL's or rules. It uses powerful statistical analysis and machine learning, and a business user-friendly UI help users define what is correct vs incorrect data and then use these definitions to proactively monitor data and alert when deviations from the normal are detected.
Click to know more about our Google Cloud expertise or connect with us today to learn more about BigQuery for your data migration.
Sanjay Kalra, VP, Digital Transformation and Head, Alliances
Sanjay is a highly experienced subject matter expert in Information Technology, Product Engineering, Customer Experience, Cloud, Automation, Data Analytics, and AI – which enables him to serve as a trusted advisor to enterprises and growing organizations.
Share
