Adapting to a New Cloud Ecosystem

For much of my career, I primarily worked within the AWS ecosystem—not necessarily out of preference, but because it was the dominant platform in my workplaces. While I encountered Azure occasionally—much to the chagrin of developers—I had little exposure to GCP. I knew of its existence, of course, as it's Google after all.

Initially, I held a fondness for AWS. It felt familiar, despite the often convoluted connections between services. Navigating through endless links for the correct settings and wading through documentation frequently left me perplexed. AWS had become a part of my professional identity; the term "bucket" instantly brought S3 to mind. I had a love-hate relationship with tools like EMR and Athena, while I found Glue’s elegant data manipulation capabilities indispensable.

Reflecting on my AWS experience, I can liken it to a personal journey—a bit like moving from South Africa to the UK and realizing that living in fear, fortified by bars and electric fences, wasn't the norm everywhere.

Over the years, I toyed with the idea of developing side projects using GCP but always found excuses to delay. However, an exciting opportunity came my way: I accepted a new role at a company that exclusively uses GCP. I was thrilled to collaborate with exceptionally skilled professionals on a fresh Data Engineering venture. Though I had some reservations about transitioning to GCP, after six weeks, I can confidently say:

GCP is fantastic! It simplifies processes in a way that AWS sometimes complicates. In GCP, the focus is on creating innovative and functional solutions rather than getting bogged down by unnecessary complexities.

While I appreciate AWS for being a pioneer and transformative in many respects, I believe GCP offers distinct advantages in several areas. For instance, BigQuery stands out as one of GCP's top features I've experienced so far. The platform’s machine learning and AI capabilities appear well-crafted and forward-thinking. According to conversations with fellow engineers, Kubernetes excels in the GCP environment, and the Google Console is significantly more user-friendly than AWS. The project structure simplifies management compared to account-based systems, and the command-line interface tools make interfacing with infrastructure surprisingly easy.

These are my initial impressions of GCP. I plan to provide an update in six months to see if my perspective shifts, but for now, I’m thoroughly enjoying my experience with GCP.

For further insights, I found an amusing article on Medium discussing why GCP might surpass AWS—definitely worth a read!

If you’d like to connect, feel free to reach out on LinkedIn.

Chapter 1: The Shift to GCP

The first video titled "Which Cloud Should You Learn As A Data Engineer? - AWS Vs Azure Vs GCP" provides a comprehensive overview of cloud options for data engineers, outlining the strengths and weaknesses of each platform.

Chapter 2: GCP's Advantages

The second video, "Data Engineering with GCP - AWS vs Azure vs GCP," dives deeper into the comparative benefits of using GCP for data engineering tasks, highlighting its unique features and capabilities.