Senior Serverless Spark Migration Engineer

This a Full Remote job, the offer is available from: North America, United States, Latin America, Canada

Senior Platform Engineer, Cloud Infrastructure

Type: Remote (Brazil, Mexico)
Coverage: Pacific Hours (8:00 AM – 5:00 PM PST)

About Virtasant

Virtasant is a global cloud and technology services company helping organizations modernize, optimize, and build at scale. We work with enterprise customers on complex cloud, data, infrastructure, and AI initiatives, bringing together deep technical expertise and hands-on delivery.

About the Role

We’re looking for a Senior Serverless Spark Migration Engineer to help modernize a large-scale enterprise data platform.

You’ll lead the migration of production Apache Spark workloads from on-premise Hadoop/Spark environments to cloud-native and serverless architectures across AWS and GCP. This is a hands-on engineering role spanning workload assessment, architecture, application refactoring, migration execution, performance optimization, automation, and production readiness.

The goal is not simply to lift and shift existing workloads. You’ll determine the right target architecture for each workload and establish repeatable patterns that can eventually support migration at significant enterprise scale.

What You’ll Do

  • Lead migrations of enterprise Spark workloads from on-premise environments to AWS and GCP.

  • Assess Spark applications, clusters, configurations, dependencies, data flows, and resource utilization.

  • Determine the right migration approach across rehost, replatform, refactor, modernize, or retire.

  • Modernize traditional cluster-based workloads for serverless Spark where appropriate.

  • Design and implement architectures using technologies such as AWS EMR Serverless, S3, Glue, Lake Formation, GCP Dataproc Serverless, GCS, and BigQuery.

  • Refactor legacy PySpark/Scala/Spark SQL applications for cloud portability, scalability, and reliability.

  • Migrate workloads from environments using Hadoop, HDFS, YARN, Hive, and on-prem Spark clusters.

  • Troubleshoot and optimize Spark workloads across partitioning, shuffle behavior, joins, data skew, execution plans, executor configuration, serialization, and SQL execution.

  • Benchmark performance and optimize serverless workloads for performance, reliability, and cloud cost.

  • Build reusable migration tooling, automation, templates, and frameworks.

  • Implement CI/CD and Infrastructure as Code using tools such as Terraform.

  • Define testing, validation, cutover, rollback, observability, and production-readiness patterns.

  • Partner with Data Engineering, ML, Cloud Architecture, Platform Engineering, DevOps/SRE, Security, Governance, and FinOps teams.

What We’re Looking For

  • 8+ years of experience across data engineering, distributed systems, cloud engineering, or platform engineering.

  • 5+ years of hands-on Apache Spark experience in enterprise environments.

  • Strong PySpark and/or Scala development experience.

  • Proven experience migrating large-scale Spark workloads between infrastructure platforms.

  • Hands-on experience with both AWS and GCP.

  • Experience with on-premise Hadoop/Spark ecosystems, including technologies such as HDFS, YARN, and Hive.

  • Deep understanding of Spark internals and distributed processing.

  • Strong SQL and data engineering fundamentals.

  • Experience with cloud data lakes and object storage.

  • Strong production troubleshooting and performance-tuning experience.

  • Experience with CI/CD, Git, and Infrastructure as Code.

  • Ability to own migration work end-to-end, from discovery and architecture through production cutover and optimization.

Nice to Have

Experience with EMR/EMR Serverless, Dataproc/Dataproc Serverless, Glue, Lake Formation, BigQuery, Delta Lake, Iceberg, Kafka, Airflow, Terraform, Docker, or Kubernetes is valuable.

What Success Looks Like

You can take ownership of the complete migration lifecycle:

Discover → Assess → Design → Refactor → Migrate → Validate → Optimize → Operate

You understand both the legacy Hadoop/Spark world and modern cloud-native data platforms, and can make pragmatic architecture decisions based on workload characteristics rather than simply reproducing an existing environment in the cloud.

This offer from "Virtasant" has been enriched by Jobgether.com and got a 0% flex score.
Back to blog

Other Jobs To Apply

No other job posts for this day.

Common Interview Questions And Answers

1. HOW DO YOU PLAN YOUR DAY?

This is what this question poses: When do you focus and start working seriously? What are the hours you work optimally? Are you a night owl? A morning bird? Remote teams can be made up of people working on different shifts and around the world, so you won't necessarily be stuck in the 9-5 schedule if it's not for you...

2. HOW DO YOU USE THE DIFFERENT COMMUNICATION TOOLS IN DIFFERENT SITUATIONS?

When you're working on a remote team, there's no way to chat in the hallway between meetings or catch up on the latest project during an office carpool. Therefore, virtual communication will be absolutely essential to get your work done...

3. WHAT IS "WORKING REMOTE" REALLY FOR YOU?

Many people want to work remotely because of the flexibility it allows. You can work anywhere and at any time of the day...

4. WHAT DO YOU NEED IN YOUR PHYSICAL WORKSPACE TO SUCCEED IN YOUR WORK?

With this question, companies are looking to see what equipment they may need to provide you with and to verify how aware you are of what remote working could mean for you physically and logistically...

5. HOW DO YOU PROCESS INFORMATION?

Several years ago, I was working in a team to plan a big event. My supervisor made us all work as a team before the big day. One of our activities has been to find out how each of us processes information...

6. HOW DO YOU MANAGE THE CALENDAR AND THE PROGRAM? WHICH APPLICATIONS / SYSTEM DO YOU USE?

Or you may receive even more specific questions, such as: What's on your calendar? Do you plan blocks of time to do certain types of work? Do you have an open calendar that everyone can see?...

7. HOW DO YOU ORGANIZE FILES, LINKS, AND TABS ON YOUR COMPUTER?

Just like your schedule, how you track files and other information is very important. After all, everything is digital!...

8. HOW TO PRIORITIZE WORK?

The day I watched Marie Forleo's film separating the important from the urgent, my life changed. Not all remote jobs start fast, but most of them are...

9. HOW DO YOU PREPARE FOR A MEETING AND PREPARE A MEETING? WHAT DO YOU SEE HAPPENING DURING THE MEETING?

Just as communication is essential when working remotely, so is organization. Because you won't have those opportunities in the elevator or a casual conversation in the lunchroom, you should take advantage of the little time you have in a video or phone conference...

10. HOW DO YOU USE TECHNOLOGY ON A DAILY BASIS, IN YOUR WORK AND FOR YOUR PLEASURE?

This is a great question because it shows your comfort level with technology, which is very important for a remote worker because you will be working with technology over time...