4TB RAM, Yet an OOM Error? Debugging a Spark Memory Mystery
Our Spark job failed despite a 4TB RAM cluster. Scaling up wasn’t the fix—tuning executors and heap size was. Learn how we solved JVM memory inefficiencies and optimized Spark performance.
Hello👋
who am I?
Hi, I’m Amar, currently working as a Solutions Consultant at Sahaj Software in Pune, with a focus on data engineering. I enjoy building and fixing things, particularly when it involves working on large-scale data processing jobs and optimizing Spark workloads.
Over the years, I’ve worked on various software projects and continuously explore new technologies. I’ve also contributed as a past maintainer at CRI-O, DuckDuckGo, and Google Summer of Code.
find me here
- 🐙 Github — where my code lives (and occasionally breaks)
- 📝 Medium — where I pretend to be a writer
- 💼 LinkedIn — the professional me™
Recent Posts
Deep Dive into Spark Jobs and Stages
Learn how Apache Spark organizes and executes data processing through jobs and stages. Understand transformations, actions, and optimization strategies for better performance in large-scale data processing.