https://amarpandey.me/ Recent content in Amar Prakash on Amar Prakash Pandey - ᕦ(ò_óˇ)ᕤ Hugo en-us Wed, 27 May 2026 00:00:00 +0000 https://amarpandey.me/blog/your-ai-agent-forgets-everything-lore-fixes-that/ Wed, 27 May 2026 00:00:00 +0000 https://amarpandey.me/blog/your-ai-agent-forgets-everything-lore-fixes-that/ You’re building something with Claude Code. The agent asks a question, you explain the context, it makes a good decision. You commit. Session ends. Next session. Different task, same file. The agent is about to do the exact thing you just told it not to do. You explain again. This isn’t a bug. It’s just how LLM sessions work — every session starts completely fresh. No memory of the last one. https://amarpandey.me/blog/functors-and-monads-wrapping-your-data-not-your-head-around-them/ Sun, 15 Mar 2026 00:00:00 +0000 https://amarpandey.me/blog/functors-and-monads-wrapping-your-data-not-your-head-around-them/ Why these words sound scarier than they are If you’ve ever Googled “what is a monad”, you’ve probably landed on a Stack Overflow answer that says something like “a monad is just a monoid in the category of endofunctors” and immediately closed the tab. Fair. Here’s the thing: you don’t need category theory to use functors and monads. You need a pipeline, some data, and a small conceptual shift in how you think about wrapping transformations. https://amarpandey.me/blog/currying-and-partial-application-shaping-intent/ Sat, 28 Feb 2026 00:00:00 +0000 https://amarpandey.me/blog/currying-and-partial-application-shaping-intent/ Why this topic is confusing Currying and partial application are concepts that almost every Python developer has seen, but very few feel confident using. For a long time, I understood what they did, but I didn’t know when using them actually made code better. A clear distinction: currying vs partial application Currying transforms a function that takes multiple arguments into a chain of functions that each take a single argument: https://amarpandey.me/blog/choosing-functional-python-without-forcing-it/ Sat, 31 Jan 2026 00:00:00 +0000 https://amarpandey.me/blog/choosing-functional-python-without-forcing-it/ Why this matters Python gives you multiple ways to solve the same problem. Loops, comprehensions, generators, map, filter, reduce, any, sorted, and more. For a long time, I knew these tools existed, but I didn’t really know when or why to use them. I could read functional-style code, but writing it confidently was another story. This document is not a tutorial on functional programming in Python. It’s a set of notes, patterns, and mental models that helped things finally click for me. https://amarpandey.me/blog/spot-optimizer--spot-fleet---decide-smart-then-launch/ Mon, 01 Sep 2025 00:00:00 +0000 https://amarpandey.me/blog/spot-optimizer--spot-fleet---decide-smart-then-launch/ Picking EC2 Spot instances shouldn’t feel like rolling dice. Most teams either guess and get bitten by interruptions, or overbuild complicated logic to dodge them. Here’s the clean split that keeps you sane: Spot Optimizer is a Python library/CLI that figures out the right instance types and counts for your workload. EC2 Fleet / Spot Fleet are AWS services that launch and maintain that capacity. Short version: Spot Optimizer decides what to run; EC2/Spot Fleet decide how to run it. https://amarpandey.me/blog/how-not-to-open-a-pull-request--and-how-to-do-it-right/ Fri, 25 Jul 2025 00:00:00 +0000 https://amarpandey.me/blog/how-not-to-open-a-pull-request--and-how-to-do-it-right/ Picture this: You’re maintaining an open-source project, and you receive a pull request that promises to revolutionize your codebase. Security improvements, architectural enhancements, performance optimizations – it has everything! Sounds perfect, right? That’s exactly what happened with my project spot-optimizer. The contributor was skilled, the intentions were great, but there was one major problem: Changes Count Lines Added 2,200+ Lines Removed 1,200 Files Changed 25 Total Commits 15 Despite the valuable improvements it offered, I had to close it. https://amarpandey.me/blog/never-push-a-work-commit-with-your-personal-email-again/ Sat, 05 Jul 2025 00:00:00 +0000 https://amarpandey.me/blog/never-push-a-work-commit-with-your-personal-email-again/ As developers, it’s common to wear multiple hats — working on company projects during the day and tinkering with open-source or personal side-projects at night. If you’ve ever accidentally committed to Git using your personal email in a company repo (or worse, pushed your work identity to a public repo 😅), you know how frustrating it can be. I used to run into this problem often, especially since I used the same laptop for both work and personal projects. https://amarpandey.me/blog/from-bottlenecks-to-balance-dynamic-skew-join-fixes-in-spark/ Mon, 07 Apr 2025 00:00:00 +0000 https://amarpandey.me/blog/from-bottlenecks-to-balance-dynamic-skew-join-fixes-in-spark/ When working with large datasets in Spark, joins are a common operation. But what happens when data distribution isn’t uniform? Let’s dive into a real-world scenario to understand why dynamic skew join optimization is not just useful, but often essential. The Problem Setup Assume we have two large tables and we’re trying to join them using the following Spark SQL: SELECT * FROM large_table_one JOIN large_table_two ON large_table_one.key = large_table_two. https://amarpandey.me/blog/4tb-ram-yet-an-oom-error-debugging-a-spark-memory-mystery/ Sun, 30 Mar 2025 00:00:00 +0000 https://amarpandey.me/blog/4tb-ram-yet-an-oom-error-debugging-a-spark-memory-mystery/ Everything seemed right—ample resources, a well-sized cluster, and yet, the Spark job kept failing with an out-of-memory error. Logs pointed to memory allocation failures, but with a 63-node cluster, each equipped with 64GB RAM, this shouldn’t have been an issue. We tweaked configurations, analyzed logs, and even considered scaling up the cluster. But the real solution? It wasn’t what we expected. The Data Challenge Our task involved processing three datasets: https://amarpandey.me/blog/deep-dive-into-spark-jobs-and-stages/ Sun, 23 Mar 2025 00:00:00 +0000 https://amarpandey.me/blog/deep-dive-into-spark-jobs-and-stages/ When working with large-scale data processing using Apache Spark, understanding how jobs and stages work is crucial to optimizing performance. This blog is for those who already have some experience with Spark and want to dig deeper into the internal mechanics of jobs and stages. Spark Transformations and Actions In Spark, operations are classified into two main categories: Transformations and Actions. Transformations: These operations do not trigger execution but define a new dataset from an existing one. https://amarpandey.me/blog/balancing-the-rum-conjecture-navigating-database-trade-offs/ Tue, 05 Nov 2024 00:00:00 +0000 https://amarpandey.me/blog/balancing-the-rum-conjecture-navigating-database-trade-offs/ When designing databases, there’s a constant balancing act among three main factors: Read times Update cost Memory/storage overhead The RUM Conjecture suggests that optimizing any two of these factors will negatively impact the third. Essentially, you can only choose two out of the three to prioritize in any design. Example: Log-Structured Databases Consider a log-structured database: Update-optimized: Records are appended at the end of the file, allowing efficient updates. Low memory/storage overhead: There’s no additional indexing, saving on storage. https://amarpandey.me/blog/the-cap-theorem-balancing-the-big-three-in-distributed-databases/ Tue, 15 Oct 2024 00:00:00 +0000 https://amarpandey.me/blog/the-cap-theorem-balancing-the-big-three-in-distributed-databases/ The CAP theorem, also known as Brewer’s theorem (named after computer scientist Eric Brewer), defines a fundamental trade-off in distributed systems: any distributed data store can provide only two out of three guarantees at any time: C: Consistency A: Availability P: Partition Tolerance What Do These Terms Mean? Consistency (C): Every read receives the most recent write or an error. This means that the data you access is guaranteed to be the latest version, or the system will notify you that something went wrong. https://amarpandey.me/blog/fine-tuning-shuffle-partitions-in-apache-spark-for-maximum-efficiency/ Fri, 24 May 2024 00:00:00 +0000 https://amarpandey.me/blog/fine-tuning-shuffle-partitions-in-apache-spark-for-maximum-efficiency/ Apache Spark’s shuffle partitions play a critical role in data processing, especially during operations like joins and aggregations. Properly configuring these partitions is essential for optimizing performance. Default Shuffle Partition Count By default, Spark sets the shuffle partition count to 200. While this may work for small datasets (less than 20 GB), it is usually inadequate for larger data sizes. Besides, who would work with just 20 GB of data on Spark? https://amarpandey.me/blog/handling-large-broadcast-joins-in-apache-spark/ Wed, 22 May 2024 00:00:00 +0000 https://amarpandey.me/blog/handling-large-broadcast-joins-in-apache-spark/ In Apache Spark, efficient data processing often relies on the use of broadcast joins. However, when the dataset exceeds a certain size, specifically 8GB, you may encounter the following error: Caused by: org.apache.spark.SparkException: Cannot broadcast the table that is larger than 8GB: 13 GB This error arises because Spark is attempting to broadcast a dataset that is larger than the maximum threshold allowed for broadcast joins. By default, Spark’s threshold for broadcasting is set to 8GB. https://amarpandey.me/blog/symptoms-of-bad-code/ Tue, 19 Jul 2022 00:00:00 +0000 https://amarpandey.me/blog/symptoms-of-bad-code/ 1. Rigidity Rigidity is the tendency of the system to be hard to change. Code that has dependencies that snake out in so many directions and you cannot make an isolated change without changing everything around it. Rigidity causes compile time error. 2. Fragility A system is fragile when a small change in one module causes other unrelated modules to misbehave. It is the tendency of the code to break in many places even when you make changes in one place. https://amarpandey.me/blog/docker---the-right-way/ Sun, 23 Jan 2022 00:00:00 +0000 https://amarpandey.me/blog/docker---the-right-way/ Docker is a software framework for building, running, and managing containers on servers and the cloud. Here are the several best practices for using Docker in production to improve security, optimize image size and write cleaner and more maintainable Dockerfiles. 1. Use Official Docker Image as Base Image Always use the official or verified base image when writing the docker file. Let’s say you are developing a java application and want to build it and run it as a docker image. https://amarpandey.me/blog/gitops---the-easy-way/ Tue, 04 Jan 2022 00:00:00 +0000 https://amarpandey.me/blog/gitops---the-easy-way/ What is GitOps? Treat the infrastructure as code the same way as application code. Separate repository for Infrastructure as code. DevOps pipeline. How does GitOps works? Infrastructure as Code hosted on Git repository. Version controlled. Team collaboration. Use branching strategy to merge code in git repository. With CI pipeline to test the code. With CD pipeline to apply the changes to the Infrastructure. With the above steps we achieve: Automated Process. https://amarpandey.me/blog/finger-detection-and-tracking-using-opencv-and-python/ Sat, 28 Jul 2018 00:00:00 +0000 https://amarpandey.me/blog/finger-detection-and-tracking-using-opencv-and-python/ TL;DR. Code is here. Finger detection is an important feature of many computer vision applications. In this application, A histogram based approach is used to separate out the hand from the background frame. Thresholding and Filtering techniques are used for background cancellation to obtain optimum results. One of the challenges that I faced in detecting fingers is differentiating a hand from the background and identifying the tip of a finger. https://amarpandey.me/blog/what-is-google-summer-of-code---how-to-prepare-for-it/ Sun, 02 Jul 2017 00:00:00 +0000 https://amarpandey.me/blog/what-is-google-summer-of-code---how-to-prepare-for-it/ We will talk about Google Summer of Code but before that let’s talk about what Open Source Development is. Yes, it’s very important. What is open source development? Open-source software development is the process by which open-source software, or similar software whose source code is publicly available, is developed. These are software products available with its source code under an open-source license to study, change, and improve its design. https://amarpandey.me/about/ Mon, 01 Jan 0001 00:00:00 +0000 https://amarpandey.me/about/ Hello, I’m Amar Prakash Pandey :) I’m a developer and hacker with a knack for solving problems in unconventional ways—finding simpler solutions where conventional methods fall short. Curiosity drives me, and I’m always experimenting with new ideas, whether it’s building efficient software, designing data platforms, or creating tools to make developers’ lives easier. You’ll often find me diving into distributed computing and large-scale data processing while sharing my learnings through open-source contributions and blogs. https://amarpandey.me/projects/ Mon, 01 Jan 0001 00:00:00 +0000 https://amarpandey.me/projects/ Here are some of the projects that I have worked on. You can find the source code and demo link for each project below.