This blog post will cover my internship experience at Wingify from Dec ’17 to April ’18. The three reasons I am writing this blog post: A record for myself so that I always value my energy. To attempt providing value to others. Paras Chopra did not forget my promise of…

What is Flink? Apache Flink is an open-source distributed system platform that performs data processing in stream and batch modes. Being a distributed system, Flink provides fault tolerance for the data streams. What is Stateful Streaming Service? To make our platform fault tolerance, we want to preserve the system’s state from time to time. Achieving fault tolerance means…

A large scale distributed system that can support cyclic dataflows. Back in 2013 and even now, every distributed programming model was tightly coupled with the engine that executed this model and so interoperability was an issue. It is an issue even today but not that much — thanks to systems…

After working on MongoDB for about six years, I am sharing certain practices that have worked well for me. Introduction to MongoDB In relational database designs, the schema is statically defined. In document-based databases such as MongoDB, the schema is dynamic and is based on the document structure. MongoDB is a schemaless database. Designing MongoDB Document Stores …

This blog post will discuss the research paper, Zookeeper: Wait-free coordination for Internet-scale systems. Click here for downloading the paper. What problem does Zookeeper solve? Coordination is of utmost importance in distributed applications. While building distributed systems, we need primitives like distributed lock services, shared registers, group messaging, etc. Instead of implementing these on our…

This article explains the research paper, “Patience is a Virtue: Revisiting Merge and Sort on Modern Processors” published by Microsoft Research. I intend to explain the research paper in a simple manner. The reason I chose this algorithm is due to its application scope — infrastructure monitoring and historical analysis…

A blog post that explains the research paper: The Five-Minute Rule Ten Years Later, and Other Computer Storage Rules of Thumb While working as a data engineer, I often encounter the problem of data storage: whether data access should be kept faster or should I go for cold-storage or something…

Why do you need dimensional data modeling and how to implement it? — What is dimensional data modeling? Dimensional modeling (DM) is part of the Business Dimensional Lifecycle methodology developed by Ralph Kimball which includes a set of methods, techniques and concepts for use in data warehouse design. The approach focuses on identifying the key business processes within a business and modelling and implementing these first before adding…

Working as a Data Engineer, you are directly connected to your data science teams and some senior-level personnel who give you business use-cases. You have to be involved in the complete process to gain awareness of why a particular action should be taken. …