Created the Spark Framework for Machine Learning Automation (AutoML). It's built to be scalable and efficient for a variety of tasks (binary/multi-class classification and regression) on tabular datasets with a variety of characteristics, including numeric, categorical, dates, texts, and so on. Within three months, the framework was developed and integrated into the frontend of L&T's LymByc product, and it was used by five major clients for key driver analysis, data exploration, and insights development. Designed and implemented a Model Management System for storing and retrieving PySpark models through S3. Leading the project to develop the Auto-Tune framework, which tunes Spark tasks on clusters automatically. For tuning Spark workloads, the approach employs a Heuristics-based approach (Rule-based approach) and an Optimization-based strategy (Machine Learning). Without requiring any human intervention, AutoTuning saves 30% of cluster resources and significantly improves the Spark Job success rate. It is used extensively in L&T's Lymbyc product. Extensively worked with Spark, AWS EMR, and AWS S3

Now acquired by CSG Designed and built a complete package for handling end-to-end machine learning tasks, including data preprocessing, advanced feature development, cross validation, and hyperparameter tuning for various models. Also allows the user to generate model training and profiling reports in order to assess model outcomes and uncover insights not apparent from the initial dataset. Developed scalable and modular microservices and optimized APIs utilizing multi threading in Python, reducing response time to less than 1 second Developed mechanisms to launch, monitor, and terminate stateless Spark Clusters thereby saving 30\% in VM cost Built ETL workflows on Spark achieving a 5X improvement from traditional Python workflow performance

As a decision scientist, I built and optimized data analytics infrastructures to help multiple global enterprise clients turn raw data into actionable insights. Creating and executing data pipelines and streamlining data operations. I built a big data pipeline for a telecom client to generate key insights from users' web interactions data. For this project, I conducted extensive research into the Lambda architecture for processing both real-time and batch data. I built a real-time processing pipeline using AWS Kinesis and Spark Streaming to process the data at a rate of over 1 million records per second. Reduced query times by pre-computing batch and real-time views of the data, resulting in a significant decrease in query runtime from over 5 seconds to 300 milliseconds on average. Created python notebooks and packages to solve NLP problems including intent classification, entity extraction, and topic modeling to be used across the organization Built MuSigma’s Artificial Intelligence-based assistant which acts as a layer of intelligence over MuSigma's CMS Developed and maintained several backend services for client needs using REST APIs. Improved algorithms and experimented with ML models for intent classification