Renien John Joseph

From Data Platforms to Agentic AI: Building AI-Native Platforms at Scale 🚀

2025-12-31T19:39:55+00:00

Looking Back: Building Data Foundations

My work in Data and AI began with a deceptively simple but hard question:

How do you build data platforms that scale with the business?

In 2023, most of my focus was on:

Designing and operating large-scale data platforms
Improving data reliability, freshness, and trust
Enabling analytics, experimentation, and machine learning
Reducing friction between engineering, product, and data teams

At that stage, success wasn’t measured by how advanced the models were, but by whether data systems were dependable enough to support real decisions.

This period reinforced a belief I still hold strongly:

Strong AI systems are built on strong data platforms.

ELT-Based Big Data Stacks as a Foundation

One of the most impactful architectural decisions was adopting ELT-based big data stacks.

ELT enabled:

Scalable ingestion of diverse data sources
Clear separation of data movement from data modeling
Faster iteration on transformations and schemas
Better support for analytics and ML workloads

This approach allowed teams to focus on using data, not fighting pipelines.
More importantly, it created a flexible foundation capable of supporting increasingly advanced AI use cases.

High-Level Platform Architecture

The diagram below shows how ELT-based data platforms evolve into AI-native platforms that support agents.

AI-Native Platform at Scale (High Level)

From Predictive Models to Agentic Systems

Traditional ML systems fit neatly into batch processing or request–response APIs.

Agentic AI changes the paradigm.

Agentic systems:

Reason continuously over time
Plan and execute multi-step actions
Use tools and data dynamically
Learn from outcomes and feedback

This reframes AI from prediction services into systems that think and act.

At this stage, platform design becomes more important than any individual model.

Data Platforms as Cognitive Infrastructure

Agentic AI raises expectations for what data platforms must provide.

Beyond storage and pipelines, platforms now need to support:

Contextual access to historical and real-time data
Clear data semantics and lineage
Durable memory grounded in reliable storage
Feedback loops tied to real outcomes
Strong access controls and safety boundaries

Data platforms become:

Cognitive infrastructure that grounds autonomous intelligence in reality

They are what make agent reasoning auditable, explainable, and trustworthy.

Open-Source Observability: Making AI Operable

As systems become more autonomous, observability becomes foundational.

A significant part of my work has involved deploying open-source observability stacks spanning data, ML, and agents.

These stacks monitor:

Data pipelines and freshness
Feature generation and serving
Model performance, drift, and bias
Agent reasoning, decisions, and tool usage

Observability Across Data, ML, and Agents

Observability for Data + ML + Agents

Open-source observability provides transparency, extensibility, adherence to regulatory rules, and deep platform integration.

More importantly, it turns AI from a black box into an operable system.

Data, ML, and Agent APIs as Platform Interfaces

At scale, intelligence must be exposed through clear, stable interfaces.
Modern platforms achieve this through a combination of data, ML, and agent APIs that separate implementation from usage while enabling governance, control, and observability.

Platform APIs: Data, ML, and Agent Interfaces

Modern intelligent platforms typically provide:

Data APIs for features, aggregates, and real-time signals
ML APIs for predictions, embeddings, and scoring
Agent APIs for tool access, memory, and controlled actions

These APIs are where governance meets autonomy—ensuring that intelligent systems remain auditable and controllable.

Architecture Overview

Platform APIs: Data, ML, and Agent Interfaces

APIs decouple intelligence from implementation details and make agent behavior controllable and auditable.

ELT, APIs, and Observability: Enabling Agentic AI

Agentic AI does not emerge from models alone—it is the result of platform design choices.
A reliable foundation is created by combining three core capabilities:

ELT-based data platforms that support scalable, replayable, and auditable data flows
Strong Data and ML APIs that expose features, predictions, and embeddings through stable interfaces
Open-source observability tools that provide visibility into decisions, actions, and outcomes

Together, these components transform AI from isolated predictions into operational intelligence.

What This Enables

A platform built on ELT, APIs, and observability allows teams to:

Roll out intelligent systems incrementally, starting with human-in-the-loop workflows
Evaluate human and agent decisions side by side, using the same data and metrics
Continuously monitor, learn, and improve, based on real-world feedback
Build trust through visibility and control, rather than blind automation

Scaling Intelligence Responsibly

This approach ensures that intelligence scales in a way that is:

Powerful, by leveraging modern data and ML capabilities
Transparent, through observable behavior and decision traces
Accountable, with clear ownership, governance, and guardrails

This is how agentic systems move from experimentation to production—
not as black boxes, but as trusted, auditable platform capabilities.

Looking Toward 2026: Building AI-Native Platforms

Looking ahead, my focus is shifting toward AI-native platform design—systems where intelligence is not an add-on, but a foundational capability.

Rather than centering on individual models, AI-native platforms are built to support reasoning, action, and learning as continuous processes.

Areas of Focus

I’m particularly interested in advancing platforms that emphasize:

Agent-Aware Data Architectures
Data systems designed to support planning, memory, feedback loops, and long-running agent workflows.
Observability for Reasoning and Actions
Visibility not just into outputs, but into why decisions were made and how actions unfolded over time.
Data, ML, and Agent APIs as First-Class Citizens
Stable interfaces that make intelligence accessible, governable, and composable across the organization.
Human-in-the-Loop Autonomy
Systems that blend automation with oversight, enabling agents to act independently while remaining accountable.
Intelligence as Infrastructure
Platforms that treat reasoning, learning, and decision-making as shared capabilities—just like compute, storage, and networking.

A Platform-First Future

The future of AI will be defined less by individual models and more by the systems that surround them.

AI-native platforms will be the differentiator—enabling organizations to scale intelligence responsibly, adapt continuously, and build trust in autonomous systems.

This is the direction where durable, production-grade AI will be built.

From Data Platforms to Agentic AI: Building AI-Native Platforms at Scale 🚀 was originally published by Renien Joseph at Renien John Joseph on December 31, 2025.

Refelction on 2023 🚀

2023-12-31T19:39:55+00:00

Fork me on GitHub

The end of a year is the beginning of another new journey with exciting plans. It’s been a great year with lots of learning that always leads to reflection and plans for the coming year.

Friends/subscribers of my blog know the whole journey of our Data Platform-in-a-Box. Personally, it has changed my perspective on how I look at Data Platform/Data Team back in 2021: Read More.

It used to be a BIG mammoth and now it’s tied up to a SMALL box 😀

Transition journey

Never thought building and operating a Data Platform will become sweet as pie. The key secret of success is stepping out of your comfort zone, which means learning new things, meeting new people, seeing new places, and trying new experiences.

Most importantly keep consistent on the following,

Keep your bars higher…
Incrementally move the bars to higher…

Achieving all your aspirational goals is always not an easy thing. It’s a lifelong journey. The journey will be a bumpy ride, keep breaking all the obstacles/mountains to climb the next mountain. But always remember to celebrate the small victories.

Next year 2024, is going to be a transaction journey to Data & AI in a box.

Data & AI Transition / Data & AI in-a-Box

I will keep sharing the journey with Data & AI in-a-box. Keep following and follow my github profile to get notified on all the new projects.

Stay Tuned!

I can’t wait to crush 2024 together… cheers! 🍺

Refelction on 2023 🚀 was originally published by Renien Joseph at Renien John Joseph on December 31, 2023.

[Modern] On-Prem Data Platform

2023-08-27T19:39:55+00:00

An on-premises data warehouse refers to a data storage and management solution that is physically hosted within an organization’s own data center or facilities, as opposed to being hosted in the cloud.

In recent years, the trend has been shifting toward cloud-based data warehousing solutions due to their scalability, flexibility, and reduced upfront costs. However, on-premises data warehouses continue to be relevant for organizations with specific requirements around data control, security, compliance, and performance.

Data Privacy

In recent times I’ve been solutioning/designing on-prem data platforms due to data residency law. When it comes to on-prem data platforms, the first thing that a data geek will think about is managing Hadoop clusters. But the intention on a SaaS data platform is to reduce the operational overhead and managing Hadoop clusters is actually a big overhead.

If it’s only for canned reports that can be done easily with RDBMS databases. But nowadays a data centric approach is a key thing for any business. If we want to really do data driven business it’s vital to have a proper data lake, data warehouse or data mesh strategy to enable it.

Hadoop Data Lake & Warehouse

The technology growth has changed a lot of pieces on big data and data platform spectrum. As I said, managing Hadoop clusters is an overkill to a company and team. So basically we need to find reliable data lake storage similar to cloud bucket storage.

Minio & Chep / Rook both are very popular open source distributed storage that can be managed easily and it supports on top of the kubernetes cluster. This actually changed the whole thinking pattern of the Hadoop ecosystem. In our recent on-prem solution we wiped out the Hadoop ecosystem idea and managed to build a fully managed on-prem data platform on top of the kubernetes cluster.

S3 like bucket

Since managing and querying big data on Hadoop can be challenging. We found alternative solutions like Minio/CEPH S3 object stores which you can access and query with Presto or Trino. Therefore, we ended up relying on S3 object stores that’s deployed on K8 clusters and Hive metastore, Trino for data warehousing.

Modern On-Prem DWH Solution

So far the journey looks really great and moto on building Data Platform-in-a-Box rolling out to be real and enjoying the performance and scalability on the k8 cluster.

The current setup on on-prem data platform is somewhat equivalent to all cloud data platforms. One of the missing pieces in this blog is on masking policies and solutions. We have already solved it as well and in my next blog post I will explain about it.

Stay Tuned!… Cheers! 🍺

[Modern] On-Prem Data Platform was originally published by Renien Joseph at Renien John Joseph on August 27, 2023.

Introducing Light-Weight (EL) Extractor Loader

2023-02-05T19:39:55+00:00

Data engineering field is evolving and transforming in many ways, driven by the increasing amount of data being generated and the need to MAKE SENSE of it. In data engineering there are two concepts ETL (Extract, Transform, Load) and ELT (Extract, Load, Transform) that are used to move between two systems. ETL is a traditional approach and ELT is a more modern approach with the influence on the cloud platform where the data is first extracted from the source system and loaded into the target system and then transformed.

Extract Load Transform

In the ELT approach, data extraction is always a challenging part for data engineers. Data extraction refers to the process of retrieving specific information from a large or small dataset. It can be done manually or through the use of various tools and techniques.

Sqoop is one of the important tools that is used across the industries. I was one of the biggest fans with Sqoop and it has helped me throughout many years to extract data from relational databases. Still we are relying on Sqoop in most of the legacy platforms.

In this journey Sqoop was not the right tool for us and currently it’s been retired by the apache community as well.

Apache Sqoop - Retired

So I had to look for a better tool to extract data from multiple sources and found a lot of proprietary tools where we need to pay a lot of money. But after a while staying in this field we always pop up with crazy ideas. One of them was to create a self service ELT tool to support our data platform.

Self Service Product

After a couple of months, we managed to build a lightweight EXTRACTOR<>LOADER tool that will support any type of database. The outcome of the product was great and the performance of EXTRACTOR-LOADER is really impressive.

The concept we relied from the old school books theories and improvised a bit to create the lightweight library. Actually you can build it using any programming language that you prefer and we mostly relied on python.

Producer Consumer Restaurant Example [Ref:https://levelup.gitconnected.com/producer-consumer-problem-using-mutex-in-c-764865c47483]

This idea got triggered from the very old theory of producer-consumer concept. Like shown in the above figure where the cook will keep preparing the food and place it on a shared resource plate and the kid keeps enjoying the food.

Producer Consumer Pattern For Data Ingestion Problem

Similarly, to our EXTRACTOR LOADER problem we can apply the producer-consumer approach. Extractor job will be the producer who extracts the data from source and consumer will be a loader job that keeps consuming (loading the data) the extracted data when it’s available in the shared resource.

Light-Weight Framework

Currently with one producer and consumer the LIGHT-WEIGHT library; it’s showing massive improvement in the performance. Our next step will be to improve the framework to implement a multi producer-consumer solution to increase the performance of EXTRACT LOADER tasks. If we manage to crack the solution I will share the concept in my blog.

Stay Tuned!… Cheers! 🍺

Introducing Light-Weight (EL) Extractor Loader was originally published by Renien Joseph at Renien John Joseph on February 05, 2023.

Data Platform-in-a-Box

2022-12-14T19:39:55+00:00

Fork me on GitHub

The software industry is gradually evolving and everyone likes to have quick delivery similar to fast food. In order to stay competitive, organizations are engaged in developing highly scalable products and services.

I recently prepared a quick fast food and full meals with different dishes. Actually, a full meal with different dishes stays in our memory for a long time compared with quick fast food. Similarly, a well-designed product always helps the company to scale up easily by attracting a lot of customers and also it gives us the flexibility to customize based on their choices/needs. If you look at below pics, potato wedges vs banana leaf meals; who will not like a full meals with different flavors 😀.

Fast Food vs Full Meals

From the inception data has always been a critical component. Data platform is gradually becoming/evolving to support the mainstream system to give the power back to business. Being in a SaaS hyper growth company is a fun ride. There were multiple key challenges to build a Data Platform to serve the SaaS business. It’s like sailing in the rough sea 🙂.

Sailing in the rough sea

There were lot of questions to be answered to achieve this mission,

How to manage multiple flavors of data warehouse?
How to maintain multiple data platforms/multiple code bases?
How about the data accuracy?
How about managing PII data and data governance?
How do we maintain the systems? Do we need a big army?
How fast can we bring up a data platform?
How can we build a platform to empower data science to work?

The answer relies on two strategies.

Strategy 1: Build a team with a culture of engineering excellence

Culture is not something you build overnight but it’s created by everyone and everything that happens on your team. It’s actually a journey with lots of ups and downs. But if you have a clear vision that can be explained to your team and show them by practicing in your day-to-day activities to reach the goals. After a while the people around you will understand the importance of it and gradually their actions influence others as well.

Motto: “Should be cathedral builders, not bricklayers”

Strategy 2: Build data platform in a box

Motto: “Data platform in one button click with 360 view on quality”

Data is scattered all around the platform and around us. The following are the key modern principles of data platform.

View data as shared asset
Provide the right interfaces to the users to consume data
Ensure the security of data and access controls
Establish a common vocabulary
Curate the data
Eliminate data copies and movements
Data quality

It’s essential to follow these above key modern principles to build a stable data platform.

It’s important first to build a stable and reliable data platform then bring DevOps culture to the team to bundle the platform into a BOX.

Upskill your team!

During ‘data platform in a box’ journey it’s actually demands to have skills not only on DevOps side but the knowledge about the data tools and architectures/solutions. So the only option is to upskill the data team on DevOps skills to MAKE IT HAPPEN.

Actually without achieving Strategy 1; we will not be able to achieve Strategy 2.

If I rewind back for demi-decade period, the love towards big data and data science is to the highest level and I still didn’t lose interest in it. Because it’s evolving and keeps me really engaged with very complex systems.

But during the journey on bringing a ‘data platform in a box’ just expanded my interest in DevOps and it will be part of my journey forever. Understanding devops to control big data and data science playground just drives me crazy.

Data + Data Science + DevOps 💗

This year the journey taught me so many things. Not only on the tech side, but also life lessons. I’m sorry, I was unable to spend time sharing some ground level work on how to bring the data platform in a box. I was really busy building the data platform. But for sure, I will spend some time next year sharing some interesting topics on ‘data platform with devops’.

I will share sample templates and code to kick start the journey with data platform tools and devops in my github. Keep following and follow my github profile to get notified on all the new projects.

Stay Tuned!

I can’t wait to crush 2023 together… cheers! 🍺

Data Platform-in-a-Box was originally published by Renien Joseph at Renien John Joseph on December 14, 2022.

Life Hacks #1: My Wife Starting To Like Linux Distro

2022-05-28T19:39:55+00:00

During working hours or in an office call my family looks at me a bit differently. Especially when I always look at linux terminals, my wife keeps on bugging me as a weird guy! For the past two year I actually tried to explain in different ways but still she keeps on asking the same question.

What is this black screen that you are staring at?

Almost two years back (2019) the coronavirus COVID-19 affected all aspects of society and all dimensions of sustainable development. It actually created a paradigm shift in our work life as well where we had to work from home. But we were not used to it and faced multiple problems. But In fact there is a saying

Difficult Times Can Bring Out the Greatest Innovations

Due to the sudden lockdowns my wife had to work from home and she has been working at the office all these years and she is only familiar with Windows PCs. Since her work is into personal financial data, the company requested to install Citrix System in her personal machine to work in a cloud environment.

But unfortunately we didn’t have a high-end Windows 10 laptop to support the Citrix version. We have an old machine and it works well with the Windows 7 operating system. Now, unfortunately I got all the tech related complaints. I tried so many different versions of the Citrix but it was not compatible with the Windows 7 machine and the machine was not performing well and needed a fresh install as well. But my current laptop specs do not support Windows 10/11 😢…

So I took this chance to experiment and create an environment to introduce a linux based system to my wife 😀. First I installed elementary OS. Because I thought it will be good for beginners and she can learn quickly. But I faced so much trouble to find relevant drivers to support audio and wifi.

After almost 4-5 hours I gave up and installed Ubuntu and with the help of the open source community I managed to install all relevant drivers to support our old laptop. But now another problem, basically citrix application does not support linux base system.

😓

When one door closes, another door opens

My experiment box is always a Virtual Machine (VM). So I installed a VM inside my Ubuntu machine and installed Windows 11 and it worked without any issues 🎇. Finally managed to set up Citrix successfully in Windows 11.

Dell with Linux

This is a great chance that I had to show the power of open source technology and she is kind of getting familiar with Ubuntu and started to like it alot 😃

Yes! she’s a keeper!

Key learnings: we shouldn’t not force the OS on anyone. People should find out for themselves if it suits their needs.

Life Hacks #1: My Wife Starting To Like Linux Distro was originally published by Renien Joseph at Renien John Joseph on May 28, 2022.

SaaS - Data Platform

2022-05-15T19:39:55+00:00

If you think to become a good Software Engineer means being a good thinker. But to become a master means willing to learn new challenges and improving the way we think.. Right? I think it’s applicable for any field of work. Because recently I tried out some unique dishes and it ended up really well! 😊. My wife and little princess 👼 liked it alot and approved it 💯

Wedding-style Chicken Biryani

At a certain point, the things that get in the way help to improve problem solving skills. Basically when something is actually happening in our mind we should engage with the problem to find the root cause of it to fix it permanently.

If I look at my journey my love toward solving problems is gradually expanding. I started as Desktop application developer; then moved to full stack developer and ended up settling down at big data and data science. I think probably you all might have guessed it based on my last two articles.

Yes!, SRE (DevOps) is becoming one of my interest areas for quite some time now.

Site reliability engineering (SRE) was born at Google in 2003, prior to the DevOps movement, when the first team of software engineers was tasked to make Google’s already large-scale sites more reliable, efficient, and scalable.

In my current workplace - Circles.Life we are creating SaaS (Software as a Service) products and for some time focusing on building SaaS-Data Platform. To achieve it we need to focus on two key important things.

Single Code Base - It will out-of-the-box support high configuration to deliver multiple launches
Automate End-to-End - It will include infrastructure creation, delete, update and deploy applications

Folks on big data & data science teams current reaction will be:

“Dude, how can we do it? Because, There are so many tools and applications to build a futuristic data platform”

Yes! I know I was in the state for almost 3 months. But after getting to know about DevOps and the tools took me to a different type of world. Starting to love the journey to automate end-to-end Data Platform. If you have a good understanding about big data and data science tools with DevOps magic wands we will be able to automate it with one button click!

I have already started posting a few items on DevOps work with Data Platform and stay tuned for more interesting posts.

PS: As new parents just puts your whole world into perspective. Everything else just disappears 😫. But I will try to find some time to share the knowledge.

SaaS - Data Platform was originally published by Renien Joseph at Renien John Joseph on May 15, 2022.

Trino - Popular Distributed Interactive Query Engine

2021-06-05T19:39:55+00:00

Fork me on GitHub

Trino(formerly PrestoSQL) is a popular distributed interactive query engine in data platform. Trino can be used as a fast query engine.

As a data platform component, Trino is becoming one of my favorite components. Here I am going to show you the way to deploy/explore trino in your local machine.

You can find a simplified diagram of how trino is integrated with the data platform.

This tool can enric Data fabric architecture and Data mesh architecture.

The project is built and deployed using docker. It has simply 3 module trino-base where you will find the base image and two important components for trino.

trino-coordinator
trino-worker

To get more detail deployment steps please refere to this README file.

Thanks to Trino where we can have one query engine and write federated query 🔥 to communicate with different data sources 🤯

Trino POC: Trino

Trino - Popular Distributed Interactive Query Engine was originally published by Renien Joseph at Renien John Joseph on June 05, 2021.

Installing Spark & Livy with Ansible

2021-01-22T19:39:55+00:00

Fork me on GitHub

Deploying your own Spark Standalone cluster with Livy is a time consuming task due to the complexity of the deployments. Even for the guys who want to play around with spark and livy in their local machine or even in virtual machines this project will be useful.

In my recent project we are building a SAAS platform where we need to deploy dev and stage environments in different cloud providers. Therefore, I used docker and ansible for deployments. Spark can be run on top of the Hadoop ecosystem and as stand alone mode. Therefore, I created the latest version (2.4.7) of Spark Standalone docker image and the latest version of Livy (0.7.0 Incubating) image.

The following starter kit repository contains a simple ansible playbook to install a Spark Standalone cluster and Livy in docker. Current playbook contains only local ip to install docker and deploy spark cluster and livy. Based on your needs you will be able add more environments and automate your big data dev, stage and prod environments.

The repository README file contains more details about setting up the environments.

Thanks to Ansible we can scale the platform with one button click.

Starter Kit: Ansible Spark Livy

Installing Spark & Livy with Ansible was originally published by Renien Joseph at Renien John Joseph on January 22, 2021.

Building Apache Spark with Play Framework

2020-03-10T19:39:55+00:00

Fork me on GitHub

Nowadays businesses are hiring data scientists in droves to make rigorous, scientific, unbiased, data-driven decisions. The data is huge, therefore the biggest challenge is to build faster and scalable systems for prediction and decision making engines.

Without faster and scalable decision-making engine business will not be able to create data driven business models

There are situations where business needs real time predictions for daily trained models. One of the most popular high velocity framework is Play Framework. It allows to build web applications which follow the model-view-controller (MVC) architectural pattern. Apache Spark is a lightning-fast cluster computing engine that supports well matured many distributed machine learning algorithms.

What if we can run Apache Spark on a local machine along with Play Framework controllers?

If we can manage to run Apache Spark application along with Play application we can leverage Spark ML algorithms for real time predictions.

In my recent work I had to setup play with spark and I faced multiple dependency issues in Akka and guava versions. Therefore I have created a starter kit for anyone who is interested in working with play-spark.

In this starter kit you will find an API where it will count the words on top of spark context and the result will be returned as response (http://localhost:9000/prediction).

In this example it includes, Spark Model loading guice module where it creates a Spark Context/Spark Session as singleton object during the Application startup time.

By following this example we can integrate any complex Spark ML models to provide real time prediction.

.
├── app
│   ├── controllers
│   │   └── HomeController.java
│   ├── guice
│   │   └── module
│   │       └── MLLibModule.java              --> Machine Learning guice module
│   ├── ml
│   │   └── ModelPrediction.scala             --> Model prediction class (word count example)
│   ├── response
│   ├── startup
│   │   └── AppLoader.java                    --> App loader module
.
.

Starter Kit: Play-With-Spark

Building Apache Spark with Play Framework was originally published by Renien Joseph at Renien John Joseph on March 10, 2020.

Life is a Journey

2017-12-31T19:39:55+00:00

Most of them not know me, but I’ve written this article specifically for you. Yes, you. I know you’re feeling pretty down right now and that current situation makes the world seem like full of misery.

Friends who knows me would sound like,

“Common man this supposed to be a TECH blog, what are all these scrap?”

Response,

“Hold on guys! Its Year end, I can take one more step to make another mistake and learn from it :P”

Hey there, don’t worry this blog always will be about tech information and I just want to share this years’ experience which I learned from 2017.

Sharing is Caring

Most of us will always have great plans in the beginning of the year but after couple of months including me they will forget the plan and act based on reflection on your past days. It will become kind of feedback loop at the year-end and you will realise all the mistakes and freak out.

I’m not here to tell you next year you can plan again and achieve it and that everything is going to be okay – you already know that. But despite knowing it, you still feel doubtful about it. I was always in an impression; I was able handle all the pressure but just realised within last 3 weeks what I faced was terrible. I was so patient and handling work life and family but some actions can end with biggest disaster.

My dear friends, but always don’t lose your hope. Experience matters because always keep in mind,

“Life is a Journey, Not a Destination”

Happy New Year 2018 ..

Life is a Journey was originally published by Renien Joseph at Renien John Joseph on December 31, 2017.

Cloud-DataSync

2017-05-09T19:39:55+00:00

There are plenty of good reasons to move towards cloud, but mainly it make good business sense. Service like AWS Elastic MapReduce (EMR Cluster) and GCP DataProc makes life easy for the companies to maintain and extract intent from large data.

In big data platform Lamda Architecture (LA) is a well-known concept for scalable and fault-tolerant data processing platform. Nathan Marz address the importans of ‘vertically partitioned data’ – Big Data

If you determine the requirement of big data storage the main thing is reliable and It is recommended to store the data as different partitioned (eg: daily, hourly partitioned). There are many advantages by partitioning vertically the data. One of the important scenario is while processing the data, if the extracted click stream data is corrupted after a deployment change due a bug it will be really easy to skip those corrupted partitioned than spending long hours to debug the large data.

Therefore, during the migration process the biggest challenge is to move all the partitioned/non-partitioned HDFS data to Cloud buckets. Considering the migration process and to sync the data,

Introducing: cloud-datasync

The cloud-datasync tool currently supports to incrementally copy the partitioned and bulk copy the non-partitioned data to GCP/AWS cloud data storage from any local linux/mac machines. These are new features I have planned for the tool,

Using ‘distcp’ command effectively copy the data from local HDFS cluster to any cloud service
Automatically generate Azkaban job flow and schedule the data copy jobs in Azkaban servers. We can use Azkaban Python Package to implement this feature.

Let me know what you think about ‘cloud-datasync’ below in the comments and share your thoughts. If you want to share any new features/issues, feel free to open an issue in the GitHub repository.

Cloud-DataSync was originally published by Renien Joseph at Renien John Joseph on May 09, 2017.

Doc-Diff (Python Library)

2017-04-02T19:39:55+00:00

Fork me on GitHub

Python is often the choice for developers who need to apply data analysis in their work or mainly data scientists/data engineers whose tasks are more related deriving insight from the data.

One of Python’s greatest assets is its extensive set of libraries. Recently, I was working on very popular Data Mining algorithms (i.e: FP-Growth and Custom A-Priori). There was a situation I wanted to get comprehensive analysis report on results generated by these algorithms.

As a support lib for Data Science work introducing “doc-dff — Generate the diff data between two files”

doc-diff supports the following features:

Generate the following comparison reports
- common_in_doc1-and-doc2-%Y-%m-%d.csv
- common_key_with_diff_values-%Y-%m-%d.csv
- exclusive_in_doc1-%Y-%m-%d.csv
- exclusive_in_doc2-%Y-%m-%d.csv
Compare two files and return following ‘dicts(prodCode, recommendation)’
- common_in_doc1_and_doc2_list = dicts()
- common_key_with_diff_values_list = dicts()
- exclusive_in_doc1_list = dicts()
- exclusive_in_doc2_list = dicts()

Install

$ pip install doc-diff

Implementation

from doc_diff import Diff
from doc_diff import gen_comp_report

if __name__ == '__main__':
    # Data file location
    a_priori_csv_location = "./data/a-priori.csv"
    pfp_csv_location = "./data/pfp.csv"

    # Process a-priori.csv data file
    a_priori_diff = Diff(a_priori_csv_location)
    a_priori_diff.process_file()

    # Process pfp.csv data file
    pfp_diff = Diff(pfp_csv_location)
    pfp_diff.process_file()

    gen_comp_report(a_priori_diff, pfp_diff)

I’m looking forward to open source all my supportive lib for Data Science/Data Engineering work. Let me know what you think about ‘doc-diff’ below in the comments and share your thoughts. If you want to share any new features/issues, feel free to open an issue in the GitHub repository.

Doc-Diff (Python Library) was originally published by Renien Joseph at Renien John Joseph on April 02, 2017.

ETL-Starter-Kit (Extract Transform Load)

2017-03-20T19:39:55+00:00

Fork me on GitHub

Lambda Architecture(LA) is a well-known data processing architecture designed to handle massive amount of data and it’s commonly known as Big Data. It contains both batch and stream processing methods.

When I started my journey in Data Science to process massive data came across an excellent book,

“Big Data Principles and best practices of scalable real-time data systems by Nathan Marz and James Warren.”

I recommend to read this book for all Big Data/Data Science developers.

In Big Data/Data Science project LA aims to satisfy the need to scale up and shrink indecently and also the system that is fault tolerant. Therefore, it is essential to get a correct structure for your project before starting over the implementation.

I spent almost couple of years building Lambda Architecture design for client specific projects. I realized in Big Data domain starter-kit projects are lacking when comparing to JavaScript world. So, I have published minimal skeleton project to process Big Data,

Introducing: ETL-Starter-Kit

Since the repository is to keep only the structure; different type of many sample jobs are not implemented. Based on your requirement be free to modify and implement different type of batch/streaming jobs (Spark, Hive, Pig etc)

ETL-Starter-Kit (Extract Transform Load) was originally published by Renien Joseph at Renien John Joseph on March 20, 2017.

Gradle build and Jar Hell

2017-03-09T00:00:00+00:00

Overview

One of the challenging task is to setup the ground level structure for a project and build scripts focusing a large growing project. I was spending almost a week incrementally building the product features and updating the build scripts for a data science project.

To make sure all my dependencies are available for my Hadoop MapReduce jobs I’m creating FAT JAR. Due to multiple dependencies in the project and Hadoop cluster I faced a Jar Hell problem. JAR Hell is an endearing term referring to the problems that arise from the characteristics of Java’s class loading mechanism. Simply, it is a dependency version conflict issue.

So I decided to move few libraries to use only at compilation time and during the Hadoop job execution it will get resolved from the HDP libraries. Since the provided scope was not supported in some Gradle version I implemented a workaround using the groovy script.

Create your own configuration:

configurations {
   providedCompile
}

and set it to be used with the compilation classpath:

sourceSets {
    main {
        compileClasspath += configurations. providedCompile
    }
}

After ‘providedCompile’ task is added modified my build script to use few libs during only compilation. Eg:

providedCompile([
        libs.pig
])

But unfortunately Gradle build failed for some reason with the following error message:

You can’t change configuration ‘providedCompile’ because it is already resolved!

The issue was in groovy : Gradle Community Forums

Since I’m using Gradle 2 version I faced this issue. Because they have updated Gradle 2 with Groovy 2.3, which is no longer supported to add single element to a collection. Instead of adding a single element we need to add it as a collection passing the single element.

sourceSets {
    main {
        compileClasspath += [configurations. providedCompile]
    }
}

After fixing the issues in build script, I was able to build and execute the job successfully.

Gradle build and Jar Hell was originally published by Renien Joseph at Renien John Joseph on March 09, 2017.

Gradle: Invalid method Code length 66047 in class file jobs_51dck7bh02aiqhihu32e2hl475

2017-02-27T00:00:00+00:00

Overview

Recently, I came across the following issue (Figure 1) in my build script when I started to add more custom tasks into my build script. The current project code base is huge and lot of custom build process needs to be implemented to automate the project build.

Figure 1 - Invalid Method Length

After reading so many articles got to know that these issues can happen due to the method length in our java code. Basically, JVM has a limit on method size in bytes, but during byte code transformation which is required to make possible mocking static, final size of the method could exceed this limit. But even though when I shorten my groovy custom tasks it was throwing the same error.

I don’t know for some reason, I thought to modularize the build script and import the different sub.gradle files to the main gradle file so that number of lines of code will be reduced. At last I was able to build successfully.

Basically, while transforming the code to Java byte code due to the big lengthy build script it exceeded the JVM method size limit.

Gradle: Invalid method Code length 66047 in class file jobs_51dck7bh02aiqhihu32e2hl475 was originally published by Renien Joseph at Renien John Joseph on February 27, 2017.

Interest Towards Data Science and Lessons Learnt

2016-12-30T19:39:55+00:00

It’s been a long time I shared some valuable information. Because half-year in 2016, I was spending long time assertively building myself to fill my gaps to work on some very interesting projects.

Being in IT field there are numerous areas and opportunities to work. Having almost one-time experience in most of the areas nothing interested me like working with data. Actually, the interest was started in mid 2015 and I wrote a first article on “Learn about Big Data”. After reading so many blogs and articles I ended in one image. The Figure 1 shows the skill sets that you need to work on Data Science field. I fear a lot looking at Big Data and Data Science.

Figure 1: Skills for a Data Science (Image from a another source)

“Of Course who will not get scared when they look at this Venn Diagram :P”

I always believe technology, should give us superpowers and the data is the soul of everything, so thought of let’s give it a try.

It is still not an easy journey, every day we learn new things from everywhere. But when I look back from mid 2016 up to now it surprises me that I didn’t know much stuff back in that time. It’s all matter of identifying your weaknesses and try hard to overcome it. The moment you can identify your weaknesses believe me you’re going in the correct path. Many people will have their own different goals. But I like to put it in a different way,

Fix your goals as what you really like to do and “makes you happy”

In year 2016, spoke to lot of professionals and students about goals and being happy in what they are doing. Most of them are still confused about it. I believe, it is a good sign because I always have it in my self too.

The confusion helps to open up the untouched energy in your brain and before taking a decision you would consider lots of factors. But then decisions can go wrong, it is up to you to overcome and work towards the next steps.

The current situation is different. The growth of technology has brought everything to our door steps. If you think to learn new stuffs in your field, there are plenty and more resources available out there in internet and it is in our hand to find all our diamonds.

Later if you realise what you really actually want to do in your life, My dear friends just don’t think you’re too late. Take that day as your start date and work hard on them and results will follow you.

“Life is about growing and improving and getting better”

Happy New Year 2017 ..

Interest Towards Data Science and Lessons Learnt was originally published by Renien Joseph at Renien John Joseph on December 30, 2016.

Exclude particular fields from SELECT queries in Hive

2016-06-28T00:00:00+00:00

Hive is a high level language to analyse large volumes of data. The easiest way to select specific columns in Hive query is by specifying the column name in the select statement.

SELECT col1, col3, col4 .... 
FROM Table1;

But imagine your table contains many columns (i.e : more than 100 columns) and you need to only exclude a few columns in the select statement. Therefore, Hive query should be able to select all the columns excluding the defined columns in the query. To achieve it you need to follow these steps.

In ‘hive-site.xml’ add the following configuration,

hive.support.quoted.identifiers=none

and execute the query as

SELECT `(extract_date)?+.+` FROM <TABLE_NAME>;

with appropriate column and table name.

i.e Think you need to exclude only ‘transaction_date’ column in a select statement from a ‘cart’ table. Then the query will be,

SELECT `( transaction_date)?+.+`
FROM cart;

Exclude particular fields from SELECT queries in Hive was originally published by Renien Joseph at Renien John Joseph on June 28, 2016.

Scoping Rules of R

2016-04-06T19:39:55+00:00

When we use some symbol to implement our logics, how does R know which value to assign?

R functions are treated much like any other R objects.

Assume we are running the following code,

> c <- 100 ## Assign a value to 'c'
> (c+1) ## Increment 
[1] 101
> 
> vec <- c(1:10) ## Create a vector
> vec
 [1]  1  2  3  4  5  6  7  8  9 10

If we observe properly, we can still use c() to create vectors. Now the question is how does R know to use which c symbol to use and when? It’s because,

R has separate namespace for functions and non-functions.

Lets try to understand it clearly. When R tried to ‘bind/connect’ a value to a symbol (in our case c(), it search for the corresponding symbol in an order.

Search the global environment (workspace) for a symbol name matching the request.
Search the namespaces of each of the packages on the search list.

So the search list will have the follow packages (packages which we have loaded) including “.GlobalEnv” as a first item in the search list and the “base” is always at the very end.

> search()
 [1] ".GlobalEnv"        "tools:rstudio"     "package:stats"    
 [4] "package:graphics"  "package:grDevices" "package:utils"    
 [7] "package:datasets"  "package:methods"   "Autoloads"        
[10] "package:base"

“.GlobalEnv” is just our workspace. If there is a symbol matching it will get it from workspace. If nothing found it will search from the rest of the namespace in each of the packages.

Rules of scoping

R uses scoping rules called Lexical scoping (static scoping).

It will determine the value associated with free variable function.

> fun <- function(x,y){
+   x^2 + y / z
+ }
> fun(2,3)

In the above function we have two arguments and they are x, y. But inside the function body we can find another symbol ‘z’. In this case z is called free variable.

According to scoping rules in R it first searches in the environment where the function was defined. An environment is collection of symbols and values. Environments have patents.

> parent.env(globalenv())
<environment: 0x10390d6e0>
attr(,"name")
[1] "tools:rstudio"

Since we defined fun function in global environment, R will look for z in that scope (environment).

> environment(fun) ## to check the scope
<environment: R_GlobalEnv>

These rule are matters because we can define some complex logics and function.

> y <- 10 ## y Symbol
> 
> f1 <- function(x) {
+   y <- 2 ## Binding value to 'y' symbol
+   y^2 + f2(x) ## 'f2' function is used
+ }
> 
> f2 <- function(x) {
+   x * y
+ }
> 
> f1(2) ## Execute 'f1' function 
[1] 24
> f2(2) ## Execute 'f2' fucntion
[1] 20

Blog Series

Scoping Rules of R was originally published by Renien Joseph at Renien John Joseph on April 06, 2016.

Functions in R

2016-04-05T19:39:55+00:00

We need functions to avoid the repetitive same few lines of code.

R functions are treated much like any other R objects.

Like in JavaScript language the functions can be passed as arguments and it can be nested too. The functions are defined using the function() directive.

> func <- function(){
+   ## Body of the function [empty]
+ }
> class(func) ## Identify the class type
[1] "function"
> func() ## Execute the function
NULL

function arguments are another important option for a function.

> func <- function(num){ ## pass the num argument
+   for (c in num) {
+     cat("function reduces the # of code\n")
+   }
+ }
> func(1:3) ## Execute the function
function reduces the # of code
function reduces the # of code
function reduces the # of code

We can modify our code a bit by setting a default value for the arguments. Therefore we can call the function without passing any argument value to the interface.

> func() ## Execute the function
Error in func() : argument "num" is missing, with no default

> func <- function(num=1:2){ ## pass the num arguments
+   for (c in num) {
+     cat("function reduces the # of code\n")
+   }
+ }
> func() ## Execute the function
function reduces the # of code
function reduces the # of code

Lazy Evaluation

Function arguments are evaluated lazily. The below code example clearly explains it.

> add <- function(a, b){ ## Evaluated lazily
+   cat(a) 
+ }
> add(3) ## Passing one argument value
3

When both arguments are used in the body then R will check for the next argument too.

> add <- function(a, b){ ## Evaluated lazily
+   cat(a + b) ## Both arguments are used 
+ }
> add(3) ## Passing one argument value
Error in cat(a + b) : argument "b" is missing, with no default

Arguments with ‘…’

In R we can find a special argument … , which indicate a number of arguments that are usually passed on to other functions. The … argument is often used when extending another function.

> plot ## Execute to see the arguments 
function (x, y, ...) 
UseMethod("plot")
<bytecode: 0x10195dad0>
<environment: namespace:graphics>
> 
> myplot <- function(x, y, type = 'l', ...){
+   plot(x, y, type = type, ...) ## pass '...' to plot function
+ }
> 
> ## Create the x, y points to plot
> x=seq(0,2*pi,0.01)
> y=sin(x)
> 
> # Draw the plot graph [Figure 1]
> myplot(x,y)

Figure 1: Sample Plot Graph

Blog Series

Functions in R was originally published by Renien Joseph at Renien John Joseph on April 05, 2016.