The software industry is gradually evolving and everyone likes to have quick delivery similar to fast food. In order to stay competitive, organizations are engaged in developing highly scalable products and services.
I recently prepared a quick fast food and full meals with different dishes. Actually, a full meal with different dishes stays in our memory for a long time compared with quick fast food. Similarly, a well-designed product always helps the company to scale up easily by attracting a lot of customers and also it gives us the flexibility to customize based on their choices/needs. If you look at below pics, potato wedges vs banana leaf meals; who will not like a full meals with different flavors 😀.
From the inception data has always been a critical component. Data platform is gradually becoming/evolving to support the mainstream system to give the power back to business. Being in a SaaS hyper growth company is a fun ride. There were multiple key challenges to build a Data Platform to serve the SaaS business. It’s like sailing in the rough sea 🙂.
There were lot of questions to be answered to achieve this mission,
- How to manage multiple flavors of data warehouse?
- How to maintain multiple data platforms/multiple code bases?
- How about the data accuracy?
- How about managing PII data and data governance?
- How do we maintain the systems? Do we need a big army?
- How fast can we bring up a data platform?
- How can we build a platform to empower data science to work?
The answer relies on two strategies.
Strategy 1: Build a team with a culture of engineering excellence
Culture is not something you build overnight but it’s created by everyone and everything that happens on your team. It’s actually a journey with lots of ups and downs. But if you have a clear vision that can be explained to your team and show them by practicing in your day-to-day activities to reach the goals. After a while the people around you will understand the importance of it and gradually their actions influence others as well.
Motto: “Should be cathedral builders, not bricklayers”
Strategy 2: Build data platform in a box
Motto: “Data platform in one button click with 360 view on quality”
Data is scattered all around the platform and around us. The following are the key modern principles of data platform.
- View data as shared asset
- Provide the right interfaces to the users to consume data
- Ensure the security of data and access controls
- Establish a common vocabulary
- Curate the data
- Eliminate data copies and movements
- Data quality
It’s essential to follow these above key modern principles to build a stable data platform.
It’s important first to build a stable and reliable data platform then bring DevOps culture to the team to bundle the platform into a BOX.
During ‘data platform in a box’ journey it’s actually demands to have skills not only on DevOps side but the knowledge about the data tools and architectures/solutions. So the only option is to upskill the data team on DevOps skills to MAKE IT HAPPEN.
Actually without achieving Strategy 1; we will not be able to achieve Strategy 2.
If I rewind back for demi-decade period, the love towards big data and data science is to the highest level and I still didn’t lose interest in it. Because it’s evolving and keeps me really engaged with very complex systems.
But during the journey on bringing a ‘data platform in a box’ just expanded my interest in DevOps and it will be part of my journey forever. Understanding devops to control big data and data science playground just drives me crazy.
This year the journey taught me so many things. Not only on the tech side, but also life lessons. I’m sorry, I was unable to spend time sharing some ground level work on how to bring the data platform in a box. I was really busy building the data platform. But for sure, I will spend some time next year sharing some interesting topics on ‘data platform with devops’.
I will share sample templates and code to kick start the journey with data platform tools and devops in my github. Keep following and follow my github profile to get notified on all the new projects.
Stay Tuned!
I can’t wait to crush 2023 together… cheers! 🍺