Challenges:
- – Microsoft SQL Server platform was not capable enough to handle customers with huge data loads from various data sources.
- – Scalability hindered due to technical platforms not able to keep up with business scale.
- – High downtime YoY due to unreliable systems.
Solution:
- – Building a new platform on Snowflake for high scalability on cloud.
- – Design and build automated pipeline for smooth transition and migration.
- – Run all services on Kubernetes pods, which facilitates scalability.
- – Reduce Data Doctor (DDS) scope was to just a reporting tool for Data Quality issues.
- – Migrating the legacy Stratosphere platform to Snowflake, use data cleanup andvalidation to ensure consistent, structured data.
Impact:
- – Created and maintained Cost tracking Dashboard and added new entities.
- – Automated pipeline for Data migration.
- – Assisting on Snowflake best practises, performance tuning and issue resolution.
- – RBAC development and creating roles, warehouses and users with appropriateprivileges.
- – All services became serverless and multi tenant, now all clients per region could accessthe same services, thus doing away with Virtual Machine based deployments.
- – Extract Load (EL) service SQL loader service was launched for historical load for SupplyChain Planning & Optimization (SCPO)