Data delivery and orchestration at Foursquare scale - good and bad practices, and lessons learned
Track
Best Practicies & Success Stories
Date and time
Monday, 16. October 2023., 17:00
Room
Hall C
Duration
45'
Thousands of ETL delivery jobs. Hundreds of clusters.Computational infrastructure hitting AWS region limits, legacy technology meeting bleeding edge improvements, throttled AWS API calls, tight SLAs, ... Squeezing the last drop from Airflow, Luigi and other orchestration tools to meet the demands.On top of that, three acquisitions, three different tech stacks being merged into one...This is a talk about bad choices made, good decisions implemented, but above all, about what has been learned along the way.
Lecture details
About speaker
Dusan Zamurovic, author
Now data engineering specialist, systems architect and engineering manager, but previously software consultant and body to be leased.Enjoying working with all things cloud while building all kinds of pipelines and heavy lifting of data, getting it structured, preprocessed and ready for people smarter than me to analyse it... Usually using Python, Scala or Java.Living the "customers first" principle - leading an amazing group of engineers responsible for company's orchestration tools. We deliver the platform almost all other Foursquare engineers are relying on.
Like us on Facebook
Follow us on Twitter
Keep up to date with all the news on Twitter! Follow @JavaCroPlatinasti Pokrovitelj
Zlatni pokrovitelj
Srebreni pokrovitelji
Medijski pokrovitelji
