Data delivery and orchestration at Foursquare scale - good and bad practices, and lessons learned
Track
Best Practicies & Success Stories
Datum i vrijeme
ponedjeljak, 16. listopad 2023., 17:00
Dvorana
Dvorana C
Trajanje
45'
Thousands of ETL delivery jobs. Hundreds of clusters.Computational infrastructure hitting AWS region limits, legacy technology meeting bleeding edge improvements, throttled AWS API calls, tight SLAs, ... Squeezing the last drop from Airflow, Luigi and other orchestration tools to meet the demands.On top of that, three acquisitions, three different tech stacks being merged into one...This is a talk about bad choices made, good decisions implemented, but above all, about what has been learned along the way.
Detalji o predavanju
O predavaču
Dusan Zamurovic, autor
Now data engineering specialist, systems architect and engineering manager, but previously software consultant and body to be leased.Enjoying working with all things cloud while building all kinds of pipelines and heavy lifting of data, getting it structured, preprocessed and ready for people smarter than me to analyse it... Usually using Python, Scala or Java.Living the "customers first" principle - leading an amazing group of engineers responsible for company's orchestration tools. We deliver the platform almost all other Foursquare engineers are relying on.
Prati nas na Facebooku
Prati nas na Twitteru
Sve novosti i zanimljivosti vezane uz JavaCro objavljujemo i na Twitteru! Follow @JavaCroPlatinasti Pokrovitelj
Zlatni pokrovitelj
Srebreni pokrovitelji
Medijski pokrovitelji
