Del is currently working at pinterest.com ads-infra team. Shoot me an email if you want to chat about numbling-exciting geeky things
First, need to learn to monitor the status of your spark jobs. There is an excellent guide of spark UI from Databricks