There are no two ways about it, Hadoop development iterations are slow. Traditional programmers have always had the benefit of re-compiling their app, running it, and seeing the results within seconds. They have near instant validation that what they’re building is actually working. When you’re working with Hadoop, dealing with gigabytes of data, your development iteration time is more like hours.
Inspired by the amazing real-time feedback experience of Light Table, we’ve built Mortar Watchtower to bring back that almost instant iteration cycle developers are used to. Not only that, Watchtower also helps surface the semantics of your Pig scripts, to give you insight into how your scripts are working, not just that they are working.
Watchtower is a daemon that sits in the background, continuously flowing a sample of your data through your script while your work. It captures what your data looks like, and shows how it mutates at each step, directly inline with your script.
Better Error Handling
Watchtower helps flush out the errors in your script. A common problem for Pig developers is making sure your UDFs don’t have errors. When we’re talking about a job that can take eight hours to run, finding an error in your UDF that runs at the end of your execution can be painful. Since Watchtower is constantly sending data through your entire script, errors are surfaced and displayed instantly. No longer will you be surprised when at the eleventh hour your Hadoop job fails.
Watchtower Sees Everything
Watchtower is always looking for changes in your Mortar Project. If you edit anything that alters the functionality of your script (including data samples, UDFs, or parameters), Watchtower will know, start recompiling the results, and almost instantly feed the results back to you. All you have to do is save.
Getting Started is Easy
Best of all Watchtower is incredibly simple to get started with. If you’re already setup with Mortar Development Framework then all you have to do is:
$ mortar plugins:install https://github.com/mortardata/watchtower.git
Then start watching a script. For example:
$ git clone https://github.com/mortardata/mortar-examples.git $ cd mortar-examples $ mortar watch pigscripts/coffee_tweets.pig -f params/coffee_tweets/local.large.params
We hope you love getting real-time feedback on your Pig scripts. Enjoy!