Mortar Data Blog

Hadoop Weekly - May 20, 2013

Mon, 20 May 2013 09:10:07 -0400

Scott Haylon

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer on Foursquare’s big data team, where he focuses on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

Both Apache Hadoop and Apache Hive crowned new releases this week, and there are a number of interesting technical articles covering YARN, NFS access to HDFS, and Apache Flume. With so much happening so quickly in the Hadoop-ecosystem, it can be a difficult to keep up — so please let me know if I missed anything, and I’ll include it next week.

Technical
Apache HDFS is getting support for the Network FileSystem (NFS) protocol. This an exciting new feature, and one of the authors working on the feature details the what, why, how, and when of Hadoop’s NFS support, which is being developed in trunk.
http://hortonworks.com/blog/simplifying-data-management-nfs-access-to-hdfs/

Cloudera’s blog has the second in their “meet the founders” series. This post features Roman Shaposhnik who founded and works on Apache BigTop. Aside from having one of the best names of projects in the Hadoop ecosystem, BigTop is beginning to have a lot of influence in making sure that components in the stack are compatible with one another when released.
http://blog.cloudera.com/blog/2013/05/meet-the-project-founder-roman-shaposhnik/

If you’ve ever tried to put together a patch for Hadoop, it can be very intimidating (and a slow process) just to configure your development environment. This post provides an overview of setting up Eclipse for developing Hadoop — covering all the major versions and flavors under development.
http://blog.cloudera.com/blog/2013/05/how-to-configure-eclipse-for-hadoop-contributions/

You might remember the “stinger initiative” which was introduced a while back by Hortonworks with the goal of making Hive 100x faster. With the release of Hive 0.11 (more below), they summarize some of the work that’s already been done towards this goal, as well as some of the new features in Hive 0.11 (such as RANK and other analytical functions).
http://hortonworks.com/blog/apache-hive-0-11-stinger-phase-1-delivered/

The Manning Early Access Program (MEAP) is now available for the new book, “Pig in Action”, by M. Tim Jones. With MEAP, you pre-order the book but get access to the content as the author is writing and uploading it.
http://www.manning.com/tjones/

Apache Flume is a system for transferring data from application servers or other event-generators to HDFS or HBase. In this post, the author gives an overview of the Flume architecture — both at the component and system scale.
http://www.drdobbs.com/database/acquiring-big-data-using-apache-flume/240155029

The Natural Language Toolkit (NLTK) is a set of Python libraries for natural language processing. This post describes how to tie them to Hadoop MapReduce for parallel processing using Hadoop Streaming.
http://datacommunitydc.org/blog/2013/05/nltk-hadoop/

Arun C. Murthy, one of the leads on the Apache Hadoop YARN project, gives an update on the progress of the project plus background on what YARN can enable. In particular, YARN turns Hadoop into a multi-application system, allowing more than just MapReduce to run on Hadoop. Arun highlights that we’ll be able to run SQL in Hadoop rather than SQL on Hadoop (via MapReduce).
http://hortonworks.com/blog/moving-hadoop-beyond-batch-with-apache-yarn/

Hortonworks has compiled a list of links for the Hadoop on Windows developer. In particular, the .NET SDK, the Microsoft Hive ODBC driver, and HDInsight’s Preview (Hadoop on Azure).
http://hortonworks.com/blog/hadoop-sdk-and-tutorials-for-microsoft-net-developers/

Platfora’s product provides a combines a UI and low-latency data store to do interactive analysis on data stored in S3 or HDFS. If the data isn’t already in Platfora’s store, the system can generate a MapReduce job to load the data. This article gives a good overview of how all of the technology components in the Platfora system work together.
http://cloudcomputing.sys-con.com/node/2663726

HUE provides a UI for interacting with Hadoop, Hive, Pig, and more. This post describes how to leverage HUE’s python API to execute queries against Hive (via HiveServer2) or Impala (which must implement the same Thrift API).
http://gethue.tumblr.com/post/49882746559/executing-hive-or-impala-queries-with-python

Storm is sometimes called the real-time version of MapReduce. With a lot of interest in getting Storm running on YARN, now’s a good time to get familiar with the system. The inaugural London Storm Meetup featured an overview of Storm as well as a discussion of the presenter’s use-case. This post has a summary of the event, including links out to the presentation and code examples.
http://partners.peerindex.com/london-storm-meetup-big-data/

News
Contexti and MapR have joined forces to provide training, consulting, and professional services for MapR’s distribution in Asia-Pacific.
http://www.businesswire.com/news/home/20130512005046/en/Contexti-Expands-Hadoop-NoSQL-Portfolio-MapR

Drawn to Scale, a SQL-on-Hadoop vendor, has announced that they’re closing their doors. They had an interesting system, which is built to be performant on many types of SQL operations, and they even had a compatibility layer for MongoDB. It should be interesting to see what happens to that team and their technology.
http://www.roadtofailure.com/?p=11

Concurrent and MapR announced that Concurrent’s Cascading framework is now certified to run on MapR’s distribution.
http://www.concurrentinc.com/posts/2013/05/15/concurrent-inc-partners-with-mapr-technologies-to-drive-mass-enterprise-hadoop-adoption/

Releases
Hadoop 1.2.0 featuring DistCP v2 backport, web services for the JobTracker, the offline image viewer, and a bunch of other enhancements.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201305.mbox/%3CCA%2Bz3%2B9Er-fx6XwZ%3DrefL1aa70qSKKREhBc3Rz0XP3aSOhaVh6w%40mail.gmail.com%3E

WibiData announced Albacore/BentoBox v1.0.4. This version has some new features, including a whole new component — KijiREST, which provides a REST interface to KijiSchema.
http://www.kiji.org/announcing-the-albacore-bentobox-v1.0.4/

Hive 0.11 was released with over 350 Jira issues closed. This is the first release since HCatalog was integrated as a subproject of Hive, and it has a bunch of new features such as HiveServer2, ORCFile, and analytics functions.
http://mail-archives.apache.org/mod_mbox/hive-user/201305.mbox/%3CCAHfHakGC9pkVV5V_oZBB4kzB_nQ2RRibZ9cqSGPMTp2Qe3%2BABw%40mail.gmail.com%3E

Talend’s Open Studio was updated to version 5.3 a few weeks ago. This post has a quick overview of the new features, which include a new integration with Apache Pig, as well as support for Amazon’s Elastic MapReduce and RedShift.
http://www.h-online.com/open/news/item/Talend-5-3-focused-on-Hadoop-usability-1864844.html

Events (curated by Mortar Data)
Monday, May 20
MySQL to Cassandra: Big Data, High Scale, Data Migration… Oh My! (New York, NY)
http://www.meetup.com/mysqlnyc/events/114879742/

Monday, May 20
Automating the Hadoop Stack (Los Angeles, CA)
http://www.meetup.com/LA-HUG/events/117428702/

Tuesday, May 21
Data & Drinks - Member Networking Meetup (New York, NY)
http://www.meetup.com/Analytics-and-Data-in-Financial-Services/events/112520472/

Tuesday, May 21
Recommendation Engines & Accumulo (Denver, CO)
http://www.meetup.com/Data-Science-Business-Analytics/events/116790372/

Tuesday, May 21
Thoughts on machine learning (New York, NY)
http://www.meetup.com/NYC-Machine-Learning/events/119204802/

Tuesday, May 21
How we use Scala on Hadoop @ eBay (New York, NY)
http://www.meetup.com/ny-scala/events/113168872/

Wednesday, May 22
Cloudera Impala: An Open Source Real-Time Query Engine for Apache Hadoop (Boulder, CO)
http://www.meetup.com/Boulder-Denver-Big-Data/events/114501572/

Wednesday, May 22
Big Data, NoSQL, Now What? (New York, NY)
http://www.meetup.com/mysqlnyc/events/114883642/

Saturday, May 25
Big Data Science Meetup Event (Fremont, CA)
http://www.meetup.com/Big-Data-Science/events/71084702/

Hadoop Weekly - May 13, 2013

Mon, 13 May 2013 06:58:11 -0400

Scott Haylon

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer on Foursquare’s big data team, where he focuses on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

This week’s newsletter is a little lighter than normal in technical news (some fascinating articles, though!), but there are a quite a few interesting releases and upcoming events. Hope you enjoy, and please let me know if you find anything that I missed! Also, thanks to everyone that has been spreading the word about this newsletter — the number of new subscribers each week has been really encouraging.

Technical
LinkedIn has open-sourced a number of big data projects built on or to coexist with Hadoop. In celebration of LinkedIn’s 10th anniversary, this post covers 10 of those projects (such as Voldemort and DataFu), including a brief overview of each.
http://www.hadoopsphere.com/2013/05/hadoops-10-in-linkedins-10.html

Following the release of version 0.2.0 of the Cloudera Development Kit, the Cloudera blog has a new post with an overview of the project, an FAQ, and a list of future plans. They plan on having monthly releases and focusing on documentation in addition to software libraries. It should be an interesting project to watch.
http://blog.cloudera.com/blog/2013/05/cloudera-development-kit-cdk/

An overview of eBay’s data warehouse, which ingests as much as 100TB/day and stores over 90PB. To power internal analytics, they use a combination of Hadoop, Teradata, and a custom built system as data stores plus front-end tools Tableau, Excel, Microstrategy (and more).
http://www.itnews.com.au/News/342615,inside-ebay8217s-90pb-data-warehouse.aspx

RCFile is a columnar format that’s part of the Hive project. This post describes the motivation for RCFile as well as the benefits. In a follow-up post, the author will talk about the successor to RCFile — ORCFile, which has similar features to the Parquet format.
http://www.bigdatarepublic.com/author.asp?section_id=2840&doc_id=262756

An honest review of a new Hadoop book, the “Hadoop Beginner’s Guide” talks about both the good and the bad in the book. Overall, the review is positive but notes that there are a few technical issues that could be improved.
http://architects.dzone.com/articles/review-hadoop-beginners-guide

Russell Jurney, the author of Agile Data, has posted slides to accompany his book. The slides cover a number of principles for Agile big data as well as a bunch of example code covering everything from data analysis with Pig to visualization with Bootstrap and D3.
http://www.slideshare.net/rjurney/agile-analytics-applications-on-hadoop-20839095

When an HBase RegionServer fails, it can take a few seconds or minutes for the regions owned by that RegionServer to recover. Reducing this time, known as the Mean Time to Recover (MTTR), has been the subject of a lot of work on both the HBase and HDFS projects. This post has a good overview of the technical challenges and their solutions.
http://hortonworks.com/blog/introduction-to-hbase-mean-time-to-recover-mttr/

Releases
Snakebite is a new python project that makes use of protocol buffers to talk to HDFS without going through the JVM. In addition to an API, it supplies a command line utility with similar functionality to “hadoop fs” (without the JVM startup overhead, it’s a lot faster) and a script to startup a mini HDFS cluster, which it uses for testing.
http://labs.spotify.com/2013/05/07/snakebite/

Azkaban 2.1 was released. This is the first point release to the Hadoop workflow management software since the version 2 rewrite. It has a bunch of new features, like JMX support, auto-retries, and SLA aware notifications.
https://groups.google.com/d/msg/azkaban-dev/WN5LWbqtsxE/PhUmmcO2lZIJ

DataStax has made their ODBC driver for Hadoop/Hive free and has announced a new ODBC driver for Cassandra, currently in beta.
http://www.datastax.com/dev/blog/free-odbc-drivers-for-cassandra-and-hadoop-now-available

Apache Giraph, the computation framework for bulk synchronous parallel programming (often used for network graph algorithms), had its version 1.0 release, the first since graduating from the Apache Incubator. This release has a bunch of features, including support for running within YARN, support for accessing Hive tables, and improved performance and memory efficiency.
http://mail-archives.apache.org/mod_mbox/giraph-user/201305.mbox/%3C51888B3D.5040100@apache.org%3E

Apache Curator (incubating), is a set of Java libraries for Apache Zookeeper. The project was originally started at, and open sourced by, Netflix. This week they had their 2.0.0-incubating release, the first since joining the Apache incubator.
http://mail-archives.apache.org/mod_mbox/incubator-curator-user/201305.mbox/%3CEDA26D30-171C-481F-B740-4284A1C3B417%40jordanzimmerman.com%3E

GigaOm notes that IBM is also an entrant into the SQL-on-Hadoop field, with a preview open to a limited number of participants.
http://gigaom.com/2013/05/06/look-ibm-is-doing-sql-on-hadoop-too/

Events (curated by Mortar Data)
Monday, May 13
Hackathon: Develop/MapReduce Your Dream Predictive Analytics/BigData App (Cambridge, MA)
http://www.meetup.com/Predictive-Analytics/events/109954442/

Tuesday, May 14
Storm, Brickhouse, ElasticSearch (San Francisco, CA)
http://www.meetup.com/Real-time-Big-Data/events/113867452/

Tuesday, May 14
Big Data is Not About The Data! (Chestnut Hill, MA)
http://www.meetup.com/intelligence/events/110894352/

Tuesday, May 14
Smarter Big Data Integration for Hadoop (London, UK)
http://www.meetup.com/hadoop-users-group-uk/events/115450432/

Tuesday, May 14
Introduction to Distributed Search using Cassandra with Solr and Lightning Talks (Columbia, MD)
http://www.meetup.com/Hadoop-DC/events/117691352/

Wednesday, May 15
R and Hadoop (Free Webinar) (Internet)
http://www.meetup.com/OC-HUG/events/113336802/

Wednesday, May 15
37th Bay Area Hadoop User Group (HUG) Monthly Meetup (Sunnyvale, CA)
http://www.meetup.com/hadoop/events/69997352/

Thursday, May 16
Hackathon for Hadoop Administrators (Hyderabad, India)
http://www.meetup.com/hyderabad-hadoop/events/118712352/

Thursday, May 16
STORM for streaming analytics at scale (London, UK)
http://www.meetup.com/storm-london/events/118169452/

How to get Hilary Mason to build your recommender for free

Wed, 08 May 2013 10:29:00 -0400

Scott Haylon

If you want Hilary Mason, Drew Conway, or Max Shron to build your recommender for free, enter your email address here.

As a platform for working with data, we’ve seen users tackle lots of interesting use-cases: log analysis, natural language processing, pattern detection, and many more.

However, perhaps no use-case is in greater demand than recommender systems. If you have more “inventory” than your users can easily find (whether it’s news, jobs, videos, restaurants, vacations, recipes, apps, etc.), a great recommender is crucial to driving engagement.

The problem is that recommender systems are really hard to implement, so most companies either don’t have one or aren’t happy with what they have.

What makes recommenders so tough?

It’s painful to Obtain, Scrub, and Explore. These systems often require a lot of data, and the “OSE” steps from “A Taxonomy of Data Science” can take 75% of the effort in building a recommender system.
Processing data at scale is difficult (at best). Those who have ever attempted setting up and maintaining a Hadoop cluster know how challenging it can be.
Data scientists are nearly impossible to find. Looking at every user and everything that could be recommended to them typically requires a very good data scientist. Unfortunately, there is a massive data scientist shortage, and the problem is only getting worse.

Although our users were finding Mortar hugely helpful for OSE and processing their data at scale, they still found they needed extra help to build the recommenders they wanted due to the scarcity of data science talent available to them.

To remedy this problem, we’re custom-building recommenders for 10 companies by partnering with three world-class data scientists:

Hilary Mason – Chief Scientist at Bitly
Drew Conway – Author of Machine Learning for Hackers, Scientist in Residence at IA Ventures and Co-founder of DataKind
Max Shron – Data strategy consultant (clients include Amazon.com, the Guardian, and Warby Parker), formerly lead data scientist at OkCupid

We’re going to select 10 companies to work with, building each of them a custom recommender system created with open technologies (Pig, Python, Java), for free. We are open sourcing the generic components and giving the custom pieces to their respective companies to keep.

Why would we do this?

We want to provide future users reusable, open source components that work on real problems, at scale. So while 10 companies will get custom-built solutions, all of our customers benefit by being able to more easily build recommenders on Mortar.

We’ve already accepted two companies to our program during our beta rollout, and now we’re looking for eight more. Want a free recommender custom-built for you? Enter your email address here.

Hadoop Weekly - May 6, 2013

Mon, 06 May 2013 07:00:00 -0400

Scott Haylon

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer on Foursquare’s big data team, where he focuses on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

There were two big and exciting releases this week from Hadoop vendors — Cloudera with Impala and MapR with M7. In addition, this week marks the 500th subscriber to Hadoop Weekly! Thanks everyone for subscribing, and please send anything my way that you think might make a good addition to this newsletter.

Technical
In the third and final part of his “Introduction to Hadoop” series, Tom White covers higher-level frameworks, anatomy of a Hadoop cluster, and data application pipelines. In terms of frameworks, he covers Pig, Hive, and Crunch (there’s a nice example of computing top-K with Crunch).
http://www.drdobbs.com/database/introduction-to-hadoop-real-world-hadoop/240153375/

HBase uses Hadoop’s metrics framework to expose metrics via JMX, HTTP, etc. This post covers the migration from the old metrics framework (“metrics1”) to the new framework (“metrics2”). Like most things that make use of a Hadoop API, there are extra complications because the hadoop-1 and hadoop-2 branches have diverged significantly.
https://blogs.apache.org/hbase/entry/migration_to_the_new_metrics

Cloudera Impala (low-latency SQL on HDFS) hit GA this week (more below), and some of the first benchmarks are starting to come out. Unsurprisingly, this presentation shows that snappy-compressed parquet files performed the best on their 11-node cluster, with average performance 12x that of Hive.
http://www.slideshare.net/sudabon/performance-evaluation-of-cloudera-impala-ga

An interesting presentation about some real-world experience using Hadoop and Vertica for offline analytics at AdTech. They have some useful information about using Avro with Flume and Pig and the interaction between Hadoop and Vertica.
http://prezi.com/6r0rsx6hluu1/mapreduce-in-action-large-scale-reporting-based-on-hadoop-and-vertica/

Andy Feng and Robert Evans recently presented on using Storm (a realtime distributed computation system) at Yahoo. In particular, their presentation covers collocating storm and hadoop on the same cluster using YARN.
http://www.slideshare.net/ydn/april-2013-hug

A case-study of a Hive query that started out taking over 2 hours but was optimized to take under 10 minutes. They used a bunch of tricks to speed it up, including enabling map-side joins and optimizing the number or reducers.
http://dennyglee.com/2013/04/26/optimizing-joins-running-on-hdinsight-hive-on-azure-at-gfs/

Ambari is an Apache Incubator project for provision, configuring, and monitoring Hadoop (and related services) clusters. Hortonworks posted a walkthrough for setting up your first Ambari cluster. Tools like this (another option is Cloudera Manager) are a must for managing a Hadoop cluster.
http://hortonworks.com/kb/get-started-setting-up-ambari/

News
Cloudera announced Impala 1.0 GA this week. It supports many of the same commands and datasets as Hive, but with much lower latency. Cloudera is reporting performance gains anywhere from 6x to 68x vs. Hive, and also better-than-linear scaling for multi-tenant queries. Given that Cloudera announced Impala less than a year ago, the fact that Impala has already reached version 1.0 is quite impressive and exciting (even if not all the features are there) — especially so since Impala is open-source.
http://blog.cloudera.com/blog/2013/05/cloudera-impala-1-0-its-here-its-real-its-already-the-standard-for-sql-on-hadoop/

MapR announced Mapr M7 Edition. This version touts features like 5-9 availability and other features targeting the NoSQL market. Main highlights include “no manual administrative tasks such as table merges or splits,” data snapshots, and data mirroring.
http://www.mapr.com/products/mapr-editions/m7-edition

SAS, the business analytics software company, has partnered with Cloudera to integrate their software with Hadoop.
http://www.marketwire.com/press-release/cloudera-announces-strategic-alliance-with-sas-1783697.htm

MapR and LucidWorks announced a partnership to include LucidWorks Search with MapR’s distributions. LucidWorks Search provides a number of advance features for Lucene/Solr. It’s a little unclear what the technical details of this integration look like, but it sounds like they are providing indexing of data stored in MapR’s FileSystem.
http://www.mapr.com/press-release/mapr-technologies-distributes-enterprise-grade-search-with-hadoop-platform

Hadoop Summit is coming up in less than 2 months in San Jose. The Hadoop Summit organizers, Hortonworks, have unveiled the full scheduled for the conference.
http://hortonworks.com/blog/hadoop-summit-schedule-is-now-available/

While many business analytics solutions focus on structured data accessible via SQL (over JDBC or ODBC), Precog is focusing on providing the same capabilities on unstructured data sitting in HDFS or elsewhere. They announced this week that their product is leaving beta.
http://gigaom.com/2013/05/01/precog-launches-with-a-plan-to-simplify-analytics-on-unstructured-data/

Releases
Cloudera Development Kit 0.2.0 was released with experimental support for the new Parquet columnar file format as well as integration with Hive/HCatalog for metadata management.
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/Q9teffdaJzY/rmtMpXoBMA4J

Amazon Elastic MapReduce now has support for S3’s server-side encryption. This includes S3DistCp, the EMR-optimized distcp implementation.
http://aws.amazon.com/about-aws/whats-new/2013/05/01/amazon-elastic-mapreduce-now-supports-S3-server-side-encryption/

Events (curated by Mortar Data)
Monday, May 6
Deep Dive into Cloudera Impala! (New York, NY)
http://www.meetup.com/Hadoop-NYC/

Tuesday, May 7
NoSQL & Hadoop with Couchbase server (Tel Aviv, Israel)
http://www.meetup.com/Big-Data-Israel/events/115467802/

Wednesday, May 8
San Francisco Hadoop Meetup (San Francisco, CA)
http://www.meetup.com/hadoopsf/events/114514922/

Thursday, May 9
Hadoop At Spotify (Warsaw, Poland)
http://www.meetup.com/warsaw-hug/events/111492272/

Thursday, May 9
Analyzing Twitter: An End-to-End Data Pipeline (Baltimore, MD)
http://www.meetup.com/Data-Science-MD/events/111081282/

Hadoop Weekly - April 29, 2013

Mon, 29 Apr 2013 06:53:00 -0400

Scott Haylon

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer on Foursquare’s big data team, where he focuses on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

The last full week of April was pretty busy for the Hadoop ecosystem — two core projects (Hadoop and HBase) saw releases, there was also some exciting funding news (congrats to Qubole!), and there were plenty of interesting technical articles.

Technical
The naming of components in Hadoop-related projects have often caused confusion (e.g. HDFS’ secondary namenode). Apache HBase is no exception — the HMaster is often misunderstood, because unlike its name suggests, not all writes go through the HMaster. This article elaborates on the role of the HMaster in HBase.
https://blogs.apache.org/hbase/entry/hbase_who_needs_a_master

Tachyon is the in-memory distributed file system from Berkeley’s AMPLab that recently had its initial release. This article provides more details about the system, including how it might be good for MapReduce jobs and fit into the Hadoop ecosystem.
http://strata.oreilly.com/2013/04/tachyon-open-source-distributed-fault-tolerant-in-memory-file-system.html

“Meet the Project Founder” is a new blog series from Cloudera. Their first story features Doug Cutting, the founder of Hadoop and Cloudera’s Chief Architect. Doug is incredibly prolific in open-source — he’s started the Apache Lucene, Nutch, Hadoop, and Avro projects.
http://blog.cloudera.com/blog/2013/04/meet-the-project-founder-doug-cutting-first-in-a-series/

Gravity migrated from MySQL to HBase as their primary data store. In addition to talking about their use-case of both online and batch processing via MapReduce, this article speaks to the recent improvements in the ease of deployment of HBase and the Hadoop stack, and how it’s changing the data storage landscape.
http://gigaom.com/2013/04/22/how-hbase-converted-myspaces-mysql-champion-and-is-driving-hadoop-mainstream/

Apache MRQL (incubating) is another low-latency analytics solution on HDFS solution. Unlike other systems, it’s SQL-like but not SQL and can take advantage of Hama’s Bulk Synchronous Parallel (BSP). These differences make it possible to do iterative processing, e.g. computing k-means (of which there is an example in this post).
http://www.hadoopsphere.com/2013/04/mrql-sql-on-hadoop-miracle.html

In the second part in his series on Dr. Dobbs, Tom White gives a thorough walkthrough of writing your first MapReduce job. He covers the classic “hello world” of MapReduce — word count.
http://www.drdobbs.com/database/hadoop-writing-and-running-your-first-pr/240153197/

Since entering the Hadoop-game, VMWare has been improving Hadoop on virtualized hardware. This article covers 7 myths related to Hadoop — some related to virtualization (and probably controversial) and more generally applicable.
http://blogs.vmware.com/vfabric/2013/04/myths-about-running-hadoop-in-a-virtualized-environment.html

It can be difficult to keep up with all the SQL-on-Hadoop solutions (it seems like there is a new one each week!). This article covers four of them — Impala, Hadapt, Hawq, and Berkely Data Analytics Suite (BDAS) — including the trade-offs you make when selecting one or the other (and importantly, the maturity of the product).
http://www.openbi.com/content/sql-hadoop

This article covers converting the Hortonworks Sandbox virtual machine image to a Rackspace instance. It’s a pretty interesting idea, and the process is appears to be quite easy.
http://devops.rackspace.com/getting-started-with-hadoop-using-hortonworks-sandbox.html

Based upon a Dell Whitepaper, Hortonworks has highlighted six important hardware decisions for designing a Hadoop cluster — from the operating system to storage to network.
http://hortonworks.com/blog/6-key-hardware-considerations-for-deploying-hadoop-in-your-environment/

These slides give a good technical overview of the HAWQ (Greenplums SQL-on-HDFS solution) architecture (starting on page 21) as well as the features of Spring’s Hadoop integration (starting on page 37).
http://www.slideshare.net/marklpollack/pivotal-hd-and-spring

On the WANdisco blog, Konstantin Boudnik provides an interesting analysis of the Hadoop 2.0-alpha series, which as he notes is on its 5th release in the past 11 months.
http://blogs.wandisco.com/2013/04/22/hadoop-2-alpha-elephant-or-not/

In the paper, “Nobody ever got ﬁred for using Hadoop on a cluster” (no link, due to copyright restrictions), the authors observe that while MapReduce is great for many tasks, there are a growing number of situations (mostly due to the dropping price of memory) in which data can fit in RAM on a single machine.

When getting started with Hadoop, it can be a challenge just to decide which distribution to use — it seems like each week a new vendor is announcing their new distribution. This fragmentation has been a source of discussion recently, and this article speaks a bit about that and also celebrates Apache BigTop as the system making all of these releases possible.
http://blogs.wandisco.com/2013/04/22/on-coming-fragmentation-of-hadoop-platform/

News
Qubole, the Hadoop-as-a-Service startup from a team of former Facebook employees, has raised $7 million in Series A financing. It’s fantastic to see a vote of confidence in a company that’s lowering the barrier to entry of Hadoop.
http://gigaom.com/2013/04/23/hadoop-startup-qubole-raises-7m-for-hive-as-a-service/

Cloudera’s HUE has some interesting new features in the recent 2.3 release. They include Oozie improvements and a new Pig Editor.
http://blog.cloudera.com/blog/2013/04/whats-new-in-hue-2-3/

Spring XD is a new project from the folks at SpringSource focusing on tools for data ingestion, real-time analytics, workflow management, and data export.
http://blog.springsource.org/2013/04/23/introducing-spring-xd/

Sematext’s Performance Monitoring (SPM) suite is adding support for Hadoop. SPM is a proactive monitoring tool that can be self-hosted or used as a service.
http://blog.sematext.com/2013/04/23/sneak-peek-hadoop-monitoring-comes-to-spm/

Releases
HBase 0.94.7 was released and is the new stable version. The release contains performance improvements and bug fixes.
http://mail-archives.apache.org/mod_mbox/hbase-user/201304.mbox/%3C1366934770.31322.YahooMailNeo%40web140601.mail.bf1.yahoo.com%3E

Hadoop 2.0.4-alpha was released. This is intended to be the final alpha release, with a beta following up soon.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201304.mbox/%3C5B011941-90CA-4EEF-BAB9-39A6BFE99B1D%40hortonworks.com%3E

KijiSchema and KijiMR, the HBase schema management software and accompanying MapReduce libraries were updated to versions 1.0.2 and rc61. These releases include bug fixes and improvements.
http://www.kiji.org/2013/04/22/announcing-kijischema-1-0-2-and-kijimr-rc61/

The MySQL project has announced a new product (currently in the MySQL labs) called the MySQL Hadoop Applier for replaying the mysql binlog onto a file in HDFS. Notably, it currently only supports INSERT commands, but data can be inserted into HDFS in near-real time.
http://dev.mysql.com/tech-resources/articles/mysql-hadoop-applier.html
http://innovating-technology.blogspot.com/2013/04/mysql-hadoop-applier-part-1.html

Cloudera released CDH 4.2.1, which includes a number of improvements and bug fixes.
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/yM2tng8-kqI/LinM89A4vhUJ

Events (curated by Mortar Data)
Tuesday, April 30
Big Data Jobs in London Meetup (London, UK)
http://www.meetup.com/Big-Data-Jobs-in-London/events/110496712/

Improving Hive; MapR Hbase M7 (Washington D.C.)
http://www.meetup.com/Hadoop-DC/events/114264532/

Cloudera - Enterprise Big Data Platform (Hamilton Township, NJ)
http://www.meetup.com/nj-hadoop/events/113996532/

Wednesday, May 1
Hadoop Hackathon! (Houston, TX)
http://www.meetup.com/Houston-Hadoop-Meetup-Group/events/102954462/

Thursday, May 2
Big Data, Data Science, and Hadoop (San Francisco, CA)
http://www.meetup.com/San-Francisco-Bay-Area-Microsoft-BI-User-Group/events/114347422/

Thursday, May 2
Data Science for Sustainability (Redwood City, CA)
http://www.meetup.com/Data-Science-for-Sustainability/events/113231972/

Saturday, May 4
Accumulo Hackathon (Washington D.C.)
http://www.meetup.com/Hadoop-DC/events/112435332/

Hadoop Weekly - April 22, 2013

Mon, 22 Apr 2013 06:50:51 -0400

Scott Haylon

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer on Foursquare’s big data team, where he focuses on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

There were a number of exciting announcements and releases this week (e.g. Hadoop on OpenStack, Impala 0.7) as well as some fantastic technical articles and tutorials. It’s great to see more technical articles about how folks are doing things with Hadoop — this week covering Hadoop internals, data formats, and MapReduce-based mobile UI customization. A big thanks to those that share their insights and experiences for making this newsletter possible!

News
Cloudera has announced the Cloudera Academic Partnership program with seven universities part of the initial program. Cloudera cites the need for Hadoop-related expertise as the main motivation for the program.
http://siliconangle.com/blog/2013/04/19/closing-the-gap-on-big-data-education-cloudera-teams-with-top-universities-around-the-world/
http://blog.cloudera.com/blog/2013/04/cloudera-academic-partnership-program-creating-hadoop-lovers-in-universities-worldwide/

Apache BigTop is a system for testing the components of the Hadoop stack in conjunction with one another. This post highlights why it’s becoming popular (particularly with vendors) even though it’s not in the spotlight.
https://blogs.apache.org/bigtop/entry/bigtop_the_way_to_grow

We’ve been seeing a lot of new products focusing on running SQL on HDFS. Most of these products distribute worker-nodes alongside the datanodes. Teradata has taken a different approach (they kind of have to since they ship an appliance). This week, in addition to announcing a new set of hardware, they announced SQL-H which gives Teradata access to data stored on HDFS by using HCatalog to get metadata about the files in HDFS.
http://www.asterdata.com/blog/2013/04/the-enterprise-approach-to-interactive-sql-on-hadoop-data-teradata-sql-h/
http://www.dbms2.com/2013/04/15/teradata-sql-h/ (see the first comment)

Mirantis, Hortonworks, and Red Hat are working on project Savanna to bring Hadoop support to OpenStack (OpenStack is software for managing cloud computing software). It sounds like they’re targeting an initial release for June in time for Hadoop Summit, and that they have some grand plans — everything from provisioning bare-metal hardware to enabling something like Amazon’s Elastic MapReduce.
http://www.theregister.co.uk/2013/04/18/project_savanna_hadoop_on_openstack/
http://hortonworks.com/blog/hadoop-perect-app-for-openstack/
https://wiki.openstack.org/wiki/Savanna

Technical
This is a great overview of the components in the Hadoop stack other than HDFS and MapReduce — in particular, HBase, Cassandra, Pig, Hive, and Impala. It also discusses a few other SQL-on-Hadoop solutions.
http://binalytics.wordpress.com/2013/04/20/quick-hadoop-overview/

While its architecture is easy to understand, HDFS is a complex piece of software that oftentimes seems to work as if by magic. This article discusses the architecture and starts diving into the software stack — providing a map for someone trying to navigate the source code.
http://www.javacodegeeks.com/2013/04/how-hadoop-works-hdfs-case-study.html

Tom White, the author of Hadoop, The Definitive Guide, is writing a series of posts for Dr. Dobb’s about Hadoop. The first article has an overview of HDFS and MapReduce as well as an introduction to various other systems in the Hadoop stack like Flume, Pig, Hive, and HBase.
http://www.drdobbs.com/database/hadoop-the-lay-of-the-land/240150854

LinkedIn is using Hadoop-based algorithms to customize the UI on their mobile apps, where real estate is limited. Their infrastructure includes Kafka for data ingestion, a Hadoop workflow for building recommendations (which they describe in some detail), and Voldemort for serving the data in real-time.
http://engineering.linkedin.com/mobile/linkedin-mobile-introducing-personalized-navigation

At the Twitter Seattle Open House, Julien Le Dem presented on Parquet, the new columnar storage format that Twitter is building in collaboration with Cloudera. The slides include a great overview of the use-case, the file format, and some initial benchmarks.
http://www.slideshare.net/julienledem/parquet-twitter-seattle-open-house

Tutorials
Cloudera HUE provides a web interface to interact with Hadoop to upload and browse data as well as run Hive and MapReduce jobs. In this tutorial, you’ll load a dataset from the Yelp challenge into Hive, run some SQL queries on it, and then run a python streaming MapReduce job using MrJob.
http://blog.cloudera.com/blog/2013/04/demo-analyzing-data-with-hue-and-hive/

Redis is a key-value store that supports various data structures such as lists, sets, strings, and more. This tutorial covers getting data in and out of Redis from MapReduce, including the code for custom input formats, record readers, and output formats.
http://www.greenplum.com/blog/topics/hadoop/making-hadoop-mapreduce-work-with-a-redis-cluster

A tutorial covering running Apache Mahout on HDInsight (HDInsight is the Hadoop Distribution running on Windows Azure). Covers install, setup, and running a Mahout MapReduce job.
http://bluewatersql.wordpress.com/2013/04/12/installing-mahout-for-hdinsight-on-windows-server/

Releases
Cloudera Impala 0.7 was released (and a few days later the 0.7.1 release with some critical bug fixes was announced). Version 0.7.1 has a bunch of new features, including support for the Parquet columnar file format and avro, plus distributed aggregations and top-n computations. This release supports CDH4.1 and 4.2 as well as a number of different linux distributions.
https://groups.google.com/a/cloudera.org/d/msg/impala-user/2EUowODoRIk/44S7Q044-_UJ
https://groups.google.com/a/cloudera.org/d/msg/impala-user/LuWvbjUY0EU/mHKJYc2Blm4J

Apache MRUnit, the MapReduce unit-testing library reached version 1.0.0. It supports both hadoop 1 and hadoop 2.
http://mail-archives.apache.org/mod_mbox/hadoop-common-user/201304.mbox/%3CCAFZSZPsXMx6taV-%2B3SC7GKaV1M%2BkRE3TMLk%3DcDZ-5GAn8UceZQ%40mail.gmail.com%3E

Last week, Amazon announced support for Elastic MapReduce on their GovCloud service.
http://www.theregister.co.uk/2013/04 /09/amazon_adds_hadoop_to_govcloud/

UC Berkeley’s AMPLab, the same lab that develops Spark, has announced the Tachyon Project. Tachyon is a distributed file system that can cache some datasets in memory, but it checkpoints data to an underlying file system (it currently supports HDFS or a single node local file system).
http://tachyon-project.org/

Events
Monday, April 22
Cloudera Sessions (Toronto, Canada)
http://www.meetup.com/TorontoHUG/events/114398482/

Tuesday, April 23
Natural Language Processing and Big Data (Washington, DC)
http://www.meetup.com/Data-Science-DC/events/109386702/

Wednesday, April 24
Big Data @ Yelp — taming the reviews & recommendations (San Jose, CA)
http://www.meetup.com/BigDataGurus/events/114645332/

Thursday, April 25
Data in the Big City (New York, NY)
http://www.meetup.com/DataKind-NYC/events/112727792/

Thursday, April 25
Power in Numbers: Growing Atlanta’s Data Science Talent (Atlanta. Georgia)
http://www.meetup.com/Data-Science-ATL/events/109289502/

Thursday, April 25
Bigvis: visualising 100,000,000 observations in R with Hadley Wickham (New York, NY)
http://www.meetup.com/nyhackr/events/112271042/

Saturday, April 27
Map Reduce Programming - Deep Dive (Santa Clara, CA)
http://mapreduce-deep-dive-es2005.eventbrite.com/?rank=6

Hadoop Weekly - April 15, 2013

Mon, 15 Apr 2013 07:57:00 -0400

Scott Haylon

Hadoop Weekly is a recurring guest post by Joe Crobak. Joe is a software engineer on Foursquare’s big data team, where he focuses on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

This week’s newsletter features fewer releases than normal (let me know if I missed something!) but has a lot of interesting technical articles. In addition, I’m pleased to announce the return of an events section. Thanks to the folks at Mortar Data for curating this list! They’ve found a number of great Hadoop-related events taking place all over the world this week.

Technical
Apache Pig provides support for expressive SQL-like join operations. In this post, Matthew Rathbone shows how to implement a left-outer join in Pig and write a unit test to check for correctness. This is his third article that demos a framework — he previously covered MapReduce and Hive. This trifecta is quite an interesting comparison, so be sure to read all three if you missed the previous articles.
http://blog.matthewrathbone.com/2013/04/07/real-world-hadoop—-implementing-a-left-outer-join-in-pig.html

If you’re reading this newsletter, you probably don’t need convincing, but Ofer Mendelevitch from Hortonworks offers some compelling reasons to use Hadoop for Data Science. Each reason (such as “Data exploration with full datasets”) includes a discussion, and some reasons include a discussion of the tools available to aid a data scientist.
http://hortonworks.com/blog/4-reasons-to-use-hadoop-for-data-science/

Apache Ambari is a system for managing and configuring Hadoop and related projects such as Apache Zookeeper. This tutorial covers configuring a 6 node test cluster on EC2 with HDFS, Mapreduce, Nagios, Ganglia, HBase, ZooKeeper, Hive, HCatalog, and Zookeeper.
http://hortonworks.com/kb/ambari-on-ec2/

Vagrant provides a command line interface and tools to spin up and configure virtual machines with a Virtualbox, VMWare, or a cloud provider. This post explains using Vagrant to build and configure a (virtual machine) Hadoop cluster. The recipe from the post lets one build a 6-node cluster with a single command, ‘vagrant up’.
http://blog.cloudera.com/blog/2013/04/how-to-use-vagrant-to-set-up-a-virtual-hadoop-cluster/

Krishnan Raman of Twitter presented on using Scalding (a scala DSL for Cascading) and Algebird (Twitter’s open-source abstract algebra framework) at BigData TechCon in Boston. In addition to the slides, the code and materials for the presentation have been posted to github.
https://github.com/krishnanraman/bigdata/blob/master/ProgrammingScaldingAlgebird.pdf?raw=true
https://github.com/krishnanraman/bigdata

Splout is a SQL data store that includes tight-coupling with Hadoop and is suitable for serving real-time, web-scale traffic. This post is the 3rd in a series (the first two covered Hive and Cascading), and it covers loading data into Splout from Pig.
http://www.datasalt.com/2013/04/pig-splout-sql-for-a-retail-coupon-generator-a-big-data-love-story/

HCatalog provides access to Hive’s metadata to other portions of the Hadoop stack (e.g. MapReduce and Pig) as well as via REST. This allows HCatalog to act as a glue between many different components in the stack. This blog post has a great overview of HCatalog’s features and benefits.
http://hortonworks.com/blog/hivehcatalog-data-geeks-big-data-glue/

AirBnB has followed up their recent post about Chronos, their workflow and scheduling software, with an overview of their big data stack. They’re using Storm and Hadoop, in addition to Chronos, on a single Mesos cluster. They have some information about each and promise a follow up post with more details.
http://nerds.airbnb.com/distributed-computing-at-airbnb

News
The Hadoop Summit selection committees have created the initial program for Hadoop Summit taking place this June in San Jose. More sessions will be posted over the coming weeks.
http://hadoopsummit.org/san-jose/program/

The Call for Proposals is open through May 16th for Strata + Hadoop World. The conference takes place in New York in October, and covers big data, data science, and pervasive computing. Proposals for 40-minute sessions as well as 3-hour tutorials are accepted for one of these topics.
http://strataconf.com/stratany2013/public/cfp/264

Releases
A few weeks ago, Wibidata announced version 1.0.0 of KijiSchema. KijiSchema is a data management system atop of Apache HBase focussed on real-time retrieval of diverse datasets. The 1.0.0 version marks a commitment to maintaining API compatibility going forward.
http://www.kiji.org/2013/04/02/announcing-kijischema-1-0-0/

Also announced a few weeks ago was the general availability of Platfora. Unlike other solutions that provide either an SQL interface or focus on BI tools, Platfora is trying to do both. A user asks a question via a web UI, and Platfora imports and caches data via MapReduce jobs in order to find an answer.
http://www.platfora.com/hadoop-ecosystem-blog/

Events (curated by Mortar Data)
Tuesday, April 16
Amazon Elastic Map Reduce - Hadoop Cloud Service (Hamilton Township, NJ)
http://www.meetup.com/nj-hadoop/

Tuesday, April 16
St. Louis Hadoop Users Group Meetup (Saint Louis, MO)
http://www.meetup.com/St-Louis-Hadoop-Users-Group/events/110159842/

Wednesday, April 17
Automating the Hadoop Stack with Chef (San Diego, CA)
http://www.meetup.com/sd-hug/events/112475312/

Thursday, April 18
A recommendation system and MapReduce (New York, NY)
http://www.meetup.com/NYC-Machine-Learning/

Thursday, April 18
Hadoop 2.0: What’s coming? (Toronto)
http://www.meetup.com/TorontoHUG/events/112153292/

Thursday, April 18
Big Data in the AWS Cloud + More (Norwich, UK)
http://www.syncnorwich.com/events/110574642/?eventId=110574642&action=detail

Friday, April 19
Big Hadoop Jobs on AWS (Munich, Germany)
http://www.meetup.com/Hadoop-User-Group-Munich/events/102940592/

Friday, April 19
Hadoop At Spotify (Kraków, Poland)
http://www.meetup.com/datakrk/events/113175722/

Data Science at Tumblr

Tue, 09 Apr 2013 13:41:22 -0400

Scott Haylon

Our second NYC Data Science Meetup featured Tumblr data scientist Adam Laiacano, who discussed the analytics stack at Tumblr and the tools he and his team use to organize and analyze data.

Here are the video and slides from Adam’s talk, which cover Tumblr’s use of Scribe, Hive & Pig, Hue, and Vowpal Wabbit:

Big thanks to Adam for his talk and to great group of data enthusiasts that attended.

Hadoop Weekly - April 8, 2013

Mon, 08 Apr 2013 07:03:00 -0400

Scott Haylon

Happy 7th birthday to Apache Hadoop! The first release of Hadoop was made in April 2006. This week’s newsletter caps that anniversary by representing many parts of the Hadoop ecosystem. It’s quite impressive how far the project and the ecosystem have come in those 7 short years!

News
April 2nd marked the 7-year anniversary of the first release of Apache Hadoop. In this post, Doug Cutting (the founder of Hadoop) provides 7 thoughts and predictions for Hadoop. He touches everything from open-source, to the name of the project, to where he sees Hadoop heading in the next 7 years.
http://blog.cloudera.com/blog/2013/04/seven-thoughts-on-hadoops-seventh-birthday/

The folks at LiveRamp have come up with a clever technique to speed up joins/cogroups by filtering map-side using Bloom filters. If you haven’t seen Bloom filters before, the post explains their usefulness in this context. With this technique, they see performance improvements of 2x for a large job. They have open-sourced an implementation of this technique for Cascading.
http://blog.liveramp.com/2013/04/03/bloomjoin-bloomfilter-cogroup/

Videos and slides from Hadoop Summit EU are beginning to arrive online. Hortonworks highlights the keynotes from the events which include presentations from 451 Research, Hortonworks, and a panel featuring HSBC, eBay and others. You can find many more talks (and more being added every week) on the Hadoop Summit YouTube page, too.
http://hortonworks.com/blog/keynotes-from-hadoop-summit-amsterdam-2013/
http://www.youtube.com/user/HadoopSummit?feature=watch

YARN is the new resource scheduler in Hadoop 2.0 for building applications other than vanilla MapReduce on a Hadoop cluster. Josh Patterson has started a new open-source project called Metronome built upon YARN. The software is based upon former projects IterativeReduce and Knitting Boar, and it provides an implementation of parallel linear regression.
http://www.slideshare.net/jpatanooga/hadoop-summit-eu-2013-parallel-linear-regression-iterativereduce-and-yarn
https://github.com/jpatanooga/Metronome

VMWare has released version 0.8.0 of Serengeti, their open-source initiative to improve Hadoop for virtualization. This release includes support for CDH4 and MapR’s distributions as well as improved support for HBase.
http://blogs.vmware.com/vfabric/2013/04/new-serengeti-release-extends-cloud-computing-support-for-hadoop-community.html

Luigi is an open-source Hadoop framework from the folks at Spotify. We’ve been using it at Foursquare for a few months and really like it. In this presentation, Elias gives an overview of Luigi as well as the evolution of Spotify’s thinking about workflow management which explains how they arrived at Luigi.
http://www.slideshare.net/EliasFreider/luigi-pydata-presentation
https://github.com/spotify/luigi

Datanami has a good summary of Eric Baldeschwieler’s (aka Eric14) keynote from Hadoop Summit. The synopsis includes Eric’s views on the future of Hadoop, from scaling to 10,000 nodes to lots of younger projects in the Hadoop ecosystem like HCatalog, Ambari, Tez, and more.
http://www.datanami.com/datanami/2013-04-03/baldeschwieler:_looking_at_the_future_of_hadoop.html

Microsoft has posted an in-depth analysis of how they use Hadoop on Azure (HDInsight) with Halo 4. The case study includes everything from analytics and BI to email targeting. It’s a pretty interesting and impressive analysis considering that just a few months ago Hadoop didn’t run on Windows at all.
http://www.microsoft.com/enterprise/it-trends/big-data/articles/Changing-the-Game-Halo-4-Team-Gets-New-User-Insights-from-Big-Data-in-the-Cloud.aspx#fbid=OAmTkNNsaBu

Falcon is a new Apache Incubator project from the folks at InMobi and Hortonworks focussing on ETL. It has a number of use cases, such as disaster recovery, multi-cluster management, and SLA management. It seems to have some overlap with existing projects (e.g. Oozie or Sqoop) but is focused on just ETL within or between Hadoop clusters so far.
http://hortonworks.com/blog/project-falcon-tackling-hadoop-data-lifecycle-management-via-community-driven-open-source/
http://www.inmobi.com/inmobiblog/2013/04/02/inmobi-works-with-hortonworks-to-incubate-falcon-with-apache-software-foundation-to-provide-huge-benefits-to-the-big-data-community/

Did you know that Windows Azure can run Linux VMs? This tutorial shows how to build a Linux (CentOS) Hadoop cluster in Windows Azure. After booting a Windows Server for DNS, the rest of the tutorial focuses on Hadoop (they use HDP 1.2.2) on Linux.
http://blogs.msdn.com/b/benjguin/archive/2013/04/05/how-to-install-hadoop-on-windows-azure-linux-virtual-machines.aspx

April Fools’ Day was this week, and there were a few fake Hadoop-related product announcements. Here are a couple in case you missed them.
http://www.hadoopsphere.com/2013/04/yas-1000x-faster-sql-on-hadoop-engine.html
http://www.wibidata.com/blog/real-time-is-reckless-slow-and-steady-wins-the-race

Releases
Apache Pig 0.11.1 was released. This update includes fixes to Avro, HCatalog, and HBase integrations (and more) as well as improvements including documentation polish.
http://pig.apache.org/releases.html#1+April%2C+2013%3A+release+0.11.1+available

KairosDB is a rewrite of OpenTSDB (the time series database and visualization system from StumpleUpon) with a pluggable backend (defaults to Cassandra but also supports HBase and H2). KairosDB uses Flot for visualization and provides REST APIs for retrieving data. The release on their website is 1.0.0-alpha-4a, so I assume it’s still considered alpha quality.
http://nosql.mypopescu.com/post/47102531877/kairosdb-fast-scalable-time-series-database
https://code.google.com/p/kairosdb/

Hama is a computing system on top of HDFS that is specialized for matrix and graph problems. Version 0.6.1, which includes improvements, bug fixes and new features, was released this week.
http://mail-archives.apache.org/mod_mbox/hadoop-general/201304.mbox/%3CCAGQgZQQ1x3w5tRB3eVs-ZNdsBKGz5Qdwy%3DhW5JOOjcmadOUC6Q%40mail.gmail.com%3E
https://issues.apache.org/jira/issues/?jql=fixVersion%20%3D%20%220.6.1%22%20AND%20project%20%3D%20HAMA

Cloudera Manager 4.5.1 was released. This bug fix release contains fixes for HDFS, MapReduce, and Hive.
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/qYgMASROJWQ/ctFfmzE1TF0J
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.1/Cloudera-Manager-Enterprise-Edition-4.5.x-Release-Notes/Cloudera-Manager-Enterprise-Edition-4.html

Cloudera announced the Cloudera Developer Kit. The goal of the project is to make it easier to write applications on top of Hadoop. Unlike other high-level frameworks for Hadoop, CDK is focusing on all layers of the stack, not just MapReduce. For example, the data API is one of the first under development, and it focuses on easing the burden of implementing data integration services which would normally have to muck with the nuances of the HDFS APIs.
https://groups.google.com/a/cloudera.org/d/msg/cdh-user/xJV77baI4Ss/5oPZzcaIe7wJ
https://github.com/cloudera/cdk

ElasticSearch Hadoop is a set of libraries for MapReduce, Pig, Hive, and Cascading from the folks at ElasticSearch. They haven’t yet made a release, so you have to build it yourself, but it includes a bunch of features. The drivers support read from and write to ElasticSearch over REST/JSON, and they’ve made the binary small and independent for easy integration (just add a single jar!).
https://github.com/elasticsearch/elasticsearch-hadoop

CDH3u6 was released. This version includes a handful of fixes in MapReduce, Flume, and HBase.
http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH3/CDH3u6/CDH3-Release-Notes/CDH3-Release-Notes.html

Hadoop Weekly - April 1, 2013

Mon, 01 Apr 2013 08:16:00 -0400

Scott Haylon

Hadoop Weekly is a new (recurring) guest post by Joe Crobak. Joe is a software engineer on Foursquare’s big data team, where he focuses on Hadoop and analytics. You can follow Joe on Twitter at @joecrobak.

News
Apache Hadoop’s Distributed File System and MapReduce were originally based upon research papers written by Google. Google owns a number of patents in these spaces, including 10 related to MapReduce. This week, they pledged “not to sue any user, distributor or developer of open-source software on specified patents, unless first attacked.”
http://google-opensource.blogspot.com/2013/03/taking-stand-on-open-source-and-patents.html

MapR and Canonical announced that MapR’s M3 Hadoop distribution will be an integrated offering into Ubuntu 12.04 LTS and 12.10 via the Ubuntu Partner Portal.
http://www.businesswire.com/news/home/20130328005139/en/MapR-Teams-Canonical-Deliver-Hadoop-Ubuntu

MapR also made some new this week by open-sourcing their forks of a number of projects in the Hadoop stack (but not MapR FS). The list includes, sqoop, pig, mahout, hive, hbase, oozie, opentsdb, scribe and more. Some of these projects haven’t been updated in a year (scribe) but the majority were updated in the past month.
http://www.businesswire.com/news/home/20130328005137/en/MapR-Source-Code-GitHub
https://github.com/mapr/

Netflix has an in-depth blog post about building recommendation systems. They discuss the three types of systems that they use — offline, nearline, and online, as well as the trade-offs and design decisions you have to consider for each. The post contains a number of detailed system diagrams with thorough explanations about the systems that they use from hadoop to cassandra to mysql.
http://techblog.netflix.com/2013/03/system-architectures-for.html

GIS Tools is a set of open-source tools for spatial data analytics on Hadoop from the folks at Esri. It includes java libraries for integration into MapReduce as well as lots of Hive UDFs for spatial and geometric processing.
http://blogs.esri.com/esri/arcgis/2013/03/25/gis-tools-for-hadoop/
http://esri.github.com/gis-tools-for-hadoop/

HUE is an open-source web interface to Hadoop, which includes a number of applications such as an HDFS browser and a web-based interface into Hive called Beeswax. This tutorial describes loading tweet data into HDFS, creating a table for that data in Hive, and running an analysis of the data using Hive.
http://blog.cloudera.com/blog/2013/03/how-to-analyze-twitter-data-with-hue/

Microsoft’s HDInsight platform includes a developer edition for local development and testing. In this post, the author describes the process of setting up a local development environment on Windows, writing a map reduce job using C#, running your job on the cluster, and loading the data into Hive.
http://www.amazedsaint.com/2013/03/taming-big-data-with-c-using-hadoop-on.html

Matt Walker presented on Etsy’s data stack at Data Day Texas. He covers the evolution of their offline infrastructure over several years, systems that they’ve built, and the tools & frameworks that they use for offline analytics.
http://www.slideshare.net/mwalkerinfo/data-daytexas2013

The Call for Speakers for HBaseCon 2013 ends today (4/1). Cloudera interviewed some members of the HBaseCon Program Committee about HBase and HBaseCon.
http://blog.cloudera.com/blog/2013/03/meet-the-hbasecon-2013-program-committee/

Debugging MapReduce is usually a matter of adding counters or System.out.println statements to determine what is happening. If you’re developing on a local box, though, a simpler solution is to attach a debugger. This walkthrough has all you need to know about debugging a MapReduce job in IntelliJ.
http://vichargrave.com/debugging-hadoop-applications-with-intellij/

Releases
Apache Oozie 3.3.2 was released with a number of improvements and bug fixes. Among the highlights are: uberjar support for MapReduce actions, improvements to the ooze web ui, and improvements to the command line for coordinators.
http://mail-archives.apache.org/mod_mbox/oozie-user/201303.mbox/%3CCAHz+ZFd2dmk4U8Z8M41QQS-4o8nQea6CfZ5uzLpuR3oz9XjAdA@mail.gmail.com%3E

Dirty Secrets of Data Science

Tue, 12 Mar 2013 11:35:00 -0400

Scott Haylon

Thanks to everyone who came out to our inaugural NYC Data Science Meetup. For those who couldn’t attend, Hilary Mason fought off jetlag and a tough cold to give a great presentation.

Below is a 12-minute clip from Hilary’s talk, which she called “Dirty Secrets of Data Science.”

P.S. - If you’re in NYC and haven’t yet, make sure you join the NYC Data Science Meetup Group.

Pig Eye for the SQL Guy

Fri, 01 Mar 2013 15:16:00 -0500

Cat Miller

For anyone who came of programming age before cloud computing burst its way into the technology scene, data analysis has long been synonymous with SQL. A slightly awkward, declarative language whose production can more resemble logic puzzle solving than coding, SQL and the relational databases it builds on have been the pervasive standard for how to deal with data.

As the world has changed, so too has our data; an ever-increasing amount of data is now stored without a rigorous schema, or must be joined to outside data sets to be useful. Compounding this problem, often the amounts of data are so large that working with them on a traditional SQL database is so non-performant as to be impractical.

Enter Pig, a SQL-like language that gracefully tolerates inconsistent schemas, and that runs on Hadoop. (Hadoop is a massively parallel platform for processing the largest of data sets in reasonable amounts of time. Hadoop powers Facebook, Yahoo, Twitter, and LinkedIn, to name a few in a growing list.)

This then is a brief guide for the SQL developer diving into the waters of Pig Latin for the first time. Pig is similar enough to SQL to be familiar, but divergent enough to be disorienting to newcomers. The goal of this guide is to ease the friction in adding Pig to an existing SQL skillset.

WHAT’S SIMILAR?

The basic concepts in SQL map pretty well onto Pig. There are analogues for the major SQL keywords, and as a result you can write a query in your head as SQL and then translate it into Pig Latin without undue mental gymnastics.

WHERE → FILTER

The syntax is different, but conceptually this is still putting your data into a funnel to create a smaller dataset.

HAVING → FILTER

Because a FILTER is done in a separate step from a GROUP or an aggregation, the distinction between HAVING and WHERE doesn’t exist in Pig.

ORDER BY → ORDER

This keyword behaves pretty much the same in Pig as in SQL.

JOIN

In Pig, joins can have their execution specified, and they look a little different, but in essence these are the same joins you know from SQL, and you can think about them in the same way. There are INNER and OUTER joins, RIGHT and LEFT specifications, and even CROSS for those rare moments that you actually want a Cartesian product.

Because Pig is most appropriately used for data pipelines, there are often fewer distinct relations or tables than you would expect to see in a traditional normalized relational database.

CONTROL OVER EXECUTION

SQL performance tuning generally involves some fiddling with indexes, punctuated by the occasional yelling at an explain plan that has inexplicably decided to join the two largest tables first. It can mean getting a different plan the second time you run a query, or having the plan suddenly change after several weeks of use because the statistics have evolved, throwing your query’s performance into the proverbial toilet.

Various SQL implementations offer hints to combat this problem—you can use a hint to tell your SQL optimizer that it should use an index, or to force a given table to be first in the join order. Unfortunately, because hints are dependent on the particular SQL implementation, what you actually have at your disposal varies by platform.

Pig offers a few different ways to control the execution plan. The first is just the explicit ordering of operations. You can write your FILTER before your JOIN (the reverse of SQL’s order) and be clever about eliminating unused fields along the way, and have confidence that the executed order will not be worse.

Secondly, the philosophy of Pig is to allow users to choose implementations where multiple ones are possible. As a result, there are three specialized joins that a can be used when the features of the data are known, and are less appropriate for a regular join. For regular joins, the order of the arguments dictates execution—the larger data set should appear last in this type of join.

As with SQL, in Pig you can pretty much ignore the performance tweaks until you can’t. Because of the explicit control of ordering, it can be useful to have a general sense of the “good” order to do things in, though Pig’s optimizer will also try to push up FILTERs and LIMITs, taking some of the pressure off.

WHAT’S DIFFERENT?

A Row By Any Other Name

The SQL paradigm is very straightforward—there are tables, and tables contain rows. Every select statement yields a set of rows, and each field in a row is a basic data type. Conceptually, the result of any SQL select statement can be imported into Excel without loss of information.

Pig introduces a mature nesting notion in tuples and data bags that changes the game significantly. Pig consists of data sets called relations (sometimes called aliases for the names they are given), and those contain records that are data tuples, which can in turn recursively contain data bags, data tuples, or data items. There is distinct lack of flatness in Pig, and the best way to see it is to explore how GROUP works.

Not Your Grandmother’s GROUP

In the handling of GROUP, SQL and Pig diverge significantly. SQL’s GROUP doesn’t exist outside of the aggregation performed on it; you would never SELECT * GROUP by field1–it just doesn’t make sense. Because everything in SQL is a row, the grouping created isn’t persistent—only the data produced aggregating over it remains.

Pig’s GROUP is an entirely different beast, albeit used for the same purpose. It is a persistent relation that can be used agan, independent of what aggregations you might choose to perform on it later.

student_grades = GROUP grades by student;

This student_grades relation has two fields: one called group and populated with the value of student, the other called grades populated with a data bag containing tuples for all the entries with the same value of student.

group	grades
alyssa 	{< hacking101, 95 >, < english, 60 >}
ben	{< math, 90 >}

Now to do an aggregation, perform it on the student_grades alias.

average_grades = FOREACH student_grades GENERATE group, AVG(grades.value);

Procedural Paradise

This is the first thing any Google search on Pig will tell you, and it is the most glaringly obvious change from SQL. After having taught your brain for years how to turn an idea inside-out and mash all of its pieces into one query, Pig makes query writing feel like writing Java or C++. In addition to to obvious potential cognitive benefits, this has some technical ones as well.

Subquery Reuse

Ever write a query with a subselect, and then realize you actually needed to use that subtable twice in the query? Did you feel absolutely awful as you cut and pasted that subtable? Did you wonder whether your SQL plan would successfully manage to not calculate it twice? (Note: the WITH clause mitigates this pain in a lot of cases, but isn’t available in all flavors of SQL.)

Because in Pig Latin every step has a declared alias, reusing “subquery” tables is natural and intuitive, and generally does not involve building them twice.

Getting multiple queries out of one pipeline

In SQL you can find yourself in a place where you want to use the data, do some manipulation on it, and then take it in a few different directions. To do this in one query requires profligate use of JOIN, and enough parens to intimidate a LISP hacker.

In a Pig pipeline, any and all aliases produced along the way can be stored, and all it takes is adding a new STORE statement to the script.

User-Defined Functions

SQL has had decades for people to figure out what analytic functions they need for arbitrary data analysis, and so when you find yourself suddenly interested in extracting the day of the week from a date, that function is ready and waiting.

Pig’s list of built-in functions is growing, but is still dwarfed by what Oracle or MYSQL provides. What turns this into a tolerable constraint is that Pig allows the user to define aggregate or analytic functions in other languages (Java, Python, and others) and then apply them in Pig quickly and without fuss.

REGISTER udf.jar;

new_data = FOREACH my_data GENERATE udf.ImportantFunction(field1);

The Well-Disguised SQLer

In general, if you can think about it in SQL, you can do it in Pig. Be aware of the nested data structures, have a cheat sheet for syntax, and relish the ability to write queries the way your brain thinks them, and not the way SQL demands.

As a final thought, let’s resurrect our old friend the emp table, and take a look at some SQL to Pig Latin examples.

Average Salary by Location

SQL

SELECT loc, AVG(sal) FROM emp JOIN dept USING(deptno) WHERE sal > 3000 GROUP BY loc;

Pig Latin

filtered_emp = FILTER emp BY sal > 3000;

emp_join_dept = JOIN filtered_emp BY deptno, dept BY deptno;

grouped_by_loc = GROUP emp_join_dept BY loc;

avg_salary = FOREACH grouped_by_loc GENERATE group, AVG(emp_join_dept.sal);

Ordered Average Salary by Location

Suppose now that the following is true.

● The ‘loc’ field is a string/varchar field, and we have two pieces of software that automatically populate it. One stores values as lowercase, one as uppercase. (If this seems like a contrived example to you, you have chosen your employers and software vendors well.)

The new parts of the queries appear in bold.

SQL

SELECT standard_loc, AVG(sal) avg_salary FROM
(SELECT UPPER(loc) standard_loc, sal FROM emp JOIN dept USING(deptno) WHERE sal > 3000) std_table GROUP BY standard_loc;

Pig Latin

filtered_emp = FILTER emp BY sal > 3000;

emp_join_dept = JOIN filtered_emp BY deptno, dept BY deptno;

grouped_by_loc = GROUP emp_join_dept BY UPPER(loc);

avg_salary = FOREACH grouped_by_loc GENERATE group, AVG(emp_join_dept.sal);

This kind of change is friendlier to Pig because of the limitations in SQL’s GROUP BY clause; the trade-off of verboseness for clarity increases in value the more complex your query gets.

Above Average Salary for Location

Now suppose instead of arbitrarily selecting 3,000 as a threshold, which is going to overselect people living in large expensive cities, we want to select those employees who make more than twice the average for their location.

SQL

There are several ways to accomplish this, of course, but for illustration purposes the most vanilla version is shown here.

SELECT empno FROM 
(SELECT empno, UPPER(loc) standard_loc, sal FROM emp JOIN dept USING(deptno)) std_table1 JOIN
(SELECT standard_loc, AVG(sal) avg_salary FROM
(SELECT UPPER(loc) standard_loc, sal FROM emp JOIN dept USING(deptno)) std_table2 GROUP BY standard_loc) grp_table2
USING(standard_loc) WHERE sal > (2 * avg_salary);

Pig Latin

emp_join_dept = JOIN emp BY deptno, dept by deptno;

grouped_by_loc = GROUP emp_join_dept BY UPPER(loc);

loc_avg_salary = FOREACH grouped_by_loc GENERATE AVG(emp_join_dept.sal) as avg_salary, FLATTEN(emp_join_dept);

highly_paid = FILTER loc_avg_salary BY sal > (2 * avg_salary);

Here the dreaded subquery reuse rears its unattractive head in SQL and leads to a bit of a frankenquery. In contrast, the Pig script stays the same length, because we can effectively just shift the FILTER from the beginning to the end of our data flow. FLATTEN is a new entry to our function arena, for which there is no SQL analogue. What FLATTEN does is unnest tuples and data bags; in this example it’s taking the data bag ‘emp_join_dept’ created by the GROUP function, and removing the nesting so that the fields within it will be at the same level as avg_salary.

Run It Yourself for the Swine-Curious

If you want to to try out some Pig examples hands-on, you can get a free Mortar account here. We’ve generated a one million row emp table data set, so to run these examples on your own all you need to do is:

Sign up at app.mortardata.com for a free account.
Go to Web Projects
Select My Web Projects -> New Blank project

To load the two data files, you need LOAD statements, and to store the resulting data you need a STORE statement, so in the end your pig script should look like this:

dept = LOAD 's3n://mortar-example-data/employee/dept.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage(',', 'NO_MULTILINE', 'NOCHANGE', 'SKIP_INPUT_HEADER')
    AS (deptno:int, dname:chararray, loc:chararray);

emp = LOAD 's3n://mortar-example-data/employee/emp.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage(',', 'NO_MULTILINE', 'NOCHANGE', 'SKIP_INPUT_HEADER')
    AS (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, deptno:int);

-- Pig example code here

STORE avg_salary INTO 's3n://mortar-example-output-data/$MORTAR_EMAIL_S3_ESCAPED/emp_example’ USING PigStorage('\t');

Click Run and select a 2-node cluster. Even with one million rows, it runs in under three minutes. Sit back and watch the power of modern computing tackle the problems of the 1980s.

MongoDB + Hadoop: Why & How

Thu, 14 Feb 2013 11:08:00 -0500

Jeremy Karn

You have MongoDB, so you have this tremendously scalable database. You’re collecting a ton of data, but you know you need to do more with it (okay, a lot more). You think you want to use Hadoop, but it doesn’t sound easy.

To keep it simple, we’ve divided the article into three parts:

“WHY” explains the reasons for using Hadoop to process data stored in MongoDB
“HOW” helps you get get set up
“DEMO” shows you MongoDB and Hadoop working together. If you’re a tldr; type, you’ll want to start with this section.

WHY

Mongo was built for data storage and retrieval, and Hadoop was written for data processing. So naturally, data processing is often better offloaded to Hadoop. Here’s why:

Easier, more expressive languages

MongoDB supports native MapReduce, but MapReduce is a pain in the ass. The Hadoop community has created Pig, Hive, and the Cascading family of languages—all of which compile to MapReduce, but are expressive and high-level.
Libraries to build on

Very few popular data processing libraries are written in Javascript, so you’ll often find yourself without access to the libraries you need, such as NumPy, NLTK, etc.
Big performance improvements

Hadoop is purpose-built for fully distributed, multi-threaded execution of data processing, so it performs much, much better for this sort of work.
Separate workloads mean less load

If you’re doing significant data processing on MongoDB, it can add substantial load. You may need an order of magnitude more power to process data than to store it, so it works really well to separate those concerns and separate those workloads.

So if you want to easily write distributed jobs that perform well and don’t add load to your primary storage system, Hadoop is probably the way to go.

Bonus: We’ve included ready-to-use Hadoop code to extract the “schema” of your MongoDB, and characterize how that schema is used in the demo section.

HOW

We used Mortar for this demo because it’s free for this purpose, and you won’t need to set up any infrastructure. Mortar is an open source framework for easily writing/developing your Hadoop jobs coupled with Hadoop-as-a-Service for running Mortar jobs on Hadoop clusters. If you’re going to use Mortar, skip to the DEMO section.

Otherwise, you need:

Hadoop Cluster

There are many ways to run Hadoop, here are a couple:
1. Set up your own cluster: You’ll need some machines and you’ll need to follow these instructions. Warning, this is not for the faint of heart, and probably should be reserved for companies with substantial resources and serious sys admin chops.
2. Use Amazon Elastic MapReduce (EMR): EMR is an offering by Amazon Web Services (AWS) that allows you to run a Hadoop-based job on a Hadoop cluster in the cloud. Aside from all of the typical cloud benefits that you get from doing this, you also get to skip the setup and configuration of a Hadoop cluster. There’s a step-by-step guide to setting up EMR here.
Input Connector

To load your data from MongoDB into Pig, you’ll need the Pig loader. Here’s documentation on how you can use the loader.
Data Processing Language

In this demo you’ll process your data on Hadoop with Apache Pig, a high-level data flow language that compiles down into Hadoop MapReduce jobs. It was designed to be easy to learn and simple to write. If you’ve written SQL, Pig will feel familiar—it is like procedural SQL. For more details on Pig, check out “8 reasons you should be using Apache Pig”.

If you don’t want to write a Pig script and would prefer to stick with raw Hadoop MapReduce jobs, the Mongo-Hadoop project will support that, but we won’t cover raw MapReduce in this article since it requires 10x more code without much benefit.
Output Connector

OK, now it’s time to choose an output destination for your processed data. You have a lot of options here:
1. MongoDB
  
  If your want to write your final results back to MongoDB, the Mongo-Hadoop project also contains support for this with MongoStorage.
  
  For documentation on how to do this read the “Storing Data to MongoDB” section under “Using MongoDB Data”.
2. Amazon’s S3
  
  One of the most useful output locations - If you’re running your Hadoop job on a cluster that’s running in AWS, then your data transfer will be extremely fast and you get all of the benefits of storing your data in the cloud.
3. HDFS
  
  If you’re running your own long-lived cluster, you can write the results to the cluster’s distributed file system. This can be useful if you’re going to be doing more processing on the data later.

DEMO

The fun part! Here’s a quick step-by-step example that should take just a few minutes.

To get started, we’ve already set up a small MongoDB instance on MongoLab, populated it with a random sampling of Twitter data from a single day (around 120,000 tweets), and created a read-only user for you.

We’ve also set up a public Github repo with a Mortar project that has three Pig scripts ready to run. Here’s what you need to do:

If you don’t already have a free Github account - create one. You’ll need a github username in step 4.
Sign into (or create) your free Mortar account.
After you receive the confirmation email, log into Mortar at https://app.mortardata.com.
Install the Mortar Development Framework:
```
    $ gem install mortar
```
(full installation details here)

Clone the example git project and register it as a mortar project:

    $ git clone git@github.com:mortardata/mongo-pig-examples.git
    
    $ cd mongo-pig-examples
    
    $ mortar register mongo-pig-examples

Script 1 - Characterize Collection

If you’re like most MongoDB users, you may not have a great sense of the different fields, data types, or values in your collection. We built characterize_collection.pig to deeply inspect your collection to extract that information.

From the base directory of the mongo-pig-examples project you just cloned take a look at pigscripts/characterize_collection.pig. It loads all the data in the collection as a map, sends the map to Python (udfs/python/mongo_util.py) to gather a bunch of metadata, calculates some basic information about the collection, and then it writes the results out to an S3 bucket.

To see this script in action let’s run it on a 4 node Hadoop cluster. In your terminal (from the base directory of your mongo-pig-examples project) run:

    $ mortar run characterize_collection --clustersize 4

This job will take about 10 minutes to finish. You can monitor the job’s status on the command line or by going to https://app.mortardata.com/jobs.

Once the job has finished, you’ll receive an email with a link to your job results. Clicking on this link will bring you into the Mortar web app, where you can download the results from s3. The output is described at the top of the characterize_collection script but as an example you can scroll down the output and find:

    …

    user.is_translator      2     false unicode     118806

    user.is_translator      2     true  unicode     31

    user.lang   26    en    unicode     114108

    user.lang   26    es    unicode     3462

    user.lang   26    fr    unicode     532

    user.lang   26    pt    unicode     281

    user.lang   26    ja    unicode     79

    user.listed_count 398   0     int   73757

    user.listed_count 398   1     int   18518

    …

Looking at the values for user.lang - we see that there are 26 unique values for the field in our dataset. The most common was “en” with 114108 occurrences, the next most common was “es” with 3462 occurrences, and so on. To see the full results without running the job you can view the output file here.

Script 2 - Mongo Schema Generator

It can be tricky to properly declare Mongo’s highly nested schemas in Pig. Now, Pig is graceful—it can roll without a schema, or with inconsistent, or incorrect schemas. But it’s easier to read and write your Pig code if you have a schema because it allows you (and the Pig optimizer) to focus on just the relevant data.

So this next script automatically generates a Pig schema by examining your MongoDB collection. If you don’t need the whole schema, you can easily edit it to keep just the fields you want.

Running this script is similar to running the previous one. If you ran the Characterize Collection script in the past hour, the same cluster you used for that job should still be running. In that case, you can just run:

    $ mortar run mongo_schema_generator

If you don’t have a cluster that’s still running, just run the job on a new 4 node cluster like this:

    $ mortar run mongo_schema_generator --clustersize 4

Script 3 – Twitter Hourly Coffee Tweets

Using the pigscripts/hourly_coffee_tweets.pig script, we’re going to demonstrate how we can use a small subset of the fields in our MongoDB collection. For our example, we’ll look at how often the word “coffee” is tweeted throughout the day. As with the Mongo Schema Generator script, you can run this job on an existing cluster or start up a new one.

Next Steps

If you already have a mongo instance/cluster based in US-East EC2, the first two example scripts should run on one of your collections with only minor modifications. You’ll just need to:

Update the MongoLoader connection strings in the pig scripts to connect to your MongoDB collections with one of your own users. If your mongo instance is on a non-standard port (any port other than 27017), just email us at support@mortardata.com to allow your Mortar account to access that port.
If you’d like your jobs to write to one of your own S3 buckets, you can update the AWS keys associated with your Mortar account by following these instructions to enable s3 access.
If you run out of free cluster hours with Mortar, you can upgrade your account to get additional free hours each month.
You can find more resources for learning Pig here.
If you have any questions or feedback, please contact us at support@mortardata.com or, if you’re a Mortar user, ping us on in-app chat at app.mortardata.com.

Why we started the NYC Data Science Meetup

Tue, 22 Jan 2013 09:56:00 -0500

Scott Haylon

New York’s data science community has been building since long before “data science” was used to describe it. In addition to a long history of advertising and adtech companies, the recent startup explosion here in NYC has been largely led by companies built to leverage data science (including FourSquare, Tumblr, AppNexus, and Knewton, to name just a few).

This community has been fortunate to have a handful of fantastic advocates – people like Hilary Mason, Drew Conway, John Myles White, Jake Porway, and others – who have worked tirelessly to create great educational opportunities like DataGotham and amazing initiatives like DataKind. Yet our conversations with people in the data science community (and the traffic explosion resulting from our recent data education-focused post) helped us realize that people wanted even more opportunities to learn and collaborate.

So we started the NYC Data Science meetup to help meet this growing demand. Even then, we were surprised when the group grew to 160 members in just 24 hours … all without a scheduled event.

And it’s not just current or aspiring data scientists who are thirsty to learn more. Software engineers, statisticians, hackers, and many others are increasingly recognizing the value of rounding out their own skills with a broader education in data science.

Given its focus on education, we couldn’t think of a better speaker for the inaugural NYC Data Science meetup than Hilary Mason, bitly’s Chief Scientist and one of the most influential data scientists in New York (or anywhere else, for that matter). Hilary is not only a tremendous advocate for NYC’s data science community, she also has a unique gift for making challenging subject matter accessible and exciting to audiences of all experience levels.

If you love working with data and want to meet other awesome people who share your passion, sign up and join us.

7 books to supercharge your data education

Tue, 15 Jan 2013 10:04:00 -0500

Scott Haylon

Working with data is HARD. Let’s face it, you’re brave to even attempt it, let alone make it your everyday job.

Fortunately, some incredibly talented people have taken the time to compile and share their deep knowledge for you.

Here are 7 books we recommend for picking up some new skills in 2013:

Hadoop: The Definitive Guide by Tom White

Tom works at Cloudera and is one of the foremost experts on Hadoop, having been an Apache Hadoop committer since February 2007. He is a Hadoop PMC member and a member of the Apache Software Foundation.

What people have said: “A comprehensive, ‘roll up your sleeves, here’s some Java’ deep dive into Hadoop… No single book will do Hadoop justice, but this book is the best attempt so far.” (via Amazon)

Programming Pig by Alan Gates (free online version!)

Alan open-sourced Pig while at Yahoo! and later designed HCatalog. He’s currently a co-founder at Hortonworks, where he continues his extensive work on open-source projects.

What people have said: “[T]his is an excellent book that covers the details of using Pig, from basic to advanced features. It saved my bacon (if you’ll pardon the expression…) numerous times on a recent, challenging project.” (via O’Reilly)

(Bonus: We’ve compiled some additional Pig resources here.)

NoSQL Distilled by Pramod Sadalage and Martin Fowler

Pramod is as a DBA and developer at ThoughtWorks, an enterprise application development and integration company. He pioneered the practices and processes of evolutionary database design and database refactoring.

Martin is Thoughtworks’ Chief Scientist and pioneered various topics around object-oriented technology and agile methods. He’s an active speaker and author, having written six books on software development.

What people have said: “The authors of this book present a wonderful, accessible, product-agnostic introduction to the world of NoSQL… This book has demystified much of NoSQL for me and made it seem quite common-sensical.” (via Amazon)

Python for Data Analysis by Wes McKinney.

Wes is Python’s pied piper of data analysis. The MIT math major is the main developer of pandas, a Python data analysis library, and co-founder of Lambda Foundry.

What people have said: “One of the best and most practical programming books I’ve ever read. Amazing job at introducing tools (ipython, pandas) that aren’t well covered on the web.” (via O’Reilly)

Machine Learning for Hackers by Drew Conway and John Myles White

Drew is kind of a big deal in NYC’s data community: in addition to being a PhD candidate at NYU, he is IA Ventures’ “Scientist-in-Residence”, a co-organizer of Data Gotham, and co-founder of DataKind.

John is a Ph.D. candidate in the Department of Psychology at Princeton University, where he leverages his mathematical modeling and machine learning chops to understand human decision-making.

What people have said: “Drew and John have written an excellent book on presenting machine learning concepts like classification, clustering, recommendation, network graphs, and SVMs to name a few. The authors do a great job of presenting how to apply these machine learning algorithms and explain the general concepts of the algorithms.” (via Amazon)

Hadoop Operations by Eric Sammer

Eric is a principal architect at Cloudera and an active speaker on large scale data processing, integration, and system management. Prior to Cloudera, he worked at various startups for over a decade as a DBA, SysAdmin, software engineer, and system architect.

What people have said: “Whether the topic is HDFS and how data is ingested and replicated, or how Map/Reduce “finds” the most suitable node to run it’s tasks on, or what the cost and performance advantages are of adopting the shared-nothing, commodity model recommended for Hadoop clusters, etc., etc., etc., this book provides the how, what, when, where and why of Hadoop (the missing manual, of sorts).” (via Amazon)

Agile Data by Russell Jurney (free online version!)

Those who have met Russell (or followed him on Twitter) know him as a hilarious force of nature, but his data science chops are no joke. After working at Ning and LinkedIn, Russell is now Hortonworks’ Hadoop Evangelist.

What people have said: “This is definitely the best book I’ve ever written.” (Nice review, Russell…) (via O’Reilly)

Want an easy way to test your skills for free? Try Mortar’s web projects and get up-and-running quickly on Hadoop (no previous experience required).

Mortar co-founder Jeremy Karn gave this talk on using MongoDB...

Wed, 12 Dec 2012 14:28:00 -0500

Mortar co-founder Jeremy Karn gave this talk on using MongoDB data with Hadoop (and specifically with Apache Pig) at MongoSV.

Jeremy’s presentation covers the steps needed to read JSON from Mongo into Pig, parallel process it on Hadoop with sophisticated functions, and write back to Mongo.

Jeremy was a big part of our contributions to the Mongo Hadoop connector, which we extended it to make it work with Pig. MongoDB creator (and 10gen founder) Dwight Merriman also gave Mortar a nice shout out.

AWS re: Invent Startup Launch

Tue, 11 Dec 2012 18:21:00 -0500

Scott Haylon

Here are the slides and video from our public launch at AWS re: Invent, Amazon Web Services’ first-ever user conference.

We had a great time and met a lot of awesome people, including Donnie Berkholz from Redmonk, who shared his thoughts about Mortar on Twitter:

.@mortardata — Rails approach to Big Data. Opinionated open-source framework + PaaS, Python/Java. One of the coolest things at #reinvent.
— Donnie Berkholz (@dberkholz) November 30, 2012

Thanks to Donnie for the kind words and AWS for a great event. Video and slides are below the cut:

Hadoop, Pig, and Python at PyData

Thu, 06 Dec 2012 17:49:00 -0500

Scott Haylon

Our CEO, K Young, spoke at PyData NYC abpit using real Python with Pig, and why we integrated these two awesome languages. The audience asked some great questions, many of which you can see at the end of the video.

Here is the video (with slides just below):

MongoDB + Pig talk at MongoSV

Sat, 01 Dec 2012 10:49:00 -0500

Scott Haylon

Mortar is pleased to sponsor MongoSV on December 4^th, an annual one-day conference in Silicon Valley, CA dedicated to the open source, non-relational database MongoDB.

Our very own lead engineer, Jeremy Karn, will deliver a talk entitled MongoDB + Pig, which will teach attendees how to process MongoDB data with Hadoop—specifically with Apache Pig. As many of you know, we’ve committed to the MongoDB+Hadoop connector, extending it to work with Pig. Jeremy’s talk will cover the steps needed to read JSON from Mongo into Pig, parallel process it on Hadoop with sophisticated functions, and write back to Mongo.

For more information about MongoSV, check out the agenda or 10gen’s blog post Get Ready for MongoSV. And if you want to save a few bucks, enter the discount code “mortar20” and get 20% off.

Announcing our public launch

Wed, 28 Nov 2012 07:00:00 -0500

Scott Haylon

Last week, we announced our $1.8 million fundraising. For those of you who follow big data startups, our blog post probably felt…underwhelming. Startups typically come out and make a huge publicity splash, jam-packed with buzzwords and vision galore. While we feel very fortunate to have what we need to help us grow, we know that VC funding is merely a means, and not an end.

But now you get to see us get really excited, because Mortar’s Hadoop PaaS and open source framework for big data is now publicly available. This means if you want to try it, you can activate your trial right now on our site without having to talk to anyone (unless you want to!).

You can get started on Mortar using Web Projects (using Mortar entirely online through the browser) or Git Projects (using Mortar locally on your own machine with the Mortar development framework). You can see more info about both here.

All trial accounts come with our full Hadoop PaaS, unlimited use of the Mortar framework, our site, and dev tools, and 10 free Hadoop node-hours. (You can get another 15 free node-hours per month and additional support at no cost by simply adding your credit card to the account.)

Where we started…

Our team has been banging our heads against big data for a long time. We’ve felt the pain of cumbersome ETL systems and other tools, and we were so excited about the potential of Hadoop. However when we started using it, we quickly realized it was really only accessible to large companies with deep pockets and big teams, and even then it took a VERY long time to get from “Hey, we want to use Hadoop!” to “Sweet! We’re using Hadoop!”

Our first iteration of Mortar was a browser-based version that made it easy to write jobs on our Hadoop PaaS using Apache Pig and Python. It was a great way for us (and our users) to get started quickly, but we always had a bigger vision, which we’re just now bringing to the public.

…and what’s new.

Mortar is an open source, code-based platform for big data. As a company, our Hadoop PaaS hosts and executes Mortar projects as a service. We’ve partnered with Amazon Web Services and built Mortar entirely on AWS, backed by their Elastic MapReduce offering.

You use Mortar by writing a combination of Pig (which is like SQL) and real Python. Historically, Hadoop has only worked with Jython, but the data science community told us over and over how deeply they depended on libraries like NumPy, SciPy, and NLTK. They were dying to use Hadoop, but they were being forced to choose between one set of tools or another. We wanted to fix this, so we made Hadoop and Python play nice together.

Our key focuses in building Mortar’s Hadoop PaaS are:

Ease of use – engineers and data scientists can get started quickly using the tools they love, without training
Collaboration – share/repeat/maintain your code using Git or other revision control
Open source – customers should never be locked in
Convention over configuration – do automated testing, find errors quickly
Removing all non-core elements of building data pipelines – you shouldn’t waste time worrying about infrastructure and operations

Does it work for you?

I could write at length about how Mortar works, but you’re better off seeing it in action. Of course, the easiest way to really know if any software works for you is to try it for yourself. So if you want to do ETL, natural language processing, aggregations, machine learning, regression analysis, or some other big data analysis, you can try Mortar free.