Friday Hacks #70, April 11

At Viki we collect and analyze over 60M user-generated data points everyday. This talk goes into deep technical and architecture design to share about the infrastructure, the different tools and process we’ve built with our analytics system. Specifically we’ll talk about how we collect data, how we process data and how we present data.

The current system is capable of supporting a variety of business and product needs: automated business reporting, a/b testing, cohort analysis, funnel analysis, recommendation engine, etc.

Technology stack: Ruby, Python, R, NodeJS, Golang, PostgreSQL, Java, Storm, Hadoop with Apache Hive.

We are also looking for summer interns – http://bit.ly/viki-summer-internship .

We will talk more about our summer internship offers for students, what a typical intern’s life looks like and what works our past interns have done at the end of the talk.

Speaker Profile

Jason Grendus is Director of Analytics at Viki (a Rakuten company), a global TV site that streams TV shows, movies and other premium content. Jason has many years of experience developing database solutions to solve business and financial analysis problems, with a depth of experience in many database and data warehouse technologies. In his current role, Jason’s primary focus is the scalability of Viki’s analytics and reporting systems. Jason made his first foray into large-scale analytics / big data as the first analytics engineer hired at Slide (acquired by Google for $200m), where his role evolved to encompass load balancing the data pipeline for 1TB of logging data per day across a distributed network of servers.Ishan Agrawal works as a software engineer at Viki in the Data team, exploring and learning about building scalable data infrastructure. Previously he interned at Paypal Labs researching on new sound based technologies. He also spent a year at NOC Silicon Valley working for Rocket Fuel Inc. building their real time bidding system.

Talk 2: ElasticSearch and the cloud – data analysis and visualization.

Talk Description:

Harness the power of the cloud and ElasticSearch to create a flexible and scalable distributed infrastructure to store, analyze, and visualize large amounts of data! I will share my experience working with ElasticSearch and discuss various setup and configuration options. This talk will be useful for people who are relatively new to ElasticSearch and are interested in visualizing their application logs.

Visit http://www.elasticsearch.org/ and get familiarized with elasticsearch, kibana, and logstash. Download these packages if you wish to setup your own installation, along with a recent Java Runtime and any web server (nginx, or node http-server)

Kok Hong has been with PayPal since 2011, working on analyzing and improving the performance of mid-tier services to read and manage users’ financial instrument information. He’s been working with Linux systems since 1997! More information is available at his LinkedIn profile: http://www.linkedin.com/pub/kokhong-cheng/4/973/411