Past Details

Self-Tuning Hadoop for Big Data Analytics

Date / Time:
24/02/2014 Download ICS File
Location:
Carnegie Mellon Qatar, room 1202
Category:
Lectures and Activities
QCRI-Logo-new-2016.jpg Dr. Ashraf Aboulnaga is a Principal Scientist in the Qatar Computing Research Institute, where he heads the Distributed Systems group. His current research interests are in the area of data intensive cloud computing, with a current focus on platforms for Big Data, distributed storage systems, and data integration on the web. Dr. Aboulnaga obtained M.S. and Ph.D. degrees from the University of Wisconsin - Madison, and B.S. and M.S. degrees from Alexandria University, Egypt. He holds an Associate Professor position at the University of Waterloo in Canada, where he has worked since 2004.


Hadoop has become one of the most popular platforms for Big Data analytics, enabling flexible and powerful analysis of massive data sets on commodity computing clusters. However, tuning and administering Hadoop clusters is still not a well-studied area. In this talk, I will present two projects that aim to make Hadoop more self-tuning. The first is ReStore, a system for reusing MapReduce results. ReStore focuses on queries that are expressed in high-level query languages such as Pig Latin or Hive. The compilers of these languages translate queries into workflows of MapReduce jobs. Each job in these workflows reads its input from the Hadoop file system (HDFS) and produces output that is stored in HDFS.  The current practice is to delete all intermediate results from HDFS at the end of executing the workflow. ReStore keeps these intermediate results and reuses them for future queries, resulting in significant performance improvement. I will also present PStorM, a system for tuning Hadoop configuration parameters based on the history of previously observed MapReduce jobs. The two projects presented in the talk share the goal of automatically improving the performance of Hadoop based on workload characteristics.

Follow Us

  • YouTube
  • Twitter
  • Facebook
  • RSS Feed
  • Linkedin
  • github-web.png
Back to Top

In the Media

roadtracer.png

MIT/QCRI system uses machine learning to build road maps

22/04/2018

Map apps may have changed our world, but they still haven’t mapped all of it yet. Specifically, mapping roads can be difficult and tedious: even after taking aerial images, companies still have to ...

Read More

Economist story pic.JPG

Improving disaster response efforts through data

08/02/2018

Extreme weather events put the most vulnerable communities at high risk. How can data analytics strengthen early warning systems and and support relief efforts for communities in need? The size and ...

Read More

Yazan Wired story pic.jpg

Your sloppy bitcoin drug deals will haunt you for years

28/01/2018

Perhaps you bought some illegal narcotics on the Silk Road half a decade ago, back when that digital black market for every contraband imaginable was still online and bustling. You might already ...

Read More

Events

Past Events

2018

Eman interns pic 2017.jpg

QCRI Summer Internship Program

Download ICS File 06/05/2018  - 05/07/2018 , Hamad Bin Khalifa Research Complex

Each year, Qatar Computing Research Institute organizes a summer internship program for undergraduate students studying computer science, computer engineering and other disciplines. The internship is unpaid, and QCRI does not provide any visa support.

Read More

Regina

Public Talk by Prof. Regina Barzilay "Artificial Intelligence for Oncology: Learning to Cure Cancer from Images and Text"

Download ICS File 27/03/2018 ,

Artificial Intelligence for Oncology: Learning to Cure Cancer from Images and Text A talk by Professor Regina Barzilay, MIT CSAIL Winner of 2017 MacArthur ‘genius grant’ At Education City Student ...

Read More

Slide1.JPG

QCRI & MIT-CSAIL Annual Project Review 2018

Download ICS File 27/03/2018 ,

Executive Overview Sessions Open to public Date:    Tuesday, March 27, 2018 Time:    9:00AM – 3:00PM Venue:  HBKU Research Complex Multipurpose Room To view full agenda, please click here . To RSVP, ...

Read More

News

Cozmo camp.jpg

QCRI announces 2018 summer camps for kids

09/07/2018

Register soon as places fill quickly.

Read More

ahmed pic.JPG

QCRI to launch Qatar Center for Artificial Intelligence (QCAI)

14/05/2018

Initiative plans to ramp up development of artificial intelligence technologies to position Qatar as a global leader in AI.

Read More

humdata15.jpg

QCRI hosts humanitarian data skills workshop

09/05/2018

Joint event with UNOCHA's Centre for Humanitarian Data, the Qatar Red Crescent Society and Education Above All Foundation held to devise common practices for data use in humanitarian action.

Read More