`
zhangyafei_kimi
  • 浏览: 261561 次
  • 性别: Icon_minigender_1
  • 来自: 武汉
社区版块
存档分类
最新评论

Why Map/Reduce?

阅读更多
从Google在OSDI'04上发表了Map/Reduce这篇Paper;到Yahoo!在06年雇佣Doug Cotting,
投入对Hadoop最大的关注,以期提高其计算能力对抗Google;再到CMU CS的系主任发表Tec
h report提出DISC(Data-Intensive Super Computing,在最近一次的Hadoop summit上,
他演讲的slides将Super改成了Scalable)的概念,以及Intel、IBM与Stanford、CMU合作
基于Hadoop搞云计算。Map/Reduce引入的并行计算模型得到了前所未有的关注和应用。

究竟什么是Map/Reduce,为什么需要Map/Reduce,孟岩(此孟岩非彼孟岩,此孟岩是财帮
子的创始人之一)的这篇文章:Map Reduce - the Free Lunch is not over?阐述得很清
楚。(http://www.mengyan.org/blog/archives/2006/11/15/138.html)

随着du-core,multi-core甚至Stanford在做的1024核的出现,我们的Programming Model
会慢慢从OOP转到COP(concurrency-oriented programming)上来。而如何抽象这种progr
amming model的转化,让程序员更加方便的编写并行化程序利用多核甚至分布式系统的计
算能力,成为了很多研究的热点。Map/Reduce就是在这方面做出了不错的努力,这是它的
价值!

于是,在Map/Reduce发表没多久中,很多基于multi-core甚至GPU上的研究也在火热的展开


Phoenix: from Stanford

"Evaluating MapReduce for Multi-core and Multiprocessor Systems" —
paper by Colby Ranger, Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, and
Christos Kozyrakis; from Stanford University

很有趣的是,这篇在当年会议上被推荐为best paper的论文居然出自于Stanford开的一门
课”Advanced Processor Design”上的project。(强大的stanford学生啊!)

Mars:from Hongkong University Science and Technology
(GPGPU很猖獗啊~)

# "Mars: A MapReduce Framework on Graphics Processors" — paper by Bingsheng
He, Wenbin Fang, Qiong Luo, Naga K. Govindaraju, Tuyong Wang; from Hong Kong
University of Science and Technology

Cell B.E. Architecture: from University of Wisconsin-Madison(不愧是CS
Architecture No 1的学校啊!)

# "MapReduce for the Cell B.E. Architecture" — paper by Marc de Kruijf and
Karthikeyan Sankaralingam; from University of Wisconsin-Madison

此外,还有一些项目也在应用Map/Reduce的思想。

QtConcurrent :http://labs.trolltech.com/page/Projects/Threads/QtCo
ncurrent
Open Source C++ MapReduce (non-distributed) implementation from Trolltech

Skynet:http://skynet.rubyforge.org/
Ruby Map/Reduce Framework

最牛的不是语言,不是代码,而是思想!还有Simplified的哲学!
学术界的力量也是不可低估的,很多很神奇很美妙的东西都来自于学术界。大家要好好珍
惜还在学习、做研究的机会。

Thanks to map/Reduce!
Thanks to lisp&other fp!
Thanks to lambda calculus!
Thanks to Alonzo Church!
分享到:
评论
1 楼 gjlzjb 2011-11-07  
非常有用,谢谢哈。
另外问下,您是否用过Pheonix System(MapReduce),最近在尝试,不过老出问题,因为我对P-thread不熟悉~~

相关推荐

    大数据云计算技术 优酷网Hadoop及Mapreduce入门教程(共35页).pptx

    Map-reduce Programming Paradigm Hadoop Map-reduce Job Scheduler Resources Hadoop, Why? 数据太多了,需要能存储、快速分析Pb级数据集的系统 单机的存储、IO、内存、CPU有限,需要可扩展的集群 使用门槛低,...

    利用micemeat实现mapreduce

    The assignment is to write a parallel map-reduce program for the above task using either octo.py, or mincemeat.py, each of which is a lightweight map-reduce implementation written in Python. ...

    Java 8 Lambdas

    reduce 28 Putting Operations Together 30 Refactoring Legacy Code 31 iii Multiple Stream Calls 34 Higher-Order Functions 36 Good Use of Lambda Expressions 36 Key Points 37 Exercises 37 Advanced ...

    Data-Intensive Text Processing with MapReduce Jimmy Lin and Chris Dyer

    1.3 Why Is This Dierent? 1.4 What This Book Is Not 2 MapReduce Basics 2.1 Functional Programming Roots 2.2 Mappers and Reducers 2.3 The Execution Framework 2.4 Partitioners and Combiners 2.5 The ...

    ComponentOne Studio for Silverlight 2012 v3 1/3

    This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...

    ComponentOne Studio for Silverlight 2012 v3 3/3

    This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...

    ComponentOne Studio for Silverlight 2012 v3 2/3

    This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...

    ComponentOne 2012 V2 Studio for Silverlight 3/3

    This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...

    ComponentOne 2012 V2 Studio for Silverlight 2/3

    This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...

    Functional Programming in R

    Use Filer, Map and Reduce functions to express the intent behind code clearly and safely Build new functions from existing functions without necessarily writing any new functions, using point-free ...

    Beginning Python (2005).pdf

    Why Text Processing Is So Useful 175 Searching for Files 176 Clipping Logs 177 Sifting through Mail 178 Navigating the File System with the os Module 178 Try It Out: Listing Files and Playing ...

    ComponentOne 2012 V2 Studio for Silverlight(完整安装)

    This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...

    Dependency Injection Principles, Practices, and Patterns

    PART 1 Putting Dependency Injection on the map Chapter 1. The Basics Of Dependency Injection: What, Why, And How Chapter 2. Writing Tightly Coupled Code Chapter 3. Writing Loosely Coupled Code PART 2 ...

    大数据分析

    1. Introduction: Why Look Beyond Hadoop Map-Reduce?.........................................14 Hadoop Suitability..........................................................................................

Global site tag (gtag.js) - Google Analytics