从Google在OSDI'04上发表了Map/Reduce这篇Paper;到Yahoo!在06年雇佣Doug Cotting,
投入对Hadoop最大的关注,以期提高其计算能力对抗Google;再到CMU CS的系主任发表Tec
h report提出DISC(Data-Intensive Super Computing,在最近一次的Hadoop summit上,
他演讲的slides将Super改成了Scalable)的概念,以及Intel、IBM与Stanford、CMU合作
基于Hadoop搞云计算。Map/Reduce引入的并行计算模型得到了前所未有的关注和应用。
究竟什么是Map/Reduce,为什么需要Map/Reduce,孟岩(此孟岩非彼孟岩,此孟岩是财帮
子的创始人之一)的这篇文章:Map Reduce - the Free Lunch is not over?阐述得很清
楚。(http://www.mengyan.org/blog/archives/2006/11/15/138.html)
随着du-core,multi-core甚至Stanford在做的1024核的出现,我们的Programming Model
会慢慢从OOP转到COP(concurrency-oriented programming)上来。而如何抽象这种progr
amming model的转化,让程序员更加方便的编写并行化程序利用多核甚至分布式系统的计
算能力,成为了很多研究的热点。Map/Reduce就是在这方面做出了不错的努力,这是它的
价值!
于是,在Map/Reduce发表没多久中,很多基于multi-core甚至GPU上的研究也在火热的展开
。
Phoenix: from Stanford
"Evaluating MapReduce for Multi-core and Multiprocessor Systems" —
paper by Colby Ranger, Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, and
Christos Kozyrakis; from Stanford University
很有趣的是,这篇在当年会议上被推荐为best paper的论文居然出自于Stanford开的一门
课”Advanced Processor Design”上的project。(强大的stanford学生啊!)
Mars:from Hongkong University Science and Technology
(GPGPU很猖獗啊~)
# "Mars: A MapReduce Framework on Graphics Processors" — paper by Bingsheng
He, Wenbin Fang, Qiong Luo, Naga K. Govindaraju, Tuyong Wang; from Hong Kong
University of Science and Technology
Cell B.E. Architecture: from University of Wisconsin-Madison(不愧是CS
Architecture No 1的学校啊!)
# "MapReduce for the Cell B.E. Architecture" — paper by Marc de Kruijf and
Karthikeyan Sankaralingam; from University of Wisconsin-Madison
此外,还有一些项目也在应用Map/Reduce的思想。
QtConcurrent :http://labs.trolltech.com/page/Projects/Threads/QtCo
ncurrent
Open Source C++ MapReduce (non-distributed) implementation from Trolltech
Skynet:http://skynet.rubyforge.org/
Ruby Map/Reduce Framework
最牛的不是语言,不是代码,而是思想!还有Simplified的哲学!
学术界的力量也是不可低估的,很多很神奇很美妙的东西都来自于学术界。大家要好好珍
惜还在学习、做研究的机会。
Thanks to map/Reduce!
Thanks to lisp&other fp!
Thanks to lambda calculus!
Thanks to Alonzo Church!
分享到:
相关推荐
Map-reduce Programming Paradigm Hadoop Map-reduce Job Scheduler Resources Hadoop, Why? 数据太多了,需要能存储、快速分析Pb级数据集的系统 单机的存储、IO、内存、CPU有限,需要可扩展的集群 使用门槛低,...
The assignment is to write a parallel map-reduce program for the above task using either octo.py, or mincemeat.py, each of which is a lightweight map-reduce implementation written in Python. ...
reduce 28 Putting Operations Together 30 Refactoring Legacy Code 31 iii Multiple Stream Calls 34 Higher-Order Functions 36 Good Use of Lambda Expressions 36 Key Points 37 Exercises 37 Advanced ...
1.3 Why Is This Dierent? 1.4 What This Book Is Not 2 MapReduce Basics 2.1 Functional Programming Roots 2.2 Mappers and Reducers 2.3 The Execution Framework 2.4 Partitioners and Combiners 2.5 The ...
This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...
This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...
This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...
This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...
This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...
Use Filer, Map and Reduce functions to express the intent behind code clearly and safely Build new functions from existing functions without necessarily writing any new functions, using point-free ...
Why Text Processing Is So Useful 175 Searching for Files 176 Clipping Logs 177 Sifting through Mail 178 Navigating the File System with the os Module 178 Try It Out: Listing Files and Playing ...
This will reduce development time, cost and eliminate the learning curve of having to learn controls from multiple platforms. You can also use your XAML skills to develop LightSwitch extensions and ...
PART 1 Putting Dependency Injection on the map Chapter 1. The Basics Of Dependency Injection: What, Why, And How Chapter 2. Writing Tightly Coupled Code Chapter 3. Writing Loosely Coupled Code PART 2 ...
1. Introduction: Why Look Beyond Hadoop Map-Reduce?.........................................14 Hadoop Suitability..........................................................................................