Hadoop实战-PHP-MapReduce
admin 发布于:2018-6-8 8:51 分类:系统架构 有 2188 人浏览,获得评论 0 条 标签: Hadoop MapReduce
1. 编写Mapper的代码
#vi WordMap.php
#!/usr/bin/php <?php while (($line = fgets(STDIN)) !== false) { $words = preg_split('/(\s+)/', $line); foreach ($words as $word) { echo $word."\t"."1".PHP_EOL; } } ?>
2.编写Reducer的代码
#vi WordReduce.php
#!/usr/bin/php <?php $result=array(); while (($line = fgets(STDIN)) !== false) { $line = trim($line); list($k, $v) = explode("\t", $line); $result[$k] += $v; } ksort($result); foreach($result as $word => $count) { echo $word."\t".$count.PHP_EOL; } ?>
3.运行WordMapReduce
#chmod 0777 WordMap.php #chmod 0777 WordReduce.php #bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.9.1.jar -mapper WordMap.php -reducer WordReduce.php -input HdfsInput/* -output HdfsOutput
4.查看运行结果
#hadoop fs -ls HdfsOutput #hadoop fs -cat HdfsOutput/* #hadoop fs -get HdfsOutput LocalOutput #cat LocalOutput/*