Hadoop实战-PHP-MapReduce
admin 发布于:2018-6-8 8:51 分类:系统架构 有 3192 人浏览,获得评论 0 条 标签: Hadoop MapReduce
1. 编写Mapper的代码
#vi WordMap.php
#!/usr/bin/php
<?php
while (($line = fgets(STDIN)) !== false)
{
$words = preg_split('/(\s+)/', $line);
foreach ($words as $word)
{
echo $word."\t"."1".PHP_EOL;
}
}
?>
2.编写Reducer的代码
#vi WordReduce.php
#!/usr/bin/php
<?php
$result=array();
while (($line = fgets(STDIN)) !== false)
{
$line = trim($line);
list($k, $v) = explode("\t", $line);
$result[$k] += $v;
}
ksort($result);
foreach($result as $word => $count)
{
echo $word."\t".$count.PHP_EOL;
}
?>
3.运行WordMapReduce
#chmod 0777 WordMap.php #chmod 0777 WordReduce.php #bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.9.1.jar -mapper WordMap.php -reducer WordReduce.php -input HdfsInput/* -output HdfsOutput
4.查看运行结果
#hadoop fs -ls HdfsOutput #hadoop fs -cat HdfsOutput/* #hadoop fs -get HdfsOutput LocalOutput #cat LocalOutput/*
