Проблема с MapReduce в среде Hadoop
Осваиваю вычислительную парадигму MapReduce в среде Hadoop. Я создал два файла Python, содержащие трансформатор и редуктор.
with open('mapper_hadoop.py', 'w') as fh:
fh.write("""#!/usr/bin/env python
import sys
for line in sys.stdin:
print "chars", len(line.rstrip('\\n'))
print "words", len(line.split())
print "lines", 1
""")
with open('reducer_hadoop.py', 'w') as fh:
fh.write("""#!/usr/bin/env python
import sys
counts = {"chars": 0, "words": 0, "lines": 0}
for line in sys.stdin:
kv = line.rstrip().split()
counts[kv[0]] += int(kv[1])
for k,v in counts.items():
print k,v
""")
Выставил права на файлы, а затем попробовал запустить эти два сценария локально, не используя Hadoop. Сценарии оказались рабочие. После этого я захотел воспользоваться вычислительной парадигмой MapReduce в Hadoop, чтобы посмотреть результат работы двух сценариев. В Jupyter я выполнил следующий код:
!hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.4.jar \
-files mapper_hadoop.py,reducer_hadoop.py \
-mapper mapper_hadoop.py -reducer reducer_hadoop.py \
-input hadoop_git_readme.txt \
-output /tmp/mr.out
В результате получил следующую ошибку:
packageJobJar: [/tmp/hadoop-unjar8018849492984741801/] [] /tmp/streamjob7225804067539527250.jar tmpDir=null
20/11/12 18:44:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/12 18:44:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/12 18:44:05 INFO mapred.FileInputFormat: Total input paths to process : 1
20/11/12 18:44:05 INFO mapreduce.JobSubmitter: number of splits:2
20/11/12 18:44:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1605187077393_0005
20/11/12 18:44:06 INFO impl.YarnClientImpl: Submitted application application_1605187077393_0005
20/11/12 18:44:06 INFO mapreduce.Job: The url to track the job: http://sparkbox:8088/proxy/application_1605187077393_0005/
20/11/12 18:44:06 INFO mapreduce.Job: Running job: job_1605187077393_0005
20/11/12 18:44:17 INFO mapreduce.Job: Job job_1605187077393_0005 running in uber mode : false
20/11/12 18:44:17 INFO mapreduce.Job: map 0% reduce 0%
20/11/12 18:44:17 INFO mapreduce.Job: Job job_1605187077393_0005 failed with state FAILED due to: Application application_1605187077393_0005 failed 2 times due to AM Container for appattempt_1605187077393_0005_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://sparkbox:8088/proxy/application_1605187077393_0005/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1605187077393_0005_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
20/11/12 18:44:17 INFO mapreduce.Job: Counters: 0
20/11/12 18:44:17 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!
В чем заключается проблема? Страничка с логом "http://sparkbox:8088/proxy/application_1605187077393_0005/Then" не открывается.