Проблема с MapReduce в среде Hadoop

Осваиваю вычислительную парадигму MapReduce в среде Hadoop. Я создал два файла Python, содержащие трансформатор и редуктор.

with open('mapper_hadoop.py', 'w') as fh:
    fh.write("""#!/usr/bin/env python

import sys

for line in sys.stdin:
    print "chars", len(line.rstrip('\\n'))
    print "words", len(line.split())
    print "lines", 1
    """)
with open('reducer_hadoop.py', 'w') as fh:
    fh.write("""#!/usr/bin/env python

import sys

counts = {"chars": 0, "words": 0, "lines": 0}
for line in sys.stdin:
    kv = line.rstrip().split()
    counts[kv[0]] += int(kv[1])

for k,v in counts.items():
    print k,v
    """)

Выставил права на файлы, а затем попробовал запустить эти два сценария локально, не используя Hadoop. Сценарии оказались рабочие. После этого я захотел воспользоваться вычислительной парадигмой MapReduce в Hadoop, чтобы посмотреть результат работы двух сценариев. В Jupyter я выполнил следующий код:

!hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.6.4.jar \
        -files mapper_hadoop.py,reducer_hadoop.py \
        -mapper mapper_hadoop.py -reducer reducer_hadoop.py \
        -input hadoop_git_readme.txt \
        -output /tmp/mr.out

В результате получил следующую ошибку:

packageJobJar: [/tmp/hadoop-unjar8018849492984741801/] [] /tmp/streamjob7225804067539527250.jar tmpDir=null
20/11/12 18:44:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/12 18:44:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
20/11/12 18:44:05 INFO mapred.FileInputFormat: Total input paths to process : 1
20/11/12 18:44:05 INFO mapreduce.JobSubmitter: number of splits:2
20/11/12 18:44:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1605187077393_0005
20/11/12 18:44:06 INFO impl.YarnClientImpl: Submitted application application_1605187077393_0005
20/11/12 18:44:06 INFO mapreduce.Job: The url to track the job: http://sparkbox:8088/proxy/application_1605187077393_0005/
20/11/12 18:44:06 INFO mapreduce.Job: Running job: job_1605187077393_0005
20/11/12 18:44:17 INFO mapreduce.Job: Job job_1605187077393_0005 running in uber mode : false
20/11/12 18:44:17 INFO mapreduce.Job:  map 0% reduce 0%
20/11/12 18:44:17 INFO mapreduce.Job: Job job_1605187077393_0005 failed with state FAILED due to: Application application_1605187077393_0005 failed 2 times due to AM Container for appattempt_1605187077393_0005_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://sparkbox:8088/proxy/application_1605187077393_0005/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1605187077393_0005_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
20/11/12 18:44:17 INFO mapreduce.Job: Counters: 0
20/11/12 18:44:17 ERROR streaming.StreamJob: Job not successful!
Streaming Command Failed!

В чем заключается проблема? Страничка с логом "http://sparkbox:8088/proxy/application_1605187077393_0005/Then" не открывается.


Ответы (0 шт):