如何部署Apache Hadoop 2.2.0 Eclipse插件

日期: 2014-08-27 作者:Abhijit Ghosh翻译:Ranma 来源:TechTarget中国 英文

Hadoop DevelopmentTools(HDT)是针对Hadoop平台开发而推出的Eclipse IDE的一套插件。目前我们正在从Apache Hadoop项目移植现有的MapReduce工具以使一个IDE中多个版本的Hadoop可以协同工作。

然而,Apache Hadoop Development Tools (HDT) 仍然处在开发阶段。因此,现在Hadoop2.2.0 Eclipse插件还没有正式的官方版本放出。但是我们可以用winghc/hadoop2x-eclipse-plugin搭建出相同的环境。在本文中,我们会为Eclipse或任何基于Eclipse的IDE(如SpringTool Suite)搭建,安装并配置插件来缓解使用Hadoop框架的开发活动。

本文中我们所用到的工具和技术:

1.    Apache Hadoop 2.2.0

2.    Spring Tool Suite 3.5.0

3.    Windows7 OS

4.    JDK 1.6

5.    ApacheAnt

1.下载

下载winghc/hadoop2x-eclipse-plugin 压缩包。

2.提取

提取到本地路径(如“C:hadoop2x-eclipse-plugin”)。

3.搭建

在命令行窗口中打开'<hadoop2x-eclipse-plugin-directory>srccontribeclipse-plugin’ 

C:>cd C:hadoop2x-eclipse-pluginsrccontribeclipse-plugin

运行ANT搭建

C:hadoop2x-eclipse-pluginsrccontribeclipse-plugin>ant jar -Dversion=2.2.0 -Declipse.home=C:/IDE/sts-3.5.0 -Dhadoop.home=c:/hadoopBuildfile: C:hadoop2x-eclipse-pluginsrccontribeclipse-pluginbuild.xml

check-contrib:init:     [echo] contrib: eclipse-plugin    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-plugin    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginclasses    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-plugintest    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginsystem    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginsystemclasses    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginexamples    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-plugintestlogsinit-contrib:ivy-download:      [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar      [get] To: C:hadoop2x-eclipse-pluginivyivy-2.1.0.jarivy-probe-antlib:ivy-init-antlib:ivy-init:[ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::[ivy:configure] :: loading settings :: file = C:hadoop2x-eclipse-pluginivyivysettings.xmlivy-resolve-common:ivy-retrieve-common:[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead[ivy:cachepath] :: loading settings :: file = C:hadoop2x-eclipse-pluginivyivysettings.xmlcompile:     [echo] contrib: eclipse-plugin    [javac] C:hadoop2x-eclipse-pluginsrccontribeclipse-pluginbuild.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds    [javac] Compiling 45 source files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginclasses    [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found    [javac] c:hadoopsharehadoophdfshadoop-hdfs-2.2.0.jar(org/apache/hadoop/hdfs/DistributedFileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'    [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'    [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'    [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'    [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'    [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate'    [javac] Note: Some input files use or override a deprecated API.    [javac] Note: Recompile with -Xlint:deprecation for details.    [javac] Note: Some input files use unchecked or unsafe operations.    [javac] Note: Recompile with -Xlint:unchecked for details.    [javac] 7 warningsjar:    [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying 9 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-app-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-app-2.2.0.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-common-2.2.0.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-core-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-core-2.2.0.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-hs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-hs-2.2.0.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-hs-plugins-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-hs-plugins-2.2.0.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-jobclient-2.2.0-tests.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-jobclient-2.2.0-tests.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-jobclient-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-jobclient-2.2.0.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-shuffle-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-shuffle-2.2.0.jar     [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-examples-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-examples-2.2.0.jar     [copy] Copying 3 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonhadoop-common-2.2.0-tests.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-common-2.2.0-tests.jar     [copy] Copying c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-common-2.2.0.jar     [copy] Copying c:hadoopsharehadoopcommonhadoop-nfs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-nfs-2.2.0.jar     [copy] Copying 3 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoophdfshadoop-hdfs-2.2.0-tests.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-hdfs-2.2.0-tests.jar     [copy] Copying c:hadoopsharehadoophdfshadoop-hdfs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-hdfs-2.2.0.jar     [copy] Copying c:hadoopsharehadoophdfshadoop-hdfs-nfs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-hdfs-nfs-2.2.0.jar     [copy] Copying 11 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-api-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-api-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-applications-distributedshell-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-applications-distributedshell-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-client-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-client-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-common-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-common-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-nodemanager-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-nodemanager-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-resourcemanager-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-resourcemanager-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-tests-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-tests-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-web-proxy-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-web-proxy-2.2.0.jar     [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-site-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-site-2.2.0.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibprotobuf-java-2.5.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibprotobuf-java-2.5.0.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonliblog4j-1.2.17.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginliblog4j-1.2.17.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibcommons-cli-1.2.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibcommons-cli-1.2.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibcommons-configuration-1.6.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibcommons-configuration-1.6.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibcommons-lang-2.5.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibcommons-lang-2.5.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibjackson-core-asl-1.8.8.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibjackson-core-asl-1.8.8.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibjackson-mapper-asl-1.8.8.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibjackson-mapper-asl-1.8.8.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibslf4j-log4j12-1.7.5.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibslf4j-log4j12-1.7.5.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibslf4j-api-1.7.5.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibslf4j-api-1.7.5.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibguava-11.0.2.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibguava-11.0.2.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibhadoop-auth-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-auth-2.2.0.jar     [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib     [copy] Copying c:hadoopsharehadoopcommonlibnetty-3.6.2.Final.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibnetty-3.6.2.Final.jar      [jar] Building jar: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginhadoop-eclipse-plugin-2.2.0.jarBUILD SUCCESSFULTotal time: 42 seconds

eclipse.home:

Eclipse IDE的安装路径。

hadoop.home:

Hadoop 安装路径。

注意:下载“ivy-2.1.0.jar”需要有网络连接

4. 安装

在搭建成功之后,'srccontribeclipse-plugin'.内会生成'hadoop-eclipse-plugin-2.2.0.jar' 。拷贝此jar文件并将其粘贴到你IDE的“插件”所在路径。

5. 配置

如果Eclipse是开启状态就重新启动。否则只要启动就可以了。

a. 进入Windows-->Open Perspective-->Other并选择“Map/Reduce”。

b. 点击“New Hadoop location...”(蓝色大象图标)并定义Hadoop路径来运行MapReduce应用程序。点击“Finish”按钮。

Map/Reduce(V2) Master:

Map/Reduce主节点地址(任务追踪器)

DFS Master:

分布式文件系统主节点地址(名称节点)。

要了解“Port”数量的话,就启动Hadoop并在浏览器中打开http://localhost:8088/cluster。点击Tools --> Configuration 并查找以下属性。

DFS Master

			<property>				<name>fs.defaultFS</name>				<value>hdfs://localhost:9000</value>				<source>core-site.xml</source>			</property>		

Map/Reduce(V2) Master

			<property>				<name>mapreduce.jobtracker.http.address</name>				<value>0.0.0.0:50030</value>				<source>mapred-default.xml</source>			</property>

c. 现在我们可以浏览Hadoop文件系统并只使用GUI来执行不同的文件/文件夹操作。

同样,我们还可以用向导(File --> New --> Other... --> Map/Reduce)轻松创建Map/Reduce Project,Mapper,Reducer以及MapReduce Driver并进入Hadoop编程。

我们一直都在努力坚持原创.......请不要一声不吭,就悄悄拿走。

我原创,你原创,我们的内容世界才会更加精彩!

【所有原创内容版权均属TechTarget,欢迎大家转发分享。但未经授权,严禁任何媒体(平面媒体、网络媒体、自媒体等)以及微信公众号复制、转载、摘编或以其他方式进行使用。】

微信公众号

TechTarget微信公众号二维码

TechTarget

官方微博

TechTarget中国官方微博二维码

TechTarget中国

电子邮件地址不会被公开。 必填项已用*标注

敬请读者发表评论,本站保留删除与本文无关和不雅评论的权力。

翻译

Ranma
Ranma

相关推荐