Hadoop DevelopmentTools(HDT)是针对Hadoop平台开发而推出的Eclipse IDE的一套插件。目前我们正在从Apache Hadoop项目移植现有的MapReduce工具以使一个IDE中多个版本的Hadoop可以协同工作。
然而,Apache Hadoop Development Tools (HDT) 仍然处在开发阶段。因此,现在Hadoop2.2.0 Eclipse插件还没有正式的官方版本放出。但是我们可以用winghc/hadoop2x-eclipse-plugin搭建出相同的环境。在本文中,我们会为Eclipse或任何基于Eclipse的IDE(如SpringTool Suite)搭建,安装并配置插件来缓解使用Hadoop框架的开发活动。
本文中我们所用到的工具和技术:
3. Windows7 OS
4. JDK 1.6
5. ApacheAnt
1.下载
下载winghc/hadoop2x-eclipse-plugin 压缩包。
2.提取
提取到本地路径(如“C:hadoop2x-eclipse-plugin”)。
3.搭建
在命令行窗口中打开'<hadoop2x-eclipse-plugin-directory>srccontribeclipse-plugin’
C:>cd C:hadoop2x-eclipse-pluginsrccontribeclipse-plugin
运行ANT搭建
C:hadoop2x-eclipse-pluginsrccontribeclipse-plugin>ant jar -Dversion=2.2.0 -Declipse.home=C:/IDE/sts-3.5.0 -Dhadoop.home=c:/hadoopBuildfile: C:hadoop2x-eclipse-pluginsrccontribeclipse-pluginbuild.xml
check-contrib:init: [echo] contrib: eclipse-plugin [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-plugin [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginclasses [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-plugintest [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginsystem [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginsystemclasses [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginexamples [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-plugintestlogsinit-contrib:ivy-download: [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar [get] To: C:hadoop2x-eclipse-pluginivyivy-2.1.0.jarivy-probe-antlib:ivy-init-antlib:ivy-init:[ivy:configure] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/ ::[ivy:configure] :: loading settings :: file = C:hadoop2x-eclipse-pluginivyivysettings.xmlivy-resolve-common:ivy-retrieve-common:[ivy:cachepath] DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead[ivy:cachepath] :: loading settings :: file = C:hadoop2x-eclipse-pluginivyivysettings.xmlcompile: [echo] contrib: eclipse-plugin [javac] C:hadoop2x-eclipse-pluginsrccontribeclipse-pluginbuild.xml:76: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds [javac] Compiling 45 source files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginclasses [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/Path.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate': class file for org.apache.hadoop.classification.InterfaceAudience not found [javac] c:hadoopsharehadoophdfshadoop-hdfs-2.2.0.jar(org/apache/hadoop/hdfs/DistributedFileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FileSystem.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataInputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar(org/apache/hadoop/fs/FSDataOutputStream.class): warning: Cannot find annotation method 'value()' in type 'LimitedPrivate' [javac] Note: Some input files use or override a deprecated API. [javac] Note: Recompile with -Xlint:deprecation for details. [javac] Note: Some input files use unchecked or unsafe operations. [javac] Note: Recompile with -Xlint:unchecked for details. [javac] 7 warningsjar: [mkdir] Created dir: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying 9 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-app-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-app-2.2.0.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-common-2.2.0.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-core-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-core-2.2.0.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-hs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-hs-2.2.0.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-hs-plugins-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-hs-plugins-2.2.0.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-jobclient-2.2.0-tests.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-jobclient-2.2.0-tests.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-jobclient-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-jobclient-2.2.0.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-client-shuffle-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-client-shuffle-2.2.0.jar [copy] Copying c:hadoopsharehadoopmapreducehadoop-mapreduce-examples-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-mapreduce-examples-2.2.0.jar [copy] Copying 3 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonhadoop-common-2.2.0-tests.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-common-2.2.0-tests.jar [copy] Copying c:hadoopsharehadoopcommonhadoop-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-common-2.2.0.jar [copy] Copying c:hadoopsharehadoopcommonhadoop-nfs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-nfs-2.2.0.jar [copy] Copying 3 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoophdfshadoop-hdfs-2.2.0-tests.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-hdfs-2.2.0-tests.jar [copy] Copying c:hadoopsharehadoophdfshadoop-hdfs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-hdfs-2.2.0.jar [copy] Copying c:hadoopsharehadoophdfshadoop-hdfs-nfs-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-hdfs-nfs-2.2.0.jar [copy] Copying 11 files to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-api-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-api-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-applications-distributedshell-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-applications-distributedshell-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-applications-unmanaged-am-launcher-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-client-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-client-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-common-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-common-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-common-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-nodemanager-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-nodemanager-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-resourcemanager-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-resourcemanager-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-tests-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-tests-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-server-web-proxy-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-server-web-proxy-2.2.0.jar [copy] Copying c:hadoopsharehadoopyarnhadoop-yarn-site-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-yarn-site-2.2.0.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibprotobuf-java-2.5.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibprotobuf-java-2.5.0.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonliblog4j-1.2.17.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginliblog4j-1.2.17.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibcommons-cli-1.2.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibcommons-cli-1.2.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibcommons-configuration-1.6.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibcommons-configuration-1.6.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibcommons-lang-2.5.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibcommons-lang-2.5.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibjackson-core-asl-1.8.8.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibjackson-core-asl-1.8.8.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibjackson-mapper-asl-1.8.8.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibjackson-mapper-asl-1.8.8.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibslf4j-log4j12-1.7.5.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibslf4j-log4j12-1.7.5.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibslf4j-api-1.7.5.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibslf4j-api-1.7.5.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibguava-11.0.2.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibguava-11.0.2.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibhadoop-auth-2.2.0.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibhadoop-auth-2.2.0.jar [copy] Copying 1 file to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlib [copy] Copying c:hadoopsharehadoopcommonlibnetty-3.6.2.Final.jar to C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginlibnetty-3.6.2.Final.jar [jar] Building jar: C:hadoop2x-eclipse-pluginbuildcontribeclipse-pluginhadoop-eclipse-plugin-2.2.0.jarBUILD SUCCESSFULTotal time: 42 seconds
eclipse.home:
Eclipse IDE的安装路径。
hadoop.home:
Hadoop 安装路径。
注意:下载“ivy-2.1.0.jar”需要有网络连接
4. 安装
在搭建成功之后,'srccontribeclipse-plugin'.内会生成'hadoop-eclipse-plugin-2.2.0.jar' 。拷贝此jar文件并将其粘贴到你IDE的“插件”所在路径。
5. 配置
如果Eclipse是开启状态就重新启动。否则只要启动就可以了。
a. 进入Windows-->Open Perspective-->Other并选择“Map/Reduce”。
b. 点击“New Hadoop location...”(蓝色大象图标)并定义Hadoop路径来运行MapReduce应用程序。点击“Finish”按钮。
Map/Reduce(V2) Master:
Map/Reduce主节点地址(任务追踪器)
DFS Master:
分布式文件系统主节点地址(名称节点)。
要了解“Port”数量的话,就启动Hadoop并在浏览器中打开http://localhost:8088/cluster。点击Tools --> Configuration 并查找以下属性。
DFS Master
<property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> <source>core-site.xml</source> </property>Map/Reduce(V2) Master
<property> <name>mapreduce.jobtracker.http.address</name> <value>0.0.0.0:50030</value> <source>mapred-default.xml</source> </property>c. 现在我们可以浏览Hadoop文件系统并只使用GUI来执行不同的文件/文件夹操作。
同样,我们还可以用向导(File --> New --> Other... --> Map/Reduce)轻松创建Map/Reduce Project,Mapper,Reducer以及MapReduce Driver并进入Hadoop编程。
我们一直都在努力坚持原创.......请不要一声不吭,就悄悄拿走。
我原创,你原创,我们的内容世界才会更加精彩!
【所有原创内容版权均属TechTarget,欢迎大家转发分享。但未经授权,严禁任何媒体(平面媒体、网络媒体、自媒体等)以及微信公众号复制、转载、摘编或以其他方式进行使用。】
微信公众号
TechTarget
官方微博
TechTarget中国
翻译
相关推荐
-
Quest发布最新Oracle Java开发工具
Quest Software近期发布了一款新的Oracle应用开发工具,这款工具可以在Eclipse IDE framework中使用Java。
-
Eclipse连接SQL 2000主要操作及可能遇到的问题
这个学期开了数据库,用的是sql server 2000,因为前面都是在sql server的企业管理器里面写一些程序,也就不要用具体的程序设计语言来实现了。这段时间要搞数据库课程……