Hadoop环境已经安装完毕
(1)到https://archive.apache.org/dist/pig/下载对应的tar包,如果是hadoop0.20.x之前的版本,则直接可以用,如果Hadoop2.x之后的,则需要重新编译,在pig的根目录下执行如下命令:
- ant clean jar-withouthadoop -Dhadoopversion=23
ant clean jar-withouthadoop -Dhadoopversion=23
否则执行MapReduce时,会报如下的异常
- 2013-10-24 09:35:19,300 [main] WARN
- org.apache.pig.backend.hadoop20.PigJobControl - falling back to default
- JobControl (not using hadoop 0.20 ?)
- java.lang.NoSuchFieldException: runnerState
- at java.lang.Class.getDeclaredField(Class.java:1938)
- at
- org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
- at
- org.apache.p
2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.hadoop20.PigJobControl - falling back to default JobControl (not using hadoop 0.20 ?) java.lang.NoSuchFieldException: runnerState at java.lang.Class.getDeclaredField(Class.java:1938) at org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51) at org.apache.p
(2)配置Pig的环境变量:
- export PIG_CLASSPATH=$HADOOP_HOME/etc/hadoop
- export PATH=/home/search/pig-0.12.1/bin:$PATH
export PIG_CLASSPATH=$HADOOP_HOME/etc/hadoop export PATH=/home/search/pig-0.12.1/bin:$PATH
(3)直接在linux终端执行pig命令,即可进入grunt界面:
- 2015-05-01 12:44:58,573 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.2-SNAPSHOT (r: unknown) compiled May 01 2015, 12:28:37
- 2015-05-01 12:44:58,574 [main] INFO org.apache.pig.Main - Logging error messages to: /home/search/pig-0.12.1/build/pig_1430498698551.log
- 2015-05-01 12:44:58,602 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/search/.pigbootup not found
- 2015-05-01 12:44:59,244 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
- 2015-05-01 12:44:59,244 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
- 2015-05-01 12:44:59,244 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://h1:8020
- 2015-05-01 12:44:59,247 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
- 2015-05-01 12:45:00,465 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: h1:8021
- 2015-05-01 12:45:00,469 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
- grunt>
2015-05-01 12:44:58,573 [main] INFO org.apache.pig.Main - Apache Pig version 0.12.2-SNAPSHOT (r: unknown) compiled May 01 2015, 12:28:37 2015-05-01 12:44:58,574 [main] INFO org.apache.pig.Main - Logging error messages to: /home/search/pig-0.12.1/build/pig_1430498698551.log 2015-05-01 12:44:58,602 [main] INFO org.apache.pig.impl.util.Utils - Default bootup file /home/search/.pigbootup not found 2015-05-01 12:44:59,244 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2015-05-01 12:44:59,244 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS 2015-05-01 12:44:59,244 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://h1:8020 2015-05-01 12:44:59,247 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.used.genericoptionsparser is deprecated. Instead, use mapreduce.client.genericoptionsparser.used 2015-05-01 12:45:00,465 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: h1:8021 2015-05-01 12:45:00,469 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS grunt>
(4)pig -i查看pig的版本,
pig --help 查看pig的一些帮助命令
pig -x local 执行local模式
pig -x mapreduce 执行MapReduce模式
相关推荐
在Windows上使用Cygwin作为类Unix支撑环境来安装Hadoop, 网上介绍的还是比较多的。...其实,Hadoop环境搭建的主要内容是Cygwin的安装,估计要占总时间的2/3, 其它工作我会手把手的教你,come on,follow me!
hadoop环境搭建
hadoop环境搭建和eclipse开发
适合零基础的初学者,在window系统下搭建hadoop环境,
史上最详细的Hadoop环境搭建,CentOS+Hadoop等详解 第一部分:Linux环境安装 第二部分:Hadoop本地模式安装 第三部分:Hadoop伪分布式模式安装 第四部分:完全分布式安装 第五部分:Hadoop HA安装
Hadoop环境搭建Hadoop环境搭建 Hadoop环境搭建
Hadoop环境搭建.pdf Hadoop环境搭建.pdf Hadoop环境搭建.pdf
Hadoop环境搭建手册(包含所有基本信息,本人亲测)
hadoop的环境搭建手册 分布式计算开源框架Hadoop介绍
hadoop的环境搭建过程.方便第一次使用hadoop的同学.快速搭建环境
Hadoop部署平台全套组件,包含hdfs,hive,hbase,sqoop,mysql,flume等,包含大数据处理的全部组件功能! 现成虚拟机,安装好的镜像,下载直接可运行!!!
windows下面的hadoop的bin目录; Failed to locate the winutils binary in the hadoop binary path
成功在windows的eclipse连接hadoop 具体步骤如下: 1.使用对应版本的Hadoop jar包 ...因为是Hadoop2.6环境,所以后者覆盖前者 解决:创建HADOOP_HOME环境变量,值为解压目录 将 %HADOOP_HOME%/bin添加至Path下。
Hadoop环境安装设置(最简单的hadoop单机环境部署教程) 安装前设置 SSH设置和密钥生成 安装Java.下载Java (JDK<最新版> - X64 ... 下载Hadoop.下载来自Apache基金会软件,使用下面 ... Hadoop操作模式 在单机模式下...
hadoop2.9.2 zookeeper 3.4.14 搭建环境,亲测可用。。
全程教导,分享快乐 centos下搭建hadoop全分布式.2.5.0环境.doc
Hadoop超详细介绍,详情介绍了Hadoop的安装
离线安装HADOOP环境
hadoop环境搭建资源 hadoop环境搭建资源 hadoop环境搭建资源 hadoop环境搭建资源
linux搭建hadoop环境,介绍如何搭配hadoop