Hadoop(四)Sqoop安装

139 阅读 0 评论 92 点赞

我是靠谱客的博主难过母鸡，这篇文章主要介绍Hadoop(四)Sqoop安装，现在分享给大家，希望可以做个参考。

Sqoop 主要用于在Hadoop(Hive)与传统的数据库(如mysql)间进行数据的传递，可以将一个关系型数据库（例如： MySQL ,Oracle等）中的数据导进到Hadoop的HDFS中，也可以将HDFS的数据导进到关系型数据库中。

数据库的操作使用可视化工具会更方便，这里使用Navicat。
Navicat 下载与安装 https://www.jianshu.com/p/5f693b4c9468
使用Navicat连接linux中的MySQL数据库出现错误：Access denied for user ‘root’@’localhost’ (using password:YES)
这里是因为MySQL的权限只能本地连接

复制代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
mysql -uhadoop -p
show databases;//第一步
mysql> use mysql//第二步
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

mysql> select host,user from user;//第三步
+-----------+--------+
| host      | user   |
+-----------+--------+
| %         | root   |
| 127.0.0.1 | root   |
| ::1       | root   |
| localhost |        |
| localhost | hadoop |
| master    |        |
| master    | hadoop |
| master    | root   |
+-----------+--------+
8 rows in set (0.00 sec)

mysql> update user set host = '%' where user = 'hadoop' and host='localhost';//第四步
Query OK, 0 rows affected (0.00 sec)
Rows matched: 0  Changed: 0  Warnings: 0

mysql> flush privileges;//第五步
Query OK, 0 rows affected (0.00 sec)

Sqoop官网：http://sqoop.apache.org/
下载地址：http://mirrors.hust.edu.cn/apache/sqoop/

从官网看主要有两个版本，1.4.7和1.99.7两者并不兼容。而最新的版本还是16年发布的，两年没更新新版本。刚学大数据后面的知识还没看，猜测Sqoop可能以后用到的不多吧。不过还是得了解它。

这里安装1.99.7版本

下载gz文件上传到linux中并解压，为了方便这里吧解压后的目录改名为sqoop

在解压目录中创建文件夹extra来存放MySQL的驱动jar包
mkdir extra
把mysql的驱动jar文件复制到这个目录下，跟上一篇hive中驱动一样

配置环境变量

复制代码

1
2
3
vim /etc/profile
SQOOP_HOME=/home/chs/software/sqoop
SQOOP_SERVER_EXTRA_LIB=/home/chs/software/extra

进入到解压目录中的 conf目录中更改sqoop.properties文件

复制代码

1
2
3
4
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/chs/hadoop-2.7.3/etc/hadoop 
org.apache.sqoop.security.authentication.type=SIMPLE 
org.apache.sqoop.security.authentication.handler=org.apache.sqoop.security.authentication.SimpleAuthenticationHandler 
org.apache.sqoop.security.authentication.anonymous=true

配置完成后验证
在安装目录下面
bin/sqoop2-tool verify

复制代码

Setting conf dir: /home/chs/software/sqoop/bin/../conf
Sqoop home directory: /home/chs/software/sqoop
Sqoop tool executor:
    Version: 1.99.7
    Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
    Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
14   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/chs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/chs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Setting conf dir: /home/chs/software/sqoop/bin/../conf
Sqoop home directory: /home/chs/software/sqoop
Sqoop tool executor:
    Version: 1.99.7
    Revision: 435d5e61b922a32d7bce567fe5fb1a9c0d9b1bbb
    Compiled on Tue Jul 19 16:08:27 PDT 2016 by abefine
Running tool: class org.apache.sqoop.tools.tool.VerifyTool
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
14   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/chs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/chs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Verification was successful.
Tool class org.apache.sqoop.tools.tool.VerifyTool has finished correctly.

开启服务器
bin/sqoop2-server start

复制代码

1
2
3
4
5
6
7
8
9
10
Setting conf dir: /home/chs/software/sqoop/bin/../conf
Sqoop home directory: /home/chs/software/sqoop
Starting the Sqoop2 server...
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
6    [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/chs/hadoop-2.7.3/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/chs/apache-hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Sqoop2 server started.

开启服务器后生成了两个目录
@BASEDIR@ 和 @LOGDIR@
查看运行日志
cd cd @LOGDIR@/ ll

复制代码

1
2
3
4
-rw-r--r--. 1 root root     0 May 23 20:10 audit.log
-rw-r--r--. 1 root root   669 May 23 20:11 derbyrepo.log
-rw-r--r--. 1 root root 81227 May 23 20:11 sqoop.log

关闭服务器
bin/sqoop2-server stop

复制代码

1
2
3
4
5
Setting conf dir: /home/chs/software/sqoop/bin/../conf
Sqoop home directory: /home/chs/software/sqoop
Stopping the Sqoop2 server...
Sqoop2 server stopped.

开启客户端：
bin/sqoop2-shell

复制代码

1
2
3
4
5
6
7
8
Setting conf dir: /home/chs/software/sqoop/bin/../conf
Sqoop home directory: /home/chs/software/sqoop
May 23, 2018 8:16:06 PM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
Sqoop Shell: Type 'help' or 'h' for help.

sqoop:000>