Showing posts with label hive-database. Show all posts
Showing posts with label hive-database. Show all posts

Thursday, August 13, 2020

Sqoop Import All - Copying MySQL Database to Hive

At the point when I began learning Sqoop, I confronted such huge numbers of blunders for which I was unable to get a lot of help from the discussions or sites. The beneath is one of them which killed parcel of time. 

sqoop import-all-tables

--connect=jdbc:mysql://localhost/retail_db

--username=root

--password=cloudera

--hive-import --hive-database retaildb

--warehouse-dir=/user/hive/warehouse/retaildb.db

-m 1

I wanted to import all the tables and data into Hive database all at once i.e., a database to database import. 

The execution looks fine and I see tables are creating and even I see the data is importing. Be that as it may, after execution, I see just barely any tables created in Hive and not many table's information imported. To confound me more, there is not really any information being shown in Impala considerably even after refreshing the metadata. On the other hand, Hue couldn't open a portion of the documents. 

I did spend a lot of time to figure out what wrong I was doing. 

Finally I got the code right. 

I have assigned a single mapper which caused the issue. The size of the retail_db database is around 450 MB for which assigning a single mapper is inadequate.

         sqoop import-all-tables

--connect=jdbc:mysql://localhost/retail_db

--username=root

--password=cloudera

--hive-import --hive-database retaildb

--warehouse-dir=/user/hive/warehouse/retaildb.db 

The above command creates the internal tables in Hive automatically and data will be imported without any issue.

Big Data & SQL

Hi Everybody, Please do visit my new blog that has much more information about Big Data and SQL. The site covers big data and almost all the...