How to export remote pc’s csv file to HDFS file system?

Last night I had to work on hdfs where I need to import remote pc’s csv file to hdfs file system. Here are the steps what I have followed.

STEP1
First I have conformed password less login

mohiulalamprince@ubuntu:/home/postgres$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/mohiulalamprince/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/mohiulalamprince/.ssh/id_rsa.
Your public key has been saved in /home/mohiulalamprince/.ssh/id_rsa.pub.
The key fingerprint is:
d1:17:73:19:74:15:81:15:6a:d4:f8:18:9b:02:f5:6b mohiulalamprince@ubuntu
The key's randomart image is:
+--[ RSA 2048]----+
|          ..ooOBB|
|         o  oBoo |
|        . o .+*  |
|         . o.+.. |
|        S   .E   |
|            .    |
|                 |
|                 |
|                 |
+-----------------+

STEP2

mohiulalamprince@ubuntu:/home/postgres$ ssh-copy-id admin@172.16.0.215
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
admin@172.16.0.215's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'admin@172.16.0.215'"
and check to make sure that only the key(s) you wanted were added.

STEP3

mohiulalamprince@ubuntu:/home/postgres$ cat waypoints_road_area.csv | ssh admin@172.16.0.215 "hadoop dfs -put - /user/admin/waypoints_road_area.csv"
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Leave a Reply