Last night I had to work on hdfs where I need to import remote pc’s csv file to hdfs file system. Here are the steps what I have followed.
First I have conformed password less login
mohiulalamprince@ubuntu:/home/postgres$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/mohiulalamprince/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/mohiulalamprince/.ssh/id_rsa. Your public key has been saved in /home/mohiulalamprince/.ssh/id_rsa.pub. The key fingerprint is: d1:17:73:19:74:15:81:15:6a:d4:f8:18:9b:02:f5:6b mohiulalamprince@ubuntu The key's randomart image is: +--[ RSA 2048]----+ | ..ooOBB| | o oBoo | | . o .+* | | . o.+.. | | S .E | | . | | | | | | | +-----------------+
mohiulalamprince@ubuntu:/home/postgres$ ssh-copy-id firstname.lastname@example.org /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys email@example.com's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'firstname.lastname@example.org'" and check to make sure that only the key(s) you wanted were added.
mohiulalamprince@ubuntu:/home/postgres$ cat waypoints_road_area.csv | ssh email@example.com "hadoop dfs -put - /user/admin/waypoints_road_area.csv" DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it.