File exists error in HDFS - CopyFromLocal

HDFS is a distributed file system designed to run on top of the local file system. Many times we may need to copy files from different sources i.e. from the internet, remote network, or from the local file system. There are  "CopyFromLocal" and "Put" commands to help us in performing the task. While copying a file from the local file system to HDFS, if the file exists in the destination, the execution will fail and we will receive 'the file exists' error.

Let's assume the file "emp.txt" already exists in the path /user/cloudera.

Hadoop fs -put Desktop/emp.txt /user/cloudera/emp.txt

This returned “the file already exists” error

Hadoop fs -copyFromLocal Desktop/emp.txt /user/cloudera/emp.txt

This also returned “the file already exists” error.

Hadoop fs -copyFromLocal -f Desktop/Documents/emp.txt /user/cloudera/emp.txt
This is succeeded. The file is copied to the destination without any errors.

The usage of the "-f" option with -copyFromLocal will overwrite the destination if it already exists.

Hope you find this article helpful.

Difference between CopyFromLocal, Put, CopyToLocal and Get

The purpose of this article is to let you know about few HDFS commands that are identical in behavior but distinct.

CopyFromLocal and Put: These two commands help in copying the file from one location to another. The difference between these two is that the "CopyFromLocal" command will help copy the file from local file system to HDFS, while the "Put" command will copy from anywhere (local or network) to anywhere (HDFS or local file system).

hadoop fs -put <Local system directory path or network path> <HDFS file path>

hadoop fs -copyFromLocal <Local system directory path>  <HDFS file path>

"Put" allows us to copy several file paths to HDFS at once (files or folders from 

local or remote locations), while copyFromLocal, on the other hand, is limited to a local file reference.

A choice exists to overwrite an existing file using -f when using copyFromLocal. However, an error is returned if the file persists when "put" is executed.

In short, anything you do with copyFromLocal, you can do with "put", but not vice-versa.

CopyToLocal and Get: These two commands are just opposite to "CopyFromLocal" and "Put".
The destination is restricted to a local file reference when we use copyToLocal. While using "Get" there are no such restrictions.

Anything you do with copyToLocal, you can do with "get" but not vice-versa.
hadoop fs -get <HDFS file path> <Local system directory path> hadoop fs -copyToLocal <HDFS file path> <Local system directory path>

For complete HDFS commands please click here. For complete Hive DDL commands please click here.