windows os

windows os

Build and Install Hadoop on Windows with Native Binaries & Run Without Cygwin

1d ago
SOURCE  

Description

This tutorial aims to provide a step by step guide to Build Hadoop from Hadoop source on Windows OS. Tutorial for Building Hadoop 2.7.2 for Windows with Native Binaries. Hadoop installation on windows Without Cygwin Solution for Spark Error: Many of you may tried running spark on Windows OS and faced below error on console. This is because your hadoop distribution does not contains native binaries for Windows OS as they are not included in official Hadoop Distribution. So you need to build hadoop from its source for your OS. Error like: 16/04/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/04/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. Solution for Hadoop Error: This error is also related to the Native Hadoop Binaries for Windows OS. So solution is same as above Spark problem that you need to build it for your Windows OS from Hadoop Source code. Error Like 16/04/03 19:59:10 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable C:\hadoop\bin\winutils.exe in the Hadoop binaries. So just follow this tutorial video and at the end you will be able to get rid of these errors. Build Command Used : mvn package -Pdist,native-win -DskipTests -Dtar Download Links : 1. Download Hadoop source from http://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-2.7.2/hadoop-2.7.2-src.tar.gz 2. Download Microsoft .NET Framework 4 (Standalone Installer) from https://www.microsoft.com/en-in/download/details.aspx?id=17718 3. Download Windows SDK 7 Installer from https://www.microsoft.com/en-in/download/details.aspx?id=8279 or you can also use offline installer ISO from https://www.microsoft.com/en-in/download/details.aspx?id=8442 You will find 3 different ISO’s to download. a. GRMSDK_EN_DVD.iso (x86) b. GRMSDKX_EN_DVD.iso (AMD64) c. GRMSDKIAI_EN_DVD.iso (Itanium) Please choose based on your OS type. 4. Download JDK according to your OS & CPU architecture from http://www.oracle.com/technetwork/java/javase/downloads/index.html 5. Download and install 7-zip from http://www.7-zip.org/download.html 6. Download & extract Maven 3.0 or later from https://maven.apache.org/download.cgi 7. Download ProtocolBuffer 2.5.0 from https://github.com/google/protobuf/releases/download/v2.5.0/protoc-2.5.0-win32.zip 8. Download CMake 3.5.0 or higher from https://cmake.org/download/ 9. Download Cygwin installer from https://cygwin.com/ Official Hadoop On Windows Configuration guide : https://wiki.apache.org/hadoop/Hadoop2OnWindows Official Hadoop building guide : https://svn.apache.org/viewvc/hadoop/common/branches/branch-2/BUILDING.txt?view=markup