Skip to content

bcleenders/hops

 
 

Repository files navigation

Hops

Join the chat at https://gitter.im/hopshadoop/hops

Hops (Hadoop Open Platform-as-a-Service) is a new distribution of Apache Hadoop with scalable, highly available, customizable metadata. HopsFS is a next generation distribution of the Hadoop Distributed File System (HDFS) that replaces HDFS’ single node in-memory metadata service with a distributed metadata service built on a NewSQL database (MySQL Cluster). HopsYARN is a next generation distribution of YARN that replaces YARN's single node in-memory cluster state with a distributed metadata service built on a NewSQL database (MySQL Cluster). Together, HopsFS and HopsYARN enable Hadoop clusters to scale to larger volumes and higher throughput.

Information

Introduction

Hops consists internally of two main sub projects, Hops-Fs and Hops-Yarn. Hops-FS is a new implementation of the the Hadoop Filesystem (HDFS), that supports multiple stateless NameNodes, where the metadata is stored in an in-memory distributed database (MySQL Cluster). Hops-FS enables more scalable clusters than Apache HDFS (up to ten times larger clusters), and enables NameNode metadata to be both customized and analyzed, because it can now be easily accessed via a SQL API. Hops-YARN introduces a distributed stateless Resource Manager, whose state is migrated to MySQL Cluster, a replicated, partitioned, in-memory NewSQL database. This enables our YARN architecture to have no down-time, with failover of a ResourceManager happening in a few seconds.

For the latest information about Hops, please visit our website at:

http://hops.io

For the latest information about Hadoop, please visit their website at:

http://hadoop.apache.org/core/

and their wiki, at:

http://wiki.apache.org/hadoop/

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software: Hadoop Core uses the SSL libraries from the Jetty project written by mortbay.org.

===============================================================================

How to build

sudo  aptitude install cmake libprotobuf-dev libprotobuf-c0-dev

mvn clean generate-sources

cd hadoop-maven-plugins
mvn install

build the hop-metadata-dal project first, then build the associated metadata-dal implementation which for now is hop-metadata-dal-ndb-impl

cd ..
mvn package -Pdist -DskipTests -Dtar

for native deployment, you need to install these libraries first

sudo aptitude install zlib1g-dev libssl-dev
cd ..
mvn package -Pdist,native -DskipTests -Dtar

How to add RPC messages

Add first a protocol buffer msg and rpc to the .proto file. Then add a wrapper class.

Finally, run: mvn generate-sources to generate the java classes from the protocol buffer files.

===============================================================================

License

Hops is released under an Apache 2.0 license.

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Java 91.0%
  • HTML 4.9%
  • C 2.6%
  • Shell 0.4%
  • Protocol Buffer 0.4%
  • C++ 0.2%
  • Other 0.5%