Skip to content

carlosdotdanger/HadooperPooper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

21 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Reimplementing some of the hadoop-ec2 tools without so much shell script as a way to get cozy with the AWS SDK.

Building - 
it's currently setup as a Netbeans project, will be getting the Netbeans stuff out soon.

depends on:
--aws-sdk 
http://aws.amazon.com/sdkforjava/

--commons-codec v. 1.4 (aws includes 1.3 but I had issues using it)
http://commons.apache.org/codec/

--log4j
http://logging.apache.org/log4j/

--ganymed (ssh lib)
http://www.cleondris.ch/opensource/ssh2/

setup the required libraries and build.

Configuring --
there's sample config files in dist/config
to launch a cluster you need to choose or build an AMI, preferably with hadoop installed, but you could install hadoop on startup without too much effort.

Running -- 

run pooper.sh in dist/bin after building.

As of this writing, HaddooperPooper 's command-list:
	command
	create-groups
	delete-cluster
	describe-cluster
	get-login-command
	launch-cluster
	launch-master
	launch-slaves
	list-clusters
	login
	push-file
	terminate-cluster
	terminate-slaves

Still to come:
port forwarding, 
automated job submission/termination, 
support for a map-reduce only (non-dfs) slave group that can be scaled separately,
GUI.










About

Hadoop-ec2 tool written in java.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published