Giranking project contains Link ranking algorithms developed with Apache Giraph to be used in Apache Nutch 2.x. It can be used in different applications as well.
Apache Giraph allows us perform graph operations efficiently in a distributed manner.
For sample jobs with proper configurations, look at https://github.com/AGMLab/nutch/tree/2.1-emre/src/java/org/apache/nutch/scoring. There exists UsageRank in this package as well which just normalizes visit counts of hosts.
- Cycles cause unfair advantage to some vertices, they get 9.9+ score out of 10.0
Derivation of PageRank. Ported from Apache Nutch 1.x LinkRank to make it more efficient with Giraph.
- Uses full URLs (pages).
- Uses 10 iteration.
- Removes duplicate links.
- Performs cumulative probability distribution normalization, yields scores [0..10].
- Use Nutch2WebpageInputFormat and Nutch2WebpageOutputFormat when HBase is your DB.
- Use LinkRankVertexInputFormat (if you have initial scores for vertices) or LinkRankVertexUniformInputFormat (gives 1.0 for each vertex). LinkRankEdgeInputFormat for edge list.
- Use LinkRankEdgeFilter for skipping malformed URLs.
Derivation of LinkRank. Works on HBase host table.
- Use LinkRankComputation with Nutch2HostInputFormat and Nutch2HostOutputFormat.
- Uses hosts from the HBase host table generated by Apache Nutch 2. (./nutch updatedb -linkDb)
- Use TrustRankComputation with Nutch2HostTrustInputFormat and Nutch2HostOutputFormat with output fields set as mtdt:_tr_