public AggregatedDfs aggregateDfs(AtomicArray<DfsSearchResult> results) {
ObjectObjectOpenHashMap<Term, TermStatistics> termStatistics = HppcMaps.newNoNullKeysMap();
ObjectObjectOpenHashMap<String, CollectionStatistics> fieldStatistics =
HppcMaps.newNoNullKeysMap();
long aggMaxDoc = 0;
for (AtomicArray.Entry<DfsSearchResult> lEntry : results.asList()) {
final Term[] terms = lEntry.value.terms();
final TermStatistics[] stats = lEntry.value.termStatistics();
assert terms.length == stats.length;
for (int i = 0; i < terms.length; i++) {
assert terms[i] != null;
TermStatistics existing = termStatistics.get(terms[i]);
if (existing != null) {
assert terms[i].bytes().equals(existing.term());
// totalTermFrequency is an optional statistic we need to check if either one or both
// are set to -1 which means not present and then set it globally to -1
termStatistics.put(
terms[i],
new TermStatistics(
existing.term(),
existing.docFreq() + stats[i].docFreq(),
optionalSum(existing.totalTermFreq(), stats[i].totalTermFreq())));
} else {
termStatistics.put(terms[i], stats[i]);
}
}
final boolean[] states = lEntry.value.fieldStatistics().allocated;
final Object[] keys = lEntry.value.fieldStatistics().keys;
final Object[] values = lEntry.value.fieldStatistics().values;
for (int i = 0; i < states.length; i++) {
if (states[i]) {
String key = (String) keys[i];
CollectionStatistics value = (CollectionStatistics) values[i];
assert key != null;
CollectionStatistics existing = fieldStatistics.get(key);
if (existing != null) {
CollectionStatistics merged =
new CollectionStatistics(
key,
existing.maxDoc() + value.maxDoc(),
optionalSum(existing.docCount(), value.docCount()),
optionalSum(existing.sumTotalTermFreq(), value.sumTotalTermFreq()),
optionalSum(existing.sumDocFreq(), value.sumDocFreq()));
fieldStatistics.put(key, merged);
} else {
fieldStatistics.put(key, value);
}
}
}
aggMaxDoc += lEntry.value.maxDoc();
}
return new AggregatedDfs(termStatistics, fieldStatistics, aggMaxDoc);
}
public static void main(String[] args) {
int NUMBER_OF_KEYS = (int) SizeValue.parseSizeValue("20").singles();
int STRING_SIZE = 5;
long PUT_OPERATIONS = SizeValue.parseSizeValue("5m").singles();
long ITERATIONS = 10;
boolean REUSE = true;
String[] values = new String[NUMBER_OF_KEYS];
for (int i = 0; i < values.length; i++) {
values[i] = RandomStrings.randomAsciiOfLength(ThreadLocalRandom.current(), STRING_SIZE);
}
StopWatch stopWatch;
stopWatch = new StopWatch().start();
ObjectIntOpenHashMap<String> map = new ObjectIntOpenHashMap<>();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
map.clear();
} else {
map = new ObjectIntOpenHashMap<>();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
map.addTo(values[(int) (i % NUMBER_OF_KEYS)], 1);
}
}
map.clear();
map = null;
stopWatch.stop();
System.out.println(
"TObjectIntHashMap: "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
stopWatch = new StopWatch().start();
// TObjectIntCustomHashMap<String> iMap = new TObjectIntCustomHashMap<String>(new
// StringIdentityHashingStrategy());
ObjectIntOpenHashMap<String> iMap = new ObjectIntOpenHashMap<>();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
iMap.clear();
} else {
iMap = new ObjectIntOpenHashMap<>();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
iMap.addTo(values[(int) (i % NUMBER_OF_KEYS)], 1);
}
}
stopWatch.stop();
System.out.println(
"TObjectIntCustomHashMap(StringIdentity): "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
iMap.clear();
iMap = null;
stopWatch = new StopWatch().start();
iMap = new ObjectIntOpenHashMap<>();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
iMap.clear();
} else {
iMap = new ObjectIntOpenHashMap<>();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
iMap.addTo(values[(int) (i % NUMBER_OF_KEYS)], 1);
}
}
stopWatch.stop();
System.out.println(
"TObjectIntCustomHashMap(PureIdentity): "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
iMap.clear();
iMap = null;
// now test with THashMap
stopWatch = new StopWatch().start();
ObjectObjectOpenHashMap<String, StringEntry> tMap = new ObjectObjectOpenHashMap<>();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
tMap.clear();
} else {
tMap = new ObjectObjectOpenHashMap<>();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
String key = values[(int) (i % NUMBER_OF_KEYS)];
StringEntry stringEntry = tMap.get(key);
if (stringEntry == null) {
stringEntry = new StringEntry(key, 1);
tMap.put(key, stringEntry);
} else {
stringEntry.counter++;
}
}
}
tMap.clear();
tMap = null;
stopWatch.stop();
System.out.println(
"THashMap: "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
stopWatch = new StopWatch().start();
HashMap<String, StringEntry> hMap = new HashMap<>();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
hMap.clear();
} else {
hMap = new HashMap<>();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
String key = values[(int) (i % NUMBER_OF_KEYS)];
StringEntry stringEntry = hMap.get(key);
if (stringEntry == null) {
stringEntry = new StringEntry(key, 1);
hMap.put(key, stringEntry);
} else {
stringEntry.counter++;
}
}
}
hMap.clear();
hMap = null;
stopWatch.stop();
System.out.println(
"HashMap: "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
stopWatch = new StopWatch().start();
IdentityHashMap<String, StringEntry> ihMap = new IdentityHashMap<>();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
ihMap.clear();
} else {
hMap = new HashMap<>();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
String key = values[(int) (i % NUMBER_OF_KEYS)];
StringEntry stringEntry = ihMap.get(key);
if (stringEntry == null) {
stringEntry = new StringEntry(key, 1);
ihMap.put(key, stringEntry);
} else {
stringEntry.counter++;
}
}
}
stopWatch.stop();
System.out.println(
"IdentityHashMap: "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
ihMap.clear();
ihMap = null;
int[] iValues = new int[NUMBER_OF_KEYS];
for (int i = 0; i < values.length; i++) {
iValues[i] = ThreadLocalRandom.current().nextInt();
}
stopWatch = new StopWatch().start();
IntIntOpenHashMap intMap = new IntIntOpenHashMap();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
intMap.clear();
} else {
intMap = new IntIntOpenHashMap();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
int key = iValues[(int) (i % NUMBER_OF_KEYS)];
intMap.addTo(key, 1);
}
}
stopWatch.stop();
System.out.println(
"TIntIntHashMap: "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
intMap.clear();
intMap = null;
// now test with THashMap
stopWatch = new StopWatch().start();
IntObjectOpenHashMap<IntEntry> tIntMap = new IntObjectOpenHashMap<>();
for (long iter = 0; iter < ITERATIONS; iter++) {
if (REUSE) {
tIntMap.clear();
} else {
tIntMap = new IntObjectOpenHashMap<>();
}
for (long i = 0; i < PUT_OPERATIONS; i++) {
int key = iValues[(int) (i % NUMBER_OF_KEYS)];
IntEntry intEntry = tIntMap.get(key);
if (intEntry == null) {
intEntry = new IntEntry(key, 1);
tIntMap.put(key, intEntry);
} else {
intEntry.counter++;
}
}
}
tIntMap.clear();
tIntMap = null;
stopWatch.stop();
System.out.println(
"TIntObjectHashMap: "
+ stopWatch.totalTime()
+ ", "
+ stopWatch.totalTime().millisFrac() / ITERATIONS
+ "ms");
}
