October 26, 2015: Version 2.0 of SQUARE now released! (new GIT repo to avoid impacting those using Version 1.0). See 2.0 page for details of what's new! -- [SQUARE-2.0] (https://github.com/utir/square-2.0.git).

Build and Run Instructions: -- Project conforms to a maven build. -- Run mvn install from the root directory. -- See _CLA files for example usage. See: http://ir.ischool.utexas.edu/square/ for results analysis and API.

Processing scripts: -- This folder includes Matlab scripts to interface with data generated by the SQUARE algorithms to plug and play with other algorithms. — Generating data 1. Create a folder of your choice 2. The folder needs to contain the files: a)categories.txt - each line names the category b)responses.txt - space separated values with each line of the format workerId question response c)groundTruth.txt - space separated values with each line of the format question response Note that its not necessary to have the groundTruth.txt file 3. Depending on whether you have ground truth or not run generateSQUAREData.m or generateSQUAREDataWOGT.m -- Sub Folders -- resultScripts - Scripts to read, process and generate label aggregation results. -- statisticalSig - Scripts to prove statistical significance of aggregation results. -- statistics - Scripts to analyze dataset statistics. -- synthetic - Scripts to generate synthetic data. -- utils - General utility scripts. -- interfaceScripts - Scripts to interface with third-party algorithms.

To run all the algorithms use the Matlab interface scripts — see below (remember to generate SQUARE data scripts prior to this). — interfaceScripts/printAllResults.m (when ground truth is available) - interfaceScripts/runAllWOGT.m (when ground truth isn’t available)

Additionally note that printAllResults saves aggregated files such that original question, categories and responses are mapped to integers. The map files are stored under model. Use printMappedFiles.java to print new aggregated files that are mapped to original questions and categories.

If you use the code please cite:

@inproceedings{Sheshadri13, author = {Aashish Sheshadri and Matthew Lease}, title = {{SQUARE: A Benchmark for Research on Computing Crowd Consensus}}, booktitle = {{Proceedings of the 1st AAAI Conference on Human Computation (HCOMP)}}, year = {2013} }