Interactive benchmarking of Text-to-SQL systems


Numerous search systems have been implemented that allow users to pose unstructured queries over databases without the need to use a query language, such as SQL. Unfortu- nately, the landscape of efforts is fragmented with no clear sight of which system is best, and what open challenges we should pursue in our research. To help towards this direction, we present THOR that makes 4 important contributions: a query benchmark, a framework for comparing different sys- tems, several search system implementations, and a highly interactive tool for comparing different search systems.

More detailed information about Thor can be found here.

System Architecture

The user can submit a query concurrently to several search systems. Since system execution times may vary, it is important that the results of every system can be sent and rendered to the front-end separately as soon as it finishes.