LLM Routing (Query Complexity Estimation)

Assignment:

The size of LLMs is largely determined by the number of their parameters. It is generally accepted that LLMs with a higher parameter count tend to be more powerful. However, this is also accompanied by increased computing power and correspondingly higher costs. In parallel, there are smaller LLMs that are either optimized in specific domains or designed for less demanding tasks. Interacting with an LLM is through requests, also known as “queries”. To make the most of the resources of LLMs, it is crucial to estimate the complexity of a request. This assessment allows for the targeted assignment of queries to models of different sizes.

Tasks:

The main idea of this thesis is the development of a heuristic for the evaluation of the complexity of queries to Language Models (LLMs) in the context of increasing parameter size:

• Literature survey on size and complexity of several LLMs
• Developing a heuristic for assessment of the complexity of requests, for example a use of another LLM as a tool to analyze the complexity of requests, thus facilitating efficient mapping to large or small models.
• Creating a set of quality metrics and the corresponding evaluation mechanisms to systematically analyze the correlation between complexity of prompts and the achieved results.

Contact person: Odej Kao (odej.kao@tu-berlin.de)

Start: immediately

Resources:

• https://arxiv.org/abs/2311.08692 (This is about routing to the right LLM [expertise], not about the complexity of the query)