Local search-based approach for cost-effective job assignment on large language models

Large Language Models (LLMs) have garnered significant attention due to their impressive capabilities. However, leveraging LLMs can be expensive due to the computational resources required, with costs depending on invocation numbers and input prompt lengths. Generally, larger LLMs deliver better performance but at a higher cost. In addition, prompts that provide more guidance to LLMs can increase the probability of correctly processing the job but also tend to be longer, increasing the processing cost. Therefore, selecting an appropriate LLM and prompt template is crucial for achieving an optimal trade-off between cost and performance. This paper formulates the job assignment on LLMs as a multi-objective optimisation problem and proposes a local search-based algorithm, termed LSAP, which aims to minimise the invocations cost while maximising overall performance. First, historical data is used to estimate the accuracy of each job submitted to a candidate LLM with a chosen prompt template. Subsequently, LSAP combines heuristic rules to select an appropriate LLM and prompt template based on the invocation cost and estimated accuracy. Extensive experiments on LLM-based log parsing, a typical software maintenance task that utilizes LLMs, demonstrate that LSAP can efficiently generate solutions with significantly lower cost and higher accuracy compared to the baselines.