Adaptive Optimization of Prompts, Models and Generation Strategies

Provides provider-agnostic tools for jointly comparing and optimizing prompts, language-model providers, and generation strategies for generative artificial intelligence workflows. Candidate configurations can be evaluated using user-supplied scoring functions, cost and latency measurements, robustness perturbations, Pareto-front screening, budget and latency constraints, prompt evolution, adaptive routing, self-consistency, and text-output ensembles. The core workflow is designed to run offline with deterministic mock providers, while external model application programming interfaces can be connected through user-defined provider functions. Evolutionary search concepts are described by Goldberg (1989, ISBN:0201157675), and multi-objective optimization concepts are related to Deb, Pratap, Agarwal and Meyarivan (2002) .

AutoGenAI

AutoGenAI is a provider-agnostic R framework for joint optimization of prompts, model providers, and generation strategies.

Main ideas

• compare prompt-model-strategy configurations;
• optimize quality, robustness, cost, and latency;
• identify Pareto-efficient configurations;
• evolve prompts with mutation and crossover;
• stress-test prompts with reproducible perturbations;
• learn a lightweight adaptive model router;
• build majority or weighted text ensembles;
• enforce budget, latency, and abstention constraints;
• run all package examples offline with deterministic mock providers.

Minimal offline example

library(AutoGenAI)

ex <- autogenai_example()
fit <- optimize_ai(
  task = ex$task,
  data = ex$data,
  providers = ex$providers,
  prompts = ex$prompts,
  temperatures = c(0, 0.2),
  strategies = c("single", "self_consistency")
)

fit
pareto_ai(fit)
select_configuration(fit, preference = "balanced")

Connecting a real model

my_provider <- ai_provider(
  name = "my-model",
  generate = function(prompt, input, params) {
    # Call any external model here and return text.
    paste(prompt, input)
  },
  input_cost_per_1k = 0,
  output_cost_per_1k = 0
)

The package itself does not require API keys and does not contact external services during checks.