Budget Configuration

The multi-fidelity budget system is the mechanism by which AGILE allocates computational resources during hyperparameter optimization. Each budget level defines how much data, how many random seeds, and how much wall-clock time to use when evaluating candidate configurations. Configurations are promoted from cheaper to more expensive budget levels based on their performance, ensuring that the majority of computation is spent on the most promising candidates.

The spires_opt_budget Struct

Each budget level is defined by:

struct spires_opt_budget {
    double data_fraction;   /* fraction of training data to use (0, 1] */
    int    num_seeds;       /* number of random seeds for evaluation */
    double time_limit_sec;  /* wall-clock time limit in seconds */
};

Fields

Multi-Fidelity Rationale

Evaluating a reservoir configuration is expensive: the reservoir must be created, driven with data, trained, and evaluated on a test set. Doing this multiple times with different random seeds and with the full dataset is very costly.

The key insight of multi-fidelity optimization is that cheap, low-accuracy evaluations can quickly identify bad configurations. There is no need to spend minutes evaluating a configuration with 500 neurons, 5 random seeds, and the full dataset if a quick 5-second evaluation with 100 neurons and 20% of the data shows it performs poorly.

The budget levels implement this principle:

1. Level 1 (cheap): Evaluate many candidates quickly with a small data fraction and few seeds. Discard the worst performers.
2. Level 2 (moderate): Evaluate the survivors with more data and more seeds. Discard again.
3. Level 3 (expensive): Evaluate the final survivors at full fidelity to determine the best configuration.

Example Budget Arrays

Minimal (2 levels)

struct spires_opt_budget budgets[] = {
    { .data_fraction = 0.3, .num_seeds = 2, .time_limit_sec = 60.0  },
    { .data_fraction = 1.0, .num_seeds = 5, .time_limit_sec = 300.0 },
};
int num_budgets = 2;

This is the simplest useful configuration. The first level uses 30% of the data and 2 seeds to quickly screen candidates. The second level uses the full dataset and 5 seeds for final evaluation.

Standard (3 levels)

struct spires_opt_budget budgets[] = {
    { .data_fraction = 0.2, .num_seeds = 2, .time_limit_sec = 30.0  },
    { .data_fraction = 0.5, .num_seeds = 3, .time_limit_sec = 120.0 },
    { .data_fraction = 1.0, .num_seeds = 5, .time_limit_sec = 600.0 },
};
int num_budgets = 3;

Three levels provide a good balance between screening efficiency and final accuracy. This is the recommended starting point for most applications.

Aggressive (5 levels)

struct spires_opt_budget budgets[] = {
    { .data_fraction = 0.1, .num_seeds = 1, .time_limit_sec = 10.0   },
    { .data_fraction = 0.2, .num_seeds = 2, .time_limit_sec = 30.0   },
    { .data_fraction = 0.4, .num_seeds = 3, .time_limit_sec = 120.0  },
    { .data_fraction = 0.7, .num_seeds = 4, .time_limit_sec = 300.0  },
    { .data_fraction = 1.0, .num_seeds = 5, .time_limit_sec = 1200.0 },
};
int num_budgets = 5;

Five levels are useful when the search space is very large or when individual evaluations are very expensive (e.g., large reservoirs, long time series). The additional levels provide more aggressive early screening.

Design Guidelines

Data Fraction Progression

The data fractions should increase monotonically. Common progressions:

• Geometric: 0.1, 0.2, 0.4, 0.8, 1.0
• Linear: 0.2, 0.4, 0.6, 0.8, 1.0
• Aggressive: 0.1, 0.3, 1.0

The first level should use enough data to distinguish clearly good from clearly bad configurations. Using less than 10% risks making evaluations too noisy to be informative.

Number of Seeds

More seeds produce more reliable performance estimates but cost more time. A useful progression:

Single-seed evaluations at the first level are acceptable because the goal is only to eliminate the worst candidates, not to precisely rank the survivors.

Time Limits

Time limits serve as a safety valve. They should be generous enough that the optimizer can evaluate a reasonable number of candidates at each level:

If the time limit is too tight, AGILE may be forced to move to the next level before exploring the configuration space adequately. If it is too loose, no harm is done — the optimizer will move on when it converges.

As a rough guide, the total time across all levels should be at least 10x the cost of a single full-fidelity evaluation (full data, max seeds).

How Promotion Works

At each budget level, AGILE maintains a population of candidate configurations. After evaluating candidates at level $k$:

1. Candidates are ranked by their composite score (see Scoring Metrics).
2. The top fraction of candidates is promoted to level $k + 1$.
3. The remaining candidates are discarded.
4. At level $k + 1$, promoted candidates are re-evaluated at the higher fidelity, and new candidates generated by Levy flights are added to the population.

This successive halving ensures that each subsequent level evaluates fewer but better candidates with more computational resources.

Memory and Compute Costs

The budget configuration directly controls the total optimization cost:

For a rough estimate of total optimization time:

where $t_k$ is the time limit, $n_k$ is the number of candidates, $s_k$ is the number of seeds, $f_k$ is the data fraction, and $T_{\text{base}}$ is the cost of a single full-fidelity evaluation.

← AGILE Overview | Scoring Metrics →