# Prospective forecast and workflow test

This is a protocol for a later live pilot, not a completed evaluation. Freeze the event definitions, years, forecast timestamps, baselines and analysis code before seeing pilot outcomes. Rhine observation and source-cutoff rules are unchanged; its current ten-market ladder has a separate methodology identity. California retains its earlier contract definition.

## Forecast comparison

At each chosen forecast date, collect timestamped executable YES bids and asks with available size for every selected threshold, plus fees and any collateral or funding charge. Collect an insurer or actuary's probability forecast for the exact same threshold and source rule, documented before the outcome. Construct a simple public baseline from only earlier observations, with the same forecast date and no look-ahead. Record source release dates. Evaluate Rhine and California separately; never pool their scores.

Use common predeclared lead times, such as before the observation season or year and regular updates during it. At each lead time, compare only forecasts that existed together at that timestamp. Do not call a stale last trade an executable forecast. Translate the market quote into a documented bid, ask or midpoint signal and report sensitivity to that choice; a traded price is not automatically a physical probability. Record missing quotes and whether missingness is concentrated near extreme thresholds.

For all resolved questions report Brier score, log loss with a prespecified numerical clipping rule, calibration by forecast band, and errors on severe outcomes. Compare paired differences by observation year; all strikes from one year stay in the same resampling block (ten for the current Rhine ladder, 100 for California). Report uncertainty, all thresholds and exclusions. A small number of years, shared weather across strikes, risk premia and sparse trading can leave superiority inconclusive. The six frozen Rhine seasons and 20 California report years are settlement examples, not a backtest of nonexistent market prices.

Market quality is a separate gate: report executable basket size, all-leg completion rate, full-basket spread after fees, stale-quote rate, active participant counts if a venue can supply them, and quote availability during developing events. A forecast comparison that relies on untradeable midpoint prices cannot establish a usable insurance signal.

## Insurance decision and model-lifecycle comparison

A participating insurer defines one real decision before the test: an exact-trigger Rhine quote or risk update is the preferred first case. Match the event, information date, forecast horizon, update frequency, customer facts, policy terms and review standard. Compare internal development, vendor provision, a simple public baseline and the market-derived input.

Document the setup and recurring commitments for that forecast: source licences and preparation, model development and deployment, model/software licences, compute and storage, recalibration, validation and governance. Mark each component avoided, reduced, retained or added. Support the classification with model architecture, licence terms, operating commitments or supplier evidence. Separate new development that could be avoided from sunk development costs and existing shared infrastructure that cannot be recovered.

Record the market route's integration, data rights, quote archives, liquidity/staleness checks and fallbacks. Include market creation or liquidity support if the user must fund it. A retained fallback engine can preserve the supposed maintenance costs. No generic salary rate, assumed hours or arbitrary weighted score will establish economic value. Actual supplier quotes, operating bills and approved model budgets can support a later financial comparison.

Assess decision quality with the insurer's usual acceptance checks, plus whether trigger probabilities and resulting quote or risk decisions remain defensible. Blind a reviewer to workflow where feasible. A qualitative finding of avoidable modeling commitments supports testing the economic thesis; it is not a quantified savings result. See the model-lifecycle comparison for the component checklist and possible findings.

## Interpretation gate

A supported forecast claim needs better same-horizon scores with uncertainty and usable two-sided execution. A qualitative opportunity needs a meaningful component that can be avoided or retired, manageable new obligations and maintained decision quality. A supported monetary savings claim later needs lower observed total lifecycle cost without worse decision quality. If partners cannot provide comparable forecasts, executable quotes or workflow records, report the test as unavailable rather than interpreting a historical settlement replay as validation. The [NAIC catastrophe-model overview](https://content.naic.org/insurance-topics/catastrophe-models-property) explains why a peril signal cannot replace exposure, vulnerability and policy terms.

## Public cost evidence follow-up

The [cost assessment](/peril-markets/downloads/forecasting-cost-research.md) defines separate three-year comparisons for the Rhine duration curve and a hypothetical California statewide index cover. Neither currently supports a percentage. Match question, information date, forecast horizon and daily update frequency before comparing prices. Require public fixed-scope build costs, recurring commitments, licence cancellation terms, integration, monitoring, validation and fallback costs. Distinguish awards from actual payments and projected budgets. Include recipient-funded market creation or liquidity support. Missing amounts remain unknown; no salary proxy or arbitrary allocation is permitted. Any numerical planning scenario remains conditional on equivalent forecast and decision quality. No outreach is performed in this phase.