Data Providers
A DataProvider supplies candlestick history and symbol metadata to the VM. It serves two purposes:
- Main chart data —
Instance::run()pulls the primary K-line stream from it to drive bar-by-bar execution. request.security()data — MTF and cross-symbol calls also pull from the same provider.
Without a provider, Instance::run() returns immediately with no bars executed, and all request.security() calls return na.
Pass it as the first argument to Instance::builder(), then run:
let instance = Instance::builder(MyProvider::new(), source, timeframe, "NASDAQ:AAPL")
.build().await?;
instance.run_to_end("NASDAQ:AAPL", timeframe).await?;Trait Definition
#[async_trait(?Send)]
pub trait DataProvider {
type Error: fmt::Display + fmt::Debug + 'static;
async fn symbol_info(
&self,
symbol: String,
fields: SyminfoFields,
) -> Result<PartialSymbolInfo, DataProviderError<Self::Error>>;
fn candlesticks(
&self,
symbol: String,
timeframe: TimeFrame,
from_time: i64,
count: Option<usize>,
) -> Result<
impl Stream<Item = Result<CandlestickItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
>;
// Optional — implement to support tick-based timeframes (1T, 2T, …)
fn ticks(
&self,
symbol: String,
from_time: i64,
count: Option<usize>,
) -> Result<
impl Stream<Item = Result<TickItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
> {
Ok(stream::empty()) // default: no tick data
}
// --- Auxiliary data streams (all optional, default to empty) ---
fn currency_rate(&self, from: Currency, to: Currency, from_time: i64)
-> Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>;
fn financial(&self, symbol: String, financial_id: String, period: String,
currency: Option<Currency>, from_time: i64)
-> Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>;
fn earnings(&self, symbol: String, field: EarningsField,
currency: Option<Currency>, from_time: i64)
-> Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>;
fn dividends(&self, symbol: String, field: DividendsField,
currency: Option<Currency>, from_time: i64)
-> Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>;
fn economic(&self, country_code: String, field: String, from_time: i64)
-> Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>;
fn splits(&self, symbol: String, field: SplitsField, from_time: i64)
-> Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>;
fn data(&self, function: String, args: DataArgs, from_time: i64)
-> Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>;
}symbol_info uses async_trait(?Send) — implement it as a plain async fn. All stream-returning methods (candlesticks, ticks, currency_rate, etc.) return Result<impl Stream<Item = ...> + 'static, DataProviderError<Self::Error>> — return Err immediately when the symbol/request is invalid (see Error type below). On success, return Ok(stream) where the stream must not borrow self; clone any shared state before returning. For simple cases you can return an iterator-based stream directly; for branching logic, use Box::pin() internally.
All stream-returning methods support both finite and infinite streams:
- Finite — the stream yields all items and terminates. Typical for historical backtests. If a realtime bar/item is still open when the stream ends, the VM confirms it automatically.
- Infinite — the stream never terminates and keeps producing items after
HistoryEndas new data arrives. Typical for live trading and real-time monitoring.
Both modes use the same HistoryEnd-boundary protocol: items before HistoryEnd are loaded eagerly (blocking); items after HistoryEnd are consumed with non-blocking polling so a stalled live symbol never freezes execution. All auxiliary methods default to an empty stream.
candlesticks
Returns a stream of CandlestickItem values for the requested (symbol, timeframe) pair. count is Some(n) when a request stream asks for only a recent window; main-chart runs pass None.
Both modes use the same CandlestickItem protocol described below.
CandlestickItem
CandlestickItem is an enum with two variants:
CandlestickItem::Bar(candle)— a bar whose role (historical vs. realtime) is determined by its position relative toHistoryEnd. All bars beforeHistoryEndmust be fully closed and confirmed historical bars. AfterHistoryEnd,Baritems are treated as realtime bars — they may carry the same timestamp as the last historical bar (if the current period was still forming at query time) or a later timestamp (new bar). Consecutive items with the same timestamp update the bar in place (tick update). An item with a later timestamp implicitly confirms the previous bar and opens a new one.CandlestickItem::HistoryEnd— emitted exactly once after all confirmed historical bars have been sent, before any realtime updates. The DataProvider must ensure every bar beforeHistoryEndis a fully closed period. If the current period is still forming whenHistoryEndis sent, begin streaming it as realtimeBaritems afterHistoryEnd. If the stream ends with an open realtime bar, the VM confirms it automatically.
Stream sequence
The expected item sequence is:
Bar(t0) Bar(t1) … Bar(tN) ← all confirmed historical bars (all closed)
HistoryEnd ← boundary marker (emit once)
Bar(tA) ← first realtime bar
(tA may equal tN if period was forming at query time)
Bar(tA) ← same timestamp → update in place
Bar(tB) (tB > tA) ← later timestamp:
VM auto-confirms tA, opens tB
Bar(tB) ← update tB
… ← finite: stream ends here
infinite: keeps producingDo not emit an explicit Bar to confirm a realtime bar that has just closed. The VM handles confirmation internally when it receives the next Bar(tB) with a later timestamp.
| Requirement | Detail |
|---|---|
| Ascending order | Bars must be yielded in strictly ascending time order |
Start before from_time | Begin at or one bar before from_time (epoch ms of the first chart bar) so the VM can seed MTF state before the first chart bar opens |
Respect count when set | Return only the most recent n historical bars for that request stream |
HistoryEnd | Emit after all confirmed historical bars have been sent and before any realtime updates; all bars before HistoryEnd must be fully closed |
'static stream | The stream must not borrow self — clone handles before returning |
InvalidSymbol | Return Err(DataProviderError::InvalidSymbol(_)) from the method itself (not as a stream item) when the symbol is not supported |
| Other errors | Use Err(DataProviderError::Other(e)) — either from the method or as a stream item — for network failures and similar |
Error type
Use type Error = std::convert::Infallible when the only possible failure is InvalidSymbol:
type Error = std::convert::Infallible;ignore_invalid_symbol
When candlesticks returns Err(DataProviderError::InvalidSymbol(_)):
- If the Navi script called
request.security(..., ignore_invalid_symbol = true), the call silently returnsnafor every bar. - Otherwise the VM raises a runtime error.
This lets you distinguish "symbol not in dataset" (soft, ignorable) from "connection failed" (hard, always an error):
fn candlesticks(
&self,
symbol: String,
timeframe: TimeFrame,
_from_time: i64,
_count: Option<usize>,
) -> Result<
impl Stream<Item = Result<CandlestickItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
> {
if !self.supports(&symbol, timeframe) {
// Soft: script can opt in to receiving na instead of an error
return Err(DataProviderError::InvalidSymbol(
format!("{symbol}/{timeframe} not found"),
));
}
// ... normal path
Ok(my_stream)
}ticks
Returns a stream of TickItem values for the given symbol. Implement this to support tick-based timeframes (1T, 2T, …). When the chart or a request.security() call uses a tick timeframe, the VM calls ticks() instead of candlesticks() and synthesizes bars internally.
The default implementation returns an empty stream (no bars executed). Like candlesticks, this stream supports both finite and infinite modes.
TickItem
TickItem is an enum with two variants:
TickItem::Tick(tick)— a single market tick. TheTickstruct carriestime(epoch ms),price,volume,ask, andbid.TickItem::HistoryEnd— emitted exactly once, after all historical ticks and before any realtime ticks.
Bar synthesis rules
The VM synthesizes CandlestickItem::Bar values from the tick stream according to the requested timeframe quantity n:
| Timeframe | Rule | ask / bid |
|---|---|---|
1T | Each tick → one bar; open = high = low = close = price | Preserved from the tick |
nT (n > 1) | Every n ticks → one confirmed bar; OHLCV accumulated | Always na |
For nT, if HistoryEnd arrives while a partial bar is accumulating (tick count < n), the partial bar is flushed before HistoryEnd is propagated.
The ask and bid built-in variables are only meaningful on 1T charts; they return na on all other timeframes.
Example
fn ticks(
&self,
symbol: String,
from_time: i64,
count: Option<usize>,
) -> Result<
impl Stream<Item = Result<TickItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
> {
let items = self.load_ticks(&symbol, from_time, count);
Ok(async_stream::stream! {
for tick in items {
yield Ok(TickItem::Tick(tick));
}
yield Ok(TickItem::HistoryEnd);
// continue yielding TickItem::Tick(...) for realtime ticks
})
}symbol_info
Returns metadata for the given symbol string. All fields are Option — supply only what you know; the VM merges the rest with defaults derived from the exchange prefix.
Return PartialSymbolInfo::default() if you have no additional metadata:
async fn symbol_info(
&self,
_symbol: String,
_fields: SyminfoFields,
) -> Result<PartialSymbolInfo, DataProviderError<Self::Error>> {
Ok(PartialSymbolInfo::default())
}Using fields to optimize data fetching
fields is a bitset of the syminfo.* properties the compiled script actually uses. You may use it to skip expensive lookups for fields the script never reads. You are free to ignore it and always return all fields — the VM uses only what it needs.
async fn symbol_info(
&self,
symbol: String,
fields: SyminfoFields,
) -> Result<PartialSymbolInfo, DataProviderError<Self::Error>> {
let mut info = PartialSymbolInfo::default();
// Basic price metadata — cheap, always fetch
let meta = self.pool.query_basic(&symbol).await
.map_err(DataProviderError::Other)?;
info.min_move = Some(meta.min_move);
info.price_scale = Some(meta.price_scale);
info.currency = Some(meta.currency);
// Analyst data — expensive remote call, skip if not needed
if fields.intersects(
SyminfoFields::RECOMMENDATIONS_BUY
| SyminfoFields::RECOMMENDATIONS_SELL
| SyminfoFields::RECOMMENDATIONS_HOLD
| SyminfoFields::TARGET_PRICE_AVERAGE,
) {
let rec = self.pool.query_analyst_data(&symbol).await
.map_err(DataProviderError::Other)?;
info.recommendations_buy = rec.buy;
info.recommendations_sell = rec.sell;
info.target_price_average = rec.target_avg;
}
Ok(info)
}All fields
| Field | SyminfoFields flag | Description |
|---|---|---|
market | — | Exchange/market (Option<Market>). Variants: NYSE, NASDAQ, SHSE, SZSE, HKEX, SGX |
description | DESCRIPTION | Human-readable name, e.g. "Apple Inc." |
type_ | TYPE | Instrument type (Option<SymbolType>). See variants below |
country | COUNTRY | ISO 3166-1 alpha-2 country code of the exchange, e.g. "US", "HK" |
isin | ISIN | International Securities Identification Number |
root | ROOT | Root ticker for derivatives, e.g. "ES" for "ESZ4" |
min_move | MIN_MOVE, MIN_TICK | Numerator of mintick: mintick = min_move / price_scale |
price_scale | PRICE_SCALE, MIN_TICK | Denominator of mintick. E.g. min_move=1, price_scale=100 → mintick 0.01 |
point_value | POINT_VALUE | Point value multiplier. Usually 1.0; e.g. 50.0 for ES futures |
currency | CURRENCY | Price currency (Option<Currency>). See variants below |
expiration_date | EXPIRATION_DATE | Expiration date of the current futures contract |
current_contract | CURRENT_CONTRACT | Ticker identifier of the underlying contract |
base_currency | BASE_CURRENCY | Base currency for forex/crypto pairs, e.g. BTC in BTCUSD |
employees | EMPLOYEES | Number of employees (stocks only) |
industry | INDUSTRY | Industry classification |
sector | SECTOR | Sector classification |
min_contract | MIN_CONTRACT | Minimum contract size |
volume_type | VOLUME_TYPE | Volume interpretation: Base (base currency), Quote (quote currency), Tick (trade count) |
shareholders | SHAREHOLDERS | Number of shareholders |
shares_outstanding_float | SHARES_OUTSTANDING_FLOAT | Free-float shares outstanding |
shares_outstanding_total | SHARES_OUTSTANDING_TOTAL | Total shares outstanding |
recommendations_buy | RECOMMENDATIONS_BUY | Analyst buy recommendation count |
recommendations_buy_strong | RECOMMENDATIONS_BUY_STRONG | Analyst strong-buy recommendation count |
recommendations_hold | RECOMMENDATIONS_HOLD | Analyst hold recommendation count |
recommendations_sell | RECOMMENDATIONS_SELL | Analyst sell recommendation count |
recommendations_sell_strong | RECOMMENDATIONS_SELL_STRONG | Analyst strong-sell recommendation count |
recommendations_date | RECOMMENDATIONS_DATE | Date of latest analyst recommendations update |
recommendations_total | RECOMMENDATIONS_TOTAL | Total number of analyst recommendations |
target_price_average | TARGET_PRICE_AVERAGE | Average analyst price target |
target_price_date | TARGET_PRICE_DATE | Date of latest price target update |
target_price_estimates | TARGET_PRICE_ESTIMATES | Number of price target estimates |
target_price_high | TARGET_PRICE_HIGH | Highest analyst price target |
target_price_low | TARGET_PRICE_LOW | Lowest analyst price target |
target_price_median | TARGET_PRICE_MEDIAN | Median analyst price target |
See the SymbolType and Currency enum definitions in the Rust API docs for all variants.
Symbol Format
Symbols use the "EXCHANGE:TICKER" format. The VM passes the exact string from the Navi script without modification:
| Navi script | Passed to provider |
|---|---|
syminfo.tickerid | "NASDAQ:AAPL" (from the chart symbol) |
"BINANCE:BTCUSDT" | "BINANCE:BTCUSDT" |
Ticker expressions (e.g. "NASDAQ:AAPL*0.5+NYSE:SPY*0.5") are decomposed into individual symbol lookups before being passed to the provider.
TimeFrame
TimeFrame is passed directly from the Navi script's timeframe string argument:
| Navi string | Meaning |
|---|---|
"1" | 1-minute |
"5" | 5-minute |
"60" | 60-minute |
"D" | Daily |
"W" | Weekly |
"M" | Monthly |
"1T" | 1-tick (each tick is one bar; ask/bid available) |
"2T" | 2-tick (every 2 ticks form one bar) |
Vec<Candlestick>
Vec<Candlestick> implements DataProvider directly. For single-symbol offline backtests, pass the vector straight to Instance::builder() — no wrapper needed:
use navi_vm::{Candlestick, TimeFrame, TradeSession};
let bars = vec![
Candlestick::new(
1_700_000_000_000, // epoch ms
150.0, 155.0, 149.0, 153.0,
1_000_000.0, 0.0,
TradeSession::Regular,
),
// ... more bars in ascending time order
];
let mut instance = Instance::builder(bars, source, TimeFrame::days(1), "NASDAQ:AAPL")
.build()
.await?;
instance.run_to_end("NASDAQ:AAPL", TimeFrame::days(1)).await?;For lazy or streaming sources, collect into a Vec<Candlestick> first:
// Collect from an iterator before building
let bars: Vec<Candlestick> = load_bars_from_disk().collect();
let mut instance = Instance::builder(bars, source, TimeFrame::days(1), "NASDAQ:AAPL")
.build().await?;When you only need script metadata (inputs, script type, etc.), use the standalone script_info() function instead — it requires no data provider, symbol, or timeframe:
use navi_vm::script_info;
let info = script_info(source)?;Auxiliary Data Streams
In addition to candlesticks and symbol info, DataProvider has optional methods for supplying auxiliary data used by request.currency_rate(), request.financial(), request.earnings(), request.dividends(), request.economic(), request.splits(), and request.data().
All methods have default empty-stream implementations, so you only need to override the ones your data source supports.
AuxDataItem stream protocol
Auxiliary streams yield AuxDataItem values following the same protocol as candlestick streams (finite and infinite both supported):
Data(t0) Data(t1) … Data(tN) ← historical data points
HistoryEnd ← boundary marker
Data(tA) Data(tB) … ← optional realtime updatesBefore HistoryEnd is received, the VM drains the stream eagerly (blocking) to build the full historical dataset. After HistoryEnd, items are consumed non-blocking.
Available methods
currency_rate(from, to, from_time) — Streams timestamped exchange-rate data points so the VM can resolve request.currency_rate() on a per-bar basis. Also consumed internally when strategy.convert_to_account() / strategy.convert_to_symbol() need cross-currency conversion during backtesting. Each AuxDataPoint.value is the rate for converting one unit of from into to.
financial(symbol, financial_id, period, currency, from_time) — Streams periodic financial-metric values that the VM maps onto chart bars for request.financial(). financial_id is a provider-defined metric key (e.g. "TOTAL_REVENUE"), period is the reporting cadence ("FQ" / "FY"), and currency requests value conversion. Points are typically sparse — one per filing date.
earnings(symbol, field, currency, from_time) — Streams per-event earnings figures for request.earnings(). field is an EarningsField enum selecting which figure the provider should return (Actual, Estimate, or Standardized). The VM applies gaps and lookahead logic on top of the raw stream.
dividends(symbol, field, currency, from_time) — Streams per-event dividend amounts for request.dividends(). field is a DividendsField enum selecting Gross (pre-withholding) or Net (post-withholding). Like earnings, subject to gaps / lookahead processing by the VM.
economic(country_code, field, from_time) — Streams timestamped macroeconomic indicator values for request.economic(). country_code is an ISO 3166-1 alpha-2 code and field is a provider-defined indicator key.
splits(symbol, field, from_time) — Streams stock-split ratio components for request.splits(). field is a SplitsField enum selecting Numerator or Denominator. Like earnings / dividends, subject to gaps / lookahead processing.
data(function, args, from_time) — Routes custom data requests from request.data() calls in Navi scripts. function is a provider-defined routing key identifying which dataset to fetch. args is a DataArgs map built from the map<string, any> argument passed in Navi; use args.deserialize::<T>() with a #[derive(serde::Deserialize)] struct for type-safe parameter extraction. Returns series float data points aligned to chart bars via the same AuxDataItem protocol as the other auxiliary methods.
Each method returns Result<impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static, DataProviderError<Self::Error>>. Return Err(DataProviderError::InvalidSymbol(_)) from the method itself when the identifier is not recognised — the VM handles it the same way as for candlestick streams (ignore_invalid_symbol / ignore_invalid_currency).
Example
fn currency_rate(
&self,
from: Currency,
to: Currency,
_from_time: i64,
) -> Result<
impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
> {
let pool = Arc::clone(&self.pool);
Ok(async_stream::stream! {
match pool.query_fx_rates(from, to).await {
Err(e) => yield Err(DataProviderError::Other(e)),
Ok(rates) => {
for rate in rates {
yield Ok(AuxDataItem::Data(AuxDataPoint {
time: rate.timestamp_ms,
value: rate.rate,
}));
}
yield Ok(AuxDataItem::HistoryEnd);
}
}
})
}fn data(
&self,
function: String,
args: DataArgs,
from_time: i64,
) -> Result<
impl Stream<Item = Result<AuxDataItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
> {
#[derive(serde::Deserialize)]
struct MyMetricParams {
symbol: String,
period: i64,
adjusted: bool,
}
match function.as_str() {
"my_metric" => {
let params = args
.deserialize::<MyMetricParams>()
.map_err(|e| DataProviderError::Other(MyError::from(e)))?;
Ok(Box::pin(self.fetch_my_metric(params, from_time)))
}
_ => Ok(Box::pin(futures_util::stream::empty())),
}
}The corresponding Navi script:
// No arguments
vix = request.data("vix")
// With typed arguments
args = Map.new<string, any>()
args.put("symbol", "AAPL")
args.put("period", 14)
args.put("adjusted", true)
val = request.data("my_metric", args)Custom In-Memory Provider Example
When you need multi-symbol support (e.g. for request.security()), implement DataProvider directly:
use std::{collections::HashMap, convert::Infallible};
use futures_util::{Stream, stream};
use navi_vm::{
Candlestick, CandlestickItem, DataProvider, DataProviderError, PartialSymbolInfo, TimeFrame,
TradeSession,
};
struct InMemoryProvider {
data: HashMap<(String, TimeFrame), Vec<Candlestick>>,
}
#[async_trait::async_trait(?Send)]
impl DataProvider for InMemoryProvider {
type Error = Infallible;
async fn symbol_info(
&self,
_symbol: String,
_fields: SyminfoFields,
) -> Result<PartialSymbolInfo, DataProviderError<Self::Error>> {
Ok(PartialSymbolInfo::default())
}
fn candlesticks(
&self,
symbol: String,
timeframe: TimeFrame,
from_time: i64,
count: Option<usize>,
) -> Result<
impl Stream<Item = Result<CandlestickItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
> {
let bars = self
.data
.get(&(symbol.clone(), timeframe))
.cloned()
.unwrap_or_default();
// Start one bar before from_time so the VM can seed MTF state.
let start = bars
.partition_point(|b| b.time < from_time)
.saturating_sub(1);
let sliced = bars[start..].to_vec();
let recent = if let Some(count) = count {
let start = sliced.len().saturating_sub(count);
sliced[start..].to_vec()
} else {
sliced
};
Ok(stream::iter(
recent
.into_iter()
.map(|c| Ok(CandlestickItem::Bar(c)))
.chain(std::iter::once(Ok(CandlestickItem::HistoryEnd))),
))
}
}Usage:
let mut provider = InMemoryProvider { data: HashMap::new() };
provider.data.insert(
("NASDAQ:AAPL".to_string(), TimeFrame::weeks(1)),
vec![
Candlestick::new(
1_700_000_000_000, // epoch ms
150.0, 155.0, 149.0, 153.0,
1_000_000.0, 0.0,
TradeSession::Regular,
),
// ... more bars in ascending time order
],
);
let mut instance = Instance::builder(provider, source, TimeFrame::days(1), "NASDAQ:AAPL")
.build()
.await?;
instance.run_to_end("NASDAQ:AAPL", TimeFrame::days(1)).await?;Async Database Provider Example
For providers that fetch data from a remote API or database:
use std::sync::Arc;
use futures_util::Stream;
struct DatabaseProvider {
pool: Arc<DbPool>,
}
#[async_trait::async_trait(?Send)]
impl DataProvider for DatabaseProvider {
type Error = DbError;
async fn symbol_info(
&self,
symbol: String,
_fields: SyminfoFields,
) -> Result<PartialSymbolInfo, DataProviderError<Self::Error>> {
self.pool
.query_symbol_info(&symbol)
.await
.map_err(DataProviderError::Other)
}
fn candlesticks(
&self,
symbol: String,
timeframe: TimeFrame,
from_time: i64,
count: Option<usize>,
) -> Result<
impl Stream<Item = Result<CandlestickItem, DataProviderError<Self::Error>>> + 'static,
DataProviderError<Self::Error>,
> {
let pool = Arc::clone(&self.pool);
Ok(async_stream::stream! {
match pool.query_candles(&symbol, timeframe, from_time, count).await {
Err(e) => yield Err(DataProviderError::Other(e)),
Ok(rows) => {
for row in rows {
yield Ok(CandlestickItem::Bar(row.into_candlestick()));
}
// Signal end of history so the VM switches to non-blocking poll
yield Ok(CandlestickItem::HistoryEnd);
}
}
})
}
}See Also
- Executing Scripts — running the VM with your data provider
- Security & Limits —
max_security_depthandmax_security_callslimits