Forex Data Cleaning: Handling Missing Bars and Bad Ticks
Proof path
Before trusting any backtest idea from this article, start from the historical forex data hub, then inspect a sample file and the current coverage report. The paid bundle path stays proof-first: sample, coverage, then order help if the data fits.
The phrase "garbage in, garbage out" applies directly to algorithmic trading. You can have a sophisticated model, but if your forex data cleaning process is weak, your backtests can be misleading before the first trade rule is evaluated.
Identifying Common Data Issues
Before you run a strategy, audit the dataset. The common problems are missing bars, duplicate timestamps, invalid OHLC rows, unrealistic outliers, inconsistent timezones, and source-specific session behavior. Missing bars can happen around market closes, holidays, low-liquidity periods, source outages, or incomplete downloads. Outliers can happen when a feed records a price that does not belong in the observed market sequence.
The important rule is simple: do not hide uncertainty. If an interval is missing from the source, mark it as missing. If a value is impossible, flag it. If a candle is synthetic because your platform created it, keep that fact visible.
Handling Missing Bars
There is a difference between analysis-time convenience and sellable historical data. For a chart preview, you might resample a series so your plotting library has regular timestamps. For a backtest or a commercial dataset, you should not create source history that was not observed.
Short gaps can be excluded, marked, or handled by explicit strategy rules. Multi-hour or multi-year gaps should be backfilled only from real source-observed data. Interpolation can be useful for some analytics, but it should not be used to pretend missing price action happened.
A Safer Pandas Audit Pattern
import pandas as pd
REQUIRED = ['timestamp', 'open', 'high', 'low', 'close']
def audit_forex_data(df):
missing_columns = [column for column in REQUIRED if column not in df.columns]
if missing_columns:
raise ValueError(f'Missing columns: {missing_columns}')
checked = df.copy()
checked['timestamp'] = pd.to_datetime(checked['timestamp'], utc=True)
checked = checked.sort_values('timestamp')
duplicate_count = checked.duplicated(subset=['timestamp']).sum()
invalid_ohlc = checked[
(checked['high'] < checked[['open', 'close']].max(axis=1)) |
(checked['low'] > checked[['open', 'close']].min(axis=1)) |
(checked['high'] < checked['low'])
]
expected = pd.date_range(
checked['timestamp'].min(),
checked['timestamp'].max(),
freq='1min',
tz='UTC'
)
missing_minutes = expected.difference(checked['timestamp'])
return {
'rows': len(checked),
'duplicate_timestamps': int(duplicate_count),
'invalid_ohlc_rows': len(invalid_ohlc),
'missing_minutes': len(missing_minutes),
}
Why Clean Data Matters for Backtesting
A single bad tick can trigger a fake stop-loss or take-profit. A hidden gap can make an indicator carry stale state across a period where the strategy should not have been allowed to trade. A timezone mismatch can make daily bars disagree with the platform you use for execution.
HistoricalFX packages source-observed OHLCV files with coverage reporting so buyers can see what is present and what is not. The clean forex data proof path explains how sample files, Major-8 QA, known-gap caveats, and audit routes fit together before a backtest depends on the data. You should still validate the files in your own environment, but the paid value is that the download, normalization, Parquet packaging, and baseline audit work have already been done.
If you already have broker exports, CSV archives, or MetaTrader history and want a second opinion, start with the sample forex data audit report. It shows the exact style of findings a paid forex data quality audit can return before any repair work is quoted.
Related Articles
Inspect the dataset before you buy
Start with the historical forex data hub, then the free EUR/USD sample and release coverage. If the schema and proof layer fit your workflow, the major-pair bundle is the practical first paid download.