Forex Data for Machine Learning: Preprocessing Guide

Machine learning does not rescue weak market data. It usually amplifies it. When working with forex machine learning data, the preprocessing workflow should make missing periods, duplicate timestamps, source changes, and label leakage visible before model training starts.

Start with Coverage, Not Features

Before building indicators or model inputs, inspect the date range and coverage profile for each pair. A model trained across a hidden outage, broker-specific artifact, or duplicated timestamp sequence can look impressive in a notebook and fail immediately in live testing.

import pandas as pd

df = pd.read_parquet("EURUSD_M1.parquet")
df["timestamp"] = pd.to_datetime(df["timestamp"], utc=True)
df = df.sort_values("timestamp")

print(df["timestamp"].min(), df["timestamp"].max())
print("duplicates", df["timestamp"].duplicated().sum())
print(df[["open", "high", "low", "close"]].isna().sum())

Feature Engineering Beyond OHLC

Raw OHLC bars are only the base layer. Common features include returns, rolling volatility, range, session flags, day of week, hour of day, and lagged values. Cross-pair features can be useful, but only if every pair is aligned on the same timestamp rules and missing-data policy.

df["return_1m"] = df["close"].pct_change()
df["range"] = (df["high"] - df["low"]) / df["close"]
df["hour_utc"] = df["timestamp"].dt.hour
df["dow"] = df["timestamp"].dt.dayofweek

Use Time-Based Splits

Random train/test splits leak future market regimes into the training set. Use time-based or walk-forward validation instead. The exact windows depend on the strategy, but the principle is fixed: train only on data that would have been available at the time.

Do Not Invent Missing Market History

If a source-backed candle is missing, treat that as a data-quality issue to investigate. Do not fill multi-hour or multi-year market-history gaps with averages and pretend the sequence is continuous. Short feature-level transformations such as returns or rolling windows are different from fabricating source candles.

Make the Dataset Reproducible

Pin your model run to a dataset release ID, coverage report, pair list, timeframe, and preprocessing script commit. That makes a model result auditable instead of just a one-off notebook output.

HistoricalForexPrices is built for that workflow: source-backed Parquet files, visible coverage metadata, sample downloads, and an optional data-quality audit for teams that need help evaluating their own archive. Review the coverage report, test the sample, or request a data-quality audit.