submitted by jtimperio to DataHoarder [link] [comments]
Intro:In my exploration of the world of big data and I became curious about tick data. Unfortunately, market data is almost always behind a paywall or de-sampled to the point of uselessness. After discovering the Dukascopy API, I knew I wanted to make this data available for all in a more accessible format. Over the course of a few months, I downloaded, cleaned, parsed, and compressed over a decade of Forex tick data on 37 currency pairs and commodities. Today I am happy to finally release the final result of my work to the DataHoarder community!
Download Links:Warning: I have rented a seedbox for the next 3 months from seedbox.io but I have been having some issues. If you have any issues with the torrent please leave a comment. Also, PLEASE SEED when you are done. This is quite a large data set and I can only push so much data on my own.
Torrent File: https://drive.google.com/file/d/18ymZWeFLJK7FggK_iiWZ-TxgWIVdJVvv/view?usp=sharingCompanion Blog Post: https://www.driftinginrecursion.com/post/dukascopy_opensource_data/
About the Data:The data was collected from https://www.dukascopy.com/ via a public API that allows for the download of tick data on the hour level. These files come in the form of a .bi5 file. The data starts as early as 2004 all the way to 2019.
These files were decompressed, then merged into yearly CSV’s named in the following convention. “AUDCHF_tick_UTC+0_00_2011.csv” or ‘Pair_Resolution_Timezone_Year.csv’
These CSV’s are split into 3 categories “Majors”, “Crosses”, “Commodities”.
Majors, Crosses, and Commodities have had their timestamps modified so that they are in the official UTC ISO standard. This was originally done for a Postgresql database that quickly became obsolesced. Any files that have been modified are appended with a “-Parse”. These timestamps have been modified in the following format.
Millisecond timestamps to UTC +00:00 time [2017.01.01 22:37:08.014] -- [2017-01-01T22:37:08.014+00:00]
User Resources:For those looking to use this data in a live context or update it frequently, I have included a number of tools for both Windows and Linux that will be useful.
WindowsThe ~/dukascopy/resources/windows folder contains a third party tool written in java that can download and convert Dukascopy’s .bi5 files. I have also included the latest zstd binaries from Zstandard Github page.
LinuxLinux is my daily driver in 99% of cases, so I have developed all my scraping tools using Linux only tools. In the ~/dukascopy/resources/linux folder you will find a number of shell script and pyhton3 files that I used to collect this data. There are quite a few files in this directory but I will cover the core ones below.
download-day.py:This file is used to download a single symbol for a single day and then convert and merge all 24 .bi5 files into a single CSV.
download-year.pyThis file is used to download a single symbol for a full year and then convert and merge all .bi5 files into a single CSV.
dukascopy.pyThis file contains all the core logic for downloading and converting data from dukascopy.
utc-timestamp-convert.pyThis tad slow but works well enough. It requires the pandas project and parses timestamps into the UTC ISO standard. This is useful for those looking to maintain the format of new files with the those in this repo, or those looking to use this in a SQL database.
|Rate (per day)||9.17||95.73|
Generated with BBoe's Subreddit Stats (Donate)
Download free historical data. Import historical data in MetaTrader, Excel, Forex Strategy Builder, Expert Advisor Studio. Load Historical Forex Data in Excel. Loading CSV (Comma Separated Values) files in Excel is straightforward. Download the necessary forex symbol files in Excel (CSV) format. Open a new Spreadsheet in Excel. Click on the Data tab in the navigation menu. You can import forex data from Get External Data-> From Text If you’re looking for Free Forex Historical Data, you’re in the right place! Here, you’ll be able to find free forex historical data ready to be imported into your favorite application like MetaTrader, NinjaTrader, MetaStock or any other trading platform.. Since the data is delivered in .CSV format (comma separated values), you can use it in any almost any application that allows you to ... Forex Tester allows you to import an unlimited number of currency pairs and years of history data in almost any possible text format (ASCII *.csv, *.txt). We strongly recommend importing 1-minute data for accurate testing (it is possible to import higher timeframes but testing results may not be as good). Chart data (including ticker symbols & indicators) can now be saved into a CSV file. It can then be imported into Microsoft Excel or other apps for further analysis, if that’s what you are into. To download the data choose Export Chart Data… from the menu: Select the chart & hit Export: Exporting becomes available as soon as chart data is ... Forex Historical Data App is absolutely free for all the traders who want to download Forex data CSV and use it to backtest trading strategies and Robots. Forex Historical Data App is FREE! The Forex Historical Data app is developed to solve one of the biggest problems that the beginner algo traders meet – the brokers do not provide a lot of bars. With this App, you will have Daily Data ... Download Free Forex Data. Download Step 1: Please, select the Application/Platform and TimeFrame! In this section you'll be able to select for which platform you'll need the data. MetaTrader 4 / MetaTrader 5. This platform allows the usage of M1 (1 Minute Bar) Data only. These files are well suited for backtesting trading strategies under MetaTrader 4 and MetaTrader 5 platform. Please, select ...
[index]          
How to download Royal Bank of Scotland RBS bank transactions to OPX and CSV for import to XERO and Kashflow accounting systems How to Backtest and download forex history data to you computer - Duration: 30:02. Fit Money 62,454 views. 30:02 . Get FREE historical data for Amibroker in 3 Simple Steps - Duration: 5:05 ... The Data Downloader is a MetaTrader 4 Tool that allow the user to export the forex data in CSV format. Available on MQL5 Market: https://mql5.com/3eq6p. In this video I show you how to delete previous MetaTrader 4 currency pair data and upload your own data to perform consistent back tests. Steps 1. Open MT4 ... How to export and download CSV in Node JS We are using fast-csv module to create CSV files. And a simple HTML anchor tag to download the file. Text tutorial:... This video shows you how to download your real-time Valarm sensor data in CSV format. Using the Valarm mobile app you can connect thousands of different sens... CommBank Simplify provides a single solution that lets you accept payments online securely and easily. Watch how to download a CSV file with CommBank Simplif...