WebSocket Market Data Server Latency Report at Market Close

Update of Latency Measures — January 18, 2021

Contents

1. Background
2. Test and Measurement
3. Market Data Server Locations
4. US Market Data Servers
5. Non-US Market Data Servers

Appendices

• Cryptocurrency Exchanges
• Outlier Detection Methodology

A.1 Z-score Method

B.2 IQR Method

API and Database Access

List of Figures

2.1 Latency Test, Measurement, and Daily Report Generation: latencies between 3 different ExxaBlock test sites(T1, T2, and T3) and exchanges (EX1..n) are measured at 7:30 a.m. and 4:00 p.m for market-open and market-close reports, respectively (mt). All test results (J1t, J2t, and J3t) are copied into a main test site (T1). A daily latency report is generated by aggregating and analyzing those results at 20 minute after tests start — 7:50 a.m. for market-open and 4:20 p.m. for market-close reports (Rt+20). An individual test measures the difference between two time points — (1) time just after
a “ping” message(pn) is sent to a WebSocket market data server and (2) time just after receiving a “pong”message(pn)from the server.

3.1 Exchange WebSocket Streaming Market Data Server Hosting and Connectivity Providers: CloudFlare (51.9%) dominates the US market. AWS (22.78%) and Barefruit (11.39%) follow the lead. Others are a collection of all ISPs that have less than 3% market share, which are Akamai (2.53%), A100ROW (1.27%), OVH SAS (1.27%), Hetzner (1.27%), Quantil (1.27%), and Microsoft (1.27%).

B.1 Latency Outlier Detection with Z-scores 3 (Z3): All unique latencies that are away from the mean(μ) of their distribution (X) more than ±3 standard deviation (σ) are classified into outliers. Only 0.27% or less of unique latencies are considered to outliers in Z3.

B.2 Latency Outlier Detection with IQR 1.5 (IQR1.5): m􏰂 is an median of latency data(X). Q1 is the median between Xmin and m􏰂. Q3 is the median between m􏰂 and Xmax. IQR is Q3 — Q1. IQR1.5 is the aggregation of all latencies that are located at lower than Q1–1.5*IQR and higher than Q3 + 1.5*IQR.

List of Tables

4.1 Best Server Connectivity at the US East Test Sites (Excluding non-US ISPs): It lists WebSocket streaming market data servers that provide the minimum latencies at the US East test sites. aet, awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc 1 test sites.

4.2 Best Server Connectivity at the US West Test Site (Excluding non-US ISPs): It lists WebSocket streaming market data servers that provide the minimum latencies at the US West test sites. aet, awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc1.

4.3 Statistical Summary on Latency Measurement Tests on Exchange WebSocket Market Data Servers hosted on US ISPs: WNn is the number of market data servers(n) that shows lower latencies on either SE or SW . N is the total number of exchanges WebSocket market data servers that are reachable through US domestic hosting service providers (e.g., CDNs, ISPs, and Data centers). μ, m􏰂σ are mean, median, and standard deviation of the latencies of the n servers. μ±1σ and μ±2σ are the number of market data servers among the n servers that are located within 1 and 2σ from the μ, respectively. Z3 and IQR1.5 are the numbers of outliers identified by Z-score and IQR outlier detection methods.

5.1 Average Latencies and Jitters of WebSocket Streaming Market Data Servers hosted on US ISP: aet, awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc 1.

5.1 Average Latencies and Jitters of WebSocket Streaming Market Data Servers hosted on Non-US ISP: aet,awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc 1.

A.1 A List of Exchanges Supporting WebSocket Market Data Streaming.

1 | Background

The majority of cryptocurrency exchanges publish their real time streaming market data through WebSocket1 client-server connections. A connection is established directly between an exchange and their customer, or indirectly channel over a TCP/IP . It allows the customer to subscribe a specific market data channel (a.k.a. endpoint). Most exchanges provide public WebSocket endpoints for their market data access. A small number of exchange allows only registered customers to access their endpoints. WebSocket streaming market data server are hosted at Content Delivery Networks (CDNs) (e.g. CloudFlare), Cloud Computing Providers (e.g., AWS4), or Data Centers (e.g., Equinix 5 ).

Real Time streaming market data include but are not limited to quotes, trades, orderbook snapshots, and orderbook updates. Most exchanges adopt JSON6 format to support a wide range of audience with a high degree of flexibility and easy-to-use (e.g., nested/hierarchical specifications of parent-children and trench orders). Other formats such as FIX7 (e.g., itBit8) and JSON-RPC9 (e.g., bitFlyer10) are also used at some exchanges.

Latencies 11 and jitters 12 of streaming market data play an essential role in algorithmic (or systematic) trading and real time market surveillance. They are directly involved in (risk-free) latency arbitrages, order fill ratio improvement, and market manipulation detection (e.g., wash trading and quote spoofing). As a result, identifying exchanges that provide the minimum latency with the stable jitter becomes critical for institutional and retail market participants. In spite of their imperative roles, latencies and jitters of exchange streaming market data servers are not fully investigated yet.

Latency and Jitter are measured in a few different ways. The easiest way is to use the timestamp that exchanges 13 is no time synchronization 14 is guaranteed between exchanges and their customers. Without the synchronization, through third-party WebSocket proxy providers . The connection is a full-duplex (i.e., bidirectional) communication 3 provide as part of their streaming market data . However, it has a few drawbacks. The most critical problem the measurement errors are typically 10th milliseconds (e.g., 10 to 40 milliseconds) with or without negative time 15 differences . Most exchanges does not publish a list of time synchronization servers that their market servers are associated with. It makes institutional market participants not to rely on this approach. Instead, pragmatically, the “ping-pong” method is widely used. The method computes the latency by measuring the time difference between two time points — one for time when “ping” data are sent to the market data server and the other for time when “pong” data are received.

ExxaBlock publishes daily reports on latencies and jitters of exchange WebSocket streaming market data servers twice a day — one at market open and the other at market close. It is expected that the reports help many market participants in cryptocurrency capital markets. The measurement data and their analytics are also available in 16 real time using ExxaBlock real time streaming APIs.

1. https://tools.ietf.org/html/rfc6455
2. Pusher.com(https://pusher.com/) is a widely used WebSocket proxy service provider.
3. 3https://tools.ietf.org/html/rfc1180
4. https://aws.amazon.com/
5. https://www.equinix.com/data-centers/
6. https://www.json.org/json-en.html
7. https://www.fixtrading.org/what-is-fix/
8. https://api.itbit.com/marketdata
9. https://www.jsonrpc.org/specification
10. https://bf-lightning-api.readme.io/docs/endpoint-json-rpc
11. https://en.wikipedia.org/wiki/Low_latency_(capital_markets)
12. https://en.wikipedia.org/wiki/Jitter
13. some exchange does not provide timestamp in a certain market data components.
14. https://tf.nist.gov/sim/2017_Seminar/SIM_2017_Time_in_Financial_Markets.pptx
15. With the adaptation of a new fine-resolution time synchronization method, a.k.a. Precision Time Protocol(PTP) — https://standards.ieee.org/standard/1588–2019.html, the synchronization errors could be reduced into a sub-microsecond range. However, the protocol is not widely deployed into the market, yet.
16. Please contact ExxaBlock Support (jwiehl@exxablock.com) for more information on access to ExxaBlock real time streaming data.

2 | Test and Measurement

ExxaBlock measures and computes the latencies and jitters between ExxaBlock test sites and 79 exchanges Websocket market data streaming servers hosted on US(50), Europe(16), and Asia-Pacific(13) CDNs, Cloud Computing Providers, and data centers (Hereafter, we use the term ISP to represent all of them). For an individual exchange market data server,

1. latencies are measured for 30 times, followed by removing the best and worst 5 measurements.
2. Average latency is computed over 20 measurements.
3. Average jitter is computed over 19 differences of 20 measurements.
4. Above steps are repeated at three different geological locations, which are

• Amazon Web Service (AWS) us-east-1 datacenter at Ashbury, VA, US.
• Amazon Web Service (AWS) us-west-1 data center at San Francisco, CA.US.
• IBM Cloud wdc1 datacenter at Washington D.C., US.

All tests are executed on three different ExxaBlock test sites independently at 7:30 a.m. and 4:00 p.m. for daily market-open and market-close report respectively 1 (hereafter, all times are US Eastern time). Aggregated test results and their analytics are published in the form of a daily report and a JSON file at 7:50 a.m. and 4:20 p.m. ExxaBlock daily latency reports are currently published at ExxaBlock Web Store and AWS Marketplace.

Figure 2.1: Latency Test, Measurement, and Daily Report Generation: latencies between 3 different ExxaBlock test sites(T1, T2, and T3) and exchanges (EX1..n) are measured at 7:30 a.m. and 4:00 p.m for market-open and market close reports, respectively (mt). All test results (J1t, J2t, and J3t) are copied into a main test site (T1). A daily latency report is generated by aggregating and analyzing those results at 20 minute after tests start — 7:50 a.m. for market-open and 4:20 p.m. for market-close reports (Rt+20). An individual test measures the difference between two time points — (1) time just after a “ping” message(pn) is sent to a WebSocket market data server and (2) time just after receiving a “pong” message(pn) from the server.

3 | Market Data Server Locations

Among 79 exchange WebSocket streaming market data servers, 41 servers (51.90%) are hosted at Cloudflare. Cloudflare provides industry-leading Web securities (e.g., DDos attack protection), global data caching (a.k.a. geocaching), and dynamic content delivery through its world-wide private backbone networks. 18 servers (22.78%) are hosted or connected at AWS. AWS provides two options — (i) hosting at the AWS data centers and (ii) private 1 connections between the customer site and AWS Direct Connect
private connections between customer data centers (e.g., Equinix) and AWS Direct Connect facilities. 9 servers. AWS Direct Connect provides the low-latency 23 (11.39%) are hosted at Barefruit . 4 servers (5.06%) are hosted at AliCloud . Other servers are hosted at Internet Service Providers (ISPs) and CDNs(e.g., Akamai).

Figure 3.1: Exchange WebSocket Streaming Market Data Server Hosting and Connectivity Providers: CloudFlare (51.9%) dominates the US market. AWS (22.78%) and Barefruit (11.39%) follow the lead. Others are a collection of all ISPs that have less than 3% market share, which are Akamai (2.53%), Quantil (1.27%), Hetzner (1.27%), Microsoft (1.27%), A100ROW (1.27%), and OVH SAS (1.27%).

It is notable that Cloudflare hides real IP addresses of servers hosted at its networks. It forces all public IP address records of the servers to point to Cloudflare proxy. As a result, it is hard/impossible to find geological locations of the servers hosted at Cloudflare. However, some servers can be located through CrimeFlare site 4,5 6 . Downlink speeds and their latencies to major Cloud computing providers can be tested on CloudHarmony. Specifically, download speeds and their latencies to AWS, Google Cloud, IBM Cloud, Microsoft Azure and Oracle Cloud are measured through http://cloudharmony.com/speedtest-for-aws-google-ibm-oracle-and-azure. Link-level 7 latencies between any two AWS data centers are published in Cloudping.io.

1. https://aws.amazon.com/directconnect/
2. https://www.barefruit.com/
3. https://www.alibabacloud.com/
4. http://www.crimeflare.org
5. ExxaBlock identified real IP addresses and their geological information on a collection of exchange WebSocket market data servers hosted at Cloudflare using textit CrimeFlare data and other resources. The information will be provided to ExxaBlock customers in the form of database or text files in the future, upon request, after the full verification of the information completes.
6. http://cloudharmony.com/
7. https://www.cloudping.co/grid

4 | US Market Data Servers

50 of 77 (64.94%) exchange WebSocket market data servers are reachable through US ISPs. Among 50 servers, 18 servers show the better connectivity (i.e., lower latencies) from the US East test sites (i.e. AWS us-east-01(16) and IBM wdc1(2). 32 servers does from the US West test site (i.e., AWS us-west-1) as shown in Table 4.1.

Table 4.1: Best Server Connectivity at the US East Test Sites (Excluding non-US ISPs): It lists WebSocket streaming market data servers that provide the minimum latencies at the US East test sites. aet, awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc1 test sites.

Market data servers at the US East test sites(SE) show 35.17% lower latency than market data servers at the US West test site (SW ) in terms of the average latency for exchanges shown in the above table. binanceus exhibits the most significant latency reduction (87.73%) at SE . Statistically, mean(μSE ) and standard deviation(σSE ) of SE (a.k.a. standard measures) are 41.766 and 54.193, respectively. median(m􏰃SE) and median absolute deviation 1 (σ􏰃SE ) (a.k.a. robust measures) are 21.155 and 6.523. coinall(11.438), nlexch(13.203), coinbasepro(14.058), bitforex(15.207), binanceus(16.563), bigone(17.197), bittrex(17.885), ftx(18.98), bitfinex(20.941), cex(21.37), gem- ini(22.057), idex(25.245), p2pb2b(33.255), poloniex(34.772), coinbene(54.463), and graviex(62.607) are within 1σSE from the μSE. zbg(121.62) is within 2σSE from the μSE. 1 outlier is found in the Z-score method (Z3) — paymium(230.934). Z3 considers all latencies that are located at the outside of the μSE ±3σSE band. 2 outliers are found in the IQR method (IQR1.5) — zbg(121.62), and paymium(230.934). IQR1.5 takes all latencies that are lower than Q1–1.5*IQR or higher than Q3+1.5*IQR. Q1 and Q3 are the first and third quartile of sorted latency data in ascending order, respectively. IQR is a positive/absolute difference between them (i.e., Q3 — Q1). Please See Appendix B for more information on outlier detection methods used in this paper.

Table 4.2: Best Server Connectivity at the US West Test Site (Excluding non-US ISPs): It lists WebSocket streaming market data servers that provide the minimum latencies at the US West test sites. aet, awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc1.

Table 4.3: Average Latencies and Jitters of WebSocket Streaming Market Data Servers hosted on US ISP: aet, awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc1.

5 | Non-US Market Data Servers

As expected, latencies increases linearly as the location of exchange WebSocket market data servers far away from measurement sites. Measurements at US East test sites (i.e., AWS us-east-1, and IBM wdc1) reveals lower latencies to servers hosted at Europe. Similarly, measurements at US West test site (i.e., AWS us-west-1) shows lower latencies to servers hosted at Asia-Pacific region. Most exchange WebSocket servers are hosted at Barefruit in Europe and AWS in Asia-Pacific region.

Table 5.1: Average Latencies and Jitters of WebSocket Streaming Market Data Servers hosted on Non-US ISP: aet, awt and iet are AWS us-east-1, AWS us-west-1 and IBM wdc1.

Appendices

A | Cryptocurrency Exchanges

Table A.1 lists all exchanges that use the WebSocket protocol to publish their streaming market data 1

1. WebSocket endpoints of individual exchanges are available upon signing ExxaBlock non-disclosure agreement on data use and redistribution.

B | Outlier Detection Methodology

Two outlier detection methods, Z-score and InterQuartile Range(IQR), are applied to latency measurement data in this report.

B.1 Z-score Method

All latencies that their Z-scores are less than -3 or greater than +3 (Z3) are considered as outliers in the report. Alternatively, outliers are defined by all latencies located at 3 standard deviation (3σ) away from the mean (μ) of latency distribution. That is,

Figure B.1: Latency Outlier Detection with Z-scores 3 (Z3): All unique latencies that are away from the mean (μ) of their distribution (X) more than ±3 standard deviation (σ) are classified into outliers. Only 0.27% or less of unique latencies are considered to outliers in Z3.

B.2 IQR Method

IQR1.5 considers all latencies that are lower than Q1–1.5*IQR and higher than Q3+1.5*IQR. In a standard normal distribution (i.e., N(0,σ2)), Q1–1.5·IQR and Q3 + 1.5·IQR are equivalent to -2.698σ and 2.698σ, respectively. As a result, IQR1.5 detects slightly more outliers than Z3 for the normal data distribution. IQR1.5 is visually described by

Figure B.2: Latency Outlier Detection with IQR 1.5 (IQR1.5): m􏰂 is an median of latency data(X). Q1 is the median between Xmin and m􏰂. Q3 is the median between m􏰂 and Xmax. IQR is Q3 — Q1. IQR1.5 is the aggregation of all latencies that are located at lower than Q1–1.5*IQR and higher than Q3 + 1.5*IQR.

IQR1.5 is based on a robust measure median (m􏰂). The median is less sensitive to outliers than the mean (μ). As a small number of outliers are far away from the mean of the data (i.e., |μ − m􏰂 | ≫ 0), Z3 becomes skewed to the direction where those outliers exist. Meanwhile, IQR1.5 becomes less effective from those outliers.

C | API and Database Access

ExxaBlock supports real time and historic APIs and Database to support programmable access to daily latency reports as well as raw measurement and analytic data. Please contact ExxaBlock Support (jwiehl@exxablock.com) if interested.

Copyright Notice Copyright © 2020 ExxaBlock Inc.

ALL RIGHT RESERVED.

ALL INTELLECTUAL PROPERTY RIGHTS IN THE REPORT AND THE INFORMATION PUBLISHED HEREIN ARE THE EXCLUSIVE PROPERTY OF EXXABLOCK INC., AND/OR ITS LICENSORS (INCLUDING EXCHANGES) AND MAY ONLY BE USED UNDER LICENSE FROM EXXABLOCK INC. WITHOUT LIMITING THE FOREGOING, BY ACCESSING THIS REPORT YOU AGREE THAT YOU WILL NOT COPY OR REPRODUCE OR USE ANY PART OF ITS CONTENTS (INCLUDING, BUT ARE NOT LIMITED TO, SINGLE MEASUREMENTS OR ANY OTHER INDIVIDUAL ITEMS OF DATA) IN ANY FORM OR FOR ANY PURPOSE WHATSOEVER EXCEPT UNDER VALID LICENSE FROM EXXABLOCK INC.

Disclaimer And Liability

THIS REPORT IS FOR INFORMATION PURPOSE ONLY AND REPRESENTS NEITHER INVESTMENT ADVICE, NOR AN INVESTMENT ANALYSIS OR AN INVITATION TO BUY OR SELL FINANCINSTRUMENTSENTS. SPECIFICALLY, THE DOCUMENT DOES NOT SERVE AS A SUBSTITUTE FOR INDIVIDUAL INVESTMENT OR OTHER ADVICE. THE STATEMENTS CONTAINED IN THIS REPORT ARE BASED ON KNOWLEDGE AS OF THE TIE OF PREPARATION (”AS-IS”) AND ARE SUBJECT TO CHANGE AT ANY TIME WITHOUT FURTHER NOTICE. EXXABLOCK DISCLAIMS ALL WARRANTIES OF ANY RELATEDATED TO THE PRODUCTS, WHETHER EXPRESS OR IMPLIED. EXXABLOCK INC. AND ITS OFFICERS AND EMPLOYEES DO NOT PROVIDE ANY LIABILITY TO ANYONE RELATED TO THE PRODUCTS.

Customer Support and Sales
Jack Wiehl (jwiehl@exxablock.com)
244 5th Avenue, Suite #P284, New York, NY 10001, United States of America.