Latencies and Jitters of Cryptocurrency Real Time Market Data Feeds
Sunwoo Park, Ph.D. (spark@exxablock.com)
ExxaBlock, Inc.
June 28, 2020
The majority of cryptocurrency exchanges publish their realtime market data through WebSocket[1] client-server connections. These connections are established directly between exchanges and their customers, or indirectly through third-party WebSocket proxy providers[2]. Each connection allows the customer to subscribe to specific data channels. Then receive streaming data from an exchange via the channel. This data includes but is not limited to quotes, trades, orderbook snapshots, and orderbook updates. Depending on geological locations and system/network configurations of an exchanges market data WebSocket servers, individual customers can see varying network latencies and jitters from the exchange.
Latencies and jitters of market data play an essential role in algorithmic (or systematic) trading and realtime market surveillance. They are directly involved in (risk-free) latency arbitrages, order fill ratio improvement, and market manipulation detection (e.g., wash trading and quote spoofing). As a result, identifying exchanges that provide the minimum latency with the stable jitter becomes critical for institutional and retail market participants.
We measured and computed the average latencies and jitters of 81 exchange WebSocket market data servers hosted on US, Europe, Asia, and Australia sites. This quantitative task has been achieved through the following steps.
· For each exchange, latencies are measured for 30 times, followed by removing the best and worst 5 experiments.
· Average latency is computed over 20 measurements.
· Average jitter is computed over 19 differences of 20 measurements.
· Above steps are repeated at four different geological locations, which are
o Amazon Web Service (AWS) us-east-1 Datacenter at Ashbury, VA, US
o Amazon Web Service (AWS) us-west-1 Datacenter at San Francisco, CA, US
o IBM Cloud wdc1 Datacenter at Washington D.C., US
o Verizon Fios residential site at Jersey City, NJ, US
Among 81 exchange WebSocket market data servers, 41 servers (50.6%) are hosted at Cloudflare (https://www.cloudflare.com/). Cloudflare provides industry-leading web-securities (e.g. DDoS Attack protection), global data caching, and dynamic content delivery through its world-wide private backbone networks. 18 servers (22.2%) are hosted or connected at Amazon Web Services (AWS) (https://aws.amazon.com/). AWS provides two options — (i) hosting at the AWS data centers and (ii) private connections between the customer sites and AWS network hubs (a.k.a., AWS Direct Connect[3]). 9 servers (11.1%) are hosted at Barefruit (https://www.barefruit.com/). 4 servers (4.9%) are hosted at AliCloud (https://www.alibabacloud.com/). Other servers are hosted at Internet Service Providers (ISPs) and content delivery networks (e.g., Akamai).
WebSocket market data servers accessible through US ISPs
50 of 81 (61.7%) exchange WebSocket market data servers are reachable through US ISPs. Among 50 servers, 19 servers show the better connectivity (i.e., lower latencies) from the US East coast (i.e., AWS us-east-1(10), IBM wdc-01(3), and Fios(6)) as shown Table 1; and 31 servers does from the US West coast (i.e., AWS us-east-1) in Table 2.
Based on average latencies and jitters shown in table 1 and 2, market data of some exchanges (e.g., zbg, graviex, crex24, paymium, bitbay, ant bitz) seems dynamically delivered from out-side US (e.g., Europe and Asia) servers through Cloudflare global content delivery networks.
In most measurements, exchanges servers hosted at Cloudflare provide less/lower latencies than others. It indicates that Cloudflare geological data caching and dynamic content delivery networks deliver what they claim. Measurements at AWS us-east-1 provide less/lower latencies than AWS us-west-1 measurements. However, the latter shows better jitters than the former. It is interesting to see AWS us-west-1 has less/lower latencies than AWS us-east-1 for the servers connected through AWS Direct Connect at Newark, NJ, US. It is not clear whether it comes from the better network infrastructure between AWS us-west-1 and AWS Direct Connect, or higher network traffic between AWS us-east-1 and AWS Direct Connect.
It is notable that Cloudflare hides real IP addresses of servers hosted at its network. It forces all public IP address records of the servers to point to Cloudflare proxy. As a result, it is hard/impossible to find real geological locations of the servers hosted at Cloudflare. However, some servers can be located through CrimeFlare site (http://www.crimeflare.org/)[5].
Based on all measurements followed by quantitative analysis on them our team believes,
· run market data consumers at US east coast servers (e.g., AWS us-east-1) If you want to receive exchange market data as soon as possible (e.g., to perform latency arbitrage, smart order routing, and to improve order fill ratios),.
· run market data consumers at US west coast servers (e.g., AWS us-west-1), instead, if you prefer more predictable/stable jitters,.
· AWS is preferable to IBM Cloud[6].
WebSocket market data servers accessible through ISPs in Europe and Asia-Pacific Region
As expected, latencies increase linearly as the locations of exchange WebSocket market servers far away from measurement sites. Measurements at US east coast sites (i.e., AWS us-east-1, IBM wdc-1, and Fios residential site) reveals lower latencies to servers hosted at Europe. Similarly, measurement at US west coast site (i.e., AWS us-west-1) shows lower latencies to servers hosted at Asia and Pacific region. Most of the exchange WebSocket servers are hosted at Barefruit in Europe and AWS in Asia-Pacific region.
[1] https://tools.ietf.org/html/rfc6455
2 Pusher.com (https://pusher.com/) is a widely used WebSocket proxy service provider.
[3] AWS Direct Connect (https://aws.amazon.com/directconnect/) provides the low-latency private connections between customer data centers (e.g, Equinix) and AWS Direct Connect facilities.
[4] ExxaBlock identified real IP addresses and their geological information on a collection of exchange WebSocket servers hosted at Cloudflare using CrimeFlare data and other resources. The information will be provided to ExxaBlock customers in the future, upon request, after the full verification of the information completes.
[5] (https://www.crimeflare.org/)
[6] Upon Customers’ request, ExxaBlock will provide them more extensive test reports on all major Cloud providers that include but are not limited to Google Cloud, Microsoft Azure, and Oracle Cloud.
