More Nanex Research
Update (2/15): Added charts (#2 and #5) showing data from the exchange
perspective - duplicate trades and quotes
removed and plotted with queued (old) exchange timestamps. Added chart (#3) of USO aligned in time with other 1 second charts.
NYMEX Black Swan ~ The Energy Complex Data Feed Loop
On February 13, 2012, starting 13:59:57, quotes and trades for crude oil began queuing. At 14:00:35,
all of the queued messages were sent at once. Again at 14:01:08 the same 38 second block
of messages sent earlier was sent again -- old timestamps and all plus a few new messages.
Again at 14:01:18, all messages since 13:59:57 were sent again. This repeated 12 times.
From a programmer's perspective, it looks like a system problem led to a synchronization
corruption in either a circular queue (head/tail pointers), or in the send acknowledgment
mechanism, causing the software to believe the queue was full all the time.
To the exchange's credit, the timestamps were accurate and reflected the problem. Algos
that reacted poorly should probably pay more attention to timestamps.
We have not yet found any unusual message rates at the time the event began.
Here is a list of symbols affected.
Tick chart of bid prices (red) along with quote age (blue).
Note that as the cycle repeats, it includes a few more quotes (the new quotes + those
since 13:59:57). There are 500 quotes between time axis labels.
More Nanex Research