More Nanex Research


Update (2/15):
Added charts (#2 and #5) showing data from the exchange perspective - duplicate trades and quotes removed and plotted with queued (old) exchange timestamps. Added chart (#3) of USO aligned in time with other 1 second charts.


NYMEX Black Swan ~ The Energy Complex Data Feed Loop

On February 13, 2012, starting 13:59:57, quotes and trades for crude oil began queuing. At 14:00:35, all of the queued messages were sent at once. Again at 14:01:08 the same 38 second block of messages sent earlier was sent again -- old timestamps and all plus a few new messages. Again at 14:01:18, all messages since 13:59:57 were sent again. This repeated 12 times.

From a programmer's perspective, it looks like a system problem led to a synchronization corruption in either a circular queue (head/tail pointers), or in the send acknowledgment mechanism, causing the software to believe the queue was full all the time.

To the exchange's credit, the timestamps were accurate and reflected the problem. Algos that reacted poorly should probably pay more attention to timestamps.

We have not yet found any unusual message rates at the time the event began.

Here is a list of symbols affected.


Tick chart of bid prices (red) along with quote age (blue).
Note that as the cycle repeats, it includes a few more quotes (the new quotes + those since 13:59:57). There are 500 quotes between time axis labels.

Filled in squares are implied prices (from spreads), empty squares are regular trades. Shaded area is quote spread.





More Nanex Research