Shook, Hardy & Bacon
Distributed database job data skew detection
A system and method for identifying whether data skew is causing delays in a map phase and/or a reduce phase of a query of a distributed database. The system and method identify the values of various metrics relating to a database query. These metrics include map phase and reduce phase durations and various related metrics. The system and method gather statistics of multiple queries to determine correlation levels between the metrics and the map phase and reduce phase durations. Based on the statistics, the system and method determine whether one or both of the map and reduce phases for a query/response are taking longer than expected. If the durations are longer than expected, the system identifies the delay as caused by data skew and informs the originator of the query.
eBay Inc.
G G G
G06F G06F G06F
inventional inventional inventional
Yong Libin Xin Hao Tan
Zhang Sun Li Chen Chen