Scully, Scott, Murphy & Presser, P.C.
System and method for transferring data between RDBMS and big data platform
A system for transferring data from a Relational Database Management System (“RDBMS”) to a big data platform and methods for making and using the same. The system can acquire a partitioning execution scheme of a selected table from the RDBMS and submitting partitioned queries from the big data platform to each mapper of partitions. The partitioned queries are generated based on the partitioning execution scheme. The partitioning execution scheme can be acquired by submitting a query explain request to an optimizer of the RDBMS to generating a parallel query plan. The partitioning execution scheme can also be acquired by querying statistics from a statistics catalog of the RDBMS or by user inputs. The system can use RDBMS capabilities and statistics for parallel data fetching. Thereby, the system can increase efficiency of the fetching and can avoid straggling when target data is not evenly distributed and can avoid table query-in-serial.
International Business Machines Corporation
G G G
G06F G06F G06F
inventional inventional inventional
Gong Ke Wei Xi Long Serge Xiao Hong
Su Wei Zheng Bourbonnais Li Min