Adaptive map-reduce pipeline with dynamic thread allocations
In an approach to adaptively pipeline a MapReduce job, a processor receives one or more data records from a storage and inserts the one or more data records into a map queue, wherein a size of the map queue is adaptive to one or more utilizations of one or more resources in the processor. One or more processors apply a map function to the one or more data records in the first buffer and sort the records that are output from the map function and store the sorted records. One or more processors receive and insert the sorted records into a reduce queue, wherein a size of the reduce queue is adaptive to one or more utilizations of resources in the one or more processors. One or more processors apply a reduce function to the sorted records in the reduce queue and store a result in a storage.
International Business Machines Corporation
G G G G G G G G G G G G G
G06F G06F G06F G06F G06F G06F G06F G06F G06F G06F G06F G06F G06F
inventional inventional additional inventional additional inventional inventional inventional inventional inventional inventional inventional inventional
Xiao Ming Jian Zhenhua Wen Yan Haohai
Bao Liu Hu Bai Ma