Stratified sampling using adaptive parallel data processing
A computer-implemented method includes partitioning a plurality of records into a plurality of splits. Each split includes at least a portion of the plurality of records. The method further includes providing at least one split of the plurality of splits to a mapper. The mapper scans the input data set, transforms each input record using a map function, and extracts a grouping key in parallel. The method further includes assigning at least a portion the records of the at least one split to a group. Each assignment to the group is based on a strata of the assigned record, and filtering the records of the group. Each filtering is based on a comparison of a weight of a record to a local threshold of the mapper. The method further includes shuffling the group to a reducer and providing a stratified sampling of the plurality of records based on the group.
International Business Machines Corporation
G G G G G G
G06F G06F G06F G06F G06F G06F
inventional inventional inventional inventional inventional inventional
Liping Peter J. John Vuk Andrey
Peng Haas Sismanis Ercegovac Balmin