Edell, Shapiro & Finnan, LLC
Transforming and loading data utilizing in-memory processing
A system includes at least one processor and processes an ETL job. The system analyzes a specification of the ETL job including one or more functional expressions to load data from one or more source data stores, process the data in memory, and store the processed data to one or more target data stores. One or more data flows are produced from the specification based on the one or more functional expressions. The one or more data flows utilize in-memory distributed data sets generated to accommodate parallel processing for loading and processing the data. The one or more data flows are optimized to assign operations to be performed on the one or more source data stores. The optimized data flows are executed to load the data to the one or more target data stores in accordance with the specification. Present invention embodiments further include methods and computer program products.
International Business Machines Corporation
G
G06F
inventional
Lawrence A. Xiaoyan Yong Yeh-Heng
Greene Pu Li Sheng