Load Manager in a Data Warehouse
The load manager is the system component that performs all the operations necessary to support the extract and load process. This system may be constructed using a combination of off-the-shelf tools, bespoke coding, C programs and shell scripts. The size and complexity of the load manager will vary between specific solutions from data warehouse to data warehouse but, as an indication, the larger the degree of overlap between source systems, the larger the load manager will be. However, it is worth noting that third-party tools will probably contribute a maximum of 20 to 25% of the total system functionality.
The bulk of the effort to develop a load manager should be planned within the first production phase. Although the entire system does not have to be built, a significant proportion of the functionality should be provided in the first phase. Each of these functions has to operate automatically, and recover from any errors it encounters, to a very large extent with no human intervention. This is because this process tends to run overnight at the close of the business day. In a situation where the load manager encounters a major error, it may be necessary for it to flag that human intervention is required. A system that automatically pages system operators DBAs is fairly typical.