Adhoc Multifile in Abinitio
Adhoc Multifile in Abinitio
Ad hoc multifiles treat several serial files having the same record format as a
single graph component.
Frequently, the input of a graph consists of a set of serial files, all of which
have to be processed as a unit. An Ad hoc multifile is a multifile created 'on
the fly' out of a set of serial files, without needing to define a multifile sys
tem to contain it. This enables you to represent the needed set of serial files
with a single input file component in the graph. Moreover, the set of files used
by the component can be determined at runtime. This lets the user customize whi
ch set of files the graph uses as input without having to change the graph itsel
f, even after it goes into production.
Ad hoc multifiles can be used as output, intermediate, and lookup files as well
as input files.
The simplest way to define an Ad hoc multifile is to list the files explicitly a
s follows:
1. Insert an input file component in your graph.
2. Open the properties dialog. Select Description tab.
3. Select Partitions in the Data Location of the Description tab
4. Click Edit to open the Define multifile Partitions dialog box.
5. Click New and enter the first file name. Click New again and enter the second
file name and so on.
6. Click OK.
If you have added 'n' files, then the input file now acts something like a file
in a n-way multifile system, whose data partitions are the n files you listed. I
t is possible for components to run in the layout of the input file component. H
owever, there is no way to run commands such as m_ls or m_dump on the files, bec
ause they do not comprise a real multifile system.
There are other ways than listing the input files explicitly in an Ad hoc multif
ile.
1. Listing files using wildcards - If the input file names have a common pattern
then you can use a wild card for all the files. E.g. $AI_SERIAL/ad_hoc_input_*.
dat. All the files that are found at the runtime matching the wild card pattern
will be taken for the Ad hoc multifile.
2. Listing files in a variable. You can create a runtime parameter for the graph
and inside the parameter you can list all the files separated by spaces.
3. Listing files using a command - E.g. $(ls $AI_SERIAL/ad_hoc_input_*.dat), whi
ch produces the list of files to be used for the ad hoc multifile. This method g
ives maximum flexibility in choosing the input files, since you can use complex
commands also that involves owner of file or date time stamp.