Vision Administration: Batch Processing

Overview

The Vision Administrator Module performs various Database Administration functions in an interactive Windows environment. You may find it useful to use scripts to perform these same functions in a batch environment. Such sample scripts are described in this document and can serve as a basis for designing your own. These scripts work outside the VAdmin module and allow you to perform multiple functions and/or load multiple files simultaneously, as well as allow you to keep an 'audit trail' of your scripting iterations.

The examples as provided assume certain environment variables have been set to the standards established in PMA Release 1.7. If you are on a prior release, you will need to modify these variables before running your scripts as described in this document:

On Unix:

Modify the general defaults for Vision Database Administrator processing in /localvision/include/DBAVision.env:

    Add setenv VisionAdm 1
    Add setenv UserOSI 3
    Change the VisionMaxSegSize from 50000000 to 33554432

On NT:

Modify the System Variables:

    Add %VisionRoot%bin to the values listed for the Path variable
    Add the new variable NDFPathName with a value of %LocalVisionRoot%network\NDF
Modify the User Variables for the Vision Administrator:
    Add the new variable UserOSI with a value of 3
    Add the new variable VisionAdm with a value of 1
    Add the new variable VisionMaxSegSize with a value of 33554432

If you need additional help setting up these variables, please contact your Insyte consultant.

Creating Batch Scripts

The server-based software that manages your Vision database is the component known as batchvision. When you use VAdmin or one of the Vision editors, you are communicating with a batchvision process interactively. The batchvision program is also used to execute non-interactive programs. Requests can be submitted to batchvision via Vision script files. A Vision script file is simply a standard text file containing one or more Vision requests. Any number of requests can be included in a Vision script file. Each request is terminated by the ?g command which instructs Vision to begin executing the request.

For example, assume the message displayWeeklyReports has been saved in the Vision database and you want to run this program. You could create the file testfile which contains the following lines: 

  displayWeeklyReports ;
  ?g
Note that the characters ?g must be the only characters on the line followed by a carriage return. To execute this Vision script file, it can be used as standard input to the batchvision program as shown below for both Unix and NT: 
    batchvision < testfile > testfile.out
The requests in a Vision Script file can access, create, and manipulate any Vision structures that have been saved in the database or defined in the script.

If you want your Vision script to commit any new or changed information to the database, you need to start batchvision as the Vision Administrator and include an explicit instruction to update the database in your Vision script. For example, the following script defines a new display method for the class Object and saves the method to the database: 

  #---  Vision Script to define and save a new method at Object
  Object defineMethod: [ | display |
     ^self whatAmI print ; ^self creationDate print ;
     ^self code printNL ;
  ] ;
  ?g

  #---  Now save the results - don't forget the ?g on its own line
  Utility updateNetworkWithAnnotation: "define a new display method" ;
  ?g
Assume this script is saved as the file testupdate. You can run and save this script on Unix or NT using: 
    batchvision < testupdate > testupdate.out
Once this script has completed, you will be able to access the new method any time you connect to the Vision database. You must be logged in as the Vision Database Administrator to save the results of this script.

NOTE: On Unix systems, you need to source the following file before you begin your batch processing: 

    source /localvision/include/DBAVision.env

!!! Warnings !!!

  • When you are testing your Vision scripts you should make sure that they do not include the step that updates the database.

  • Scripts that contain the ?g characters cannot be loaded via VAdmin.

Using BatchScripts to Run DataFeeds

The VAdmin module limits DataFeed loading to one feed at a time. Once you have established some basic procedures for loading your data, you may prefer to create some batch scripts for yourself that perform the same function for multiple feeds at a time.

The Vision method for loading a data file via a data feed requires that you provide the name of the feed, the name of the file, and an optional configuration file name. For example, if you want to load data using the SecurityMaster feed from the file /localvision/upload/feeds/secmaster.dat, use the expression: 

  Interface BatchFeedManager
  upload: "SecurityMaster"
  usingFile: "/localvision/upload/feeds/secmaster.dat"
  withConfig: NA ;
If you have an explicit configuration file to use, replace the third parameter with the actual file name. To update the Security Master via a batch process, do the following:
  1. Create a Vision script that loads and saves the data 
      #--  Load the data
  Interface BatchFeedManager
      upload: "SecurityMaster"
      usingFile: "/localvision/upload/feeds/secmaster.dat"
      withConfig: NA ;
  ?g

  #--  and Update the database - don't forget the ?g on its own line
  Utility updateNetworkWithAnnotation: "load single secmaster file" ;
  ?g

    Note: On NT systems, if your data files are on a different drive than your script, you need to use the absolute path name.

  2. Save the Vision script as secmaster.vis
  3. Submit this script to batchvision for update. You must be logged in as the Vision Database Administrator to update the database. For both Unix and NT use: 
      batchvision < secmaster.vis > secmaster.out
Your Vision script can be modified to include multiple updates. For example, you may want to update a standard set of files each day in a specific order. Assume on a daily basis you receive a security master, company master, portfolio master, prices, and holdings file. If you copy these files to a known directory and name them the same way each day, you can use a single Vision script to update the set: 
  #--  Load the data
  Interface BatchFeedManager
      upload: "CompanyMaster"
      usingFile: "/localvision/upload/feeds/compmaster.dat"
      withConfig: NA ;
  ?g
  Interface BatchFeedManager
      upload: "SecurityMaster"
      usingFile: "/localvision/upload/feeds/secmaster.dat"
      withConfig: NA ;
  ?g
  Interface BatchFeedManager
      upload: "PortfolioMaster"
      usingFile: "/localvision/upload/feeds/portmaster.dat"
      withConfig: NA ;
  ?g
  Interface BatchFeedManager
      upload: "PriceFeed"
      usingFile: "/localvision/upload/feeds/prices.dat"
      withConfig: NA ;
  ?g
  Interface BatchFeedManager
      upload: "HoldingsFeed"
      usingFile: "/localvision/upload/feeds/holdings.dat"
      withConfig: NA ;
  ?g

  #--  and Update the database - don't forget the ?g on its own line
  Utility updateNetworkWithAnnotation: "load daily file" ;
  ?g

Defining New Classes and Properties

In addition to loading data, the menus in the VAdmin module provide tools that allow you to define new classes and relationships. The actual Vision code invoked by VAdmin to perform these functions is supplied below: 

  #--------------------
  #  VAdmin Schema Choice: Create Entity Class
  #      Creates Entity subclass and associated MasterFeed
  #  Inputs -
  #        createEntityClass:      name of subclass to create
  #                     from:      Entity class to subclass
  #--------------------
  Interface BatchFeedManager
     createEntityClass: "SPIndustry" from: "Industry" ;
  #--------------------
  #  VAdmin Schema Choice: Create DataRecord Class
  #      Creates DataRecord subclass and EntityExtenderFeed
  #  Inputs -
  #        createDataRecordClass:     name of subclass to create
  #                         from:     DataRecord class to subclass
  #                     linkedTo:     Entity class to link to
  #                          via:     property to define at linkedTo class
  #                         asTS:     should property be a t/s?
  #--------------------
  Interface BatchFeedManager
    createDataRecordClass: "ModelData" from: "DataRecord"
       linkedTo: "Company" via: "modelData" asTS: "YES" ;

  #------------------------------------------
  #  VAdmin DataFeeds Choice: PropertySetup
  #      Define properties for new DataRecord Class
  #  Inputs -
  #               classId:   name of class/subclass properties are assigned to
  #              property:   name of property
  #                tsflag:   is property a t/s? Y or N
  #              datatype:   class of object returned by this property
  PropertySetup updateFromString:
      #--  these are tab-delimited
      "classId  property  tsflag  datatype
      ModelData	model1	N	Number
      ModelData	model2	N	Number
      ModelData	model3	N	Number
      ";

  #--------------------
  #  VAdmin Schema Choice: Create Relationship
  #      Creates MembershipFeed and property tracking relationship
  #              at the group and member entity classes supplied
  #  Inputs -
  #        createRelationship:     name of MembershipFeed class to create
  #             withGroupType:     Entity class of Groups
  #                       via:     list property defined at group class
  #             andMemberType:     Entity class of Members
  #                       via:     property defined at member class
  #                      asTS:     should properties be t/s?
  #--------------------
  Interface BatchFeedManager
    createRelationship: "CompanyToSPIndustry"
       withGroupType: "SPIndustry" via: "memberList"
       andMemberType: "Company" via: "spIndustry" asTS: "YES" ;
You can include these functions in any batch script you submit for update.

ModelData Sample

Assume that you want to track several model scores about a company over time. You need to:

  1. Create the ModelData class as a DataRecord linked to Company over time.
  2. Define the model properties for your new class.
  3. Load model data for specific company/date combinations.
The following Vision script can be used to define the new class: 
  Interface BatchFeedManager
    createDataRecordClass: "ModelData" from: "DataRecord" 
       linkedTo: "Company" via: "modelData" asTS: "YES" ;
  ?g

  PropertySetup updateFromString:
     "classId     property     tsflag     datatype
      ModelData   model1       N          Number
      ModelData   model2       N          Number
      ModelData   model3       N          Number
     " ;
  ?g

  Utility updateNetworkWithAnnotation: "Define ModelData class" ;
  ?g
This creates the class ModelData and defines three properties. Model data for the three model values can be updated using the ModelDataFeed. For example, if you create a file named models.dat, that contains: 
  id     date     model1     model2     model3
  ibm    9801     1          2          3
  ibm    9802     2          2          3
  dell   9801     10         12         14
you can update it using: 
  Interface BatchFeedManager 
      upload: "ModelDataFeed"
      usingFile: "models.dat
      withConfig: NA ;
  ?g
Access to the data is provided via the Vision expression: 
  Named Company IBM modelData
  do: [ date print: 15 ; 
        model1 print ;
        model2 print ;
        model3 printNL ;
      ] ;

Database Maintenance Functions

A set of database maintenance functions needs to be performed periodically to reclaim space and compress structures that have become sub-optimal as the result of many updates to the database. These functions are described in the table below:

DBA FunctionDescription
Database Cleanups and Garbage Collection Performs various cleanup activities that are standard for all installations.
Compaction Physically moves unused information out of the database and temporarily relocates it elsewhere on the disk.
Delete Segments Removes the unused information identified by the compaction process.

In a starter environment, these maintenance activities are invoked interactively using the VAdmin module as needed. As you move towards a production environment, you may prefer to create a batch process that performs these maintenance tasks.

The adminScripts directory in the localvision directory contains batch scripts that perform these functions. On Unix systems, this directory is accessed via /localvision/adminScripts. On NT systems, this directory is located in c:\localvision by default, but is usually relocated during the installation process. The environment variable LocalVisionRoot refers to the location chosen for the localvision directory on NT.

The following scripts are included in the adminScripts directory:

Maintenance scripts in /localvision/adminScripts
Script Name DBA Function
garbageCollect.cmd Database Cleanups and Garbage Collection
fullCompact.cmd Compaction
deleteCompactSegments.cmd Delete Segments
batchCleanup.cmdIncludes above 3 functions in one script.

These scripts can be run directly from the operating system prompt. You must be logged in as the Vision Administrator.

Running Maintenance Scripts in Unix

On Unix systems, you need to source the following file before you start your batchvision session: 

    source /localvision/include/DBAVision.env

To run a script, type the script name and redirect its output to another file. For example: 

    /localvision/adminScripts/garbageCollect.cmd > /localvision/logs/gc.yymmdd

Note that if you setup an automated procedure to run the maintenance tools that bypasses the standard login procedure, you should include the following "source" statement in your procedure: 

    source /localvision/include/DBAVision.env
Running Maintenance Scripts on NT

To run a script, type the script name and redirect its output to another file. For example: 

    %LocalVisionRoot%adminScripts\garbageCollect.cmd > %LocalVisionRoot%logs\gc.yymmdd
When to Run the Maintenance Scripts

Large production environments typically run the maintenance functions nightly in order to optimize their space utilization. In this type of environment we recommend the steps run in the sequence employed by the batchCleanup.cmd script: 

    deleteCompactSegments.cmd
garbageCollect.cmd
fullCompact.cmd
This sequence introduces a one day lag in the removal of segments flagged for deletion, allowing for deleted segments to be backed up in case they are needed for recovery activities.

Your environment may not require the maintenance functions to be run as frequently. If you find you only need to run them on a sporadic basis, you will probably want to run the deleteCompactSegments (using the batch script or the VAdmin module) after Compaction to immediately reclaim the space. The following guidelines can be used to determine the optimum frequency for running the maintenance functions: TBD

Creating New Object Spaces

Object spaces are used to organize functional components of the Vision database. They correspond to physical directories in the localvision/network branch of the file system. By default, classes, instances, and data will be stored in object space 3. This is fine for prototyping and small systems. If you plan to load a large amount of historical data from multiple sources and update on a regular basis, multiple object spaces are normally appropriate. The primary reason to do this is that it will allow the compaction process to operate more efficiently and effectively. Most users and applications will not need to know about the object space structure in order to use the database since messages encapsulate the navigations for you.

We recommend creating a separate object space for each data source that loads as a unit on a regular basis. For example, Barra data should be physically stored in its own object space. Note that your initial installation includes object spaces that have been pre-allocated for pricing and holdings so there is no need to create separate object spaces for these sources initially.

The script installSpace.cmd is available in the /localvision/adminScripts directory and is used to create new object spaces. The script requires that you provide a code that identifies the object space and an option type that indicates the role of the object space. The type can be supplied as one of DB, DBA, or User. If omitted, the type is assumed to be DB. DB object spaces are used to store instances and property values for one or more classes. DBA object spaces are used to store tools to manage various aspects of the database. User object spaces are used to store private data for a specific user.

For example, to install a new object space for storing model data, run the following command:

    On Unix: 
      /localvision/adminScripts/installSpace.cmd Models DB
    On NT: 
      %LocalVisionRoot%adminScripts\installSpace.cmd Models DB
After this runs, you should see an 'object space created' message as well as the normal update message.

Once you have created your new Models object space you can access it via: 

   Environment DB Models

NOTE:  If you rollback through a space installation, you will need to manually delete the associated physical sub-directory under localvision/network. If you are unsure of what the number of the object space to be deleted is, refer to the NDF.JOURNAL file in the localvision/network directory.

Assigning New Classes to an Object Space

By default, new Entity classes are created in object space 3 and new DataRecord classes are created in object space 6. The instances of a class and the property values associated with each of these instances are stored in the object space in which the class is originally installed.

We recommend creating a separate object space for each data source that loads as a unit on a regular basis. For example, your Model data could be physically stored in its own object space. You need to specify the object space location of a new class prior to its creation using the following expression: 

  #---  Set the object space of the next class created to 
  #--     the space define to hold the Models
  Interface BatchFeedManager setObjectSpaceTo: Environment DB Models;
You can use the object space name or the object space number as the parameter to setObjectSpaceTo:. You must precede each Entity and DataRecord subclass creation request with the setObjectSpaceTo: expression if you want the new class set up in a specific object space. For example: 
  Interface BatchFeedManager
     setObjectSpaceTo: Environment DB Models;
  Interface BatchFeedManager 
     createEntityClass: "ModelType" from: "Classification" ;

  Interface BatchFeedManager
     setObjectSpaceTo: Environment DB Models;
  Interface BatchFeedManager
    createDataRecordClass: "ModelData" from: "DataRecord"
       linkedTo: "Company" via: "modelData" asTS: "YES" ;
This request will create the new entity class ModelType and the new data record class ModelData in the object space named Environment DB Models if the classes do not already exist. Note that this approach will not recreate the classes if they already exist and it will not relocate existing classes to the object space specified.

Creating New DataRecord Stores

By default, the instances of a class and the property values associated with these instances are stored in the object space in which the class was originally installed. For DataRecord subclasses that will store a large number of instances over time, we recommend that data be physically stored in a new storage structure in its own object space.

Once you have created the space, you will need to define a "new store" for the class you wish to store in that space. For example, to create a new store for ModelData in the Models object space, create a Vision script that contains the following: 

  #--  Create a new store for ModelData in Models object space
  ModelData createNewStoreAt: Environment DB Models ;
  ?g
Once you have saved this script, any new instances created via the ModelDataFeed will be stored in the Models object space.

Note that instances and data that existed prior to the creation of the new store will continue to reside in its original location. Normally, you would create the new object space and store immediately after creating the new DataRecord subclass and prior to updating any actual data.

If you use a DataFeed to update your data, the information will automatically get placed in the last "store" created. If you are writing your own Vision methods to load data directly, you will need to modify any code that creates new instances so it is in the form: 

  ^global ModelData currentStore createInstance: "xxx" ;

Creating Bridge Classes and Feeds

The subclasses of the Bridge class are used to manage the protocol for connecting an entity to one or more DataRecord instances. It is useful when there are complex relationships that need to be managed and you do not want to overburden the entity class with all the required information. An instance of a Bridge subclass is normally referenced via the entity instance with which it is associated.

To create a subclass of Bridge, you provide the Entity class with which the new class is associated, the name of the property that establishes the connection between an instance of entity's class and an instance of the bridge's class, and an indication whether the relationship should vary over time. For example, you may want to create a bridge class called FundamentalBridge which links to separate data record classes that track annual, quarterly, and monthly fundamental data. To create the FundamentalBridge class and associate it with the Company entity via the fixed property fundamentalData, do the following: 

   #--------------------
   #  Create Bridge Class
   #      Creates Bridge subclass and establishes relationship
   #          with existing Entity subclass
   #  Inputs -
   #        createBridgeClass:     name of subclass to create
   #                     from:     Bridge class to subclass
   #                 linkedTo:     Entity class to link to
   #                      via:     property to define at linkedTo class
   #                     asTS:     should property be a t/s?
   #--------------------
   Interface BatchFeedManager
     createBridgeClass: "FundamentalBridge" from: "Bridge"
        linkedTo: "Company" via: "fundamentalData" asTS: "NO" ;
You can create and attach the different data records to this class using: 
  Interface BatchFeedManager
    createDataRecordClass: "AnnualData" from: "DataRecord"
       linkedTo: "FundamentalBridge" via: "annual" asTS: "YES" ;
  Interface BatchFeedManager
    createDataRecordClass: "QuarterlyData" from: "DataRecord"
       linkedTo: "FundamentalBridge" via: "quarterly" asTS: "YES" ;
The PropertySetup feed can be used to define updateable properties for the AnnualData and QuarterData classes and the AnnualDataFeed and QuarterlyDataFeed classes can be used to update these properties for specific companies on specific dates.

Once these classes have been established, you can access the specific fundamental data record using: 

  Named Company IBM fundamentalData annual  #insert property name
or 
  Named Company IBM fundamentalData quarterly   #insert property name

LocalEntity Bridges and Private Object Spaces

A LocalEntity is an abstract subclass of Bridge designed to provide special protocol for a subset of instances of a specific entity class to enable private saves. LocalEntity subclasses can be created in a way that allows them to connect with their underlying entity dynamically. As a result, these bridge classes, and their related data record classes can be stored in a way that allows them to be updated privately.

For example, suppose the Equity Research group wants to update information about a subset of companies without requiring the intervention of the global database administrator. A separate object space can be created for this group. The ERCompany bridge class and one or more data record classes can be created in this space. Data associated with this bridge class can then be updated privately.

Here are the steps involved with setting up a private space that can be updated using data feeds:

  1. Use the installSpace.cmd script to create a new object space: 
      /localvision/adminScripts/installSpace.cmd MySpace

  2. Set the permissions on this space so that the private user and the Vision Administrator can update it.

  3. As the Vision Administrator, define and save a LocalEntity bridge in this space for any entity class you wish to extend with private data: 
      Interface BatchFeedManager 
     setObjectSpaceTo: Environment DB MySpace ;
  Interface BatchFeedManager
       createBridgeClass: "MyCompany" from: "LocalEntity" 
       linkedTo: "Company" via: "myCompany" asTS: "Method" ;

  4. As the Vision Administrator, define and save one or more DataRecord classes attached to this bridge in this space: 
      Interface BatchFeedManager
      setObjectSpaceTo: Environment DB MySpace ;
  Interface BatchFeedManager
      createDataRecordClass: "AnalystRating" from: "DataRecord"
        linkedTo: "MyCompany" via: "rating" asTS: "Y" ;

  Interface BatchFeedManager
      setObjectSpaceTo: Environment DB MySpace ;
  Interface BatchFeedManager
      createDataRecordClass: "CompanyAnnualOverride" from: "DataRecord"
        linkedTo: "MyCompany" via: "companyAnnualOverride" asTS: "Y" ;

  5. If you want to create private Entity classes in your private space, you must save the following Vision code in the private space: 
      Environment DB MySpace 
       define: 'Entity' toBe: CoreWorkspace Entity newPrototype ;
  Environment DB MySpace 
       define: 'Named' toBe: CoreWorkspace Named newPrototype ;
    This code must be saved by the Vision Administrator.

  6. As the Vision Administrator, define and save one or more Entity classes. Note that you must start your batchvision with the -U# option, where # indicates the private object space number: 
        Interface BatchFeedManager
        setObjectSpaceTo: Environment DB MySpace ;
    Interface BatchFeedManager 
       createEntityClass: "MyIndustry" from: "Entity" ;
Once the Vision Administrator has saved the appropriate classes, the private user can use the feeds to update the database using the upload:usingFile:withConfig: message. The private user must follow these steps:

  • Log in as the private user

  • Start batchvision with the -U## option (or set the UserOSI environment variable) where ## is the private object space number

  • When the Utility updateNetworkWithAnnotation: runs, the message User Space Updated should display instead of Object Network Updated.

Note that private updates should not run when general vision database maintenance is in progress. This includes garbage collection and compaction.

Creating Estimate Bridges and Feeds

The subclasses of the EstimateBridge class are used to manage the relationship between an entity or bridge and data that can be estimated for multiple time periods. To create a subclass of Bridge, you provide the Entity or Bridge class with which the new class is associated, the name of the property that establishes the connection between an instance of this class and an instance of the estimate bridge's class, the type of estimate record, and the periodicity of the estimate. For example, you may want to create an estimate bridge for tracking annual company eps estimates made by an analyst as well as consensus estimate provided by a source such as I/B/E/S: 

  #--------------------
  #  Create EstimateBridge Class
  #  Inputs -
  #    createEstimateBridgeClass: name of subclass to create
  #                         from: EstimateBridge class to subclass
  #                     linkedTo: Entity or Bridge class to link to
  #                          via: t/s property to define at linkedTo class
  #      withEstimateRecordClass: subclass of EstimateRecord
  #                      andFreq: periodicity of estimates
  #--------------------

  Interface BatchFeedManager
      createEstimateBridgeClass: "EpsABridge" from: "EstimateBridge" 
      linkedTo: "Company" via: "analystEpsEst" 
      withEstimateRecordClass: "AnalystEstimateRecord" 
      andFreq: 12 monthEnds ;

  Interface BatchFeedManager
      createEstimateBridgeClass: "IbesEpsABridge" from: "EstimateBridge" 
      linkedTo: "Company" via: "ibesEpsEst" 
      withEstimateRecordClass: "ConsensusEstimateRecord" 
      andFreq: 12 monthEnds ;

Once the estimate bridges have been created you can use the appropriate feed to update the database using the upload:usingFile:withConfig: message.

Access to estimate structures and additional examples are described elsewhere.

Related Topics