Vision Class: DataFeed

DataFeed Overview

The DataFeed class is an abstract class that is used to organize the classes that translate data from sources external to Vision into Vision objects. An external feed corresponds to a flat, tabular structure. It may be loaded into Vision from a file, a spreadsheet, a relational database, or any other source capable of presenting records of information.

Several subclasses of DataFeed have been defined to encapsulate different ways to map data from external formats into Vision objects. These include:

--- DataFeed Subclasses ---
MasterFeed	used to create new instances in an Entity subclass and refresh key properties for instances of that class.
EntityExtenderFeed	used to update properties and DataRecord instances associated with a specific Entity or Bridge, potentially over time.
TransactionFeed	used to create and cross reference instances of a LinkRecord subclass, classes that associate two or more entities.
AliasFeed	used to establish multiple aliases for existing Entity instances.
XRefFeed	used to load alternative identifiers for existing Entity instances.
MembershipFeed	used to update and cross reference one-to-many relationships between two Entity instances over time.
RangeGroupFeed	used to define and update numeric ranges and categorize Entity instances into the appropriate range group.
SchemaFeeds	used to define new core classes, properties, and data feed classes.

The initial DataFeed class hierarchy is illustrated below:

           Object
             |
             IncorporatorPrototype
                |
                DataFeed
                   |
                   |-- MasterFeed
                   |   |-- CurrencyMaster
                   |   |-- UniverseMaster
                   |
                   |-- EntityExtenderFeed
                   |   |-- ExchangeRateFeed
                   |   |-- EstimateRecordFeed
                   |
                   |-- TransactionFeed
                   |
                   |-- AliasFeed
                   |-- XRefFeed
                   |
                   |-- MembershipFeed
                   |   |-- UniverseMembers
                   |
                   |-- RangeGroupFeed
                   |
                   |-- SchemaFeeds
                   |   |-- ClassSetup
                   |   |-- PropertySetup
                   |   |-- MessageSetup
                   |   |-- DataFeedSetup
                   |
                   |-- GlobalsFeed

A separate subclass is defined for each format of external data you wish to load into your Vision database. For example, subclasses of the MasterFeed class are defined to create and update instances of a specific Entity, such as Currency or Security. Subclasses of the EntityExtenderFeed class are defined to update a specific relationship between an Entity subclass and associated data, such as pricing data for a security.

Basic Interactions

Each DataFeed subclass defines one or more data items that you can update. Some feeds create new instances of Entity, Bridge, DataRecord, and/or LinkRecord subclasses; other feeds only refresh properties in existing objects. Some feeds support updates for multiple time periods; other feeds only refresh properties as of the current date.

At its simplest, a feed is a tab or vertical bar delimited string containing one or more columns of information for one or more rows. For example, to create and update Currency instances, you could use:

     CurrencyMaster updateFromString: 
     "entityId | name                 | shortName
      USD      | United States Dollar | US Dollar
      CAD      | Canadian Dollar      | CA Dollar
      GBP      | Great British Pound  | GB Pound
     " ;

This example loads data using the CurrencyMaster feed. This feed will create Currency instances for any entityId not already defined and will update the name and shortName properties for the three instances included.

The message updateFromString: can be sent to any DataFeed subclass. The parameter to this message is a String with the following structure:

Records are separated by carriage-return/line-feed.
Records that are blank and records that start with a # as the first non-blank character are ignored.
Fields in the record are delimited by tabs or vertical bars (| character).
The first record is a header containing delimited field identifiers which correspond to the properties to update.
The header record can contain any number of identifiers in any order. If an identifier is invalid or blank, the data in this column will be ignored.
The field identifiers Entity, EntityId, and Id can be used interchangeably to identify a column containing an entity identifier. The field identifiers Currency and CurrencyId can be used interchangeably to identify a column containing a currency identifier. Field identifiers are not case sensitive.
Numeric values can be supplied with any number of decimal places.
Dates can be supplied in a number of formats including CCYYMMDD and MM/DD/CCYY. The initial date can be indicated as 10101 or Default.
Values that are not available should be supplied as blanks or the string NA.

The GlobalsFeed can be used to temporarily set or alter the field delimiter, header location, and other characteristics of your feed.

The message getUploadProperties returns the list of properties that can be updated via a specific feed. For example:

     CurrencyMaster getUploadProperties
     do: [ code printNL ] ;

Any of these property identifiers can be included in your header record. Properties can be included in the header record in any order. The message describeFeed can also be sent to any DataFeed subclass and displays a description of the subclass, including these properties. The message showInheritance can be sent to the DataFeed class or any of its subclasses to display the DataFeed classes defined for your installation.

The message loadFromFile: can be sent to any DataFeed subclass to read the data from the file name supplied as a parameter. The data in the file should be in exactly the same format as the updateFromString: version. The first line of the file should contain tab or |-delimited field identifiers. The remaining lines in the file should contain delimited data records. For loading data from large files, you should use the bulkLoadFromFile: options described elsewhere.

A number of specialized interfaces have been defined that package feeds from different sources including The Vision Administrator Module, Excel spreadsheets, and batch processing scripts.

The MasterFeed Class

Subclasses of MasterFeed are used to create new instances in an Entity subclass and refresh key properties for instances of that class. The properties updated by these feeds usually store basic descriptive information about the underlying Entity. Time series properties, supplemental data, and relationships are usually updated using an Entity Extender or a Membership feed.

A separate MasterFeed subclass should be created for any Entity subclass that you wish to update from an external source. The following items can be included in the header for any MasterFeed upload:

entityId - id of entity to create or refresh
currencyId - id of an existing currency
name - descriptive name of entity
numericCode - numeric id for entity
shortName - short descriptive name
sortCode - id used for sorting

Any combination of these items may be included in any order. The entityId must be included. The field identifier id or entity can be used interchangeably with entityId.

For example, if the CountryMaster class has been defined, the expression:

     CountryMaster updateFromString:
     "id |   name
      US |   United States
      CA |   Canada
     " ;

can be used to create country objects and/or update their names.

This expression will perform the following steps:

Use the first line (i.e., the header) to determine the data items present in this feed and their order. In this case, the id and name are supplied.
For each line in the supplied string, create an instance in the DataFeed subclass. In this case, two instances are created in the CountryMaster class.
For each instance, look up the supplied id in the feed's underlying entity class, in this case Country.
For each id that does not map to an existing entity, create a new instance in the correct Entity subclass.
For each feed instance, refresh the corresponding properties in the entity instance with the data supplied by this feed. If the property has a default data type, the supplied value is first converted to a value of this type.
Display exception reports.
Run any special wrapup procedures.

If time series properties are included in a MasterFeed, the value is updated on the current "as of" date if it reflects a change from the value currently stored. By default, the "as of" date is "today". You can reset the "as of" date for the entire feed using the GlobalsFeed.

The MasterFeed subclasses are used to create new instances and refresh data for instances of an entity subclass. If you want to use a feed to refresh data without creating any instances, you can set the disableEntityCreation flag prior to executing the feed. This will prevent any new instances from being created but will update any existing instances with the information supplied in the feed. For example:

     CurrencyMaster disableEntityCreation ;
     CurrencyMaster updateFromString: 
     "entityId | name                 | shortName
      USD      | United States Dollar | US Dollar
      CAD      | Canadian Dollar      | CA Dollar
      XYZ      | Dummy Currency       | Dummy
     " ;

This feed will refresh the name and short name for the existing currencies but will not create the new currency XYZ. Note that the disableEntityCreation only applies to the next execution of this feed. Once executed, entity creation is re-enabled.

The enableInternalIds flag can be set for any MasterFeed subclass if you want to automatically generate a permanent, unique id for any instances in the underlying entity class. By default, these identifiers are not created. When enabled, the XRef dictionary named Internal is established for the entity class and the property internalId is created to track the permanent identifier. For example, to enable the feed for the CurrencyMaster use:

     CurrencyMaster enableInternalIds ;

Once enabled, internal identifiers will automatically get generated every time this feed is executed, unless this option is explicitly disabled using the disableInternalIds flag.

If duplicate underlying records exist in a feed, neither will be processed and they will be included in the exception report and reject file.

If you supply a blank string or the string NA in a feed, the property value for the entity for that item will be set to NA or the appropriate default value for the item. If the property is time varying and it currently has a non-default value, its value will be set to default as of the current date. You cannot delete entity instances but they can be flagged as inactive via a MasterFeed. These rules are described later in this document.

Two MasterFeed subclasses are included as part of the basic installation: CurrencyMaster and UniverseMaster. These feeds are used to create and refresh Currency and Universe instances and define several additional properties specific to these classes.

You can define new MasterFeed subclasses using the MasterFeedSetup class. This feed includes two fields: feedId and baseClassId. The feedId represents the name of the class that will be used to temporarily store the data from your feed and translate it to the appropriate entities. The baseClassId names the entity class to associate with this feed. The fields must be supplied in this order. You should include a header line in this feed, but the header values are ignored. Both fields must be supplied. For example, to define MasterFeed classes to update countries and industries, you could use:

     MasterFeedSetup updateFromString:
     "feedId         |  baseClassId
      CountryMaster  |  Country
      IndustryMaster |  Industry
     " ;

This upload will create CountryMaster and IndustryMaster as subclasses of MasterFeed (if they do not already exist) and link them to the entity classes Country and Industry.

Note that if the Country and/or Industry class does not exist, the corresponding MasterFeed subclass will not be created. Use the ClassSetup class to create new Entity subclasses.

The EntityExtenderFeed Class

Subclasses of EntityExtenderFeed are used to update properties for an Entity that are directly defined at the entity's class or linked to the entity via a property returning a subclass of DataRecord. An EntityExtenderFeed may also be defined to update properties that are directly defined at a Bridge class associated with a specific Entity class or linked to the bridge via a property returning a subclass of DataRecord. Data may be tracked for a fixed point in time or over time. Whereas MasterFeed subclasses are normally used to update basic, non-time varying descriptive information about an Entity, EntityExtenderFeed subclasses are usually created to update supplemental information about the underlying Entity.

A separate EntityExtenderFeed subclass should be created for any sets of data that get updated as a unit. The following items can be included in the header for any EntityExtenderFeed upload:

entityId - id of entity to update
date - date to update (for time series properties)
currencyId - id of an existing currency

All subclasses of EntityExtenderFeed define additional items that can be updated. Any or all items defined for a feed can be included in any order. The entityId must be included. The field identifier id or entity can be used interchangeably with entityId. The date should be included with any feed that updates time varying data. The currencyId should be included with any feed that includes data items representing monetary values.

The simplest form of EntityExtenderFeed is used to update properties defined directly for an Entity subclass. For example, if the time series property usExchange is defined for the Currency class to track the rate of exchange into US dollars over time, then the ExchangeRateFeed class could be used to update this property using:

     ExchangeRateFeed updateFromString:
     "id   |   date     |  usExchange
      CAD  |  19970930  |    1.3828	
      CAD  |  19971031  |    1.4090	
      CAD  |  19971128  |    1.4234	
      DEM  |  19971031  |    1.7212	
      DEM  |  19971128  |    1.7638	
     " ;

This expression will perform the following steps:

Use the first line (i.e., the header) to determine the data items present in this feed and their order. In this case, the id, date, and usExchange are supplied.
For each line in the supplied string, create an instance in the DataFeed subclass. In this case, five instances are created in the ExchangeRateFeed class.
For each instance, look up the supplied id in the feed's underlying entity class, in this case Currency.
For each id that references an existing entity, refresh the corresponding properties in the entity instance with the data supplied by this feed. If the property has a default data type, the supplied value is first converted to a value of this type. In this case, the usExchange property is updated as of the supplied dates.
Display exception reports.
Run any special wrapup procedures.

EntityExtenderFeed subclasses can also be defined to create and refresh instances in a DataRecord subclass associated with an Entity or Bridge subclass. For example, PriceRecord may be defined as a subclass of DataRecord used to track the high, low, and closing prices for a security over time. The time series property pricingData could be defined for the Security class to store a separate PriceRecord instance for each date the security is priced. If PricingFeed is defined as an EntityExtenderFeed subclass that establishes this relationship, then it could be used to update pricing data using:

     PricingFeed updateFromString:
     "id   |  date   |  high   |    low   |   close
      GM   |  971215 |  65.125 |  62.75   |   64.00
      GM   |  971216 |  66.5   |  63.75   |   65.25
      GM   |  971217 |  65.25  |   62.5   |   64.75
      IBM  |  971215 | 105.125 | 102.75   |  104.00
      IBM  |  971216 | 106.5   | 103.75   |  105.25
      IBM  |  971217 | 105.25  |  102.5   |  104.75
     " ;

This expression will perform the following steps:

Use the first line (i.e., the header) to determine the data items present in this feed and their order. In this case, the high, low, and close are supplied in addition to the id and date.
For each line in the supplied string, create an instance in the DataFeed subclass. In this case, six instances are created in the PricingFeed class.
For each instance, look up the supplied id in the feed's underlying entity class, in this case Security.
For each id that references an existing entity, create a new DataRecord instance for any date included in this feed that does not currently exist. In this case, a new PriceRecord is created for any date that does not exist in the security's pricingData time series.
For each instance in this feed that references a valid entity/date combination, refresh the corresponding properties in the DataRecord associated with the entity/date. Supplied values are converted to the correct type for the property if appropriate. In this case, the high, low, and close properties are updated in the PriceRecord stored in the security's pricingData time series as of the supplied date.
Expire old data, if appropriate.
Display exception reports.
Run any special wrapup procedures.

The autoExpire option can be set in the GlobalsFeed for feeds associated with a DataRecord to automatically add a default value to a time series property after the last "real" value, based on a supplied expiration period. By default, Vision returns the last available value for a time series. This option is used to expire the data if it is older than a specified duration. For example, assume your database includes a time series of monthly fundamental data records stored in the property funDataM for the class Company. If you load your June 30, 1999 data and specify an expiration of 1 monthEnds, any company whose last "real" data point is on or before May 31, 1999 will have a default value inserted in the time series on the month-end date following the last real data point.

If you supply a blank string or the string NA in a feed, the property value will be set to NA or the appropriate default value for the item. In addition, there are a number of ways to purge information using an EntityExtenderFeed. These rules are described later in this document.

The message enableOnlyUpdateOnChange allows you to update time series properties in an Entity Extender Feed only if the value has changed. This works with Entity Extender Feeds that are used to update fields that are time series properties, not feeds that update time series of records. You can set the switches enableOnlyUpdateOnChange and disableOnlyUpdateOnChange to control the behavior of the next update for the feed. This value is reset to disable after each update.

One subclass is included as part of the basic installation: The ExchangeRateFeed is used to update the usExchange time series property for Currency instances. The PricingFeed described above was included for illustration purposes and is not part of the basic installation. An actual PriceFeed class is included as part of the Portfolio Management Application Layer.

You can define new EntityExtenderFeed subclasses using the EntityExtenderFeedSetup class. This feed includes the fields:

feedId - name of feed class
baseClassId - name of associated Entity, Bridge or DataRecord class
baseEntityId - name of Entity or Bridge class if base class is a DataRecord
frequency - offset used to expire old data

The feedId represents the name of the class that will be used to temporarily store the data from your feed and translate it to the correct entities or data records. The baseClassId indicates the base class associated with this feed. It can refer to an Entity/Bridge or to a DataRecord class. The fields associated with this specific feed will update properties defined at this class. The baseEntityId indicates the Entity or Bridge class associated with the feed for feeds where the base class is a DataRecord. This field should be omitted if the base class is an Entity/Bridge.

The fields must be supplied in this order. You can include a header line in this feed, but the header values are ignored. The feedId and baseClassId must be supplied. The baseEntityId should be supplied for feeds that update DataRecord instances.

For example, to define the ExchangeRateFeed and PricingFeed described above, you could use:

     EntityExtenderFeedSetup updateFromString:
     "
     feedId            |  baseClassId  |     baseEntityId | frequency
     ExchangeRateFeed  |  Currency     |
     PricingFeed       |  PriceRecord  |     Security | 3 businessDays
     " ;

This upload will create ExchangeRateFeed as a subclass of EntityExtenderFeed if it does not already exist. This feed will be linked directly to the Entity subclass Currency. This feed can be used to update any properties defined directly for the Currency class. You can use the PropertySetup class to define new Currency properties which can then be updated by including these property names in the header record of this feed.

This update will also create PricingFeed as a subclass of EntityExtenderFeed if it does not already exist. This feed will be used to create and update properties in the DataRecord subclass PriceRecord via instances of the Entity subclass Security. Note that if the Security and/or PriceRecord class does not exist, the PricingFeed subclass will not be created. Use the ClassSetup class to create these subclasses and the PropertySetup class to define the property connecting these subclasses. The PropertySetup class should also be used to define the PriceRecord properties which can then be updated by this feed.

The frequency field is optional and is used in conjunction with the autoExpire option to expire old data for feeds that update DataRecords over time. If the frequency is present and the autoExpire option is in effect, the check for expiration is based on the criteria:

last observation + frequency <= current run date

Otherwise, the expiration is based on the criteria:

last observation + autoExpire value <= current run date

The frequency, if set, should be supplied as a DateOffset such as 3 businessDays or 18 monthEnds. The frequency can be optionally defined in the feed setup as shown or can be set prior to an update using the setFrequencyTo: message:

     PricingFeed setFrequencyTo: 3 businessDays ;

     GlobalsFeed updateFromString: 
     "option      | setting        | settingType
      asofDate    | 9906           | Date
      autoExpire  | 1 businessDays | DateOffset
     " ;

     PricingFeed updateFromString:
     "id | date | high | low | close
      #--   data goes here
     " ;

This example will add a default PriceRecord instance to any security that was last priced at least 3 business days prior to 6/30/99. The default instance will be added to these time series 1 business day after the last date priced.

The TransactionFeed Class

Subclasses of TransactionFeed are used to create and cross reference instances of a LinkRecord subclass, classes that associate two or more entities.

Currently, there is no special protocol defined to manage subclasses of the TransactionFeed class. You can follow the procedures outlined for custom feeds, to define protocol for these subclasses.

You can define new TransactionFeed subclasses using the TransactionFeedSetup class.

The MembershipFeed Class

Subclasses of MembershipFeed are used to update and cross reference one-to-many relationships between a group and its members over time. For example, one or more companies can be members of the same industry. Over time, a company can move from one industry to another. In this case, instances of the entity class Company represent members and instances of the entity class Industry represent groups. To track this relationship, you could define a time series property industry at Company that returns an Industry object and a time series property companyList at Industry that returns a List of Company objects.

Two forms of relationship can be established:

Track member lists over time for a group - this approach defines a time series property at the group class for tracking lists of member entities. There is no corresponding property at the member class. For example, SP500 could represent a Universe of Security instances that is tracked in the time series property list at Universe.
Track a bi-directional relationship between a group and its members - this approach stores the list of entities at the group and also stores a specific instance of the group as the member's value. For example, US could represent a Country that tracks Company instances in the property companyList and country stores the value of a specific Company's country.

Each MembershipFeed subclass defines the entity class of the group, the entity class of the members, the property defined at the group's class that returns the list of members defined for the group, and optionally, the property defined at the member's class that returns the group defined for a member. In addition, a MembershipFeed can be defined as Replace or Append indicating whether all members for a group must be supplied in an update or if changes only are supplied.

A separate MembershipFeed subclass should be created for any membership relationship that you wish to update from an external source. The following items can be included in the header for any MembershipFeed upload:

memberId - id of member
groupId - id of group
date - optional date

These fields can be supplied in any order. The memberId and groupId are required; the date is optional.

For example, if the CompanyToIndustry feed has been defined, the expression:

     CompanyToIndustry updateFromString:
     "groupId  |   memberId  |   date
      HARDWARE |   IBM       |   Default
      HARDWARE |   HWP       |   Default
      AUTO     |   GM        |   Default
      AUTO     |   F         |   Default
      AUTO     |   C         |   Default
      AUTO     |   XYZ       |   Default

      HARDWARE |   XYZ       |   12/31/97
     " ;

can be used to setup the initial relationship between the industries and their members and to indicate that company XYZ was reassigned from the AUTO industry to the HARDWARE industry on 12/31/97.

This expression will perform the following steps:

Use the first line (i.e., the header) to determine the data items present in this feed and their order. In this case, the groupId memberId and date are supplied.
For each line in the supplied string, create an instance in the MembershipFeed subclass. In this case, seven instances are created in the CompanyToIndustry class.
For each instance in the feed, identify the group instance (i.e., the industry) and the member instance (i.e., the company) associated with the supplied identifiers.
For each instance in this feed that references a valid group/member combination, update the relationships using the following rules:
- If feed is setup as a bi-directional relationship, update the group associated with supplied members and the member lists associated with the supplied groups. If this is a Replace mode update, all relationships stored for any supplied group/date pair will be deleted. Each supplied member's group and each supplied group's member list will be updated as of each supplied date. If no date is supplied and this is a time varying relationship, member lists will be updated as of the earliest possible date (1/1/1).
- If feed is not setup as a bi-directional relationship, update the member lists associated with the supplied groups. If this is a Replace mode update, all members associated with any supplied group/date pair will be deleted and replaced with the members supplied with this update. If this is an Append mode update, the supplied members will be added to the member list for the group/date pair. If no date is supplied and this is a time varying relationship, member lists will be updated as of the earliest possible date (1/1/1).
Display exception reports.
Run any special wrapup procedures.

You can define new MembershipFeed subclasses using the MembershipFeedSetup class. This feed includes the following fields:

feedId - name of the feed class
groupId - entity class of the group
groupPath - property defined at the group's class that stores members
memberId - entity class of the group's members
memberPath - property defined at the member's class that returns the group defined for a member
mode - Replace or Append to indicate whether all members for a group must be supplied in an update or if only changes are supplied

The fields must be supplied in this order. You can include a header line in this feed but the header values will be ignored. The memberPath field is only required for bi-directional relationships. If the mode field is not supplied, it will default to append. The groupPath and memberPath properties should be defined through the Property Setup feed.

For example, to define the CompanyToIndustry feed from the previous example, use:

MembershipFeedSetup updateFromString:
"feedId	| groupId | groupPath | memberId | memberPath
CompanyToIndustry | industry |companyList | Company |Industry	
";

Membership feeds can also be set up to store the relationship information at a Bridge instance associated with the member or group entity.

The AliasFeed and XRefFeed Classes

Subclasses of AliasFeed are used to establish multiple aliases for existing Entity instances. For example, PortfolioAliases may be defined as a subclass of AliasFeed to establish multiple aliases for accessing your portfolios:

PortfolioAliases updateFromString:
"oldId   |	newId
PORT001	 |      IndexAccount  | TopFund
PORT002	 |      G&IAccount
PORT003	 |      BondFund	
";

Note that you can provide more that one alias in the data record.

You can define new AliasFeed subclasses using the AliasFeedSetup class. This feed must include the following fields:

feedId - name of feed
baseClassId - name of the entity class to associate with this feed

The fields must be supplied in this order. You can include a header line in this feed, but the header values are ignored.

For example, to define the alias feed for the Portfolio class, use:

AliasFeedSetup updateFromString:
"feedId  |   baseClassId
PortfolioAliases  |  Portfolio
";

Subclasses of XRefFeed are used to load alternative identifiers for an entity. These alternative identifiers are stored in Cross Reference Dictionaries (XRefs). XRefs provide a convenient way to track identifiers that are uniquely assigned by a specific source but may overlap with identifiers assigned by other sources. For example, suppose you want to establish the MSCI code as a alternative identifier for Security. You can use a SecurityXRef feed to do the following:

SecurityXRef updateFromString;
"entityId | sourceId | symbol
00000001  | MSCI     | M0000001
00000002  | MSCI     | M0000002
";

The sourceId must first be defined through the IdSourceMaster Feed.

You can define new XRefFeed subclasses using the XRefFeedSetup class. The following fields must be provided for any XRefFeedSetup:

feedId - name of feed
baseClassId - name of the associated class

For example, to define the XRefFeed for the Security Class, you can use:

XRefFeedSetup updateFromString:
"feedId      |  baseClassId
SecurityXRef |  Security
";

The RangeGroupFeed Class

Subclasses of RangeGroupFeed are used to define and update numeric ranges and categorize Entity instances into the appropriate range group.

You can define new RangeGroupFeed subclasses using the RangeGroupFeedSetup class.

The SchemaFeeds Class: Creating New Classes and Properties

Subclasses of SchemaFeeds are used to define new core classes, properties, and data feed classes. The class hierarchy is illustrated below:

          Object
             |
             IncorporatorPrototype
                |
                DataFeed
                   |
                   |-- SchemaFeeds
                       |
                       |-- ClassSetup
                       |
                       |-- PropertySetup
                       |
                       |-- MessageSetup
                       |
                       |-- DataFeedSetup
                           |-- MasterFeedSetup
                           |-- EntityExtenderFeedSetup
                           |-- TransactionFeedSetup
                           |-- MembershipFeedSetup
                           |-- AliasFeedSetup
                           |-- RangeGroupFeedSetup
			   |-- XRefFeedSetup

The ClassSetup Class

This feed is used to define subclasses of Entity, Bridge, DataRecord, and LinkRecord and includes the following fields:

classId - name of core class
parentId - name of class from which to create the new class
description - description of class
ospace - object space number for locating a new class

The classId represents the name of the class to create. If it does not exist, the class identified by the parentId is used as the parent class to subclass. The parentId is usually one of Entity, Bridge, DataRecord, or LinkRecord, but it can indicate a subclass of one of these, if appropriate. Since the classId is used to name Vision classes, the names should conform to the convention of starting with a capital letter, although this is not enforced.

The fields can be supplied in any order. If the classId does not reference an existing class, the parentId must be supplied. If the classId already exists, no action is taken. The description is optional and can be used to supply an initial description for a new class or to update the description of an existing class.

For example, to define the Entity subclasses Security and Country and the DataRecord subclass PriceRecord, you could use:

    ClassSetup updateFromString:
     "
     classId    |    parentId
     Security   |    Entity
     Country    |    Entity
     PriceRecord|    DataRecord
     " ;

This upload will create classes for any classId values that do not already exist.

By default, new classes created with the ClassSetup feed are stored in object space 3 except for new subclasses of DataRecord which are stored in the object space defined by Environment DBA DataRecordClasses . If you want a new class to be created in a different object space, you can include the ospace field in the feed header and provide the number of an existing object space as its value. You must create the object space prior to specifying it in the ClassSetup feed.

The PropertySetup Class

This feed is used to define new properties for Entity, Bridge, DataRecord, and LinkRecord subclasses and to set or update descriptive information for existing properties. Any property that you wish to update via a DataFeed subclass should be defined by this feed.

This feed uses the following fields:

classId - name of core class
property - name of the property
tsFlag - indicates if this is a time series property
adjustments - currency and application specific adjustment flags
defaultValue - default value of property for new instances
dataType - class of object returned by this property
description - description of the property
propertyXRef - name of property that cross references the values of this property at the class indicated by defaultValue

The fields can be supplied in any order as long as the header line indicates which fields are present. The classId and property fields must be present. The tsFlag and adjustments fields are only examined when a new property is created so these should be present if applicable. The other fields are optional and can be used to update new or existing properties.

The classId field identifies the class at which to define the property. This id must reference an existing class. The property field identifies the property to create or update. If the property does not exist it is created for the class indicated by classId. The classId and property fields must be supplied. The tsFlag field indicates whether the property should be created as a time series property. A value of Yes or True indicates that the property should be a time series. Properties are created as fixed properties by default. Note that if the property is already defined for the class, this field is ignored.

The adjustments field is used to indicate if the property is being used to store data that needs special adjustments. If this field is used, the underlying property name is created with a _ prepended to it and a cover method is defined using the property name without the prefix. The cover method is used to access this property with the appropriate adjustments. This field can contain one or more of the keywords defined to perform adjustments. The following keywords have been defined:

--- Adjustment Keywords ---
CURRENCY	data is in monetary units; adjust for currency
PERSHARE	data is supplied as per share values ; adjust for splits (investment applications only)
SHARES	data is supplied in terms of shares outstanding; adjust for splits (investment applications only)

Any number of keywords can be included for a property. This value is normally blank, indicating that no special adjustments are needed. This field is only used during new property creation. For example:

     PropertySetup updateFromString:
     "classId   property     dataType     adjustments
      GVData    price        Double       CURRENCY PERSHARE
      GVData    sharesOut    Double       SHARES
     " ;

The defaultValue field indicates the default value to set for the property when a new instance of its class is created. If this field is omitted, the new instances will have the value NA for this property. For time series properties, this default value is set as of the earliestPossibleDate (i.e., 1/1/0001). This value can be supplied as the name of a class (e.g., Currency to indicate that the property should return the default Currency object) or a specific value (e.g., 0 or -999.99). If a default value is supplied, all inputs are converted to a value of this type when updated from a feed.

Note: If the classId references an Entity subclass and the defaultValue is a DataRecord subclass, this property is used to establish the update relationship between these two classes.

If you want to supply a default type but do not want to automatically initialize the value of new instances for this property, use the dataType field instead. This field will set the returnObjectType for this property (which is used to convert inputs to the appropriate class) but will not initialize this property for new instances. If the defaultValue is supplied, the dataType can be omitted. In general, you will want to set the defaultValue when the property references an instance of an application class such as an Entity. This will initialize the property with the default instance of the appropriate class. If the property is intended to return a Double, Integer, String, or Date, you would indicate the appropriate class as the dataType and omit the defaultValue so that the initial value of the property is NA. Values supplied from the feed will be converted to the class specified by the defaultValue, if present, or the dataType otherwise.

The description field supplies a description of the property. It can be used to set the initial description for a new property or to update the description for an existing property.

The propertyXRef field is used to establish a one-to-many relationship between the supplied class and the class indicated by the defaultValue. If present, this field is used to define a property at the defaultValue's class that returns an IndexedList of the instances that are members of that class. For example:

Field	Value
classId	Company
property	country
defaultValue	Country
propertyXRef	companyList

implies the following:

The property country is defined at the class Company. This property will return Country instances. When a new Company instance is created, the initial value of its country property will be the default Country instance.
The property companyList is defined at the class Country. This property will return an IndexedList of the Company instances associated with the country. When a new Country instance is created, the initial value of this property will be an empty IndexedList. Note that if the tsFlag indicates time series, this relationship is tracked over time.

For example:

    PropertySetup updateFromString:
     "
     classId   |     property  |   tsFlag  |   defaultValue
     Country   |     x         |   N       |   0
     Country   |     y         |   Y       |   
     Currency  |     z         |   Y       |   Country        
     " ;

This update defines two properties x and y for the class Country if they do not already exist. Property x is defined to be a fixed property and y is time varying. Property x will initially be set to 0 when new instances of Country are created. Property z will be defined for the class Currency if it does not already exist. It will be defined as a time varying property. This property value will be set to the default Country for new instances.

The MessageSetup Class

This feed is used to add descriptive information to messages defined in your database. It can be used to supplement information supplied in using the PropertySetup feed or to describe methods that have been installed.

This feed uses the following fields:

classId - name of the class
message - name of the message
keyType - message can be thought of as a "key"
returnType - type of object returned by message
containerType - form object returned in (Object, List, IList, or TS)
level - usage level (Basic, Advanced, DBA)
function - function type
tvFlag - time-varying indicator (for methods)
brief - brief description
description - full description
paramList - comma separated list of parameter types for keyword messages

The fields can be supplied in any order as long as the header line indicates which fields are present. The classId and message fields must be present.

The classId and message fields identify the message. Both must be present and must identify a message that already exists.

The keyType field is optional and is used to indicate that the message can be thought of as a "key" for the class. Vision does not use keys explicitly but this field can be used as an indicator on schema based reports. Valid values for this field are FULL and PARTIAL indicating the message represents the only (FULL) key or a partial key.

The returnType field indicates the class of object returned by the message. The containerType field indicates the form of the object returned when the message returns a collection. Valid values are List, IndexedList, TimeSeries, and Object indicating the type of collection, if applicable. If no value or a bad value is supplied, the default value of Object is used, indicating that the value is NOT returned as a collection.

The function field is used to define a functional categorization for the message. By default, messages uploaded via this feed are set to a function type of Data. The level field is used to indicate expected user level for the message. Defined values are Basic, Advanced, or DBA.

The tvFlag field indicates if the message returns different values over time. If the message is a property or constant, this field is ignored since the time varying attribute is based on the type of property. If this message refers to a method, you can use this field to indicate that the method may return different values as of different dates by supplying the value Yes or True. By default, methods are not flagged as time-varying.

The brief field is used to define a brief description. The description field is used to define a long description.

The paramList field contains a comma-separated list of types, one for each parameter in binary and keyword messages. This field can also be used to set the index type for a message that return an IndexedList.

The GlobalsFeed Class

The GlobalsFeed class is used to override the default settings for loading a data feed file. The overrides only apply to the next feed run. The loadFromFile: and updateFromString: messages can be used to load global values with this feed. The first line of the feed may contain the header line record however it is ignored as the fields are assumed to be in the following order:

     option   | setting   | settingType

where option is a configuration option such as delimiter, fieldOrderList, or autoExpire and setting is the value you wish to use as an override. The settingType is optional and is used to indicate the type of data supplied in the setting field such as Date or DateOffset. For example:

     GlobalsFeed updateFromString: 
     "option      | setting        | settingType
      asofDate    | 9906           | Date
      autoExpire  | 1 businessDays | DateOffset
     " ;

Note: the data must be supplied in this order.

The GlobalsFeed is used by configuration files. The Global options used in the built-in feeds are described elsewhere.

Purging Data Using DataFeeds

DataFeed classes can be used to flag entities as inactive (MasterFeed subclasses) and to purge actual DataRecords and points in time series properties (EntityExtenderFeed subclasses). To indicate that you want to use a feed to purge data, you can set the enablePurge flag prior to executing the feed. The next time the flagged feed is executed, it will be run in "purge" mode. After it executes, the purge flag is automatically disabled.

For example:

     CurrencyMaster enablePurge ;
     CurrencyMaster updateFromString: 
     "id |
      CAD
     " ;

flags the Canadian currency as inactive.

MasterFeed Purges

You cannot purge the entity instances associated with a master feed but you can flag them as inactive. If you set the enablePurge flag prior to running a MasterFeed update, all entities included in the feed will be flagged for deletion, performing the following steps:

The entity's deletionFlag property is set to TRUE.
The entity's deletionDate property is set to the current date.
The entity's deletionReason property is set to "Purge Enabled".

Because the entity's deletionFlag has been set, the entity will no longer be included in the list returned by the activeList message. The entity will continue to be accessible via the masterList and instanceList messages as well as its class' naming dictionary.

If you want to store a specific reason for the entity's deletion, you can include the terminateFlag field in your feed. The value in this field will be used to update the deletionReason property in the entity. Note that if you include this field explicitly, you do not need to set the enablePurge option prior to executing the feed.

EntityExtenderFeed Purges

EntityExtenderFeed classes can be used to purge information in a number of ways depending on the style of feed. If the feed is used to update time series properties defined directly for an Entity or Bridge class, the update will delete dates from each time series specified when the enablePurge option is set. If the feed is used to update a time series of DataRecords, the update will delete dates in this time series when the enablePurge option is set. When the feed is used to update a DataRecord containing time series properties, the update will delete dates from the individual time series specified when the enablePurge option is set.

The entityId value can be specified as *, indicating that you wish to purge all entities for the supplied date. The date value can also be specified as a *, indicating that you wish to purge all dates for the supplied entity. Alternatively, the date can be specified as a range in the form:

date1:date2 (all dates between date1 and date2)
date1: (all dates from date1 on)
:date2 (all dates through date2)

For example:

     #--  Purge specific date for specific entity
     PricingFeed enablePurge;
     PricingFeed updateFromString: "id | date
     IBM | 990709
     " ;

     #--  All entities; explicit date
     PricingFeed enablePurge;
     PricingFeed updateFromString: "id | date
     * | 990709
     " ;

     #--  1 entity ; date range
     PricingFeed enablePurge;
     PricingFeed updateFromString: "id | date
     IBM | 990701 : 990729
     " ;

     #--  1 entity ; all points before a date    
     PricingFeed enablePurge;
     PricingFeed updateFromString: "id | date
     IBM | :990729
     " ;

     #--  1 entity ; all date
     PricingFeed enablePurge;
     PricingFeed updateFromString: "id | date
     IBM | *
     " ;