Notice:
  The views and information expressed in this document reflect the opinions and experience of the author Robert C. Pelletier.  Neither CSI nor the author undertake or intend to provide tax advice or trading advice in any market or endorse any outside individual or firm.  All recommendations are provided for their informational value only.  Readers should consult competent financial advisors or outside counsel before making any software purchase or investment decision.  CSI does not stand behind or endorse the products of any outside firms.

Copyright (c) 2001 Commodity Systems Inc. (CSI).  All rights are reserved.

Holiday Schedule CSI's Formula For Data Perfection

    CSI will be closed for voice communication on Monday, September 3 for the U.S. Labor Day Holiday. The CSI host computer will be accessible as usual, and data from those exchanges that remain open will be available at their normal times.

CSI's Formula For Data Perfection

    People are always asking us how we keep such an accurate database. We hope our customers and competitors alike will forgive us for keeping the specific list of sources as our own little trade secret. We will, however, share with all of you the basics of how we have compiled and how we maintain the cleanest financial database on earth. 

    There have been many data sources since CSI's beginnings in early 1969. At first we relied heavily upon newspapers for historical materials. Sources included The Wall Street Journal, The Financial Times of London and The US Journal of Commerce. Another important source of market information was a set that included bound editions of the out-of-print Chicago Daily Trade Bulletin and other vintage publications such as stock and futures exchange yearbooks that covered the period from 1899 to the present. A significant part of this set of invaluable source material was bartered from a friend who secured bound documents from the estate of a late Chicago Board of Trade official. Other sources include extensive newspaper microfilm libraries, and photocopies of selected data sources. 

    As many as 52 original CSI employees researched our first data bank of futures data from both the U.S. and London commodity exchanges and keypunched the material into electronic form. Much of the data was obtained from the above newspaper source material that I had carefully cataloged and saved in years past. The data was entered into multiple teletype terminals and stored on an IBM 7090 mainframe computer through a service offered by The General Electric Company. Our Florida offices hold all of this source data in various forms, as well as daily stories and commentary about the markets reaching deeply into the past. The historical commentary has been valuable to CSI whenever we are challenged about an obscure data point from the past.

    We got our first big break as a data vendor in the late 1970's, when CSI became the sole source supplier of market data for Tim Slater's original CompuTrac computer models. CSI outlasted CompuTrac and countless data and service firms in the futures industry. Today we service several thousand individual financial investors who use our analysis software and dozens of compatible programs for in-depth market research. We also have many commercial customers, including about 100 websites who post CSI data for visitor review, many data banking firms, universities, oil companies, mutual fund management companies, advisory services and many more.

    As vendors, we quickly learned that providing daily updates is an entirely different proposition from researching historical data. Newspapers simply won't do for keeping customers updated after the markets close. In the early 1970's, CSI was obtaining daily data from the national Quotron service. Soon Quotron went out of business and CSI replaced them with the Bunker Ramo Company service, which was later acquired by ADP. Finally ADP was acquired, but not replaced, by Bridge. Today we have many options for data sources, including some of the above, plus individual exchange sources, HyperFeed, SunGard, Reuters, S&P ComStock and Bloomberg.

    CSI now obtains data directly from up to 80 different exchanges around the world and from five of the commercial data firms mentioned above. To be certain that we are vending accurate information, we actually compile the data from our five sources plus the exchanges, and then compare all reports with each other before release. Errors do get through sometimes, but they are quite rare. 

    You might ask, "Why so many redundant sources?" If you were a data vendor, you would know why. Just about every day when CSI compiles data from the many international firms plus all of the exchanges, one or more (sometimes all) of the sources will commit critical errors that can influence the trading decisions of our ultimate customers. CSI could not, in good conscience let this happen.

    If an exchange gives bad data to all vendors, you might believe that all would vend incorrect information. This is not the case with CSI. Such an opportunity for widespread disinformation occurred at the end of June 2001. For two days NASDAQ failed to turn off a prototype software test system when sending out their market feed to vendors, and consequently sent wrong prices to all of us. 

    Most website data vendors captured the bad information from the exchange and unknowingly posted if for their regular site visitors. Fortunately for CSI customers, including 100 or so website re-vendors, our very knowledgeable data department staff picked up the error before releasing the flawed reports and held off on posting our NASDAQ data until the exchange had resent corrected data. Our data was a trifle late on those days and a few errors still occurred, but they were corrected as soon as they were identified.

    Regular CSI customers are accustomed to our very low error rate, and in a testament to our service, over 80% of new customer referrals stem directly from existing users. Our many data sources and the automation machinery behind those sources have made it possible for CSI to improve our regular end-of-day data posting time by perhaps a full hour or more. Some data elements such as mutual fund net asset value reports still slow us down a bit, but we often get all of the data ready before 7:00 PM Eastern time (US) each evening. CSI's csidata.com website monitors data posting progress as various data categories are completed, reducing the chance that less than a 100%-fruitful download attempt will occur.

    Our current mix of products and features makes CSI particularly well suited to service large corporations who manage their own assets, and companies who may repackage the data for sale to traders or investors. CSI's commercial data feed allows the corporate IT manager to quickly download the universe of market information for processing by appropriate software and to perform analytical modeling tasks. CSI's exceptional data quality greatly reduces the liability risks associated with analysis based upon unverified data. It is important to recognize that if you are using one of CSI's competitors who fails to employ several redundant sources and fails to exercise the necessary diligence, you will suffer many serious data errors that will compromise your trading plans and models. 

    To provide customers with the most accurate and complete service possible, CSI spends more on data than any vendor we know about. Multiple redundant data sources mean multiple redundant fees. We also pay full real-time exchange fees for world markets, which further add to our costs. We do this for our customers because we know that there is an implicit presumption of accuracy, rendering decisions that are based upon bad data far worse than decisions that are based upon no data. 

    In September 1999, Futures Magazine conducted an in-depth study of about a dozen prominent data firms. The study included CSI and some firms whose advertisements regularly boast of their "clean data." Also included were at least two of the most prestigious commercial firms in the world market. CSI won the contest by a substantial margin, beating out the closest firm by a factor of well over 2:1, and was named "the best" (the cleanest) of the studied lot. 

    Exactly what kinds of data does CSI supply? Originally we concentrated on commodity data, but we eventually branched out by adding stocks, options, mutual funds, indices and fundamental economic information. CSI's data reserves are principally arranged in a daily open-high-low-close summary form and include trading volume and open interest. We offer data on about 1,000 futures, all world futures options, currencies and bond markets representing nearly 100 countries. Cash (non-futures) data is also provided. Commitments of Traders data from the U.S. government (compliments of Steve Briese's Bullish Review) are also provided to help traders make informed market decisions. Mutual fund data includes dividend reports, sales charges and capital gains payments. All stock data includes price and volume figures and is accompanied by historical split information. Fact sheet information on most equity series holds CUSIP identifiers, company name, industry, country and sector identification information. 

    In addition, CSI provides world index data and important monthly economic data that will keep you informed on fundamental economy factors such as the national debt, CPI and PPI by sector. Numerous statistics such as unemployment rates, retail sales, etc. are provided in auxiliary files. Finally, we have written software to provide data on the single-stock futures markets that are coming soon to U.S. exchanges.

    One unique characteristic of CSI is that although we service thousands, every subscriber's copy of the CSI database is identical. Our remote correction facility promotes unparalleled excellence in data quality. It is not uncommon to find an error somewhere in such a vast database, but our technology lets us correct each error as it is discovered, even if the bad data point was entered decades earlier.

    Experience tells us what may appear to be an error is often a misinterpretation of the time series involved. For example, many exchanges report data in multiple forms, such as day-only pricing or 24-hour pricing, and we often report such markets in both forms. Customers sometimes observe day-only data and attempt to compare the range with 24-hour data that may be reported in their local newspaper. Twenty-four-hour price reports often show an expanded range of trading over the day-only reports. Such discrepancies sometimes lead customers to unknowingly report false "errors" to us. Customers help us detect errors, but we don't take anyone's word about a suspicious data point until we have called the exchange or examined other source alternatives. Questionable data points are typically researched and resolved within 24 hours.

    With so much data, it may be interesting to note the computer storage resources necessary to hold all of the above information. In ASCII (text) form, all of the historical data CSI provides would take up just under 4,000 megabytes. In compressed form, the database size reduces to around 400 megabytes, an easily manageable quantity by today's computer standards. The extensive work involved in compiling, researching and certifying data through our comparison of multiple redundant sources allows CSI to claim a copyright on all our data (see sidebar). 

    We hope this overview of our data sources and procedures will help quell any doubts about the accuracy of your database or the commitment of your data vendor. We have come a long way from our humble beginnings and look forward to many more decades as the premier data supplier for serious investors and serious entrepreneurs who wish to attract web site visitors with flawless world-wide financial information.
 

   CSI has a right to claim a copyright for the data we provide because our data is a compendium of data from many sources. A single source, according to the World Trade Organization, is on much shakier ground when it comes to copyrights. The argument centers upon what is considered to be "artistic content." A compilation of many sources passes that test, but a given exchange who wants to both capture the promotional value of making their information public and also claim a copyright cannot, it appears, have it both ways.