An Essay By Bob Pelletier
Reprinted from the CSI Technical Journal - Jan. 2001
Tools. Where would we be without them? From the can opener to the computer on your desk, tools are indispensable. This holiday season finds me reminiscing about a college professor I had who spoke at length on the internal tools we carry throughout our lives. What tools did we possess? Did we speak English? French? Spanish? These were all tools that helped us communicate, we were told. He went on to explain that programming languages were also tools. Soon we would own those tools and they would enhance our careers in ways never imagined by our own parents.
Many years have passed since my introduction to programming tools in the 1950s, but I suggest that, as we welcome the new year, we all sit back and examine the tools we bring to the task of trading the markets. The most important tools for the technical investor include computer, data and analytical software. For those of us who delve into the mechanics of developing trading systems, additional tools include the programming languages and corresponding skills we use in our craft. Is your set of tools growing and improving to meet the demands of the new millennium?
I am pleased to say that I have recently learned a new programming language called Perl, which offers a unique set of advantages to help me reach my goals. This is not to suggest that Fortran, VisualBasic® or other programming languages have been replaced in my personal tool kit, but that new tools may provide some rewarding results and improve overall capability as a programmer and market analyst. This article is to help the many who, like me, are accustomed to programming in Fortran or even Basic, and who may be a bit reluctant to try the Perl language for use with Unfair Advantage's® MarketScanner.™ This tutorial just may be all you need to get up and running with Perl. It is our hope that it will shorten the learning curve.
Having used the Perl language for a few months to drive CSI's MarketScanner, I feel qualified to speak as someone who knows and shares the struggle of transition. I have found pitfall after pitfall, and hope that by sharing lessons learned, I can help you avoid many of them. If you can program in VisualBasic or Fortran, then Perl won't be all that difficult to learn. However, if you have no programming experience whatsoever, then you might want to pass this on to your programmer friends; this introduction may be a trifle difficult for non-programmers to understand.
Perl is worth learning for several reasons. As you become accustomed to the language, you will begin to appreciate its compactness. A moderately complex program can usually be written on a single page. This is a positive feature in favor of Perl. Because it is an interpretive language, it doesn't require compiling (like Fortran) to be run and debugged. Perl includes its own DOS debugger, so you can complete the process without going into additional programs.
On the flip side, you will soon see that even the simplest program may be difficult to debug and run. Inevitably, I find myself missing a semicolon at the end of a line, using a parenthesis instead of a square bracket, or perhaps missing a curly bracket somewhere. Other pitfalls involve incompatible calls to subroutines with missing or undefined arguments. Perl is the least forgiving language I have encountered.
Program statements (commands) must always end in a semi-colon; left and right curly brackets are used to delimit conditional statements and various levels of nesting. Variables must be preceded by dollar signs, arrays must be preceded with an @ sign and much of the arithmetic is done in floating point. A structure must be followed for subroutine usage and the passing of variables. Square brackets are used to reference array elements, and parentheses are used for function calls to subroutines.
The language makes use of symbol operators, such as +, -, <, >, !, eq, ne, and operators such as "and" and "or" may be used in statements. For example, the symbol "!" is to be interpreted as "not" within the Perl code. C++ conventions are also permitted such as the use of "--" to decrement or "++" to increment when used adjacent to variable names.
An important requirement of Perl is the need to spell all variables with the same case everywhere they are used. The variable "$PositionPrice" may not be referenced as "$Positionprice" because the portion "Price" must be capitalized in the same manner in every such reference or Perl will not understand which variable you are attempting to reference.
We'll make broad use of examples from MarketScanner's working code throughout this tutorial. Simply launch MarketScanner through UA, then pull down the "Insert Custom" option under the "Columns" menu and click the "List" option. You'll see a great deal of sample code – all of it functional. This code will be an integral part of our attempt to help you use this language. Please refer to it as the many examples and references to Perl conventions are mentioned. This way you will obtain a quicker understanding of Perl, and how it may benefit you. The extensive set of examples available within UA's Market Scanner should be helpful in demonstrating how to use and navigate the Perl language.
Whenever a Perl program is written within MarketScanner, a supplementary package of Perl code subroutines that address database issues is included, courtesy of CSI's Steven Davis. This code is always appended to the code supplied by you. Therefore, when you write some code to analyze stocks or futures markets, your code — call it "myperlstudy.pl" — will always be followed by Davis' Perl code to produce the working code to be run in the name of perlstdy.pl.
The code that is added to your code includes subroutines to deal with the CSI database. I won't produce an exhaustive list, but some of the subroutines are: getOpen[ ], getHigh[ ], getLow[ ], getClose[ ], getDate[ ], getVolume[ ], getOi[ ], GetSymbol[ ], GetName[ ], GetSectorCode[ ], GetIndustryCode[ ], GetExchange[ ], GetCVF[ ], GetFact(), getNumDays(), etc. These subroutines are necessary for you to get into the database for many kinds of analytical purposes. getNumDays(), for example, is a routine that returns the number of days available for the given stock or future currently being processed.
Inside the brackets of array references, such as those identified above, would be a variable or an integer referencing the array. For example, to reference the very last day on file (the most current day of data), the integer supplied would be a zero (0). If a variable is given such as the variable $i inside the brackets, then the contents of $i would be a 0 to accomplish the same current day element. To access the day before the last day on file would require a 1 or a variable reference which holds a 1.
In reviewing the sample code, you'll see that a "sub" statement followed by the selected study name and a left curly bracket ({) introduces the list of studies that you or CSI supplied, together with each study's list of variables. At the very end of the subroutine, after all supplied code has been inserted, a closing right curly bracket (}) is required. Should "if" statements be used requiring another set of curly brackets, it is advisable to place the required pair of curly brackets in the same column of different lines to keep your logic from being too confusing. A pound sign (#) is used to introduce a comment, either following a program statement or on a line by itself. If the comment continues to a subsequent line, another pound sign is required.
In some cases, an array of "Indicator Labels" is introduced within the subroutine. These pertain to the Excel spreadsheet where stock-by-stock or future-by-future results are displayed. These items can best be defined through the examples shown. The indicator labels normally present column headings. The IndicatorDisplayFormat shows the number of decimal places of display, and whether data is presented in a centered, left-adjusted or right-adjusted format. The characters used indicate "l" for long, "f" for float, "s" for string, and "L," "R" or "C" for a left-adjusted, right-adjusted or center-adjusted display, respectively. All local variable definitions must be introduced with a "my" statement. For example, to define $PositionPrice, you must have introduced it with my($PositionPrice) when it is first referenced. Please note the generous use of "my(…)" statements in the example code.
Perl has several reserved words, such as "defined," "my," "return," "chop," "split," "for," "if," "else," "while," "reverse," "return," etc. The "return" statement delivers columns of information to the Excel spreadsheet for the given stock analyzed. The Perl code supplied through your main subroutine (sub) is executed to completion for each time series (stock or future) supplied by MarketScanner. When the list of all stocks and/or futures time series is exhausted by repeated entry into the scanner matrix, the job will be completed and the spreadsheet will be filled with the details of the study's objective.
Some subroutines may have a list or an array of arguments. The given Perl subroutine refers to that list through use of the underscore character. For example, a routine with a single variable such as the variable $Np in a call to stti($Np) would be accessed by the stti subroutine as "my(@t) = @_; and my($Np) = $t[0];" Of course, the @ sign introduces an array, and the underscore (_) refers to subroutine input.
The above statement will transfer the input array @_ into the internal array @t and store the contents of $t[0] into local variable $Np. In effect, there is no array $Np, but there could have been one. $Np is a single variable that has been passed for use by the subroutine stti for use within stti. The "my" command above simply tells the stti subroutine that this is a variable local to the stti routine to be used within stti.
Perl makes extensive use of "for" loops, which are much like Fortran "Do loops" or Basic 'for' loops. A typical "for" loop might read, "$sum=0; for($i=$#t;$i>=0;$i--) {$sum += $t[$i];}" When the two characters "$#" precede an array reference (such as t), Perl obtains the array's highest possible index reference. The above loop will accumulate into the variable $sum from the element $i of an array t, where element $i ranges from 0 to $#t. In other words, the sum ($sum) of all the values will result from adding together the $#t element through the 0th element of the t array. Summing will stop after $i reaches 0, the very first element of the $t array of $#t + 1 elements. In Perl, an array of N elements would range from element 0 to element N-1.
As you build on the MarketScanner custom study list, you can access and run any Perl code that you have supplied within the perlstdy.pl file. The custom programs will be presented in a list when you right click your mouse on the blank white portion of the MarketScanner screen. Click the sub program that you wish to run and enter the arguments, if any, required by your program. It should be apparent to you by now that the code you might write or employ from the "Insert Custom…" list is all pretty much targeted to feeding an Excel spreadsheet. The right-click of your mouse simply alphabetizes all the subroutines in your custom list. When you add your own indicator or study, it will appear in that custom list.
This little summary of Perl usage may be helpful to you, but don't despair if it leaves you unsure of how to proceed. It isn't always easy to add a new tool to your set. Perhaps you would benefit from reading one of the many books available on the subject (see "Perl Tips," below). We don't want to drag anyone kicking and screaming into the world of Perl, nor do we want to alienate those who just don't like it. In fact, we continue to offer VisualBasic support within MarketScanner for those who prefer that language. Please consider this an invitation to begin the new year by joining me in enhancing the tools that will help us all be successful in 2001.