Page Last Updated 7/27/2001
SI Reader
User Manual

Table of Contents

**Update Information**
System Diagram
Synopsis
General
Critical-Message Notification

System Command

Environment Variables
Installation and Maintenance

Installation
Directories
Switching to Production
Applying Updates
Testing Updates

Files and File Formats

Log File
SI Deal Configuration File
SI Deal Record File
Cache File
Client Deal Record File
Client Deal Sequence Number File
Last Deal Id File
Critical Message File
Script Files

TOF Mimic Considerations

TOF Limitations
Special Script Transformations

Operator Interaction

System Startup
Process Status
System Shutdown
Daemon Mode
Interactive Mode
Signal Handling

Periodic Cleanup

New Features

Script Configuration
Default Script Configuration Settings

Bulk Fund Deals (Allocations) and Single Fund Deals
TOF Back Office Reference
Front Office Loading

Sybase
IBM MQ
MQ Configuration File

TCP-IP Stream Dump/Replay
TCP-IP Full Reset on Error
TCP-IP Failover
Kill Protection

 


SI Reader
User Manual

System Diagram

Synopsis

ctksi IP port tcid script_ext cache_delay [controlport]

mqctksi ignored ignored tcid script_ext cache_delay [controlport]

General

Ctksi is a TCP-IP client and mqctksi is an MQ client that opens a port resp. queue on a Cognotec server machine, establishes an on-line session and reads deals as they are sent from the Cognotec dealing system. As deals are received they are cached to the local disk where they wait for a configurable time for the arrival of payment instructions. If payment instructions arrive, or if the "cache delay" timeout period expires, the consolidated deals are fed to the Softek scripting system. Deal data may be loaded into any Back or Front office systems (or any number of combinations thereof by use of a deal splitter).

On startup ctksi reads a deal configuration file (DCF) supplied by Cognotec, checking it for any obvious errors. Once this check is completed, ctksi opens a socket, binds it to the IP and port numbers, that are associated with its first two arguments, and then repeatedly attempts to connect (as a client) to an SI Server that is expected to be listening at the bound IP location. If the connection to the SI Server fails for any reason, ctksi returns to this "attempting to connect" state.

Once connected, ctksi responds to SI Server "PING" messages and ACKs, or NACKs deals (according to the SI specification) as they are presented. Ctksi handles any duplicate deals that are presented sequentially by ACKing them and not passing them on to the Scripting System. If the same deal is received more than a few times (currently 5 times) in a row, ctksi NACKs the deal, in an attempt to persuade the SI Server to move on.

If "PING" messages stop arriving for a configurable time, the TCP-IP or MQ connection to the SI Server is reset and ctksi repeatedly attempts to reconnect using fresh resources.

Ctksi runs on any hardware and operating system that supports an ANSI C compiler.

The parameters for ctksi are as follows:

IP the IP name or number of the remote machine to be used to present the network protocol. This parameter is ignored if configured for an MQ connect.

port the port number (or a port name present in the "services" file) that ctksi should attempt a connect on. This parameter is ignored if configured for an MQ connect.

tcid a 4 character code (usually "LITE") which will be the ID of any associated Reuters Server Mimic.

script_ext the 3 character filename extension of the set of scripts that will be associated with this SI reader.

cache_delay the number of seconds that this SI reader should wait for associated Payment Instruction records before sending whatever information it has to the downstream processor (e.g. a TOF Mimic).

controlport this optional parameter may be set to the number or "services" name of the control port that will accept external TCP-IP connections and respond to status requests (7/27/2001).

Critical-Message Notification

System Command

Critical Messages are delivered by a “system” command which is driven by the environment variable MRG_ECMD. The shell command specified in this variable is executed with a minimum granularity of 10 minutes and is passed any critical messages issued since the last time the command was executed. Unsent critical messages are stored in the file <TCID>.msg which is passed on the standard-input to the command in the MRG_ECMD environment variable. The execution of this command is noted in the logfile (see later). To prevent unnecessary repetition of identical messages, critical messages stored in <TCID>.msg may be processed with “uniq” for example, as follows:

setenv MRG_ECMD “(uniq -6 | mail -S \"Critical Softek Message\" salist)”

Each line in the <TCID>.msg file is timestamped and its process-origin is identified. <TCID>.msg is deleted each time after the command in MRG_ECMD is executed.

To prevent short-term error conditions from producing “orphaned notifications” a 45 second delay is implemented for new batches of error notifications.

Environment Variables

The default behavior for certain interface functions may be modified by the use of “environment variables”. These variables must be set prior to the interface execution (usually in a “shell” or “batch” file). Command parameters take precedence over equivalent environment variables. The file ~softek/ctksi/set_my_env is used to contain all the critical ctksi environment variables. This file is sourced-in before any of the ctksi binaries are executed. The environment variables, used by the interface, all start with “MRG_” or "CTK_" and are as follows:

MRG_ECMD The command to be used to process critical (information and error) messages (including all arguments to run the command) is taken from this variable. The command must take its input from the “standard input” stream.
CTK_MSGDIR The directory where transitory work files will be stored. Both critical messages and conversation details (Reuters only) make a temporary appearance in this directory.
CTK_LOGDIR The directory that will be used to store the logfiles generated by this program.
CTK_DLSDIR The directory that will be used to store the dealfiles generated by this program.
CTK_CACHEDIR The directory in which deals awaiting Payment Instructions should be stored. Each deal is stored in its own file and named according to the "SI Deal Reference Number"
CTK_CACHEDELAY Same as the cache_delay program argument
CTK_SCREXT Script extension specifier, usually set as an argument to ctksi
CTK_ACKLATEPI By default, this option is switched on and results in payment instruction records, that are not received within the cache-delay period, being ACKd to the SI Server. If this variable is set to "no" or "NO", these "late" payment instruction records will be NACKd. See later for a discussion of this issue.
CTK_PULSETIMEOUT The value in this variable represents the number of seconds that ctksi may wait, without receiving a PING (heartbeat) from the SI Server. If this number of seconds is exceeded, the TCP-IP port to the SI Server is completely reset and attempts are made to reconnect with a newly opened socket. If this variable is not present or it cannot be decoded as a number, the default value is set to 300 seconds.
CTK_VIRTUALHOSTNAME Enables the TCP-IP Failover feature.
CTK_TCPMSGDUMP Enables the TCP-IP Stream Dump feature.
CTK_TCPMESSAGES Enables the TCP-IP Stream Replay feature.
MQSERVER If MQ deal processing is configured, this variable must be set up in accordance with IBM's guidelines. It is used by the IBM's MQ client software to configure an MQ connection to a remote MQ server.

Installation and Maintenance

Installation

The ctksi distribution comes as a gzipped tar file. The command:

gzcat ctksi.tar.gz | tar xvf -

or

gunzip ctksi.tar.gz; tar xvf ctksi.tar

when executed from within the Softek home directory while logged in as "softek" should completely build the runtime directory hierarchy. The file "~softek/ctksi/i_s" will need to be edited to make sure the first two arguments (IP and port) are correct for this customer. The "softek" login should be /bin/csh and have ". ~/bin" as the first two entries in its search path and, if a TOF Mimic is to be used, the device "/dev/ttya" should have permissions set to "world rw". In addition, this port (/dev/ttya) must not have a login shell enabled on it.

Directories

Once unzipped the directory hierarchy should look like this:

~softek
~softek/bin
~softek/ctksi
~softek/ctksi/deals
~softek/ctksi/logs
~softek/ctksi/cache

The ~softek/bin directory should contain symbolic links to executable binaries and shell scripts that are also hosted in ~softek/ctksi. An important configuration file (which should usually be left as distributed) is the shell command file that defines the Environment Variables ~softek/bin/set_my_env which usually contains the lines:

#!/bin/sh
CTK_PRINT=no; export CTK_PRINT
# change the "yes" to a "no" below to NACK late Payment Instrs.
CTK_ACKLATEPI=yes; export CTK_ACKLATEPI
DL_DLSDIR=./deals; export DL_DLSDIR
CTK_CACHEDIR=./cache; export CTK_CACHEDIR
CTK_DLSDIR=$DL_DLSDIR; export CTK_DLSDIR
DL_LOGDIR=./logs; export DL_LOGDIR
CTK_LOGDIR=$DL_LOGDIR; export CTK_LOGDIR
MMC_LOGDIR=$DL_LOGDIR; export MMC_LOGDIR
CTK_VIRTUALHOSTNAME=Virtual_RRS; export CTK_VIRTUALHOSTNAME
# MQSERVER=ctksi/TCP/peizo.shappys.com; export MQSERVER
# CTK_TCPMESSAGES=TCPmsginput.bin; export CTK_TCPMESSAGES
# CTK_TCPMSGDUMP=TCPmsgdump.bin; export CTK_TCPMSGDUMP

Note: Turning on the MQ facility involves setting the MQSERVER variable and uncommenting it in the ~softek/bin/set_my_env file.

Switching to Production

  1. Pick a date/time when you want to go live.
  2. When the moment arrives shut down the Softek interfaces (i_kill command).
  3. Delete all the files from the ~softek/deals directory and the ~softek/cache directory.
  4. Switch over the Bank's TOF client to production.
  5. Make sure no more "pretend" deals are entered :-).
  6. Restart the Softek interfaces (i_start).

The Reuters sequence number file (LITE.seq) or the "last-SI-deal-received" record file (LITE.lst) should not need to be touched.

If, however you want to start the production run at a specific Reuters sequence number the file LITE.seq, in the $CTK_DLSDIR (~softek/deals) directory, contains the last assigned Reuters ticket number. Providing you are deleting the LITE*.dls files from the directory (while the interfaces are down) you should be able to put any number in here and have the SI reader assign the NEXT number to the next deal (wraps back to 1 at 999,999 just like a Reuters server).

Applying Updates

Before software updates are made available to Banks, they are tested at Cognotec. If the Bank does not wish to perform their own testing, the updates, when distributed in "tar" format may be installed over the existing directory hierarchy using a command similar to:

cd ~softek; tar xvf ctksi_upd.tar

To ensure continuity of deal presentation, the following files should be preserved before the upgrade has been installed and their contents should be checked after the upgrade has been installed and before the Softek processes are restarted, to verify that they have not changed.

~softek/ctksi/deals/LITEmmdd.dls (where mmdd represent the last 5 days (mm=monthnumber and dd=daynumber))
~softek/ctksi/deals/LITE.seq
~softek/ctksi/deals/LITE.lst
~softek/ctksi/i_s (to preserve the IP and port designations of the SI Server)

Testing Updates

If the Bank requires their own testing before they go live with the update, the following procedure should preserve a "reversion" path in the event of test failure.

First

i_kill
cd ~softek
mv ctksi preserve_ctksi

This makes sure you will not change any files in the old (production) ctksi directory hierarchy.

Then

mkdir ctksi
cd preserve_ctksi; tar cf - . | (cd ../ctksi; tar xvfBp -)

Now follow the directions for applying an update. You must make sure the Bank's TOF client is targeted at a test system. You should now be able to use i_start, i_query, and i_kill as normal (these are implemented with symbolic links from the ~/bin directory and should always be pointing at binaries within the ~softek/ctksi hierarchy).

Once successful testing is complete you may:

i_kill
(retarget the TOF client at a production system)
cd ~softek
rm -rf preserve_ctksi
i_start

or if it is necessary to revert back to the previous production system

i_kill
(retarget the TOF client at a production system)
cd ~softek
rm -rf ctksi
mv preserve_ctksi ctksi
i_start

Files and File Formats

Log File

A logfile named SIDL<mmdd>.log in the directory pointed to by the CTK_DLSDIR environment variable (where <mmdd> stands for the digits representing the month and day that the file was created) is appended to as log-events occur. All "critical" and non-critical messages are logged.

If ctksi is left running overnight, a new logfile is automatically created as soon as necessary after midnight.

SI Deal Configuration File

When ctksi starts up it looks in its execution directory for a file named "dcf.ini". This file is a direct copy of the file used to provide deal configuration information to the SI server. Ctksi parses this file and extracts SI field names on a per-deal-type basis. Field names referencing data at similar offsets in a deal record may have different names. All that is required is that all correspondingly indexed fields, for all deal-types occupy the same space in the deal record. On startup, ctksi checks that this is the case (that the "dcf.ini" file is legal) and outputs the number of fields found in each deal-type along with the name of the deal-type. The format of this file is not provided here but a sample might look like this:

SP-ACTIVE
SPEC_STX,1,0,0 (* Start of Message *)
LIST_DEAL_NO,19,0,0 (* Key - Deal Num - Branch 8, Year 4, Seq 7 - *)
LIST_DEAL_ID,1,0,0 (* Key - Deal Component *)
SPEC_FX_MESSAGE_TYPE,2,0,0 (* Key - ND or UD *)
SPEC_RAB_FX_DEALNO_SEQ_NO,7,0,0 (* 501 - Source Ref *)
...
...

SI Deal Record File

A file named SIDL<mmdd>.dls is appended to as each deal record arrives from the SI server. Each record, representing the actual deal data received from the SI server, comprises three main parts,

  1. a timestamp on a line by itself, together with the word "Start" and the SI ticket number.
  2. the fields of the record (each field begins a new line and starts with the SI field identifier (a name) followed by a colon (':') character and then followed by the field data). If the field data contains newline characters, these are also output,
  3. the timestamp in 1. repeated (together with the word "End" and the ticket number).

Cache File

When a deal is received from the SI interface its data record is stored in a file whose name is based on the SI deal type and reference number. A header is written into this file that comprises:

  1. The system representation of the time when this deal should be presented to the TOF
  2. The number of possible legs for this deal type
  3. The number of legs received so far
  4. The type of the last leg received

Raw deal data is appended to this header on a separate line. Whenever a new leg is received a new header is written and new deal data is appended to the old on its own line. When either all the possible deal legs have been received or the timer in the record header has expired, the data in this file is presented to the script system which, when configured to feed a Reuters TOF mimic, appends a new record to the appropriate dealstore file and the Cache File is deleted. The ticket number for this new Reuters deal is taken from the Client Deal Sequence Number File after is has been incremented.

Client Deal Record File

The default set of scripts, provided by Softek causes the creation of a Reuters Deal Store in text file form. This text file is indexed by Softek Reuters Deal Processors and Reuters Deal Mimics, any number of which may be simultaneously processing data from this Deal Store. The format of the Deal Store is as follows:

A file named <TCID><mmdd>.dls (where <TCID> is the string used as the third parameter to ctksi) is appended to as each deal record is cleared from the SI reader's deal cache. Each record, representing the transformed deal data received from the SI server, comprises three main parts,

  1. a timestamp on a line by itself, together with the word "Start", the created Reuters ticket number and optionally followed by the letter "C" (to indicate a "conversation" record).
  2. the fields of the record (each field begins a new line and starts with a Reuters field identifier (a 3 digit number in the range 500 to 599) followed by a colon (':') character and then followed by the transformed field data). If the field data contains newline characters, these are also output,
  3. the timestamp in 1. repeated, together with the word "End" and the repeated ticket number.

Client Deal Sequence Number File

A file named <TCID>.seq is rewritten with the number of the last unique Reuters-like ticket number written, by the Script System, with a deal to the Client Deal Record File. Since the deals may be written to the Client Deal Record File in a different sequence than they are presented by the SI (as a result of ctksi waiting for possible payment instruction records) downstream processors (such as a TOF Mimic) need access to a guaranteed sequential sequence of numbers. The Client Deal Sequence Number stored in this file is a number between 1 and 999,999 that wraps back to 1 at 999,999.

Last Deal Id File

A file named <TCID>.lst contains a unique representation of the last deal that was ACKd to the SI. The unique information comprises a combination of the following SI fields

dealno_branch

Branch of a Deal

dealno_year

Current Year

dealno_seq_no

Deal reference number

dealno_id

Deal Id. for Near/Far leg (A resp. B)

message_type

Type of Message (ND = New Deal, UD = Update)

deal_type

SP, FR, FO, SL, SD, SN, TN, EN, SF, TF, EF or DL

e.g.

DEALROOM19991000178AUDSD

This information is used to determine if a sequential duplicate deal has been received from the SI Server. In this event, the duplicate is ACKd unless it has been received more than five times in a row, in which case it is NACKd with a "%BXX" (an anonymous NACK).

Critical Message File

(UNIX and NT installations only, see “Critical-Message Notification” above)
The file <tcid><mmdd>.msg contains one line for each critical message. Each line comprises four sections:

  1. <application name>
  2. [<DateTime>]
  3. <a critical message>
  4. #<a critical message number>. Error numbers produced by ctksi are in the 140,000 to 149,999 range.

For Example:
ctksi [Wed Sep 29 12:02:24 1999] Payment instructions for SP/A deal <1000168> did not arrive in time for TOF #141233

Script Files

Currently a different script file is executed for each of the Reuters main deal types that SI handles i.e. Spot/Forwards, Swaps, and Money Market. In Reuters parlance, each of these deal types is defined by the corresponding information fields that may be valid (called Field Lists).

Record Type

Name of Script File

SP, FR, FO, DL

si501.<ext>

SN, TN, EN, SF, TF, EF

si502.<ext>

SD, SL

si503.<ext>

TOF Mimic Considerations

Cognotec and Softek provide a set of Front/Back Office Deal Processing Scripts with ctksi that will generate sufficient information to drive a Reuters Server Mimic. This is just one of the many deal processing mechanisms that the Softek scripting system is capable of implementing and some constraints that are associated with its choice should be considered by the Bank.

TOF Limitations

TOF (Ticket Output Feed) is a serial line (RS232) protocol that is defined to run at 9600 baud. At a maximum throughput below 1kbyte per second, and considering the overheads involved in the protocol itself, each deal may take several seconds just to travel down the serial line to its destination. That said, the nature of the TOF protocol results in reliable transmission of deals (even if slow) through an inherently unreliable (and generally short-haul) medium (RS232). Tunneling this short-haul transport through one which has longer range (LAN or WAN) may introduce character latencies that will cause packet repetition which, even on a fast medium could lead to slower than expected data throughput. Packet duplication is dealt with by the protocol, however, and should not result in any deal duplication at the application level.

At the data level, there are certain SI fields that have no TOF equivalent and vica versa. These are addressed in more detail in a document from Cognotec, but certain, fairly TOF fundamental information (e.g. Counterparty TCID, which is defined for all real Reuters customers worldwide) cannot be mapped to SI equivalents. Usually, an alternative representation is provided, but some extra effort is involved, on the part of Bank systems implementers, to integrate these alternatives.

TOF deals are self contained, that is, each deal contains all the data fields needed to describe the legal deal entity. The TOF cannot supply data that might arrive later about an already sent deal. As a result, any SI Payment instruction records that arrive after the designated cache-time cannot be sent through the TOF mechanism to the Bank. Setting the CTK_ACKLATEPI environment variable to "no" or "NO" will cause these "late" messages to be NACKd when they are received by ctksi. This, however has the undesirable side-effect of not allowing these records to be printed via Cognotec's reporting system. By default, this variable is unset, and "late" PI records are ACKd to the SI Server (having the other undesirable side-effect that Cognotec systems are not aware these records have not been delivered to the Bank). There are ongoing discussions dealing with ways around these TOF limitations (e.g. providing separate Back Office and Front Office TOF feeds).

Special Script Transformations

To facilitate deal loading (into TOF or directly into other Bank deal-loading systems), some Script Transformations have been developed that are specific to the format of data provided in SI records.

loaddlsname
The workarea is loaded with the name of the current client deal record file

loaddlsstart

The workarea is loaded with the word "Start" followed by a "#" character and the currently available TOF ticket number.

loaddlsend

The workarea is loaded with the word "End" followed by a "#" character and the currently available TOF ticket number.

ctkdate2reuters

The Cognotec-format date in the workarea is replaced by the equivalent date in Reuters TOF format.

ctkdealtype

The SI fields "SPEC_RAB_FX_DEAL_TYPE", "FXM_FIXED_BANK_SELL_CCY_NEAR", "FXM_FIXED_COUNT_CCY_NEAR" and "FXM_FIXED_FIXED_CCY_NEAR" are checked, decoded and used to determine the Reuters equivalent "deal type" (Reuters TOF field 514). Since FRAs are not implemented in the SI, a number between 1 and 6 is loaded into the workarea. If any of the required fields are not present, this transform will indicate failure (testable using the if statement).

ctk2r2kcur

Converts an SI currency volume into its minimal Reuters form, stripping off a leading '+' and leading zeroes before a decimal point (leaving the last if necessary). Trailing zero are stripped from the RHS of a decimal point leaving only one if necessary. The resulting floating point number is placed back in the workarea.

ctksubrates

The floating point number found in the workarea is taken away from the floating point number found in the savearea. The resulting floating point number is placed back in the workarea. Equivalent to the expression:
workarea = savearea - workarea

If the workarea or the savearea does not contain a number, this transform will indicate failure (testable using the if statement).

ctkpuredealtype

The SI field "SPEC_RAB_FX_DEAL_TYPE" is checked, decoded and used to determine the equivalent Reuters "Pure Deal Type" (TOF field 569). Since FRAs are not implemented in the SI, a number in the set [2, 4, 8, 16] is loaded into the workarea. If the required SI field is not present or does not contain a legal value, this transform will indicate failure (testable using the if statement).

ctkfieldlist

The SI field "SPEC_RAB_FX_DEAL_TYPE" is checked, decoded and used to determine the equivalent Reuters Field List Number. If everything checks out, a number in the set [501, 502, 503] is loaded into the workarea. If the required SI field is not present or does not contain a legal value, this transform will indicate failure (testable using the if statement).

ctkperiod

An SI Deal Period designation in the workarea is converted into its Reuters equivalent, suitable for loading into TOF field 515 or 516.

ctkloaddltype

The current SI deal type (as specified in the Deal Configuration File) is loaded into the workarea. After this transform, the workarea will contain one of the following string tokens (without the quotes):
"SP", "FR", "FO", "SN", "TN", "EN", "SF", "TF", "EF", "SL", "SD", "DL"

loadrawrecord

The current SI TCP-IP deal record is stripped of its STX and ETX framing characters and loaded into the workarea.

loadlabeledrecord

The current SI TCP-IP deal record is stripped of its STX and ETX framing characters and labels are inserted before each data field with an intervening colon ':' character. Each field is terminated by an ASCII NL (newline) character (dec 10, hex 0x0A). The last field is also terminated by a NL character. The labels for this transform are deal-type dependent and are taken from the current dcf.ini file. The resulting new record is loaded into the workarea.

mq_write

The contents of the workarea is sent to the IBM MQ queue whose name is specified in the savearea. The mq.conf file is used to store information about the default queue name and any name mapping that will be used by the namemap data transform. mq_write establishes a new connect to the designated Queue Manager as necessary. Remote server access is governed by the contents of the MQSERVER environment variable, the black art of correctly setting up the value for this variable is, unfortunately, beyond the scope of this document. As of 6/15/2000 the savearea contains the name of the MQ queue into which data is inserted. If the value of the savearea is "default" (no quotes), the MQ queue name used will be the first non-comment token from the mq.conf file.

namemap (6/15/2000)

The first time this function is used the mq.conf file is parsed and all lines after the first non-comment line are used to supply "from"/"to" mapping tokens. The workarea is compared against each of these "from" values in the order of appearance in the file. If a match is found the workarea is replaced with the corresponding whitespace separated "to" value on the same line in this file. If no match is found, the workarea is set to the default QM queue name, which is specified by the first non-comment string token in the mq.conf file. Note: since the mq_write script transform uses the savearea to designate the MQ queue name to write into, the script function
save:

should be called after this namemap transform.

popen (UNIX only 7/27/2001)

The contents of the workarea is sent used as the argument to a UNIX popen() call. The results of the popen() (it's standard-output) are loaded back into the workArea.

Operator Interaction

System Startup

The UNIX command i_start causes ctksi and (if appropriate) the TOF Mimic (ctkrmc) to start up in Daemon Mode. Since only one ctksi process may connect to an SI Server, i_start will fail to startup any processes if any other Softek processes are running on the execution machine. For convenience, the command i_s will startup ctksi in Interactive Mode and the command i_r will likewise startup any associated TOF Mimic. Interactive Mode is intended for use as a debugging aid, since all messages will appear on the screen as they are output. The use of Interactive Mode is strongly discouraged in production since it bypasses measures to prevent multiple invocation and inadvertant keyboard entry may be used to shut the process down.

Process Status

The UNIX command i_query will print to the "standard output" a status line for any Softek processes currently running on the execution machine.

System Shutdown

The UNIX command i_kill will shutdown any Softek processes currently running on the execution machine. The command i_query is used internally by i_kill to determine which processes must be shutdown.

Daemon Mode

The program automatically detects if it is to be run non-interactively. If the standard-input is closed during program execution or is redirected to /dev/null at startup, ctksi stops reading the standard-input and stops delivering output to the standard-output and standard-error channels.

Interactive Mode

When the program is run in interactive mode, the following commands are available, all operating systems except MacOS and Windows NT require a <CR> to be appended for the command to be accepted:

q or Q quit the program gracefully
eof ctksi enters Daemon Mode (see above).

Interactive Mode is intended for use as a debugging aid, since all messages will appear on the screen as they are output. The use of Interactive Mode is strongly discouraged in production since simple keyboard entry may be used to shut the process down.

Signal Handling

On UNIX operating systems, the program catches SIGINT, SIGHUP, and SIGTERM. When one of these signals is detected, the program exits cleanly, closing sockets and writing shutdown information to the logfile. A Critical Message is also generated. On MacOS, only SIGINT is caught, causing clean program termination. Under MSWindows, both SIGINT and SIGTERM receive this treatment.

On UNIX operating systems, SIGPIPE is caught and its passage is noted in the logfile.

Periodic Cleanup

Disk space is consumed according to the number of errors detected by ctksi. Each day that ctksi is executed, three new files (a Log File, an SI Deal File and a Client Deal File) may be created in the ctksi deals directory. It is suggested that these files are archived and deleted from the drive every few months.

The following UNIX shell script has been used to create archive files in the login directory (these archive files will still need to be moved off the filestore). The script should be run periodically (monthly perhaps) and will archive and delete logfiles and dealfiles older than 60 days. The script may be run in a "cron" job as the "softek" user.

#!/bin/sh
date=`date '+%m%d%Y'`
archfile=`echo softek${date}.tar`
if [ -f ${HOME}/${archfile}.gz ]; then
	echo ${archfile}.gz already exists...skipping overwrite
else
	echo will create ${archfile}.gz
	cd ${HOME}; files=`find . \( -name "*[0-9][0-9][0-9][0-9].log" -o -name "*[0-9][0-9][0-9][0-9].dls" \) -a -mtime +60 -a -print`
	cd ${HOME}; tar cvf ${archfile} ${files}; gzip ${archfile}
	cd ${HOME}; rm ${files}
fi

New Features

The features described from here to the end of this document are in the process of being integrated into this document. Meanwhile, they are described here. The dates in parentheses against some of the headings in this section are "handoff dates" for Cognotec testing and release. The facilities described in this section should be regarded as Beta.

Script Configuration (5/18/2000)

A new directory has been introduced named ~/ctksi/scriptcfg. This directory contains template script files with configuration options that are understood by the C-compiler preprocessor. A Makefile is included which will, when invoked (using make some_target) at a UNIX command prompt, overwrite the existing scripts (files that end in .ctk) in the parent directory (~/ctksi). The existing script files are overwritten with new scripts that implement certain combinations of options that are associated in two places

  1. ~/ctksi/scriptcfg/Makefile
  2. ~/ctksi/scriptcfg/ctkscript.h

The Makefile contains targets that are customer bank names and major feature combinations. It is intended that special feature requests made by customer banks will result in this set of targets being enlarged and associated with the switch combinations necessary to satisfy the bank's requests. For example:

btmjapansi:
make "CUST=JPYSPOTFROMSI NOFIRST4IN508" ${CTKSCRIPTS}

This example illustrates a recursive call to the Makefile that will attempt to build scripts for the target btmjapandsi. The "make" recursion is used to allow the system to substitute some high level customization groupings and associate them with a single target token. The groupings are assigned to the local Makefile variable CUST which defaults to the value STANDARD. The above example illustrates the use of quotation marks to group multiple configuration options into the CUST variable.

The bank, in the above example, wanted a JYP spot rate supplied by the SI feed and did not want the first four characters of the SI client code stored as TOF field 508. These two customer requirements translate into two options loaded into the make variable CUST which are used in the script header file ctkscript.h to cause certain #define variables to be switched on or off (see just below).

The file ctkscript.h contains the cluster of C Preprocessor switches necessary to turn on and off the standard script features and override them with special options. e.g.

#if defined(JPYSPOTFROMSI)
# define VAL_CUR_CODE_LABEL_544 SPEC_JPY_TEXT
# define VAL_CUR_RATE_LABEL_543 SPEC_JPY_USD_SP
# undef SPOT_CURRENCY
# undef SPOT_RATE_FILE
#endif /* defined(JPYSPOTFROMSI) */

#if defined(NOFIRST4IN508)
# undef FIRST_4CHARS_OF_CLIENT_CODE_IN_508
#endif /* defined(NOFIRST4IN508) */

which illustrates how the Makefile variables JPYSPOTFROMSI and NOFIRST4IN508 are associated with a subset of the C Preprocessor switches (macros).

Default Script Configuration Settings

The default settings in ctkscript.h and their meanings are:

#define FIRST_4CHARS_OF_CLIENT_CODE_IN_508

Use the first 4 characters of the SI's CLIENT_CODE field as the TOF TCID (field 508).

#define FILL_541_AND_542
#define FILL_561_AND_562
#define FILL_563_AND_564

Populate fields 541, 542, 561, 562, 563 and 564.

#define WANT_FORWARD_OPTION_DATA_IN_516_527_528

Populate fields 516, 527, and 528 with SI forward option data.

#define AUTOTRADER AUTODL

Use the string AUTODL as the TOF Trader Id in field 504 unless a dealer intervened, in which case populate the field with the SI's INTER_DEALER_NAME field.

#undef BROKERDEALINGCODE510

Use the value in this preprocessor macro as the token to store in TOF field 510. If this macro is not defined (default), populate field 510 with nothing (leave it blank).

#undef INTERDIN507_USERNMIN504

Populate TOF field 507 with the intervening broker's Id and store the SI's USER_NAME in TOF field 504 (switching fields 507 and 504). Off by default.

#undef SPOT_CURRENCY
#undef SPOT_RATE_FILE

By default there is no base currency defined and we do not use a file to get the base currencies spot rate.

#undef VAL_CUR_CODE_LABEL_544

If this macro is given the name of an SI field, that SI field's value would be loaded into TOF field 544 (Base rate currency code). Off by default.

#undef VAL_CUR_RATE_LABEL_543

If this macro is given the name of an SI field, that SI field's value would be loaded into TOF field 543 (base rate against USD). Off by default.

#undef MARKETSPOTRATE_IN_565_AND_566
#undef FARMARKETRATE_IN_565_AND_566

By default we do not load TOF fields 565 or 566.

#undef ONLY_DEALS_WITH_INSTRS_MAY_PASS

If this macro is switched on (default is off), the TOF will not get this deal unless settlement instructions are included. If you enable this macro, you should probably increase (from the default 30 seconds) the value of the environment variable CTK_CACHEDELAY or its associated program argument.

#undef SPOTMKTRATE_IN_560

If this macro is switched on (default is off), FXM_R_T4_RP_NR will be loaded into TOF field 560 for all deal types except Money Market.

#undef APPEND_F_TO_572

If this macro is switched on (default is off), value loaded into TOF field #572 will have an "F" character appended to it if the value of the originating SI field (M_RTS_MRKT_PRFLE_CCY_YEAR) was 365. Please note: This change was made to facilitate deal loading into the "Infinity" Front Office System and will result in a TOF field #572 being an out-of-spec. 4 characters wide instead of the Reuters standard 3 characters wide. (6/15/2000)

#undef LARGEID

If this macro is switched on (default is off), the value loaded into fields 504 and 507 are not restricted to just 6 characters. BEWARE, this does not conform to the TOF spec.

#undef SHORTUSER

If this macro is switched on (default is off), the value loaded into fields 504 is always FXM_USER_SHORTNAME. If the INTERDIN507_USERNMIN504 option is also used, this (SHORTUSER) option is automatically disabled.

#define REPOS_IN_509

Enables the processing of "Repository Funds" information within deals. This will result in the characters "|1" or "|0" being appended to field 509 in the presence resp. absence of a Repository Fund deal. This option is enabled by default.

Bulk Fund Deals (Allocations) and Single Fund Deals (5/18/2000)

When a Bulk Fund (Allocation) message (deal type BF_XX) is received from the SI Server, ctksi first generates a TOF "contra" deal to negate the effects of the original TOF position deal, received earlier from the SI. This "contra" deal is identical to the original "position" deal except that it's BUY/SELL flag is flipped (buy becomes sell and sell becomes buy in TOF field 514). Subsequently, ctksi generates one new TOF deal for each of the indicated funds allocated to by the Fund Manager. Up to 1000 allocation deals plus one "contra" may result from a single Bulk Funds deal. Each of these allocation deals is assigned a TOF field 567 whose first seven (zero padded) digits are the SI deal reference number and the last four (zero padded) digits are the allocation number starting from zero (max 999). The same source reference field value (TOF field 501) used in the original deal is used in the "contra" and all the new allocation deals. In the allocation deals

TOF field 509 is replaced with the "Fund Code"
TOF field 519 is replaced with the "Fixed Amount"
TOF field 545 is replaced with the "Contra Amount".

Updated (10/23/2000)
TOF field 514 is flipped for each allocation where the "Allocation Buy Currency" is the same as the value of the SI "FXM_FIXED_COUNT_CCY_NEAR" field.

Since a Back Office Reference number is assigned when an allocation deal arrives, it is important that TOF dealfile processing (using this assigned ticket number) occurs immediately. Deals which have an associated Back Office Reference number spend minimal time in the TOF cache. They are designated as "write-through" and are processed immediately. Coincidentally, a previously cached deal may becomes current just as a "write-through" deal is written to the cache directory. To prevent this cached (but now current) deal taking the TOF ticket number assigned to the "write-through" deal, a file ~softek/cache/priority.dat is written in the cache directory. This priority.dat file contains the filename of the "write-through" entry. Cachefile processing always starts (or resumes) with the file whose name is contained in this priority.dat file whenever priority.dat exists.

When a Single Fund message (deal type SF_XX) is received from the SI Server, it is not designated as "write-through" (no priority.dat file is generated) however, since there can be no updates to this deal type, it is processed through the cache as soon as possible. There is no "contra" deal, nor are there any special TOF fields, associated with this deal type. Only a single ticket is created which must suffice for both position and settlement information. The Back Office System must have associated settlement details for the specified fund.

TOF Back Office Reference (5/18/2000)

Ctksi provides (TOF only) Back Office Reference information to allow the customer to more easily track Allocation deals that are expanded by ctksi and submitted to the bank's Back Office System. The TOF ticket number that corresponds to the first, non-contra, segment of the allocation (the one that is assigned a TOF contra reference (field 567) ending in 0000) is passed back to the SI Server as the Back Office Reference.

Exception condition processing may result in the allocation deal being represented to ctksi. This is handled correctly. The most recent Back Office Reference is preserved and resent to the SI Server. See here for more details.

Front Office Loading

Ctksi has the ability to perform a primary data load directly into a Front Office System using mechanisms other than the TOF serial-line based feed. This Front Office load is accomplished using two new script files:

siscript.ctk

generated from the script configurator file siscript.h, this script file is called immediately a deal is received by ctksi. This script is responsible for performing the primary load into the Front Office System. If this script generates an error, the deal will not be ACKd to the SI Server and it will be repeated by the SI Server in a few minutes.

ackscript.ctk

generated from the script configurator file ackscript.h, this script file is called after siscript.ctk and is responsible for reading any response to a deal presented by siscript.ctk. The response does not have to come via the same mechanism that supplied the deal to the Front Office (e.g. a deal, supplied to the Front Office via a direct database write could result in an MQ confirmation message being read by this script and passed back to the SI Server). This script is responsible for decoding any Back Office Reference and related information sent from the Bank's Front Office System and passing it on to the SI Server. ***Although the script is called at the appropriate time, it currently does nothing...as of 8/29/00***.

Currently the following direct deal-loading mechanisms have been implemented at customer sites. The whole range of deal loading mechanisms are available as currently provided by Softek's scripting system including (but not limited to) the following:

IBM MQ (5/28/2000)

Two forms of MQ data loading have been provided, but, using the scripting system, any record format may be crafted. Using this mechanism, the Back Office Reference will be available in a future release. The two provided record formats are:

  1. SI raw data format. A record is sent to MQ in exactly the format it is received from the SI TCP-IP stream except without the framing STX and ETX characters. To enable this "raw" format, by configuring the appropriate script, type:

    cd ~softek/ctksi/scriptcfg; make mq

    when logged in as softek on the Cognotec SPARCstation.

  2. Labeled data format. The SI record is provided to MQ with the associated SI field labels. Each field has the appropriate SI label and a colon (':') prepended to it and each field is terminated by an ASCII NL (newline) character (dec 10, hex 0x0A). Note: The last field also ends with a NL character. The customer bank should be aware that the the field labels change with the deal type. Ctksi uses the field label definition set stored in the file ~softek/ctksi/dcf.ini on the Cognotec SPARCstation. To enable this "labeled" format, by configuring the appropriate script, type:

    cd ~softek/ctksi/scriptcfg; make mqlabeled

    when logged in as softek on the Cognotec SPARCstation.

    MQ Configuration File

    Ctksi uses a file named mq.conf to enter MQ configuration details. This file contains two white-space separated strings on the same line, the second of which is optional.

    1. QueueName
    2. QueueManagerName

    If the QueueManagerName is omitted, system defaults are used for this value. It is a fatal program error if MQ deal loading is attempted and the mq.conf file cannot be found or parsed.

    From 6/15/2000, any line whose first non-whitespace character is '#' is treated as a comment line. The first non-comment line is decoded as described above. All subsequent non-comment lines are used to supply name mapping values for the namemap script transform, each such "mapping" line comprising two whitespace separated tokens, the first being the "from" name and the second being the "to" name. Since the output from the namemap script transform is used to drive the mq_write script transform (which uses the savearea to denote the queue name to be used) the script function

    1. save:

    should be called after this namemap transform and before mq_write is called..

The MQ deal loader script will repeatedly attempt to place the most recently available deal record from the SI stream on the designated MQ queue until a successful return is received from the MQPUT() function. Once a successful return value is received from this function, ctksi will allow the SI deal server to move on to the next available deal. It may take up to two minutes (while SI TCP-IP comms are reset) for a failed deal load to be repeated. This mode of operation obviates the need for the bank's client application to be running all the time and allows MQ to act as the reliable delivery mechanism for SI-originated deals.

The ctksi MQ loader holds an MQ queue manager connection open (if possible) once a connection is established. If the queue manager connection is dropped for any reason, ctksi attempts to reestablish the connection with the next deal that is received from the SI TCP-IP stream. Writing to an MQ queue is accomplished using the mq_write script transform.

For your convenience an environment variable MQSERVER has been included (but commented out) in the ~softek/ctksi/set_my_env file. This variable should be configured according to IBM's instructions, to allow a connection to the bank's remote MQ server that is hosting the QueueName selected in the mq.conf configuration file.

Any MQ errors that are detected are reported in the ctksi logfile together with the IBM MQ reason code. If an MQ client is loaded on your browsing system this link (UNIX) may work to get you to a description of IBM's reason codes.

Sybase (5/18/2000)

Any record structure into any tables with any (reasonable) data transformations. See here for more details.

TCP-IP Stream Dump/Replay (5/18/2000)

The environment variables

CTK_TCPMESSAGES and
CTK_TCPMSGDUMP

each enable a new feature.

If CTK_TCPMSGDUMP is set to a writeable filename, ctksi will append a timestamped record of all TCP-IP data transmissions to the designated file.

If CTK_TCPMESSAGES is set to a readable filename, ctksi will use the contents of this file in place of the designated TCP-IP stream and will replay the input portion of this file as if it came from the TCP-IP stream. Records will be received into ctksi with the same tempo as they were stored in the file. Please note that this is a fairly dumb replay mechanism. The original record order (including any repeats) will be faithfully replayed into ctksi even if different server behavior would subsequently be appropriate.

The output from CTK_TCPMSGDUMP is usable as input into CTK_TCPMESSAGES. The easiest way to create a properly formatted CTK_TCPMESSAGES file is to run (or have someone else run) ctksi with a value set in the CTK_TCPMSGDUMP environment variable.

Ctksi will automatically quit when an available data stream from a CTK_TCPMESSAGES file has been exhausted. It is often useful to wait for a couple of PING messages to appear in this file when it is created (or edit them in by hand) for the replay mechanism to have enough time to flush the ctksi TOF deal cache.

The file ~softek/ctksi/set_my_env contains a commented-out template for these two environment variables.

TCP-IP Full Reset on Error (4/8/2000)

For some, as yet unknown, reason, if the SI TCP-IP server suffers a power failure, ctksi's select() calls return no error condition to inform us of the communication outage. To circumvent this problem, versions of ctksi after the above availability date, implement an active "heartbeat" check. If an SI heartbeat has not been received within a reasonable time (usually two minutes), a complete reset of ctksi's TCP-IP subsystem is performed and a new connection is attempted with fresh sockets. This reset is repeated about every two minutes until a heartbeat from the SI server is reestablished.

TCP-IP Failover (5/18/2000)

A facility has been implemented that works with Cognotec's strategy for automatic TCP-IP failure recovery. Every time the TCP-IP channel is reset (either automatically, perhaps due to a heartbeat failure, or manually by restarting the interface) the environment variable CTK_VIRTUALHOSTNAME is looked up and, if it exists, it is used instead of the IP program argument for Cognotec server connects. To disable this feature, either comment out the CTK_VIRTUALHOSTNAME entry in the ~softek/bin/set_my_env file or set the value of this variable to an nonexistent host (one not present in the Operating System's hosts database).

Kill Protection (5/18/2000)

To further increase the fail-safe nature of this interface and ensure that no deals are lost, processing of the TOF dls file of multipart allocation deals is performed in two stages:

  1. all changes are made to a temporary file in the deals directory
  2. the contents of the temporary file is appended to the TOF deal file and then the temporary file is deleted.

If, when the interface starts up, if TOF deal-file writing was interrupted during the processing of a multipart deal, the cache file that the TOF deal-file creation scripts were being run against will still be present and processing will begin anew. The old temporary file is deleted and a new one is created, eventually, processing will complete and both the temporary file and the cachefile will be deleted. During this, recovery scenario, the originating SI deal has been acknowledged and the only record of the deal is within the ctksi SPARCstation.

If, for any reason, the SI sends a deal that was a duplicate of the deal most recently processed by ctksi, this is detected and an ACK is sent back to the SI with no further processing on the part of ctksi. If this response (ACK) warrants a "Back Office Reference", the reference is obtained from the file ~softek/cache/lastbor.dat which is written by ctksi whenever a Back Office Reference number is generated.

When ctksi starts up, any currently cached but time-expired TOF deals are processed into the TOF dealfile. If a file called priority.dat exists in the cache directory (~softek/ctksi/cache) the content of this file is used as the cachefile to process first. The presence of this file at ctksi startup may indicate that a prior run of ctksi was interrupted before a Back Office Reference associated deal was fully processed. Ctksi is designed to recover from this scenario.